Recently I have been discussing this question of ‘What can you expect static analysis to find compared to other forms of testing?’ with some of my colleagues and customers. It is obviously an interesting question to warrant lengthy discussions andresearch papers. Since every software project is different depending on the size, the development team, the purpose of the software, and various other reasons, it’s understandably not a simple answer.
In its simplest form, one can think of static analysis as a really smart friend that tests my code and finds mistakes I have made and tells me about it, reports defects. So to answer the question, one approach would be to look at what kind of defects good static analysis finds that are not possible using other forms of testing.
Systemic defects versus idiosyncratic errors
Every software project, regardless of its size, the size of the development team working on it, the quality and experience of the developers, or the function that the software is intended to perform, typically is expected to have a certain number of software defects. Coding errors such as using a pointer before it has been initialized, or mistakes such as leaving a large chunk of allocated memory on the heap unreleased are typical mistakes in all software development projects. These systemic mistakes exist regardless of the best intentions of the developers, and good static analysis finds them. Taking the analogy of investing, you invest in the stock market knowing the systemic risk that such a financial system has. The systemic risk that investing in a stock has doesn’t depend on the individual company, it’s carried within any stock trading in the market. That risk is different than the idiosyncratic risk which is specific to a particular company and depends on the management of that company, the business strategy decisions, the market it is in, etc. This risk is similar to idiosyncratic bugs that a software project might have. These are functional issues, scale issues, and design issues that will need the strengths of unit testing, functional testing, and automated QA testing to uncover.
A memory corruption or a leaked system resource is an invisible defect. The execution is usually way past the place in the program where the defect is when it manifests into a visible error. Testing for such defects using traditional testing is like trying to find the fatal disease that caused the death by figuring out the time of the death. By looking for the defect like a null pointer dereference that leads to a visible error like a program crash, static analysis is testing code for invisible defects.
Defects on uncommonly executed paths
Coding defects exist in uncommonly executed parts of the program and error handling code. In fact, most developers focus less on the code in error handling routines. Error handling routines can be complicated when it has to undo the state that the system has gotten into. Most traditional testing tends to focus on creating and executing tests to evaluate the commonly executed paths. In addition, it is hard to create tests to test error handling code. Static analysis doesn’t differentiate between commonly executed paths and uncommonly executed paths when testing code.
So in essence, to answer the question of ‘what can you expect static analysis to find?’ it’s probably best to understand what types of defects are not tested and found by any other way.