How do you measure software quality?
It’s a simple question with a not-so-simple answer. For instance, depending on who you ask, software quality depends on one or more of the following:
- The metric or combination of metrics you consider as accurate indicators of quality
- Whether the software is pre-release or post-release
- The complexity of the code base
- The number of issues reported in the field, and the trend
- Even if there few visible bugs, the amount of technical debt built in
From our experience with over 1100 customers, who are creators of some of the most fascinating products and software of our generation, defect density is consistently ranked as a good indicator of software quality, especially pre-release.
So why defect density? I believe there are a few strong reasons for this:
- With the strengths of modern static analysis you can now automatically, quickly, accurately, and consistently analyze your entire codebase, regardless of which stage of development you are in. You don’t have to be dependent on “feature completeness” to understand your defect density.
- Finding out your defect density does not require you to change anything in the development workflow – static analysis is typically easy to integrate in your build.
- Research shows the pre-release defect density as reported by static analysis is a very accurate indicator of software quality
The obvious question is then, what is a good number to aim for?
Defect density is typically measure per thousand lines of code (KLOC). The Coverity SCAN team has done a study over the last few years on open source projects, and estimate that 1.0 defects/KLOC typically represents a good quality project – a project that is actively evaluating the reported defects, and making fixes. There are several books that have estimates of software quality based on defect density – e.g. the book ‘Code Complete’ by Steve McConnell has a section on error expectations using defect density that provides an example for Microsoft Applications which have about 10 – 20 defects/KLOC during in-house testing and 0.5/KLOC in released product.
There is no silver bullet for measuring software quality, especially when in development pre-release. But there are certainly good indicators, and defect density is a strong indicator that you can use to guide your software development ship in the right direction.