Unit Testing Costs Benefits
Unit Testing Costs BenefitsPairProgrammingCostsBenefits gives very convincing evidence that PairProgramming is where it's at. But is there any evidence besides thought experiments (or perhaps including thought experiments) that UnitTesting is definitely a good investment?
The answer can be found sarcastically in "BigBangTesting". More specifically, when you have a bug in a class, that bug will manifest later as incorrect behavior. That manifestation will be noticed by a human. Humans are much more costly than computers, so I'd prefer to get my notice:
But these are all reasons based on intuition. Is there any evidence that CommonSense is correct? For example, can we find 2 similar projects, one of which used pervasive unit testing; the other relied on compiler errors and acceptance tests. Which of these
Over the past 9 months I rewrote an application that had become a BigBallOfMud. Using UnitTests made the program's design better, and ensured that there would be fewer bugs. The new program had the same functionality in 1/10 the LinesOfCode. A reduction from 215,000 lines to 22,000 lines. I am sure that I would not have been able to accomplish this without UnitTests. Our customer also claims that the application looks and performs better. The original took 3 people a year, and this took just me 9 months. The rate of bug reports has dropped off by more than 90%.
It is very hard to talk about benefits of UnitTesting without knowing the answer to the following question: How many defects per 1000 lines of code remain in the code covered by UnitTests when it is put into production?
Does anyone have some statistics regarding this? -- DmitryJemerov
Why does this statistic help? Not all lines of code are equal nor are all "defects" equal and there are a lot of things which may or may not be covered under the term "defects."
The PersonalSoftwareProcess book has data on programmer defect rates. One of the extreme books has a conference paper showing when there's cost-benefit of refactoring (implying unit tests), from a model of software as disease propagation. Can't remember the title...
I finally found a study comparing a project with and without unit testing. It was done by Monster Consulting in Utah. Interestingly, the programmers were a group of 5 engineers, four of whom were practicing pair programming. They had tried unit testing a year ago but gave up on it because "the architecture of their software did not lend itself to unit testing." Monster Consulting had them write a toy program, then trained them on testing, and had them write a second toy program. Here's the study: http://www.agilealliance.com/articles/articles/IntegratingUnitTesting.pdf
I will summarize the results and comment on them. Each "result" compares the pre-training toy program with the post-training toy program. The numbers I am typing in are my rough estimates from reading off their charts.
Total lines of code, including testing code, doubled on average
Development time in minutes
Where I see the value of testing is on longer, larger, more complex projects. My personal rough estimate is that testing becomes more efficient than not testing once your project is at least three days' work. Other people would say it has to be at least a week long. I would love to see empirical results on this. But I bet there is a point at which a project has so many interacting parts that having tests saves you major debugging time, which would mean you spend overall less time coding. Also, when a project is big enough to include refactorings, the size of the code base should quickly drop. On these larger projects, you don't have to choose between getting done fast without testing and slowly developing quality code with testing. Instead, on large projects unit testing both increases quality and drops development time.
(By the way, I skipped their results on productivity, measured as number of lines of code per hour, because I care only about getting features done, not on generating more lines of code per hour.)
when a project is big enough to include refactorings, the size of the code base should quickly drop
Not to mention that the unit tests save you a lot of time by telling you right away if your refactorings broke anything. Of course, they are not 100% accurate at this, but they're much much better than no tests at all.
EditText of this page
(last edited October 15, 2010)
or FindPage with title or text search