In some of the other discussion on UnitTestsReconsidered, SunirShah claims that UnitTests become mud. Consider the following points, stolen from CodeComplete, chapter 25:
- Complete domain verification is impossible. Often the domain is too large to enumerate, say larger than the number of atoms in the universe. Imagine testing strstr() on every possible pair of strings.
- "Eighty percent of the errors are found in 20 percent of a project's routines . . . Fifty percent of the errors are found in 10 percent of a project's routines." p. 606
- "Regardless of the exact proportion of the cost contributed by highly defective routines, highly defective routines are extremely expensive. In the 1960s, IBM performed a study of its OS/360 operating system and found that errors were not distributed evenly across all routines but were concentrated into a few. Those error-prone routines were found to be "the most expensive entities in programming" [Jones 1986] They contained as many as 50 defects per 1000 lines of code, and fixing them often cost 10 times what it took to develop the whole system." p. 607
- "The implication of avoiding troublesome routines for maintenance is equally clear. Maintenance activities should be focused on identifying, redesigning, and rewriting from the ground up those routines that have been identified as error-prone." p. 607. See WhatIsReworking.
- "Many errors are outside the domain of construction. Researchers conducting a series of 97 interviews found that the three most common sources of errors were thin application-domain knowledge, fluctuating and conflicting requirements, and communication and coordination breakdown. [Curtis, Krasner, and Iscoe 1988]" p. 608 Ed note: this is to say that errors don't come primarily from functional failure, but from failure to understand the system; thus the tests themselves will be wrong, especially in TestFirstProgramming. Defects in tests have to be fixed as well. --ss
- "Test cases are often more likely to contain errors than the code being tested. [Weiland 1983, Jones 1986]" p. 612. (See BugsInTheTests.)
References
Curtis, Bill, H. Krasner, and N. Iscoe. "A Field Study of the Software Design Process for Large Systems." Communications of the ACM 31, no. 11 (November): 1268-87. (1988)
Jones, Capers. Programming Productivity. New York: McGraw-Hill. (1986)
McConnell, Steve. Code Complete. Redmond, WA: Microsoft Press. p.612 (1993)
Weiland, Richard J. The Programmer's Craft: Program Construction Computer Architecture, and Data Management. Reston, VA: Reston Publishing. (1983)
By the way, not testing is also expensive. There are two BalancingForces at play, and they resolve at some point in the middle. -- SunirShah.
See also TestEverythingThatCouldPossiblyBreak - I don't think it attempts to define that point, but it has more debate on exactly where to put it. -- DanBarlow
Complete Coverage Is Expensive