Unit Tests Reconsidered

To improve focus, the discussion of "XP does not use the StandardDefinitionOfUnitTests" moved to XpVsStandardDefinitionOfUnitTest.

[UnitTestIsolation] The whole point of UnitTests is to reduce the scope of the system under test to a small subset that can be tested in isolation. Thus, by the ScientificMethod, we reduce the number of variables that may affect the results so we can derive useful information on whether the unit is actually correct. [Discussion in UnitTestIsolation]

Furthermore, UnitTests are often used in a sort of "DesignByContract" (although that's once again using the terms wrong), in order to ensure that an interface conforms to its predefined specification. These sorts of UnitTests aim to be complete, testing boundary, failure, average, and interesting cases. Thus, if the implementation coded to this reference test suite passes, we can be assured it conforms to the specification.

Taking this even further, we can code a constraint harness for a unit after we've written it to ensure functional equivalence before and after refactoring. See WhatIsRefactoring to understand how functional equivalence is important to refactoring.

On this page, I consider some arguments for UnitTests and rebut them. The point is to show that UnitTests are not the right solution in every case - and to be "eXtremely" anal about showing that. On the other hand, it should be clear by now that they do serve a purpose - like the Java Compliance Tests, to isolate bugs to a given unit, and to force functional equivalence during tricky refactoring, and definitely others that should be obvious to every software engineer by now. -- SunirShah

In the future, we (i.e. I) should make this discussion less anal, and more constructive. It's more valuable to demonstrate not just UnitTests can fail, but where and how, and what to do about that. After all, one of the main points was that UnitTests are not a SilverBullet, so what else can we do? -- SunirShah

Most of the discussion below is anecdotal. Please upgrade/downgrade statements as facts are discovered. A good resource is CodeComplete, chapter 25, which has a lot of hard data.

Discussion moved to...

IsXpUnitTestingBest -- Are XP's testing practices "the best," or are there other testing approaches that produce better results for less cost?
DoBothUnitAndAcceptanceTests. (AcceptanceTests, by themselves, are not sufficient.)
Should we be TestingInterfaces?
UnitTestsTellYouWhenYoureDone.
Is it desirable or required to do only OneUnitTestAtaTime? [Was "TestFirstProgramming forces you to work on only one thing at a time."]
StrictTypingIsaTest -- one that the compiler executes.
CompleteCoverageIsExpensive.
ExtremeProgrammingChallengeFourteenSplit may be an example of the cost of UnitTesting being too high; it took 16 hours to test.
LukeGorrie got TestInfected and wrote some testing functions -- filed under LispUnit. (It's not an "StarUnit" library, but it might inspire someone! ;-) (Actually I'm not TestInfected, and IIRC the code didn't work as claimed! *shame* -- LukeGorrie)

[If you're looking for some part of the discussion that used to be on this page, the bulk of it moved to XpVsStandardDefinitionOfUnitTest. Some comments moved to IfXpIsntWorkingYoureNotDoingXp. Other, more independent lines of discussion were moved to the pages in the bullet points above; try the one or two that best match what was being discussed.]

[I think that IsXpUnitTestingBest may be the best place for the main thrust of the UnitTestsReconsidered argument. -- JeffGrigg]

CategoryTesting