Should Unit Tests Test Interoperations

We say UnitTests are testing one specific unit (maybe a class) independently. MockObjects and stubs are all about it. We say, however, UnitTests enable us to carry on CollectiveCodeOwnership because the tests will tell you whether you broke anything. If UnitTests are independent tests of the parts, they can't guarantee you whether you broke the whole system or not -- if you've changed your unit and its test code and they work well, the whole thousands of independent UnitTests will tell you OK, even if some other units, which uses MockObjects to gain independency from your unit, might break nicely. Do we need massive AcceptanceTests here?

Is it right to test interoperations between units in a UnitTest?

Any comments from experiences would be appreciated.

On a current project implemented mostly in C, there are just a few natural boundaries where MockObjects and stubs can be easily applied. For example at the network interface, and the disk device interface. These interfaces already make use of polymorphism, so MockObjects are a natural fit. Above the interface, tests go in layers. First the objects closest to the interface. Then the objects which use those objects, etc. This has worked well.

Of course part of the difficulty has been the use of C rather than C++. In C I have to find a function-pointer interface or something similar before I can apply MockObjects. In C++ I have a choice of virtual function interfaces or templates, so it is easier to test in isolation.

Actually it is just as easy to test thing in isolation in C as any other language. You build a calling program for a specific source file and build stub source files as needed. You do not have to change any of the header files.

Is this page asking whether (XP?) needs to employ "massive acceptance testing", or is it asking whether a unit test (by definition) should test more than a unit in isolation? Tests of interoperations are usually called "integration tests", while "acceptance testing" usually refers to a formal activity in which the customer ends up signing off. What, really, is the question?

The question is,

In XP, it is encouraged that unit tests are independent, and test one single class taken out. Yet XPers say CollectiveCodeOwnership works thanks to unit tests. It sounds a bit contradictory. So, is it desirable to test interoperations between classes in UnitTests in order to support CollectiveCodeOwnership well? How do experienced XPers guarantee the whole program's correctness with CollectiveCodeOwnership, if not testing the interoperations among units?

See http://www.xpdeveloper.com/cgi-bin/wiki.cgi?DoesUnitTestingMissTheBigBugs and http://www.xpdeveloper.com/cgi-bin/wiki.cgi?CanSmallFunctionalTestsReplaceUnitTests, where similar questions are raised.

(Not claiming to be a true XPer...) Can the interactions between classes "break?" If you only test each class in absolute isolation, have you done TestEverythingThatCouldPossiblyBreak?

When Spiderman class has SmallGadget? objects, in a UnitTest for the Spiderman class, we'd try to test it in isolation making some mock objects for the SmallGadget? objects. Now, A pair has changed the signature of SmallGadget? class and they run all UnitTests getting OK -- since everything works in isolation. However the system fails to work right. So unit testing every class in isolation can't be a guarantee in CollectiveCodeOwnership.

I don't understand this example. If you're using a mock object in place of some "real" instance of SmallGadget?, how can you change the signature of SmallGadget? without the mock object then failing to compile? Shouldn't the mock and the "real" objects both implement the same interface? If they don't, what are you really testing, then?

Not all "units" in a system are the same size. The larger units are composed of smaller units. Does this answer the question?

Unless the larger unit is tested interdependently with real "small units" not with MockObjects in isolation.

So what is a "unit"? And how many levels of "units" need to be tested?

Whoa. I believe the premise here ("larger units...composed of smaller units") is misleading and incorrect in a practical sense. A "unit" is a bunch of code found in one place. A "UnitTest" is a devised means of exercising that code in isolation from other code that you wrote, more precisely from other code that you have to test. If class A and class B are units, and class A's methods calls methods in class B, then it is incorrect to assert that the class A unit is a "larger" unit by dint of the fact that it "includes" class B. No, that's not what unit testing means. The size of a unit does not depend on the (recursive) size of the code it calls. The aim of unit testing is to exercise the code of the unit and as little other code as possible. In a "pure" unit test, the only code other than the unit being tested is the test driver code itself. Only "terminal" classes can be tested this way, for obvious reasons. For "non-terminal" classes, the way to approach pure unit testing is to scaffold the hell out of it (see MockObject). "Unit testing" that uses real objects (ones you developed, and therefore ones you have to test) as the callees of the target object is really integration testing. The level of faith in the "scaffolding", whether it be composed of MockObjects or real ones, is what really determines what kind of testing you're doing.

I don't speak for XP, for those who ask the question on this page. In my opinion, any system that hasn't been integration tested isn't going to work, because the work of devising and honoring contracts is as error prone as any other. -- WaldenMathews

Walden, unit tests in the XP sense are all the tests written by the programmers. By this definition "Unit testing" that uses real objects (ones you developed, and therefore ones you have to test) as the callees of the target object... is really unit testing.

If you think this begs the question, please consider: why should the term "unit" only be applicable to a class? (I was going to link to UnitInUnitTestIsntTheUnitYouAreThinkingOf?, but that page doesn't really address the term "unit". Oh well.) If we have a group of three classes which are supposed to do something that's well defined, why shouldn't we call them a "unit" and test them in isolation... isolation from the rest of the system, that is? That doesn't mean we shouldn't also test the individual classes in isolation - using MockObjects or whatever. -- LaurentBossavit

You can use whatever name you want, although I'd prefer that software developers stop redefining the software lexicon with each new generation: there's no need for that.

Naming issues aside, there is value in testing things apart, and there is value in testing things together, I feel sure we're agreed.

In a broader sense, the questioner is asking what sort of testing framework is required in order that CodeOwnership can be avoided. Interestingly, we are focused on testing in answering that, but the real answer is simply that if the specification is written down, then it's not in any one person's head, and no one "owns" it. Coded tests, unit or whatever, are an example of "specifications written down". You're right about UnitInUnitTestIsntTheUnitYouAreThinkingOf?. It should be TestInUnitTestIsntTheTestYouAreThinkingOf? instead, since that's what the page really discovers. -- WaldenMathews

I think the above discussion highlights my confusion about unit tests in the XP meaning. It seems every writer redefines the term to mean whatever he chooses it to mean at the time. Let me echo the question written above, what is a unit for a unit test? -- WayneMack

I'd say that we have things and interactions between things. So you test the things, and you test the interactions (they are both "units"). If the code follows LawOfDemeter, then interactions are always between pairs of classes, so the scope does not increase too much. You don't have to test interactions between every possible pair of objects because you don't have a fully-connected graph: each object only interacts with a small number of other objects.

Consider testing a visitor interaction. What do you need to test to be sure that a specific concrete visitor (V) will operate correctly with a specific concrete element (E). You could implement a complex test, that V can visit E. But is there a better way? Is it sufficient to test three things: E is a valid element; V is a valid visitor; A valid visitor can visit a valid element. (Of course, static type checking will do a lot of this for you). -- DaveWhipp

I've adopted the idea of SubsystemTests when faced with this problem. First there are UnitTests; these test each class in isolation as much as possible. When I see a question as to whether the interactions of various units are functioning correctly I create a SubsystemTest that tests a group of units. I find this distinction useful because it clarifies what UnitTests should be doing (testing a unit in isolation) and still allows us to test unit interoperations. -- BrianRobinson

See ShouldUnitTestsTestInternalFunctions

CategoryTesting