Unit Tests That Dont Break

I have recently started using TestFirstDesign (CodeUnitTestFirst). I often find myself in the following situation: I need to implement some new function. First, I write a fairly simple UnitTest to test the most basic case handled by the new function. I run the test, and it obviously breaks. Then I implement the functionality (handling all the required cases) and run the test again. This time, it passes.

But I don't have the incentive to write tests covering all the aspects of the new functionality. I suppose that they should pass the first time they're run. Therefore, I feel that writing them is a waste of time that should better be spent on new functionality.

(Suppose that the function in question consists of 20 lines of C++ and has 5 separate cases that can be tested.)

What am I doing wrong?

Writing tests for all cases in advance obviously violates WriteJustOneTest.
Implementing the functionality in many small steps will take much longer than writing the whole function at once. Moreover, it is often impossible to write the code in such a way that it will satisfy only one of the testable cases and will not satisfy another.
Should I overcome my reluctance and write UnitTestsThatDontBreak?

-- DmitryJemerov

The solution here is:

work hard to ensure the test fails for the correct reason
write the stupidest simplest possible code to pass the test, even if that code could not satisfy any other test
refactor
write another test that tries to "force out the lie"

Practice this technique, to cover situations where you don't know the code lies.

-- PhlIp

You should either write more than one test at at time (reluctantly), or really, really, really write one aspect of the functionality at a time.

Bear in mind, however, that it is okay to write a test, and see that it already works. That's not the final step, however. You still need to make sure that the code is as simple as possible, via refactoring and OnceAndOnlyOnce, etc.

Just don't go out and write a bunch of functionality that's clearly beyond the minimum required for your test case. If you want to give the example you have in mind, we can work together and see if we can find a good way to do it test-first, if you like. -- RobertWatkins

That's why members of the ExtremeProgramming community DoTheSimplestThingThatCouldPossiblyWork: An XP team wouldn't write code "handling all the required cases" in the first pass; they'd DoTheSimplestThingThatCouldPossiblyWork. Then the next test would force them to handle more cases, etc.

You have to make a choice to do one of the following:

Write tests for all the functionality you intend to implement, then run them and write the code.
Write one test, DoTheSimplestThingThatCouldPossiblyWork, and then write another test. You'll know that you've succeeded when the 2nd test fails.

Read the XP books: The first implementation of the "Total" method on a collection-type class typically returns zero. "But that's not right!" you say. Well, at that point in the implementation we haven't added the functionality needed to actually put anything into the collection, so the answer at that point can only correctly be zero. Later, when we add the ability to put elements into the collection, we'll have to "rewrite" the Total method. -- JeffGrigg

Keep in mind, also, that you don't really write UnitTests for today -- you write them for tomorrow. Imagine yourself six months from now, revisiting the class you wrote today. Imagine that you have a major internal refactoring you'd like to do. You probably won't be able to remember every detail and every behavior of your class. Without UnitTests, you'd have to worry about breaking something and not noticing right away. With lots of UnitTests, you know when you break something.

Also, the tests are a little bit of you that gets to travel around with your code and watch that everyone else that touches it keeps your thoughts in mind.

Also, if you write the tests after the code needed to pass them, how do you know that you've throughly tested all the functionality? It's very easy to sluff off and say "it's tested enough" -- even though a later refactoring might break it, causing clients to fail, but not causing this class' inadequate UnitTests to fail.

Implementing the functionality in many small steps will take much longer than writing the whole function at once. Moreover, it is often impossible to write the code in such a way that it will satisfy only one of the testable cases and will not satisfy another.

This turns out not to be the case. Implementing the functionality CORRECTLY in many small steps, test first, will not take longer than writing the whole function at once.

No, in general it is difficult to say whether it will be the case. For a normal application, I can write 80% of the methods without too many problems because they are very short (I too like ShortMethods). I can also write a linked list all at once without trouble. On the other hand, when I recently had to write an efficient algorithm to map from a whole bunch of different color systems to whole bunch of other color systems, I found the best way was one if statement at a time. Otherwise, there would have been too many bugs and I would never have gotten it done. The crossing point is your level of confusion. The second you start feeling confused, you've bitten off more than you can chew. -- SunirShah

I don't understand why this comment supports the idea that writing things in small steps will take "much longer" than writing them in big chunks.

Don't be so binary. It's not one or the other. Sometimes it takes longer, sometimes it's faster. Sometimes it depends on how sleepy you are, or what you had for breakfast (seriously--inability to focus requires extra tools to keep moving forward). Remember, XP aims for no risk, but that doesn't mean it is efficient. It just means that it will eventually produce. TestFirstProgramming means that everything will be covered. It forces the programmers to be disciplined. However, if you are God's gift to CowboyCoding, this may not be necessary. By the way, this is why XP is generally better than WaterFall, which has > 90% failure rate. Then again, I think everything is better than Waterfall, but that's my opinion. --ss

I have experienced that test-first-tiny-steps never seems substantially slower than taking bigger bites. You seemed to be saying that it sometimes is substantially slower, but the details didn't seem to support that notion. The next rant didn't improve my understanding of the basis of your position, or even of what your position might be. Are tiny steps sometimes substantially slower? How do you know? How might we know in advance which way to go?

I suppose it depends how experienced you are in writing what you are writing. If you know what to do, there isn't much point in going on a journey of self-discovery. I know in advance by watching myself. If I get confused, I stop and write another, simpler test that I know I can pass.

Actually, I don't think it does depend on how experienced you are - it depends on how experienced the people who follow you are. Developers hate documentation (as a rule), but it is another necessary evil. Perhaps they could test later if they want to change something, but you know better now about what it is meant to do. Success in development is like success in life - it's only really worth it if you've left a legacy. No tests may well mean your legacy is ended when they completely replace the code. --Stephen Kestle

What about this scenario: I write a unit test "If input contains non-numeric characters, then throw an exception", and then I write code that passes that test. I look at my next task card, and it says "If input is blank, then throw an exception"."Hmmm, " I think, "the existing implementation uses the toNumber() method, which will throw an exception on empty input just like it does for non-numeric characters, so I think the existing code will pass the new test." I write a unit test to verify it, and it passes without any code changes. I consider the task finished, and go on to the next one.

Have I done something wrong in this scenario? Should I have not written that second non-failing unit test? Should I have rewritten the implementation so that it would fail the second unit test, and then changed it back?

Maybe just refactor your tests to "If the input is anything but 1 or more numeric characters, then throw an exception". In this case, instead of writing your tests in a negative manner (defining what is bad), you could write one test in a positive manner (define what is good, and reject anything else).

OK, integrating the new test into the unit test associated with the last code change makes sense. But the refactored tests would still pass on the first try--I wouldn't see a failure. Rewriting UnitTestsThatDontBreak as new UnitTestsThatDontBreak seems to be the same as just writing a new non-failing unit test.

Above, RobertWatkins wrote "it is okay to write a test, and see that it already works." I agree with that and that is how I do my work, but I'm wondering if this is counter to TestFirstDesign, TestFirstProgramming, or XP philosophy. Some of the test-related content in the wiki implies that if you ever write a unit test that doesn't fail, you have either (a) written a test you need not have written, or (b) taken too large a step. In the scenario I gave, the second task was fulfilled "accidentally" without considering it or doing any unnecessary coding during implementation of the first task. If such "happy accidents" occur often, is it a sign that I am taking too-big steps, that I am not writing simple code, or that I am over-testing? And what is the right thing to do in this situation: leave the test as is, remove it, change the implementation to force a failure, ...?

I find I often comment out a line which should cause a break, in order to exercise the test. Not always, but often; mainly when I'm curious about whether or not the fact/constraint I specified in the test is actually related to the model/implementation I'm creating.

There is nothing wrong with writing a test that does not fail. This usually occurs when using an existing library or class that may already do more than one absolutely needs. If the design considers the two test cases different, keep the test to ensure any future changes do not break this condition. The rule in test first design is not to write code that is more complex than needed to handle the current test. Bear in mind, that the code will do something in response to the next condition tested, one has just not defined what the expected outcome should be. When one writes the next test, he defines the expected outcome; if the current simplest possible code happens to provide the desired result, it is merely a lucky break. Enjoy the good luck, check in the new test, and go onto the next item to be addressed.

I don't understand why this comment supports the idea that writing things in small steps will take "much longer" than writing them in big chunks.

Don't you find that there is a roughly-constant overhead associated with writing "a thing", taking "a step"?

I have experienced that test-first-tiny-steps never seems substantially slower than taking bigger bites.

Perhaps you would be so kind as to let me shift the emphasis from never to seems. I happen to be pretty sure, based on what I know of myself, that I would find the approach excruciatingly slow, whether it actually turned out to be or not. Thus I couldn't possibly force myself to do things this way.

Well, at that point in the implementation we haven't added the functionality needed to actually put anything into the collection, so the answer at that point can only correctly be zero. Later, when we add the ability to put elements into the collection, we'll have to "rewrite" the Total method.

I can't see why I would implement Total before add, since I know I will have to reimplement it almost immediately. Similarly, having implemented add, might I not reasonably find the implementation of Total so trivial that it takes less time (especially given that I avoid the overhead time of the context switch from write<->test) than testing add without Total?

HelpMeToUnderstand, how do the extremists follow these kinds of policies and not feel like they are moving more slowly than they might? -- KarlKnechtel

I agree with Karl that very small steps can be intolerable for some folks. I find it particulary hard when pairing with someone who likes smaller steps, since I have to follow them jumping back and forth from test class to implementation class and running the test every minute or so - it completely derails my train of thought. -- DavidAllsopp

CategoryTesting