Refactoring Unit Tests Is Expensive

Extensively using CRC Cards is still making design decisions before you have enough information to make them. It's hard to hold a whole system in your head. Some of us (like me) find it barely possible to hold a single class in our heads. Taking the refactoring way out is expensive because refactoring UnitTests is very expensive. In fact, it's such a drag force that most people don't do them. So, to summarize, DesignByContract is difficult because the contract is far too static and assumes that you can describe the system before you're done writing it, which we all know is false. CRC Cards don't change this fact either.

Can you give an example where this is true?

And maybe offer a conclusion?

Well, when I was playing with Perl and trying to write an object store, writing the UnitTests really slowed me down because they required that I knew where everything went. Even though I've serialized object graphs before, I was doing it much differently this time and it became annoying to change all the UnitTests every time I learned something new (i.e. every five minutes). Especially since the UnitTests were several times the size of the actual code. I should note here that my iterations were on average 30 minutes long with a median of around 10 minutes.

UnitTests didn't detect any non-trivial bugs either. Trivial bugs were things like typos in method invocations; things that would be detected without UnitTests. About 90% of all trivial bugs were in the tests themselves. All the non-trivial errors would have required massively complex UnitTests to write (versus minimal hand debugging). Complex UnitTests require more UnitTests!

UnitTests only saved me once when I lost a whole version of the source code but still had the latest tests. Of course, an IDE integrated with my source control system is a better solution. And besides, that was only a few hours worth of work that I lost.

The whole problem with UnitTests is the whole static-dynamic issue in programming. The more dynamic you are, the better you are to handle decisions. Anything that hammers you into a design or a decision is bad. UnitTests add a negative value to changing decisions, so they are at least staples if not nails.

I quickly decided to not write the UnitTests first because they were asking me to make the same decisions CASE tools do: What method goes where? What is it called? What member variables do you need? What signatures do you need? What are the constraints on the arguments? I didn't know, so I couldn't write them. It wasn't that I didn't want to write them, I just couldn't. My motto: One decision at a time. Maybe they didn't work for me because my mind is low wattage.

Anyway, it's not that I think that UnitTests are bad. They apparently work for you. I just don't understand how. I was hoping someone could explain. -- SunirShah

See http://members.pingnet.ch/gamma/junit.htm and http://www.xprogramming.com/publications/testfirstexample.pdf for examples of test driven programming.

DesignByContract is difficult because the contract is far too static and assumes that you can describe the system before you're done writing it, which we all know is false.

That depends on what you think "the system" is. If you're meaning "the whole finished software product (as an next release)", then you are right. If you mean "the nexus of classes I need to think about for the current sub-goal", then you are less right. If you mean the two (let's say) classes on either side of the particular interface I'm working on right this minute", less right again. And so on. When we get down to if you mean the current argument of the current method of the current class of the current interface of the current sub-goal", they you must be absolutely wrong, or else how do you know what to type (irrespective of whether or not you wrote a test first)?

This indicates that in order to have value test-first unit tests need to be small, simple and focussed as possible. But still big enough to be worth writing-- there is , as always, a sweet spot. How big/complex/wide-ranging were yours? I ask this in particular since you complain that it became annoying to change all the UnitTests every time I learned something new. Well, I'm sure it would be, but I'm puzzled as to why any increment in knowledge would require you to change all the tests. (Or was that just hyperbole?)

The more dynamic you are, the better you are to handle decisions. Anything that hammers you into a design or a decision is bad. UnitTests add a negative value to changing decisions, so they are at least staples if not nails. This I find puzzling. In my experience the growing body of test-first tests enable dynamism in design by providing a level of semantic DeltaIsolation.

Uh, now I'm confused. In test-first we write the test for a method immediately before writing the method. If you don't know where the method goes, what it is called, what variables it needs etc. immediately before writing the method, how are you going to write the method anyway? At this micro level, one thing that a test-first test does is to let you capture what you think as many of those bits of information as you need are, immediately before you start writing the method itself. If, while writing the method you learn something (quite likely), then having the test to record your earlier thought might help you not get lost (it does me). You will have to change the test, true. But if the test is small, simple and focussed, that's the work of seconds (and the same benefit applies in the new situation, inductively).

UnitTests didn't detect any non-trivial bugs either.

Of course they didn't, that's not what they're for.

My motto: One decision at a time. Maybe they didn't work for me because my mind is low wattage.

A good motto. I find that test-driven development enables me to serialize decisions.

As to wattage, I'd suspect that it's quite the reverse. In the years that I've been using and teaching test-first techniques I've consistently found that the developers who have most difficulty with the technique are either the very, very smart ones, and/or the ones that know themselves to be very, very smart (the intersection of those two sets is not empty, but the union of them is much bigger than either).

Personally, one of the things that I enjoy about test-first is that it enables me to produce working software at least as quickly as I did with the old methods, but without having to run my brain so hard as before. YMMV --KeithBraithwaite

: [...] writing the UnitTests really slowed me down because they required that I knew where everything went. Even though I've serialized object graphs before, I was doing it much differently this time

Usually when this happens, I decide it's time for a quick SpikeSolution session. Sort of to scout the road ahead, y'know.

The UnitTests required you to know where everything went, but you could write the code without knowing where everything went? This doesn't seem to make a lot of sense to me. I agree, it sounds as if the unit tests were testing that the code followed a particular implementation, rather than it exhibiting a particular behavior. My best tests test the code from the point of view of the consumer, or client of that code. It's when I mock some behavior behind the scenes or, worse, test some expectation behind the scenes, that my tests become brittle and tied to the implementation. Sometimes that's appropriate, but not usually.

I'm confused as to why people say that small UnitTests are usless. It is true that larger UnitTests are difficult sometimes (especially if you're intoducing them to a project not made with testing in mind). But, small unit tests (testing very small units of any given component you are writing) let you make assumptions about the larger test. If you have a test saying method X will handle the conditions you throw at it without fail, then when you write a larger test, you can trust that method.

I was recently working on a nightmare of a project. We were using an untested, difficult to reproduce C library with no tests (MEL's Bufr encoder library). The code on top of it was legacy, poorly designed, and buggy as well. The only tests for these encoders was to see if they actually encoded a file correctly. Not very useful for finding bugs. As I began to extend the system, I'd start testing small units of functionality. Does this date parse function work? How does it work with a malformatted string? Does this initialization routine work? How well?

I found several bugs in the individual methods which were trivial, but when added up in the larger sense produced bewildering debugger outputs and random segfaulting. People had unsuccessfully "identified" these bugs in the past by identifying and working around their oddities, assuming the old C library was at fault. Without dozens of tiny unit tests, I probably would have made the same mistake.

Instead, I brought that leg of the project to a close in less than 2 months, roughly 8 months ahead of schedule. :)

So, I like small unit tests, eif they only catch small bugs. -- DaveFayram

CategoryTesting CategoryRefactoring RefactoringUnitTestsPaysOff