Two Year Itch

I've been a testing consultant since '91 or '92. I have worked with companies that used to have a lot of UnitTests. When I ask them where those tests are now, they say "on backup tapes in a vault somewhere". What happened?

Programmers wrote the tests with great enthusiasm. In due course, code interfaces changed and broke the tests. The programmers spent some time FixingBrokenUnitTests?. They did it with nearly as much enthusiasm.

Later, the tests broke again. The programmers fixed them again. With less enthusiasm.

At some point in this cycle, some programmer said, "I gotta get these changes out. I'll set the tests aside, just for now, and fix them when I have more time." And that, as they say, is all she wrote.

For some reason, I've gotten the impression that the whole cycle - from enthusiasm to mass disillusionment and abandonment - takes about two years. So I've dubbed it the TwoYearItch, by analogy to what's called "the seven year itch" in US culture. In the vernacular, that's a period when a couple finds the work of maintaining a relationship increasingly difficult, to the point where a divorce happens. "According to the most current figures available from the National Center for Health Statistics, the median duration of marriage was 7.2 years for couples who divorced in 1989 and 1990." [1]

Because of the TwoYearItch, I advise programmers that DeletingBrokenUnitTests is sometimes OK, as is MovingBrokenUnitTests.

-- BrianMarick

It was shown in a study (*) that programmers will gladly do anything they perceive of having real value. Note the difference from doing something that they have been told is valuable. Since the UnitTests don't experience the RatchetEffect (HoldingTheGains), which is the goal any professional programmer, naturally they should be/will be chucked.

(*) I wish I had a reference to it. I saw it somewhere on Wiki, I believe. Anyone?

-- SunirShah

It sounds to me as though those guys weren't doing CodeUnitTestFirst. --

Yeah, If you changed the UnitTest as part of changing the interface, you wouldn't gripe about the additional work, in fact you'd find that the UnitTests were making your job much easier. Consider this: changing the interface in one place in your program means making lots of little edits in many different places throughout your program. In order to get your program working again (or even to compile again) you've got to make all of the edits. That means doing a substantial amount of work before getting any positive feedback (I've done this more than once only to discover that there is a fatal flaw in the new interface halfway through the editing process). If you CodeUnitTestFirst then you will start by changing the UnitTests to reflect the new interface. This will put you in a position to consider the ramifications of the proposed change in a small, controlled, environment. You will be able to make the change in small increments, one test at a time, and observe that the new implementation will work before exposing the rest of the system to it. Each client of the interface can be upgraded the same way. If you wait until the new interface is in place and operational before discarding the old interface you can do the whole thing incrementally over time and keep the whole system working while you do it. -- PhilGoodwin

I'm feeling dense. I fail to see what UnitTests add in this case over the actual client code. If you want to verify if your interface changes are sane, I don't see how your UnitTests help nearly as well as the actual context in which the interface is used. In other words, when, if ever, is a simulation better than reality? -- SunirShah

When the simulation is easier or cheaper than reality?

And UnitTests aren't cheaper than actual code. At least that's the point of this page. (By the way, I really meant better in terms of accuracy, not in terms of economics.) -- ss

UnitTests are much more flexible than the actual client code and more compact as well. The UnitTest for a class is a bundle of examples of how the class is meant to be used and what is expected to happen when it's used that way. If you want to change something about a class you have one-stop-shopping. Just go to the UnitTest, change it to reflect the new way that you want the class to act and then go make the class act that way. This is better than the client code because it is more focused: the client code is only using the class as a means to an end and may exercise only a narrow slice of its functionality, a UnitTest, in contrast, is explicitly focused on defining the limits of what the class is supposed to do. -- PhilGoodwin

What happened to YouArentGonnaNeedIt? If the class is doing something that isn't used, delete the functionality. --

You miss the point - UnitTests put it all together in one place. They also show not only the inputs but the outputs as well. Client code is spread out all over the place. Also, a UnitTest allows you to test the change to the interface in an isolated environment. Without them you have to pump the input through half the system just to get it to your new code. -- PhilGoodwin

There is a fundamental misunderstanding of the definition of UnitTest here. Quick summary: when I use the term UnitTests, I really mean unit tests. The true, formal definition. I gather that XP folk really just mean test code.

I agree, writing test code is much easier than a lengthy session with an interactive debugger, so lazy programmers will take that route. No disagreements there. However, UnitTests are bloody expensive. So, before you continue arguing, please define your terms. -- SunirShah

I am using the XP definition of UnitTest as defined on UnitTest. I only test for new functionality and things that I feel nervous about. I thought that the "...those guys weren't doing CodeUnitTestFirst" at the start of this section made that context clear. I agree that the "real, formal" definition of unit test requires far more effort than it pays back. -- pg

I've argued before (http://www.testing.com/writings/new-models.pdf) that "unit test" is a hopelessly confusing term. Plato said that good terminology "carves nature at its joints" by making real and important distinctions. The term "unit testing" blurs important distinctions:

between test implementation and test design
between test design as a program design tool and test design as a bug-finding tool
between test design (and implementation) done before the code and design done after the code

That makes productive conversation hard. Everyone means something different, and checking what each person means is exhausting.

We have (in StandardDefinitionOfUnitTest) a definition that is all about implementation and barely mentions design. That is a definition that is enshrined in certain standards (IEC61508, implicitly in IEEE528), but I think it's a mistake to assume that those standards reflect real practice. They're relics of the days of punched cards, when things like the order in which you integrated "units" actually might have mattered.

I would not assume the people I talked about in the beginning of this page were blindly following that standard definition. And I believe that even people who do unit testing right - test-first design, use test design to constrain program design and to find bugs, use stubs only when needed to make tests easy to implement and run fast - could still find themselves getting tired of updating tests. (See DeletingBrokenUnitTests for a reason.)

Were I responsible for a project, I'd worry about that. XP has some practices that would reduce my worry - pair programming, for example - but I'd still worry. It's one of the places I'd devote energy to determining whether people were thinking or merely following rules ("fix all broken UnitTests"). -- BrianMarick