Ytwok Unit Testing Triumph

As noted elsewhere, the rollover didn't just exceed my expectations. It wiped them out. As wiki's foremost paranoid, it's time I wear some egg and eat some crow. I'm not sitting down the bottom of a mineshaft, but it's certainly fair to say I don't expect to purchase any rice or beans until about 2100. So please have a good laugh at my expense, but let's talk seriously about why things have gone so well so far.

ExtremeProgrammers have observed many times here that, so long as they keep the discipline to CodeUnitTestFirst, large systems often seem to work miraculously well even the first time they're run. This is most surprising because a large system seems much more complex than a bunch of isolated units.

Perhaps we shouldn't be surprised. If we ignore the complexity of each unit, which we can when each unit is well tested ahead of time, the complexity of a large system may in fact be less than that of an individual unit. An egg carton, or even a whole egg farm, for example, is vastly less complex than an individual egg.

Now with Ytwok it was a prime cause of my concern that end-to-end functional testing could not be attempted for large infrastructure systems. Individual machines and systems could be tested and replaced, but meaningful tests of most large integrated systems were both too expensive and too inconvenient to be done across the board. Effectively, we could only rely on our UnitTests. Amazingly, the UnitTests worked great, even on these global scales.

The lesson I'm taking away, then, is that UnitTest(ing) is more than just a way to enable ReFactoring. It's a way to build in great fault-tolerance too. Pass me that black feathered drumstick over there won't you? --PeterMerel

I think this topic is deep. I'm engaged in a number of discussions around the web where people are trying to convince us that certain things are hard. SecurityIsHard and the related pages here, for example. Another was a long exchange on why proper testing is O(infinity) because you have to test all classes in combination with other classes. Another on whether you should purchase a defect database the first day you start your project.

In each of these discussions, I've been accused of trying to solve this particularly complex screw problem with my standard hammer. Well, I am, but the insult aside, it's not without serious consideration.

Modularity is part of what is going on here. If A and B are sufficiently independent, then a broad class of errors in A can't affect B. (A certain class still can, mostly relating to assumptions about their interconnection, e.g. the Mars Orbiter units problem.

Clearly if one module has 3 cases and the other 4, there are 12 combinations. However, there are certainly conditions under which it is sufficient to test just the 3 and the 4, i.e. 7 covers 12.

Our fears tell us that it isn't always that way - sometimes you have to check everything in combination. Yes, this is clearly true in that we could write a program with that property. But in programming, everything doesn't really affect everything, especially in objects.

So what happens is that generally (but not always) errors don't cascade, and disasters don't occur.

My hypothesis is much like yours: UnitTests are much more powerful than people realize, because the case we have isn't the worst case.

Further hypothesis: if you build with this in mind, you can build software where you can be quite confident that you don't have the worst case.

ExtremePhoneSwitch?, anyone? --RonJeffries

Hi, I only just got back to the big city from my own Y2K bunker - cunningly disguised as a remote Scottish island (without physical power, water or phone links) on which my in-laws have lived for many years. I never read a better "Happy New Year" message than this one.

I agree that this issue runs deep. Ron, your instincts (and those of most writers on Wiki) were right, that's extremely important.

Maybe there was a small WikiCollectiveSanityTriumph? here, showing the real world benefit of true CollectiveIntelligence. Kinda analogous to barely coordinated, distributed unit testing putting paid to (enough of) the two digit century bugs for the big "system of systems" we depend on to hold together. The WikiNature helped deal with psychological or social bugs in a few sincere but ineptly-extrapolating (try saying that a few times after a traditional Scottish Hogmanay celebration) software engineers, as I was at least some of the last eighteen months.

Certainly PeterMerel's "I admit that my paranoia is wearing a bit thin" has my vote as the GreatestPhraseOfWikiNinetyNine.

--RichardDrake

CategoryYtwok