Diff Test

Maybe your UnitTests pay for themselves, or maybe they don't. Sometimes it's hard to tell. However, there is another approach that always comes out way in the black, and I call it Diff-Testing.

The concept is simple: capture the output of a program's successful run, make some changes, compare the outputs (outputii?).

This takes just a minute to set up. Literally. Here, time me:

  ./run > out.green
  vim run
  ./run > out.questionable
  diff -u out.{green,questionable} | less -F

36 seconds. From no tests to some surprisingly thorough RegressionTests.And the first time it helps you catch a mistake you would've otherwise sent to a customer, it saved you time, possibly money, embarrassment, and an adrenal discharge.

Usually during the first session I'll go ahead and put the above stuff in a script, probably even make a VimKeymap to do it for me. That first line goes in a script called "./snapshot" or something - used after changes that intentionally affect output. The bottom to are usually just "./test". In this file, I like to have the unit tests run first, if they're present. If they pass, go ahead and show the diff.

Problems:

Yes, it is possible to write bugs that this will not catch. Not only in the theoretical "Can you ever really test everything?" sense... but in the "You should keep your brain turned on." sense. You might have to run the program a large number of times to cover all combinations of significant conditions -

The main problem is retraining ourselves to run it between nice, atomic changes. If you introduce a subtle bug, and then decide to toss in some noisy change (such as converting your output's tabs to spaces), you'll probably not have the patience to notice the diff.

Another consideration is constantly-changing stuff, such as dates, randomly generated stuff (e.g., advertisements), external data. If this is happening, you probably would benefit from a "diff-ignore" marker. So, in HTML, you could wrap your random quote generator like this: <!--diffskip-->"It was the best of times..."<!--/diffskip-->Then, in your snapshot and test, you filter anything between the two tags out.

Also, it's slow. This might DogFood you to optimize a bit, or it might just plain discourage execution of the script. C'est la vie.

Not only is it a great way to get something off the ground quickly, also, a large percentage of the best regression test suites I've ever seen (and I've seen a lot) were elaborations on this idea. The worst regression test suites had no trace of this idea, no matter how fancy their gui.

It's about regression, that's all, it's not the same topic as other kinds of tests.

---

[ The following was deleted, presumably because it was considered personal-spam or something, however, I downloaded the tar file in question and looked it over, and it seems to me that it is on topic for this page. -- DougMerritt ]

A simple example is RyRes: http://panoptic.com/rking/pkg/ryres.tgz, my resume scrift. I feel nervous making changes to that thing without diffs, and the fact that I generate it from code makes me a little more nervous. I slap down a little 5 minute test framework, and my fears are put to rest. Also I keep it under VersionControl, because I'm not crazy. In case you don't use version control to keep everything that's important that's not absolutely immutable, you need to change your ways. Don't even argue.

Yeah, I deleted it, because I'm the one that hosts the file, and after I left I thought it was spammish. Here before too long I'll have a non-commercial link.

I actually like the fact that it's an unusual example: a resume. Such things can trigger creativity in others. But, as you like.