Asynchronous Unit Testing

CategoryAntiPattern

You are developing software that must run correctly in a number of target environments. You have decided to be aggressive about testing, and so you have a growing suite of UnitTests that run at 100% on the developers' machines all the time. Building the system and running the UnitTests on every one of the target environments requires the programmer to perform a dozen or so non-automated steps, takes a couple of hours to complete, and completely breaks the flow.

Therefore, write a batch job for each environment, that runs every hour and mails a log to every developer if there was a build or test failure. The batch job should automatically build the latest version of the system, and will therefore pick up all newly integrated changes every time. Now your team need only UnitTest in the development environment; any problems in the target environment will show up within an hour of integration.

On the contrary, the notice of problems goes to everyone, not just to the perpetrator of the bug. For a brief few minutes, every programmer in the team checks the log and looks at the failures, wondering "did I cause this somehow?". Productivity decreases briefly, and there is a real chance that everyone will assume that someone else is fixing the problem. When problems go unfixed for a while, they mount up and everyone begins to ignore the emailed reports. Your system is now degenerating. Eventually, a new task must be scheduled to get rid of those pesky reports. Aggressive testing has led to an incorrect system and an inefficient development process.

Therefore, take the time (and pain and cost) and develop a one-touch tester that can run the UnitTests on every environment. Find a way to make it run fast, say by doing incremental builds. All the required UnitTests can now be run "in-line" in the programmer's task again, so only the perpetrator gets disturbed - and she knows it must be her bug.

Sometimes it's not possible to have a one-touch tester that runs on all the targets environments. The MozillaTinderbox system at mozilla.org demonstrates an interesting way to deal with developers distributed over the net without access to all the target machines. -- MathieuGervais

The problem here isn't the asynchronous nature of the unit testing environment, but the fact that the test system is performing PolledUnitTesting?, which results in the test system having no knowledge of who made which changes. Instead, the test environment could be designed to be poked by a developer when they have made their change, run the tests asynchronously in the same manner, and then report the results to the developer who poked it.

CategoryDevelopmentAntiPattern