Cost Reliability Tradeoff

For the vast majority of software, there is no tradeoff between cost and reliability.

It is well-known (perhaps even true) that finding and fixing defects later in the software development process is orders of magnitude more costly than catching them earlier.

Therefore, do the math, an approach that finds the bulk of the defects on the first day will cost less for any given level of reliability than an approach that finds them on any other day.

One such approach is to have UnitTests for all code, and to run the UnitTests incredibly frequently, such as every ten minutes. When this is done, the bulk of defects inserted are caught within ten minutes. Only a few statements have been added, so that only a few statements need to be considered as the cause of the defect. The developer's mind is fresh on the subject, and she can easily find the defect.

It takes overall less time and therefore less money, to bring the product to any given level of reliability.

There is no tradeoff between cost and reliability. Increased reliability done wisely decreases cost.

-- RonJeffries

How often should the acceptance tests be run? In a way aren't they more important in determining whether an application has been broken over all? -- MichaelFeathers (will move this to an appropriate acceptance test page once I find it)

We find that running the entire suite of UnitTests and requiring them all to run at 100% gets most of the job done. We also run a subset of the AcceptanceTests before release to the developer config, and all the AcceptanceTests before release to the customer. The idea above isn't pure XP, but an example that goes to the effect of the earliest testing possible. We run the UnitTests for what we're working on every few minutes, the entire suite every now and again, and all of them before release to the config (ideally at least daily for each developer). -- RonJeffries

I believe the above may contradict TestFirstDesign. If one follows test first design, he will avoid putting in the defects in the first place. This is the true way to avoid the cost reliability trade-off, not by merely using frequent regression UnitTests as a means of quickly catching problems after they occur.

CategoryTesting