Defect Seeding

I'm simply going to describe what this is and let others refactor it into the desired pattern format ...

Also Known As Bebugging by TomGilb Circa 1980.

DefectSeeding is a commonly used and published strategy of deliberately and purposely injecting known defects of non-trivial severity into the artifact that is to be reviewed. Care must be taken to ensure that its not injected into the real thing, but some reasonable facsimile instead (or else make darn sure you are always able to remove all of them before checkin-time).

When I've seen this done (as an experiment, not as standard development procedure) the defects were added offline and distributed to the reviewers in paper copies only. -- JeremyCromwell

DefectSeeding allegedly serves two purposes: First, the percentage of seeded defects detected during inspection can be used as a reasonable predictor of the effectiveness of the review. Someone somewhere (I forget who) published some data to suggest that the percentage of seeded defects found had a close correspondence to the percentage of total defects found (both seeded and unseeded). This meant that if we didn't do a very good job of finding all the defects that someone already knew about, we probably did just as bad (or worse) at finding all the ones that have yet to be discovered, and perhaps take some additional measures.

Secondly, some shops that use DefectSeeding successfully have claimed that it is a better motivator for the reviewers to more thoroughly seek out and uncover defects. Apparently it's a sort of challenge to everyone to see how many of the known/seeded defects they can find; it supposedly adds more synergy and zeal to their inspection efforts. The issue here is how many defects are inserted and do you tell the reviewers how many there are, or just give them a vague idea but no hard number or limit.

In general, deciding on the typical number/range, type, and severity of defects to insert requires a bit of thought. Effort is required not only to think them up, but to make sure they don't stay inserted after the inspection is over if they weren't uncovered. One wants to get a good enough sample size to give a meaningful percentage-found estimation. At the same time, one wants a fairly diverse set of injected defect-types for use not only as a predictor of the percentage of kinds of defects found, but also so its not too easy to distinguish what is likely to have been a seeded defect from what isn't.

Numerous shops that use DefectSeeding, and manage to reputably avoid letting any seeded defects slip through the cracks, report great satisfaction with this technique.

"Name two." Hey, I'm serious here. Does anyone here really do this? Does anyone anywhere in the real world really do this? Confess, Harlequin! --RonJeffries

Yes Ron - they really and truly do: Real shops in the real world writing real software for real users and real paying customers with real deadlines and real market cycles and schedules really do this stuff! And they rave about it too! Folks at IBM have actually published hard data to back up their claims of it's benefits. So have folks at HP, and Motorola, and a few other places as well. Dig around somewhere in the web pages and technical publications long enough and you'll probably find the stuff. This practice was originated in a non-software field and there are loads of published results for its use in Hardware as well as software. A good search in IEEE publications would probably turn something up. (Ooops - you said you only wanted two, that was more than two - sorry :-).

Motorola, really? I did a gig there a while back and didn't see the practise. I personally know at least a half dozen people who work or have worked recently at Schaumburg, Mundelein, Harvard, or Arlington Heights. All of them participated in inspections. None of them reported this practise.

Response: Yeah. Really. That doesnt mean everyone there does it. Just that there is ample evidence of folks that do. Ive worked there awhile myself and have spoken with folks from three different sectors across about 5 different locations that reported having used it in their group.

This sounds similar to MutationTesting, where one intentionally inserts errors into code and runs the code through the RegressionTesting suite, to see how effective the test suite is. Various methods of automating the mutation of code are considered; things like changing "<" to "<=", for instance.

I have to admit this sounds like a great idea. In some of the places where I wish I had it however, I dared not use it. You see, OptimizeLater had come and I needed to replace the main report loop (originally written in Word automation) with a version that walked some serialized form (in this case, we noticed that Word exports to HTML and imports losslessly) with some very fast version. I ended up writing that code in C, using almost pure conditional GoTo statements. If I had seeded any defects, nobody would have noticed as, unfortunately, there is nobody else who could maintain that code. Yeah I know, a TruckNumber of one is a bad idea. At least that code only ever had one bug since it was released (six months before I wrote this). - JoshuaHudson (Update four years later: two more bugs since, both outside the goto cluster, one 64 bit only).

See: ProjectSaboteur

The simple way to be sure that you can remove the defects is to add them all in a single commit to your code repository. I make a list of files that were revised recently, use a random number generator to select files from it, then mutate statements that are not covered by the automated tests. To be exact, I make a list of files sorted by number of updates, not in the iteration under test but in the previous iteration. This focuses the measure on discovery of defects that were missed in the previous testing round.

-- chrishmorris

CategoryTesting