Mutation Testing

Mutation Testing involves running slightly corrupted versions of your target program through your test suite to see if any test cases flag the variations as defects.

---

MutationTesting turns the "testing" question upside-down: Instead of asking how if we've tested the program enough, we ask a more useful question: Are the tests sufficient to detect bugs in the program?

So, we "test the tests" by introducing bugs in the code and running the tests. If the changed program passes all the tests, just like the original program, you have to ask "why doesn't it make any difference if this variable is off-by-one??? Most likely, it's an area of the program that hasn't been tested sufficiently.

---

I'd be interested to know if anyone has used mutation testing in a practical way in industry. Very few tools support this technique and writing your own is not an option unless you are in the compiler business. But you could use a perl script to generate a few variations from a checklist of mutations. Can the technique be usefully adapted for industry?

As I understand it, MutationTesting is intended as an iterative process. On each pass, you tighten up the tests to catch more mutant programs. In contrast to DefectSeeding, MutationTesting seems to be targeted more to the test suite than to the testers or the review process. Also, the type of defects used in MutationTesting tend to be simple variations like a plus instead of a minus sign. The theory is that complicated bugs cluster around small bugs, so that if your suite catches simple defects, if will find both.

http://www.isse.gmu.edu/faculty/ofut/rsrch/abstracts/practical.html contains some ideas for limiting the cost of mutation testing.

Cayte

There's a very important application of MutationTesting (aka ActiveNonlinearTesting) in working with academic or other first-principles models. See http://www.santafe.edu/sfi/publications/Abstracts/96-03-011abs.html for the first place I've ever encountered it. I'm not sure exactly how it can be used in a production coding setting, but it's a vital aspect of working with models, and anytime GeneticAlgorithms are involved. -- BillTozier

While we have not done production mutation testing, the DMS Software Reengineering Toolkit http://www.semanticdesigns.com/Products/DMS/DMSToolkit.html could be used to implement this pretty easily. DMS enables one to specify as set of arbitrary source to source transformation rules. This is much safer and easier than using PERL to do this, as the transformation rules operate on the abstract syntax of the program. By definining a set of mutation transforms, it would be easy to produce mutants. As a bonus, DMS works with virtually all the languages in common use. -- Ira Baxter, CTO Semantic Designs

---

If your UnitTests are automated (automated RegressionTesting), an automated MutationTesting tool can tell you how well you've managed to TestEverythingThatCouldPossiblyBreak.

Tools:

• JesTer (described on the XPDeveloper site - follow the link)

---

See also DefectSeeding, which is targeted at people, rather than being automated.

---

For JavaLanguage programs, maybe something like BCEL (http://jakarta.apache.org/bcel/index.html) could be used to do "mutation automation". I have used it to do some simple code instrumentation work before, it could easily be used to make the code "always return null" or "always return without making any modifications" or "subtract instead of add", ...

---

PiTest? (http://pitest.org/) is actively developed as a JavaLanguage mutation testing tool. It has a good integration with SonarQube?.

---

CategoryTesting