Fixture Infection

This arose from a discussion of PerlUnit. An implementation of the TestingFramework stopped all tests when an exception occurs during a test; the 'die' (exception) is not handled and a stack trace gets printed.

What Should Happen Here is that the tests continue after trapping the exception. In JavaUnit, this is achieved by passing back the exception as an 'Error' in the TestResult, as opposed to a Failure. This is deemed to be safe, because, though I haven't seen this written explicitly before, the fixture is torn down immediately after each test.

In JavaUnit, a TestCase is a set of tests related by them sharing the same fixture. This does not mean that the fixture exists across the tests. On the contrary, the fixture is set up and torn down for each and every test. This is the key point.

If the TestingFramework is not written to keep fixtures 'uninfected' as you run over tests, then you cannot predict the state on entry to a test, and you will almost certainly get ugly cascading errors. In essence, you have made the test coupled: it no longer satisfies UnitTestDefined?.

Working this way means that 100% of tests should return a meaningful result (a Pass, Fail, or Error), which is surely better than not trapping the exception (because you worry about FixtureInfection) and seeing the result of only 57% of tests?

Why only 57%?

As it happens, the code that caused me to write this *is* safe. I just haven't seen the reasoning behind the way setUp and tearDown are called made explicit before.

The reasoning above implies something else, too: TestSuites should not have Fixtures. A test suite having a setUp called before it runs tests, and a tearDown afterwards, provides an unpredictable, infected environment to its child tests. However, as well as coupling tests, it violates what we would expect the TestCase class to have responsibility for. If one of its tests was run from another suite, or at a later date when the suite has changed, they may magically fail. And this happened because the TestCase was no longer responsible for the data (ALL of the data) that it uses.

-- BrianEwins