Test Inventory

If you collect tests automatically using a TestCollector (and even if you don't), it's also a good idea to use a TestInventory to detect if tests have silently disappeared.

A TestInventory is, well, a list, or inventory, of tests that have run in the past, and which can be expected to run in the future. Usually, a test framework implementing TestInventory will record the names of tests run, e.g. into a file, say CurrentTestInventory?. This file will be diffed against the ExpectedTestInventory.

If no changes, all seems well.

If tests have been added, the diff will report this, and the human running the tests has the option of making the CurrentTestInventory? into the ExpectedTestInventory?.

If tests have been accidentally deleted, the diff will report this, and the human running the tests can go look for the missing tests, and why they were not run.

I started using the TestInventory pattern after bugs in my TestCollector resulted in tests that were written not being run. Also after I accidentally deleted some tests while editing, not noticing it for a while because I was using a TestCollector and not paying attention to the TestSuiteConsistencyCheck.

See HandlingBrokenTests

TestInventory is a TestSuiteConsistencyCheck. A form of TestPositiveConfirmation.

TestInventory is related to ProjectBillOfMaterials.

TestInventory to some degree goes against OnceAndOnlyOnce. To add a new test, you have to code it, and then add it to the TestInventory. This is two steps, but the latter is (or should be) automated.

It guards against accidental deletion of tests by requiring the same two steps, first deleting the test, and then confirming. You're unlikely to do both by accident.

In some ways, this is like DoubleEntryBookkeeping.

CategoryTesting