Test Collector

When using a unit testing framework like JavaUnit, forget adding each TestCase to the TestSuite. It's tedious and error-prone.

Just have some code loop over the directory with your TestCases, and pattern match for your tests (i.e, if your convention is to have "Foo.java" and "FooTest.java", match everything named "*Test.java").

Bonus points go to implementations that put the TestCases with the most recent modification times at the top of the list (that way, your most likely failing tests come faster, it's more feedbacky that way).

Implementations:

• Java
  • http://www.panoptic.com/rking/TestCollector/javaTestCollector.zip <-- This is back - sorry about the downtime!
  • Thanks to AdamWilliams and the folks at RoleModelSoftware
  • Needs timestamp sorting

• Ruby
  • http://www.panoptic.com/rking/TestCollector/ruby-directory-test-suite.tar.gz
  • This even has a README, a "make install" target, and a handy run_ruby_tests script
  • Needs timestamp sorting
  • Thanks to RyanKing, DuffOmelia, RoyMiller, and the rest at RoleModelSoftware
• Php
  • http://www.panoptic.com/rking/TestCollector/php.txt
  • Sorts by timestamp
  • Courtesy Panoptic.com
• Python: PythonTestCollector. No time stamping; instead I run the pieces manually as needed.

Implementations built-in to the *Unit:

• NunitFramework -- deprecates manual TestSuite creation in favour of an automatic collection mechanism. Each assembly is scanned for tests, and suites are built according to namespace hierarchy.
• CsUnit -- another unit testing tool for the .NET framework. Also replaces the need of manually maintaining a TestSuite by automatically scanning assemblies, and also by allowing to combine multiple assemblies into a recipe (see http://www.csunit.org).
• CppUnit -- The more recent versions automatically register all tests (http://cppunit.sourceforge.net/doc/lastest/cppunit_cookbook.html - TestFactoryRegistry?) is something like a TestCollector built purely within C++ without any external scripts. However, something is still necessary to get an object file linked in, so the static objects can register themselves. An external script TestCollector is still necessary to do this. Plus, external scripts can provide the "most recent tests first" property.
• VisualCeePlusPlus - this uses the TEST_(suite, case) macro to collect test cases, without sweeping your hard drive for them...
  • I presume this is referring to NanoCppUnit, which is for VC++ only.
• OCUnit -- Automatically adds subclasses of SenTestCase to the list of test case objects, and instance methods of these objects returning void and with no arguments whose names start with 'test' are added to the suite. Phew! MyTestCase:SenTestCase>>- (void)testSomething, for instance, will be automatically run.
• SUnit Browser -- When run from the class browser, collects the tests in the selected TestCase subclass. If the selected class is abstract, collects the concrete subclasses of the selected class. When run standalone, collects all the TestCase subclasses in the image.
• Eclipse + JUnit -- Allows you to run all tests in a Project, Source Folder, Package, File or Class. Unfortunately, it doesn't seem to give a pleasant way to run tests from a project that the current project depends on, and it doesn't allow you to specify a specific order (i.e., newest first).
• Cutter [CutterTestingFramework] -- Collects symbols from dynamic loaded shared libraries and selects test function from them. So you don't need to write any registering codes or macros in spite of using CeeLanguage.

---

If you collect tests automatically using a TestCollector (and even if you don't), it's also a good idea to use a TestInventory to detect if tests have silently disappeared.

---

Is a TestCollector necessary if you NeverWriteaLineOfCodeWithoutaFailingTest? You should be expecting a RedBar when you run your test for the first time, and if it's not, you'll realise you forgot to register the new suite.

See ThreeStrikesAndYouAutomate. Why should I keep adding test suites when the computer is perfectly capable of doing it for me?

---

I've been thinking about writing a Java test suite collector for a while. I recently put my first attempt at a build collector on SourceForge. It builds three different test suites- AllCollectedFunctionalTests?, AllCollectedAcceptanceTests?, and AllCollectedUnitTests?.

Test collectors will work because your codebase doesn't include tests that depend on the order that they're run. Only a complete tool would commit something like that into a repository.

--

---

In ruby, I just require all the *.rb files under a tree, and then I go through all of the objects looking for ones marked appropriately. (In my case, the convention is to test Foo with a TestCase named TC_Foo.) I used actually just to run everything that inherited from TestCase, until I started creating abstract TestCases descending from TestCase (DBTestCase, WebTestCase, etc.).

suite = Test::Unit::TestSuite.new("ruby unit tests")
test_regex = Regexp.compile('^(.*::)?TC_[^:]+$', Regexp::EXTENDED)
ObjectSpace.each_object(Class) { | klass |
suite << klass.suite if test_regex.match(klass.to_s)
}

---

Requiring all the *.rb files (or some other) is a test pattern that seems quite prevalent. Maybe I should call it something like TestAllClassesTogether?. CppUnit and CppUnitLite do something like this, by automatically registering all tests with the testrunner.

However, in languages like C++ this does not test that the classes (and their associated files) can be used in isolation.

Conjecture: SmalltalkLanguage and its friends have no concept of file structure. All is workspace. Now, I know that I personally have had trouble moving stuff from one workspace to another - making sure I have all the dependencies.

---

CategoryTesting