Considerations For And Comparison Of Cee Plus Plus Test Frameworks

This page tries to list and discuss several considerations for C++ testing frameworks, and occasionally mentions which testing frameworks have what features.

It is intended to help me, AndyGlew, decide which of the widely available C++ Unit testing frameworks I may want to use or, failing that, steal features from for inclusion in my own test framework.

Most of these considerations are not specific to C++ unit testing frameworks, but in fact apply to any C++ library intended to be widely used.

See also WhySoManyCeePlusPlusTestFrameworks.

Summary

What started out as a fairly small attempt to compare test suites is turning into BigDesignUpFront. Or, maybe not BDUF, since there really are only a few design alternatives being considered, but perhaps AnalysisParalysis.

Basically, I want the following:

There are two basic ways to structure tests The former work marginally better with TestCollector scripts and multiple test frameworks, but cannot self-register. The latter imposes conventions on the TestCollector scripts, but can self register and are thus easier to use in the absence of scripts.

My listing of pros and cons goes on and on. It may actually contain useful design principles for C++ libraries. But there doesn't seem to be one single best solution for C++, so I think I must get on with life.

(Updated May 2005: Noel Llopis has written a nice survey of C++ unit test frameworks, http://www.gamesfromwithin.com/articles/0412/000061.html, using criteria very similar to my own, concluding that cxxtest is the easiest to use.)

After a 72 hour break, I realized that it does matter, that I was responding to a CodeSmell in CppUnit (CppUnitCodeSmell), and that a small "nugget" satisfied my goals: AndyGlewMinimalCeePlusPlusTestFramework.

HeaderOnlyCeePlusPlusPackages

It is especially convenient if a library or package can be expressed completely in C/C++ header files.

Dependence on "Advanced" C++ Language Features

Unfortunately, C++ is very inconsistently implemented.

I.e. G++ is my standard of portability.

Programming in the least common denominator of all C++ implementations would be too constraining.

Certain language features are frequently problematic:

OnceAndOnlyOnce'ing

C++ is a language that requires lots of repetitive code.

For example, you may need to define a class in one place, and declare it in another, with a forward declaration still somewhere else.

Or you may need to define a testcase in one place, and link it onto a test suite somewhere else.

Or, you may need to type a test case name in both the class and as a printable string.

There are several ways of reducing this repetition, but they have varying degrees of ugliness.

OnceAndOnlyOnceingPrintableNamesOfCodeObjects

Test frameworks often want to be able to report a name for a passing (or failing) test, or suite, or... Often the "print name" and the internal C++ name for the test class or method or function are almost identical.

There are several ways of doing this:

TestCollector

A TestCollector that makes it unnecessary to first code a test, and then enter the test into a list of tests, is a good idea.

Unfortunately, such a tool has to be written outside the C/C++ language as a script.

TestCollector scripts require naming and data structure conventions. For example, a TestCollector may

Minimize External Library Dependencies

The more non-standard libraries a package depends on, the less the chance of being able to use it on a different platform.

You should endeavour to only depend on libraries in the language standard, such as the STL (and even that is problematic), or on libraries whose source code can be distributed with your package.

Separate GUI Dependencies

In particular, as a special case of minimizing external library dependencies, C/C++ have no global standards for Graphical User Interfaces. Therefore, try to separate GUI code out.

Text Interface

The only reasonably portable I/O system for C and C++ is to try to provide a text-only or command line interface using C++ iostreams or C stdio FILE*s.

C++ iostreams are preferable, because they can be extended.

Thus, for example, standard iostreams can be used in a text only interface. But, a GUI oriented interface might extend the ostream to scroll results in a window, or parse the stream into a still more graphical interface like a folding browser, or one that pattern matches to reports tests passed and failed.

HTML/XML Text Interface If you must try for "advanced" features, vanilla HTML, and slightly less vanilla XML, are a good place to start.

E.g. XML can indicate arbitrarily nested test suites 

<testsuite name="ComplexNumberTest">
  <testcase name="testEquality"
started="10:29"
result="passed"/>
  <testcase name="testAddition"
started="10:31"
result="passed"/>
  .
  .
  .
  <summary ran="4" passed="2" failed="2"/>
</testsuite>
These are fairly easily understandable as ordinary text, can be reformatted by standard tools, and can be converted into useful GUI visualizations.

Script Dependencies

In limited languages like C and C++, it is often convenient to write extra-linguistic scripts, in Perl, etc., to avoid repetitive typing. (The motivation is usually to have a tool that can run before anything can be compiled. In interpretive languages like LISP (or Java with local filesystem access, there is much less reason to write such scripts --- except that scripting languages are often easier to code in, a secondary motivation.)

However, such script dependencies can make it harder to port your code. Now you have to port the C/C++ compiler *and* the scripting language :-(

I prefer to use or abuse language facilities such as the pre-processor (not m4) in order to avoid script dependencies. Occasionally, I will write C/C++ code that performs the sort of file munging actions usually performed by scripting languages.

For testing frameworks, however, pretty much the only way to implement a TestCollector is to use a scripting language.

Peaceful Coexistence

Big monolithic projects may be able to enforce a single, standard, big monolithic test framework.

However, libraries that are intended to be used in many different projects must have minimal dependencies, and must be prepared to coexist peacefully with other code.

C++ NameSpaces help. Unfortunately, they often have compiler bugs.

From the point of view of test frameworks, peaceful coexistence means, again, minimal dependencies. The library may be embedded in a project that uses a completely different test framework. It should therefore be possible to adapt or wrapperize the library's tests, so that they can be called from the enclosing project's test framework.

(I am assuming that the library is also under development along with the project. The library is not read-only golden code. The library's unit tests should be run.)

Conversely, a project's test framework should not make assumptions that make it difficult to embed a library with a different test framework within the project.

For example:

Testing Extensibility via Inheritance and Templates

Often, the whole purpose of the library is as a basis for extension and derivation. Therefore, one of the most important things the library can do is provide tests so that the derived and extended code can be verified.

I.e. the library's tests don't just test the library itself. The library's tests also test the user code that extends the library. Particular extensions or implementations get tested.

In my opinion, this is the key missing factor for code reuse. It has often been hard to extend libraries because the libraries have lacked code to test the extensions. Fragile base class problems arise in the lack of tests for the contract between library and users that extend the library.

If the extension mechanism is inheritance, such testing requires use of virtual functions.

Inheritance, however, is often deprecated. Similarly, virtual functions are often deprecated. They are deprecated both for efficiency reasons, and for code maintainability reasons.

Templates are a WONDERFUL C++ mechanism for tests, allowing tests to be applied to classes unrelated by inheritance.

VoidFuncVsTestCaseObjects?

There are two main ways of defining a test case:

(1) Void functions 

void testComplexEquality(void) {
TEST_ASSERT( Complex(10,1) == Complex(10,1) );
TEST_ASSERT( !(Complex(10,1) == Complex(10,2)) );
}
void testComplexAddition(void) {
TEST_ASSERT( Complex(10,1) + Complex(1,1) == Complex(11,2) );
}
(2) Test case objects 

  class ComplexNumberTest : public CppUnit::TestFixture  {
  private:
Complex *m_10_1, *m_1_1, *m_11_2;
  protected:
void setUp()
{
m_10_1 = new Complex( 10, 1 );
m_1_1 = new Complex( 1, 1 );
m_11_2 = new Complex( 11, 2 );  
}
void tearDown() 
{
delete m_10_1;
delete m_1_1;
delete m_11_2;
}
void testEquality()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
  };


  CppUnit::TestCaller<ComplexNumberTest> 
test( "testEquality",
&ComplexNumberTest::testEquality );
  CppUnit::TestResult result;
  test.run( &result );
Now, the difference in complexity between these two examples is pretty damned extreme!!!!It's not quite fair, though: CppUnit's test jigs, etc., are quite useful.

In general, however, anything that can be done with a test object can be done with a VoidFuncTest.

VoidFuncTest-s may lead to NameSpace collisions. Objects conserve NameSpace. However, collisions can be avoided be avoided by using file static. Moreover, naming conventions such as test_CLASSNAME are unlikely to lead to collisions if the classes don't collide.

Test Object Structure

When using test objects, there are several different ways to relate the test cases (test functions) and the test case class.

One might do as CppUnit did, making the test functions methods of the class. 

  class ComplexNumberTest : public CppUnit::TestFixture  {
  private:
Complex *m_10_1, *m_1_1, *m_11_2;
  protected:
void setUp()
{
m_10_1 = new Complex( 10, 1 );
m_1_1 = new Complex( 1, 1 );
m_11_2 = new Complex( 11, 2 );  
}
void tearDown() 
{
delete m_10_1;
delete m_1_1;
delete m_11_2;
}
void testEquality()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
  };
Or one might make each test case inherit from the test jig that it uses: 

  class ComplexNumberTestFixture? : public CppUnit::TestFixture  {
  private:
Complex *m_10_1, *m_1_1, *m_11_2;
  protected:
void setUp()
{
m_10_1 = new Complex( 10, 1 );
m_1_1 = new Complex( 1, 1 );
m_11_2 = new Complex( 11, 2 );  
}
void tearDown() 
{
delete m_10_1;
delete m_1_1;
delete m_11_2;
}
};


class ComplexTestEquality? : public ComplexNumberTestFixture? {
void run()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
};
In fact, when CppUnit does 
CppUnit::TestCaller<ComplexNumberTest> test( "testEquality", 
&ComplexNumberTest::testEquality );
it is doing basically this. However, TestCaller<>::run takes a result argument, and so succeeds in hiding the details of the result from the user.

Aside: a related convention is that a class have a "test()" method, or a test subclass. Taxonomically

Parameterized Test Objects Test objects (or classes) can be parameterized, so that something like a loop can be used to do range testing. With void functions, the loop would have to be embedded.

However, such parameterized testing is unlikely to be accessible to a TestCollector script.

It seems that VoidFuncTest may be useful as the interface that a TestCollector handles. Class and object test interfaces may be useful inside a test suite, but are less convenient for TestCollectors?.

Test Collections

In C++, objects in a collection must have compatible types. E.g. all objects in a test suite must be related by inheritance, and must share the same virtual function "run".

This allows suites and testcases to be intermixed, as long as they are related by inheritance.

But it gets in the way when different test frameworks are used in a package P, and its sublibraries L1 and L2. Since the frameworks are different, you cannot place P's, L1's, and L2's tests in the same collection. You must write adapters.

One Possible Moral: if you use free void functions as your top level test object, a TestCollector script can likely collect them. I.e. VoidFuncTest are a lowest common denominator that can allow different test frameworks to coexist with minimal adapter writing.

Test Registries

Registries are a C++ idiom that allows instances of a class to be registered centrally. Each file contains a static variable that performs the registration. The main program does not need to enumerate all the tests; the addition of a test or suite to the registry is decentralized.

Unfortunately, only objects can register themselves in this way. Void functions cannot.

This sort of registration is convenient when we do not have a TestCollector script that greps test cases out of the source file. It works in combination with a simpler TestCollector, that simply compiles and links all source files obeying a naming convention like foo-test.{cc,o}

However, it does require that the main program of the test have an explicit call to the test registry's run command.

If we have two different test frameworks in the same program, it might be possible to write a main() by hand that calls the registry for each framework, separately. (As opposed to VoidFuncTests? that mush all the different framework's tests together.)

TANSTAAFL I kept hoping that there was a way to get the best of both worlds -- the generic nature of a VoidFuncTest, and the flexibility of a TestCaseObject. Unfortunately, there does not seem to be such a way in C++.

If C++ had reflection, it would probably be pretty easy to get both.

Test Object Structure and TestCollector Scripts

After a night's sleep, and a day doing something else, I realized that using free void functions, and/or using test objects with a test() method that inherit their test fixture, are easier to use with TestCollector scripts than the CppUnit style where test case are methods in the fixture, and must be bound to create a test case object.

Consider CppUnit style testcases implemented as methods of a fixture: 

  class ComplexNumberTest : public CppUnit::TestFixture  {
  protected:
void testEquality()
{
CPPUNIT_ASSERT( *m_10_1 == *m_10_1 );
CPPUNIT_ASSERT( !(*m_10_1 == *m_11_2) );
}
void testAddition() { ... }
  };
to create a test case object, one must "bind" the various test methods: 
  CppUnit::TestCaller<ComplexNumberTest> 
test( "testEquality",
&ComplexNumberTest::testEquality );
  CppUnit::TestResult result;
  test.run( &result );
A TestCollector script that recognizes such methods and tries to automatically extract them into test case objects must understand enough of C++ language syntax to recognize what class they are methods of. That's a real pain.

For a simpler approach, consider testcases as free void functions 

void test_foo(void) {
TEST_ASSERT( foo(1) = "foo" );
}
and consider testcases as objects with a test() method: 
class ComplexNumberTestFixture? { ... };
class TextComplexNumberEquality? : public ComplexNumberTestFixture?
{
void test() { TEST_ASSERT( m_1_2 == m_1_2 ); }
}
A TestCollector script does not need to understand much C++ syntax. It does not need to figure out what class they are a member of. In fact, it might simply use naming conventions, grepping for a string in a function name, like VoidFuncTest 
void 
VoidFuncTest_foo(void) {
or a string in a class name, like 
class ClassWithTestMethod_ComplexEqualityTest
or similarly with comments 
void 
test_foo(void)  // extract test VoidFuncTest
{
or 
class 
ComplexEqualityTest?// extract test ClassWithTestMethod?
: public arbitrary_test_case
The TestCollector script can then simply generate 
testsuite.add( voidfunc_to_testcase( test_foo, "test_foo" ) );
testsuite.add( cwtm_to_testcase( new ComplexEqualityTest?, "ComplexEqualityTest?" ) );
with the suite arranging to call the function, or instantiate the class as an object and call the test method.

I.e. the tests cases need make no assumptions about the test suite that is running them. The test suite can wrapperize arbitrary code. This promotes peaceful coexistence and makes it easier to automate writing adapters.

Location of Test Code Where should the test code be located? There are several common ways:

Test Output Verbosity As noted above, positive confirmation of tests is good.

But different test suites have different standards for test output. E.g. some may expect to see messages such as "TEST PASSED", and others "test completed successfully". Such small differences in test output make peaceful coexistence of test suites harder.

One advantage of TestCollectors? is that they may be able to adapt one test case to any test suite. The test suite driver then would be responsible for generating output in the format it desires.

If this is done, then we want the test cases to be relatively quiet: 

void test_complex_equality(void) {
TEST_ASSERT( Complex(1,2) == Complex(1,2) );
}
and not 
void test_complex_equality(void) {
cout << "TEST STARTING: test_complex_equality\n";
if( Complex(1,2) == Complex(1,2) ) {
cout << "TEST ENDING SUCCESSFULLY: test_complex_equality\n";
}
else {
cout << "TEST ENDING UNSUCCESSFULLY: test_complex_equality\n";
}
}
The test driver will also be responsible for handling exceptions.

What should you do if you want to wrapperize or adapt a set of test cases that is verbose?

MORAL: write silent tests. Let the rest suite handle printing and exceptions.

TestAssertions

Most suites use TestAssertions. The assertions may themselves be verbose, and/or record how many have been run, a form of TestInventory.

Handling Memory Access Errors

Something you might want to consider is whether the test framework can trap memory access errors in the tested code and report them as test outcomes in the GUI or text trace. CppUnit cannot do this - a segfault will abort the entire test run. The 'check' library for C runs tests in a separate process and can therefore detect crashes caused by bad pointer usage, etc.

AndyGlew responds:

I've done exactly this, running tests in a separate process, for all sorts of errors that may cause the program to abort.

Note: this is particularly useful to do in a regression test. You can indicate "Bug #2333 that caused a segmentation violation is probably still present".

I have also found separate process debugging to be useful for non-object oriented C code. It may not be possible to create multiple instances of the C data structures in a program. But, fork a separate process, exec if necessary, and you can test them and verify that they haven't been broken, as you refactor them into objects.

However, if you don't bother with separate process debugging, take care to look at the TestSummary? and/or TestInventory. I have been burned by tests that silently crashed after one sub-test was finished but before the next had started, or in a subtest that had not announced "Test starting". There was no apparent error, just a lack of failure messages. TestPositiveConfirmation (e.g. the bar turning green) is always a good idea.

TestingPrivateMethodsInCeePlusPlus

Some people may debate (1) whether private methods should be used at all, and (2) whether it should be necessary to test private methods, or whether the whole class should be exercisable from the public interface.

Nevertheless, if you want to, here's a discussion of some ways to do so: TestingPrivateMethodsInCeePlusPlus.

TestSpeed

The topic TestSpeed describes some techniques to manage the amount of time tests take to run.

TestIntegrationWithBuildSystems?

Another good criteria is to see if each test framework integrates well with your build system.

CategoryCpp CategoryTesting CategoryFramework

EditText of this page (last edited May 17, 2013) or FindPage with title or text search