Cee Unit Testing

(Insert Seussian jingle here)

[...] while I can see how to test to see if a CpuSimulator? (or a BytecodeInterpreter?, for that matter) is doing what it's supposed to, I don't see a good way of testing that it isn't doing anything it's not supposed to (stuff like extra writes to weird memory areas and rewriting the interpreted code and whatnot).

-- AlastairBridgewater

If you could figure that out you'd have solved the halting problem, which would certainly change the face of the universe. On the other hand, for the actual set of problems you list, the easiest step is to invest in one of the many hardcore bounds-checking packages. I emphasize hardcore here: cheap packages that can't test without relinks are only worth it if you can't find anything else. For all the basic Win32 languages (all C++ flavors, Delphi, VB, and check their web site for what's new), I can't recommend BoundsChecker highly enough. That's by Numega (http://www.NuMega.com I believe), with whom I have zero affiliation. If you have to roll your own, look into ways to hook into the dynamic memory allocation system, and learn how to overallocate w/sentinel bytes as a first step.

-- MichaelHill

Actually, I was referring specifically to writing things it wasn't supposed to within the problem domain. That is, if I have an 8-bit CPU like the 6502, I don't want an STA $04 to write to both $0004 and $0204 or something. I also remember having to convince someone that his trick of having an instruction rewrite itself and its operand depending on if it was pointing to RAM or ROM or I/O was dangerous (self-modifying code would break it). It's not so much solving the halting problem as avoiding a 64k memcmp() per test or similar. I think I see how to do it now, though.

As for BoundsChecker, I develop on Linux and FreeBSD and I have no money for tools (because it keeps getting spent on books). If I want anything it has to be free, otherwise I write it myself or do without. -- AlastairBridgewater

Alastair, how about if you log every memory read and write, then compare the log with your expected memory changes? --KentBeck

I'm actually doing it the other way around. I assign a value to every memory location that will be accessed, and if any other location gets accessed the test program screams bloody murder. I should probably come up with a way to mark memory as read-only, though. -- AlastairBridgewater

I've done lots of C and C++ Unit testing by embedding the test code in the source code of the modules I'm testing.

Use "#ifdef DEBUG" / "#if defined(DEBUG)" to make the testing code appear only in DEBUG builds.
Use "assert()" statements to test results.
Use statically initialized arrays of structures to hold sets of test cases to run.

To clarify the last point:

    struct TestCase {
        char *pzInput;
        char *pzExpectedOutput;
    }
    static struct TestCase asTests[] = {
        { "Input String", "Expected Output" },
        { "Another Input", "Another Output" },
        { NULL, NULL }    /* EOF */
    };

Then, in a DEBUG-only function, I loop through the elements of asTests[], calling the function I'm testing and assert()-ing that results are what I expect. (I'm lazy, and like to "NULL-terminate" my arrays, rather than doing sizeof() computations to determine the number of elements.) -- JeffGrigg

When I write UnitTests for my C code I just use #ifdef DEBUG/#endif around a main function in the .o file. Then I include a test target in the applications Makefile that does a ${CC} -DDEBUG ... and runs the resulting executable. make will automatically quit should any program it runs fail.

-- AndraeMuys

Andrae, I don't like that style. Instead, I like to keep the main() (or WinMain() or whatever) in it's own separate file. Then if I want the unit test, it is a separate executable that can be managed independantly. Or even shipped with the final product as a platform tester! And the code being tested can have/not have debugging or other instrumentation turned on (e.g. BoundsChecker, Code Coverage, etc.)

This really helps with larger projects. If you have to build a LIB/DLL, accidently leaving a main() in the .o is a real pain to ContinuousIntegration.

Finally, when building Debug/Release/Whatever, it ALWAYS builds the unit tests as well. Period. No extra compilation step is needed to make the stubs, and I am testing the actual code that is being delivered. (Which QA groups really like) Sure, I end up with lots of little executables. tstx1, tstx2, etc. But those are sure easier to debug than the whole App.

This process surprised me by uncovering, then being able to test for different compiler bugs for cross-platform projects. That was over 12 years ago. Today: Are you absolutely sure all JVM's are created equal?

--Richard Vireday (richard@vireday.com)

Another approach to unit testing in C: Build most of your code as a library (or set of libraries), and build the code for your application's main() into a small excutable which links against the library.

Now, write a test suite application which also links against the library. Your test suite can be as simple or as elaborate as you like, but make that it returns 0 from main() if all tests pass, and returns 1 otherwise. I've found it's very useful for your test suite to handle command line arguments (for controlling the verbosity of the suite's output, or to run just a particular test or group of tests). Try to write tests for each component of the application in isolation (if you're having trouble breaking the application into isolated components, that should be a cause for concern).

Add a 'test' target to your build, which compiles and runs the test suite. Once you have your code set up like this, it's very easy to get into the "tweak the code, run the unit tests" kind of flow. You just need to make some edits to the code, and then run 'make test'. If any tests are broken, back out your last change, or try to fix them. Run 'make test' again. Repeat until all tests pass.

Another benefit of the test suite is that it should be deterministic (multiple threads notwithstanding), so if the test suite fails, asserts or crashes - it should fail the same way every time, a 100% repeatable error. (My personal favourite kind of error :) So just fire up your debugger on the test suite, run it till it dies, and try to figure out what went wrong. If you mess things up in the debugger too badly, just run again from the start. Then it's back to: edit the code, 'make test' and repeat...

''Another approach to unit testing in C: Build most of your code as a library (or set of libraries), and build the code for your application's main() into a small executable which links against the library.''

I HaveThisPattern too. This is one of the modular ways of building full applications that has worked pretty well for me. Whenever I have to build a larger program that performs several tasks, I often start by thinking in how many (reusable) small libraries I can break it. Then I go off into the test or regression sub-tree and sketch a few UnitTests for the public functions of the library. Part of the design process for the unit tests usually reveals what these public functions should be too. So I tend to write the header file (or multiple header files) for the library at about the same time the unit tests are beginning to get shape and a mostly usable form.

When the UnitTests are close to becoming usable I start the loop of running the "make test" or "make check" target of the project and fixing the tests I can fix "easily". The PositiveFeedbackFirst effect of seeing the first unit tests pass often makes me feed good about the project and the work I've done so far. This tends to work as a powerful incentive to keep going. Admittedly this last part is probably a flaw of my character, but it's there and I know it's there so I am taking advantage of it.

Another good thing about writing larger programs as lots of tiny libraries that call each other is that it tends to lead (at least me) down the path of thinking a bit harder about the functional parts of a program and their interactions. The extra time it takes to split something in smaller functional blocks and delegate these to small libraries usually helps me with the overall design of the program.

Finally, if a CeeLanguage program is constructed by a playful LegoToy-like combination of many small parts, these very same parts can be used again in other projects later—including their accompanying UnitTest's. This is a GoodThing as it allows me to re-use the work I've already done a dozen times, without having to go through the same TrialAndError process and repeat the same mistakes all over again. LifeIsShort enough as it is, without having to waste large gobs of our precious little time re-doing what we have already accomplished. -- GiorgosKeramidas

CategoryCee CategoryTesting