Poor Mans Testing Framework

Please allow me to set something straight:

You don't need a fancy-schmancy TestingFramework to UnitTest [or do TestDrivenDevelopment].

The only tools you need to write a Poor Man's Testing Framework are: That's it. Anything on top of that is just gravy. Now stop being mystified by unit tests and just do it!

Let me walk through this to make sure I'm clear:

Pretend you're writing multiply() (I don't know why you would do this, but it's the example everyone uses, so bear with me.). Let's work a poor man's unit testing example in CeeLanguage, knowing that we can apply most of the same techniques in other HighLevelLanguages.

The first test looks like this:

 int main() {
if (6 != multiply(2, 3))
puts("6 != multiply(2, 3)");
return 0;
 }
Attempt to compile and run it. We get a failure, of course (because we haven't written multiply()).

Jot a multiply() down (hard-coding the result to get rid of the evil "Red Bar" as quick as possible):

 int multiply(int a, int b) {
return 6;
 }
Compile and run. Good. (SilenceImpliesConsent in the Poor Man's Testing Framework).

Now then, before we move on, we look around for duplication. Sure enough, we see it, right there in main(): the "6 != multiply(2, 3)" is repeated once. Honestly, I can't think of a clean way of getting rid of it yet, but since it's only one duplication, I'll just forge ahead and see if I can't figure out a way to fix it after the next test.

Now, we hard-coded the "6" as the return value. Better add another test to bring out the need for real code inside multiply():

 int main() {
if (6 != multiply(2, 3))
puts("6 != multiply(2, 3)");

if (42 != multiply(6, 7)) puts("42 != multiply(6, 7)");

return 0; }
Compile and attempt to run. Of course, it fails. So we flesh out multiply:
 int multiply(int a, int b) {
return a * b;
 }
Compile and run. Good, it passes. So let's get right to the refactoring.

First of all, I hope you noticed the violation of ThreeStrikesAndYouAutomate in "compile and run". Hopefully, by this point, you would write a Makefile or equivalent to automate this step. From now on, I'll just say "run".

Next we need to get to that icky pile of duplicated conditions filling our main() function. Let's pull it out:

 void assertMultiplication(int expected, int a, int b) {
if (expected != multiply(a, b))
printf("%d != multiply(%d, %d)\n", expected, a, b);
 }

int main() { assertMultiplication(6, 2, 3); assertMultiplication(42, 6, 7); return 0; }
Run tests. Pass. Good. However, we're not sure if the tests actually worked the way we expected. So, let's break our implementation for a moment:
 int multiply(int a, int b) {
return 1234;
 }
Run tests, expecting two failures. Good, we got them. Now, let's hit our editor's undo button until we get back to the correct implementation. Re-run tests, expecting pass. Good, got it.

When we write the test cases and implementation for divide() (again, I have no idea why we need these functions), we'll end up with something like this (skipping ahead a bunch):

 void assertMultiplication(int expected, int a, int b) {
ASSERT_EQUALS("multiply", multiply, expected, a, b);
 }

void assertDivision(int expected, int a, int b) { ASSERT_EQUALS("divide", divide, expected, a, b); }

#define ASSERT_EQUALS(functionName, function, expected, a, b) \ int actual = function(a, b); \ if (expected != actual) \ printf("%d != %s(%d, %d)\n", expected, functionName, a, b);

After I wrote this, I realized that I wasn't outputting the "actual" number anywhere. Fortunately, since we refactored, all I had to do was change the one printf:
printf( \
"Expected %d from %s(%d, %d), but got %d\n", \
expected, functionName, a, b, actual \
)
Baby steps, running the tests, refactoring in between. That's it! Eventually, we won't have a Poor testing framework at all.

(Note: Of course, you'd probably want to check out an actual unit testing framework for your language after you've bootstrapped. I just want to make it clear that you don't have to have one to get going.)


Hey poor, you don't have to be poor anymore. PMTF is here! (Sorry, couldn't help myself)


The impoverished of spirit may seek a hand up from OnceAndOnlyOnce.


Too much work. diff expected.output actual.output

I've done this one on some projects. But then you have to output everything. Internal subroutine calls and comparisons give you much better control. -- JeffGrigg


Thanks, you! This is a really good simple example of how you can create simple tests for your language. Of course, you can go to the TestingFramework but this piece of explanation is really good for newbies.

Thanks !!!! -- Fabrice


Good start. But the missing ingredient is ThreeStrikesAndYouAutomate. The code may be minimal, but at error time you still must...

OnceAndOnlyOnce does not cover programmer activities, such as looking at the file name of the failure, looking at the line number, memorizing them, opening the file in the editor, and going to the line. But this too must be automated.

The next step bonds with your editor. VisualCeePlusPlus contains a minimal CppUnit knock-off that obeys both OnceAndOnlyOnce (roughly) and ThreeStrikesAndYouAutomate (aggressively).


I use the C preprocessor 'stringize' operator to remove the "6 != multiply(2, 3)" duplication:

 #define ASSERT_EQUAL(e, a) if ((e) != (a)) printf("%s != %s\n", #e, #a)
 #define ASSERT(e) if (!(e)) puts(#e)

int multiply(int a, int b) { return 0; }

int main(int ac, char **av) { ASSERT(6 == multiply(2, 3)); ASSERT_EQUAL(10, multiply(2, 5)); return 0; }
-- PhilippeAntras

Way! -- PhlIp


The beef I have with poor man's Testing (not that I don't do it in time of need) is that automation of the entire testing suite must be maintained by hand as you go. When there must be manual maintenance of a list of tests, more effort is required, fewer tests will be written, and mistakes will be made. It's better if the act of writing the test automatically informs a testing framework of the existence of another test to run. For my compromise, see VbLiteUnit. VbLiteUnit may be inelegant in that it keeps all the tests for a test case in one big procedure, but it has the advantage of not requiring (sometimes unreliable) heroics to identify the tests in a language that doesn't do introspection well, and it still does have sufficient safeguards against common copy/paste errors, etc. -- SteveJorgensen


For small projects or for algorithm experiments, separate test files using a full-blown UnitTest framework can be overkill. PythonLanguage's built-in "assert" is handy for creating poor-man's tests in the same file as the code. I often use "assert" to write some tests which run when the module is called as the main program. The tests are designed to pass silently and fail noisily. I can even do TestDrivenDevelopment this way. An example:

 #!/usr/bin/env python

def factorial(n): if long(n) != n: raise ValueError("factorial arg must be integer: %f" % n) if n < 0: raise ValueError("factorial arg must be >= 0: %d" % n) if n >= sys.maxint: raise ValueError("you must be crazy to want %d!" % n) fact = 1 for i in xrange(2, int(n)+1): fact *= i return fact

# # # # # # # # # # # # # # # # # # # #

if __name__ == "__main__": import sys, math

for input, expected in (0,1), (1,1), (2,2), (3,6), (4,24), (5,120): assert factorial(input) == expected assert factorial(float(input)) == expected

big_arg = int(math.sqrt(math.sqrt(math.sqrt(sys.maxint)))) assert factorial(big_arg) > sys.maxint

for bad_input in -5, 3.14, -3.14, 0.2, -0.2, sys.maxint, sys.maxint+1: try: factorial(bad_input) print "factorial failed to raise error on bad argument value!", bad_input except ValueError: pass except: print "should never get here!", bad_input

If this particular file is supposed to be called as main, then I encapsulate the tests. I put the tests in a _test() function and call it only when I pass "-t" on the CommandLine.

-- ElizabethWiethoff


I HaveThisPattern. Unit tests and wiki share the fact that the concept is more valuable than the codebase. Here is the UnitTest framework I wrote (and I use) for MicrosoftAccess VisualBasic. I throw an error, and use the debugger to find where the error occurs. -- StanSilver

 '===========
 'Test Engine
 '===========
 Sub Assert(blnDesiredTrue As Boolean)

Dim intMakeError

If blnDesiredTrue = False Then intMakeError = 1 / 0 End If

End Sub

Sub AssertNot(blnDesiredFalse As Boolean)

Dim intMakeError

If blnDesiredFalse = True Then intMakeError = 1 / 0 End If

End Sub

Function RunAllTests()

'Execute all functions that are defined in the "mdlTests" module 'If an error occurs, use the debugger to find the line that makes the error AllProcs ("mdlDevTests")

End Function

'From Microsoft Access Help 'ProcOfLine Property Example Function AllProcs(strModuleName As String) (lots of code here to run all procs in a file...)

'===== 'Tests '=====

Function TestAge()

Assert (Age(#7/15/1959#) = 46) Assert (Age(#1/15/1959#) = 46) Assert (Age(#1/1/2000#) = 5) Assert (Age(#12/1/2000#) = 5)

End Function

Function TestFormatTime()

Assert (DevClassFormatTime("10 AM", "dummy") = "10:00 am") Assert (DevClassFormatTime("10PM", "dummy") = "10:00 pm") Assert (DevClassFormatTime("10:20PM", "dummy") = "10:20 pm") Assert (DevClassFormatTime("10:30 am", "dummy") = "10:30 am") Assert (DevClassFormatTime("9:00 PM", "dummy") = "9:00 pm")

End Function


CategoryDefensiveProgramming


EditText of this page (last edited June 24, 2006) or FindPage with title or text search