Programmer Test

XP terminology, not quite synonymous with UnitTest. (See Also DeveloperTest)

Failure of a UnitTest shall implicate one (1) and only one (1) unit. (A method, class, module, or package.)

Failure of a ProgrammerTest, under TestDrivenDevelopment, implicates only the most recent edit.

http://www.XProgramming.com/xpmag/whatisXP.htm#test

Often a ProgrammerTest is better than a comment, to help us understand why a particular function is needed, to demonstrate how a function is called and what are the expected results are, and to document bugs in previous versions of the program that we want to make sure don't come back.

ProgrammerTests give us confidence that after we improve one facet of the program (adding a feature, or making it load faster, ...), we haven't made some other facet worse.

If it's worth doing, it's worth testing to make sure it was done right.

I'm starting to understand why ProgrammerTests are so important. When there's a couple of useless-looking lines in the code, with some cryptic comment about how we "need to do this or else it doesn't work" ( VoodooChickenCoding ), but I take them out and it still seems to work - the UnitTests help explain why we need it, they show that while it may work in some cases, but fails other cases. However, I still find it difficult to write UnitTests for my own code. Let's take the example in the ThingsYouShouldNeverDo article - a bunch of crufty-looking code to handle 60 kinds of FTP servers. How in the world do I UnitTest that code? I suppose I could write unit tests that download a short file from 60 different servers -- but those unit tests would give a false error if one of those servers went off-line. Or worse, server number 60 could upgrade to the same software used by server number 33, then when I remove the code that was absolutely necessary to get my software working with the software that used to be running on server number 60, those unit tests would give a false pass.

Of course, the right way to do it would be to write a simulator for each of the FTP servers, or for each of the distinctive characteristics they have. This is the yin and yang of coding, the same way every server security hole has a corresponding exploit. (See NeverWriteaLineOfCodeWithoutaFailingTest)

What you would do is put together a very trivial simulator, which might run as a separate process or simply as a replacement for the function that reads a line from the FTP server. You would put together sixty simple "scripts", one for each type of FTP server: client says A, server says B, client says C, ... If the client says anything other than its next line, it fails the test; otherwise the simulator replies with the next line for the server.

That's what unit testing is all about: you test each unit IN ISOLATION.

Another possibility would be to perceive the whole network as the system to work on. Then the test actually connects to the FTP servers, and it can be made succeeding by fixing either client or server.

See HistoryOfProgrammerTest

What process should I follow if all my ProgrammerTests pass, but my code does not work? Obviously, once I find the error I should write a test that exposes it, but how do I get to that point? Should I fire up a debugger? Start logging? Write random tests? --WayneMiller

You should inspect the process by which you wrote those tests. TestDrivenDevelopment's outer cycle is frequent review and manual program massaging. That was the discovery process to write more tests. If you do BigTestDrivenDevelopmentUpFront, you will merrily TDD many bugs and dead-spots into your code. PairProgramming also helps defeat the DarkSide.

If, however, your code still doesn't work, and a small amount of debugging can't find the problem, throw the code away and start again. --PhlIp

How does one know his code does not work short of it failing to provide an expected result based on a specified input? Known input and expected output is the definition of a test in a nutshell, whether one chooses to automate the test or run it manually is a separate issue.

''When one's code provides an unexpected result, one should use all the tools at one's disposal to determine why the unexpected result occurred. Review the tests written for the calling code, why do they pass?

Review the calling code, is it doing something not tested? Use a debugger, if that is one's preference. Put in message boxes or log files or flash lights on a front panel. One piece of advice, though, once one identifies the condition causing an anomalous result, add an automated test for that condition before fixing the problem.''

My thought here would be in the case of a customer-discovered bug, where the bug is reproducible but the user IO is not practical as a developer test (user input is much too large, and the job takes far too long). Throwing the code away is not really an option, because it is not yet evident what code is wrong. Sounds like the opinion is to solve with traditional techniques and mark afterward with a test? -WayneMiller

Close. The consensus seems to be to discover the location of the bug in the code with traditional techniques, then write a failing test, then fix the code, and finally make sure the test passes. NeverWriteaLineOfCodeWithoutaFailingTest

Usually, after you find the bug, you can figure out a small ProgrammerTest which will make the code fail. Yes, some long sequence of user input is needed to trigger the bug. But generally the bug is small and easily triggered with a defined set of conditions. You can create those conditions and call the offending code much more easily than re-creating the full sequence of user input. That's your test. --EricHopper

Don't test through the user interface. The common approach is to have a separate test executable that has the production code files linked into it. An alternative is to add the test code to the production code and have the test mode launched through a special command or a command line switch. You may choose to expose this to the end user or not. Typically programmer tests will be written at a lower level of access than exposed to the end user. This avoids many of the issues of large amounts of set up and job run time, and with an automated program, well computers are real good at doing the repetitive grunt work to set up a test condition. The advice within programmer tests is always to write a test, individuals, however, are always free to ignore this advice.

--- 1/14/05 If I know that those tests pass, I should use the tested code exactly as written, to provide only the functionality I tested them for. I should then think of the new criteria that my code generates, knowing that if those other tests pass, then it must be the thing I added that doesn't deal with what those other tests are testing. For example, suppose tests for "add user" and "remove user" pass within the same app scope, but if I "add user, remove user" made up of identical code, that fails. My very first thought should be that they work separately when I click one and then the other, but not when executed in immediate succession in code. My tests were flawed because they implied some time delay by user actions. If I am using code for extended purposes besides what the tests that code pass on, no matter how minor that extension, then I need new tests, or the functionality is not guaranteed to provide the expected results. One should strive to have numerous failing tests, to demonstrate the precise limits of a function, versus one passing test, demonstrating a special case where the functionality does work.

If, however, your code still doesn't work, and a small amount of debugging can't find the problem, throw the code away and start again... This reminds me of a phrase seen often in PlanNine discussions: BuildTwoYou'llThrowOneAway -- the TemporalPrimeDirective doesn't seem to care about the order of the builds and the throws. -- ChrisGarrod

CategoryTesting