Testing By Poking Around

Testing by poking around is a term invented by Robert V. Binder in his paper Object-Oriented Testing: Myth and Reality (http://www.rbsc.com/pages/myths.html). Although I don't much like the paper (the myths are too unbelievable to be worth debunking), I do like his term. Testing by poking around means asking a program to do a few things, and if it does them well then it's tested. This is NOT a method of TacticalTesting.

Unfortunately, testing by poking around is one of the standard testing techniques in use these days. There has to be a better way, please help me find it. -- JohnFarrell

Poking around becomes more useful, though still far from ideal, if you fill the code with assertions. -- MartinPool (See also: ExceptionsAsConstraints, DesignByContract)

I have seen some of Binder's myths in action. It may seem incredible, but these myths do need debunking. -- DonWells

Much code is so bad that TestingByPokingAround will in fact find plenty of problems.

Smalltalk-80 was once distributed as a virtual machine specification and a memory dump of the development environment. Vm implementors would measure progress by seeing how many instructions they could go before failing. Once the environment came up, one would just poke around. This worked amazingly well, except for the few machine operations that weren't used in the environment itself, such as less than for negative floating point numbers, which operated incorrectly for many months of heavy use. In fact, the BitsOfHistory? book included a cartoon to this effect: a port was complete when one could type 2+3 and evaluate it. Never mind that the result was 6.

Similarly, I once attended an all-day workshop on testing. Most speakers held up GlenMeyer?'s bible on the subject before offering their particular slant. What most people didn't realize was that Glen had a company across the parking lot and they were porting a dos kernel to their own particular hardware. So I went over at lunch and asked the head of the os group how they did their testing. He said, we just boot Windows and poke around a little bit. -- WardCunningham

On the subject of ST-80, is it still possible to obtain such a memory dump? I for one would like to try playing around with one. -- AlastairBridgewater

Could we interest you in SqueakSmalltalk, instead?

I'm not sure whether this is what was meant by TestingByPokingAround, but I see it in operation when testing is mixed in with debugging and even "watching the code with a debugger to see how it works" (even when it's your own code). This sort of testing is time consuming and often unrepeatable, but it is what is often used early on in software (where UnitTests are not already present) as a first pass at testing/debugging. A little bit of this sort of testing is reassuring and I'd forgive anybody for working in this way. Spending too much time on it, however, could result in UnitTests not happening at all (alas, all too common). -- RichardDevelyn

This is a very common testing method on projects with no formal methodology. "Let's just try it and see what happens." Sometimes it almost works, when the users of the software don't deviate from the ways they have been trained. There is almost always something, however, that surfaces and bites you later. -- RobertField

TestingByPokingAround is like statistical sampling, only that it isn't completely random and the bugs are not completely uncorrelated. The tests are informally structured based on the know-how of the testers and the requirements of the project. And one bug discovered this way can lead to a single fix that affects many externally evidenced problems. What is beta-testing but TestingByPokingAround?

The problem lies in assuming a program is bug free after it has passed through this phase. But the same problem is inherent in any test regime. There is no way of proving a program bug free, and throwing random-search out of your bag of testing tricks would be a mistake.

ExploratoryTesting is a lot like TestingByPokingAround, except that it is much more than just "trying out a few things" and if you don't find any errors, you failed. The purpose of ExploratoryTesting is to find bugs, while the purpose of regression testing is to make sure that you haven't added any. ExploratoryTesting should be done by testers, who should have been trained in how to discover bugs in odd corners of a program. -- RalphJohnson

I use a variation of TestingByPokingAround all the time, but I prefer to think of it as OneShotTesting (in contrast to AutomaticTesting). After I implement some functionality, I make a mental list of all the cases involving the functionality and try them all out.

AutomaticTesting is better than OneShotTesting, since it detects CodeRegressions?, but it also takes 100 times as long to implement. Exhaustive OneShotTesting combined with selective AutomaticTesting seems like the most efficient way for me to develop high-quality code. -- JaredLevy

TestingByPokingAround is great, absolutely necessary and anyone who avoids it is nuts! UnitTests and all that are fine and dandy but who's testing the test writer? Get someone from another department, the UPS man, whoever and have them use your app. My 20 month old daughter can make lots of apps crash faster than you can hit Ctrl-S.

When I was working on an AppleTalk stack and file server for the PC we passed all the applicable automatic tests that the Apple guys themselves gave us. The real test, however, was to run all the popular Mac apps and do goofy things. You wouldn't believe how many bizarre bugs popped up because the Mac app writers didn't do things as the AppleTalk spec expected. --AndrewQueisser

Anybody ever consider keeping a log of the poking around? Works great in console apps, or when the 'poking around' involves poking around in a ReadEvalPrintLoop, in that the same checks can be applied, or you can simply check for identical output. Not perfect, but if the log is editable, it's a good start to get people TestInfected. --WilliamUnderwood

CategoryAntiPattern TestingPatterns