Statement Trap Handler

A program frozen at the point of failure detection may have already erased the evidence that explains the failure.

Therefore, trap each statement of the program and record crucial information in a CircularBuffer. Inspect this buffer when the program halts to determine the circumstances leading to the failure.

The concept is similar to the flight voice and data recorders recovered from the wreckage of airplane crashes. Information is easy to capture, but, since failures are rare, all but the most recent records are discarded.

I used a variation on this concept to hunt bugs in multi-microprocessor hardware and software. After each statement in the program I mechanically inserted ...

• A call on a trap handler
• the source line number
• a pointer to a table of in-scope variables

As inserted, the calls were to a subroutine that would freeze the program, record and display the line number, and wait for interactive requests which included ...

• display in-scope variables
• disassemble code for the statement (to find compiler bugs)
• examine system message queues
• display the buffer of recorded line numbers
• modify the trap to call a more passive handler

In essence, the program started with a breakpoint on every statement and this last interactive command removed them. Except it didn't remove the breakpoints completely. It changed them to breakpoints that silently recorded the progress of the program in the circular buffer.

I would step through a loop once watching each statement go by. The second time through the loop I would remove the breakpoints as I went. Then, zip, I would be out of the loop, at the next statement's breakpoint. A quick check of the buffer would make sure the final passes through the loop were as well behaved as the first.

This may all sound trivial compared to the heroics that modern debuggers will perform on your behalf. Where it saved me was when the systems crashed in bizarre ways. I was never left without a clue. And, I could trust the clues because of the absolute simplicity of the debugging tool. -- WardCunningham

You're my hero, Ward. I have probably used plenty of stuff you touched over the years. -- MartySchrader, long time and extensive user of Tektronix instruments, development systems, and software

---

Hugh Blair-Smith Describes the same technique used to debug the ApolloGuidanceComputer? in the 60's.

Another function served by these auxiliary erasable memories was called the "Coroner," which recorded every instruction executed, with its inputs and results, writing over the oldest such record when it filled up. When a program "crashed," you could punch out a full record of what it was doing in most of its last second and analyze the problem at your ease. I have often wished that PCs offered such an advanced feature.

http://hrst.mit.edu/hrs/apollo/public/blairsmith3.htm