A program frozen at the point of failure detection may have already erased the evidence that explains the failure.
Therefore, trap each statement of the program and record crucial information in a CircularBuffer. Inspect this buffer when the program halts to determine the circumstances leading to the failure.
The concept is similar to the flight voice and data recorders recovered from the wreckage of airplane crashes. Information is easy to capture, but, since failures are rare, all but the most recent records are discarded.
I used a variation on this concept to hunt bugs in multi-microprocessor hardware and software. After each statement in the program I mechanically inserted ...
I would step through a loop once watching each statement go by. The second time through the loop I would remove the breakpoints as I went. Then, zip, I would be out of the loop, at the next statement's breakpoint. A quick check of the buffer would make sure the final passes through the loop were as well behaved as the first.
This may all sound trivial compared to the heroics that modern debuggers will perform on your behalf. Where it saved me was when the systems crashed in bizarre ways. I was never left without a clue. And, I could trust the clues because of the absolute simplicity of the debugging tool. -- WardCunningham
You're my hero, Ward. I have probably used plenty of stuff you touched over the years. -- MartySchrader, long time and extensive user of Tektronix instruments, development systems, and software
Hugh Blair-Smith Describes the same technique used to debug the ApolloGuidanceComputer? in the 60's.
Another function served by these auxiliary erasable memories was called the "Coroner," which recorded every instruction executed, with its inputs and results, writing over the oldest such record when it filled up. When a program "crashed," you could punch out a full record of what it was doing in most of its last second and analyze the problem at your ease. I have often wished that PCs offered such an advanced feature.