Problems Testing Performance Simulators

I work on performance simulators - programs which estimate how fast a new computer hardware design is going to be. The problem with performance simulators is that there is no single right answer - there are only reasonable answers. For example, it is semantically correct for a cache to always miss, just not very good performance. It is possible to test for the expected performance behaviour of something like a 4 way associative cache with LRU replacement on a machine that processes requests in-order. It is much more complicated to test for the performance behaviour for a 4 way associative cache with pseudo-random-LRU-biased behaviour, with an MRU quick-return policy, that is capable of responding to requests out-of-order and in a non-blocking manner. Also, the hardest performance simulator bugs deal with interacting features, such as when a memory subsystem composed of several caches works for all component cache parameters *except* when the greatest common denominator of all cache associativies is equal to the pipeline depth. (Again, this probably means that the hardware design is too intertwined, but that's another issue.)

The problem is that performance estimates, e.g. cycle counts, are the bottom line, but that cycle counts will change whenever a feature is added or tweaked. This will mean that a test that evaluates the most important result, cycle counts, will change its expected answer whenever a feature is addeds or tweaked during normal program development, if the unit under test is being tested in conjunction with any other real units. If the unit under test is being tested only in conjunction with mock units that do not change during normal program development, the tests endure, but it inspires significantly less confidence in me. Similarly, if the unit tests do not test the most important results, it inspires less confidence.

I suspect that these characteristics of my problem domain, performance simulation, that make detailed tests hard, also are present in some other problem domains.

---

Because of such ProblemsTestingPerformanceSimulators I occasionally use the TemporaryDetailedTestingSupportingRefactoring technique to increase confidence during refactoring.