Benchmarks Are Evil

Are they?

No.

Comparing performance can be useful, but not if it's a goal in and of itself. I benchmark my programs against earlier versions of themselves, to see if changes to algorithms help improve overall performance. I don't worry about benchmarking my code in comparison to other code, unless my goal is to write a program that does a given function more quickly than an existing solution.

I think this more or less falls under StatisticsLie.

What's the alternative? I've put together some hand-rolled benchmarks before using scripts, macros, and duct-tape to run "typical" apps. They may not be thorough tests, but they are tests that the vendor couldn't pre-fudge for.

An example of this (for reference) is: http://blogs.zdnet.com/Ou/?p=322

The summary: PC enthusiasts are paying twice the money for a slower hard drive with a quarter of the storage capacity

Except the author didn't actually benchmark the results of partial stroking before making said summarized conclusion. According to one response to the article, the cost of a seek is dominated by the cost to start, stop, and settle the head; however, the author assumed it was dominated by traversal (i.e. that seeking 1/4 the distance results in 1/4 the seek time). According to the respondent, the benefits of partial stroking in his experience amounts to 15%-20%.

EditHint: Can this page be merged into BenchMarking? (Especially since neither makes a strong case that bench marks are "evil", just that they have limited usefulness.)