Looking for benchmark test tool 2321

dbone I used to work on the Bell Labs optimizer for the C compiler. Our marketing department gave us a bunch of about a dozen benchmarks to run so we could prove our compiler, running code for our processor, was faster and better than the compebreastion. We could easily do this.

But the thing is that those benchmarks were pathetic and showed lack of understanding. About half of them were copied from BYTE Magazine (IIRC), and the editor who though them up had no clue what compilers did. So they tended to have a loop executed 10,000 times that did what they thought they were testing. But I wrote a Loop Invariant Code Motion optimization that noticed if the same thing was computed each time through a loop, and if so, it moved it out of the loop. Thus the test was run only once instead of 10,000x. Then my partner's Live-Dead Analysis program noticed that the stuff outside the loop, and the loop code itself, were dead after being computed (i.e., never used) so it eliminated them too. Thus a benchmark that the editor thought measured something 10,000x executed in about 12 microseconds (we had 10 MHz processors in those days). USELESS.

There was a benchmark called Whetstone that was appropriate in the 1960s, but is still used today. It is thought to measure floating point performance, which, with a decent optimizer, it does not. That benchmark was designed to measure all the heavily-used code types generated by the Whetstone Algol-60 Interpreter. One of the test modules, the only one with any floating point computation, was used to measure call and return overhead. It worked by calling a function 10,000 times (or so). Well, my partner's Inline Function Expansion expanded that function inline (so it no longer tested what it was meant to test), and then my LICM moved the computation out of the loop, after which Live-Dead Analysis removed the last reminant of that test. So any decent optimizer (we are talking about mid 1980s, by the way) would ensure that no floating point computation was done at all in that benchmark.

We had another idea for making better benchmarks. Our company had a large System-360 or the Amdahl equivalent running UNIX and we just profiled the thing to see which programs were using the most processor time for a week. We then took the top 10 of those (no surprise: troff headed the list). We then took those 10 programs and re-coded them to do no IO but to get their data from built-in tables, and write their output to built-in tables.

Those were more real benchmarks for the load our main-frame UNIX systems were running. Unfortunately, our optimizer could not improve them by factors of 10000:1 like is the stupid benchmarks provided by our marketing department. We did improve them some. But we had to compete with our compebreastion, and they used the BYTE benchmarks, too, so we clobbered them. But not in sales. People did not believe our tests, though the tests were OK on the benchmarks provided. Well, Bell Labs is no longer in the computer business in any way. That is what happens when bean-counters reign supreme.

-- .~. Jean-David Beyer Registered Linux User 85642. V PGP-Key: 9A2FC99A Registered Machine 241939. ^^-^^ 21:50:01 up 94 days, 11:23, 4 users, load average: 4.40, 4.44, 4.21