Page 80 -

2.5 / PERFORMANCE ASSESSMENT 55

execution times. But this is not inversely proportional to the sum of execution times.
In other words, the arithmetic mean of the instruction rate does not cleanly relate to
execution time. On the other hand, the harmonic mean instruction rate is the in-
verse of the average execution time.
SPEC benchmarks do not concern themselves with instruction execution
rates. Rather, two fundamental metrics are of interest: a speed metric and a rate met-
ric. The speed metric measures the ability of a computer to complete a single task.
SPEC defines a base runtime for each benchmark program using a reference
machine. Results for a system under test are reported as the ratio of the reference
run time to the system run time.The ratio is calculated as follows:
Tref
r = i (2.5)
i
Tsut i
where Tref is the execution time of benchmark program i on the reference system
i
and Tsut is the execution time of benchmark program i on the system under test.
i
As an example of the calculation and reporting, consider the Sun Blade 6250,
which consists of two chips with four cores, or processors, per chip. One of the SPEC
CPU2006 integer benchmark is 464.h264ref. This is a reference implementation of
H.264/AVC (Advanced Video Coding), the latest state-of-the-art video compres-
sion standard. The Sun system executes this program in 934 seconds. The reference
implementation requires 22,135 seconds.The ratio is calculated as: 22136/934 23.7.
Because the time for the system under test is in the denominator, the larger
the ratio, the higher the speed. An overall performance measure for the system
under test is calculated by averaging the values for the ratios for all 12 integer
benchmarks. SPEC specifies the use of a geometric mean, defined as follows:
n 1/n
r
r = a q i b (2.6)
G
i=1
where r is the ratio for the ith benchmark program. For the Sun Blade 6250, the
i
SPEC integer speed ratios were reported as follows:
Benchmark Ratio Benchmark Ratio
400.perlbench 17.5 458.sjeng 17.0
401.bzip2 14.0 462.libquantum 31.3
403.gcc 13.7 464.h264ref 23.7
429.mcf 17.6 471.omnetpp 9.23
445.gobmk 14.7 473.astar 10.9
456.hmmer 18.6 483.xalancbmk 14.7

The speed metric is calculated by taking the twelfth root of the product of the
ratios:
(17.5 * 14 * 13.7 * 17.6 * 14.7 * 18.6 * 17 * 31.3 * 23.7 * 9.23 * 10.9 * 14.7) 1 12 = 18.5
The rate metric measures the throughput or rate of a machine carrying out a
number of tasks. For the rate metrics, multiple copies of the benchmarks are run si-
multaneously. Typically, the number of copies is the same as the number of proces-
sors on the machine.Again, a ratio is used to report results, although the calculation