Processor Whispers: About benchmarks and sore losers
by Andreas Stiller
AMD, NVIDIA and VIA Technologies pull out of the benchmark consortium BAPCo and thereby present IT procurement managers with problems. No suggestions for a better benchmark for measuring the GPGPU technology were made. And Intel postpones the Haswell processor to 2014.
That's the way it goes with benchmarks. If you are the number 1 on the Top500 list, Linpack is great. If you are somewhere further down, its significance is very limited. SYSmark, probably the most important Windows applications benchmark, is a similar case. With a lot of hoopla, the CPU and GPU/APU developers AMD and VIA have now cut their ties with the BAPCo – and we're not talking about the Irish schnapps company here, but about the industry consortium Business Applications Performance Corporation responsible for SYSmark. NVIDIA apparently dropped out quietly a few weeks ago; in any case, it was already missing from the BAPCo's members list in May. Meanwhile, however, the names Sandisk and Apple have also stopped appearing in the BAPCo announcements...
Has AMD, in particular, done itself a favour by leaving? According to AMD, the new SYSmark 2012 fails to correctly illustrate the performance of modern computers: it ignores the potential of graphics processors (GPUs) while at the same time giving too much weight to typical CPU tasks like data compression and OCR, as these are "rarely if ever" utilised by the average PC user today.
Change of Mind
AMD's change of mind over SYSmark comes as a surprise. Not only had AMD been involved in the development of the benchmark explicitly intended for the evaluation of office PCs, but it also had the procurement managers of public sector entities in Europe commit themselves to make their calls for bids for PCs and Notebooks product-neutral. To this end, at the beginning of 2006, AMD explicitly recommended SYSmark 2004 SE "for its accuracy, objectivity, simplicity, and widespread adoption" in its procurement guidelines. The then-current Athlon 64 easily beat the Pentium 4 in the SYSmark benchmark. However, in 2006, Intel started to turn the tide with the Core 2 Duo – and in 2008, AMD criticised the SYSmark 2007 Preview: "Most of the applications SYSmark 2007 Preview uses are more expensive, graphics-intensive applications like Adobe Photoshop, Flash, and Illustrator."
Today, on the other hand, AMD's head of marketing Nigel Dessau finds fault with the benchmark because it "virtually ignores the parallel processing performance of the GPU. In particular, SM2012 scores do not take into account GPU-accelerated applications that are widely used in today’s business environments." As examples, he names Internet Explorer 9, "Office 11" and Adobe Flash – among others.
AMD intends to bring to life an open and transparent benchmark – Nigel Dessau didn't provide any more specific information, leaving unanswered the question of how one can now compare the performance of GPUs concerning general-purpose computational tasks. Current GPGPU benchmarks partially fail to deliver satisfying results because of the highly varying GPU functionality beyond conventional 2D and 3D acceleration. For instance, the GPU part of the Llano APUs doesn't support dual-precision floating-point operations, just like many of the cheaper NVIDIA chips.
That just programming a GPGPU benchmark is difficult throws into sharp relief the development of "real" software for the heterogeneous machines of the future that AMD and NVIDIA are betting on. And maybe not everyone shares the belief that GPU-accelerated software is "widely used in today’s business environments". Even if future applications require more GPU performance than current DirectX 10 compatible onboard graphics can deliver, it's much easier to retrofit a PC with a graphics card than to upgrade a system's main processor. Many buyers of typical company PCs that are supposed to serve for a rather long time will probably give more importance to the CPU performance than to that of the GPU – all the more so as hardly anything is predictable about the requirements of future GPGPU software.
In any case, in the end, there's the suspicion that AMD primarily criticises SYSmark 2012 because the CPU performance of the desktop PC Llano is disappointing. In spite of the supposed advantages of the 32 nm technology – not only smaller but also high-k dielectric plus metal gate (HKMG) structures – the performance increase in comparison to the 45 nm Athlons is marginal. It can only be hoped that everything will look better with Bulldozer, which, at least in the high-end desktop sector, won't have to rely on GPU performance – but not everyone is optimistic about that either.
Meanwhile, new rumours about the next AMD CEO have surfaced. Ex-CEO Hector Ruiz is unlikely to return: he has just founded the management consulting firm Bull Ventures LLC. Oracle's vice president and former HP boss Mark Hurd has signalled indifference, just like Apple's COO Tom Cook. Then, the ball went to EMC²'s president and COO Pat Gelsinger, who, for many years, played a decisive role as development manager, technology strategist, and manager of the business unit at Intel and who had, among other instances, expressed his feelings toward the then-competitor with his "I hate AMD" password. Gelsinger would be a doozie for new bets – but he has already, and decidedly, said "no" and "no" again. Theo Valich of Bright Side of News already knows who AMD's next CEO is going to be: someone from IBM, but he is unwilling to reveal a name yet.
At least concerning manufacturing technology, Intel is a big step ahead of its competitor, but the ambitious metronome is now slowing down a bit. As could be heard through closed doors at the edge of the ISC11, Intel has postponed the launch of its Haswell processor to 2014. The old "Tick-Tock" plan from 2008/2009 indicated 2012; at the release of the vector extension AVX2, a few weeks ago, the news was 2013 and everyone dutifully reported "Haswell on Track for 2013" – but now it's apparently going to be 2014. Whether the processor itself or its infrastructure is to blame for the delay is anyone's guess. For instance, iSuppli has reported a delay concerning its DDR4 RAM, which is not going to roll out before 2014. However, it wouldn't make sense to bring it to the market before the corresponding Intel chip has been launched, anyway.