Browser debate: Did Microsoft cheat?

When Microsoft released the seventh Platform Preview of Internet Explorer 9 last Wednesday, Microsoft said that the browser had overtaken its competition in terms of JavaScript performance. Microsoft based its statement on the results of the well-known WebKit SunSpider JavaScript benchmark: according to Microsoft's tests, IE9 PP7 came out on top.

The dramatic JavaScript performance improvements of IE9 over the browser's version 8 (which is the only current browser that doesn't have a compiling engine) soon raised the suspicion that Microsoft might have cheated to achieve better benchmark results. Mozilla developer Rob Sayre found that making minimal modifications to one of the individual tests (math-cordic) in SunSpider had huge effects on the required computing time in IE9 – but not in Chrome or Opera.

On the official IE blog, project leader Dean Hachamovitch hastened to provide clarification: Internet Explorer 9 apparently uses the "dead code elimination" compilation trick, which apparently had an impact on the test in question. The Mozilla developer's modifications apparently cancelled out this optimisation. Sayre criticised the dead code elimination implemented by Microsoft as an approach that can behave unexpectedly, or produce flawed results.

To get a better idea, The H's associates at heise Open ran tests on a work laptop under Windows Vista and compared the JavaScript performance of IE9 with that of Firefox 3.6.12, Chrome 7, Safari 5.0.2, Opera 10.63 and the IE9 beta released two months ago. In addition to SunSpider, the Google V8, Peacekeeper, SlickSpeed and Speed-Battle JavaScript benchmarks were also run.

IE9 PP7 did indeed complete SunSpider slightly faster than Opera and Chrome, with only Firefox coming in behind in this tightly-packed field. However, IE9 couldn't sustain this result in the other tests: In V8 and SlickSpeed, Chrome came first, in Peacekeeper it was Opera, and in Speed-Battle it was Safari. In V8, SlickSpeed and Peacekeeper, IE9 PP7 came in second-to-last before Firefox (whose version 4 will offer a further optimised JavaScript engine). However, the results are close; the current Platform Preview offers consistent improvements over the beta version.

Our conclusion can only be that none of the current JavaScript engines clearly outperforms the others. All current browsers except IE8 and perhaps Firefox 3.x are doing very well. On the IEBlog, Microsoft repeatedly points out that a browser's JavaScript performance is only a small aspect of its overall performance, and that, in Microsoft's opinion, such tests as SunSpider are "at best not very useful, and at worst misleading."

(crve)