Critics say that Microsoft compares apples and browsers in its security study

We did not have to wait long for criticism of Microsoft's study, which claims that Internet Explorer is safer than Firefox. First and foremost, Mozilla's security head Window Snyder reported a critical hole in Microsoft's measurement procedures. Above all, she criticizes the mere counting of flaws. As she puts it, Mozilla documents and remedies bugs as quickly as possible, whereas companies like Microsoft do not list flaws that they find internally and remedy before they become public in their statistics. In other words, from the outside you only see security holes that become public.

Snyder also harshly criticizes Microsoft for taking so long to release the next Service Pack or major update to remedy vulnerabilities discovered by external service providers in penetration tests. She writes that users often remain unprotected for a year or longer, allowing attackers to localize and exploit the problem in the meantime. Snyder probably knows what she's talking about; after all, she was Senior Security Strategist at Microsoft up to 2005.

Her colleague Mike Schroepfer, Firefox's Development Director, also chimed in with figures provided by independent security provider Secunia showing that users of Internet Explorer have been regularly exposed to unpatched security holes in the past few years, while there were only two brief such periods for users of Firefox.

On the other hand, another possible explanation for the surprising findings in the Jones study has hardly even been mentioned: the way Microsoft categorizes vulnerabilities in various products. After all, Redmond does not necessarily categorize everything that endangers users of IE as a vulnerability in Internet Explorer. Unfortunately, Jeff Jones did not publish a list of the vulnerabilities counted, nor did he specify the exact criteria used when, for instance, a vulnerability is assigned to Internet Explorer.

When contacted by heise Security, Jones said that he did not, for instance, include the URI vulnerability as an Internet Explorer 7 problem because the flaw was actually part of the Windows function ShellExecute(). Jones pointed out that the problem was also not attributed to Firefox, but in the next sentence he said that patch MFSA 2007-27 - Un-escaped URI is passed to external programs was included in the statistics because it has its own CVE number. As a result, Jones attributed a critical vulnerability caused by a security flaw in Windows to Firefox, but not to Internet Explorer, even though the flaw was not possible to exploit unless Internet Explorer was installed.

It is also not clear whether Jones included the February patch for HTML help in his statistics for IE. Jones has yet to respond to heise Security's query on this matter. Although the patch technically does not remedy a vulnerability in Internet Explorer, the flaw mainly concerns users of IE. Microsoft even says so itself in its security advisory on the flaw, answering the question how an attacker could exploit the vulnerability:

An attacker could host a specially crafted Web site that is designed to exploit this vulnerability through Internet Explorer and then persuade a user to view the Web site.

Indeed, ActiveX controls are a weak point in the study. At least up until IE 6, under the default settings any website can call all ActiveX controls installed on a Windows system, which repeatedly causes security problems. Jones reassured heise Security that he counted the vulnerabilities in ActiveX components provided in IE.

Furthermore, Microsoft includes multiple flaws in a single patch for ActiveX components, such as in MS05-038/CAN-2005-1990. The CVE entry counts 17 COM objects for this patch – and a full 32 in CAN-2005-2127. Jones has yet to explain whether he took the trouble of counting all 49 flaws in full or merely the two patches. In light of his tally for Internet Explorer 6 SP2, in which he found a total of 50 critical vulnerabilities, it is more likely that he only counted a single CAN/CVE entry in those two cases.

But despite all the criticism, Jeff Jones has once again fulfilled his actual mission: he managed to get the public interested in the claim that Microsoft's Internet Explorer is safer than Firefox. No doubt, Jones will see this as a success in its own right.

(mba)