Few security products gain certification at the first attempt

Just four per cent of the security products tested by independent tester ICSA Labs are certified on the first attempt. The figure comes from a Product Assurance Report produced by the US company, which is now part of Verizon. ICSA Labs are know for their stringent, in-depth, standardised tests and guidelines for anti-virus solutions, firewalls, VPNs and intrusion detection systems. The report evaluated thousands of certification processes spanning the last 20 years.

According to ICSA Labs, the main reason for non-certification was in many cases the failure of a product to fulfil its core functionality – for example, anti-virus programs which failed to prevent infection, or intrusion detection systems which failed to detect attacks. Big differences were seen between different product categories. Whereas for SSL VPNs, only 36 per cent of problems were due to failures relating to core functionality, for web application firewalls (WAF) and intrusion prevention systems (IPS) the figure was 100 per cent. For anti-virus solutions, 85 per cent of rejected products failed to offer adequate protection. Nonetheless, 27 per cent of anti-virus products passed muster at the first attempt – in stark contrast to WAFs and IPS which, according to the report, almost never pass without some improvements having to be made.

As well as core functionality, ICSA also tested logging functions, whether the product itself contained vulnerabilities, the installation process, interoperability, patching, operation and reliability. According to the report, logging functionality problems were frequently found the first time a product was submitted for testing. Despite the fact that deficits in this area are particularly problematic for firewall applications, 97 per cent of firewalls failed to carry out reliable logging.

According to the report, 82 per cent of all products pass the certification process at the second attempt. Typically two to four attempts are required to pass. The report notes that one reason for requiring multiple attempts is a lack of vendor-side quality assurance and the pressure to get products with specific functions to market quickly.

(crve)