Testing security suites in real-life conditions is essential

Anti-malware security suite testing receives a lot of attention, but it's important to appreciate that different testing methods can produce vastly different and even misleading results.

A decent security suite is greater than the sum of its parts: it's designed in layers to thwart various attacks at various stages. Testing individual components of a security suite on their own - or testing in an environment that doesn't reflect real-world conditions - often fails to give an accurate picture of a security suite's overall performance.

"Think of layered security like Swiss cheese," says Lloyd Borrett, Security Evangelist for AVG (AU/NZ). "Each layer can have holes in it, but if you stack the slices together they cover each other's holes. Testing one layer in isolation doesn't accurately reflect the performance of the stack."

A case in point, AV-Comparatives recently conducted proactive/retrospective tests of antivirus products for heuristic and generic detection using on-demand scanning. In other words, a test to see how well security suites perform when faced with previously unseen threats. No definition updates are downloaded for a defined period of time, during which malware samples are still collected. An on-demand scan is then carried out to see if the antivirus product detects the new malware.

It is important to understand that the report specifies clearly the limitations of the test methodology. It states, "Some products maybe had the ability to detect some samples eg, on-execution or by other monitoring tools." So in other words, many security suites were forced to take the test with one arm tied behind their backs - hardly a reflection of real-world performance.

Similarly, the German publication c’t Magazine recently published AV-Test.org's results from pitting security suites against 'Zoo' malware - a collection of malware samples stored on a hard drive. Unfortunately, samples stored in this way do not reflect real-world conditions; if a piece of malware can only infect a machine through execution in a browser, then it should only be tested that way. It seems unfair to mark down a security suite for failing a test that it will never face in the real world.

To ensure security suites are tested in real-world conditions, AVG actively promotes the changing of test methodology through its membership of the Anti-Malware Testing Standards Organisation (AMTSO - www.amsto.org). AMTSO is an international non-profit association which focuses on improving the objectivity, quality and relevance of anti-malware testing methodologies. Membership is open to academics, reviewers, publications, testers and vendors.

AMTSO recently published two sets of guidelines that, for the first time, set recognised standards for testing security software. The guidelines are part of a series of documents on AMTSO's website aimed at introducing a set of standards broadly agreed throughout the industry as appropriate for ranking the effectiveness of security software.

AMTSO's 'Whole Product Protection Testing Guidelines' advocates a more balanced look at the effectiveness of products, taking into account multiple layers of detection and protection. "This Guidelines document marks an important step in developing tests which accurately measure how an entire product actually functions when exposed to threats," says Dr Igor Muttik, representing McAfee. "Too many current tests focus on individual technologies, such as 'On Demand Scans'. Only by testing all of a product's protection capabilities in a comprehensive test can one provide a more realistic view of the security offered to computer users by contemporary security suites."

AMTSO's 'Performance Testing Guidelines' highlight how easy it is for testers to overemphasise irrelevant metrics by implementing poorly conceived benchmarking methodologies. "It is very tempting to take a simplistic approach to measuring the speed and footprint of an antivirus program," says Mikko Hypponen, Chief Research Officer of F-Secure. "However, there is as much art as there is science in understanding the various elements which can skew the results for the unwary tester. This document will help testers understand these issues and allow them to take the necessary steps to minimise them and take them into account."

The new guidelines are voluntary, but many testing organisations have agreed to go along with them, such as Virus Bulletin, AV-Comparatives, West Coast Labs, ICSA Labs and AV-Test.org. Hopefully, AMTSO's bipartisan push for improved testing standards will see security suites tested in real-world conditions to give an accurate picture of how they perform at the coal face.

"Today's internet threats from hackers and cybercriminals are both transient and rapidly evolving," says AVG Technologies' CTO Karel Obluk. "To combat them, security products must also evolve. As such, real-time testing of security products is critical, and it takes both reactive and proactive technologies to protect against today's threats."

*Lloyd Borrett, Security Evangelist for AVG (AU/NZ)