McAfee Labs

Testing and Accountability

By on Jul 07, 2010

The Anti-Malware Testing Standards Organization (AMTSO) is a coalition of security professionals, including many anti-malware product vendors, product testing organizations and publishers, and some interested individuals. Given the highly technical nature of its activities, it is inevitable that the organization owes some of its authority to the expertise of the security specialists within its ranks, but that doesn’t make it a vendor lobby group. As Kurt Wismer (not himself a member) points out here ( “many of them are employed by vendors precisely because that’s one of the primary places where one with expertise in this field would find employment.” Given some recent negative publicity aimed at AMTSO (example:, we want to collectively clarify the following points on behalf the anti-malware industry, where we come from, and indirectly on behalf of AMTSO.
We find it strange that expertise in the testing field is somehow seen as a disqualification, given the specialist expertise that characterizes the group.

Although some distrust anything a vendor says and accept uncritically anything a tester says, others are puzzled that different tests can vary so dramatically in their evaluation of the same product. Though this may sometimes be simply due to poor testing practices, there are other, deep-seated reasons, one being the high volume of malware and new attacks seen every day. Vendors work hard to close the gap between the ideal of 100 percent detection and what is actually achievable–by developing a range of technologies, both proactive and reactive. The capabilities of products can change, while tests using broadly similar methodology can generate dramatically “conflicting” results due to different approaches to the selection, classification, and validation of samples and URLs, among other factors.

AMTSO aims to promote precisely the kinds of tests that clearly demonstrate these variations, and its members were flying the flag for real-world testing before AMTSO ever formally existed, believing that sound testing benefits vendors and customers as well as testers. As an industry, we are all too aware that we cannot currently offer detection of all known and unknown malware. The relatively high scores achieved in established tests by major vendors do not necessarily reflect real-world performance, but real-world detection cannot be measured in product comparisons with no checks on selection, classification, and validation of malicious samples and URLs.

Another misconception is that AMTSO members simply don’t like tests done by non-AMTSO members. This is not the case: None of the undersigned have a problem with labs that intend to provide objective, real-world testing. (However, other testers are entitled to object vehemently when one company claims to be the only one doing live, Internet-connected testing, and that all other testers are doing static testing based on the WildList.)

However, charging consultancy fees for the release of any information relating to a test (even to participants) is very different to the transparency that AMTSO advocates, although we recognize that full-time testers must generate revenue like any other business. However, when a tester claims to have shared information about methodology in advance and subsequently fails to provide methodological and sample data, even to vendors prepared to pay the escalating consultancy fees required for such information, this suggests that the tester is not prepared to expose its methodology to informed scrutiny and validation. This stance compromises the tester’s aspirations to be taken seriously as a testing organization in the same league as the mainstream testing organizations committed to working with AMTSO.

No one believes that AMTSO has all the answers and can “fix” testing all by itself, but the organization has compiled and generated resources that have made good testing practices far more practicable and understandable. The way for testers (and others) to improve those resources is by talking to and working with AMTSO in a spirit of cooperation: The need for transparency is not going to go away.

Roel Schouwenberg, Kaspersky Lab
Luis Corrons, Panda Security
David Harley, ESET
Mark Kennedy, Symantec
Igor Muttik, McAfee