Many Facets of AV Testing

Following the news from my colleague Dr. Igor Muttik about his recent trip to Bilbao, Spain, to participate in the Anti-Virus Testing Workshop, AV-Test.org just released the results of their latest comparative test. It was picked up by many media outlets:

1. PC Magazine (USA)
2. Dark Reading (USA)
3. PC Welt (Germany)
4. CHIP (Germany)
5. Security.nl (The Netherlands)

Unlike in many previous reviews, AV-Test.org ran various types of tests, and McAfee scored well in most of them:

• We are pleased that we made the most progress of any vendor from AV-Test.org’s last test, which was published by c’t magazine (Germany) earlier this month. Our detection-rate improvement was +7.3%.
• We are proud that we did not detect any false positives. (We are one of only three vendors that can make that claim.)
• We received the second-best rating in the Rootkits test.

Signature-Based Tests are usually an on-demand scan (ODS) by anti-malware products on a computer system against a set of known malware. We have discussed the challenges in making this test fair in the past.

Proactive Tests are similar to signature-based tests, except that they attempt to measure how well an anti-malware product can detect samples that it has never seen before–by taking an old DAT version and scanning with malware that was discovered after the DAT release date. This test often gives a sense of how well an anti-malware vendor does in writing generic, heuristic, or behavioral signatures. The caveat with this is that if a product ventures too far into this realm, the likelihood of false-positives increases.

False-Positive Tests are also an ODS test, except with a sample set of clean files instead of malicious files. False positives are the bane of the anti-malware industry as they could have far worst collateral damage than a false-negative (missed detection) depending on the severity. Because of our large customer base, we take this metric very seriously and have an internal zero-tolerance policy.

Rootkits Tests are one of the most complex and time-consuming tests that a tester can run, and are similar to the behavioral tests described above. However, these require even more intimate knowledge of both the target operating system and known rootkit techniques to accurately judge whether an anti-malware product was able to properly remediate the rootkit infection.

Response Times tests attempt to determine how quickly an anti-malware vendor responds to a new threat with their definition updates and heuristic detections.

Individually, each of these tests gives us a way to gauge one of the many facets of measuring the value of an anti-malware product. However, when grouped together, they can give a holistic picture of how well we balance the many criteria by which we are judged.