About Me

Hiep Dang

Hiep Dang
Hiep Dang is the director of operations for McAfee Labs. He is currently in charge of opening a new research & ...

Read More

Feeds & Podcasts

Blogs

Meet the Bloggers

Archive

Tags

#SecChat $1 million guarantee 12 Scams of Christmas access to live fraud resolution agents Acquisition Alex Thurber Android antivirus Apple botnet Channel Partners cloud security Compliance Consumer counter identity theft credit card fraud and protection credit fraud alerts credit monitoring credit monitoring and resolution critical infrastructure Cyber Security Mom cyberbullying Cybercrime cybermom data breach data center data center security Data Protection Dave DeWalt DLP Email & Web Security embedded encryption Endpoint Protection enterprise facebook fake anti-virus software Family Safety Friday Security Highlights global threat intelligence google government Hacktivism how to talk to kids how to talk to teens identity fraud identity fraud scams identity protection identity protection $1 million guarantee identity protection fraud identity protection surveillance identity surveillance identity theft identity theft expert identity theft fraud identity theft protection identity theft protection product Identity thieves and cybercriminals intel iphone kids online behavior lost wallet protection malware McAfee McAfee Channel McAfee Family Protection McAfee Identity Protection McAfee Initiative to Fight Cybercrime McAfee Labs McAfee security products Mid-Market Mobile mobile malware mobile security monitor credit and personal information Network Security online personal data protection online safety Operation Aurora PCI personal identity theft fraud personal information loss personal information protection phishing privacy proactive identity protection proactive identity surveillance Public Sector restore credit and personal identity Risk and Compliance scam scams scareware security smartphones social media social networking social networks spam Stuxnet twitter vulnerability Web 2.0 work with victim restore identity

What a "Tangled" Web…

Sunday, August 12, 2007 at 9:14pm by Hiep Dang
Hiep Dang

We’ve been receiving inquiries about a recent anti-virus comparative test that was performed at LinuxWorld by a start-up network gateway vendor named Untangle. Their goal was to prove that open-source anti-virus solutions (in this case ClamAV) were just as effective, if not better than commercial anti-virus products. It seems that they were highly motivated to prove this because evidently they use ClamAV in their gateway product (more on this conflict of interest later).

As with any comparative test involving McAfee, we analyzed the results and testing methodology used. Here is a basic breakdown of the testing that was performed by this vendor (their presentation can be found here):

  • 10 anti-virus vendors were tested (ClamAV, FProt, Fortinet, Global Hauri, Kaspersky, McAfee, SonicWall, Sophos, Symantec, Watchgaurd)
  • 35 samples were used (6 EICAR samples, 12 from Untangle, and 17 user-submitted samples)
  • It appears they performed an on-demand scan of the sample set.

Before delving into the many problematic facets of this test, if you would like a quick primer on comparative tests, please read “Comparing the Comparatives”. We recently discussed “(Mis)interpreting Reviews” here. For an introduction into false statistical interpretation go here.

Now let’s take a look at the flaws in the methodology used in this test:

  1. Blatantly False Results - We ran our own scan on the exact same files and our results showed we detected everything that was not a password-protected zip or 0-byte file. How many other AV vendors’ results were wrong? I’m sure I’ll soon find out from my counterparts.
  2. Small Sample Size – One of the first rules of Statistical Method is using a large enough sample size for your test in order to accurately represent the entire population. In this case, they used only 35 samples of the hundreds of thousands malware samples in the wild today.
  3. Biased Samples - The fact that 12 of the 35 samples came from the CTO of Untangle’s mailbox negates the fact that they used a random sampling that accurately represents the true population of malware samples.
  4. Comparing the Wrong Products – The test compares 5 Linux, 2 Windows, and 3 Gateway products. This is like comparing apples with oranges with kumquats.
  5. Misconfigurations of Vendor’s Products – The CTO admits in his blog that “In fact, one audience participant significantly improved one vendor’s performance, Sophos, by pointing out that I needed to add a command-line option. Others pointed out mistakes I made recording results.” What other misuses occurred during this test?
  6. Conflict of Interest – One of the first rules of comparative tests is that it needs to be performed by an arbitrary third party with no vested interests in the outcome of the results. The fact that this test was performed by Untangle who develops, markets, and sells an anti-virus solution with their gateway product is a blatant example of a conflict of interest.
  7. Improper Handling & Distribution of Viruses – By offering a link to these live viruses on their company’s public website, they are in violation of the Computer Fraud & Abuse Act which prohibits the distribution of computer viruses because it is endangering public safety. There is a reason why only trained security professionals should handle computer viruses. I have already contacted them to remove the links. Note: If the link is not removed, SiteAdvisor’s webcrawlers will automatically identify the company’s site as red due to the malicious link.

Needless to say, the methodology used in this test was riddled with flaws. So if anyone thinks this test proves anything, we have $14 million in a Nigerian bank account waiting to be transferred to a US account. ;-)

* Rather than adding to the confusion of proper anti-virus comparative testing, those in the industry are working towards a better solution. McAfee Avert Labs recently participated in an Anti-Virus Testing Workshop in order to define more robust testing methodologies.

Bookmark and Share

Submit your own comments / message for this post

Your email is never published nor shared. Required fields are marked *

 

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comments (12)

  • jeffs January 14, 2008 2:59PM

    I think it’s amazing that McAfee took all this time to say “we still don’t know”. Don’t they have the resources to simply pick any number of samples and do their own test? Surely they have done that already. And surely they would have mentioned it here… but only if it showed that ClamAV was *not* more effective !

  • Ffoeg December 19, 2007 9:28PM

    I’ve been using AVG free edition and Spybot S & D for a few years and haven’t ever detected a single virus with either of them because I DON’T DOWNLOAD AND RUN THEM TO BEGIN WITH!!! Who cares what AV you use or which one works 2% better. If you pay for one it will cost more. If you download every stupid thing your friends send you and open every file you can find your computer is going to get hosed regardless!

  • agus budianto November 14, 2007 9:50PM

    hi, i am using mcafee in my office, and it works with all the virus so far, so i have no disappointment with mcafee at all, just for information.

  • Beto October 13, 2007 2:58PM

    Well.. I have a conclusion: Kaspersky and NOD32 are the best antivirus.

    McAfee used to be good in the V4.0.3 times… now is bloatware, with low detection rates…

  • Dave September 29, 2007 5:32AM

    McAfee’s inability to detect viruses is why I no longer use Mcafee. So much got through that my Windows operating system was corrupted to the point that it was useless. I scanned all my documents with Clamav, backed up any that weren’t infected, formatted the drive, reinstalled Windows, partitioned the drive and installed Linux Fedora. I just don’t let Windows have any internet access. To make sure I unplug the cable when I use it. Since then no problem. Unfortunately there are a couple programs I still need Windows for. Those are Sonic Foundry’s Acid and Windows Video Maker (just for ease of use). I’m sure there are programs out there that do similar things but I have many Acid files that I still need to edit and I haven’t found a Linux video program that allows me to edit the audio track as easily.

  • idbeholda September 28, 2007 11:24PM

    There are several problems with the aforementioned logic. I will now systematically pwn all vendors for everyone’s enjoyment.

    Blatantly False Results (lol) – Anyone can run results on an arbitrary archive, and the results will always be different. This is precisely why all test fail under the scrutiny of logic 101. If memory serves me correctly, the idea was to have a *RANDOM* sampling of a few files submitted by the audience as a whole. To an extent, this is a statistically better idea, as it keeps the vendors on their feet to the fact that at some point in time, no two archives or submissions will be identical. Theoretical Practice. When developers understand this, they’ll stop failing so hard.

    As for the password protected or 0-byte files. This can be counteracted in two ways. Skip the 0-byte files to begin with unless the filesystem on that particular install allows (and has enabled) resource forking like the ADS for NTFS on windows NT based systems. Password protection? Not an actual threat because in order for the files to be extracted one has to know the password to begin with. Problem is, no security vendor will decide to add in brute force password cracking on protected archives because of the so-called “ethics” behind it.

    I believe the euphamism you’re looking for is “in order to defeat hackers, you must think and act as one of their own.” This cannot be argued.

    Biased Samples (zLOL)- Look into the idea of honeypotting/honeynetting. These are more accurate representation of a statistical enumerations of a large population than it is of an archive. Remember, no two archives are identical. Lest we forget Mr. Bontchev’s article regarding the pitfalls of “in the wild” lists. That was something even I have said for years and nobody else listened. In any regard to that, any sort of sampling submission is ultimately flawed and biased *TOWARDS* the particular vendor providing the samples.

    Comparing the Wrong Products (facepalm.jpg)- Like everyone is going to have the same operating system installed? TRY AGAIN. Databases should be interpreted/factored to the same degree on any system regardless of what is installed.

    Misconfigurations of Vendor’s Products (lmfao… sophos sure got told, didn’t they?) – This is why vendors need to fully publish the requirements of command-line arguments for their scanners. If you don’t fully disclose any critical information regarding a scanner functions, how can you reasonably expect it to work flawlessy in an arbitrary test?

    Conflict of Interest (????)- The samples were taken from audience members. If you don’t consider each audience member as a statistical third, fourth, fifth, etc party, how can you be expected to pass basic math? At this point, somebody needs to buy a vowel from Pat Sajak.

    Improper Handling & Distribution of Viruses (PROFIT)- You’ve not only managed to misquote the CFAA, but you’ve simultaneously managed to misquote it out of context, and then further compounded the issue by discrediting the entire test by basically quoting the metaphorical fine print after the fact. Secondly, since the samples were provided by audience members, the beginning parameter of the argument invalidates itself by default. Since the “public” was the audience, exactly how does the public endanger itself?

    Further more, if one were to actively search for the link, that would inherently imply that they are security and technologically savvy to begin with, provided they’re not a script kiddie looking for an easy way to “distribute malware”, which again, defeats the purpose altogether, as there are several well known sites (basically the entire internet) that in some way, shape, or form allows public access to said material in the first place. Putting a lock on a glass display case is pointless, as the glass itself can be broken: Learn first before you start touching.

    To misquote the CFAA out of context is completely inexcusable, and shows not only a blatant disregard for logic, but also education, basic reading comprehension, and a complete understanding of fantasy .vs. reality. It also does nothing more than imply that the detractors involved are nothing more than racketeers themselves, and ultimately are no better than the malevolent users they are trying to stop. Some information may be slightly dangerous to the wrong individual, but if everyone knows the same information, the threat will “mysteriously” vanish.

    As a final note, this is not simply directed at one company/vendor, but all of them. I say this, because I can guaruntee at least 5-10 years (maybe more) down the road, I will ultimately be proven right. Do we have any bets on this?

  • dp September 23, 2007 8:54PM

    The methods are published – we can decide how valid they are as well as you can. We may not agree with you as you might suspect. You lost in this test – get over it. Go find somebody else’s test that has published methodologies and see how you do, but quitcher bitchin.

  • bobo September 22, 2007 6:46AM

    LOL!??!?!?? is there someone who still buys something from McAfee??!?!?!?!?!?!?
    :) haahah

  • ooo September 21, 2007 10:51PM

    5 Linux, 2 Windows, and 3 Gateway products.

  • g0d0t September 20, 2007 5:03AM

    http://www.clamav.net/ has something to say about that :

    http://www.clamav.org/2007/09/20/different-views-on-av-testing-methodology/

    Quote :
    “While the methodology in this test has been debated, we believe that all tests should be as open to review as the Untangle test was!”

    Indeed it seems funny to criticise a methodology used in a test (even justified) while not pubishing detailed methodology for your own tests.

  • Mario Rossi September 6, 2007 3:15AM

    Considering that SiteAdvisor, up to two weeks ago, was detecting clamav.net as yellow that threat sounds pointless

  • Mike August 17, 2007 1:26AM

    Yub. Fully agree. I wrote similar things at my personal weblog.
    http://weblog.vircop.org/?p=52