McAfee used to be good in the V4.0.3 times… now is bloatware, with low detection rates…

Blatantly False Results (lol) – Anyone can run results on an arbitrary archive, and the results will always be different. This is precisely why all test fail under the scrutiny of logic 101. If memory serves me correctly, the idea was to have a *RANDOM* sampling of a few files submitted by the audience as a whole. To an extent, this is a statistically better idea, as it keeps the vendors on their feet to the fact that at some point in time, no two archives or submissions will be identical. Theoretical Practice. When developers understand this, they’ll stop failing so hard.

As for the password protected or 0-byte files. This can be counteracted in two ways. Skip the 0-byte files to begin with unless the filesystem on that particular install allows (and has enabled) resource forking like the ADS for NTFS on windows NT based systems. Password protection? Not an actual threat because in order for the files to be extracted one has to know the password to begin with. Problem is, no security vendor will decide to add in brute force password cracking on protected archives because of the so-called “ethics” behind it.

I believe the euphamism you’re looking for is “in order to defeat hackers, you must think and act as one of their own.” This cannot be argued.

Biased Samples (zLOL)- Look into the idea of honeypotting/honeynetting. These are more accurate representation of a statistical enumerations of a large population than it is of an archive. Remember, no two archives are identical. Lest we forget Mr. Bontchev’s article regarding the pitfalls of “in the wild” lists. That was something even I have said for years and nobody else listened. In any regard to that, any sort of sampling submission is ultimately flawed and biased *TOWARDS* the particular vendor providing the samples.

Comparing the Wrong Products (facepalm.jpg)- Like everyone is going to have the same operating system installed? TRY AGAIN. Databases should be interpreted/factored to the same degree on any system regardless of what is installed.

Misconfigurations of Vendor’s Products (lmfao… sophos sure got told, didn’t they?) – This is why vendors need to fully publish the requirements of command-line arguments for their scanners. If you don’t fully disclose any critical information regarding a scanner functions, how can you reasonably expect it to work flawlessy in an arbitrary test?

Conflict of Interest (????)- The samples were taken from audience members. If you don’t consider each audience member as a statistical third, fourth, fifth, etc party, how can you be expected to pass basic math? At this point, somebody needs to buy a vowel from Pat Sajak.

Improper Handling & Distribution of Viruses (PROFIT)- You’ve not only managed to misquote the CFAA, but you’ve simultaneously managed to misquote it out of context, and then further compounded the issue by discrediting the entire test by basically quoting the metaphorical fine print after the fact. Secondly, since the samples were provided by audience members, the beginning parameter of the argument invalidates itself by default. Since the “public” was the audience, exactly how does the public endanger itself?

Further more, if one were to actively search for the link, that would inherently imply that they are security and technologically savvy to begin with, provided they’re not a script kiddie looking for an easy way to “distribute malware”, which again, defeats the purpose altogether, as there are several well known sites (basically the entire internet) that in some way, shape, or form allows public access to said material in the first place. Putting a lock on a glass display case is pointless, as the glass itself can be broken: Learn first before you start touching.

To misquote the CFAA out of context is completely inexcusable, and shows not only a blatant disregard for logic, but also education, basic reading comprehension, and a complete understanding of fantasy .vs. reality. It also does nothing more than imply that the detractors involved are nothing more than racketeers themselves, and ultimately are no better than the malevolent users they are trying to stop. Some information may be slightly dangerous to the wrong individual, but if everyone knows the same information, the threat will “mysteriously” vanish.

As a final note, this is not simply directed at one company/vendor, but all of them. I say this, because I can guaruntee at least 5-10 years (maybe more) down the road, I will ultimately be proven right. Do we have any bets on this?

http://www.clamav.org/2007/09/20/different-views-on-av-testing-methodology/

Quote :
“While the methodology in this test has been debated, we believe that all tests should be as open to review as the Untangle test was!”

Indeed it seems funny to criticise a methodology used in a test (even justified) while not pubishing detailed methodology for your own tests.