Blogs
- Consumer (534)
- Corporate (162)
- Enterprise (524)
- McAfee Labs (1031)
Feeds & Podcasts
Meet the Bloggers
- Aditya Kapoor
- Anindita Mishra
- Archive
- Arun Sabapathy
- Brenda Moretto
- Brian Contos
- Carlos Castillo
- Chintan Shah
- Cindy Chen
- Consumer Threat Alerts
- Craig Schmugar
- Cybermum Australia
- Dan Wolff
- David Marcus
- David Small
- Doug McLean
- Dr. Phyllis Schneck
- DThakkar
- Ed Metcalf
- Eric Peterson
- Eric Schou
- Fernando Ruiz
- Francisca Moreno
- Francois Paget
- Gabriel Acevedo
- Gary Davis
- Gavin Struthers
- Gert Jan Schenk
- Greg Brown
- Guilherme Venere
- Haowei Ren
- Hardik Shah
- Itai Liba
- Jan Volzke
- Jim Walter
- Jimmy Shah
- Jonelle Hester
- Kevin Beets
- Kim Eichorn
- Lang Tibbils
- Lianne Caetano
- Madhurima Pawar
- Martin Pivetta
- Martin Ward
- Michelle Dennedy
- MihwaLee
- Oliver Devane
- PageOne Pr
- Paras Gupta
- Pat Calhoun
- Paula Greve
- Peter Szor
- Rachit Mathur
- Raj Samani
- Robert Siciliano
- Ryan Permeh
- Sean Roth
- Shinsuke Honjo
- Simon Cooper
- Simon Hunt
- Stuart McClure
- Tad Heppner
- Tim Roddy
- Todd Gebhart
- Toralv Dirro
- Tracy Ross
- Umesh Wanve
Archive
- May 2012 (42)
- April 2012 (38)
- March 2012 (48)
- February 2012 (46)
- January 2012 (47)
- 2011 (477)
- 2010 (429)
- 2009 (378)
- 2008 (271)
- 2007 (354)
- 2006 (143)
Tags
Anti-Virus Testing 2.0
|
|
A colleague of mine from McAfee Avert QA and I have just returned from a summit in Bilbao, Spain where more then 40 experts gathered together for almost two full days. Security researchers, QA people from many AV companies, independent AV testing bodies and magazine reviewers were present. The purpose of the meeting was to form a non-profit organization that would work towards improving testing standards for anti-malware products. We want to give help to everybody who is eager to be involved in the area of testing anti-malware security solutions. Helping computer users is what we do on a 24x7x365 basis and we very much want to promote quality independent testing because bad tests mislead, confuse, and frustrate everybody.
The necessity to create such an organization rose due to occasionally seeing AV reviews that compared apples and oranges or, sometimes, not even saying what kind of fruit were compared! It would be beneficial for everybody if there were some minimal requirements for the tests. For example, an ability to contact the reviewer is very important and so is publishing the testing methodology. If there were many competent independent testing bodies it would be ideal but, unfortunately, contemporary anti-malware tests require big well-equipped research labs and it takes literally years to set them up. Neither the governments nor academic institutions have had much success on this front so far.
The delegates agreed on the name of this organization. Many suggestions were considered – such as “The Bilbao group”, “CATS” (Computer Anti-Malware Testing Standards) and “iTOSS” (International Testing of Security Software). In the end we all settled on a rather simple and straightforward name. The formal press release will follow soon where the name and the participants will be announced.
We all also worked together on a charter of this newly formed body and it was mutually agreed upon. I would expect a Web site to be set up soon and the charter published there.
We set up temporary committees – they will be in charge of organizing future meetings, establishing new contacts, and drafting the standards. The plan of action is to meet several times a year and seek agreement on the ways to objectively compare anti-malware security solutions, taking the output of the working committees as working drafts.
The idea is to have the organization open for the anti-malware companies, the academic institutions, testing bodies, magazine reviewers and everybody, who would wish to participate in improving the standards of testing for security software.
I could hear objections along the line – “How can we trust AV industry to set standards of how they themselves should be tested? Can say, car manufacturers be allowed to set their own testing standards?” I would reply that, firstly, the security software makers are vitally interested in the existence of comparative tests – they show us our strengths and weaknesses and only then can we better decide where to invest our resources in order to improve protection and increase the number of users. Secondly, the spectrum of security software is getting wider all the time (heuristics, generics, sandboxing, behavioral protection, herd-intelligence, HIPS, NIPS, etc. – new technologies are being constantly added) and traditional focus of testing on scanning piles of files (sometimes very old piles) is stifling innovation. Improvements in AV testing are urgently needed and involving the developers of the new technologies in the discussions sooner rather then later should benefit everybody. And finally, independent testers are already represented in the new organization and it will be them actually running the tests – nobody can force them to do what they disagree with.
It was a pleasure to work together with so many talented people and I really do hope that our joint efforts would not only improve testing standards but would also stimulate innovation in testing methodologies and in the security technology.
Avert Labs will keep you updated on the developments.
|
|
Comments (5)
There are various professional companies and organizations who are only into the lab testing of antivirus programs and they work primarily for IT companies by testing the programs provided to them. So before an organization decides to implement particular antivirus software widely in the organization, they contact the lab and ask them to perform a test on the program. The lab will perform professional test cases with malware and adware programs on the antivirus and give the result to the organization.
Virus naming is always going to be a problem, unfortunately in this day and age it is impossible for AV vendors to create the same names, what is needed is a library of all the names that the vendors have put to code so that people can refer to it in order to find out what each vendor calls a specific piece of malware. While this is done in several places at present it is not as good as it should be as generally they are not updated frequently enough.
With reference to the actual article and the previous comments, todays independant testers do not do a good job at present, and it is important that someone comes up with better testing methodologies. Most tests today are of static collections and are essentially just a test of a particular vendors signatures at a particular point in time. This is no good, no really it is actually quite misleading. What needs to be created is a way of testing whether AV software can actually stop a particular (or preferably all) malware from actually infecting / compromising the system in the first place. This is afterall what AV should be doing. Whether it identifies it by this name or not is fairly irrelevant as long as the infection does not actually occur. We all know that signatures are important in identification and cleaning, but it is not necessarily the best form of protection from malware. What needs to be highlighted in tests is whether or not machines were actually protected from the word go whether it be by IPS / firewall type technologies within the the product through to generic and heuristic type detection methods.
It shoudl also be noted that if you are going to test the importance of signatures, it should be on actually infected machines, not machines with just infected files on them. In this way then the true capabilities of the AV products can be put to test by acknowledging which products are actually capable of removing the infection from the machine and what is required to do so. Not all vendors are the same, several require that you actually download and install extra executables onto machines in order to clean them…..customers need to be made aware of this before they actually spend a load of money only to discover that they are unable to clean their machines easily.
I strongly applaud the industry in trying to rectify the testing methodoligies to give a better picture of product capability to all, i just fear that it is going to be an incredibly hard thing to do.
One other point i would like to make – whoever suggested the name that abbreviates to iTOSS was obviously not taking the whole thing too seriously. 
AV industry designing the tests, that’s priceless.
“The plan of action is to meet several times a year”
Preferably in luxorious, warm locales such as Spain.
i wish av industry would give as much resources and time on a more centralized database and sharing of samples (between the av vendors of course). for me this current “fight” of av industry about av testing seems to be quite useless (don kihots like), since there are few av testing bodies that are already doing quite a good job. and please don’t tell me that you (the av vendors) are already sharing the samples, for more then 10 years i am quite close to this industry and i know that this is by far not done as good as it could/should be.
and about the comment above, from chris… i believe this comment was on the right place about 5 years ago, today however when we have several thousands of new samples per day the names of this samples IMHO have no value and people should realize this and just get over it.
What about virus naming standards???
Submit your own comments / message for this post