Industry Questions Consumer Reports' Testing Practices
Security vendors are up in arms over Consumer Reports' "State of the Net 2006." Just days after the September issue hit newsstands, the magazine's anti-virus testing procedures were raising eyebrows.
Lavasoft's CEO, Ann-Christine Åkerlund, finds the testing practices of Consumer Reports "highly suspicious. We're keenly aware of the reputable anti-spyware programs and how they detect spyware. That is why we question how one program receives top ranking while Ad-Aware SE anti-spyware is ranked fifth, according to this Consumer Reports analysis."
Industry analyst, Mary Landesman, agrees. She takes on the Consumer Reports methods in her article, Testing Hocus Pocus, and also refers to the 5,500 new viruses created in order to support the tests.
McAfee AVERT's Igor Muttik posted a blog on the security company's website taking the publication to task for hiring a lab to design new virus variants. "Creating new viruses for the purpose of testing and education is generally not considered a good idea. Viruses can leak and cause real trouble," Muttik wrote.
Adware Report also criticized the technique, saying, "Basing test results on fabricated viruses is misleading. The testers claim that viruses are the "kind you'd most likely encounter in real life." However, they have no way of knowing this. There is no substitute for real-world conditions."
To rate anti-spyware software capabilities, CR used the public suite of Spycar scripts, whose own website states the product uses "tools designed to mimic spyware-like behavior, but in a benign form."
"It's not a serious testing tool," said Alex Eckelberry, chief executive of Sunbelt Software, whose product CounterSpy rated seventh on the list. "It (Spycar) is specifically designed to test how well anti-spyware programs block unknown applications, not (how they) scan and remove."
Consumer Reports defended its testing methods to Eckelberry in a letter, "We chose this approach because we felt it best captured the flexibility of the software."
Lavasoft did contact CR for a comment, but nothing had been received at publication time.