Tuesday, January 6, 2015

"Tests" of Security Software

Again it is a customer question that triggers new information on this blog; Thank you Frank C.

The customer asked what I think about the results of a test of Security Software in Consumer Reports' June 2014 issue.

I am not subscribed to Consumer Reports and the contents of their publication is not available online. Luckily the customer had attached a pdf file of the article. Without permission from CR I can not publish it here.

Needless to say that Microsoft Security Essentials/Windows Defender ended up on one of the last places in the rankings. That is very relevant to me because all my home customers use either one of these anti virus programs.

Here is my reply, almost verbatim from the email.
Thank you for the question. A few points in no special order as a reply.

Who actually ran these tests?
And who financed them?
Consumer Reports certainly does not have a proper test lab; that takes years to develop and a big lot of money to finance and run.

I have seen dozens and dozens of "tests" that were paid for by manufacturers of "security software".
And guess what, the result was always that their specific product ended up on top of the list.

Microsoft Security Essentials and Windows Defender on Windows 8 are not "security" programs, they are classic anti virus programs. Anti virus programs protect against getting virus infected files on your computer. And in my limited experience of 12 years and ca. 6000 distinct home customers these two programs do an excellent job at that.

To compare the two MS programs 1:1 against security suites is ridiculously wrong and done to dupe the un-informed into wrong conclusions.
Security suites try to supervise every click and input in web pages.
An endeavor that brings additional computing burdens but is doomed to fail because most errors are or are a result of an EBKAC (Error Between Keyboard And Chair).
Please see an irreverent remark below.

Most security suites are a very noticeable additional work load even for well equipped computers.

Just today I had been called to "slow" computer. After removing the PuPs the machine was still sluggish. After removing an older version of Norton Internet Security (about 4 years old)  the computer suddenly worked just fine. It was a BIG perceivable difference; I have seen that many, many times. This effect is not specific to Norton, it applies to many brands of security suites; in my experience especially (but not limited) to AVG, Avast, Norton, McAfee and Trend Micro.

Many of these "tests" do not talk about the curse of free security suites, that is false positives. Erroneously marking a benign program as malicious leaves the non-geek home user clueless and helpless.

Avast especially has last year broken quite a few computers with insufficiently tested updates.

only one of the programs in the CR test can even detect Poweliks, the worst and best hidden virus currently around.

AFAIK the only AV program that currently detects Poweliks is MS's Security Essentials/Defender! Although I use third party tools to remove it completely and terminally.

Re. EBKAC errors:IMHO no software in the world can protect irresponsible people from themselves.

We need to pay attention to the details and we need to heed #6 of my 10 commandments for safe computing.

Frank, please do not take the last paragraph personally; it only reflects general observations that I make all too often.
Please let me know in the comments what you think; thank you in advance

As usual I welcome suggestions and comments right here in the blog.

Click here for a categorized Table Of Contents.


popeye363 said...

I agree with your point about the challenges and expense of a full-blown malware testing lab. However, I think you're off base with your implication that Consumer Reports is influenced by some vendor's money.

CR is an independent, not-for-profit organization that's been around since 1936, before your or I were born (and I date back to the punched-card era). They maintain a strict policy against commercial use of their reviews, and defend their copyrighted reviews earnestly.

Their consumer product testing labs, the largest in the world, are supported by subscriptions. CR has never accepted advertising, to avoid even the appearance of impropriety.

Their credibility depends on maintaining their independence from commercial influence. The idea that they would risk everything by allowing manufacturers to rig the testing (as our government often does) is simply laughable. Indeed, manufacturers who attempt to trade on a favorable review can expect to be challenged by CR and its eight million subscribers.

I haven't seen the lab where CR tested software, and it may not be the equal of the best of those whose sole business is malware testing. However, if their world-class approach to auto testing is any indication, I'm confident it's superior to the testing performed by any mass-market computer magazine... you know, the ones who take ad money from software vendors.

Eike Heinze said...

Dear reader Popeye363,
Thank you for reading my blog.
I applaud your fervent belief in CR as THE honest and trustworthy institution in this great country.

Yet I still beg to differ. The fact that they uncritically compare true anti virus directly to security suites IMHO shows either:
- a deplorable lack of technical know how or
- an unwillingness to inform correctly and completely about this issue or
- an agenda of some sort.

I look forward to an extended discussion with you but PLEASE not here with comments; send me an email (address under "Welcome" in the left side bar). This discussion may with your permission even become another entry on the blog.