I just want to give up at times!!!!!!!!

[sobranie]

http://www.matousec.com/projects/proactive-security-challenge/results.php

[Niall]

Is it me, or is that site full of cr*p?

[Simon]

I couldn't quite work out what it was full of. 

[Gary]

Is it me, or is that site full of cr*p?

From what I remember that site was pretty spot on. Not sure what its like now.

[Niall]

Well as it's listing the most unreliable software as the best, I think it's safe to say it's cr*p

I might look into outpost to see what's up with that these days. I used to use that, and it WAS good. Then suddenly it became awful, almost overnight, so I ditched it.

Bit defender is rated highly there, and that has been notorious in it's entire life for blocking specific things, until you rename them!

[armadillo]

It does look strangely suspect. As you suggest, how likely is it that all the most respected players score values like 3%?

Outpost has featured recently on the ESET forums since the latest versions of Outpost cause the PC to crash when ESET is present.

I used to use Comodo until I found it was so successful because it liked to block virtually everything.

The most successful security of all is not to turn the PC on. I find no malware gets through when it's switched off. 

[Simon]

I've recently installed BitDefender Internet Security 2012, as I was having problems with a conflict between F-Secure and iTunes.  I find it OK, although, it does have some annoyances.  It has a bit of an 'all or nothing' approach when it comes to informing the user.  If 'Paranoid Mode' is activated, multiple alerts and permission requests are produced, which can soon become tiresome and overwhelming.  Switch off 'Paranoid Mode', and it runs practically silently, but I've found that it blocks certain 'good' applications, with only a 5 second 'flag' to warn you, which you miss if you don't happen to be sitting in front of the PC at the time.  It's easy enough to allow 'blocked' applications, but you have to check every so often to see what 'Events' there are listed.  Anyway, other than that, it seems to run smoothly enough, and no problems with iTunes.

[armadillo]

Just did a bit of looking into Matousec.

The way they score is highly crazy.

Tests are divided into levels. If a product fails on a level 1 test, it does not get tested against higher levels but instead gets allocated a score of zero for all tests in all higher levels that are not tested.

The percentage score is then calculated on the basis of successes achieved as a percentage of all available tests. Therefore a product that fails in level 1 automatically gets a very low percentage because its score only includes level 1 results but the total of available tests includes all tests, even those not performed at all.

This explains why well known products can score 3% when they are not tested at all against 97% of the tests.

Those tests are very carefully constructed and carefully arranged in levels.
Arranging the tests in levels according to a different arrangement could result in very different scores.

Vendors can pay to have all tests evaluated when they fail a lower level test.
Vendors can pay to be retested after altering their product to pass a test.

Tests are carried out by different testers who are not trained and whose results are not independently verified.

http://www.techsupportalert.com/freeware-forum/security/315-evaluation-of-matousec-firewall-tests-editorial.html

[zappaDPJ]

Well that certainly puts some perspective on the matter, nice find

[Technical Ben]

Yep. If it's was an obvious reason such as "level 1 = viruses" and "level 2 = malware" you could perhaps understand. "Scanner A failed on viruses, so there is no point testing for malware".
However, if they do it where they choose what to add or remove from test 1, 2, 3 etc, then they could run a nice little scam where they purposely failover programs on the first test (an out of the blue virus etc).

Not a good way to do your test, either if done in incompetence or insidiousness. 

[armadillo]

Quite right, Ben. If you look at the arrangement of their tests, there is nothing at all obvious about them. Moreover, they are outgoing leak tests which you have to instruct the security software to allow to run if it flags the test as malware. It tests what the security software does once you have permitted it to run the test.

I very much like the analogy in the link I posted. It is like testing how your body reacts after deliberately swallowing some poison. Yeah, it tastes horrible. Drink it up. Oh dear, you got sick.

[Gary]

I thought Matousec tests are mainly for proactive/HIPS features 

Found this though

"Some have criticized Matousec for demanding more of a firewall than what is traditionally expected. A PC firewall should stop leaks, and some argue that it must not protect against other vulnerabilities such as key logging.

Matousec has also been criticized on the basis of conflict of interest, due to being paid by security software makers for re-testing. Results may then, at the discretion of the software vendor, be disclosed or kept private. However, Matousec is transparent about this, and the testing procedure is described in detail on the website:

    Every vendor has a right to request a paid Proactive Security Challenge testing, in which case its product will be tested in all levels regardless the results on each of the levels. After the vendor receives the results of the paid testing, it can either keep them private or request their publishing on our website, but such a request will be satisfied only if the previously published results for the tested product, if any, are at least one month old and if the tested version is stable and publicly available. There are no limits of the frequency of the paid tests"