MITRE added a new wrinkle to its latest endpoint detection and response (EDR) evaluations, a test of endpoint security products’ ability to stop an adversarial attack.
Previous MITRE evaluations and the first part of the latest evaluation, Carbanak+FIN7, focused on the ability of vendors to detect attacks and alert security staff. That detection approach focuses more on the strengths of EDR tools, which essentially add a centralized management layer to endpoint security, the ability to detect and respond to attacks that evade endpoint security and other defensive tools.
The new MITRE protection tests are aimed more at endpoint protection platforms (EPP), which are somewhat like traditional antivirus software, except with the greater sophistication that enterprise IT security requires. EDR and EPP tools have been merging over the years, yet they retain their distinct functions.
What’s especially intriguing about the MITRE protection evaluation is it used adversarial attack simulations, not just malware samples used in traditional antivirus testing. A number of vendors, among them Palo Alto Networks and Symantec, have focused on the protection results, saying they’re more important than the detection evaluations. Detection can produce “noise,” or alerts that aren’t always important, whereas protection represents actual attacks stopped. This is especially true with the MITRE detection tests, which don’t track false positives, so vendors can set detection thresholds so they’re more sensitive than they might be in production environments. This makes the new protection tests an important addition to the MITRE evaluations.
MITRE protection results
Only 17 of the 29 vendors in the latest MITRE evaluations participated in the protection test, mainly vendors with strong EPP offerings.
The protection results are valuable on their own, and also add interesting perspective when combined with the detection results.
Six of the vendors stopped all 10 attacks: Palo Alto, Cybereason, Symantec, McAfee, Microsoft and Fortinet. Five vendors stopped nine of the 10 attacks: SentinelOne, Trend Micro, Cynet, ESET and Cisco.
Below you’ll find a chart of the results of 16 of the vendors that participated in both the MITRE detection and protection results. We left out BlackBerry Cylance, which experienced a target machine crash in half the protection tests, but otherwise stopped the other five attacks.
The first column of results is the protection scores, and the second the detection percentage. In the third column we did something completely unscientific and equally weighted both scores. Nonetheless, a few names stand out when looking at both scores together.
Palo Alto, Cybereason and Symantec were the only vendors scoring 100% on the protection results and above 90% on the detection results. SentinelOne and Trend Micro scored above 90% in both tests, with McAfee, Microsoft, Cynet and ESET not far behind.
The old guard stands tall
Those nine vendors comprise a mix of old and new. There are a number of things we could point out, like the persistent excellence of Palo Alto, the rapid ascent of Cybereason, SentinelOne and Cynet, or Microsoft’s firm commitment to cybersecurity.
But instead we’d like to note the presence of the first-generation antivirus vendors on that list, Symantec, McAfee and Trend Micro. They may have been overshadowed in recent years by some of the flashy marketing of the upstarts, but that long history gives the old guard a product depth that’s tough to beat. Just one example: Symantec was prepared for last year’s SolarWinds hack because it long ago faced attacks when hackers tried to disable endpoint agents, a primary vector for the Sunburst malware.
In cybersecurity, experience still counts for something. And given the market share of Symantec, McAfee and Trend Micro, that’s good news for all of us.
