Comparing MITRE EDR evaluations results for APT3

Which EDR vendor detects most APT3 steps? This is an analysis using Python and Splunk of the data MITRE published.

Why does EDR matter?

Laptops and PC’s are an important point of entry for cyber attackers because they are operated by humans that can be easily tricked into clicking on malicious links or email attachments. When those laptops and PC’s are used in an enterprise they are called endpoints, and often run special software to mitigate cyber threats.

Which EDR performs best?

Thanks to MITRE , a US non-profit organisation, we can now compare performance of various Endpoint Detection and Response solutions. This evaluation is unique because it puts a well documented cyber threat in a lab environment and tracks detection throughout the attack path. MITRE published the results, but deliberately without ranking, scoring or rating. Make up your own mind.

This is MITRE. They solve problems and apparently they’re not afraid to point out that "cyber" is the fifth domain in warfare besides land, see, air and space.

MITRE solves problems

In the cyber domain they’re famous for creating the MITRE ATT&CK matrix , an information product that helps organisations think about their cyber defence in a more attacker oriented way: from initial access via privilege escalation and lateral movement to impact.

We approve of us!

MITRE doesn’t assign scores in their EDR evaluation and in this ranking vacuum you can only imagine what most vendors did:

  • "Carbon Black outperforms all other EDR solutions" (source)
  • "CounterTack Platform leads with fast automated detections" (source)
  • "CrowdStrike Falcon […] the most effective EDR solution" (source)
  • "[…] Endgame as the first zero training endpoint protection" (source)
  • "FireEye Endpoint Security […] the most effective EDR solution" (source)
  • "[…] evaluation showcases the effectiveness of SentinelOne’s platform" (source)
  • "[…] Windows Defender ATP demonstrated industry-leading optics […]" (source)
  • "[…] Cybereason best enables defenders […]" (source)
  • "Cortex XDR and Traps Outperform in MITRE Evaluation" (source)

What are the results?

Loading the evaluation results into Splunk, via this Python script , leads to the charts below.

The evaluation simulated 136 steps of an advanced persistent threat. For example the first chart shows that 60 attacker steps on a total of 136 weren’t detected by the chart leader of main detection type "None". You can read up on the main detection types here .

Who detected the least APT3 steps?

It may not be a coincidence that RSA is missing from the cheering press release crowd.

Detection fails

Which EDR system logs most telemetry?

If you just want the logs and create your own rules in a SIEM, look no further than this chart (or Sysmon)

Who enriches events the most?

Enrichment is telemetry++.

Telemetry

Whose IOC triggered on APT3?

Telemetry

Who detects most general attacker behaviour?

Telemetry

Who detects most specific attacker behaviour?

Telemetry

Another dimension

Three other modifiers are included in the EDR evaluation. Read up on them here .

Tainted

Tainted means correlated, which is helpful for analysts.

Telemetry

Delayed

This means non-realtime.

Telemetry

Configuration change

This means APT3 steps were detected, but only after special vendor assistance.

Telemetry

Which EDR is most suitable as ultimate noise generator?

If you only care about raw data and do correlation on your own outside of the EDR solution, this chart ranks vendors based on Telemetry, Enrichment and IOC results:

Telemetry

Which EDR has the best signal/noise ratio?

If you look for EDR solutions based on signal/noise ratio, this chart ranks vendors based on General or Specific Behaviour divided by the amount of Telemetry,Enrichment or IOC results:

Telemetry

Draw your own conclusions

There are endless possibilities to slice and dice the EDR evaluation results. Drop me a line with how and why if you did.

  1. Download Splunk for free
  2. Download and install the EDR evals app from Splunkbase