1.2 Detection Approaches - MAD20 Threat Hunting & Detection Engineering Course

Summary

This lesson on threat hunting introduces the concepts of precision and recall in detecting malicious activity. Precision refers to the accuracy of an analytic in avoiding false positives, while recall measures its ability to identify relevant events. The lesson explores how improving one can often reduce the other and provides a visual example. It also reviews three traditional detection approaches: signature-based, allow list, and anomaly-based detection. Each method is discussed with its pros and cons, emphasizing the challenge of balancing these metrics in the dynamic landscape of cybersecurity.

Takeaways

• 😀 Precision is a metric that measures how few false positives an analytic produces, aiming for fewer irrelevant results.
• 😀 Recall indicates how well an analytic detects all relevant events without missing important ones.
• 😀 Improving precision often reduces recall, and vice versa, requiring a balance between these two metrics when developing analytics.
• 😀 A good analytic should ideally have high precision (few false positives) and high recall (detects most relevant events).
• 😀 An analytic with good precision ensures fewer false positives, which means less analyst fatigue and more accurate detections.
• 😀 An analytic with good recall detects more of the malicious events, but might include more false positives, leading to potential missed detections.
• 😀 Signature-based detection involves looking for predefined malicious patterns (e.g., file hashes or IP addresses) but can struggle with rapidly changing threats.
• 😀 The allow list approach focuses on defining what is allowed and blocking deviations, but can be inefficient due to the need for constant updates.
• 😀 Anomaly-based detection defines 'normal' activity statistically and flags deviations, though it is prone to high false alarms and can be challenging to implement effectively.
• 😀 Understanding the difference between precision, recall, and detection approaches is critical for developing effective threat hunting strategies in cybersecurity.
• 😀 Adversaries may change their tactics to blend with normal behavior, making anomaly-based detection harder to implement and more vulnerable to evasion tactics.

Q & A

What is Precision, and how is it calculated?

Precision is a metric that indicates how few false positives an analytic returns. It is calculated by dividing the number of true positives by the total number of results (both true and false positives). A higher precision means fewer false positives.

What is Recall, and how is it calculated?

Recall is a metric that indicates how few relevant events an analytic misses. It is calculated by dividing the number of true positives by the total number of relevant events that should have been detected. A higher recall means fewer false negatives.

How are Precision and Recall related?

Precision and recall are often in tension: improving one metric usually results in the other one decreasing. For example, improving precision can reduce recall by making the analytic more specific and missing more events, while improving recall can lead to more false positives and lower precision.

What is the base rate fallacy in threat detection?

The base rate fallacy occurs when there is significantly more benign activity than malicious activity. In such cases, even a precise detection system will often produce more false positives, making it harder to identify actual threats.

What is a signature-based detection approach?

Signature-based detection involves defining malicious activity using specific indicators such as file hashes, IP addresses, or domain names. It's effective for known threats but has limited use for detecting new or modified attacks.

What are the main drawbacks of signature-based detection?

Signature-based detection requires up-to-date signatures, and attackers often alter their infrastructure to evade detection. Additionally, it doesn't help with the first victim of an attack, as signatures only emerge after the attack is discovered.

What is an allow list detection approach?

An allow list approach defines what is allowed on a system and blocks anything that deviates from this list. It can be highly restrictive but requires constant maintenance and can grow large, reducing its effectiveness.

What is anomaly-based detection?

Anomaly-based detection defines what is 'normal' based on statistical models and flags deviations as suspicious. It can detect new or previously unknown threats but is prone to high false positive rates if the definition of 'normal' is too broad or narrow.

What are the challenges in implementing anomaly-based detection?

Anomaly-based detection faces challenges in achieving a good balance between precision and recall. Networks and systems often have high variability in benign activity, making it hard to define a narrow enough normal baseline without triggering many false alarms.

How can precision and recall be visualized in threat detection?

Precision and recall can be visualized using a circle in a rectangle where malicious events are represented as red triangles and benign events as green smiley faces. The circle detects events within its radius, and the trade-off between precision and recall can be seen by adjusting the size of the circle.