2.2 Hypothesis Considerations - MAD20 Threat Hunting & Detection Engineering Course

MAD20Tech

25 Apr 202406:40

Summary

TLDRThis lesson delves into the influence of bias in threat hunting, teaching how to recognize and mitigate its impact on intelligence reporting. It emphasizes the importance of being aware of cognitive biases like visibility and victim bias, and how defenders can introduce availability and anchoring biases. The lesson guides on formulating hypotheses, choosing attack techniques wisely, leveraging existing data, and engaging with the community to refine analytic approaches and avoid redundancy.

Takeaways

🔍 Bias in threat hunting can occur and must be recognized to minimize its impact on intelligence reporting.
🧠 Cognitive biases, such as visibility and victim bias, can skew the perception of the full scope of attacks.
👀 Availability and anchoring biases can lead to a narrow focus on familiar or currently accessible data, potentially overlooking other important information.
📝 Documenting and sharing assumptions with the team is crucial for validating and revisiting them during the analytic development process.
🔑 When generating hypotheses, be specific about known facts, inferences, chosen hypotheses, discarded options, and the environment being defended.
💡 Focus analytic efforts on techniques that are not commonly covered, have a significant impact if used, and leverage existing data collection for efficient implementation.
🚀 Consider techniques that are not typically employed by users and system administrators to avoid high false alarm rates.
🔎 Engage with the community and existing resources to avoid redundant work and to uncover potential flaws in your approach.
🔗 Investigate if there are precursor, follow-on, or correlated techniques to the one being analyzed, as grouping them can improve precision and recall.
🛠 Define the scope of the behavior under examination based on platforms, implementations, and functionality to focus research effectively.
⚖️ Be prepared to revisit and adjust the scope as needed to ensure the full range of the technique is covered according to the environmental terrain.

Q & A

What is the main focus of lesson 2.2 in the provided transcript?
-The main focus of lesson 2.2 is to discuss how bias can occur in threat hunting, how to recognize it, and considerations for choosing an attack technique on which to focus the hypothesis.
What is the significance of understanding cognitive biases in the context of threat intelligence reporting?
-Understanding cognitive biases is crucial to minimize their impact in threat intelligence reporting, as they can lead to skewed analysis and false impressions of the full scope of attacks or activities.
Can you give an example of a cognitive bias mentioned in the transcript?
-One example mentioned is visibility bias, which occurs when threat intelligence produced by an organization is only focused on the subset of adversarial activity that they can detect.
What is the importance of being aware of inherent biases in models used for threat hunting?
-Being aware of inherent biases in models is important to ensure accurate and comprehensive threat analysis, as these biases can influence the focus and interpretation of data.
How can victim bias affect the threat intelligence reports?
-Victim bias can affect reports by focusing more on high-profile victims and skewing the data based on what is allowed to be published, which may not represent the full range of threats.
What is the impact of novelty bias in the context of threat hunting?
-Novelty bias can lead to more coverage and attention being given to new or flashy adversary groups, potentially overshadowing long-standing threats that may be more prevalent or significant.
Why is it important for threat hunters to document and share their assumptions with their team?
-Documenting and sharing assumptions is important for validation and to revisit them through the analytic development process, ensuring a more accurate and objective threat hunting approach.
What is the advice given for focusing analytic efforts when choosing a technique in threat hunting?
-The advice is to focus on techniques that are not already commonly used by adversaries, would create a significant impact if used successfully, and capitalize on existing data collection, documentation, or analytics.
Why is it beneficial to check for existing analytics, mitigations, or other defensive ideas online before conducting research?
-Checking for existing work can save time and effort, help avoid redundant work, and may highlight gaps that can be focused on, leveraging the knowledge and findings of other security researchers.
What is the purpose of engaging with the community in the context of threat hunting?
-Engaging with the community helps improve work by sharing new discoveries, getting feedback on approaches, and uncovering flaws early on, which can save time and prevent potential issues.
Why is it necessary to define the scope of the behavior when preparing to conduct research on a technique?
-Defining the scope helps to focus research on relevant systems and behaviors, ensuring that the analysis is accurate and tailored to the specific environment and requirements of the threat hunting process.

Outlines

00:00

🕵️‍♂️ Bias in Threat Hunting and Hypothesis Considerations

This paragraph discusses the occurrence of bias in threat hunting and the importance of recognizing and addressing it. It mentions the MITRE ATT&CK Defender Cyber Threat Intelligence course, which covers cognitive biases that can affect threat intelligence reporting. The script emphasizes the need to be aware of inherent biases such as visibility bias, victim bias, and novelty bias. It also points out that defenders can introduce biases like availability bias and anchoring bias. The importance of documenting and sharing assumptions with the team is highlighted, as well as the need for specificity when generating hypotheses. The paragraph concludes with advice on choosing a technique to focus on, considering the return on investment, and leveraging existing data and analytics to minimize false alarms.

05:03

🔍 Technique Selection and Hypothesis Scoping

The second paragraph delves into the process of selecting and scoping techniques for threat hunting. It advises focusing on techniques that are not commonly covered, have a significant impact if used, and can be implemented easily without triggering many false alarms. The paragraph also suggests considering techniques that are not typically used by system administrators. It encourages researchers to check for existing analytics, mitigations, and defensive ideas to avoid redundant work and to engage with the community to improve their work. The importance of defining the scope of the behavior to be examined is highlighted, including considering factors like platforms, implementations, and functionality. The paragraph concludes by emphasizing the need to be aware of biases when developing hypotheses and the significance of technique choice and hypothesis scoping for setting up for success in the long run.

Mindmap

Keywords

💡Bias

Bias refers to a systematic error or deviation from a standard, often in judgment or decision-making. In the context of the video, bias is discussed as a potential issue in threat hunting and intelligence reporting, where it can lead to skewed perceptions or incomplete understanding of threats. The video mentions several types of cognitive biases, such as visibility bias, victim bias, and novelty bias, which can distort the analysis of threats.

💡Threat Hunting

Threat hunting is the proactive search for threats that have evaded initial detection by security measures. The video emphasizes the importance of recognizing and dealing with bias in this process to ensure accurate and comprehensive threat intelligence. Threat hunting involves generating hypotheses about potential threats and then seeking evidence to validate or refute these hypotheses.

💡Cognitive Biases

Cognitive biases are systematic patterns of deviation from rational judgment that occur due to the way the human mind processes information. The video highlights the relevance of cognitive biases in the field of cybersecurity, particularly in threat intelligence, where they can influence the interpretation of data and the prioritization of threats.

💡Visibility Bias

Visibility bias occurs when an organization focuses only on the subset of adversarial activity that they can detect, potentially leading to a false impression of the full scope of attacks. The video uses this term to illustrate how bias can affect the comprehensiveness of threat intelligence, as it may overlook threats that are not currently detectable by the organization's systems.

💡Victim Bias

Victim bias is a type of bias where reports tend to focus on more high-profile victims, potentially skewing the perception of threat severity or prevalence. The video script mentions this bias as an example of how the focus of threat reports can be influenced by the prominence of the victims, rather than an objective assessment of the threats themselves.

💡Novelty Bias

Novelty bias is the tendency to give more attention to new or novel information, often at the expense of more established or routine information. In the video, this bias is discussed in the context of cybersecurity, where new adversary groups may receive more coverage than long-standing ones, potentially leading to an overemphasis on newer threats.

💡Availability Bias

Availability bias is the tendency to rely on the most readily available information, often neglecting other potentially relevant data. The video describes how a threat hunter might fall into this bias by focusing only on the data they currently have access to, which could result in a skewed prioritization of techniques or threats.

💡Anchoring Bias

Anchoring bias is the tendency to rely too heavily on the first piece of information encountered when making decisions. In the context of the video, this bias can cause defenders to focus on initial reports or discussions, potentially overlooking other valuable data sources that could provide a more comprehensive understanding of threats.

💡Hypothesis

A hypothesis is a proposed explanation for a phenomenon, made as a starting point for further investigation. In the video, the development of hypotheses is a key part of the threat hunting process, where specific hypotheses about potential threats are formulated and then tested against evidence gathered through research and analysis.

💡Technique

In the context of cybersecurity, a technique refers to a specific method or approach used by adversaries to compromise systems or achieve their objectives. The video discusses the importance of choosing techniques to focus on in threat hunting, considering factors such as their prevalence, impact, and how they can be detected or mitigated.

💡Investment Return

The term 'investment return' in the video refers to the benefit or value gained from the effort and resources invested in a particular activity, such as threat hunting. The video advises focusing on techniques that are not commonly covered by adversaries or that could have a significant impact if successfully exploited, to ensure a good return on the investment of analytic efforts.

💡False Alarms

False alarms in cybersecurity are alerts triggered by security systems that incorrectly identify benign activity as malicious. The video mentions the need to balance the selection of techniques to avoid generating too many false alarms, which can lead to alert fatigue and reduced effectiveness of security measures.

💡Community Engagement

Community engagement in the video refers to the interaction with other security researchers and professionals to share findings, ideas, and to receive feedback. This engagement is encouraged as a way to improve work, avoid redundancy, and uncover potential flaws in approaches early on.

💡Scope

Scope in the context of the video refers to the boundaries or limitations set for a particular research or analysis project. Defining the scope is crucial for focusing research efforts effectively, and the video discusses factors such as platforms, implementations, and functionality that can help in determining the scope of the behavior under examination.

Highlights

Bias can occur in threat hunting and recognizing it is crucial.

Cognitive biases can be present in threat intelligence and user actions.

The goal is not to memorize all cognitive biases but to understand their impact.

Visibility bias gives a false impression of the full scope of attacks.

Victim bias skews focus towards high-profile victims.

Novelty bias leads to more attention on new adversary groups.

Defenders may introduce bias through availability, focusing only on accessible data.

Anchoring bias can cause defenders to miss useful information from other sources.

Threat hunters should document and share assumptions with their team for validation.

When generating hypotheses, be specific about what is known and inferred.

Focus analytic efforts on techniques not commonly used by adversaries.

Choose techniques that capitalize on existing data collection for easier implementation.

Avoid techniques that trigger too many false alarms in the system.

Engage with the community to improve work and avoid redundant efforts.

Consider precursor, follow-on, or correlated techniques for a comprehensive approach.

Define the scope of the behavior to examine within the context of desired factors.

Scoping helps in focusing research towards relevant systems and behaviors.

Be aware of biases when developing hypotheses and conducting research.

Technique choice and hypothesis scoping are crucial for setting up for success.