Microsoft Sentinel Data tiering best practices

Summary

In this episode of the Virtual Ninja Show, Microsoft Sentinel's new **Auxiliary Logs** tier is introduced, offering a cost-effective solution for storing low-value, high-volume security data. Hosts Maria Sza and Jel Bergman explain how Auxiliary Logs differentiate from primary logs, focusing on low-Fidelity data like firewall logs. The show highlights the power of **Summary Rules**, which aggregate data and send it to the Analytics Tier for alert generation and deeper analysis. Real-world use cases like **Threat Intelligence Lookups** and **Anomaly Detection** are discussed, showcasing how organizations can save costs while enhancing security. The episode concludes with a demo of setting up and using Summary Rules in Microsoft Sentinel.

Takeaways

• 😀 Microsoft Sentinel is a Security Information and Event Management (SIEM) tool that helps collect, analyze, and provide insights into security data.
• 😀 Sentinel's pricing model is based on the volume of data ingested, with customers paying based on the number of gigabytes processed.
• 😀 Data in Sentinel is categorized into primary (high-security value) and secondary (low-security value) tiers to optimize cost and processing.
• 😀 Primary data includes high-value logs like EDR, antivirus, authentication, and Windows security events, which are closely monitored.
• 😀 Secondary data includes verbose logs like proxy or net flow logs, which are not continuously monitored but kept for context in case of incidents.
• 😀 The new 'auxiliary logs' tier allows low-cost ingestion of high-volume, low-value logs while still enabling queries and analytics with some limitations.
• 😀 Auxiliary logs have a 30-day retention period, after which they can be moved to long-term retention (archive) for continued access.
• 😀 Summary rules in Sentinel can aggregate data from auxiliary logs at specific intervals, then send summarized results to the analytics tier for deeper analysis.
• 😀 While auxiliary logs do not support real-time analytics, summary rules help run pre-built queries that extract valuable insights from low-fidelity data.
• 😀 Use cases for auxiliary logs include threat intelligence lookups, anomaly detection in firewall logs, and data exfiltration monitoring using aggregated data.
• 😀 The integration of summary rules with auxiliary logs enables detection and alerts based on patterns in the summarized data, ensuring security insights at a reduced cost.

Q & A

What is Microsoft Sentinel, and what role does it play in security operations?

Microsoft Sentinel is a Security Information and Event Management (SIEM) tool that aggregates and analyzes security data from various sources to provide insights and detect potential threats. It helps organizations monitor their security infrastructure, run queries, and manage alerts based on security events.

What are the two main types of data in Microsoft Sentinel, and how are they categorized?

In Microsoft Sentinel, data is categorized into two main tiers: 'Primary Data' (hot tier) and 'Secondary Data' (cold tier). Primary data includes high-value logs, such as EDR, antivirus, and authentication logs, which require close monitoring. Secondary data includes lower-value, high-volume logs, such as proxy or net flow logs, which are stored for context during investigations but do not need constant monitoring.

What are auxiliary logs, and how do they differ from analytics logs in Sentinel?

Auxiliary logs are a new data tier introduced in Microsoft Sentinel for handling secondary, low-value data at a reduced cost. Unlike analytics logs, which are used for real-time analysis and detection, auxiliary logs are designed for storing high-volume, verbose logs that are not frequently monitored but may be valuable for investigation if needed.

How does the pricing structure for auxiliary logs differ from that of analytics logs?

The pricing for auxiliary logs is significantly cheaper than analytics logs. While analytics logs are priced based on the volume of data processed for real-time analysis, auxiliary logs are designed for lower-cost storage and analysis of secondary data, providing a more cost-effective solution for handling less critical logs.

What are summary rules in Microsoft Sentinel, and how do they work with auxiliary logs?

Summary rules are a new feature in Microsoft Sentinel that allows users to aggregate and summarize secondary data from auxiliary logs. These summary rules run queries at set frequencies (e.g., hourly, daily) and send the aggregated results to the analytics tier for further analysis. This enables cost-effective processing of large volumes of data by summarizing them before ingesting them into the more expensive analytics tier.

Can you run real-time analytics on data in the auxiliary log tier?

No, auxiliary logs do not support real-time analytics. However, summary rules allow for periodic aggregation of data into the analytics tier, where full analytics and detections can be performed based on the summarized results.

How does a customer benefit from using summary rules with auxiliary logs?

Customers benefit by being able to store large volumes of low-value data in the cheaper auxiliary log tier, and then use summary rules to aggregate and send only the valuable results to the more expensive analytics tier. This reduces costs while still enabling valuable insights and detections from the data.

What are some practical use cases for summary rules and auxiliary logs in Sentinel?

Some practical use cases include: 1) Threat Intelligence matching, where summary rules can match logs against known malicious IPs and send results to the analytics tier; and 2) Anomaly detection, where firewall logs are aggregated to identify deviations from normal traffic patterns, such as unusual data transfers, which could indicate potential security incidents.

How can you set up a summary rule to analyze firewall logs in Sentinel?

To set up a summary rule for firewall logs, you need to create a rule that specifies the data you want to summarize (e.g., bytes sent to public IPs). You define the frequency of the summary rule (e.g., once per day), and the results are stored in a custom table within the analytics tier, where they can be used for further analysis or detection rules.

What happens to auxiliary logs after 30 days, and how can you access them?

After 30 days, data in the auxiliary log tier is moved to long-term retention (archive). You can still access this archived data by running search jobs, though the data is no longer part of the active log analysis in Sentinel.