A Crash Course in Audit Logs

Summary

Justin Massey, a product manager at Datadog, discusses the importance of audit logs for security and analytics. He emphasizes the need for detailed logs that include user actions, IP addresses, and outcomes to detect threats like brute-force attacks. He also highlights the challenges of log management, such as responding to customer requests for logs and the 'right to be forgotten'. Massey suggests using a structured log format and central audit log function to improve log quality and monitoring.

Takeaways

• 🔍 Audit logs are essential for analytics and threat detection, allowing you to detect outliers and suspicious activities, such as brute-force attacks.
• 👥 It's crucial to have audit logs in a good format to quickly provide them to customers, especially during security incidents or investigations.
• 🗑️ Customers may request deletion of audit logs (Right to be Forgotten), so logs should be structured to facilitate easy deletion.
• ❓ Effective logs should answer basic questions: Who did what, where, when, and why. These details help in analyzing and troubleshooting.
• 🗃️ Logs should be structured in a standardized format, such as using key-value pairs, to enable easy searching and parsing by machines.
• ⚙️ A centralized audit log function should be used for consistent logging across different parts of an application.
• 📏 Monitor changes to the audit log function file using code owners or CI checks to ensure integrity and consistency.
• 🌐 Correctly logging IP addresses is vital. Ensure the original client IP is logged, not an intermediary, like a load balancer's IP.
• 🔗 Use request IDs to correlate logs across different services and components, ensuring traceability of all actions within a system.
• 📈 While CRUD operations (Create, Read, Update, Delete) are essential to log, reading operations may require special handling due to high volume.

Q & A

What is Justin Massey's role at Datadog?

Justin Massey is a product manager for Datadog.

What is the main focus of Justin's work at Datadog?

Justin's work focuses on threat detection, specifically dealing with audit logs and their analysis.

Why are audit logs important for analytics?

Audit logs are important for analytics because they can help detect outliers and unusual activities, such as a high number of failed logins by a specific user.

What is the purpose of threat detection using audit logs?

The purpose of threat detection using audit logs is to identify potential security threats such as brute-force attacks by analyzing patterns and anomalies in the logs.

Why do customers typically request audit logs?

Customers typically request audit logs during security incidents when they are investigating issues within their environment.

What is the 'right to be forgotten' in the context of audit logs?

The 'right to be forgotten' refers to the ability to quickly delete audit logs when requested by customers, ensuring compliance with data privacy regulations.

What common issues does Justin observe with application logs?

Justin observes that application logs often lack sufficient information such as IP addresses and user actions, making it difficult to analyze and search through them.

What basic questions should be answered by audit logs?

Audit logs should answer who is doing something, what the user is doing, where it is occurring from, and when it occurred.

Why is a structured format for audit logs recommended?

A structured format for audit logs is recommended because it makes it easier for machines to read and parse the logs, allowing for quick searching and filtering.

What is the purpose of a centralized audit log function?

The purpose of a centralized audit log function is to standardize the format of logs across different parts of an application, making it easier to monitor and analyze them.

Why is it important to monitor changes to the audit log function?

Monitoring changes to the audit log function ensures that any deviations from the standard format are noticed, maintaining consistency and reliability in log data.