A Crash Course in Audit Logs

Summary

Justin Massey, a product manager at Datadog, discusses the importance of audit logs for security and analytics. He emphasizes the need for detailed logs that include user actions, IP addresses, and outcomes to detect threats like brute-force attacks. He also highlights the challenges of log management, such as responding to customer requests for logs and the 'right to be forgotten'. Massey suggests using a structured log format and central audit log function to improve log quality and monitoring.

Takeaways

• 🔍 Audit logs are essential for analytics and threat detection, allowing you to detect outliers and suspicious activities, such as brute-force attacks.
• 👥 It's crucial to have audit logs in a good format to quickly provide them to customers, especially during security incidents or investigations.
• 🗑️ Customers may request deletion of audit logs (Right to be Forgotten), so logs should be structured to facilitate easy deletion.
• ❓ Effective logs should answer basic questions: Who did what, where, when, and why. These details help in analyzing and troubleshooting.
• 🗃️ Logs should be structured in a standardized format, such as using key-value pairs, to enable easy searching and parsing by machines.
• ⚙️ A centralized audit log function should be used for consistent logging across different parts of an application.
• 📏 Monitor changes to the audit log function file using code owners or CI checks to ensure integrity and consistency.
• 🌐 Correctly logging IP addresses is vital. Ensure the original client IP is logged, not an intermediary, like a load balancer's IP.
• 🔗 Use request IDs to correlate logs across different services and components, ensuring traceability of all actions within a system.
• 📈 While CRUD operations (Create, Read, Update, Delete) are essential to log, reading operations may require special handling due to high volume.

Q & A

What is Justin Massey's role at Datadog?

Justin Massey is a product manager for Datadog.

What is the main focus of Justin's work at Datadog?

Justin's work focuses on threat detection, specifically dealing with audit logs and their analysis.

Why are audit logs important for analytics?

Audit logs are important for analytics because they can help detect outliers and unusual activities, such as a high number of failed logins by a specific user.

What is the purpose of threat detection using audit logs?

The purpose of threat detection using audit logs is to identify potential security threats such as brute-force attacks by analyzing patterns and anomalies in the logs.

Why do customers typically request audit logs?

Customers typically request audit logs during security incidents when they are investigating issues within their environment.

What is the 'right to be forgotten' in the context of audit logs?

The 'right to be forgotten' refers to the ability to quickly delete audit logs when requested by customers, ensuring compliance with data privacy regulations.

What common issues does Justin observe with application logs?

Justin observes that application logs often lack sufficient information such as IP addresses and user actions, making it difficult to analyze and search through them.

What basic questions should be answered by audit logs?

Audit logs should answer who is doing something, what the user is doing, where it is occurring from, and when it occurred.

Why is a structured format for audit logs recommended?

A structured format for audit logs is recommended because it makes it easier for machines to read and parse the logs, allowing for quick searching and filtering.

What is the purpose of a centralized audit log function?

The purpose of a centralized audit log function is to standardize the format of logs across different parts of an application, making it easier to monitor and analyze them.

Why is it important to monitor changes to the audit log function?

Monitoring changes to the audit log function ensures that any deviations from the standard format are noticed, maintaining consistency and reliability in log data.

Outlines

00:00

🔍 Audit Logs and Threat Detection

Justin Massey, a product manager for Datadog, emphasizes the importance of audit logs for threat detection. He points out that application logs often lack essential information needed for analytics and threat detection. Massey suggests structuring audit logs to include who performed an action, what action was taken, where it was done (e.g., IP address), and why it was done. He highlights the need for audit logs to be in a good condition for quick retrieval during security incidents and for compliance with 'right to be forgotten' requests. Massey also discusses common issues with logs, such as missing IP addresses or unclear user actions, and proposes a structured format for logs to facilitate easier searching and analysis.

05:01

💻 Centralized Audit Log Function and Monitoring

The speaker discusses the creation of a centralized audit log function, which standardizes the logging format to include attributes like user, event, and outcome. This function helps in monitoring changes to the log file and ensures that logs are consistent and machine-readable. The speaker also advises monitoring the log file for any changes and being aware of such changes promptly. They touch upon the importance of logging the correct IP address, especially in scenarios involving load balancers, and the use of request IDs to trace logs across the application stack. The paragraph also covers considerations for which events to log, distinguishing between state-changing and non-state-changing API calls, and the need to log reads for potential data breach investigations.

10:02

🗂️ Storage and Handling of Audit Logs

In the final paragraph, the focus shifts to how audit logs should be stored and handled. The speaker raises questions about the storage method, whether on-disk or replicated to a secure location like S3, and the importance of ensuring logs can only be read once. They also discuss how to handle deletion requests efficiently if logs are well-formatted. The speaker encourages considering both security and operational use cases when deciding what to log and emphasizes the need to clearly define what should be monitored and logged.

Keywords

💡Audit Logs

Audit logs are records of events that occur within a system, capturing who did what, when, and where. In the video, Justin emphasizes their importance for analytics, threat detection, and compliance, noting that well-structured audit logs can help identify anomalies like failed logins, support investigations during security incidents, and comply with regulations like the 'right to be forgotten'.

💡Threat Detection

Threat detection involves identifying suspicious or malicious activities within an environment, often using data from logs. The video discusses using audit logs to detect security threats, such as brute-force attacks, by analyzing authentication events and failed login attempts to identify outliers or unusual patterns.

💡Structured Log Format

Structured log format refers to a consistent and standardized way of formatting log entries, making them easier to read and analyze. Justin suggests using key-value pairs (e.g., 'user=John Doe') to enhance the readability and searchability of logs, allowing teams to more easily parse and filter data for analytics and security purposes.

💡Log Analytics

Log analytics is the process of analyzing log data to gain insights into system behavior, security, and performance. In the video, Justin highlights the importance of having logs in good condition for effective analytics, allowing users to detect anomalies, drill down into specific events, and visualize trends in audit data.

💡Event Outcome

The event outcome in logging refers to the result of an action, such as 'success' or 'failure'. In the video, Justin emphasizes the need to standardize outcomes (e.g., success vs. failure) to avoid ambiguity and ensure that log entries are consistently interpretable, aiding in analysis and monitoring.

💡IP Address Logging

IP address logging involves recording the source IP address associated with an event, such as a login attempt. Justin warns about common pitfalls, such as logging the IP of a load balancer instead of the actual client, which can mislead security analysis. Properly capturing the originating IP is crucial for accurate auditing and threat detection.

💡Centralized Logging Function

A centralized logging function is a standard method or tool used across an application to create consistent log entries. Justin suggests implementing such a function to enforce log formats, manage data integrity, and ensure consistent logging practices across different modules, which simplifies monitoring and analytics.

💡Right to be Forgotten

The right to be forgotten is a privacy principle that allows individuals to request the deletion of their data from systems. In the video, Justin mentions the importance of structuring logs in a way that enables quick identification and deletion of relevant data when customers exercise this right, ensuring compliance with privacy regulations.

💡CRUD Operations

CRUD stands for Create, Read, Update, and Delete—common operations in API and database contexts. The video discusses the importance of logging these operations, particularly the 'Read' actions, which are often high-volume but necessary to capture for security purposes, such as tracking data access during a breach investigation.

💡X-Forwarded-For Header

The X-Forwarded-For header is used in HTTP requests to identify the originating IP address of a client connecting through a proxy or load balancer. Justin highlights that improper use of this header can result in logging incorrect IP addresses, which could undermine the accuracy of security audits and threat detection processes.

Highlights

Audit logs are essential for performing analytics and threat detection on authentication events.

Outlier detection: Identifying unusual behaviors, such as a specific user with more failed logins, is critical for security analysis.

Audit logs need to be in a structured format to easily perform analytics, drill down into specific users or IP addresses, and filter events.

Threat detection: Properly formatted audit logs are key to detecting potential security incidents, such as brute-force attacks.

Audit logs should be quickly retrievable in the event of security incidents or customer requests, especially when investigating suspicious activities.

Handling data deletion requests, such as the right to be forgotten, requires structured audit logs to efficiently remove specific records.

Many application logs lack critical information such as IP addresses, making it difficult to answer essential security questions.

To improve log searchability, logs should use a consistent format, such as key-value pairs with clear identifiers like user and IP.

A structured log format that includes user identity, event type, outcome, and IP address is more easily parsed by machines for analytics.

A centralized audit log function should be implemented to standardize the log format across different types of authentication (e.g., OAuth, SAML).

It's important to monitor changes to audit log configurations and formats, ensuring any adjustments are tracked and audited.

Correctly logging IP addresses is crucial to avoid logging load balancer IPs, which can lead to misinterpretations of audit data.

Use request IDs to join logs from various parts of the application stack, enabling better tracing of user actions across systems.

Considerations for logging read events: While non-state-changing, read operations should still be logged for security purposes.

Define and monitor for specific security and operational events to ensure relevant data is logged for analysis and threat detection.

Transcripts

so my name is Justin Massey I'm a

product manager for data dog

and rather than dive into the demo the

data dog product or talked too much

about it we have a sponsor booth for

that you can check out that but let's

dive deep into audit logs I work on our

threats about threat detection product

and I I work with a lot of our customers

I looked to them about their application

logs and best practices and I've worked

on many different environments before

and the application logs always seem to

be lacking some information so one thing

we want to do with these application

logs is we want to do analytics on them

and analytics we have like our analytics

on the right-hand side and our actual

logs on the left-hand side now these are

all of our authentication events so

imagine you wanted to detect a an

outlier and you see that this outlier

this user admin has more is it has more

failed logins than the rest if you

wanted to drill down to this specific

user you want to be able to ask

questions about that why and you may

want to drill down to a specific IP

address and you want to take a look at

that now you want to you you've now

quickly drilled down you can see all of

this information regarding this specific

audit blog but first to get here you

have to have your audit logs in a good

condition the next thing you want to do

on these audit logs is you want to do

threat detection on them you're going to

want to look at they sim audit logs and

ensure for this example you may be

wanting to detect it like a potential

potential brute-force attack now another

thing that you use your audit logs for

is customers ask for them and then when

they're asking for them it's typically

not at a good situation your audit the

customers are asking for those auto logs

because something else in the

environment is going going on maybe they

have a security incident and they are

investigating so you want to be able to

return these audit logs in a quick

manner next after the customers ask for

them they're going to ask you to delete

them so this is the right to be

forgotten so you want your audit logs in

a good format where you can quickly

delete

when the customers have requested that

so yes what's the problem well I've

logged into many different applications

when I'm starting a new job and I've

taken a look at our application logs and

something like this happens you see John

Doe logs in and it doesn't necessarily

say what IP address it was from the next

one is a login from that's likely

addressed but what user just did it then

you have password reset was cents it to

who you're not answering all the

questions here and even though that this

John Doe logged in and this failed login

these proposed the communication events

where the outcome may have been

successful or unsuccessful there are

very similar logs but you're not showing

that and it's going to be very hard to

present this in an analytical way or

search for these logs once they're

stored so if you want to do things like

show your top audit logs you need a

better format so how do we talk to

developers about these audit logs what

what can we suggest well I'm gonna pose

some very very very basic questions we

want to know who is doing something

typically this is a user what is that

user doing where is that occurring from

maybe that's what an IP address when did

it occur

why so you may don't always be able to

answer the question of why why did John

do login I don't know but if your audit

logs said we lock this user out because

the user failed to login five times

that's your reason why but you may not

have a Y for every single log so we take

this example log we have the timestamp

John Doe logged in from 1.2.3.4 with

OAuth so this gives us a little bit more

information it asks it answers all of

those questions however your developers

may have done this for OAuth but for

something like sam'l the format may have

changed even though the login was still

a login it was just a different type so

being able to search these it makes it

very difficult

so I'm gonna pose that we use a little

bit more structured format what if we

said user equals John Doe and we've used

the equal as a delimiter and we've put

quotations around that users name and

we've said that the event is an

authentication event the outcome of that

event was successful so we answered to

what questions here what was the event

and what was the outcome now we have an

IP address where they logged in from the

type of authentication and the type type

people's on it we've appended this audit

so that we can quickly search to search

and filter down to all of our audit

events now a machine can easily read

this you're seeing this imagine if you

had a grok parser or some sort of reg X

you can quickly filter down these key

value pairs and you get something that

looks like this user John Doe event and

then you can see how much easier it is

to gather that information about what

the specific event was and graph this

and display this and whatever way you're

doing your log management now how do you

get there

you want some sort of central audit log

function this is a very basic function

[Music]

pseudocode looks something like python

it takes in a couple of different

attributes it would take in the request

object the event and the outcome now

you'll see that the first thing I do in

the outcome is I put it allow list here

for success in failure and the success

and failure I do this because there's

many different ways to say success you

can say success successful or failure

you can say fail failed failure that

there's a lot of different ways to say

this so we just put a little allow list

here and require that they are in this

list the next thing we do is we put it

in that exact format and we're gonna let

the let the logger handle the timestamp

format so once you have this centralized

audit function I want you to monitor it

and what I mean by monitor is if there

are changes to this file maybe this is

just in like

if this was like say it's like lager pie

if there's ever a change to this file

you should know so whether that's

setting a code owners file in github or

having something in a CI check please

monitor make sure that it's when it has

changed

you are aware of that change so a couple

things to keep in mind when you're

talking about users may be you may be

asking the question you know was this

request from an API key or was it from a

browser session or what happens if the

user has five API keys and I need to one

of these API keys is compromised so I

need to know which one was the one

making the request so you should take us

in mind when adding this into your log

file next thing I want to talk about is

IP issues I've seen many many logs where

they are in the wrong where they're

logging the wrong IP address and

typically this happens in HTTP logs due

to the improper use of x-forwarded-for

headers you take a look at this if John

Doe makes a request to a web application

through a load balancer you may be

getting the IP address of the load

balancer and logging that into your

application looks that is not good that

is not what you want you want the

original IP address of John Doe so

please please please make sure if your

web application should only be external

to external applications maybe you can

make sure that the IP address matches an

external IP address format next thing I

want to talk about are joints if you

take that same scenario where you have a

load balancer in a web application you

may want to gather other information

from that initial request in your web

application log you may not have the

user agent but you want to know that

user agent information how do you gather

that information well if you don't have

an ID to join them together it's not

going to be possible

so I recommend adding a some sort of

request ID in here so you can trace it

through the entire application stack and

that should be logged in all the all the

logs so that you can quickly filter down

and see every

single every every single log that was

associated with that request so next

thing I want to talk about are some

considerations with which events to log

when we're talking about API we commonly

talked about crud create reads updates

deletes one of these is not like the

other reads reads are non state changing

API calls however they should be logged

but you're going to have a large number

of reads very very likely you're gonna

have a high number of reads so you may

need to consider the storage of the

reads differently than the stateful

configurations now don't know don't

think you don't have to log the reads

you should log the reads because you're

going to need say the data breach

happens and you want to identify what

content was accessed you want those

reads so what should you monitor for

this is just a when you're when you're

figuring out what's a log you need to

think about like what exactly are you

monitoring for and if you're in security

it's not only security use cases

there's also operational use cases you

need to ask so just think about this

write down exactly what you want to

monitor for and make sure that those

events make it into the logs a couple of

things extra things to ponder is how

you're storing your audit logs you know

are you storing these on disk and then

replicating them to you know some

offline or like s3 are you making sure

that these can only be like read once

right many like how are you storing

these audit logs how do you handle those

deletion request I mentioned earlier if

you have logged that information and in

a nice format it maybe is going to make

your life a lot easier to run a script

and delete that content that has to be

deleted with that that's that's all the

talk today Sam I'll hand it back to you