Log Data - CompTIA Security+ SY0-701 - 4.9

Summary

This script delves into the critical role of log files in network security, detailing how they document traffic, exploit attempts, and potential threats. It highlights the insights gained from firewall logs, endpoint devices, and SIEM systems, and underscores the importance of analyzing metadata and packet captures for a holistic security approach.

Takeaways

• 🗂️ Storing Security Logs: The script emphasizes the importance of storing a vast amount of security-related information in log files across various network components like servers and devices.
• 🚫 Traffic Analysis: Log files document traffic flows, including both allowed and blocked traffic, providing insights into network security and potential threats.
• 🔎 Intrusion Detection: Intrusion prevention devices can offer detailed lists of exploit attempts, helping in identifying and mitigating security breaches.
• 🌐 URL Categorization: The script highlights the ability to view categories of URLs that may be blocked on a user's workstation, which is crucial for understanding web traffic control.
• 🔍 DNS Sinkhole Traffic: Monitoring DNS sinkhole traffic can indicate malicious processes within the network, a valuable insight for security analysts.
• 🔒 Firewall Logs: Firewall logs are a rich source of information about traffic flows, including source and destination IP addresses, port numbers, and the actions taken by the firewall.
• 🌟 Next-Gen Firewalls: Next Generation Firewalls (NGFW) provide detailed insights into applications in use and can identify suspicious data or anomalies within traffic flows.
• 📝 Application Logs: Logs from applications like Windows Event Viewer or Linux /var/log directory are crucial for analyzing security events and can be integrated into a SIEM for comprehensive analysis.
• 📲 Endpoint Device Logs: Endpoint devices such as laptops and smartphones contain detailed logs that can be aggregated to a SIEM for a holistic view of network activities.
• 🛡️ Operating System Logs: Operating systems maintain security event logs that can alert to potential security issues, such as unauthorized service disablement or system file changes.
• 🚀 IPS/IDS Events: Intrusion prevention and detection systems provide logs on known vulnerabilities and attacks, which are essential for proactive security measures.
• 📑 Metadata Analysis: The script points out the value of metadata in documents and emails, which can reveal hidden information about file origins and transfer processes.
• 🔄 Vulnerability Scans: Logs from vulnerability scans are vital for identifying and rectifying security weaknesses in the network, such as misconfigured devices or unsupported operating systems.
• 📊 Automated Reporting: SIEMs can generate automated reports that summarize security data, but their effectiveness depends on regular review and action by security teams.
• 📊 Dashboard Overview: SIEMs and security devices often offer customizable dashboards for quick, at-a-glance insights into network security status and active alerts.
• 🔎 Packet Analysis: Network packet analysis with tools like Wireshark provides detailed insights into traffic flows at the packet level, aiding in the identification of security issues.

Q & A

What types of information are typically stored in security log files?

Security log files store information such as traffic flows that were blocked or allowed, exploit attempts, categories of URLs that may be blocked, and DNS sinkhole traffic, which can indicate malicious processes within the network.

How do firewall logs contribute to network security?

Firewall logs provide detailed information about traffic flows, including the source and destination IP addresses, port numbers, and the actions taken by the firewall, such as allowing or blocking the traffic. This helps in documenting and analyzing potential security events.

What is a Next Generation Firewall (NGFW) and how does it enhance security?

A Next Generation Firewall (NGFW) is an advanced firewall that not only monitors traffic but also provides information about the applications in use, feedback on URLs or URL categories, and can identify suspicious data or anomalies within traffic flows.

How can endpoint devices contribute to security monitoring?

Endpoint devices such as laptops, desktops, phones, and tablets contain log details about login and logout events, system events, processes, device management like password changes, and directory services. These logs can be aggregated to a SIEM for comprehensive security analysis.

What is a Security Information and Event Manager (SIEM) and its role in security?

A Security Information and Event Manager (SIEM) is a system that aggregates log data from various sources within the network, allowing for the parsing and correlation of data to identify and respond to security incidents.

What kind of information can be extracted from operating system logs?

Operating system logs can provide information about security events, such as brute force attacks, changes to critical system files, and authentication-related activities. They can also alert to unusual activities like the disabling of essential services.

How do Intrusion Prevention Systems (IPS) or Intrusion Detection Systems (IDS) contribute to network security?

IPS and IDS systems monitor network traffic for suspicious activities and known vulnerabilities or attack patterns. They log detailed information about potential threats, which can be integrated into a SIEM for further analysis.

What insights can be gained from analyzing metadata in documents transferred over a network?

Metadata in documents can reveal information about the file's creation, the creator's details, GPS coordinates for images, and other hidden data that can be crucial for understanding the context and origin of the document.

How do vulnerability scans help in identifying security weaknesses in a network?

Vulnerability scans identify devices without proper security configurations, such as missing firewalls or antivirus software, misconfigured shares, and operating systems with guest access enabled, which can be exploited by attackers.

What is the importance of automated reporting in SIEM systems?

Automated reporting in SIEM systems helps in efficiently generating security insights and summaries, which are crucial for making informed security decisions. However, the value of these reports depends on their regular review and action by the security team.

How can network packet analysis tools like Wireshark contribute to security monitoring?

Network packet analysis tools capture and analyze data at the packet level, providing detailed insights into traffic flows, applications, and potential security issues. This granular data can be invaluable for diagnosing and responding to network security incidents.

Outlines

00:00

🔒 Network Security Log Analysis

This paragraph discusses the importance of analyzing log files for network security. It covers the types of information stored in these logs, such as traffic flows, blocked and allowed traffic, exploit attempts, and DNS sinkhole traffic. The role of firewall logs, especially Next Generation Firewalls (NGFWs), in providing detailed traffic information, including source and destination IP addresses, port numbers, and application usage, is highlighted. The paragraph also touches on the integration of these logs into a Security Information and Event Manager (SIEM) for comprehensive security event analysis.

05:00

🕵️‍♂️ Gathering Security Intelligence from Various Sources

The second paragraph delves into the various sources from which security intelligence can be gathered, including IPS/IDS events, network infrastructure devices, and metadata from documents and emails. It explains how these sources provide valuable insights into potential security threats and vulnerabilities, such as denial of service attacks, authentication errors, and misconfigurations. The paragraph emphasizes the integration of this information into a SIEM for automated reporting and analysis, as well as the importance of not ignoring these reports for effective security management.

10:00

📊 Utilizing SIEM Dashboards and Network Analysis Tools

The final paragraph focuses on the practical use of SIEM for security monitoring and analysis. It discusses the generation of automated reports and the challenges of creating efficient and relevant reports from vast amounts of data. The importance of dashboards for quick, at-a-glance insights into the network's security status is highlighted, along with the customization options available. The paragraph concludes with a discussion on network packet analysis using tools like Wireshark, which provides detailed traffic flow information at the packet level, essential for identifying and understanding complex security issues.

Keywords

💡Log Files

Log files are records of events that occur within an IT system. They are essential for tracking and analyzing security-related information, as they document actions such as traffic flows that were allowed or blocked. In the video's context, log files are mentioned as a primary source of data for understanding network activity and potential security threats, such as exploit attempts and DNS sinkhole traffic.

💡Exploit Attempts

Exploit attempts refer to efforts by attackers to take advantage of vulnerabilities in a system to gain unauthorized access or perform malicious actions. The script mentions that one can get a list of such attempts, particularly from intrusion prevention devices, which is crucial for identifying and mitigating potential security risks.

💡Intrusion Prevention System (IPS)

An Intrusion Prevention System is a security solution designed to detect and prevent malicious network activities in real time. The script highlights the importance of IPS in providing feedback on suspicious data or anomalies within traffic flows, which helps in identifying and stopping attacks before they cause damage.

💡Next Generation Firewall (NGFW)

A Next Generation Firewall is an advanced form of firewall that not only filters traffic based on IP addresses and port numbers but also examines the content of the traffic for potential threats. The video script describes how NGFWs can provide detailed information about applications in use and URL categories, enhancing the security monitoring capabilities of a network.

💡Security Information and Event Management (SIEM)

SIEM systems aggregate and analyze log data from various sources to identify security threats and provide real-time analysis of security alerts. The script discusses how all the information from different devices can be rolled up into a SIEM, allowing for comprehensive monitoring and correlation of security events across the network.

💡Endpoint Devices

Endpoint devices are any devices that connect to a network, such as laptops, desktops, phones, and tablets. The script mentions that these devices contain a wealth of log detail, including information on log in and log off events, system events, and device management activities, all of which can be crucial for understanding the security posture of a network.

💡Metadata

Metadata is data that provides information about other data. In the context of the video, metadata in documents, emails, and images can reveal details about their origin, creation, and handling. The script gives examples of metadata in emails, such as IP addresses and SPF information, which can be vital for tracing the source of security threats.

💡Vulnerability Scans

Vulnerability scans are processes used to identify, quantify, and prioritize vulnerabilities in a system. The script describes how these scans generate extensive log information, which can reveal misconfigurations, unsupported operating systems, and other potential security weaknesses that need to be addressed.

💡Dashboard

A dashboard in the context of security operations is a user interface that provides a real-time overview of the system's status, highlighting important information for quick assessment. The script mentions that SIEM and other security devices often include customizable dashboards to display critical data at a glance, aiding in efficient monitoring and response to security events.

💡Packet Analysis

Packet analysis involves examining the individual packets of data that are transmitted over a network to understand traffic flows and identify potential security issues. The script describes using utilities like Wireshark for capturing and analyzing network packets, providing detailed insights into network operations and helping to detect anomalies or attacks.

💡Automated Reports

Automated reports are generated by SIEM systems to provide periodic summaries of security events and log data. The script emphasizes the importance of these reports for ongoing security monitoring but also points out the potential issue of organizations ignoring them, highlighting the need for effective report management and attention.

Highlights

Storing massive amounts of security-related information in log files across network components allows documenting traffic flows and potential attacks.

Log files can provide lists of exploit attempts, especially from intrusion prevention devices.

Monitoring categories of URLs blocked on user workstations can identify potential security threats.

DNS sinkhole traffic may indicate malicious processes occurring within the network.

Firewall logs provide detailed information about traffic flows, including source and destination IP addresses, port numbers, and whether traffic is allowed or blocked.

Next Generation Firewalls (NGFW) offer insights into applications in use and can flag suspicious URLs or anomalies.

Firewall logs from NGFWs display traffic flow details such as time, date, source IP, MAC address, destination IP, application, and disposition.

Application logs from tools like Windows Event Viewer or /var/log directories on Linux/MacOS can be crucial for analyzing security events.

Endpoint devices like laptops, phones, and tablets contain extensive log data on events like log in/out, system events, and device management.

Endpoint logs can be aggregated in a Security Information and Event Manager (SIEM) for correlation with network and device data.

Operating systems maintain security event logs that can alert on issues like brute force attacks or unauthorized changes to critical system files.

Intrusion Prevention Systems (IPS) or Intrusion Detection Systems (IDS) provide detailed logs on known vulnerabilities and attack types.

Network infrastructure devices like switches and routers generate logs that can identify changes to routing tables or authentication errors.

Metadata in documents and emails can reveal important information about file origins, transfer processes, and user details.

Vulnerability scans create logs detailing identified issues like misconfigured devices, unsupported operating systems, and unpatched vulnerabilities.

SIEMs can generate automated reports and dashboards for efficient analysis and real-time monitoring of security data.

Packet analysis tools like Wireshark provide deep insights into network traffic at the packet level, aiding in the identification of security issues.

Transcripts

We store a massive amount of security related information

in log files contained in servers, devices,

and other components on our network.

Those log files contain information such as the traffic

flows that were blocked and the traffic

flows that were allowed.

We can get a list of exploit attempts,

especially from intrusion prevention devices.

We might want to see what categories of URLs

may be blocked on a particular user's workstation.

And we can see DNS sinkhole traffic,

which could point to some type of malicious process occurring

within our own network.

This allows us to document every traffic flow

on the network, where we can provide information

on what attacks may be occurring,

and we can correlate that information

with other logs contained in other devices.

One of those devices that contains an amazing amount

of detail about the traffic on our network

is the firewall log.

Our firewalls are often monitoring all traffic

that goes from the inside to the outside of our network and vice

versa.

And from there, we can get information

about the source and destination IP addresses associated

with those traffic flows.

We can see what port numbers are being used.

And we can see what the firewall does with those traffic flows.

Does the firewall allow the traffic flow to proceed?

Or is that traffic flow blocked on the firewall?

If you're using a Next Generation Firewall, or NGFW,

you can also get information about the applications

that are in use.

These next generation firewalls are also

very good at providing feedback on URLs

or URL categories that are being used.

And they may also be able to point us

to suspicious data or any anomalies

with the information within these traffic flows.

Here's a view of firewall logs from a next generation

firewall.

Each one of these lines is a separate flow

of traffic traversing the network.

Each one of these traffic flows contains the time and date

of the traffic flow, the source IP address,

and in the case of this firewall,

the MAC address that received that data.

You can also see the destination IP

address and the application that is used for this traffic flow.

And then you finally get to see the disposition, or result

of this traffic flow whether it was

accepted through the firewall or whether it was blocked.

The applications that we use can also

create log files that may be very useful when

analyzing security events.

This might include information from the Windows Event Viewer

log, specifically the application log

section of the Event Viewer.

If you're using Linux or Mac OS, you can look in the /var/log

directory.

And all of this information would probably

be rolled up into one single security information and event

manager, or SIEM.

And all of this information can then

be filtered out or viewed in different ways inside

of the SIEM itself.

You may not realize it, but the endpoint devices

you're using also contain a great deal of log detail.

If you're using a laptop, a desktop, a phone, a tablet,

or any other endpoint device, there

is an extensive amount of log information available.

For example, you may be able to view information on log

in and log off events.

Maybe there's information on system events or processes

running on that endpoint.

And there might be details about the management of the device,

such as password changes or lock outs.

And you can also view information

on any directory services.

All of these endpoint logs can also be rolled up to a SIEM

so that you can then parse out that data

and view different correlations between what's

happening on the endpoint, what's happening

on the network, and any other devices that you

may be monitoring.

Once you have all of this endpoint log

information in the SIEM, you can now

start comparing and correlating that data against log

file information from other devices.

So you can track each step of the way

for a particular traffic flow or a potential security event

inside of the SIEM, all by consolidating these log

files together into one single source.

There's also an extensive amount of security information stored

in the operating systems themselves.

Many operating systems keep a log

file associated with security events.

So you can monitor individual applications.

You can see if there are brute force attacks or any changes

to critical system files, and anything

relating to authentication is usually also stored

in these security log files.

This log file information inside the operating system

may be able to provide you with a heads up

of any type of security events.

For example, this log file might show that a particular service

was disabled.

And that service is not one that would normally be manually

disabled by the administrator.

That single log file event may cause a security alert

to be generated, and you may be able to stop a particular event

from occurring just by monitoring this log data.

As you can imagine, we are collecting a very large amount

of data across all of our systems.

And so we may not want to send all of our log information

into a SIEM, but instead, only send the information

that's important for us to be able to make security

decisions.

Another great place to gather information

is from IPS or IDS events.

This would be associated with intrusion prevention systems

or intrusion detection systems.

These days, we don't tend to use standalone IPS or IDS systems.

Instead, that functionality is often

built into a next generation firewall.

An IPS log is going to provide information

about known vulnerabilities or known types of attacks.

So it might look something like the log

file I've taken from an open source IPS called Snort.

In this IPS, we have a timestamp.

It shows us the class of the alert,

and it tells us that it's a possible denial

of service attack, specifically a SYN flood attack.

It has a priority inside the IPS of two.

It gives us a source IP address, a source port number,

and a destination IP address and destination port number.

We can now take all of these individual events

and also roll those up into our SIEM

so that we can now start extracting and correlating

this data with all of the other devices on our network.

There's also a great deal of log information

we can gather from our network infrastructure devices.

This might include our switches, our routers,

our wireless access points, or even VPN concentrators.

These log files can identify any changes

that might occur to any of our routing tables.

We might be able to identify authentication errors that

are occurring to someone trying to gain access to a switch

or to a router.

And we might also be able to identify

other types of attacks that are occurring on the network.

For example, in this log, we can see

there is an informational entry that

shows a TCP SYN attack was identified on port gigabit

eight.

And we can see that the TCP SYN traffic destined

to the local system has been automatically blocked

for 60 seconds.

Sometimes, you can gather important information

that is contained within the documents that were

transferring over the network.

Stored inside of the documents that we are often

creating in our word processors, our spreadsheets, our graphics

programs, and others is information that

describes that particular file.

For example, if you're reading an email in your email client,

you don't often see the large amount of metadata

that's stored within the header of that email.

But if you ask your email client to show you the entire email

document, you can often see information

that's hidden within the email headers, things

such as the servers that sent the email or the addresses that

were specified as the destination.

You can also see this if you look

at the description of pictures taken with a mobile device.

You may be able to tell what mobile device was used

and even information about the GPS

coordinates that were associated with this location

of the picture.

There's even metadata in the browsers that we're using,

things like the operating system that we're using,

the browser type, or the IP address that you're using.

And if you look at the metadata that's inside

of a word processing document or a spreadsheet,

you may find information on the person that

created that document, their address, their phone number,

and perhaps their title.

To give you an idea of what some of this metadata

might look like, let's look at the header of an email message.

You can see there is extensive information

in this email header including the IP addresses

that this message was received by.

We can see SPF information, other details about signatures

and information that can help you determine where this email

message originated, and the process

that it used to be transferred into your inbox.

If you're performing vulnerability scans

on your network, then you're creating an extensive amount

of log information.

This is going to give you details

about what this vulnerability scam was able to identify.

For example, it may identify devices

on your network that don't have a firewall configured.

There may be no antivirus on that device or anti-spyware.

And we're able to identify that in the logs

of our vulnerability scans.

We might also be able to identify

devices that are misconfigured.

For example, there may be shares that

are available that you can access

without using any type of username or password.

Or perhaps there's an operating system

that has the guest access turned on when

the best practice for your organization

might be to completely disable all guest accounts.

And of course, once you update the signatures inside

of your vulnerability scanner, it

can identify any operating systems or applications

which may have known vulnerabilities

that need to be patched.

Here's a summary of the results from a vulnerability scan.

This log information shows that there are certain operating

systems that were running on our network that are currently

unsupported.

And there might even be NFS shares

that are on our network that are readable by anyone

in the world.

As you've probably seen already, there

is an extensive amount of log information

that you would need to go through

to be able to find details hidden within all of this data.

Fortunately, most SIEMs have the ability

to create a set of automated reports.

This may be a feature that is built into the SIEM itself,

or you may be able to use a third party report

generator to simply access the information that is currently

stored in the SIEM.

Of course, these reports aren't very valuable

unless someone actually reads them.

And one stumbling block that many organizations will find

is they will create these automated reports,

but then simply ignore them when they arrive in their inbox.

This might also take a bit of finesse

to be able to find the right mix between the type of report

that you need and the amount of time

that it takes to create that report.

When you have a SIEM that may contain terabytes and terabytes

of data, it may take an extensive amount

of processing power just to create a single report.

So you may need to be very specific about the reports

that you'd like to generate so that you can create them

as efficiently as possible.

Instead of waiting for a day or two

to receive reports that are generated by the SIEM,

it might be useful to have a summary of information

that you could view at a glance.

This is often available through a dashboard.

Many SIEM and other security devices

allow you to customize a screen, containing

information that will be important to review

at a glance.

These dashboards may be customizable,

or there might be predefined dashboards

built into the SIEM itself.

This is often the type of data that's

useful to see at a glance or on the main screen

of your security operations center.

This commonly doesn't show long-term information primarily

because it takes such a long amount of time

to be able to compile all of that data together.

The dashboard is designed to give you something

that you can view instantly and get an understanding of how

the current status might be.

For example, you can see a breakdown of the system itself,

any active firewall rules, you can

see warnings that have come through,

and information about users and devices

that might be on the network.

One of the best places to gather data on your network

is from the network itself.

Being able to analyze the packets going over the network

can give you a great deal of insight into the operations

of the networking equipment, the applications, and any security

issues.

This may require that you use a third party

utility, such as the Wireshark utility that we see here.

This may be able to provide you with the ability

to capture data that's running across the wired network

and the wireless network.

And in some cases, devices like switches, routers, or firewalls

may have the ability to capture packets inside

of those devices themselves.

These captures give us detailed information about traffic flows

at the packet level.

Everything going across the network is captured,

and that allows us to analyze every bit and byte that's

transferred over the network.

The Wireshark summary view here in the top pane

shows a packet by packet breakdown of everything that's

being sent over the network.

You can see the highlighted frame is one that's

being sent as HTTP traffic.

And you can see the GET command is written inside of the packet

capture itself.

You can then see all of the following packets that

describe the transfer of this file that was requested

with the GET command.

The bottom half of the screen is the detail pane.

This provides us with a breakdown

of everything highlighted in that single frame

at the top of the page.

In this case, we can see the ethernet data.

We can see what's involved in the IPv4 header.

We have details about the TCP header.

And then we have the HTTP data itself

being shown all in that detail view.