Installing and Configuring Logstash to Ingest Fortinet Syslogs

Summary

This tutorial video guides viewers through the process of installing and configuring Logstash to send syslogs from a Fortigate firewall to Elasticsearch. The presenter, building on a previous video, demonstrates setting up the Logstash input, filter, and output plugins, including parsing and enriching data. The video also covers troubleshooting steps, such as adjusting firewall settings and ensuring correct permissions for certificates, to ensure a smooth data flow into Elasticsearch for indexing and visualization in Kibana.

Takeaways

• 😀 The video is a tutorial on installing and configuring Logstash to send syslogs from a firewall to Elasticsearch.
• 🔧 Part two of a previous video series, where Elasticsearch and Kibana were installed and configured to work together.
• 💻 The presenter uses Oracle Linux and installs Java 11 or 17 as a prerequisite for Logstash, using the command `yum install java-1.8.0-openjdk`.
• 📝 Logstash is introduced as an open-source data collection engine that can enrich data, like adding geo-locations.
• 📑 The presenter guides through the process of setting up Logstash using a `yum` repository and a public signing key.
• 🔌 The video demonstrates configuring the firewall to send syslogs to Logstash, specifying the IP address and port 5144.
• 📡 The Logstash configuration file is detailed, explaining the input, filter, and output plugins necessary for processing the syslog data.
• 📚 The use of the 'grok' filter is highlighted for parsing syslog messages into structured data.
• 🛠 The 'mutate' filter is used to clean up the log data by removing unnecessary fields like the original syslog priority value.
• 🗓️ The 'kv' filter is introduced to parse key-value pairs from the log messages, and a 'date' filter is used to convert log timestamps into a usable format for Elasticsearch.
• 🔒 The video concludes with securing data transmission to Elasticsearch using SSL, including copying the CA certificate and setting appropriate file permissions.
• 📊 Finally, the presenter shows how to view the ingested data in Kibana and mentions future plans to create dashboards for data visualization.

Q & A

What is the purpose of the video?

The purpose of the video is to demonstrate the installation and configuration of Logstash to send syslogs from a firewall to Elasticsearch, which is a continuation of a previous video where Elasticsearch and Kibana were installed and configured.

What are the prerequisites for installing Logstash as mentioned in the video?

The prerequisites for installing Logstash include having Java installed, specifically JDK 11 or JDK 17, and the video uses JDK 17.

How does the video demonstrate the installation of Logstash?

The video demonstrates the installation of Logstash by using the yum repository, downloading the public signing key, creating a repository file, and then executing the 'yum install logstash' command.

What is the default port used for syslog in the video?

The default port used for syslog in the video is 5144.

How does the video handle the firewall configuration for sending syslogs?

The video shows how to log in to the firewall, configure syslog settings, enable the syslog server, set the IP address of the syslog server to the Logstash server, and specify the port to 5144.

What are the three main components of Logstash configuration files?

The three main components of Logstash configuration files are the input plugin, filter plugin, and output plugin.

What is the purpose of the 'grok' filter plugin used in the video?

The 'grok' filter plugin is used to parse arbitrary text and structure it, allowing the user to define patterns for extracting data from logs.

How does the video address the issue of viewing the data before applying filters?

The video suggests sending the output to the console (stdout) initially to view the data before applying any filters for further processing.

What is the significance of setting the correct permissions for the certificate in the video?

Setting the correct permissions for the certificate is crucial for the Logstash service to run without issues, as it ensures that the service can access and use the certificate for secure communication with Elasticsearch.

How does the video demonstrate troubleshooting steps for Logstash?

The video demonstrates troubleshooting by showing the process of giving the correct permissions to the certificate file, changing the file owner to 'logstash', and then successfully restarting the Logstash service.

What is the final step shown in the video for confirming that data is being sent to Elasticsearch?

The final step shown in the video is logging into Kibana, navigating to the 'Discover' section, and confirming that the data from the firewall is being displayed in real-time.