Splunk Components | universal forwarder | Heavy forwarder

Summary

This video introduces key components of Splunk, focusing on the Universal Forwarder and Enterprise software packages. It explains the roles of Heavy Forwarder, Indexer, Search Head, and Deployment Server, highlighting the importance of event parsing and filtering to optimize data indexing. The script also covers the architecture of a Splunk deployment, including the functions of the Cluster Master and License Master, and the significance of the Deployment Server in managing configurations across the system.

Takeaways

• 📚 The video introduces Splunk components and its two main software packages: Splunk Universal Forwarder and Splunk Enterprise.
• 🔄 Splunk Enterprise can perform various roles including Heavy Forwarder, Indexer, Search Head, Deployment Server, Cluster Master, and License Master.
• 🌐 Splunk Universal Forwarder (UF) is a separate, free software package that collects events from servers or endpoints without requiring a license.
• 🔍 UF is used for scenarios like monitoring continuously updated files or NTP service synchronization and can be managed by a Deployment Server.
• 🚫 UF cannot parse events, which involves breaking data into blocks, identifying timestamps, and adding meta fields like source and host.
• 🔑 Heavy Forwarder (HF) is a role of Splunk Enterprise that can parse events and apply filters to remove unwanted data, thus saving on Splunk licensing costs.
• 📈 The need for HF is recommended in larger deployments to offload the indexer's workload and improve performance.
• 🗂️ Splunk Indexer stores, indexes, and serves event data to the Search Head, which is crucial for handling search queries and generating reports.
• 🔑 Splunk Cluster Master manages the indexer cluster, including data replication and adjusting cluster buckets in case of peer node failure.
• 🔍 The Search Head is the interface for non-admin users to interact with Splunk, allowing them to run queries, generate reports, and create knowledge objects.
• 🛠️ Splunk Deployment Server acts as a centralized configuration manager, deploying updates to other instances and managing deployment clients.
• 🏢 Server Classes in Splunk are combinations of Deployment Clients and Deployment Apps, allowing for targeted configuration updates.
• 🛡️ Splunk License Master controls access to licenses for one or more license slaves, managing licensing volume and defining stacks and pools.

Q & A

What are the two main Splunk software packages mentioned in the video?

The two main Splunk software packages mentioned are Splunk Universal Forwarder and Splunk Enterprise.

What is Splunk Universal Forwarder (UF) and what is its purpose?

Splunk Universal Forwarder (UF) is a separate software package used for collecting events from servers or endpoints. It is free to download and does not require a license.

Can Splunk UF parse events?

No, Splunk UF is not capable of parsing events. Event parsing, which includes breaking data into blocks, identifying timestamps, and adding meta fields, is a capability of the Heavy Forwarder or Indexer.

What is the role of the Heavy Forwarder (HF) in Splunk Enterprise?

The Heavy Forwarder (HF) is an optional component of Splunk Enterprise that can parse and filter events, offloading some of the workload from the indexer and potentially saving on Splunk license costs.

How does the Deployment Server in Splunk Enterprise manage configurations?

The Deployment Server in Splunk Enterprise acts as a centralized configuration manager, deploying configuration updates to other instances, including Universal Forwarders and Heavy Forwarders.

What is an indexer in the context of Splunk Enterprise?

An indexer in Splunk Enterprise is responsible for storing, indexing, and serving the events to the search head. It is also referred to as a search peer if it is part of an indexer or cluster.

What is the function of the Cluster Master in a Splunk Enterprise setup?

The Cluster Master in Splunk Enterprise manages the indexer cluster, instructing where to stream replica data and adjusting cluster buckets. It also coordinates search head requests to the appropriate indexers.

What is the role of the Search Head in Splunk Enterprise?

The Search Head in Splunk Enterprise is the component that users interact with to run queries, generate reports, searches, dashboards, and create knowledge objects such as field aliases, calculated fields, lookups, event types, and tags.

What is the purpose of the Search Head Deployer in Splunk Enterprise?

The Search Head Deployer in Splunk Enterprise is used to deploy apps to the search head members of a cluster. It is recommended to use the Search Head Deployer instead of installing apps directly on search members.

What is a Deployment App and how does it relate to Server Classes in Splunk Enterprise?

A Deployment App is a unit of content deployed to the members of one or more server classes. A server class is a combination of deployment clients and deployment apps, allowing for the centralized management of configurations across similar systems.

What is the License Master role in Splunk Enterprise and how does it interact with License Slaves?

The License Master in Splunk Enterprise controls one or more License Slaves, providing them access to Splunk Enterprise licenses and managing the licensing volume. It allows for the definition of stacks, pools, and management of license slaves.

Outlines

00:00

🔍 Introduction to Splunk Components

This paragraph introduces the main components of Splunk, focusing on two primary software packages: Splunk Universal Forwarder and Splunk Enterprise. It explains the roles of Splunk Enterprise, such as Heavy Forwarder, Indexer, Search Head, Deployment Server, Cluster Master, and License Master. The paragraph also details the function of the Universal Forwarder (UF), its installation on servers or endpoints for event collection, and its limitations, such as the inability to parse events or apply filters. The role of the Heavy Forwarder in offloading indexer workload and its management by the Deployment Server is also highlighted.

05:03

🛠 Deep Dive into Splunk Enterprise Roles and Deployment

This paragraph delves deeper into the roles within the Splunk Enterprise package, including the Indexer, Search Head, Cluster Master, Search Head Deployer, Deployment Server, and License Master. It describes the indexer's role in storing and serving event data, the search head's function in displaying events and generating reports, and the Cluster Master's responsibility in managing the indexer cluster and data replication. The paragraph also explains the Search Head Deployer's role in deploying apps to search head members and the Deployment Server's function as a centralized configuration manager. Lastly, it discusses the License Master's role in controlling license slaves and managing licensing volumes.

Keywords

💡Splunk

Splunk is a suite of software used for searching, monitoring, and analyzing machine-generated big data, via a web-style interface. In the video's context, it is the central theme, as all discussed components and roles are part of the Splunk ecosystem.

💡Splunk Universal Forwarder (UF)

The Splunk Universal Forwarder is a lightweight software package that collects data from various servers or endpoints. It's an essential component in the Splunk architecture for gathering events, as mentioned in the script with examples of monitoring file updates and NTP service synchronization.

💡Event Parsing

Event parsing is the process of breaking down a data stream into individual events, typically lines of data, and adding metadata such as timestamps and source types. It's a crucial step for Splunk to index data correctly, as highlighted in the script when discussing the capabilities of the Universal Forwarder and Heavy Forwarder.

💡Heavy Forwarder

The Heavy Forwarder is a role that can be configured within the Splunk Enterprise package. It is more capable than the Universal Forwarder, with the ability to parse events and apply filters to remove unwanted data, thus saving on indexing resources and license usage, as explained in the script.

💡Deployment Server

The Deployment Server in Splunk is responsible for managing configurations and deploying applications to other Splunk instances, such as Universal Forwarders and Heavy Forwarders. It's a centralized configuration manager that streamlines the deployment process, as described in the script.

💡Indexer

An Indexer is a role within the Splunk Enterprise package that is responsible for storing, indexing, and serving search requests for data. It's a key component in the architecture, especially when discussing the performance implications of not having a Heavy Forwarder, as mentioned in the script.

💡Search Head

The Search Head is a role in Splunk that non-admin users interact with to run queries, generate reports, searches, dashboards, and manage knowledge objects. It's the interface through which users retrieve and analyze data stored in the Indexers, as explained in the script.

💡Cluster Master

The Cluster Master is a role that manages an indexer cluster in Splunk, ensuring data replication and load balancing. It directs the Search Head to the correct Indexers for data retrieval, as depicted in the script's discussion on Splunk architecture.

💡Search Head Deployer

The Search Head Deployer is responsible for deploying apps to the members of a Search Head cluster. Unlike standalone Search Heads, apps are not installed directly on Search Head cluster members, which is a point emphasized in the script.

💡Deployment App

A Deployment App in Splunk is a unit of content that is deployed to the members of one or more server classes. It's part of the deployment mechanism managed by the Deployment Server, as detailed in the script.

💡Server Class

A Server Class in Splunk is a combination of Deployment Clients and a Deployment App. It's used to categorize and manage similar configurations across different instances, as illustrated in the script with the example of deploying a TA to Windows servers.

💡License Master

The License Master is a role within the Splunk Enterprise package that controls access to Splunk licenses for one or more License Slaves. It manages licensing volume and can define stacks and pools for efficient license usage, as explained in the script.

Highlights

Introduction to Splunk components and two main software packages: Splunk Universal Forwarder and Splunk Enterprise.

Explanation of Splunk Universal Forwarder (UF) as a free, downloadable software for collecting events from servers or endpoints.

Examples of Universal Forwarder usage, including monitoring file updates and NTP service synchronization.

Description of how the Universal Forwarder can be managed by a Deployment Server.

Clarification that UF cannot parse events but Heavy Forwarder can, with an explanation of event parsing.

Details on Heavy Forwarder's ability to apply filters to remove unwanted events, saving on Splunk license usage.

Introduction to Splunk Enterprise's Heavy Forwarder role and its optional nature in small deployments.

Overview of Splunk Indexer's role in storing, indexing events, and serving search head requests.

Discussion on the importance of Heavy Forwarder in offloading indexer workload to maintain performance.

Architecture diagram explanation featuring Cluster Master, Search Head, Indexers, and Forwarders in a Splunk setup.

Function of Splunk Search Head in displaying events and generating reports, searches, dashboards.

Role of Splunk Cluster Master in managing the indexer cluster and data replication.

Description of how Search Head interacts with Cluster Master to fulfill user search requests.

Introduction to Splunk Search Head Deployer for deploying apps to search head members in a cluster.

Explanation of Splunk Deployment Server as a centralized configuration manager.

Definition of Deployment Client, Deployment App, and Server Class in the context of Splunk configurations.

Role of Splunk License Master in controlling license slaves and managing licensing volume.

Conclusion and call to action for likes, shares, and subscriptions for more educational content.

Transcripts

hello everyone my name is balaji in this video you will learn splunk components

let's look at two main splunk software packages which are used mainly in splunk

deployment first one is splunk universal forwarder

second one is splunk enterprise splunk enterprise software package can

be set up to perform any of the roles which i'll explain here

heavy forwarder indexer search head deployment server

cluster master license master deployer before understanding splunk enterprise

different roles first let's understand what is splunk universal forwarder

splunk universal forwarder in short we call it as splunk uf

splunk uf is a separate software package it is free to download

license is not required

we install splunk universal forwarder on servers or endpoints from where we want

to collect events let's understand few examples of

universal forwarder usage first example is you have a file which

is continuously getting updated with new events

you want to collect events as soon as the file is updated with new events

another example is you want to monitor ntp service

to check whether the server is syncing with your company ntp server or not

you can write a script add it to an application the application can be

deployed to universal forwarder installed on the server uf can be

managed by deployment server i'll explain what is deployment

server in this video later as i mentioned earlier application can

be deployed to universal forwarder the application deployment is possible

by deployment server UF cannot parse events let's

understand what is event parsing while uf reads the data stream from its

source breaks into 64k blocks breaking the stream of data

into individual lines is one of the feature of even parsing

identifying timestamp of each event happens at the time of parsing only

timestamp is crucial for each event in splunk because events are indexed with

their timestamps adding meta fields source source type

and host to each event happens at the time of event passing only

uf is not capable of parsing events but heavy forward is capable

uf cannot apply filters to remove unwanted events for example

you have a file to read events from you don't want to index certain events

based on some keyword criteria this is not possible with uf

having said that it can apply filters on windows events

splunk heavy power is one of the roles of splunk enterprise package

to have heavy forwarder in your deployment you should download splunk

enterprise only and then configure splunk enterprise to

act as heavy forwarder

splunk hf is an optional component because hf role can be added to an

indexer if splunk deployment is very small where

not much data is indexed per day

but am as a splunk consultant i recommend heavy forwarders to offload

indexes workload hf can be managed by deployment server

updating or creating new configurations or

applications on heavy forwarder can be done by deployment server

hf can parse events i have already discussed what is event parsing

event parsing can be done by heavy forwarder or indexer if there is no hf

hf can apply filters to remove unwanted events this will save splunk license

filtering of events cannot be done by universal forwarder but hf can do the

job this will save splunk license splunk

license works based on how much volume of data we are indexing per day

splunk indexer is another role of splunk enterprise package

this is example splunk architecture diagram where the architecture is set up

with cluster master search head three indexers two

forwarders peer nodes highlighted in red color box

are nothing but indexers

splunk indexer is mainly used to store events indexes them

and serve requests of splunk search head if no hf parsing and filtering of events

will be done by indexer this is the main reason i have mentioned

splunk heavy power is an optional but having no hf will

overload indexer and reduce performance of splunk search

if it is not properly sized its cpu and memory

indexers are also called search peers if they are part of indexer or cluster

you can see in the diagram indexers are called peer nodes because they are

participating in indexer cluster

indexers are managed by cluster master node if they are an indexer or cluster

you can see red color double sided arrow connecting master node and each peer

node in the right side diagram splunk search

head is another role of splunk enterprise package

splunk users who are not splunk admins mainly interacts with splunk search

splunk search head shows events that are stored in indexer when

user runs query

splunk search head is used to generate reports searches

dashboards and also to create field aliases calculated fields

lookups event types and tags in splunk terms we call reports searches

dashboards fields lookups event types tags knowledge objects

splunk cluster master is another role of splunk enterprise package

splunk cluster master manages indexer cluster

you can see there are three indexers and replication factor is set to 3

which means when forwarder starts sending data to indexer

the data should be replicated in 3 indexers let's say there are 10 indexers

and indexer which is receiving data doesn't know that to which indexer

replication copy of data should go instruction will be given to indexer by

cluster master where to stream replica in case of peer

node failure adjusting cluster buckets will also be

done by cluster master when user types something in search of

search head and hit search button search head will first contact cluster

master to understand where the data exactly resides

cluster master will instruct search head for example

the data you are looking for is available in indexer 3

and 4 then search head will contact indexer 3 and 4

to fulfill user search request

creating new index applying props or transforms

will be done on cluster master and then they will be deployed to peer nodes

unlike indexers master node doesn't index data

only one master node for one index or cluster

splunk search head deployer is another role of splunk enterprise package

search head deployer is used to deploy apps to search head members of such a cluster

unlike we install apps on standalone search head directly

it's not recommended to install apps directly on search members

in search head cluster

splunk deployment server is another role of splunk enterprise package

you can see here in the picture single deployment server below

and then server class and deployment clients

a splunk enterprise instance that acts as centralized configuration manager

it deploys configuration updates to other instances

also refers to the overall configuration update facility

compromising deployment server clients and apps

what is deployment client a remotely configured splunk universal forwarder

or enterprise instance it receives updates from the deployment server

what is deployment app a unit of content deployed to the members of one or more

server classes what is server class server class is

nothing but combination of deployment client and deployment app for

example you have windows ta which contains

inputs to collect windows event logs and you want to deploy this ta to all

windows servers the new server class will combine ta which is basically

deployment app and windows servers which are basically deployment clients

the deployment client can belong to multiple server classes

splunk license master is another role of splunk enterprise package

license master controls one or more license slaves

what is license slave the license master provides

its slaves access to splunk entropage licenses and

in the case of indexers associated licensing volume

from the license master we can define stacks pools

on licensing capacity and manage license slaves

thanks for watching please like share and subscribe for more videos