Introducing Ingest Processor: An Evolution in Splunk Data Management

Summary

In today's challenging business environment, leaders must navigate macroeconomic pressures and budget constraints while maintaining compliance. Splunk’s Ines Processor addresses these challenges by enabling efficient data management. It allows businesses to build pipelines that connect data sources to platforms like Splunk and Amazon S3, facilitating data filtering, masking, cleansing, and log-to-metrics conversion. With a focus on infrastructure monitoring, the Ines Processor helps streamline Kubernetes log data, ensuring only relevant information is sent to the right destinations. This system enhances operational efficiency and long-term data storage, all while supporting regulatory compliance.

Takeaways

• 😀 Business leaders today face challenges like navigating macroeconomic conditions and budgetary constraints.
• 😀 To overcome these challenges, businesses must run more efficiently while staying compliant with tightening regulations.
• 😀 Splunk's new data management capabilities help businesses operate more efficiently and maintain compliance.
• 😀 The Ines processor by Splunk allows you to build data pipelines that connect sources to Splunk or third-party platforms.
• 😀 Key features of the Ines processor include filtering, masking, data cleansing, and converting logs to metrics.
• 😀 The Ines processor routes data to various destinations such as Splunk, observability cloud, or Amazon S3 for long-term storage.
• 😀 Kubernetes logs are often voluminous, but the Ines processor helps focus on essential data like HTTP response codes and levels for infrastructure monitoring.
• 😀 The Ines processor can filter out unnecessary data and send relevant metrics to Splunk Observability Cloud for easier monitoring.
• 😀 The pipeline preview feature allows users to test and see how the processed data looks before sending it to destinations.
• 😀 Raw logs can be sent to Amazon S3, and Splunk Federated S3 can be used to read that data back into Splunk for analysis.
• 😀 Splunk's Ines processor streamlines data management and ensures businesses can navigate compliance challenges effectively.

Q & A

What are the primary challenges businesses face today, as mentioned in the script?

Businesses today are faced with navigating macroeconomic conditions and budgetary constraints, which compel them to operate more efficiently. Additionally, they must maintain compliance with tightening regulations.

How does Splunk's new data management capability help businesses manage these challenges?

Splunk's new data management capabilities allow businesses to run their workloads more efficiently while also ensuring compliance with regulations. This is achieved through advanced features like the INI Processor, which optimizes data processing and routing.

What is the role of the Splunk INI Processor?

The Splunk INI Processor enables users to build data pipelines that connect data sources to Splunk or third-party platforms. It provides features like data filtering, masking, cleansing, and normalizing, and can convert logs to metrics, routing them to appropriate destinations like Splunk Index or Observability Cloud.

How does the INI Processor handle Kubernetes logs?

The INI Processor extracts key-value pairs from Kubernetes logs, such as 'level' and 'HTTP response code,' which are important for monitoring infrastructure. This allows users to efficiently process and route only the relevant data to their desired destinations.

What are the two destinations configured in the demo for the INI Processor?

In the demo, the two destinations configured for the INI Processor are an Amazon S3 bucket and Splunk Observability Cloud.

How can users test their data pipelines in the Splunk INI Processor?

Users can test their data pipelines using the 'preview' feature, which shows how the data will be processed and sent to the configured destinations. This helps ensure the pipeline functions as expected before going live.

What happens when you switch the preview from S3 bucket data to the Observability Cloud in the INI Processor?

Switching the preview from the S3 bucket to the Observability Cloud allows users to see the transformed data in the cloud environment. This provides a real-time view of how the data will appear in the Observability Cloud for monitoring purposes.

What is the significance of using Splunk Federated S3 with the INI Processor?

Splunk Federated S3 allows users to read the raw logs stored in the Amazon S3 bucket back into Splunk. This is useful for future data analysis and ensures that long-term storage is seamlessly integrated with Splunk for ongoing monitoring and reporting.

How does the Splunk Observability Cloud help with monitoring Kubernetes infrastructure?

The Splunk Observability Cloud helps monitor Kubernetes infrastructure by providing a dashboard where metrics from the INI Processor are visualized. Users can create charts and track key metrics to ensure the health of their infrastructure.

What are the benefits of using the INI Processor for data routing?

The INI Processor simplifies the process of data routing by allowing users to filter and transform data on-the-fly, ensuring that only relevant data is sent to monitoring platforms like Splunk Observability Cloud, or stored efficiently in Amazon S3 for long-term use.