Managing Observability Data at the Edge with the OpenTelemetry Collector and OTTL - Evan Bradley

Summary

In this insightful session, Evan, a contributor to the OpenTelemetry Collector and OTL, introduces the audience to the Collector's capabilities and OTL's role in observability pipelines. He presents a case study of 'Global Telescopes' to demonstrate how to use OTL for data processing, including filtering, parsing, and redacting sensitive data. The session also covers routing and sampling strategies, showcasing the flexibility of OTL for telemetry data management. Evan wraps up with a sneak peek into upcoming features and concludes with a Q&A, inviting participants to explore and test OTL's capabilities.

Takeaways

• 📈 Evan has been contributing to The otel Collector and OTL for about two years.
• 🔄 The otel Collector is a middleware in observability pipelines, processing and routing data through its internal pipeline model.
• 🔍 OTL is a language used for reading from and writing to data as it flows through the otel Collector, offering flexibility and a common configuration format.
• 🏭 A hypothetical case study is presented involving a company called Global Telescopes, which has applications hosted worldwide and deals with data privacy and telemetry processing.
• 🛠 The Collector processes data through receivers, processors, and exporters, allowing for filtering, enriching, and routing of data.
• 🔐 Data redaction and handling of PII are demonstrated, including the use of hashing (SHA-256) to protect sensitive information.
• 📊 The routing and sampling of data in a centralized collector are covered, with a focus on handling errors and payment service data.
• 📄 Examples of OTL statements and configurations are provided, illustrating the setup and processing within the otel Collector.
• 🆕 New features include optional parameters, functions as parameters, and additional functions to enhance the capabilities of OTL.
• 🔧 Future improvements aim to handle list values and stabilize the transform processor, making the system more user-friendly.

Q & A

What is the main focus of Evan's presentation?

Evan's presentation focuses on The Otel Collector and OTL, including an introduction to these tools, a hypothetical case study, and how to apply these components to various setups.

What are The Otel Collector and OTL?

The Otel Collector is a middleware in observability pipelines that processes and routes data. OTL is an easy-to-read language that allows for reading from and writing to data as it flows through the collector.

How does the internal pipeline model of The Otel Collector work?

The internal pipeline model of The Otel Collector consists of different components, such as receivers, processors, and exporters. Data enters through receivers, is processed by processors, and is sent out by exporters.

What is the purpose of connectors in The Otel Collector?

Connectors in The Otel Collector are used to connect pipelines and perform tasks such as routing data. They add flexibility to the pipeline model.

What is the main advantage of using OTL in The Otel Collector?

The main advantage of using OTL is its flexibility and common configuration format, which allows users to work with data in the collector without worrying about input or output formats.

What is the hypothetical case study presented by Evan?

The case study involves a company called Global Telescopes, which needs to handle data from applications hosted worldwide, comply with local data privacy laws, and process telemetry data through sidecar collectors and a centralized collector.

How does the filter processor help in the case study?

The filter processor is used to filter out unnecessary data, such as noisy logs at debug level, to reduce the amount of data that needs to be processed further.

What is the purpose of redacting data in the case study?

Redacting data, such as purchase IDs considered PII, ensures that sensitive information is not exposed when data leaves the region. It also helps in handling data deletion requests by hashing the PII.

How does the routing connector function in the case study?

The routing connector directs data to the appropriate backend based on annotations. Data from newly acquired teams is routed to their old backend, while other data is routed to the company-wide backend.

What new features have been added to OTL recently?

Recent additions to OTL include optional parameters, functions as parameters, and 15 new functions. Future improvements are planned for handling list values and stabilizing OTL in the transform processor.