Fine-Tuning Auto-Instrumentation - Jamie Danielson, Honeycomb

Summary

Jamie Danielson from Honeycomb discusses fine-tuning automatic instrumentation across different programming languages, focusing on zero and low-code options. The talk covers enabling/disabling specific libraries, ignoring spans based on criteria, and enriching spans with headers. It also highlights the capabilities of the OpenTelemetry collector for processing telemetry data, such as filtering, redacting, and transforming attributes. Danielson emphasizes the evolving nature of instrumentation and the goal of meaningful, customizable telemetry.

Takeaways

• 😀 Jamie Danielson from Honeycomb discusses various options for fine-tuning automatic instrumentation across different programming languages.
• 🔍 Automatic instrumentation can be complex and is an ongoing issue within the project, as it involves deciding what to expect and what options are available for app instrumentation.
• 🛠 Two main approaches to instrumentation are covered: zero code (no source code changes) and low code (some programmatic setup).
• 🌐 The talk highlights the need to include only necessary instrumentation libraries to avoid unnecessary overhead and to start simple before customizing.
• 📚 Java is noted for having a comprehensive set of instrumentation options and often leads in implementing new features, which other languages then adopt.
• ⚙️ Enabling and disabling specific libraries can be done through meta-packages that bundle multiple instrumentations, allowing selective inclusion.
• 🚫 Options for ignoring spans based on criteria are available and can differ significantly between languages, affecting how traces are constructed.
• 🔑 The OpenTelemetry Collector is highlighted as a powerful tool for processing telemetry data, including filtering out noisy span events and redacting sensitive information.
• 🔄 The Collector's transform processor can be used to update attributes, ensuring consistency across different languages and versions.
• 📝 The script emphasizes the importance of meaningful telemetry and the ongoing evolution of instrumentation options, aiming for built-in, native integration.
• 🔄 The discussion also touches on the possibility of adjusting instrumentation dynamically using feature flags or other remote configuration methods.

Q & A

What is the main topic of Jamie Danielson's talk?

Jamie Danielson's talk is about fine-tuning automatic instrumentation across different programming languages and exploring the options available for various languages such as Java, JavaScript (specifically Node.js), and Python.

What does Jamie Danielson do at Honeycomb?

Jamie Danielson works at Honeycomb as an approver on OpenTelemetry JS and is in the process of becoming a maintainer, as indicated by an open PR.

What is the general idea behind zero-code instrumentation?

The general idea of zero-code instrumentation is that it allows you to instrument your application without touching the actual source code, often through the use of environment variables or running an agent next to the application.

What is the difference between zero-code and low-code instrumentation?

Zero-code instrumentation means not modifying the source code for instrumentation, while low-code instrumentation involves making some changes to the source code, usually for more flexible configuration options during SDK initialization.

Why is it recommended to only include the instrumentation libraries that are needed for an application?

It is recommended to only include necessary instrumentation libraries to avoid unnecessary complexity and potential performance overhead, and to make the most efficient use of resources.

How can one disable specific instrumentation libraries in Java?

In Java, you can disable specific instrumentation libraries by setting the default enabled state to false and then selectively enabling only the desired libraries.

What is a feature of the OpenTelemetry collector that Jamie Danielson highlighted in her talk?

Jamie Danielson highlighted the ability of the OpenTelemetry collector to process, transform, batch, and export telemetry data to a backend, as well as its capability to filter out specific span events that are not useful.

What is the purpose of the span suppression strategy in Java instrumentation?

The span suppression strategy in Java is used to avoid creating duplicate spans that provide the same information, thus reducing the verbosity and potential cost associated with unnecessary spans.

How can one ignore spans based on criteria in JavaScript instrumentation?

In JavaScript, one can ignore spans based on criteria by using environment variables or code hooks to disable instrumentation for specific URLs or to only instrument file system calls if they are part of an existing trace.

What is a common configuration change when it comes to HTTP instrumentation in different languages?

A common configuration change is the ability to capture and include HTTP request and response headers as attributes on spans, which can be done through specific environment variables or code configurations in languages like Java and Python.

What is the potential downside of dropping spans at the collector level?

The potential downside of dropping spans at the collector level is that it can lead to broken traces by removing context that is not expected to be missing, which can make debugging and monitoring more difficult.

What is the role of the transform processor in the OpenTelemetry collector?

The transform processor in the OpenTelemetry collector allows for the updating of attributes on spans, which can be useful for aligning telemetry from different languages that may be using different attribute names for the same data.

How can one remotely configure the OpenTelemetry SDK?

One can remotely configure the OpenTelemetry SDK using the OpenTelemetry Configuration service (OTL), which is part of an ongoing project to allow dynamic configuration changes.