The "Zen" of Python Exemplars - Paige Cruz, Chronosphere

Summary

Paig Cruz from Chronosphere introduces 'Exemplars' in the context of Python's Zen, emphasizing their role in observability. Exemplars, as trace data linked to metrics, offer a straightforward way to investigate issues, especially beneficial for new users. Cruz explains the technical aspects, current limitations, and the process of adding exemplars to metrics in both OpenTelemetry and Prometheus. The talk advocates for the use of exemplars as an entry point for trace analysis, providing a practical guide for those new to distributed tracing.

Takeaways

• 🐍 'Zen of Python' is a set of guiding principles for Python development, accessible via 'import this' in Python.
• 🔍 Exemplars are trace data points linked to specific metrics, serving as entry points for deeper investigation into system issues.
• 💎 Exemplars are described as 'beautiful little diamonds' in observability tools, inviting users to explore the cause behind metrics.
• 🛑 The speaker emphasizes that exemplars are not a silver bullet but a useful tool in the observability toolkit.
• 📈 Exemplars are especially beneficial for newcomers or those resistant to tracing, providing an easier on-ramp to understanding system behavior.
• 🔬 The OpenTelemetry (OTel) specification supports exemplars across all metric types, but Python's implementation currently does not.
• 📊 Open Metrics, which Prometheus intends to implement, defines exemplars slightly differently, supporting only counters and histograms, not gauges or summaries.
• 🔄 The process of attaching exemplars to metrics involves ensuring proper trace ID formatting and enabling feature flags for storage in Prometheus.
• 🔗 Exemplars can bridge siloed telemetry data, linking separate instances of trace and metric data for easier navigation.
• 🚀 The speaker advocates for using exemplars in daily workflows to improve the initial response from alert to understanding system issues.
• 📚 For those interested in the technical details of translating between different observability data formats, the Open Metrics specification and OTel-Prometheus compatibility documentation are recommended reading.

Q & A

What is the Zen of Python and how is it related to PEPs?

The Zen of Python is a collection of 19 aphorisms that outline the guiding principles of the Python programming language. It can be accessed by typing 'import this' in a Python interpreter. PEP stands for Python Enhancement Proposal, which are design documents that propose new features or processes for the Python language, similar to how the Zen of Python provides philosophical guidance.

What are exemplars in the context of observability?

Exemplars are specific instances or 'diamonds' within a distributed trace or request that are associated with a metric. They provide a way to drill down from a metric alert to the underlying trace data, offering more context and helping to identify the cause of an issue.

Why might there be multiple ways to do something in Open Source observability?

In the world of Open Source observability, there can be multiple ways to achieve a task due to the variety of ecosystems and setups that different organizations use. This diversity can sometimes make it challenging to standardize on a single method for performing certain operations.

How are exemplars defined in the OpenTelemetry metric specification?

In the OpenTelemetry metric specification, an Exemplar is defined as a record value that associates OpenTelemetry context to a metric event within a metric. It includes a value, a timestamp in Unix Epoch nanoseconds, a filtered attribute, and optionally a Trace ID or Span ID.

What is the current status of Python's support for exemplars in OpenTelemetry?

As of the script's context, Python does not support exemplars in OpenTelemetry metrics. There is an open issue, and contributions are welcome to add support for exemplars in the Python SIG (Special Interest Group).

What is the OpenMetrics specification and how does it relate to Prometheus?

The OpenMetrics specification is intended to be a neutral metric specification, not tied to any specific system like Prometheus. However, Prometheus intends to implement it. It defines exemplars as references to data outside of the metric set, commonly used for program IDs of program traces.

Which types of metrics in Prometheus support exemplars?

In Prometheus, only counters and histograms support exemplars. Gauges, summaries, and info types do not support exemplars as per the OpenMetrics specification.

What is the practical process of instrumenting to add exemplars to metrics?

The process involves ensuring that you have a zero-padded hexadecimal string for the trace ID and then adding the trace ID and optional data, such as a trace URL, to the metric. It also requires enabling the feature flag for exemplar storage in Prometheus, as the implementation is relatively new.

What are the benefits of using exemplars in observability workflows?

Exemplars are discoverable within common workflows, provide a quick way to compare traces before and after an event, and can act as a bridge between siloed telemetry systems. They are particularly useful for onboarding new users to tracing and integrating trace data into familiar metric workflows.

What are the limitations of exemplars when used for debugging?

While exemplars provide a good entry point for investigation by linking metrics to trace data, they are a single representative and cannot contain all the answers. Further aggregate trace analysis, event logs, and metric investigation are still required for comprehensive debugging.

What resources are available for those interested in learning more about exemplars and their implementation?

For those looking to delve deeper, the OpenMetrics specification and the compatibility documentation between OpenTelemetry and Prometheus/OpenMetrics are recommended. Additionally, there is an open-source workshop available that covers instrumentation in Python.