OpenTelemetry eBPF Instrumentation

Learn how to use OpenTelemetry eBPF Instrumentation for automatic instrumentation.

eBPF application auto-instrumentation

OpenTelemetry libraries provide telemetry collection for popular programming languages and frameworks. However, getting started with distributed tracing can be complex. In some compiled languages like Go or Rust, you must manually add tracepoints to the code.

OpenTelemetry eBPF Instrumentation is an auto-instrumentation tool to easily get started with Application Observability. OBI uses eBPF to automatically inspect application executables and the OS networking layer, and capture trace spans related to web transactions and Rate Errors Duration (RED) metrics for Linux HTTP/S and gRPC services. All data capture occurs without any modifications to application code or configuration.

OBI offers the following features:

  • Wide language support: Java, .NET, Go, Python, Ruby, Node.js, C, C++, and Rust
  • Lightweight: No code changes required, no libraries to install, no restarts needed
  • Efficient instrumentation: Traces and metrics are captured by eBPF probes with minimal overhead
  • Distributed tracing: Distributed trace spans are captured and reported to a collector
  • Kubernetes-native: Provides configuration-free auto-instrumentation for Kubernetes applications
  • Visibility into encrypted communications: Capture transactions over TLS/SSL without decryption
  • Context propagation: Propagate trace context across services automatically
  • Protocol support: HTTP/S, gRPC, and gRPC-Web
  • Low cardinality metrics: Prometheus-compatible metrics with low cardinality for cost reduction
  • Network observability: Capture network flows between services
  • SQL database traces: Capture database queries and connections
  • Trace context propagation: Propagate trace context across services automatically

Requirements

OBI requires the following to run:

  • Linux kernel version 4.19 or later
  • An x86_64 or arm64 processor
  • Runtime support for eBPF (most modern Linux distributions)
  • Administrative privileges (root access) or the specific capabilities listed in the configuration reference
OBI eBPF architecture

Compatibility

OBI is tested with the following Linux distributions:

  • Ubuntu 20.04 LTS, 21.04, 22.04 LTS and 23.04
  • CentOS 7, 8, and 9
  • AlmaLinux 8, 9
  • Rocky Linux 8, 9
  • Red Hat Enterprise Linux 8, 9
  • Debian 11, 12
  • openSUSE Leap 15.3, 15.4
  • SUSE Linux Enterprise Server 15 SP4
  • OBI also supports RedHat-based distributions: RHEL8, CentOS 8, Rocky8, AlmaLinux8, and others, which ship a Kernel 4.18 that backports eBPF-related patches.

  • For instrumenting Go programs, compile with at least Go 1.17. OBI support Go applications built with a major Go version no earlier than 3 versions behind the current stable major release.

  • Administrative access rights to execute OBI.

Limitations

OBI has its limitations too. It only provides generic metrics and transaction level trace span information. Language agents and manual instrumentation is still recommended, so that you can specify the custom attributes and events you want to capture.

While most eBPF programs require elevated privileges, OBI allow you to specify finer grained permissions to run with minimum required permissions, such as: CAP_DAC_READ_SEARCH, CAP_SYS_PTRACE, CAP_PERFMON, CAP_BPF, CAP_CHECKPOINT_RESTORE, and others.

Some OBI functionality requires further permissions, for example using the network observability probes with Linux Traffic Control requires CAP_NET_ADMIN, but it’s a feature you have to optionally enable.

For a comprehensive list of capabilities required by OBI, refer to Security, permissions and capabilities.

Get started with OBI

  • Follow the setup documentation to get started with OBI either with Docker or Kubernetes.
  • For application observability, refer to the language specific quickstart guides for examples about how to set up OBI to instrument applications written in a particular language.

OBI Quickstart guides

Learn how to set up and run OBI to instrument multiple languages.

Configure Beyla

Learn how to configure Beyla.

Network metrics

Configuring OBI to observe point-to-point network metrics.

Set up OBI

Learn how to set up and run OBI.

OBI RED metrics dashboard

Learn how to use the OBI RED metrics dashboard.

Beyla exported metrics

Learn about the HTTP/gRPC metrics Beyla can export.

Distributed traces with Beyla

Learn about Beyla’s distributed traces support.

Measuring total request times, instead of service times

How to measure total request times from the point of view of the client

OBI security, permissions, and capabilities

Privileges and capabilities required by OBI

OBI and Cilium compatibility

Compatibility notes when running OBI alongside Cilium

Beyla performance overhead

Beyla runs alongside your applications and has a minimal impact on performance. Find out about the methodology we used to measure the performance overhead.

OBI metrics cardinality

Overview of how to calculate the cardinality of metrics produced by a default OBI installation, considering the size and complexity of the instrumented environment.