Back to Haproxy

README

addons/otel/README.md

3.4-dev109.3 KB
Original Source

HAProxy OpenTelemetry Filter (OTel)

The OTel filter enables HAProxy to emit telemetry data -- traces, metrics and logs -- to any OpenTelemetry-compatible backend via the OpenTelemetry protocol (OTLP).

It is the successor to the OpenTracing (OT) filter, built on the OpenTelemetry standard which unifies distributed tracing, metrics and logging into a single observability framework.

Features

  • Distributed tracing -- spans with parent-child relationships, context propagation via HTTP headers or HAProxy variables, links, baggage and status.
  • Metrics -- counter, histogram, up-down counter and gauge instruments with configurable aggregation and bucket boundaries.
  • Logging -- log records with severity levels, optional span correlation and runtime-evaluated attributes.
  • Rate limiting -- percentage-based sampling (0.0--100.0) for controlling overhead.
  • ACL integration -- fine-grained conditional execution at instrumentation, scope and event levels.
  • CLI management -- runtime enable/disable, rate adjustment, error mode switching and status inspection.
  • Context propagation -- inject/extract span contexts between cascaded HAProxy instances or external services.

Dependencies

The filter requires the OpenTelemetry C Wrapper library, which wraps the OpenTelemetry C++ SDK.

Building

The OTel filter is compiled together with HAProxy by adding USE_OTEL=1 to the make command.

Using pkg-config

PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 TARGET=linux-glibc

Explicit paths

make -j8 USE_OTEL=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc

Build options

VariableDescription
USE_OTELEnable the OpenTelemetry filter
OTEL_DEBUGCompile in debug mode
OTEL_INCForce path to opentelemetry-c-wrapper include files
OTEL_LIBForce path to opentelemetry-c-wrapper library
OTEL_RUNPATHAdd opentelemetry-c-wrapper RUNPATH to executable
OTEL_USE_VARSEnable context propagation via HAProxy variables

Debug mode

PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_DEBUG=1 TARGET=linux-glibc

Variable-based context propagation

PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_USE_VARS=1 TARGET=linux-glibc

Verifying the build

./haproxy -vv | grep -i opentelemetry

If the filter is built in, the output contains:

Built with OpenTelemetry support (C++ version 1.26.0, C Wrapper version 1.0.0-842).
	[OTEL] opentelemetry

Library path at runtime

When pkg-config is not used, the executable may not find the library at startup. Use LD_LIBRARY_PATH or build with OTEL_RUNPATH=1:

LD_LIBRARY_PATH=/opt/lib ./haproxy ...
make -j8 USE_OTEL=1 OTEL_RUNPATH=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc

Configuration

The filter uses a two-file configuration model:

  1. OTel configuration file (.cfg) -- defines the telemetry model: instrumentation settings, scopes and groups.
  2. YAML configuration file (.yml) -- defines the OpenTelemetry SDK pipeline: exporters, samplers, processors, providers and signal routing.

Activating the filter

Add the filter to a HAProxy proxy section (frontend/listen/backend):

frontend my-frontend
    ...
    filter opentelemetry [id <id>] config <file>
    ...

If no filter id is specified, otel-filter is used as default.

OTel configuration file structure

The OTel configuration file contains three section types:

  • otel-instrumentation -- mandatory; references the YAML file, sets rate limits, error modes, logging and declares groups and scopes.
  • otel-scope -- defines actions (spans, attributes, metrics, logs) triggered by stream events or from groups.
  • otel-group -- a named collection of scopes triggered from HAProxy TCP/HTTP rules.

Minimal YAML configuration

yaml
exporters:
  my_exporter:
    type:     otlp_http
    endpoint: "http://localhost:4318/v1/traces"

samplers:
  my_sampler:
    type: always_on

processors:
  my_processor:
    type: batch

providers:
  my_provider:
    resources:
      - service.name: "haproxy"

signals:
  traces:
    scope_name: "HAProxy OTel"
    exporters:  my_exporter
    samplers:   my_sampler
    processors: my_processor
    providers:  my_provider

Supported YAML exporters

TypeDescription
otlp_grpcOTLP over gRPC
otlp_httpOTLP over HTTP (JSON or Protobuf)
otlp_fileLocal files in OTLP format
zipkinZipkin-compatible backends
elasticsearchElasticsearch
ostreamText output to a file (for debugging)
memoryIn-memory buffer (for testing)

Scope keywords

KeywordDescription
spanCreate or reference a span
attributeSet key-value span attributes
eventAdd timestamped span events
baggageSet context propagation data
statusSet span status (ok/error/ignore/unset)
linkAdd span links to related spans
injectInject context into headers or variables
extractExtract context from headers or variables
finishClose spans (supports wildcards: *, *req*, *res*)
instrumentCreate or update metric instruments
log-recordEmit a log record with severity
otel-eventBind scope to a filter event with optional ACL
idle-timeoutSet periodic event interval for idle streams

CLI commands

Available via the HAProxy CLI socket (prefix: flt-otel):

CommandDescription
flt-otel statusShow filter status
flt-otel enableEnable the filter
flt-otel disableDisable the filter
flt-otel hard-errorsEnable hard-errors mode
flt-otel soft-errorsDisable hard-errors mode
flt-otel logging [state]Set logging state
flt-otel rate [value]Set or show the rate limit
flt-otel debug [level]Set debug level (debug build only)

When invoked without arguments, rate, logging and debug display the current value.

Performance

Benchmark results from the standalone (sa) configuration, which exercises all events (worst-case scenario):

Rate limitReq/sAvg latencyOverhead
100.0%38,202213.08 us21.6%
50.0%42,777190.49 us12.2%
25.0%45,302180.46 us7.0%
10.0%46,879174.69 us3.7%
2.5%47,993170.58 us1.4%
disabled48,788167.74 us~0
off48,697168.00 usbaseline

With a rate limit of 10% or less, the performance impact is negligible. Detailed methodology and additional results are in the test/ directory.

Test configurations

The test/ directory contains ready-to-run example configurations:

  • sa -- standalone; the most comprehensive example, demonstrating spans, attributes, events, links, baggage, status, metrics, log records, ACL conditions and idle-timeout events.
  • fe/be -- distributed tracing across two cascaded HAProxy instances using HTTP header-based context propagation.
  • ctx -- context propagation via HAProxy variables using the inject/extract mechanism.
  • cmp -- minimal configuration for benchmarking comparison.
  • empty -- filter initialized with no active telemetry.

Quick start with Jaeger

Start a Jaeger all-in-one container:

docker run -d --name jaeger -p 4317:4317 -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest

Run one of the test configurations:

./test/run-sa.sh

Open the Jaeger UI at http://localhost:16686 to view traces.

Documentation

Detailed documentation is available in the following files:

Copyright 2026 HAProxy Technologies

Author

Miroslav Zagorac [email protected]