OpenTelemetry Emitter

The OpenTelemetry emitter generates OpenTelemetry Spans for queries.

How OpenTelemetry emitter works

The OpenTelemetry emitter processes ServiceMetricEvent events for the query/time metric. It extracts OpenTelemetry context from the query context. To link druid spans to parent traces, the query context should contain at least traceparent key. See context propagation for more information. If no traceparent key is provided, then spans are created without parentTraceId and are not linked to the parent span. In addition, the emitter also adds other druid context entries to the span attributes.

Configuration

Enabling

To enable the OpenTelemetry emitter, add the extension and enable the emitter in common.runtime.properties.

Load the plugin:

properties

druid.extensions.loadList=[..., "opentelemetry-emitter"]

Then there are 2 options:

You want to use only opentelemetry-emitter

properties

druid.emitter=opentelemetry

You want to use opentelemetry-emitter with other emitters

properties

druid.emitter=composing
druid.emitter.composing.emitters=[..., "opentelemetry"]

*More about Druid configuration here.

Testing

Part 1: Run zipkin and otel-collector

Create docker-compose.yaml in your working dir:

yaml

version: "2"
services:

  zipkin-all-in-one:
    image: openzipkin/zipkin:latest
    ports:
      - "9411:9411"

  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=otel-local-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ${PWD}/config.yaml:/otel-local-config.yaml
    ports:
      - "4317:4317"

Create config.yaml file with configuration for otel-collector:

yaml

version: "2"
receivers:
receivers:
  otlp:
    protocols:
      grpc:

exporters:
  zipkin:
    endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
    format: proto

  logging:

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, zipkin]

*How to configure otel-collector you can read here.

Run otel-collector and zipkin.

bash

docker compose up

Part 2: Run Druid

Build Druid:

bash

mvn clean install -Pdist
tar -C /tmp -xf distribution/target/apache-druid-0.21.0-bin.tar.gz
cd /tmp/apache-druid-0.21.0

Edit conf/druid/single-server/micro-quickstart/_common/common.runtime.properties to enable the emitter ( see Configuration section above).

Start the quickstart with the apppropriate environment variables for opentelemetry autoconfiguration:

OTEL_SERVICE_NAME="org.apache.druid" OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" bin/start-micro-quickstart

*More about opentelemetry autoconfiguration here

Load sample data - example.

Part 3: Send queries

Create query.json:

json

{
   "query":"SELECT COUNT(*) as total FROM wiki WHERE countryName IS NOT NULL",
   "context":{
      "traceparent":"00-54ef39243e3feb12072e0f8a74c1d55a-ad6d5b581d7c29c1-01"
   }
}

Send query:

bash

curl -XPOST -H'Content-Type: application/json' http://localhost:8888/druid/v2/sql/ -d @query.json

Then open http://localhost:9411/zipkin/ and you can see there your spans.