Readme

This directory contains some docker files to help you set up metrics tracking on an alluxio cluster. The scripts use OpenTelemetry(OTEL)'s auto instrumentation java agent to generate tracing information for GRPC and S3 calls, which are visualized using Jaeger and Prometheus.

For reference, please refer to documentations listed at opentelemetry GitHub page

1. On one of the Alluxio master or the node where you want to run Jaeger and Prometheus, run

docker-compose -f docker-compose-master.yaml up -d

This will run four services on this node, a Jaeger service to visualize traces, a Prometheus service to visualize metrics such as counters, an OTEL agent to collect traces and counters, and an OTEL collector to aggregate traces and counters from this node and other nodes.

2. On all other alluxio nodes, run (if you are running alluxio in local mode or on a single host, skip this step)

MASTER_IP=xxx.xx.xx.xx docker-compose -f docker-compose-worker.yaml up -d

3. Download the auto instrumentation jar and place it in the alluxio/conf directory. The jar can be found at

https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent-all.jar

4. Edit conf/alluxio-env.sh and add

ALLUXIO_MASTER_JAVA_OPTS+=" -javaagent:./conf/opentelemetry-javaagent-all.jar \
     -Dotel.resource.attributes=service.name=AlluxioMaster \
     "
ALLUXIO_WORKER_JAVA_OPTS+=" -javaagent:./conf/opentelemetry-javaagent-all.jar \
     -Dotel.resource.attributes=service.name=AlluxioWorker \
     "

5. Copy the content of alluxio/conf dir to all nodes using alluxio utility.

bin/alluxio copyDir conf

6. Restart alluxio master and workers
7. Point your browser to MASTER_IP:16686 for tracing and MASTER_IP:9090 for metrics
8. Stop alluxio master and workers

docker-compose -f docker-compose-master.yaml down
docker-compose -f docker-compose-worker.yaml down