doc/operations/observability.md
{{< details >}}
{{< /details >}}
{{< history >}}
{{< /history >}}
[!flag] The availability of this feature is controlled by a feature flag. For more information, see the history. This feature is available for testing, but not ready for production use.
Use GitLab Observability (O11y) to:
[!disclaimer]
By adding observability to GitLab itself users can gain these (planned) features.
<i class="fa-youtube-play" aria-hidden="true"></i> For an overview, see GitLab Experimental Observability (O11y) Introduction.
<!-- Video published on 2025-06-18 -->Join the conversation about interesting ways to use GitLab O11y in the GitLab O11y Discord channel.
{{< tabs >}}
{{< tab title="GitLab.com" >}}
If GitLab Observability is not yet enabled for your group:
The email includes your OpenTelemetry (OTEL) endpoint URL for instrumenting your applications.
Once access is granted:
If Observability isn't displayed in the left sidebar, go directly to https://gitlab.com/groups/<group_path>/-/observability/services.
Use the OTEL endpoint URL provided in your access confirmation email to configure your application's OpenTelemetry instrumentation.
Replace YOUR_OTEL_ENDPOINT_URL with the URL from your confirmation email:
require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
OpenTelemetry::SDK.configure do |c|
resource = OpenTelemetry::SDK::Resources::Resource.create({
'service.name' => 'your-service-name',
'service.version' => '1.0.0',
'deployment.environment' => 'production'
})
c.resource = resource
c.add_span_processor(
OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
OpenTelemetry::Exporter::OTLP::Exporter.new(
endpoint: 'YOUR_OTEL_ENDPOINT_URL'
)
)
)
end
For other programming languages, refer to the OpenTelemetry documentation.
{{< /tab >}}
{{< tab title="Self Hosted" >}}
Observability data is collected in a separate application outside of your GitLab.com instance. Problems with your GitLab instance do not impact collecting or viewing your observability data and vice-versa.
Prerequisites:
For AWS EC2:
sudo mkdir -p /mnt/data
sudo mount /dev/xvdbb /mnt/data # Replace xvdbb with your volume name
sudo chown -R $(whoami):$(whoami) /mnt/data
For permanent mounting, add to /etc/fstab:
echo '/dev/xvdbb /mnt/data ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab
For Ubuntu/Debian:
sudo apt update
sudo apt install -y docker.io docker-compose
sudo systemctl enable docker
sudo systemctl start docker
sudo usermod -aG docker $(whoami)
For Amazon Linux:
sudo dnf update
sudo dnf install -y docker
sudo systemctl enable docker
sudo systemctl start docker
sudo usermod -aG docker $(whoami)
Log out and log back in, or run:
newgrp docker
sudo mkdir -p /mnt/data/docker
sudo bash -c 'cat > /etc/docker/daemon.json << EOF
{
"data-root": "/mnt/data/docker"
}
EOF'
sudo systemctl restart docker
Verify with:
docker info | grep "Docker Root Dir"
cd /mnt/data
git clone -b main https://gitlab.com/gitlab-org/embody-team/experimental-observability/gitlab_o11y.git
cd gitlab_o11y/deploy/docker
docker-compose up -d
If you encounter timeout errors, use:
COMPOSE_HTTP_TIMEOUT=300 docker-compose up -d
If you'd prefer, you can use your own ClickHouse database.
Prerequisites:
Before you run docker-compose up -d, complete the following steps:
docker-compose.yml file.docker-compose.yml and comment out:
clickhouse and zookeeper services.x-clickhouse-defaults and x-clickhouse-depend sections.clickhouse:9000 with your relevant ClickHouse endpoint and TCP port (for example, my-clickhouse.example.com:9000) in the following files. If your ClickHouse instance requires authentication, you may also need to update connection strings to include credentials:
docker-compose.ymlotel-collector-config.yamlprometheus-config.ymlTo properly receive telemetry data, you need to open specific ports in your GitLab O11y instance's security group:
Access the GitLab O11y UI at:
http://[your-instance-ip]:8080
Configure the GitLab O11y URL for your group and enable the feature flag using the Rails console:
Access the Rails console:
docker exec -it gitlab gitlab-rails console
Configure the observability settings for your group and enable the feature flag:
group = Group.find_by_path('your-group-name')
Observability::GroupO11ySetting.create!(
group_id: group.id,
o11y_service_url: 'your-o11y-instance-url',
o11y_service_user_email: '[email protected]',
o11y_service_password: 'your-secure-password',
o11y_service_post_message_encryption_key: 'your-super-secret-encryption-key-here-32-chars-minimum'
)
Feature.enable(:observability_sass_features, group)
Feature.enabled?(:observability_sass_features, group)
Replace:
your-group-name with your actual group pathyour-o11y-instance-url with your GitLab O11y instance URL (for example: http://192.168.1.100:8080)The last command should return true to confirm the feature is enabled.
{{< /tab >}}
{{< /tabs >}}
After you have configured GitLab O11y, to access the dashboard embedded in GitLab:
If Observability isn't displayed in the left sidebar,
go directly to http://<gitlab_instance>/groups/<group_path>/-/observability/services.
You can test your GitLab O11y installation by sending sample telemetry data using the OpenTelemetry SDK. This example uses Ruby, but OpenTelemetry has SDKs for many languages.
Prerequisites:
Ruby installed on your local machine
Required gems:
gem install opentelemetry-sdk opentelemetry-exporter-otlp
Create a file named test_o11y.rb with the following content:
require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
OpenTelemetry::SDK.configure do |c|
# Define service information
resource = OpenTelemetry::SDK::Resources::Resource.create({
'service.name' => 'test-service',
'service.version' => '1.0.0',
'deployment.environment' => 'production'
})
c.resource = resource
# Configure OTLP exporter to send to GitLab O11y
c.add_span_processor(
OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
OpenTelemetry::Exporter::OTLP::Exporter.new(
endpoint: 'http://[your-o11y-instance-ip]:4318/v1/traces'
)
)
)
end
# Get tracer and create spans
tracer = OpenTelemetry.tracer_provider.tracer('basic-demo')
# Create parent span
tracer.in_span('parent-operation') do |parent|
parent.set_attribute('custom.attribute', 'test-value')
puts "Created parent span: #{parent.context.hex_span_id}"
# Create child span
tracer.in_span('child-operation') do |child|
child.set_attribute('custom.child', 'child-value')
puts "Created child span: #{child.context.hex_span_id}"
sleep(1)
end
end
puts "Waiting for export..."
sleep(5)
puts "Done!"
Replace [your-o11y-instance-ip] with your GitLab O11y instance's IP address or hostname.
Run the script:
ruby test_o11y.rb
Check your GitLab O11y dashboard:
http://[your-o11y-instance-ip]:8080To add OpenTelemetry instrumentation to your applications:
Refer to the OpenTelemetry documentation for language-specific guidelines.
GitLab provides pre-built dashboard templates to help you get started with observability quickly. These templates are available at Experimental Observability O11y Templates.
Standard OpenTelemetry dashboards: If you instrument your application with standard OpenTelemetry libraries, you can use these plug-and-play dashboard templates:
GitLab-specific dashboards: When you send GitLab OpenTelemetry data to your GitLab O11y instance, use these dashboards for out-of-the-box insights:
CI/CD observability: The repository includes an example GitLab CI/CD pipeline with OpenTelemetry instrumentation that works with the GitLab O11y CI/CD dashboard template JSON file. This helps you monitor your CI/CD pipeline performance and identify bottlenecks.
GitLab Observability automatically instruments your CI/CD pipelines when enabled, providing visibility into pipeline performance, job durations, and execution flow without any code changes.
To enable automatic pipeline instrumentation, add the GITLAB_OBSERVABILITY_EXPORT CI/CD variable to your project or group:
GITLAB_OBSERVABILITY_EXPORTtraces, metrics, logs (comma-separated for multiple values)The GITLAB_OBSERVABILITY_EXPORT variable accepts the following values:
traces: Exports distributed traces showing pipeline execution flow, job dependencies, and timingmetrics: Exports metrics about pipeline duration, job success rates, and resource usagelogs: Exports structured logs from pipeline executionYou can enable multiple types by separating them with commas:
traces,metrics,logs
Once the variable is set, GitLab automatically:
No changes to your .gitlab-ci.yml file are required. The instrumentation happens automatically in the background.
After running pipelines with instrumentation enabled:
gitlab-ci service.The CI/CD dashboard template from Experimental Observability O11y Templates provides pre-built visualizations for pipeline performance analysis.
Check container status:
docker ps
View container logs:
docker logs [container_name]
Verify the feature flag is enabled for your group:
Feature.enabled?(:observability_sass_features, Group.find_by_path('your-group-name'))
Check that the O11Y_URL environment variable is set:
group = Group.find_by_path('your-group-name')
group.observability_group_o11y_setting&.o11y_service_url
Ensure the routes are properly registered:
Rails.application.routes.routes.select { |r| r.path.spec.to_s.include?('observability') }.map(&:path)
If experiencing SSH connection issues or poor performance:
If your telemetry data isn't appearing in GitLab O11y:
Verify ports 4317 and 4318 are open in your security group.
Test connectivity with:
nc -zv [your-o11y-instance-ip] 4317
nc -zv [your-o11y-instance-ip] 4318
Check container logs for any errors:
docker logs otel-collector-standard
docker logs o11y-otel-collector
docker logs o11y
Try using the HTTP endpoint (4318) instead of gRPC (4317).
Add more debugging information to your OpenTelemetry setup.