GitLab Observability development guidelines

GitLab Observability development setup

There are several options for developing and debugging GitLab Observability:

• Run GDK and GitLab Observability Backend locally all-in-one: This is the simplest and recommended approach for those looking to make changes, or verify changes to Rails, Sidekiq or Workhorse.
• Run GDK locally and connect to the staging instance of GitLab Observability Backend. This is an alternative approach for those looking to make changes, or verify changes to Rails, Sidekiq or Workhorse.
• Use the purpose built devvm. This is more involved but includes a development deployment of the GitLab Observability Backend. This is recommended for those who want to make changes to the GitLab Observability Backend component.
• Run GDK with mocked Observability data. This could be useful in case you just need to work on a frontend or Rails change and do not need the full stack, or when providing reproduction steps for an MR reviewer, who probably won't want to set up the full stack just for an MR.

Run GDK and connect to the staging instance of GitLab Observability Backend

This method takes advantage of our Cloud Connected Observability Backend. Your GitLab instance will require a valid Cloud License and will be treated as a GitLab Self-Managed instance, connected to a multi-tenant GitLab-hosted instance of the GitLab Observability Backend. See this design document for more details on how this works.

How to enable:

1. Add a GitLab Ultimate Self-Managed subscription to your GDK instance.

   1. Sign in to the staging Customers Portal by selecting the Continue with GitLab.com account button. If you do not have an existing account, you are prompted to create one.
   2. If you do not have an existing cloud activation code, click the Staging Self-Managed Ultimate Subscription link on the new subscription purchase links page.
   3. Select enough seats to cover the number of users in your GDK instance (200 should be plenty).
   4. Purchase the subscription using a test credit card.

   After this step is complete, you will have an activation code for a GitLab Ultimate Self-Managed subscription.

2. Set environment variables to point customers-dot to staging, and the Observability URL to staging. For GDK, this can be done in <gdk-root>/env.runit:

   shellCopy

   export GITLAB_SIMULATE_SAAS=0
   export GITLAB_LICENSE_MODE=test
   export CUSTOMER_PORTAL_URL=https://customers.staging.gitlab.com
   export OVERRIDE_OBSERVABILITY_QUERY_URL=https://observe.staging.gitlab.com
   export OVERRIDE_OBSERVABILITY_INGEST_URL=https://observe.staging.gitlab.com
   

   On a non-GDK/GCK instance, you can set the variables using gitlab_rails['env'] in the gitlab.rb file:

   shellCopy

   gitlab_rails['env'] = {
   'GITLAB_LICENSE_MODE' => 'test',
   'CUSTOMER_PORTAL_URL' => 'https://customers.staging.gitlab.com',
   'OVERRIDE_OBSERVABILITY_QUERY_URL' => 'https://observe.staging.gitlab.com',
   'OVERRIDE_OBSERVABILITY_INGEST_URL' => 'https://observe.staging.gitlab.com'
   }
   

3. Enable the feature flag for GitLab Observability features:

   1. Start a rails console session:
      • GDK: gdk rails console
      • GCK: make console
      • GitLab Distribution: Start a Rails console session
   2. Run Feature.enable(:observability_features);

4. Restart your instance (for example, gdk restart).

5. Follow the instructions to activate your new license.

6. Test out the GitLab Observability feature by navigating to a project and selecting Tracing, Metrics, or Logs from the Monitor section of the navigation menu.

7. If you are seeing 404 errors you might need to manually refresh your license data.

Use the purpose built devvm

Visit devvm and follow the README instructions for setup and developing against it.

Use the OpenTelemetry Demo app to send data to a project

The OpenTelemetry Demo app is a great way to run several Docker containers (representing a distributed system), and to send the logs, metrics, and traces to your local GDK instance.

You can reference the instructions for running the demo app.

OpenTelemetry Demo app Quickstart

1. Clone the Demo repository:

   shellCopy

   git clone https://github.com/open-telemetry/opentelemetry-demo.git
   

2. Change to the demo folder:

   shellCopy

   cd opentelemetry-demo/
   

3. Create a project in your local GDK instance. Take note of the project ID.

4. In the newly created project, create a project access token with Developer role and API scope. Save the token for use in the next step.

5. With an editor, edit the configuration in src/otelcollector/otelcol-config-extras.yml. Add the following YAML, replacing:

   • $GDK_HOST with the host and $GDK_PORT with the port number of your GitLab instance.
   • $PROJECT_ID with the project ID and $TOKEN with the token created in the previous steps.
   yamlCopy

   exporters:
      otlphttp/gitlab:
         endpoint: http://$GDK_HOST:$GDK_PORT/api/v4/projects/$PROJECT_ID/observability/
         headers:
            "private-token": "$TOKEN"
   
   service:
      pipelines:
         traces:
            exporters: [spanmetrics, otlphttp/gitlab]
         metrics:
            exporters: [otlphttp/gitlab]
         logs:
            exporters: [otlphttp/gitlab]
   

[!note] For GDK and Docker to communicate you may need to set up a loopback interface.

1. Save the configuration and start the demo app:

   shellCopy

   docker compose up --force-recreate --remove-orphans --detach
   

2. Visit the UI to generate data.

3. Verify Telemetry by exploring logs, metrics, and traces under the Monitor menu in your GitLab project.

Run GDK with mocked Observability data

Apply the following patch to override Observability API calls with local mocks:

git apply < <(curl --silent "https://gitlab.com/gitlab-org/opstrace/opstrace/-/snippets/3747939/raw/main/mock.patch")