End-to-end test pipelines

Common architecture

All E2E tests are executed within separate child pipelines. To support different dynamic features of E2E testing pipelines, all child pipeline YAML files are generated by e2e-test-pipeline-generate CI/CD job and triggered by respective trigger job.

e2e-test-pipeline-generate

The e2e-test-pipeline-generate job generates CI/CD YAML file definitions which are used to trigger child pipelines that run E2E tests.

The generate_e2e_pipelines Rake task:

Determines which e2e specs should be executed in a particular merge request pipeline.
Generates a CI/CD YAML file definition for each E2E test pipeline type.

This Rake task:

Analyzes changes in a particular merge request and determines which specs must be executed using selective test execution with these criteria. Based on that, a dry-run of every scenario executes and determines if a scenario contains any executable tests.
The total run time for each scenario is calculated.
Based on the run time value, dynamic job scaling calculates the necessary number of parallel CI/CD jobs for each scenario type and generates pipeline YAML file with appropriate values.

`e2e:perf-on-cng`

The e2e:perf-on-cng child pipeline runs tests against a Cloud Native GitLab installation.

Deployment is managed by the orchestrator CLI tool, which you can also use to locally recreate CI/CD deployments.

The e2e:perf-on-cng child pipeline is executed in merge requests and is a non blocking job. If any test fails, it won't block your merge requests from being merged.

Setup

This E2E test child pipeline is triggered by the e2e:perf-on-cng job using dynamically generated CI/CD YAML file stored as artifacts in the e2e-test-pipeline-generate CI/CD job. CI/CD YAML files are generated using a template.

child pipeline jobs

The child pipeline consists of several stages that support E2E test execution.

.pre

build-cng-env job is responsible for setting up all environment variables for CNG downstream pipeline
build-cng job triggers CNG downstream pipeline which is responsible for building all necessary images

prepare

dotenv-vars job is responsible to create an artifact out of the qa/performance_test folder which consists of all the k6 performance test. This artifact will be downloaded by downstream pipeline jobs to run the tests. It also stores some env vars like CI_JOB_NAME, CI_JOB_ID and GITLAB_HELM_CHART_REF as can be seen in the job definition, which will be used by run-performance-tests job

test

`run-performance-test` job

This job triggers a downstream multi-project pipeline in Component Performance testing project. This pipeline performs the following actions

Creates two gpc instances in the same region and zone.
1. server instance: where CNG instance of GitLab is created using orchestrator
  1. local k8s cluster setup using kind
  2. GitLab installation using official helm chart
2. test runner instance: which runs the k6 tests download as an artifact from dotenv-vars job and runs it against the CNG GitLab instance created on the server instance.

Adding a new test

You can add a new k6 tests which will then run as a part of run-performance-test job.

To add a new test:

Create a new .js file under qa/performance_test/k6_test.
Copy and paste the following boilerplate to it before beginning to write the test.
Update the comment sections with the respective values and code.

javascript

export const TTFB_THRESHOLD= /* TTFB THRESHOLD VALUE EXPECTED */;
export const RPS_THRESHOLD= /* RPS THRESHOLD VALUE EXPECTED */;
export const TEST_NAME=/* 'NAME OF THE TEST IN QUOTES' */;
export const LOAD_TEST_VUS = 2; /* THE NUMBER OF THREADS OF ACTUAL TEST */
export const LOAD_TEST_DURATION = '50s'; /* THE DURATION FOR THE ACTUAL TEST RUN */
export const WARMUP_TEST_VUS = 1; /* THE NUMBER OF THREADS FOR WARMING UP THE SYSTEM */
export const WARMUP_TEST_DURATION = '10s'; /* THE DURATION FOR THE WARMUP RUN */
export const LOAD_TEST_START_TIME = '10s'; /* THE TIME TO WAIT AFTER WHICH THE LOAD TEST STARTS
                                              USUALLY THIS WOULD BE EQUAL TO WARMUP_TEST_DURATION */

export const options = {
scenarios:  {
    warmup: {
      executor: 'constant-vus',
      vus: WARMUP_TEST_VUS,
      duration: WARMUP_TEST_DURATION,
      gracefulStop: '0s',
      tags: { scenario: 'warmup' },
    },
    load_test: {
      executor: 'constant-vus',
      vus: LOAD_TEST_VUS,
      duration: LOAD_TEST_DURATION,
      startTime: LOAD_TEST_START_TIME,
      tags: { scenario: 'load_test' },
    },
  },
  thresholds: {
    'http_req_waiting{scenario:load_test}': [
      { threshold: `p(90)<${TTFB_THRESHOLD}`, abortOnFail: false }
    ],
    'http_reqs{scenario:load_test}': [
      { threshold: `rate>=${RPS_THRESHOLD}`, abortOnFail: false }
    ]
  },
};

export default function () {

  // WRITE THE TEST HERE


}

You can currently give any value for TTFB_THRESHOLD and RPS_THRESHOLD as they are currently not being used for reporting and will be removed as a part of https://gitlab.com/gitlab-org/quality/component-performance-testing/-/issues/75

You can refer to other test present in qa/performance_test/k6_test to write your tests.

If you tests required data in the environment you can update the mr_seed.rb file to add the resources you need to create.

[!note] Do not remove any existing resources being created form mr_seed.rb.

Component Performance testing currently doesn't support multiple seed files but this will be addressed as a part of https://gitlab.com/gitlab-org/quality/component-performance-testing/-/issues/76

Skipping `e2e:perf-on-cng` test job

To skip the e2e:perf-on-cng test job to run on MR pipeline, add the pipeline:skip-performance label while creating the merge request.

Troubleshooting

Sometimes the multi-project job under run-performance-test job may fail during data seeding with gem incompatibility error.

An example of the error is

shell

/usr/lib/ruby/gems/3.3.0/gems/bundler-2.6.9/lib/bundler/vendor/pub_grub/lib/pub_grub/version_solver.rb:225:in `resolve_conflict': Could not find compatible versions (Bundler::PubGrub::SolveFailure)
Because every version of gitlab-backup-cli depends on grpc ~> 1.74.0
  and Gemfile depends on gitlab-backup-cli >= 0,
  grpc ~> 1.74.0 is required.

This happens because there might have been an update on the Gemfile which your merge request doesn't contain. Rebasing the merge request branch with the master branch should fix this issue.

`e2e:test-on-cng`

The e2e:test-on-cng child pipeline runs tests against a Cloud Native GitLab installation.

Deployment is managed by the orchestrator CLI tool, which you can also use to locally recreate CI/CD deployments.

The e2e:test-on-cng child pipeline is executed in merge requests and is part of pre-merge validation lifecycle. If any test fails, you can't merge introduced code changes.

Setup

This E2E test child pipeline is triggered by the e2e:test-on-cng job using dynamically generated CI/CD YAML file stored as artifacts in the e2e-test-pipeline-generate CI/CD job. CI/CD YAML files are generated using a template.

child pipeline jobs

The child pipeline consists of several stages that support E2E test execution.

.pre

build-cng-env job is responsible for setting up all environment variables for CNG downstream pipeline
build-cng job triggers CNG downstream pipeline which is responsible for building all necessary images

test

Jobs in test stage perform following actions:

local k8s cluster setup using kind
GitLab installation using official helm chart
E2E test execution against performed deployment

Jobs in test stage:

cng-instance runs the full e2e suite, excluding orchestrated tests, against CNG
cng-qa-min-redis-version runs validation for minimal supported Redis version using Test::Instance::HealthCheck scenario against CNG
cng-registry runs tests in the registry scenario Test::Integration::Registry against CNG
cng-relative-url runs the smoke suite, Test::Instance::Smoke, but with a relative URL set for CNG
cng-oauth runs e2e spec for authentication between GitHub and GitLab with OmniAuth enabled.

report

This stage is responsible for allure test report generation.

Debugging

To help with debugging:

Each test job prints a list of arguments that you can pass to the orchestrator to exactly recreate the same deployment for local debugging.
Cluster events log and all pod logs are saved in E2E test job artifacts.
orchestrator automatically outputs all cluster events with errors in the case of failed deployment.

`e2e:test-on-omnibus-ee`

The e2e:test-on-omnibus-ee child pipeline runs tests against an Omnibus installation. This pipeline type is not executed in merge request pipelines by default and can be triggered manually by triggering the e2e:test-on-omnibus-ee job.

This pipeline type is allowed to fail and even in the case of a manual trigger inside of a merge request pipeline, failing tests will not block the ability to merge.

Linux package deployment is managed by gitlab-qa.

Setup

This E2E test child pipeline is triggered by the e2e:test-on-omnibus-ee job using dynamically generated CI/CD YAML file stored as artifacts in e2e-test-pipeline-generate CI/CD jobs. The CI/CD YAML file is generated using a template.

child pipeline jobs

E2E test execution pipeline consists of several stages which all support execution of E2E tests.

.pre

This stage is responsible for the following tasks:

Triggering downstream pipeline which builds the omnibus-gitlab Docker image.

test

Jobs in test stage perform following actions:

Installs GitLab by using a built Linux package.
Executes E2E tests against the Linux package installation.

report

This stage is responsible for allure test report generation.

`e2e:test-on-gdk`

The e2e:test-on-gdk child pipeline supports development of the GitLab platform by providing feedback to engineers on end-to-end test execution faster than via e2e:test-on-omnibus-ee.

This is achieved by running tests against the GitLab Development Kit (GDK), which can be built and installed in less time than when testing against Omnibus GitLab. The trade-off is that Omnibus GitLab can be used to deploy a production installation, whereas the GDK is a development environment. Tests that run against the GDK might not catch bugs that depend on part of the process of preparing GitLab to run in a production environment, including pre-compiling assets, assigning configuration defaults as part of an official installation package, deploying GitLab services to multiple servers, and more. On the other hand, engineers who use the GDK day-to-day can benefit from automated tests catching bugs that only appear on the GDK.

Setup

This E2E test child pipeline is triggered by the e2e:test-on-gdk job using dynamically generated CI/CD YAML file stored as artifacts in the e2e-test-pipeline-generate CI/CD job. The CI/CD YAML file is generated by using a template.

`build-gdk-image`

The build-gdk-image job uses the code from the merge request to build a Docker image that can be used in test jobs to launch a GDK instance in a container. The image is pushed to the container registry.

The job also runs in pipelines on the default branch to build a base image that includes the GDK and GitLab components. This avoids building the entire image from scratch in merge requests. However, if the merge request includes changes to certain GitLab components or code the job will rebuild the base image before building the image that will be used in the test jobs.

child pipeline jobs

The child pipeline consists of several stages that support E2E test execution.

test

Jobs in the test stage perform following actions:

Start an instance of GDK by using the Docker image built by the build-gdk-image job.
Run E2E tests against the running instance of GDK.

report

This stage is responsible for allure test report generation.

Test Licenses

For more information on the licenses these pipelines use, see test licenses.

Adding new jobs to E2E test pipelines

E2E test pipelines use dynamic scaling of jobs based on their runtime. To create a mapping between job definitions in pipeline definition YAML files and a particular test scenario, scenario classes are used. These classes are located in qa/qa/scenario folder.

A typical job definition in one of the e2e test pipeline definition YAML files would look like:

yaml

my-new-test-job:
  # ...
  variables:
    QA_SCENARIO: Test::Integration::MyNewTestScenario

In this example:

QA_SCENARIO: Test::Integration::MyNewTestScenario: name of the scenario class that is passed to the qa/bin/qa test execution script. While the full class name would be QA::Scenario::Test:Integration::MyNewTestScenario, QA::Scenario is omitted to have shorted definitions.

Considering example above, perform the following steps to create a new job:

Create a new scenario my_new_job.rb in the integration directory of the e2e test framework. The scenario class should define a pipeline mapping that couples the scenario to a specific job in a specific pipeline type. If job was added to the test-on-cng pipeline, this scenario would define RSpec tags that should be executed and pipeline mapping:
ruby
```
module QA
  module Scenario
    module Test
      module Integration
        class MyNewJob < Test::Instance::All
          tags :some_special_tag

          pipeline_mappings test_on_cng: %w[my-new-test-job]
        end
      end
    end
  end
end
```

Add the new job definition in the main.gitlab-ci.yml pipeline definition:

yaml

my-new-test-job:
  extends:
    - .cng-test
  variables:
    QA_SCENARIO: Test::Integration::MyNewTestScenario

Such a definition ensures that my-new-test-job has automatic parallel job scaling based on predefined runtime threshold.

Common architecture

e2e-test-pipeline-generate

e2e:perf-on-cng

Setup

child pipeline jobs

.pre

prepare

test

run-performance-test job

Adding a new test

Skipping e2e:perf-on-cng test job

Troubleshooting

e2e:test-on-cng

Setup

child pipeline jobs

.pre

test

report

Debugging

e2e:test-on-omnibus-ee

Setup

child pipeline jobs

.pre

test

report

e2e:test-on-gdk

Setup

build-gdk-image

child pipeline jobs

test

report

Test Licenses

Adding new jobs to E2E test pipelines

`e2e:perf-on-cng`

`run-performance-test` job

Skipping `e2e:perf-on-cng` test job

`e2e:test-on-cng`

`e2e:test-on-omnibus-ee`

`e2e:test-on-gdk`

`build-gdk-image`