doc/development/testing_guide/end_to_end/_index.md
End-to-end (e2e) testing is a strategy used to check whether your application works as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together.
To test GitLab, we:
Additionally, we use the GitLab Development Kit (GDK) as a test environment that can be deployed quickly for faster test feedback.
We run scheduled pipelines each night to test nightly builds created by Omnibus. You can find these pipelines at https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules (requires the Developer role). Results are reported in the #e2e-run-master Slack channel.
We run scheduled pipelines each night to test staging. You can find these pipelines at https://gitlab.com/gitlab-org/quality/staging/pipelines (requires the Developer role). Results are reported in the #e2e-run-staging Slack channel.
End-to-end test pipelines describes pipeline setup responsible for running E2E testing within merge requests.
It is possible to run end-to-end tests for a merge request by triggering the e2e:test-on-omnibus-ee manual action in the qa stage (not available for forks).
This runs end-to-end tests against a custom EE (with an Ultimate license) Docker image built from your merge request's changes.
Manual action that starts end-to-end tests is also available in gitlab-org/omnibus-gitlab merge requests.
In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch.
The end-to-end tests on a merged results pipeline would use the new ref instead of the head of the merge request source branch.
graph LR
A["x1y1z1 - master HEAD"]
B["d1e1f1 - merged results (CI_COMMIT_SHA)"]
A --> B
B --> C["Merged results pipeline"]
C --> D["E2E tests"]
The existing scenarios
that run in the downstream gitlab-qa-mirror pipeline include many tests, but there are times when you might want to run a
test or a group of tests that are different than the groups in any of the existing scenarios.
For example, when we dequarantine a flaky test we first want to make sure that it's no longer flaky. We can do that by running _ee:quarantine manual job. When selecting the name (not the play icon) of manual job, you are prompted to enter variables. You can use any of the variables that can be used with gitlab-qa as well as these:
| Variable | Description |
|---|---|
QA_SCENARIO | The scenario to run (default Test::Instance::Image) |
QA_TESTS | The tests to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests by using RSpec, for example, qa/specs/features/ee/browser_ui would include all the EE UI tests. |
QA_RSPEC_TAGS | The RSpec tags to add (default --tag quarantine) |
For now,
manual jobs with custom variables don't use the same variable when retried,
so if you want to run the same tests multiple times,
specify the same variables in each custom-parallel job (up to as
many of the 10 available jobs that you want to run).
In order to limit amount of tests executed in a merge request, dynamic selection of which tests to execute is present. Algorithm of which tests to run is based on changed files and merge request labels. Following criteria determine which tests will run:
qa framework code would execute the full suite_spec.rb file in qa folder would execute only that particular test. In this case knapsack will not be used to run jobs in parallel.coverband gem is used in a non-standard way for E2E selective test execution in backend MRs.coverband is enabled in the GitLab application only when COVERBAND_ENABLED ENV variable is set. This is set only in the scheduled e2e:test-on-gdk pipeline on master and not in MR pipelines.backend MRs.Mapping based selective test execution is currently in use for test-on-gdk pipeline. For more information, see
epic 47.
To override selective test execution and trigger the full suite, label pipeline:run-all-e2e should be added to particular merge request.
In some cases, it may not be necessary to run the end-to-end test suite.
Examples could include:
Skip running end-to-end tests by applying the pipeline:skip-e2e label to the merge request.
[!warning] There is a risk in skipping end-to-end tests. Use caution and discretion when applying this label. The end-to-end test suite is the last line of defense before changes are merged into the default branch. Skipping these tests increases the risk of introducing regressions into the codebase.
To maintain consistent pipeline run times, the CI/CD job count for each particular E2E test suite is scaled dynamically based on total run time of tests in the suite.
The generate_e2e_pipelines Rake task creates CI/CD YAML files that:
This functionality works in tandem with selective test execution to optimize pipeline run time and costs as much as possible based on particular changes within merge request.
Dynamic job scaling relies on Test Scenario classes. This abstraction encapsulates the following:
The PipelineCreator class generates pipeline YAML files with
dynamically adjusted parallel job counts. Before pipeline YAML generation, PipelineCreator iterates over all defined Test Scenario classes and creates a mapping
which contains a total of calculated test run time for each CI/CD job. Based on this information, the pipeline YAML file is generated with correctly adjusted parallel job count.
PipelineCreator additionally takes input from selective test execution to further reduce the total number of tests that
would be executed.
For an example of how to create a new scenario that would run this scenario in merge request pipelines and scale parallel jobs dynamically, see Adding new jobs to E2E test pipelines.
Our CI setup uses the knapsack gem to enable test parallelization. Knapsack reports are automatically generated and stored in the knapsack-reports GCS bucket within the gitlab-qa-resources project. The KnapsackReport helper manages the report generation and upload process.
To enhance test health visibility, a custom setup exports the pipeline's test execution results to a GCS bucket, with results visualized in Snowflake dashboard.
For additional test results visibility, tests that run on pipelines generate and host Allure test reports.
The QA framework is using the Allure RSpec gem to generate source files for the Allure test report. An additional job in the pipeline:
S3 bucket gitlab-qa-allure-report located in AWS group project eng-quality-ops-ci-cd-shared-infra.A common CI template for report uploading is stored in allure-report.yml.
When these tests are executed in the scope of merge requests, the Allure report is uploaded to the GCS bucket and a bot comment is added linking to their respective reports.
Scheduled pipelines for these tests contain a generate-allure-report job under the Report stage. They also output a link to the current test report. Each type of scheduled pipeline generates a static link for the latest test report according to its stage. You can find a list of this in the GitLab handbook.
Provisioning of all components is performed by the engineering-productivity-infrastructure project.
Use these environment variables to configure metrics export:
| Variable | Required | Information |
|---|---|---|
QA_RUN_TYPE | false | Arbitrary name for test execution, like e2e:test-on-omnibus-ee. Automatically inferred from the project name for live environment test executions. No default value. |
QA_EXPORT_TEST_METRICS | false | Flag to enable or disable metrics export to GCS. Defaults to false. |
If you are not testing code in a merge request, there are two main options for running the tests. If you want to run the existing tests against a live GitLab instance or against a pre-built Docker image, use the GitLab QA orchestrator. See also examples of the test scenarios you can run by using the orchestrator.
On the other hand, if you would like to run against a local development GitLab environment, you can use the GitLab Development Kit (GDK). Refer to the instructions in the QA README and the section below.
Learn how to perform tests that require special setup or consideration to run on your local environment.
Before you write new tests, review the GitLab QA architecture.
After you've decided where to put test environment orchestration scenarios and instance-level scenarios, take a look at the GitLab QA README, the GitLab QA orchestrator README, and the already existing instance-level scenarios.
We should follow these best practices for end-to-end tests:
Flows used to capture reusable sequences of actions in testsResources class that used for creating test datagitlab-qa: For information regarding the use of the GitLab QA orchestratorcustomers-gitlab-com (internal only): For guides that are specific to the CustomersDot platformYou can ask question in the #s_developer_experience channel on Slack (GitLab internal) or you can find an issue you would like to work on in the gitlab issue tracker.