ContextQMD
Back to Gitlabhq

Merge trains

doc/ci/pipelines/merge_trains.md

19.3.019.8 KB
Original Source

{{< details >}}

  • Tier: Premium, Ultimate
  • Offering: GitLab.com, GitLab Self-Managed, GitLab Dedicated

{{< /details >}}

In projects with frequent merges to the default branch, changes in different merge requests might conflict with each other. Use merge trains to put merge requests in a queue. Each merge request is compared to the other, earlier merge requests, to ensure they all work together.

A merged results pipeline tests one merge request's changes combined with the target branch. A merged results pipeline does not account for other merge requests that merge around the same time. Two merge requests can each pass their own pipeline, but their combined changes can still conflict. If both merge, the target branch can break, even though every pipeline succeeded.

mermaid
%%{init: { "fontFamily": "GitLab Sans" }}%%
graph LR
accTitle: Two merge requests that pass individually but conflict together
accDescr: Merge request A and merge request B each pass a pipeline that tests their changes combined with the target branch alone. When both merge, the combined changes break the target branch.

  subgraph Without merge trains
    target[Target branch] --> pipeline_a[Pipeline for A: passes]
    target --> pipeline_b[Pipeline for B: passes]
    pipeline_a --> merge_both[Both merge]
    pipeline_b --> merge_both
    merge_both -.-> broken[Target branch breaks]
  end

Merge trains prevent this by testing each merge request against the combined changes of every merge request ahead of it in the queue. This catches conflicts before they reach the target branch.

Use merge trains if your project has:

  • Frequent merges to the default branch
  • Multiple merge requests that are often ready to merge around the same time
  • A requirement to keep pipelines passing on the default branch at all times

Merge train workflow

A merge train starts when there are no merge requests waiting to merge and you select Merge or Set to auto-merge. GitLab starts a merge train pipeline that verifies that the changes can merge into the default branch. This first pipeline is the same as a merged results pipeline, which runs on the changes of the source and target branches combined together. The author of the internal merged result commit is the user that initiated the merge.

To queue a second merge request to merge immediately after the first pipeline completes, select Merge or Set to auto-merge to add it to the train. This second merge train pipeline runs on the changes of both merge requests combined with the target branch. Similarly, if you add a third merge request, that pipeline runs on the changes of all three merge requests merged with the target branch. The pipelines all run in parallel.

mermaid
%%{init: { "fontFamily": "GitLab Sans" }}%%
graph LR
accTitle: Merge train pipelines test combined changes
accDescr: Pipeline 1 tests merge request A against the target branch. Pipeline 2 tests merge request A and B together against the target branch. Pipeline 3 tests merge request A, B, and C together against the target branch. The three pipelines run in parallel.

  subgraph Merge train
    target[Target branch] --> pipeline_1[Pipeline 1: A]
    target --> pipeline_2[Pipeline 2: A + B]
    target --> pipeline_3[Pipeline 3: A + B + C]
  end

Each merge request merges into the target branch only after:

  • The merge request's pipeline completes successfully.
  • All other merge requests queued before it are merged.

If a merge train pipeline fails, the merge request is not merged. GitLab removes that merge request from the merge train, and starts new pipelines for all the merge requests that were queued after it.

For example:

Three merge requests (A, B, and C) are added to a merge train in order, which creates three merged results pipelines that run in parallel:

  1. The first pipeline runs on the changes from A combined with the target branch.
  2. The second pipeline runs on the changes from A and B combined with the target branch.
  3. The third pipeline runs on the changes from A, B, and C combined with the target branch.

If the pipeline for B fails:

  • The first pipeline (A) continues to run.
  • B is removed from the train.
  • The pipeline for C is canceled, and a new pipeline starts for the changes from A and C combined with the target branch (without the B changes).

If A then completes successfully, it merges into the target branch, and C continues to run. Any new merge requests added to the train include the A changes now in the target branch, and the C changes from the merge train.

<i class="fa-youtube-play" aria-hidden="true"></i> Watch this video for a demonstration on how parallel execution of merge trains can prevent commits from breaking the default branch.

Automatic pipeline cancellation

GitLab CI/CD detects redundant pipelines, and cancels them to conserve resources.

Redundant merge train pipelines happen when:

In these cases, GitLab must create new merge train pipelines for some or all of the merge requests on the train. The old pipelines were comparing against the previous combined changes in the merge train, which are no longer valid, so these old pipelines are canceled.

Enable merge trains

Prerequisites:

To enable merge trains:

  1. In the top bar, select Search or go to and find your project.
  2. In the left sidebar, select Settings > Merge requests.
  3. In the Merge options section, ensure Enable merged results pipelines is enabled and select Enable merge trains.
  4. Select Save changes.

Start a merge train

Prerequisites:

  • You must have permissions to merge or push to the target branch.

To start a merge train:

  1. Go to a merge request.
  2. Select:

The merge request's merge train status displays under the pipeline widget with a message similar to A new merge train has started and this merge request is the first of the queue. View merge train details. You can select the link to view the merge train.

Other merge requests can now be added to the train.

View a merge train

{{< history >}}

  • Merge train visualization introduced in GitLab 17.3.

{{< /history >}}

You can view the merge train to gain better insight into the order and status of merge requests in the queue. The merge train details page shows active merge requests in the queue and merged merge requests that were part of the train.

To access the merge train details from the list of merge requests:

  1. In the top bar, select Search or go to and find your project.
  2. In the left sidebar, select Code > Merge requests.
  3. Above the list of merge requests, select Merge trains.
  4. Optional. Filter the merge trains by target branch.

You also access this view by selecting View merge train details from:

  • The pipeline widget and system notes on a merge request added to a merge train.
  • The pipeline details page for a merge train pipeline.

You can also remove ({{< icon name="close" >}}) a merge request from the merge train details view.

Add a merge request to a merge train

{{< history >}}

  • Auto-merge for merge trains introduced in GitLab 17.2 with a feature flag named merge_when_checks_pass_merge_train. Disabled by default.
  • Auto-merge for merge trains enabled on GitLab.com in GitLab 17.2.
  • Auto-merge for merge trains enabled by default in GitLab 17.4.
  • Auto-merge for merge trains generally available in GitLab 17.7. Feature flag merge_when_checks_pass_merge_train removed.

{{< /history >}}

Prerequisites:

  • You must have permissions to merge or push to the target branch.

To add a merge request to a merge train:

  1. Visit a merge request.
  2. Select:

The merge request's merge train status displays under the pipeline widget with a message similar to This merge request is 2 of 3 in queue.

Each merge train can run a maximum number of pipelines in parallel. The default limit is 20. If you add more merge requests to the merge train than the limit, the extra merge requests are queued until a pipeline completes. The number of queued merge requests is unlimited.

After a merge request joins a merge train, new conversation threads do not remove it from the train or prevent the merge, even when all threads must be resolved is enabled. This behavior is intentional. For more information, see issue 220916.

Remove a merge request from a merge train

When you remove a merge request from a merge train:

  • All pipelines for merge requests queued after the removed merge request restart.
  • Redundant pipelines are canceled.

You can add the merge request to a merge train again later.

To remove a merge request from a merge train:

  • From a merge request, select Cancel auto-merge.
  • From the merge train details, next to the merge request, select {{< icon name="close" >}}.

Skip the merge train and merge immediately

If you have a high-priority merge request, like a critical patch that must be merged urgently, you can select Merge immediately.

[!warning] Merging immediately can use a lot of CI/CD resources. Use this option only in critical situations.

When you merge a merge request immediately:

  • The commits from the merge request are merged, ignoring the status of the merge train.
  • The merge train pipelines for all other merge requests on the train are canceled.
  • A new merge train starts and all the merge requests from the original merge train are added to this new merge train, with a new merge train pipeline for each. These new merge train pipelines now contain the commits added by the merge request that was merged immediately.

[!note] The merge immediately option may not be available if your project uses the fast-forward merge method and the source branch is behind the target branch. See issue 434070 for more details.

Merge immediately without restarting merge train pipelines

{{< details >}}

  • Status: Experiment

{{< /details >}}

{{< history >}}

{{< /history >}}

[!flag] On GitLab Self-Managed, by default this feature is available. To hide the feature, an administrator can disable the feature flag named merge_trains_skip_train. On GitLab.com and GitLab Dedicated, this feature is available.

You can allow merge requests to be merged without completely restarting a running merge train. Use this feature to quickly merge changes that can safely skip the pipeline, for example minor documentation updates.

You cannot skip merge trains for fast-forward or semi-linear merge methods. For more information, see issue 429009.

Skipping merge trains is an experimental feature. It may change or be removed completely in future releases.

[!warning] You can use this feature to quickly merge security or bug fixes, but the changes in the merge request that skipped the train are not verified against any of the other merge requests in the train. If these other merge train pipelines complete successfully and merge, there is a risk that the combined changes are incompatible. The target branch could then require additional work to resolve the new failures.

Prerequisites:

To enable skipping the train without pipeline restarts:

  1. In the top bar, select Search or go to and find your project.
  2. In the left sidebar, select Settings > Merge requests.
  3. In the Merge options section, ensure the Enable merged results pipelines and Enable merge trains options are enabled.
  4. Select Merge immediately without restarting the merge train.
  5. Select Save changes.

To merge a merge request by skipping the merge train, use the merge requests merge API endpoint to merge with the attribute skip_merge_train set to true.

The merge request merges, and the existing merge train pipelines are not canceled or restarted.

Merge train parallel pipeline limit

{{< history >}}

{{< /history >}}

By default, each merge train can run a maximum of 20 pipelines in parallel. When this limit is reached, additional merge requests are queued until a pipeline slot is available.

To modify this limit for your project:

  1. In the top bar, select Search or go to and find your project.
  2. In the left sidebar, select Settings > Merge requests.
  3. In the Merge options section, set a value for Maximum parallel pipelines per merge train. The minimum value is 1. A value of 1 processes merge requests sequentially with no parallelism.
  4. Select Save changes.

The project limit cannot exceed the instance limit.

You can also use the projects API, or the GraphQL API.

Enforce merge trains

{{< history >}}

{{< /history >}}

By default, if you have permission to merge, you can bypass the merge train. Enforcement requires every merge request to go through the train.

When enforcement is enabled:

  • GitLab hides the Merge immediately options, including Merge now and don't restart train.
  • The REST API and GraphQL API reject direct merges.
  • Auto-merge routes all merges to the train.

Merge train enforcement has three levels:

  • Allow bypass (default): Users with permission to merge can bypass the merge train through the UI or API.
  • Enforce for all users: All merge requests must go through the merge train. No one can bypass the merge train, including Owners and administrators.
  • Enforce with Owner override: All merge requests must go through the merge train, but Owners and administrators can bypass the merge train for individual merge requests.

Prerequisites:

To configure merge train enforcement:

  1. In the top bar, select Search or go to and find your project.
  2. In the left sidebar, select Settings > Merge requests.
  3. In the Merge options section, under Merge train enforcement, select an enforcement level.
  4. Select Save changes.

Troubleshooting

Merge request dropped from the merge train

If a merge request becomes unmergeable while a merge train pipeline is running, the merge train drops your merge request automatically. Common causes include:

  • Changing the merge request to a draft.
  • A merge conflict.

You can find the reason the merge request was dropped from the merge train in the system notes. Check the Activity section in the Overview tab for a message similar to: User removed this merge request from the merge train because ...

Cannot use auto-merge

You cannot use auto-merge (formerly Merge when pipeline succeeds) to skip the merge train, when merge trains are enabled. See issue 12267 for more information.

Cannot retry merge train pipeline

When a merge train pipeline fails, the merge request is dropped from the train and the pipeline can't be retried after it fails. Merge train pipelines run on the merged result of the changes in the merge request and changes from other merge requests already on the train. If the merge request is dropped from the train, the merged result is out of date and the pipeline can't be retried.

You can:

  • Add the merge request to the train again, which triggers a new pipeline.
  • Add the retry keyword to the job if it fails intermittently. If it succeeds after a retry, the merge request is not removed from the merge train.

Cannot add a merge request to the merge train

When Pipelines must succeed is enabled, but the latest pipeline failed:

  • The Set to auto-merge or Merge options are not available.
  • The merge request displays The pipeline for this merge request failed. Please retry the job or push a new commit to fix the failure.

Before you can re-add a merge request to a merge train, you can try to:

  • Retry the failed job. If it passes, and no other jobs failed, the pipeline is marked as successful.
  • Rerun the whole pipeline. On the Pipelines tab, select Run pipeline.
  • Push a new commit that fixes the issue, which also triggers a new pipeline.

For more information, see issue 35135.

Automation tools fail to merge with a 405 error

If merge train enforcement is enabled, any tool that calls the merge requests API without auto_merge=true receives a 405 Method Not Allowed response. This includes scripts, CI/CD jobs, and bots.

To resolve this, update the tool to pass auto_merge=true, which adds the merge request to the merge train instead of merging it directly. For example, if you use Renovate, enable the platformAutomerge configuration option.