Troubleshooting background migrations

{{< details >}}

• Tier: Free, Premium, Ultimate
• Offering: GitLab Self-Managed

{{< /details >}}

Database migrations failing because of batched background migration not finished

When updating to GitLab version 14.2 or later, database migrations might fail with a message like:

StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
  {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob",
   :table_name=>"push_event_payloads",
   :column_name=>"event_id",
   :job_arguments=>[["event_id"],
   ["event_id_convert_to_bigint"]]
  }

To resolve this error:

• If you followed the version-specific upgrade instructions for 14.2, manually finish the batched background migration.
• If you didn't follow those instructions, you must either:
  • Roll back and upgrade through one of the required versions before updating to 14.2+.
  • Roll forward, staying on the current version and manually ensuring that the batched migrations complete successfully.

Roll back and follow the required upgrade path

To roll back and follow the required upgrade path:

1. Roll back and restore the previously installed version.
2. Update to either 14.0.5 or 14.1 before updating to 14.2+.
3. Check the status of the batched background migrations and make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, manually finish them.

Roll forward and finish the migrations on the upgraded version

The process for rolling forward depends on whether no downtime is required or not.

For a deployment with downtime

Running all batched background migrations can take a significant amount of time depending on the size of your GitLab installation.

1. Check the status of the batched background migrations in the database, and manually run them with the appropriate arguments until the status query returns no rows.

2. When the status of all of all them is marked as complete, re-run migrations for your installation.

3. Complete the database migrations from your GitLab upgrade:

   plaintextCopy

   sudo gitlab-rake db:migrate
   

4. Run a reconfigure:

   plaintextCopy

   sudo gitlab-ctl reconfigure
   

5. Finish the upgrade for your installation.

For a no-downtime deployment

Because the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded version and wait for the batched background migrations to finish.

1. Check the status of the batched background migration from the error message, and make sure it is listed as finished. If the migration is still active, either:
   • Wait until it is finished.
   • Manually finish it.
2. Re-run migrations for your installation so the remaining post-deployment migrations finish.

Advanced search migrations are stuck

In GitLab 15.0, an advanced search migration named DeleteOrphanedCommit can be permanently stuck in a pending state across upgrades. This issue is corrected in GitLab 15.1.

If you are a GitLab Self-Managed customer who uses GitLab 15.0 with advanced search, you will experience performance degradation. To clean up the migration, upgrade to 15.1 or later.

For other advanced search migrations stuck in pending problems, retry the halted migrations.

If you upgrade GitLab before all pending advanced search migrations are completed, any pending migrations that have been removed in the new version cannot be executed or retried. In this case, you must re-create your index from scratch.

Error: Elasticsearch version not compatible

To resolve this problem, confirm that your version of Elasticsearch or OpenSearch is compatible with your version of GitLab.