ContextQMD
Back to Gitlabhq

Before you upgrade

doc/update/plan_your_upgrade.md

18.11.29.2 KB
Original Source

{{< details >}}

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab Self-Managed

{{< /details >}}

Before you upgrade a GitLab instance, you must:

  1. Gather pre-upgrade information to prepare yourself for the upgrade.
  2. Perform pre-upgrade steps before you upgrade GitLab itself.

Gather pre-upgrade information

When planning the upgrade:

  1. Review the GitLab release and maintenance policy.
  2. Consult the GitLab upgrade notes for different versions of GitLab to ensure compatibility.
  3. If relevant, check OS compatibility with the target GitLab version.
  4. If you're using Geo:
  5. Determine the appropriate upgrade path for your instance, including any required upgrade stops. Upgrade stops might require you to perform multiple upgrades.
  6. Run upgrade health checks to find and resolve potential issues early.
  7. Create an upgrade plan that documents:
  8. Test your upgrade plan in a clone of your production environment. This helps reduce the risk of unplanned outages and helps you measure the duration of potentially long-running migrations.

With all pre-upgrade information gathered, you can move on to performing pre-upgrade steps.

Create a rollback plan and backup

Something might go wrong during an upgrade, so it's critical that you have a rollback plan. A proper rollback plan creates a clear path to bring a GitLab instance back to its last working state and comprises:

  • The process to back up the instance.
  • The process to restore the instance.

You should test the rollback plan before you need it. For an overview of the steps required for rolling back, see roll back to earlier GitLab versions.

Create a GitLab backup

To make it possible to roll back GitLab if there's a problem with the upgrade, either:

  • Create a GitLab backup. You must follow the instructions based on your installation method and make sure to back up the secrets and configuration files.
  • Create a snapshot of your instance. If you instance is multi-node installation, you must snapshot every node. This process is out of scope for GitLab Support.

Roll back GitLab

If you have a test environment that mimics production, test the restoration to ensure that everything works as you expect.

To restore your GitLab backup:

  1. Refer to restore prerequisites. Most importantly, the versions of the backed up and the new GitLab instance must be the same.
  2. Restore GitLab by following the instructions based on your installation method.
  3. Confirm that the secrets and configuration files are also restored.

If restoring from a snapshot, you must already know how to do this. This process is out of scope for GitLab Support.

Perform pre-upgrade steps

Shortly before you perform the upgrade:

  1. Rerun upgrade health checks.
  2. Perform upgrades for any optional features that you use.

Run upgrade health checks

Immediately before and after the upgrade, run upgrade health checks to ensure the major components of GitLab are working:

  1. Check the general configuration:

    shell
    sudo gitlab-rake gitlab:check

  2. Check the status of all background database migrations. All migrations must finish running before each upgrade. You must spread out upgrades between major and minor releases to allow time for background migrations to finish.

  3. Confirm that encrypted database values can be decrypted:

    shell
    sudo gitlab-rake gitlab:doctor:secrets

  4. In the GitLab UI, check that:

    • Users can sign in.
    • The project list is visible.
    • Project issues and merge requests are accessible.
    • Users can clone repositories from GitLab.
    • Users can push commits to GitLab.

  5. For GitLab CI/CD, check that:

    • Runners pick up jobs.
    • Docker images can be pushed and pulled from the registry.

  6. If using Geo, run the relevant checks on the primary site and each secondary site:

    shell
    sudo gitlab-rake gitlab:geo:check

  7. If using Elasticsearch, verify that searches are successful.

If something goes wrong, get support.

Upgrades for optional features

Depending on how your GitLab instance is configured, you might be required to perform these additional steps before upgrading GitLab:

  1. If using external Gitaly servers, upgrade the Gitaly servers to the newer version before upgrading GitLab itself. This prevents the gRPC client on the application server from sending RPCs that the old Gitaly version does not support.

  2. If you have Kubernetes clusters connected with GitLab, upgrade your GitLab agents for Kubernetes to match your new GitLab version.

  3. If you use advanced search (Elasticsearch), confirm advanced search migrations are complete by checking for pending migrations.

    After upgrading GitLab, you might have to upgrade Elasticsearch if the new version breaks compatibility. Updating Elasticsearch is out of scope for GitLab Support.

Pause CI/CD pipelines and jobs

During upgrades for most types of GitLab instances, you should pause CI/CD pipelines and jobs.

If you upgrade your GitLab instance while GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. If a trace update does not self-heal, depending on the error, GitLab Runner either retries or terminates job handling.

GitLab Runner attempts to upload job artifacts three times, after which the job fails.

To pause CI/CD pipelines and jobs:

  1. Pause the runners.

  2. Block new jobs from starting by adding the following to your /etc/gitlab/gitlab.rb file:

    ruby
    nginx['custom_gitlab_server_config'] = "location = /api/v4/jobs/request {\n deny all;\n return 503;\n}\n"

  3. Reconfigure GitLab:

    shell
    sudo gitlab-ctl reconfigure

  4. Wait until all jobs are finished.

When you've completed your GitLab upgrade:

  1. Unpause your runners.
  2. Unblock new jobs from starting by reverting the previous /etc/gitlab/gitlab.rb change.

Working with Support

If you are working with Support to review your upgrade plan, document and share it with the answers to the following questions:

  • How is GitLab installed?
  • What is the operating system of the node? Check the supported platforms to confirm that later updates are available.
  • Is it a single-node or a multi-node setup? If multi-node, document and share any architectural details about each node. Which external components are used? For example, Gitaly, PostgreSQL, or Redis?
  • Are you using Geo? If so, document and share any architectural details about each secondary site.
  • What else might be unique or interesting in your setup that might be important?
  • Are you running into any known issues with your current version of GitLab?

Get support

If something goes wrong during your upgrade:

  1. Copy any errors and gather any logs to analyze later. Use the following tools to help you gather data:
    • gitlabsos if you installed GitLab with the Linux package or Docker.
    • kubesos if you installed GitLab using the Helm Charts.
  2. Roll back to the last working version.

For support: