Migrate to a new server - Gitlabhq

Tier: Free, Premium, Ultimate
Offering: GitLab Self-Managed

You can use GitLab backup and restore to migrate your instance to a new server. This section outlines a typical procedure for a GitLab deployment running on a single server using the Linux package.

If you're running GitLab Geo, an alternative option is Geo disaster recovery for planned failover. You must make sure all sites meet the Geo requirements before selecting Geo for the migration.

[!warning] Avoid uncoordinated data processing by both the new and old servers, where multiple servers could connect concurrently and process the same data. For example, when using incoming email, if both GitLab instances are processing email at the same time, then both instances miss some data. This type of problem can occur with other services as well, such as a non-packaged database, a non-packaged Redis instance, or non-packaged Sidekiq.

Prerequisites:

A broadcast message banner published in advance to notify your users of the upcoming migration.
Complete and current backups. Create a complete system-level backup, or take a snapshot of all servers involved in the migration, in case destructive commands (like rm) are run incorrectly.
Administrator access.

Prepare the new server

To prepare the new server:

Copy the SSH host keys from the old server to avoid man-in-the-middle attack warnings. See Manually replicate the primary site's SSH host keys for example steps.
Install GitLab.
Configure by copying /etc/gitlab files from the old server to the new server, and update as necessary. Read the Linux package installation backup and restore instructions for more detail.
If applicable, disable incoming email.
Block new CI/CD jobs from starting upon initial startup after the backup and restore. Edit /etc/gitlab/gitlab.rb and set the following:
ruby
```
nginx['custom_gitlab_server_config'] = "location = /api/v4/jobs/request {\n    deny all;\n    return 503;\n  }\n"
```
Reconfigure GitLab:
shell
```
sudo gitlab-ctl reconfigure
```
Stop GitLab to avoid any potential unnecessary and unintentional data processing:
shell
```
sudo gitlab-ctl stop
```
Stop Redis:
shell
```
sudo gitlab-ctl stop redis
```

Configure the new server to allow receiving the Redis database and GitLab backup files:

shell

sudo rm -f /var/opt/gitlab/redis/dump.rdb
sudo chown <your-linux-username> /var/opt/gitlab/redis /var/opt/gitlab/backups

Prepare and transfer content from the old server

Ensure you have an up-to-date system-level backup or snapshot of the old server.
Enable maintenance mode, if supported by your GitLab edition.

Block new CI/CD jobs from starting:

Edit /etc/gitlab/gitlab.rb, and set the following:

ruby

nginx['custom_gitlab_server_config'] = "location = /api/v4/jobs/request {\n    deny all;\n    return 503;\n  }\n"

Reconfigure GitLab:
shell
```
sudo gitlab-ctl reconfigure
```

Disable periodic background jobs:
1. In the upper-right corner, select Admin.
2. In the left sidebar, select Monitoring > Background jobs to show the Sidekiq dashboard.
3. On the Sidekiq dashboard, on its top menu, select Cron.
4. On the Sidekiq dashboard, on its upper right, select Disable All.
Wait for the running CI/CD jobs to finish, or accept that jobs that have not completed may be lost. To view all running jobs:
1. In the left sidebar, select CI/CD > Jobs.
2. In the filter bar, select Status > Running.
Wait for Sidekiq jobs to finish:
1. In the left sidebar, select Monitoring > Background jobs.
2. On the Sidekiq dashboard, on its top menu, select Queues.
3. On the Sidekiq dashboard, on its upper right, select Live Poll. Wait for Busy and Enqueued to drop to 0. These queues contain work that has been submitted by your users; shutting down before these jobs complete may cause the work to be lost. Make note of the numbers shown in the Sidekiq dashboard for post-migration verification.

Flush the Redis database to disk, and stop GitLab other than the services needed for migration:

shell

sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && \
sudo gitlab-ctl stop && \
sudo gitlab-ctl start postgresql && \
sudo gitlab-ctl start gitaly

Create a GitLab backup:
shell
```
sudo gitlab-backup create
```

After the backup is complete, disable the following GitLab services and prevent unintentional restarts by adding the following to the bottom of /etc/gitlab/gitlab.rb:

ruby

alertmanager['enable'] = false
gitaly['enable'] = false
gitlab_exporter['enable'] = false
gitlab_pages['enable'] = false
gitlab_workhorse['enable'] = false
grafana['enable'] = false
logrotate['enable'] = false
gitlab_rails['incoming_email_enabled'] = false
nginx['enable'] = false
node_exporter['enable'] = false
postgres_exporter['enable'] = false
postgresql['enable'] = false
prometheus['enable'] = false
puma['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
registry['enable'] = false
sidekiq['enable'] = false

Reconfigure GitLab:
shell
```
sudo gitlab-ctl reconfigure
```
Verify everything is stopped, and confirm no services are running:
shell
```
sudo gitlab-ctl status
```

Transfer the Redis database and GitLab backups to the new server:

shell

sudo scp /var/opt/gitlab/redis/dump.rdb <your-linux-username>@new-server:/var/opt/gitlab/redis
sudo scp /var/opt/gitlab/backups/your-backup.tar <your-linux-username>@new-server:/var/opt/gitlab/backups

For instances with a large volume of Git and object data

If your GitLab instance has a large amount of data on local volumes, for example greater than 1 TB, backups may take a long time. In that case, you may find it easier to transfer the data to the appropriate volumes on the new instance.

The main volumes that you might need to migrate manually are:

The /var/opt/gitlab/git-data directory which contains all the Git data. Be sure to read the moving repositories documentation section to eliminate the chance of Git data corruption.
The /var/opt/gitlab/gitlab-rails/shared directory which contains object data, like artifacts.
The /var/opt/gitlab/gitlab-rails/uploads directory which contains upload data, like user profile photos.
If you are using the bundled PostgreSQL included with the Linux package, you also need to migrate the PostgreSQL data directory under /var/opt/gitlab/postgresql/data.

After all GitLab services have been stopped, you can use tools like rsync or mounting volume snapshots to move the data to the new environment.

Restore data on the new server

Restore appropriate file system permissions:

shell

sudo chown gitlab-redis /var/opt/gitlab/redis
sudo chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis/dump.rdb
sudo chown git:root /var/opt/gitlab/backups
sudo chown git:git /var/opt/gitlab/backups/your-backup.tar

Start Redis:
shell
```
sudo gitlab-ctl start redis
```
Redis picks up and restores dump.rdb automatically.
Restore the GitLab backup.
Verify that the Redis database restored correctly:
1. In the upper-right corner, select Admin.
2. In the left sidebar, select Monitoring > Background jobs.
3. Under the Sidekiq dashboard, verify that the numbers match with what was shown on the old server.
4. While still under the Sidekiq dashboard, select Cron and then Enable All to re-enable periodic background jobs.
Test that read-only operations on the GitLab instance work as expected. For example, browse through project repository files, merge requests, and issues.
Disable Maintenance Mode, if previously enabled.
Test that the GitLab instance is working as expected.
If applicable, re-enable incoming email and test it is working as expected.
Update your DNS or load balancer to point at the new server.

Unblock new CI/CD jobs from starting by removing the custom NGINX configuration you added previously:

ruby

# The following line must be removed
nginx['custom_gitlab_server_config'] = "location = /api/v4/jobs/request {\n    deny all;\n    return 503;\n  }\n"

Reconfigure GitLab:
shell
```
sudo gitlab-ctl reconfigure
```
Remove the scheduled maintenance broadcast message banner.