doc/administration/backup_restore/restore_gitlab.md
{{< details >}}
{{< /details >}}
GitLab restore operations recover data from backups to maintain system continuity and recover from data loss. Restore operations:
The restore process requires an existing GitLab installation of the same version as the backup. Follow the prerequisites and test the complete restore process before using it in production.
You need to have a working GitLab installation before you can perform a
restore. This is because the system user performing the restore actions (git)
is usually not allowed to create or delete the SQL database needed to import
data into (gitlabhq_production).
The restore process handles existing data differently depending on the data type:
For a reliable restore process, for example when automating production-to-staging restores, use a fresh GitLab installation at the same version as the backup.
Restoring SQL data skips views owned by PostgreSQL extensions.
You can only restore a backup to exactly the same version and type (CE or EE) of GitLab on which it was created. For example, CE 15.1.4.
If your backup is a different version than the current installation, you must downgrade or upgrade your GitLab installation before restoring the backup.
To restore a backup, you must also restore the GitLab secrets. If you are migrating to a new GitLab instance, you must copy the GitLab secrets file from the old server. These include the database encryption key, CI/CD variables, and variables used for two-factor authentication. Without the keys, multiple issues occur, including loss of access by users with two-factor authentication enabled, and GitLab Runners cannot sign in.
[!warning] WebAuthn devices are disabled when restoring to a different FQDN: WebAuthn registrations (such as YubiKeys) are cryptographically bound to the origin (domain/hostname) where they were created. If you restore a backup to a GitLab instance with a different FQDN than the original instance, all WebAuthn devices will be disabled. Users will need to re-register their WebAuthn devices after the restore is complete.
For more information about WebAuthn and hostname requirements, see Two-factor authentication.
Based on your installation method, restore the following:
{{< tabs >}}
{{< tab title="Linux package" >}}
/etc/gitlab/gitlab-secrets.json
{{< /tab >}}
{{< tab title="Helm chart (Kubernetes)" >}}
GitLab Helm chart secrets can be converted to the Linux package format, if required.
{{< /tab >}}
{{< tab title="Docker" >}}
If you have mounted /etc/gitlab under /srv/gitlab/config:
/srv/gitlab/config/gitlab-secrets.json
{{< /tab >}}
{{< tab title="Self-compiled (source)" >}}
/home/git/gitlab/.secret
{{< /tab >}}
{{< /tabs >}}
See also:
You likely want to separately restore your previous /etc/gitlab/gitlab.rb (for Linux package installations)
or /home/git/gitlab/config/gitlab.yml (for self-compiled installations) and
any TLS or SSH keys and certificates.
Certain configuration is coupled to data in PostgreSQL. For example:
default, my-storage-1, and my-storage-2), then the target environment must also have at least those storage names defined in configuration.For more information, see Data not included in backup.
If you're restoring into directories that are mount points, you must ensure these directories are empty before attempting a restore. Otherwise, GitLab attempts to move these directories before restoring the new data, which causes an error.
Read more about configuring NFS mounts.
This procedure assumes that:
sudo gitlab-ctl reconfigure at least once.sudo gitlab-ctl start.First ensure your backup tar file is in the backup directory described in the
gitlab.rb configuration gitlab_rails['backup_path']. The default is
/var/opt/gitlab/backups. The backup file needs to be owned by the git user.
sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups/
sudo chown git:git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar
Stop the processes that are connected to the database. Leave the rest of GitLab running:
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
# Verify
sudo gitlab-ctl status
Next, ensure you have completed the restore prerequisites steps and have run gitlab-ctl reconfigure
after copying over the GitLab secrets file from the original installation.
Next, restore the backup, specifying the ID of the backup you wish to restore:
[!warning] The following command overwrites the contents of your GitLab database!
# NOTE: "_gitlab_backup.tar" is omitted from the name
sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
If there's a GitLab version mismatch between your backup tar file and the installed version of GitLab, the restore command aborts with an error message:
GitLab version mismatch:
Your current GitLab version (16.5.0-ee) differs from the GitLab version in the backup!
Please switch to the following version and try again:
version: 16.4.3-ee
Install the correct GitLab version, and then try again.
[!warning] The restore command requires additional parameters when your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster.
Run reconfigure on the PostgreSQL node:
sudo gitlab-ctl reconfigure
Next, start and check GitLab:
sudo gitlab-ctl start
sudo gitlab-rake gitlab:check SANITIZE=true
Verify that the database values can be decrypted
especially if /etc/gitlab/gitlab-secrets.json was restored, or if a different server is
the target for the restore.
sudo gitlab-rake gitlab:doctor:secrets
For added assurance, you can perform an integrity check on the uploaded files:
sudo gitlab-rake gitlab:artifacts:check
sudo gitlab-rake gitlab:lfs:check
sudo gitlab-rake gitlab:uploads:check
After the restore is completed, it's recommended to generate database statistics to improve the database performance and avoid inconsistencies in the UI:
Enter the database console.
Run the following:
SET STATEMENT_TIMEOUT=0 ; ANALYZE VERBOSE;
There are ongoing discussions about integrating the command into the restore command, see issue 276184 for more details.
Post-restore verification guides:
For GitLab installations using the Docker image or the GitLab Helm chart on a
Kubernetes cluster, the restore task expects the restore directories to be
empty. However, with Docker and Kubernetes volume mounts, some system level
directories may be created at the volume roots, such as the lost+found
directory found in Linux operating systems. These directories are usually owned
by root, which can cause access permission errors because the restore Rake task
runs as the git user. To restore a GitLab installation, users have to confirm
the restore target directories are empty.
For both these installation types, the backup tarball has to be available in
the backup location (default location is /var/opt/gitlab/backups).
The GitLab Helm chart uses the process documented in restoring a GitLab Helm chart installation
If you're using Docker Swarm, the container might restart during the restore process because Puma is shut down, and so the container health check fails. To work around this problem, temporarily disable the health check mechanism.
Edit docker-compose.yml:
healthcheck:
disable: true
Deploy the stack:
docker stack deploy --compose-file docker-compose.yml mystack
For more information, see issue 6846.
The restore task can be run from the host:
# Stop the processes that are connected to the database
docker exec -it <name of container> gitlab-ctl stop puma
docker exec -it <name of container> gitlab-ctl stop sidekiq
# Verify that the processes are all down before continuing
docker exec -it <name of container> gitlab-ctl status
# Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name
docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
# Restart the GitLab container
docker restart <name of container>
# Check GitLab
docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true
First, ensure your backup tar file is in the backup directory described in the
gitlab.yml configuration:
## Backup settings
backup:
path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/)
The default is /home/git/gitlab/tmp/backups, and it needs to be owned by the git user.
Begin the backup procedure:
# Stop processes that are connected to the database
sudo service gitlab stop
sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production
Example output:
Unpacking backup... [DONE]
Restoring database tables:
-- create_table("events", {:force=>true})
-> 0.2231s
[...]
- Loading fixture events...[DONE]
- Loading fixture issues...[DONE]
- Loading fixture keys...[SKIPPING]
- Loading fixture merge_requests...[DONE]
- Loading fixture milestones...[DONE]
- Loading fixture namespaces...[DONE]
- Loading fixture notes...[DONE]
- Loading fixture projects...[DONE]
- Loading fixture protected_branches...[SKIPPING]
- Loading fixture schema_migrations...[DONE]
- Loading fixture services...[SKIPPING]
- Loading fixture snippets...[SKIPPING]
- Loading fixture taggings...[SKIPPING]
- Loading fixture tags...[SKIPPING]
- Loading fixture users...[DONE]
- Loading fixture users_projects...[DONE]
- Loading fixture web_hooks...[SKIPPING]
- Loading fixture wikis...[SKIPPING]
Restoring repositories:
- Restoring repository abcd... [DONE]
- Object pool 1 ...
Deleting tmp directories...[DONE]
Restore /home/git/gitlab/.secret if necessary.
Restart GitLab:
sudo service gitlab restart
Although the Rake task used to restore a GitLab instance doesn't support restoring a single project or group, you can use a workaround by restoring your backup to a separate, temporary GitLab instance, and then export your project or group from there:
A feature request to provide direct restore of individual projects or groups is being discussed in issue #17517.
When you create an incremental repository backup by using
gitlab-backup, the resulting backup archive contains all repository data needed for a complete restore.
To restore, use the same instructions as restoring any other regular backup archive.
Internally, incremental repository backups store only the changes made after the previous backup.
When you create an incremental backup, gitlab-backup bundles every step from the original full backup
onward into the backup archive. This means the archive is self-contained, even though the individual repository
backup bundles depend on each other.
With server-side repository backups, the backup archive does not contain repository data. Instead, repository data is stored in object storage by each Gitaly node, and each increment is stored as a separate object. In a server-side restore, Gitaly reads the backup manifest and applies each increment in order.
[!warning] Do not delete incremental backup files from object storage. If an intermediate file is deleted (for example, through an object storage lifecycle policy), the backup chain is broken and the backup cannot be restored.
The command-line tool GitLab provides to restore from backup can accept more options.
Backup files use a naming scheme starting with a backup ID. When more than one backup exists, you must specify which
<backup-id>_gitlab_backup.tar file to restore by setting the environment variable BACKUP=<backup-id>.
During a restore from backup, the restore script prompts for confirmation:
<!-- vale gitlab_base.Spelling = NO -->authorized_keys file.To disable these prompts, set the GITLAB_ASSUME_YES environment variable to 1.
Linux package installations:
sudo GITLAB_ASSUME_YES=1 gitlab-backup restore
Self-compiled installations:
sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production
The force=yes environment variable also disables these prompts.
You can exclude specific tasks on restore by adding the environment variable SKIP, whose values are a comma-separated list of the following options:
db (database)uploads (attachments)builds (CI job output logs)artifacts (CI job artifacts)lfs (LFS objects)terraform_state (Terraform states)registry (Container registry images)pages (Pages content)repositories (Git repositories data)packages (Packages)To exclude specific tasks:
Linux package installations:
sudo gitlab-backup restore BACKUP=<backup-id> SKIP=db,uploads
Self-compiled installations:
sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=<backup-id> SKIP=db,uploads RAILS_ENV=production
{{< history >}}
{{< /history >}}
[!warning] GitLab 17.1 and earlier are affected by a race condition that can cause data loss. The problem affects repositories that have been forked and use GitLab object pools. To avoid data loss, only restore backups by using GitLab 17.2 or later.
When using multiple repository storages,
repositories from specific repository storages can be restored separately
using the REPOSITORIES_STORAGES option. The option accepts a comma-separated list of
storage names.
For example:
Linux package installations:
sudo gitlab-backup restore BACKUP=<backup-id> REPOSITORIES_STORAGES=storage1,storage2
Self-compiled installations:
sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=<backup-id> REPOSITORIES_STORAGES=storage1,storage2
{{< history >}}
{{< /history >}}
[!warning] GitLab 17.1 and earlier are affected by a race condition that can cause data loss. The problem affects repositories that have been forked and use GitLab object pools. To avoid data loss, only restore backups by using GitLab 17.2 or later.
You can restore specific repositories using the REPOSITORIES_PATHS and the SKIP_REPOSITORIES_PATHS options.
Both options accept a comma-separated list of project and group paths. If you
specify a group path, all repositories in all projects in the group and
descendant groups are included or skipped, depending on which option you used.
Both the groups and projects must exist in the specified backup or on the target instance.
[!note] The
REPOSITORIES_PATHSandSKIP_REPOSITORIES_PATHSoptions apply only to Git repositories. They do not apply to project or group database entries. If you created a repositories backup withSKIP=db, by itself it cannot be used to restore specific repositories to a new instance.
For example, to restore all repositories for all projects in Group A (group-a), the repository for
Project C in Group B (group-b/project-c), and skip the Project D in Group A (group-a/project-d):
Linux package installations:
sudo gitlab-backup restore BACKUP=<backup-id> REPOSITORIES_PATHS=group-a,group-b/project-c SKIP_REPOSITORIES_PATHS=group-a/project-d
Self-compiled installations:
sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=<backup-id> REPOSITORIES_PATHS=group-a,group-b/project-c SKIP_REPOSITORIES_PATHS=group-a/project-d
If an untarred backup (made with SKIP=tar) is found,
and no backup is chosen with BACKUP=<backup-id>, the untarred backup is used.
For example:
Linux package installations:
sudo gitlab-backup restore
Self-compiled installations:
sudo -u git -H bundle exec rake gitlab:backup:restore
{{< history >}}
gitlab-backup in GitLab 16.3.gitlab-backup for restoring a specified backup instead of the latest backup introduced in GitLab 16.6.gitlab-backup for creating incremental backups introduced in GitLab 16.6.backup-utility introduced in GitLab 17.0.{{< /history >}}
When a server-side backup is collected, the restore process defaults to use the server-side restore mechanism shown in Create server-side repository backups. You can configure backup restoration so that the Gitaly node that hosts each repository is responsible for pulling the necessary backup data directly from object storage.
{{< tabs >}}
{{< tab title="Linux package (Omnibus)" >}}
sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
{{< /tab >}}
{{< tab title="Self-compiled" >}}
sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=11493107454_2018_04_25_10.6.4-ce
{{< /tab >}}
{{< tab title="Helm chart (Kubernetes)" >}}
kubectl exec <Toolbox pod name> -it -- backup-utility --restore -t <backup_ID> --repositories-server-side
When using cron-based backups,
add the --repositories-server-side flag to the extra arguments.
{{< /tab >}}
{{< /tabs >}}
The following are possible problems you might encounter, along with potential solutions.
If you're using backup restore procedures, you may encounter the following warning messages:
ERROR: must be owner of extension pg_trgm
ERROR: must be owner of extension btree_gist
ERROR: must be owner of extension plpgsql
WARNING: no privileges could be revoked for "public" (two occurrences)
WARNING: no privileges were granted for "public" (two occurrences)
Be advised that the backup is successfully restored in spite of these warning messages.
The Rake task runs this as the gitlab user, which doesn't have superuser
access to the database. When restore is initiated, it also runs as the gitlab
user, but it also tries to alter the objects it doesn't have access to.
Those objects have no influence on the database backup or restore, but display
a warning message.
For more information, see:
While restoring from backup, you can encounter an error when the following are true:
custom_hook) is configured using the method for GitLab version 15.10 and earlierThe error looks like:
{"level":"fatal","msg":"restore: pipeline: 1 failures encountered:\n - @hashed/path/to/hashed_repository.git (path/to_project): manager: restore custom hooks, \"@hashed/path/to/hashed_repository/<BackupID>_<GitLabVersion>-ee/001.custom_hooks.tar\": rpc error: code = Internal desc = setting custom hooks: generating prepared vote: walking directory: copying file to hash: read /mnt/gitlab-app/git-data/repositories/+gitaly/tmp/default-repositories.old.<timestamp>.<temporaryfolder>/custom_hooks/compliance-triggers.d: is a directory\n","pid":3256017,"time":"2023-08-10T20:09:44.395Z"}
To resolve this, you can update the Git server hooks for GitLab version 15.11 and later, and create a new backup.
fapolicydWhen using fapolicyd for increased security, GitLab can report that a restore was successful but repositories show as empty. For more troubleshooting help, see
Gitaly Troubleshooting documentation.