docs/production/upgrade.md
This page explains how to upgrade a Zulip server, including:
settings.py inline documentationWe recommend always running the latest Zulip server release. We work extremely hard to make sure these releases are stable and have no regressions, and that the upgrade process Just Works.
If you do have any questions or problems with your upgrade process, best-effort community support is available in the Zulip development community.
:::{note}
For professional support, upgrade to Zulip Business, or reach out to [email protected].
:::
:::{important}
Be sure to follow the additional instructions if you're {doc}using Docker <docker:how-to/compose-upgrading>, have patched
Zulip, or have modified Zulip-managed configuration
files.
:::
To upgrade to a new Zulip release:
Read the upgrade notes for all releases between your current release and the one you're upgrading to.
Download the appropriate release tarball. You can get the latest release (Zulip Server {{ LATEST_RELEASE_VERSION }}) with the following command:
curl -fLO https://download.zulip.com/server/zulip-server-latest.tar.gz
Log in to your Zulip, and run as root:
/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-latest.tar.gz
Once the Zulip upgrade is complete, you may also wish to upgrade the version of PostgreSQL.
The upgrade process will:
apt-get upgrade.puppet apply.Upgrading will result in brief downtime for the service, which should be under 30 seconds unless there is an expensive database migration involved. Such migrations will be documented in the release notes, and can usually can be avoided with some care. If downtime is problematic for your organization, consider testing the upgrade on a backup in advance, doing the final upgrade at off hours, or arranging support.
:::{note}
If you run into any issues or need to roll back the upgrade, see the troubleshooting guide.
:::
:::{important}
If your deployment is using Docker, please follow {ref}these instructions <docker:how-to/compose-upgrading:upgrading from a git repository>.
:::
Zulip supports upgrading a production installation to any commit in a Git repository. This lets you get unreleased features and bugfixes, or maintain a fork of Zulip.
The git versions documentation describes some branches you may choose to upgrade to, depending on your requirements.
If you are considering upgrading to main, see our upgrading to main
guide for detailed information. You can also
apply a small change to get a fix for an
issue that matters for your organization.
To upgrade to a branch from Git, simply run:
# Upgrade to an official release
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 11.5
# Upgrade to a maintenance branch
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 11.x
# Upgrade to the Zulip Cloud branch
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current
# Upgrade to the `main` branch
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git main
Zulip will automatically fetch the relevant Git commit and upgrade to that version of Zulip.
In addition to the steps described above,
the upgrade process will also build Zulip's frontend assets using webpack.
By default, this uses the main upstream Zulip server repository, but
you can configure any other Git repository by adding a section like
this to /etc/zulip/zulip.conf:
[deployment]
git_repo_url = https://github.com/zulip/zulip.git
settings.py inline documentationZulip installations often upgrade many times over their lifetime, and we strive
to keep all configuration files backwards-compatible. The
/etc/zulip/settings.py file is not automatically modified during upgrades.
After an upgrade, new features may thus be missing from that file, as it is based on a template provided by an earlier version of Zulip.
After upgrading across major versions of Zulip Server, we recommend
comparing your /etc/zulip/settings.py file to the current settings
template, which can be found in
/home/zulip/deployments/current/zproject/prod_settings_template.py. We
suggest using that updated template to update
/etc/zulip/settings.py:
Copy the current settings.py to make a backup (especially if you
do not have a recent complete backup), and make a copy
of the current template:
cp -a /etc/zulip/settings.py ~/zulip-settings-backup.py
cp -a /home/zulip/deployments/current/zproject/prod_settings_template.py /etc/zulip/settings-new.py
Open both /etc/zulip/settings.py and /etc/zulip/settings-new.py
files in an editor; for each setting set in settings.py, find its
section in /etc/zulip/settings-new.py and copy the setting from
settings.py into there.
To help with this process, the following tool finds the most likely version
of the template that your /etc/zulip/settings.py was installed from, and
shows the modifications you've made:
/home/zulip/deployments/current/scripts/setup/compare-settings-to-template
If there are settings which you cannot find documented in
/etc/zulip/settings-new.py, check the changelog to see
if they have been removed.
Overwrite the configuration with the updated file, and restart the server to pick up the updates; this should be a no-op, but it is much better to discover immediately if it is not:
cp -a /etc/zulip/settings-new.py /etc/zulip/settings.py
su zulip -c '/home/zulip/deployments/current/scripts/restart-server'
The upgrade scripts are idempotent, so there's no harm in trying again after resolving an issue. The most common causes of errors are:
upgrade-zulip-from-git, systems with the
minimal RAM for running Zulip can run into out-of-memory issues
during the upgrade process (generally tools/webpack is the step
that fails). You can get past this by shutting down the Zulip
server with ./scripts/stop-server to free up RAM before running
the upgrade process.Useful logs are available in a few places:
/var/log/zulip/upgrade.log./var/log/zulip/errors.log.See also the general Zulip server troubleshooting guide.
This rollback process is intended for minor releases (e.g., 9.4 to
9.3); a more complicated process is required to roll back database
migrations before downgrading to an older major release.
The Zulip upgrade process works by creating a new deployment under
/home/zulip/deployments/ containing a complete copy of the Zulip server code,
and then moving the symlinks at /home/zulip/deployments/{current,last,next}
as part of the upgrade process.
This means that if the new version isn't working,
you can quickly downgrade to the old version by running
/home/zulip/deployments/last/scripts/restart-server, or to an
earlier previous version by running
/home/zulip/deployments/DATE/scripts/restart-server. The
restart-server script stops any running Zulip server, and starts
the version corresponding to the restart-server path you call.
Zulip's upgrades have a hook system which allows for arbitrary user-configured actions to run before and after an upgrade.
Files in the /etc/zulip/pre-deploy.d and /etc/zulip/post-deploy.d
directories are inspected for files ending with .hook, just before and after
the critical period when the server is restarted. Files are called from the
working directory of the new version in alphabetical order, with
environment variables as described below. If any of them exit with non-0 exit
code, the upgrade will abort.
The hook is run with the following environment variables set:
ZULIP_OLD_VERSION: The version being upgraded from, which may either be a
release name (e.g., 10.0 or 10.0-beta1) or the output from git describe
(e.g., 10.0-beta1-2-abcd158b18f2).ZULIP_NEW_VERSION: The version being upgraded to, in the same format as
ZULIP_OLD_VERSION.If the upgrade is upgrading between versions in git, then
the following environment variables will also be present:
ZULIP_OLD_COMMIT: The full commit hash of the version being upgraded from.ZULIP_NEW_COMMIT: The full commit hash of the version being upgraded to.ZULIP_OLD_MERGE_BASE_COMMIT: The full commit hash of the merge-base of the
version being upgraded from, and the public branch in
zulip/zulip. This will be the closest commit in standard
Zulip Server to the version being upgraded from.ZULIP_NEW_MERGE_BASE_COMMIT: The full commit hash of the merge-base of the
version being upgraded to, and the public branch in
zulip/zulip. This will be the closest commit in standard
Zulip Server to the version being upgraded to.See the deploy documentation for hooks included with Zulip.
:::{warning}
If you have modified service configuration files installed by
Zulip (e.g., the nginx configuration), the Zulip upgrade process will
overwrite your configuration when it does the puppet apply.
:::
You can test whether any files will be overwritten assuming no upstream changes
to the configuration using scripts/zulip-puppet-apply (without the -f
option), which will do a test Puppet run and output and changes it would make.
Using this list, you can save a copy of any files that you've modified, do the
upgrade, and then restore your configuration.
That said, Zulip's configuration files are designed to be flexible
enough for a wide range of installations, from a small self-hosted
system to Zulip Cloud. Before making local changes to a configuration
file, first check whether there's an option supported by
/etc/zulip/zulip.conf for the customization you need. And if you
need to make local modifications, please report the issue so that we
can look into making the Zulip Puppet configuration flexible enough to
handle your setup.
If you need to extend Zulip's nginx configuration, there are a few different
include directories you can use, in different contexts:
/etc/nginx/conf.d is in the http context/etc/nginx/zulip-include/app.d is in the server context
for the public-facing server/etc/nginx/zulip-include/localhost.d is in the server
context for the server listening on 127.0.0.1:80, which is
used for internal inter-process communicationThe major version of PostgreSQL is upgraded separately from the Zulip server version. Further, the version of PostgreSQL included with a Zulip server is not linked to that of the host OS; the Zulip installer uses the latest available version of PostgreSQL at installation time (currently, version 16).
The following table details which versions each major Zulip Server version supports:
To upgrade the version of PostgreSQL on the Zulip server:
Upgrade your Zulip server, at least to the latest Zulip maintenance release for your major Zulip version (e.g., upgrade 9.1 to 9.4). This ensures you're using the most robust version of the PostgreSQL upgrade tool.
Stop the server, as the zulip user:
/home/zulip/deployments/current/scripts/stop-server
Take a backup, in case of any problems:
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/postgresql-upgrade.backup.tar.gz
As root, run the database upgrade tool:
/home/zulip/deployments/current/scripts/setup/upgrade-postgresql
As the zulip user, start the server again:
/home/zulip/deployments/current/scripts/start-server
You should now be able to navigate to the Zulip server's URL and confirm everything is working correctly.
When you upgrade the operating system on which Zulip is installed (e.g., Ubuntu 22.04 to Ubuntu 24.04), you need to take some additional steps to update your Zulip installation, documented below.
The steps are largely the same for the various OS upgrades aside from the versions of PostgreSQL, so you should be able to adapt these instructions for other supported platforms.
Upgrade your server to the latest Zulip 8.x release (at
least 8.3, which adds support for Ubuntu 24.04).
As the Zulip user, stop the Zulip server and run the following to back up the system:
/home/zulip/deployments/current/scripts/stop-server
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
One of Zulip's dependencies, RabbitMQ, is used to store deferred work
in queues. RabbitMQ's Ubuntu packaging has problems
upgrading from version 3.9 in Ubuntu 22.04 to 3.12 in Ubuntu
24.04. To work around this bug, you'll need to uninstall
rabbitmq-server, purging its database, before upgrading the OS;
the steps after the OS upgrade will reinstall the new version and
configure it properly. You can do this uninstallation process
safely via the following process:
rabbitmqctl list_queues
zulip user
/home/zulip/deployments/current/manage.py process_queue --all
rabitmqctl list_queues
shows that no queues contain events anymore. You can also decide
to skip this step if you're OK losing a bit of data of the
relevant type.apt purge rabbitmq-server to remove the RabbitMQ
package, including, critically, its database and configuration
state, which would otherwise cause installation of the Ubuntu
24.04 package to crash.Switch to the root user and upgrade the operating system, following the prompts until it completes successfully:
sudo -i # Or otherwise get a root shell
do-release-upgrade
When do-release-upgrade asks you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
nginx, and memcached, the best choice is N to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
The do-release-upgrade tool will complete by prompting you to
restart the system; press N, as we will do so later.
Next, we need to reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python and rewrite Zulip's full-text search indexes to work with the upgraded dictionary packages. This will also take care of re-installing and re-configuring RabbitMQ which we removed earlier.
rm -rf /srv/zulip-venv-cache/* /home/zulip/deployments/current/.venv /root/.cache/uv
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
This process may show a dialog about a "pending kernel upgrade", which can safely be ignored. It may also prompt about "daemons using outdated libraries"; you should select "cancel".
As root, upgrade the database to the latest version of PostgreSQL:
/home/zulip/deployments/current/scripts/setup/upgrade-postgresql
As root, restart the server:
reboot
You should now be able to navigate to your Zulip server's URL and confirm everything is working correctly.
Upgrade your server to the latest Zulip 5.x release (at
least 5.3, which adds support for Ubuntu 22.04 and above).
As the Zulip user, stop the Zulip server and run the following to back up the system:
supervisorctl stop all
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
Switch to the root user and upgrade the operating system, following the prompts until it completes successfully:
sudo -i # Or otherwise get a root shell
do-release-upgrade
When do-release-upgrade asks you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
nginx, and memcached, the best choice is N to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
The do-release-upgrade tool will complete by prompting you to
restart the system; press N, as we will do so later.
Next, we need to reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python and rewrite Zulip's full-text search indexes to work with the upgraded dictionary packages:
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
This process may show a dialog about a "pending kernel upgrade", which can safely be ignored. It may also prompt about "daemons using outdated libraries"; you should select "cancel".
As root, upgrade the database to the latest version of PostgreSQL:
/home/zulip/deployments/current/scripts/setup/upgrade-postgresql
As root, restart the server:
reboot
You should now be able to navigate to your Zulip server's URL and confirm everything is working correctly.
Upgrade your server to the latest Zulip 3.x or 4.x release (at
least 3.0, which adds support for Ubuntu 20.04). You can only
upgrade to Zulip 5.0 and newer after completing this process, since
newer releases don't support Ubuntu 18.04.
As the Zulip user, stop the Zulip server and run the following to back up the system:
supervisorctl stop all
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
Switch to the root user and upgrade the operating system, following the prompts until it completes successfully:
sudo -i # Or otherwise get a root shell
do-release-upgrade
When do-release-upgrade asks you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
nginx, and memcached, the best choice is N to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
The do-release-upgrade tool will complete by prompting you to
restart the system; press N, as we will do so later.
Next, we need to reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python and rewrite Zulip's full-text search indexes to work with the upgraded dictionary packages:
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
As root, upgrade the database to the latest version of PostgreSQL:
/home/zulip/deployments/current/scripts/setup/upgrade-postgresql
Finally, Ubuntu 20.04 has a different version of the low-level glibc library, which affects how PostgreSQL orders text data (known as "collations"); this corrupts database indexes that rely on collations. Regenerate the affected indexes by running:
/home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
As root, restart the server:
reboot
Upgrade from Ubuntu 20.04 to 22.04, so that you are running a supported operating system.
Upgrade your server to the latest Zulip 2.1.x release. You can
only upgrade to Zulip 3.0 and newer after completing this process,
since newer releases don't support Ubuntu 16.04 Xenial.
Same as for Ubuntu 18.04 to 20.04.
Same as for Ubuntu 18.04 to 20.04.
As root, upgrade the database installation and OS configuration to match the new OS version:
touch /usr/share/postgresql/10/pgroonga_setup.sql.applied
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
pg_dropcluster 10 main --stop
systemctl stop postgresql
pg_upgradecluster 9.5 main
pg_dropcluster 9.5 main
apt remove postgresql-9.5
systemctl start postgresql
systemctl restart memcached
Finally, we need to reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python:
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets
This will finish by restarting your Zulip server; you should now be able to navigate to its URL and confirm everything is working correctly.
As root, verify the contents of the full-text indexes:
/home/zulip/deployments/current/manage.py audit_fts_indexes
Upgrade from Ubuntu 18.04 to 20.04, the next in chain of upgrades leading to a supported operating system.
Upgrade your server to the latest 11.x release.
As the Zulip user, stop the Zulip server and run the following to back up the system:
/home/zulip/deployments/current/scripts/stop-server
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
One of Zulip's dependencies, RabbitMQ, is used to store deferred work
in queues. RabbitMQ's Debian packaging has problems
upgrading from version 3.10 in Debian 12 to 4.0 in Debian 13. To
work around this bug, you'll need to uninstall rabbitmq-server,
purging its database, before upgrading the OS; the steps after the
OS upgrade will reinstall the new version and configure it
properly. You can do this uninstallation process safely via the
following process:
rabbitmqctl list_queues
zulip user
/home/zulip/deployments/current/manage.py process_queue --all
rabitmqctl list_queues
shows that no queues contain events anymore. You can also decide
to skip this step if you're OK losing a bit of data of the
relevant type.apt purge rabbitmq-server to remove the RabbitMQ
package, including, critically, its database and configuration
state, which would otherwise cause installation of the Debian 13
package to crash.Follow Debian's instructions to upgrade the OS.
When prompted for you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
nginx, and memcached, the best choice is N to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
Next, we need to reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python and rewrite Zulip's full-text search indexes to work with the upgraded dictionary packages. This will also take care of re-installing and re-configuring RabbitMQ which we removed earlier.
rm -rf /srv/zulip-venv-cache/* /home/zulip/deployments/current/.venv /root/.cache/uv
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
As an additional step, you can also upgrade the PostgreSQL version.
As root, restart the server:
reboot
You should now be able to navigate to your Zulip server's URL and confirm everything is working correctly.
Upgrade your server to the latest 7.x release.
As the Zulip user, stop the Zulip server and run the following to back up the system:
/home/zulip/deployments/current/scripts/stop-server
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
Follow Debian's instructions to upgrade the OS.
When prompted for you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
nginx, and memcached, the best choice is N to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
As root, run the following steps to regenerate configurations for services used by Zulip:
apt remove upstart -y
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
Reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python:
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
This will finish by restarting your Zulip server; you should now be able to navigate to its URL and confirm everything is working correctly.
As an additional step, you can also upgrade the PostgreSQL version.
Upgrade your server to the latest 5.x release. You can only
upgrade to Zulip Server 6.0 and newer after completing this
process, since newer releases don't support Debian 10.
As the Zulip user, stop the Zulip server and run the following to back up the system:
/home/zulip/deployments/current/scripts/stop-server
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
Follow Debian's instructions to upgrade the OS.
When prompted for you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
nginx, and memcached, the best choice is N to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
As root, run the following steps to regenerate configurations for services used by Zulip:
apt remove upstart -y
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
Reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python:
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets --audit-fts-indexes
This will finish by restarting your Zulip server; you should now be able to navigate to its URL and confirm everything is working correctly.
Debian 11 has a different version of the low-level glibc library, which affects how PostgreSQL orders text data (known as "collations"); this corrupts database indexes that rely on collations. Regenerate the affected indexes by running:
/home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
As an additional step, you can also upgrade the PostgreSQL version.
Upgrade from Debian 11 to 12, so that you are running a supported operating system.
Upgrade your server to the latest Zulip 2.1.x release. You can
only upgrade to Zulip 3.0 and newer after completing this process,
since newer releases don't support Debian 9.
As the Zulip user, stop the Zulip server and run the following to back up the system:
supervisorctl stop all
/home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
Follow Debian's instructions to upgrade the OS.
When prompted for you how to upgrade configuration
files for services that Zulip manages like Redis, PostgreSQL,
nginx, and memcached, the best choice is N to keep the
currently installed version. But it's not important; the next
step will re-install Zulip's configuration in any case.
As root, upgrade the database installation and OS configuration to match the new OS version:
apt remove upstart -y
/home/zulip/deployments/current/scripts/zulip-puppet-apply -f
pg_dropcluster 11 main --stop
systemctl stop postgresql
pg_upgradecluster -m upgrade 9.6 main
pg_dropcluster 9.6 main
apt remove postgresql-9.6
systemctl start postgresql
service memcached restart
Finally, we need to reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python:
rm -rf /srv/zulip-venv-cache/*
/home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
/home/zulip/deployments/current/ --ignore-static-assets
This will finish by restarting your Zulip server; you should now be able to navigate to its URL and confirm everything is working correctly.
Upgrade to the latest 5.x release, now
that your server is running a supported operating system.
Debian 10 has a different version of the low-level glibc library, which affects how PostgreSQL orders text data (known as "collations"); this corrupts database indexes that rely on collations. Regenerate the affected indexes by running:
/home/zulip/deployments/current/scripts/setup/reindex-textual-data --force
As root, finish by verifying the contents of the full-text indexes:
/home/zulip/deployments/current/manage.py audit_fts_indexes
Upgrade from Debian 10 to 11, the next in chain of upgrades leading to a supported operating system.