UPGRADING/v0_16.md
v0.15.x to v0.16.xStalwart v0.16.x introduces significant breaking changes that make its configuration and management layer completely incompatible with every previous release. The database layout used to store user data (emails, calendars, contacts, files, blobs, search indexes) is not affected by this change, so message bodies, mailboxes, calendar events, and shared files remain on disk unchanged. What does change is how the server is configured and managed, and because those records live inside the same database, a multi-step migration is required.
Before continuing, please read this document in full. Skipping steps will leave the server in an unrecoverable state and will require restoring from a backup.
If any step below raises questions, a dedicated discussion thread for the v0.16 upgrade is open at https://github.com/stalwartlabs/stalwart/discussions/3004. The earlier design discussion that led to these changes is also public at https://github.com/stalwartlabs/stalwart/discussions/2892 and describes the user-reported problems that motivated each breaking change.
Email is a critical service, and we understand that a forced maintenance window is disruptive: in some environments it is simply not an option. The breaking changes in v0.16 are not cosmetic. Stalwart has been under continuous development for close to five years; in that time the feature set and the user base have both grown well beyond what the original configuration and management layer was designed for. The gap between what users need and what the old architecture can cleanly support has widened to the point where a redesign was unavoidable: and the redesign itself unlocks a long list of frequently-requested features that were simply not implementable under the previous model. The storage layer is untouched by all of this: emails, calendars, contacts, files, and every other piece of user data stay exactly where they are. The migration is about configuration, not about data.
Operators who cannot accept downtime should wait. In the next two to three weeks we plan to release two tools that work together:
v0.15.x or below) deployment to a freshly-installed v0.16.x deployment one account at a time, while both servers are running.Together, these let operators migrate a live production deployment on an account-by-account basis with no scheduled maintenance window. When those tools are available, the instructions in this document will be superseded for most deployments. Everyone else can follow the manual steps below during a scheduled maintenance window.
The previous server used one or more TOML files, with some settings living on local disk and others living in the database. In v0.16 there is a single small config.json on disk that describes only the datastore (the database Stalwart uses to keep everything else). Every other configuration and management setting: domains, accounts, mail routing, DKIM signatures, storage backends, rate limits, spam rules, and so on: is now stored inside that datastore as a JMAP object. JMAP ("JSON Meta Application Protocol") is the JSON-based API Stalwart uses to expose its data; treating configuration as JMAP objects means the same API that serves email metadata also serves server configuration.
This change is driven by two real problems with the old model. First, in a clustered deployment every node had to carry its own copy of the configuration file and stay in lockstep with every other node. Divergence was easy to introduce and hard to debug, and it made distributed deployments unnecessarily fragile. Centralising everything in the database means configuration is consistent across the cluster by definition. Second, the split between "settings in the file" and "settings in the database" was a persistent source of user confusion: the same conceptual setting had to be documented in two places depending on where it happened to live, and administrators routinely edited the wrong one. A single unified model removes that entire category of mistake, and it gives management tooling (the WebUI and the CLI) a complete view of the system.
For Ansible, NixOS, Terraform, and other declarative tooling: the small config.json is still a plain file and can be managed with existing tooling exactly as before. Everything that used to live in TOML is now managed through stalwart-cli apply, which accepts a declarative plan file and idempotently reconciles the live server state to match it, creating what is missing, updating what has changed, and removing what the plan no longer declares. This is the same pattern used by CockroachDB (cluster settings via SQL/CLI), Consul (KV store), Elasticsearch (PUT /_cluster/settings), and HashiCorp Vault (CLI/API for policies and secrets); infrastructure-as-code tooling targets the API rather than a file. The workflow becomes: commit the declarative plan to version control, deploy config.json through existing tooling, and invoke stalwart-cli apply as an idempotent step in a playbook or activation script.
The /api/... endpoints from previous releases no longer exist. All management operations happen through JMAP objects reachable at /jmap. JMAP (RFC 8620) is a well-specified, transport-efficient protocol with first-class support for batch operations, push notifications, and fine-grained change tracking. Stalwart already speaks JMAP for email: extending it to administration gives operators and integrators a single consistent protocol for interacting with the entire server. In practice this means dozens of configuration changes can be applied in a single round-trip (the apply command uses this), any JMAP client library works against the management surface, and the same authentication flow covers both mail access and administration. Existing scripts and integrations that called the old REST endpoints must be updated; the new CLI is the straightforward replacement for most of them.
Every user and group principal now has a local part (the name) and an associated domain. In previous releases an account could be a bare string such as alice; in v0.16 it must be [email protected]. The migration script handles this automatically: accounts without a domain are assigned the default domain of the deployment (chosen by scanning existing principals for the most common domain), so no users are lost during conversion.
To avoid locking existing users out of their mail clients on the first login after the upgrade, v0.16 automatically appends the default domain when a client authenticates with a bare username. Administrators running an external directory (LDAP, SQL, etc.), however, do need to update their directory filters to query by full email address rather than by bare account name; the old filters will no longer match.
CalDAV, CardDAV, and WebDAV clients need one manual adjustment. These protocols use the account name as part of the URL path (for example /dav/cal/alice), and because the account name is now a full email address, that path changes. The @ character is reserved in URLs and must be encoded as %40, so the equivalent path in v0.16 becomes /dav/cal/alice%40example.com. Authentication itself still works (the server accepts the bare username and appends the default domain, as described above), but calendar, contact, and file sync will stop working until each client is reconfigured to point at the new path. It is a good idea to notify users before the upgrade so that they can update their calendar and contacts accounts in Apple Calendar, Thunderbird, DAVx⁵, and similar clients.
Two reasons drove this requirement. The first is support for multiple external directories simultaneously: when account names are bare strings there is no reliable way to tell which directory owns a given username, whereas email addresses are naturally namespaced by domain and make that mapping unambiguous. The second, and more consequential, reason is the PACC specification (draft-ietf-mailmaint-pacc): the IETF's replacement for the fragmented collection of autoconfig / autodiscover / SRV-record mechanisms that mail clients use today to discover server settings. PACC expects login names shaped like email addresses; when they are not, the server has to reveal whether a given account exists just to disambiguate the login, which is exactly the privacy leak the spec is designed to prevent. Aligning account names with email addresses is what lets Stalwart implement PACC correctly.
PACC also brings OAuth into the autodiscovery flow, and because the draft originates from Apple, a correct PACC implementation is the path to supporting Apple Mail clients with OIDC and MFA: a long-standing user request that only becomes possible once this groundwork is in place.
stalwart-cli) that also uses the JMAP API and can be used for day-to-day administration, scripted deployments, and infrastructure-as-code workflows. Full documentation is available at https://stalw.art/docs/management/cli.v0.16 before migratingBecause so much has changed, v0.16 will feel like a different product at first contact. Concepts have been renamed, some have been removed, and several new ones have been introduced.
It is strongly recommended that operators first install a fresh v0.16 instance in a Docker container or a throwaway virtual machine, log into the new WebUI, and spend time becoming familiar with how configuration works in the new release. This avoids the situation where a critical production upgrade is the first time an operator sees the new interface.
A second, equally important benefit: any settings created in the test deployment (directory integrations, SMTP listeners, spam rules, rate limits, TLS providers, etc.) can be exported using the snapshot command. The resulting JSON file is an apply plan that can be fed directly into the production instance after the migration completes. Time spent on a test deployment is not thrown away.
The migration is a multi-step, offline process. At a high level:
v0.15.x, it must first be upgraded to v0.15.x. The v0.16 migration tooling does not support anything older. Operators who cannot upgrade to v0.15.x now should wait for the zero-downtime proxy described above, which will perform a direct migration from older releases.v0.15.x server. It downloads the current settings and principals, converts them to the new format, and produces two files: config.json (the new on-disk datastore configuration) and export.json (a snapshot of everything else, in a format that the new CLI can replay).v0.15.x server is stopped and its database is backed up.v0.16 binary (or Docker image) is started in recovery mode. On first start it detects the old data, wipes the pieces that are no longer compatible, migrates the spam classifier model, and comes up listening on a single HTTP port (8080) exposing the management API.stalwart-cli apply replays export.json (and, optionally, any snapshots from the test deployment) against the recovery-mode server.config.json, and the server is restarted normally.The following sections describe each step in detail.
Note for clustered deployments. Before starting the migration, every node in the cluster must be stopped. If even one node is left running on
v0.15.xwhile another is being upgraded, it will write records in the old format and cause data corruption that can only be repaired by manually deleting the offending keys. This requirement is repeated in the binary and Docker sections below, but it applies globally.
This step is independent of how Stalwart is deployed and does not require stopping the server. The migration script talks to the running v0.15.x server over its management API and produces two JSON files on the machine where it is run. Running this step early is encouraged: it gives the operator a chance to review the generated files before touching the server, and to rerun the conversion with different options if needed.
Download the script from the Stalwart repository:
$ curl -fLO https://raw.githubusercontent.com/stalwartlabs/stalwart/refs/heads/main/resources/scripts/migrate_v016.py
Review the script before running it. It is a single self-contained Python file and makes no changes to the running server: it only reads configuration and principal data.
A virtual environment (venv) is a self-contained Python setup that keeps installed libraries out of the system-wide Python install. This avoids polluting the host Python and lets the script run on systems where pip installs are restricted.
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install requests urllib3
The first command creates the environment in a .venv/ directory. The second activates it (the shell prompt usually gains a (.venv) prefix). The third installs the only two libraries the script needs.
v0.15.x settingsThe script has two subcommands. The first, dump, connects to the running server and downloads its settings and principals into two files on disk:
(.venv) $ python migrate_v016.py dump \
--url https://mail.example.com \
--username admin \
--password adminPassword \
--settings settings.json \
--principals principals.json
Replace the URL and credentials with those of the v0.15.x server. The admin account must have permission to read all settings and principals. Output files default to settings.json and principals.json in the current directory. These files are plain JSON: opening them in a text editor to inspect their contents is encouraged.
The second subcommand, convert, reads the two dump files and produces the two files that the new server will consume:
(.venv) $ python migrate_v016.py convert \
--settings settings.json \
--principals principals.json \
--config config.json \
--output export.json
This produces:
config.json: the new on-disk datastore configuration. This is the file the v0.16 server will be pointed at on startup. It is small, because it describes only the datastore (data store, blob store, search store, in-memory store).export.json: a snapshot of every other piece of state the script could convert, in the format consumed by stalwart-cli apply. This file will be replayed against the v0.16 server once it is running in recovery mode.The conversion is intentionally conservative. Only the following settings are migrated, because the rest have changed enough that automatic mapping would do more harm than good:
rsa-sha1, which is obsolete and not supported in v0.16)Everything else: SMTP listeners, mail routing rules, rate limits, connection limits, spam filter settings, logging and telemetry configuration, authentication backends other than the ones listed above, session scripts, Sieve preludes, milter/MTA hook configuration, etc.: must be recreated on the new server.
This is the reason the test deployment recommended above is so useful: recreating the remaining settings on a test v0.16 instance, then using stalwart-cli snapshot to export them, turns what would otherwise be manual post-migration work into a second apply run. If the production deployment is close to the defaults, this is straightforward. If it has extensive customisation, plan for the time this takes.
When the v0.16 server starts for the first time, it will wipe the parts of the database that are no longer compatible with the new schema. No user mail is touched, but everything below is deleted unconditionally:
apply, the new directory entries produced from export.json recreate these records with the same identities.export.json replays what the script was able to convert; anything the script could not convert needs to be recreated manually (or via a snapshot from the test deployment).v0.16 exposes a task panel in the WebUI, and the equivalent tasks can be triggered manually from there.Because the wipe is irreversible, a full backup of the existing data must exist before the new server is started.
These store everything in a single directory on disk (typically /var/lib/stalwart/data or /opt/stalwart/data). A file-level copy while the server is stopped is sufficient:
$ sudo systemctl stop stalwart # or the equivalent for the service manager in use
$ sudo cp -a /var/lib/stalwart /var/lib/stalwart.v015-backup
Record the path of the backup somewhere safe. If the migration fails, restoring this directory and starting the old binary returns the system to its previous state.
The database holds many tables, but only a subset needs to be captured to be able to undo the migration. Each table is a single ASCII character that corresponds to an internal Stalwart subspace. The destructive part of the migration touches the following tables:
| Table | Purpose | Priority |
|---|---|---|
s | Settings | Critical: contains all server configuration |
d | Directory | Critical: users, groups, domains, tenants, mailing lists, OAuth clients |
r | Incoming reports (DMARC, TLS, ARF) | Recommended |
h | Outgoing reports | Recommended |
b | Legacy bitmap index | Recommended |
g | Legacy full-text-search index | Recommended |
j | Legacy blob-extra metadata | Recommended |
f | Pending task queue | Recommended |
u | Quotas (partially reset) | Recommended |
o | Telemetry spans (traces) | Optional: can be very large |
x | Telemetry metrics | Optional: can be very large |
w | Legacy telemetry/spam-sample index | Optional: can be very large |
The telemetry tables (o, x, w) can grow into tens of gigabytes on busy servers. Skipping them from the backup is reasonable unless there is a specific need to preserve historical metrics or traces.
For PostgreSQL, a per-table dump looks like this:
$ pg_dump -U stalwart -d stalwart \
-t s -t d -t r -t h -t b -t g -t j -t f -t u \
-f /var/backups/stalwart-v015-critical.sql
The equivalent with mysqldump:
$ mysqldump -u stalwart -p stalwart \
s d r h b g j f u \
> /var/backups/stalwart-v015-critical.sql
A full database dump (pg_dump / mysqldump without the -t flags, or pg_dumpall) is the safest option if disk space allows.
This step is the only one that requires downtime. The sequence has moving parts, and each moving part must complete before the next begins. Reading this entire section before starting is strongly encouraged.
Clustered deployments: stop every node before beginning. Leaving a single
v0.15.xnode running while the migration is in progress will corrupt the database.
The following instructions assume the standard FHS layout (/usr/local/bin/stalwart, /etc/stalwart/config.toml, /var/lib/stalwart). Operators using a custom prefix (for example /opt/stalwart) should substitute their paths accordingly.
1. Download the v0.16 binary. Grab the release matching the target platform from https://github.com/stalwartlabs/stalwart/releases/latest. Do not replace the running binary yet.
2. Stop the old service.
$ sudo systemctl stop stalwart
$ sudo service stalwart stop
Verify the process is gone with ps before continuing. In a cluster, repeat this on every node.
3. Back up the old binary and install the new one.
$ sudo mv /usr/local/bin/stalwart /usr/local/bin/stalwart.v015
$ sudo mv /path/to/downloaded/stalwart /usr/local/bin/stalwart
$ sudo chmod 0755 /usr/local/bin/stalwart
$ sudo chown root:root /usr/local/bin/stalwart
4. Install the new config.json. The file produced by the migration script in Step 1 goes where the old TOML configuration used to live:
$ sudo mv /path/to/config.json /etc/stalwart/config.json
$ sudo chown stalwart:stalwart /etc/stalwart/config.json
$ sudo chmod 0640 /etc/stalwart/config.json
The old config.toml can be kept as a reference but is no longer read by the server.
5. Start the new binary in recovery mode from the foreground. Running the initial migration under the service manager is discouraged: if something goes wrong, the output scrolls past in journalctl and the restart loop masks the cause. Instead, run it directly as the stalwart user so that stdout and stderr are visible in the current terminal:
$ sudo -u stalwart env \
STALWART_RECOVERY_MODE=1 \
STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword \
/usr/local/bin/stalwart --config=/etc/stalwart/config.json
STALWART_RECOVERY_MODE=1 tells the server to enter the one-shot migration path: wipe the incompatible subspaces listed above, migrate the spam classifier model, and then bring up only the management HTTP endpoint on port 8080. Mail ports stay closed. STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword provisions a temporary admin credential that the CLI can authenticate against: this is needed because the converted export.json does not grant admin rights to any user (that is deliberate; admin assignment is a deployment decision). Replace someTemporaryPassword with a strong value; this account exists only until a real admin is created.
The migration output will scroll past. When it finishes, the process stays in the foreground, listening on port 8080. Leave this terminal open.
6. Apply the exported snapshot. From a second terminal (on the same host or any machine that can reach the server on port 8080), install the new CLI (make sure to install v1.0.2 or later): instructions at https://stalw.art/docs/management/cli/overview: and run:
$ export STALWART_URL=http://127.0.0.1:8080
$ export STALWART_USER=admin
$ export STALWART_PASSWORD=someTemporaryPassword
$ stalwart-cli apply --file /path/to/export.json
A summary similar to the following should appear:
Plan: 0 destroy, 5 update, 6 create (…)
✓ created Tenant (…)
✓ created Domain (…)
✓ created Account (…)
…
Done: 0 destroyed, 5 updated, … created (0 failed)
If any operation fails, the CLI stops immediately and prints the error. Fix the root cause (usually a conflict with an object created in an earlier attempt) and rerun. apply is re-entrant with --continue-on-error when needed.
At this point, snapshots exported from the test deployment with stalwart-cli snapshot can also be applied, in order:
$ stalwart-cli apply --file /path/to/test-deployment-snapshot.json
7. Shut down recovery mode. Return to the terminal running the foreground server and press Ctrl+C. The process will exit cleanly.
8. Reconfigure the service manager. The systemd unit or init.d script still references the old TOML path. Update it to point at the new JSON file:
/etc/systemd/system/stalwart.service), locate the ExecStart= line and change the --config= argument:
ExecStart=/usr/local/bin/stalwart --config=/etc/stalwart/config.json
$ sudo systemctl daemon-reload
/etc/init.d/stalwart), update the DAEMON_ARGS line similarly.9. Decide how to handle the recovery admin. The recovery admin credential must be available the first time a real administrator logs in to create a proper admin account. Two options:
STALWART_RECOVERY_ADMIN in place until a real admin is created through the WebUI, then remove it. For systemd, set it via the environment file referenced by EnvironmentFile= in the service unit (the default Stalwart install creates /etc/stalwart/stalwart.env for exactly this purpose: uncomment the STALWART_RECOVERY_ADMIN line and set the value). For init.d, export the variable in /etc/default/stalwart or the distribution's equivalent. Do not set STALWART_RECOVERY_MODE=1: that is for the migration only and would put the server back into recovery mode at every restart.10. Start the service.
$ sudo systemctl start stalwart # or: sudo service stalwart start
Verify it comes up cleanly and is listening on its normal ports. The deployment is now on v0.16. For reference on how a fresh v0.16 Linux install is expected to look, see https://stalw.art/docs/install/platform/linux.
The new Docker image uses different mount points than the old one. Where the previous image mounted a single /opt/stalwart volume, the new image mounts two:
| Volume | Purpose |
|---|---|
/etc/stalwart | Configuration directory (contains config.json) |
/var/lib/stalwart | Persistent application data (RocksDB, local blobs, bootstrap registry) |
The Docker migration uses the same recovery-mode pattern as the binary case: stop the old container, run a throwaway container in recovery mode, apply the snapshot, stop the throwaway, then start the real container.
Clustered deployments: stop every container running
v0.15.xbefore starting the migration on any node.
1. Stop the old container.
$ docker stop stalwart
2. Prepare the new volumes. Two named volumes (or two host directories, if bind-mounting) are required:
$ docker volume create stalwart-etc
$ docker volume create stalwart-data
For deployments where the embedded database holds user mail (RocksDB / SQLite), the contents of the old /opt/stalwart/data directory must be placed in the new stalwart-data volume before starting the recovery-mode container. The simplest way is a helper container:
$ docker run --rm \
-v <OLD_STALWART_DIR>:/old \
-v stalwart-data:/new \
alpine sh -c 'cp -a /old/data/. /new/ && chown -R 2000:2000 /new'
Replace <OLD_STALWART_DIR> with the host path that the previous container had mounted at /opt/stalwart. The chown step is required because the new image runs as UID 2000. For deployments that use external databases (PostgreSQL, MySQL, FoundationDB, S3, Azure, Redis, NATS), skip the copy: the data already lives outside the container.
3. Install config.json in the new config volume.
$ docker run --rm \
-v /path/to/local/config.json:/src/config.json:ro \
-v stalwart-etc:/dst \
alpine sh -c 'cp /src/config.json /dst/config.json && chown 2000:2000 /dst/config.json'
Update embedded paths inside
config.jsonandexport.jsonfor the new mount points. The migration script writes the on-disk paths it found in the v0.15 deployment, which on the previous Docker image typically pointed at/opt/stalwart/data(and/opt/stalwart/data/blobsfor the filesystem BlobStore). The new image mounts persistent data at/var/lib/stalwartinstead, so any path referencing the old location must be rewritten before the recovery container is started; otherwise the container exits withPermission denied: /opt/stalwart/databecause UID2000cannot create that directory inside the container's filesystem.The migration script ships with a
--patch-pathsflag that handles the rewrite duringconvert:bash$ python migrate_v016.py convert \ --settings settings.json --principals principals.json \ --config config.json --output export.json \ --patch-paths /opt/stalwart=/var/lib/stalwart
--patch-paths SOURCE=DESTwalks both emitted files and rewrites any string value beginning with the source prefix. The flag may be supplied multiple times for deployments that mount data under several legacy paths. When the script detects/opt/stalwartin the source settings and the flag was not passed, it prints a notice with the exact command to rerun.For deployments that already produced
config.jsonandexport.jsonwithout the flag, the equivalent in-place edit is:bash$ sed -i.bak \ -e 's|/opt/stalwart/data/blobs|/var/lib/stalwart/blobs|g' \ -e 's|/opt/stalwart/data|/var/lib/stalwart|g' \ config.json export.json $ grep -n /opt/stalwart config.json export.json # verify cleanThe blob-path substitution runs first so the more general data-path rewrite does not double-rewrite it. The
.bakfiles left behind by-i.bakare the rollback if the substitution went wrong.Skip this paragraph entirely on deployments that use external databases (PostgreSQL, MySQL, FoundationDB) and external blob backends; those deployments have no on-disk paths to rewrite.
4. Start a temporary container in recovery mode. This container exists only for the duration of the migration:
$ docker run -d --name stalwart-recovery \
-e STALWART_RECOVERY_MODE=1 \
-e STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword \
-p 8080:8080 \
-v stalwart-etc:/etc/stalwart \
-v stalwart-data:/var/lib/stalwart \
stalwartlabs/stalwart:v0.16
Only port 8080 (management API) is published: mail ports stay closed in recovery mode. Watch the logs to confirm the migration completes successfully:
$ docker logs -f stalwart-recovery
Wait until the logs stop scrolling and settle on the message indicating the HTTP endpoint is listening.
5. Apply the exported snapshot. From the host (or any machine that can reach http://<docker-host>:8080):
$ export STALWART_URL=http://127.0.0.1:8080
$ export STALWART_USER=admin
$ export STALWART_PASSWORD=someTemporaryPassword
$ stalwart-cli apply --file /path/to/export.json
Follow with any snapshots captured from the test deployment:
$ stalwart-cli apply --file /path/to/test-deployment-snapshot.json
6. Stop and remove the temporary container.
$ docker stop stalwart-recovery
$ docker rm stalwart-recovery
7. Start the production container. Same image, without STALWART_RECOVERY_MODE, with all mail ports published:
$ docker run -d --name stalwart \
--restart unless-stopped \
-e STALWART_RECOVERY_ADMIN=admin:someTemporaryPassword \
-p 443:443 -p 8080:8080 \
-p 25:25 -p 587:587 -p 465:465 \
-p 143:143 -p 993:993 \
-p 110:110 -p 995:995 \
-p 4190:4190 \
-v stalwart-etc:/etc/stalwart \
-v stalwart-data:/var/lib/stalwart \
stalwartlabs/stalwart:v0.16
The STALWART_RECOVERY_ADMIN variable is retained deliberately so that a real administrator account can still be created through the WebUI after the first login. Once a permanent admin exists, restart the container without that environment variable to remove the back-door credential. If the test deployment snapshot applied in step 5 already includes an administrator account, the variable can be omitted from this step entirely.
For reference on the standard Docker deployment, see https://stalw.art/docs/install/platform/docker.
With the server running on v0.16, a few follow-up actions are required to complete the upgrade.
Open a browser and navigate to:
https://mail.example.org/admin
Replace mail.example.org with the server's hostname. Log in either with the recovery admin credential (if it is still active) or with the administrator account created via the test-deployment snapshot.
A few behavioural changes from v0.15.x are worth flagging before the first sign-in:
v0.16 publishes use https://<defaultHostname>/... exclusively in normal mode. Loading the WebUI by IP address, by container name, or over plain HTTP (for example http://192.168.1.10:8080/admin) will appear to load the sign-in page but will fail at the OAuth callback. Use the same hostname that was entered in Step 1 of the wizard, or that already lives on defaultHostname from the migrated settings.http://...:8080 is no longer the right URL for day-to-day administration. Port 8080 carries the recovery / bootstrap HTTP listener and is intended for the migration window; once the server is running normally it stops being a valid sign-in entry point.443 (for example a reverse proxy on :8443, or a Docker host port mapping that diverges from the container's 443), set the STALWART_HTTPS_PORT environment variable to that port and restart the server. Without it, the discovery documents will publish https://<host>/... (port 443 implied) and clients will be sent to a port the proxy is not listening on.587 submission, port 143 IMAP) are no longer added by default. This is required for compliance with the PACC autoconfig draft, which only advertises implicit-TLS ports. Mail clients that were configured to connect over 587 STARTTLS will silently stop working until either the listener is recreated through the WebUI / CLI or the clients are pointed at the implicit-TLS ports (465 for submission, 993 for IMAPS).If the deployment sits behind a reverse proxy (NGINX, Traefik, Caddy, HAProxy, or similar), this is the part of the migration where proxy-related issues most often surface. The migrated defaultHostname, the proxy's public hostname, the proxy's listening port, and the proxy's TLS configuration all have to line up before the first sign-in completes; if any of them is off, the OAuth flow fails partway through with errors that are hard to relate back to the proxy.
The most reliable way through this step is to bypass the proxy temporarily for the duration of the recovery-mode apply, the first sign-in, and the creation of a permanent administrator. Concretely:
stalwart-cli apply and creating the permanent admin, point the CLI and the browser at Stalwart directly: http://<stalwart-host>:8080 for the recovery-mode CLI session, then https://<stalwart-host>/admin (accepting any self-signed certificate warning) for the first WebUI sign-in.A full description of how v0.16 composes the published URLs, how the proxy can talk to Stalwart on either HTTP or HTTPS, and where to set STALWART_HTTPS_PORT for non-standard public ports lives at https://stalw.art/docs/server/reverse-proxy/overview.
Disk quotas were reset to zero during the wipe and need to be rebuilt from the actual mailbox contents. Navigate to the Tasks section of the admin panel and trigger the "Recalculate disk quotas" task. This spawns one subtask per user account, each of which scans that user's storage and updates the counter. On large deployments this may take a while to complete: progress is visible in the Tasks panel.
Only applicable when per-tenant disk quotas are in use. After the per-account recalculation has finished for every user, trigger a second task from the Tasks panel: "Recalculate tenant quotas". This rolls the per-account totals up into the tenant-level counters.
If the migration was performed without a snapshot from a test deployment, the only administrative credential at this point is the recovery admin defined by STALWART_RECOVERY_ADMIN. This credential is a back door: as long as the environment variable is set, the username and password it specifies can log in regardless of directory state. Create a real administrator account through Management → Accounts, verify the new account can log in, and then remove STALWART_RECOVERY_ADMIN from:
/etc/stalwart/stalwart.env) and restart the service, or-e flag (redeploy the container without it).The migration script converts directory, domain, storage, DKIM, and certificate state. Everything else: SMTP listeners, mail routing, spam rules, rate limits, retention policies, ACME, authentication backends other than those listed above: must be reviewed in the WebUI and either recreated by hand, applied from a test-deployment snapshot, or validated against the defaults that v0.16 ships with.
config.jsonConfirm the file is valid JSON (python -m json.tool config.json or jq . config.json) and that the datastore described in it is reachable with the provided credentials. The daemon's logs will name the offending field when a field is missing or malformed.
stalwart-cli apply fails partway throughMost failures come from trying to create an object whose parent does not exist yet (for example, a DkimSignature referencing a Domain that is missing from the plan). The error message names the object and the missing reference. Either edit the plan to include the missing parent, or split the apply into two runs using the individual snapshot files produced by the script and the test deployment.
applyapply runs operations in plan order and stops on the first error. When a create fails halfway through, every prior create in the same run has already been committed to the database. Re-running the same plan now fails with primaryKeyViolation (the objects exist) or invalidForeignKey (a parent that did not get created the first time is still missing).
The cleanest recovery is to wipe the registry and apply again. While the server is still in recovery mode, list and delete the migrated objects:
$ stalwart-cli query Domain --json | jq -r '.[].id' \
| stalwart-cli delete Domain --stdin
$ stalwart-cli query Account --json | jq -r '.[].id' \
| stalwart-cli delete Account --stdin
$ stalwart-cli query Tenant --json | jq -r '.[].id' \
| stalwart-cli delete Tenant --stdin
$ stalwart-cli query DkimSignature --json | jq -r '.[].id' \
| stalwart-cli delete DkimSignature --stdin
$ stalwart-cli query Certificate --json | jq -r '.[].id' \
| stalwart-cli delete Certificate --stdin
Then fix the underlying cause in export.json (most often a domain that fails the v0.16 hostname check, an account whose local-part contains @, or a stale /opt/stalwart path embedded by the migration script) and rerun:
$ stalwart-cli apply --file export.json
If the failure was caused by data that the migration script itself produced incorrectly, also rerun the script with the latest version from main before applying. Fixes during the v0.16.0 / v0.16.1 window addressed several edge cases (group names containing @, ACME base64 padding, single-URL Redis stores, paths embedded in custom storage backends).
For deployments where individual objects are easier to identify than to wipe wholesale, use stalwart-cli query <type> to list ids and stalwart-cli delete <type> --ids <id> to remove a specific one.
When the WebUI is unreachable for any reason (TLS not yet in place, reverse proxy misconfigured, OAuth callback failing), the CLI is the supported escape hatch for promoting the first real administrator. Authenticate as the recovery admin and run:
$ export STALWART_URL='http://127.0.0.1:8080'
$ export STALWART_USER='admin'
$ export STALWART_PASSWORD='someTemporaryPassword'
$ stalwart-cli query Domain --fields id,name
$ stalwart-cli create account/user \
--field name=admin \
--field domainId=<domain-id>
$ stalwart-cli query Account --where name=admin --fields id
$ stalwart-cli update Account <account-id> \
--field 'credentials={"0":{"@type":"Password","secret":"<NEW-PASSWORD>"}}'
$ stalwart-cli update Account <account-id> \
--field 'roles={"@type":"Admin"}'
Once the new account can sign in to the WebUI, remove STALWART_RECOVERY_ADMIN from the service environment and restart the service.
primaryKeyViolation on a rerun of apply: see Recovering from a partial apply above.Domain: create failed for create-N: invalidPatch | Invalid domain name: the domain in export.json does not pass the v0.16 hostname check (typically a missing or non-public TLD). Either correct the domain in v0.15 before redumping, or hand-edit the offending block in export.json./admin redirects to http://<random>:8080/: fixed in v0.16.x; upgrade to the latest patch release.Data corruption detected after migrationThis error means one node in a cluster was left running on v0.15.x while another was being migrated, and the old node wrote records in the obsolete format into the shared database. Stop every node in the cluster, ensure every binary is on v0.16, and restart. If corruption persists, the logs name the offending keys and they can be removed with the stalwart-cli delete command.
/admin (or /account) returns 404 Not FoundThe WebUI is delivered as a downloadable Application bundle that the server fetches from https://github.com/stalwartlabs/webui/releases/latest/ on first start, and then refreshes on a schedule. When the very first download fails, no bundle has been unpacked locally and every request to a WebUI mount path returns 404 Not Found. This is the most common cause of "the server is running, port 8080 answers, but /admin returns 404" reports during the migration.
The fix is to make sure outbound HTTPS from the Stalwart host can reach GitHub's release storage (github.com and objects.githubusercontent.com). On a host that genuinely cannot reach the public internet, stage the WebUI bundle on an internal HTTPS server and update the resourceUrl field on the WebUI's Application record to point at the internal location. After the first successful download, subsequent failures are non-fatal: the previously installed bundle stays in service and /admin keeps working until the next successful refresh. The full description, including the precise hosts involved, is at https://stalw.art/docs/management/webui/overview#outbound-network-requirement.
If the migration cannot be completed within the available maintenance window, the database backup captured in Step 2 can be restored and the old binary (preserved as /usr/local/bin/stalwart.v015 in the example) started again. The v0.16 binary will refuse to start a second time against a database that has already been migrated, so restoring the pre-migration backup is the only path back to v0.15.x.
If any part of this migration is unclear, or if something does not behave as documented, please post in the dedicated upgrade discussion at https://github.com/stalwartlabs/stalwart/discussions/3004. Include:
stalwart --version)We would rather answer a question than watch a deployment break. There is no such thing as an obvious question for a migration of this size.