docs/Features/Admin-Panel/Problems/Recovery.md
This page uses the shared Table Page design. The layout, search, pagination, column spec and per-page data loading are defined there and are not repeated here.
Status: Implemented · Owner: xet7 · Related (#6492):
models/lib/recoveryPlan.js, models/lib/recoveryEventsJsonl.js,
models/recoveryEvents.js, server/recovery.js,
server/publications/recoveryReport.js, client/components/settings/adminProblems.*,
snap-src/bin/ferretdb-control, releases/ferretdb/*, and the FerretDB fork
(internal/backends/sqlite/metadata/pool/opendb.go,
internal/handler/handler.go).
When WeKan stores its data in FerretDB v1 (SQLite), the text data lives in
wekan.sqlite. Attachments and avatars live on the filesystem, not in the
database. This subsystem keeps that text data safe: it prevents the database from
bloating, detects corruption, keeps a ready-to-use backup, can restore or re-migrate
automatically, and shows the whole remediation history in Admin Panel → Problems →
Recovery.
PRAGMA quick_check; anything but ok is logged prominently.
Corruption cannot be repaired in place, so this only reports it — recovery is done
by WeKan (restore/re-migrate).VACUUMs a file whose free pages
dominate it (≥ ~1 MiB and ≥ ¼ free), rebuilding it compactly. Content-preserving.local.oplog.rs is capped small (16 MiB) so the transient
local.sqlite cannot bloat and drive CPU high.FERRETDB_SQLITE_AUTO_REPAIR=false.Before FerretDB opens the files, every launch path (snap ferretdb-control, the
bundled release start-wekan.sh, the Docker wekan-entrypoint.sh):
wekan.sqlite* into a backup/ subfolder of the
same data dir, keeping the previous generation under backup/prev. It only ever
copies the live database — never moves or deletes it — and never copies
attachments/avatars. Toggle: WEKAN_SQLITE_BACKUP=false.WEKAN_FORCE_RESTORE env or
a RESTORE_REQUESTED marker file containing backup, prev or remigrate — it
copies the chosen known-good backup into the live database (dropping the stale
WAL side-files so the copy is used cleanly) and records the action. The backup copies
are never deleted; the main wekan.sqlite is only ever overwritten, never removed.
remigrate requests a re-migration of the text data from MongoDB (handled by
migration-control); attachments/avatars on the filesystem are untouched.local.sqlite) so a bloated/corrupt OpLog cannot
persist across a restart. Toggle: WEKAN_FERRETDB_RESET_OPLOG=false.models/lib/recoveryPlan.js (decideRecovery) is the pure, unit-tested brain: given
the integrity result and what backups / MongoDB are available, it chooses the
least-invasive recovery — latest good backup → previous backup → re-migrate → (else)
manual. It never chooses anything destructive unless the database is known corrupt.
Recovery must never look like a broken site. Two layers cover the whole window, on every FerretDB v1 platform (snap, bundled release, Docker):
RECOVERY_IN_PROGRESS marker
when they restore/re-migrate; the server keeps the spinner up until it has
health-probed the database (a real read), then clears the marker and hides the
spinner — or, if it still cannot read, keeps the spinner and records that manual
recovery is required. Admins can also toggle it for a server-initiated re-migration.
Because the server owns clearing the marker, the spinner behaves identically on all
platforms.wekan-maintenance-page.mjs (with a recovery wording), and the release/Docker paths
serve the portable releases/ferretdb/recovery-bridge.mjs. The bridge is
time-bounded (WEKAN_RECOVERY_BRIDGE_SECONDS, default 20s) and is only a visual
bridge — it can never block WeKan from starting, and it hands straight over to the
in-app spinner above. It is skipped cleanly if its page file or the marker is absent.The startup scripts append one JSON line per action to recovery-events.jsonl in the
data dir. On startup the WeKan server imports the new lines into the recoveryEvents
collection (server/recovery.js), and the Recovery report
(server/publications/recoveryReport.js, admin-only) lists them newest-first with the
same search + pagination as the other admin reports. Admins can also record a manual
event with the recordRecoveryEvent method.
Event types include backup-created, corruption-detected, restore-backup,
restore-prev, remigrate, bloat-repaired, integrity-ok, manual-required, each
with a severity (info / warning / error).
Recovery also keeps the audit trail for irreversible board deletion. Changing
Admin Panel → Problems → Delete records permanent-delete-setting-changed with
the Global Admin username, user ID and whether the setting was enabled or
disabled. Every successfully purged archived board records
board-permanently-deleted with that actor, the board ID and its title. No-op,
unauthorized and failed operations are not logged as successful actions.
Attempted setting changes and board purges are always recorded with a Boolean
done, the user ID and username when known, and the proxy-aware IPv4 or IPv6
address resolved through HTTP_FORWARDED_COUNT. Board attempts also keep
bounded boardIds and boardTitles arrays; an ID that does not resolve uses an
unknown-title marker rather than disappearing from the audit.
The Recovery table begins with Done. true renders a green check, false a
red warning triangle, and a successful operation that physically deleted data
adds a yellow trashcan. A batch that partly completes therefore shows yellow
successful rows for the boards already removed and a red failed-attempt row for
the whole requested batch.
The filter dropdown above the table selects All, Done, Failed or
Deleted events. Filtering is applied by the server before counting and
pagination, and combines with the search term. Older events written before the
done field existed count as Done because those event types represented completed
recovery actions.
Below the existing database-recovery description, the pane has a second paragraph explaining that Recovery also records permanent-delete setting changes and every successful, failed or unauthorized purge attempt, together with its Done status, actor, trusted address and attempted board IDs and titles.
Admin Panel → Problems → Delete repeats that same paragraph immediately below its existing permanent-delete setting description. Both panes use one shared source sentence, so the explanation at the control and the explanation at its audit trail cannot drift apart.
To force a restore on the next start, set WEKAN_FORCE_RESTORE=backup (or prev, or
remigrate) in the environment, or create a RESTORE_REQUESTED file containing that
word in the SQLite data dir, then restart WeKan. The action is recorded in the
Recovery report.
tests/recoveryPlan.test.cjs — the decision logic (positive + negatives: never act
on a healthy/unknown database; never restore a known-bad backup; nothing to restore
→ manual, non-destructive).tests/recoveryEventsJsonl.test.cjs — the JSONL parser (skips junk, normalizes
severity, bounds line size; never throws).tests/recoveryReportQuery.test.cjs — the report search and outcome selectors
(including combined filters, legacy Done rows and escaped regex metacharacters).tests/recoveryReportWiring.test.cjs — the Recovery report is wired and the
publication/count/method are admin-gated.tests/permanentDeleteRecoveryAudit.test.cjs — permanent-delete setting
changes and board-purge attempts record status, actor, address and affected
boards; it also pins the Done/deletion icons and proves failed operations
cannot produce success records.tests/ferretdbTextDataBackup.test.cjs — the backup/restore scripts (critical
negatives: never delete the live text data or a backup copy, never copy
attachments/avatars).opendb_test.go (corruption check + bloat VACUUM) and
msg_replset_test.go (OpLog cap).