Job & Scheduler — Architecture Overview

Two independent main-process lifecycle services:

Layering rule: SchedulerService is unaware of Jobs. JobManager uses SchedulerService to arm schedules. Business modules pick one based on need:

• Need cron + persistent observability + retry → register a JobHandler + use jobManager.registerJobSchedule()
• Need cron only (heartbeat-style, no persistence) → schedulerService.registerSchedule() directly
• Need recurring service-internal GC / self-check → BaseService.registerInterval (project convention, not SchedulerService)

DB-driven dispatch

jobTable is the single source of truth. Memory state (handlers Map, queues Map, AbortControllers) is a derived view that JobManager rebuilds on every startup.

Each queue has a DispatchQueue instance holding { name, concurrency, mutex }. The dispatch loop (JobManager.dispatch):

1. Acquire Layer 1 per-queue mutex first
2. Enter Layer 0 — the synchronous BEGIN IMMEDIATE write transaction (withWriteTx) — second
3. Inside that one DB transaction:
   • Count queue-active jobs → check queue.concurrency
   • Count globally-running jobs → check globalMaxConcurrency
   • SELECT next pending → UPDATE to running (claim)
4. The Layer 0 transaction commits, then release the Layer 1 per-queue mutex
5. Spawn handler.execute outside the lock
6. Queue a microtask to dispatch the same queue again (fill next slot)

Spawning happens outside the lock — the handler executes for seconds/minutes while new dispatches proceed.

Acquisition order is fixed (Layer 1 mutex, then the Layer 0 write transaction). Layer 0 holds no async lock — it is a synchronous transaction — so Layer 1 is the only mutex in the dispatch path and the two layers cannot deadlock against each other.

Six-state state machine

                  ┌── retry backoff (delayed) ──┐
                  ▼                              │
   enqueue → pending → running → completed       │
                  │       │                      │
                  │       └→ failed ─────────────┘ (if retryable && attempt < max)
                  │       └→ cancelled (terminal)
                  └→ delayed → (scheduledAt ≤ now) → pending

Terminal states (completed / failed / cancelled) are never reopened. Retry re-enters delayed then transitions back to pending when scheduledAt elapses.

Startup Recovery

Startup recovery is JobManager's deferred sweep that reconciles the DB-driven state machine with the freshly booted process. It is service-level business work, not a bootstrap initialization side effect — see onAllReady business work pattern for the framework-level rationale.

Sequence

1. JobManager.onAllReady() schedules a setTimeout with a 60-second "quiet window" and returns synchronously. LifecycleManager.allReady() is fire-and-forget; bootstrap is not blocked.
2. After 60 s, the timer callback assigns the recovery flow promise to this._recoveryDone (only if shutdown has not been requested) and the flow starts running.
3. The flow runs four IO steps in order:
   1. runStartupRecovery(handlers, isJobInFlight) — resets non-terminal rows per handler recovery strategy (abandon / retry / singleton); cancelRequested=true overrides every strategy. Rows the current process is already executing (reported via isJobInFlight, backed by JobManager.inFlightExecuted) are excluded before any strategy, so a job enqueued during the quiet window and still running when the sweep fires is never reset or re-dispatched (#16291).
   2. Resurrect queues — walks distinct (queue, type) pairs over non-terminal rows and ensures a DispatchQueue exists for each. Without this step dispatchAll would iterate an empty queues map and pending rows would wait until the next enqueue.
   3. Catch-up THEN arm — calls detectAndDispatchOverdue(schedules) before armSchedule(schedule) for every enabled schedule. The order is load-bearing: if we armed first, a cron with protect: true could fire its natural calendar concurrently with a catch-up enqueue (protect only blocks overlapping callbacks, not external callers). Sequencing catch-up first guarantees the make-up enqueue lands before croner's first natural fire.
   4. dispatchAll() kicks every per-queue pump so pending rows reset by step 1 start running immediately rather than waiting on the next enqueue.

The 60 s quiet window

The delay (JOB_MANAGER_STARTUP_DELAY_MS = 60_000, hardcoded) gives cold-start IO — DB warm-up, window paints, client bootstrap — time to settle before scheduled work piles on. Tests bypass it via vi.useFakeTimers + advanceTimersByTimeAsync(60_000), then await _recoveryDone.

Shutdown safety — three layers

The flow can be interrupted at any point by onStop. Three mechanisms cooperate:

The join is bounded from the outside: shutdown caps every service at SERVICE_STOP_TIMEOUT_MS (5s). If the in-flight recovery step outlasts that, the framework stops waiting on JobManager and moves on — no SERVICE_STOPPED, and the pass is recorded as unclean. The teardown after the join still runs whenever recovery does finish, just too late to count: it now overlaps the shutdown of services further down the order. Whether onDestroy still runs on top of it depends on that late finish beating destroyAll() to JobManager — a race, not a guarantee (see Lifecycle Overview — teardown time contract). Acceptable either way because none of it is load-bearing — enqueue writes to the DB immediately and startup recovery repairs whatever was left mid-flight.

Handler registration timing

Handlers must be registered in the owning service's onInit (see handler-authoring.md — Registration Timing). By the time the 60-second timer fires every consumer has finished onInit / onReady, so runStartupRecovery sees the full handler set. Registering a handler from another service's onAllReady is unsafe: that hook runs in parallel with JobManager's, and any non-terminal job for an unregistered type during recovery gets treated as an orphan and cancelled.

Pause and drain (write quiesce)

Serves backup restore (#16850): after the restore snapshot is taken at time T, any JobManager write to the old live DB fails the fingerprint re-check and wastes the whole restore attempt. pause() stops autonomous writes to avoid that waste; the fingerprint gate stays the correctness backstop. The restore orchestrator must NOT run as a JobManager job — a handler that pauses and drains its own manager deadlocks until timeout.

const hold = jobManager.pause('backup restore')
const verdict = await jobManager.drainInFlight({ timeoutMs: 15_000 })
const clean = verdict.stragglerIds.length === 0 && !verdict.startupRecoveryPending
if (!clean) {
  hold.dispose() // abort path ONLY — give the manager back its autonomy
  return abortRestoreAttempt()
}
await createSnapshot()
// Happy path: NEVER dispose. The release pass writes to the old live DB
// (promotion, markFired, catch-up enqueues) — post-snapshot that fails the
// fingerprint re-check and voids the attempt. The hold stands until the
// process relaunches into the restored DB (a lost hold fails closed).

Blocked while paused (autonomous writes): dispatch claims (entry check + post-mutex re-check), schedule fire callbacks (crons are additionally paused at the croner layer so limit quotas survive the window), GC / delayed-promotion ticks, delayed/retry promotion fires, and new startup-recovery steps — a started step (one schedule's onMissed + catch-up enqueue, atomic) runs to completion and is awaited by drain.

Allowed while paused (request-driven): enqueue / enqueueTx (rows land at rest and the snapshot captures them), cancel / cancelMany, schedule mutations, and triggerJobScheduleNow* — forced onto its direct-enqueue fallback (row lands pending + markFired; true still means "row persisted").

Missed cron fires are skipped, not caught up (croner semantics). A suppressed once fire is re-armed on release from the recorded id set — exactly once; never rebuild by scanning "enabled ∧ missing scheduler entry", which also matches historical completed one-shots.

Why DB-driven and not in-memory queue?

We considered BullMQ / bee-queue / better-queue / agenda / graphile-worker / bree etc. and selected this design because:

• All persistence already in SQLite (no Redis / MongoDB / PostgreSQL dependency)
• Restart recovery is automatic — memory replays from DB
• Race safety needs only one mutex pair (Layer 0 + Layer 1) around count → claim
• No double-source-of-truth bookkeeping (PQueue + DB) and its sync discipline

Throughput: ~200 dispatch/s at single-process better-sqlite3 throughput, well above Cherry Studio's largest scenario (1000+ knowledge bases, each with concurrency=5, never exceeds globalMaxConcurrency=50 simultaneous running jobs).

Strongly-typed JobRegistry

Business modules use TypeScript declaration merging to register type → payload mapping:

declare module '@main/core/job/jobRegistry' {
  interface JobRegistry {
    'agent.task': AgentTaskPayload
    'knowledge.index-leaf': IndexLeafPayload
  }
}

After this declaration:

• jobManager.enqueue('agent.task', payload) is compile-time type-checked
• Renaming a type surfaces every call site via the TypeScript error pipeline
• Wrong payload shape is a compile error

Transactional enqueue (enqueueTx)

enqueue persists the row on the bare connection — fine when the enqueue is the only write. When a business-state flip and the job INSERT must commit atomically (e.g. mark items deleting and enqueue the purge job), use the transactional variant inside a DbService.withWriteTx callback:

application.get('DbService').withWriteTx((tx) => {
  itemService.setStatusTx(tx, ids, 'deleting') // business write
  return jobManager.enqueueTx(tx, 'my.purge', { ids }) // job INSERT, same tx
})

Post-commit side effects (state publish, dispatch / delayed arming) are deferred one microtask past the synchronous transaction. On rollback the row never existed: the returned handle's finished never resolves, and an idempotency-key unique-index collision aborts the whole caller transaction. See the enqueueTx JSDoc for the full contract.

Transactional schedule mutation (registerJobScheduleTx / updateJobScheduleTx + syncJobScheduleTimerById)

When a schedule row and a related business write must commit atomically, compose the transactional primitives inside a DbService.withWriteTx callback, then sync the timer after the transaction returns:

const { id } = application.get('DbService').withWriteTx((tx) => {
  const created = jobManager.registerJobScheduleTx(tx, { type: 'agent.task', ... }) // schedule row
  agentChannelService.replaceTaskSubscriptionsTx(tx, created.id, channelIds) // business write, same tx
  return created
})
jobManager.syncJobScheduleTimerById(id) // post-commit timer sync (create: always; update: when the patch carried trigger/enabled)

The *Tx primitives validate up front (handler, name, trigger semantics → JOB_SCHEDULE_TRIGGER_INVALID) and never touch the timer, so a rollback has zero timer side effects. Timer sync is the caller's explicit post-commit step — enqueueTx's post-commit re-read cannot be reused here because its rollback test ("row absent") only holds for INSERT; an UPDATE rollback would read as committed and re-arm, resetting the interval phase. See the registerJobScheduleTx JSDoc for the full contract.

Renderer-side consumers

The renderer never enqueues, cancels, or otherwise mutates jobs through the DataApi. It only observes job state read-only:

• useJob(jobId) → current JobSnapshot (status / counters / error / ...). Source: shared cache jobs.state.${id} with GET /jobs/:id as a cold-start fallback.
• useJobProgress(jobId) → fine-grained progress. Source: shared cache jobs.progress.${id} only.

Triggering a job is owned by the relevant business module in main:

1. The business service decides the semantics — which job type, what payload, queue, idempotency key, max attempts, timeout.
2. It calls application.get('JobManager').enqueue(...) directly.
3. If the renderer needs to initiate the work, the business module exposes a dedicated IPC route (e.g. the knowledge.add_items IpcApi route); the route handler internally calls JobManager.enqueue(...).

Schedule mutations (CRUD / pause / resume / run-now) follow the same pattern: renderer → dedicated IpcApi route (e.g. ai.agent.task.* → AgentJobsService) → JobManager schedule APIs; schedule reads stay on the GET-only DataApi.

This keeps JobRegistry's compile-time JobPayloadOf<K> type safety intact and prevents the renderer from depending on JobManager infrastructure details (queue names, retry policies, idempotency keys).

See also

• concurrency-and-locks.md — The full four-layer lock model
• handler-authoring.md — How to write a handler
• migration-checklist.md — Migrating existing services