Back to Langfuse

Blob Storage Integration — Sync Status State Machine

web/src/features/blobstorage-integration/README.md

3.197.15.8 KB
Original Source

Blob Storage Integration — Sync Status State Machine

The blob storage exporter's UI status is derived from five DB fields on BlobStorageIntegration. No explicit status column exists — the status is computed at read time by deriveSyncStatus.ts.

DB fields (the state vector)

FieldTypeWritten by
enabledbooleanWeb (save)
lastErrorstring | nullWorker (catch / success)
lastSyncAtDate | nullWorker (success)
nextSyncAtDate | nullWorker (success / empty-window), Web (save)
runStartedAtDate | nullWorker (start / end), Web (save clears it)

Derived states (precedence top-to-bottom)

disabled    ← enabled = false
error       ← lastError != null
running     ← runStartedAt != null AND age < 2h
queued      ← nextSyncAt <= now
idle        ← lastSyncAt = null (never exported)
up_to_date  ← fallthrough

Transition diagram

                      ┌──────────────────────────────────────────┐
                      │           User saves (web)               │
                      │  runStartedAt = null                     │
                      │  if errored+enabled: nextSyncAt = now    │
                      │  if mode changed: lastSyncAt = null,     │
                      │                   nextSyncAt = now       │
                      └────────────────┬─────────────────────────┘
                                       │
    ┌──────────┐                       ▼
    │ disabled │◄──── enabled = false (any state)
    └──────────┘
         │ user enables
         ▼
    ┌──────────┐   nextSyncAt = null, lastSyncAt = null
    │   idle   │◄──── freshly created, never synced
    └──────────┘
         │ save sets nextSyncAt = now (on mode change or save while errored)
         ▼
    ┌──────────┐   nextSyncAt <= now
    │  queued  │◄──── scheduler finds row (lastSyncAt=null OR nextSyncAt<=now)
    └──────────┘      and enqueues a BullMQ job (no DB write)
         │
         │ worker picks up job, sets runStartedAt = new Date()
         ▼
    ┌──────────┐
    │ running  │   runStartedAt set, age < 2h
    └──────────┘
        ╱    ╲
  success    failure
      ╱        ╲
     ▼          ▼
┌───────────┐  ┌───────┐
│ up_to_date│  │ error │
└───────────┘  └───────┘

Transition detail

FromTriggerWritesTo
anyUser saves with enabled=falserunStartedAt=nulldisabled
anyUser saves with enabled=truerunStartedAt=null; nextSyncAt=now if errored or mode changedidle, queued, up_to_date, or stays error (lastError is not cleared by save)
disabledUser saves enabled=true(as above)idle, queued, up_to_date, or stays error
idleScheduler finds lastSyncAt=nullEnqueues BullMQ job (no DB write)stays idle
queuedScheduler finds nextSyncAt<=nowEnqueues BullMQ job (no DB write)stays queued
idle/queuedWorker starts jobrunStartedAt=nowrunning
runningWorker: integration disabledrunStartedAt=nulldisabled
runningWorker: empty time windowrunStartedAt=null, nextSyncAt=now+frequency, lastError=nullup_to_date (or idle if never synced)
runningWorker: export succeeds, caught uplastSyncAt=max, nextSyncAt=max+freq, lastError=null, runStartedAt=nullup_to_date
runningWorker: export succeeds, not caught uplastSyncAt=max, nextSyncAt=now, lastError=null, runStartedAt=null + re-enqueues jobqueued (immediately)
runningWorker: export failslastError=msg, lastErrorAt=now, runStartedAt=nullerror
errorUser saves (enabled, same mode)runStartedAt=null, nextSyncAt=nowstays error (lastError preserved; scheduler re-enqueues via nextSyncAt)
errorUser clicks Run NowEnqueues manual job (no DB write)stays error until worker clears lastError (success or empty-window)
runningStale runStartedAt > 2h(no write — derived only)falls through to queued, idle, or up_to_date
up_to_dateClock passes nextSyncAt(no write — derived only)queued

Safety valve

If a worker crashes without clearing runStartedAt, the 2h TTL in deriveSyncStatus lets the status fall through to whatever the underlying fields indicate (typically queued, since the scheduler will re-enqueue on the next tick). No explicit cleanup is needed.

Run Now (manual trigger)

Run Now only enqueues a BullMQ job with a unique manual- jobId. It does not write any DB state. The worker then follows the normal running → success/failure path. The UI shows the current status (error/up_to_date/queued) until the worker picks up the job and sets runStartedAt.

Key files

FileRole
deriveSyncStatus.tsDerives display status from DB fields
types.tsBlobStorageSyncStatus type
service.tsUpsert logic (web save path)
blobstorage-integration-router.tstRPC router (save, runNow)
worker/src/features/blobstorage/handleBlobStorageIntegrationProjectJob.tsWorker job handler
worker/src/features/blobstorage/handleBlobStorageIntegrationSchedule.tsScheduler (enqueues due jobs)