Railway Seed Consolidation Runbook

Date: 2026-04-10 PR: #2891 Current services: 100 (at Railway limit) Target services: 65 (~35 slots freed)

Single source of truth for Railway-deployed scripts: scripts/railway-services.json. When adding a new Railway service (nixpacks or Dockerfile), add an entry to the registry before merging. Both tests/scripts-railway-nixpacks-no-escape-import.test.mts and tests/dockerfile-digest-notifications-imports.test.mjs derive their entry lists from the registry, and tests/railway-services-registry-coverage.test.mts fails if a Dockerfile.* CMD, runbook "Start command:" entry, or standalone service row references a script the registry doesn't know about. The scripts-root guard also conservatively scans unregistered legacy seeders.

Prerequisites

1. Merge PR #2891 to main
2. Verify the bundle scripts are in the deployed branch
3. Have Railway dashboard access and gh CLI authenticated

Deployment safety guardrails

Watch paths are a live contract, and an accurate one

Railway stores watch paths in each service's environment configuration, not in the repository. The repo-side contract is scripts/railway-services.json: every registry-managed production seeder pins its cron and the exact repository-relative files in its runtime dependency closure. tests/railway-watch-path-audit.test.mjs walks each entry point's imports and fails when that closure grows without a matching registry update. This keeps the declared closure complete without making unrelated changes under scripts/** or shared/** rebuild every seeder.

The filter matches accurately — read meta.skippedReason before concluding otherwise. Railway records a refused push as a SKIPPED deployment carrying meta.commitHash and a reason, and the two reasons it uses mean opposite things. Re-measured 2026-08-04 across all 77 repository-backed services and 600 commits of main, of 7,391 No changes to watched files skips, 7,331 were plainly correct, 57 matched a pattern pointing outside the service's build context (a rootDirectory: scripts container cannot see repository-root shared/, so the shared/** several of them carry is unreachable by construction), and 3 were a registry closure not yet applied to Railway. None were unexplained.

The lag comes from the other reason. CI check suite failed — 1,504 skips — is Railway refusing to build a commit whose whole GitHub check suite is failing, including scheduled workflows that re-report onto main's head SHA long after the merge gates went green. Over closure-relevant merges that is p90 4.7h against p90 0.01h when Railway simply builds, and it is self-reinforcing: the freshness monitor goes red exactly when the fleet is behind.

So the closures stay, and .github/workflows/railway-deploy-trigger.yml deploys the services whose closure actually changed, gated on main's own gate status rather than on Railway's reading of the entire suite. Clearing the filters fleet-wide remains rejected on cost: roughly 75 build-minutes per push across 77 services at ~30 merges a day, and three always-on services (ais-relay, notification-relay, scenario-worker) restarting on every merge, dropping the AIS websocket connections among them — and it would not fix the dominant cause anyway. The full measurement is in Railway defers deploys on a red check suite, not on watch paths.

The always-on bootstrap publisher is the deliberate exception: its empty watch path list means Railway watches the whole repository. That broader trigger covers its Dockerfile and future bootstrap inputs without rebuilding the cron seeder fleet.

scripts/audit-railway-watch-paths.mjs compares the registry with live production configuration. It reports exact watch-path and cron drift, missing registered services, and missing required source-routing variables. Apply mode refuses a partial mutation while a service or required variable is absent.

After linking the CLI to the world-monitor production environment, audit the live settings with:

node scripts/audit-railway-watch-paths.mjs

To reconcile only drifted seeders and verify the read-back:

node scripts/audit-railway-watch-paths.mjs --apply

The apply mode changes only drifted build.watchPatterns, build.dockerfilePath and deploy.cronSchedule fields, uses one environment config commit, and waits for Railway's eventually consistent config read-back before reporting success. It does not assign a cron to explicitly always-on services such as the bootstrap publisher, while still auditing their watch paths and required environment. Run the audit after adding or replacing a standalone seeder, changing a bundle dependency, or changing a production cron.

The audit only proves the trigger config matches the registry. Proving a merge actually reached production is the separate deploy-drift check below.

The scheduled operational-acceptance workflow performs the same audit in read-only mode, then the deploy-drift check, before checking compact health. Create the dedicated GitHub Actions environment ingestion-acceptance-production, restrict its deployment branch policy to main, and configure:

• environment secret RAILWAY_PRODUCTION_TOKEN: a Railway project token scoped to the production environment;
• environment variable RAILWAY_PROJECT_ID: the world-monitor project ID.

Do not define the Railway token as a repository or organization secret: workflow_dispatch can target another ref, while the environment's server-side branch policy keeps the production credential unavailable there. The workflow references the environment with deployment tracking disabled, maps the project token to the CLI's standard RAILWAY_TOKEN variable only for the link and audit steps, links only inside the ephemeral runner, and never passes --apply. Do not use the broader account-scoped RAILWAY_API_TOKEN. Missing or inaccessible context intentionally fails the acceptance run rather than silently skipping the live audit.

Reconciliation control-plane provisioning

The reconciliation controller is a dedicated Cloudflare Worker backed by one SQLite Durable Object. Its foundation can be deployed while the existing Railway integration remains unchanged. Keep both RAILWAY_RECONCILE_CUTOVER_ACTIVE and RAILWAY_RECONCILE_AUTO_RECOVERY_ENABLED set to false until the lease-aware target workflow, protected environments, role credentials, and authenticated status probe are ready. Activate the flags separately with the staged cutover below.

Provision these GitHub Actions environments with a main-only deployment branch policy:

Set the non-secret repository variable RAILWAY_RECONCILE_CONTROL_URL to the dedicated HTTPS origin compiled into the shipped control client; the client rejects every other host even if a workflow variable is misconfigured.

Both Railway tokens are distinct project tokens with identical capability; the names record intended use, not an enforced boundary. See the scope note below.

The ordinary lease-aware mutation and verifier jobs receive only their own HMAC roles in their separately protected environments.

The Viewer token is read-only by convention, not by credential scope. Railway issues exactly two token types and neither accepts a role or scope: apiTokenCreate takes only {name, workspaceId} and inherits the creating user's permissions, and projectTokenCreate takes only {projectId, environmentId, name}. The VIEWER value on ProjectRole and TeamRole applies to project members — user accounts — not to tokens. A genuinely read-only token would require a separate Railway user account joined to the project as a VIEWER member, which this project does not maintain.

So mint the Viewer and deploy tokens as two distinct project tokens on the one owner account: distinct tokens still give independent revocation, a separate audit trail, and blast-radius containment if one leaks. What they do not give is an inability to deploy. That property is enforced instead by the --workflow-authorized fence in scripts/trigger-railway-deploys.mjs and the workflow contract tests, so treat any change to those guards as a change to a security boundary. Note also that projectTokenCreate rejects a CLI session with Not Authorized; both tokens must be created from the Railway dashboard.

Breakglass authorization is one-person by deliberate choice. The environment keeps its main-only branch policy and isolated secrets, but has no required reviewer. Requiring that same operator to review their own dispatch adds no independent authorization and sends one approval email for every recovery attempt. The approver workflow input records the delegated operator identity in the immutable controller audit and can equal the dispatch actor; it is not a verified second party. Real two-person control requires a second named operator and prevent_self_review.

The protected resolver deliberately repeats the GitHub, convergence, and provider-inactivity reads after environment approval; the earlier proof is not fresh enough to authorize a state transition by itself.

GET /version proves only that the expected Worker revision propagated. Authenticated /v1/watchdog/status is the readiness check because it also opens the canonical Durable Object and exercises the deployed watchdog HMAC.

HMAC rotation is a coordinated maintenance operation. First disable automatic recovery, confirm authenticated status has no lease, barrier, or dispatch hold, and confirm no target mutation or verifier job is active. Then update the Worker and the one matching consumer environment together and repeat both the version and authenticated status probes. Never rotate across an active lease, hold, or barrier: the old run would lose its only authorization path and the result would require protected operator recovery.

Staged cutover

Do not enable both activation flags in one operation.

1. Confirm the current main head has an exact green gate, no target mutation or verifier run is active, and authenticated controller status has no lease, barrier, attempt, or dispatch hold. Confirm every protected environment has only the role credentials its jobs read.
2. Set RAILWAY_RECONCILE_CUTOVER_ACTIVE=true while keeping RAILWAY_RECONCILE_AUTO_RECOVERY_ENABLED=false. This enables the ordinary lease-fenced mutation and verifier cycle plus protected manual retry, but it cannot authorize a watchdog dispatch. Run one exact-green-head cycle and require terminal acceptance to populate lastAccepted. Unset the cutover flag to roll back admission of new mutation jobs.
3. Enable RAILWAY_RECONCILE_AUTO_RECOVERY_ENABLED only after lastAccepted has been observed fresh, watchdog read failures fail the job after their bounded streak, and a baseline exists for green deferrals caused by a live paginated GitHub inventory. Run the watchdog in dispatch mode and require HEALTHY with no dispatch. Unset this flag first to roll back automatic recovery authority.

The native Railway source integration is not the lease-aware mutation path. It can continue to accept ordinary builds: the reconcile trigger reads the live deployment inventory and treats an accepted or in-progress build for the exact commit as a no-op. The repository has no separate legacy GitHub workflow that deploys with RAILWAY_PRODUCTION_TOKEN; that token remains only in read paths. If either fact changes, disable and drain the new competing mutation path before the staged cutover.

To print the authenticated raw controller state without dispatching anything, run the watchdog's status mode from main:

gh workflow run railway-deploy-trigger-watchdog.yml --ref main -f mode=status

The resulting Read or classify reconciliation control state log contains the closed STATUS_REPORTED envelope. An operator with the separately provisioned watchdog environment variables can run the same reader locally with node scripts/dispatch-stale-railway-reconcile.mjs --phase status; never copy the watchdog HMAC into an issue, command history, or shared shell profile.

Manual recovery evidence contract

evidence_json is a closed JSON object: unknown or omitted fields fail the request. Every decision uses these common fields:

{
  "version": 1,
  "evidenceId": "incident-evidence-0001",
  "runEvidenceId": "github-run-evidence-0001",
  "environmentEvidenceId": "breakglass-review-0001",
  "priorKind": "dispatch_hold",
  "priorCreatedAt": "2026-08-08T08:00:00.000Z",
  "targetRunId": null,
  "targetRunAttempt": null,
  "decisionEvidence": {
    "kind": "pre_mutation_hold",
    "mutationBoundaryCrossed": false
  }
}

The decision-specific decisionEvidence schemas are:

The target must be an exact run of the Railway deploy-trigger workflow on the main branch. All decisions separately require an exact current green main authorization. The target can be an older main run: observed convergence is verified against that run's immutable manifest and exact incident head, while the protected resolution also records the separately revalidated current head. This separation keeps the convergence-acceptance route reachable after main moves, including when it must reset a full 64-run mutation lineage.

The protected retry hold carries an immutable capability that is the only path that can replace a same-head Railway deployment in FAILED. Ordinary runs and watchdog-created recovery holds continue to report that failure without retrying it. A recovery still adopts any active same-head deployment, or a running same-head replacement newer than the failure, so the one-use controller hold cannot duplicate work that Railway already accepted.

retryEvidence has exactly one of these forms. Its evidenceId must differ from the outer evidence ID:

{"kind":"terminal_inactive","evidenceId":"retry-audit-0001","auditedAt":"2026-08-08T08:45:00.000Z"}

{
  "kind": "outage_wait",
  "evidenceId": "retry-audit-0002",
  "automaticEntrantsDisabledAt": "2026-08-08T08:00:00.000Z",
  "allJobsTerminatedAt": "2026-08-08T08:02:00.000Z",
  "lastPossibleLeaseAcquiredAt": "2026-08-08T08:01:00.000Z",
  "auditedAt": "2026-08-08T08:44:00.000Z"
}

The protected resolver rebuilds all GitHub and Railway proof after environment approval, then rechecks it immediately before recording the immutable resolution. The operator HMAC is removed from the process environment before either read-only Railway subprocess can start.

Watchdog outcome decisions

Bootstrap R2 publisher contract

The public bootstrap tiers use the dedicated private bucket worldmonitor-bootstrap. Managed r2.dev access stays disabled and the bucket has no custom domain; clients continue to enter through /api/bootstrap so the WAF, origin policy, rate limits, telemetry, and future access controls remain in the request path.

This service is an always-on publisher, not a Railway cron. Configure it with Dockerfile.publish-bootstrap-tiers (the root application Dockerfile does not contain the publisher) and start command node scripts/publish-bootstrap-tiers.mjs --loop, no cron schedule, and an empty watch-path list (whole-repository watching). It publishes both tiers on startup, then fast every two minutes and slow every ten minutes. Keep Redis authoritative: until the publisher and later rollout gates pass, /api/bootstrap continues to serve its existing Redis assembly.

The environment contract is deliberately split by consumer:

Preview and development do not receive either credential; missing credentials must use the Redis path. The publisher must not fall back to any CLOUDFLARE_R2_* account, bucket, key, secret, or API token. Never copy the publisher credential into Vercel or the edge credential into Railway, and never add a VITE_ alias for any bootstrap R2 credential. Set IRAN_EVENTS_ENABLED explicitly to the same value in both production services; otherwise the publisher and edge handler resolve different tier contents.

Provision and release in this order:

1. Create the repo-root Railway service, install only the shared and publisher variables above, and confirm the live watch paths and lack of a cron schedule.
2. Deploy the publisher before enabling shadow measurement or serving from R2.
3. Parse both fast.json and slow.json, then verify generatedAt advances in two successive publishes for each tier.
4. Install only the shared and read-only variables in Vercel production. Keep them absent from preview and development.
5. Run the negative permission probes: publisher cannot access worldmonitor-data; edge cannot write/delete in worldmonitor-bootstrap and cannot read worldmonitor-data.

Rotate one consumer at a time: create a replacement token, update that consumer, verify its publish or read with the replacement, then revoke the old token. On suspected compromise, revoke first; Redis fallback preserves availability while a replacement is issued. Never log, commit, or copy credential values into an incident note.

Merged does not mean deployed

.github/workflows/seed-freshness-monitor.yml runs every 15 minutes on the default branch. Scheduled runs first require the latest main commit's gate status to be green; a missing, pending, or failed gate makes the workflow fail closed instead of producing a green skipped run. Manual runs execute directly. The workflow runs two independent jobs. Read the one that answers your question.

• monitor is the gated job. After the repository gate, it checks live Railway watch paths, cron schedules, required routing variables, and service presence against scripts/railway-services.json, then checks public compact health.
• Railway deploy drift is a separate job. It has no needs: and no gate condition, so it runs in parallel with monitor and publishes its own conclusion. A failed gate turns monitor red but leaves this job's verdict intact (#6523).

During a gate outage, read the Railway deploy drift job, not monitor. A red monitor alone says nothing about the fleet. Note that the run's overall conclusion and its badge stay red whenever monitor fails, so you must open the run and read the job conclusions to tell a clean fleet from a stranded one.

monitor fails on every actionable problem, including SEED_ERROR, STALE_SEED, STALE_CONTENT, and degraded composed coverage. Statuses that explicitly end in _ON_DEMAND remain informational. It deliberately does not run on an ingestion push because Railway may not have deployed or executed that revision yet. This is the operational acceptance gate for the "merged and green, but production data is still unhealthy or running under stale deployment controls" gap.

Deploy-drift check

node scripts/check-railway-deploy-drift.mjs        # add --json for the machine-readable form

The watch-path filter is one way a merge fails to reach production; a GitHub integration that stopped delivering (#6064) and a build that failed after the merge landed are others. This check is deliberately agnostic about which. For every service whose Railway source is this repository it takes the newest deployment that actually reached a running state, reads meta.commitHash off it, and asks whether that source contains everything that can reach the service. Four verdicts are healthy — CURRENT, CURRENT_FOR_CLOSURE, AHEAD, PENDING_BUILD — and the problem set is derived from them by negation, so the reported verdicts are REJECTED_PUSH, BEHIND, CLOSURE_UNKNOWN, BUILD_FAILED, UNKNOWN_SOURCE, UNKNOWN_STATUS, NO_DEPLOYMENTS, NO_BUILD_IN_WINDOW and QUERY_FAILED. The file's header comment and exported constants are the exact semantics.

CURRENT_FOR_CLOSURE is what makes this compatible with watch-path filtering: the service is not on head, and that is correct, because nothing it can see has changed since. REJECTED_PUSH now means Railway refused a push that did reach the service, and the reason Railway gave is printed with it — a CI check suite failed refusal and a path refusal have different owners.

Two questions need local history: ancestry (git merge-base --is-ancestor) and the diff between the commit each service is running and head. A service legitimately weeks behind sits outside any fixed depth, so both workflows check out full history with filter: blob:none — the diff walks trees and never needs blobs. Never re-fetch with --depth afterwards: that re-shallows the clone and strands exactly those commits. An unanswerable question reports the service (CLOSURE_UNKNOWN) rather than excusing it.

Accepted degradations go in scripts/railway-deploy-drift-baseline.json, each with an owner issue and the whole file with an expiry, split by the same applyAcceptanceBaseline that scripts/check-seed-freshness.mjs applies to compact health — so expiry, prune-on-recovery, and "a service failing with a different verdict than the one baselined still blocks" cannot acquire two meanings. It held 62 REJECTED_PUSH entries — every filtered service — until #6142 made the check closure-aware; those were not degradations, they were the check demanding that a filtered service run head. The last remaining entry — umami at BEHIND against #6064 — was pruned on 2026-08-06 once the service was running head again, so the file is now empty. Acknowledged entries are printed on every run and do not fail it, so a green monitor here means "nothing new went stale", not "every service is on head". A service that recovers is printed as recovered — prune it. The expiry does not fire on an empty list: it exists to stop a suppression outliving its cause, and with nothing suppressed there is nothing to re-review.

Recovering a stale service

Use the Railway Deploy Trigger workflow. Its manual dry-run reports the closure plan without mutation; a normal manual dispatch runs the same green-main authorization as the automatic path. Do not run the trigger script directly for production recovery:

# Local inspection only; this cannot authorize a production mutation.
node scripts/trigger-railway-deploys.mjs --only <service-name> --dry-run

The workflow builds from an exact green main commit, so the resulting deployment carries a SHA the drift check can compare. It is a no-op for a service Railway already took. After the lease-aware cutover, non-dry-run direct invocation fails closed: only the protected workflow can acquire the 30-minute non-renewing owner lease and bind the attempt manifest. The script's remaining local guarantees are still useful for previews:

• The deployed commit defaults to origin/main, never your local HEAD, so standing on a feature branch cannot ship it to production.
• A --head that is not reachable from origin/main is refused outright.
• --only throws on a name the fleet does not have rather than silently selecting nothing — a typo that reported "no service needs a build" would read exactly like a healthy fleet.

Run git fetch origin before a local preview so origin/main is current.

The reconciliation control plane deliberately separates two failures:

• A runner-less or pre-mutation attempt owns no unbounded GitHub lock. Once any bounded LEASED/PREPARED lease expires and no dispatch hold is ambiguous, the independent watchdog can dispatch a replacement. It never cancels the old run.
• Once MUTATION_STARTED is durable, a project/environment-wide barrier blocks every automatic head until the exact result passes terminal deployment convergence plus strict zero drift, or the protected Railway Reconcile Manual Recovery workflow records an audited resolution. Lease expiry alone never clears this barrier.

The watchdog cannot create a hold or dispatch a new recovery unless both RAILWAY_RECONCILE_CUTOVER_ACTIVE and RAILWAY_RECONCILE_AUTO_RECOVERY_ENABLED are exactly true. Classification is deliberately read-and-repair: if an earlier authorized watchdog or manual recovery dispatch left a DISPATCH_HELD record, an ordinary scheduled run may bind its exact accepted workflow run or close it after definitive pre-dispatch rejection. That repair is independent of both flags because it completes an authorization that already exists; it never creates a hold or sends a dispatch. The first flag enables the lease-aware mutation contract and protected manual retry, while the second authorizes new automatic recovery. RECOVERY_AUTHORIZED means the controller persisted a one-use dispatch hold but has not yet dispatched it; RECOVERY_DISPATCHED means GitHub accepted the request and the controller durably bound its exact workflow run and attempt. A green watchdog run means only that observation did not poison main; authoritative success is the replacement's final strict-acceptance step, including a verified no-op. The run summary prints the current prior attempt or hold ID for protected manual recovery.

The observer combines run summaries from the preceding 24 hours with separate queries for every active target status, including runs that started before that window. It repeats the active sweep around the history read and defers if the inventory changes. With a durable mutation barrier, it reads attempt jobs only for active and durably referenced runs because the barrier already forbids recovery. After strict terminal acceptance, lastAccepted.acceptedAt retires older failures while a failure that finishes after the watermark is still read. Before the first trusted acceptance, it keeps the full fail-closed non-success scan. It follows at most 10 pages per query, makes at most 250 GitHub API requests, and bounds each request to 10 seconds. Durable barriers and dispatch holds remain authoritative beyond that window; exhausting any read budget defers recovery rather than dispatching on partial history.

Do not use railway redeploy: Railway documents it as rebuilding the most recent deployment with the same code, so it cannot pick up a newer fixed commit.

railway up is not an ordinary recovery path. If the control plane itself is unavailable, first disable every automated mutation entrant, inventory all mutation jobs, wait for them to terminate, and preserve the durable state. A separately audited manual action may be considered only after the 30-minute lease plus the documented 5-minute termination grace, 1-minute clock/network margin, and 6-minute safety margin have elapsed. Time does not override a restored MUTATION_STARTED barrier. If an approved break-glass recovery still requires railway up, use a clean detached worktree at origin/main:

git fetch origin
git worktree add --detach /tmp/railway-deploy origin/main
cd /tmp/railway-deploy
git rev-parse HEAD                       # must equal origin/main
railway up --service <service-name> --environment production --detach

An upload carries no commit SHA, so check-railway-deploy-drift.mjs reports that service as UNKNOWN_SOURCE until the next git-triggered build replaces it. That is expected after a recovery upload, not a second failure — and the deploy trigger treats it as a reason to deploy, so it self-heals on the next run.

After any recovery path, verify the deployment commit SHA and the relevant compact-health problem have both advanced. See Railway's official redeploy CLI reference and deployment actions reference.

railway run is also not production-network evidence: Railway documents it as executing locally after injecting service variables. For an immediate long-cron backfill, use a controlled temporary Railway cron execution, verify its terminal run plus seed metadata and compact health, then restore the captured command and schedule and rerun the operational-config audit. The full rollback-safe sequence is documented in A merged seeder fix is not live until its cron fires.

How It Works

Each "bundle" is a single Railway cron service that replaces N individual services. The bundle script spawns each member seed sequentially via child_process.execFile, checking Redis seed-meta: timestamps to skip seeds that ran recently. Original seed scripts are unchanged.

The derived-signals bundle also owns the final China composition (seed-china-decision-signals.mjs). It runs after the cross-Strait source lane, calls the public six-domain RPC, publishes intelligence:china-decision-signals:v1, and records seed-meta:intelligence:china-decision-signals. It does not add providers or recompute any source-domain method. Before rollout, run node scripts/audit-china-decision-parity.mjs; after staging is deployed, pass --require-live --url <public-staging-api-base>. Against production that live probe is already enforced every six hours by .github/workflows/china-decision-parity-live.yml, so the manual run is for pre-production environments that workflow does not reach. The probe output is intentionally sanitized to reachability, latency, generation time, and group states.

Graceful fetch failures: runSeed now treats transient upstream fetch failures as non-zero graceful failures after extending the last-good Redis TTL. This applies to bundled members and standalone runSeed cron seeders: Railway may mark that cron run failed, but /api/health and seed-contract probes still read the preserved seed-meta: freshness. Alerting should either tolerate these transient cron failures or key sustained data-health pages off those freshness checks. Bundle member logs use status=GRACEFUL_FAIL; external log consumers that match only status=FAILED should include GRACEFUL_FAIL. The bundle summary still reports these under failed:N, so use per-section status when distinguishing graceful upstream outages from hard failures.

Standalone follow-up: scripts/seed-military-flights.mjs and scripts/seed-service-statuses.mjs still have manual graceful failure paths that exit 0. Track those separately if the standalone graceful-failure contract needs to be made fully uniform beyond shared runSeed users.

Per-bundle migration:

1. Delete ONE old member first (to free a slot under the 100 limit)
2. Create the bundle service on Railway
3. Wait 2-3 cron cycles, verify /api/health shows OK for all member seeds
4. Delete remaining old member services
5. Monitor 24h before proceeding to next bundle

Rollback: Delete the bundle service, re-create individual services. Scripts are unchanged in the repo.

Services to DELETE (46 total)

Standalone service retired before bundle restoration

Replaced by seed-bundle-ecb-eu

Replaced by seed-bundle-portwatch

Replaced by seed-bundle-static-ref

Replaced by seed-bundle-resilience

Replaced by seed-bundle-derived-signals

Replaced by seed-bundle-climate

Replaced by seed-bundle-energy-sources

Replaced by seed-bundle-macro

Replaced by seed-bundle-health

Replaced by seed-bundle-market-backup

Replaced by seed-bundle-relay-backup

Services to CREATE (11 total)

All new services share these settings:

• Root directory: . (repo root, so npm ci installs all deps)
• Build command: (default nixpacks, uses scripts/nixpacks.toml)
• Source branch: main
• Resources: 1 vCPU / 1 GB RAM
• NODE_OPTIONS: --dns-result-order=ipv4first

Watch paths: Use scripts/**, shared/** for all bundles. scripts/** covers all seed scripts and their helpers. shared/** is needed because loadSharedConfig() in _seed-utils.mjs resolves ../shared/ (repo root) before ./shared/ (scripts dir), so config JSON files like country-names.json, iso3-to-iso2.json, and others live at the repo root shared/ directory. Without shared/**, config-only edits won't trigger redeploys.

Bundle 1: seed-bundle-ecb-eu

Why 13:00 UTC (not 06:00): the daily ECB SDMX series (€STR, yield curve, CISS) are rebuilt during ECB's early-morning refresh window. A 0 6 * * * run (08:00 CEST — exactly €STR's publication moment) intermittently hit that window and got empty/incomplete datasets, so those three sections failed gracefully (TTL extended, no data loss) while the bundle exited non-zero and showed red on Railway. 13:00 UTC (15:00 CEST) clears €STR (08:00 CET), the yield curve (~12:00 CET) and CISS morning publication. Changed 2026-07-01.

Bundle 2: seed-bundle-portwatch

Bundle 3: seed-bundle-static-ref

Defense Patents is an intentional data-series migration, not a continuation of the former grant/issue series. USPTO ODP Patent File Wrapper records represent applications, so date is the application filing date and abstract remains empty for wire compatibility. The producer marks the discontinuity with sourceVersion: uspto-odp-v1 and schemaVersion: 2; operational comparisons must not treat pre-migration grant dates and post-migration filing dates as one continuous metric.

Defense Industrial Base writes military:industrial-base:v1 from World Bank MS.MIL.* series and military:arms-suppliers:v1 from SIPRI-derived five-year supplier shares. Both values have a 30-day TTL. Each source is eligible every 10 days, and the daily service evaluates that interval before day 10, which keeps the canonical TTL above the three-refresh safety floor. The sources run as separate bounded processes under a 570-second bundle budget, so one failure cannot block the other source's publication. A SIPRI portal failure preserves last-good supplier rows with their original timestamps. A separate SIPRI-completion marker stays old after a partial pass, so the next daily tick retries the portal instead of treating the partial pass as complete. Strict health-probe registration is a staged follow-up after the first Railway run publishes both seed-meta keys; the follow-up must cite real Railway pre-seed evidence under the health-probe cutover contract.

Bundle 4: seed-bundle-resilience

Bundle 5: seed-bundle-derived-signals

Staged correlation runtime modes

Correlation uses one Redis control key, correlation:runtime-mode:v1, whose value is a JSON object with one strict field, for example {"mode":"legacy"}, {"mode":"exact"}, or {"mode":"fuzzy"}. The browser reads the public GET /api/correlation-runtime-mode contract with cache: "no-store" at startup and before every correlation refresh. The correlation seeder reads the Redis key again on every compute cycle; it does not reuse a previous cycle's decision.

Every missing key, malformed JSON or shape, unknown mode, missing Redis credentials, failed Redis request, non-OK browser response, or failed browser payload parse resolves to legacy. legacy remains the current keyword clustering behavior. Exact entity clustering and fuzzy resolution are staged follow-ups owned by #5984 and #5989; this control slice does not activate either mode or change live configuration.

Changing the key is an operational activation or rollback and requires separate operator approval. Keep that approval, the observed validation evidence, and the rollback decision outside the code deployment; the code path is only the fail-closed read and hand-off contract.

Japan MOD discovery surface and recovery gate

The official discovery URL is the Japanese Joint Staff homepage, https://www.mod.go.jp/js/. The runtime makes one direct request and, after a transport failure or an empty allowlisted index, one request through JAPAN_MOD_PROXY_URL (falling back to PROXY_URL). It never downloads linked PDFs during a scheduled run.

Japan MOD's Cloudflare rule is path-level, not egress-level. Measured 2026-08-01, from both direct egress and the configured Decodo path:

Note that /js/ succeeds while /js/index.html does not. Cloudflare fronts the succeeding path too — the 200 response carries a window.__CF$cv$params beacon — so this is a rule that exempts /js/, not a zone the CDN does not cover. Changing the discovery URL to any other path on this host is a regression, not a refinement. This is the general lesson for other Japanese government sources: probe the bare directory before concluding the host is blocked.

Do not derive an English companion PDF by inserting e before .pdf. The English series carries its own counter, so the mapping resolves to unrelated releases. Measured 2026-08-01:

That day Japanese published _01/_02 while English published _01e–_05e. Every check that mapping would be validated against — HTTP 200, application/pdf, %PDF magic, Joint Staff publisher marker — passes on the wrong document, so it cannot be validated into correctness. The correct counterpart is only resolvable from the English index, which is the surface Cloudflare blocks. parseJapanModIndex therefore accepts only /js/pdf/<year>/p<YYYYMMDD>_<NN>.pdf, which structurally excludes the English series, and the source reports companionResolution: english_index_blocked_no_derivable_companion.

Discovery records candidates for manual review; it never admits an observation. admittedDocumentCount counts hand-reviewed rows and unreviewedCandidateCount counts the discovered backlog. A 200 carrying no allowlisted release is JMOD_INDEX_EMPTY, not success.

The blocked English index stays wired as shadowIndexUrl: a direct probe that runs at most once per 24 hours, only after the homepage request already succeeded. It is diagnostic only — it never contributes to requestCount, errorCodes, transportStatus, or lastSuccessAt. Watch shadowIndexProbe.status flip from blocked to reachable; that is the signal that English provenance can be restored and the +e constraint above revisited. candidates and shadowIndexProbe are operator-only and stripped from the anonymous bootstrap projection.

If a future provider change is needed instead, the concrete external dependency for the current provider is an active Decodo Site Unblocker subscription with source-specific credentials and a successful target test. The ordinary residential gateway credential is not a substitute for that product. If the provider requires disabling TLS verification, do not weaken the adapter; provision a trusted provider CA or use an approved authenticated HTTPS integration instead.

Recovery is accepted only when:

1. crossStraitActivityJapanMod reports OK for two consecutive scheduled three-hour runs from distinct scheduled executions;
2. lastSuccessAt is non-null, later than the deploy, and advances between those two successful runs;
3. the published _seed.sourceVersion reads taiwan-mnd-html+japan-joint-staff-homepage-v3, proving the resilient homepage-discovery adapter is the code that ran rather than a merged-but-not-deployed PR;
4. the source reports transportMode: japanese_homepage_candidate_discovery and at least one newly discovered candidate tied to each successful fetch — retained rows do not count;
5. the combined cross-Strait publication remains available and explicitly source-degraded when a later Japan MOD request fails.

Bundle 6: seed-bundle-climate

Bundle 7: seed-bundle-energy-sources

Bundle 8: seed-bundle-macro

Bundle 9: seed-bundle-health

Bundle 10: seed-bundle-market-backup

Bundle 11: seed-bundle-relay-backup

Registry-covered live resilience services

These live Country Resilience services are not slot-saving consolidation migrations and should not be counted in the 35-slot savings plan above. They are listed here so their Railway start commands are first-class registry-covered entries.

seed-bundle-resilience-recovery

seed-bundle-resilience-energy-v2

Services that STAY unchanged (54 total)

Infrastructure (4)

Long-running services (4)

Consumer prices pipeline (3)

Standalone seed crons (43, not bundled)

Inventory check: 4 infra + 4 long-running + 3 consumer + 46 delete + 43 standalone = 100

Standalone seed crons added after this snapshot

These data seeds were added after the 2026-04-10 inventory above. The rows marked planned are registry/documentation entries for services that are not provisioned in production; they remain excluded from the live audit and --apply until an explicit lifecycle activation. The four planned rows below are repository-root nixpacks-root-repo cron candidates (root directory ., start command node scripts/<file>), so their eventual packaging can include valid imports outside scripts/. Active rows must instead follow the deploy mode and exact watchPatterns recorded in scripts/railway-services.json. These rows are intentionally not part of the 100-service inventory count above and are registered in scripts/railway-services.json with deploy mode nixpacks-root-repo.

Cadence below is inferred from each seed's cache TTL as a documentation aid; confirm the live cron schedule and Service ID against the Railway dashboard before relying on it. Rows showing a bold cron expression with a verified date were read from the Railway API rather than inferred.

To verify one yourself (reads cronSchedule for every service in the project; the CLI stores the token at ~/.railway/config.json):

railway whoami   # confirm you are logged in, then query the API:
node -e "const c=require(require('os').homedir()+'/.railway/config.json');
fetch('https://backboard.railway.com/graphql/v2',{method:'POST',
 headers:{'Content-Type':'application/json',Authorization:'Bearer '+(c.user.token||c.user.accessToken)},
 body:JSON.stringify({query:'query(\$id:String!){project(id:\$id){services{edges{node{name serviceInstances{edges{node{cronSchedule}}}}}}}}',
 variables:{id:'29419572-0b0d-437f-8e71-4fa68daf514f'}})})
 .then(r=>r.json()).then(d=>d.data.project.services.edges.forEach(e=>{
   const cs=e.node.serviceInstances.edges.map(x=>x.node.cronSchedule).filter(Boolean);
   if(cs.length)console.log(e.node.name.padEnd(40),cs.join(','));}))"

The bilateral HS4 cron uses COMTRADE_API_KEYS and a 480-request hard budget under the provider's 500-call monthly quota. The authenticated route requests one four-year window (Y-2 through Y-5) in each of two HS4 batches and keeps the newest row per product/partner. The public-preview fallback cannot accept that period list and tries Y-2, then Y-3. A 24-day freshness gate prevents accidental repeat runs; health reports COVERAGE_PARTIAL below 110 country shards and stale after 35 days. Country payloads live for 40 days so a missed monthly tick becomes visible before last-good data expires.

Not standalone services (documented here to avoid confusion):

• scripts/seed-chokepoint-flows.mjs — spawned in-process by the AIS relay (ais-relay.cjs), not deployed as its own cron.
• scripts/seed-military-maritime-news.mjs — this is the script behind the existing seed-military-maritime standalone cron (USNI/NGA warm-ping) listed in the inventory above.

Execution Order (recommended)

Start with lowest-risk, highest-savings bundles.

Running total: 3 + 3 + 1 + 3 + 4 + 5 + 5 + 3 + 1 + 4 + 3 = 35 slots freed

Verification Checklist (per bundle)

After deploying each bundle and before deleting old services:

• Bundle service shows "Active" in Railway dashboard
• First cron fire produced logs (check Railway logs)
• Logs show expected [Bundle:X] Starting (N sections) and Finished lines
• Each member seed shows Done or Skipped (not all failed)
• /api/health shows OK for all member seed-meta keys (not STALE_SEED)
• Wait at least 2 full cron cycles before deleting old services
• After deleting old services, verify health still shows OK on next cycle

Env Vars

Each bundle service inherits the same env vars as the individual seeds it replaces. Copy these from any existing seed service in Railway:

• UPSTASH_REDIS_REST_URL
• UPSTASH_REDIS_REST_TOKEN
• NODE_OPTIONS=--dns-result-order=ipv4first
• Plus any API keys used by member seeds (GIE_API_KEY, ICAO_API_KEY, etc.)
• SAM_GOV_API_KEY for the Global Tenders SAM.gov adapter. The other initial procurement adapters do not require credentials.

The simplest approach: use Railway's "shared variables" or copy all env vars from the worldmonitor (ais-relay) service, which has a superset of all API keys.

Import-HHI Comtrade 429 Runbook

Issue #3979 covers the residual operational failure mode for the Country Resilience Index importConcentration dimension: AE/RU/NO/CH can still remain absent from resilience:recovery:import-hhi:v1 when UN Comtrade rejects the monthly recovery bundle for key budget, pacing, or reporter metadata reasons.

Decision: treat this as Comtrade quota/pacing while the seed logs show HTTP 429 or quota-exhausted HTTP 403 responses. Do not change importConcentration scoring until the rate-limit path has been addressed and a force-refresh proves that Comtrade is returning non-quota responses for the watched reporters.

Controls

Set these on the Railway service that runs node scripts/seed-bundle-resilience-recovery.mjs:

Reporter cohort splitting is the last resort. Prefer more COMTRADE_API_KEYS, then wider per-key delay, then lower concurrency. The import-HHI seeder fetches the watched #3979 reporters first when they are missing, so a replenished force-refresh should recover AE/RU/NO/CH before unrelated registry backfill can consume the hourly provider budget. Aggressive incident pacing such as IMPORT_HHI_PER_KEY_DELAY_MS=15000 with IMPORT_HHI_MAX_CONCURRENCY=1 can exceed the 30-minute bundle window; that mode intentionally relies on checkpoint/resume across ticks, not one-pass completion. Cohort splitting should only be used if a single full pass still exhausts the provider budget after the first three controls.

The import-HHI publish gate requires AE/RU/NO/CH as well as the normal country-count floor. If one of those watched reporters is still absent, the seed run fails validation with emptyDataIsFailure: true, does not refresh seed-meta, and leaves the bundle eligible to retry instead of stranding a fresh-but-incomplete canonical payload for the full monthly interval.

If a watched reporter is still missing and the seed log says status=200 rows=0, stop treating that reporter as a key-budget problem. Inspect Comtrade reporter metadata, data availability, and query-shape filters (customsCode, motCode, cmdCode) before considering any scoring change. The known non-M49 reporter-code exceptions are pinned in scripts/shared/comtrade-reporter-overrides.json; as of the #3979 follow-up this includes Norway (NO=579) and Switzerland (CH=757). Russia (RU=643) currently needs the seed-only stale period fallback (Y-5..Y-8) because Comtrade returns zero annual import rows for the standard Y-1..Y-4 window but still exposes 2018 rows.

Force-Refresh

After deploying a pacing/key-budget change, bypass the 30-day freshness gate:

IMPORT_HHI_VERBOSE=1 FORCE_RESEED=true node scripts/seed-recovery-import-hhi.mjs

Then warm live scores so importConcentration reads the refreshed canonical key:

API_BASE_URL=https://api.worldmonitor.app \
WORLDMONITOR_SEED_REFRESH_KEY=<seed-refresh-key> \
WORLDMONITOR_API_KEY=<read-key> \
node scripts/seed-resilience-scores.mjs

WORLDMONITOR_SEED_REFRESH_KEY is required: the resilience score seeder uses it for the seed-only get-resilience-ranking?refresh=1 recompute path. Keep WORLDMONITOR_API_KEY or WORLDMONITOR_VALID_KEYS available too so laggard per-country score warms can fall back to the normal premium read endpoint. In Railway, the service environment should already provide the Upstash Redis credentials; for a local force-run, export UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN as well.

If the run is fixing missing interval data, the success signal is the seed_complete log for domain="resilience:scores" with intervalsWritten > 0 and no status="ERROR". A failed interval recovery sets status="ERROR" plus intervalFailureReason and includes the diagnostic counts intervalMissingScorePayloadCount, intervalStaleScorePayloadCount, intervalInvalidScorePayloadCount, intervalMalformedScorePayloadCount, intervalFormulaSkipCount, and intervalPayloadSkipCount.

Verify the public audit surfaces after the run:

curl -fsS https://api.worldmonitor.app/api/resilience/v1/get-runtime-manifest \
  | jq '{formulaTag, rankingCache, constructVersions, intervals}'
curl -fsS https://api.worldmonitor.app/api/health \
  | jq '.checks.resilienceIntervals'

Pass condition for interval recovery: runtime manifest reports intervals.available=true, and /api/health reports resilienceIntervals.status="OK" with records > 0.

Verification

Verify both Redis and the live score API:

WORLDMONITOR_API_KEY=<key> node scripts/verify-import-hhi-coverage.mjs

Pass condition for AE/RU/NO/CH:

• resilience:recovery:import-hhi:v1.countries.<ISO2> is present.
• seed-meta:resilience:recovery:import-hhi is fresh.
• Live GetResilienceScore has importConcentration.coverage > 0.
• Live importConcentration.imputationClass is empty.

If the live API key is not available during Redis-only triage, use:

IMPORT_HHI_VERIFY_REDIS_ONLY=1 node scripts/verify-import-hhi-coverage.mjs

Redis-only verification is not sufficient to close #3979; it only confirms that the seeder recovered the canonical payload before score warmup.