Health Endpoints

/api/health

Primary health endpoint. Checks all Redis-backed data keys and seed freshness metadata in a single pipeline call.

Authentication: Compact health (?compact=1) is public for uptime and keyword monitors. Detailed health (/api/health without compact=1) and the operator history view (?history=1) require a valid operator/enterprise API key because they expose canonical Redis key names, record counts, and freshness thresholds. Browser origins must still pass the CORS allowlist in api/_cors.js; requests with no Origin header, such as server-side monitors, are allowed only for compact health unless they include an operator key. Health responses are never cached (Cache-Control: private, no-store, max-age=0 and CDN-Cache-Control: no-store).

HTTP Method: GET

Query Parameters

Response Status Codes

The overall health verdict lives in the JSON status field, not the HTTP code. Every state except REDIS_DOWN returns 200 so warn-level seed jitter doesn't flap HTTP-status monitors (see PR #2699). REDIS_DOWN returns 503 because with Redis unreachable the endpoint can assess nothing, so a plain HTTP probe must see a failure.

Response Body

{
  "status": "HEALTHY | WARNING | DEGRADED | UNHEALTHY | REDIS_DOWN",
  "summary": {
    "total": 262,
    "ok": 262,
    "warn": 0,
    "onDemandWarn": 0,
    "staleContent": 0,
    "rolloutPending": 0,
    "crit": 0
  },
  "checkedAt": "2026-08-07T08:25:57.776Z",
  "checks": {
    "earthquakes": {
      "status": "OK",
      "records": 142,
      "seedAgeMin": 8,
      "maxStaleMin": 30
    }
  }
}

summary fields: total is one entry per probed key, and grows as panels are added — read it from your own response rather than from this page. warn excludes on-demand-empty keys — those are surfaced separately as onDemandWarn so they don't drive the overall verdict to WARNING. staleContent is a subset of warn (fresh seeder but the upstream feed stopped advancing), and rolloutPending is another subset of warn (a newly deployed schema still inside its bounded deploy-before-cron window). Only crit (EMPTY/EMPTY_DATA) drives DEGRADED/UNHEALTHY.

With ?compact=1, the checks object is replaced by problems containing only non-OK keys.

Consumer-price source coverage

The consumer-prices-core service exposes GET /wm/consumer-prices/v1/coverage?market=ae (the service API key is required). Its response includes attempted, completed, failed, and validator-rejected page counts for the market and each active retailer. status is healthy, partial, or degraded; a partial result remains publishable, while a rejected observation is never admitted merely to improve the coverage count. The publisher writes the same snapshot to consumer-prices:coverage:<market> and includes its aggregate completion ratio in seed-meta, so /api/health can distinguish a fresh partial run from a stopped producer.

This operational contract complements, rather than replaces, the source-quality work in #5445 and #5811.

Coverage schema rollout handshake

The Edge health registry deploys with the PR that adds a market, but the keys it reads can only appear on consumer-prices-core's next daily scrape (02:00 UTC) → aggregate (02:15) → publish (02:30) window. #6059 closes that window with two independent gates, both required:

1. Activation. After it writes a coverage snapshot that actually attempted pages for a market, the daily publisher (consumer-prices-core/src/jobs/publish.ts, and only it) SETs seed-activated:consumer-prices:coverage:v1:<market> with no TTL. The marker is durable and one-way — once it exists, that market is strict forever and absent, empty, stale, or below-threshold coverage classifies exactly as it would without the handshake. A truthful-but-empty snapshot (no active retailers, or retailers with zero recorded runs) does not activate. The v1 segment is the coverage schema version, so a future schema change opens its own separately-reviewed window rather than inheriting activation earned by the old shape.
2. Deadline. Softening also stops at a per-market wall-clock timestamp compiled into api/health.js, whether or not the producer ever ran. A missed or failed first tick therefore escalates to EMPTY (crit) on its own, within one complete scrape/aggregate/publish window.

Every one of the eight coverage checks also carries activated (true once its durable marker exists) in all statuses, so rollout progress is auditable from the payload and an EMPTY market can be told apart as "activated, then broke" rather than "never ran" without probing Redis by hand.

If a market is still ROLLOUT_PENDING as its deadline approaches, the producer — not the health reader — is what needs attention. Check the seed-consumer-prices-publish service logs for that market: coverage-activation:<market> failed means the snapshot published but the marker write did not (it retries next run), while coverage:<market> published without attempted pages — activation marker withheld means no retailer page was attempted for that market at all, which is a scrape problem, not a rollout one. scripts/seed-consumer-prices.mjs --force can republish a single market's data as a last resort, but it deliberately does not activate: its coverage key carries a 30-minute TTL, and a permanent activation earned from a 30-minute artifact would leave that market in an unrecoverable EMPTY (crit) once the data expired. Only the daily publisher, which writes a 26h key, activates a market.

Only a wholly absent coverage key is softened. Once the key exists the producer has run, so EMPTY_DATA, COVERAGE_DEGRADED, COVERAGE_PARTIAL, and STALE_SEED all stay strict — coverage counts are never synthesized, and success is never inferred from unrelated canonical consumer-price keys. scripts/check-seed-freshness.mjs re-checks each rolloutPendingUntil against the wall clock rather than trusting the status string, so an expired or unparseable deadline (or a stale cached compact snapshot) reports the problem normally.

Two limits are deliberate. First, softening keys on "the coverage key is absent" is not identical to "the producer has not run": if a scheduled run publishes every other snapshot for a market but buildCoverageSnapshot alone throws, that market reads ROLLOUT_PENDING rather than a coverage failure until its deadline passes. Narrowing this by inferring health from sibling keys is deliberately not done — it is the fabricated-success inference the design forbids, and seven of the eight markets have no sibling health keys registered anyway. Second, overall becomes WARNING (never UNHEALTHY) for the duration of the window, so a keyword monitor configured per Integration with Monitoring Tools — which alerts whenever "status":"HEALTHY" is absent — will fire during a rollout. That threshold already fires on any ordinary STALE_SEED warn, so this adds no new alert class, but it is worth knowing before a planned schema rollout.

Relay ingestion telemetry

The relay's public GET /health remains HTTP 200 for Railway/process-liveness probes, but its top-level JSON status now mirrors the aggregate ingestion verdict and can be degraded; ingestion.status carries the same application-level result. It becomes degraded when aviation or RSS served coverage falls below its configured floor, or when configured AIS is disconnected, has no usable current-connection position data, or serves zero snapshots after they are requested. AIS WebSocket upgrades time out after 30 seconds by default, and position readiness expires when an open stream stops delivering accepted position reports for five minutes by default; either failure re-enters the bounded reconnect schedule, while retained vessel data remains a stale fallback. Deployments intentionally missing AIS credentials report enabled: false and status: "disabled"; that optional adapter does not degrade aggregate health.

GET /health also adds upstream attempt, throttle, failure, cooldown, position age, and position-freshness threshold details to explain the current verdict. A run of consecutive HTTP 429 upgrade rejections is reported as consecutiveThrottles, and once it reaches AIS_THROTTLE_ESCALATE_AFTER the reconnect ceiling switches from AIS_RECONNECT_MAX_MS to the much longer AIS_THROTTLE_RECONNECT_MAX_MS and throttleEscalated becomes true. A 429 arrives before the API key is sent, so it identifies an egress-IP rate limit rather than a transient stream fault; escalating stops the relay contributing refused requests to the provider's sliding window, which can otherwise sustain the block. Any non-throttle outcome — a different handshake error, a close without a recorded error, or an accepted frame — resets the counter and restores the ordinary ceiling, so ordinary disconnects keep the responsive schedule. GET /metrics remains the operational counter surface: it exposes rolling-window per-route counters plus process-lifetime AIS connection attempts, successful streams, throttles, terminal failures, reconnect state, and the same position-freshness telemetry. It also reports timeout, authentication rejection, fallback, served coverage, and RSS feed backoff. AIS snapshot unauthorizedClient is tracked separately from upstream authentication failures so client traffic cannot be mistaken for provider health. The theaterPosture section attributes each successful theater-posture publication cycle to the upstream that actually fed it (opensky, adsb.lol, wingbits, or vessel-only), so healthy publication through the fallback chain is never mistaken for OpenSky recovery; the same source is written as sourceVersion on seed-meta:theater-posture (with a producer field distinguishing the relay loop from the seed-military-flights writer). Unlike the rolling-window route counters, theaterPosture.sourceCountsSinceBoot and emptyRejectionsSinceBoot are process-lifetime because the seed cadence exceeds the metrics window. lastRun carries the latest cycle's source and counts. Published cycles require both the canonical envelope and seed metadata to succeed and include published: true, seededAt, and write outcomes; a replica-envelope failure is exposed through redisOk without hiding a successful canonical publication. Failed canonical or seed-metadata writes include published: false, attemptedAt, and reason: "write-failed", and do not increment the source counter. Rejected zero-input cycles also include published: false and attemptedAt, use reason: "no-input-records", and do not replace the last-known-good envelopes or seed-meta:theater-posture.

Key Classifications

Keys are grouped into three tiers that determine alert severity:

Per-Key Statuses

Producers That Serve Last-Known-Good

Some producers are expected to miss individual runs — an LLM synthesis stage whose provider times out, for example — while the payload they last published stays useful for hours. For these keys a single miss is not an outage, so the producer does not write status: "error". Instead it holds fetchedAt at the vintage it is still serving, reports recordCount for those served records, and records the miss as diagnostics:

That independence has a ceiling. Once the streak reaches the key's warnAfterConsecutive — 2 for both keys today — the status itself becomes SEED_ERROR, so a streak of 1 is the largest one that can coexist with OK. </Warning>

Each such key declares thresholds sized to its own cadence, and health reports SEED_ERROR once either is crossed: warnAfterConsecutive misses in a row, or warnAfterAgeMin since the last attempt with a miss on record. Because fetchedAt is not advanced by a miss, the ordinary maxStaleMin age gate keeps escalating independently — a producer that stops entirely still ages into STALE_SEED. A producer that stops after recording a miss therefore reports SEED_ERROR while retained data is still available. A miss with nothing left to serve skips all of this and writes status: "error" immediately. What happens once the canonical data key itself disappears is not specific to this contract — see the fleet-wide rule below.

Two keys use this contract today: newsInsights and marketImplications.

Producer Faults vs. Missing Data

seed-meta outlives the data key it describes — 7 days against hours for most canonical keys — so a producer that faulted once and then stopped leaves health holding two true statements at the same time: the producer is unhappy and nothing is being served. Every fault signal is affected, not just the last-known-good contract above: status: "error", a non-ok sourceState, a blocked source, and a crossed failure streak.

The stronger verdict wins, and ties go to the fault — it is the only one of the pair that carries a cause. Concretely:

• A blank key whose absence is critical reports EMPTY (crit), never SEED_ERROR (warn). Without this rule a vanished homepage panel would report a warning for the seed-meta's full 7-day life.
• A blank key whose absence is not critical keeps SEED_ERROR. This covers keys where an empty payload is a valid state (OK/STALE_SEED), keys covered by a cascade sibling (OK_CASCADE), on-demand keys (EMPTY_ON_DEMAND), and keys inside a rollout window (ROLLOUT_PENDING) — none of which should be able to silence a fault the producer actually reported.

errorCode is published whenever a fault of the SEED_ERROR kind fired and the producer recorded a code — including when the missing-data verdict outranks that fault and the status ends up EMPTY. An escalation to crit never costs you the reason. (SOURCE_BLOCKED publishes no errorCode; its cause is the status itself.) lastSynthesisFailureCode is broader still: like the other last-known-good diagnostics above, it is published on every status whenever the producer recorded one.

The resilienceStaticIndex check also publishes failedDatasets when its static-index seed metadata names failed source adapters. This is a validated, deduplicated list of at most 50 adapter keys. A non-empty list makes the check SEED_ERROR and remains visible in the public ?compact=1 problem projection. The sibling resilienceStaticFao check shares the producer heartbeat but does not inherit failures for unrelated adapters.

JODI China Row

jodiOil, jodiGas and lngVulnerability publish an extra chinaRow block whenever their producer recorded one:

The block never moves the status. China's rows going unusable is an upstream fact no retry clears — every China row in both JODI files has carried ASSESSMENT_CODE 3 ("null/uncertain") since at least 2026-08 — so grading it would be a warning nobody can act on. What it must not be is silent: the seeders publish the other 50-57 countries regardless of China, and this block is what names the country that dropped out.

It describes the producer's last completed publish, like every other seed-meta-derived field: on a run that refused or failed, the previous block stays in place and ages alongside seedAgeMin. A producer that reported status: "error" publishes no block at all rather than relaying a verdict it never reached this run. The block is also operator-only — it names a country, so ?compact=1 strips it under the same rule as contentFreshness and the decision-group breakdown.

Whether the dataset is still advancing is a separate, graded signal: both seeders declare newestItemAt/maxContentAgeMin from the newest month a quorum of countries reports a measurement for, so a JODI file that stops publishing reads STALE_CONTENT rather than passing as fresh, and one fast-reporting country cannot vouch for a frozen file. Countries whose every field parsed to null are not published at all, so the seeders' country floors count coverage rather than rows.

Do not confuse checks.jodiGas.chinaRow with the checks.chinaCoverage entry below: the first is one country's row inside one source, the second is the fleet-wide China coverage summary.

China Coverage Projection

chinaCoverage projects the hourly Railway summary at health:china-coverage:v1. The evaluator checks each launched China contract for both a fresh producer heartbeat and fresh, substantive China content; a fresh seed cannot hide stale or missing source content. CHINA_DEGRADED is a warning projection for partial or stale coverage, while CHINA_UNAVAILABLE is critical when the summary is invalid or the launched content is unavailable.

The final public composition is monitored separately as chinaDecisionSignals. Its canonical payload must contain all six stable groups even when individual groups are explicitly unavailable. Health requires six group records and a seed no older than 60 minutes. Per-source transport details for policy, exchanges, and cross-Strait publishers remain visible only in this authenticated operator view; they are not copied into the public country summary or Pro MCP result.

The six required records count operationally covered groups, which is not the same as populated groups. A group whose state is unavailable with the cause healthy_quiet_window is covered: the upstream answered and simply had nothing qualifying to report, which is not a source failure and needs no operator action. Its public state and zero-item payload are unchanged — no event is invented to fill it. Every other unavailable cause (insufficient_data, provenance_rejected, upstream_unavailable, unknown) is a real failure and stays uncovered, and a cause that is absent or malformed fails closed as uncovered.

Both health surfaces publish a group breakdown so a shortfall names its own work item instead of reading as a bare 4/6: quietGroups (nothing to do), staleGroups (chase the source's content), and unavailableGroups, each carrying the group id and its unavailableCause.

Blocked China contracts remain visible in the audit with their stable reason code but are excluded from the strict launched-entry health count.

The launched cross-Strait activity contract audits the durable archive military:cross-strait-activity:v1 independently for producer transport and latest Taiwan MND reporting-window freshness. A fresh seed with a stale official report therefore remains degraded. Japan Joint Staff reviewed observations are regional augmentation and do not satisfy the Taiwan MND content requirement. /api/health separately monitors military:cross-strait-activity-bootstrap:v1; fresh canonical data cannot hide a missing compact UI projection. It also exposes dedicated MND and Japan Joint Staff transport records. Japan Joint Staff alone reports SOURCE_BLOCKED, and only when retained reviewed records exist and one of two evidenced conditions holds. HTTP_403 means the direct request and an upstream response received after a successful proxy CONNECT both returned HTTP 403 — the publisher itself refused both paths. PROXY_TARGET_FORBIDDEN means the direct request returned HTTP 403 and the proxy refused CONNECT for the target while a control CONNECT to a different contracted host succeeded in the same run through the same credentials — the proxy provider forbids this destination specifically, so no configured transport path exists. An uncorroborated CONNECT refusal, a control tunnel that also fails, PROXY_AUTH_FAILED, stale metadata, a missing source record, or any other source using the blocked state still fail closed through the existing STALE_SEED, EMPTY, or SEED_ERROR statuses. The distinction matters operationally: PROXY_TARGET_FORBIDDEN is durable and needs a different egress to change, whereas a bare CONNECT refusal is a proxy fault to remediate. Other current fetch failures report SEED_ERROR while the last-good archive remains available. The bundle freshness gate advances only after the archive, projection, and both source-health records publish successfully. The public bootstrap retains the bounded reason codes used for disclosure but omits proxy response diagnostics; full sanitized diagnostics remain in the authenticated operator source record.

Operators can obtain the same sanitized, read-only audit with node scripts/audit-china-coverage.mjs --json; add --strict to return a nonzero exit code unless every launched entry is healthy. The audit reads only the compact Redis contracts and emits status, age, and reason-code summaries— never credentials or raw upstream payloads.

Cascade Groups

Some keys use fallback chains. If any sibling has data, empty siblings report OK_CASCADE:

• Theater Posture: theaterPostureLive -> theaterPosture (stale) -> theaterPostureBackup
• Military Flights: militaryFlights -> militaryFlightsStale
• Displacement: displacement (current UTC year) -> displacementPrev (prior year, covers the Jan-1 window before the new-year seed runs)

riskScores is intentionally stricter than a raw feed heartbeat. Its recordCount is realtime signal-density coverage: the count of score-relevant Tier-1 conflict, news, and cyber signal families present during the CII refresh. The conflict family is satisfied by either the ACLED path or the UCDP event feed, matching the CII v8 scorer. When those feeds are reachable but quiet, riskScores can still report COVERAGE_PARTIAL; underlying feed freshness is tracked by the source-specific health entries where those feeds publish seed metadata.

portwatchPortActivity also uses minRecordCount. A fresh seed-meta:supply_chain:portwatch-ports record below 174 countries reports COVERAGE_PARTIAL instead of OK; partial runs may still refresh per-country PortWatch cache entries, but the canonical country list and healthy seed-meta signal do not advance until full 174-country coverage returns.

portwatchPortActivity additionally requires per-country content freshness, which is a different question from transport freshness and country cardinality. The seeder reuses a cached country payload whenever upstream max(date) has not advanced, so a run can report a fresh heartbeat and a complete 174/174 country list while an individual country's observation is days old. The producer therefore publishes a contentFreshness block, and the check verdict is:

The verdict keys on the decision-critical countries — currently CN and HK, the two the China corridor control towers read — rather than on all 174. Health pins both that set and the 144-hour budget in its own config rather than accepting whatever the producer declares, so a producer-side change cannot narrow the alarm scope or widen its threshold silently: dropping CN from the seeder's list, or publishing a 30-day budget, would otherwise report OK with China days stale — the exact failure this check exists to catch. A producer set that covers the pinned countries and adds more is accepted.

The producer's counts are a measurement taken at seeder-run time, and seed metadata is only rewritten on a canonical-advancing 12-hour run. Health therefore re-ages the oldest decision-critical observation against the current time rather than trusting the count, so an observation that was inside budget when the seeder measured it still alarms once it crosses the boundary between runs. criticalOldestAgeMinutes on the wire is the recomputed age, not the producer's. A block claiming every critical country is fresh but carrying no usable observation timestamp cannot be re-aged, so it reads as unusable.

unusableReasons names which condition failed (declared_scope_narrowed, fresh_exceeds_covered, critical_observation_time_unusable, …) so a consumer never has to reconstruct the verdict from the raw counts; expectedCriticalCountries publishes the pinned scope alongside the declared one.

Because the edge function redeploys within minutes of a merge while the producer is a 12-hour cron, the absent-block case had a bounded deployment-order grace. The durable marker seed-activated:supply_chain:portwatch-ports:content-freshness could grant it only inside the compiled window 2026-08-03T10:24:42Z → 2026-08-04T06:00:00Z: one complete producer interval plus six hours of scheduling slack after the schema shipped.

That window has closed, and the PortWatch grace is now permanently spent. A clean EXISTS=0 no longer softens anything: a missing PortWatch content block is COVERAGE_DEGRADED, unconditionally. The window is kept in source as the audit record of what was granted and until when — it is deliberately not reopened, because the producer has since published the block and re-granting the softening would undo exactly the bound #6111 asked for.

In steady state, therefore, contentFreshnessPendingUntil is not emitted at all. It appears only while some key's window is open, which today means only if a NEW entry is added to CONTENT_FRESHNESS_ROLLOUT in api/_content-freshness.js for a newly deployed content schema. Add one there and the deadline is published automatically on every surface below; do not extend the PortWatch entry.

While a window is open, the grace needs positive proof, not merely the lack of a marker. The EXISTS read is three-valued: read-and-present revokes the softening, read-and-absent grants it while the window is open, and a read that failed or returned a malformed pipeline entry is unknown state and grants nothing. A pending health entry publishes contentFreshnessPendingUntil; compact health responses repeat the same per-key deadlines in summary.contentFreshnessPendingUntil, so the bound is auditable without reading the implementation. /api/seed-health publishes the same deadline on its entry, and MCP applies the same shared window to its stale boolean while exposing the optional top-level contentFreshnessPendingUntil field in cache-tool output. The cache refresh path also refuses to serve a warm snapshot past this deadline. The softening covers absence only — a block that is present is always evaluated, and once the marker exists a block that disappears fails closed.

The cfg.activationKey pending-activation path in /api/seed-health (and the matching ON_DEMAND policy in /api/health) intentionally remains separate and does not reuse this PortWatch window. Those markers describe optional or operator-triggered producers for which “no metadata because the producer has never run” is the expected state, not a newly deployed data schema; there is no single producer cadence from which to derive a safe deadline. This path never softens a meta-bearing content-freshness failure. A scheduled content schema must use contentFreshnessActivation and its reviewed window instead. This closes the clean-absent activation gap in #6111.

Any check whose verdict rested on an unreadable marker carries activationUnknown: true, on both endpoints and in every status. Without it the payload is identical whether the marker read failed or the producer genuinely never published — two different remediations (check Upstash for per-command errors, versus check why the producer stopped writing). The flag reports which evidence the verdict used; it never softens or hardens anything by itself.

MCP cache tools publish the same flag on their envelope for the same reason. A tool that consults an activation marker and cannot read it returns activationUnknown: true alongside stale, so a caller can tell an unreadable marker from a producer that regressed — previously that failure was reported only to Sentry and the two were indistinguishable on the wire. Like contentFreshnessPendingUntil the field is optional: it is declared on every cache envelope, but only a tool whose freshness check names a contentFreshnessActivationKey can ever populate it.

The seeder reserves cold-fetch slots for the decision-critical countries so they refresh every run they are a cache miss. The content budget deliberately spans two full producer rotations: ceil(174 / 30) runs at a 12-hour cadence is about 72 hours, leaving 72 hours of headroom before the 144-hour alarm. The reservation still matters because it keeps CN/HK from waiting behind the rest of the bounded queue when many countries become misses together.

The refresh deadline uses the content clock, not the retrieval timestamp. A successful refetch of unchanged upstream data updates fetchedAt but carries contentAsOfChangedAt forward, so a frozen feed remains due for another decision-critical refresh and cannot hide behind a recent retrieval. If the upstream still does not advance, the check stays STALE_CONTENT: the content is genuinely old, the seeder cannot fix it, and the alarm reports an upstream outage rather than an internal rotation delay. Fleet-wide content staleness is normal by construction: the seeder refreshes at most 30 countries per 12-hour run, so a full sweep takes about 72 hours; the two-rotation budget leaves normal tail lag visible without making it an actionable failure. Alarming on the fleet-wide age would produce a warning that is always lit and never actionable. The fleet-wide counts (coveredCount, freshCount, staleCount, unknownCount, a bounded staleCountries list, and the oldest observation with its age) stay published for visibility; only the critical subset moves the status.

The 144-hour budget is deliberately the same one china-corridor-source-adapters.ts applies to a PortWatch observation, so this alarm explains the China activity-nowcast's marked_stale exclusion rather than contradicting it. Health is inclusive at the boundary and treats a future-dated observation as stale, both one step more conservative than the data gate — an alarm should fire no later than the contract it protects.

The MCP freshness envelope over the same seed-meta key carries this dimension too. get_chokepoint_status declares the identical pinned scope, budget, and activation marker, and calls the same assessor — so it ages the same contentAsOfChangedAt clock described above, and an MCP consumer cannot read stale: false for a key this endpoint calls STALE_CONTENT. The parity is asserted field-for-field against this config rather than left to a comment, and both surfaces read the activation marker as an EXISTS, so its stored value can never make them disagree.

/api/seed-health mirrors the same contract on its own supply_chain:portwatch-ports entry, reporting coverage_degraded where this endpoint reports COVERAGE_DEGRADED and stale_content where it reports STALE_CONTENT. All three surfaces read the marker three-valued and grant the grace only on a read-and-absent result. One test loop drives every marker outcome — read-present, read-absent, and unreadable — against the block shapes that decide the verdict (fresh, content-stale, absent, and present-but-unusable) through all three at once, pinning the expected verdict and the status-name mapping above rather than mere agreement between them.

An MCP caller still sees a single boolean: stale does not say which dimension failed, so /api/health remains the surface that names the stale country.

Both sides age the same clock, and it is not the retrieval timestamp. They read contentAsOfChangedAt, which advances only when the upstream's own max(date) advances, falling back to fetchedAt only for payloads written before that field existed. This matters because the seeder force-refetches every country once its cache passes MAX_CACHE_AGE_MS, and that refetch resets fetchedAt while returning unchanged content. Ageing fetchedAt would therefore have admitted a frozen observation as current for one budget window out of every cache lifetime — the same transport-for-content substitution this check exists to end, one layer below it.

Named entities are operator-only. contentFreshness and the chinaDecisionSignals group breakdown are stripped from the anonymous ?compact=1 projection, which keeps the status but not which source is degraded.

predictionMarkets also requires at least one published market in each geopolitical, tech, and finance pool. Its seed metadata publishes poolCounts; missing, malformed, or under-floor counts report COVERAGE_PARTIAL even when the aggregate market count is healthy.

Staleness Thresholds (maxStaleMin)

Selected thresholds from SEED_META:

These are illustrative; SEED_META in api/health.js is the source of truth and each entry documents its own cadence rationale.

Example Requests

# Full health check (requires an operator API key)
curl -s https://api.worldmonitor.app/api/health \
  -H "X-WorldMonitor-Key: $WORLDMONITOR_API_KEY" | jq .

# Compact (problems only)
curl -s "https://api.worldmonitor.app/api/health?compact=1" | jq .

# UptimeRobot / monitoring: check HTTP status code
curl -o /dev/null -s -w "%{http_code}" "https://api.worldmonitor.app/api/health?compact=1"
# Returns 503 only when Redis is unreachable (REDIS_DOWN); 200 for every other
# state — the verdict (HEALTHY/WARNING/DEGRADED/UNHEALTHY) is in the body's `status`.

/api/seed-health

Focused endpoint for seed loop freshness. Checks only seed-meta:* keys without fetching actual data payloads.

Authentication: Requires valid API key or allowed origin.

HTTP Method: GET

Response Status Codes

Response Body

{
  "overall": "healthy | warning | degraded",
  "checkedAt": 1710158400000,
  "seeds": {
    "seismology:earthquakes": {
      "status": "ok",
      "fetchedAt": 1710158100000,
      "recordCount": 142,
      "sourceVersion": null,
      "ageMinutes": 5,
      "stale": false
    },
    "market:stocks": {
      "status": "stale",
      "fetchedAt": 1710150000000,
      "recordCount": 85,
      "sourceVersion": null,
      "ageMinutes": 140,
      "stale": true
    },
    "supply_chain:portwatch-ports": {
      "status": "coverage_partial",
      "fetchedAt": 1710158100000,
      "recordCount": 139,
      "minRecordCount": 174,
      "sourceVersion": null,
      "ageMinutes": 5,
      "stale": true
    },
    "prediction:markets": {
      "status": "coverage_partial",
      "fetchedAt": 1710158100000,
      "recordCount": 87,
      "poolCounts": {
        "geopolitical": 52,
        "tech": 0,
        "finance": 35
      },
      "minPoolCounts": {
        "geopolitical": 1,
        "tech": 1,
        "finance": 1
      },
      "coveragePartial": true,
      "sourceVersion": null,
      "ageMinutes": 5,
      "stale": false
    }
  }
}

Staleness Logic

A seed is considered stale when its age exceeds 2x the configured interval. This accounts for normal jitter in cron/relay timing. Seeds below an aggregate minRecordCount report coverage_partial and stale: true. Seeds below a subgroup floor such as prediction-market minPoolCounts also report coverage_partial, but retain stale: false while their producer heartbeat remains fresh so freshness and coverage stay distinct.

Consumers: treat status and overall as authoritative for coverage. Do not rely on stale alone — pool shortfalls keep stale: false by design. When either coverage floor fails, the entry also sets coveragePartial: true so clients that only inspect booleans still see the shortfall.

History ingestion (intel-history:*)

Domains prefixed intel-history: do not describe a seeder's canonical publish. They track whether that collector's post-publish append to the historical intelligence store is still landing:

The append is fail-open by design — the canonical publish has already committed when it runs, so a failure must never fail the run. That means the collector's own entry (conflict:acled-intel, …) stays ok while history silently stops accumulating. These entries are the separate signal:

• fetchedAt is the last healthy append, never the last attempt. A run that reached the relay advances it; a run that delivered nothing (every chunk rejected, or the wall-clock budget died before the first request) does not. So a broken relay freezes it and the entry goes stale on the ordinary 2x-interval rule.
• status: "error" means the append failed on two consecutive runs — or, on the very first tick, that a relay credential present during an earlier successful append has since been removed.
• lastErrorCode names the cause when there is one: http_401, budget_exhausted, all_chunks_failed, config_removed, or a clamped error-class name. Absent when the ingest has never failed.
• status: "not_configured" means this deployment has never had relay credentials. It is visible but never an alarm — no operator action clears it except provisioning the relay. Losing credentials after a successful append is not this state; it reports error with lastErrorCode: "config_removed".
• recordCount is the volume the relay accepted on the last successful append. Zero is valid: a run whose records were all deduped still proves the pipeline works.

The richer per-run detail — lastErrorReason, consecutiveFailures, missingConfig, and the inserted/deduped/abandoned counts — is not returned by either endpoint. It lives in the Redis record intel-history:ingest-health:<domain>:<resource>:v1, which the endpoints project from.

A stale or error here alongside an ok collector means canonical data is fine and the history store is the thing to investigate.

Example Request

curl -s https://api.worldmonitor.app/api/seed-health \
  -H "Origin: https://worldmonitor.app" | jq .

Integration with Monitoring Tools

UptimeRobot

Use /api/health?compact=1 as the public monitor URL. The HTTP status code only distinguishes a total Redis outage from everything else:

• 503 = REDIS_DOWN (Redis unreachable — a true hard outage)
• 200 = every other state, including DEGRADED and UNHEALTHY

So an HTTP-status-only monitor catches a full backend outage but not degraded/unhealthy data. For those, add a keyword monitor.

Point the keyword monitor at https://api.worldmonitor.app/api/health?compact=1 and alert when the compact token "status":"HEALTHY" (no space after the colon) is absent from the response body. Compact mode serializes with no indentation, so this exact token is stable regardless of formatting.

The bare /api/health URL is now an operator view and returns 401 without an API key. Public monitoring should always use ?compact=1.

Custom Alerting

Parse the JSON response to build granular alerts:

# Alert on any critical keys
STATUS=$(curl -s "https://api.worldmonitor.app/api/health?compact=1")
CRIT=$(echo "$STATUS" | jq '.summary.crit')
if [ "$CRIT" -gt 0 ]; then
  echo "CRITICAL: $CRIT data keys empty"
  echo "$STATUS" | jq '.problems'
fi

Differences Between Endpoints