kura/docs/architecture.md
This document explains how Kura works, starting from a very high level and going deeper as you read on. Skim the first sections for an overview; read further for the runtime, replication, and rollout details.
Kura is a Rust service that builds low-latency cache meshes. A mesh is a small set of Kura nodes that each serve cache traffic from local disk and replicate writes to one another in the background. Clients (Bazel, Buck2, Xcode, Gradle, Tuist Module Cache, Nx, Metro) talk to whichever node is closest. Reads come back fast because they are local; writes propagate to peers asynchronously.
The project name comes from the Japanese word 蔵 ("storehouse"). The role of a node fits the name: keep artifacts and metadata stored durably and close at hand.
Build caches are read-heavy and latency-sensitive. A central cache hundreds of milliseconds away wastes more time than it saves. Kura puts a writable cache node next to each cluster of clients and keeps the nodes loosely consistent in the background. There is no leader, no global lock, and no synchronous fan-out on the hot path.
clients (Bazel/Buck2/Xcode/Gradle/...)
│
▼
┌──────────────────────────────────────────┐
│ Kura node (region X) │
│ │
│ co-hosted HTTP + gRPC (REAPI) │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ request handlers │ │
│ └────────┬─────────┬───────┘ │
│ │ │ │
│ ┌────────▼─┐ ┌───▼──────────┐ │
│ │ RocksDB │ │ segment files│ │
│ │ metadata │ │ (blob bodies)│ │
│ │ + outbox │ │ │ │
│ └────┬─────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ outbox worker ──► internal HTTP ──► peers
│ │
│ membership/discovery worker ◄── peers' │
│ /_internal/status
└──────────────────────────────────────────┘
Each node owns one persistent volume, runs one writer process, and exchanges traffic with peers over a separate internal plane that can be optionally protected with mTLS.
| Concern | Code |
|---|---|
| Process entry, server wiring | src/main.rs, src/app.rs |
| Public HTTP + gRPC handlers, readiness/rollout endpoints | src/http.rs, src/reapi/ |
| Storage (metadata, outbox, segments) | src/store.rs |
| Replication (membership, bootstrap, outbox processing) | src/replication/ |
| Cluster membership and readiness state | src/state.rs |
| Traffic state, drain, single-writer fencing | src/runtime.rs |
| Configuration + auto-derived defaults | src/config.rs, src/constants.rs |
| Metrics, traces, logs, error reporting | src/metrics.rs, src/telemetry.rs, src/analytics.rs |
| Usage metering | src/usage.rs |
| Optional Lua extension hook | src/extension.rs |
| Helm chart, rollout scripts, observability config | ops/ |
| End-to-end and shell-based tests | test/e2e/, spec/e2e/ |
Kura splits durable state into two planes so that the hot path is simple and the cold path can compact freely:
The metadata store uses tunable RocksDB budgets (KURA_METADATA_STORE_*) that auto-derive from the host's memory and FD limits.
Every public HTTP cache write and read is scoped by tenant_id, with an optional namespace_id. Namespace-scoped requests land in that namespace directly. Tenant-scoped requests omit namespace_id and Kura stores them under an internal empty namespace key, so policy hooks can still distinguish tenant-only traffic from project-like traffic without a special reserved namespace.
Kura treats memory as an admission-controlled shared resource, not just a set of independent caches:
httparse without consuming bytes. Only known artifact GET routes that match the local tenant, pass extension access checks, and resolve to a file-backed local artifact are consumed by the accelerator. Everything else, including HTTPS, HTTP/2, non-GET requests, inline artifacts, cold misses, unsupported routes, saturated accelerator capacity, and non-Linux builds, falls through to the normal Axum/Hyper path. Accelerated transfers are bounded by KURA_ACCELERATED_FILE_SERVING_MAX_CONCURRENT and use splice by default, with sendfile available as a runtime mode. Accelerated responses are framed with content-length and keep the connection alive, so a client can pipeline further requests on the same socket; the accelerator only forces connection: close when the client requests it or sends an unconsumed request body, and an idle reused connection is dropped after a bounded keep-alive timeout. A follow-up request that is not accelerable is handed back to the Axum/Hyper path mid-connection without consuming its bytes.Bytes chunks for file-backed artifacts only when a bounded memory budget is available and the region is already resident in the page cache (checked with mincore), so hot fallback responses avoid copying artifact bytes into heap buffers. Hot mmap responses are yielded in 1 MiB chunks to keep Hyper body overhead low. Cold or budget-constrained reads fall back to the streaming reader path, which isolates blocking reads with spawn_blocking and keeps each read at 512 KiB. REAPI ByteStream reads stay streaming, so they do not materialize whole artifacts in memory.BatchReadBlobs, GetActionResult inline expansions, action-cache proto loads) use two gates:
The pressure sensor reads the container charge every 200 milliseconds and removes clean file-backed cache before evaluating admission pressure. Shared memory, dirty file pages, and pages being written back remain charged. This avoids pressure-state changes when the kernel merely reclassifies clean artifact pages between active and inactive cache. A separate cache-reclaim signal activates on either of two arms. The working-set arm runs through the same hysteretic pressure state machine as admission, so a working set that reaches the soft watermark turns reclaim on and only clears once it recovers roughly 10% below that watermark; this stops mmap serving and drop-behind from flipping per sample as the kernel reclassifies clean artifact pages. The container-charge arm turns reclaim on whenever raw memory.current reaches the hard watermark and is deliberately not hysteresed: on a warm serving node the kernel keeps clean page cache charged until forced to reclaim, so once the cumulative footprint crosses the hard watermark this arm stays effectively steady state, holding drop-behind and mmap-serving denial on so request serving keeps borrowing from clean file cache rather than the container sitting close to its limit. Either arm disables mapped-file serving and makes accelerated reads and foreground writes release completed file ranges, allowing demand to trade file-cache warmth for request capacity without constraining admission. The pressure charge that drives admission counts anonymous, kernel, shared, socket-buffer, dirty, and writeback memory while excluding clean file-backed cache. Raw memory.current, the conventional working set, the pressure charge, memory events, and the heap/file/kernel breakdown remain metrics. A separate one-second actuator performs lock-taking heap-cache cleanup, keeping cleanup latency away from pressure observation. Admission has one enforceable invariant: live permits for unreclaimable transient work never exceed the fixed soft-to-hard watermark gap. Waiting acquisitions use Tokio's fair queue, a permit follows the allocation or transfer that owns its bytes, and code that already holds a permit may only grow it with a non-blocking attempt. Sampled usage never enters this equation; it trims optional caches, gates background work, and pauses the usage outbox. Mapped-file serving has a separate try-only bound because it covers already-resident reclaimable pages and always falls back to streaming. The controller bounds admitted work, not untracked allocations by RocksDB, the network stack, or allocator fragmentation. The allocator runs one background page-reclamation thread with a four-second dirty and muzzy-page decay; startup fails if that configuration is not active, preventing unused allocator pages from holding an idle node in critical pressure after a burst.
Normal artifact and ByteStream response readers use weighted sublimits within the shared transient budget. The response-stream capacity scales with both the soft-to-hard watermark gap and the hard-watermark-to-runtime-limit gap instead of stopping at a fixed absolute ceiling. Full-size and degraded response streams can therefore use memory released by pressure-driven cache trimming while their combined worst-case reservation remains inside the runtime reserve. File-backed Hyper responses reserve four reader, encoding, and transport buffers sized from 8 KiB to 512 KiB according to the remaining response. Inline artifacts also reserve their complete value. Materialized Remote Execution responses reserve the source payload and encoded transport copy. The reservation is acquired before opening a reader and follows the response through Tonic encoding and every outgoing byte buffer, releasing only when the last transport owner drops. A public read that cannot reserve its full buffers promptly can use a bounded degraded slot, but still charges the complete 512 KiB per-stream transport send-buffer cost. Internal bootstrap responses use a non-queuing background sublimit and cannot bypass a waiting public response. They remain admissible under constrained pressure so a stable working set above the soft watermark cannot permanently stop mesh convergence, but critical pressure still sheds them.
Disk-backed foreground writes use a source-plus-destination lease of up to 32 MiB, reduced to fit the fixed transient budget on smaller profiles. A body larger than the active window uses an 8 MiB synchronized file-cache window in both its temporary staging file and append-only segment. A smaller write keeps its warm staging pages on the uncontended path, but switches to the same bounded policy after it has queued for permits or overlaps another foreground reservation. The owned permit follows the request through staging, multipart assembly when applicable, and segment persistence. At each completed range Kura synchronizes and closes the writer before issuing page-aligned DONTNEED advice through Rustix, then resumes with a fresh append-only descriptor; this preserves later buffered bytes and the append-only segment invariant. Multipart parts synchronize and release their clean pages while waiting for completion, and assembly releases each input part after it is copied. ByteStream preserves the existing 64 MiB decode limit with an admission body in front of Tonic. The body scans every five-byte gRPC envelope header, including headers split across transport frames or following another message in the same frame, and non-blockingly grows the stream permit to twice the largest encoded message observed before forwarding that header to Tonic. After the first resource name reveals the blob size, the handler adds its bounded staging and segment file-cache window. Failed growth surfaces RESOURCE_EXHAUSTED immediately and never waits while consuming shared HTTP/2 connection flow-control. Other foreground uploads retain the 30-second admission deadline so bounded work can queue briefly without waiting indefinitely.
When a request would exceed either REAPI response gate, Kura returns RESOURCE_EXHAUSTED instead of continuing toward an out-of-memory path. Upload admission waits for bounded headroom and returns RESOURCE_EXHAUSTED or 503 Service Unavailable only when its deadline expires. Under constrained pressure Kura pauses new bootstrap and the snapshot build's manifest scan and action-result load, and halves retained optional caches. The snapshot presence gate still runs: it is correctness, not background work, so a sustained pressure window cannot freeze the served snapshot advertising blobs CAS eviction has since removed. Under critical pressure it trims opportunistic caches to zero and clears extension authorization and authentication caches because they are performance state, not correctness state. Replication delivery is deliberately exempt at every tier, and also through the container hard-limit arm. The outbox is depth-capped and a full outbox fails cache writes, so pausing the drain does not defer work — it strands the queue and ends up rejecting writes, leaving the node divergent from peers for as long as they can accept its deliveries. The state that walks into the cap is the hard-limit arm below a critical tier, where writes are still admitted while the drain is held: both write gates test only the critical tier, and test it before outbox depth, so at critical the outbox is frozen rather than growing. The memory a pause could reclaim does not justify either case — the drain loop is serial and node-wide, so exactly one delivery is in flight regardless of peer count or backlog depth (a queued message costs RocksDB, not RAM), and it takes no transient reservation; that delivery holds one 512 KiB segment-read chunk, or for an inline artifact the whole value, bounded by the 4 MiB inline ceiling. The usage (metering) outbox has no such feedback and stays sheddable.
Replication is leaderless and eventually consistent:
OutboxMessage in RocksDB inside the same atomic batch as the metadata commit.dropped_stale_target replication results): the fetched peer view is authoritative and the control plane withholds a peer only after a full staleness window of missed heartbeats, so the removal is deliberate — and a peer that later rejoins does so through a recovery re-enrollment that re-bootstraps the full dataset, so dropped deltas are recovered. Targets known only through discovery (in-cluster siblings, cross-region pods) are treated like the static seeds and never pruned within a process lifetime: their absence usually means a network flap rather than departure, and nothing re-bootstraps them afterwards. The protection is process-scoped — a genuinely removed pod (scale-down, region move) is never rediscovered after the observer's next restart, and since enqueues stop within one membership tick of unreachability, its small frozen backlog is dropped after the next deploy. An empty peer view never prunes — it means the node has no view (control plane unreachable), not that every peer left.KURA_REPLICATION_BANDWIDTH_LIMIT_BYTES_PER_SECOND. The limiter is shared per node across live replication uploads, replication ingests, and bootstrap artifact fetches/responses. The configured value is a ceiling; the effective sync rate is divided by the larger of public_inflight + 1 and the recent public request latency EWMA over KURA_REPLICATION_PUBLIC_LATENCY_TARGET_MS. The latency EWMA is sampled at time-to-first-byte (when the response is ready to start streaming), not at body completion, so large but healthy downloads do not register as latency and over-throttle sync; their concurrency is already captured by public_inflight. Public inflight includes non-probe public HTTP requests plus gRPC cache RPCs. Internal replication and probe requests do not count as public load. This lets sync work use its full budget while the node is quiet and back off automatically when public cache traffic is active or slow.Two observability surfaces support capacity and sharding decisions: kura_public_request_latency_seconds is a histogram of time-to-first-byte for public requests across both transports (transport is http or grpc, labeled by route), and kura_artifact_egress_throughput_bytes_per_second is a histogram of achieved per-response egress throughput by producer. Together with the aggregate kura_artifact_egress_bytes_total rate they indicate when a region is bandwidth-bound and a good candidate for sharding across more primary pods.
version_ms (last-writer-wins per key). A rejected apply reports which case it hit, surfaced as the outcome label on kura_replication_apply_results_total: ignored_stale when the incoming version is strictly older than the stored one (a genuine LWW rejection — the peer is behind on this key), ignored_equal when it equals the stored one (both sides already hold the identical entry, so the walk re-shipped converged data rather than reconciling skew), and — bootstrap only — ignored_missing when a peer advertised an artifact in a manifest page but 404s the body fetch (e.g. it evicted the segment between the two round trips; no version comparison happened). ignored_equal has a structural baseline: bootstrapping from several overlapping peers and concurrent replication of one key both produce it by design (every fetch-gate/apply-lock loser lands there), so its diagnostic value is in the ratio against ignored_stale, not in being non-zero. The apply path takes a per-key write lock so that concurrent applies of the same key serialize: the first commits the manifest and the rest re-read it and short-circuit to IgnoredEqual rather than each appending their own copy to a segment. Without it, simultaneous applies of one key leave all but the last copy orphaned on disk.Newly joined nodes catch up by bootstrapping from a peer: paginated manifest and tombstone fetches followed by lazy artifact body fetches. Bootstrap re-uses the same apply paths as live replication, so the same conflict rules apply. A fresh node bootstraps from every known peer concurrently (so it captures artifacts that exist on only one peer), which means the same artifact is offered by several peers at once. Three layers keep that overlap bounded: a per-artifact fetch gate single-flights the body download so only the first peer-task to claim a key transfers it while the rest re-check presence and skip the network; a live-memory reservation admits only the transfer windows that fit below the soft watermark; and the per-key apply lock is the last-line guarantee that even a body that does get fetched twice is written to a segment only once. A peer that temporarily rejects an artifact fetch with rate-limit or service-unavailable backpressure is retried in place with bounded exponential delay and the peer's numeric Retry-After hint. The joining node releases its fetch gate and memory reservation before waiting, so another peer can satisfy the artifact and the failed source does not force a complete manifest pass to be walked again. Large transfers do not reserve their full object size because a valid object may approach the container limit. Instead, once pressure rises, completed temporary-file and append-only segment ranges are synchronized and released from Linux file cache in bounded intervals before the stream waits for recovery. The cache advice is an optional optimization and never changes storage correctness.
Before walking a peer's manifests, bootstrap runs range-digest anti-entropy so the walk costs O(delta) rather than O(peer dataset). The manifests column family is a sorted keyspace over artifact_id (a content/storage hash, identical across nodes for identical content); both nodes summarize it as per-prefix-bucket digests (GET /_internal/bootstrap/digest?prefix_len=3 → 4096 buckets of (prefix, count, hash), where the hash folds the sorted (artifact_id, version_ms) pairs so adds, removes, and version bumps all flip a bucket). The joining node diffs the peer's digest against its own and enumerates only the mismatching buckets — scoping the existing paginated walk to each divergent prefix range (&prefix=). For a mostly-in-sync pair this collapses a full-keyspace page walk into one digest exchange plus a handful of small range walks. The digest endpoint is additive and negotiated: a peer that returns 404 Not Found because it is one version behind falls back to the full walk. Manifest pages also negotiate across adjacent versions: a new node prefers 2048 manifests per request and retries once with the legacy 256-manifest limit when an older peer returns 400 Bad Request. Namespace tombstones stay at the legacy limit. These fallbacks keep mixed-version rollouts and rollbacks correct. The digest only decides which ranges to enumerate; apply semantics are unchanged, so a digest bug can at worst cause an unnecessary walk, never a wrong apply.
A per-peer bootstrap runs under a no-progress watchdog rather than a fixed total-runtime cap. KURA_BOOTSTRAP_TIMEOUT_MS is the maximum time the bootstrap may go without forward progress (a fetched page or applied artifact); a bootstrap that keeps advancing runs to completion however long that takes, and only a genuinely stalled one is abandoned and retried. A single wall-clock cap could never let a large cold pull finish — it would be killed and restarted from scratch every window — so a node whose backlog exceeds one window's worth of transfer would stay NotReady indefinitely.
Under KURA_BACKFILL_ENABLED the catch-up walker in src/backfill/ replaces bootstrap, applying peer records through the same LWW/outcome/index code paths — but with batched durability instead of the live paths' per-record fsyncs (which dominated cold-node catch-up: ~2 fsyncs × every record). Each spooled bodies batch applies in four phases: (1) every Present segmented body is appended to the active segment with no per-record sync while inline bodies stage in memory; (2) one group-commit fsync makes the batch's segment bytes durable (a segment that rotated out mid-batch was already fsynced by the rotation); (3) staged records commit in groups of up to BACKFILL_APPLY_GROUP_RECORDS (64) — each group re-runs the authoritative LWW/tombstone checks under the records' write locks and lands ONE shared non-sync WriteBatch, and each record's claim resolves right after its group's WriteBatch (a mid-commit failure leaves later groups unresolved for the re-list); (4) one synced WAL flush is the batch's durability barrier — ~2 fsyncs and ~ceil(records/64) WAL appends per batch instead of one-plus per record (a live measurement showed the per-record WAL appends alone saturating disk IOPS). Ordering matters: an unrelated concurrent sync write can flush the WAL at any moment, so segment bytes are always fsynced before any of the batch's manifests enters the WAL, preserving the live invariant that a durable manifest implies durable segment bytes. Losing the tail of un-flushed applies in a crash is absorbed by the pass contract — no pass completion means no watermark advance, so the restart re-lists the window and LWW absorbs replays — and the watermark written on completion is itself a sync commit through the same WAL, whose prefix-ordered sync guarantees every apply it covers is durable. Live replication, client writes, oversized per-artifact backfill fetches, and namespace tombstone applies (the latter two a handful per pass) keep per-record sync durability. See ApplyDurability in src/store.rs.
See src/replication/mod.rs for the membership/outbox/bootstrap loops, and src/replication/operation.rs + outbox_message.rs for the message types.
A node finds peers in three ways:
KURA_PEERS. Static config is immutable for the process lifetime, so it carries platform-stable peers only (the managed regions' public peer gateways); volatile membership lives in the dynamic layer below.src/mesh_heartbeat.rs): enrolled self-hosted nodes send a mesh heartbeat every ~60s (cadence server-advertised) whose response carries the current peer list; managed pods fetch the same view read-only when KURA_MESH_PEERS_SYNC is set, with serving gated on the first successful fetch (a pod booting blind would accept writes without enqueuing replication for peers it cannot see). Additions and removals propagate at heartbeat cadence with no restart; a failed heartbeat keeps the last-known view, so a degraded control plane can never shrink the mesh.KURA_DISCOVERY_DNS_NAME, which resolves to the addresses of the other pods (typical when running as a Kubernetes StatefulSet behind a headless service).A spawn_membership_task loop polls each candidate's GET /_internal/status every two seconds. Only peers that respond with the same tenant_id and a different node_url are admitted as members. The local node never lists itself.
Mesh membership itself is control-plane state for enrolled nodes: a node that stops sending mesh heartbeats is deactivated (withheld from every peer's view) and its row is purged once its peer certificate can no longer be valid. Heartbeats never create or restore membership — a withheld node is answered mesh_member: false and recovers with a recovery re-enrollment (backoff-limited), which reactivates or recreates its membership server-side, clears local bootstrap progress, and briefly takes the node out of serving. A node with usable local data can serve again after discovery settles while reconciliation continues; an empty node stays joining until the bootstrap completes. The writes missed while the node was out of the mesh were never enqueued for it (replication targets are computed at write time), so a full re-bootstrap still runs to reconcile the gap.
Each tick produces a MembershipUpdate and feeds it into ReadinessState (src/state.rs). The state tracks:
known_peers (peers that responded successfully),initial_discovery_completed only flips once we have actually observed a successful peer status check (or there are no discovery targets at all). This avoids promoting a node to "ready" when the seed peers were transiently unreachable.
A node moves through three explicit traffic states (src/runtime.rs). Ordinary membership changes do not demote a serving node; newly discovered or returning peers reconcile in the background:
┌──────────┐ restart or recovery ┌──────────┐
│ │ re-enrollment │ │
│ joining │ ◄──────────────────────────│ serving │
│ │ │ │
└────┬─────┘ └─────┬────┘
│ discovery settled and either │
│ local data exists or an empty node │
│ completed every peer bootstrap │
▼ ▼
┌──────────┐ drain request ┌──────────┐
│ serving │ ─────────────────────────► │ draining │
└──────────┘ └──────────┘
joining — public reads/writes are accepted but /ready returns 503. A node that entered the joining cycle with usable persistent data becomes ready after discovery settles and continues reconciliation in the background. A truly empty node waits until bootstrap completes for every known peer. Under KURA_BACKFILL_ENABLED the catch-up clause is different: after the same discovery gates, the node becomes ready once its segment ring is at least KURA_BACKFILL_READY_RING_PERCENT full or its initial backfill cycle has settled (zero discovered peers settle immediately; peers that exhaust their failure budget stop gating while background retries continue). Readiness then latches for the process lifetime — later re-join backfills and recovery re-enrollments never regress it — while the orthogonal inputs (writer lock, draining) still gate /ready. The /status/rollout report adds a backfill_initial_cycle mode (pending / complete / degraded) that consumers use instead of the legacy bootstrap counters.serving — /ready returns 200. Public APIs handle traffic normally.draining — public HTTP rejects new requests and stops reusing HTTP/1.1 connections; established HTTP/2 connections (gRPC included) receive a GOAWAY so channels finish in-flight streams and reconnect elsewhere. Inflight work continues until a shared drain deadline (KURA_DRAIN_COMPLETION_TIMEOUT_MS) elapses.Independent of draining, the co-hosted listener's hyper path recycles every connection after CONNECTION_MAX_AGE (300s): the server sends GOAWAY and allows in-flight streams CONNECTION_MAX_AGE_GRACE (900s) to finish before severing. Without recycling, a long-lived Bazel channel would pin to a demoted-but-alive NodePort primary indefinitely after failover. Both public listeners — plaintext and TLS, with acceleration on or off — share this per-connection serving path, so recycling and drain GOAWAY apply uniformly; the internal mTLS peer listener is a separate plane with its own lifecycle.
The action cache also serves an instance-wide snapshot through a reserved action key (tuist-actioncache-snapshot/v2, intercepted by digest comparison inside GetActionResult — no dedicated RPC): the response inlines the namespace's complete key→value map as a deduplicated node table plus per-key node-index lists and a write-time watermark, so a cold client primes every association in one round trip and fetches content through ordinary batched blob reads. Serving is backed by an incrementally maintained per-namespace index (bounded, LRU): reconciliation runs as a detached task shared by every concurrent request (a client or gateway that gives up on a slow first build cannot throw the work away — the build completes and caches regardless, and the next request serves from memory) and diffs the cached index against the newest slice of the action-cache keyspace; a request with no cached index waits only briefly for the build before answering UNAVAILABLE (the client stays on the per-key path and refetches shortly — pinning requests to a long first build walked every one of them into its deadline). Enumeration reads a dedicated action_cache_index column family ordered newest-first by write time (maintained at publish/delete, lazily backfilled per namespace with one legacy full scan) so a reconcile touches at most its entry cap rather than the whole namespace — the scan keeps only the most recent entries by write time, bounding the build's memory the way the byte ceiling bounds the response; only new-or-changed entries read their stored ActionResult — every referenced blob is presence-gated by manifest existence, which tracks eviction exactly, and the node table is compacted once entry churn strands enough unreferenced nodes. The gate runs on every reconcile regardless of memory pressure — only the action-result load is denied under pressure — so the served view stays honest through a pressure window instead of freezing stale (a frozen snapshot that references an evicted blob fails the build on the first missing object). A tuist-snapshot-after:<watermark> hint in inline_output_files returns a delta of entries written at or after the client's watermark (inclusive — millisecond versions are not unique, and re-sent boundary entries merge idempotently); deltas only add, so clients periodically refetch the full view to pick up retractions. The payload is capped at 48 MiB: the full view encodes newest-first (the ceiling sheds the oldest keys — a recency window) while a delta encodes oldest-first with the watermark set to the newest entry actually included, so an oversized delta paginates across refreshes instead of skipping what it dropped. A server without the feature answers a plain not-found the client degrades from — safe under mixed-version rollouts. Bump the key's version suffix on any encoding change.
Action-cache entries are also the one artifact class with their own lifecycle: clients publish new keys on every source change and the tiny records never face segment capacity pressure, so without intervention a namespace's keyspace grows forever. Two mechanisms bound it, both node-local (peers apply the same rules over the replicated version_ms and converge on their own): the snapshot serve path cascade-deletes entries whose blobs were evicted (unserveable by construction; a grace window spares young entries whose blobs may still be mid-replication), and a periodic background sweep expires entries whose write time predates the TTL — an expired entry that is still used costs its next cold reader one recompile + republish, which refreshes it fleet-wide.
An action-cache entry and its output blobs form one integrity unit: the entry (a REAPI ActionResult) is meaningless once any blob it references leaves the CAS. The publish path already enforces this ordering (FindMissingBlobs → BatchUpdateBlobs → UpdateActionResult last), but blob eviction is capacity-driven and independent, so an entry could outlive its blobs and strand — served as a hit that then fails the client's blob fetch (CAS error: missing object), which clang-compiled targets do not tolerate mid-build the way Swift does. To close that at the source, eviction honors the same invariant: a node-local, derived reverse index (individual blob_ref/{blob}\0{entry} pair rows kept in the existing key_value column family — prefixed rather than given a dedicated column family so a rollback to a binary that predates it can still reopen the database — maintained in the same atomic batch as every entry write and delete, mirroring the segment_artifacts shape) lets evict_segment cascade — dropping a blob removes, in one batch, every action-cache entry that references it, re-validating each candidate against the entry's current ActionResult so a reverse row left stale by a re-publish only deletes itself. The map is purely derived (never replicated) and rebuilt once at startup from the entries on disk, which widens coverage to entries that predate it. That backfill does not gate the cascade: it waits on background headroom, and a warm node holding clean page cache at the hard watermark never grants it, so gating on it left the cascade inert in production across every eviction sweep. The cascade instead acts on whatever rows exist (the write path maintains them for every entry written since boot) and re-validates each one, so an incomplete map can only under-cascade, never remove a live entry. KURA_ACTION_CACHE_EVICTION_CASCADE_ENABLED is the only gate. The serve-side presence gates (per-key GetActionResult and the snapshot reconcile) stay on regardless: they remain correct under the residual publish-versus-eviction race — where an entry commits just after eviction scanned its blob and returns NOT_FOUND, which is an expected client outcome — and they are the backstop the cascade lifts load from rather than replaces. A symmetric race exists in the other direction: the cascade validates and stages an entry's deletion without the per-entry write lock (it runs from segment rotation while that path already holds a write-lock stripe, so taking the lock could self-deadlock), so a re-publish that commits a newer version between the validation read and the batch write has that fresh version removed. Every branch is fail-safe and self-healing — the entry degrades to a NOT_FOUND the client recomputes and republishes, the reconcile's stale-row cleanup drops the dangling index row, and the newer reverse rows are re-validated as stale on the next eviction — so the cascade must never be assumed to leave a live newer version in place. Kura deliberately does not refuse an entry whose blobs are momentarily absent at commit, because metadata-lane replication intentionally delivers fresh action-cache entries ahead of the bulk-blob backlog, so a receiving node legitimately sees an entry before its blobs.
The REAPI gRPC services (src/reapi/mod.rs) are mounted into the co-hosted listener rather than a dedicated gRPC server (reapi::routes returns an axum::Router that run_with_config merges with the HTTP router). The listener advertises raised HTTP/2 flow-control windows — a 4 MiB stream window and a 16 MiB connection window — so a single large ByteStream.Write is not throttled to roughly window / RTT under WAN latency (without them the kura hop becomes the next bottleneck after the gateway nginx window). The window is FIXED, never adaptive: hyper's adaptive flow control would override the fixed size and ramp a single stream up from ~64 KiB, halving single-stream upload throughput under WAN latency. A per-upload stall timeout (60s, keyed on byte progress so trickled keepalive frames cannot hold a stalled stream open) reclaims a vanished or stalled writer without cutting an upload that keeps making progress. Temporary files also carry drop guards, so transport cancellation and bootstrap watchdog cancellation schedule partial-file removal on Tokio's blocking pool even when they drop a future before its asynchronous cleanup path runs.
/up is a liveness signal that does not depend on any of this — it stays healthy as long as the process is alive.
Each PVC is owned by exactly one Kura process. On startup, DataDirLock (src/runtime.rs) takes an OS file lock on .kura.writer.lock inside KURA_DATA_DIR. If another process holds the lock, startup fails fast. Public readiness depends on the lock being held, so a node that loses the lock cannot serve traffic.
The Kubernetes layer reinforces this with ReadWriteOncePod PVC access by default; the app-level lock is the source of truth and works even when the CSI driver only supports ReadWriteOnce.
Rollouts are deliberately conservative because each node is stateful and serves cache traffic that should not regress to misses during an upgrade.
The pieces:
preStop hook sends SIGUSR1 to start drain. The pod stays alive long enough to finish inflight work, then exits.terminationGracePeriodSeconds is computed from the application's own drain timeout plus small lifecycle buffers, so Kubernetes never cuts shutdown short.ops/rollout/gate.sh) polls each node's /status/rollout and only advances when every node reports the same membership generation, all are back in serving, ring size matches, no bootstrap is inflight, the outbox is near baseline, no node is under critical memory pressure, and there is no new file-descriptor timeout activity.ops/helm/kura/rollout.sh) stages the new revision behind a StatefulSet partition, rolls the highest ordinal first, and delegates health gating to the generic gate. The adapter is a thin transport layer; it does not own rollout semantics.test/e2e/kura_compatibility_rollout.sh) validates PREVIOUS_REF → HEAD → PREVIOUS_REF on the same persistent Docker volumes for the artifact CAS path.The rollout gate explicitly assumes only that it can fetch /status/rollout from each node. It does not depend on Kubernetes probes or Prometheus.
Each node exposes:
/metrics (replication latency, FD pressure, manifest cache, RocksDB internals, outbox depth, traffic state, rollout-relevant counters).kura_http_client_requests_total, and kura_http_request_duration_seconds aggregates public non-probe latency without a route label to avoid multiplying route cardinality by histogram buckets.KURA_CONTROL_PLANE_URL and client credentials. Kura aggregates bytes and request counts into bounded in-memory windows, persists closed windows into a dedicated RocksDB usage outbox, and pushes batches to /_internal/kura/usage. Delivery pauses under critical memory pressure and is at least once, with deterministic event ids for control-plane deduplication. Both transports are metered and tagged by protocol: the HTTP cache path emits protocol = "http", and the REAPI (gRPC) path emits protocol = "grpc" with artifact_kind = "reapi". A batch RPC (BatchReadBlobs/BatchUpdateBlobs) books one request carrying the aggregate bytes, matching the one-request-per-call accounting of the HTTP and ByteStream paths, and re-uploads of an already-present blob are not billed.X-Forwarded-For / X-Real-IP headers and resolved against the DB-IP Lite City MMDB vendored into the container image at /opt/geoip/dbip-city-lite.mmdb (src/geoip.rs). Country (ISO 3166-1) lands on both the kura_http_client_requests_total metric (as the client_country label) and http.request spans (as geo.country.iso_code); subdivision (ISO 3166-2) lands on spans only (as geo.region.iso_code), deliberately kept off metrics to bound label cardinality. Span and Resource attributes follow the OpenTelemetry geo.* semantic conventions. On by default; soft-fails when the database file is absent. A background task refreshes the in-memory copy every KURA_GEOIP_REFRESH_INTERVAL_SECS seconds (default 86400, 0 disables) with download/decompress sizes capped (128 MiB / 256 MiB) to stay within Kura's resource discipline.src/node_location.rs), probing the egress IP only when the operator overrides do not already cover the missing fields. Country falls back to an explicit mapping for the synthetic deployment labels we use today (eu-central -> DE, us-east / us-west -> US) or to a real country prefix already embedded in the region label (fr-par -> FR, nl-ams -> NL). The result stamps geo.country.iso_code and geo.region.iso_code on the OTel Resource (next to the unchanged kura.region cloud deployment region) so every span carries them, and also lands on the low-cardinality kura_node_geo_info metric for Grafana maps. Combined with the request-side geo.country.iso_code / geo.region.iso_code, traces have both endpoints needed to compute geographic distance.tracing::error! events.Helm and the local docker-compose stack ship a complete Grafana/Prometheus/Loki/Tempo setup. See ops/AGENTS.md for layout.
All configuration is environment-driven (src/config.rs). The full table lives in README.md. Highlights:
KURA_TENANT_ID, KURA_REGION, KURA_NODE_URL, KURA_PORT, KURA_INTERNAL_PORT, KURA_DATA_DIR, KURA_TMP_DIR.reapi::routes, everything else to the HTTP router), so a single client-facing URL speaks both protocols. It serves plaintext on KURA_PORT and — when public_tls (KURA_PUBLIC_TLS_*) is configured — TLS on KURA_HTTPS_PORT, ALPN-negotiated (h2 for gRPC, http/1.1 for HTTP). The plaintext listener runs through the accelerated server, so HTTP/1 artifact GETs get the sendfile/splice fast path while gRPC (h2c) and other non-accelerable requests fall through to hyper; the TLS listener uses the plain hyper path (TLS is incompatible with sendfile). Both use the fixed REAPI-sized HTTP/2 windows so co-hosted uploads are not throttled.KURA_PEERS, KURA_DISCOVERY_DNS_NAME, optional KURA_INTERNAL_TLS_* for peer mTLS.KURA_REPLICATION_BANDWIDTH_LIMIT_BYTES_PER_SECOND sets the aggregate peer artifact body traffic ceiling per node when set above 0; Kura adapts the effective rate downward under public HTTP or gRPC load, and KURA_REPLICATION_PUBLIC_LATENCY_TARGET_MS controls the latency target for additional backoff.KURA_ACCELERATED_FILE_SERVING_ENABLED, KURA_ACCELERATED_FILE_SERVING_MODE, KURA_ACCELERATED_FILE_SERVING_MAX_CONCURRENT, and KURA_ACCELERATED_FILE_SERVING_CHUNK_BYTES bound the Linux plaintext HTTP/1 artifact fast path while preserving the Axum/Hyper fallback path on the same public port.KURA_DRAIN_COMPLETION_TIMEOUT_MS.When budget vars are unset Kura inspects RLIMIT_NOFILE, the cgroup memory limit, and detected CPU count to pick safe defaults.
src/http.rs and src/reapi/mod.rs.src/store.rs is the single entry point.src/replication/mod.rs and src/state.rs.ops/helm/kura/ and ops/rollout/gate.sh.spec/e2e/ exercises the live stack.