kura/docs/architecture.md
This document explains how Kura works, starting from a very high level and going deeper as you read on. Skim the first sections for an overview; read further for the runtime, replication, and rollout details.
Kura is a Rust service that builds low-latency cache meshes. A mesh is a small set of Kura nodes that each serve cache traffic from local disk and replicate writes to one another in the background. Clients (Bazel, Buck2, Xcode, Gradle, Tuist Module Cache, Nx, Metro) talk to whichever node is closest. Reads come back fast because they are local; writes propagate to peers asynchronously.
The project name comes from the Japanese word 蔵 ("storehouse"). The role of a node fits the name: keep artifacts and metadata stored durably and close at hand.
Build caches are read-heavy and latency-sensitive. A central cache hundreds of milliseconds away wastes more time than it saves. Kura puts a writable cache node next to each cluster of clients and keeps the nodes loosely consistent in the background. There is no leader, no global lock, and no synchronous fan-out on the hot path.
clients (Bazel/Buck2/Xcode/Gradle/...)
│
▼
┌──────────────────────────────────────────┐
│ Kura node (region X) │
│ │
│ co-hosted HTTP + gRPC (REAPI) │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ request handlers │ │
│ └────────┬─────────┬───────┘ │
│ │ │ │
│ ┌────────▼─┐ ┌───▼──────────┐ │
│ │ RocksDB │ │ segment files│ │
│ │ metadata │ │ (blob bodies)│ │
│ │ + outbox │ │ │ │
│ └────┬─────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ outbox worker ──► internal HTTP ──► peers
│ │
│ membership/discovery worker ◄── peers' │
│ /_internal/status
└──────────────────────────────────────────┘
Each node owns one persistent volume, runs one writer process, and exchanges traffic with peers over a separate internal plane that can be optionally protected with mTLS.
| Concern | Code |
|---|---|
| Process entry, server wiring | src/main.rs, src/app.rs |
| Public HTTP + gRPC handlers, readiness/rollout endpoints | src/http.rs, src/reapi/ |
| Storage (metadata, outbox, segments) | src/store.rs |
| Replication (membership, outbox processing) | src/replication/ |
| Peer catch-up walker (backfill) | src/backfill/ |
| Cluster membership and readiness state | src/state.rs |
| Traffic state, drain, single-writer fencing | src/runtime.rs |
| Configuration + auto-derived defaults | src/config.rs, src/constants.rs |
| Metrics, traces, logs, error reporting | src/metrics.rs, src/telemetry.rs, src/analytics.rs |
| Usage metering | src/usage.rs |
| Cache authorization | src/auth/ |
| Helm chart, rollout scripts, observability config | ops/ |
| End-to-end and shell-based tests | test/e2e/, spec/e2e/ |
Kura splits durable state into two planes so that the hot path is simple and the cold path can compact freely:
The metadata store uses tunable RocksDB budgets (KURA_METADATA_STORE_*) that auto-derive from the host's memory and FD limits.
Every public HTTP cache write and read is scoped by tenant_id, with an optional namespace_id. Namespace-scoped requests land in that namespace directly. Tenant-scoped requests omit namespace_id and Kura stores them under an internal empty namespace key, so policy hooks can still distinguish tenant-only traffic from project-like traffic without a special reserved namespace.
Kura treats memory as an admission-controlled shared resource, not just a set of independent caches:
httparse without consuming bytes. Only known artifact GET routes that match the local tenant, pass authorization, and resolve to a file-backed local artifact are consumed by the accelerator. Everything else, including HTTPS, HTTP/2, non-GET requests, inline artifacts, cold misses, unsupported routes, saturated accelerator capacity, and non-Linux builds, falls through to the normal Axum/Hyper path. Accelerated transfers are bounded by KURA_ACCELERATED_FILE_SERVING_MAX_CONCURRENT and use splice by default, with sendfile available as a runtime mode. Accelerated responses are framed with content-length and keep the connection alive, so a client can pipeline further requests on the same socket; the accelerator only forces connection: close when the client requests it or sends an unconsumed request body, and an idle reused connection is dropped after a bounded keep-alive timeout. A follow-up request that is not accelerable is handed back to the Axum/Hyper path mid-connection without consuming its bytes.Bytes chunks for file-backed artifacts only when a bounded memory budget is available and the region is already resident in the page cache (checked with mincore), so hot fallback responses avoid copying artifact bytes into heap buffers. Hot mmap responses are yielded in 1 MiB chunks to keep Hyper body overhead low. Cold or budget-constrained reads fall back to the streaming reader path, which isolates blocking reads with spawn_blocking and keeps each read at 512 KiB. REAPI ByteStream reads stay streaming, so they do not materialize whole artifacts in memory.BatchReadBlobs, GetActionResult inline expansions, action-cache proto loads) use two gates:
The pressure sensor reads the container charge every 200 milliseconds and removes clean file-backed cache before evaluating admission pressure. Shared memory, dirty file pages, and pages being written back remain charged. This avoids pressure-state changes when the kernel merely reclassifies clean artifact pages between active and inactive cache. A separate cache-reclaim signal activates on either of two arms. The working-set arm runs through the same hysteretic pressure state machine as admission, so a working set that reaches the soft watermark turns reclaim on and only clears once it recovers roughly 10% below that watermark; this stops mmap serving and drop-behind from flipping per sample as the kernel reclassifies clean artifact pages. The container-charge arm turns reclaim on whenever raw memory.current reaches the hard watermark and is deliberately not hysteresed: on a warm serving node the kernel keeps clean page cache charged until forced to reclaim, so once the cumulative footprint crosses the hard watermark this arm stays effectively steady state, holding drop-behind and mmap-serving denial on so request serving keeps borrowing from clean file cache rather than the container sitting close to its limit. Either arm disables mapped-file serving and makes accelerated reads and foreground writes release completed file ranges, allowing demand to trade file-cache warmth for request capacity without constraining admission. The pressure charge that drives admission counts anonymous, kernel, shared, socket-buffer, dirty, and writeback memory while excluding clean file-backed cache. Raw memory.current, the conventional working set, the pressure charge, memory events, and the heap/file/kernel breakdown remain metrics. A separate one-second actuator performs lock-taking heap-cache cleanup, keeping cleanup latency away from pressure observation. Admission has one enforceable invariant: live permits for unreclaimable transient work never exceed the fixed soft-to-hard watermark gap. Waiting acquisitions use Tokio's fair queue, a permit follows the allocation or transfer that owns its bytes, and code that already holds a permit may only grow it with a non-blocking attempt. Sampled usage never enters this equation; it trims optional caches, gates background work, and pauses the usage outbox. Mapped-file serving has a separate try-only bound because it covers already-resident reclaimable pages and always falls back to streaming. The controller bounds admitted work, not untracked allocations by RocksDB, the network stack, or allocator fragmentation. The allocator runs one background page-reclamation thread with a four-second dirty and muzzy-page decay; startup fails if that configuration is not active, preventing unused allocator pages from holding an idle node in critical pressure after a burst.
Normal artifact and ByteStream response readers use weighted sublimits within the shared transient budget. The response-stream capacity scales with both the soft-to-hard watermark gap and the hard-watermark-to-runtime-limit gap instead of stopping at a fixed absolute ceiling. Full-size and degraded response streams can therefore use memory released by pressure-driven cache trimming while their combined worst-case reservation remains inside the runtime reserve. File-backed Hyper responses reserve four reader, encoding, and transport buffers sized from 8 KiB to 512 KiB according to the remaining response. Inline artifacts also reserve their complete value. Materialized Remote Execution responses reserve the source payload and encoded transport copy. The reservation is acquired before opening a reader and follows the response through Tonic encoding and every outgoing byte buffer, releasing only when the last transport owner drops. A public read that cannot reserve its full buffers promptly can use a bounded degraded slot, but still charges the complete 512 KiB per-stream transport send-buffer cost. Internal backfill responses use a non-queuing background sublimit and cannot bypass a waiting public response. They remain admissible under constrained pressure so a stable working set above the soft watermark cannot permanently stop mesh convergence, but critical pressure still sheds them.
Disk-backed foreground writes use a source-plus-destination lease of up to 32 MiB, reduced to fit the fixed transient budget on smaller profiles. A body larger than the active window uses an 8 MiB synchronized file-cache window in both its temporary staging file and append-only segment. A smaller write keeps its warm staging pages on the uncontended path, but switches to the same bounded policy after it has queued for permits or overlaps another foreground reservation. The owned permit follows the request through staging, multipart assembly when applicable, and segment persistence. At each completed range Kura synchronizes and closes the writer before issuing page-aligned DONTNEED advice through Rustix, then resumes with a fresh append-only descriptor; this preserves later buffered bytes and the append-only segment invariant. Multipart parts synchronize and release their clean pages while waiting for completion, and assembly releases each input part after it is copied. ByteStream preserves the existing 64 MiB decode limit with an admission body in front of Tonic. The body scans every five-byte gRPC envelope header, including headers split across transport frames or following another message in the same frame, and non-blockingly grows the stream permit to twice the largest encoded message observed before forwarding that header to Tonic. After the first resource name reveals the blob size, the handler adds its bounded staging and segment file-cache window. Failed growth surfaces RESOURCE_EXHAUSTED immediately and never waits while consuming shared HTTP/2 connection flow-control. Other foreground uploads retain the 30-second admission deadline so bounded work can queue briefly without waiting indefinitely.
When a request would exceed either REAPI response gate, Kura returns RESOURCE_EXHAUSTED instead of continuing toward an out-of-memory path. Upload admission waits for bounded headroom and returns RESOURCE_EXHAUSTED or 503 Service Unavailable only when its deadline expires. Under constrained pressure Kura pauses new backfill passes, the snapshot build's manifest scan and action-result load, serve-path segment refresh (read-triggered lifetime extension keeps running at this tier; see the action-cache integrity paragraph below), and manifest-cache admission, and halves retained optional caches. Every admission gate tests only the pressure tier; none consults the raw container charge. Raw memory.current is dominated by reclaimable clean file cache on a warm serving node and parks at the hard watermark as steady state — the kernel reclaims it only under allocation demand, and only as much as the allocation needs — so a raw-charge admission gate closes shortly after boot and never reopens. An earlier arm that gated background admission on the raw charge latched exactly that way, starving backfill, segment refresh, and the usage outbox for the life of the process, and was retired: the tier already tracks the memory that can actually kill the container (anonymous, unreclaimable kernel, shared, socket-buffer, dirty, and writeback bytes), and background work admitted against a charge-full but reclaimable cgroup forces the kernel to hand clean cache back, trading cache warmth for progress rather than safety. The raw charge still drives the cache-reclaim serving mode described above, where staying latched on is the intended behaviour. Transitions of backfill scheduling into and out of the memory-blocked state are logged. The snapshot presence gate still runs: it is correctness, not background work, so a sustained pressure window cannot freeze the served snapshot advertising blobs CAS eviction has since removed. Under critical pressure it trims opportunistic caches to zero and clears the authorization cache, which also drops the confirmed access levels the engine reuses when the control plane cannot be reached, because they are performance state, not correctness state — dropping them only costs the node its cover for a control-plane outage, which fails closed. Replication delivery is deliberately exempt at every tier. The outbox is depth-capped and a full outbox fails cache writes, so pausing the drain does not defer work — it strands the queue and ends up rejecting writes, leaving the node divergent from peers for as long as they can accept its deliveries. Both write gates test only the critical tier, and test it before outbox depth, so any pause below critical would hold the drain while writes keep arriving and walk the queue into its cap, while at critical the write gates already refuse work at the door and the outbox is frozen rather than growing. The memory a pause could reclaim does not justify either case — the drain loop is serial and node-wide, so exactly one delivery is in flight regardless of peer count or backlog depth (a queued message costs RocksDB, not RAM), and it takes no transient reservation; that delivery holds one 512 KiB segment-read chunk, or for an inline artifact the whole value, bounded by the 4 MiB inline ceiling. The usage (metering) outbox has no such feedback and stays sheddable.
Replication is leaderless and eventually consistent:
OutboxMessage in RocksDB inside the same atomic batch as the metadata commit.KURA_REPLICATION_UPLOAD_STALL_MS (60 seconds by default) — so a slow-but-progressing transfer of any size completes while a stalled receiver still fails fast; upload failures log the artifact size. The window is re-armed when the body stream ends, so the wait for the response gets a whole one of its own: the receiver copies the staged body into a segment and fsyncs it under the node-wide segment write lock before it answers, and that tail scales with the artifact rather than with the network. On success, the message is deleted; on failure it stays queued and the worker retries. Messages whose target is absent from the node's current peer set are dropped immediately (observable as dropped_stale_target replication results): the fetched peer view is authoritative and the control plane withholds a peer only after a full staleness window of missed heartbeats, so the removal is deliberate — and a peer that later rejoins reconciles the gap through its own backfill passes, so dropped deltas are recovered. Targets known only through discovery (in-cluster siblings, cross-region pods) are treated like the static seeds and never pruned within a process lifetime: their absence usually means a network flap rather than departure, and the re-join backfill reaches back only to the backfill window, so anything older would be lost outright. The protection is process-scoped — a genuinely removed pod (scale-down, region move) is never rediscovered after the observer's next restart, and since enqueues stop within one membership tick of unreachability, its small frozen backlog is dropped after the next deploy. An empty peer view never prunes — it means the node has no view (control plane unreachable), not that every peer left.KURA_REPLICATION_BANDWIDTH_LIMIT_BYTES_PER_SECOND. The limiter is shared per node across live replication uploads, replication ingests, and backfill body fetches/responses. The configured value is a ceiling; the effective sync rate is divided by the larger of public_inflight + 1 and the recent public request latency EWMA over KURA_REPLICATION_PUBLIC_LATENCY_TARGET_MS. The latency EWMA is sampled at time-to-first-byte (when the response is ready to start streaming), not at body completion, so large but healthy downloads do not register as latency and over-throttle sync; their concurrency is already captured by public_inflight. Public inflight includes non-probe public HTTP requests plus gRPC cache RPCs. Internal replication and probe requests do not count as public load. This lets sync work use its full budget while the node is quiet and back off automatically when public cache traffic is active or slow.Two observability surfaces support capacity and sharding decisions: kura_public_request_latency_seconds is a histogram of time-to-first-byte for public requests across both transports (transport is http or grpc, labeled by route), and kura_artifact_egress_throughput_bytes_per_second is a histogram of achieved per-response egress throughput by producer. Together with the aggregate kura_artifact_egress_bytes_total rate they indicate when a region is bandwidth-bound and a good candidate for sharding across more primary pods.
version_ms (last-writer-wins per key). A rejected apply reports which case it hit, surfaced as the outcome label on kura_replication_apply_results_total: ignored_stale when the incoming version is strictly older than the stored one (a genuine LWW rejection — the peer is behind on this key), ignored_equal when it equals the stored one (both sides already hold the identical entry, so the walk re-shipped converged data rather than reconciling skew), and ignored_tombstone when a namespace delete already covers the key. ignored_equal has a structural baseline: concurrent replication of one key produces it by design (every apply-lock loser lands there), so its diagnostic value is in the ratio against ignored_stale, not in being non-zero. The family is written only by the live replication endpoints — every series carries source="replication" — so backfill applies never appear in it. The apply path takes a per-key write lock so that concurrent applies of the same key serialize: the first commits the manifest and the rest re-read it and short-circuit to IgnoredEqual rather than each appending their own copy to a segment. Without it, simultaneous applies of one key leave all but the last copy orphaned on disk.Newly joined nodes catch up through the backfill walker (src/backfill/): a newest-first walk of each peer's per-entry index, fetching only the bodies the local node is missing. The walk is driven by the membership loop, one pass per peer at a time, with a durable per-peer watermark advanced only on a completed pass — so a restart mid-pass re-lists from the watermark rather than starting over. Records apply through the same LWW/outcome/index code paths as live replication, so the same conflict rules hold. claims.rs holds the node's one shared claim set, which single-flights a record across concurrent peer passes; lifecycle.rs schedules passes and owns the initial-cycle failure budget; pass.rs runs a pass's pipelined list/fetch/apply stages; window.rs derives the horizon and the capacity rules. See README.md for the wire protocol (GET /_internal/backfill/entries, POST /_internal/backfill/bodies, and the per-artifact route for oversized entries).
The window bound applies to every listed kind, namespace tombstones included, because the peer's index is a single version-ordered stream — exempting a kind would mean walking to the peer's oldest entry on every pass, which is the unbounded walk this design removed. The bound is max(horizon, watermark), so a delete newer than the node's last completed pass is always carried; the residual is a node whose own ring turned over while it was away, which can keep serving a namespace deleted mesh-wide until those artifacts rotate out. That is the recency guarantee's stated drawback applied to deletes, and it is the first thing to revisit if tombstone coverage ever has to be unconditional.
Catch-up applies with batched durability instead of the live paths' per-record fsyncs (which dominated cold-node catch-up: ~2 fsyncs × every record). Each spooled bodies batch applies in four phases: (1) every Present segmented body is appended to the active segment with no per-record sync while inline bodies stage in memory; (2) one group-commit fsync makes the batch's segment bytes durable (a segment that rotated out mid-batch was already fsynced by the rotation); (3) staged records commit in groups of up to BACKFILL_APPLY_GROUP_RECORDS (64) — each group re-runs the authoritative LWW/tombstone checks under the records' write locks and lands ONE shared non-sync WriteBatch, and each record's claim resolves right after its group's WriteBatch (a mid-commit failure leaves later groups unresolved for the re-list); (4) one synced WAL flush is the batch's durability barrier — ~2 fsyncs and ~ceil(records/64) WAL appends per batch instead of one-plus per record (a live measurement showed the per-record WAL appends alone saturating disk IOPS). Ordering matters: an unrelated concurrent sync write can flush the WAL at any moment, so segment bytes are always fsynced before any of the batch's manifests enters the WAL, preserving the live invariant that a durable manifest implies durable segment bytes. Losing the tail of un-flushed applies in a crash is absorbed by the pass contract — no pass completion means no watermark advance, so the restart re-lists the window and LWW absorbs replays — and the watermark written on completion is itself a sync commit through the same WAL, whose prefix-ordered sync guarantees every apply it covers is durable. Live replication, client writes, oversized per-artifact backfill fetches, and namespace tombstone applies (the latter two a handful per pass) keep per-record sync durability. See ApplyDurability in src/store.rs.
See src/replication/mod.rs for the membership and outbox loops, src/backfill/ for the catch-up walker, and src/replication/operation.rs + outbox_message.rs for the message types.
A node finds peers in three ways:
KURA_PEERS. Static config is immutable for the process lifetime, so it carries platform-stable peers only (the managed regions' public peer gateways); volatile membership lives in the dynamic layer below.src/mesh_heartbeat.rs): enrolled self-hosted nodes send a mesh heartbeat every ~60s (cadence server-advertised) whose response carries the current peer list; managed pods fetch the same view read-only when KURA_MESH_PEERS_SYNC is set, with serving gated on the first successful fetch (a pod booting blind would accept writes without enqueuing replication for peers it cannot see). Additions and removals propagate at heartbeat cadence with no restart; a failed heartbeat keeps the last-known view, so a degraded control plane can never shrink the mesh.KURA_DISCOVERY_DNS_NAME, which resolves to the addresses of the other pods (typical when running as a Kubernetes StatefulSet behind a headless service).A spawn_membership_task loop polls each candidate's GET /_internal/status every two seconds. Only peers that respond with the same tenant_id and a different node_url are admitted as members. The local node never lists itself.
Mesh membership itself is control-plane state for enrolled nodes: a node that stops sending mesh heartbeats is deactivated (withheld from every peer's view) and its row is purged once its peer certificate can no longer be valid. Heartbeats never create or restore membership — a withheld node is answered mesh_member: false and recovers with a recovery re-enrollment (backoff-limited), which reactivates or recreates its membership server-side. Nothing local is torn down for it and readiness is not clawed back: the writes missed while out of the mesh were never enqueued for the node (replication targets are computed at write time), and the backfill watermarks are durable, so the next pass re-walks from them and reconciles the gap in the background while the node keeps serving.
Each tick produces a MembershipUpdate and feeds it into ReadinessState (src/state.rs). The state tracks:
known_peers (peers that responded successfully),initial_discovery_completed only flips once we have actually observed a successful peer status check (or there are no discovery targets at all). This avoids promoting a node to "ready" when the seed peers were transiently unreachable.
A node moves through three explicit traffic states (src/runtime.rs). Ordinary membership changes do not demote a serving node; newly discovered or returning peers reconcile in the background:
┌──────────┐ restart ┌──────────┐
│ │ │ │
│ joining │ ◄──────────────────────────│ serving │
│ │ │ │
└────┬─────┘ └─────┬────┘
│ discovery settled and either the ring │
│ is full enough or the initial backfill │
│ cycle has settled │
▼ ▼
┌──────────┐ drain request ┌──────────┐
│ serving │ ─────────────────────────► │ draining │
└──────────┘ └──────────┘
joining — public reads/writes are accepted but /ready returns 503. After discovery settles, the node becomes ready once its segment ring is at least KURA_BACKFILL_READY_RING_PERCENT full or its initial backfill cycle has settled (zero discovered peers settle immediately; peers that exhaust their failure budget stop gating while background retries continue). Readiness then latches for the process lifetime — later re-join backfills and recovery re-enrollments never regress it — while the orthogonal inputs (writer lock, draining) still gate /ready. The /status/rollout report carries a backfill_initial_cycle mode (pending / complete / degraded) that rollout consumers gate on.serving — /ready returns 200. Public APIs handle traffic normally.draining — public HTTP rejects new requests and stops reusing HTTP/1.1 connections; established HTTP/2 connections (gRPC included) receive a GOAWAY so channels finish in-flight streams and reconnect elsewhere. Inflight work continues until a shared drain deadline (KURA_DRAIN_COMPLETION_TIMEOUT_MS) elapses.Independent of draining, the co-hosted listener's hyper path recycles every connection after CONNECTION_MAX_AGE (300s): the server sends GOAWAY and allows in-flight streams CONNECTION_MAX_AGE_GRACE (900s) to finish before severing. Without recycling, a long-lived Bazel channel would pin to a demoted-but-alive NodePort primary indefinitely after failover. Both public listeners — plaintext and TLS, with acceleration on or off — share this per-connection serving path, so recycling and drain GOAWAY apply uniformly; the internal mTLS peer listener is a separate plane with its own lifecycle.
The action cache also serves an instance-wide snapshot through a reserved action key (tuist-actioncache-snapshot/v2, intercepted by digest comparison inside GetActionResult — no dedicated RPC): the response inlines the namespace's complete key→value map as a deduplicated node table plus per-key node-index lists and a write-time watermark, so a cold client primes every association in one round trip and fetches content through ordinary batched blob reads. Serving is backed by an incrementally maintained per-namespace index (bounded, LRU): reconciliation runs as a detached task shared by every concurrent request (a client or gateway that gives up on a slow first build cannot throw the work away — the build completes and caches regardless, and the next request serves from memory) and diffs the cached index against the newest slice of the action-cache keyspace; a request with no cached index waits only briefly for the build before answering UNAVAILABLE (the client stays on the per-key path and refetches shortly — pinning requests to a long first build walked every one of them into its deadline). Enumeration reads a dedicated action_cache_index column family ordered newest-first by write time (maintained at publish/delete, lazily backfilled per namespace with one legacy full scan) so a reconcile touches at most its entry cap rather than the whole namespace — the scan keeps only the most recent entries by write time, bounding the build's memory the way the byte ceiling bounds the response; only new-or-changed entries read their stored ActionResult — every referenced blob is presence-gated by manifest existence, which tracks eviction exactly, and the node table is compacted once entry churn strands enough unreferenced nodes. The gate runs on every reconcile regardless of memory pressure — only the action-result load is denied under pressure — so the served view stays honest through a pressure window instead of freezing stale (a frozen snapshot that references an evicted blob fails the build on the first missing object). A tuist-snapshot-after:<watermark> hint in inline_output_files returns a delta of entries written at or after the client's watermark (inclusive — millisecond versions are not unique, and re-sent boundary entries merge idempotently); deltas only add, so clients periodically refetch the full view to pick up retractions. The payload is capped at 48 MiB: the full view encodes newest-first (the ceiling sheds the oldest keys — a recency window) while a delta encodes oldest-first with the watermark set to the newest entry actually included, so an oversized delta paginates across refreshes instead of skipping what it dropped. A server without the feature answers a plain not-found the client degrades from — safe under mixed-version rollouts. Bump the key's version suffix on any encoding change.
Action-cache entries are also the one artifact class with their own lifecycle: clients publish new keys on every source change and the tiny records never face segment capacity pressure, so without intervention a namespace's keyspace grows forever. Two mechanisms bound it, both node-local (peers apply the same rules over the replicated version_ms and converge on their own): the snapshot serve path cascade-deletes entries whose blobs were evicted (unserveable by construction; a grace window spares young entries whose blobs may still be mid-replication), and a periodic background sweep expires entries whose write time predates the TTL — an expired entry that is still used costs its next cold reader one recompile + republish, which refreshes it fleet-wide.
An action-cache entry and its output blobs form one integrity unit: the entry (a REAPI ActionResult) is meaningless once any blob it references leaves the CAS. The publish path already enforces this ordering (FindMissingBlobs → BatchUpdateBlobs → UpdateActionResult last), but blob eviction is capacity-driven and independent, so an entry could outlive its blobs and strand — served as a hit that then fails the client's blob fetch (CAS error: missing object), which clang-compiled targets do not tolerate mid-build the way Swift does. To close that at the source, eviction honors the same invariant: a node-local, derived reverse index (individual blob_ref/{blob}\0{entry} pair rows kept in the existing key_value column family — prefixed rather than given a dedicated column family so a rollback to a binary that predates it can still reopen the database — maintained in the same atomic batch as every entry write and delete, mirroring the segment_artifacts shape) lets evict_segment cascade — dropping a blob removes, in one batch, every action-cache entry that references it, re-validating each candidate against the entry's current ActionResult so a reverse row left stale by a re-publish only deletes itself. The map is purely derived (never replicated) and rebuilt once at startup from the entries on disk, which widens coverage to entries that predate it. That backfill does not gate the cascade: it waits on background headroom, which the since-retired raw-charge admission arm denied indefinitely on a warm node (leaving a cascade gated on it inert in production across every eviction sweep), and even under tier-only admission a sustained pressure window would stall it. The cascade instead acts on whatever rows exist (the write path maintains them for every entry written since boot) and re-validates each one, so an incomplete map can only under-cascade, never remove a live entry. KURA_ACTION_CACHE_EVICTION_CASCADE_ENABLED is the only gate. The serve-side presence gates (per-key GetActionResult and the snapshot reconcile) stay on regardless: they remain correct under the residual publish-versus-eviction race — where an entry commits just after eviction scanned its blob and returns NOT_FOUND, which is an expected client outcome — and they are the backstop the cascade lifts load from rather than replaces. A symmetric race exists in the other direction: the cascade validates and stages an entry's deletion without the per-entry write lock (it runs from segment rotation while that path already holds a write-lock stripe, so taking the lock could self-deadlock), so a re-publish that commits a newer version between the validation read and the batch write has that fresh version removed. Every branch is fail-safe and self-healing — the entry degrades to a NOT_FOUND the client recomputes and republishes, the reconcile's stale-row cleanup drops the dangling index row, and the newer reverse rows are re-validated as stale on the next eviction — so the cascade must never be assumed to leave a live newer version in place. Kura deliberately does not refuse an entry whose blobs are momentarily absent at commit, because metadata-lane replication intentionally delivers fresh action-cache entries ahead of the bulk-blob backlog, so a receiving node legitimately sees an entry before its blobs. Presence-gating alone still leaves a window, because the gate answers from manifest metadata and never routes through the read path that keeps a blob alive: eviction can collect a vouched-for blob between the node answering GetActionResult and the client's BatchReadBlobs. Both REAPI read paths therefore extend the lifetimes of the blobs they vouch for, which is what the spec asks of them (GetActionResult: referenced blobs stay available "for some period of time afterwards" and their lifetimes "SHOULD be increased if necessary and applicable"; FindMissingBlobs: "Servers SHOULD increase the lifetimes of the referenced blobs"). A served entry's referenced blobs, and the blobs a FindMissingBlobs reported present, are queued for the same background copy-forward promotion the serving path uses when they sit in an Old segment. The Old band is what makes the segment ring LRU-like rather than plain FIFO: a blob only earns a copy-forward if it is read while sitting in that band, so the band's share of the ring (a fifth, see resolve_segment_ring_limits) sets how much use history the cache can act on. Copy-forward rather than a reservation table: promotion makes the blob genuinely young again, so the ordinary eviction policy needs no special case, nothing has to be honored later, and the extension survives a restart, whereas a reservation only defers eviction and must either be broken (not closing the gap it exists to close) or honored (stalling eviction exactly when the node is under pressure). The work is bounded by the single promotion worker and segment_refresh_lock, stays off the request path, and skips entirely on a node holding no Old segments (FindMissingBlobs accepts a client-controlled digest count up to the 64 MiB decode ceiling, so it extends inside its presence lookup rather than over a collected key set: no per-digest allocation is retained and a present blob costs one manifest lookup rather than two). A slice of the promotion queue is reserved for vouched refreshes so a burst of serve-path reads cannot fill it ahead of one, and vouched work drains in its own lane ahead of serve-path work (reserved admission alone is not enough: a refresh admitted behind a long serve backlog still waits for the whole backlog to copy, which is the window the vouch exists to close), and a promotion the queue still has no room for is counted on kura_promotion_drops_total by trigger. That counter is the signal that extension is not keeping up: because promotion completes asynchronously, a queue that is full or draining slowly can leave a vouched blob in its Old segment until rotation reclaims it, which reproduces the missing-object failure without a crash or a critical-pressure window. Because these RPCs promise the blob will still be there, they promote one pressure tier deeper than serve-path promotion, which carries no such promise: they continue through constrained and stop at critical, where a read-path write would compound the squeeze. Refreshes are observable through kura_segment_refreshes_total, kura_segment_refresh_bytes_total, and kura_segment_refresh_duration_seconds, each carrying a trigger label (serve, action_cache, find_missing) so the read-time write amplification each path adds is separable, with declined refreshes counted as result="pressure_skipped". This narrows the population of blobs evicted between check and use; it does not eliminate it, since a crash, a critical-pressure window, or a promotion queue that cannot drain fast enough can still drop a blob the node vouched for.
The REAPI gRPC services (src/reapi/mod.rs) are mounted into the co-hosted listener rather than a dedicated gRPC server (reapi::routes returns an axum::Router that run_with_config merges with the HTTP router). The listener advertises raised HTTP/2 flow-control windows — a 4 MiB stream window and a 16 MiB connection window — so a single large ByteStream.Write is not throttled to roughly window / RTT under WAN latency (without them the kura hop becomes the next bottleneck after the gateway nginx window). The window is FIXED, never adaptive: hyper's adaptive flow control would override the fixed size and ramp a single stream up from ~64 KiB, halving single-stream upload throughput under WAN latency. A per-upload stall timeout (60s, keyed on byte progress so trickled keepalive frames cannot hold a stalled stream open) reclaims a vanished or stalled writer without cutting an upload that keeps making progress. Temporary files also carry drop guards, so transport cancellation and backfill pass cancellation schedule partial-file removal on Tokio's blocking pool even when they drop a future before its asynchronous cleanup path runs.
/up is a liveness signal that does not depend on any of this — it stays healthy as long as the process is alive.
Each PVC is owned by exactly one Kura process. On startup, DataDirLock (src/runtime.rs) takes an OS file lock on .kura.writer.lock inside KURA_DATA_DIR. If another process holds the lock, startup fails fast. Public readiness depends on the lock being held, so a node that loses the lock cannot serve traffic.
The Kubernetes layer reinforces this with ReadWriteOncePod PVC access by default; the app-level lock is the source of truth and works even when the CSI driver only supports ReadWriteOnce.
Rollouts are deliberately conservative because each node is stateful and serves cache traffic that should not regress to misses during an upgrade.
The pieces:
preStop hook sends SIGUSR1 to start drain. The pod stays alive long enough to finish inflight work, then exits.terminationGracePeriodSeconds is computed from the application's own drain timeout plus small lifecycle buffers, so Kubernetes never cuts shutdown short.ops/rollout/gate.sh) polls each node's /status/rollout and only advances when every node reports the same membership generation, all are back in serving, ring size matches, no node's initial backfill cycle is still pending, the outbox is near baseline, no node is under critical memory pressure, and there is no new file-descriptor timeout activity.ops/helm/kura/rollout.sh) stages the new revision behind a StatefulSet partition, rolls the highest ordinal first, and delegates health gating to the generic gate. The adapter is a thin transport layer; it does not own rollout semantics.test/e2e/kura_compatibility_rollout.sh) validates PREVIOUS_REF → HEAD → PREVIOUS_REF on the same persistent Docker volumes for the artifact CAS path.The rollout gate explicitly assumes only that it can fetch /status/rollout from each node. It does not depend on Kubernetes probes or Prometheus.
Each node exposes:
/metrics (replication latency, FD pressure, manifest cache, RocksDB internals, outbox depth, traffic state, rollout-relevant counters).kura_http_request_duration_seconds aggregates public non-probe latency without a route label to avoid multiplying route cardinality by histogram buckets.KURA_CONTROL_PLANE_URL and client credentials. Kura aggregates bytes and request counts into bounded in-memory windows, persists closed windows into a dedicated RocksDB usage outbox, and pushes batches to /_internal/kura/usage. Delivery pauses under critical memory pressure and is at least once, with deterministic event ids for control-plane deduplication. Both transports are metered and tagged by protocol: the HTTP cache path emits protocol = "http", and the REAPI (gRPC) path emits protocol = "grpc" with artifact_kind = "reapi". A batch RPC (BatchReadBlobs/BatchUpdateBlobs) books one request carrying the aggregate bytes, matching the one-request-per-call accounting of the HTTP and ByteStream paths, and re-uploads of an already-present blob are not billed.KURA_NODE_COUNTRY / KURA_NODE_SUBDIVISION, set per fleet from the datacenter the node runs in) onto the OTel Resource as geo.country.iso_code and geo.region.iso_code, next to the unchanged kura.region, so every span carries them; the same pair lands on the low-cardinality kura_node_geo_info metric for Grafana maps (src/node_location.rs). Resolution is a pure function of configuration — no egress-IP probe, no geographic database — falling back only to a country prefix already embedded in the region label (fr-par -> FR). Client-side geographic attribution does not exist: Kura never geolocates a client IP.tracing::error! events.Helm and the local docker-compose stack ship a complete Grafana/Prometheus/Loki/Tempo setup. See ops/AGENTS.md for layout.
All configuration is environment-driven (src/config.rs). The full table lives in README.md. Highlights:
KURA_TENANT_ID, KURA_REGION, KURA_NODE_URL, KURA_PORT, KURA_INTERNAL_PORT, KURA_DATA_DIR, KURA_TMP_DIR.reapi::routes, everything else to the HTTP router), so a single client-facing URL speaks both protocols. It serves plaintext on KURA_PORT and — when public_tls (KURA_PUBLIC_TLS_*) is configured — TLS on KURA_HTTPS_PORT, ALPN-negotiated (h2 for gRPC, http/1.1 for HTTP). The plaintext listener runs through the accelerated server, so HTTP/1 artifact GETs get the sendfile/splice fast path while gRPC (h2c) and other non-accelerable requests fall through to hyper; the TLS listener uses the plain hyper path (TLS is incompatible with sendfile). Both use the fixed REAPI-sized HTTP/2 windows so co-hosted uploads are not throttled.KURA_PEERS, KURA_DISCOVERY_DNS_NAME, optional KURA_INTERNAL_TLS_* for peer mTLS.KURA_REPLICATION_BANDWIDTH_LIMIT_BYTES_PER_SECOND sets the aggregate peer artifact body traffic ceiling per node when set above 0; Kura adapts the effective rate downward under public HTTP or gRPC load, and KURA_REPLICATION_PUBLIC_LATENCY_TARGET_MS controls the latency target for additional backoff.KURA_ACCELERATED_FILE_SERVING_ENABLED, KURA_ACCELERATED_FILE_SERVING_MODE, KURA_ACCELERATED_FILE_SERVING_MAX_CONCURRENT, and KURA_ACCELERATED_FILE_SERVING_CHUNK_BYTES bound the Linux plaintext HTTP/1 artifact fast path while preserving the Axum/Hyper fallback path on the same public port.KURA_DRAIN_COMPLETION_TIMEOUT_MS.When budget vars are unset Kura inspects RLIMIT_NOFILE, the cgroup memory limit, and detected CPU count to pick safe defaults.
src/http.rs and src/reapi/mod.rs.src/store.rs is the single entry point.src/replication/mod.rs and src/state.rs; for catch-up pass scheduling and retry behavior see src/backfill/lifecycle.rs.ops/helm/kura/ and ops/rollout/gate.sh.spec/e2e/ exercises the live stack.