Kura

Kura is a Rust server for building low-latency cache meshes for tenants, handling distributed cache traffic for binary artifacts and metadata.

[!NOTE] Kura comes from the Japanese word 蔵 (kura), which refers to a storehouse or warehouse. The name fits the system's role: keeping build artifacts and cache metadata stored durably and close at hand so they can be served with low latency.

License and Contributing

Kura is licensed under the GNU Affero General Public License, version 3 only. See LICENSE.md for the full license text.

Contributions to Kura require signing the Kura Contributor License Agreement (CLA). Please see CLA.md before submitting pull requests that modify Kura components.

Summary ✨

• ⚡ Hot reads come from local disk
• 🪨 Local metadata, multipart state, and the replication outbox live in RocksDB
• 🔁 Blobs and cache metadata replicate to peer nodes with eventual consistency
• 🔎 Nodes can discover peers through DNS and bootstrap themselves from already-running nodes
• 📦 Kura actively supports Bazel and Buck2 REAPI, Xcode Cache, Gradle, and module-cache protocols
• 🧪 Compatibility endpoints for Nx and React Native Metro are available, but they are not a primary focus today
• 🧰 The gRPC API exposes the Bazel Remote Execution cache services used by Bazel and Buck2
• 📊 The local stack includes Grafana, Prometheus, Loki, Promtail, and Tempo traces

Supported cache protocols

Actively supported:

• Bazel and Buck2: Bazel Remote Execution API v2 over gRPC on KURA_PORT
• Xcode Cache: HTTP CAS artifacts on POST/GET /api/cache/cas/{id} and action-cache style entries on PUT/GET /api/cache/keyvalue
• Gradle: PUT/GET /api/cache/gradle/{cache_key}
• Module Cache: multipart uploads on POST /api/cache/module/start, POST /api/cache/module/part, POST /api/cache/module/complete, and HEAD/GET /api/cache/module/{id}

Compatibility surfaces:

• Nx: self-hosted remote cache API on GET/PUT /v1/cache/{hash}
• React Native Metro: HttpStore / HttpGetStore on GET/PUT /api/metro/cache/{cache_key}

Local stack 🧪

Run:

docker compose up --build -d

Useful endpoints:

• http://localhost:4101/up
• http://localhost:4101/ready
• http://localhost:4101/status/rollout
• http://localhost:4102/up
• http://localhost:4102/ready
• http://localhost:4102/status/rollout
• http://localhost:4103/up
• http://localhost:4103/ready
• http://localhost:4103/status/rollout
• grpc://localhost:4101 for Bazel/Buck2 REAPI against kura-us
• grpc://localhost:4102 for Bazel/Buck2 REAPI against kura-eu
• grpc://localhost:4103 for Bazel/Buck2 REAPI against kura-ap
• http://localhost:3000 for Grafana with admin / admin
• http://localhost:9090 for Prometheus
• http://localhost:3100 for Loki
• http://localhost:3200 for Tempo

Toolchain 🛠️

Install the toolchain (Rust + Bazel) from mise.toml:

mise trust mise.toml
mise install

Build and test with Bazel (the path CI gates on):

mise run compile         # build all targets for the host
mise run test-unit       # bazel test //...

rules_rs resolves the crate graph from Cargo.toml/Cargo.lock on each build, so changing Rust deps just updates Cargo.lock as usual.

If you have access to the tuist/kura project on Tuist, run tuist bazel setup (and re-run it after changing location) to use the closest Kura remote cache; otherwise Bazel builds fine against the local cache.

Cargo works as a fallback when Bazel is unavailable, and the end-to-end suite runs under shellspec:

mise x [email protected] -- cargo test
mise x [email protected] -- shellspec

Runtime configuration is summarized in the table under Runtime Model And Limits. Kura now derives sensible defaults for the main FD, memory, and metadata-store budgets at startup when you do not set them explicitly.

🗺️ Project Areas

Kura is easier to read by subsystem than by tutorial step. The sections below group the project by the main areas you operate or extend.

• 🔌 Protocol surfaces
• 🗄️ Storage and replication
• ⚙️ Runtime model and limits
• 📊 Observability
• 📣 Runtime analytics
• ☸️ Deployment options
• 🧩 Extensions and policy

🔌 Protocol Surfaces

Kura exposes multiple cache protocols behind one service. Public HTTPS supports HTTP/2 so clients can multiplex concurrent artifact downloads on long-lived connections. The actively supported surfaces are:

• 🛠️ Bazel and Buck2: REAPI over gRPC on KURA_PORT
• 🍎 Xcode Cache: POST/GET /api/cache/cas/{id}?tenant_id=...&namespace_id=...
• 🗂️ KeyValue / action-cache entries: PUT /api/cache/keyvalue?tenant_id=...&namespace_id=...
• 🐘 Gradle: PUT/GET /api/cache/gradle/{cache_key}?tenant_id=...&namespace_id=...
• 📦 Module Cache: POST /api/cache/module/start?..., POST /api/cache/module/part?..., POST /api/cache/module/complete?..., HEAD/GET /api/cache/module/{id}?...

For those HTTP cache routes, tenant_id is always required and namespace_id is optional. When namespace_id is present, the request is namespace-scoped. When it is omitted, the request is tenant-scoped and Kura stores it under an internal empty namespace key. REAPI requests carry their namespace explicitly through the gRPC instance_name/resource_name, and may declare the account with the x-kura-tenant-id metadata header (the gRPC analog of the tenant_id query param above).

Kura extends the REAPI ActionCache with a wildcard form of the standard GetActionResult.inline_output_files hint: a literal "*" entry asks Kura to inline the contents of every output file the response budget affords (the per-request REAPI materialization budget, 8–64MB depending on the node's memory limits). It exists for clients whose output-file paths are digests unknown before the response — the Xcode CAS plugin — collapsing the action lookup and the blob fetch into one round-trip. Semantics: wildcard-matched files inline best-effort (a file the budget cannot afford stays un-inlined and the client falls back to BatchReadBlobs); explicitly listed paths keep the standard hard RESOURCE_EXHAUSTED error on budget exhaustion; servers without the extension match no literal "*" path and inline nothing, so mixed client/server versions interoperate unchanged. Note the trade-off: inlining happens before the server can know which blobs the client already holds, so every inlined byte counts as metered download egress even when a warm client discards it.

Kura also exposes compatibility endpoints that are not a primary focus today:

• 🧱 Nx: PUT/GET /v1/cache/{hash}
• 📱 Metro: PUT/GET /api/metro/cache/{cache_key}

The local compose stack is still the quickest way to exercise all of those surfaces together:

docker compose up --build -d

Example Xcode artifact round trip:

curl -X POST \
  "http://localhost:4101/api/cache/cas/artifact-1?tenant_id=acme&namespace_id=ios" \
  -H "content-type: application/octet-stream" \
  --data-binary "xcode-binary"

curl \
  "http://localhost:4102/api/cache/cas/artifact-1?tenant_id=acme&namespace_id=ios"

Example keyvalue entry round trip:

curl -X PUT \
  "http://localhost:4101/api/cache/keyvalue?tenant_id=acme&namespace_id=ios" \
  -H "content-type: application/json" \
  -d '{"cas_id":"cas-1","entries":[{"value":"hello"},{"value":"world"}]}'

curl \
  "http://localhost:4103/api/cache/keyvalue/cas-1?tenant_id=acme&namespace_id=ios"

Example tenant-scoped Xcode artifact round trip without a namespace:

curl -X POST \
  "http://localhost:4101/api/cache/cas/account-artifact?tenant_id=acme" \
  -H "content-type: application/octet-stream" \
  --data-binary "account-binary"

curl \
  "http://localhost:4102/api/cache/cas/account-artifact?tenant_id=acme"

🗄️ Storage And Replication

Kura splits storage into two planes:

• 🪨 RocksDB stores metadata, keyvalue payloads, multipart state, tombstones, segment lifecycle state, and the replication outbox.
• 📦 Segment files store large immutable binary artifacts for the hot path. The segment ring's capacity derives from the data-dir filesystem size (or KURA_CAS_CAPACITY_BYTES), and rotating in a new segment evicts the oldest one once the budget is reached.

Replication is leaderless and eventually consistent:

• 🔁 local writes become durable together with their outbox work
• 🌍 peers bootstrap by pulling manifests, tombstones, and artifact bodies, reconciling only the diverging ranges via a per-bucket manifest digest exchange
• 🔎 DNS discovery can expand the peer set automatically
• 🧠 the outbox is processed incrementally so queue depth does not blow up heap usage during backlog

Peer-to-peer traffic always uses the dedicated internal plane:

• KURA_INTERNAL_PORT

Peer-to-peer mTLS is optional on that plane:

• KURA_INTERNAL_TLS_CA_CERT_PATH
• KURA_INTERNAL_TLS_CERT_PATH
• KURA_INTERNAL_TLS_KEY_PATH

When peer mTLS is disabled:

• KURA_NODE_URL and every value in KURA_PEERS must use http://...:<KURA_INTERNAL_PORT>
• /_internal/* is only served on the internal HTTP listener
• 🌍 the public API still stays on KURA_PORT

When peer mTLS is enabled:

• 🔒 KURA_NODE_URL and every value in KURA_PEERS must use https://...:<KURA_INTERNAL_PORT>
• 🧱 /_internal/* is only served on the internal mTLS listener
• 🌍 the public API still stays on KURA_PORT
• 🪪 the certificate configured through KURA_INTERNAL_TLS_CERT_PATH should be valid for both server and client auth
• 🏷️ the certificate SANs must cover the hostname used in KURA_NODE_URL

⚙️ Runtime Model And Limits

Kura is designed around explicit resource budgets instead of relying on ambient process limits.

When Optional is Yes, the Default column shows what Kura uses today. auto means Kura derives the value at startup from detected file-descriptor limits, memory limits, or CPU count.

Backfill operations

• KURA_BACKFILL_ENABLED is flipped per account, and only after backfill/meta/build_complete is set on every node of that account's meshes — observable as GET /_internal/backfill/entries no longer answering 503 index_building, or via the kura_backfill_* metrics. On the hosted platform the flip is the account-scoped :kura_backfill feature flag: the provisioner renders the env (and a +backfill manifest-revision suffix) for gated accounts, and the reconciler rolls every managed instance the account owns — public mesh instances and private runner-cache (co-located) instances alike. Ungating rolls back the same way. Self-hosted operators set the env on their own instance configuration. Tuist's own account flips first; customer accounts follow.
• The rollout report shows the initial cycle per node as pending (passes still running or retrying with budget left), complete (every in-cycle peer resolved cleanly), or degraded (a peer exhausted its failure budget on real failures).
• Region-move promotion gates on instance readiness only; the initial-cycle mode is not consumed by the control plane. A flag-on move target can latch ready before its full transfer settles, so before initiating a move where completeness matters, check backfill_initial_cycle: complete on the target's rollout report first. To abort a move, destroy the move TARGET server (Kura.destroy_server via the server ops surface); the source keeps serving.
• Rollback is flipping the flag off plus a pod restart: the node resumes the legacy bootstrap walker, and backfill/ watermarks and index rows sit inert on disk.
• Index-build progress: a node still building answers listing requests with 503 index_building; rebuilds (rollback-window staleness, cumulative crash forgiveness) are logged with the reason.

Kura also enforces a few hard-coded budgets that are not configurable:

• Replication ingest bodies on /_internal/replicate/artifact are capped at four times MAX_SEGMENT_BYTES (2 GiB) so a misbehaving peer cannot fill the data PVC. Bootstrap-from-peer fetches enforce the same ceiling for segment-backed artifacts and a 4 MiB ceiling for inline artifacts; bootstrap manifest and tombstone pages, and the manifest digest response, are capped at 32 MiB each. When KURA_REPLICATION_BANDWIDTH_LIMIT_BYTES_PER_SECOND is positive, Kura also applies a shared per-node bandwidth ceiling to peer artifact body traffic. The effective rate shrinks as public HTTP and gRPC requests are in flight or recent public latency rises above KURA_REPLICATION_PUBLIC_LATENCY_TARGET_MS, so background sync yields network capacity to public cache reads.
• Cache writes are rejected with retryable transport-specific backpressure when memory pressure reaches Critical, when the outbox is at KURA_OUTBOX_MAX_DEPTH, when the file-descriptor pool is exhausted, or when the data volume has insufficient free space for a new segment.
• Replication delivery is never paused for memory pressure, at any tier and regardless of the raw hard-watermark arm. Because a full outbox rejects cache writes, pausing the drain does not defer work — it strands the queue and ends up rejecting writes, leaving the node divergent from its peers for as long as they can accept its deliveries. The state that walks into the cap is the hard-watermark arm below Critical, where writes are still admitted while the drain is held; at Critical the write gates already reject before they look at outbox depth, so the outbox is frozen rather than growing. The drain loop is serial and node-wide, so exactly one delivery is in flight regardless of peer count or backlog depth, and it takes no transient reservation — that delivery holds one 512 KiB segment-read chunk, or for an inline artifact the whole value up to the 4 MiB inline ceiling. The usage (metering) outbox has no such coupling and still pauses under critical pressure.
• Kura samples the container charge every 200 milliseconds and removes clean file-backed cache before evaluating pressure, while exporting the complete charge and conventional working set as separate metrics. A file-cache reclaim signal activates on two arms: a working-set arm routed through the hysteretic pressure state machine so it does not flip per sample near the soft watermark, and a raw hard-watermark arm on memory.current that intentionally stays steady state on warm serving nodes so they keep trading clean file-cache warmth for request capacity. Either arm makes request paths release completed file ranges without constraining admission. Sampling only drives pressure state, cache trimming, and coarse background load shedding; it never participates in per-request admission arithmetic. Response materialization, foreground uploads, multipart assembly, and bootstrap transfers share one fair Tokio byte budget derived from the soft-to-hard watermark gap. Response-stream capacity scales with both that gap and the reserve between the hard watermark and runtime limit, allowing request serving to use memory released by pressure-driven cache trimming instead of stopping at a fixed absolute ceiling. Owned permits remain attached to the allocation or transfer that consumed them, and growth while already holding a permit is always non-blocking. A foreground upload reserves a source-plus-destination working set of up to 32 MiB, reduced automatically on smaller memory profiles. Objects larger than the active window, smaller uploads that had to queue, and overlapping foreground uploads synchronize and release completed staging and append-only segment ranges every 8 MiB. Kura closes the synchronized writer before using aligned DONTNEED file advice through Rustix, then reopens it in append mode, so cache reclamation cannot invalidate later buffered bytes. Waiting upload admission times out after 30 seconds with 503 Service Unavailable or gRPC RESOURCE_EXHAUSTED. REAPI ByteStream keeps its existing 64 MiB decode limit. A request-body scanner reads every five-byte gRPC envelope header and non-blockingly grows the owned permit to twice the largest message observed before Tonic allocates its retained wire buffer and decoded byte vector. Once the first resource name reveals the blob size, Kura adds only its bounded disk working set. Excess growth returns retryable RESOURCE_EXHAUSTED without waiting behind a shared HTTP/2 connection window. Mapped-file serving remains a separate try-only bound over already-resident reclaimable pages and always falls back to streaming. Constrained pressure pauses locally initiated bootstrap and snapshot work, but a bounded peer-response pool continues serving bootstrap reads so the mesh can converge; critical pressure sheds those responses too. A joining node retries temporary rate-limit and service-unavailable responses in place, honoring numeric Retry-After hints and releasing its fetch lock and memory reservation while it waits. Temporary upload, assembly, and bootstrap files are owned by cancellation-safe cleanup guards, so aborted futures cannot strand disk usage; cancellation cleanup runs on Tokio's blocking pool instead of a runtime worker. The allocator reclaims unused pages on one background thread with a four-second decay, so a quiet node returns memory after a burst without relying on a later request to trigger maintenance.
• Normal artifact and ByteStream readers also use weighted sublimits within that shared transient budget. File-backed responses reserve four buffers sized from 8 KiB to 512 KiB according to the response size, while inline responses include the complete value. Materialized Remote Execution responses reserve both their source payload and encoded transport copy. One transport guard follows each permit through encoding and every Hyper-owned byte buffer, so a stalled or cancelled client cannot release capacity early. When the guaranteed response pool is full and memory pressure is normal, public reads may borrow a second bounded tier from unused transient capacity, while retaining one quarter of that capacity for uploads, materialization, and allocator growth. Public reads that cannot reserve either tier promptly degrade to the 8 KiB chunk floor while still charging the 512 KiB per-stream transport send buffer. The degraded queue and slot wait are bounded; when either capacity or transient headroom is exhausted, Kura returns a retryable unavailable response instead of opening an unaccounted stream. Bootstrap reads never queue, cannot bypass public waiters, and use only their separate reserved progress quantum, leaving the guaranteed foreground capacity for public binary serving.
• Public plaintext HTTP/1 artifact downloads can use the same-port Linux accelerator after the request has been parsed, matched to a known artifact route, authorized through the extension hook, and resolved to a local file. The accelerator owns only a bounded pool of blocking transfer workers and falls back to the normal Axum/Hyper serving path whenever classification is incomplete or unsafe.
• RocksDB column families are configured with explicit level-0 slowdown/stop triggers and pending compaction limits so backlog turns into write-side backpressure instead of unbounded write-buffer growth.
• Inline keyvalue payloads are buffered in memory before being written. Total RAM committed to inline payloads is bounded by KURA_FILE_DESCRIPTOR_POOL_SIZE * KURA_MAX_KEYVALUE_BYTES; both knobs are tuned together when sizing per-pod memory.
• On startup, the soft RLIMIT_NOFILE is raised to the hard limit so the FD pool, RocksDB file descriptors, and socket budget all share the maximum the container runtime allows.

Auto-derived defaults currently follow these rules:

• file_descriptor_limit comes from RLIMIT_NOFILE when available, otherwise Kura falls back to a conservative host default.
• memory_limit_bytes comes from the exact cgroup memory limit when available, otherwise Kura falls back to physical host memory.
• cpu_count comes from detected parallelism via the runtime.
• KURA_FILE_DESCRIPTOR_POOL_SIZE is usable_fds / 8, clamped to [64, 256], where usable_fds is the detected FD limit minus reserved headroom.
• KURA_SEGMENT_HANDLE_CACHE_SIZE is KURA_FILE_DESCRIPTOR_POOL_SIZE / 4, clamped to [16, 64], and then capped below the FD pool so transient work keeps headroom.
• KURA_MEMORY_SOFT_LIMIT_BYTES is 60% of detected memory, rounded down to MiB boundaries. The wider gap to the hard watermark is the fixed transient-admission budget.
• KURA_MEMORY_HARD_LIMIT_BYTES is 85% of detected memory, rounded down to MiB boundaries. Both watermarks are validated below the exact runtime limit.
• KURA_SNAPSHOT_CACHE_MAX_BYTES is KURA_MEMORY_SOFT_LIMIT_BYTES / 4, rounded down to MiB boundaries and capped at 256 MiB.
• KURA_MANIFEST_CACHE_MAX_BYTES is KURA_MEMORY_SOFT_LIMIT_BYTES / 16, rounded down to MiB boundaries and clamped to [8 MiB, 64 MiB].
• KURA_METADATA_STORE_MAX_OPEN_FILES is usable_fds / 2, clamped to [128, 1024].
• KURA_METADATA_STORE_MAX_BACKGROUND_JOBS is cpu_count, clamped to [1, 8].
• KURA_METADATA_STORE_READ_CACHE_BYTES is memory_limit_bytes / 32, rounded down to MiB boundaries and clamped to [16 MiB, 128 MiB].
• KURA_METADATA_STORE_WRITE_BUFFER_POOL_BYTES follows the same memory_limit_bytes / 32 rule as the metadata-store read cache.
• KURA_METADATA_STORE_WRITE_BUFFER_BYTES is KURA_METADATA_STORE_WRITE_BUFFER_POOL_BYTES / 4, rounded down to MiB boundaries and clamped to [4 MiB, 32 MiB].
• KURA_METADATA_STORE_MAX_WRITE_BUFFERS is KURA_METADATA_STORE_WRITE_BUFFER_POOL_BYTES / KURA_METADATA_STORE_WRITE_BUFFER_BYTES, clamped to [2, 8].
• KURA_MAX_KEYVALUE_BYTES defaults to 1048576, KURA_FILE_DESCRIPTOR_ACQUIRE_TIMEOUT_MS defaults to 5000, KURA_DRAIN_COMPLETION_TIMEOUT_MS defaults to 240000, KURA_ACCELERATED_FILE_SERVING_ENABLED defaults to true, KURA_ACCELERATED_FILE_SERVING_MODE defaults to splice, KURA_ACCELERATED_FILE_SERVING_MAX_CONCURRENT defaults to 32, KURA_ACCELERATED_FILE_SERVING_CHUNK_BYTES defaults to 1048576, KURA_ACTION_CACHE_EVICTION_CASCADE_ENABLED defaults to true, KURA_REPLICATION_BANDWIDTH_LIMIT_BYTES_PER_SECOND defaults to 536870912, and KURA_REPLICATION_PUBLIC_LATENCY_TARGET_MS defaults to 100.

A minimal direct-binary deployment still looks like:

KURA_PORT=4000 \
KURA_INTERNAL_PORT=7443 \
KURA_TENANT_ID=default \
KURA_REGION=eu-central \
KURA_TMP_DIR=/var/cache/kura/tmp \
KURA_DATA_DIR=/var/cache/kura \
KURA_NODE_URL=http://cache-1.internal:7443 \
KURA_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318/v1/traces \
KURA_OTEL_SERVICE_NAME=kura-eu-central \
KURA_OTEL_DEPLOYMENT_ENVIRONMENT=production \
./target/release/kura

Set KURA_SENTRY_DSN to also forward panics and tracing::error! events to Sentry. Kura uses KURA_OTEL_DEPLOYMENT_ENVIRONMENT as the Sentry environment, so set it to values such as production, staging, or canary when separating events by deployment. In the standalone Helm chart, inject the DSN via extraEnv or extraEnvFrom. In controller-managed Tuist deployments, set kuraController.telemetry.deploymentEnvironment and sync the DSN into kura-shared-secrets with kuraController.sentry.externalSecret. KURA_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT accepts either an OTLP HTTP signal path such as http://otel-collector:4318/v1/traces or an OTLP gRPC root endpoint such as http://otel-collector:4317.

📊 Observability

Kura ships with a fairly complete local observability story:

• 📈 Prometheus metrics
• 📉 Grafana dashboards
• 🪵 Loki and Promtail logs
• 🧭 Tempo traces
• 🚨 Optional Sentry error reporting for panics and error-level tracing events

Prometheus exposes live metadata-store memory gauges:

• kura_rocksdb_block_cache_usage_bytes
• kura_rocksdb_block_cache_pinned_usage_bytes
• kura_rocksdb_block_cache_capacity_bytes
• kura_rocksdb_write_buffer_usage_bytes
• kura_rocksdb_write_buffer_capacity_bytes

Kura also exports:

• 📦 artifact read and write counters by kind, client, artifact_class, and result
• 🔁 replication latency and result metrics
• 💾 file descriptor pool pressure metrics
• 🧠 manifest cache occupancy and admission metrics

HTTP request counters keep bounded route and status labels by using Axum route templates such as /api/cache/cas/{id} and folding unmatched paths into /_unmatched. Request methods stay on OpenTelemetry spans instead of Prometheus labels, and client country counts live on the separate kura_http_client_requests_total counter. The kura_http_request_duration_seconds histogram intentionally has no route label and records only public non-probe requests. Keeping route-level latency in Prometheus would multiply every route by every histogram bucket, so route-specific latency belongs in sampled traces instead.

GeoIP enrichment

The Kura container image vendors a DB-IP IP-to-City Lite MMDB at /opt/geoip/dbip-city-lite.mmdb, so client geographic attribution is on by default. Location is resolved from the first hop in X-Forwarded-For (or X-Real-IP) at two granularities:

• country (ISO 3166-1): the client_country Prometheus label on kura_http_client_requests_total and the geo.country.iso_code OTel span attribute on http.request spans
• subdivision (ISO 3166-2, e.g. US-CA): the geo.region.iso_code OTel span attribute on http.request spans

Span and Resource attributes follow the OpenTelemetry geo.* semantic conventions so standard Grafana/Tempo tooling understands them out of the box. The Prometheus label stays client_country (short and Prometheus-idiomatic; semantic conventions cover spans/logs/resource, not metric label names).

Subdivision is intentionally not a Prometheus label. ISO 3166-2 has thousands of codes, and multiplying it across route × status would inflate active series. It lives on sampled traces only, which is enough to compute geographic distance per request. Country stays on a dedicated low-cardinality metric because it is bounded (~250 codes).

Lookups that miss (no header, private IP, or DB missing) fall back to client_country="unknown" and an unset geo.region.iso_code. If the vendored database is absent (custom image builds), Kura logs a startup warning and quietly runs without geographic attribution.

A background task refreshes the in-memory database from https://download.db-ip.com/free/dbip-city-lite-YYYY-MM.mmdb.gz every KURA_GEOIP_REFRESH_INTERVAL_SECS seconds (default 86400). Set the interval to 0 to keep the vendored copy for the pod's lifetime. The swap takes the in-process RwLock write guard for the few microseconds needed to replace the reader; concurrent lookups never observe a partial state. The City dump is ~60 MiB compressed / ~125 MiB decompressed today; each download is bounded to 128 MiB compressed / 256 MiB decompressed with a 60-second timeout, so refresh memory stays predictably capped well within the pod's limit. Outcomes are tracked in kura_geoip_refresh_total{result="ok|http_error|parse_error"}.

DB-IP data is © DB-IP, released under CC BY 4.0.

Node geographic attribution

Each pod resolves its own country and subdivision once at startup and stamps them on every exported OTel span as the geo.country.iso_code and geo.region.iso_code Resource attributes, alongside the existing kura.region (the cloud deployment region, e.g. fr-par) and kura.tenant_id. The same resolved country/subdivision also lands on the low-cardinality kura_node_geo_info Prometheus info metric so Grafana can map serving nodes without parsing traces. Combined with geo.country.iso_code / geo.region.iso_code on each request span, traces carry both endpoints of the request and Grafana can compute geographic distance directly off Tempo data.

Country resolution chain, tried in order:

1. KURA_NODE_COUNTRY env var (operator override; must be a 2-letter ISO code).
2. Public egress IP discovered via https://api.ipify.org (3-second timeout, best-effort), looked up against the vendored GeoIP database.
3. Explicit deployment-region mapping for the managed labels we use today (eu-central -> DE, us-east / us-west -> US), otherwise a real country prefix already present in KURA_REGION (fr-par -> FR, nl-ams -> NL).

Subdivision resolution: KURA_NODE_SUBDIVISION env var (operator override; ISO 3166-2 code such as US-CA), otherwise the same single egress-IP lookup. If the subdivision override is present, Kura derives the country directly from it and skips the extra probe unless subdivision itself is still missing. There is no deployment-region fallback for subdivision, so when neither the override nor the GeoIP lookup yields one, geo.region.iso_code is simply not stamped (the same is true of geo.country.iso_code when all three steps fail).

Disabling OTLP tracing

OTLP tracing is optional. Leaving KURA_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT unset (or empty) makes Kura skip exporter initialization and run without distributed traces — useful in environments without a collector (local kind, isolated edge nodes). When it is set, Kura auto-detects OTLP HTTP vs gRPC from the endpoint shape: /v1/traces paths use HTTP, while root collector endpoints such as http://collector:4317 use gRPC. Helm operators control it by setting config.telemetry.otlpTracesEndpoint: "" in a values overlay; the chart only renders the env when the value is non-empty, so an empty overlay disables tracing without crashlooping the pod.

📣 Runtime Analytics

Analytics webhooks are a separate optional subsystem for Tuist's current project-scoped cache analytics contract for Xcode and Gradle traffic.

Kura emits those webhook events only for namespace-scoped Xcode and Gradle HTTP requests, using the request's tenant_id and namespace_id as account_handle and project_handle in the payload. Tenant-scoped requests skip analytics until Tuist grows account-scoped binary analytics.

When enabled:

• 🍎 Xcode upload and download events are sent to /webhooks/cache
• 🐘 Gradle upload and download events are sent to /webhooks/gradle-cache
• ✍️ requests are signed with x-cache-signature
• 🧭 requests also include x-cache-endpoint
• 🪶 delivery stays in-memory and best-effort, so analytics never block the hot path
• 🧯 a per-pipeline circuit breaker opens after repeated delivery failures so Kura sheds analytics instead of backing up under a misbehaving upstream

Configure it with:

• KURA_ANALYTICS_SERVER_URL
• KURA_ANALYTICS_SIGNING_KEY
• optional KURA_ANALYTICS_BATCH_SIZE default 100
• optional KURA_ANALYTICS_BATCH_TIMEOUT_MS default 5000
• optional KURA_ANALYTICS_QUEUE_CAPACITY default 1000
• optional KURA_ANALYTICS_REQUEST_TIMEOUT_MS default 5000
• optional KURA_ANALYTICS_CIRCUIT_BREAKER_FAILURE_THRESHOLD default 5
• optional KURA_ANALYTICS_CIRCUIT_BREAKER_OPEN_MS default 30000

It also exposes analytics-specific runtime metrics for:

• 📣 queue depth and drops
• 📦 batch sizes and flush outcomes
• 🧯 circuit-breaker state and open events

Usage Metering

When KURA_CONTROL_PLANE_URL, KURA_CONTROL_PLANE_CLIENT_ID, and KURA_CONTROL_PLANE_CLIENT_SECRET are set, Kura records first-party usage rollups for public cache traffic and pushes them to:

POST {KURA_CONTROL_PLANE_URL}/_internal/kura/usage

Both surfaces are metered: the HTTP cache path records rollups with protocol = "http", and the REAPI (gRPC) path — ByteStream read/write, CAS BatchReadBlobs/BatchUpdateBlobs, and ActionCache GetActionResult (including inlined stdout/stderr/output files) / UpdateActionResult — records them with protocol = "grpc" and artifact_kind = "reapi", so Bazel and other REAPI clients count toward the same usage surface.

The hot path increments bounded in-memory counters keyed by tenant, namespace, node, region, traffic plane, direction, operation, protocol, artifact kind, and fixed time window. Closed windows are persisted to a dedicated RocksDB usage outbox, then delivered in bounded batches with HTTP Basic client credentials. Delivery is at least once; the control plane deduplicates by deterministic event_id.

The usage pipeline follows Kura's resource discipline: bucket count, durable outbox depth, and delivery batch size are capped; delivery pauses under critical memory pressure; and a full usage outbox causes new closed windows to remain in memory until the in-memory bucket cap is reached, after which new buckets are rejected and counted through memory-action metrics.

☸️ Deployment Options

Helm And Kubernetes

The repository includes a Helm chart at ops/helm/kura that deploys Kura as a StatefulSet with:

• 💾 one PVC per pod for metadata-state and segment storage
• 🔒 single-writer fencing through a process-held data-dir lock plus ReadWriteOncePod by default
• 🧭 a headless service for stable pod DNS and peer discovery
• 🌐 a regular service exposing both HTTP and gRPC
• 🚪 optional ingress for the HTTP API
• 🚪 optional ingress for the gRPC Remote Execution API
• 🧩 optional inline extension script mounting through a ConfigMap
• 🔐 optional peer mTLS for /_internal/* traffic via a mounted Kubernetes Secret
• 🚦 /ready for public readiness and /up for liveness, with a preStop SIGUSR1 drain hook that removes pods from traffic before SIGTERM
• ⏱️ a pod grace period derived from Kura's own drain timeout plus small lifecycle buffers so Kubernetes does not cut shutdown short

Lint and render the chart:

helm lint ops/helm/kura
helm template kura ops/helm/kura --namespace kura

Enable grpcIngress when the Bazel Remote Execution API should be reachable outside the cluster. It renders a separate ingress that routes to the service's grpc port so you can attach controller-specific gRPC annotations without changing the HTTP API ingress:

grpcIngress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  hosts:
    - host: kura-grpc.example.com
      paths:
        - path: /
          pathType: Prefix

Install it on a generic cluster:

helm upgrade --install kura ./ops/helm/kura \
  --namespace kura \
  --create-namespace \
  --set image.repository=<registry>/kura \
  --set image.tag=latest \
  --set config.region=fr-par \
  --set config.telemetry.otlpTracesEndpoint=http://otel-collector.monitoring.svc.cluster.local:4318/v1/traces

The chart defaults persistence to ReadWriteOncePod so one Kura process owns each PVC. If your CSI driver does not support it, override persistence.accessModes[0]=ReadWriteOnce; Kura will still fence the volume with its app-level writer lock.

The chart computes terminationGracePeriodSeconds from config.shutdown.drainCompletionTimeoutMs, podLifecycle.preStopDelaySeconds, and podLifecycle.terminationGraceExtraSeconds. That keeps the platform budget aligned with the application's shared shutdown deadline instead of relying on a separate hard-coded Kubernetes timeout.

For a local kind smoke test, the repo includes:

./test/e2e/kura_helm_kind.sh

For a gated in-place StatefulSet rollout, the repo also includes:

./ops/helm/kura/rollout.sh kura kura --set image.tag=<new-tag>

That script is the Kubernetes adapter. The rollout gate itself lives in ops/rollout/gate.sh and only assumes it can fetch Kura's rollout status endpoint once per node per poll. The Helm adapter stages the new revision behind a StatefulSet partition, rolls the highest ordinal first, and only advances after every node reports the same membership generation, all nodes are back in serving, the updated pod stays ready, ring membership is restored cluster-wide, outbox depth stays near baseline, no node is under critical memory pressure, and the cluster is not introducing new file-descriptor timeout activity.

If the Kura container listens on a non-default HTTP port, set KURA_HTTP_PORT=<port> when invoking the rollout helper so the adapter samples the correct loopback endpoint inside each pod.

For adjacent-version mixed rollout and rollback validation on the same persistent Docker volumes, use:

PREVIOUS_REF=origin/main ./test/e2e/kura_compatibility_rollout.sh

That harness proves PREVIOUS_REF -> HEAD -> PREVIOUS_REF across a mixed-version window, but it validates protocol and on-disk compatibility only. It does not try to model Kubernetes PVC reattachment behavior.

To enable peer mTLS in Kubernetes, set:

• peerTls.enabled=true
• peerTls.internalPort=<port>
• peerTls.secretName=<secret-with-ca-cert-and-key-material>

The referenced secret should contain the files configured by:

• peerTls.caCertFileName
• peerTls.certFileName
• peerTls.keyFileName

When enabled, the chart advertises peer URLs over https on the internal port and mounts the secret into /etc/kura/peer-tls.

Scaleway Kapsule

For Scaleway, start from the bundled overrides in ops/helm/kura/values-scaleway.yaml:

helm upgrade --install kura ./ops/helm/kura \
  --namespace kura \
  --create-namespace \
  -f ./ops/helm/kura/values-scaleway.yaml \
  --set image.repository=<registry>/kura \
  --set image.tag=latest \
  --set config.region=fr-par \
  --set config.telemetry.otlpTracesEndpoint=http://otel-collector.monitoring.svc.cluster.local:4318/v1/traces

That values file does two important things:

• 🚪 uses a LoadBalancer service, which is the simplest way to expose Kura on Kapsule
• 💾 pins persistence to scw-bssd, which Scaleway documents as the default block storage class for Kapsule multi-AZ clusters

🧩 Extensions And Policy

Kura can load one operator-provided extension script at startup to customize authentication, authorization, and response headers without recompiling the binary.

Core env vars:

• KURA_EXTENSION_ENABLED=true
• KURA_EXTENSION_SCRIPT_PATH=/etc/kura/extensions/hooks.lua
• KURA_EXTENSION_HOOK_TIMEOUT_MS=25
• KURA_EXTENSION_AUTH_CACHE_ALLOW_TTL_SECONDS=600
• KURA_EXTENSION_AUTH_CACHE_DENY_TTL_SECONDS=3
• KURA_EXTENSION_FAIL_CLOSED_AUTHENTICATE=true
• KURA_EXTENSION_FAIL_CLOSED_AUTHORIZE=true
• KURA_EXTENSION_FAIL_OPEN_RESPONSE_HEADERS=true

Generic host resources are also env-driven:

• ✍️ signers:
  • KURA_EXTENSION_SIGNER_<ID>_ALGORITHM
  • KURA_EXTENSION_SIGNER_<ID>_SECRET
• 🪪 JWT verifiers:
  • KURA_EXTENSION_JWT_VERIFIER_<ID>_ALGORITHM
  • KURA_EXTENSION_JWT_VERIFIER_<ID>_SECRET
  • KURA_EXTENSION_JWT_VERIFIER_<ID>_ISSUER
  • KURA_EXTENSION_JWT_VERIFIER_<ID>_AUDIENCES
• 🌐 HTTP clients:
  • KURA_EXTENSION_HTTP_CLIENT_<ID>_BASE_URL
  • KURA_EXTENSION_HTTP_CLIENT_<ID>_CONNECT_TIMEOUT_MS
  • KURA_EXTENSION_HTTP_CLIENT_<ID>_REQUEST_TIMEOUT_MS

The script may define these hooks:

• authenticate(ctx)
• authorize(ctx, principal)
• response_headers(ctx, principal)

The runtime exports these host helpers to the script:

• kura.sign_hmac_base64(id, payload)
• kura.jwt_verify(id, token)
• kura.http_json(id, request)
• kura.env(key)

For tenant-aware deployments, ctx carries the request target (tenant_id, namespace_id) and the node's configured tenant as server_tenant_id (derived from KURA_TENANT_ID). Namespace-scoped requests set namespace_id; tenant-scoped requests leave it unset. Concrete auth and policy decisions are hook-specific and should be documented with the hook implementation rather than in the generic runtime contract.

The runtime keeps decision caching, metrics, timeouts, and cryptographic primitives in Rust, while the script supplies policy.

Hook results are cached per credentials: the cache key fingerprints only the credential-bearing request headers (authorization, proxy-authorization, cookie, x-api-key), so per-request noise such as gRPC deadlines (grpc-timeout) or trace propagation (traceparent) does not defeat the cache. Hooks that authenticate from other inputs must return a short ttl_seconds (or none) instead of relying on the cache key to separate requests.