Back to Tuist

Self-hosted cache {#self-hosted-cache}

server/priv/docs/en/guides/features/cache/self-hosting.md

4.205.016.2 KB
Original Source

Self-hosted cache {#self-hosted-cache}

Self-hosted cache nodes let you keep build artifacts and cache metadata close to the machines that produce and consume build outputs. Use them when cache latency matters across CI, developer offices, remote workstations, or regional compute clusters, while keeping endpoint discovery centralized through Tuist.

The goal is low-latency caching everywhere, not only in the one environment where a central cache happens to be nearby. Each cache node serves reads and writes from local disk, while the mesh replicates artifacts and metadata between peers so other locations can benefit from the same cache over time.

[!NOTE] Tuist's self-hosted cache nodes are powered by Kura, Tuist's decentralized cache mesh. Kura is the data plane for cache nodes: it serves cache reads and writes, stores local state on disk, and replicates artifacts and metadata to peer nodes.

How self-hosted cache fits with Tuist {#how-self-hosted-cache-fits-with-tuist}

The Tuist server tells clients which cache endpoints to use. This keeps endpoint discovery centralized while allowing the cache itself to stay decentralized and close to the compute that needs it.

Deploy on Kubernetes {#deploy-on-kubernetes}

Kura is distributed as a Helm chart through GitHub Container Registry. It deploys Kura as a StatefulSet with persistent volumes, a headless service for peer discovery, and a regular service for HTTP and gRPC traffic.

bash
helm upgrade --install kura oci://ghcr.io/tuist/charts/kura \
  --namespace kura \
  --create-namespace \
  --version <version> \
  --set image.tag=<tag> \
  --set config.region=local

For a self-hosted Tuist server running in the same cluster, tell the server which cache endpoints to hand to clients:

yaml
server:
  cacheEndpointUrl: "http://kura.kura.svc.cluster.local:4000"

This renders TUIST_CACHE_ENDPOINTS in the server pod. On a self-hosted server the CLI is routed to whatever TUIST_CACHE_ENDPOINTS lists, so point it at your Kura service. For multiple nodes, use a comma-separated list.

[!IMPORTANT] Every Kura node must own its own KURA_DATA_DIR. Kura takes an application-level writer lock on the data directory and expects exactly one process to own it. In Kubernetes, use one persistent volume per pod. Outside Kubernetes, do not point multiple processes at the same mounted directory.

Deploy without Kubernetes {#deploy-without-kubernetes}

Kura can also run as a regular container on VMs or bare-metal hosts. In this mode, you are responsible for process supervision, persistent storage, routing, and peer discovery.

At minimum, each node needs a persistent data directory, a temporary directory, a public cache port (one listener serves both the HTTP cache API and REAPI gRPC), an internal peer URL, and either a static peer list or a discovery mechanism:

bash
docker run -d --name kura \
  -p 4000:4000 \
  -p 7443:7443 \
  -v /var/lib/kura:/var/cache/kura \
  -e KURA_PORT=4000 \
  -e KURA_INTERNAL_PORT=7443 \
  -e KURA_TENANT_ID=default \
  -e KURA_REGION=local \
  -e KURA_TMP_DIR=/var/cache/kura/tmp \
  -e KURA_DATA_DIR=/var/cache/kura \
  -e KURA_NODE_URL=http://kura-1.internal:7443 \
  -e KURA_PEERS=http://kura-1.internal:7443,http://kura-2.internal:7443 \
  ghcr.io/tuist/kura:<tag>

Then configure the Tuist server with the URLs that clients can reach:

bash
TUIST_CACHE_ENDPOINTS=https://kura-1.example.com,https://kura-2.example.com

Build a cache mesh {#build-a-cache-mesh}

A cache mesh lets you place cache capacity next to the compute that needs it. A company might run one node near its main CI runners, another close to developers in Europe, and another near a US office or regional build cluster. Each location reads and writes against the closest node, while Kura replicates artifacts and metadata in the background so later builds in other locations can reuse the same outputs.

The mesh only works if nodes can reach each other on Kura's internal peer port. That peer plane is separate from the public cache endpoints that Tuist clients use. Kura uses it to check membership, backfill newly joined nodes, and replicate artifacts after local writes are accepted.

We strongly recommend securing the peer plane with mTLS when nodes communicate across regions, clouds, VPCs, offices, or any network that is not fully private to the cache deployment. With mTLS enabled, Kura only serves internal replication endpoints to peers presenting a certificate signed by the configured CA. The peer certificates must cover the DNS names nodes use to call each other, and peer URLs must use https:// on the internal port.

For example, this generates a private CA and one peer certificate that can be mounted by every node in a small mesh. Replace the DNS names with the internal names your nodes use in KURA_NODE_URL and KURA_PEERS.

bash
mkdir -p kura-peer-tls
cd kura-peer-tls

openssl ecparam -genkey -name prime256v1 -noout -out ca.key
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
  -subj "/CN=kura-peer-ca" \
  -out ca.pem

openssl ecparam -genkey -name prime256v1 -noout -out tls.key
openssl req -new -key tls.key \
  -subj "/CN=kura-peer" \
  -out peer.csr

cat > peer.ext <<'EOF'
subjectAltName = DNS:kura-0.kura-headless.kura.svc.cluster.local,DNS:kura-1.kura-headless.kura.svc.cluster.local,DNS:kura-2.kura-headless.kura.svc.cluster.local,DNS:kura-1.internal,DNS:kura-2.internal
extendedKeyUsage = serverAuth,clientAuth
keyUsage = digitalSignature,keyEncipherment
EOF

openssl x509 -req -in peer.csr \
  -CA ca.pem \
  -CAkey ca.key \
  -CAcreateserial \
  -out tls.crt \
  -days 730 \
  -sha256 \
  -extfile peer.ext

Catch-up between nodes {#catch-up-between-nodes}

A node that joins or rejoins a mesh backfills: it walks each peer's entries newest-first and pulls what it is missing, while live replication covers writes made while the peer is in view. A backfill pass reaches back from the peer's newest entry to the older of the node's segment-ring horizon and its last completed pass, so a node is guaranteed to hold the recent data, not necessarily all of it. KURA_BACKFILL_MARGIN_PERCENT sets how far back the horizon sits.

Two consequences are worth knowing before you run a mesh:

  • A node reports /ready once its segment ring is KURA_BACKFILL_READY_RING_PERCENT full or its first backfill cycle settles, and readiness then holds for the rest of the process lifetime. A later peer flap never takes a serving node out of rotation.
  • A cycle also settles when a peer is unreachable for long enough. A node whose only peer is down therefore becomes Ready while holding little or no data. It serves misses rather than errors. /status/rollout reports backfill_initial_cycle (pending, complete or degraded) if you want to gate a rollout on the difference.

Upgrading a mesh across the backfill change. Kura releases before backfill catch up through a different, now-removed peer protocol. Roll every node onto a release that runs backfill before you take a release that only speaks backfill. Nodes on the two protocols cannot catch up from each other: a warm node keeps serving from its volume but stops closing its gap, and a node that starts on an empty volume in that window never becomes Ready. The chart sets KURA_BACKFILL_ENABLED=true for you, so a chart-first upgrade covers this. A single helm upgrade that moves the chart and the image together does not.

Use network-level restrictions in addition to mTLS. In Kubernetes, run Kura as a StatefulSet with one persistent volume per pod and a headless service for peer discovery, then allow the internal peer port only between pods that belong to the same cache deployment, for example with a NetworkPolicy. Outside Kubernetes, give each node a stable DNS name or IP address, seed the mesh with the internal URLs of the other nodes, and use firewall rules or security groups so only cache nodes can reach the peer port. Public cache traffic should enter through the public HTTP or gRPC endpoints, not through the internal peer plane.

Configuration {#configuration}

The Helm chart renders the common runtime settings from values.yaml. If you run Kura without Kubernetes, set the same variables directly on the process. Variables that the chart does not map directly can be injected through extraEnv or extraEnvFrom.

Environment variableDescriptionRequiredDefaultHelm value
KURA_PORTPublic cache port. One listener serves the HTTP cache API, health endpoints, and Bazel/Buck2 REAPI gRPC (h2c).YesNo defaultservice.httpPort
KURA_INTERNAL_PORTInternal HTTP or mTLS port used by Kura peers.YesNo defaultpeerTls.internalPort
KURA_TENANT_IDDefault tenant identifier for the node.YesNo defaultconfig.tenantId
KURA_REGIONRegion label used in metrics and replication state.YesNo defaultconfig.region
KURA_TMP_DIRTemporary directory for staged request bodies and multipart assembly.YesNo defaultFixed to /var/cache/kura/tmp
KURA_TMP_DIR_MAX_BYTESMaximum staged bytes admitted into KURA_TMP_DIR before requests receive backpressure.No8589934592config.tmpDirMaxBytes
KURA_DATA_DIRPersistent directory for metadata state and segment files.YesNo defaultFixed to /var/cache/kura
KURA_NODE_URLCanonical internal URL other peers use to reach this node.YesNo defaultDerived from the pod DNS name and peerTls.internalPort
KURA_PEERSSeed peer list used before discovery converges.NoKURA_NODE_URLDerived from the StatefulSet replicas
KURA_DISCOVERY_DNS_NAMEDNS name used for automatic peer discovery.NoDisabledEnabled by config.discovery.enabled
KURA_INTERNAL_TLS_CA_CERT_PATHCA certificate used to verify peer mTLS.NoDisabledpeerTls.enabled and peerTls.caCertFileName
KURA_INTERNAL_TLS_CERT_PATHCertificate used by the internal peer mTLS listener.NoDisabledpeerTls.enabled and peerTls.certFileName
KURA_INTERNAL_TLS_KEY_PATHPrivate key used by the internal peer mTLS listener.NoDisabledpeerTls.enabled and peerTls.keyFileName
KURA_PUBLIC_TLS_CERT_PATHCertificate used to terminate TLS on the co-hosted HTTPS listener (KURA_HTTPS_PORT).NoDisabledextraEnv
KURA_PUBLIC_TLS_KEY_PATHPrivate key paired with KURA_PUBLIC_TLS_CERT_PATH.NoDisabledextraEnv
KURA_HTTPS_PORTTLS port serving the same co-hosted HTTP + gRPC surface (ALPN-negotiated). Only bound when the public TLS paths are set.No4443extraEnv
KURA_FILE_DESCRIPTOR_POOL_SIZEFile-descriptor budget for request and background I/O.NoAuto-derivedconfig.fileDescriptors.poolSize
KURA_FILE_DESCRIPTOR_ACQUIRE_TIMEOUT_MSHow long a request waits before FD backpressure fails the checkout.No5000config.fileDescriptors.acquireTimeoutMs
KURA_SEGMENT_HANDLE_CACHE_SIZEMaximum number of pinned segment read handles.NoAuto-derivedconfig.fileDescriptors.segmentHandleCacheSize
KURA_DRAIN_COMPLETION_TIMEOUT_MSGrace window for in-flight HTTP and gRPC work during shutdown.No240000config.shutdown.drainCompletionTimeoutMs
KURA_MEMORY_SOFT_LIMIT_BYTESSoft memory watermark where Kura starts reducing optional memory use.NoAuto-derivedconfig.memory.softLimitBytes
KURA_MEMORY_HARD_LIMIT_BYTESHard memory watermark where Kura pauses replication and trims hot caches.NoAuto-derivedconfig.memory.hardLimitBytes
KURA_MANIFEST_CACHE_MAX_BYTESMaximum size of the in-memory manifest cache.NoAuto-derivedconfig.memory.manifestCacheMaxBytes
KURA_MAX_KEYVALUE_BYTESMaximum per-request keyvalue payload size.No1048576config.memory.maxKeyvalueBytes
KURA_METADATA_STORE_MAX_OPEN_FILESFile descriptor budget reserved for the metadata store.NoAuto-derivedconfig.metadataStore.maxOpenFiles
KURA_METADATA_STORE_MAX_BACKGROUND_JOBSBackground flush and compaction concurrency for the metadata store.NoAuto-derivedconfig.metadataStore.maxBackgroundJobs
KURA_METADATA_STORE_READ_CACHE_BYTESCapacity of the metadata-store read cache.NoAuto-derivedextraEnv
KURA_METADATA_STORE_WRITE_BUFFER_POOL_BYTESTotal memory budget reserved for metadata write buffering.NoAuto-derivedextraEnv
KURA_METADATA_STORE_WRITE_BUFFER_BYTESSize of each metadata write buffer before flush.NoAuto-derivedextraEnv
KURA_METADATA_STORE_MAX_WRITE_BUFFERSMaximum number of metadata write buffers kept in memory.NoAuto-derivedextraEnv
KURA_OUTBOX_MAX_DEPTHMaximum replication outbox depth before public writes return backpressure.No100000extraEnv
KURA_MULTIPART_UPLOAD_TTL_MSHow long an in-progress multipart upload may sit before expiring.No86400000extraEnv
KURA_MULTIPART_JANITOR_INTERVAL_MSHow often Kura scans for stale multipart uploads.No600000extraEnv
KURA_BACKFILL_MARGIN_PERCENTShare of the age-ordered segment ring, counted from the newest, that bounds how far back a backfill pass reaches.No40config.backfill.marginPercent
KURA_BACKFILL_READY_RING_PERCENTSegment-ring fullness at which a node still running its first backfill cycle reports itself ready.NoHalf of KURA_BACKFILL_MARGIN_PERCENTconfig.backfill.readyRingPercent
KURA_BACKFILL_BATCH_BYTESByte threshold one backfill bodies batch is composed against, and the size above which an entry is fetched on its own.No33554432config.backfill.batchBytes
KURA_TOKIO_WORKER_THREADSNumber of Tokio worker threads.NoAuto-derivedextraEnv
KURA_OTEL_EXPORTER_OTLP_TRACES_ENDPOINTOTLP traces endpoint. Leave empty to disable tracing.NoDisabledconfig.telemetry.otlpTracesEndpoint
KURA_OTEL_SERVICE_NAMEOpenTelemetry service name.YesNo defaultPod name in Helm
KURA_OTEL_DEPLOYMENT_ENVIRONMENTOpenTelemetry deployment environment.YesNo defaultconfig.telemetry.deploymentEnvironment
KURA_SENTRY_DSNSentry DSN for panic and error reporting.NoDisabledextraEnv or extraEnvFrom
KURA_NODE_COUNTRYISO 3166-1 alpha-2 country of the datacenter the node runs in, stamped on exported traces as geo.country.iso_code.NoDerived from KURA_REGION when its prefix is a country codeextraEnv
KURA_NODE_SUBDIVISIONISO 3166-2 subdivision of that datacenter (e.g. US-CA), stamped as geo.region.iso_code.NoUnsetextraEnv
KURA_AUTH_ENABLEDRequires callers to present a valid Tuist token. Setting KURA_AUTH_TUIST_URL enables authorization on its own, and setting this to false does not turn it back off — unset the URL to run a node without authorization.NoDisabledauth.enabled
KURA_AUTH_TUIST_URLTuist server the node authorizes against. Enables authorization on its own.Required when authorization is enabledNo defaultauth.tuistUrl
KURA_AUTH_TUIST_CONNECT_TIMEOUT_MSConnect timeout for calls to that server.No500extraEnv
KURA_AUTH_TUIST_REQUEST_TIMEOUT_MSRequest timeout for calls to that server.No1500extraEnv
KURA_AUTH_JWT_SECRETVerification key for tokens the node can read itself, skipping a round trip. Self-hosted nodes normally leave this unset.NoDisabledextraEnv or extraEnvFrom
KURA_AUTH_JWT_ALGORITHMAlgorithm for that key (HS256, HS384 or HS512).NoHS256extraEnv
KURA_AUTH_JWT_ISSUERIssuer that tokens must carry.NoUncheckedextraEnv
KURA_AUTH_JWT_AUDIENCESComma-separated audiences that tokens must carry.NoUncheckedextraEnv
KURA_AUTH_CACHE_MAX_ENTRIESMaximum entries kept in each of the authentication and authorization caches.No100000extraEnv

If you enable internal peer mTLS, set KURA_INTERNAL_TLS_CA_CERT_PATH, KURA_INTERNAL_TLS_CERT_PATH, and KURA_INTERNAL_TLS_KEY_PATH together. KURA_NODE_URL and every value in KURA_PEERS must then use https:// with the internal peer port.