docs/current_docs/reference/deployment/kubernetes.mdx
This section covers different strategies for deploying Dagger on a Kubernetes cluster. Its primary focus are teams that looking forward to self-host their CI infrastructure and use Dagger at its core to run their pipelines.
Running Dagger in Kubernetes is generally a good choice for:
This section describe a few common architecture patterns to consider when setting up a Continuous Integration (CI) environment using Dagger on Kubernetes.
The base pattern consists of persistent Kubernetes nodes with ephemeral CI runners.
The minimum required components are:
In this architecture:
graph LR
subgraph External APIs
SCM
end
rc --> SCM
controlplane --> runner-nodes
subgraph Kubernetes cluster
controlplane{{ Kubernetes control plane }}
rc -.-> runner1
rc -.-> runnern.1
rc -.-> runnern.2
subgraph support-nodes[Support nodes]
rc{{ Runner controller }}
cert-manager
end
subgraph runner-nodes[Runner nodes]
direction BT
subgraph node-n[Node n]
runnern.1{{ CI runner }} --> dagger-engine2[Dagger Engine]
runnern.2{{ CI runner }} --> dagger-engine2[Dagger Engine]
end
subgraph node-1[Node 1]
runner1{{ CI runner }} --> dagger-engine[Dagger Engine]
end
end
end
The base architecture pattern described previously can be optimized by the addition of a node auto-scaler. This can automatically adjust the size of node groups based on the current workload. If there are a lot of CI jobs running, the auto-scaler can automatically add more runner nodes to the cluster to handle the increased workload. Conversely, if there are few jobs running, it can remove unnecessary runner nodes.
This optimization reduces the total compute cost since runner nodes are added & removed based on the number of concurrent CI jobs. The trade-off is that we lose all Dagger Engine state stored on the runner node that is de-provisioned. Configuring persistent volumes is worth considering if the platform supports it.
In this architecture:
graph LR
subgraph External APIs
SCM
end
rc --> SCM
controlplane --> runner-nodes
as --> controlplane
subgraph Kubernetes cluster
controlplane{{ Kubernetes control plane }}
rc -.-> runner1
rc -.-> runnern.1
rc -.-> runnern.2
subgraph support-nodes[Support nodes]
rc{{ Runner controller }}
cert-manager
as{{Node auto-scaler}}
end
subgraph runner-nodes[Runner nodes]
direction BT
subgraph node-n[Node n]
runnern.1{{ CI runner }} --> dagger-engine2[Dagger Engine]
runnern.2{{ CI runner }} --> dagger-engine2[Dagger Engine]
end
subgraph node-1[Node 1]
runner1{{ CI runner }} --> dagger-engine[Dagger Engine]
end
end
end
When deploying Dagger on a Kubernetes cluster, it's important to understand the design constraints you're operating under, so you can optimize your configuration to suit your workload requirements. Here are two key recommendations.
The Dagger Engine cache is used to store intermediate build artifacts, which can significantly speed up your CI jobs. However, this cache can grow very large over time. By choosing nodes with large NVMe drives, you ensure that there is plenty of space for this cache. NVMe drives are also much faster than traditional SSDs, which can further improve performance. These types of drives are usually ephemeral to the node and much less expensive relative to EBS-type volumes.
Although this will obviously vary based on workloads, a minimum of 2 vCPUs and 8GB of RAM is a good place to start. One approach is to set up the CI runners with various sizes so that the Dagger Engine can consume resources from the runners on the same node if needed.
kubectl profile.Dagger provides a Helm chart to create a Dagger Engine DaemonSet on a Kubernetes cluster. A DaemonSet ensures that all matching nodes run an instance of Dagger Engine.
The following command uses the Dagger Helm chart to create a Dagger Engine DaemonSet on the cluster:
helm upgrade --install --namespace=dagger --create-namespace \
dagger oci://registry.dagger.io/dagger-helm
This Dagger Engine DaemonSet configuration is designed to:
Wait for the Dagger Engine to become ready:
kubectl wait --for condition=Ready --timeout=60s pod \
--selector=name=dagger-dagger-helm-engine --namespace=dagger
Find more information on what was deployed using the following command:
kubectl describe daemonset/dagger-dagger-helm-engine --namespace=dagger
Get a Dagger Engine pod name:
DAGGER_ENGINE_POD_NAME="$(kubectl get pod \
--selector=name=dagger-dagger-helm-engine --namespace=dagger \
--output=jsonpath='{.items[0].metadata.name}')"
export DAGGER_ENGINE_POD_NAME
Next, set the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable so that the Dagger CLI knows to connect to the Dagger Engine that you deployed as a Kubernetes pod:
_EXPERIMENTAL_DAGGER_RUNNER_HOST="kube-pod://$DAGGER_ENGINE_POD_NAME?namespace=dagger"
export _EXPERIMENTAL_DAGGER_RUNNER_HOST
Finally, run an operation that shows the kernel info of the Kubernetes node where this Dagger Engine runs:
dagger query <<EOF
{
container {
from(address:"alpine") {
withExec(args: ["uname", "-a"]) { stdout }
}
}
}
EOF
Below are some resources from the Dagger community that may help as well. If you have any questions about additional ways to use Kubernetes with Dagger, join our Discord and ask your questions in our Kubernetes channel.
Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation.