ContextQMD
Back to Nerdctl

Example: Node-to-Node image sharing on Kubernetes using `nerdctl ipfs registry`

examples/nerdctl-ipfs-registry-kubernetes/ipfs/README.md

2.2.23.3 KB
Original Source

Example: Node-to-Node image sharing on Kubernetes using nerdctl ipfs registry

This directory contains an example Kubernetes setup for node-to-node image sharing.

Usage:

Example on kind

Prepare cluster (make sure kind nodes contain containerd >= v1.5.8).

console
$ cat <<EOF > /tmp/kindconfig.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
$ kind create cluster --image=kindest/node:v1.25.2 --config=/tmp/kindconfig.yaml
$ ./bootstrap.yaml.sh > ./bootstrap.yaml
$ kubectl apply -f .

Prepare kind-worker (1st node) for importing an image to IPFS

(in kind-worker)

console
$ docker exec -it kind-worker /bin/bash
(kind-worker)# NERDCTL_VERSION=0.23.0
(kind-worker)# curl -fsSL --proto '=https' --tlsv1.2 --output /tmp/nerdctl.tgz https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VERSION}/nerdctl-${NERDCTL_VERSION}-linux-amd64.tar.gz
(kind-worker)# tar zxvf /tmp/nerdctl.tgz -C /usr/local/bin/

Add an image to kind-worker.

console
$ docker exec -it kind-worker /bin/bash
(kind-worker)# mkdir -p /tmp/ipfsapi ; echo -n /ip4/127.0.0.1/tcp/5001 >  /tmp/ipfsapi/api
(kind-worker)# export IPFS_PATH=/tmp/ipfsapi
(kind-worker)# nerdctl pull ghcr.io/stargz-containers/jenkins:2.60.3-org
(kind-worker)# nerdctl push ipfs://ghcr.io/stargz-containers/jenkins:2.60.3-org
(kind-worker)# nerdctl rmi ghcr.io/stargz-containers/jenkins:2.60.3-org

The image added to kind-worker is shared to kind-worker2 via IPFS. You can run this image on all worker nodes using the following manifest. CID of the pushed image is printed when nerdctl push succeeded (we assume that the image is added to IPFS as CID bafkreictyyoysj56v772xbfhyfrcvmgmfpa4vodmqaroz53ytvai7nof6u).

console
$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 2
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
      - name: jenkins
        image: localhost:5050/ipfs/bafkreictyyoysj56v772xbfhyfrcvmgmfpa4vodmqaroz53ytvai7nof6u
        resources:
          requests:
            cpu: 1
EOF

NOTE: Kubernetes doesn't support ipfs://CID URL on YAML as of now so we need to use localhost:5050/ipfs/CID form instead. In the future, this limitation should be eliminated.

The image runs on all nodes.

console
$ kubectl get pods -owide | grep jenkins
jenkins-7bd8f96d79-2jbc6          1/1     Running   0          69s    10.244.1.3   kind-worker    <none>           <none>
jenkins-7bd8f96d79-jb5lm          1/1     Running   0          69s    10.244.2.4   kind-worker2   <none>           <none>