doc/user/clusters/agent/getting_started_deployments.md
This page introduces you to deploying to Kubernetes using methods supported by GitLab. In the end, you will understand:
This tutorial builds on the project you created in Get started connecting a Kubernetes cluster to GitLab. You'll use the same project you created in that tutorial. However, you can use any project with a connected Kubernetes cluster and a bootstrapped Flux installation.
The agent for Kubernetes integrates with GitLab CI/CD pipelines. You can use CI/CD to run commands like kubectl apply and helm upgrade against your cluster in a secure and scalable way.
In this section, you'll use the GitLab pipeline integration to create a secret in the cluster and use it to access the GitLab container registry. The rest of this tutorial will use the deployed secret.
Create a deploy token with the read_registry scope.
Save your deploy token and username as CI/CD variables called CONTAINER_REGISTRY_ACCESS_TOKEN and CONTAINER_REGISTRY_ACCESS_USERNAME.
container-registry-secret*.CONTAINER_REGISTRY_ACCESS_TOKEN:
Add the following snippet to your .gitlab-ci.yml file, and update both AGENT_KUBECONTEXT variables to match your project's path:
stages:
- setup
- deploy
- stop
create-registry-secret:
stage: setup
image: "portainer/kubectl-shell:latest"
variables:
AGENT_KUBECONTEXT: my-group/optional-subgroup/my-repository:testing
before_script:
# The available agents are automatically injected into the runner environment
# You need to select the agent to use
- kubectl config use-context $AGENT_KUBECONTEXT
script:
- kubectl delete secret gitlab-registry-auth -n flux-system --ignore-not-found
- kubectl create secret docker-registry gitlab-registry-auth -n flux-system
--docker-password="${CONTAINER_REGISTRY_ACCESS_TOKEN}" --docker-username="${CONTAINER_REGISTRY_ACCESS_USERNAME}" --docker-server="${CI_REGISTRY}"
environment:
name: container-registry-secret
on_stop: delete-registry-secret
delete-registry-secret:
stage: stop
image: ""
variables:
AGENT_KUBECONTEXT: my-group/optional-subgroup/my-repository:testing
before_script:
# The available agents are automatically injected into the runner environment
# You need to select the agent to use
- kubectl config use-context $AGENT_KUBECONTEXT
script:
- kubectl delete secret -n flux-system gitlab-registry-auth
environment:
name: container-registry-secret
action: stop
when: manual
Before you continue, consider how you might run other commands with CI/CD.
For production use cases, it is a best practice to use an OCI repository as a caching layer between the Git repository and FluxCD. FluxCD checks for new images in the OCI repository, while GitLab pipeline builds the Flux-compliant OCI images. To learn more about enterprise best practices, see enterprise considerations.
In this section, you'll build a simple Kubernetes manifest as an OCI artifact, then deploy it to your cluster.
Run the following flux CLI commands to tell Flux where to retrieve the specified OCI image and deploy its content.
Adjust the --url value for your GitLab instance. You can find the container registry URL under Deploy > Container registry.
You can inspect the created clusters/testing/nginx.yaml file to better understand how Flux finds the manifests to deploy.
flux create source oci nginx-example \
--url oci://registry.gitlab.example.org/my-group/optional-subgroup/my-repository/nginx-example \
--tag latest \
--secret-ref gitlab-registry-auth \
--interval 1m \
--namespace flux-system \
--export > clusters/testing/nginx.yaml
flux create kustomization nginx-example \
--source OCIRepository/nginx-example \
--path "." \
--prune true \
--target-namespace default \
--interval 1m \
--namespace flux-system \
--export >> clusters/testing/nginx.yaml
Deploy NGINX as an example. Add the following YAML to clusters/applications/nginx/nginx.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-example
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: nginx-example
template:
metadata:
labels:
app: nginx-example
spec:
containers:
- name: nginx
image: nginx:1.25
ports:
- containerPort: 80
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: nginx-example
namespace: default
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
app: nginx-example
Now, let's package the previous YAML into an OCI image.
Extend your .gitlab-ci.yml file with the following snippet, and again update the AGENT_KUBECONTEXT variable:
nginx-deployment:
stage: deploy
variables:
IMAGE_NAME: nginx-example # Image name to push
IMAGE_TAG: latest
MANIFEST_PATH: "./clusters/applications/nginx"
IMAGE_TITLE: NGINX example # Image title to use in OCI annotation
AGENT_KUBECONTEXT: my-group/optional-subgroup/my-repository:testing
FLUX_OCI_REPO_NAME: nginx-example # Flux OCIRepository to reconcile
NAMESPACE: flux-system # Namespace for the OCIRepository resource
# This section configures a GitLab environment for the nginx deployment specifically
environment:
name: applications/nginx
kubernetes:
agent: $AGENT_KUBECONTEXT
dashboard:
namespace: default
flux_resource_path: kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations/nginx-example # You will deploy this resource in the next step
image:
name: "fluxcd/flux-cli:v2.4.0"
entrypoint: [""]
before_script:
- kubectl config use-context $AGENT_KUBECONTEXT
script:
# This line builds and pushes the OCI container to the GitLab container registry.
# You can read more about this command in https://fluxcd.io/flux/cmd/flux_push_artifact/
- flux push artifact oci://${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${IMAGE_TAG}
--source="${CI_REPOSITORY_URL}"
--path="${MANIFEST_PATH}"
--revision="${CI_COMMIT_SHORT_SHA}"
--creds="${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD}"
--annotations="org.opencontainers.image.url=${CI_PROJECT_URL}"
--annotations="org.opencontainers.image.title=${IMAGE_TITLE}"
--annotations="com.gitlab.job.id=${CI_JOB_ID}"
--annotations="com.gitlab.job.url=${CI_JOB_URL}"
# This line triggers an immediate reconciliation of the resource. Otherwise Flux would reconcile following its configured reconciliation period.
# You can read more about the various reconcile commands in https://fluxcd.io/flux/cmd/flux_reconcile/
- flux reconcile source oci -n ${NAMESPACE} ${FLUX_OCI_REPO_NAME}
Commit and push the changes to your project, and wait for the build pipeline to finish.
In the left sidebar, select Operate > Environments and check the available dashboard for Kubernetes.
The applications/nginx environment should be healthy.
{{< details >}}
{{< /details >}}
The previously deployed agent is configured using the .gitlab/agents/testing/config.yaml file.
By default, the configuration enables access to the clusters configured in the project where the GitLab pipelines run.
By default, this access uses the deployed agent's service account to run commands against the cluster.
This access can be restricted either to a static service account identity or by using the CI/CD job as an identity in the cluster.
Finally, regular Kubernetes RBAC can be used to limit the access of the CI/CD jobs in the cluster.
This section shows how to restrict CI/CD access by adding an identity to every CI/CD job, and impersonating the job in the cluster.
To configure the CI/CD job impersonation, edit the .gitlab/agents/testing/config.yaml file, and add the following snippet to it (replacing path/to/project):
ci_access:
projects:
- id: my-group/optional-subgroup/my-repository
access_as:
ci_job: {}
Because the CI/CD jobs don't have any cluster bindings yet, you can't run any Kubernetes commands from GitLab CI/CD.
Let's enable CI/CD jobs to create Secret objects in the flux-system namespace.
Create the clusters/testing/gitlab-ci-job-secret-write.yaml file with the following content:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secret-manager
namespace: default
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: gitlab-ci-secrets-binding
namespace: default
subjects:
- kind: Group
name: gitlab:ci_job
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: secret-manager
apiGroup: rbac.authorization.k8s.io
Let's enable CI/CD jobs to trigger a FluxCD reconciliation too.
Create the clusters/testing/gitlab-ci-job-flux-reconciler.yaml file with the following content:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ci-job-admin
roleRef:
name: flux-edit-flux-system
kind: ClusterRole
apiGroup: rbac.authorization.k8s.io
subjects:
- name: gitlab:ci_job
kind: Group
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ci-job-view
roleRef:
name: flux-view-flux-system
kind: ClusterRole
apiGroup: rbac.authorization.k8s.io
subjects:
- name: gitlab:ci_job
kind: Group
For more information about CI/CD access, see Using GitLab CI/CD with a Kubernetes cluster.
To finish, remove the deployed resources and delete the secret you used to access the container registry:
clusters/testing/nginx.yaml file.
Flux will take care of removing the related resources from the cluster.container-registry-secret environment.
Stopping the environment will trigger its on_stop job, removing the secret from the cluster.You can use the techniques in this tutorial to scale deployments across projects. The OCI image can be built in a different project, and as long as Flux is pointed at the right registry, Flux will retrieve it. This exercise is left for the reader.
For more practice, try changing the original Flux GitRepository in /clusters/testing/flux-system/gotk-sync.yaml to an OCIRepository.
Finally, see the following resources for more information about Flux and the GitLab integration with Kubernetes: