⚠️ DEPRECATED

This chart was moved to https://github.com/jenkinsci/helm-charts

Jenkins Helm Chart

Jenkins master and agent cluster utilizing the Jenkins Kubernetes plugin

• https://plugins.jenkins.io/kubernetes

Inspired by the awesome work of Carlos Sanchez mailto:[email protected]

Chart Details

This chart will do the following:

• 1 x Jenkins Master with port 8080 exposed on an external LoadBalancer
• All using Kubernetes Deployments

Installing the Chart

To install the chart with the release name my-release:

$ helm install my-release stable/jenkins

Upgrading an existing Release to a new major version

A major chart version change (like v0.40.0 -> v1.0.0) indicates that there is an incompatible breaking change needing manual actions.

2.0.0 Configuration as Code now default + container does not run as root anymore

Configuration as Code new default

Configuration is done via Jenkins Configuration as Code Plugin by default. That means that changes in values which result in a configuration change are always applied. In contrast the XML configuration was only applied during the first start and never altered.

:exclamation::exclamation::exclamation: Attention: This also means if you manually altered configuration then this will most likely be reset to what was configured by default. It also applies to securityRealm and authorizationStrategy as they are also configured using configuration as code. :exclamation::exclamation::exclamation:

Image does not run as root anymore

It's not recommended to run containers in Kubernetes as root.

:exclamation: Attention: If you had not configured a different user before then you need to ensure that your image supports the user and group id configured and also manually change permissions of all files so that Jenkins is still able to use them.

Summary of updated values

As version 2.0.0 only updates default values and nothing else it's still possible to migrate to this version and opt out of some or all new defaults. All you have to do is ensure the old values are set in your installation.

Here we show which values have changed and the previous default values:

master:
  enableXmlConfig: false  # was true
  runAsUser: 1000         # was unset before
  fsGroup: 1000           # was unset before
  JCasC:
    enabled: true         # was false
    defaultConfig: true   # was false
  sidecars:
    configAutoReload:
      enabled: true       # was false

Migration steps

Migration instructions heavily depend on your current setup. So think of the list below more as a general guideline of what should be done.

• Ensure that the Jenkins image you are using contains a user with id 1000 and a group with the same id. That's the case for jenkins/jenkins:lts image, which the chart uses by default

• Make a backup of your existing installation especially the persistent volume

• Ensure that you have the configuration as code plugin installed

• Export your current settings via the plugin: Manage Jenkins -> Configuration as Code -> Download Configuration

• prepare your values file for the update e.g. add additional configuration as code setting that you need. The export taken from above might be a good starting point for this. In addition the demos from the plugin itself are quite useful.

• test drive those setting on a separate installation

• Put Jenkins to Quiet Down mode so that it does not accept new jobs <JENKINS_URL>/quietDown

• change permissions of all files and folders to the new user and group id

  shellCopy

  kubectl exec -it <jenkins_pod> -c jenkins /bin/bash
  chown -R 1000:1000 /var/jenkins_home
  

• Update Jenkins

1.0.0

Breaking changes:

• values have been renamed to follow helm chart best practices for naming conventions so that all variables start with a lowercase letter and words are separated with camelcase https://helm.sh/docs/chart_best_practices/#naming-conventions
• all resources are now using recommended standard labels https://helm.sh/docs/chart_best_practices/#standard-labels

As a result of the label changes also the selectors of the deployment have been updated. Those are immutable so trying an updated will cause an error like:

Error: Deployment.apps "jenkins" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/component":"jenkins-master", "app.kubernetes.io/instance":"jenkins"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

In order to upgrade, delete the Jenkins Deployment before upgrading:

kubectl delete deploy jenkins

Configuration

The following tables list the configurable parameters of the Jenkins chart and their default values.

Jenkins Master

Jenkins Configuration as Code (JCasC)

Jenkins Configuration Files & Scripts

Jenkins Global Security

Jenkins Global Settings

Jenkins In-Process Script Approval

Jenkins Plugins

Jenkins Kubernetes Plugin

Kubernetes Deployment & Service

Kubernetes Health Probes

Kubernetes Ingress

GKE BackendConfig

OpenShift Route

Prometheus

HTTPS Keystore

Kubernetes Secret

Kubernetes NetworkPolicy

Kubernetes RBAC

Kubernetes ServiceAccount - Master

Kubernetes ServiceAccount - Agent

Deprecated

Some third-party systems, e.g. GitHub, use HTML-formatted data in their payload sent to a Jenkins webhooks, e.g. URL of a pull-request being built. To display such data as processed HTML instead of raw text set master.enableRawHtmlMarkupFormatter to true. This option requires installation of OWASP Markup Formatter Plugin (antisamy-markup-formatter). The plugin is not installed by default, please update master.installPlugins.

Jenkins Agent(s)

Pod Configuration

Side Container Configuration

Other

Specify each parameter using the --set key=value[,key=value] argument to helm install.

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

$ helm install my-release -f values.yaml stable/jenkins

Tip: You can use the default values.yaml

Mounting volumes into your Agent pods

Your Jenkins Agents will run as pods, and it's possible to inject volumes where needed:

agent:
  volumes:
  - type: Secret
    secretName: jenkins-mysecrets
    mountPath: /var/run/secrets/jenkins-mysecrets

The supported volume types are: ConfigMap, EmptyDir, HostPath, Nfs, PVC, Secret. Each type supports a different set of configurable attributes, defined by the corresponding Java class.

NetworkPolicy

To make use of the NetworkPolicy resources created by default, install a networking plugin that implements the Kubernetes NetworkPolicy spec.

For Kubernetes v1.5 & v1.6, you must also turn on NetworkPolicy by setting the DefaultDeny namespace annotation. Note: this will enforce policy for all pods in the namespace:

kubectl annotate namespace default "net.beta.kubernetes.io/network-policy={\"ingress\":{\"isolation\":\"DefaultDeny\"}}"

Install helm chart with network policy enabled:

$ helm install stable/jenkins --set networkPolicy.enabled=true

You can use master.networkPolicy.internalAgents and master.networkPolicy.externalAgents stanzas to fine grained controls over where internal/external agents can connect from. Internal ones are allowed based on pod labels and (optionally) namespaces, and external ones are allowed based on IP ranges.

Adding customized securityRealm

master.securityRealm in values can be used to support custom security realm instead of default LegacySecurityRealm. For example, you can add a security realm to authenticate via keycloak.

securityRealm: |-
  <securityRealm class="org.jenkinsci.plugins.oic.OicSecurityRealm" plugin="[email protected]">
    <clientId>testId</clientId>
    <clientSecret>testsecret</clientSecret>
    <tokenServerUrl>https:testurl</tokenServerUrl>
    <authorizationServerUrl>https:testAuthUrl</authorizationServerUrl>
    <userNameField>email</userNameField>
    <scopes>openid email</scopes>
  </securityRealm>

Adding additional configs

master.additionalConfig can be used to add additional config files in config.yaml. For example, it can be used to add additional config files for keycloak authentication.

additionalConfig:
  testConfig.txt: |-
    - name: testName
      clientKey: testKey
      clientURL: testUrl

Adding customized labels

master.serviceLabels can be used to add custom labels in jenkins-master-svc.yaml. For example,

ServiceLabels:
  expose: true

Persistence

The Jenkins image stores persistence under /var/jenkins_home path of the container. A dynamically managed Persistent Volume Claim is used to keep the data across deployments, by default. This is known to work in GCE, AWS, and minikube. Alternatively, a previously configured Persistent Volume Claim can be used.

It is possible to mount several volumes using persistence.volumes and persistence.mounts parameters.

Persistence Values

Existing PersistentVolumeClaim

1. Create the PersistentVolume
2. Create the PersistentVolumeClaim
3. Install the chart

$ helm install my-release --set persistence.existingClaim=PVC_NAME stable/jenkins

Storage Class

It is possible to define which storage class to use:

$ helm install my-release --set persistence.storageClass=customStorageClass stable/jenkins

If set to a dash (-, as in persistence.storageClass=-), the dynamic provision is disabled.

If the storage class is set to null or left undefined (persistence.storageClass=), the default provisioner is used (gp2 on AWS, standard on GKE, AWS & OpenStack).

Configuration as Code

Jenkins Configuration as Code is now a standard component in the Jenkins project. Add a key under configScripts for each configuration area, where each corresponds to a plugin or section of the UI. The keys (prior to | character) are just labels, and can be any value. They are only used to give the section a meaningful name. The only restriction is they must conform to RFC 1123 definition of a DNS label, so may only contain lowercase letters, numbers, and hyphens. Each key will become the name of a configuration yaml file on the master in /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin during Jenkins startup. The lines after each | become the content of the configuration yaml file. The first line after this is a JCasC root element, eg jenkins, credentials, etc. Best reference is the Documentation link here: https://<jenkins_url>/configuration-as-code. The example below creates ldap settings:

configScripts:
  ldap-settings: |
    jenkins:
      securityRealm:
        ldap:
          configurations:
            - server: ldap.acme.com
              rootDN: dc=acme,dc=uk
              managerPasswordSecret: ${LDAP_PASSWORD}
              groupMembershipStrategy:
                fromUserRecord:
                  attributeName: "memberOf"

Further JCasC examples can be found here.

Config as Code with and without auto-reload

Config as Code changes (to master.JCasC.configScripts) can either force a new pod to be created and only be applied at next startup, or can be auto-reloaded on-the-fly. If you choose master.sidecars.autoConfigReload.enabled: true, a second, auxiliary container will be installed into the Jenkins master pod, known as a "sidecar". This watches for changes to configScripts, copies the content onto the Jenkins file-system and issues a POST to http://<jenkins_url>/reload-configuration-as-code with a pre-shared key. You can monitor this sidecar's logs using command kubectl logs <master_pod> -c jenkins-sc-config -f If you want to enable auto-reload then you also need to configure rbac as the container which triggers the reload needs to watch the config maps.

master:
  JCasC:
    enabled: true
  sidecars:
    configAutoReload:
      enabled: true
rbac:
  create: true

RBAC

RBAC is enabled by default if you want to disable it you will need to do the following:

• helm install stable/jenkins --set rbac.create=false

Backup

Adds a backup CronJob for jenkins, along with required RBAC resources.

Backup Values

Restore from backup

To restore a backup, you can use the kube-tasks underlying tool called skbn, which copies files from cloud storage to Kubernetes. The best way to do it would be using a Job to copy files from the desired backup tag to the Jenkins pod. See the skbn in-cluster example for more details.

Run Jenkins as non root user

The default settings of this helm chart let Jenkins run as root user with uid 0. Due to security reasons you may want to run Jenkins as a non root user. Fortunately the default jenkins docker image jenkins/jenkins contains a user jenkins with uid 1000 that can be used for this purpose.

Simply use the following settings to run Jenkins as jenkins user with uid 1000.

master:
  runAsUser: 1000
  fsGroup: 1000

Providing jobs xml

Jobs can be created (and overwritten) by providing jenkins config xml within the values.yaml file. The keys of the map will become a directory within the jobs directory. The values of the map will become the config.xml file in the respective directory.

Below is an example of a values.yaml file and the directory structure created:

values.yaml

master:
  jobs:
    test-job: |-
      <?xml version='1.0' encoding='UTF-8'?>
      <project>
        <keepDependencies>false</keepDependencies>
        <properties/>
        <scm class="hudson.scm.NullSCM"/>
        <canRoam>false</canRoam>
        <disabled>false</disabled>
        <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
        <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
        <triggers/>
        <concurrentBuild>false</concurrentBuild>
        <builders/>
        <publishers/>
        <buildWrappers/>
      </project>
    test-job-2: |-
      <?xml version='1.0' encoding='UTF-8'?>
      <project>
        <keepDependencies>false</keepDependencies>
        <properties/>
        <scm class="hudson.scm.NullSCM"/>
        <canRoam>false</canRoam>
        <disabled>false</disabled>
        <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
        <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
        <triggers/>
        <concurrentBuild>false</concurrentBuild>
        <builders/>
        <publishers/>
        <buildWrappers/>

Directory structure of jobs directory

.
├── _test-job-1
|   └── config.xml
├── _test-job-2
|   └── config.xml

Docs taken from https://github.com/jenkinsci/docker/blob/master/Dockerfile: Jenkins is run with user jenkins, uid = 1000. If you bind mount a volume from the host or a data container,ensure you use the same uid

Adding custom pod templates

It is possible to add custom pod templates for the default configured kubernetes cloud. Add a key under agent.podTemplates for each pod template. Each key (prior to | character) is just a label, and can be any value. Keys are only used to give the pod template a meaningful name. The only restriction is they may only contain RFC 1123 \ DNS label characters: lowercase letters, numbers, and hyphens. Each pod template can contain multiple containers. There's no need to add the jnlp container since the kubernetes plugin will automatically inject it into the pod. For this pod templates configuration to be loaded the following values must be set:

master.JCasC.enabled: true
master.JCasC.defaultConfig: true

The example below creates a python pod template in the kubernetes cloud.

agent:
  podTemplates:
    python: |
      - name: python
        label: jenkins-python
        serviceAccount: jenkins
        containers:
          - name: python
            image: python:3
            command: "/bin/sh -c"
            args: "cat"
            ttyEnabled: true
            privileged: true
            resourceRequestCpu: "400m"
            resourceRequestMemory: "512Mi"
            resourceLimitCpu: "1"
            resourceLimitMemory: "1024Mi"

Best reference is https://<jenkins_url>/configuration-as-code/reference#Cloud-kubernetes.

Adding pod templates using additionalAgents

additionalAgents may be used to configure additional kubernetes pod templates. Each additional agent corresponds to agent in terms of the configurable values and inherits all values from agent so you only need to specify values which differ. For example,

agent:
  podName: default
  customJenkinsLabels: default
  # set resources for additional agents to inherit
  resources:
    limits:
      cpu: "1"
      memory: "2048Mi"

additionalAgents:
  maven:
    podName: maven
    customJenkinsLabels: maven
    # An example of overriding the jnlp container
    # sideContainerName: jnlp
    image: jenkins/jnlp-agent-maven
    tag: latest
  python:
    podName: python
    customJenkinsLabels: python
    sideContainerName: python
    image: python
    tag: "3"
    command: "/bin/sh -c"
    args: "cat"
    TTYEnabled: true

Running behind a forward proxy

The master pod uses an Init Container to install plugins etc. If you are behind a corporate proxy it may be useful to set master.initContainerEnv to add environment variables such as http_proxy, so that these can be downloaded.

Additionally, you may want to add env vars for the Jenkins container, and the JVM (master.javaOpts).

master:
  initContainerEnv:
    - name: http_proxy
      value: "http://192.168.64.1:3128"
    - name: https_proxy
      value: "http://192.168.64.1:3128"
    - name: no_proxy
      value: ""
  containerEnv:
    - name: http_proxy
      value: "http://192.168.64.1:3128"
    - name: https_proxy
      value: "http://192.168.64.1:3128"
  javaOpts: >-
    -Dhttp.proxyHost=192.168.64.1
    -Dhttp.proxyPort=3128
    -Dhttps.proxyHost=192.168.64.1
    -Dhttps.proxyPort=3128

Custom ConfigMap

The following configuration method is deprecated and will be removed in an upcoming version of this chart. We recommend you use Jenkins Configuration as Code to configure instead. When creating a new parent chart with this chart as a dependency, the customConfigMap parameter can be used to override the default config.xml provided. It also allows for providing additional xml configuration files that will be copied into /var/jenkins_home. In the parent chart's values.yaml, set the jenkins.master.customConfigMap value to true like so

jenkins:
  master:
    customConfigMap: true

and provide the file templates/config.tpl in your parent chart for your use case. You can start by copying the contents of config.yaml from this chart into your parent charts templates/config.tpl as a basis for customization. Finally, you'll need to wrap the contents of templates/config.tpl like so:

{{- define "override_config_map" }}
    <CONTENTS_HERE>
{{ end }}

Https keystore configuration

This configuration enable jenkins to use keystore inorder to serve https: https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777

Here is the value file section related to keystore configuration.

Keystore itself should be placed in front of jenkinsKeyStoreBase64Encoded key and in base64 encoded format. To achive that after having keystore.jks file simply do this: cat keystore.jks | base64 and paste the output in front of jenkinsKeyStoreBase64Encoded .

After enabling httpsKeyStore.enable make sure that httpPort and targetPort are not the same as targetPort will serve https.

Do not set master.httpsKeyStore.httpPort to -1 because it will cause readiness and liveliness prob to fail.

If you already have a kubernetes secret that has keystore and its password you can specify its' name in front of jenkinsHttpsJksSecretName, You need to remember that your secret should have proper data key names jenkins-jks-file and https-jks-password.

master:
   httpsKeyStore:
       enable: true
       jenkinsHttpsJksSecretName: ''
       httpPort: 8081
       path: "/var/jenkins_keystore"
       fileName: "keystore.jks"
       password: "changeit"
       jenkinsKeyStoreBase64Encoded: ''