By default, Nomad doesn't allow containers to use privileged mode. We must
configure Nomad so that it allows containers to use privileged mode. Edit the
Nomad configuration file by adding the following configuration block to
`/etc/nomad.d/nomad.hcl`::

    plugin "docker" {
      config {
        allow_privileged = true
      }
    }

Loading the rbd module
~~~~~~~~~~~~~~~~~~~~~~

Nomad must have the `rbd` module loaded. Run the following command to confirm that the `rbd` module is loaded:

.. code-block:: console

  $ lsmod | grep rbd
  rbd                    94208  2
  libceph               364544  1 rbd

If the `rbd` module is not loaded, load it:

.. prompt:: bash $

  sudo modprobe rbd

Restarting Nomad
~~~~~~~~~~~~~~~~

Restart Nomad:

.. prompt:: bash $

  sudo systemctl restart nomad


Create ceph-csi controller and plugin nodes
===========================================

The `ceph-csi`_ plugin requires two components:

- **Controller plugin**: communicates with the provider's API.
- **Node plugin**: executes tasks on the client.

.. note::
    We'll set the ceph-csi's version in those files. See `ceph-csi release`_
    for information about ceph-csi's compatibility with other versions.

Configure controller plugin
---------------------------

The controller plugin requires the Ceph monitor addresses of the Ceph
cluster. Collect both (1) the Ceph cluster unique `fsid` and (2) the monitor
addresses:

.. code-block:: console

  $ ceph mon dump
  <...>
  fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
  <...>
  0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
  1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
  2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c

Generate a ``ceph-csi-plugin-controller.nomad`` file similar to the example
below. Substitute the `fsid` for "clusterID", and the monitor addresses for
"monitors"::

    job "ceph-csi-plugin-controller" {
      datacenters = ["dc1"]
      group "controller" {
        network {
          port "metrics" {}
        }
        task "ceph-controller" {
          template {
            data        = <<EOF
    [{
        "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
        "monitors": [
            "192.168.1.1",
            "192.168.1.2",
            "192.168.1.3"
        ]
    }]
    EOF
            destination = "local/config.json"
            change_mode = "restart"
          }
          driver = "docker"
          config {
            image = "quay.io/cephcsi/cephcsi:v3.3.1"
            volumes = [
              "./local/config.json:/etc/ceph-csi-config/config.json"
            ]
            mounts = [
              {
                type     = "tmpfs"
                target   = "/tmp/csi/keys"
                readonly = false
                tmpfs_options = {
                  size = 1000000 # size in bytes
                }
              }
            ]
            args = [
              "--type=rbd",
              "--controllerserver=true",
              "--drivername=rbd.csi.ceph.com",
              "--endpoint=unix://csi/csi.sock",
              "--nodeid=${node.unique.name}",
              "--instanceid=${node.unique.name}-controller",
              "--pidlimit=-1",
              "--logtostderr=true",
              "--v=5",
              "--metricsport=$${NOMAD_PORT_metrics}"
            ]
          }
          resources {
            cpu    = 500
            memory = 256
          }
          service {
            name = "ceph-csi-controller"
            port = "metrics"
            tags = [ "prometheus" ]
          }
          csi_plugin {
            id        = "ceph-csi"
            type      = "controller"
            mount_dir = "/csi"
          }
        }
      }
    }

Configure plugin node
---------------------

Generate a ``ceph-csi-plugin-nodes.nomad`` file similar to the example below.
Substitute the `fsid` for "clusterID" and the monitor addresses for
"monitors"::

    job "ceph-csi-plugin-nodes" {
      datacenters = ["dc1"]
      type        = "system"
      group "nodes" {
        network {
          port "metrics" {}
        }
        task "ceph-node" {
          driver = "docker"
          template {
            data        = <<EOF
    [{
        "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
        "monitors": [
            "192.168.1.1",
            "192.168.1.2",
            "192.168.1.3"
        ]
    }]
    EOF
            destination = "local/config.json"
            change_mode = "restart"
          }
          config {
            image = "quay.io/cephcsi/cephcsi:v3.3.1"
            volumes = [
              "./local/config.json:/etc/ceph-csi-config/config.json"
            ]
            mounts = [
              {
                type     = "tmpfs"
                target   = "/tmp/csi/keys"
                readonly = false
                tmpfs_options = {
                  size = 1000000 # size in bytes
                }
              }
            ]
            args = [
              "--type=rbd",
              "--drivername=rbd.csi.ceph.com",
              "--nodeserver=true",
              "--endpoint=unix://csi/csi.sock",
              "--nodeid=${node.unique.name}",
              "--instanceid=${node.unique.name}-nodes",
              "--pidlimit=-1",
              "--logtostderr=true",
              "--v=5",
              "--metricsport=$${NOMAD_PORT_metrics}"
            ]
            privileged = true
          }
          resources {
            cpu    = 500
            memory = 256
          }
          service {
            name = "ceph-csi-nodes"
            port = "metrics"
            tags = [ "prometheus" ]
          }
          csi_plugin {
            id        = "ceph-csi"
            type      = "node"
            mount_dir = "/csi"
          }
        }
      }
    }

Start plugin controller and node
--------------------------------

To start the plugin controller and the Nomad node, run the following commands:

.. prompt:: bash $

  nomad job run ceph-csi-plugin-controller.nomad
  nomad job run ceph-csi-plugin-nodes.nomad

The `ceph-csi`_ image will be downloaded.

Check the plugin status after a few minutes:

.. code-block:: console

  $ nomad plugin status ceph-csi
  ID                   = ceph-csi
  Provider             = rbd.csi.ceph.com
  Version              = 3.3.1
  Controllers Healthy  = 1
  Controllers Expected = 1
  Nodes Healthy        = 1
  Nodes Expected       = 1

  Allocations
  ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
  23b4db0c  a61ef171  nodes       4        run      running  3h26m ago  3h25m ago
  fee74115  a61ef171  controller  6        run      running  3h26m ago  3h25m ago

Using Ceph Block Devices
========================

Create rbd image
----------------

``ceph-csi`` requires the cephx credentials for communicating with the Ceph
cluster. Generate a ``ceph-volume.hcl`` file similar to the example below,
using the newly created nomad user id and cephx key::

    id = "ceph-mysql"
    name = "ceph-mysql"
    type = "csi"
    plugin_id = "ceph-csi"
    capacity_max = "200G"
    capacity_min = "100G"

    capability {
      access_mode     = "single-node-writer"
      attachment_mode = "file-system"
    }

    secrets {
      userID  = "admin"
      userKey = "AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ=="
    }

    parameters {
      clusterID = "b9127830-b0cc-4e34-aa47-9d1a2e9949a8"
      pool = "nomad"
      imageFeatures = "layering"
      mkfsOptions = "-t ext4"
    }

After the ``ceph-volume.hcl`` file has been generated, create the volume:

.. prompt:: bash $

  nomad volume create ceph-volume.hcl

Use rbd image with a container
------------------------------

As an exercise in using an rbd image with a container, modify the Hashicorp
`nomad stateful`_ example.

Generate a ``mysql.nomad`` file similar to the example below::

    job "mysql-server" {
      datacenters = ["dc1"]
      type        = "service"
      group "mysql-server" {
        count = 1
        volume "ceph-mysql" {
          type      = "csi"
          attachment_mode = "file-system"
          access_mode     = "single-node-writer"
          read_only = false
          source    = "ceph-mysql"
        }
        network {
          port "db" {
            static = 3306
          }
        }
        restart {
          attempts = 10
          interval = "5m"
          delay    = "25s"
          mode     = "delay"
        }
        task "mysql-server" {
          driver = "docker"
          volume_mount {
            volume      = "ceph-mysql"
            destination = "/srv"
            read_only   = false
          }
          env {
            MYSQL_ROOT_PASSWORD = "password"
          }
          config {
            image = "hashicorp/mysql-portworx-demo:latest"
            args  = ["--datadir", "/srv/mysql"]
            ports = ["db"]
          }
          resources {
            cpu    = 500
            memory = 1024
          }
          service {
            name = "mysql-server"
            port = "db"
            check {
              type     = "tcp"
              interval = "10s"
              timeout  = "2s"
            }
          }
        }
      }
    }

Start the job:

.. prompt:: bash $

  nomad job run mysql.nomad

Check the status of the job:

.. code-block:: console

  $ nomad job status mysql-server
  ...
  Status        = running
  ...
  Allocations
  ID        Node ID   Task Group    Version  Desired  Status   Created  Modified
  38070da7  9ad01c63  mysql-server  0        run      running  6s ago   3s ago

To check that data are persistent, modify the database, purge the job, then
create it using the same file. The same RBD image will be used (re-used,
really).

.. _ceph-csi: https://github.com/ceph/ceph-csi/
.. _csi: https://www.nomadproject.io/docs/internals/plugins/csi
.. _Create a Pool: ../../rados/operations/pools#createpool
.. _Placement Groups: ../../rados/operations/placement-groups
.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
.. _RBD image features: ../rbd-config-ref/#image-features
.. _nomad stateful: https://learn.hashicorp.com/tutorials/nomad/stateful-workloads-csi-volumes?in=nomad/stateful-workloads#create-the-job-file
.. _ceph-csi release: https://github.com/ceph/ceph-csi#ceph-csi-container-images-and-release-compatibility