Pod Annotations

Kata Containers gives users freedom to customize at per-pod level, by setting a wide range of Kata specific annotations in the pod specification.

Some annotations may be restricted by the configuration file for security reasons, notably annotations that could lead the runtime to execute programs on the host. Such annotations are marked with (R) in the tables below.

Kata Configuration Annotations

There are several kinds of Kata configurations and they are listed below.

Global Options

Runtime Options

Agent Options

Hypervisor Options

Hypervisor annotations must be explicitly whitelisted in the Kata runtime config. Example:

# List of valid annotation names for the hypervisor
enable_annotations = ["enable_iommu", "virtio_fs_extra_args", "kernel_params"]

Container Options

containerd Configuration

For containerd, annotations specified in the pod spec are passed down to Kata starting with version 1.3.0 of containerd. Additionally, extra configuration is needed for containerd, by providing pod_annotations field and container_annotations field in the containerd config file. The pod_annotations field and container_annotations field are two lists of annotations that can be passed down to Kata as OCI annotations. They support golang match patterns. Since annotations supported by Kata follow the pattern io.katacontainers.*, the following configuration would work for passing annotations to Kata from containerd:

$ cat /etc/containerd/config
....

         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
           runtime_type = "io.containerd.kata.v2"
           pod_annotations = ["io.katacontainers.*"]
           container_annotations = ["io.katacontainers.*"]
....

Additional documentation on the above configuration can be found in the containerd docs.

Example

As mentioned above, not all containers need the same modules, therefore using the configuration file for specifying the list of kernel modules per POD can be a pain. Unlike the configuration file, annotations provide a way to specify custom configurations per POD.

The list of kernel modules and parameters can be set using the annotation io.katacontainers.config.agent.kernel_modules as a semicolon separated list, where the first word of each element is considered as the module name and the rest as its parameters.

Also users might want to enable guest seccomp to provide better isolation with a little performance sacrifice. The annotation io.katacontainers.config.runtime.disable_guest_seccomp can used for such purpose.

In the following example two PODs are created, but the kernel modules e1000e and i915 are inserted only in the POD pod1. Also guest seccomp is only enabled in the POD pod2.

apiVersion: v1
kind: Pod
metadata:
  name: pod1
  annotations:
    io.katacontainers.config.agent.kernel_modules: "e1000e EEE=1; i915"
spec:
  runtimeClassName: kata
  containers:
  - name: c1
    image: busybox
    command:
      - sh
    stdin: true
    tty: true

---
apiVersion: v1
kind: Pod
metadata:
  name: pod2
  annotations:
    io.katacontainers.config.runtime.disable_guest_seccomp: "false"
spec:
  runtimeClassName: kata
  containers:
  - name: c2
    image: busybox
    command:
      - sh
    stdin: true
    tty: true

Restricted annotations

Some annotations are restricted, meaning that the configuration file specifies the acceptable values. Currently, only hypervisor annotations are restricted, for security reason, with the intent to control which binaries the Kata Containers runtime will launch on your behalf.

The configuration file validates the annotation name as well as the annotation value.

The acceptable annotation names are defined by the enable_annotations entry in the configuration file.

For restricted annotations, an additional configuration entry provides a list of acceptable values. Since most restricted annotations are intended to control which binaries the runtime can execute, the valid value is generally provided by a shell pattern, as defined by glob(3). The table below provides the name of the configuration entry: