design/ceph/object/ceph-cosi-driver.md
Container Object Storage Interface (COSI) is a specification for container orchestration frameworks to manage object storage. Even though there is no standard protocol defined for Object Store, it has flexibility to add support for all. The COSI spec abstracts common storage features such as create/delete buckets, grant/revoke access to buckets, attach/detach buckets, and more. COSI is released v1alpha1 with Kubernetes 1.25. More details about COSI can be found here. It is projected that COSI will be the only supported object storage driver in the future and OBCs will be deprecated after some time.
The COSI controller is deployed as container in the default namespace. It is not deployed by Rook.
The Ceph COSI driver is deployed as a deployment with a COSI sidecar container. The Ceph COSI driver is deployed in the same namespace as the Rook operator. The Ceph COSI driver is deployed with a service account that has the following RBAC permissions:
The aim to support the v1alpha1 version of COSI in Rook v1.12. It will be extended to beta and release versions as appropriate. There will be one COSI Driver support from Rook. The driver will be started automatically with default settings when first Object Store gets created. The driver will be deleted when Rook Operator is uninstalled. The driver will be deployed in the same namespace as Rook operator. The user can provide additional setting via new CephCOSIDriver CRD which is owned by Rook.
kubectl apply -k github.com/kubernetes-sigs/container-object-storage-interface-api
kubectl apply -k github.com/kubernetes-sigs/container-object-storage-interface-controller
Following contents need to be append to common.yaml :
The ceph-object-cosi-controller will start the Ceph COSI Driver pod with default settings on the Rook Operator Namespace. The controller will bring up the Ceph COSI driver when first object store is created and will stop the COSI driver when Rook Operator is uninstalled only if it detect COSI Controller is running on default namespace. The controller will also watch for CephCosiDriver CRD, if defined the driver will be started with the settings provided in the CRD. If the Ceph COSI driver if up and running,it will also create CephObjectStoreUser named cosi for each object store which internally creates a secret rook-ceph-object-user-<objectstore-name>-cosi provides credentials for the object store. This can be specified in the BucketClass and BucketAccessClass. Also this controller ensures maximum one CephCosiDriver CRD exists in the cluster. For v1.12 the Ceph COSI Driver will be supported only for single site CephObjectStore aka object stores not configured with multisite settings like zone/zonegroup/realm.
The users can define following CRD so that configuration related the Ceph COSI driver can be passed. This is not mandatory and the driver will be started with default settings if this CRD is not defined.
apiVersion: ceph.rook.io/v1
kind: CephCosiDriver
metadata:
name: rook-ceph-cosi
namespace: rook-ceph
spec:
deploymentStrategy: "Auto"
placement:
# nodeAffinity:
# podAffinity:
# tolerations:
resource:
# requests:
# cpu: "100m"
# memory: "128Mi"
# limits:
# memory: "128Mi"
The deploymentStrategy can be Auto or Force or Disable. If Auto is specified the driver will be started automatically when first object store is created and will be stopped when last object store is deleted. If Force is specified the driver will be started when the CRD is created and will be stopped when the CRD is deleted. The user can also disable the driver by specifying Disable as the value. The placement field can be used to specify the placement settings for the driver. The resource field can be used to specify the resource requirements for the driver. Both can refer from CephClusterCRD Documentation.
There are five different kubernetes resources related to COSI. They are Bucket, BucketAccess, BucketAccessClass, BucketClass and BucketClaim. The user can create these resources using following kubectl command. All the examples will be added to deploy/examples/cosi directory in the Rook repository.
kubectl create -f deploy/examples/cosi/bucketclass.yaml
kubectl create -f deploy/examples/cosi/bucketclaim.yaml
kubectl create -f deploy/examples/cosi/bucketaccessclass.yaml
kubectl create -f deploy/examples/cosi/bucketaccess.yaml
User need refer Secret which contains access credentials for the object store in the Parameter field of BucketClass and BucketAccessClass CRD as below:
Parameter
objectUserSecretName: "rook-ceph-object-user-<objectstore-name>-cosi"
objectStoreNamespace: "<objectstore-store-namespace>"
The user needs to mount the secret as volume created by BucketAccess to the application pod. The user can use the secret to access the bucket by parsing the mounted file.
spec:
containers:
volumeMounts:
- name: cosi-secrets
mountPath: /data/cosi
volumes:
- name: cosi-secrets
secret:
secretName: ba-secret
# cat /data/cosi/bucket_info.json
{
apiVersion: "v1alpha1",
kind: "BucketInfo",
metadata: {
name: "ba-$uuid"
},
spec: {
bucketName: "ba-$uuid",
authenticationType: "KEY",
endpoint: "https://rook-ceph-my-store:443",
accessKeyID: "AKIAIOSFODNN7EXAMPLE",
accessSecretKey: "wJalrXUtnFEMI/K...",
region: "us-east-1",
protocols: [
"s3"
]
}
}
Currently the ceph object store provisioned via Object Bucket Claim (OBC). They both can coexist and can even use same backend bucket from ceph storage. No deployment/configuration changes are required to support both. The lib-bucket-provisioner is deprecated and eventually will be replaced by COSI when it becomes more and more stable. The CRDs used by both are different hence there is no conflicts between them. The user can create OBC and COSI BucketClaim for same backend bucket which have always result in conflicts. Even though credentials for access the buckets are different, both have equal permissions on accessing the bucket. If Rooks creates OBC with Delete reclaim policy and same backend bucket is used by COSI BucketClaim with same policy, then bucket will be deleted when either of them is removed.
This applied to OBC with reclaim policy is Retain otherwise the bucket will be deleted when OBC is deleted. So no point in migrating the OBC with Delete reclaim policy.