design/design-doc-external-secret-stores.md
Kubernetes suggests using Secret resource to store sensitive information and
when requested from the API Server, one gets a base64 encoded value of sensitive
data rather than encrypted. RBAC is used as a security measure for access
control, instead of dealing with a decryption process whenever this data needed
to be consumed. However, in practice it is not easy to ensure proper access to
sensitive data with RBAC. For example, when an application requires read/write
access to some secrets in a namespace, it is common pitfall to deploy with
access to any secret in the namespace which could result in unintended access to
another secret in the same namespace. Hence, Kubernetes secrets usually
considered as not so secure.
Kubernetes has a solution to encrypt secrets at rest, however, this is only to keep secret data as encrypted in etcd which changes nothing at Kubernetes API level. Requests for supporting additional external secret stores at API level were rejected for various reasons.
Another point is, Kubernetes clusters often considered as ephemeral resources as opposed to being a reliable data store not only for sensitive data but also non-sensitive data like application manifests and this increased popularity of GitOps tools.
Hence, storing sensitive information in external secret stores is a common practice. Since applications running on K8S need this information as well, it is also quite common to sync data from external secret stores to K8S. There are quite a few tools out there that are trying to resolve this exact same problem. However, Crossplane, as a producer of infrastructure credentials, needs the opposite, which is storing sensitive information to external secret stores.
Today, Crossplane only supports storing connection details for managed resources
as Kubernetes secrets. This is configured via the
spec.writeConnectionSecretToRef field. However, there is an increasing demand
to store infrastructure credentials on external secret stores rather than
relying on Kubernetes secrets.
This document aims to propose a design for supporting external secret stores in Crossplane. The solution should apply to both Crossplane itself (for composite connection details) and providers (for managed resource connection details). Throughout the document, we will mostly focus on Vault as the most popular secret store today.
We would like to come up with a design that:
We would like Crossplane to be able to store connection details to external secret stores and still satisfy the following user stories:
To achieve this, it should still be possible to pass partial inputs for connection details secret for dynamically provisioned resources:
writeConnectionSecretsToNamespace)The following is out of scope for this design:
Reading provider credentials from external secret stores.
This design focuses on writing/publishing connection details to external stores and not about reading them as provider credentials. There are already ways to consume secrets in external stores from Kubernetes and is out of the scope for this document. Check the Vault credential injection guide to see how one can configure Vault and Crossplane to consume provider credentials from Vault.
Reading managed resource input secrets from external secret stores.
There are some resources which requires sensitive information like initial password as input. Typically, this type of input is received from a Kubernetes secret. This design focuses on storing credentials produced by Crossplane hence input secrets are out of scope.
A secret is a resource keeping sensitive information typically as a set of key value pairs. To access a secret instance in a secret store we would need the following information:
external-name). This identifier could be split into two
different parts:
namespace in Kubernetes, a parent path in Vault and a region for
AWS Secret Manager.kubeconfig for
Kubernetes, a server endpoint + auth config for Vault and access/secret
keys for AWS Secret Manager.We will deprecate the existing writeConnectionSecretToRef field in favor of
publishConnectionDetailsTo field which would support publishing connection
details to the local Kubernetes cluster or to an external secret store.
We will take the name of secret and some per secret metadata like tags in
our secret configuration API (i.e. publishConnectionDetailsTo). The rest of
the configuration will go to a separate store specific config (StoreConfig).
This classification enables building a flexible API that satisfies the
separation of concerns between platform operators and consumers which Crossplane
already enables today for Kubernetes Secrets.
We will end up having a unified configuration spec for all external secret store
types which contains the name field (name) and a reference to any additional
store specific configuration (configRef). This would be enough to uniquely
identify and access any secret instance, however there could still be some
additional metadata specific to store type that might be desired
to be set per secret instance. For example, labels, annotations and type
of the secret in Kubernetes; Tags and EncryptionKey in AWS.
A StoreConfig named default will be created during installation which is
configured to write secret to a Kubernetes cluster in the Crossplane
installation namespace.publishConnectionDetailsTo.configRef will be optional
and in its absence, it will be late initialized as configRef.name=default.
Examples:
Publish Connection details to a Kubernetes secret (already existing case):
spec:
publishConnectionDetailsTo:
name: my-db-connection
configRef:
name: default
Publish Connection details to a Vault secret:
spec:
publishConnectionDetailsTo:
name: my-db-connection
configRef:
name: vault-dev
Publish Connection details to a Kubernetes secret with some labels/annotations:
spec:
publishConnectionDetailsTo:
name: my-db-connection
metadata:
labels:
environment: production
annotations:
acme.example.io/secret-type: infrastructure
configRef:
name: default
Publish Connection details to an AWS Secret Manager secret with some tags:
spec:
publishConnectionDetailsTo:
name: my-db-connection
metadata:
tags:
environment: production
configRef:
name: aws-secret-manager-prod
Currently, platform operators could specify where should composite resource
secrets land using the partial input writeConnectionSecretsToNamespace. We
will similarly deprecate this field in favor of
publishConnectionDetailsToStoreConfigRef.
Examples:
Composition configuring to publish to default Store which is created during installation time, i.e. publish to a Kubernetes secret in the namespace where crossplane installed.
spec:
publishConnectionDetailsToStoreConfigRef: default
Composition configuring to publish to another namespace, e.g.
infrastructure-staging, where a StoreConfig named
store-infrastructure-staging created with defaultScope parameter as
infrastructure-staging.
spec:
publishConnectionDetailsToStoreConfigRef: store-infrastructure-staging
Composition configuring to publish to vault.
spec:
publishConnectionDetailsToStoreConfigRef: vault-production
For Claim resources, claim namespace will be used as a scope instead of the
defaultScope in StoreConfig.
spec:
publishConnectionDetailsTo:
name: database-creds
External secret store configuration will contain the required information other
than the name (and optional metadata) of secret. Thanks to
the standardization efforts on a declarative API for syncing secrets from
external stores into Kubernetes, there is already an existing schema that we
can follow here. However, there will be slight differences since we want to
receive the name of secret as input in our API whereas, they expect a full
identifier of the secret instance within the secret store
(in ExternalSecret spec at spec.dataFrom.key). Another reason for
differences is the direction of operation; Crossplane needs to publish to
external stores whereas those tools targets the opposite, that is
fetching from external stores.
Since we are mapping namespaced secret resources to a cluster scoped
StoreConfig, we need to handle same secret names coming from different
namespaces with some store specific scoping. This would be namespace for
Kubernetes, a parent directory for Vault or simply prefixing the name of the
secret if the secret store does not have a concept for scoping. We will have
a spec.defaultScope field in StoreConfig to be used for cluster scoped
resources.
Examples:
Publish to Vault under parent path secret/my-cloud/dev/ using Kubernetes auth:
apiVersion: secrets.crossplane.io/v1alpha1
kind: StoreConfig
metadata:
name: vault-default
spec:
type: Vault
# defaultScope used for scoping secrets for cluster scoped resources.
# For example, secrets for MRs will land under
# "secret/my-cloud/dev/crossplane-system" path in Vault with this StoreConfig.
# However, secret claims in `team-a` namespace will go to
# "secret/my-cloud/dev/team-a".
defaultScope: crossplane-system
vault:
server: "https://vault.acme.org"
# parentPath is the parent path that will be prepended to the secrets
# created with this store config.
parentPath: "secret/my-cloud/dev/"
version: "v2"
caBundle: "..."
auth:
# Kubernetes auth: https://www.vaultproject.io/docs/auth/kubernetes
kubernetes:
mountPath: "kubernetes"
role: "demo"
Publish with an out-of-tree Secret Plugin (for future support, if needed):
apiVersion: secrets.crossplane.io/v1alpha1
kind: StoreConfig
metadata:
name: acme-secretstore
spec:
type: Plugin
defaultScope: crossplane-system
plugin:
name: plugin-x
endpoint: unix:///tmp/plugin-x.sock
config:
host: secretstore.acme.org
port: 9999
caBundle: "..."
some:
other:
arbitrary-config: true
With the API definition above, it is mostly clear how the user interaction would be. But one important point that worth mentioning is, both Crossplane core pod and provider pods would need to access to the external secret stores. The credentials that is made available needs to be authorized to read, write and delete the secrets living at the configured scope.
Here is an example flow for configuring Vault as an external secret store with Kubernetes Auth:
crossplane-system) with permissions
to read, write and delete the secrets under a parent path
(e.g. secret/my-cloud/dev/).crossplane).
ControllerConfig)StoreConfig CRs as follows:For core Crossplane:
apiVersion: secrets.crossplane.io/v1alpha1
kind: StoreConfig
metadata:
name: vault-default
spec:
type: Vault
defaultScope: crossplane-system
vault:
server: "https://vault.acme.org"
parentPath: "secret/my-cloud/dev/"
version: "v2"
caBundle: "..."
auth:
kubernetes:
mountPath: "kubernetes"
role: "crossplane"
For the provider pod:
apiVersion: aws.secrets.crossplane.io/v1alpha1
kind: StoreConfig
metadata:
name: vault-default
spec:
type: Vault
defaultScope: crossplane-system
vault:
server: "https://vault.acme.org"
parentPath: "secret/my-cloud/dev/"
version: "v2"
caBundle: "..."
auth:
kubernetes:
mountPath: "kubernetes"
role: "crossplane"
publishConnectionDetailsTo for resources:spec:
publishConnectionDetailsTo:
name: <secret-name>
configRef:
name: vault-default
We will define a new interface, namely ConnectionSecretStore, which satisfies
slightly modified versions of the existing ConnectionPublisher and
ConnectionDetailsFetcher interfaces. This interface will be satisfied by any
secret store including the local Kubernetes. We will need this interface to be
defined in crossplane-runtime repository since both managed and Crossplane
composite reconcilers would use this interface. This will require some
refactoring since the existing interfaces defined in different
packages/repositories today.
type ConnectionSecretStore interface {
ConnectionDetailsPublisher
ConnectionDetailsFetcher
}
type ConnectionDetailsPublisher interface {
PublishConnection(ctx context.Context, p ConnectionSecretPublisher, c managed.ConnectionDetails) error
UnpublishConnection(ctx context.Context, p ConnectionSecretPublisher, c managed.ConnectionDetails) error
}
type ConnectionDetailsFetcher interface {
FetchConnectionDetails(ctx context.Context, p ConnectionSecretPublisher) (managed.ConnectionDetails, error)
}
Implementations of any function in this interface will first fetch StoreConfig
and configure its client before any read/write/delete. This is required to
ensure any changes in the StoreConfig resources to be reflected in the next
reconcile. Local Kubernetes store is an exception here since it already uses
in cluster config which does not depend on a StoreConfig and would use the
same client as it is doing today.
Other types to complete the picture:
type ConnectionSecretPublisher interface {
Object
ConnectionSecretPublisherTo
}
type ConnectionSecretPublisherTo interface {
SetPublishConnectionSecretTo(c *xpv1.ConnectionSecretConfig)
GetPublishConnectionSecretTo() *xpv1.ConnectionSecretConfig
}
type ConnectionSecretConfig struct {
Name string `json:"name"`
Metadata map[string]any `json:"metadata"`
ConfigRef *SecretStoreConfig `json:"configRef"`
}
External secret store support will be introduced in a phased fashion, with
initially being off by default behind a feature flag like
--enable-alpha-external-secret-stores.
By adding support for Kubernetes as an external secret store in StoreConfig,
we could enable publishing connection secrets to another Kubernetes cluster.
For example, platform consumers could publish database connection details they
were provisioned on control plane cluster to their existing application clusters
or even to a new Kubernetes cluster provisioned together with the database.
Example claim spec:
spec:
publishConnectionDetailsTo:
name: my-db-connection
configRef:
name: kubernetes-cluster-1
Example StoreConfig:
apiVersion: secrets.crossplane.io/v1alpha1
kind: StoreConfig
metadata:
name: kubernetes-cluster-1
spec:
type: Kubernetes
defaultScope: crossplane-system
kubernetes:
namespace: backend-dev
auth:
kubeconfig:
secretRef:
name: "db-claim-cluster-conn"
key: "kubeconfig"
There are two types of input secrets in Crossplane, provider credentials and sensitive fields in managed resource spec. These are provided with Kubernetes secrets today, and intentionally left out of scope for this design to limit the scope. However, the types and interfaces defined here would be leveraged to satisfy these two cases which will result in a unified secret management with external stores no matter it is input or output.
Not directly related to supporting external secret stores but thanks to the extensible API proposed in this design, we might consider adding an interface that supports adding new keys to connection secret content from existing connection detail keys according to a given template (similar to Vault agent inject template). This would be helpful to prepare a secret content in an expected format like SQL connection strings.
Another possible use case is, combined with support for adding annotations to Kubernetes secrets, creating a ArgoCD cluster could be enabled with a spec as follows:
spec:
publishConnectionDetailsTo:
name: my-db-connection
metadata:
annotations:
acme.example.io/secret-type: infrastructure
template: |
name: my-crossplane-managed-cluster
server: {{ .ConnectionDetails.Endpoint }}
config: |
{
"bearerToken": "{{ .ConnectionDetails.Bearer }}",
"tlsClientConfig": {
"insecure": false,
"caData": "{{ .ConnectionDetails.CABundle }}"
}
}
Crossplane enables managing external resources from Kubernetes API via custom resources. Considering each secret living in an external secret store, is indeed an external resource which could be managed by Crossplane just like any other managed resources, we could leverage Crossplane providers to create connection details secrets in external secret stores.
For example, to store a connection detail secret in AWS secret manager, all we need to do is to create a provider-aws Secret resource with proper configuration and reference to the Kubernetes secret with connection details. Similarly, to store them in Vault, we would need to implement a provider-vault and create a GenericSecret resource.
This approach would allow us to avoid reimplementing ways to connect and authenticate different providers at different layers and rely on ProviderConfigs which is exactly responsible for this.
Despite this being architecturally the cleanest solution, it has a major limitation which is the need for storing sensitive information in a Kubernetes secret which violates one of the main motivations of this design.
This option proposes a pluggable secret backend in upstream Crossplane which would allow out-of-tree secret store plugins. When configured, Crossplane and providers will communicate with the secret store plugin over gRPC. The plugin would then be responsible for communicating with the secret store. We will follow a similar approach as KMS plugin support in Kubernetes API server.
In this option, we will only implement support for a plugin API in Crossplane Core. Actual plugins and utilities to deploy and configure the environment properly could be build independently, i.e. as separate components.
This approach has the advantage of not introducing hard dependencies to Crossplane and also would be more scalable if/once we want to add support for more secret stores. However, this option introduces an upfront complexity especially around deployment of Crossplane and providers mostly related to securing the communication between plugins.
The main motivation behind this approach is hiding the complexity of interacting with external secret stores from Crossplane by putting an intermediate layer between Crossplane and Kubernetes intercepting Secret operations. In this option there will be no (or very minimal) changes in Crossplane, but the heavy lifting will be handled by the Intermediate Layer.
There are two different approaches that we have considered in the scope of this option:
While this option would require minimal changes in Crossplane and sounds more attractive technically, in addition to performance concerns, we prefer to be clear at Crossplane API level instead of hiding the information of where the connection information actually landed.
The idea of Crossplane writing secrets to the filesystem in the pod and a proper CSI driver syncs these secrets to an external secret store sounds fancy. I have investigated this possibility and came across kubernetes-sigs/secrets-store-csi-driver repository which already supports different secret backends like aws, gcp, azure and vault. However, all the work is around making a secret available in an external secret store in the filesystem of the pod and not the opposite direction that we need here. It is also debatable whether this would be possible at all and could not find any related discussion or issue.