design/one-pager-pluggable-secret-stores.md
In the v1.7.0 release, we have introduced support for publishing connection details secrets to External Secret Stores as an alpha feature. With this initial iteration, we have focused on the in-tree implementation of secret stores by adding Hashicorp's Vault support as our first External Secret Store. However, this approach introduces additional dependencies to Crossplane core and would not scale once the community wants to support more secret stores. Even further, it won't be possible to use an in-house secret store with Crossplane since adding support for such a store to Crossplane core wouldn't make sense. For this purpose, with the initial design, we set it as a goal to support switching to an out-of-tree (plugin) model but intentionally left it out of scope to avoid the upfront complexity.
With the gained confidence, we can now iterate with a more scalable option by supporting pluggable secret stores.
The goal of this document is to propose a pluggable secret store mechanism that enables the development and consumption of out-of-tree secret store plugins.
This new mechanism should:
This document proposes introducing a plugin mechanism for communicating with External Secret Stores. We will follow a similar approach as KMS plugins used by Kubernetes API Server for encryption at rest. A Secret Store plugin will be serving a well-defined API via gRPC. To have a secure communication, we will deploy plugins as a separate deployment and enforce mTLS.
In addition to the existing Secret Store types, Kubernetes and Vault, we
will define a new type named as Plugin. We will deprecate the existing type
Vault in favor of a plugin supporting Vault to remove the dependencies
already introduced. With this, Kubernetes and Plugin will be the only two
built-in types going forward.
Configuration of this type would require passing a typed reference to another configuration custom resource defined by the deployed secret store plugin.
apiVersion: gcp.crossplane.io/v1alpha1
kind: StoreConfig
metadata:
name: vault
spec:
type: Plugin
defaultScope: crossplane-system
external:
endpoint: ess-plugin-vault:4040 # Service endpoint of the plugin.
configRef:
apiVersion: secrets.crossplane.io/v1alpha1
kind: VaultConfig
name: vault-internal
---
apiVersion: secrets.crossplane.io/v1alpha1
kind: VaultConfig
metadata:
name: vault-internal
spec:
server: http://vault.vault-system:8200
mountPath: secrets/
version: v2
auth:
method: Token
token:
source: Secret
secretRef:
namespace: crossplane-system
name: vault-token
key: token
There won't be any API changes at MR/XR/XRC level as they would still refer to
a StoreConfig resource as before.
We will implement the Plugin Secret Store type in Crossplane Runtime which
will use a gRPC client to communicate with the plugin server using the service
definition below:
syntax = "proto3";
// This service defines the APIs for an External Secret Store plugin.
service ExternalSecretStoreService {
rpc GetSecret(GetSecretRequest) returns (GetSecretResponse) {}
rpc ApplySecret(ApplySecretRequest) returns (ApplySecretResponse) {}
rpc DeleteKeys(DeleteKeysRequest) returns (DeleteKeysResponse) {}
}
message ConfigReference {
string api_version = 1;
string kind = 2;
string name = 3;
}
message Secret {
string scoped_name = 1;
map<string, string> metadata = 2;
map<string, bytes> data = 3;
}
message GetSecretRequest {
ConfigReference config = 1;
Secret secret = 2;
GetOptions options = 3;
}
message GetSecretResponse {
Secret secret = 1;
}
message ApplySecretRequest {
ConfigReference config = 1;
Secret secret = 2;
ApplyOptions options = 3;
}
message ApplySecretResponse {
bool changed = 1;
}
message DeleteKeysRequest {
ConfigReference config = 1;
Secret secret = 2;
DeleteOptions options = 3;
}
message DeleteKeysResponse {}
message GetOptions {}
message ApplyOptions {}
message DeleteOptions {
bool keep_empty_secret = 1;
}
A Secret Store Plugin:
Deployment of a Secret Store Plugin will be similar to the current provider deployment, with one exception. We need to generate a server certificate during deployment time and mount it to the plugin pod. This is required to enable mTLS communication.
In addition to the server certificates, we will generate a client certificate for each provider and store it in a k8s secret. These certificates will be generated by the package manager during the package installation.
Since the core Crossplane communicates with the secret stores as well, it will also act as a gRPC client to communicate with the plugin. Hence, it will need a client certificate. To provide this certificate, we will create an empty secret with the Crossplane helm chart and mount it to the Crossplane pod. The init container will then fill the secret with the client certificate similar to how webhook certificates are provided today.
All these certificates will be signed by the cluster's root CA certificate with proper subject and DNS names and stored as k8s secrets. Rotation of these certificates can be triggered by deleting those secrets and restarting the Crossplane pod.
A typical flow during a Managed Resource publishing a connection secret to a pluggable Secret Store will be as follows:
connection.DetailsManager.WriteKeyValues method of Plugin Secret Store, ApplySecret gRPC
method called with configReference and secret parameters:func (ss *SecretStore) WriteKeyValues(ctx context.Context, s *store.Secret, _ ...store.WriteOption) (changed bool, err error) {
// TODO(turkenh): handle write options.
resp, err := pb.NewExternalSecretStoreServiceClient(ss.conn).ApplySecret(ctx, &pb.ApplySecretRequest{
Config: &pb.ConfigReference{
ApiVersion: ss.pluginConfigRef.APIVersion,
Kind: ss.pluginConfigRef.Kind,
Name: ss.pluginConfigRef.Name,
},
Secret: &pb.Secret{
ScopedName: filepath.Join(s.Scope, s.Name),
Metadata: s.Metadata.Labels,
Data: s.Data,
},
})
if err != nil {
return false, err
}
return resp.Changed, nil
}
ApplySecret request:
VaultConfig,
from the Kubernetes API.Instead of deploying plugins as separate deployments, it was initially proposed to deploy them as sidecars. The main motivation behind this was eliminating authentication complications and having secure communication via sockets. However, adding sidecars to Crossplane and providers will make the packaging and deployment processes more complicated. For example, when we want to install a new plugin, we will need to inject it to the core Crossplane and existing providers. Injecting the plugin to the Crossplane would require a helm upgrade which is not possible by the package manager or a mutating webhook. The second complication is that if we want to install multiple plugins on a single cluster, we end up having multiple sidecars for the Crossplane and providers which causes having multiple processes for the same purpose.
We could use a Mutating Webhook to inject plugin sidecars instead of modifying the Crossplane helm chart and package manager code similar to the Vault agent injector. However, this would mean introducing one more component for a work that we could already accomplish with some simple changes in the existing components. In other words, considering plugin sidecars will not need to be injected to some arbitrary pods other than Crossplane and Providers, there is not much need for this approach.
We propose to use Kubernetes CA to sign the certificates instead of using the self-signed CA created for the webhooks. The main reason for this is that we do not want to couple two different features and make them dependent on each other.