docs/root/configuration/security/secret.rst
.. _config_secret_discovery_service:
TLS certificates, the secrets, can be specified in the bootstrap.static_resource
:ref:secrets <envoy_v3_api_field_config.bootstrap.v3.Bootstrap.StaticResources.secrets>.
But they can also be fetched remotely by secret discovery service (SDS).
The most important benefit of SDS is to simplify the certificate management. Without this feature, in k8s deployment, certificates must be created as secrets and mounted into the proxy containers. If certificates are expired, the secrets need to be updated and the proxy containers need to be re-deployed. With SDS, a central SDS server will push certificates to all Envoy instances. If certificates are expired, the server just pushes new certificates to Envoy instances, Envoy will use the new ones right away without re-deployment.
If a listener server certificate needs to be fetched by SDS remotely, it will NOT be marked as active, its port will not be opened before the certificates are fetched. If Envoy fails to fetch the certificates due to connection failures, or bad response data, the listener will be marked as active, and the port will be open, but the connection to the port will be reset.
Upstream clusters are handled in a similar way, if a cluster client certificate needs to be fetched by SDS remotely, it will NOT be marked as active and it will not be used before the certificates are fetched. If Envoy fails to fetch the certificates due to connection failures, or bad response data, the cluster will be marked as active, it can be used to handle the requests, but the requests routed to that cluster will be rejected.
If a static cluster is using SDS, and it needs to define a SDS cluster (unless Google gRPC is used which doesn't need a cluster), the SDS cluster has to be defined before the static clusters using it.
The connection between Envoy proxy and SDS server has to be secure. One option is to run the SDS server on the same host and use Unix Domain Socket for the connection. Otherwise the connection requires TLS with authentication between the proxy and SDS server. Credential types in use today for authentication are:
A SDS server needs to implement the gRPC service :repo:SecretDiscoveryService <api/envoy/service/secret/v3/sds.proto>.
It follows the same protocol as other :ref:xDS <xds_protocol>.
.. note::
`SPIRE <https://spiffe.io/docs/latest/spire-about/>`_,
an open-source reference implementation of the
`SPIFFE specification for production identity <https://spiffe.io/>`_,
can act as an SDS server for Envoy. See the
`SPIRE documentation <https://spiffe.io/docs/latest/microservices/envoy/>`_
for how to configure Envoy with SPIFFE and SPIRE.
:ref:SdsSecretConfig <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.SdsSecretConfig> is used to specify the secret. Its field name is a required field. If its sds_config field is empty, the name field specifies the secret in the bootstrap static_resource :ref:secrets <envoy_v3_api_field_config.bootstrap.v3.Bootstrap.StaticResources.secrets>. Otherwise, it specifies the SDS server as :ref:ConfigSource <envoy_v3_api_msg_config.core.v3.ConfigSource>. When using a remote SDS service, the api_config_source must specify a grpc_service as only gRPC is supported.
SdsSecretConfig is used in two fields in :ref:CommonTlsContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.CommonTlsContext>. The first field is tls_certificate_sds_secret_configs to use SDS to get :ref:TlsCertificate <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.TlsCertificate>. The second field is validation_context_sds_secret_config to use SDS to get :ref:CertificateValidationContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.CertificateValidationContext>.
.. _sds_key_rotation:
It's usually preferrable to perform key rotation via gRPC SDS, but when this is not possible or desired (e.g. during bootstrap of SDS credentials), SDS allows for filesystem rotation when secrets refer to filesystem paths. This currently is supported for the following secret types:
TlsCertificate <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.TlsCertificate>CertificateValidationContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.CertificateValidationContext>By default, directories containing secrets are watched for filesystem move events. For example, a
key or trusted CA certificates at /foo/bar/baz/cert.pem will be watched at /foo/bar/baz.
Explicit control over the watched directory is possible by specifying a watched_directory path in
:ref:TlsCertificate <envoy_v3_api_field_extensions.transport_sockets.tls.v3.TlsCertificate.watched_directory> and
:ref:CertificateValidationContext <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.watched_directory>.
This allows watches to be established at path predecessors, e.g. /foo/bar; this capability is
useful when implementing common key rotation schemes.
An example of key rotation is provided :ref:below <xds_certificate_rotation>.
This example show how to configure secrets in the static_resource:
.. code-block:: yaml
static_resources: secrets: - name: server_cert tls_certificate: certificate_chain: filename: certs/servercert.pem private_key: filename: certs/serverkey.pem - name: client_cert tls_certificate: certificate_chain: filename: certs/clientcert.pem private_key: filename: certs/clientkey.pem - name: validation_context validation_context: trusted_ca: filename: certs/cacert.pem verify_certificate_hash: E0:F3:C8:CE:5E:2E:A3:05:F0:70:1F:F5:12:E3:6E:2E:97:92:82:84:A2:28:BC:F7:73:32:D3:39:30:A1:B6:FD clusters: - connect_timeout: 0.25s load_assignment: cluster_name: local_service_tls ... transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext common_tls_context: tls_certificate_sds_secret_configs: - name: client_cert listeners: .... filter_chains: transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificate_sds_secret_configs: - name: server_cert validation_context_sds_secret_config: name: validation_context
In this example, certificates are specified in the bootstrap static_resource, they are not fetched remotely. In the config, secrets static resource has 3 secrets: client_cert, server_cert and validation_context. In the cluster config, one of hosts uses client_cert in its tls_certificate_sds_secret_configs. In the listeners section, one of them uses server_cert in its tls_certificate_sds_secret_configs and validation_context for its validation_context_sds_secret_config.
.. _sds_server_example:
This example shows how to configure secrets fetched from remote SDS servers:
.. literalinclude:: _include/sds-source-example.yaml
:language: yaml
:lines: 1-97
:linenos:
:lineno-start: 1
:caption: :download:sds-source-example.yaml <_include/sds-source-example.yaml>
For illustration, above example uses three methods to access the SDS server. A gRPC SDS server can be reached by Unix Domain Socket path /tmp/uds_path and 127.0.0.1:8234 by mTLS. It provides three secrets, client_cert, server_cert and validation_context. In the config, cluster example_cluster certificate client_cert is configured to use Google gRPC with UDS to talk to the SDS server. The Listener needs to fetch server_cert and validation_context from the SDS server. The server_cert is using Envoy gRPC with cluster sds_server_mtls configured with client certificate to use mTLS to talk to SDS server. The validate_context is using Envoy gRPC with cluster sds_server_uds configured with UDS path to talk to the SDS server.
.. _xds_certificate_rotation:
Managing certificates for xDS gRPC connection between Envoy and xDS server introduces a bootstrapping problem: SDS server cannot manage certificates that are required to connect to the server.
This example shows how to set up xDS connection by sourcing SDS configuration from the filesystem.
The certificate and key files are watched with inotify and reloaded automatically without restart.
In contrast, :ref:sds_server_example requires a restart to reload xDS certificates and key after update.
.. code-block:: yaml
clusters:
- name: control_plane
type: LOGICAL_DNS
connect_timeout: 1s
load_assignment:
cluster_name: control_plane
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: controlplane
port_value: 8443
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
transport_socket:
name: "envoy.transport_sockets.tls"
typed_config:
"@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext"
common_tls_context:
tls_certificate_sds_secret_configs:
name: tls_sds
sds_config:
path_config_source:
path: /etc/envoy/tls_certificate_sds_secret.yaml
validation_context_sds_secret_config:
name: validation_context_sds
sds_config:
path_config_source:
path: /etc/envoy/validation_context_sds_secret.yaml
Paths to client certificate, including client's certificate chain and private key are given in SDS config file /etc/envoy/tls_certificate_sds_secret.yaml:
.. code-block:: yaml
resources:
- "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
name: tls_sds
tls_certificate:
certificate_chain:
filename: /certs/sds_cert.pem
private_key:
filename: /certs/sds_key.pem
Path to CA certificate bundle for validating the xDS server certificate is given in SDS config file /etc/envoy/validation_context_sds_secret.yaml:
.. code-block:: yaml
resources:
- "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
name: validation_context_sds
validation_context:
trusted_ca:
filename: /certs/cacert.pem
In the above example, a watch will be established on /certs. File movement in this directory
will trigger an update. An alternative common key rotation scheme that provides improved atomicity
is to establish an active symlink /certs/current and use an atomic move operation to replace the
symlink. The watch in this case needs to be on the certificate's grandparent directory. Envoy
supports this scheme via the use of watched_directory. Continuing the above examples:
.. code-block:: yaml
resources:
- "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
name: tls_sds
tls_certificate:
certificate_chain:
filename: /certs/current/sds_cert.pem
private_key:
filename: /certs/current/sds_key.pem
watched_directory:
path: /certs
.. code-block:: yaml
resources:
- "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret"
name: validation_context_sds
validation_context:
trusted_ca:
filename: /certs/current/cacert.pem
watched_directory:
path: /certs
Secret rotation can be performed with:
.. code-block:: bash
ln -s <path to new secrets> /certs/new && mv -Tf /certs/new /certs/current
SSL socket factory outputs following SDS related statistics. They are all counter type.
For downstream listeners, they are in the listener.<LISTENER_IP>.server_ssl_socket_factory. namespace.
.. csv-table:: :header: Name, Description :widths: 1, 2
ssl_context_update_by_sds, Total number of ssl context has been updated.
downstream_context_secrets_not_ready, Total number of downstream connections reset due to empty ssl certificate.
For upstream clusters, they are in the cluster.<CLUSTER_NAME>.client_ssl_socket_factory. namespace.
.. csv-table:: :header: Name, Description :widths: 1, 2
ssl_context_update_by_sds, Total number of ssl context has been updated.
upstream_context_secrets_not_ready, Total number of upstream connections reset due to empty ssl certificate.
SDS has a :ref:statistics <subscription_statistics> tree rooted in the sds.<SECRET_NAME>.
namespace. In addition, the following statistics are tracked in this namespace:
.. csv-table:: :header: Name, Description :widths: 1, 2
key_rotation_failed, Total number of filesystem key rotations that failed outside of an SDS update.
By default SDS certificate fetching blocks initialization of the listeners and the clusters that
reference them. In some cases, it is preferable to accept connections without having the SDS secret
and request the certificate on-demand using the peer hello message fields, such as SNI, to derive
the secret name. This is useful for multi-tenant deployments, where a single listener or an upstream
cluster can present a variety of certificates to the peer. Envoy provides an :ref:on-demand certificate selector <extension_envoy.tls.certificate_selectors.on_demand_secret> that pauses the
TLS handshake to issue an SDS request for a certificate if not present, and then continues the
handshake after receiving the response.
A certificate obtained via the on-demand SDS is configured the same way as a regular TLS certificate defined in the context, e.g. all parent settings are applied. If there is a dynamic update to the parent TLS context, e.g. a validation context SDS update, on-demand certificate contexts also receive it and get updated. As a consequence, the handshake uses the latest version of the CA secret when resuming the handshake.
On-demand SDS should be used with DELTA_GRPC to manage the deletion of the secrets from the data plane. A resource removal sent via the xDS response will cancel the data plane subscription for the specific secret name. When using the regular GRPC xDS protocol, the subscription for each mapped secret remains active until the removal of the parent resource (listener or cluster).
In addition to the standard SDS subscription statistics <subscription_statistics>, the following
statistics are produced by the on-demand certificate extension. For downstream listeners, they are
in the listener.<stat_prefix>.on_demand_secret. namespace. For upstream clusters, the stat prefix
is cluster.<stat_prefix>.on_demand_secret..
.. csv-table:: :header: Name, Type, Description :widths: 1, 1, 2
cert_requested, Counter, Total number of new SDS subscriptions created
cert_updated, Counter, Total number of certificate updates
cert_active, Gauge, Number of active certificate subscriptions and certificates
.. note::
Session resumption is currently not supported for on-demand certificates.
Examples ^^^^^^^^
The following downstream TLS context configuration uses the SNI field as the secret name in the SDS request:
.. validated-code-block:: yaml :type-name: envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context: custom_tls_certificate_selector: name: on-demand typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.cert_selectors.on_demand_secret.v3.Config config_source: api_config_source: api_type: DELTA_GRPC grpc_services: - envoy_grpc: cluster_name: some_xds_cluster certificate_mapper: name: sni typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.cert_mappers.sni.v3.SNI default_value: "default_host" # Starts fetching the secret prior to any requests. prefetch_secret_names: - default_host disable_stateless_session_resumption: true disable_stateful_session_resumption: true
The following downstream TLS context configuration is analogous to the one with a regular SDS TLS certificate, but does not block the listener from listening until the SDS response arrives. Instead, connections are accepted and paused during the TLS handshake, and resumed once the certificate is received.
.. validated-code-block:: yaml :type-name: envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context: custom_tls_certificate_selector: name: on-demand typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.cert_selectors.on_demand_secret.v3.Config config_source: api_config_source: api_type: DELTA_GRPC grpc_services: - envoy_grpc: cluster_name: some_xds_cluster certificate_mapper: name: sni typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.cert_mappers.static_name.v3.StaticName name: secret_0 prefetch_secret_names: - secret_0 disable_stateless_session_resumption: true disable_stateful_session_resumption: true
The following upstream TLS context configuration uses a dynamic filter state value passed from the downstream listener:
.. validated-code-block:: yaml :type-name: envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context: custom_tls_certificate_selector: name: on-demand typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.cert_selectors.on_demand_secret.v3.Config config_source: api_config_source: api_type: DELTA_GRPC grpc_services: - envoy_grpc: cluster_name: some_xds_cluster certificate_mapper: name: filter_state_override typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.cert_mappers.filter_state_override.v3.Config default_value: "default_secret"
For the upstream filter state override configuraton above to work, the value must be written in the downstream filter chain, e.g. using the following filter configuration:
.. validated-code-block:: yaml :type-name: envoy.config.listener.v3.Filter
name: envoy.filters.network.set_filter_state typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.set_filter_state.v3.Config on_new_connection: - object_key: envoy.tls.certificate_mappers.on_demand_secret factory_key: envoy.hashable_string format_string: text_format_source: inline_string: my_secret_name shared_with_upstream: ONCE