.. _arch_overview_ssl:

TLS

Envoy supports both :ref:TLS termination <envoy_v3_api_field_config.listener.v3.FilterChain.transport_socket> in listeners as well as :ref:TLS origination <envoy_v3_api_field_config.cluster.v3.Cluster.transport_socket> when making connections to upstream clusters. Support is sufficient for Envoy to perform standard edge proxy duties for modern web services as well as to initiate connections with external services that have advanced TLS requirements (TLS1.2, SNI, etc.). Envoy supports the following TLS features:

Configurable ciphers: Each TLS listener and client can specify the ciphers that it supports.
Client certificates: Upstream/client connections can present a client certificate in addition to server certificate verification.
Certificate verification and pinning: Certificate verification options include basic chain verification, subject name verification, and hash pinning.
Certificate revocation: Envoy can check peer certificates against a certificate revocation list (CRL) if one is :ref:provided <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.crl>.
ALPN: TLS listeners support ALPN. The HTTP connection manager uses this information (in addition to protocol inference) to determine whether a client is speaking HTTP/1.1 or HTTP/2.
SNI: SNI is supported for both server (listener) and client (upstream) connections.
Session resumption: Server connections support resuming previous sessions via TLS session tickets (see RFC 5077 <https://www.ietf.org/rfc/rfc5077.txt>_). Resumption can be performed across hot restarts and between parallel Envoy instances (typically useful in a front proxy configuration).
BoringSSL private key methods: TLS private key operations (signing and decrypting) can be performed asynchronously from :ref:an extension <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.PrivateKeyProvider>. This allows extending Envoy to support various key management schemes (such as TPM) and TLS acceleration. This mechanism uses BoringSSL private key method interface <https://github.com/google/boringssl/blob/c0b4c72b6d4c6f4828a373ec454bd646390017d4/include/openssl/ssl.h#L1169>_.
OCSP Stapling: Online Certificate Stapling Protocol responses may be stapled to certificates.

Underlying implementation

Currently Envoy is written to use BoringSSL <https://boringssl.googlesource.com/boringssl>_ as the default TLS provider.

OpenSSL <https://openssl.org>_ can also be used as an alternative, when Envoy is built using the --config=openssl Bazel option. OpenSSL libraries are not statically linked into the main Envoy executable; they must be present at runtime and are loaded dynamically. HTTP/3 (QUIC) is not available with OpenSSL builds. OpenSSL builds are not currently covered by the :repo:Envoy security policy <SECURITY.md>.

.. _arch_overview_ssl_fips:

FIPS 140-2

BoringSSL can be built in a FIPS-compliant mode <https://boringssl.googlesource.com/boringssl/+/main/crypto/fipsmodule/FIPS.md>, following the build instructions from the Security Policy for BoringCrypto module <https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3678.pdf>, using --config=boringssl-fips Bazel option. Currently, the BoringSSL/FIPS build will only work Linux-x86_64.

AWS-LC FIPS can also be used with --config=aws-lc-fips, and has wider architecture support.

When Envoy has been built for FIPS, you should see BoringSSL-FIPS or AWS-LC-FIPS in the :option:--version output. When built with OpenSSL, the output will show OpenSSL.

It's important to note that while using FIPS-compliant module is necessary for FIPS compliance, it's not sufficient by itself, and depending on the context, additional steps might be necessary. The extra requirements may include using only approved algorithms and/or using only private keys generated by a module operating in FIPS-approved mode. For more information, please refer to the Security Policy for BoringCrypto module <https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3678.pdf>_ and/or an accredited CMVP laboratory <https://csrc.nist.gov/projects/testing-laboratories>_.

Please note that the FIPS-compliant build is based on an older version of BoringSSL than the non-FIPS build, and it doesn't support the most recent QUIC APIs.

.. _arch_overview_ssl_enabling_verification:

Enabling certificate verification

Certificate verification of both upstream and downstream connections is not enabled unless the validation context specifies one or more trusted authority certificates.

Example configuration ^^^^^^^^^^^^^^^^^^^^^

.. literalinclude:: _include/ssl.yaml :language: yaml

/etc/ssl/certs/ca-certificates.crt is the default path for the system CA bundle on Debian systems. :ref:trusted_ca <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.trusted_ca> along with :ref:match_typed_subject_alt_names <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.match_typed_subject_alt_names> makes Envoy verify the server identity of 127.0.0.1:1234 as "foo" in the same way as e.g. cURL does on standard Debian installations. Common paths for system CA bundles on Linux and BSD are:

/etc/ssl/certs/ca-certificates.crt (Debian/Ubuntu/Gentoo etc.)
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (CentOS/RHEL 7)
/etc/pki/tls/certs/ca-bundle.crt (Fedora/RHEL 6)
/etc/ssl/ca-bundle.pem (OpenSUSE)
/usr/local/etc/ssl/cert.pem (FreeBSD)
/etc/ssl/cert.pem (OpenBSD)

See the reference for :ref:UpstreamTlsContexts <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.UpstreamTlsContext> and :ref:DownstreamTlsContexts <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.DownstreamTlsContext> for other TLS options.

.. attention::

If only :ref:trusted_ca <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.trusted_ca> is specified, Envoy will verify the certificate chain of the presented certificate, but not its subject name, hash, etc. Other validation context configuration is typically required depending on the deployment.

Custom Certificate Validator ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The configuration explained above is used by the "default" certificate validator. Envoy also supports custom validators in envoy.tls.cert_validator extension category which can be configured on :ref:CertificateValidationContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.CertificateValidationContext>.

For example, Envoy can be configured to verify peer certificates following the SPIFFE <https://github.com/spiffe/spiffe>_ specification with multiple trust bundles in a single listener or cluster. For more detail, please refer to :ref:the documentation of custom_validator_config field<envoy_v3_api_field_extensions.transport_sockets.tls.v3.CertificateValidationContext.custom_validator_config>.

.. _arch_overview_ssl_cert_select:

Certificate selection

:ref:DownstreamTlsContexts <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.DownstreamTlsContext> support multiple TLS certificates. These may be a mix of RSA and ECDSA certificates for multiple server name patterns.

Certificate config/loading rules:

DNS SANs or Subject Common Name is extracted as server name pattern to match SNI during handshake. Subject Common Name is not used if DNS SANs are present in the certificate.
FQDN like "test.example.com" and wildcard like "*.example.com" are valid at the same time, which will be loaded as two different server name patterns.
If multiple certificates of a particular type (RSA or ECDSA) are specified for the same name or name pattern, the first one loaded is used for that name.
Non-P-256, P-384 or P-521 server ECDSA certificates are rejected.
Static and SDS certificates may not be mixed in a given :ref:DownstreamTlsContext <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.DownstreamTlsContext>.

Certificate selection rules:

If the client supports SNI, e.g. SNI is "test.example.com", it looks for a cert that exactly matches to the SNI. If the certificate adheres to the OCSP policy and matches to key type, it is selected for handshake. If the certificate adheres to the OCSP policy, but key type is RSA while client is ECDSA capable, it is marked as as the candidate and continues searching until a cert is selected with perfect match or certs exhausted. Candidate will be selected for handshake if there is no perfect match.
If the client supports SNI, but no cert is selected from certs that exactly matches to SNI, it matches on wildcard server name. e.g. if SNI is "test.example.com", a certificate with "test.example.com" will be preferred over "*.example.com". And wildcard matching only works for 1 level of depth, so "*.com" will not be a match for "test.example.com". Afterwards, it execuates OCSP and key type checking on each cert which is the same as what happens after exact SNI matching.
If no cert is selected from certs that matches wildcard name, the candidate cert is selected for handshake if it is present. If there is no candidate, check :ref:full_scan_certs_on_sni_mismatch <envoy_v3_api_field_extensions.transport_sockets.tls.v3.DownstreamTlsContext.full_scan_certs_on_sni_mismatch>, go to full scan all certificates if it is enabled, otherwise pick the first certificate for handshake.
If the client does not provide SNI at all, go to full scan no matter :ref:full_scan_certs_on_sni_mismatch <envoy_v3_api_field_extensions.transport_sockets.tls.v3.DownstreamTlsContext.full_scan_certs_on_sni_mismatch> is false or true.
Full scan execuates OCSP and key type checking on each cert which is the same as described above in exact SNI matching. It falls back to the first cert in the whole list if there is no cert selected.
Currently only two kinds of key type are supported, RSA or ECDSA. If the client supports P-256, P-384 or P-521 ECDSA, the P-256, P-384 or P-521 ECDSA certificate is preferred over RSA. The certificate that it falls back to might result in a failed handshake. For instance, a client only supports RSA certificates and the certificate only support ECDSA.
The final selected certificate must adhere to the OCSP policy. If no such certificate is found, the connection is refused.

.. note:: With the support of SNI-based certificate selection, it allows configuring large number of certificates for multiple hostnames. :ref:full_scan_certs_on_sni_mismatch <envoy_v3_api_field_extensions.transport_sockets.tls.v3.DownstreamTlsContext.full_scan_certs_on_sni_mismatch> is introduced to determine if we continue full scan on SNI mismatch when the client provides SNI. SNI mismatch contains two cases in this context, one is there is no cert that matches to SNI, another one is there are certs matches to SNI while OCSP policy fails on those certs. The :ref:full_scan_certs_on_sni_mismatch <envoy_v3_api_field_extensions.transport_sockets.tls.v3.DownstreamTlsContext.full_scan_certs_on_sni_mismatch> defaults to false, so full scan is disabled by default. If full scan is enabled, it will look for the cert from the whole cert list on SNI mismatch, this could be a problem for a potential DoS attack because of O(n) complexity.

Only a single TLS certificate is supported today for :ref:UpstreamTlsContexts <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.UpstreamTlsContext>.

Secret discovery service (SDS)

TLS certificates can be specified in the static resource or can be fetched remotely. Certificate rotation is supported for static resources by sourcing :ref:SDS configuration from the filesystem <xds_certificate_rotation> or by pushing updates from the SDS server. Please see :ref:SDS <config_secret_discovery_service> for details.

.. _arch_overview_ssl_ocsp_stapling:

OCSP Stapling

:ref:DownstreamTlsContexts <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.DownstreamTlsContext> support stapling an Online Certificate Status Protocol (OCSP) response to a TLS certificate during the handshake. The ocsp_staple field allows the operator to supply a pre-computed OCSP response per-certificate in the context. A single response may not pertain to multiple certificates. If provided, OCSP responses must be valid and affirm the certificate has not been revoked. Expired OCSP responses are accepted, but may cause downstream connection errors depending on the OCSP staple policy.

:ref:DownstreamTlsContexts <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.DownstreamTlsContext> support an ocsp_staple_policy field to control whether Envoy should stop using a certificate or continue without stapling when its associated OCSP response is missing or expired. Certificates marked as must-staple <https://tools.ietf.org/html/rfc7633>_ require a valid OCSP response regardless of the OCSP staple policy. In practice, a must-staple certificate causes cEnvoy to behave as if the OCSP staple policy is :ref:MUST_STAPLE<envoy_v3_api_enum_value_extensions.transport_sockets.tls.v3.DownstreamTlsContext.OcspStaplePolicy.MUST_STAPLE>. Envoy will not use a must-staple certificate for new connections after its OCSP response expires.

OCSP responses are never stapled to TLS requests that do not indicate support for OCSP stapling via the status_request extension.

OCSP responses are ignored for :ref:UpstreamTlsContexts <envoy_v3_api_msg_extensions.transport_sockets.tls.v3.UpstreamTlsContext>.

.. _arch_overview_ssl_auth_filter:

Authentication filter

Envoy provides a network filter that performs TLS client authentication via principals fetched from a REST VPN service. This filter matches the presented client certificate hash against the principal list to determine whether the connection should be allowed or not. Optional IP allowlisting can also be configured. This functionality can be used to build edge proxy VPN support for web infrastructure.

Client TLS authentication filter :ref:configuration reference <config_network_filters_client_ssl_auth>.

.. _arch_overview_ssl_custom_handshaker:

Custom handshaker extension

The :ref:CommonTlsContext <envoy_v3_api_field_extensions.transport_sockets.tls.v3.CommonTlsContext.custom_handshaker> has a custom_handshaker extension which can be used to override SSL handshake behavior entirely. This is useful for implementing any TLS behavior which is difficult to express with callbacks. It is not necessary to write a custom handshaker to use private key methods, see the :ref:private key method interface <arch_overview_ssl> described above.

To avoid reimplementing all of the Ssl::ConnectionInfo <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/include/envoy/ssl/connection.h#L19>_ interface, a custom implementation might choose to extend Envoy::Extensions::TransportSockets::Tls::SslHandshakerImpl <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/source/extensions/transport_sockets/tls/ssl_handshaker.h#L40>_.

Custom handshakers need to explicitly declare via HandshakerCapabilities <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/include/envoy/ssl/handshaker.h#L68-L89>_ which TLS features they are responsible for. The default Envoy handshaker will manage the remainder.

A useful example handshaker, named SslHandshakerImplForTest, lives in this test <https://github.com/envoyproxy/envoy/blob/64bd6311bcc8f5b18ce44997ae22ff07ecccfe04/test/extensions/transport_sockets/tls/handshaker_test.cc#L174-L184>_ and demonstrates special-case SSL_ERROR handling and callbacks.

.. _arch_overview_ssl_trouble_shooting:

Trouble shooting

When Envoy originates TLS when making connections to upstream clusters, any errors will be logged into :ref:UPSTREAM_TRANSPORT_FAILURE_REASON<config_access_log_format_upstream_transport_failure_reason> field or :ref:AccessLogCommon.upstream_transport_failure_reason<envoy_v3_api_field_data.accesslog.v3.AccessLogCommon.upstream_transport_failure_reason> field.

When Envoy listener gets connection and perform TLS with downstream, any errors will be logged into :ref:DOWNSTREAM_TRANSPORT_FAILURE_REASON<config_access_log_format_downstream_transport_failure_reason> field or :ref:AccessLogCommon.downstream_transport_failure_reason<envoy_v3_api_field_data.accesslog.v3.AccessLogCommon.downstream_transport_failure_reason> field.

Common errors are:

Secret is not supplied by SDS: Envoy is still waiting SDS to deliver key/cert or root CA.
SSLV3_ALERT_CERTIFICATE_EXPIRED: Peer certificate is expired and not allowed in config.
SSLV3_ALERT_CERTIFICATE_UNKNOWN: Peer certificate is not in config specified SPKI.
SSLV3_ALERT_HANDSHAKE_FAILURE: Handshake failed, usually due to upstream requires client certificate but not presented.
TLSV1_ALERT_PROTOCOL_VERSION: TLS protocol version mismatch.
TLSV1_ALERT_UNKNOWN_CA: Peer certificate CA is not in trusted CA.

More detailed list of error that can be raised by BoringSSL can be found here <https://github.com/google/boringssl/blob/main/crypto/err/ssl.errordata>_