Sidecar or ambient? - Istio Io

An Istio service mesh is logically split into a data plane and a control plane.

The {{< gloss >}}data plane{{< /gloss >}} is the set of proxies that mediate and control all network communication between microservices. They also collect and report telemetry on all mesh traffic.

The {{< gloss >}}control plane{{< /gloss >}} manages and configures the proxies in the data plane.

Istio supports two main {{< gloss "data plane mode">}}data plane modes{{< /gloss >}}:

sidecar mode, which deploys an Envoy proxy along with each pod that you start in your cluster, or running alongside services running on VMs.
ambient mode, which uses a per-node Layer 4 proxy, and optionally a per-namespace Envoy proxy for Layer 7 features.

You can opt certain namespaces or workloads into each mode.

Sidecar mode

Istio has been built on the sidecar pattern from its first release in 2017. Sidecar mode is well understood and thoroughly battle-tested, but comes with a resource cost and operational overhead.

Each application you deploy has an Envoy proxy {{< gloss "injection" >}}injected{{< /gloss >}} as a sidecar
All proxies can process both Layer 4 and Layer 7

Ambient mode

Launched in 2022, ambient mode was built to address the shortcomings reported by users of sidecar mode. As of Istio 1.22, it is production-ready for single cluster use cases.

All traffic is proxied through a Layer 4-only node proxy
Applications can opt in to routing through an Envoy proxy to get Layer 7 features

Choosing between sidecar and ambient

Users often deploy a mesh to enable a zero-trust security posture as a first-step and then selectively enable L7 capabilities as needed. Ambient mesh allows those users to bypass the cost of L7 processing entirely when it’s not needed.

<table> <thead> <tr> <td style="border-width: 0px"></td> <th><strong>Sidecar</strong></th> <th><strong>Ambient</strong></th> </tr> </thead> <tbody> <tr> <th>Traffic management</th> <td>Full Istio feature set</td> <td>Full Istio feature set (requires using waypoint)</td> </tr> <tr> <th>Security</th> <td>Full Istio feature set</td> <td>Full Istio feature set: encryption and L4 authorization in ambient mode. Requires waypoints for L7 authorization.</td> </tr> <tr> <th>Observability</th> <td>Full Istio feature set</td> <td>Full Istio feature set: L4 telemetry in ambient mode; L7 observability when using waypoint</td> </tr> <tr> <th>Extensibility</th> <td>Full Istio feature set</td> <td>Via <a href="/docs/ambient/usage/extend-waypoint-wasm">WebAssembly plugins</a> (requires using waypoint) The EnvoyFilter API is not supported.</td> </tr> <tr> <th>Adding workloads to the mesh</th> <td>Label a namespace and restart all pods to have sidecars added</td> <td>Label a namespace - no pod restart required</td> </tr> <tr> <th>Incremental deployment</th> <td>Binary: sidecar is injected or it isn't</td> <td>Gradual: L4 is always on, L7 can be added by configuration</td> </tr> <tr> <th>Lifecycle management</th> <td>Proxies managed by application developer</td> <td>Platform administrator</td> </tr> <tr> <th>Utilization of resources</th> <td>Wasteful; CPU and memory resources must be provisioned for worst case usage of each individual pod</td> <td>Waypoint proxies can be auto-scaled like any other Kubernetes deployment. A workload with many replicas can use one waypoint, vs. each one having its own sidecar. </td> </tr> <tr> <th>Average resource cost</th> <td>Large</td> <td>Small</td> </tr> <tr> <th>Average latency (p90/p99)</th> <td>0.63ms-0.88ms</td> <td>Ambient: 0.16ms-0.20ms Waypoint: 0.40ms-0.50ms</td> </tr> <tr> <th>L7 processing steps</th> <td>2 (source and destination sidecar)</td> <td>1 (destination waypoint)</td> </tr> <tr> <th>Configuration at scale</th> <td>Requires <a href="/docs/ops/configuration/mesh/configuration-scoping/">configuration of the scope of each sidecar</a> to reduce configuration</td> <td>Works without custom configuration</td> </tr> <tr> <th>Supports "server-first" protocols</th> <td><a href="/docs/ops/deployment/application-requirements/#server-first-protocols">Requires configuration</a></td> <td>Yes</td> </tr> <tr> <th>Support for Kubernetes Jobs</th> <td>Complicated by long life of sidecar</td> <td>Transparent</td> </tr> <tr> <th>Security model</th> <td>Strongest: each workload has its own keys</td> <td>Strong: each node agent has only the keys for workloads on that node</td> </tr> <tr> <th>Compromised application pod gives access to mesh keys</th> <td>Yes</td> <td>No</td> </tr> <tr> <th>Support</th> <td>Stable, including multi-cluster</td> <td>Stable, only single-cluster</td> </tr> <tr> <th>Platforms supported</th> <td>Kubernetes (any CNI) Virtual machines</td> <td>Kubernetes (any CNI)</td> </tr> </tbody> </table>

Layer 4 vs Layer 7 features

The overhead for processing protocols at Layer 7 is substantially higher than processing network packets at Layer 4. For a given service, if your requirements can be met at L4, service mesh can be delivered at substantially lower cost.

Security

<table> <thead> <tr> <td style="border-width: 0px" width="20%"></td> <th width="40%">L4</th> <th width="40%">L7</th> </tr> </thead> <tbody> <tr> <th>Encryption</th> <td>All traffic between pods is encrypted using {{< gloss "mutual tls authentication" >}}mTLS{{< /gloss >}}.</td> <td>N/A—service identity in Istio is based on TLS.</td> </tr> <tr> <th>Service-to-service authentication</th> <td>{{< gloss >}}SPIFFE{{< /gloss >}}, via mTLS certificates. Istio issues a short-lived X.509 certificate that encodes the pod's service account identity.</td> <td>N/A—service identity in Istio is based on TLS.</td> </tr> <tr> <th>Service-to-service authorization</th> <td>Network-based authorization, plus identity-based policy, e.g.: <ul> <li>A can accept inbound calls from only "10.2.0.0/16";</li> <li>A can call B.</li> </ul> </td> <td>Full policy, e.g.: <ul> <li>A can GET /foo on B only with valid end-user credentials containing the READ scope.</li> </ul> </td> </tr> <tr> <th>End-user authentication</th> <td>N/A—we can't apply per-user settings.</td> <td>Local authentication of JWTs, support for remote authentication via OAuth and OIDC flows.</td> </tr> <tr> <th>End-user authorization</th> <td>N/A—see above.</td> <td>Service-to-service policies can be extended to require <a href="/docs/reference/config/security/conditions/">end-user credentials with specific scopes, issuers, principal, audiences, etc.</a> Full user-to-resource access can be implemented using external authorization, allowing per-request policy with decisions from an external service, e.g. OPA.</td> </tr> </tbody> </table>

Observability

<table> <thead> <tr> <td style="border-width: 0px" width="20%"></td> <th width="40%">L4</th> <th width="40%">L7</th> </tr> </thead> <tbody> <tr> <th>Logging</th> <td>Basic network information: network 5-tuple, bytes sent/received, etc. <a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#command-operators">See Envoy docs</a>.</td> <td><a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#command-operators">Full request metadata logging</a>, in addition to basic network information.</td> </tr> <tr> <th>Tracing</th> <td>Not today; possible eventually with HBONE.</td> <td>Envoy participates in distributed tracing. <a href="/docs/tasks/observability/distributed-tracing/overview/">See Istio overview on tracing</a>.</td> </tr> <tr> <th>Metrics</th> <td>TCP only (bytes sent/received, number of packets, etc.).</td> <td>L7 RED metrics: rate of requests, rate of errors, request duration (latency).</td> </tr> </tbody> </table>

Traffic management

<table> <thead> <tr> <td style="border-width: 0px" width="20%"></td> <th width="40%">L4</th> <th width="40%">L7</th> </tr> </thead> <tbody> <tr> <th>Load balancing</th> <td>Connection level only. <a href="/docs/tasks/traffic-management/tcp-traffic-shifting/">See TCP traffic shifting task</a>.</td> <td>Per request, enabling e.g. canary deployments, gRPC traffic, etc. <a href="/docs/tasks/traffic-management/traffic-shifting/">See HTTP traffic shifting task</a>.</td> </tr> <tr> <th>Circuit breaking</th> <td><a href="/docs/reference/config/networking/destination-rule/#ConnectionPoolSettings-TCPSettings">TCP only</a>.</td> <td><a href="/docs/reference/config/networking/destination-rule/#ConnectionPoolSettings-HTTPSettings">HTTP settings</a> in addition to TCP.</td> </tr> <tr> <th>Outlier detection</th> <td>On connection establishment/failure.</td> <td>On request success/failure.</td> </tr> <tr> <th>Rate limiting</th> <td><a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/rate_limit_filter#config-network-filters-rate-limit">Rate limit on L4 connection data only, on connection establishment</a>, with global and local rate limiting options.</td> <td><a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/rate_limit_filter#config-http-filters-rate-limit">Rate limit on L7 request metadata</a>, per request.</td> </tr> <tr> <th>Timeouts</th> <td>Connection establishment only (connection keep-alive is configured via circuit breaking settings).</td> <td>Per request.</td> </tr> <tr> <th>Retries</th> <td>Retry connection establishment</td> <td>Retry per request failure.</td> </tr> <tr> <th>Fault injection</th> <td>N/A—fault injection cannot be configured on TCP connections.</td> <td>Full application and connection-level faults (<a href="/docs/tasks/traffic-management/fault-injection/">timeouts, delays, specific response codes</a>).</td> </tr> <tr> <th>Traffic mirroring</th> <td>N/A—HTTP only</td> <td><a href="/docs/tasks/traffic-management/mirroring/">Percentage-based mirroring of requests to multiple backends</a>.</td> </tr> </tbody> </table>

Unsupported features

The following features are available in sidecar mode, but not yet implemented in ambient mode:

Sidecar-to-waypoint interoperability
Multi-cluster installations
Multi-network support
VM support