DevDocumentation/ARCHITECTURE.md
In this document you will find information about the internal design and architecture of the Emissary-ingress (formerly known as Ambassador API Gateway). Emissary-ingress provides a Kubernetes-native load balancer, API gateway and ingress controller that is built on top of Envoy Proxy.
Looking for end user guides for Emissary-ingress? You can check out the end user guides at https://www.getambassador.io/docs/emissary/.
Emissary-ingress is a Kubernetes native API Gateway built on top of Envoy Proxy. We utilize Kubernetes CRDs to provide an expressive API to configure Envoy Proxy to handle routing traffic into your cluster.
Check this blog post for additional context around the motivations and architecture decisions made for Emissary-ingress.
At the core of Emissary-ingress is Envoy Proxy which has very extensive configuration and extensions points. Getting this right can be challenging so Emissary-ingress provides Kubernetes Administrators and Developers a cloud-native way to configure Envoy using declarative yaml files. Here are the core components of Emissary-Ingress:
Kubernetes allows extending its API through the use of Customer Resource Definitions (aka CRDs) which allow solutions like Emissary-ingress to add custom resources to K8s and allow developers to treat them like any other K8s resource. CRDs provide validation, strong typing, structured data, versioning and are persisted in etcd along with the core Kuberenetes resources.
Emissary-ingress provides a set of CRD's that are applied to a cluster and then are watched by Emissary-ingress. Emissary-ingress then uses the data from these CRD's along with the standard K8's resources (services, endpoints, etc...) to dynamically generate Envoy Proxy configuration. Depending on the version of Emissary-ingress there might be multiple versions of the CRD's that are suppported.
You can read the user documentation (see additional reading below) to find out more about all the various CRDs that are used and how to configure them. For understanding, how they are defined you can take a look in pkg/getambassador.io/* directory. In this directory, you will find a directory per version of the CRDs and for each version you will see the Golang structs that define the data structures that are used for each of the Emissary-ingress custom resources. It's recommended to read the doc.go file for information about API guidelines followed and how the comment markers are used by the build system.
The build system (make) uses controller-gen to generate the required YAML representation for the customer resources that can be found at pkg/getambassador.io/crds.yaml. This file is auto-generated and checked into the respository. This is the file that is applied to a cluster extending the Kubernetes API. If any changes are made to the custom resources then it needs to be re-generated and checked-in as part of your PR. Running make generate will trigger the generation of this file and other generated files (protobufs) that are checked into the respository as well. If you want to see more about the build process take a look at build-aux/generate.mk.
Annotations: K8s allows developers to provide Annotations as well to the standard K8s Resources (Services, Ingress, etc...). Annotations were the preferred method of configuring early versions of Emissary-ingress but annotations did not provide validation and can be error prone. However, with the introduction of CRD's these are now the preferred method and annotations are only supported for backwards compatibility. We won't discuss the annotations much here due to this but rather making you aware that they exist.
Kubernetes provides the ability to have multiple versions of Custom Resources similiar to the core K8s resources but it is only capable of having a single storage version that is persisted in etcd. Custom Resource Definitions can define a ConversionWebHook that Kubernetes will call whenever it receives a version that is not the storage version.
You can check the current storage version by looking at pkg/getambassador.io/crds.yaml and searching for the storage: true field and seeing which version is the storage version of the custom resource (at the time of writing this it is v2).
The apiext container is the Emissary-ingress's server implementation for the conversion webhook that is registered with our custom resources. Each custom resource will have a section similiar to the following in the pkg/getambassador.io/crds.yaml:
conversion:
strategy: Webhook
webhook:
clientConfig:
service:
name: emissary-apiext
namespace: emissary-system
conversionReviewVersions:
- v1
This is telling the Kubernetes API Server to call a WebHook using a Service within the cluster that is called emissary-apiext that can be found in the emissary-system namespace. It also states that our server implementation supports the v1 version of the WebHook protocol so the K8s API Server will send the request and expect the response in the format for v1.
The implementation of the apiext server can be found in cmd/apiext and it leverages the controller-runtime library which is vendored in vendor/sigs.k8s.io/controller-runtime. When this process starts up it will do the following:
https.One of the major goals of the Emissary-ingress is to simplify the deployment of Envoy Proxy in a cloud-native friendly way using containers and declarative CRD's. To honor this goal Emissary-ingress is packaged up into a single image with all the necessary components.
This section will give a high-level overview of each of these components and will help provide you direction on where you can find more information on each of the components.
Emissary-ingress has evolved over many years, many contributors and many versions of Kubernetes which has led to the internal components being implemented in different programming languages. Some of the components are pre-built binaries like envoy, first-party python programs and first-party golang binaries. To provide a single entrypoint for the container startup the Golang binary called busyambassador was introduced.
/buildroot/ambassador/python/entrypoint.sh
The busyambassador binary provides a busybox like interface that dispatches the CMD's that are provided to a container for the various configured Golang binaries. This enables a single image to support multiple binaries on startup that are declartively set within a deployment in the command field when setting the image for a deployment.
The image takes advantage of the ENTRYPOINT and CMD fields within a docker image manifest. You can see this in builder/Dockerfile in the final optimized image on the last line there is ENTRYPOINT [ "bash", "/buildroot/ambassador/python/entrypoint.sh" ]. This entrypoint cannot be overriden by the user and will run that bash script. By default the bash script will run the entrypoint binary which will be discussed in the next section but if passed a known binary name, then busyambassador will run the correct command.
To learn more about busyambassador the code can be found:
cmd/busyambassadorpkg/busyNote: the bash script will just exec into the
busyambassadorGolang binary in most cases and is still around for historical reasons and advanced debugging scenarios.
Additional Reading: If you want to know more about how containers work with entrypoint and commands then take a look at this blogpost. https://www.bmc.com/blogs/docker-cmd-vs-entrypoint/
The entrypoint Golang binary is the default binary that busyambassador will run on container startup. It is the parent process for all the other processes that are run within the single ambassador image for Emissary-Ingress. At a high-level it starts and manages multiple go-routines, starts other child processes such as diagd (python program) and envoy (c++ compiled binary).
Here is a list of everything managed by the entrypoint binary. Each one is indicated by whether its a child OS process that is started or a goroutine (note: some of the OS processes are started/managed in goroutines but the core logic resides within the child process thus they are marked as such).
| Description | Goroutine | OS.Exec |
|---|---|---|
diagd - admin ui & config processor | :white_check_mark: | |
ambex - the Envoy ADS Server | :white_check_mark: | |
envoy - proxy routing data | :white_check_mark: | |
| SnapshotServer - expose in-memory snapshot over localhost | :white_check_mark: | |
| ExternalSnapshotServer - Ambassador Cloud friendly exposed over localhost | :white_check_mark: | |
| HealthCheck - endpoints for K8s liveness/readiness probes | :white_check_mark: | |
| Watt - Watch k8s, consul & files for cluster changes | :white_check_mark: | |
| Sidecar Processes - start various side car processes | :white_check_mark: |
Some of these items will be discussed in more detail but the best places to get started looking at the entrypoint is by looking at cmd/entrypoint/entrypoint.go.
To see how the container passes
entrypointas the default binary to run on container startup you can look atpython/entrypoint.shwhere it callsexec busyambassador entrypoint "$@"which will drop the shell process and will run the entrypoint process via busyambassador.
Watch All The Things (aka Watt) is tasked with watching alot of things, hence the name :smile:. Specifically, its job is to watch for changes in the K8s Cluster and potentially Consul and file system changes. Watt is the beginning point for the end-to-end data flow from developer applying the configuration to envoy being configured. You can find the code for this in the cmd/entrypoint/watcher.go file.
The watching of the K8s Cluster changes is where Emissary-ingress will get most of its configuration by looking for K8s Resources (e.g. services,ingresss, etc...) as well as the Emissary-ingress CRD Resources (e.g. Host, Mapping, Listeners, etc...). A consulWatcher will be started if a user has configured a Mapping to use the ConsulResolver. You can find this code in cmd/entrypoint/consul.go. The filesystem is also watched for changes to support istio and how it mounts certificates to the filesystem.
Here is the general flow:
flowchart TD
subgraph watchers
K8s & consulW(Consul) & file(FileSystem)
K8s -.ConsulResolver.-> consulW
K8s & consulW(Consul) & file(FileSystem) --> sgen
end
API[K8s API] --> K8s
consul[Consul Manager] --> consulW
istio[Pod Filesystem] --> file
sgen("Snapshot \n Generation") --> update("Update Snapshot \n(in-memory)") --> diagd(Notify diagD);
update -.isEdgeStack?.-> AES(Notify AES)
Provides two main functions:
/ambassador/snapshots/snapshot-tmp.yaml./ambassador/snapshots/aconf-tmp.json/ambassador/snapshots/ir-tmp.jsonaconf, econf, ir, snapshot that get persisted in the snapshot path /ambassador/snapshots.
-tmp files written above into/ambassador/bootstrap-ads.json # this is used by envoy during startup to initial config itself and let it know about the static ADS Service/ambassador/enovy/envoy.json # this is used in ambex to generate the ADS snapshots along with the fastPath items/ambassador/clustermap.json # this might not be used either...envoy and ambex that a new snapshot has been persisted using signal SIGHUP
entrypoint that starts up envoy is blocking waiting for this signal to start envoyambex process continuously listens for this signal and it triggers a configuration update for ambex.kubestatus binary found in cmd/kubestatus which handles the communication to the clusterThis is the gRPC server implementation of the envoy xDS v2 and v3 api's based on ...
envoy.json into in-memory snapshots that are cached for v2/v3We maintain our own fork of Envoy that includes some additional commits for implementing some features in Emissary-Ingress.
Envoy does all the heavy-lifting
ambexTODO: talk about testing performed by kat-client/kat-server.
TODO: discuss the purpose of kat-client
TODO: discuss the purpose of kat-client