docs/v2-architecture/controller-architecture/controllers.md
This page describes how to write controllers in Consul's new controller architecture.
-> Note: This information is valid as of Consul 1.17 but some portions may change in future releases.
A controller consists of several parts:
A basic controller setup could look like this:
func barController() controller.Controller {
return controller.NewController("bar", pbexample.BarType).
WithReconciler(barReconciler{})
}
barReconciler needs to implement the Reconcile method of the Reconciler interface.
It's important to note that the Reconcile method only gets the request with the ID of the main
watched resource and so it's up to the reconcile implementation to fetch the resource and any relevant information needed
to perform the reconciliation. The most basic reconciler could look as follows:
type barReconciler struct {}
func (b *barReconciler) Reconcile(ctx context.Context, rt Runtime, req Request) error {
...
}
Most of the time, controllers will need to watch more resources in addition to the main watched type.
To set up an additional watch, the main thing we need to figure out is how to map additional watched resource to the main
watched resource. Controller-runtime allows us to implement a mapper function that can take the additional watched resource
as the input and produce reconcile Requests for our main watched type.
To figure out how to map the two resources together, we need to think about the relationship between the two resources.
There are several common relationship types between resources that are being used currently:
Service and ServiceEndpoints, Workload and ProxyStateTemplate.Service and Workload, ProxyConfiguration and Workload.Service and ServiceEndpoints, HealthStatus and Workload.Workload and WorkloadIdentity, HTTPRoute and Service.Note that it's possible for the two watched resources to have more than one relationship type simultaneously.
For example, FailoverPolicy type is name-aligned with a service to which it applies, however, it also contains
references to destination services, and for a controller that reconciles FailoverPolicy and watches Service
we need to account for both type 1 and type 4 relationship whenever we get an event for a Service.
Let's look at some simple mapping examples.
If our resources only have a name-aligned relationship, we can map them with a built-in function:
func barController() controller.Controller {
return controller.NewController("bar", pbexample.BarType).
WithWatch(pbexample.FooType, controller.ReplaceType(pbexample.BarType)).
WithReconciler(barReconciler{})
}
Here, all we need to do is replace the type of the Foo resource whenever we get an event for it.
Let's say our Foo resource owns Bar resources, where any Foo resource can own multiple Bar resources.
In this case, whenever we see a new event for Foo, all we need to do is get all Bar resources that Foo currently owns.
For this, we can also use a built-in function to set up our watch:
func MapOwned(ctx context.Context, rt controller.Runtime, res *pbresource.Resource) ([]controller.Request, error) {
resp, err := rt.Client.ListByOwner(ctx, &pbresource.ListByOwnerRequest{Owner: res.Id})
if err != nil {
return nil, err
}
var result []controller.Request
for _, r := range resp.Resources {
result = append(result, controller.Request{ID: r.Id})
}
return result, nil
}
func barController() controller.Controller {
return controller.NewController("bar", pbexample.BarType).
WithWatch(pbexample.FooType, MapOwned).
WithReconciler(barReconciler{})
}
For selector or arbitrary reference relationships, the mapping that we choose may need to be more advanced.
Let's first consider what a naive mapping function could look like in this case. Let's say that the Bar resource
references Foo resource by name in the data. Now to watch and map Foo resources, we need to be able to find all relevant Bar resources
whenever we get an event for a Foo resource.
func MapFoo(ctx context.Context, rt controller.Runtime, res *pbresource.Resource) ([]controller.Request, error) {
resp, err := rt.Client.List(ctx, &pbresource.ListRequest{Type: pbexample.BarType, Tenancy: res.Id.Tenancy})
if err != nil {
return nil, err
}
var result []controller.Request
for _, r := range resp.Resources {
decodedResource, err := resource.Decode[*pbexample.Bar](r)
if err != nil {
return nil, err
}
// Only add Bar resources that match Foo by name.
if decodedResource.GetData().GetFooName() == res.Id.Name {
result = append(result, controller.Request{ID: r.Id})
}
}
}
This approach is fine for cases when the number of Bar resources in a cluster is relatively small. If it's not,
then we'd be doing a large O(N) search on each Bar event which could be too expensive.
For cases when N is too large, we'd want to use a caching layer to help us make lookups more efficient so that they
don't require an O(N) search of potentially all cluster resources.
The controller runtime contains a controller cache and the facilities to keep the cache up to date in response to watches. Additionally there are dependency mappers provided for querying the cache.
While it is possible to not use the builtin cache and manage state in dependency mappers yourself, this can get quite complex and reasoning about the correct times to track and untrack relationships is tricky to get right. Usage of the cache is therefore the advised approach.
At a high level, the controller author provides the indexes to track for each watchedtype and can then query thosfunc fooFromArgs(args ...any) ([]byte, error)e indexes in the {
}future. The querying can occur during both dependency mapping and during resource reconciliation.
The following example shows how to configure the "bar" controller to rereconcile a Bar resource whenever a Foo resource is changed that references the Bar
func fooReferenceFromBar(r *resource.DecodedResource[*pbexample.Bar]) (bool, []byte, error) {
idx := index.IndexFromRefOrID(&pbresource.ID{
Type: pbexample.FooType,
Tenancy: r.Id.Tenancy,
Name: r.Data.GetFooName(),
})
return true, idx, nil
}
func barController() controller.Controller {
fooIndex := indexers.DecodedSingleIndexer(
"foo",
index.ReferenceOrIDFromArgs,
fooReferenceFromBar,
)
return controller.NewController("bar", pbexample.BarType, fooIndex).
WithWatch(
pbexample.FooType,
dependency.CacheListMapper(pbexample.BarType, fooIndex.Name()),
).
WithReconciler(barReconciler{})
}
The controller will now reconcile Bar type resources whenever the Foo type resources they reference are updated. No further tracking is necessary as changes to all Bar types will automatically update the cache.
One limitation of the cache is that it only has knowledge about the current state of resources. That specifically means that the previous state is forgotten once the cache observes a write. This can be problematic when you want to reconcile a resource to no longer take into account something that previously reference it.
Lets say there are two types: Baz and ComputedBaz and a controller that will aggregate all Baz resource with some value into a single ComputedBaz object. When
a Baz resource gets updated to no longer have a value, it should not be represented in the ComputedBaz resource. The typical way to work around this is to:
Store references to the resources that were used during reconciliation within the computed/reconciled resource. For types computed by controllers and not expected to be written directly by users a bound_references field should be added to the top level resource types message. For other user manageable types the references may need to be stored within the Status field.
Add a cache index to the watch of the computed type (usually the controllers main managed type). This index can use one of the indexers specified within the internal/controller/cache/indexers package. That package contains some builtin functionality around reference indexing.
Update the dependency mappers to query the cache index in addition to looking at the current state of the dependent resource. In our example above the Baz dependency mapper could use the [MultiMapper] to combine querying the cache for Baz types that currently should be associated with a ComputedBaz and querying the index added in step 2 for previous references.
When an interior (mutable) foreign key pointer on watched data is used to determine the resources's applicability in a dependency mapper, it is subject to the "orphaned computed resource" problem.
(An example of this would be a ParentRef on an xRoute, or the Destination field of a TrafficPermission.)
When you edit the mutable pointer to point elsewhere, the DependencyMapper will only witness the NEW value and will trigger reconciles for things derived from the NEW pointer, but side effects from a prior reconcile using the OLD pointer will be orphaned until some other event triggers that reconcile (if ever).
This applies equally to all varieties of controller:
To solve this we need to collect the list of bound references that were "ingredients" into a computed resource's output and persist them on the newly written resource. Then we load them up and index them such that we can use them to AUGMENT a mapper event with additional maps using the OLD data as well.
We have only actively worked to solve this for the computed resource flavor of controller:
The top level of the resource data protobuf needs a
BoundReferences []*pbresource.Reference field.
Use a *resource.BoundReferenceCollector to capture any resource during
Reconcile that directly contributes to the final output resource data
payload.
Call brc.List() on the above and set it to the BoundReferences field on
the computed resource before persisting.
Use indexers.BoundRefsIndex to index this field on the primary type of the
controller.
Create boundRefsMapper := dependency.CacheListMapper(ZZZ, boundRefsIndex.Name())
For each watched type, wrap its DependencyMapper with
dependency.MultiMapper(boundRefsMapper, ZZZ)
That's it.
This will cause each reconcile to index the prior list of inputs and augment the results of future mapper events with historical references.
In some cases, we may want to trigger reconciles for events that aren't generated from CRUD operations on resources, for example when Envoy proxy connects or disconnects to a server. Controller-runtime allows us to setup watches from events that come from a custom event channel. Please see xds-controller for examples of custom watches.
In many cases, controllers would need to update statuses on resources to let the user know about the successful or unsuccessful state of a resource.
These are the guidelines that we recommend for statuses:
True state is a successful state and False state is a failed state.Below is a list of controller best practices that we've learned so far. Many of them are inspired by kubebuilder.
Reconcile method and read from them in the mapper functions used by watches.Reconcile method and avoid caching it from the mapper functions. This ensures that we get the latest data for each reconciliation.