design/20220118.server-side-apply.md
Server-Side Apply is a Kubernetes feature whereby clients writing to a resource that is managed by more than one client can
Server-Side Apply works by the client sending a PATCH API request with a
Content-Type header with the value application/apply-patch+yaml. The
fieldManager=<my-field-manager> URL query can be optionally sent which
instructs which field manager to use (or alternatively, will be derived from the
client's user agent). This API call tells the API server that the client wishes
to manage and set the fields and values of the resource (or alternatively remove
managed fields which are omitted) as they appear in the body. A client's managed
fields are defined as the fields that are labelled with the client's manager
name <my-field-manager> (or client's user agent equivalent).
Server-Side Apply is useful for cert-manager for 2 reasons:
cert-manager controllers, namely the certificates controllers, have multiple controllers that are writing to the same resource type. This leads to cases where UPDATE operations result in resource version conflicts. This in turn, results in error logs that regularly confuse users and miss-attribute the problem they are having, as well as needless re-queuing churn through the controller, slowing cert-manager down.
Without Server-Side Apply, it is difficult for cert-manager to reason about some fields that should be deleted on an UPDATE operation during reconciliation. One such example are Certificate SecretTemplate's Annotations and Labels. If a key was removed from either the Annotation or Label field on the SecretTemplate, the next reconciliation would not know which keys should be preserved or removed (as they may have been annotated or labelled from a third party). This problem occurs in several places where copying or transforming state from one resource to another happens.
Managed fields define ownership of applied fields. cert-manager is therefore able to observe a discrepancy occurring from, for example, a previous SecretTemplate state and performing another apply (omitting those keys), resulting in those fields being removed on the Secret. Without managed fields, cert-manager would have to create a third state store in either annotations or status fields on that resource.
The Field Manager is a string denoting the manager who owns a field or set of
fields on a resource. This string is set by the client at API call time and is
derived by either the client setting the fieldManager URL query explicitly, or
calculated from the client's user agent (characters preceding the first "/",
quote unprintable characters and then trim what's beyond the 128
limit).
cert-manager will use a consistent naming scheme for both the user agent prefix and field manager across all components. Each component will have a human readable name that describes that component, and is used for both its user agent and field manager. Using the same name enables better auditing trails that helps debugging and improves telemetry.
This is the list of known field managers. This might not be an exhaustive list as they will be discovered during the changes.
cert-manager # base field manager of cert-manager controller, used when no distinct component
cert-manager-leader-election
cert-manager-acme-[orders,challenges]
cert-manager-certificates-[issuing,trigger,keymanager,readiness]
cert-manager-certificaterequests-[acme,approver,ca,selfsigned,vault,venafi]
cert-manager-clusterissuers-[acme,ca,selfsigned,vault,venafi]
cert-manager-issuers-[acme,ca,selfsigned,vault,venafi]
cert-manager-cainjector # base field manager of cert-manager cainjector
cert-manager-webhook # base field manager of cert-manager webhook
cert-manager-cmctl
The field manager is derived from the component in cert-manager, prefixed by the
string cert-manager:
cert-manager-<cert-manager component>
e.g.
cert-manager-certificates-issuing
The user agent is inspired by the client-go default user agent, and is defined with the following:
<component field manager>/<cert-manager version> (<Operating System>/<Architecture>) cert-manager/<git commit>
e.g:
cert-manager-certificates-issuing/v1.7.0 (Linux/amd64) cert-manager/0b686b8f38c8c7442744c9224d18e780ee7f244a
All CREATE and UPDATE operations in cert-manager controllers are to be replaced with Apply. For this change, controllers will need to be modified so that for these calls, only the fields in which they manage should be included in the PATCH API call. This means each controller will need to define exactly which fields they are concerned with. A package will be created which all API calls will use. This package will implement the logic for checking the feature gate for consistency across the project whilst minimising disruption to existing code paths.
Some fields, such as the Certificate Issuing Condition are managed by more than
one controller (issuing and trigger Certificate controllers, and cmctl), and as
such, will need to make use of the force parameter in their API calls. This
option tells the API server to revoke management of that field from the previous
owner, overwrite the field, and change ownership to the new client. Since some
fields, such as the Issuing Condition, may have an undefined number of potential
managers (both internal and external to the cert-manager controller), using the
same manager for things is not a possibility. You can read more about the
force parameter on the Kubernetes documentation on
Server-Side Apply,
and in particular the
Conflicts
section.
Use of the force parameter will not overwrite other controllers progress
since, 1 controllers only update the fields that they manage, 2 controllers who
manage the same fields should never be writing to the same field at the same
time (as they should be gated by those or other fields presence and values, such
as the Issuing condition for the issuing and trigger controllers).
There is likely no Apply call that cert-manager does not set the force
parameter to true since it, never wants to give up ownership claim, and always
wants to overwrite
values.
See
Using Server-Side Apply in a controller.
The fake client-go client does not support the Apply PATCH call for mocking API calls and events. This means that significant controller unit-testing will either need to moved to testing against a real API server as integration tests, the controller test framework must add custom support for Apply, or a new testing framework should be developed. We can also PR this upstream but will take time to be released so a stop gap would always be needed.
Some API fields will need to have some metadata updated to function better under
Server-Side Apply. One such example is adding x-kubernetes-list-type=map and
x-kubernetes-list-map-keys=Type to the Certificates Status Condition
slice,
so that controllers are able to apply distinct condition types, without
conflicting with other controller conditions (i.e. the Ready and Issuing
conditions). Integration tests will be able to ensure cert-manager have set
these API type tags correctly.
Placing the Apply functionality behind a feature gate should be required. Placing this functionality behind a feature gate would allow the cert-manager authors gain confidence about its correctness, and ensure there are no regressions in the stability of controller reconciliation.
To reach graduation, the cert-manager authors should consider the Server-Side Apply implementation for cert-manager to be safe for end users. One strategy for gaining confidence that it is a safe change is to solicit feedback from end users who have the feature enabled. Graduation should ideally be done over no more than one or two releases, however the project shouldn't proceed with removing the gate until the authors are confident.
For an exiting cluster with cert-manager and resources installed, a user should be able to safely enable the feature or upgrade cert-manager with the feature enabled by default without any down time or loss of data.
Although testing is needed, it should be the case that an upgrade/feature enable that all controllers on the next reconcile that applies, will take ownership of that component's fields and set the correct values. Fields will not be deleted that should be however (i.e. Annotation or Label from a previous SecretTemplate) as the field manager has changed (though this is the same behaviour as today).