Server-Side Apply support for ArgoCD

Server-Side Apply (SSA) allows calculating the final patch to update resources in Kubernetes in the server instead of the client. This proposal describes how ArgoCD can leverage SSA during syncs.

• Open Questions
  • [Q-1] How to handle conflicts?
  • [Q-2] Should we support multiple managers?
• Summary
• Motivation
  • Better interoperability with Admission Controllers
  • Better resource conflict management
  • Better CRD support
• Goals
  • [G-1] Fine grained configuration
  • [G-2] Strategic merge patch while diffing
  • [G-3] Admission Controllers compatibility
  • [G-4] Conflict management
  • [G-5] Register a proper manager
• Non-Goals
• Proposal
  • Non-Functional Requirements
  • Use cases
    • [UC-1]: enable SSA at the controller level
    • [UC-2]: enable SSA at the Application level
    • [UC-3]: enable SSA at the resource level
  • Security Considerations
  • Risks and Mitigations
    • [R-1] Supported K8s version check
    • [R-2] Alternating Server-Side Client-Side syncs
  • Upgrade / Downgrade
• Drawbacks

---

Open Questions

[Q-1] How to handle conflicts?

When SSA is enabled, the server may return field conflicts with other managers. What ArgoCD controller should do in case of conflict? Just force the sync and log warnings (like some other controllers do?)

Conclusion

The first version should use the force flag and override even if there are conflicts. We could improve and add other options once there is a use case.

[Q-2] Should we support multiple managers?

Should Server-Side Apply support in ArgoCD be implemented allowing multiple managers for the same controller? (more details)

Summary

ArgoCD can benefit from Server-Side Apply during syncs. A few improvements to consider:

• More reliable dry-runs (as admission controller is executed) (ISSUE-804)
• Syncs always run mutating webhooks (even without diff)
• Fix big CRD sync issues
• Better interoperability with different controllers

Kubernetes SSA Proposal (KEP-555) has more details about how it works.

Motivation

ArgoCD uses kubectl library while syncing resources in the cluster. Kubectl uses by default a 3-way-merge logic between the live state (in k8s), desired state (in git) and the previous state (last-applied-configuration annotation) to calculate diffs and patch resources in the cluster. This logic is executed in the client (ArgoCD) and once the patch is calculated it is then sent to the server.

This strategy works well in the majority of the use cases. However, there are some scenarios where calculating patches in the client side can cause problems.

Some of the known problems about using client-side approach:

Better interoperability with Admission Controllers

More and more users are adopting and configuring Admission Controllers in Kubernetes with different flavors of Validating Webhooks and Mutating Webhooks. Admission Controllers will only execute in server-side. In cases where users rely on dry-run executions to decide if they should proceed with a deployment, having the patch calculated at the client side might provide undesired results. Having SSA enabled syncs also guaranties that Admission Controllers are always executed, even when there is no diff in the resource.

Better resource conflict management

Server-Side Apply will better handle and identify conflicts during syncs by analyzing the managedFields metadata available in all Kubernetes resources (since kubernetes 1.18).

Better CRD support

By not having to rely on the last-applied-configuration annotation, SSA would help with failing syncs caused by exceeded annotation size limit. This is a common issue when syncing CRDs with large schemas.

Goals

All following goals should be achieve in order to conclude this proposal:

[G-1] Fine grained configuration

• Provide the ability for users to define if they want to use SSA during syncs
• Users should be able to enable SSA at the controller level (via binary flag) (see UC-1)
• Users should be able to enable SSA for a given Application (via syncOptions) (see UC-2)
• Users should be able to enable SSA at resource level (via annotation) (see UC-3
• Relates to ISSUE-2267

[G-2] Strategic merge patch while diffing

• Diffing needs to support strategic merge patch (see ISSUE-2268)
• Make sure Services can be patched correctly (more details)

[G-3] Admission Controllers compatibility

• Allow Admission Controllers to execute even when there is no diff for a particular resource. (Needs investigation) (more details)

[G-4] Conflict management

• ArgoCD should respect field ownership and provide a configuration to allow users to define the behavior in case of conflicts (see Q-1 outcome)

[G-5] Register a proper manager

• ArgoCD must register itself with a pre-defined manager (suggestion: argocd-controller). It shouldn't rely on the default value defined in the kubectl code. (more details)

Non-Goals

TBD

Proposal

Change ArgoCD controller to accept new parameter to enable Server-Side Apply during syncs. Changes are necessary in ArgoCD as well as in gitops-engine library.

Use cases

The following use cases should be implemented:

[UC-1]: As a user, I would like enable SSA at the controller level so all Application are applied server-side

Implement a binary flag to configure ArgoCD to run all syncs using SSA. (suggestion: --server-side-apply=true). Default value should be false.

[UC-2]: As a user, I would like enable SSA at the Application level so all resources are applied server-side

Implement a new syncOption to allow users to enable SSA at the application level (Suggestion ServerSideApply=true). UI needs to be updated to support this new sync option. If not informed, the controller should keep the current behaviour (client-side).

[UC-3]: As a user, I would like enable SSA at the resource level so only a single manifest is applied server-side

Leverage the existing argocd.argoproj.io/sync-options annotation allowing the ServerSideApply=true to be informed at the resource level. Must not impact other sync-options informed in the annotation (make sure this annotation supports providing multiple options).

Security Considerations

TBD

Risks and Mitigations

[R-1] Supported K8s version check

ArgoCD must check if the target Kubernetes cluster has full support for SSA. The feature turned GA in Kubernetes 1.22. Full support for managed fields was introduced as beta in Kubernetes 1.18. The implementation must check that the target kubernetes cluster is running at least version 1.18. If SSA is enabled and target cluster version < 1.18 ArgoCD should log warning and fallback to client sync.

[R-2] Alternating Server-Side Client-Side syncs

Kubernetes SSA proposal (KEP-555) mentions about alternating between server-side and client-side applies in the Upgrade/Downgrade Strategy section. It is stated that Kubernetes will verify the incoming apply request validating if the user-agent is kubectl to decide if the last-applied-configuration annotation should be updated. ArgoCD relies on this annotation and the implementation must make sure that this agent is correctly informed when changing to server-side apply and specifying a manager different than kubectl. This is mainly to make sure that G-5 isn't impacting the client-side/server-side compatibility.

Upgrade / Downgrade

No CRD update necessary as syncOption field in Application resource is non-typed (string array). Upgrade will only require ArgoCD controller update.

Drawbacks

Slight increase in ArgoCD code base complexity.