ContextQMD
Back to Argo Cd

v3.2 to 3.3

docs/operator-manual/upgrading/3.2-3.3.md

3.4.09.6 KB
Original Source

v3.2 to 3.3

Breaking Changes

ApplicationSet CRD exceeds the size limit for client-side apply

In this version, the ApplicationSet CRD exceeds the size limit for client-side apply, meaning that the previous upgrade methods will result in The CustomResourceDefinition "applicationsets.argoproj.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes

In order to avoid this issue, it is now required to upgrade Argo CD using Server Side Apply (SSA) along with resolve conflicts. SSA does not store the ApplicationSet fields in last-applied annotation, thus not impacted by the annotation size limitation. The --force-conflicts flag allows the apply operation to take ownership of fields that may have been previously managed by other tools (such as Helm or a previous kubectl apply). This is necessary for upgrades. Note that any custom modifications you've made to fields that are defined in the Argo CD manifests (like affinity, env, or probes) will be overwritten. However, fields not specified in the manifests (like resources limits/requests or tolerations) will be preserved.

Upgrading Argo CD which is managing itself

When Argo CD is upgraded using an Argo CD Application, it is required to enable Server-Side Apply in the Application spec:

yaml
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

With this configuration, Argo CD will add the resolve conflicts option automatically.

Upgrading Argo CD manually using Kustomize or plain manifests

When Argo CD is upgraded manually using plain manifests or Kustomize overlays, it is required to upgrade it with --server-side --force-conflicts. For example, kubectl apply -n argocd --server-side --force-conflicts -f manifests/install.yaml

Upgrading Argo CD manually using Helm

Users upgrading Argo CD manually using helm upgrade are not impacted by this change, since Helm does not use client-side apply and does not result in creation of the last-applied annotation.

Users who previously upgraded to 3.3.0 or 3.3.1

In some cases, after upgrading to one of those versions and applying Server-Side Apply, the following error occured:
one or more synchronization tasks completed unsuccessfully, reason: Failed to perform client-side apply migration: failed to perform client-side apply migration on manager kubectl-client-side-apply: error when patching "/dev/shm/2047509016": CustomResourceDefinition.apiextensions.k8s.io "applicationsets.argoproj.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes.

Users that have configured the sync option ClientSideApplyMigration=false as a temporary remediation for the above error, should remove it after upgrading to 3.3.2. Disabling ClientSideApplyMigration imposes a risk to encounter conflicts between K8s field managers in the future.

Source Hydrator Now Tracks Hydration State Using Git Notes

Previously, Argo CD's Source Hydrator pushed a new hydrated commit for every DRY (source) commit, regardless of whether any manifest files (manifest.yaml) actually changed. This was necessary for the hydrator to track which DRY commit had last been hydrated: it embedded this information in the hydrator.metadata file's drySha field in each hydrated commit.

Starting with v3.3, the Source Hydrator uses git notes to record the state of the most recently hydrated DRY commit instead of creating a hydrated commit on every DRY commit. The git note is stored in a dedicated namespace reserved for the Source Hydrator, making the state both reliable and isolated from other repository operations.

This design change improves repository cleanliness, reduces unnecessary commit noise, and lowers risk from excessive branch divergence due to frequent automated commits.

How It Works

  • On each hydration run:
    1. The hydrator first pulls the existing git note in its namespace to check the last hydrated DRY commit SHA.
    2. If the note SHA matches the latest DRY SHA, the hydrator logs a debug message and skips hydration.
    3. If files such as manifest.yaml have not changed (even if DRY SHA is new), the hydrator skips committing manifests and only updates the git note to reflect the latest hydrated DRY SHA.
    4. If manifest files have changed, the hydrator commits the updated manifests and also updates the git note.

Migration Impact

  • Behavioral:
    • Users will no longer see a hydrated commit for every DRY commit. Only actual manifest changes will create a new commit; otherwise, the hydration state is tracked in the git note namespace.
  • Operational:
    • Applications and tools that depended on a hydrated commit for every DRY commit should now use the git note in the Source Hydrator namespace to determine the hydration state.
  • No action needed for most users, but if you have automations dependent on hydrated commits as signals, please update them to consult the new git note.

Rationale and Benefits

  • Reduces repository clutter: fewer unnecessary commits.
  • Improves performance for teams with high-frequency DRY changes and rare manifest changes.
  • Decreases risk of merge conflicts and branch bloat in automation scenarios.

Removal of Application Path Cleaning During Hydration

  • Behavioral Change:
    In Argo CD versions prior to v3.3, the Source Hydrator would automatically clean (delete all files in) the application's configured path before writing out new manifest files during each hydration run. Starting with v3.3, this cleaning logic has been removed. The hydrator will now only overwrite or create files that correspond to the current manifest output; any additional files or obsolete data in the application path will remain untouched unless explicitly overwritten.
  • Operational Impact:
    If your repository or automation relied on automatic cleanup of stale or obsolete files in application paths, you must now handle this cleanup manually. This change avoids the risk of accidental deletion of unrelated files (for instance, if application paths overlap with directories containing other assets or when applications are restructured).
  • Recommended Action:
    Review your automation workflows and repository maintenance scripts to ensure that old or unwanted files in application paths are cleaned up if necessary. Consider implementing a periodic manual or automated cleanup procedure if your use case requires it.
  • For more details on current behavior, see the Source Hydrator user guide.

Anonymous call to Settings API returns fewer fields

The Settings API now returns less information when accessed anonymously. It no longer returns the resourceOverrides field which is considered sensitive information.

New ENV Variable to control K8s Server Side Timeout of API Requests

The new environment variable ARGOCD_K8S_SERVER_SIDE_TIMEOUT can be used to control the K8s server side timeout of API requests. In 3.2 and before this change, the K8s server side timeout was controlled by ARGOCD_K8S_TCP_TIMEOUT which is also used to control the TCP timeout when communicating with the K8s API server. From now onwards, the Kubernetes server-side timeout is controlled by a separate environment variable.

Deprecated --self-heal-backoff-cooldown-seconds flag

The --self-heal-backoff-cooldown-seconds flag of the argocd-application-controller has been deprecated and will be removed in a future release.

Helm Upgraded to 3.19.2

Argo CD v3.3 upgrades the bundled Helm version to 3.19.2. There are no breaking changes in Helm 3.19.2 according to the release notes.

Helm 2.x is no longer supported as we are dropping the --client flag in calls to helm version.

Kustomize Upgraded to 5.8.0

Argo CD v3.3 upgrades the bundled Kustomize version from v5.7.0 to v5.8.0. According to the 5.7.1 and 5.8.0 release notes, there are no breaking changes.

However, note that Kustomize 5.7.1 introduces code to replace the shlex library used for parsing arguments in exec plugins. If any existing manifests become corrupted, please follow the instructions in the 5.7.1 release notes.

Kustomize 5.8.0 also resolves an issue where namespaces were not properly propagated to Helm charts. If you rely on Helm charts within kustomization files, please review the details in the associated fix: kubernetes-sigs/kustomize#5940.

Added Healthchecks