Sync Phases and Waves

Sync phases and hooks define when resources are applied such as before or after the main sync operation. This makes it possible to define jobs, or any other resource to run or be applied in any specific order.

Argo CD has the following hook types:

Adding the argocd.argoproj.io/hook annotation to a resource will assign it to a specific phase. During a Sync operation, Argo CD will apply the resource during the appropriate phase of the deployment. Hooks can be any type of Kubernetes resource kind, but tend to be Pod, Job or Argo Workflows. Multiple hooks can be specified as a comma separated list.

How phases work?

Argo CD will respect resources assigned to different phases, during a sync operation Argo CD will do the following:

1. Apply all the resources marked as PreSync hooks. If any of them fails the whole sync process will stop and will be marked as failed
2. Apply all the resources marked as Sync hooks. If any of them fails the whole sync process will be marked as failed. Hooks marked with SyncFail will also run
3. Apply all the resources marked as PostSync hooks. If any of them fails the whole sync process will be marked as failed.

Hooks marked with Skip will not be applied.

Here is a graphical overview of the sync process:

You can use this simple lifecycle method in various scenarios. For example you can run an essential check as a PreSync hook. If it fails then the whole sync operation will stop preventing the deployment from taking place. In a similar manner you can run smoke tests as PostSync hooks. If they succeed you know that your application has passed the validation. If they fail then the whole deployment will be marked as failed and Argo CD can then notify you in order to take further actions.

Hooks at the SyncFail phase can be used for cleanup actions and other housekeeping tasks. Note that if they themselves fail, Argo CD will not do anything special (other than marking the whole operation as failed).

Note that hooks do not run during a selective sync operation.

Hook lifecycle and cleanup

Argo CD offers several methods to clean up hooks and decide how much history will be kept for previous runs. In the most basic case you can use the argocd.argoproj.io/hook-delete-policy to decide when a hook will be deleted. This can take the following values:

Note that if no deletion policy is specified, Argo CD will automatically assume BeforeHookCreation rules.

PreDelete and PostDelete Hooks

PreDelete Hooks

PreDelete hooks execute before an Application and its resources are deleted. They run only during Application deletion (e.g., kubectl delete application or argocd app delete), not during normal sync operations, even when pruning is enabled.

Behavior:

1. When an Application is deleted, Argo CD checks for PreDelete hooks defined in the Application's manifests
2. If PreDelete hooks exist, they are created and executed before any Application resources are deleted
3. Argo CD waits for all PreDelete hooks to reach a Healthy state before proceeding with deletion
4. Once all PreDelete hooks complete successfully, they are cleaned up and the Application resources are deleted

Failure Handling:

If a PreDelete hook fails (e.g., a Job fails or a Pod crashes), the Application deletion is blocked:

• The Application will remain in a deleting state with a DeletionError condition
• Application resources will not be deleted until the hook succeeds
• The user can fix the failing hook by updating its manifest in the git repository
• After fixing the hook, Argo CD will automatically retry the deletion on the next reconciliation
• Alternatively, the user can manually delete the failing hook resource to allow deletion to proceed

PostDelete Hooks

PostDelete hooks execute after all Application resources have been deleted. They are useful for cleanup operations, notifications, or removing external resources.

Behavior:

1. Application resources are deleted first
2. Once all Application resources are fully removed, PostDelete hooks are created and executed
3. Argo CD waits for all PostDelete hooks to reach a Healthy state
4. Once all PostDelete hooks complete successfully, they are cleaned up and the Application is fully removed

Failure Handling:

If a PostDelete hook fails:

• The Application is already deleted from the cluster, but the Application CR remains with a DeletionError condition
• The user can fix the failing hook by updating its manifest in the git repository
• After fixing the hook, Argo CD will automatically retry on the next reconciliation
• Alternatively, the user can manually delete the failing hook resource to complete the Application deletion

How sync waves work?

Argo CD also offers an alternative method of changing the sync order of resources. These are sync waves. They are defined by the argocd.argoproj.io/sync-wave annotation. The value is an integer that defines the ordering (ArgoCD will start deploying from the lowest number and finish with the highest number).

Hooks and resources are assigned to wave 0 by default. The wave can be negative, so you can create a wave that runs before all other resources.

When a sync operation takes place, Argo CD will:

1. Order all resources according to their wave (lowest to highest)
2. Apply the resources according to the resulting sequence

There is currently a delay between each sync wave in order to give other controllers a chance to react to the spec change that was just applied. This also prevents Argo CD from assessing resource health too quickly (against the stale object), causing hooks to fire prematurely. The current delay between each sync wave is 2 seconds and can be configured via the environment variable ARGOCD_SYNC_WAVE_DELAY.

Combining Sync waves and hooks

While you can use sync waves on their own, for maximum flexibility you can combine them with hooks. This way you can use sync phases for coarse grained ordering and sync waves for defining the exact order of a resource within an individual phase.

When Argo CD starts a sync, it orders the resources in the following precedence:

1. The phase
2. The wave they are in (lower values first)
3. By kind (e.g. namespaces first and then other Kubernetes resources, followed by custom resources)
4. By name

Once the order is defined:

1. First Argo CD determines the number of the first wave to apply. This is the first number where any resource is out-of-sync or unhealthy.
2. It applies resources in that wave.
3. It repeats this process until all phases and waves are in-sync and healthy.

Because an application can have resources that are unhealthy in the first wave, it may be that the app can never get to healthy.

How Do I Configure Phases?

Pre-sync and post-sync can only contain hooks. Apply the hook annotation:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync

Read more about hooks.

How Do I Configure Waves?

Specify the wave using the following annotation:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"

Hooks and resources are assigned to wave zero by default. The wave can be negative, so you can create a wave that runs before all other resources.

Examples

Send message to Slack when sync completes

The following example uses the Slack API to send a Slack message when sync completes:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: app-slack-notification-
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: slack-notification
          image: curlimages/curl
          command:
            - curl
            - '-X'
            - POST
            - '--data-urlencode'
            - >-
              payload={"channel": "#somechannel", "username": "hello", "text":
              "App Sync succeeded", "icon_emoji": ":ghost:"}
            - 'https://hooks.slack.com/services/...'
      restartPolicy: Never
  backoffLimit: 2

Initialize or migrate a database

The following example runs a db initialization/migration command before the main sync operation (also in wave -1):

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: '-1'
spec:
  ttlSecondsAfterFinished: 360
  template:
    spec:
      containers:
        - name: postgresql-client
          image: 'my-postgres-data:11.5'
          imagePullPolicy: Always
          env:
            - name: PGPASSWORD
              value: admin
            - name: POSTGRES_HOST
              value: my_postgresql_db
          command:
            - psql
            - '-h=my_postgresql_db'
            - '-U postgres'
            - '-f preload.sql'
      restartPolicy: Never
  backoffLimit: 1

Work around ArgoCD sync failure

Upgrading ingress-nginx controller (managed by helm) with ArgoCD 2.x sometimes fails to work resulting in:

To work around this, a helm user can add:

ingress-nginx:
  controller:
    admissionWebhooks:
      annotations:
        argocd.argoproj.io/hook: Skip

Which results in a successful sync.