Metrics

Argo CD exposes different sets of Prometheus metrics per server.

Application Controller Metrics

Metrics about applications. Scraped at the argocd-metrics:8082/metrics endpoint.

Labels

Metrics Cache Expiration

If you use Argo CD with many application and project creation and deletion, the metrics page will keep in cache your application and project's history. If you are having issues because of a large number of metrics cardinality due to deleted resources, you can schedule a metrics reset to clean the history with an application controller flag. Example: --metrics-cache-expiration="24h0m0s".

Exposing Application labels as Prometheus metrics

There are use-cases where Argo CD Applications contain labels that are desired to be exposed as Prometheus metrics. Some examples are:

• Having the team name as a label to allow routing alerts to specific receivers
• Creating dashboards broken down by business units

As the Application labels are specific to each company, this feature is disabled by default. To enable it, add the --metrics-application-labels flag to the Argo CD application controller.

The example below will expose the Argo CD Application labels team-name and business-unit to Prometheus:

containers:
- command:
  - argocd-application-controller
  - --metrics-application-labels
  - team-name
  - --metrics-application-labels
  - business-unit

In this case, the metric would look like:

# TYPE argocd_app_labels gauge
argocd_app_labels{label_business_unit="bu-id-1",label_team_name="my-team",name="my-app-1",namespace="argocd",project="important-project"} 1
argocd_app_labels{label_business_unit="bu-id-1",label_team_name="my-team",name="my-app-2",namespace="argocd",project="important-project"} 1
argocd_app_labels{label_business_unit="bu-id-2",label_team_name="another-team",name="my-app-3",namespace="argocd",project="important-project"} 1

Exposing Application conditions as Prometheus metrics

There are use-cases where Argo CD Applications contain conditions that are desired to be exposed as Prometheus metrics. Some examples are:

• Hunting orphaned resources across all deployed applications
• Knowing which resources are excluded from ArgoCD

As the Application conditions are specific to each company, this feature is disabled by default. To enable it, add the --metrics-application-conditions flag to the Argo CD application controller.

The example below will expose the Argo CD Application condition OrphanedResourceWarning and ExcludedResourceWarning to Prometheus:

containers:
  - command:
      - argocd-application-controller
      - --metrics-application-conditions
      - OrphanedResourceWarning
      - --metrics-application-conditions
      - ExcludedResourceWarning

Application Set Controller metrics

The Application Set controller exposes the following metrics for application sets.

Similar to the same metric in application controller (argocd_app_labels) the metric argocd_appset_labels is disabled by default. You can enable it by providing the –metrics-applicationset-labels argument to the applicationset controller.

Once enabled it works exactly the same as application controller metrics (label_ appended to normalized label name). Available labels include Name, Namespace + all labels enabled by the command line options and their value (exactly like application controller metrics described in the previous section). |

Application Set GitHub API metrics

All the following argocd_github_api_* metrics can be enabled upon setting applicationsetcontroller.enable.github.api.metrics: true in argocd-cmd-params-cm ConfigMap. Note that they are disabled by default.

Labels

Exposing Cluster labels as Prometheus metrics

As the Cluster labels are specific to each company, this feature is disabled by default. To enable it, add the --metrics-cluster-labels flag to the Argo CD application controller.

The example below will expose the Argo CD Application labels team-name and environment to Prometheus:

containers:
- command:
  - argocd-application-controller
  - --metrics-cluster-labels
  - team-name
  - --metrics-cluster-labels
  - environment

In this case, the metric would look like:

# TYPE argocd_app_labels gauge
argocd_cluster_labels{label_environment="dev",label_team_name="team1",name="cluster1",server="server1"} 1
argocd_cluster_labels{label_environment="staging",label_team_name="team2",name="cluster2",server="server2"} 1
argocd_cluster_labels{label_environment="production",label_team_name="team3",name="cluster3",server="server3"} 1

API Server Metrics

Metrics about API Server API request and response activity (request totals, response codes, etc...). Scraped at the argocd-server-metrics:8083/metrics endpoint. For GRPC metrics to show up environment variable ARGOCD_ENABLE_GRPC_TIME_HISTOGRAM must be set to true.

Labels

Repo Server Metrics

Metrics about the Repo Server. The gRPC metrics are not exposed by default. Metrics can be enabled using ARGOCD_ENABLE_GRPC_TIME_HISTOGRAM=true environment variable.
Scraped at the argocd-repo-server:8084/metrics endpoint.

Commit Server Metrics

Metrics about the Commit Server. Scraped at the argocd-commit-server:8087/metrics endpoint.

Prometheus Operator

If using Prometheus Operator, the following ServiceMonitor example manifests can be used. Add a namespace where Argo CD is installed and change metadata.labels.release to the name of label selected by your Prometheus.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics
  endpoints:
    - port: metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-server-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-server-metrics
  endpoints:
    - port: metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-repo-server-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-repo-server
  endpoints:
    - port: metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-applicationset-controller-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-applicationset-controller
  endpoints:
    - port: metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-dex-server
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-dex-server
  endpoints:
    - port: metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-redis-haproxy-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-redis-ha-haproxy
  endpoints:
    - port: http-exporter-port

For notifications controller, you need to additionally add following:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-notifications-controller
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-notifications-controller-metrics
  endpoints:
    - port: metrics

For the optional Source Hydrator commit server component, you can add the following:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-commit-server-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-commit-server
  endpoints:
    - port: metrics

Dashboards

You can find an example Grafana dashboard or check the demo instance dashboard.