import FeatureAvailability from '@site/src/components/FeatureAvailability';

Monitoring Remote Executor

Overview

This guide covers all aspects of monitoring your Remote Executor deployment:

1. File-based health checks
2. UI-based health monitoring
3. Advanced Prometheus metrics configuration

Health Checks

File-Based Health Checks

The Remote Executor uses file-based health checks that can be monitored by your container platform:

• Liveness: /tmp/worker_liveness_heartbeat
• Readiness: /tmp/worker_readiness_heartbeat

These files are automatically managed by the Remote Executor and can be used by Kubernetes liveness/readiness probes or ECS health checks.

UI-Based Health Monitoring

Monitor Remote Executor health directly in the DataHub UI:

1. Navigate to Data Sources > Executors
2. View health information for each Pool:
   • Active Remote Executor instances
   • Last reported time for each executor
   • Status (Active/Stale)
   • Currently running Ingestion tasks and their details

Advanced: Prometheus Metrics

The Remote Executor exposes metrics on port 9087/tcp in Prometheus/OpenMetrics format. Metrics can be collected by Prometheus stack or compatible agents, such as DataDog.

Metric Categories

1. Ingestion Metrics

   • datahub_executor_worker_ingestion_requests - Total jobs received
   • datahub_executor_worker_ingestion_errors - Failed jobs (v0.3.9+)

2. Resource Metrics (v0.3.9+)

   • Memory: datahub_executor_memory_*
   • CPU: datahub_executor_cpu_*
   • Disk: datahub_executor_disk_*
   • Network: datahub_executor_net_*

Prometheus Configuration

Example ServiceMonitor resource to allow scraping Remote Executor metrics in Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
  name: datahub-remote-executor
spec:
  endpoints:
    - port: metrics
  selector:
    matchLabels:
      app.kubernetes.io/name: datahub-remote-executor

Discovering Available Metrics

1. View metrics endpoint directly:

   bashCopy

   curl http://your-executor:9087/metrics
   

2. Read annotations in Prometheus UI

3. Search datahub_executor_* in your monitoring system

:::note Platform-specific metrics (e.g., container restarts) should be monitored through native tooling (CloudWatch for ECS, Kubernetes metrics for K8s). :::