Agent Alert Notifications

Netdata's Agent can send alert notifications directly from each node. It supports a wide range of services, multiple recipients, and role-based routing.

How It Works

The Agent uses a notification script defined in netdata.conf under the [health] section:

script to execute on alarm = /usr/libexec/netdata/plugins.d/alarm-notify.sh

The default script is alarm-notify.sh.

This script handles:

• Multiple recipients
• Multiple notification methods
• Role-based routing (e.g., sysadmin, webmaster, dba)

Role-Based Routing Visualization

flowchart TD
    Alert("High CPU Usage Alert") --> Check("Severity Level")
    
    Check -->|"WARNING"| WarningRouting("Role: SysAdmin")
    Check -->|"CRITICAL"| CriticalRouting("Multiple Roles")
    
    WarningRouting --> SlackChannel("Slack")
    WarningRouting --> EmailOps("Email")
    
    CriticalRouting --> PagerDuty("PagerDuty")
    CriticalRouting --> EmailManagers("Email")
    CriticalRouting --> SlackUrgent("Slack")
    CriticalRouting --> SMS("SMS")
    
    %% Style definitions
    classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
    classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
    classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
    classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px

    %% Apply styles
    class Alert alert
    class Check database
    class WarningRouting,CriticalRouting complete
    class SlackChannel,EmailOps,PagerDuty,EmailManagers,SlackUrgent,SMS neutral

Health Management API Workflow

flowchart TD
    Start("Normal Operation") --> Maintenance("Maintenance Window?")
    
    Maintenance -->|"No"| NormalOps("Continue Normal Alerting")
    Maintenance -->|"Yes"| ApiAction("Choose Action")
    
    ApiAction --> SilenceAll("SILENCE ALL")
    ApiAction --> DisableAll("DISABLE ALL")
    ApiAction --> SilenceSelect("SILENCE Specific")
    
    SilenceAll --> Reset("RESET when done")
    DisableAll --> Reset
    SilenceSelect --> Reset
    
    Reset --> Restored("Normal Operations Restored")
    
    %% Style definitions
    classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
    classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
    classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
    classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px

    %% Apply styles
    class Start,Restored complete
    class Maintenance database
    class ApiAction alert
    class NormalOps,SilenceAll,DisableAll,SilenceSelect,Reset neutral

Quick Setup

:::tip

Use the edit-config script to safely edit configuration files. It automatically creates the necessary files in the right place and opens them in your editor. Learn how to use edit-config

:::

1. Open the Agent's health notification config:

   bashCopy

   sudo ./edit-config health_alarm_notify.conf
   

2. Set up the required API keys or credentials for the service you want to use.

3. Define recipients per role (see below).

4. Changes take effect immediately. The next alarm will use the updated configuration, so no restart or reload is required.

Example: Alert with Role-Based Routing

Here's an example alert assigned to the sysadmin role from the ram.conf file:

alarm: ram_in_use
   on: system.ram
class: Utilization
 type: System
component: Memory
     os: linux
  hosts: *
   calc: $used * 100 / ($used + $cached + $free + $buffers)
  units: %
  every: 10s
   warn: $this > (($status >= $WARNING)  ? (80) : (90))
   crit: $this > (($status == $CRITICAL) ? (90) : (98))
  delay: down 15m multiplier 1.5 max 1h
   info: system memory utilization
     to: sysadmin

Then, in health_alarm_notify.conf, you assign recipients per notification method:

role_recipients_email[sysadmin]="[email protected] [email protected]"
role_recipients_slack[sysadmin]="#alerts #infra"

Advanced Role-Based Routing Examples

# Backend team receives database and application server alerts
role_recipients_slack[backend]="#backend-team"
role_recipients_pagerduty[backend]="PDK3Y5EXAMPLE"

# Frontend team receives web server and CDN alerts
role_recipients_slack[frontend]="#frontend-team"
role_recipients_opsgenie[frontend]="key1example"

# Security team receives all security-related alerts
role_recipients_email[security]="[email protected]"
role_recipients_slack[security]="#security-alerts"

# SRE team receives critical infrastructure alerts 24/7
role_recipients_slack[sre]="#sre-alerts"
role_recipients_pagerduty[sre]="PDK3Y5SREXAMPLE"
role_recipients_telegram[sre]="123456789"

You can use external scripts to dynamically change recipients based on work hours, on-call schedules, etc.:

# Use a script to determine the current on-call engineer
ONCALL_EMAIL=$(get_oncall_email.sh)
role_recipients_email[oncall]="${ONCALL_EMAIL}"
role_recipients_sms[oncall]="${ONCALL_PHONE}"

# Standard business hours team gets non-critical alerts during work hours
role_recipients_slack[business_hours]="#daytime-monitoring"

Health Management API

Netdata provides a powerful Health Management API that lets you control alert behavior during maintenance windows, testing, or other planned activities.

API Authorization

The API is protected by an authorization token stored in /var/lib/netdata/netdata.api.key:

# Get your token
TOKEN=$(cat /var/lib/netdata/netdata.api.key)

# Use the token in API calls
curl "http://localhost:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: ${TOKEN}"

Common API Commands

Completely stops evaluation of health checks during maintenance:

curl "http://localhost:19999/api/v1/manage/health?cmd=DISABLE ALL" -H "X-Auth-Token: ${TOKEN}"

Continues to evaluate health checks but prevents notifications:

curl "http://localhost:19999/api/v1/manage/health?cmd=SILENCE ALL" -H "X-Auth-Token: ${TOKEN}"

Target only certain alerts by name, chart, context, host, or family:

# Silence all disk space alerts
curl "http://localhost:19999/api/v1/manage/health?cmd=SILENCE&context=disk_space" -H "X-Auth-Token: ${TOKEN}"

# Disable CPU alerts for specific hosts
curl "http://localhost:19999/api/v1/manage/health?cmd=DISABLE&context=cpu&hosts=prod-db-*" -H "X-Auth-Token: ${TOKEN}"

Check what's currently silenced or disabled:

curl "http://localhost:19999/api/v1/manage/health?cmd=LIST" -H "X-Auth-Token: ${TOKEN}"

Re-enable all health checks and notifications:

curl "http://localhost:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: ${TOKEN}"

Configuration Options

Define who receives alerts and how:

role_recipients_email[sysadmin]="[email protected]"
role_recipients_telegram[webmaster]="123456789"
role_recipients_slack[dba]="#database-alerts"

Use spaces to separate multiple recipients.

To disable a notification method for a role, use:

role_recipients_email[sysadmin]="disabled"

If left empty, the default recipient for that method is used.

You can limit certain recipients to only receive critical alerts:

role_recipients_email[sysadmin]="[email protected] [email protected]|critical"

This setup:

• Sends all alerts to [email protected]
• Sends notifications to [email protected] only once the alarm reaches CRITICAL, then continues sending status changes (including WARNING and CLEAR) until the alarm is cleared.

Works for all supported methods: email, Slack, Telegram, Twilio, Discord, etc.

When an alert returns to normal, Netdata sends a CLEAR (recovered) notification. You can control when and whether these are sent.

Default behavior: Netdata suppresses CLEAR notifications when the alert was never in a WARNING or CRITICAL state. If old_status was not WARNING or CRITICAL and the alert transitions to CLEAR, no notification is sent. This prevents noise from alerts that flap without ever reaching a problem state.

Enable CLEAR for all transitions: If your downstream system handles deduplication, set clear_alarm_always in health_alarm_notify.conf to override the default suppression and send a CLEAR notification regardless of the previous status:

clear_alarm_always='YES'

Filter by CRITICAL history with the |critical modifier: As described in Alert Severity Filtering above, |critical forwards notifications only for alerts that have reached CRITICAL status. This affects both WARNING and CLEAR:

• WARNING notifications are suppressed unless the alarm has previously reached CRITICAL.
• CLEAR notifications are only sent when the alert previously passed through CRITICAL. If the alert only went through WARNING → CLEAR, the CLEAR is not forwarded.

role_recipients_email[sysadmin]="[email protected]|critical"

Suppress all CLEAR notifications: Use the |noclear modifier to completely block CLEAR notifications for a recipient while still receiving WARNING and CRITICAL alerts:

role_recipients_email[sysadmin]="[email protected]|noclear"

You can combine modifiers. This example notifies only for alarms that have reached CRITICAL (WARNING is suppressed until then), and excludes CLEAR notifications entirely:

role_recipients_email[sysadmin]="[email protected]|critical|noclear"

To send notifications via a proxy, set these environment variables:

export http_proxy="http://10.0.0.1:3128/"
export https_proxy="http://10.0.0.1:3128/"

By default, Netdata includes public image URLs in notifications (hosted by the global Registry).

To use custom image paths:

images_base_url="http://my.public.netdata.server:19999"

Change the timestamp format in notifications:

date_format="+%F %T%:z"   # Example: RFC 3339

Common formats:

See man date for more formatting options.

By default, Netdata uses the short hostname in notifications.

To use the fully qualified domain name (FQDN), set:

use_fqdn=YES

If you've set a custom hostname in netdata.conf, that value takes priority.

Testing Your Notification Setup

You can test alert notifications manually.

# Switch to the Netdata user
sudo su -s /bin/bash netdata

# Enable debugging
export NETDATA_ALARM_NOTIFY_DEBUG=1

# Test default role (sysadmin)
./plugins.d/alarm-notify.sh test

# Test specific role
./plugins.d/alarm-notify.sh test "webmaster"

:::important

If you're running your own Netdata Registry, set:

export NETDATA_REGISTRY_URL="https://your.registry.url"

before testing.

:::

Debugging with Trace

To see the full execution output:

bash -x ./plugins.d/alarm-notify.sh test

Then look for the internal calls and re-run the one you want to trace in more detail.

Troubleshooting Alert Notifications

Here are solutions for common alert notification issues:

Email Notifications Not Working

1. Verify your email configuration:

   bashCopy

   grep -E "SEND_EMAIL|DEFAULT_RECIPIENT_EMAIL" /etc/netdata/health_alarm_notify.conf
   

2. Check if the system can send mail:

   bashCopy

   echo "Test" | mail -s "Test Email" [email protected]
   

3. Look for errors in the Netdata log:

   bashCopy

   tail -f /var/log/netdata/error.log | grep "alarm notify"
   

4. Test with debugging enabled:

   bashCopy

   sudo su -s /bin/bash netdata
   export NETDATA_ALARM_NOTIFY_DEBUG=1
   ./plugins.d/alarm-notify.sh test
   

Slack Notifications Failing

1. Verify your webhook URL is correct:

   bashCopy

   grep -E "SLACK_WEBHOOK_URL" /etc/netdata/health_alarm_notify.conf
   

2. Check for network connectivity to Slack:

   bashCopy

   curl -X POST -H "Content-type: application/json" --data '{"text":"Test"}' YOUR_WEBHOOK_URL
   

3. Confirm channel names start with # in your configuration.

PagerDuty Integration Issues

1. Verify your service key:

   bashCopy

   grep -E "DEFAULT_RECIPIENT_PD|SEND_PD" /etc/netdata/health_alarm_notify.conf
   

2. Test the PagerDuty API directly:

   bashCopy

   curl -H "Content-Type: application/json" -X POST -d '{"service_key":"YOUR_SERVICE_KEY","event_type":"trigger","description":"Test"}' https://events.pagerduty.com/generic/2010-04-15/create_event.json
   

Notification Delays

If notifications seem delayed:

• Check the delay parameter in your alarm definitions in health.d/*.conf — this is the only per-alert delay setting for notifications; there is no global delay setting in health_alarm_notify.conf.

Related Docs

• How to configure alerts
• Notification methods list
• Netdata configuration basics