Custom

Netdata Agent's alert notification feature allows you to send custom notifications to any endpoint you choose.

Setup

Prerequisites

Access to the terminal where Netdata Agent is running

Configuration

Options

The following options can be defined for this notification

<details open><summary>Config Options</summary>

Option	Description	Default	Required
SEND_CUSTOM	Set `SEND_CUSTOM` to YES	YES	yes
DEFAULT_RECIPIENT_CUSTOM	This value is dependent on how you handle the `${to}` variable inside the `custom_sender()` function.		yes
custom_sender()	You can look at the other senders in `/usr/libexec/netdata/plugins.d/alarm-notify.sh` for examples of how to modify the function in this configuration file.		no

DEFAULT_RECIPIENT_CUSTOM

All roles will default to this variable if left unconfigured. You can edit DEFAULT_RECIPIENT_CUSTOM with the variable you want, in the following entries at the bottom of the same file:

role_recipients_custom[sysadmin]="systems"
role_recipients_custom[domainadmin]="domains"
role_recipients_custom[dba]="databases systems"
role_recipients_custom[webmaster]="marketing development"
role_recipients_custom[proxyadmin]="proxy-admin"
role_recipients_custom[sitemgr]="sites"

custom_sender()

The following is a sample custom_sender() function in health_alarm_notify.conf, to send an SMS via an imaginary HTTPS endpoint to the SMS gateway:

custom_sender() {
    # example human readable SMS
    local msg="${host} ${status_message}: ${alarm} ${raised_for}"

    # limit it to 160 characters and encode it for use in a URL
    urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}"

    # a space separated list of the recipients to send alarms to
    to="${1}"

    for phone in ${to}; do
      httpcode=$(docurl -X POST \
            --data-urlencode "From=XXX" \
            --data-urlencode "To=${phone}" \
            --data-urlencode "Body=${msg}" \
            -u "${accountsid}:${accounttoken}" \
        https://domain.website.com/)

      if [ "${httpcode}" = "200" ]; then
        info "sent custom notification ${msg} to ${phone}"
        sent=$((sent + 1))
      else
        error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."
      fi
    done
}

For the complete list of supported notification variables, see the Alert Notification Variables section in the Health Reference.

</details>

via File

The configuration file name for this integration is health_alarm_notify.conf.

You can edit the configuration file using the edit-config script from the Netdata config directory.

bash

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config health_alarm_notify.conf

Examples

Basic Configuration

yaml

#------------------------------------------------------------------------------
# custom notifications

SEND_CUSTOM="YES"
DEFAULT_RECIPIENT_CUSTOM=""

# The custom_sender() is a custom function to do whatever you need to do
custom_sender() {
    # example human readable SMS
    local msg="${host} ${status_message}: ${alarm} ${raised_for}"

    # limit it to 160 characters and encode it for use in a URL
    urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}"

    # a space separated list of the recipients to send alarms to
    to="${1}"

    for phone in ${to}; do
      httpcode=$(docurl -X POST \
            --data-urlencode "From=XXX" \
            --data-urlencode "To=${phone}" \
            --data-urlencode "Body=${msg}" \
            -u "${accountsid}:${accounttoken}" \
        https://domain.website.com/)

      if [ "${httpcode}" = "200" ]; then
        info "sent custom notification ${msg} to ${phone}"
        sent=$((sent + 1))
      else
        error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."
      fi
    done
}

Troubleshooting

Test Notification

You can run the following command by hand, to test alerts configuration:

bash

# become user netdata
sudo su -s /bin/bash netdata

# enable debugging info on the console
export NETDATA_ALARM_NOTIFY_DEBUG=1

# send test alarms to sysadmin
/usr/libexec/netdata/plugins.d/alarm-notify.sh test

# send test alarms to any role
/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"

Note that this will test all alert mechanisms for the selected role.