src/health/notifications/custom/README.md
Netdata Agent's alert notification feature allows you to send custom notifications to any endpoint you choose.
The following options can be defined for this notification
<details open><summary>Config Options</summary>| Option | Description | Default | Required |
|---|---|---|---|
| SEND_CUSTOM | Set SEND_CUSTOM to YES | YES | yes |
| DEFAULT_RECIPIENT_CUSTOM | This value is dependent on how you handle the ${to} variable inside the custom_sender() function. | yes | |
| custom_sender() | You can look at the other senders in /usr/libexec/netdata/plugins.d/alarm-notify.sh for examples of how to modify the function in this configuration file. | no |
<a id="option-default-recipient-custom"></a>
All roles will default to this variable if left unconfigured. You can edit DEFAULT_RECIPIENT_CUSTOM with the variable you want, in the following entries at the bottom of the same file:
role_recipients_custom[sysadmin]="systems"
role_recipients_custom[domainadmin]="domains"
role_recipients_custom[dba]="databases systems"
role_recipients_custom[webmaster]="marketing development"
role_recipients_custom[proxyadmin]="proxy-admin"
role_recipients_custom[sitemgr]="sites"
<a id="option-custom-sender"></a>
The following is a sample custom_sender() function in health_alarm_notify.conf, to send an SMS via an imaginary HTTPS endpoint to the SMS gateway:
custom_sender() {
# example human readable SMS
local msg="${host} ${status_message}: ${alarm} ${raised_for}"
# limit it to 160 characters and encode it for use in a URL
urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}"
# a space separated list of the recipients to send alarms to
to="${1}"
for phone in ${to}; do
httpcode=$(docurl -X POST \
--data-urlencode "From=XXX" \
--data-urlencode "To=${phone}" \
--data-urlencode "Body=${msg}" \
-u "${accountsid}:${accounttoken}" \
https://domain.website.com/)
if [ "${httpcode}" = "200" ]; then
info "sent custom notification ${msg} to ${phone}"
sent=$((sent + 1))
else
error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."
fi
done
}
The supported variables that you can use for the function's msg variable are:
| Variable name | Description |
|---|---|
${alarm} | Like "name = value units" |
${status_message} | Like "needs attention", "recovered", "is critical" |
${severity} | Like "Escalated to CRITICAL", "Recovered from WARNING" |
${raised_for} | Like "(alarm was raised for 10 minutes)" |
${host} | The host generated this event |
${url_host} | Same as ${host} but URL encoded |
${unique_id} | The unique id of this event |
${alarm_id} | The unique id of the alarm that generated this event |
${event_id} | The incremental id of the event, for this alarm id |
${when} | The timestamp this event occurred |
${name} | The name of the alarm, as given in netdata health.d entries |
${url_name} | Same as ${name} but URL encoded |
${chart} | The name of the chart (type.id) |
${url_chart} | Same as ${chart} but URL encoded |
${status} | The current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
${old_status} | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
${value} | The current value of the alarm |
${old_value} | The previous value of the alarm |
${src} | The line number and file the alarm has been configured |
${duration} | The duration in seconds of the previous alarm state |
${duration_txt} | Same as ${duration} for humans |
${non_clear_duration} | The total duration in seconds this is/was non-clear |
${non_clear_duration_txt} | Same as ${non_clear_duration} for humans |
${units} | The units of the value |
${info} | A short description of the alarm |
${value_string} | Friendly value (with units) |
${old_value_string} | Friendly old value (with units) |
${image} | The URL of an image to represent the status of the alarm |
${color} | A color in AABBCC format for the alarm |
${goto_url} | The URL the user can click to see the netdata dashboard |
${calc_expression} | The expression evaluated to provide the value for the alarm |
${calc_param_values} | The value of the variables in the evaluated expression |
${total_warnings} | The total number of alarms in WARNING state on the host |
${total_critical} | The total number of alarms in CRITICAL state on the host |
The configuration file name for this integration is health_alarm_notify.conf.
You can edit the configuration file using the edit-config script from the
Netdata config directory.
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config health_alarm_notify.conf
#------------------------------------------------------------------------------
# custom notifications
SEND_CUSTOM="YES"
DEFAULT_RECIPIENT_CUSTOM=""
# The custom_sender() is a custom function to do whatever you need to do
custom_sender() {
# example human readable SMS
local msg="${host} ${status_message}: ${alarm} ${raised_for}"
# limit it to 160 characters and encode it for use in a URL
urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}"
# a space separated list of the recipients to send alarms to
to="${1}"
for phone in ${to}; do
httpcode=$(docurl -X POST \
--data-urlencode "From=XXX" \
--data-urlencode "To=${phone}" \
--data-urlencode "Body=${msg}" \
-u "${accountsid}:${accounttoken}" \
https://domain.website.com/)
if [ "${httpcode}" = "200" ]; then
info "sent custom notification ${msg} to ${phone}"
sent=$((sent + 1))
else
error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."
fi
done
}
You can run the following command by hand, to test alerts configuration:
# become user netdata
sudo su -s /bin/bash netdata
# enable debugging info on the console
export NETDATA_ALARM_NOTIFY_DEBUG=1
# send test alarms to sysadmin
/usr/libexec/netdata/plugins.d/alarm-notify.sh test
# send test alarms to any role
/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
Note that this will test all alert mechanisms for the selected role.