Nightingale (n9e) Alert Self-Healing Script (task_tpl) Generation

Nightingale alert self-healing is the ibex subsystem: when an alert rule's callbacks field is written as ${ibex}/<task_tpl_id>, after an alert event fires it pulls up the script of the corresponding task_tpl on the machine matching the current event's TargetIdent and runs it on that machine's categraf. The script receives the labels of the current alert through stdin.

This skill focuses on writing/modifying the task_tpl.script field itself—it does not cover creating alert rules, configuring recipients, or editing notification templates.

1. Scope: First Determine Which Layer the User Is Changing

The Nightingale alert pipeline has four layers, and each goes through a different skill:

Judgment criteria: the user's wording contains "script/shell/bash/python/jq/parse/execute/timeout"—this skill; contains "PromQL/threshold/trigger condition"—alert rule; contains "template/body/field rendering"—message template; contains "URL/Webhook/signature"—notification channel.

2. Data Model (What the User Can Fill In, Which Fields Nightingale Actually Uses)

TaskTpl (models/task_tpl.go:17-35)

TaskForm (models/task_tpl.go:351-365) — the Carrier Actually Dispatched

Alert-triggered self-healing goes through alert/sender/ibex.go CallIbex → constructs a TaskForm → TaskAdd writes to ibex:

• AlertTriggered: true —— edge data centers go through Redis direct-dispatch to categraf, not relying on the central DB (ibex.go:244-276)
• Stdin: <JSON string> —— what this skill cares about most, explained in detail in the next section
• Hosts: []string{event.TargetIdent} —— executes only on the machine the alert event matched
• Title: tpl.Title + " FH: " + host —— easy to recognize in execution history (FH = From Host)

CleanFields Hard Constraints (models/task_tpl.go:137-189)

These validations run before saving or dispatching; violations are rejected outright:

• timeout == 0 → automatically set to 30
• timeout > 3600*24*5 → error "longer than five days"
• title / args / pause / tags containing str.Dangerous characters (`, $(), &&, etc.) → error
• script automatically \r\n → \n (resolves the CRLF problem from Windows editor copies)
• script == "" → error "script is required"

3. ⚠️ The Truth About stdin (the Easiest Pitfall, Must Read)

3.1 stdin Is a Flat JSON of map[string]string, Not a $event Object

Many users copy syntax like {{$event.RuleName}} from the message template docs—it does not work at all. $event is the context of the notify_tpl template; there is no template rendering process inside an ibex self-healing script—what it receives is just a plain JSON string.

The construction logic of stdin (alert/sender/ibex.go:118-142):

tagsMap := make(map[string]string)
for _, pair := range event.TagsJSON {        // flatten event labels one by one
    k, v := splitOnce(pair, "=")
    tagsMap[k] = v
}
// inject 3 built-in keys
tagsMap["alert_severity"]      = strconv.Itoa(event.Severity)
tagsMap["alert_trigger_value"] = event.TriggerValue
tagsMap["is_recovered"]        = strconv.FormatBool(event.IsRecovered)

tags, _ := json.Marshal(tagsMap)             // serialize the whole thing into a string
in.Stdin = string(tags)

Example of a real stdin payload:

{
  "ident": "host01",
  "instance": "host01:9100",
  "job": "categraf",
  "alert_severity": "2",
  "alert_trigger_value": "92.5",
  "is_recovered": "false"
}

Note: all values are strings (alert_severity is "2", not 2); to parse them into numbers in the script you have to convert them yourself.

3.2 Three Fields You "Want but Can't Get", and Their Solutions

3.3 Templates for Reading stdin in Three Languages (Must Include When Generating Scripts)

shell + jq (most common):

#!/bin/bash
set -euo pipefail

PAYLOAD=$(cat)                    # read all of stdin at once to avoid blocking
[ -z "$PAYLOAD" ] && { echo "FATAL: empty stdin payload"; exit 2; }

IDENT=$(echo "$PAYLOAD" | jq -r '.ident          // empty')
SEV=$(echo   "$PAYLOAD" | jq -r '.alert_severity // "3"')
VAL=$(echo   "$PAYLOAD" | jq -r '.alert_trigger_value // "0"')

[ -z "$IDENT" ] && { echo "FATAL: no ident in stdin"; exit 2; }
echo "[$(date -Iseconds)] ident=$IDENT severity=$SEV value=$VAL"

shell fallback without jq (use grep+sed, suitable for minimal containers):

PAYLOAD=$(cat)
IDENT=$(echo "$PAYLOAD" | grep -oE '"ident"[[:space:]]*:[[:space:]]*"[^"]*"' | sed 's/.*"\([^"]*\)"$/\1/')

python (when the processing logic is complex):

#!/usr/bin/env python3
import sys, json, time

try:
    data = json.load(sys.stdin)
except json.JSONDecodeError as e:
    print(f"FATAL: bad stdin json: {e}", file=sys.stderr)
    sys.exit(2)

ident = data.get("ident", "")
sev   = int(data.get("alert_severity", "3"))
val   = float(data.get("alert_trigger_value", "0"))

if not ident:
    print("FATAL: no ident", file=sys.stderr)
    sys.exit(2)

print(f"[{time.strftime('%FT%T')}] ident={ident} sev={sev} val={val:.2f}")

go (rarely used, minimal skeleton):

package main

import (
    "encoding/json"
    "fmt"
    "os"
)

func main() {
    var d map[string]string
    if err := json.NewDecoder(os.Stdin).Decode(&d); err != nil {
        fmt.Fprintln(os.Stderr, "FATAL:", err)
        os.Exit(2)
    }
    fmt.Println("ident:", d["ident"])
}

4. ⚠️ The Truth About is_recovered (Don't Hard-Code Dead Code)

ibex self-healing does not fire on recovery events—alert/sender/ibex.go:39-42:

func (c *IbexCallBacker) CallBack(ctx CallBackContext) {
    ...
    if event.IsRecovered {
        logger.Infof("event_callback_ibex: event is recovered, event: %s", event.Hash)
        return                       // ← returns directly, never calls handleIbex at all
    }
    ...
}

Therefore is_recovered in stdin is always the string "false". Writing if [ "$IS_RECOVERED" = "true" ]; then ... is dead code.

The correct path:

• If you want to "also run an action on alert recovery (send email / close ticket / notify IM)" → use notify_rule + callback medium, check "also notify on recovery" in the notify_rule, and process the event JSON after the callback URL receives it.
• If you want to "trigger another script on a recovery event" → same as above; do not try to reuse task_tpl.

A core pain point is "I also want it to trigger on recovery", and the common idea of "adding an is_recovered check" is actually a non-solution—fundamentally you need an architectural change to ibex so it accepts recovery events, or the user must switch to the callback channel.

5. timeout / batch / tolerance / pause —— Semantics of the Numeric Fields

5.1 timeout (seconds)

• Defaults to 30 seconds (auto-filled by CleanFields)
• Maximum of 5 days (anything longer errors out; by design self-healing is a short-cycle task)
• After timeout the process is SIGKILLed; stdout/stderr is written to the task_host table before being killed
• No direct relationship to the alert evaluation interval—a 60s evaluation cycle does not mean the script only has 60s to run

Typical value reference:

5.2 Distinguish Three "Timeout-Class" Root Causes

Do not lump them together—when the user says "timeout", first ask about the scenario.

5.3 batch / tolerance / pause

• batch: number of hosts running concurrently per batch. Single-host self-healing triggered by an alert doesn't need it; leave at 0. Only manual batch tasks (ops scenarios) use it.
• tolerance: allowed failure count within a batch. Same as batch, leave at 0 for self-healing scenarios.
• pause: pause between batches (e.g. 00:00-08:00,17:00-23:59 means pause during these time ranges). Basically not used for self-healing—an alert can fire at any time, so you can't use pause to restrict working hours. If you really want to restrict working hours, do it in the alert_rule's enable_in_bg / enable_stime fields, or use the notify_rule's time window.

6. Dangerous Command List (Must Be Avoided or Guarded When Generating)

6.1 Blacklist (Refuse to Generate Outright, Even If the User Explicitly Asks)

6.2 Graylist (Generate but Must Add Guards)

6.3 Generic Guard Template

#!/bin/bash
set -euo pipefail

# (1) single-instance lock, to prevent concurrent triggers
LOCK=/var/run/$(basename "$0").lock
exec 9>"$LOCK"
flock -n 9 || { echo "another instance running, skip"; exit 0; }

# (2) dry-run switch (pass --dry-run in args to not actually do it)
DRY_RUN=0
[[ "${1:-}" == "--dry-run" ]] && DRY_RUN=1

# (3) before state
echo "=== BEFORE ==="
df -h /var/log

# (4) main body (switch by DRY_RUN)
if [[ $DRY_RUN -eq 1 ]]; then
    find /var/log -name "*.log" -mtime +7 -print | head -20
else
    find /var/log -name "*.log" -mtime +7 -delete
fi

# (5) after state
echo "=== AFTER ==="
df -h /var/log

7. Built-in Scenario Library (For When "I Can't Think of a Good Scenario" Hits)

Each scenario gives: typical alert rule name → script skeleton → suggested timeout → risk points.

7.1 Disk Space Category

S1: Clean up logs older than 7 days under /var/log

• Trigger: disk_used_percent{mountpoint="/"} > 90
• timeout: 120

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
echo "before: $(df -h /var/log | tail -1)"
find /var/log -type f -name "*.log.*" -mtime +7 -print -delete | wc -l
find /var/log -type f -name "*.gz"    -mtime +7 -print -delete | wc -l
echo "after:  $(df -h /var/log | tail -1)"

S2: Clean up docker image layer cache

• Trigger: disk full + the host has docker
• timeout: 300

#!/bin/bash
set -euo pipefail
command -v docker >/dev/null || { echo "no docker, skip"; exit 0; }
echo "before:"; df -h /var/lib/docker
docker image prune -af --filter "until=168h"     # images older than 7 days
docker builder prune -af --filter "unused-for=168h"
echo "after:"; df -h /var/lib/docker

S3: Clean up files older than 30 days under /tmp

• timeout: 60

#!/bin/bash
set -euo pipefail
find /tmp -xdev -type f -mtime +30 -print -delete | wc -l
find /tmp -xdev -type d -empty -mtime +30 -delete 2>/dev/null || true

S4: Shrink journalctl to 500MB

• timeout: 60

#!/bin/bash
set -euo pipefail
journalctl --disk-usage
journalctl --vacuum-size=500M
journalctl --disk-usage

7.2 Memory / Process Category

S5: Find the top5 memory processes and dump them (do not auto-kill)

• Trigger: mem_used_percent > 90
• timeout: 60
• ⚠️ Intentionally does not kill processes—high memory is rooted in a business problem, and auto-killing would widen the failure

#!/bin/bash
set -euo pipefail
TS=$(date +%Y%m%d-%H%M%S)
OUT=/var/log/n9e-mem-top5.$TS.txt
{
    echo "=== top5 mem proc @$TS ==="
    ps -eo pid,user,%mem,%cpu,etime,cmd --sort=-%mem | head -6
    echo; free -m
} > "$OUT"
echo "dumped to $OUT"

S6: JVM OOM auto-dump + restart

• Trigger: process_jvm_heap_used_percent > 95
• timeout: 600
• ⚠️ Requires the service label in stdin (add it to the PromQL by clause)

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
SVC=$(echo "$PAYLOAD" | jq -r '.service // empty')
[ -z "$SVC" ] && { echo "no service tag"; exit 2; }

PID=$(systemctl show -p MainPID --value "$SVC")
[ "$PID" = "0" ] && { echo "service not running"; exit 0; }

TS=$(date +%Y%m%d-%H%M%S)
DIR=/var/log/jvm-dump
mkdir -p "$DIR"

jstack "$PID" > "$DIR/jstack.$SVC.$TS.txt" || true
jmap -dump:live,format=b,file="$DIR/heap.$SVC.$TS.hprof" "$PID" || true

# lock file to prevent re-entry
LOCK=/var/run/n9e-restart-$SVC.lock
exec 9>"$LOCK"; flock -n 9 || { echo "already restarting"; exit 0; }

systemctl restart "$SVC"
sleep 5
systemctl is-active "$SVC"

S7: Zombie process cleanup

• Trigger: zombie_process_count > 5
• timeout: 30

#!/bin/bash
set -euo pipefail
ps -eo stat,ppid,pid,cmd | awk '$1 ~ /^Z/ {print $2}' | sort -u | while read ppid; do
    echo "send SIGCHLD to ppid $ppid"
    kill -SIGCHLD "$ppid" 2>/dev/null || true
done

7.3 Service Availability Category

S8: Service port down → systemctl restart (with cooldown)

• Trigger: net_tcp_connection_count{port="3306"} < 1
• timeout: 120

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
SVC=$(echo "$PAYLOAD" | jq -r '.service // empty')
[ -z "$SVC" ] && { echo "no service tag"; exit 2; }

# cooldown: skip if already restarted within 5 minutes
COOLDOWN=/var/run/n9e-restart-$SVC.cooldown
if [ -f "$COOLDOWN" ] && [ $(($(date +%s) - $(stat -c%Y "$COOLDOWN"))) -lt 300 ]; then
    echo "in cooldown, skip"
    exit 0
fi

systemctl restart "$SVC"
touch "$COOLDOWN"
sleep 5
systemctl is-active "$SVC"

S9: Reload after nginx config change

• Trigger: manual trigger / file_change_count{path="/etc/nginx"} > 0
• timeout: 30

#!/bin/bash
set -euo pipefail
nginx -t                          # validate config
nginx -s reload
sleep 1
pgrep -x nginx >/dev/null || { echo "nginx died after reload"; exit 1; }
echo "reloaded ok"

S10: Only diagnose a crashloop container without restarting it (k8s)

• Trigger: kube_pod_container_status_restarts_total > 5
• timeout: 60
• ⚠️ k8s already auto-restarts, so self-healing only does diagnosis + notification

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
NS=$(echo "$PAYLOAD" | jq -r '.namespace // "default"')
POD=$(echo "$PAYLOAD" | jq -r '.pod // empty')
[ -z "$POD" ] && { echo "no pod tag (check PromQL by clause)"; exit 2; }

TS=$(date +%Y%m%d-%H%M%S)
OUT=/var/log/n9e-crash-$NS-$POD.$TS.log
{
    kubectl -n "$NS" describe pod "$POD"
    echo "==="
    kubectl -n "$NS" logs "$POD" --tail=200 --previous || true
} > "$OUT"
echo "diagnosed → $OUT"

7.4 Network / System Category

S11: NIC down → ip link set up (whitelist)

• Trigger: net_interface_up{iface=~"eth.+|ens.+"} == 0
• timeout: 30

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
IFACE=$(echo "$PAYLOAD" | jq -r '.iface // empty')
[ -z "$IFACE" ] && { echo "no iface"; exit 2; }

# whitelist: only allow eth* / ens* / eno*, to avoid touching container/VPN NICs
[[ "$IFACE" =~ ^(eth|ens|eno) ]] || { echo "iface $IFACE not in whitelist"; exit 0; }

ip link show "$IFACE"
ip link set "$IFACE" up
sleep 2
ip link show "$IFACE"

S12: NTP offset → restart chronyd

• Trigger: system_clock_offset_seconds > 60
• timeout: 30

#!/bin/bash
set -euo pipefail
chronyc tracking | grep -E "System time|Last offset"
systemctl restart chronyd
sleep 5
chronyc makestep
chronyc tracking | grep -E "System time|Last offset"

S13: DNS resolution failure → restart systemd-resolved + verify

• timeout: 60

#!/bin/bash
set -euo pipefail
echo "before: $(dig +short @127.0.0.53 example.com || true)"
systemctl restart systemd-resolved
sleep 2
echo "after:  $(dig +short @127.0.0.53 example.com)"

S14: Recover drifted sysctl parameters

• Trigger: node_net_somaxconn != 65535
• timeout: 10

#!/bin/bash
set -euo pipefail
sysctl -w net.core.somaxconn=65535
grep -q "net.core.somaxconn=65535" /etc/sysctl.d/99-n9e.conf || \
    echo "net.core.somaxconn=65535" >> /etc/sysctl.d/99-n9e.conf

7.5 Windows Scenarios (PowerShell)

S15: Windows service down → Restart-Service

• Trigger: windows_service_state{state!="Running"} == 1
• timeout: 120
• In the task_tpl, set account to a user with permissions; script type script_type: powershell (supported by categraf-win)

$payload = [Console]::In.ReadToEnd() | ConvertFrom-Json
$svc = $payload.service
if (-not $svc) { Write-Error "no service tag"; exit 2 }

Get-Service $svc | Format-Table
Restart-Service $svc -Force
Start-Sleep -Seconds 5
$st = (Get-Service $svc).Status
Write-Output "after restart: $st"
if ($st -ne "Running") { exit 1 }

S16: Windows disk cleanup (temp + Windows.old)

• timeout: 300

$before = (Get-PSDrive C).Free / 1GB
Get-ChildItem C:\Windows\Temp -Recurse -Force -ErrorAction SilentlyContinue |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-7) } |
    Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
$after = (Get-PSDrive C).Free / 1GB
Write-Output ("freed {0:N2} GB" -f ($after - $before))

7.6 In-Container Scenarios

S17: Restart a process inside a pod via kubectl exec

• Register categraf on an "ops machine" in the central data center that has a kubeconfig, and select this machine for the task_tpl to execute on
• timeout: 120

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
NS=$(echo "$PAYLOAD" | jq -r '.namespace // empty')
POD=$(echo "$PAYLOAD" | jq -r '.pod // empty')
CTN=$(echo "$PAYLOAD" | jq -r '.container // empty')

[ -z "$NS" ] || [ -z "$POD" ] || [ -z "$CTN" ] && { echo "need namespace/pod/container in labels"; exit 2; }

kubectl -n "$NS" exec "$POD" -c "$CTN" -- supervisorctl restart app || \
kubectl -n "$NS" delete pod "$POD"     # fallback recreate

S18: Execute inside a container via docker exec

• timeout: 60

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
CNAME=$(echo "$PAYLOAD" | jq -r '.container_name // empty')
[ -z "$CNAME" ] && { echo "no container_name"; exit 2; }
docker inspect "$CNAME" >/dev/null || { echo "no such container"; exit 2; }
docker exec "$CNAME" sh -c 'kill -HUP 1'    # let PID 1 reload itself

7.7 Middleware Scenarios

S19: MySQL long transaction kill (with user whitelist)

• Trigger: mysql_long_transaction_seconds > 600
• timeout: 60

#!/bin/bash
set -euo pipefail
PAYLOAD=$(cat)
THRESH=600

# only kill user connections, bypass system users
mysql -N -e "
SELECT trx_mysql_thread_id
FROM information_schema.innodb_trx t
JOIN information_schema.processlist p ON p.ID=t.trx_mysql_thread_id
WHERE TIMESTAMPDIFF(SECOND, trx_started, NOW()) > $THRESH
  AND p.USER NOT IN ('root','repl','backup','event_scheduler')
" | while read tid; do
    [ -n "$tid" ] && mysql -e "KILL $tid"
done

S20: Redis memory over threshold → force BGREWRITEAOF + suggest scaling up

• timeout: 120

#!/bin/bash
set -euo pipefail
USED=$(redis-cli INFO memory | awk -F: '/^used_memory_rss:/{gsub(/\r/,""); print $2}')
echo "rss: $USED bytes"
redis-cli BGREWRITEAOF
echo "BGREWRITEAOF triggered, NOT auto-restarting redis; please scale up maxmemory or scale out"

8. Debugging and Troubleshooting

8.1 "Script Stays Running Forever"

Cross-reference issues in chronological order:

8.2 "Can't Get stdin"

Troubleshooting order:

1. Open a manual task test: in "Task Center" manually create a task once, enter {"ident":"host01"} as stdin, and see whether the script can echo it out
2. Check the PromQL by() clause: 90% of missing-label cases in self-healing are this; the event detail page shows what event.TagsJSON actually contains
3. Check how the script reads: read VAR only reads one line; use cat or python json.load(sys.stdin) to read it all at once

8.3 "exit code 0 but the Service Didn't Actually Recover"

• Force set -e (exit immediately if any sub-command fails)
• Add explicit verification after key steps: systemctl is-active "$SVC" || exit 1
• Output before/after to stdout so task_record is observable

8.4 SQL to Check Execution History (When the User Asks "Did the Task Run Successfully")

-- look at all self-healing tasks triggered by a given event
SELECT id, title, create_at, FROM_UNIXTIME(create_at) AS ts
FROM task_record
WHERE event_id = <event_id>
ORDER BY id DESC;

-- task_host_x is a sharded table (by task_id mod 26)
-- assume task_id=12345, 12345%26=21
SELECT id, host, status, stdout, stderr
FROM task_host_21
WHERE id = 12345;

Task history is designed to be non-deletable—don't teach users to delete it; tell them this is an audit requirement.

9. Output Style

When the user asks "help me write a self-healing script for X", follow this routine:

1. State the assumed scenario in one sentence ("Assuming the trigger condition is disk_used_percent > 90 and the machine has jq installed")
2. Fenced script block—must contain stdin parsing + set -euo pipefail + before/after echo + the main body; graylist commands must include guards
3. stdin field explanation: list only the keys actually used in the script, marking which are PromQL labels and which are ibex-injected
4. Run parameter suggestions: timeout=120 / batch=0 / tolerance=0 / pause=empty / account=root, with reasons
5. Risk and rollback: destructive actions must include a rollback method or an "irreversible" warning

When the user asks "why X": answer directly, don't dump a whole script—especially for is_recovered-class questions, correct the concept first and then mention the alternative.

When the user requests destructive commands: state the risk + give a safe alternative, and never generate it directly.