Host Onboarding Failure Diagnosis (host-onboard-diagnose)

Scope

Enter this skill:

• "My newly installed categraf host doesn't show up in Nightingale"
• "The agent is installed and running, but the host doesn't appear in the host list"
• "For this host in the list, the OS / CPU / version are all unknown"
• "I deployed 3 categraf hosts via Helm, but the platform only sees 1"
• "I installed the agent on Windows but it won't register"
• "The host disappeared right after I changed its hostname" (if categraf is still running)

Do NOT enter this skill:

• Was visible before, recently lost contact → host-health-diagnose
• ident duplicate / want to clean up residue after renaming → host-ident-cleanup (to be built)
• Want to change alert rules / mutes → creation / create-alert-rule
• Looking into why an alert didn't fire → alert-rule-troubleshoot

One-Sentence Principle

A missing host ≠ a single cause. The onboarding pipeline has 5 segments, and each segment getting stuck looks different. Looking at just one segment and telling the user to change config is a common pitfall. Gather evidence first, then localize segment by segment, and finally give fix commands.

The 5 Segments of the Onboarding Pipeline

[1] categraf local process    Is it present / is the config correct / is heartbeat.enable on
        │
[2] heartbeat report HTTP      Can it reach /v1/n9e/heartbeat (network / TLS / BasicAuth)
        │
[3] server / edge receive      token / version compatibility / hostname duplicate check
        │
[4] target table persistence   Is this ident in the DB, is the meta in redis
        │
[5] Redis + metric stream       Can the time-series store find samples for this ident

First Action: Call probe_target_onboard_status

This is the only diagnostic entry tool in this skill; it returns the footprint of all 5 segments in one shot. Always call it first, then decide the next step.

Key fields returned:

• in_target_db + target.os + target.agent_version → evidence for segments 3/4
• in_redis_beat + redis_meta.hostname + redis_meta.remote_addr → evidence for segment 4
• in_prom_target_up + target_up_last + prom_metrics_hit → evidence for segment 5
• likely_segment + likely_causes → diagnosis already aggregated at the tool layer; do not bypass it and re-derive it yourself

If the user did not provide an ident, first call list_targets and let the user pick, or filter out candidates by OS=unknown / empty agent_version (the "unassigned / partially onboarded" view).

Decision Table (branch by likely_segment)

Three Variant Queries for Segment 5 (use query_prometheus)

When likely_segment=segment_5, you must run the following 3 queries with query_prometheus to confirm whether it is an ident label problem or truly no data:

# 1. Standard ident exact match (most common)
target_up{ident="<ident>"}

# 2. Fuzzy match (use when ident has an IP prefix / alias)
target_up{ident=~".*<host>.*"}

# 3. Extreme fallback: did it land on the instance label (snmp / custom tag scenario)
{instance=~".*<host>.*"}

All three return no data → the data stream truly never reached prom; go back and check categraf writers / TLS / n9e ingest queue. Only (2) or (3) has data → ident label problem (special characters / global_labels override / snmp agent_host_tag misuse); guide the user to host-ident-cleanup (to be built) or to fix the categraf config.

Output Template (strongly enforced)

The Final Answer uses Markdown, in the user's language. Four sections:

## Conclusion
<one sentence: stuck at segment X: xxxx (or: onboarding is normal, the reason you can't see it is yyy)>

## Onboarding Pipeline Evidence
- Segment 1/2 (categraf local/HTTP): <not gathered / inferred anomaly: xxx>
- Segment 3 (server receive): target in_db=true, os=unknown, agent_version="" → heartbeat metadata not persisted
- Segment 4 (target persistence + redis): target update_at=2026-05-14 10:23:11 but no heartbeat in redis
- Segment 5 (Prom): target_up has no data / prom_metrics_hit=0

## Fix Commands
1. Run on the target host:
   `grep -E 'heartbeat|omit_hostname' /etc/categraf/conf/config.toml`
   Expected: in the heartbeat section enable=true, omit_hostname=false. If either doesn't match, change it and `systemctl restart categraf`.
2. ...
3. ...

## Self-Verification Steps
<give the user 1-2 commands for "if you want to confirm it's fixed, you can verify like this", for example:
- Within 30s after restarting categraf, refresh the host list back in the platform; the OS/CPU fields should be non-unknown
- `curl -s http://<n9e>:17000/api/n9e/self-metrics | grep <ident>`>

Anti-Patterns (do NOT do these)

• ❌ Telling the user to change heartbeat.enable directly without calling probe_target_onboard_status. Gather evidence first. Many users already have heartbeat enabled; the problem is in omit_hostname / TLS / version.
• ❌ Saying "categraf isn't installed" just because in_target_db=false. You must look at the redis segment and the segment_1_or_2 causes to judge holistically whether it's a network problem or a process problem — the recommended actions are completely different.
• ❌ When segment 3 is stuck, only telling the user to change heartbeat without mentioning omit_hostname / version. Those two are just as common.
• ❌ When segment 5 is stuck, telling the user to change writers without running the 3 variant PromQL queries. An ident label problem also gets stuck at segment 5.
• ❌ Not giving concrete commands in the output, only saying "check the categraf config". Every recommendation must be something the user can paste and run directly.

Known Failure Modes Per Segment

• Segment 1/2: categraf can't reach center, connection refused, TLS unknown authority, self-signed certificate, BasicAuth invalid, ams token mismatch, Helm multi-node only 1 seen, Windows, Win2008 not supported
• Segment 3: heartbeat enable=false, unknown fields / omit_hostname=true, categraf version too low, v6 requires v0.2.35+, identity shell fails to get IP, hostname duplicate
• Segment 4: edge redis nil, n9e and n9e-edge version mismatch, CenterApi missing, host not visible in the center under edge deployment
• Segment 5: ident with parentheses not found in dashboards, host=* bug, snmp ident conflict, omit_hostname=true causing the ident label to be lost, wrong datasource in multi-cluster, write queue full 499, global.labels override

Output Style

• Don't be coy. The conclusion goes in the first line of the first section.
• Evidence must give concrete field values; don't write "looks normal".
• Fix commands must be paste-and-run; avoid fluff like "go check it".
• Answer in the user's language (Chinese for Chinese users, English for English users).
• If likely_segment=ok but the user insists the host doesn't appear, prompt them to check: business-group filtering (the host is present but hidden by business group in the frontend), browser cache, and the visible-business-group permissions of the logged-in user.