.agents/skills/project-create-topology/topology-modes-correlation-aggregation.md
Implementation contract introduced by topology v1 producer work in PR #22496 /
commit 8afe52d8b.
PR #22496 completed the cross-repo compatibility layer for modes, modal identification, correlation classes, and table merge policy. The stronger detailed network-connections loose-side graph model remains target behavior, but requires a separate Agent/UI/aggregator execution pass before it becomes current contract.
Topology payloads must let operators inspect a topology at the right level:
The contract must remain topology-agnostic. The UI and aggregator must not learn domain words such as process, router, parent, child, endpoint, LLDP, socket, or retention as hardcoded behavior. Producers describe identities, modes, merge rules, table merge policies, and presentation recipes in the payload.
netdata.topology.v1
payload, for example network-connections, SNMP, streaming, or vSphere.netdata.topology.v1 payload.The user-facing request key for topology mode is __topology_mode.
Allowed values:
detailedaggregatedIf the key is absent, each Function uses its documented default. Producers that do not have a meaningful detailed/aggregated difference should not expose a mode selector only to return identical output.
Mode-capable producers declare data.view.supported_modes. Consumers treat an
absent field or a single-value field as mode-invariant and must not show a
detailed/aggregated toggle for that payload.
The aggregator must consume detailed payloads whenever a producer supports detail mode.
When the user asks the aggregator for __topology_mode=aggregated, the
aggregator must rewrite fanout requests to producers as
__topology_mode=detailed before correlation and aggregation. This prevents
early information loss before cross-node matching. After correlation, the
aggregator returns either detailed or aggregated output according to the
original user request.
If a producer does not expose __topology_mode, the aggregator must not invent
that parameter for it. SNMP and streaming are expected to be mode-invariant
unless a future producer change defines a real mode difference.
Aggregator internal states must not appear in final payloads. Terms such as absorbed, candidate, rewrite plan, partial class, or equivalence set may exist inside the service, but final topology output contains only normal actors, links, tables, labels, presentation, and diagnostics.
Do not copy high-cardinality evidence rows only to make modal tables easier. Modal sections must select and project existing actors, links, evidence, relationship tables, actor tables, and actor labels.
Small scalar facts may appear in more than one plane when the grains differ. For
example, a graph link and a relationship-summary row may both carry
socket_count; the graph link is the renderable relationship, while the summary
row is the modal/drilldown grain.
The modal identification area is part of the schema contract.
types.actor_types.<id>.presentation.modal.labels must be extended with an
ordered producer-selected identification list over the existing actor label
table. The UI renders those selected label keys near the actor title. The full
label table remains available in the Labels tab.
Target shape:
{
"labels": {
"enabled": true,
"table": "actor_labels",
"actor_column": "actor",
"key_column": "key",
"value_column": "value",
"identification": {
"enabled": true,
"fields": [
{ "key": "process", "label": "Process", "max_values": 1 },
{ "key": "username", "label": "User", "max_values": 1 },
{ "key": "cmdline", "label": "Command", "max_values": 1 }
]
}
}
}
Rules:
identification.fields[] selects rows from actor_labels by key.actor_column.value_index when present.max_values limits displayed values for one key. The full Labels tab still
shows all values.This spec requires these schema extensions beyond the currently deployed v1 contract:
modal.labels.identification.enabled and
modal.labels.identification.fields[].The implementation may encode these additions in the most compact shape that fits the existing JSON schema style. This spec defines semantics; exact field names are accepted when they are schema-valid, documented, and used uniformly by Agent, UI, and aggregator.
Correlation rules are declarative. The aggregator builds keys from columns and literals; it does not understand domain semantics.
Required rule classes:
resolve_loose_side: matches a loose relationship side to a known actor or
to a materialized partial actor.replace_actor: replaces weaker actors with stronger actors and rewires
incident links/tables.merge_enrich_actor: merges actors with the same identity and combines their
labels, attributes, links, evidence, and detail tables according to declared
table policies.Rules run by ascending priority number. Exact rules must run before broader or partial rules.
Example:
{
"rules": {
"socket_exact": {
"class": "resolve_loose_side",
"priority": 10,
"key_space": "socket",
"key": [
{ "column": "protocol" },
{ "literal": "|" },
{ "column": "address_space" },
{ "literal": "|" },
{ "column": "ip" },
{ "literal": ":" },
{ "column": "port" }
],
"output_link_type": "socket"
},
"ip_partial": {
"class": "resolve_loose_side",
"priority": 100,
"key_space": "ip",
"key": [
{ "column": "address_space" },
{ "literal": "|" },
{ "column": "ip" }
],
"output_link_type": "partial_endpoint"
}
}
}
If one point matches exactly one claim, apply the rule.
If one point matches multiple claims with the same priority, the aggregator must not pick randomly. It keeps the point unresolved or materializes a partial actor according to the rule, and records a diagnostic.
If no match exists, the point or loose side remains unresolved and may be materialized for UI display according to the declared materialization policy.
NAT, load balancer, or alias information is modeled as additional keys for the same point or claim. Alias rows add match possibilities; they do not mutate or delete the original observation.
Every table type that can cross producer boundaries needs a merge policy.
Required dimensions:
key: columns that identify equivalent rows.action: one of deduplicate, append, set_union, merge_metrics,
latest, or preserve.metrics: per numeric column merge operation, such as sum, min, max,
avg_weighted, or latest.conflicts: how non-key scalar conflicts are handled. Allowed policies are
prefer_claim, prefer_newest_agent, preserve_all, or diagnostic.General rules:
append or deduplicate.merge_metrics on the declared key.set_union keyed by actor, key, value, source, kind, and
value_index.preserve or append when two parents retain data
for the same node because each retaining parent is meaningful.The Agent returns the producer's local topology.
__topology_mode.The aggregator always collects the highest-detail useful input it can get.
The UI renders the payload it receives.
modal.labels.identification.fields[].actor_labels.A socket observation is associated with an observed process actor and two dependency endpoint tuples:
protocol, client_ip, client_port, server_ip, server_port, state
For inbound sockets, the observed process owns the server tuple. For outbound sockets, the observed process owns the client tuple. For local/same-node sockets, both process actors may be known, but the emitted topology direction remains client-to-server. For listening sockets, there is no remote side.
The exact remote tuple is required for cross-node correlation, but creating an
actor per IP:PORT in a single-node graph can explode actor count and force
layout noise. Therefore detailed mode must preserve the exact tuple without
requiring every tuple to be an actor.
Agent aggregated mode is a readable local process dependency map.
Rules:
direction_role: "dependency" and graph
links point from client/dependant actor to server/dependency actor.client_ip, server_ip) and merged
metrics, but not per-socket ephemeral ports.socket_count.Synthetic example:
{
"view": { "mode": "aggregated" },
"actors": [
{ "id": 1, "type": "node", "display_name": "node-a" },
{ "id": 2, "type": "process", "display_name": "api", "socket_count": 14 },
{ "id": 3, "type": "endpoint", "display_name": "198.51.100.20", "ip": "198.51.100.20" }
],
"links": [
{ "src_actor": 1, "dst_actor": 2, "type": "ownership" },
{ "src_actor": 2, "dst_actor": 3, "type": "endpoint_socket", "protocol": "tcp", "socket_count": 12 }
],
"tables": {
"relationship": {
"connections": [
{ "src_actor": 2, "dst_actor": 3, "protocol": "tcp", "client_ip": "192.0.2.10", "server_ip": "198.51.100.20", "socket_count": 12 }
]
}
}
}
The example uses row objects for readability. Production uses compact tables.
Agent detailed mode preserves exact socket evidence.
Rules:
Synthetic example:
{
"view": { "mode": "detailed" },
"actors": [
{ "id": 1, "type": "node", "display_name": "node-a" },
{ "id": 2, "type": "process", "display_name": "api" },
{ "id": 3, "type": "endpoint", "display_name": "198.51.100.20", "ip": "198.51.100.20" }
],
"evidence": {
"socket": [
{
"src_actor": 2,
"dst_actor": 3,
"protocol": "tcp",
"client_ip": "192.0.2.10",
"client_port": 50120,
"server_ip": "198.51.100.20",
"server_port": 443
}
]
}
}
The aggregator receives detailed rows.
Exact match:
node-a api claims client tcp 192.0.2.10:50120, points at server tcp 198.51.100.20:443
node-b nginx claims server tcp 198.51.100.20:443, points at client tcp 192.0.2.10:50120
Result:
node-a/api -> node-b/nginx
No endpoint actors remain for the exact match. The loose side was resolved to a known actor and the final graph is a normal process-to-process dependency.
Partial match:
node-a api outbound -> 198.51.100.20:443
node-b is known to own 198.51.100.20
node-b has no matching process/socket row at collection time
Result:
node-a/api -> node-b/[materialized endpoint for 198.51.100.20]
The graph remains truthful: the dependency points at node-b, but the exact process could not be proven.
No match:
node-a/api -> materialized endpoint 198.51.100.20
The unresolved endpoint remains visible with presentation that clearly differs from resolved process links.
Direct Agent aggregated:
Dependencies from relationship-summary rows
where the selected actor is src_actor, and Dependants where it is
dst_actor.Direct Agent detailed:
Dependencies and Dependants from exact socket
evidence using the same src_actor / dst_actor split.Aggregator aggregated:
Aggregator detailed:
SNMP topology observations are device, interface, neighbor, forwarding, ARP, bridge, VLAN, protocol, and bounded L3 adjacency facts.
Most graph links represent observed or inferred L2 relationships between two actors. SNMP may also emit logical L3 relationships with their own link and evidence types. These links MUST NOT be presented as physical or L2 adjacency.
l3_subnet represents an observed shared-subnet relationship between two
managed SNMP device actors. It is currently derived from interface IP/netmask
state collected from IP-MIB ipAddrTable, limited to IPv4 point-to-point
/30 and /31 subnets. It means both devices have managed interfaces in the
same point-to-point subnet; it does not prove that the devices are physically
connected.
l3_subnet_segment represents a broader IPv4 shared subnet observed from
managed SNMP device interface state. SNMP currently emits these segment actors
for /24 through /29 subnets. The segment actor is a logical grouping point,
not a physical switch, bridge domain, VLAN, or endpoint. Each member device is
connected to the segment by an l3_subnet_membership link. Membership evidence
contains the member actor, subnet, member IP, interface index/name/description,
network, netmask, prefix, and source. Only resolved managed SNMP network-device
actors participate; unmanaged endpoints and unresolved rows are suppression
state, not graph actors.
l3_subnet_segment actor ids must be scoped by a stable producer scope, such
as the parent Agent registry id, plus the subnet identity. If the producer
cannot obtain a stable scope id, it must omit L3 subnet segments rather than
fall back to a process-local or random id. This prevents identical private
subnets from different Agents from colliding after Cloud aggregation.
Current SNMP L3 subnet segments are single-routing-context. L3Interface does
not currently carry a VRF or routing-context field, so identical subnet/prefix
values inside one producer scope are treated as one logical segment. Producers
MUST NOT present these segments as VRF-aware until collection adds routing
context to L3 interface observations and segment identity includes it.
ospf_adjacency represents OSPF control-plane adjacency between two resolved
managed SNMP device actors. It is a logical L3 routing-protocol relationship,
not physical, L2, discovery, or port-neighbor evidence. The producer emits graph
links only for full OSPF neighbors with resolved managed endpoints. Non-full or
unresolved OSPF neighbor rows remain diagnostic actor-owned detail rows and
must not create loose router/IP graph actors.
Current SNMP OSPF topology scope is OSPFv2 non-virtual neighbors from
ospfNbrTable; OSPFv3 and OSPF virtual-neighbor tables are future scope unless
a later topology feature explicitly adds their distinct semantics.
bgp_adjacency represents BGP control-plane adjacency between two resolved
managed SNMP device actors. It is a logical routing-protocol relationship, not
physical, L2, discovery, or port-neighbor evidence. The producer emits graph
links only for established BGP peers with resolved local and remote managed
actors. Unresolved or non-established BGP peer rows remain diagnostic
actor-owned detail rows and must not create loose router/IP graph actors.
BGP adjacency graph identity should use the managed actor pair plus routing
instance. BGP identifiers, ASNs, and endpoint IPs are useful evidence and
display fields, but they must not be the primary graph deduplication key because
SNMP BGP profiles expose those fields inconsistently across vendors. Parallel
sessions between the same managed actor pair in the same routing instance should
remain actor-owned peer detail rows under one compact graph relationship unless
a future producer contract explicitly adds a more detailed BGP graph mode.
When ospf_adjacency and l3_subnet describe the same resolved actor pair and
endpoint/subnet relationship, OSPF is the stronger protocol-specific signal for
the graph. The producer may suppress the matching l3_subnet graph link while
preserving typed OSPF evidence and actor-owned neighbor detail rows.
SNMP is not a loose-side topology. Every graph link should have two actors in
both Agent and aggregator views. l3_subnet follows the same rule: unmatched
or ambiguous L3 endpoints are diagnostic/suppression state, not materialized
IP-only graph actors.
SNMP detailed and aggregated modes are currently a no-op. The producer should
not expose __topology_mode until it has a real lower/higher-grain distinction.
If a global Cloud topology request asks for aggregated output, the aggregator still consumes the same SNMP payload and returns the same semantic grain after correlation/replacement.
SNMP mainly uses actor replacement.
Examples:
Replacement example:
payload-a: switch-a port 10 -> lldp-remote(chassis=aa:bb:cc)
payload-b: managed-switch-b(chassis=aa:bb:cc)
Result:
switch-a port 10 -> managed-switch-b
The weaker LLDP remote actor is removed from the aggregated output and its incident links/tables are rewired to the managed device actor.
For l3_subnet, aggregation MUST preserve the explicit l3_subnet link and
evidence type. Replacement may rewire endpoint actors to stronger managed
device actors, but it MUST NOT reinterpret l3_subnet as discovery protocol
evidence or port-neighbor evidence.
For l3_subnet_segment, aggregation MUST preserve the explicit
l3_subnet_segment actor type and l3_subnet_membership link/evidence type.
Replacement may rewire member actors to stronger managed device actors, but it
MUST NOT reinterpret subnet membership as L2 segment membership, discovery
protocol evidence, or port-neighbor evidence. Aggregation may merge identical
scoped subnet segment actors from the same producer scope, but must not merge
identical private subnet strings across different producer scopes.
Within one producer scope, the current identity is subnet/prefix only because
SNMP topology does not yet collect VRF/routing-context for L3 interfaces.
For ospf_adjacency, aggregation MUST preserve the explicit ospf_adjacency
link and evidence type, including its semantic_role: control. Replacement may
rewire endpoint actors to stronger managed device actors, but it MUST NOT
reinterpret OSPF adjacency as discovery, physical, L2, or port-neighbor
evidence. Actor-owned OSPF neighbor detail rows should remain attached to the
local device actor and are not loose-side graph materialization input unless a
future producer contract explicitly defines such a policy.
For bgp_adjacency, aggregation MUST preserve the explicit bgp_adjacency
link and evidence type, including its semantic_role: control. Replacement may
rewire endpoint actors to stronger managed device actors, but it MUST NOT
reinterpret BGP adjacency as discovery, physical, L2, or port-neighbor
evidence. Actor-owned BGP peer detail rows should remain attached to the local
device actor and are not loose-side graph materialization input unless a future
producer contract explicitly defines such a policy.
The UI should not expose a detailed/aggregated toggle for SNMP unless the payload declares supported modes.
Device modals should remain port-centric:
if_index as the
visible real numeric port ID when known, and the port name;l3_subnet relationship
evidence, not from actor_port_links.l3_subnet_membership evidence. A focused device may show the shared subnet
segment and the focused membership without automatically fanning out to every
other managed device in that broader subnet.actor_ospf_neighbors and actor_bgp_peers, not from
actor_port_links. The modal may show unresolved or non-established protocol
neighbors as diagnostics without creating graph links.Streaming topology describes Netdata Agent streaming relationships:
child -> parent
parent <-> parent
virtual/stale/remote nodes represented as actors
All actors are real topology actors from the streaming view. Links always have two actor refs. Streaming is not a loose-side topology.
Streaming detailed and aggregated modes are currently a no-op. The producer
should not expose __topology_mode until it has a real lower/higher-grain
distinction.
The aggregator still consumes the same streaming payload for global aggregated requests and returns merged/enriched streaming topology.
Streaming uses actor enrichment and table merging by machine_guid.
Example:
child-1 -> parent-1 <-> parent-2 <- child-2
Both parents may report facts about the same node. The aggregator must not show
duplicate actors for the same machine_guid. It merges those actors and then
merges their tables according to table policy.
Required merge behavior:
Retention example:
payload-parent-a: parent-a retains child-x for tier 0
payload-parent-b: parent-b retains child-x for tier 0
payload-child-x: child-x self retention tier 0
Result:
actor child-x modal Retention table has 3 rows:
retaining actor parent-a
retaining actor parent-b
retaining actor child-x
These rows must not be deduplicated away solely because the retained child is the same.
The UI should not expose a detailed/aggregated toggle for streaming unless the payload declares supported modes.
Actor modal identification should show selected labels such as role, hostname, machine GUID when useful, and stream status. The full Labels tab remains complete.
Tables:
observer_actor. This is the
default retention view in the current modal contract;The current default modal contract does not show a separate Retention for node
section. The underlying retention table still preserves actor and
observer_actor so aggregated/cloud views can add an explicitly named
Retained by section later without changing the facts.
Highlight path must use the deduplicated stream-path table, not direct sibling selection only.
Input request to Cloud:
function=topology:network-connections __topology_mode=aggregated
Aggregator fanout:
node-a: topology:network-connections __topology_mode=detailed
node-b: topology:network-connections __topology_mode=detailed
Aggregator work:
Returned payload:
{
"view": { "mode": "aggregated" },
"actors": "... compact post-correlation actor table ...",
"links": "... compact post-correlation graph links ...",
"tables": {
"relationship": {
"connections": "... aggregated drilldown rows ..."
}
}
}
Input request to Agent:
function=topology:network-connections __topology_mode=detailed
UI work:
Input request to Cloud:
function=topology:snmp __topology_mode=aggregated
Aggregator fanout:
node-a: topology:snmp
node-b: topology:snmp
No mode parameter is sent unless the producer advertises one. Aggregator applies replacement rules and returns a normal graph.
Input request to Cloud:
function=topology:streaming __topology_mode=aggregated
Aggregator fanout:
node-a: topology:streaming
node-b: topology:streaming
No mode parameter is sent unless the producer advertises one. Aggregator merges
actors by machine_guid, preserves retention rows by retaining source, and
deduplicates stream path rows by declared path identity.
machine_guid only when identities match.Agent:
UI:
Aggregator:
__topology_mode=aggregated to detailed on fanout only when the
producer supports it.machine_guid and table policy.