Topology Schema Implementation Scope

This document scopes the work needed to move Netdata topology producers, Cloud aggregation, and the Cloud UI to the production topology schema defined in FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md and FUNCTION_TOPOLOGY_SCHEMA.json.

It is not an implementation plan for one commit. It is the work map for the backend, frontend, producer, and aggregator changes.

Ground Rules

New topology producers emit only the new schema.
Superseded topology schema support is removed from Agent/backend contracts and docs.
Temporary compatibility support may exist only as an isolated Cloud frontend adapter during Agent rollout.
Production payloads carry canonical topology facts, not reconstruction instructions for compatibility payloads.
Actor/link modals are composed from schema-declared recipes over existing actors, links, evidence, detail tables, and actor labels. Production payloads must not duplicate high-cardinality rows only for modal display.
Test-only reconstruction/projection code may derive older shapes to prove information parity, but that code must not affect production payloads.
Raw payload captures from real systems stay under .local/ and are never committed.

Shared Backend Work

Function Contract

Required changes:

add topology validation against src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json;
update Function validator tooling to recognize the new topology schema;
make topology Function examples and tests use the new contract;
remove superseded topology-schema references from Agent/backend docs once producer migration lands.

Likely files:

src/go/tools/functions-validation/
src/plugins.d/FUNCTION_UI_REFERENCE.md
src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md
src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json
src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md

Shared Encoding Helpers

The schema uses compact columnar tables. Producers should not hand-roll table encoding repeatedly.

Required helpers:

table builder for rows / columns / values;
codecs for const, values, and dict;
string dictionary builder;
validation checks for column/value length;
deterministic sorting helpers for actors, links, and evidence rows;
size measurement hooks for tests.

Likely homes:

Go: src/go/pkg/topology/v1 or src/go/pkg/funcapi/
C: small helper module for network-viewer, or a local builder until a shared C helper is justified
Rust: SDK helper if a Rust topology producer is added

Current Migration Inventory

Agent Producers

topology:network-connections:

producer path: src/collectors/network-viewer.plugin/network-viewer.c;
the Function now emits netdata.topology.v1 at src/collectors/network-viewer.plugin/network-viewer.c:2535;
the Function parses aggregated / mode:aggregated and detailed / mode:detailed, with aggregated as the default, at src/collectors/network-viewer.plugin/network-viewer.c:272;
response metadata exposes the mode selector at src/collectors/network-viewer.plugin/network-viewer.c:1451;
actors, graph links, and optional socket evidence rows are emitted as compact columnar tables at src/collectors/network-viewer.plugin/network-viewer.c:2568;
socket evidence is emitted only in detailed mode at src/collectors/network-viewer.plugin/network-viewer.c:2571;
repeated string columns use automatic dictionary encoding when it is smaller than plain values at src/collectors/network-viewer.plugin/network-viewer.c:2041;
old-schema presentation metadata and actor-nested socket tables have been removed from the Agent producer. The v1 producer now emits compact graph-presentation metadata inside type definitions plus data.presentation. Actor modal socket lists must be derived from evidence by the Cloud frontend/aggregator during rollout.
modal-composition producer work now emits actor_labels, process username, process cmdline, self local_ip_count, socket-port inventory, and modal recipes. Remaining work is integrated UI/aggregator QA.

topology:streaming:

producer paths: src/web/api/functions/function-topology-streaming.c and src/streaming/stream-path.c;
the Function now emits netdata.topology.v1 directly at src/web/api/functions/function-topology-streaming.c:1870;
actors, graph links, link evidence, and actor-detail tables are emitted as compact tables at src/web/api/functions/function-topology-streaming.c:1912;
streaming path rows are preserved as an actor_detail table at src/web/api/functions/function-topology-streaming.c:1255;
inbound and outbound drilldown rows are declared as relationship summaries at src/web/api/functions/function-topology-streaming.c:1259;
streaming, virtual, and stale links have explicit directed link-type metadata and separate evidence type ids at src/web/api/functions/function-topology-streaming.c:1226;
modal-composition producer work now emits actor_labels, complete host labels where available, host/system metadata labels, OS/architecture/CPU fields, link metric columns, and modal recipes. Remaining streaming work is parity/UX validation with the Cloud frontend and Cloud aggregator once those parallel workers are ready.

topology:snmp:

producer paths: src/go/plugin/go.d/collector/snmp_topology/ and src/go/pkg/l2topology/;
the L2 engine builds an internal, non-payload l2topology.Graph projection from l2topology.Result;
the Function handler snapshots cached SNMP topology state, applies SNMP-owned shape/enrichment passes, and renders netdata.topology.v1 through src/go/plugin/go.d/collector/snmp_topology/internal/topologyv1;
the old method-level Go presentation adapter has been retired; presentation metadata is emitted in the v1 payload type registry and data.presentation;
SNMP-owned logical enrichment now adds L3 subnet, OSPF adjacency, and BGP adjacency links after the L2 projection and before v1 rendering;
current L2 emission uses directions such as bidirectional and unidirectional at src/go/pkg/l2topology/topology_adapter_segments_builder_emit.go:60 and src/go/pkg/l2topology/topology_adapter_projection_pairs.go:230;
migration target: use observed_bidirectional or unordered aggregation policy where discovery direction is noise, preserve LLDP/CDP/FDB/ARP/STP evidence, keep interface inventory as actor detail/inventory, and move metric query definitions to overlay templates/refs.

vSphere:

producer path: src/go/plugin/go.d/collector/vsphere/;
the Function emits netdata.topology.v1 directly from the Go collector;
actor identity uses the vSphere managed-object type plus managed-object id;
inventory containment is modeled as hierarchical ownership links;
VM-to-host and host/VM-to-network relationships are graph links with typed evidence.

Cloud Frontend

The Cloud frontend compatibility work is outside this repository, but the schema rollout depends on it:

current topology fetch normalizer decodes every topology payload through normalizeTopologyPayload(response?.data || {}) and then computes render-time aggregated links at ${CLOUD_FRONTEND_REPO}/src/domains/functions/useFetch/normalizers/topology/index.js:9;
current frontend graph aggregation groups by source, target, and link type, canonicalizing reverse links if already seen, at ${CLOUD_FRONTEND_REPO}/src/domains/functions/topology/graphAggregation.js:58;
current actor modal code still branches on presentation table source values, including source === "links", at ${CLOUD_FRONTEND_REPO}/src/domains/functions/components/topology/actorModal/index.js:286;
migration target: add a new-schema decoder for compact tables, keep old-schema support isolated in one temporary adapter, derive actor drilldown relationship tables from evidence rows, render actor custom tables from typed actor-detail tables, and use link-type direction metadata instead of guessing from raw link direction strings.
zero-heuristic v1 rendering target: read actor size scale, actor repulsion, actor search policy, link semantic role, and closed icon tokens from the v1 type registry. Keep isSelfNode, isDerivedSegmentNode, isDeviceNode, LLDP/CDP protocol checks, capability icon inference, and hardcoded search paths inside the temporary legacy adapter only.

Producer Migration Scope

`topology:network-connections`

Producer path:

src/collectors/network-viewer.plugin/network-viewer.c

Required behavior:

emit actors as compact actor table rows;
emit graph links as three semantic families:
- node-to-process ownership links that keep each node cluster together;
- local process-to-process links when both process endpoints are known;
- process-to-correlation-endpoint links for unresolved or cross-node socket endpoints;
emit pure correlation endpoint actors plus data.correlation.points and data.correlation.claims rows for socket tuple resolution;
emit one socket evidence row per socket tuple needed for cross-node matching;
default to aggregated graph projection while preserving detailed evidence;
support aggregation scopes prepared for node, process name, PID, container, and Kubernetes workload labels as enrichment becomes available;
omit compatibility per-row display strings, duplicated labels, and actor modal socket tables from production payload;
keep current metrics optional and separate from topology identity.

Validation:

compare against captured corpus under .local/;
prove no truncation on large socket counts;
assert payload size at corpus scale;
assert exact reverse-tuple matching inputs remain present.

Current state:

src/collectors/network-viewer.plugin/network-viewer.c now emits compact actor rows, graph-link rows, and optional socket evidence rows directly in netdata.topology.v1;
aggregated mode is the default and omits socket evidence from the response;
detailed mode keeps socket evidence as a shared relationship-evidence table, not as actor-owned duplicated modal data;
link and evidence string columns choose dictionary encoding only when it reduces raw payload size;
PR #22496 semantic-link split and correlation endpoint/point/claim emission are implemented in the Agent producer;
remaining network-connections work is corpus-scale validation with captured Cloud payloads and Cloud/frontend integration once the parallel workers are ready.

`topology:streaming`

Producer paths:

src/web/api/functions/function-topology-streaming.c
src/streaming/stream-path.c

Required behavior:

emit streaming agents as actors;
emit parent/child streaming relationships as directed dependency links;
classify stream_path as actor detail, not relationship evidence;
keep retention and relationship summaries as typed detail tables;
make direction semantics explicit through link type definitions.

Validation:

preserve current actor modal data through new actor-detail tables;
prove graph links and custom actor tables are not conflated;
use fixtures from current streaming topology tests where possible.

Current state:

src/web/api/functions/function-topology-streaming.c now emits netdata.topology.v1 directly from the C Function;
actor rows, link rows, relationship evidence, stream_path, retention, inbound, and outbound tables are compact columnar sections;
stale stream-path hops remain signed values instead of being coerced to unsigned values;
streaming, virtual, and stale graph-link types have separate evidence types so link-type metadata and evidence metadata agree.
streaming graph-presentation metadata is emitted inside type definitions plus data.presentation, including highlight-path selection, legend, link styles, and graph port-bullet tokens.

`topology:snmp`

Producer paths:

src/go/plugin/go.d/collector/snmp_topology/
src/go/pkg/l2topology/

Required behavior:

emit devices, interfaces, bridge domains, VLANs, and endpoints as actor rows;
emit L2 adjacencies with direction policy canonicalize_unordered when direction is discovery noise;
preserve LLDP/CDP/FDB/ARP/STP facts as evidence or actor inventory depending on role;
move interface traffic/errors/state metric pointers to overlay templates and overlay refs;
avoid copying metric query fragments on every link.

Validation:

reuse existing SNMP topology golden fixtures;
add schema-level golden fixtures for devices, interfaces, ports, and bidirectional adjacency merge;
verify overlay refs can query interface metrics without recomputing topology.

Current state:

Function payload rendering is implemented in src/go/plugin/go.d/collector/snmp_topology/internal/topologyv1;
L2 graph synthesis is internal to src/go/pkg/l2topology and uses l2topology.Graph, not the legacy Go topology payload package;
the renderer emits compact actor, link, evidence, actor-label, actor-detail, and relationship-evidence tables and preserves nested custom actor cells with json columns where needed;
SNMP topology is internally split into collector/cache/registry code plus internal/topologymodel, internal/topologyoptions, internal/topologyshape, internal/topologyenrich, and internal/topologyv1; see src/go/plugin/go.d/collector/snmp_topology/ARCHITECTURE.md for the current maintainer map;
modal-composition producer work now emits actor_labels, promoted scalar/count actor fields, stable actor_ports rows, structured endpoint evidence, modal recipes, and payload-level presentation metadata. Remaining SNMP work is to migrate metric lookup fragments into first-class overlay templates/refs instead of only preserving them in actor/detail data, plus integrated UI/aggregator QA.

vSphere Topology

Producer path:

src/go/plugin/go.d/collector/vsphere/

Required behavior:

update the vSphere topology producer to the new schema in place;
use stable vSphere managed object ids as actor identity where available;
model inventory containment with hierarchical ownership links;
represent VM-to-host, cluster-to-host, datastore, and network relationships as graph links plus typed evidence where needed;
use overlay templates for refreshable utilization/state metrics.

Current state:

vSphere emits netdata.topology.v1 directly from src/go/plugin/go.d/collector/vsphere/func_topology.go;
the producer builds compact actor, link, evidence, actor-detail, and actor_labels tables with src/go/pkg/topology/v1;
actor identity uses vSphere managed-object type plus managed-object id;
containment, VM-to-host, and network relationships have explicit link types, direction roles, and evidence types;
the old method-level Go presentation adapter is retired. Presentation metadata lives in the v1 type registry and data.presentation.

Cloud Frontend Scope

Required changes:

add a decoder for the compact table schema;
build graph nodes from the actors table;
build graph edges from the links table;
derive actor drilldown relationship tables from evidence rows;
render actor custom tables from typed actor-detail tables;
decode and execute presentation.modal recipes for actor/link modals;
render actor_labels as actor labels instead of raw metadata JSON;
reuse existing topology modal/table components where practical, extending them for v1 projections rather than building a separate v1 table stack;
use link type direction metadata to decide whether links are directed, undirected, hierarchical, or observation-only;
use overlay templates and refs for metric refreshes;
isolate compatibility support in one temporary adapter;
delete the temporary adapter after Agent rollout.

Likely frontend areas:

topology payload normalizer;
graph aggregation layer;
actor modal tables;
link details;
telemetry overlay query layer;
Function response version detection.

Frontend risks:

decoding large columnar sections synchronously can still block the main thread; use streaming, workers, or chunked decode if needed;
mixed Agent versions need clear adapter selection;
actor modal tables must not duplicate evidence in memory unnecessarily.
v1 actor modals can regress visually if they bypass the existing table, port-table, labels, and navigation components. Component reuse is part of the frontend migration, not just a cleanup preference.

Cloud Aggregator Scope

The aggregator should be implemented in Go as a separate Cloud component or service, not inside charts-service request routing.

The MVP aggregator must support all topology kinds covered by the production schema contract. topology:network-connections remains the required high-cardinality benchmark, but it is not an acceptable production boundary by itself. The Cloud UI should not need separate aggregation paths for different topology kinds.

Inputs

one or more netdata.topology.v1 payloads;
requested aggregation scope, such as node, process name, container, Kubernetes workload labels, vSphere object type, or SNMP device/interface;
optional filters such as layer, link type, actor type, room, or node set.

Outputs

a netdata.topology.v1 payload with:
- merged actor rows;
- merged graph links;
- resolved correlation output as normal actors and links, with no exposed aggregator internal states;
- preserved or counted evidence rows according to schema policy;
- merged detail tables according to table type policy;
- preserved and remapped modal/table presentation recipes;
- merged actor labels according to actor table policy;
- merged overlay refs according to overlay template policy;
- stats describing input rows, output rows, evidence rows, and drops/errors.

Core Packages

Suggested package split:

schema: generated or hand-written Go structs for the topology schema;
codec: compact table decode/encode helpers;
model: canonical in-memory actors, links, evidence, tables, overlays;
aggregate: scope-based actor/link/evidence merge logic;
match: declarative correlation-key normalization, priority handling, exact and partial match resolution, and exact tuple matching;
validate: schema and semantic validation;
fixtures: sanitized corpus and synthetic scale fixtures.

Aggregation Logic

Required behavior:

merge actors by the requested scope and actor type identity;
apply data.correlation.rules without hardcoding topology-kind-specific key names in the aggregator;
remove pure correlation actors only for exact unambiguous absorb matches, rewiring incident correlation links to the matched actor with the rule's output_link_type;
keep correlation actors visible for no-match, ambiguous, and link partial matches, emitting weak semantic correlation links for visible partial matches;
preserve evidence rows when evidence policy is preserve;
count evidence rows when evidence policy is count;
preserve modal composition definitions and rewrite their type, table, evidence, and column references after namespacing/deduplication;
do not materialize modal rows during aggregation unless the underlying canonical table is already being merged;
merge actor_labels after actor reference remapping and preserve repeated values as repeated rows; string and string_ref label columns are equivalent logical strings and must be normalized before label deduplication;
never silently truncate evidence;
fail explicitly when a requested payload would exceed configured limits;
canonicalize undirected links only when link type policy allows it;
preserve directed links when direction is flow, dependency, or ownership;
merge overlay refs with set or append semantics defined by templates.

Network socket matching:

exact reverse-tuple matching should be expressed through the generic correlation contract using process claims, endpoint points, correlation link types, rule priorities, and output link types;
NAT, load balancer, and proxy inference are out of scope for the first aggregator, but later NAT evidence can add extra point/claim rows for the same rule without changing the aggregator's key-building mechanism;
unresolved endpoints can aggregate by visible endpoint identity, but the evidence row must remain available when the requested mode preserves it.

Limits And Failure Behavior

The aggregator must have explicit limits:

maximum decoded bytes;
maximum actor rows;
maximum graph links;
maximum evidence rows;
maximum output bytes;
maximum CPU time per request.

If a limit is exceeded:

return a structured error;
include stats showing which limit was exceeded;
do not return a truncated topology as if it were complete.

Paged or chunked evidence transport remains a phase-2 option. Phase 1 should make payloads small enough that this is rarely needed.

Tests

Required test classes:

schema decode/encode round-trip;
semantic validation failures;
actor identity merge by scope;
directed vs undirected link aggregation;
relationship evidence preservation;
actor-detail table aggregation;
actor-label table aggregation;
modal presentation recipe preservation and reference rewriting;
overlay ref merge;
network socket exact reverse-tuple matching;
streaming hierarchy and actor-detail custom tables;
SNMP/L2 unordered adjacency and observation evidence;
vSphere ownership/dependency topology;
generic schema-conformant custom topology passthrough;
synthetic scale benchmark near and above current corpus scale;
sanitized real-corpus replay from .local/ promoted only as non-sensitive fixtures when safe.

Rollout Plan

Land schema docs, developer project skill, and implementation scope.
Add validator support and compact-table helpers.
Add Cloud frontend new-schema decoder and temporary compatibility adapter so mixed Agent rollout is safe before producers emit the new schema broadly.
Migrate producers behind tests. topology:network-connections should be the first high-cardinality producer exercised internally, but it is not the production boundary for Cloud aggregation.
Migrate the streaming producer and complete SNMP overlay-template refinement.
Coordinate and migrate the vSphere topology producer.
Build cloud-topology-service in parallel against fixtures and new-schema payloads. Its MVP is complete only when all topology kinds covered by this contract pass service-level aggregation tests.
Hand final service ownership, environment-specific Helm values, deployment targets, and production node-instance routing strategy to Cloud backend and DevOps once the service is otherwise ready for operational integration.
Remove compatibility support from Cloud frontend after supported Agent rollout.

Resolved Phase-1 Defaults

Cloud aggregator service repository: cloud-topology-service.
Cloud aggregated topology route: POST /api/v3/spaces/{spaceID}/rooms/{roomID}/topology.
Cloud service contract: accepts and emits only netdata.topology.v1.
Phase-1 topology service MVP: all topology kinds covered by this contract, not only topology:network-connections.
Network socket snapshot metrics such as RTT and retransmissions: opt-in, not default core topology columns.
Cloud-side topology payload cache: no payload cache in phase 1; aggregate on demand and collect request-cost metrics first.
Service-local validation package is sufficient for the Cloud service MVP; producer CI may still add a separate validator binary later if needed.

External Integration Gates

These items cannot be safely invented from this repository and must be handed to Cloud backend and DevOps when cloud-topology-service is otherwise ready for operational integration:

final service owner and CODEOWNERS entries;
environment-specific Helm values and deployment targets;
approved production node-instance routing strategy.