netflow-plugin

Rust NetFlow/IPFIX/sFlow ingestion and query plugin.

It stores flow entries in journal tiers under the Netdata cache directory and exposes flows:netflow.

Offline Function test mode

Fixture harnesses can execute the Function query path directly against an existing NetFlow backend directory:

netflow-plugin --test flows:netflow --dir <flows-dir> [--timeout <seconds>] [--no-persist] < payload.json

Requirements:

• <flows-dir> is the NetFlow backend root containing the raw, 1m, 5m, and 1h tier directories.
• stdin is the JSON Function request body (non-empty, maximum 16 MiB).
• --request is not supported and fails with usage output.
• --timeout <seconds> controls the offline Function execution timeout. It defaults to 30; use --timeout 0 to map to a very large finite timeout for long-running fixture comparisons.
• stdout contains only the raw JSON Function response.
• errors are written to stderr and return non-zero.

Use --no-persist for shared fixture datasets. It prevents the test run from writing facet state or sidecar files under <flows-dir> while keeping facet data in memory for the Function response. Without --no-persist, the plugin may refresh facet state under the backend directory, matching normal runtime behavior.

Function output includes volatile fields such as collection timestamps and runtime statistics. Test harnesses should normalize those fields before comparing fixture outputs.

Configuration

When running under Netdata, config is loaded from netflow.yaml in:

• ${NETDATA_USER_CONFIG_DIR}/netflow.yaml (preferred)
• ${NETDATA_STOCK_CONFIG_DIR}/netflow.yaml (fallback)

Top-level toggle:

• enabled: true|false controls the netflow plugin itself.

If journal.journal_dir is relative (default: flows), it is resolved against NETDATA_CACHE_DIR. With the standard cache directory this becomes:

• /var/cache/netdata/flows/raw
• /var/cache/netdata/flows/1m
• /var/cache/netdata/flows/5m
• /var/cache/netdata/flows/1h

If enrichment.geoip does not define explicit MMDB paths, the plugin auto-detects packaged databases in this order:

• ${NETDATA_CACHE_DIR}/topology-ip-intel
• ${NETDATA_STOCK_DATA_DIR}/topology-ip-intel

netdata-plugin-netflow now ships a stock MMDB seed set under ${NETDATA_STOCK_DATA_DIR}/topology-ip-intel. Freshly downloaded data written by topology-ip-intel-downloader stays in ${NETDATA_CACHE_DIR}/topology-ip-intel and overrides the stock copy automatically.

Important:

• packaged installs ship the stock MMDB payload
• source installs from a Git checkout do not include the generated stock MMDBs by default
• local/source installs should run topology-ip-intel-downloader if they want a local cache copy
• packaged 32-bit installs still ship the stock MMDB payload, but do not include topology-ip-intel-downloader

protocols.decapsulation_mode

Controls packet decapsulation for datalink payload parsing:

• none (default): keep outer header view.
• srv6: enable SRv6 decapsulation for supported payloads.
• vxlan: enable VXLAN decapsulation for supported payloads.

protocols.timestamp_source

Controls which timestamp is written as _SOURCE_REALTIME_TIMESTAMP for decoded flows:

• input (default): packet receive time at ingestion.
• netflow_packet: NetFlow/IPFIX packet export timestamp.
• netflow_first_switched: first-switched timestamp from flow fields when available.

enrichment (Akvorado-style static metadata and sampling)

Optional enrichment is applied at ingestion time. When enabled, it follows Akvorado core behavior for static metadata and sampling defaults/overrides:

• metadata lookup by exporter prefix + interface index
• sampling override/default by exporter prefix
• exporter/interface classifier rules (exporter_classifiers, interface_classifiers)
• classifier cache TTL (classifier_cache_duration, default 5m, minimum 1s)
• routing static lookup for AS/mask/next-hop/path/community fields (routing_static)
• routing dynamic lookup from BMP updates (routing_dynamic.bmp)
• BMP decode-error tolerance is configurable (routing_dynamic.bmp.max_consecutive_decode_errors)
• network attribute lookup by source/destination IP prefix (networks)
• GeoIP lookup from local MMDB files (geoip)
• periodic remote network source refresh (network_sources)
• network attribute merge order: GeoIP base, then remote network_sources, then static networks
• static networks supernet/subnet inheritance (more specific prefixes override non-empty fields)
• AS and net provider order controls (asn_providers, net_providers)
• flows are dropped when metadata/sampling requirements are not met
• when both static and dynamic routes match, dynamic routes are preferred

Example:

enrichment:
  classifier_cache_duration: 5m
  default_sampling_rate:
    192.0.2.0/24: 1000
  override_sampling_rate:
    192.0.2.128/25: 4000
  metadata_static:
    exporters:
      192.0.2.0/24:
        name: edge-router
        region: eu
        role: peering
        tenant: tenant-a
        site: par
        group: blue
        default:
          name: Default0
          description: Default interface
          speed: 1000
        if_indexes:
          10:
            name: Gi10
            description: 10th interface
            speed: 1000
            provider: transit-a
            connectivity: transit
            boundary: external
  geoip:
    asn_database:
      - /usr/share/GeoIP/GeoLite2-ASN.mmdb
    geo_database:
      - /usr/share/GeoIP/GeoLite2-City.mmdb
    optional: true
  networks:
    198.51.100.0/24:
      name: customer-a
      role: customer
      site: par1
      region: eu-west
      country: FR
      state: Ile-de-France
      city: Paris
      tenant: tenant-a
      asn: 64500
    203.0.113.0/24: transit-a
  network_sources:
    amazon:
      url: "https://ip-ranges.amazonaws.com/ip-ranges.json"
      method: GET
      tls:
        enable: true
        verify: true
        ca_file: ""
        cert_file: ""
        key_file: ""
      interval: 10m
      timeout: 30s
      transform: |
        (.prefixes + .ipv6_prefixes)[] |
        { prefix: (.ip_prefix // .ipv6_prefix), tenant: "amazon", region: .region, role: .service|ascii_downcase }
      headers:
        X-Example: "value"
  asn_providers: [flow, routing, geoip]
  net_providers: [flow, routing]
  routing_static:
    prefixes:
      198.51.100.0/24:
        asn: 64600
        as_path: [64550, 64600]
        communities: [123456, 654321]
        large_communities:
          - asn: 64600
            local_data1: 7
            local_data2: 8
        next_hop: 203.0.113.9
  routing_dynamic:
    bmp:
      enabled: true
      listen: "0.0.0.0:10179"
      receive_buffer: 0 # bytes; 0 keeps kernel default
      max_consecutive_decode_errors: 8
      rds: ["0", "65000:100", "192.0.2.1:42"]
      collect_asns: true
      collect_as_paths: true
      collect_communities: true
      keep: 5m
    bioris:
      enabled: false
      ris_instances: []
      timeout: 200ms
      refresh: 30m
      refresh_timeout: 10s

routing_dynamic.bioris is implemented as stream-based route ingestion:

• the plugin periodically discovers routers with GetRouters
• it consumes DumpRIB streams per router/AFI for baseline reconciliation
• it keeps ObserveRIB streams per router/AFI for incremental updates between refreshes
• enrichment keeps lookup local/in-memory (no per-flow remote RPC on the ingestion hot path)

network_sources.*.transform accepts jq expressions (compiled/executed via jaq) and should emit objects with fields compatible with:

• prefix (required)
• name, role, site, region, country, state, city, tenant, asn, asn_name (optional)

network_sources.*.tls follows Akvorado-style source TLS controls:

• enable toggles TLS settings for this source.
• verify must remain true; disabling TLS certificate verification is not supported.
• skip_verify is accepted only for compatibility with earlier drafts and must remain false.
• ca_file sets a custom CA bundle.
• cert_file and key_file set optional client identity (if key_file is empty, cert_file is reused).

Example:

enabled: true

listener:
  listen:
    - "0.0.0.0:2055"
    - "0.0.0.0:6343"

protocols:
  v5: true
  v7: true
  v9: true
  ipfix: true
  sflow: true
  decapsulation_mode: srv6
  timestamp_source: input

journal:
  journal_dir: flows
  tiers:
    raw:
      size_of_journal_files: 200GB
      duration_of_journal_files: 24h
    minute_1:
      size_of_journal_files: 40GB
      duration_of_journal_files: 14d
    minute_5:
      size_of_journal_files: 30GB
      duration_of_journal_files: 30d
    hour_1:
      size_of_journal_files: 20GB
      duration_of_journal_files: 365d
  query_max_groups: 50000

Standalone CLI runs still accept the legacy uniform retention flags --netflow-retention-size-of-journal-files and --netflow-retention-duration-of-journal-files. They apply the same value to all tiers and exist only for standalone/CLI compatibility; YAML configuration is per-tier.

query_max_groups caps the number of distinct group keys a single aggregation query may build before extra groups are folded into a synthetic __overflow__ bucket. The response carries a warning when this happens. The limit protects the query worker from accidentally wide group-by combinations exhausting memory.

Journal rotation size is not user-configured. The plugin derives it per tier:

• if size_of_journal_files is set, rotation size is clamp(size / 20, 5MB, 200MB)
• size_of_journal_files must be at least 100MB
• if size_of_journal_files is omitted or explicitly set to null, the plugin uses a fixed internal rotation size of 100MB
• the internal time-based rotation cadence remains 1h

The plugin exposes lightweight production health charts by default:

• netflow.open_tiers
  • currently open rollup rows by tier
• netflow.decoder_scopes
  • NetFlow v9 parser scopes
  • IPFIX parser scopes
  • legacy parser scopes
  • persisted decoder namespaces
  • hydrated namespace-source mappings
• netflow.facet_values
  • total published facet values
  • exposed facet values
• netflow.facet_fields
  • populated facet fields
  • autocomplete-backed facet fields
• netflow.tier_index_entries
  • indexed hours
  • indexed rollup flows

Absolute byte-level memory diagnostics are disabled by default because they read process memory maps and estimate in-memory structures. Enable them only when debugging memory growth:

charts:
  memory_diagnostics:
    enabled: true
    interval: 10s

These charts refresh on the configured interval. With the default 10s interval, short-lived memory spikes may be missed or appear stale between samples.

When enabled, the plugin also exposes:

• netflow.memory_resident_bytes
  • rss, hwm
  • rss_anon, rss_file, rss_shmem, anon_huge_pages
• netflow.memory_resident_mapping_bytes
  • resident heap bytes
  • anonymous non-heap mappings
  • raw journal mmap bytes
  • 1m journal mmap bytes
  • 5m journal mmap bytes
  • 1h journal mmap bytes
  • GeoIP ASN MMDB resident bytes
  • GeoIP geo/country MMDB resident bytes
  • other file-backed mappings
  • shmem mappings
• netflow.memory_allocator_bytes
  • heap_in_use, heap_free, heap_arena
  • mmap_in_use, releasable
• netflow.memory_accounted_bytes
  • facet runtime archived/active/contribution/published/path buckets
  • materialized tier indexes
  • open tier rows
  • GeoIP ASN MMDB resident bytes
  • GeoIP geo/country MMDB resident bytes
  • unaccounted process RSS remainder
• netflow.memory_tier_index_bytes
  • tier row storage
  • tier field dictionaries
  • tier lookup tables
  • tier schema/index metadata

These charts are intended for debugging memory explosions under high-cardinality traffic, not for billing or hard enforcement decisions.

journal.tiers configures retention independently for:

• raw
• minute_1 (aliases: 1m, minute-1, minute1)
• minute_5 (aliases: 5m, minute-5, minute5)
• hour_1 (aliases: 1h, hour-1, hour1)

If a tier is omitted, it uses the built-in tier default (10GB with no time-based age limit). There are no top-level journal retention knobs; set retention on each tier you want to tune.

size_of_journal_files budgets journal data plus finalized per-journal facet sidecars. The active journal counts but is protected from deletion, so usage can temporarily exceed the budget until rotation. Shared facet-state.bin, temporary sidecars, and unrelated files are outside the per-tier budget.

To make a tier time-only, set size_of_journal_files: null. To make a tier size-only, set duration_of_journal_files: null.

Performance benchmarking

The plugin ships two complementary benchmarks:

• cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release ingest::bench_tests::bench_ingestion_protocol_matrix -- --ignored --nocapture unpaced full UDP→journal max throughput per protocol, plus decode-only and post-decode phases
• cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release ingest::resource_bench_tests::bench_resource_envelope_child -- --ignored --nocapture paced post-decode resource envelope at a configurable rate, controlled via env vars: NETFLOW_RESOURCE_BENCH_PROTOCOL, NETFLOW_RESOURCE_BENCH_PROFILE, NETFLOW_RESOURCE_BENCH_LAYER, NETFLOW_RESOURCE_BENCH_FLOWS_PER_SEC, NETFLOW_RESOURCE_BENCH_WARMUP_SECS, NETFLOW_RESOURCE_BENCH_MEASURE_SECS, NETFLOW_RESOURCE_BENCH_SYNC_EVERY_ENTRIES, NETFLOW_RESOURCE_BENCH_SYNC_INTERVAL_MILLIS

The resource-envelope benchmark scope:

• pre-decoded flow records pushed into the ingest pipeline (UDP receive and protocol decode are excluded; measure decode separately with the protocol matrix benchmark)
• full pipeline active: raw journal + 1-minute + 5-minute + 1-hour tier accumulation, real disk-backed journals
• enrichment is NOT loaded (no GeoIP MMDB, no static metadata, no classifiers, no static networks); cardinality fields are pre-populated by the harness
• reports achieved flows/s, CPU% of one core, peak/final RSS, real disk read and write bytes/s from /proc/self/io
• NETFLOW_RESOURCE_BENCH_LAYER=production-shaped keeps production listener sync defaults, runs tier commits on worker threads, includes low-rate cases, and reports fixed overhead buckets for sync ticks, chart sampling, raw/tier syncs, tier flushes, and decoder-state persistence

cpu_percent_of_one_core is the sum of user+system ticks across all threads of the test process during the measurement window, divided by wall time, as a percent of one core. 100% means one core's worth of CPU was consumed; values above 100% are normal for multi-threaded saturation.

Note (2026-06): the recorded tables below predate the migration of the journal backend to systemd-journal-sdk (compact file format, no compression, periodic fsync disabled by default). Spot re-runs after the migration show roughly 20-40% higher throughput at every point (e.g. high-cardinality post-decode saturation moved from ~37k to ~43-46k flows/s, low-cardinality full ingest from ~85-95k to ~110-120k flows/s) and about half the physical disk write per flow (~400 bytes/flow). Re-run the commands above for current numbers on your host.

Reference measurements:

• CPU: 12th Gen Intel(R) Core(TM) i9-12900K
• storage: ext4 on Seagate FireCuda 530
• methodology: release mode, 5s warmup, 15s measurement window, disk-backed journals, post-decode paced ingest, all-tiers-batched layer

Paced post-decode resource envelope (3A)

Per protocol, per cardinality, at 10 offered rates from 100 to 60 000 flows/s. Cardinality is synthetic: low-cardinality cycles 256 unique records, high- cardinality cycles 4 096 unique records. Real exporter data sits between the two.

Low cardinality, NetFlow v9:

Low cardinality, IPFIX:

Low cardinality, sFlow:

High cardinality, NetFlow v9 (saturates around 30 000 flows/s):

High cardinality, IPFIX (saturates around 30-40 000 flows/s):

High cardinality, sFlow (saturates around 30 000 flows/s):

Unpaced full UDP→journal protocol matrix (3B)

Single-threaded peak throughput at native fixture cardinality:

Interpretation

• The post-decode ingest hot path is currently single-threaded. CPU pins at ~98-99% of one core at saturation; it does not scale further with more cores.
• Low-cardinality saturation is above 60 000 flows/s for NetFlow v9 and IPFIX, and around 70 000 flows/s for sFlow on this host (above 100 000 flows/s post-decode after the systemd-journal-sdk migration). The matrix above does not reach those ceilings on purpose; extrapolate from the CPU% column.
• High-cardinality saturation is around 30 000 flows/s post-decode for all three protocols (~43-46 000 flows/s after the migration). Above the knee, achieved rate stays at the plateau while offered rate grows.
• Adding decode (~10 µs/flow) on top of post-decode ingest brings the practical full-path ceiling to roughly 22-25 000 flows/s at high cardinality on this host (~40 000 flows/s after the migration), with the four-tier pipeline running and no enrichment. UDP socket receive is not measured; at these flow rates packet rate (1-5k pps) is well below typical socket limits.
• Disk reads stay near zero because the benchmark isolates the ingest path from query workloads. The journals themselves are indexed and rewrite pages during normal operation; this benchmark does not exercise that overhead at steady state because it only runs for the warmup + measurement window.
• Higher cardinality raises steady memory and per-flow encoding cost, lowering the throughput ceiling.
• These numbers are specific to this host. They do not include GeoIP/MMDB or any other enrichment; loading enrichment adds per-lookup CPU cost on top.

plugins.d protocol

When stdout is not a TTY (normal plugins.d runtime), netflow-plugin emits PLUGIN_KEEPALIVE periodically to avoid parser inactivity timeouts.