ContextQMD
Back to Netdata

topology-ip-intel-downloader

src/go/tools/topology-ip-intel-downloader/README.md

2.11.06.7 KB
Original Source
<!-- markdownlint-disable-file MD013 MD043 -->

topology-ip-intel-downloader

topology-ip-intel-downloader builds the Netdata IP intelligence databases used by topology and netflow enrichment.

It always writes a fixed Netdata output set:

  • ASN MMDB: topology-ip-asn.mmdb
  • GEO MMDB: topology-ip-geo.mmdb
  • metadata JSON: topology-ip-intel.json

The GEO MMDB is not country-only. It can contain:

  • country.iso_code
  • city.names.en
  • region
  • subdivisions
  • location.latitude
  • location.longitude

Both MMDB outputs also include Netdata classification metadata under netdata.* for CIDRs that must be tracked individually.

Stock databases and refreshed databases

Netdata packages ship a stock payload under:

  • ${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-asn.mmdb
  • ${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-geo.mmdb
  • ${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-intel.json

Refreshed databases generated by this downloader are written to:

  • ${NETDATA_CACHE_DIR}/topology-ip-intel

The netflow plugin auto-detects the cache copy first and then falls back to the stock copy. This means:

  • packaged stock files provide out-of-box enrichment
  • downloader runs update the cache copy
  • the refreshed cache copy overrides the packaged stock copy automatically

Important:

  • source installs from a Git checkout do not carry the packaged stock MMDBs
  • packaged builds stage the stock payload from CI/release tooling
  • local/source installs should run the downloader if they want a local cache copy
  • 32-bit packaged builds still ship the staged stock payload, but do not build or install the downloader binary

Config lookup

If --config is not provided, the downloader loads the first existing file from:

  1. /etc/netdata/topology-ip-intel.yaml
  2. /usr/lib/netdata/conf.d/topology-ip-intel.yaml

These paths are resolved from the compiled Netdata install layout, so prefixed installs use their own prefix automatically.

Common CLI usage

The CLI is optimized for the common case:

  • --asn provider:artifact[@format]
  • --geo provider:artifact[@format]
  • --no-asn
  • --no-geo

Rules:

  • --asn affects only ASN sources
  • --geo affects only GEO sources
  • repeated flags are ordered by precedence
  • first source wins when address ranges overlap
  • --no-asn or --no-geo removes that output file if it exists

Examples:

Use the built-in defaults:

bash
topology-ip-intel-downloader

Refresh both families explicitly from DB-IP:

bash
topology-ip-intel-downloader \
  --asn dbip:asn-lite \
  --geo dbip:city-lite

Use public ASN and country providers:

bash
topology-ip-intel-downloader \
  --asn iptoasn:combined \
  --geo ip2location:country-lite

Use MaxMind GeoLite2 sources when MAXMIND_LICENSE_KEY is available in the environment:

bash
MAXMIND_LICENSE_KEY="..." topology-ip-intel-downloader \
  --asn maxmind:geolite2-asn \
  --geo maxmind:geolite2-country

Use CAIDA RouteViews prefix2as for ASN-only attribution:

bash
topology-ip-intel-downloader \
  --asn caida:prefix2as \
  --geo dbip:country-lite

Prefer a custom GEO source, then fall back to DB-IP:

bash
topology-ip-intel-downloader \
  --geo iptoasn:combined \
  --geo dbip:city-lite

Disable GEO output entirely for one run:

bash
topology-ip-intel-downloader --no-geo

The downloader prints the effective execution plan before downloading, including:

  • effective ASN source order
  • effective GEO source order
  • which output files will be written
  • which output files will be removed

Advanced config

The config file uses ordered sources[] entries. Each entry is explicit about:

  • family: asn or geo
  • provider
  • artifact
  • format
  • optional url or path

Built-in DB-IP sources can omit url and path. The downloader resolves the current monthly download URL from the official DB-IP landing page.

Built-in MaxMind sources require MAXMIND_LICENSE_KEY in the environment. The generated metadata redacts URL query strings so license keys are not written to topology-ip-intel.json.

User-provided databases can point directly to:

  • a local file with path
  • a direct URL with url

Example:

yaml
sources:
  - name: custom-asn
    family: asn
    provider: dbip
    artifact: asn-lite
    path: /srv/ip-intel/custom-asn.mmdb

  - name: fallback-asn
    family: asn
    provider: dbip
    artifact: asn-lite
    format: mmdb

  - name: custom-geo
    family: geo
    provider: dbip
    artifact: city-lite
    url: https://example.internal/geo.mmdb.gz
    format: mmdb

output:
  directory: /var/cache/netdata/topology-ip-intel
  asn_file: topology-ip-asn.mmdb
  geo_file: topology-ip-geo.mmdb
  metadata_file: topology-ip-intel.json

Important:

  • source order is per family
  • earlier entries win on overlap
  • removing every source of a family disables that family
  • when a family is disabled, its stale output file is deleted on the next run

Supported built-in sources

ASN sources:

  • dbip:asn-lite (mmdb, csv)
  • iptoasn:combined (tsv)
  • caida:prefix2as (tsv)
  • maxmind:geolite2-asn (mmdb, requires MAXMIND_LICENSE_KEY)

GEO sources:

  • dbip:country-lite (mmdb, csv)
  • dbip:city-lite (mmdb, csv)
  • iptoasn:combined (tsv, country only)
  • maxmind:geolite2-country (csv, requires MAXMIND_LICENSE_KEY, country only)
  • ip2location:country-lite (csv, country only)
  • ipdeny:country-zones (cidr, country only)
  • ipip:country (txt, country only)

Supported formats:

  • mmdb
  • csv
  • tsv for iptoasn:combined and caida:prefix2as
  • cidr for ipdeny:country-zones
  • txt for ipip:country

The downloader accepts direct file/URL config for advanced cases, but the CLI source tokens are intentionally focused on built-in source families.

Output behavior

Every successful run stages all files privately and then publishes each final output atomically with rename().

This gives two guarantees:

  • readers never see a half-written MMDB file
  • the final visible output set matches the currently enabled families

If a family is disabled:

  • its final MMDB file is removed
  • metadata is rewritten to describe the new output set

Preparing a local stock payload staging directory

From the repository root:

bash
./src/go/tools/topology-ip-intel-downloader/refresh-stock.sh

This generates a local staging directory at:

  • ./artifacts/topology-ip-intel-stock/README.md
  • ./artifacts/topology-ip-intel-stock/topology-ip-asn.mmdb
  • ./artifacts/topology-ip-intel-stock/topology-ip-geo.mmdb
  • ./artifacts/topology-ip-intel-stock/topology-ip-intel.json

This is useful for:

  • checking the current packaged stock payload locally
  • preparing a staged payload for package/release work
  • verifying DB-IP downloads without storing generated binaries in Git