AWS CloudWatch Collector Architecture

Maintainer-oriented map of cloudwatch, written to be read top to bottom as a journey: the plain model and the big picture first, then the lifecycle and the collection cycle, then each stage in detail, and finally the profile schema, invariants, and where to change things. Per-service metric and chart details live in the profile YAMLs (config/go.d/cloudwatch.profiles/) and the focused tests, not here.

The one rule that shapes most of what follows:

GetMetricData is billed by requested metrics, so the collector normally queries each series on the first shared update_every tick that observes a newer aligned effective-period window, and re-emits cached values in between. Transient failures use bounded per-query backoff. When period < update_every, collection ticks also quantize successful execution and intermediate eligible windows can be skipped.

What It Does

cloudwatch is a framework-V2 go.d collector that pulls AWS CloudWatch metrics for a curated set of AWS services and renders them as dynamic Netdata charts. It is profile-driven: each service is a YAML profile (see Profiles) declaring its CloudWatch namespace, the dimension set that identifies one instance, the metrics/statistics to query, and a chart template — so adding or adjusting a service is usually a YAML edit, not Go.

It does not generate per-account or per-resource nodes: AWS resources are chart instances keyed by by_labels. All output follows the job's configured vnode, when present, otherwise the Agent host; the collector creates no host scopes of its own.

Big Picture

Config picks the services; the collector discovers what actually exists when a profile has identifying dimensions, queries each resulting instance on a per-period schedule, and renders the results as charts under the cloudwatch. namespace. A profile whose dimensions are all constants already describes one complete instance, so compilation creates that static instance directly.

flowchart LR
    cfg("Config
credentials · targets · ordered rules")
    static("Static instance
all dimensions known")
    disc("Discover dynamic instances
ListMetrics
cheap / free tier")
    tags("Resolve tags
membership + labels")
    plan("Plan + state
stable query + policy")
    query("Query
GetMetricData
billed — the cost driver")
    activity("Collector activity
SDK invocations · calculated units · profile estimates · query items")
    store("metrix store
gauges + labels")
    charts("Dynamic charts
cloudwatch.*")

    cfg --> static --> plan
    cfg --> disc --> tags --> plan --> query --> store --> charts
    disc --> activity --> store
    query --> activity

    classDef input fill:#eef1f4,stroke:#8b949e,color:#24292f;
    classDef sched fill:#ece2ff,stroke:#8250df,color:#3b1f6b;
    classDef effect fill:#ffe6cc,stroke:#e36209,color:#5a2a00;
    classDef commit fill:#d8f3e0,stroke:#2da44e,color:#0b3d1f;

    class cfg input
    class static,disc,tags,plan sched
    class query effect
    class store,charts commit

Each collection cycle (collect.go), in order:

1. compile the raw configuration and resolve one AWS account id per target (plan.go, identity.go);
2. use compiler-created instances for all-constant profiles; for dynamic profiles, share one ListMetrics scan per (target, region, namespace) and match each participating profile against that stream (discover.go);
3. resolve resource-tag membership and optional chart labels (tags.go);
4. reuse or rebuild the stable per-series query blueprint (query_plan.go);
5. execute the due queries in concurrent batches (query_executor.go);
6. apply explicit per-query outcomes to completion, retry, and source-observation state (observe.go);
7. write retained observations or synthetic zero presentation as float gauges into metrix, stamped with {account_id, region, <dimension labels>}, and re-emit not-due series (query_emit.go);
8. publish absolute collector activity for the commit-acknowledged interval (activity.go);
9. serve a chart template built once from the selected profiles plus one collector-activity group (chart.go).

Lifecycle

Framework V2: Init → Check → repeated Collect; there is no background Run (the collector does not implement CollectorV2Runner). collector.go owns the Collector struct and lifecycle.

flowchart TD
    Init("Init
applyDefaults + validate
no AWS calls")
    Check("Check")
    Runtime("ensurePlan
load catalog · compile rules
build + cache chart template")
    Targets("ensureTargets
concurrent STS GetCallerIdentity
fail-soft · retry pending")
    Post("framework postCheck
validate ChartTemplateYAML via chartengine")
    Collect("Collect
every update_every")
    Cycle("collect cycle
discover → tags → plan → schedule → execute → observe")

    Init --> Check
    Check --> Runtime --> Targets --> Post
    Post --> Collect --> Cycle --> Collect

    classDef input fill:#eef1f4,stroke:#8b949e,color:#24292f;
    classDef sched fill:#ece2ff,stroke:#8250df,color:#3b1f6b;
    classDef commit fill:#d8f3e0,stroke:#2da44e,color:#0b3d1f;
    classDef core fill:#cfe4ff,stroke:#1f6feb,stroke-width:3px,color:#0b2948;

    class Init,Check input
    class Runtime,Targets sched
    class Post commit
    class Collect,Cycle core

• Init does config only (applyDefaults, validate) — no network.
• Check compiles the configuration, builds the chart template, and resolves up to 64 target account IDs concurrently through STS. The template is built here because the framework's postCheck validates ChartTemplateYAML() through chartengine.LoadYAML before the first Collect; an unbuilt template would fail the job.
• Collect runs the whole cycle (below). ensurePlan short-circuits once compiled; ensureTargets retries only unresolved targets.
• Cleanup resets the collection plan, resolved targets, cached template, discovery/query snapshots, client caches, and per-query observation state so a framework re-Init after failed autodetection starts clean. Pending collector activity and its known interval-gauge keys are reset as well. The metrix store is created once in New and persists — it is not recreated.
• ChartTemplateYAML returns the cached string; no work at call time.

Collection Cycle

collect.go ties the stages together, in order:

flowchart TD
    A("ensure collection plan + target account ids")
    B("refreshDiscovery
shared ListMetrics scans per compatible target scope")
    C("refreshTags
resolve filter membership + non-identity labels")
    D("currentQueryPlan
cached until target/discovery/tag inputs change")
    E("dueQueries
keep stable queries with a newer eligible window")
    F("executeQueries
batched · concurrent GetMetricData")
    G("apply outcomes + emit
write gauges · re-emit retained")
    H("publish collector activity
absolute interval gauges")

    A --> B --> C --> D --> E --> F --> G --> H

    classDef input fill:#eef1f4,stroke:#8b949e,color:#24292f;
    classDef sched fill:#ece2ff,stroke:#8250df,color:#3b1f6b;
    classDef effect fill:#ffe6cc,stroke:#e36209,color:#5a2a00;
    classDef commit fill:#d8f3e0,stroke:#2da44e,color:#0b3d1f;

    class A input
    class B,C,D,E sched
    class F effect
    class G,H commit

Configuration Compilation

config.go, config_validate.go, plan.go, plan_compiler.go, and runtime.go separate the raw operator contract, compiler state, and installed execution plan:

• YAML and JSON use normal typed decoding; unknown keys are ignored. Config.validate decides whether the resulting configuration is valid during Init / Check.
• Named credential sources describe only credential acquisition. Named targets describe monitored identities and optional role assumption. Ordered rules bind targets to profile selectors, optional per-profile metric overrides, regions, and effective resource-tag predicates. Omitting metrics selects each profile's default-enabled series. Profiles without a metric group keep those defaults. A group includes defaults unless defaults: false, then adds its exact included MetricNames; statistics resolve from the metric entry, group, or profile in that order. Group query overrides attach only to explicit include[] expansions; item query overrides attach only to that item's expanded metric/statistic series.
• rule_defaults.filters.resource_tags is inherited when a rule omits filters.resource_tags; a present list replaces the default and [] disables it. Predicates are canonicalized once: exact case-sensitive keys are ANDed and the exact values for one key are ORed.
• compileConfig coordinates a private staged compiler that resolves every reference, rejects unused credential/target definitions, applies profile defaults/include/exclude semantics, rejects duplicate profile groups and normalized statistics, pre-indexes profile MetricNames/statistics, resolves inherited and replaced statistics into canonical exact exported-series descriptors, rejects overlap between repeated explicit-item expansions, intersects intrinsic supported regions, enforces target/role partition consistency, and emits immutable ordered scopes.
• Ordered policy scopes and tag-membership identities are separate. Scopes with the same target/profile/region/predicate share one membership identity, and already-owned exported series are removed statically with one bounded aggregate diagnostic per affected rule. A fully overlapping later scope is removed, so one exported series cannot acquire two competing query policies; a partially overlapping scope retains its unshadowed series. Same-account cross-target overlap remains until discovery, where final instance identity can be evaluated correctly. Scopes with different predicates remain ordered policy scopes even when they share one discovery scan.
• A profile whose dimensions are all constants compiles one immutable static instance into each eligible target/region scope. It consumes no discovery-group budget and needs no resource-tag fetch; query ownership and every final-instance and query-work limit still apply normally.
• Fixed internal caps bound credentials, 64 targets, rules, list references, candidate-scope evaluation, and compiled scopes. Overflow fails compilation and never installs a partial plan; limits.max_instances and limits.max_discovery_groups are operator safeguards for final selected instances and intended discovery breadth.

Discovery

Which profiles to query is decided by config, not by discovery (see Profiles): the selected profiles are the CloudWatch namespaces ListMetrics runs against. Discovery then finds which instances of those profiles exist per target and region.

discover.go. refreshDiscovery re-runs only when the snapshot TTL (discovery.refresh_every, default 300s) has expired.

All-constant scopes are omitted from discovery groups. A static-only plan makes refreshDiscovery a true no-op: no ListMetrics, snapshot installation, TTL advance, or dependent tag/query invalidation.

• discoveryGroups coalesces compiled scopes by target, region, and namespace. Each target/profile/region matcher first evaluates the full horizons of its selected series; the shared namespace scan then takes the least restrictive result, so one long-horizon participant disables PT3H instead of creating a redundant filtered ListMetrics stream beside the unfiltered superset. discoverAll fans out over those groups concurrently (bounded by apiConcurrency), with one CloudWatch client per (target, region).
• discoverAll uses two phases: every non-skipped group that resolves a client attempts its first admitted ListMetrics operation before any group may request a continuation page. Skipped groups and client-resolution failures consume no operation budget. Continuations share the remaining job-level budget. Explicit authorization denial cancels queued namespace work only in the same (target, region) lane.
• Each discoveryGroupScanner pages one shared namespace and applies an index compiled from each profile's canonical exact dimension-name set. Each returned metric's dimensions are canonicalized once; profiles with a different shape are rejected by the index lookup, while same-shape profiles evaluate only their pinned constants. This collapses CloudWatch's multi-granularity fan-out to the chosen instance grain and dedups shared instances. Constant mismatches fail closed, so a constant dimension can never merge distinct instances onto one unlabeled series.
• Recently-active-only is horizon-aware: the ListMetrics RecentlyActive=PT3H filter is applied only when every selected series participating in the shared target/region/namespace scan has publication_delay + lookback + period ≤ 3h. PT3H is the only value CloudWatch accepts, so applying it to a daily profile (S3) would hide the metric most of the day. Configurable (discovery.recently_active_only, default true).
• limits.max_discovery_groups bounds unique (target, region, namespace) groups (default 64, valid 1..100). Compatible rules and profiles share one group. The default is an accidental-expansion safeguard; larger jobs may raise it to 100, while larger collection must be split across jobs. Each group is additionally bounded to 100 pages, 50,000 scanned metrics, 1,000,000 residual same-shape profile matches, and 20,000 candidate profile instances; repeated pagination tokens fail the whole group rather than truncating it. The residual bound was measured after indexed routing and prevents adversarial metric-by-profile CPU expansion without limiting profile count directly.
• One discovery refresh has aggregate ceilings of 100 admitted ListMetrics SDK operations, 50,000 scanned metrics, 1,000,000 residual matches, 20,000 retained candidates, and 64 MiB of conservatively weighted retained-candidate storage. The configured timeout is shared by the whole discovery stage. Admission happens immediately before every actual first-page or continuation call; the AWS SDK may still retry an admitted operation up to five wire attempts.
• Snapshot + carry-forward: buildDiscoverySnapshot stores instances for successful target/region/namespace groups and carries forward the previous instances for errored groups, so a transient group failure never drops series. The merged effective snapshot is rechecked against the aggregate candidate-count and weighted-memory bounds before installation, preventing rotating partial failures from accumulating retained groups beyond the refresh envelope. Only a first-ever pass where every group errors is fatal only when no executable static scope exists. With static work, the empty prior dynamic state is retained and a retry is scheduled after discovery.refresh_every.
• Aggregate discovery failure is atomic: budget/stage-timeout exhaustion or a merged-snapshot bound violation discards all results from that refresh. An existing snapshot and its dependent tag/query/observation state remain active, with the next attempt scheduled after discovery.refresh_every. A first-ever aggregate failure is returned to the job runner only when no executable static scope exists. Otherwise static queries continue while dynamic discovery waits for its retry. Parent cancellation changes neither snapshot state nor retry scheduling and is always returned.
• One terminal disposition gate owns cancellation precedence: refresh work computes exactly one install, retry-schedule, or first-pass-error outcome and prepares all diagnostics without mutating discovery state. Diagnostics run first; one final parent cancellation check then immediately applies or returns the outcome with no logging, clock read, or callback between the gate and disposition. Earlier checks only avoid unnecessary local work.
• A warning fires at ≥1000 discovered instances as an early cost signal. The separate final-instance limit is applied later, after tag filtering and overlap.

Resource Tag Filtering And Labels

tags.go, tag_fetch.go, tagjoin.go, and tagresolve.go form the stage between discovery and query planning. Resource-tag predicates decide which discovered instances a rule may query. Separately, labels.resource_tags copies selected AWS tags to non-identity chart labels.

• Focused fetches. A Resource Groups Tagging API (RGTA) client is cached per {target, region}. Policy scopes sharing target, region, and canonical predicate share a fetch group. Each request supplies native TagFilters and the union of compatible ResourceTypeFilters; pages are streamed directly into membership and label indexes, locally rechecked, and intersected with discovered candidates. Fetch topology is rebuilt only when discovery changes, and predicate groups share one candidate index per target/region/profile. Cached results retain membership, labels, shared membership ids, and freshness—not fetch-time candidate maps.
• Failure state. A failed filtered group becomes unknown. On the first failure, its selected series are withheld for every candidate and reserved from lower-priority rules; disjoint series remain eligible. After a success, last-known matched members continue to query those series while the same series remain reserved for every other candidate. Freshness and retry state are fetch-group-local: failed groups retry next collect while successful groups keep their result until its discovery TTL expires. Optional label enrichment carries last-known labels and never controls identity.
• Label resolution (tagresolve.go). resolveTagPlan turns labels.resource_tags into a per-profile awsKey → label plan once. Global config validation rejects malformed or duplicate entries; profile-specific collisions with account_id, region, or dimension labels are skipped with one warning. The optional label resolves collisions; the default is the sanitized key (Name → name).
• ARN↔dimension join (tagjoin.go). RGTA returns ARN + tags; the cache is keyed by the profile's ARN-projectable joinKey. A namespace-bound per-profile mapper (a default single-component resource-id extractor plus overrides for ALB/NLB/target-group/ECS/OpenSearch/ Step-Functions) derives the joinKey from the ARN, and the instance side projects its dimension values onto the same key. The compiler accepts a registered mapper only when an override retains its expected CloudWatch namespace and every join dimension remains identifying (not constant). Parent-resource profiles (S3, DynamoDB-operation, ALB-target, and PrivateLink endpoint-by-subnet) key on the parent dimension so children inherit its tags. A default-selected profile without a safe association is skipped when filtering is effective; explicitly including one is a config error unless that rule disables or replaces the filter. Labels-only use remains best-effort for unsupported profiles.
• Emission split. Identity labels ({account_id, region, <dims>}) and tagLabels travel separately through plannedQuery. writeSample emits labels + tagLabels in a fresh slice; the stable query key uses identity labels but excludes tag labels, so a tag change never churns query state. chartengine auto_intersection then promotes the tag labels as non-identity chart labels—no template change. A changed effective label set invalidates the plan once; the current plan supplies the new labels and the normal numeric snapshot causes chartengine to emit a label-only chart-definition update for an existing chart.

Journey Of One Stable Query

The query subsystem is easiest to understand as one series moving through the same state machine on every collection cycle:

resolved policy + discovered instance + selected exported series
  → stable query identity
  → aligned eligible window
  → due or retry decision
  → cost-aware GetMetricData batch
  → per-query outcome
  → completion, retry, and observation state
  → retained value, synthetic zero, or gap

The rolling window is end = align_down(now - publication_delay, period) and start = end - lookback. For example, with period=5m, lookback=15m, publication_delay=10m, and now=12:17, the eligible window is [11:50, 12:05). The next window becomes eligible at 12:20; intervening collection cycles after terminal completion re-emit the retained presentation without another AWS query.

This deliberately differs from YACE's align-then-subtract ordering. With a six-hour period, five-minute delay, and now=12:02, this collector computes align_down(11:57, 6h) = 06:00; an align-then-subtract implementation computes align_down(12:02, 6h) - 5m = 11:55. The collector waits for the first shared collection tick at or after 12:05 before the 12:00 boundary is eligible. Increasing lookback searches more older buckets for sparse or late data; it does not change this end boundary or repair alignment.

One batch can contain queries that finish differently. Outcomes update each stable query independently:

Query Plan And Stable Identity

query_plan.go builds the immutable per-series blueprint.

• currentQueryPlan reuses an immutable blueprint until a target resolves or a discovery/tag snapshot changes. buildQueryPlan emits one plannedQuery per instance × selected exported series when that blueprint is invalidated. Identity labels are {account_id, region} plus one label per identifying instance dimension (a constant dimension is sent in the query but not labeled). The exported series name is <profile>.<metric_id>_<statistic>.
• Resource-tag membership is applied before selected-series expansion. Compiled scope order and {final instance, exported series} ownership enforce rule precedence: the first matching rule/target owns each overlapping series. Unknown tag membership reserves only the scope's selected series, so disjoint lower-rule selections remain eligible while failures stay fail-closed. This also catches dynamic overlap when distinct targets resolve to the same account and see the same resource.
• The first scope that emits any series for a final instance supplies one immutable identity/dimension/tag-label presentation reused by sibling series, even when later siblings are owned by another target. Chart-level mutable labels therefore remain deterministic.
• limits.max_instances counts final instances that emit at least one selected series, not planned metric queries. Overflow rejects the candidate plan atomically; the current collection cycle returns an error and does not execute or emit the previous plan. The previous immutable plan and its state remain intact, and the dirty flag remains set so a later cycle can rebuild from corrected inputs. There is no first-N truncation.
• Each plannedQuery contains a fully resolved {period, lookback, publication_delay} policy. Resolution is field-by-field, from highest to lowest precedence: job metric item, job metric group, rule, rule defaults, profile metric, profile defaults, then the built-in lookback/publication-delay fallbacks. A group source applies only to explicit include[] expansions; an item source applies only to that item's expanded series. Profile query.period is required; the lookback fallback follows the resolved period. Publication delay is collector scheduling policy, not an AWS SLA. In particular, the stock S3 storage profile's 1d is a conservative choice; AWS documents that storage metrics are reported once per day but does not guarantee delivery within one day.
• Its stable key includes execution target, region, structural AWS query, final series identity, and effective policy. Rule name, scope order, request ID, batch membership, and mutable tag labels are excluded. Equivalent rule-only ownership transfers preserve state; target/query/identity/policy changes do not.

Scheduling And State Transitions

observe.go owns due-window selection, completion, retry, and retained observation state per stable query.

• Completion is per stable query. A query is due when the aligned eligible window end is newer than its last terminal completion. Adding a sibling query never makes completed siblings due, and clock jumps query only the current rolling window rather than backfilling missed buckets.
• Eligibility has no independent timer. dueQueries runs only inside the shared Netdata collection loop. If multiple period boundaries become eligible between two update_every ticks, the next tick queries only the newest rolling window.
• A transient attempt leaves completion unchanged and starts retry state for that stable query and eligible window. The first retry waits one update_every; each later delay doubles and is capped at the effective period. A new eligible window is immediately due and resets the sequence. Complete and Forbidden clear it; parent-context cancellation advances neither retry nor completion state.

Batching And AWS Responses

query_batch.go + query_executor.go + query_response.go:

• Due queries are grouped deterministically by target, region, and exact policy, then packed into bounded concurrent GetMetricData requests.
• A page failure preserves siblings already marked Complete; only unresolved queries inherit the request failure.
• Values are paired with their timestamps; response/page order is not trusted. The newest finite pair is eligible only when its full bucket lies inside the window. PartialData may contribute a candidate but remains transient unless a later result for that query is Complete.
• Complete is terminal success. Forbidden is terminal authorization failure. Client/request failure, InternalError, missing/unknown status, unresolved PartialData, and pagination overflow are transient for only the affected queries. Bounded warnings identify those unresolved results; successful siblings remain complete and are not reissued.

Emission And Sample-And-Hold

observe.go keeps per-query completion and retained raw CloudWatch values; query_emit.go writes one snapshot frame per cycle:

• meter := store.Write().SnapshotMeter(""); each sample becomes meter.WithLabels(labels...).Gauge(series, metrix.WithFloat(true)).Observe(v).
• Always a gauge. CloudWatch returns per-period aggregates (Sum, Average, …) as absolute points, so gauge last-write-wins maps directly. Every profile chart must declare algorithm: absolute (enforced by profile validation); there is no counter/delta tracking.
• Float is a metric-level hint. metrix.WithFloat(true) marks the metric family float-native; chartengine inherits that onto the chart dimension, so fractional values render at full precision without a per-dimension options.float.
• Retention re-emit (sample-and-hold). This decouples two cadences: the query eligibility follows each series' aligned effective period, while both request execution and the write cadence occur on the collect loop (update_every; metrix writes are free). Netdata expects a value per dimension per collect cycle, so a stable query that is not due, or whose current attempt is transient, is re-written from its last cached value—a long-period metric (e.g. daily S3) renders as a continuous step line instead of gaps, at no extra AWS cost. Queries sharing one request still retain and emit independently. A successful rolling-window query keeps the newest eligible datapoint while it remains inside lookback; once expired, nil_as_zero records zero and other metrics gap. Synthetic zero is presentation state, not a timestamped source observation, so it cannot outrank a real older bucket that appears in a later rolling query. Without an explicit override, only rate: true metrics at sum or sample_count default to zero; every other statistic gaps. The cache otherwise persists until a terminal success expires/replaces it or reconcilePlan drops the stable key. Transient failures intentionally replay retained values beyond lookback and follow the per-query retry backoff. Forbidden clears the value and completes only the attempted window.
• Rate totals are cached raw and divided by the effective period during emission. A collection cycle returns an error when every due query is transient and the complete current plan emits zero samples. This is not limited to the first cycle: it can also happen after a plan change leaves no retained presentation. Any retained observation or retained synthetic zero makes the cycle fail-soft and is replayed. Parent cancellation aborts before state is advanced or a frame is committed.

Query Implementation Safeguards

These details bound cost and memory, but do not change the query state machine:

• Internal ownership, observation, and billing maps use domain-separated 256-bit structural digests so maximum-length dimension payloads are hashed once instead of copied into every series key. Emitted labels, chart identity, and AWS query dimensions remain exact strings.
• Query-plan construction atomically rejects more than 20,000 queries, 600,000 all-due datapoints, 40 all-due batches, or 1,440 buckets per query. A lightweight candidate/work preflight applies those bounds before constructing AWS query structs or stable keys, so raising max_instances cannot allocate an unbounded query plan before rejection.
• Batch width is min(500, 30000 / bucket_count). Queries sharing one structural CloudWatch metric identity are divided into billing units of at most five statistics; deterministic packing keeps each unit whole while respecting width. Work preflight and execution use the same packer. Every request sets exact MaxDatapoints = queries_in_batch × bucket_count; concurrency remains bounded by apiConcurrency, with one configured timeout shared by the batch's pages.
• Request IDs (q0, q1, …) exist only inside one batch. Pagination is bounded to two calls.

Dynamic Charts

chart.go, built once by ensurePlan and cached in chartTemplateYAML.

for each selected profile:
  group := profile.Template.Clone()          # typed deep copy; never mutate the catalog
  group.Metrics = sorted(declared series)    # includes default and opt-in metrics
assemble charttpl.Spec{Version, ContextNamespace: "cloudwatch", Groups}
return spec.MarshalTemplate()                # Validate + yaml.v2 marshal, then cached

• Uses the framework charttpl primitives: Group.Clone() (typed deep copy, so the shared profile catalog is never mutated) and Spec.MarshalTemplate().
• Rate divisors are not part of the chart template because rules can override nested profile query defaults. Rate normalization happens on the emitted numeric value using the effective period.
• Opt-in metric charts stay in the template. They do not materialize until a rule selects the metric and the collector emits its first sample.

Profiles (cwprofiles/)

profile.go defines the schema; catalog.go loads and resolves; stock profiles live under config/go.d/cloudwatch.profiles/default/ (one YAML per service; a user file with the same basename overrides its stock counterpart). The public authoring contract lives in profile-format.md; keep it in lockstep with every profile-schema change.

A Profile declares: namespace (e.g. AWS/EC2), optional supported_regions (an intrinsic restriction for services such as CloudFront), required nested query.period plus optional query.lookback and query.publication_delay, instance dimensions (the CloudWatch dimension names that identify one instance; each is either mapped to a Netdata label, or pinned to a constant value — a match-and-query-only dimension that is matched and queried but not emitted as a label, for a constant CloudWatch dimension such as CloudFront's Region=Global), metrics (with statistics, optional metric-level disabled, optional rate, optional per-metric nested query field overrides, and optional nil_as_zero — record 0 vs gap on a no-datapoint result; when unset, rate-normalized sum and sample_count series default to 0 while other statistics default to a gap), and a charttpl.Group template.

Load and resolution (catalog.go):

• Stock profiles ship under the stock dir; user profiles under the user dir. A user profile overrides a stock profile with the same basename.
• Invalid stock profiles are fatal; invalid user profiles are logged and skipped.
• Decode is non-strict (unknown keys ignored), matching collector job decoding. Validation remains authoritative for supported typed profile fields and their semantic invariants.
• Profile selection is rule-driven, not discovery-driven. Omitting rules[].profiles includes every default-enabled profile. defaults: false with include selects only named profiles; defaults: true with include adds disabled opt-in profiles; exclude removes names from the union. Explicit include/exclude overlap is invalid. The compiler intersects each rule's regions with supported_regions: an incompatible defaults-selected profile is skipped with a startup diagnostic, while an explicitly included incompatible profile fails configuration.
• Metric selection is rule-driven too. Metrics without disabled: true are the per-profile defaults. A disabled metric remains a fully declared series with active chart definitions but contributes no selected-series horizon, query, or billing work until named in that profile's rules[].metrics group. Namespace discovery remains one metric-name-agnostic dimension-shape scan, so the declaration adds no ListMetrics call or discovery group. Omitting a metric group leaves a selected profile on its defaults; an omitted/true group defaults value adds exact included MetricNames, while defaults: false creates an exact-only selection for that profile.

Profile validation invariants (profile.go) — these are load-bearing:

• Every chart's instances.by_labels must include account_id, region, and every identifying instance-dimension label (a constant match-and-query-only dimension has no label and is excluded). Chart-instance identity is built solely from by_labels; a missing label would silently merge distinct AWS resources onto one chart instance.
• Every chart must declare algorithm: absolute. CloudWatch aggregates are absolute per-period values, and the explicit declaration makes that profile invariant reviewable and enforced independently of runtime metric metadata.
• template.metrics must be empty — the collector owns the visible-series list and fills it at build time.
• rate: true requires a sum or sample_count statistic (the per-second value divides a per-period total — the summed value or the observation count — by the period).

PrivateLink endpoint grains

AWS/PrivateLinkEndpoints demonstrates why one namespace can need multiple profiles. AWS publishes the same metric names at two exact dimension sets:

• privatelink_endpoint identifies the endpoint by endpoint_type, service_name, vpc_endpoint_id, and vpc_id. It is default-enabled.
• privatelink_endpoint_subnet adds subnet_id. It is opt-in because one endpoint can fan out into several chart instances.

The profiles share one namespace discovery scan. Both join RGTA ec2:vpc-endpoint resources on VPC Endpoint Id; subnet instances therefore inherit their parent endpoint's filter membership and mutable tag labels.

Each instance exports seven series across five CloudWatch MetricNames. Mixed Average and Sum entries are intentional: Average remains the raw gauge, while rate: true normalizes only the Sum sibling to a per-second value. Stock timing is period=5m, lookback=5m, and publication_delay=5m. Exact metric/statistic items in one profile group can assign one-minute Average and six-hour Sum policies. Repeating a MetricName is valid only when the expanded statistic sets are disjoint.

PrivateLink service grains

AWS/PrivateLinkServices is the provider-side companion namespace. AWS publishes traffic metrics at five exact dimension sets, so the stock catalog uses five profiles rather than merging incompatible identities:

• privatelink_service identifies the endpoint service by service_id. It is default-enabled and is the only grain with EndpointsCount.
• privatelink_service_az adds availability_zone.
• privatelink_service_load_balancer adds load_balancer_arn.
• privatelink_service_az_load_balancer adds both detail dimensions.
• privatelink_service_vpc_endpoint adds the consumer vpc_endpoint_id.

The four detailed profiles are opt-in because one service can fan out across Availability Zones, load balancers, and consumer endpoints. All five share one namespace discovery scan per target/region. They also share one RGTA association: ec2:vpc-endpoint-service joined by Service Id. Detailed instances therefore inherit the parent service's filter membership and mutable tag labels; endpoint or load-balancer tags are intentionally not consulted.

Traffic profiles export raw Average gauges and per-second Sum siblings for bytes, new connections, and sent reset packets. Stock timing is 5m/5m/5m. EndpointsCount is a five-minute service-only gauge that records zero when AWS returns no datapoint; absent traffic gauges remain gaps. Exact rules can select one-minute traffic Average, five-minute endpoint count, and six-hour byte Sum as independent per-selection query policies in one rule.

Credentials, Targets, And Account Identity

internal/awsauth is collector-local. A named credential source is either the AWS SDK default chain (environment, shared config, EC2 instance profile, EKS IRSA) or explicit static/session credentials. A target uses one source directly or layers one AssumeRole provider over it, so static credentials can bootstrap cross-account roles without environment indirection. The collector builds an aws.Config per (target, region).

Credential sources are a list of named entries. Each entry uses a required type selector. type: default has no branch-specific configuration; type: static requires a type_static object containing access_key_id, secret_access_key, and an optional session_token. Keeping branch-specific fields nested makes the runtime model match the conditional DynCfg form. After validation, the plan compiler builds a private name index for target reference resolution; list order has no credential precedence semantics.

Each target's AWS account id is resolved through STS GetCallerIdentity (identity.go, ensureTargets) and stamped on its series. Resolution is fail-soft and retried: an unresolved target stays pending while healthy targets continue; only a cycle with no resolved target fails. Named targets are never deduplicated by account id because their permissions and visible resources may differ. Dynamic final-instance overlap is deduplicated later by ordered rule/target precedence. AWS does not require an explicit IAM permission grant for GetCallerIdentity.

Partition consistency is enforced per target. A target cannot span AWS partitions, and an assumed-role ARN partition must match the selected regions (aws / aws-us-gov / aws-cn / aws-iso / aws-iso-b / aws-iso-e / aws-iso-f / aws-eusc).

Concurrency

• apiConcurrency (5) bounds ListMetrics discovery, RGTA fetch groups, and GetMetricData chunk execution via conc/pool. maxQueriesPerRequest (500) is the AWS request ceiling; the datapoint budget may reduce actual batch width. Both are internal constants, not config.
• ListMetrics discovery first pages and continuation pages use separate bounded fan-out phases so continuation depth cannot starve another group of its first call.
• clientCache builds at most one CloudWatch client per (target, region), under a mutex, caching only successes (a transient credential error is retried next call).
• The top-level collect stages run sequentially, and the per-cycle store write is single-threaded. Discovery, resource-tag lookup, and query execution fan out within their respective stages.

Cost Model

The two CloudWatch APIs bill differently, and the design leans on that:

• ListMetrics (dynamic discovery) is cheap — it falls under CloudWatch's free API tier (~1M requests/month), then ~$0.01 per 1,000 requests, and returns only metric metadata (no per-datapoint charge). So discovery.refresh_every (default 300s) barely moves the bill; the recently-active filter further trims result pages. All-constant profiles make no ListMetrics call. Raise discovery.refresh_every only to cut API load on very large dynamic catalogs.
• GetMetricData (query) is the cost driver — it is always billed (~$0.01 per 1,000 metrics requested) and scales with metric count × query frequency. Point-aware batching keeps each AWS billing unit of up to five statistics for one structural metric in a single request, avoiding duplicate billing at a batch boundary.
• Per-query aligned-window completion is the governor — on each collect tick, a metric is queried only if that tick observes a newer eligible window. When update_every <= period, successful steady-state frequency follows the period; when update_every > period, execution is limited by update_every and skipped intermediate windows are not backfilled. Between queries, values are re-emitted from cache at zero AWS cost (see Emission And Sample-And-Hold).

Narrow the bill with focused rule target/profile/region selections or longer effective periods configured through rule/default query timing; nested profile query defaults are the fallback. Discovery frequency is a minor lever. (Rates are AWS's published model — verify current per-region prices on the CloudWatch pricing page.)

Collector activity and billing inputs

activity.go keeps four bounded activity accumulators. Four chart templates under the public cloudwatch.collector_* contexts render interval counts as absolute gauges for the interval since the preceding successfully committed collector frame:

• CloudWatch SDK Invocations counts attempted collector-issued ListMetrics and GetMetricData SDK method invocations. It materializes one chart per (account_id, region, operation) with a fixed invocations dimension. Failed attempts and every requested continuation page count; SDK-internal retries do not.
• GetMetricData Calculated Metric Requests counts the calculated billing units submitted by the collector. Up to five statistics for one structural AWS metric in one request form one unit. It materializes per (account_id, region) with a fixed calculated_metric_requests dimension.
• GetMetricData Profile Metric Request Estimates independently applies that calculation to each profile's items within each submitted request. It materializes per (account_id, region, profile) with a fixed estimated_metric_requests dimension. This is a non-additive estimate for relative cost ranking: shared structural metrics can contribute to several profiles, profile values need not sum to the calculated total, and no _shared profile is emitted.
• GetMetricData Query Items counts submitted MetricDataQuery items by source profile. It materializes per (account_id, region, profile) with a fixed query_items dimension.

For one account and region over the same interval, after summing all profile instances and considering only the GetMetricData operation, the counts satisfy: query items ≥ summed profile metric-request estimates ≥ calculated metric requests ≥ GetMetricData SDK invocations. The profile sum can exceed the calculated total because cross-profile overlap is counted independently.

The calculated total is attributed only to account and region because one query batch can serve multiple profiles. Profile estimates and query items keep the source profile because every planned series has exactly one. Different effective policies produce separate batches and separate estimates. Targets that resolve to the same account aggregate.

buildQueryBatches computes one immutable activity summary per physical request after packing. Initial and continuation pages replay that same summary, so each submitted page counts the complete request footprint. Recording scans and hashes no queries. Once the bounded scope keys exist, steady-state page recording allocates nothing; first insertion or map growth may allocate while holding the activity mutex. Per-page work is proportional only to the number of profiles represented in the batch.

netdata.go.plugin.collector.cloudwatch.* names the internal metrix selectors; it is not a chart-context prefix. The chart spec's context_namespace: cloudwatch keeps all four activity contexts beside the service-profile contexts in the UI.

Pending activity lives outside the staged metrix frame. At the start of the next cycle, activity.go checks CollectMeta.LastSuccessSeq and clears the prior interval only when that sequence proves its frame committed. A failed collection or failed metric-store commit therefore carries its work into the next successful frame. Once a series is known, a successful cached interval with no real AWS work publishes zero. Cleanup, job replacement, and process restart reset the pending activity and known keys. SDK invocation counts cover CloudWatch ListMetrics and GetMetricData, not Resource Groups Tagging API, STS, or credential-provider calls. These metrics are cost inputs, not an AWS invoice: AWS owns the billing rules, may change them, and does not separately document pagination billing semantics.

Key Invariants

• Instance identity = exact dimension-name match — a metric joins a profile only if its dimension-name set equals the profile's exactly.
• Query window is policy-driven — end = align_down(now - publication_delay, period) and start = end - lookback; end is exclusive and the newest fully eligible finite bucket wins. The policy is evaluated on shared collection ticks.
• No-data policy is per-metric — a successfully-queried empty result records 0 (nil_as_zero, defaulting on only for rate: true sum/sample_count) or gaps after a terminal successful window expires the retained datapoint. Transient client/request/result failures retain and replay state; Forbidden clears state and completes the attempted window.
• Completion advances per stable query, never per batch or group.
• Fail-soft discovery — carry forward on error. A first-ever total or aggregate failure is fatal only when no executable static scope exists; static work keeps the cycle useful while dynamic discovery retries. Parent cancellation is always fatal and state-neutral.
• Job-level placement — AWS resources are chart instances via by_labels. They use the configured job vnode or the Agent host; no per-target/resource HostScope is generated.
• Tag filters fail closed — unknown membership never becomes unfiltered collection and never allows a lower-priority rule to claim the unknown higher rule's selected series; disjoint series remain eligible.
• Tag labels are non-identity — labels.resource_tags can update labels on an existing chart but cannot change its identity or overwrite an identity label.
• Compiled configuration — operator-facing credentials, targets, and ordered rules are validated and expanded once. Discovery, tag-label presentation, limits.max_instances, limits.max_discovery_groups, timeout, vnode, and the framework's common fields remain job-level settings. Concurrency and per-request/per-group AWS-work bounds are internal constants. Discovery breadth uses the explicit operator safeguard; compatible rules/profiles share a group, and larger valid installations can raise it or split across jobs.

Where To Change Things

• Add or adjust a monitored AWS service: add or edit a profile YAML under config/go.d/cloudwatch.profiles/default/ (namespace, instance dimensions, metrics, chart template). A new service that fits the profile model needs no Go change. Add a matching metadata.yaml monitored-instance entry and regenerate the integration docs.
• Change discovery: matching, fan-out, snapshot assembly/TTL, and recently-active policy are in discover.go; terminal diagnostics and install/retry/fail application are in discovery_disposition.go; aggregate admission, stage timeout, and authorization lanes are in discovery_budget.go.
• Change query identity or plan expansion: structural_id.go and query_plan.go.
• Change timing defaults or validation: internal/cwquery/config.go.
• Change collector policy resolution or aligned windows: query_policy.go.
• Change request packing and AWS work limits: query_batch.go and limits.go.
• Change result pagination, status handling, or datapoint selection: query_response.go.
• Change execution orchestration: query_executor.go.
• Change scheduling, retention, correction, or gap state: observe.go.
• Change how retained observations become metrics (rate normalization, labels, float hint): query_emit.go.
• Change chart generation (namespace or template assembly): chart.go, plus the profile template.
• Change auth: internal/awsauth (collector-local; extract to a shared package only when a second AWS consumer appears).
• Add or change a config option: config.go + config_schema.json + metadata.yaml + stock cloudwatch.conf + regenerated integrations/ docs (collector consistency). Prefer internal constants over new options unless the option names a real operator decision.

Validation Checklist

cd src/go
go test -count=1 ./plugin/go.d/collector/cloudwatch/...
go vet ./plugin/go.d/collector/cloudwatch/...
gofmt -l plugin/go.d/collector/cloudwatch/

If metadata.yaml changed, regenerate and commit the integration docs (collector consistency; see the integrations-lifecycle skill):

python3 integrations/gen_integrations.py
python3 integrations/gen_docs_integrations.py -c go.d.plugin/cloudwatch