Job Manager Architecture

This is a maintainer-oriented guide to the Job Manager (jobmgr), written to be read top to bottom. It builds up in four passes: where Job Manager sits, the machine that orders all work, the things that machine orchestrates, and how the process restarts and shuts down. The reference tables at the end are for coming back later.

It intentionally leaves collector internals opaque. How a specific collector walks SNMP, scrapes Prometheus, or renders charts lives in that collector and the framework packages it uses; Job Manager only orchestrates their lifecycle.

Path convention. Code references are relative to src/go/plugin/agent/jobmgr/, except those starting with plugin/, cmd/, or pkg/, which are relative to src/go/.

Contents

Orientation — Short Version | Where Job Manager Sits | The Big Picture

The machine — The Concurrency Model | Process Containment

What it orchestrates — Jobs | Secrets | Vnodes | Jobs With Dependencies | Service Discovery | Functions

Process lifetime — Restart and Shutdown | Runtime Metrics

Reference — Package Map | Where To Change Things | Validation

Short Version

Job Manager is the orchestration boundary for a Go data-collection plugin process (go.d, ibm.d, scripts.d). It takes everything that wants to start, stop, reconfigure, or query a collector job and turns it into an ordered, safe stream of lifecycle commands run by one single-threaded command kernel.

Three things can drive it:

• Function calls arriving on the plugin's stdin from the Netdata daemon — including dynamic configuration (DynCfg) commands to add / edit / enable / disable / test / remove jobs.
• Discovery — file configs and service discovery proposing jobs to run.
• Autodetection retries — jobs that failed to detect their target, retried later.

Everything a plugin writes back — metrics, charts, Function registrations, config state, keepalives — leaves through one serialized stdout writer.

The whole process runs one active "run generation" at a time. A reload (SIGHUP) rotates that generation without rebuilding the process when the rotation completes cleanly. If rotation cannot complete safely within its deadline, the plugin exits successfully so the daemon starts a fresh process.

Where Job Manager Sits

Every plugin binary follows the same startup path.

flowchart TD
    Plugin("cmd/godplugin · ibmdplugin · scriptsdplugin
main()")
    New("agent.New(Config)")
    Host("agenthost.Run
forwards OS signals")
    Proc("composition.NewProcess
build the process")
    Run("process.Run(ctx)
outer loop")
    Gen("run generation
command kernel + adapters")

    Plugin --> New --> Host --> Proc --> Run --> Gen
    Host -. "SIGHUP → Restart" .-> Run
    Host -. "SIGINT / SIGTERM → Terminate" .-> Run

    classDef entry fill:#dbeafe,stroke:#2563eb,color:#0b1021;
    classDef core fill:#fef3c7,stroke:#d97706,color:#0b1021;
    class Plugin,New,Host entry;
    class Proc,Run,Gen core;

• cmd/*plugin/main.go builds a RunModePolicy, registers discovery providers, and calls agent.New.
• cmd/internal/agenthost/host.go hosts one process-lifetime Agent and maps OS signals to acknowledged controls: SIGHUP → Restart (30s), SIGINT/SIGTERM → Terminate (10s).
• plugin/agent/agent.go loads config and modules, then calls composition.NewProcess and process.Run(ctx).

Run modes (plugin/agent/policy/runmode.go) flip a few gates:

• Long-lived agent (production, not a terminal): service discovery on, runtime charts on, discovered jobs wait for the daemon's enable command.
• Terminal / debug (attached to a TTY): service discovery off, runtime charts off, discovered jobs auto-enable so a developer sees output immediately.

The Big Picture

Once running, Job Manager is a funnel: several sources of intent on the left, one ordered kernel in the middle, one stdout stream on the right.

flowchart LR
    Daemon("Netdata daemon")
    Fn("Function ingress
stdin")
    Disc("Discovery
files + service discovery")
    Retry("Autodetection
retries")
    Kernel("Command Kernel
single-threaded loop")
    Jobs("Collector jobs
V1 / V2")
    Secrets("Secret resolver
+ store")
    Vnodes("Vnode registry")
    Frame("FrameOwner
one stdout writer")

    Daemon -->|"FUNCTION / DynCfg"| Fn --> Kernel
    Disc --> Kernel
    Retry --> Kernel
    Kernel --> Jobs
    Jobs -->|"resolve refs"| Secrets
    Jobs -->|"attribute metrics"| Vnodes
    Jobs --> Frame
    Kernel --> Frame
    Frame -->|"metrics · charts · CONFIG · FUNCTION"| Daemon

    classDef ext fill:#dbeafe,stroke:#2563eb,color:#0b1021;
    classDef core fill:#fef3c7,stroke:#d97706,color:#0b1021;
    classDef job fill:#dcfce7,stroke:#16a34a,color:#0b1021;
    classDef sec fill:#fee2e2,stroke:#dc2626,color:#0b1021;
    classDef vn fill:#ccfbf1,stroke:#0d9488,color:#0b1021;
    classDef out fill:#e5e7eb,stroke:#4b5563,color:#0b1021;
    class Daemon,Fn,Disc,Retry ext;
    class Kernel core;
    class Jobs job;
    class Secrets sec;
    class Vnodes vn;
    class Frame out;

A useful mental split for the rest of this document:

• The kernel decides what happens and in what order.
• Adapters (functions, joboutput, secrets, discovery) know how to do the collector-specific work, behind narrow ports.
• lifecycle provides the neutral machinery the kernel delegates to (UID ownership, tasks, permits, framing, run control, and the shutdown budget).
• containment owns collector-derived work that may outlive a run because it does not cooperate with cancellation.
• composition wires them all together.

The Concurrency Model

This is the core idea. Job Manager does almost no locking in its business logic. Instead, one goroutine — the CommandKernel run loop — owns all mutable orchestration state and is the only thing allowed to change it. Everything else either hands work in over a channel or does blocking work off to the side and reports back.

Think of an air-traffic control tower with a single controller:

• Aircraft (commands) queue on runways (lanes); one moves per runway at a time, in arrival order (FIFO).
• Before taxiing, a flight reserves airspace corridors (claims), always requested in the same order so two flights never deadlock waiting on each other.
• The controller never leaves the tower. Pilots (off-loop task goroutines) fly the actual missions and radio back completions. The controller only reads radios and updates the board.

The command lifecycle

flowchart TD
    Submit("Submit
adapter, off-loop")
    Admit("Admit
UID dedupe · route · lane")
    Stage("Pre-claim stage
optional, off-loop, no claim held")
    Lane("Lane head ready
per-resource FIFO")
    Claim("Claims
exclusive cross-lane ordering")
    Task("Run task
goroutine, off-loop")
    Complete("Apply completion
on-loop")
    Frame("FrameOwner
terminal frame → stdout")
    Dispose("Dispose
release stage · claims · lane")

    Submit --> Admit --> Stage --> Lane --> Claim --> Task --> Complete --> Frame --> Dispose
    Admit -. "no stage declared" .-> Lane
    Complete -. "advance / wake lanes" .-> Lane

    classDef offloop fill:#e5e7eb,stroke:#4b5563,color:#0b1021;
    classDef onloop fill:#fef3c7,stroke:#d97706,color:#0b1021;
    class Submit,Stage,Task offloop;
    class Admit,Lane,Claim,Complete,Frame,Dispose onloop;

1. Submit (off-loop) — an adapter validates a Request, attaches a prepared Job Manager plan or submits an unresolved Function request, pushes it onto a submission queue, and wakes the loop. command_ports.go, kernel_ingress.go.
2. Admit (on-loop) — the loop dedupes the command's UID, resolves the route, derives the lane key, and installs the operation on its lane. Duplicate UIDs and invalid routes are rejected here. kernel_admission.go, lifecycle/uid.go.
3. Pre-claim stage (off-loop, optional) — see below. The operation sits queued on its lane and holds no claim while process-owned preparation runs.
4. Lane — same-resource commands share one FIFO lane; only the lane head runs while the lane is active. Different lanes advance independently.
5. Claims — cross-lane exclusion; see below.
6. Run task (off-loop) — cooperative run-owned work executes through TaskSupervisor; collector-derived work that may ignore cancellation executes through the process containment authority. Neither executes on the kernel loop. lifecycle/task.go, containment/authority.go.
7. Apply completion (on-loop) — the task radios its result back; the loop seals it and advances the lifecycle.
8. Frame — the terminal response is committed to stdout through FrameOwner.
9. Dispose — the stage, the claims, and the lane slot are released, waking any blocked lanes.

Lanes versus claims

The two are easy to confuse, and they solve different problems.

Claim rules:

• Every claim is exclusive; there are no shared/read modes.
• Claims are acquired in a stable global key order, so the design is deadlock-free.
• Waiters are FIFO per key, so the design is starvation-free.
• claim_authority.go.

Two consequences worth remembering:

• A resource-less Function call gets a unique lane per invocation, so concurrent calls to the same Function run in parallel — unless the resolved plan declares claims. command_ports.go.
• Declaring a claim is how an adapter opts into serialization it cannot express with a lane.

Pre-claim staging

Some preparation is slow, and some of it is collector-authored code that may ignore cancellation. Running it while holding dyncfg:jobs would stall unrelated jobs. A plan may therefore declare a PreClaimStage (kernel_plan.go, kernel_stage.go):

• When it starts — at admission, immediately after the operation is queued on its lane.
• What it holds — nothing. No claim has been acquired yet.
• How the kernel waits — the lane head is not marked ready while the stage is pending; Ready() closing wakes it.
• Who owns the work — the process containment authority, not the run, so a non-cooperative worker cannot pin the kernel or the run generation.
• Cleanup — disposal calls Release(); cancellation and deadlines call Cancel(cause) first. kernel_disposal.go.

Yielding a claim during preparation

Staging solves the problem before claims are taken. The mirror-image mechanism handles preparation that must happen inside a transaction that already holds claims (claim_yield.go, claim_yield_kernel.go):

• What is yielded — only the transaction's acquisition-suffix claim, and only while bounded preparation runs. It must be reacquired before preparation completes.
• Why it is still safe — the transaction's resource lane stays active, so another command for the same resource cannot overlap the yielded work.
• Priority — reacquisition outranks ordinary waiters on that key.
• Composite parents — a composite declares the resource lanes its children may use. If one conflicts with an active yielder, the parent is parked before it acquires any claim; an already-waiting parent releases its prefix claims but retains its original ticket. Unrelated work stays serviceable on the otherwise-free claim, which prevents both child/lane cycles and head-of-line blocking.
• Children of a running composite — when such a child yields, only the matching admission-fence claim is suspended; the parent's other claims remain held.

Which mechanism each command surface uses:

Who owns what

• On-loop (exclusive to the CommandKernel run loop) — every lane, operation, deadline, claim transition, and counter. The loop is the sole mutator. architecture_test.go pins that on-loop actions are dispatched through the sanctioned kernel ownership funnel.
• Run-owned off-loop — cooperative tasks and permits belong to one run generation and must drain before that run can quiesce.
• Process-owned off-loop — contained attempts own their physical worker, per-identity exclusion, and final cleanup across run rotation.

Fairness and timeout rules

• TaskSupervisor runs two independent classes — framework-control work (lifecycle/DynCfg commands) and generic Function work — in strict round-robin. One class can never starve the other, and there is no fixed "N active Functions" cap. lifecycle/task.go.
• An ordinary timed-out run task keeps run ownership. The kernel only cooperatively cancels it; its claims, lane, and resource authority stay held until it returns, and repeated overruns can escalate to fail-stop.
• A contained attempt keeps process ownership instead. Its caller, claims, and run may settle after the containment cut because late output and mutation cannot escape the process-owned attempt boundary. The physical worker may still finish state private to that attempt; the boundary prevents publication, re-entry, or reuse until physical release.

Process Containment

Go cannot forcibly stop a goroutine. A collector's Check can block forever on a socket; a provider's Init can ignore its context. Job Manager therefore separates logical settlement (the caller stops waiting) from physical release (the goroutine actually returns).

process_attempt.go declares the contract; containment/authority.go implements it.

• StartProcessAttempt checks caller cancellation while holding the identity-registry mutex. A successful claim linearizes at that snapshot: later cancellation follows the caller's normal post-start lifecycle. The caller context gates admission only; the worker receives an independent process-owned context.
• SupersedeProcessAttempt targets the owner observed during its locked registry snapshot. Its success confirms that owner released; it does not reserve the identity against a later successor, so the subsequent start remains authoritative.

The attempt state machine

stateDiagram-v2
    [*] --> Probing: StartProcessAttempt reserves the identity
    Probing --> Admitted: Admit() — result handed over, fuse stopped
    Probing --> Released: Work returns in time
    Admitted --> Released: Work returns after handover
    Probing --> Contained: fuse expires · caller cancels · superseded · run retired
    Admitted --> Contained: cut on retire, supersede, or shutdown
    Contained --> Released: worker finally returns — possibly never
    Released --> [*]: safe terminal result — identity is free again
    Released --> Quarantined: unsafe terminal result
    Quarantined --> [*]: process exit only

• Probing — the attempt is producing a result and is bounded by the fuse.
• Admitted — the attempt has handed its result to the caller and now holds something (a running collector loop, a prepared Store mutation, a built Function bundle) until the caller decides.
• Contained — logically settled and cancelled; late effects cannot escape the attempt boundary. The worker may still finish private state and remains physically running.
• Released — Work and its cleanup returned. Only now is the identity admissible again.
• Quarantined — the physical work returned, but its terminal outcome made the identity unsafe to reuse. The quarantine is process-lifetime state, not an active attempt, and only process exit clears it.

Census reports these counts exactly (Active, Probing, Admitted, Contained, Quarantined). Containment is loud and success is quiet: entering Contained, releasing a contained worker, quarantining an identity, and a worker panic each emit a diagnostic event, while an attempt that admits or returns in time emits none. In long-lived mode, jobmgr.runtime also projects the current census as aggregate process-attempt gauges.

Containment fences receive the raw cut cause and execute synchronously before settlement becomes visible. They stay under the authority lock to preserve that ordering, so they perform only bounded in-memory state changes: they never re-enter the authority or wait for a worker, I/O, or external cleanup. A panic is redacted as ErrProcessAttemptFencePanic, joined with the original cut cause, and cannot strand the authority mutex.

Terminal outcomes quarantine an identity when an admitted worker fails, ownership remains retained, the worker panics, or the containment fence panics. Ordinary pre-admission validation failures release the identity normally.

What the fuse actually bounds

This is the subtlety that catches everyone:

• DefaultFuse (2 minutes) bounds producing a result, not holding one.
• Admit() atomically stops that fuse while keeping the identity occupied, so long-lived contained work is deliberately unbounded. An installed collector runtime that runs for weeks is a normal admitted attempt.
• DefaultSupersessionGrace (2 seconds) is how long a replacement waits for the previous owner of the same identity to release before it reports busy.

Who admits, and who does not:

joboutput/candidate_stage.go, functions/module_stage.go, secrets/store_stage.go, functions/bundle.go, composition/service_discovery.go.

What the caller sees

• Caller cancellation is propagated first; crossing the fuse or the supersession grace settles the caller and fences late results.
• The authority retains the worker and everything it owns until its complete cleanup returns, or until the plugin process exits.
• Persistent file/discovery state keeps only its latest desired replacement and retries after identity release.
• SecretStore add/update requests, including DynCfg, retain only their latest desired config after retryable contention and apply it after the identity releases.
• Other synchronous DynCfg requests are not retained or retried after a busy/contained response. Cancellation prevents queued service-discovery mutations from starting; it does not roll back work that already began while its request context was live.

No process-wide slot limit

There is intentionally no cap on concurrent attempts: one permanently stuck identity must not block an unrelated job. The trade-off is explicit — distinct permanently stuck identities can accumulate retained workers, and diagnostics report that state (including a bounded sample of retained identities at shutdown).

Namespaces

An identity is a namespace plus a key; at most one physical attempt exists per identity, and unrelated identities remain independently admissible.

Test identities are derived from the raw config and exist only in memory, so an identical repeated test returns busy instead of multiplying workers.

Attempt keys are built so a long or hostile name cannot exceed containment's internal identity bounds:

• Every variable component passes through domain-separated, length-framed fields and a fixed-size digest. Function invocations append only a bounded counter to that digest.
• Public job, Store, module, and service-discovery names are therefore preserved without bound risk.
• Diagnostics keep safe short names, and fall back to bounded generic labels for oversized, invalid-UTF-8, or control-bearing values.
• Graph IDs, lane keys, config IDs, and wire-visible names are unchanged by this hashing.

Jobs

A job is one running collector instance: a module plus a resolved config. Jobs are created from stock/user config files at startup, from discovery, or from DynCfg commands, and are retried after a failed autodetection.

The config validation boundary

External configuration is untrusted batch input. Production producers validate successfully decoded entries against the exact downstream contracts before process construction or WorkPlan submission:

• the vnode file loader and DynCfg vnode adapter validate CONFIG identity and source fields, host-emission metadata, daemon-compatible UUID syntax, and semantic hostname/GUID uniqueness;
• the secret-store file loader validates each fully stamped config, including kind type and provider support, before passing accepted entries to a run generation;
• discovered collector proposals validate their final CONFIG job name and source metadata before graph mutation.

Failure granularity is deliberate:

Strict constructor/controller checks remain as defensive invariant enforcement: programmatic state that bypasses a production producer is still rejected rather than silently normalized or recovered after admission.

Configuration source ownership

Go plugin configurations use one source order: DynCfg > user > discovered > stock > internal/unknown.

The order is shared, but there is no single generic "configuration manager." Each domain applies the order at the boundary that owns its lifecycle:

This distinction matters. Collector-job retries and fallback rules do not implicitly apply to SecretStores, service-discovery pipelines, or vnodes. Configured vnodes do not keep a stack of competing candidates: file config seeds the registry, a DynCfg upsert replaces that name, and removing the DynCfg vnode does not reveal the old file value.

A plugin-side DynCfg add is replay/upsert-capable because the daemon uses it to restore persisted configurations. Ordinary user duplicate adds remain create-only at the daemon boundary. Removal targets only the current DynCfg override and does not immediately reactivate a masked lower-priority source.

Collector source-priority event flow

One collector-job identity has one selected desired configuration. A lower source may remain known to discovery while a higher source owns the graph, but it is not allowed to probe or run.

flowchart TD
    Event("Config event for one module/job")
    Select("Select highest source priority")
    Lane("Serialize on that job identity")
    Recheck{"Higher-priority graph
owner exists?"}
    Noop("Acknowledge source state
no Check · graph unchanged")
    Probe("Prepare and Check
selected config")
    Commit("Commit selected outcome")

    Event --> Select --> Lane --> Recheck
    Recheck -->|"yes"| Noop
    Recheck -->|"no"| Probe --> Commit

    classDef ext fill:#dbeafe,stroke:#2563eb,color:#0b1021;
    classDef core fill:#fef3c7,stroke:#d97706,color:#0b1021;
    classDef quiet fill:#f3f4f6,stroke:#6b7280,color:#0b1021;
    class Event ext;
    class Select,Lane,Recheck,Probe,Commit core;
    class Noop quiet;

Common event sequences:

Here, "no-op" means do not execute the collector. Discovery may still remember the lower candidate as source state.

plugin/framework/confgroup/config.go, discovery/decision.go, joboutput/discovery.go, joboutput/pending_job.go.

Candidate lifecycle

Two different guarantees apply during replacement:

• While the candidate is preparing and probing, the incumbent keeps running.
• After a valid selected candidate settles, the selected desired state wins. Success installs it; a transient construction failure or probe failure retires the incumbent and commits the source-specific Failed or removed outcome.

A proposal rejected before it establishes desired state, or an attempt that cannot start because its physical identity is still busy, leaves the incumbent unchanged. Persistent sources retain only their latest desired retry; a synchronous DynCfg command reports busy and is not applied later.

flowchart TD
    Cmd("add / update / discovered / retry")
    Stage("Process-owned candidate
clone · secrets · construct · Init")
    AutoD{"Check + post-check
yield global jobs claim"}
    Functions("Stage job Functions
from initialized collector")
    Keep("Reject / supersede / busy
incumbent unchanged")
    Reserve("Reserve inactive
run permit")
    Retire("Fence ordinary output + detach
prior generation")
    Promote{"Acquire installed-runtime
identity"}
    Attach("Attach live projections
activate permit + output")
    Live("Start + publish Running")
    RetireFailed("Retire incumbent")
    Fail("Commit Failed / removed
schedule retry by source policy")

    Cmd --> Stage
    Stage -->|"busy / proposal rejected"| Keep
    Stage -->|"transient preparation
failure / contained"| RetireFailed
    Stage --> AutoD
    AutoD -->|"ready"| Functions
    Functions -->|"ready"| Reserve --> Retire --> Promote
    Functions -->|"failed / contained"| RetireFailed
    Promote -->|"acquired"| Attach --> Live
    AutoD -->|"failed / contained"| RetireFailed --> Fail
    Promote -->|"old runtime retained"| Fail
    Fail -. "identity release / retry due" .-> Stage

    classDef ext fill:#dbeafe,stroke:#2563eb,color:#0b1021;
    classDef core fill:#fef3c7,stroke:#d97706,color:#0b1021;
    classDef job fill:#dcfce7,stroke:#16a34a,color:#0b1021;
    class Cmd ext;
    class AutoD,Functions,Reserve,Retire,RetireFailed,Promote,Attach core;
    class Stage,Keep,Live,Fail job;

1. Stage config — validation, DynCfg test, and configuration rendering run as contained operations. A same-job validation supersedes the prior candidate; an identical DynCfg test has its own raw-config-derived, memory-only identity and returns busy rather than multiplying workers. joboutput/config_stage.go.

2. Prepare candidate (non-disruptive) — the process authority reserves the canonical job identity before cloning and secret resolution, then owns collector construction, configuration application, Init, Check, post-check validation, Function staging, and rejection cleanup. Function staging runs only after Init and Check succeed, because instance handlers may be initialized by those callbacks. The candidate has inactive output and private V2 runtime/vnode staging, but no run permit and no live run service. The incumbent keeps running. joboutput/candidate_stage.go, joboutput/runtime_staging.go.

3. Settle autodetection — caller cancellation is propagated and the transaction temporarily yields the global dyncfg:jobs claim.

   • The process-owned fuse bounds logical waiting even if the collector never returns, and late success cannot be admitted after a cut.
   • A normal detection failure commits StatusFailed. A busy/contained persistent source retains only its latest desired retry; synchronous DynCfg returns a retryable error instead.
   • One bad collector cannot dirty Job Manager. A normal Init, Check, validation, or Function-staging failure is isolated to that candidate once rejection cleanup completes. Only a failed or unprovable cleanup retains process ownership and fails the run closed.

4. Reserve and retire — only a timely successful candidate is wrapped in an inactive run permit. Replacement then fences the incumbent's ordinary output and detaches its run projections immediately; its physical managed loop, Stop, Function drain, and collector cleanup remain process-owned until they return.

5. Promote, attach, and start — the candidate acquires the separate job-runtime identity only after logical retirement. Candidate Init/Check may overlap the incumbent, but two installed runtimes for one job cannot.

   • Promotion publishes its owner transition before entering the attempt authority, then releases the owner lock for runtime-identity start, supersession, and admission. This removes the reverse edge against the authority's synchronous containment callback. Successful admission transfers process ownership even when containment wins before promotion settlement; pre-admission failure remains candidate-owned.
   • Attachment reports whether ownership transferred. The process owner adopts every transferred wrapper and partial or completed Function lifecycle even when retirement already won, so rejection finalizes them exactly once. A no-transfer failure is never inferred from cleanup fields and never triggers adoption.
   • Runtime/vnode projections attach outside the staged-owner lock because they call external registries. The owner then linearizes retirement against output-gate activation, permit activation, and accepted-cleanup freezing in that order.
   • Retirement before an otherwise-valid attachment bind returns the exact process-attempt cause even when that retirement already finalized the candidate. Independent duplicate, decided, or invalid transitions remain fail-closed.
   • Runtime-attempt containment synchronously records the first retirement cause and revokes new output admission before the authority cut returns. The worker later drains output leases and performs detach/cleanup outside the authority lock. A later stopped/retired cut never overwrites an earlier structural failure.
   • Start eligibility and retirement arbitrate under the process owner. The worker rechecks that decision after receiving the start request, so a ready channel send cannot launch StartManaged after containment has won.
   • Once output and permit acceptance wins and start remains eligible, the managed loop starts.
   • The resource transaction owns this successor until installation acknowledgement. An apply failure aborts it when possible, or returns the still-live retained generation to the kernel for fail-closed ownership.
   • If the old runtime does not release within the supersession grace, the candidate is rejected and the source-specific busy/pending policy applies.

   Rotation interacts with this step deliberately. Rotation cuts run-target attempts before draining the run, so a non-cooperative startup cannot consume the reload budget:

   When the cut arrives	Result
   Before installation reservation	The transaction aborts the pending generation and graph mutation, then returns the exact unchanged/removed disposition
   After reservation begins	Retirement cannot invalidate the short internal commit; run shutdown retires the installed generation afterward

   Only a pure target-retired/process-stopped error tree counts as normal cancellation. Any mixed, cleanup, graph, panic, or ownership error stays fail-closed.

6. Emit — resource acceptance activates the generation output gate. Two capabilities exist, and both serialize whole frames through the process's one FrameOwner (joboutput/output_gate.go):

   • Ordinary collection output is run-owned. Retirement first atomically revokes new write admission, then drains write leases outside the attempt-authority lock before detaching projections. A write admitted before the cut may finish; a write presented after the revocation cannot interleave with the successor.
   • Accepted V1/V2 cleanup output is process-owned, because physical cleanup may outlive a rotation. It emits terminal chart-obsoletion frames only after the managed loop and Function handlers physically quiesce.
   • The job-runtime identity stays occupied through cleanup, so a same-job successor cannot start before those terminal frames complete.

Job generations and fencing

Every job carries a monotonic generation number, and a staged result is consumed only by the matching transaction and run epoch. Three fences enforce that:

• The process-owned attempt target rejects late work from a retired run.
• The per-generation output gate allows ordinary frames only after installation, and fences them at logical retirement.
• The accepted-cleanup capability is process-owned rather than run-owned, because physical cleanup may outlive a rotation. Process termination fences it terminally before finalizing the output service, suppressing cleanup that arrives after the bounded shutdown budget.

V1 vs V2

Job Manager orchestrates both collector contracts identically; only the runtime adapter differs (joboutput/runtime_adapter.go, plugin/framework/jobruntime):

• V1 declares Charts() and returns Collect() map[string]int64.
• V2 writes to a metrix.CollectorStore, supplies ChartTemplateYAML() (rendered by the chart engine), and is wired to the runtime service. Each V2 scope owns its last successful host definition; the shared vnode registry only diagnoses conflicting metadata after successful output.

Autodetection retries

Retries are deliberately cheap. There is no timer or goroutine per job. Instead, one per-run map + heap + dispatcher owns all pending retries (joboutput/autodetection_retry.go, joboutput/scheduler.go):

• The process's 1-second tick advances a logical clock.
• When an entry is due, the single run-owned dispatcher resubmits it as a restart through the ordinary command port — fire-and-forget — and keeps authority over that config/retry token until the resulting transaction settles.
• A busy identity coalesces into one pending retry. Success, replacement, disable, removal, or shutdown invalidates or replaces the token.

Secrets

Secrets keep credentials out of collector configs. A config value can carry a reference instead of a literal:

• ${store:kind:name:key} — looked up in a SecretStore (Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager).
• ${env:...}, ${file:...}, ${cmd:...} — resolved from the plugin process's own environment variables, files, or command output.

Resolution happens only in memory, only when a job is built. The key property is that it is atomic — all references resolve, or none do. Picture a notary: photocopy the whole document, list every blank, check out the referenced files under one pass, fill every blank on the copy, check the files back in, and hand back a fully-filled copy or nothing at all.

flowchart TD
    Ref("config with a secret reference")
    Clone("Clone + validate whole config")
    Compile("Compile references → distinct store keys")
    Scope("Acquire ONE reader scope
pin current store generations")
    Resolve("Resolve the clone")
    Proof("Capture immutable
generation proof")
    Release("Release scope · drain readers")
    Post("Complete in-memory clone → build job")

    Ref --> Clone --> Compile --> Scope --> Resolve --> Proof --> Release --> Post

    classDef sec fill:#fee2e2,stroke:#dc2626,color:#0b1021;
    classDef job fill:#dcfce7,stroke:#16a34a,color:#0b1021;
    class Ref,Clone,Compile,Scope,Resolve,Proof,Release sec;
    class Post job;

The resolver lives in plugin/agent/secrets/resolver; it never mutates the input config and returns nil on any error, so a half-resolved config can never reach a collector.

Redaction after a secret is applied

Once a configuration containing secret references has been applied to a collector, two layers scrub its errors:

• At the Job Manager boundary — lifecycle failures are replaced with a generic redacted error before they are logged or published.
• Inside the collector's own logger — messages and newly attached attributes are sanitized, so an internally logged request error cannot bypass the runtime boundary.

What survives redaction: cancellation, DynCfg code/retryability, panic classification, and retained-ownership state. What does not: the raw collector cause.

Changing a store restarts its jobs

Backing stores are managed live over DynCfg (add / update / remove). The store (plugin/agent/secrets/secretstore) keeps numbered, immutable generations per kind:name. Each run receives a fresh Store epoch, but the process owns that epoch's preparations, generations, and reader scopes (composition/secret_epoch.go).

1. Prepare outside publication. Provider construction, configuration, and Init always run as a process-owned contained attempt, never inside the transaction that publishes the result. Only where that attempt is driven differs — at startup every initial Store starts its own attempt up front and the controller waits for all of them at one aggregate barrier before submitting any publish command (secrets/initial.go), while DynCfg commands and retained retries drive it as a pre-claim stage so the command holds no claim while it prepares (secrets/store_stage.go, secrets/pending.go). The prepared generation is then committed by compare-and-swap against the expected generation.
   • A DynCfg test creates the same temporary configured Store but never publishes it. If the Store implements dyncfg.Testable, its context-aware operational check runs inside the test's config-hash-specific contained attempt. The shared Store Test boundary rejects caller cancellation before temporary construction and makes its cause take precedence after configuration/Init and after the optional provider Test callback. Providers classify their configured timeouts but need not repeat caller cancellation checks.
   • A Store without that optional capability remains configuration-valid, and the response explicitly says that the result was validation-only. Configuration failures return 400, operational failures return 422, and busy/contained attempts return 503.
   • Provider failures retain their private causes. A marked dyncfg.PublicError may append only static, code-authored detail to Test 400/422 and add/update preparation 400 responses; unmarked failures remain generic, and every rendered SecretStore operator message stays within the existing 4 KiB bound.
   • Vault performs one real authenticated GET /v1/auth/token/lookup-self through the same token, namespace, client, TLS/proxy, timeout, and cancellation path used for secret resolution. HTTP 200 is operational success by status; its body is not inspected. An exact permission-only HTTP 403 body returns dyncfg.ErrTestUnsupported and becomes explicit validation-only success because the response cannot reliably prove token validity across Vault versions and authentication restrictions. Invalid-token, ambiguous, malformed, or oversized HTTP 403 bodies fail, as does every other status. The Test reads no secret and does not prove secret-path permission, that Vault accepted the token, or that an intermediary preserved the namespace header. It buffers at most 1 MiB plus one size-detection byte of a classified 403 body. When the request reaches Vault, it creates audit/activity evidence and can consume the final use of a limited-use service token. Vault token files use the shared regular-file-only 1 MiB configured-file reader.
   • AWS Secrets Manager Test calls the exact production credential-source dispatcher and discards the credentials. env performs no network I/O; ECS performs one logical task-metadata GET; IMDS performs token PUT, role GET, and credential GET sequentially. Metadata redirects and proxies are disabled. The bodyless GETs preserve Go's transparent retry after a qualifying reused-connection failure; Test adds no application retry. Each response is strictly capped at 1 MiB, each request uses the configured/default three-second client timeout, and IMDS can approach three sequential timeout periods before the outer caller bound. Success proves only non-empty access-key and secret-key acquisition; session tokens remain optional. It does not call Secrets Manager, STS, or KMS and therefore does not prove AWS acceptance, credential freshness, region correctness, secret access, or KMS permission.
2. Restart dependents as one composite command. If any running jobs depend on that store key: stop dependents → commit the new generation → start dependents. The parent retains dyncfg:dependency-graph throughout, and each start child temporarily yields only the dyncfg:jobs acquisition suffix while its probe runs, so unrelated job-graph work may proceed while dependency mutations remain fenced. secrets/restart.go, secrets/transaction.go.
3. Retire the old generation last. The superseded generation is retired only after its last reader scope drains, so an in-flight resolution never sees credentials vanish mid-read. A same-key mutation that encounters retirement waits on that key's mutation-readiness signal before retrying; it does not poll or use a timer.
4. Seal on reload. Reload seals the old epoch before retiring its run. Sealing rejects new scopes and late mutation commits, while already-pinned immutable generations remain readable. The old epoch closes after its exact retained-state census drains; it does not enter or dirty the retired run's census.

flowchart LR
    Change("Prepare new Store generation")
    Stop("Stop affected
running jobs")
    Store("Commit Store generation")
    Restart("Rebuild each job from
its raw graph config")
    Outcome{"Restart outcome"}
    Running("Running with new secrets")
    Failed("Job remains Failed
retry by job policy")
    Restore("Attempt to restore stopped jobs
old generation remains")

    Change --> Stop --> Store
    Store -->|"commit succeeds"| Restart --> Outcome
    Store -->|"commit fails"| Restore
    Outcome -->|"ready"| Running
    Outcome -->|"fails / busy"| Failed

    classDef sec fill:#fee2e2,stroke:#dc2626,color:#0b1021;
    classDef core fill:#fef3c7,stroke:#d97706,color:#0b1021;
    classDef job fill:#dcfce7,stroke:#16a34a,color:#0b1021;
    class Change,Store sec;
    class Stop,Restart,Outcome core;
    class Running,Failed,Restore job;

The Store change remains committed if a later job restart fails, and the graph truthfully shows that job as Failed. A retained busy/contained restart revalidates the Store dependency, source winner, desired config, resource absence, and run generation. A normal probe failure follows the collector's ordinary autodetection-retry policy. secrets/pending.go.

Two rules that surprise people:

• A Store update restarts only Running dependents. Non-running graph configs keep their raw references and resolve the current generation when next started.
• Removing a Store is rejected while any graph config references it, running or not. Only DynCfg-sourced Stores are removable.

Vnodes (Virtual Nodes)

A single job often monitors many remote things — one job scraping 50 switches, or one cloud collector pulling hundreds of resources. Netdata wants each to appear as its own node in the UI, with its own hostname and charts, not collapsed under the agent's host. A vnode is a lightweight, agent-declared "virtual host" (name, hostname, GUID, labels) that a job can attribute its metrics to.

Think of name badges at a conference: the agent prints a batch up front and can print more on demand. When a job reports a metric it wears a badge, so the dashboard files it under that identity instead of "the agent."

• Configured vnodes are loaded from vnodes/ config files at startup and passed once as InitialVnodes (plugin/agent/setup.go → plugin/agent/agent.go → composition). At startup they are published to the daemon as DynCfg config entries.
• Runtime vnodes can be added, edited, or removed live through a DynCfg vnode Function (composition/vnodes.go).
• The vnode authority (discovery/vnode.go) is revision-versioned and live-merged: a job's vnode: name is resolved against the current set of file-configured and runtime vnodes, not a frozen startup snapshot. The resolved snapshot is attached to the job so its runtime emits under that virtual host.
• A vnode cannot be removed while a job references it (409), and only runtime (DynCfg-sourced) vnodes are removable (405).

Jobs With Dependencies

A job may use no external dependency, secrets, a configured vnode, or both. All variants pass through the same source selection and candidate lifecycle.

flowchart TD
    Raw("Selected raw config
secret refs stay unresolved")
    Vnode{"Configured vnode
named?"}
    Snapshot("Capture revisioned
vnode snapshot")
    Secrets{"Secret refs
present?"}
    Resolve("Pin Store generations
resolve cloned config")
    Check("Collector Init + Check
private candidate state")
    Fresh{"No Store refs, or pinned
generations still current?"}
    Settle("Commit graph + dependency index
then attach live vnode lookup")
    Run("Running job")
    Transient("Selected job Failed / removed
normal retry policy")
    Retry("Reject stale candidate
DynCfg 503 / discovery latest retry")
    Reject("Invalid proposal rejected
incumbent unchanged")

    Raw --> Vnode
    Vnode -->|"yes and found"| Snapshot --> Secrets
    Vnode -->|"yes but missing"| Transient
    Vnode -->|"no"| Secrets
    Secrets -->|"yes"| Resolve
    Resolve -->|"all resolve"| Check -->|"ready"| Fresh
    Resolve -->|"provider / scope unavailable"| Transient
    Resolve -->|"invalid reference / config"| Reject
    Secrets -->|"no"| Check
    Check -->|"fails / contained"| Transient
    Fresh -->|"yes"| Settle --> Run
    Fresh -->|"no"| Retry

    classDef cfg fill:#dbeafe,stroke:#2563eb,color:#0b1021;
    classDef dep fill:#fee2e2,stroke:#dc2626,color:#0b1021;
    classDef core fill:#fef3c7,stroke:#d97706,color:#0b1021;
    classDef job fill:#dcfce7,stroke:#16a34a,color:#0b1021;
    class Raw cfg;
    class Vnode,Snapshot,Secrets,Resolve,Fresh dep;
    class Check,Settle core;
    class Run,Transient,Retry,Reject job;

Secret-dependent job

• The collector graph stores the raw config with references, never resolved credential values.
• Candidate preparation resolves a clone atomically: either every reference is resolved under one pinned Store scope, or no resolved config reaches the collector.
• Before releasing that scope, resolution captures the exact numbered Store generations used by the clone. After Init and Check finish and the job-graph claim is reacquired, installation validates all captured generations under one Store lock.
• A generation changed or removed during construction makes the candidate stale. It is cleaned without installation; synchronous DynCfg returns retryable 503 and preserves the incumbent, while persistent discovery retains only its latest desired config for retry.
• An unavailable provider or reader scope is a transient activation failure. An invalid reference or invalid resolved config is a proposal rejection and leaves the incumbent unchanged.
• The raw dependency set and the graph mutation commit together. This prevents a running job from becoming invisible to a later Store update.
• The generation check also covers a brand-new job and a replacement that introduces a new Store reference before either dependency postimage exists. Store mutation never waits for a non-cooperative candidate Check; the stale candidate may finish physically later, but it cannot install.
• ${store:...} dependencies participate in live Store restart orchestration. ${env:...}, ${file:...}, and ${cmd:...} are resolved at build time but have no live Store generation to watch.

Vnode-dependent job

• A named configured vnode must exist when the candidate is built. If it is missing, construction fails transiently and the selected job follows its normal retry policy.
• The candidate uses a private, revisioned vnode snapshot during Init and Check. Only a successfully installed job switches that lookup to the live vnode authority.
• Updating a vnode does not restart its jobs. Running jobs adopt a newer revision at their runtime refresh point and re-emit host metadata when needed.
• Adding a previously missing vnode does not directly push a job restart. Its retained retry or a later config event must reconcile the job.
• A collector-supplied vnode takes precedence over a configured vnode. The runtime advances past configured revisions without replacing the collector-owned identity.

Job that uses both secrets and a vnode

The two dependencies are staged independently, then join the same candidate. This matters when a vnode update overlaps a slow collector Check:

sequenceDiagram
    participant S as Selected raw config
    participant J as Candidate
    participant V as Vnode authority
    participant K as SecretStore
    participant C as Collector
    participant R as Installed runtime

    S->>J: Start selected job attempt
    J->>V: Capture current vnode revision
    J->>K: Pin generations and resolve a clone
    K-->>J: Complete resolved config + generation proof
    J->>C: Init and Check with private staging
    V-->>V: A newer vnode revision may commit
    C-->>J: Ready
    J->>K: Validate generation proof
    K-->>J: Current
    J->>R: Install and attach live vnode lookup
    R->>V: Refresh after attachment
    V-->>R: Return newest committed revision

Consequences:

• A Store update restarts the job from its raw graph config. The replacement resolves the new Store generation and captures the current vnode snapshot.
• A Store update or removal that overlaps candidate Init / Check invalidates the generation proof, so the candidate is rejected before installation and follows its source-specific retry contract.
• A vnode update alone does not restart the job or re-resolve secrets.
• A vnode update during Check is not lost: the candidate sees its staged snapshot while detached, then the installed runtime catches up through the live revisioned lookup.
• If either dependency cannot be prepared, no half-resolved or half-attached candidate becomes live.

joboutput/config_factory.go, joboutput/runtime_staging.go, secrets/dependency.go, secrets/restart.go, discovery/vnode.go, plugin/framework/jobruntime.

Service Discovery

Service-discovery configuration is materialized before a pipeline manager can start it. One contained attempt owns payload/descriptor parsing, user-config rendering, ParseJSONConfig, discoverer construction, and pipeline.New; the manager accepts only an already-prepared pipeline through StartPrepared/RestartPrepared.

• The controller materializes and applies configurations serially, after deterministic source-winner selection.
• Each materialization is individually contained, so a non-cooperative identity cannot occupy the controller loop beyond its logical containment deadline.
• DynCfg test builds a complete temporary pipeline under a payload-specific test identity and never submits it to the pipeline manager. It invokes dyncfg.Testable.Test(ctx) sequentially on every discoverer that provides the optional capability. After each callback returns, the shared Pipeline Test boundary makes the caller's cancellation cause take precedence over the discoverer result. Discoverers classify their configured timeouts but need not repeat caller cancellation checks.
• A Testable discoverer can return dyncfg.ErrTestUnsupported when the resource type has an operational test but the configured instance cannot run it safely. The pipeline continues testing later discoverers and reports the aggregate as validation-only; the sentinel does not turn real failures into success.
• Resource-authored parsing, construction, and operational failures retain their causes internally but cross one sanitized response/diagnostic boundary. Unmarked failures render only their generic phase. A dyncfg.PublicError may add static, code-authored detail; its public message must never derive from submitted/resolved values, endpoints, credentials, backend errors, or response bodies. Resource callbacks return failures rather than logging raw configuration or endpoint material.
• A discoverer without the capability produces an explicit validation-only success. Configuration/construction failures return 400, operational failures return 422, and busy/contained attempts return 503.
• Operational guarantees are discoverer-specific. Docker performs a one-item container-list query. net_listeners executes and parses one production helper snapshot. HTTP performs one production fetch only for empty/default or exact GET: it uses the configured request/client path, requires HTTP 200, enforces the 10 MiB response limit, and parses all targets. Redirect handling permits at most ten actual requests. These tests discard targets without rule rendering, cache reconciliation, publication, or installation. A non-empty HTTP method other than exact GET, plus Kubernetes and SNMP, remains validation-only. Local-listener process count and elapsed time are bounded to one invocation and the caller/configured timeout; its work and buffered output still scale with the host socket/process inventory.
• File-backed stock/user state keeps one latest pending retry after a busy/contained result; synchronous DynCfg commands do not.
• The complete service-discovery DynCfg Function is also contained, so a non-cooperative command cannot pin the Function catalog or the Job Manager run.

plugin/agent/discovery/sd/materialization.go, plugin/agent/discovery/sd/pending.go, composition/service_discovery.go.

Functions

Stable handler bundles

Collector callbacks are not called while rebuilding the shared Function catalog. Instead, each job (and each agent-level module) stages one stable process-owned handler bundle outside controller locks:

• immutable catalog generations acquire cheap references to an existing bundle;
• replacing one job never reconstructs handlers for its siblings;
• retirement closes route admission first, then cleanup waits for every catalog and in-flight callback reference to drain;
• availability polling and handler invocation run as contained attempts outside the controller and process-control loops;
• a handler invocation or availability poll that settles while its callback remains physically live quarantines only its bundle; new callbacks for that bundle are rejected until every callback covered by quarantine physically returns, while unrelated bundles remain available.
• containment runs the bundle fence before logical settlement becomes observable, so admission cannot race ahead of quarantine while an Await caller is waiting to be scheduled.
• a panic from a handler invocation is recovered at the bundle boundary and permanently closes that bundle. Later requests receive an unavailable response without re-entering the handler or accumulating one process-lifetime quarantine tombstone per request.

functions/bundle.go, functions/module_stage.go, functions/controller.go, containment/authority.go, process_attempt.go.

DynCfg is not a published Function

• DynCfg config prefix routes are private catalog routes, not Function publications. Netdata owns the global config Function that serves the tree and delegates per-config operations; go.d emits CONFIG object frames but never FUNCTION GLOBAL "config" or its withdrawal.
• Go CONFIG emission fails closed at the shared pkg/netdataapi boundary. Bare protocol identities remain strict; single-quoted metadata accepts ordinary internal backslashes (including Windows paths) but rejects controls, the quote delimiter, and a trailing escape that would consume the closing quote.

Restart and Shutdown

Job Manager separates two lifetimes:

• The process is the building. Built once by composition.NewProcess, it survives every reload: the stdin reader, the one FrameOwner, the accepted-cleanup output capability, the UID ledger, the frozen module registry, the secret resolver, the process-attempt and Store epoch authorities, the vnode registry, and the runtime metrics service.
• The run generation is the current tenant. A complete, self-contained occupant built by composition/run.go: the kernel and its loop, the task supervisor, the run supervisor, the DynCfg graph, the run-owned SecretStore controller/dependency projections, Function catalog projections and publications, the job factory, the autodetection scheduler, and the jobmgr.runtime metrics.

A SIGHUP reload tries to evict the whole tenant and move a fresh one in without touching the building. The host gives the complete rotation one 30-second budget. If it expires, or a quarantined agent-module identity makes an in-process successor unsafe, the plugin exits with status 0 and the daemon rebuilds the process. Unexpected construction, cleanup, validation, and invariant failures remain failures and exit nonzero.

flowchart TD
    HUP("SIGHUP → Restart")
    Budget("Start one 30s
rotation budget")
    Seal("Seal Store epoch")
    Cut("Cut run-target attempts")
    Ingress("Seal stdin ingress
publish stopping cut")
    Drain("Drain run-owned work
within shutdown budget")
    Census("Require exact-zero run census
detach projections")
    Next("Construct + adopt
next generation")
    Retained("Process authority retains
non-cooperative physical work")
    Recover{"Rotation disposition"}
    Fresh("Exit 0
daemon starts fresh process")
    Fail("Exit nonzero")

    HUP --> Budget --> Seal --> Cut --> Ingress --> Drain --> Census --> Next --> Recover
    Drain -->|"30s expires"| Recover
    Recover -->|"success"| NextRun("Successor running")
    Recover -->|"deadline or explicit
restart-required quarantine"| Fresh
    Recover -->|"unexpected failure"| Fail
    Cut -. "physical release later" .-> Retained

    classDef ext fill:#dbeafe,stroke:#2563eb,color:#0b1021;
    classDef core fill:#fef3c7,stroke:#d97706,color:#0b1021;
    class HUP,Fresh,Fail ext;
    class Budget,Seal,Cut,Ingress,Drain,Census,Next,Retained,Recover,NextRun core;

The rotation is an acknowledged sequence (composition/process.go, retireForSuccessor):

1. Seal the old Store epoch, so no new old-run mutation can commit.
2. Cut every process attempt targeting the retiring run. Their callers settle immediately; their physical workers stay process-owned.
3. Seal stdin ingress and arm the run shutdown budget from the remaining caller-owned rotation deadline before stopping the run.
4. Drain run-owned work — tasks, claims, permits, retries, Function publications, and projections — then drain paused ingress. A process-owned worker that ignores cancellation remains in the process census, not the retired run census.
5. Require a fully drained run authority census and finish the run finalizer.
   • Live run-owned leftovers still make the terminal state dirty.
   • Process-owned retained work does not fabricate run ownership.
   • If an ordinary frame write is the only remaining owner, the kernel arms one shutdown-only idle notification before sleeping. Frame completion wakes it to recensus, and the terminal decision validates that same snapshot rather than taking a racy second census.
6. Construct, start, and adopt the next generation. Restart acknowledges only after the successor run is running. Startup uses the rotation context, but the accepted kernel uses the process context — so cancelling an acknowledged restart cannot stop the new run.
7. Classify an incomplete rotation at the host boundary. See the table below.

An incomplete rotation is never left half-applied: there is no degraded generation and no second shutdown phase after the rotation deadline. The host maps the outcome to an exit status:

Two rules keep that classification honest:

• Typed causal provenance. Deadline-generated kernel, terminal-census, and run-non-quiescence errors are recoverable only when the same complete error tree also contains the rotation deadline. Any independent error leaf fails closed.
• Acknowledge before finalizing. A known unexpected transition failure is acknowledged before process finalization, so a slow finalizer cannot let the deadline relabel a real failure as recovery.

An old installed collector runtime that is still retained leaves its job unavailable/pending in the new run until the physical identity releases. Agent-level module Function bundles are different: their identity is canonical across generations. A released-but-quarantined module bundle cannot be reconstructed safely in the same process, so rotation requests a fresh process instead of bypassing the quarantine with an epoch-specific key.

The process command receiver exists during initial startup and rotation. Terminate can cancel an active transition instead of waiting behind its startup barrier.

Termination (SIGINT/SIGTERM) follows the same retirement path with no successor, then begins process-authority shutdown:

• One budget, not two. The host spends a single 10-second budget on both the Terminate acknowledgement and the wait for the run loop; it never starts a second 10-second wait.
• Report, don't block. Retained physical work is reported once the budget expires rather than waited on forever. Only actual process exit can reclaim a permanently blocked goroutine.
• Fence cleanup output last. After the authority drains or the budget expires, the process terminally fences accepted-cleanup output before finalizing the output service. A contained collector that returns later still releases its resources, but cannot emit a late terminal frame.

Runtime Metrics

In long-lived agent mode, one component — jobmgr.runtime — projects live orchestration counts (composition/runtime_metrics.go):

• process attempts: active, probing, admitted, contained, and quarantined identities;
• operations: admitted, active, rejected, timed out, duplicate-UID rejected, shutdown-rejected, results disposed;
• claims: keys tracked, waiters, oldest wait age;
• tasks: active, queued, oldest wait age, panics;
• jobs: active jobs, active Function invocations;
• frames: commits and failures;
• runs: dirty runs; plus the oldest live operation age.

Mutation owners write metric-owned atomics; the producer snapshots those atomics and the process authority's exact attempt census once per update. It never reads kernel-private state.

The component is registered before external admission opens and unregistered (with a final projection) when its generation retires, strictly before the successor re-registers. Run-owned samples do not cross a reload. Process-attempt gauges deliberately can: the authority survives reload, so a new run continues to expose an older physically retained attempt until it releases.

Package Map

Dependency rules

The layering is enforced by architecture_test.go, not just convention:

• lifecycle is neutral. It imports no sibling, no adapter, and no Agent or collector package — only the standard library. Domain policy (which frame is a keepalive, when to go dirty) is supplied by the caller.
• Adapters do not import each other. containment, functions, joboutput, secrets, and discovery may import the root command ports and lifecycle, but never a sibling adapter.
• composition is the only assembler. It is the single package allowed to join adapters, break construction cycles, and own the process/run-generation split. It is also the only package that imports containment.

architecture_test.go additionally checks the shipped-root/composition construction boundary and that on-loop actions are dispatched only through the sanctioned kernel funnel. Behavioral ownership guarantees belong in focused or black-box tests rather than an exact private-type or source-file manifest.

Where To Change Things

Validation

Useful focused checks after changes:

cd src/go
env GOCACHE=/tmp/netdata-go-build-cache go test -count=1 ./plugin/agent/jobmgr/...
env GOCACHE=/tmp/netdata-go-build-cache go test -race -count=1 ./plugin/agent/jobmgr/...
env GOCACHE=/tmp/netdata-go-build-cache go vet ./plugin/agent/jobmgr/...

Job Manager is concurrency-sensitive: the -race run is not optional for changes to the kernel, claims, staging, tasks, or the run/shutdown paths.

When a change touches shared framework code that Job Manager consumes (plugin/framework/jobruntime, pkg/metrix, the chart engine), also build and test a couple of representative real collectors so the change is proven against real users, not only against Job Manager's own tests.