design/ipam/ipam-gc.md
The IPAM GC lives in kube-controllers/pkg/controllers/node/, not pkg/controllers/ipam/. There is no ipam controller; if
you came looking for the GC and didn't find it, look in node.
Main file: ipam.go (~1600 lines). Supporting types in ipam_allocation.go. Pool / block mapping in pool_manager.go. Cross-component picture is in the index; the
block and handle state machines are defined in ./ipam-datastore.md and ./ipam-core-library.md - this file references them
rather than restating.
The controller is a single goroutine fed by syncerUpdates, syncChan, nodeDeletionChan, and podDeletionChan. Events are coalesced via utils.ProcessBatch. Periodic interval
is LeakGracePeriod / 2, defaulting to 5 minutes; dirty-only scans run between full scans.
Invariants:
checkAllocations (ipam.go:823) walks every allocation on every scanned node and classifies it. The decision tree:
windows-reserved: skip
not pod and not tunnel: mark node "can't delete"
tunnel address: defer until node-deletion decision
allocationIsValid(preferCache): markValid
!kubernetesNodeExists: markConfirmedLeak (immediate)
isVMAllocation: markLeak(max(5min, leakGracePeriod))
leakGracePeriod set: markLeak(leakGracePeriod)
After the walk, if every allocation on a node is released and no valid ones remain, the node's tunnel IPs are confirmed-leaked and the node is added to nodesToRelease.
allocationIsValid (ipam.go:999) is the truth check. The design-relevant rules:
pod / namespace attributes ⇒ assume valid. Conservative on purpose. Tightening this without a real reason will release allocations made by paths that don't
stamp the attrs.wep.Spec.IPNetworks to be valid. A single-WEP check would falsely release
secondary-network IPs.Candidate vs confirmed leak is timer-based: markLeak(grace) sets leakedAt; once time.Since(leakedAt) > grace the allocation is confirmed. markValid clears both fields if a
later sync re-validates the allocation. The transition is what the grace periods protect.
Review notes
allocationIsValid compares pod.Spec.NodeName against attrs["node"]. Stale source data makes valid allocations look like leaks -
https://github.com/projectcalico/calico/issues/12257. Don't expand the comparison without thinking about stale-source-data scenarios.pod / namespace attributes must remain "assume valid." Tightening this without a real reason will cause spurious releases.ipam.go:1043 TODO on pod-NodeName mismatch is unresolved. Currently releases the old allocation; the comment expresses uncertainty. Don't touch without understanding the
migration cases.Three grace periods coexist and defend against different races.
Leak grace period (leakGracePeriod, config). Generic pod-allocation grace. Defaults from KubeControllersConfiguration; zero means "GC disabled for that allocation class."
Set on markLeak; transitions candidate => confirmed only after time.Since(leakedAt) > grace. Defends against the pod-restart-between-syncs race: sync N sees the pod gone and
marks leakedAt; sync N+1 sees the new pod on the same allocation and markValid clears the timer before it expires.
VM recreation grace period (defaultVMRecreationGracePeriod = 5 * time.Minute, ipam.go:81). Floor for VM allocations. Even with a shorter leakGracePeriod, VM allocations
use max(5min, leakGracePeriod) so VM restart and live migration can complete without the GC yanking the IP out from under the new pod. Without this floor, a live migration that
paused on the destination side longer than the leak grace would release the IP and reassign it elsewhere before the migration finished.
Empty-block grace period (blockReleaseTracker, ipam_allocation.go:26). Two-observation rule: a block must be observed empty on two consecutive syncs, spanning the grace
period, before its affinity is released. First markEmpty(cidr) returns false and records the timestamp; the second returns true once the grace period has elapsed. markInUse
clears the timestamp on any allocation activity. Defends against the "block emptied transiently while a fresh pod is about to claim it" race - releasing too early forces a fresh
affinity claim, which is costly and can flap with the dataplane.
Final re-validate before release. Independent of grace periods, garbageCollectKnownLeaks (ipam.go:1229) calls allocationIsValid(a, preferCache) once more for every
confirmed leak immediately before passing it to ReleaseIPs. This is the last defence against the pod-restart race: a pod that came back between scan and release will re-validate
here and skip the release call entirely.
Review notes
./ipam-cni.md.garbageCollectKnownLeaks is non-negotiable. Removing it for "optimization" reintroduces the pod-restart-race class of bug../ipam-core-library.md on ReleaseOptions.SequenceNumber.The invariant: all-or-none per handle. Either every IP on a handle is confirmed-leaked and gets released together, or the entire handle is skipped this sync. Mixing the two
states desyncs the per-block IPAMHandle counters from the actual block bitmap and the controller permanently loses track of live IPs.
handleTracker (ipam_allocation.go:73) is the per-handle view: handle ID => set of allocations the GC believes share it. setAllocation and removeAllocation keep it in sync
with the in-memory allocation maps. The key method is isConfirmedLeak (ipam_allocation.go:91), which returns true only when every allocation associated with the handle is
confirmed-leaked.
garbageCollectKnownLeaks (ipam.go:1229) gates every release through this check:
if !c.handleTracker.isConfirmedLeak(a.handle) {
continue
}
If even one IP on a handle is still valid, the entire handle is skipped this sync. Otherwise the per-block IPAMHandle counters (Block[blockCIDR]int) would desync from the
actual block bitmap and the controller would lose track of the live IPs.
Periodic sweep: every full sync rebuilds confirmedLeaks via checkAllocations, calls garbageCollectKnownLeaks, then cold-IP GC,
releaseUnusedBlocks, and releaseNodes (syncIPAM). Unreleased items roll over to the next sync.
The GC calls ReleaseIPs rather than ReleaseByHandle because it operates at the per-allocation level (ReleaseOptions), preserving sequence-number
protection for each ordinal.
Review notes
handleTracker.isConfirmedLeak for an "optimization" -
https://github.com/projectcalico/calico/pull/12713 codifies this as a safety rule.ReleaseIPs plumbs sequence numbers through ReleaseOptions. Any new release path here must do the same.ipam.go:1281 (log.Fatalf("BUG: unable to find allocation for release options")) is reachable if released options can't be mapped back to tracked allocations.A released IP sits in cooldown - its attribute stamped with ReleasedAt, its ordinal still in Allocations - until it is deallocated. Deallocation happens in garbageCollect,
which the core library runs on every block read but only persists on write paths (see IP release and cooldown). A block that sees
no further allocation or release activity is never rewritten, so its cooled-down IPs would never be deallocated. garbageCollectColdIPs is the backstop that deallocates them.
Invariants:
The controller keeps a coldBlocks map - block CIDR to the earliest ReleasedAt among that block's cooldown IPs - built up incrementally from streamed block updates in
onBlockUpdated. The backstop visits only those blocks, and only once that earliest timestamp plus IPCooldownSeconds has passed. When nothing is in cooldown it does no work. The
global IPAM config (for IPCooldownSeconds) is cached and refreshed on the periodic sync rather than read from the datastore on every trigger.
Review notes
coldBlocks is in-memory state derived from the block stream, like every other map in this controller. It must satisfy assertConsistentState (every CIDR present in allBlocks)
and be cleaned up on block deletion. A new mutation needs the consistency check, or it joins the memory-leak class of bug.Invariants:
releaseUnusedBlocks (ipam.go:746) walks emptyBlocks and releases via ReleaseBlockAffinity(mustBeEmpty=true). The GC's job is to ensure correctness before calling - the
mustBeEmpty precondition in libcalico is a backstop, not the primary safety. Two skip rules carry design weight: a block that is the node's only affinity block is held (otherwise
the next pod allocation pays for a fresh claim) and a node mid-Flannel-migration is held entirely (the migrator is still wiring up its initial state).
Interaction with StrictAffinity. When StrictAffinity=true, a node cannot allocate from blocks it doesn't own. Releasing an empty block on a strict-affinity node forces the next
pod allocation to claim a fresh block. Under contention, multiple nodes race for the same free block and lose on CAS; with tight pools the allocation fails outright. The grace
period and single-block floor exist precisely so this churn doesn't compound.
Interaction with MaxBlocksPerHost. When MaxBlocksPerHost is non-zero, the per-node block cap is enforced at AutoAssign time (see
./ipam-core-library.md). The GC does not enforce or read this value - it releases empty blocks regardless. But the doc-vs-code mismatch on the default
is a recurring complaint (https://github.com/projectcalico/calico/issues/9462); if you touch the default, also touch the docs.
Review notes
mustBeEmpty=true is a hard precondition - the caller must have verified emptiness. The two-observation grace period is what makes the GC's verification trustworthy.StrictAffinity interaction: releasing too eagerly on strict-affinity nodes causes claim-churn between nodes. Tune the grace period, don't remove it.MaxBlocksPerHost defaults are doc-vs-code drift (https://github.com/projectcalico/calico/issues/9462). Touching the default requires a docs update.ReleaseIPs is bounded per sync (a soft DoS guard against catastrophic leaks) and unreleased items roll over via confirmedLeaks rather than failing the sync. Retry uses
workqueue.NewTypedMaxOfRateLimiter - exponential backoff plus a token bucket - so a controller hammering against a flapping datastore self-throttles instead of compounding the
storm. Constants and exact bucket sizes live in ipam.go; the design contract is that they exist, not what their current values are.
Partial-failure handling is the normal case under load: per-block CAS contention against active allocators is expected, ReleaseIPs returns the subset actually released, and the
rest roll over. Code that treats partial release as fatal will produce noisy alerts with no operational signal.
Review notes
ReleaseIPs partial-failure is normal under load.Per-pool, per-node gauges, registered lazily as pools appear. The two metrics that carry operational meaning beyond "current count" are:
ipam_allocations_gc_candidates - allocations in candidate-leak state. Returns to zero after the grace period elapses. A sustained non-zero value means the GC is stuck,
typically on a handle conflict (one IP on the handle is still valid, so the entire handle is held).ipam_allocations_gc_reclamations - counter of successful leak releases. In a healthy cluster this is near-zero and stable. Sustained increase is the signal that there's a
real leak source to investigate.ipam_allocations_in_use, ipam_allocations_borrowed, and ipam_blocks are trend metrics; their absolute values vary with pool size and pod count. Legacy single-dimension
variants exist for backward compatibility.
updateMetrics recomputes from scratch every sync - one walk over all blocks, no incremental state. The full-recompute is the consistency check; switching to incremental updates
without a separate consistency check loses the protection.
ipam_ippool_reserved is the exception to that walk: reservations make addresses unassignable without allocating them, and can cover pool space no block has been carved from, so the
number isn't in the block state the controller tracks. IPReservation is therefore a fourth kind on the controller's syncer, cached by name in reservations, and
updateReservedMetrics counts the covered addresses per pool with ipam.NumReservedIPsInCIDR (see ipam-core-library). The arithmetic is
the library's, so the gauge agrees with calicoctl ipam show; the input is the syncer's, so the sync loop makes no datastore request for it. Being per-pool rather than per-node, the
gauge is labelled ippool only, like ipam_ippool_size. It may overlap ipam_allocations_in_use, so usable capacity is
ipam_ippool_size - ipam_allocations_in_use - ipam_ippool_reserved only when no reserved address is also allocated.
Watching IPReservation needs watch in the kube-controllers ClusterRole, in the chart and in tigera/operator. With only list granted the List still succeeds and the syncer
still reaches in-sync, so the symptom is a hot re-list of IPReservations rather than a stalled controller - easy to miss in review, noisy in production.
Review notes
ipam_allocations_gc_candidates > 0 for extended periods is the canonical "GC is stuck" signal. Alert on it.ipam_allocations_gc_reclamations rate is the canonical "we have a real leak somewhere" signal. Alert on it.updateMetrics to incremental updates without a separate consistency check. The current full-recompute is the consistency check.ipam_ippool_reserved was caught doing a LIST of every block per sync in review (https://github.com/projectcalico/calico/pull/13331).assertConsistentState in ipam_test.go is the canonical invariant check; any new map mutation needs a test that exercises it.
The v3.32 memory-leak family (https://github.com/projectcalico/calico/pull/12277, /12286, /12287, /12288) all came from "added to one path, forgot another."Two harnesses cover the GC and they are not interchangeable:
assertConsistentState in kube-controllers/pkg/controllers/node/ipam_test.go is the canonical end-of-test
invariant check. It cross-walks every in-memory map (allBlocks, allocationsByBlock, allocationState, handleTracker, confirmedLeaks, nodesByBlock, blocksByNode,
emptyBlocks) and asserts they agree. Every test that mutates the controller's state must call it. The v3.32 memory-leak family
(https://github.com/projectcalico/calico/pull/12277, /12286, /12287, /12288) all came from "added to one path, forgot another" - the consistency check catches that class
directly.hack/cmd/ipam-hammer/ is the race-reproduction harness for allocation and GC paths. Use it before declaring a race-fix complete. Unit
tests can't exercise the CAS / timing windows that hammer can.Review notes
assertConsistentState call in the corresponding test. Skipping it ships a memory-leak class bug.ipam-hammer run on the before/after binaries is not done. Manual reasoning is not a substitute.libcalico-go/lib/ipam/ipam_block_test.go rather than
adding ad-hoc tests; the table is where reviewers look first../ipam-datastore.md - block / affinity state machine, sequence numbers, mustBeEmpty=true precondition../ipam-core-library.md - ReleaseIPs / ReleaseByHandle semantics, handle conventions, sequence-number protection../ipam-cni.md - CNI side of the KubeVirt persistence handshake the VM grace period defends.../../kube-controllers/pkg/controllers/node/ - a DESIGN.md stub in that directory will point back here.