felix/design/bpf-services.md
How the BPF dataplane resolves Kubernetes services: intra-cluster (cali* veth) and external (NodePort, DSR) traffic paths, Maglev consistent-hash backend selection for LB-fronting nodes, session affinity (cali_v?_nat_aff), the BPF kube-proxy replacement (felix/bpf/proxy/), and the connect-time load balancer (CTLB) attached to cgroup hooks.
This is one of several sub-designs for the eBPF dataplane. See
bpf-overview.md for the packet-path mental
model, the fast-path cost rule, and the cross-cutting review notes
that apply to every BPF change. The full set of sub-designs is
listed in felix/DESIGN.md.
Note. Everything in this subsection describes the TC path. When CTLB (Connect-Time Load Balancer (CTLB)) is enabled, a pod's service traffic never takes this path: CTLB rewrites the destination at
connect(2)time and the TC program oncali*only ever sees pod→pod packets with the backend's address. The TC path below applies when CTLB is disabled for the traffic in question (raw sockets always, UDP ifBPFConnectTimeLoadBalancing = TCP, any traffic if CTLB is off entirely).
When a pod sends a packet to a service IP, the packet leaves the pod
and enters the host on the pod's host-side veth (cali*). The TC
program on that veth — attached to the TC ingress hook on the
host-side (host-ingress from the host's point of view, policy
egress from the pod's) — runs first.
The flow is:
(dst_ip, dst_port, proto) in
the NAT frontend map (felix/bpf/nat/, consumed by
felix/bpf-gpl/nat_lookup.h and friends). On a match, it picks a
backend from the NAT backend map and rewrites the packet
(DNAT).On subsequent packets, the forward entry is hit first and there is no need to re-resolve the service. Return packets hit the reverse entry and have the service frontend IP/port restored on the way back to the pod.
When the selected backend is on the same host, the dataplane can
take a significantly shorter path than *tables does. After DNAT
and policy, the BPF program looks up the backend's veth via the
ifstate map and, on established-flow packets, hands the packet
directly to the backend's pod side with bpf_redirect_peer. The
packet never touches the host FIB, never enters the host netfilter
chains, and does not trigger the host-side program of the backend's
veth — the pod receives it straight from its own side of the veth.
fib_approve (bpf-tc-programs.md → TC program layout) is the gate: it checks that the backend's
interface is ready (attach-gap protection) and whether the flow is
confirmed in conntrack; unconfirmed traffic is fallthrough'd to the
host stack so the *tables drop rules apply.
Review note: a change that forces the same-host path through the host stack is a measurable regression. The shortcut is not optional under BPF mode; it's the point of running BPF for this case.
When the backend chosen for a service is the pod that originated the traffic, this would naively resolve to a self-to-self packet.
With CTLB on, the destination rewrite happens inside connect(2)
and the socket pair is set up pod-to-pod-same-address. No packet
is ever emitted on the network — a substantial deviation from
*tables, where every pod-service-self packet makes a MASQ
round-trip through the host. This is one of the reasons CTLB is
an important performance feature (Connect-Time Load Balancer (CTLB)).
Without CTLB, the host would have to loop the packet back, which
fails: accept_local is 0 by default on the pod's veth, and no
socket exists for the (self, self) 5-tuple anyway. Calico handles
this the same way kube-proxy does — it relies on *tables MASQ to
change the source so the packet looks like it came from the host
and the pod accepts it.
The signal is carried on the conntrack entry:
CALI_CT_FLAG_SVC_SELF on the new NAT-reverse entry (see
conntrack_types.h and the NAT-create path in tc.c).CALI_SKB_MARK_MASQ is set on the packet and FIB is disabled. The
packet falls through to *tables, which MASQ-SNATs it.CALI_F_TO_WEP),
the program detects the CALI_SKB_MARK_MASQ mark and restores
the saved original source so the CT reverse-match works.felix/bpf/nat/) — kernel will refuse to pin two layouts
under the same name.*tables on the SVC_SELF path must preserve the
CALI_SKB_MARK_MASQ + no-FIB behaviour, or re-implement the source
restoration; otherwise pod-service-self breaks without CTLB.conntrack_types.h
(CALI_CT_FLAG_*) and, if set on flow creation, must be considered
on every path that reads the flow (forward and reverse).An external client opening a connection to a NodePort lands on the node's main cluster interface (a HEP). The TC HEP-ingress program runs:
(local-host-IP, dst-port, proto) tuple in the NAT
frontend map. If the service exists, pick a backend.VXLAN ambiguity — worth flagging for readers. The VXLAN used here for NodePort forwarding is a separate use of the VXLAN device from the pod-to-pod VXLAN overlay. Calico programs both on the same
vxlan.calicodevice (flow-mode, see bpf-encap-fragments-icmp.md → VXLAN in eBPF mode), but:
- NodePort-forwarding VXLAN (this step) is always present in BPF mode, regardless of whether the overlay uses VXLAN, IPIP, WireGuard, or no encap. It carries external traffic that has hit a NodePort on a node whose selected backend is on a different node. It uses a fixed VNI of
0xca11c0(CALI_VXLAN_VNIinfelix/bpf-gpl/nat.h) — reserving that value so receivers can tell NodePort-forwarding packets from overlay packets on the same device.- Pod-to-pod overlay VXLAN is what pod→pod traffic uses when the cluster's overlay is configured as VXLAN. Its VNI is the operator-configured overlay VNI, not
0xca11c0.A reader familiar with the overlay may assume one implies the other; it doesn't. The BPF program picks per-packet which semantics apply and sets the VXLAN tunnel key (destination node IP + VNI) accordingly.
On the backend node, the VXLAN decapsulation happens and NAT is applied with local-only backend selection (so the same packet is not re-ballooned to yet another node). Policy runs on the way into the backend's veth.
On return, the backend's WEP program sees a packet whose destination is the external client, and whose conntrack entry records that the ingress came from a specific ingress node. The program wraps the return packet back in VXLAN, destined for the node that originally received the connection, and lets the host stack route it out. That node decapsulates and routes the packet to the client.
The return path follows the forward path as a deliberate choice: in non-DSR mode the ingress node holds the conntrack reverse entry keyed on the original client tuple, and returning packets there lets a single un-NAT restore the service-IP source seen by the client. DSR mode is the alternative — the backend node SNATs the return packet itself, so no round-trip to the ingress node is needed (see below).
With DSR enabled, return traffic does not go back through the ingress node. The backend node simply SNATs the return packet (swapping the backend IP for the ingress-node IP so the packet looks like it came from the service) and lets the network deliver the packet directly to the client.
DSR requires asymmetric-path tolerance in the underlay — and not just at the client. Every hop between the backend node and the client must accept an inbound packet from a node other than the one the forward packet was routed to. Cloud underlays and carefully- configured switched fabrics are typically fine; setups with strict uRPF on the first hop are not. The client's distance from the cluster often hides the asymmetry from the client itself, but the local network between the cluster and the first common hop must be accepting of it.
The cluster admin opts in via BPFExternalServiceMode = dsr (vs.
the default tunnel) with BPFDSROptoutCIDRs for per-destination
opt-out. DSR is also a prerequisite for Maglev (Maglev load balancer).
It is legal for a pod to connect to a service via a node IP:port
rather than the service clusterIP. Rather than program a NAT entry
for every node IP × every service, Calico programs a special wildcard
frontend using 255.255.255.255 (IPv4) / all-ones (IPv6) in place of
the node IP (see podNPIP / podNPIPV6 in
felix/bpf/proxy/syncer.go). When a pod's egress packet misses the
regular NAT lookup but the routing table says the destination is a
node, the WEP program retries the lookup against the wildcard entry
on the same port. On a hit, backend selection proceeds as for an
external NodePort request.
An external load balancer in front of the cluster holds connections
to several nodes at the same time. From the LB's point of view each
(LB-IP, src_port) → (node-IP, dst_port) 4-tuple is unique, so the
LB can legitimately reuse the same source port against different
node IPs: the distinct destinations make the 4-tuples distinct.
The collision appears after DNAT. Each node DNATs the incoming
packet from (node-IP, node-port) to the chosen backend pod. If
multiple of these formerly-distinct destinations resolve to the
same backend pod, the post-DNAT flows all look like
(LB-IP, src_port) → (backend-IP, backend-port) — identical 5-tuples
that were different connections when the client originally made them.
Without intervention they collide on the backend pod's conntrack.
Calico resolves this with port-SNAT on the ingress node: on
collision the TC program picks a random source port from a
reserved range (PSNATStart/PSNATEnd on the attach point) and
retries. The resulting tuple is stable for the lifetime of the
flow because the CT entry records it.
Changes to this logic need to preserve port stability within a flow — return packets must still match the CT entry.
255.255.255.255 / all-ones IPv6) fallback used for pods
addressing a NodePort via a node IP. The wildcard entry lives in
the same NAT frontend map as regular entries; changes to the key
layout need to respect both.CALI_F_DSR only with CALI_F_FROM_WEP or CALI_F_HEP
(see the COMPILE_TIME_ASSERT in felix/bpf-gpl/bpf.h). DSR
makes no sense on WEP ingress.Relationship to External traffic (NodePort, DSR). Maglev layers on top of the NodePort VXLAN-forwarding path described in External traffic (NodePort, DSR). The forwarding mechanics — VXLAN-wrap to the backend node, DSR return, conntrack bookkeeping — are reused unchanged. What Maglev adds is consistent-hash backend selection in place of the usual per-node random/round-robin, plus a re-run of policy on mid-flow packets that may have failed over from another lb-node.
Ordinary services pick a backend per connection more-or-less at random (first-available, round-robin, depending on the mode). That choice is node-local: two LB nodes balancing the same service to the same set of backends will pick different backends for the same external client 5-tuple.
Maglev-style services use a consistent-hash backend selection: the choice is a deterministic function of the 5-tuple and the LUT. If node A goes down mid-connection and the network redirects the client's packets to node B, node B picks the same backend that node A was using and the connection can continue through B's conntrack. This is what enables churn-tolerant load balancers in front of the cluster.
felix/bpf/consistenthash/consistenthash.go implements the standard
Maglev table-build: each backend generates a permutation over a prime
number of LUT slots, and the LUT is filled by sequentially picking the
next free slot from each backend's permutation. Felix publishes the
resulting table into a BPF map (key: (svc_id, ordinal), value: NAT
destination; see cali_maglev_lookup_elem /
cali_maglev_key in felix/bpf-gpl/maglev.h).
Backend selection hashes the 5-tuple (plus protocol) with Jenkins
hashing (jenkins_hash.h), reduces modulo the LUT size, and reads the
NAT destination out of the Maglev map. The LUT size is a per-service
parameter carried in globals (MAGLEV_LUT_SIZE).
The selection code is small, but the hashing code was not — the
original inlined form exceeded the kernel verifier's instruction
budget when placed inside the main program. Maglev therefore lives in
its own tail-called sub-program (calico_tc_maglev in tc.c,
registered as SubProgMaglev in felix/bpf/hook/map.go). The main
program detects that the target is a Maglev service and tail-calls
into the Maglev program before policy. Maglev fills in the post-NAT
destination and then tail-calls into policy as any other caller
would.
Only HEP-ingress programs on the main interface need Maglev; the
macro HAS_MAGLEV in felix/bpf-gpl/bpf.h expands to
(CALI_F_FROM_HEP && CALI_F_MAIN), and GetApplicableSubProgs in
hook/map.go only loads the Maglev sub-program for attach types
where this is true.
Before Maglev, a mid-flow TCP packet with no conntrack hit was either
let through to *tables (might match a pre-existing kernel CT entry;
see bpf-conntrack-flowstate.md → Switching from *tables to eBPF) or dropped as unsolicited. Maglev adds a third class: a
mid-flow packet with no BPF CT hit whose destination is a Maglev
service. In this case the lb-node has just failed over onto this
node, so the packet genuinely is mid-flow but this node doesn't know
it yet. The handling is:
CALI_CT_FLAG_MAGLEV
set (see CALI_CT_FLAG_MAGLEV in conntrack_types.h).CALI_CT_FLAG_MAGLEV flag
relaxes that check and updates the stored tunnel IP.The CALI_CT_MID_FLOW_MISS → CALI_CT_MAGLEV_MID_FLOW_MISS transition
in calico_tc_maglev is what propagates the "this was a mid-flow
miss, go through the new-flow path" signal to the rest of the chain.
internalTrafficPolicy and
externalTrafficPolicy=Local restrict the pool of eligible
backends. There is no consistent way to reconcile a deterministic
hash with a node-dependent eligible pool, so these policies are
ignored for Maglev services.CALI_CT_FLAG_MAGLEV. In particular, do not re-enable spoofing
checks for flows carrying that flag.HAS_MAGLEV must keep the Maglev sub-program
optional (not every attach type loads it). Similarly,
GetApplicableSubProgs must not force-load it where it is not
needed — the per-HEP program-array budget is finite.Kubernetes Services can opt into client-IP affinity by setting
sessionAffinity: ClientIP. While the affinity is valid, the same
client IP talking to the same service is pinned to the same backend
pod. Applies to all service types — ClusterIP, NodePort, LoadBalancer,
external or intra-cluster; any path that does BPF backend selection.
A dedicated map,
cali_v4_nat_aff / cali_v6_nat_aff
(felix/bpf/nat/maps.go, maps6.go; sized via
BPFMapSizeNATAffinity), records the affinity:
(service_id, client_IP).(backend, last_used).Backend selection for a new flow (no BPF conntrack hit) runs:
(service_id, client_IP) in the affinity map.The CT entry created for the flow carries the chosen backend, so
per-packet forwarding does not revisit the affinity map. The affinity
map's last_used is updated opportunistically on new-flow backend
resolution; flow-lifetime fast-path packets are not affected.
sessionAffinity=ClientIP, the affinity check runs first. The
Maglev consistent-hash is only reached when there is no affinity
entry.externalTrafficPolicy=Local: affinity only pins
to an eligible backend; if the previously-pinned backend is no
longer eligible (policy excluded it), the entry is re-resolved.felix/bpf/proxy/ is Calico's in-Felix replacement for kube-proxy.
It watches Kubernetes Service, Endpoints and EndpointSlice resources
and translates them into the BPF maps that the TC programs
(bpf-xdp.md → XDP programs and the XDP→TC handoff–Maglev load balancer) and the CTLB (Connect-Time Load Balancer (CTLB)) read. When BPF mode is on, Calico
disables kube-proxy and takes full responsibility for service
implementation.
This is the userspace half of "service NAT" — the TC-side view (Intra-cluster traffic & service NAT–External traffic (NodePort, DSR)) only sees "a map with a frontend pointing at a backend". The proxy package is what fills those maps, keeps them consistent as Services/Endpoints churn, and applies the Kubernetes semantics (topology, traffic policies, health, affinity) before the BPF program ever runs.
syncer.go — the central syncer. Diffs the desired state
(from watchers) against the current state (from the BPF maps)
and applies the delta. Handles frontend, backend, affinity and
Maglev LUT maps.kube-proxy.go — the Kubernetes-facing layer. Reads
Services, Endpoints, EndpointSlices; translates them into the
syncer's internal model.topology.go — implements topology-aware routing
(topologyAwareHints, preferClose, etc.). Filters backends
based on the node's zone/region.health.go — NodePort / LoadBalancer health checks for
services with externalTrafficPolicy=Local; tells external
load-balancers which nodes to avoid.lb_src_range.go — handles loadBalancerSourceRanges ACLs
for LoadBalancer services.rtcache.go — routing-table cache used by topology
decisions.proxy.go / proxy_test.go / options.go — the driver
plumbing.(All under felix/bpf/nat/.)
(service-IP, port, proto) → service_id.(service_id, ordinal) → (backend_IP, port).cali_v4_srmsg / cali_v6_srmsg) — used by
the CTLB's recvmsg hook to undo destination rewrites.A service ID indexes the service's own block of the backend map, so
no two services may hold the same ID — their backends would sit at
the same (id, ordinal) keys and overwrite each other on every sync,
leaving one frontend NATing to the other's backends. A service does
share its ID with all of its own derived (NodePort, ExternalIP,
LoadBalancer) frontends; that is how applyDerived points them at one
block of backends.
IDs are handed out by newSvcID and, on restart, re-adopted from the
maps by startupBuildPrev, which pairs each frontend entry with the
service that must have written it. Re-adoption keeps a restart from
disrupting traffic, but it means Felix trusts the maps — and Felix is
not their only writer:
bpfNetworkBootstrap enabled the ebpf-bootstrap init
container programs the API server service into them on every
calico-node start, before Felix runs (node/pkg/nodeinit).So startupBuildPrev must not adopt an ID it finds shared by two
different services: adopting it makes the conflict permanent, since
applySvc keeps an unchanged service's ID forever and every later
restart re-adopts it. Both services are instead left out of
prevSvcMap, which gives each a fresh ID and rewrites its frontends
and backends; nothing read through a duplicated ID is carried over,
because those backends may belong to the other service.
The bootstrap writer holds up the other end of the invariant: it reuses the ID already recorded for the service it is programming, and otherwise picks one no frontend entry uses, rather than assuming an ID is free.
externalTrafficPolicy=Local
(external traffic prefers local-node backends, drops if none).internalTrafficPolicy=Local similarly for cluster-internal.default/kubernetes API server service is never allowed to
drop to zero backends — see below.With bpfNetworkBootstrap enabled, Felix reaches the API server
through the default/kubernetes ClusterIP service's NAT entry
(KUBERNETES_SERVICE_HOST is the ClusterIP, and the
ebpf-bootstrap init container seeds the frontend/backend from
KUBERNETES_SERVICE_IPS_PORTS / KUBERNETES_APISERVER_ENDPOINTS
before Felix starts — see node/pkg/nodeinit/calico-init_linux.go).
This creates a hazard the generic kube-proxy model doesn't have:
if that service transiently loses all its (ready) endpoints — e.g.
the API server's own endpoint reconciler de-lists it across a
restart — the syncer would write count=0 and delete the backend,
severing Felix's own connection to the API server. Felix can
then no longer learn the restored endpoints, so the NAT stays empty
until calico-node is restarted (which re-seeds it from the init
container). The deadlock is therefore unique to bootstrap mode.
The syncer therefore retains the last-known-good backend for the
API server service whenever an update would leave it with zero ready
endpoints (apiServerFallbackEps in syncer.go, sourced from
prevEpsMap, which is rebuilt from the BPF maps on restart by
startupBuildPrev). The API server's backend (the control-plane
host IP) is stable across such an outage, so the retained backend is
correct; a later update with real ready endpoints overwrites it.
The CTLB is a set of BPF programs attached to cgroup hooks rather than to network interfaces. These hooks fire inside syscalls before any packet is built:
cgroup/connect4, cgroup/connect6 — on connect(2), the program
does a NAT lookup on the destination the application passed and, if
it matches a service frontend, rewrites the sockaddr in-place to
point at the chosen backend. The connection is then established
directly pod-to-pod; no TC program on the way out ever sees the
service frontend IP.cgroup/sendmsg4, cgroup/sendmsg6 — the equivalent for UDP
sendmsg(2).cgroup/recvmsg4, cgroup/recvmsg6 — the reverse-NAT on read, so
a UDP application sees the service IP/port in the received packet
even though the packet on the wire carried a pod IP.Source: felix/bpf-gpl/connect_balancer.c (IPv4),
connect_balancer_v6.c (IPv6 on an IPv6-only cgroup),
connect_balancer_v46.c (dual-stack), with shared helpers in
connect.h and ctlb.h. Userspace lifecycle is in
felix/bpf/nat/connecttime.go.
CTLB eliminates per-packet NAT work: one lookup at connect time and the kernel's socket is talking directly to the backend for the life of the connection.
It also solves the service-access problem for host-networked processes: a host-networked socket's traffic may leave the node on a physical interface that the TC program for its packets cannot predict (the default route could be wrong; the packet could be dropped if there is no route). With CTLB the backend is already resolved by the time the packet is built, so the routing question answers itself.
The downsides are significant enough that Calico wants CTLB to be an optimisation rather than a prerequisite. The bpfnat veth workaround (bpf-host-networking.md → Host-networked workaround (bpfnat veth)) is what lets Felix run without CTLB.
connect(2) on a UDP socket records the chosen
backend once. If the backend goes away, the socket keeps sending
to the dead backend. TCP is not affected because each new
connection runs the CTLB again.The ExcludeUDP knob (CTLB_EXCLUDE_UDP in ctlb.h,
LibBPF CTLBGlobalData.ExcludeUDP on the userspace side) lets an
operator disable the UDP sendmsg/recvmsg hooks while keeping TCP
connect-time resolution, limiting exposure to the UDP stuck-backend
issue.
sendmsg rewrites a destination, the
matching recvmsg must un-rewrite it for the same 5-tuple.
Otherwise applications see the pod IP in response messages and
reject them.ExcludeUDP
knob must preserve the ability to run without CTLB entirely — the
bpfnat workaround depends on CTLB not being required for
correctness, only for performance.A change to how the BPF dataplane works in the area this file covers must update the relevant section in the same PR — new mechanism, new flag, new map field, new config knob, or any change to the packet path. Exemptions: (a) bug fix restoring documented behaviour, (b) mechanical refactor with no observable change, (c) comment / log-message edits, (d) dependency bumps. If in doubt, update.
Cross-cutting rules that apply to every BPF change (map
versioning, mark discipline, sub-program registration, kernel-
version sensitivity) live in
bpf-overview.md → Cross-cutting review notes.