Multi-homing BGP with Default Gateway Auto Discovery

Multi-homing BGP means Cilium node is peering with two upstream routers over different links.

Common design requirements

Node is connected to two different top-of-rack switches for redundancy.
Separation of east-west and north-south traffic via different links on the node. This may require different advertisements to each peer.

BGP Resources

BGP instance configuration

In the below example, we define two eBGP peers without peer address. Peer address will be auto discovered using default gateway for each address family. It will create bgp session per peer configured so each address family will have one bgp session , both of which have identical peering configuration defined in CiliumBGPPeerConfig resource with name cilium-peer.

yaml

  bgpInstances:
  - name: "65001"
    localASN: 65001
    peers:
    - name: "ipv6-65000"
      peerASN: 65000
      autoDiscovery:
        mode: "DefaultGateway"
        defaultGateway:
          addressFamily: ipv6
      peerConfigRef:
        name: "cilium-peer"

BGP peer configuration Peer configuration contains various peering settings like transport, authentication and AFI/SAFI configurations.

yaml

  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "pod-cidr"
    - afi: ipv6
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "pod-cidr"

In the above example, both IPv4 and IPv6 address families are enabled and for each address family we advertise CiliumBGPAdvertisement resource which matches the label "advertise=pod-cidr". BGP advertisement Snippet of 'PodCIDR' advertisement is defined below. BGP control plane will advertise pod cidr prefix

yaml

    - advertisementType: "PodCIDR"

BGP node configuration override In below example, router ID is configured manually for each node. Name of the CiliumBGPNodeConfigOverride resource matches the node name on which this configuration will be applied.

yaml

---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPNodeConfigOverride
metadata:
  name: bgp-cplane-dev-multi-homing-control-plane
spec:
  bgpInstances:
    - name: "65001"
      routerID: "1.2.3.4"

---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPNodeConfigOverride
metadata:
  name: bgp-cplane-dev-multi-homing-worker
spec:
  bgpInstances:
    - name: "65001"
      routerID: "5.6.7.8"

Verification

BGP Peering

root@bgp-cplane-dev-mh-worker:/home/cilium# cilium-dbg shell -- bgp/peers         
Instance   Peer         Session State   Uptime   Family         Received   Accepted   Advertised
65001      ipv6-65000   established     1m27s    ipv4-unicast   2          0          1
                                                 ipv6-unicast   2          0          1

BGP Routes

root@bgp-cplane-dev-mh-worker:/home/cilium# cilium-dbg shell -- bgp/routes out ipv4 unicast -a
Instance   Peer         Prefix        NextHop          Age     Attrs
65001      ipv6-65000   10.1.1.0/24   fd00:11:0:2::2   2m21s   [{Origin: i} {AsPath: 65001}]

On peering routers we can see 10.1.1.0/24 prefix with appropriate route attributes and configured router ID.

FRR Router0

 docker exec -it clab-bgp-cplane-dev-mh-router0 vtysh -c 'sh bgp ipv4 10.1.1.0'
% Network not in table

FRR Router1

docker exec -it clab-bgp-cplane-dev-mh-router1 vtysh -c 'sh bgp ipv4 10.1.1.0'
BGP routing table entry for 10.1.1.0/24, version 2
Paths: (1 available, best #1, table default)
  Advertised to non peer-group peers:
  fd00:11:0:1::2 fd00:11:0:2::2
  65001
    fd00:11:0:2::2 from fd00:11:0:2::2 (5.6.7.8)               <<<<<<<<< Router ID
      Origin IGP, valid, external, best (First path received)
      Last update: Sat Apr  5 23:37:36 2025

Failover

if the default route fails for any reason like interface going down, it will trigger a reconciliation and worker will failover to the other router i.e router0

BGP Peering

root@bgp-cplane-dev-mh-worker:/home/cilium# cilium-dbg shell -- bgp/peers
Instance   Peer         Session State   Uptime   Family         Received   Accepted   Advertised
65001      ipv6-65000   established     9s       ipv4-unicast   1          0          1
                                                 ipv6-unicast   1          0          1

BGP Routes

root@bgp-cplane-dev-mh-worker:/home/cilium# cilium-dbg shell -- bgp/routes out ipv4 unicast -a
Instance   Peer         Prefix        NextHop          Age      Attrs
65001      ipv6-65000   10.1.1.0/24   fd00:10:0:2::2   10m39s   [{Origin: i} {AsPath: 65001}]

On one of the peering router we can see 10.1.1.0/24 prefix with appropriate route attributes and configured router ID.

FRR Router0

 docker exec -it clab-bgp-cplane-dev-mh-router0 vtysh -c 'sh bgp ipv4 10.1.1.0'
BGP routing table entry for 10.1.1.0/24, version 2
Paths: (1 available, best #1, table default)
  Advertised to non peer-group peers:
  fd00:10:0:1::2 fd00:10:0:2::2
  65001
    fd00:10:0:2::2 from fd00:10:0:2::2 (5.6.7.8)
      Origin IGP, valid, external, best (First path received)
      Last update: Sun Apr  6 00:27:43 2025

FRR Router1

docker exec -it clab-bgp-cplane-dev-mh-router1 vtysh -c 'sh bgp ipv4 10.1.1.0'
% Network not in table

README

Multi-homing BGP with Default Gateway Auto Discovery

BGP Resources

Verification

Failover