ContextQMD
Back to Rook

Two-Node Cluster

Documentation/Storage-Configuration/Advanced/two-node-cluster-drbd.md

1.20.15.0 KB
Original Source

Two-Node Cluster (experimental)

This guide provides installation instructions for setting up a two-node Kubernetes cluster with Rook using DRBD (Distributed Replicated Block Device) for storage replication for one of the Ceph mons.

For architecture details and design rationale, see the Two-Node Fencing Design Document.

Prerequisites

Before starting the installation:

  • 2-node Kubernetes cluster with fencing configured (e.g., OpenShift with DualReplica topology)
  • Raw block devices on each node (SSD-class, same size, no filesystem)

Installation Steps

Step 1: Install DRBD Kernel Modules

DRBD kernel modules must be installed and loaded on both nodes before running the setup script.

Follow the DRBD installation documentation for your distribution and install drbd & drbd_transport_tcp:

Verify Installation

On both nodes, verify DRBD modules are loaded:

bash
# Check loaded modules
lsmod | grep drbd

# Expected output:
# drbd_transport_tcp     16384  0
# drbd                  614400  1 drbd_transport_tcp

# Check DRBD version
drbdadm --version

Step 2: Identify Block Devices

Use the setup script to list available block devices on both nodes:

bash
./deploy/examples/drbd-setup-experimental.sh -l

Example output:

text
=== Block devices (node0=node-0, node1=node-1) ===
Use the PATH column (e.g. -d /dev/sdb or -d0 / -d1 per-node paths).

--- node-0 ---
NAME   PATH      SIZE    ROTA TYPE FSTYPE
sdb    /dev/sdb  10G    0    disk        <-- Use this

--- node-1 ---
NAME   PATH      SIZE    ROTA TYPE FSTYPE
sdb    /dev/sdb  10G    0    disk        <-- Use this

Same path on both nodes: -d <path>
Different paths (same size): -d0 <path0> -d1 <path1>

Requirements for backing devices:

  • Must be SSD/NVMe (ROTA=0, non-rotational)
  • Can be a partition of the disk
  • Same size on both nodes
  • No existing filesystem (FSTYPE empty)
  • Writable (not read-only)
  • Volume or partition is not larger than 10Gi

Step 3: Run DRBD Setup Script

The setup script configures DRBD, performs initial sync, and creates the auto-start DaemonSet.

Same Device Path on Both Nodes

bash
./deploy/examples/drbd-setup-experimental.sh -d /dev/sdb

Different Device Paths Per Node

bash
./deploy/examples/drbd-setup-experimental.sh -d0 /dev/sdb -d1 /dev/sdc

Available options:

./deploy/examples/drbd-setup-experimental.sh -h

!!! note Network connectivity on DRBD replication port (default: 7794) between nodes, we can customise this in the script.

Step 4: Verify DRBD Setup

Check auto-start DaemonSet:

bash
kubectl get daemonset drbd-autostart -n rook-ceph
kubectl get pods -n rook-ceph -l app=drbd-autostart

Check for the drbd-configure configmap

bash
kubectl get cm -n rook-ceph drbd-configure -o yaml

Step 5: Deploy Rook CephCluster

Create the CephCluster CR with floating monitor configuration:

Apply the cluster:

bash
kubectl apply -f deploy/examples/cluster-tnf.yaml

Step 6: Verify Ceph Cluster

Check monitor pods:

bash
kubectl get pods -n rook-ceph -l app=rook-ceph-mon

Expected: 3 monitor pods (mon-a, mon-b, mon-c)

Check Ceph status:

bash
kubectl exec -n rook-ceph deploy/rook-ceph-tools -- ceph status

Expected output:

text
    cluster:
        id:     <cluster-id>
        health: HEALTH_OK

    services:
        mon: 3 daemons, quorum a,b,c (age ...)
        ...

Configuration Summary

The setup creates:

  • DRBD resource (r0) with Protocol C synchronous replication
  • DRBD device (/dev/drbd0) with XFS filesystem
  • Auto-start DaemonSet (drbd-autostart) in rook-ceph namespace
  • Configuration ConfigMap (drbd-configure) in rook-ceph namespace
  • Two pinned monitors (one per node) with local storage
  • One floating monitor (mon-c) backed by DRBD

Two-Node Cluster Recommendations

Configure pools with replica-2, pool-tnf.yaml

Troubleshooting

DRBD Modules Not Loaded

Check:

bash
lsmod | grep drbd

Fix: Install DRBD kernel modules (see Step 1)

DRBD Sync Not Progressing

Check network connectivity:

bash
nc -zv <peer-node-ip> 7794

Check DRBD status:

bash
drbdadm status r0

Floating Monitor Not Starting

Check DRBD is Secondary on both nodes:

bash
drbdadm role r0
# Expected: Secondary/Secondary

Check ConfigMap:

bash
kubectl get configmap drbd-configure -n rook-ceph -o yaml

Re-run Setup Script

The script is idempotent and safe to re-run:

bash
./deploy/examples/drbd-setup-experimental.sh -d /dev/sdb