ContextQMD
Back to Redis

Cluster Recovery

content/operate/kubernetes/re-clusters/cluster-recovery.md

latest2.2 KB
Original Source

When a Redis Enterprise cluster loses contact with more than half of its nodes either because of failed nodes or network split, the cluster stops responding to client connections. When this happens, you must recover the cluster to restore the connections.

You can also perform cluster recovery to reset cluster nodes, to troubleshoot issues, or in a case of active/passive failover.

The Redis Enterprise for Kubernetes automates these recovery steps:

  1. Recreates a fresh Redis Enterprise cluster
  2. Mounts the persistent storage with the recovery files from the original cluster to the nodes of the new cluster
  3. Recovers the cluster configuration on the first node in the new cluster
  4. Joins the remaining nodes to the new cluster.

{{<warning>}}Redis Enterprise for Kubernetes 7.2.4-2 introduces a new limitation. You cannot recover or upgrade your cluster if there are databases with old module versions or manually uploaded modules. See the [Redis Enterprise Software 7.2.4 known limitations]({{< relref "/operate/rs/release-notes/rs-7-2-4-releases/rs-7-2-4-52#cluster-recovery-with-manually-uploaded-modules" >}}) for more details.{{</warning>}}

{{< note >}} If your cluster uses user-defined modules, the recovery process doesn't block on module validation errors (such as URL or credential issues). The cluster can recover successfully, and you can resolve any module configuration issues after recovery is complete. See [User-defined modules]({{< relref "/operate/kubernetes/re-databases/modules#user-defined-modules" >}}) for more information. {{< /note >}}

Prerequisites

  • For cluster recovery, the cluster must be [deployed with persistence]({{< relref "/operate/kubernetes/recommendations/persistent-volumes" >}}).

Recover a cluster

  1. Edit the REC resource to set the clusterRecovery flag to true.

    sh
    kubectl patch rec <cluster-name> -n <rec_namespace> --type merge --patch '{"spec":{"clusterRecovery":true}}'

  2. Wait for the cluster to recover until it is in the "Running" state.

    To see the state of the cluster, run:

    sh
    watch "kubectl describe rec | grep State"

  3. To recover the database, see [Recover a failed database]({{< relref "/operate/rs/databases/recover" >}}).