ContextQMD
Back to Scylladb

Remove a Data-Center from a ScyllaDB Cluster

docs/operating-scylla/procedures/cluster-management/decommissioning-data-center.rst

latest7.8 KB
Original Source

Remove a Data-Center from a ScyllaDB Cluster

Before removing (decommission) the data-center from the cluster, it is essential to verify that the data-center that is about to be removed does not hold any unique data.

Prerequisites

  • Verify that there are no client writes to nodes in the data-center that is to be decommissioned.
  • Check the cluster status using the :doc:nodetool status </operating-scylla/nodetool-commands/status> command to understand the cluster deployment.

.. code-block:: shell

Datacenter: US-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.201 112.82 KB 256 32.7% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1 UN 192.168.1.202 91.11 KB 256 32.9% 125ed9f4-7777-1dbn-lac8-23fddce9123e B1 UN 192.168.1.203 124.42 KB 256 32.6% 675ed9f4-6564-6dbd-can8-43fddce952gy B1

Datacenter: ASIA-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 50.191.1.204 112.82 KB 256 32.7% 4d5ed9f4-7764-4ded-dad8-63fdace94b7c B1 UN 50.191.1.205 91.11 KB 256 32.9% 145id9f4-7777-1dvn-nac8-83fdzce917r4 B1 UN 50.191.1.206 124.42 KB 256 32.6% 777ed9f4-6564-6dsd-can8-13fdxce999gy B1

Datacenter: EUROPE-DC Status=Up/Down State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.91.202.31 112.82 KB 256 32.7% 1d5ed8f4-7764-4dbd-rad8-44fddce94b7v B1 UN 172.91.202.32 91.11 KB 256 32.9% 525ed7g4-7437-1dbn-mac8-53fddce9123c B1 UN 172.91.202.33 124.42 KB 256 32.6% 975edbm4-6564-63bd-san8-73fddce952ga B1

Procedure

#. If there are vnode keyspaces in this DC, run the nodetool repair -pr command on each node in the data-center that is going to be decommissioned. This will verify that all the data is in sync between the decommissioned data-center and the other data-centers in the cluster.

For example:

If the ASIA-DC cluster is to be removed, then, run the nodetool repair -pr command on all the nodes in the ASIA-DC

#. If there are tablet keyspaces in this DC, run the nodetool cluster repair on an arbitrary node. The reason for running repair is to ensure that any updates stored only on the about-to-be-decommissioned replicas are propagated to the other replicas, before the replicas on the decommissioned datacenter are dropped.

#. ALTER every cluster KEYSPACE, so that the keyspaces will no longer replicate data to the decommissioned data-center.

For example:

.. code-block:: shell

  cqlsh> DESCRIBE <KEYSPACE_NAME>
  cqlsh> CREATE KEYSPACE <KEYSPACE_NAME> WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', '<DC_NAME1>' : 3, '<DC_NAME2>' : 3, '<DC_NAME3>' : 3};

  cqlsh> ALTER KEYSPACE <KEYSPACE_NAME> WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', '<DC_NAME1>' : 3, '<DC_NAME2>' : 3, '<DC_NAME3>' : 0};

For example:

There are three data-centers: US-DC, ASIA-DC and EUROPE-DC, the ASIA-DC is to be removed. The current KEYSPACE is:

.. code-block:: shell

  cqlsh> DESCRIBE nba
  cqlsh> CREATE KEYSPACE nba WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 2, 'EUROPE-DC' : 3};

Use the ALTER KEYSPACE to change the KEYSPACE and remove the decommissioned data-center.

.. code-block:: shell

  cqlsh> ALTER KEYSPACE nba WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 0, 'EUROPE-DC' : 3};

For tablet keyspaces, update the replication factor one by one:

.. code-block:: shell

  cqlsh> DESCRIBE nba2
  cqlsh> CREATE KEYSPACE nba2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 2, 'EUROPE-DC' : 3} AND tablets = { 'enabled': true };

.. code-block:: shell

  cqlsh> ALTER KEYSPACE nba2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 1, 'EUROPE-DC' : 3} AND tablets = { 'enabled': true };
  cqlsh> ALTER KEYSPACE nba2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 0, 'EUROPE-DC' : 3} AND tablets = { 'enabled': true };

.. note:: If rf_rack_valid_keyspaces option is set, a tablet keyspace needs to use rack list replication factor, so that the DC can be removed. See :ref:the conversion procedure <conversion-to-rack-list-rf>. In this case, to remove a datacenter:

     .. code-block:: shell

        cqlsh> DESCRIBE nba3
        cqlsh> CREATE KEYSPACE nba3 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : ['RAC4', 'RAC5'], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true };

     .. code-block:: shell

        cqlsh> ALTER KEYSPACE nba3 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : ['RAC4'], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true };
        cqlsh> ALTER KEYSPACE nba3 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : ['RAC1', 'RAC2', 'RAC3'], 'ASIA-DC' : [], 'EUROPE-DC' : ['RAC6', 'RAC7', 'RAC8']} AND tablets = { 'enabled': true };

     Consider :ref:`upgrading rf_rack_valid_keyspaces option to enforce_rack_list option <keyspace-rf-rack-valid-to-enforce-rack-list>` to ensure all tablet keyspaces use rack lists.

.. note::

  If table audit is enabled, the ``audit`` keyspace is automatically created with ``NetworkTopologyStrategy``.
  You must also alter the ``audit`` keyspace to remove replicas from the decommissioned data-center. For example:

  .. code-block:: shell

     cqlsh> ALTER KEYSPACE audit WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'US-DC' : 3, 'ASIA-DC' : 0, 'EUROPE-DC' : 3};

  Failure to do so will result in decommission errors such as "zero replica after the removal".

#. Run :doc:nodetool decommission </operating-scylla/nodetool-commands/decommission> on every node in the data center that is to be removed. Refer to :doc:Remove a Node from a ScyllaDB Cluster - Down Scale </operating-scylla/procedures/cluster-management/remove-node> for further information.

For example:

If ASIA-DC is to be removed, then, execute the nodetool decommission command on all the nodes in this data-center.

#. Verify that the data-center was successfully removed by using the :doc:nodetool status </operating-scylla/nodetool-commands/status> command.

For example:

.. code-block:: shell

  Datacenter: US-DC
  Status=Up/Down
  State=Normal/Leaving/Joining/Moving
  --  Address        Load       Tokens  Owns (effective)                         Host ID         Rack
  UN  192.168.1.201  112.82 KB  256     32.7%             8d5ed9f4-7764-4dbd-bad8-43fddce94b7c   B1
  UN  192.168.1.202  91.11 KB   256     32.9%             125ed9f4-7777-1dbn-mac8-43fddce9123e   B1
  UN  192.168.1.203  124.42 KB  256     32.6%             675ed9f4-6564-6dbd-can8-43fddce952gy   B1

  Datacenter: EUROPE-DC
  Status=Up/Down
  State=Normal/Leaving/Joining/Moving
  --  Address        Load       Tokens  Owns (effective)                         Host ID         Rack
  UN  172.91.202.31  112.82 KB  256     32.7%             1d5ed8f4-7764-4dbd-rad8-44fddce94b7v   B1
  UN  172.91.202.32  91.11 KB   256     32.9%             525ed7g4-7437-1dbn-mac8-53fddce9123c   B1
  UN  172.91.202.33  124.42 KB  256     32.6%             975edbm4-6564-63bd-san8-73fddce952ga   B1