================================================= Cluster Platform Migration Using Node Cycling

This procedure describes how to migrate a ScyllaDB cluster to new instance types using the add-and-replace approach, which is commonly used for:

Migrating from one CPU architecture to another (e.g., x86_64 to ARM/Graviton)
Upgrading to newer instance types with better performance
Changing instance families within the same cloud provider

The add-and-replace approach maintains data replication throughout the migration and ensures zero downtime for client applications.

.. note::

This procedure does not change the ScyllaDB software version. All nodes (both existing and new) must run the same ScyllaDB version. For software version upgrades, see :doc:Upgrade </upgrade/index>.

Overview

The add-and-replace migration follows these steps:

#. Add new nodes (on target instance type) to the existing cluster #. Wait for data to stream to the new nodes #. Decommission old nodes (on source instance type)

This approach keeps the cluster operational throughout the migration while maintaining the configured replication factor.

Key characteristics

Zero downtime: Client applications continue to operate during migration
Data safety: Replication factor is maintained throughout the process
Flexible: Works with both vnodes and tablets-enabled clusters
Multi-DC support: Can migrate nodes across multiple datacenters

.. warning::

Ensure your cluster has sufficient capacity during the migration. At the peak of the process, your cluster will temporarily have double the number of nodes.

Prerequisites

Check cluster health

Before starting the migration, verify that your cluster is healthy:

#. Check that all nodes are in Up Normal (UN) status:

.. code-block:: shell

  nodetool status

All nodes should show UN status. Do not proceed if any nodes are down.

#. Ensure no streaming or repair operations are in progress:

.. code-block:: shell

  nodetool netstats
  nodetool compactionstats

Plan the migration

Before provisioning new instances, plan the following:

Instance type mapping: Identify the source and target instance types. If your cluster uses vnodes (not tablets), consider that mismatched shard counts between source and target instance types can cause slower repairs. With tablets enabled, shard count mismatch is fully supported.

Rack assignment planning: Each new node must be assigned to the same rack as the node it will replace. This maintains rack-aware topology for:

Rack-aware replication (NetworkTopologyStrategy)
Proper data distribution across failure domains
Minimizing data movement during decommission

Example mapping for a 3-node cluster:

.. code-block:: none

Source nodes (to be decommissioned): Target nodes (to be added): 192.168.1.10 - RACK0 → 192.168.2.10 - RACK0 192.168.1.11 - RACK1 → 192.168.2.11 - RACK1 192.168.1.12 - RACK2 → 192.168.2.12 - RACK2

Create a backup

Back up the data before starting the migration. One of the following methods can be used:

ScyllaDB Manager (recommended): Use ScyllaDB Manager to perform a cluster-wide backup. See the ScyllaDB Manager documentation <https://manager.docs.scylladb.com/stable/backup/>_ for details.
Snapshots: On each node in the cluster, create a snapshot:

.. code-block:: shell

nodetool snapshot -t pre_migration_backup nodetool listsnapshots

.. note::

Snapshots are local to each node and do not protect against node or disk failure. For full disaster recovery, use ScyllaDB Manager backup.

Procedure

Adding new nodes

#. Provision new instances with the target instance type. Ensure:

The same ScyllaDB version as existing nodes
Same network configuration and security groups
Appropriate storage configuration

#. On each new node, configure /etc/scylla/scylla.yaml to join the existing cluster:

cluster_name: Must match the existing cluster name
seeds: IP address of an existing node in the cluster (used to discover cluster topology on join)
endpoint_snitch: Must match the existing cluster configuration
listen_address: IP address of the new node
rpc_address: IP address of the new node

All other cluster-wide settings (tablets configuration, encryption settings, experimental features, etc.) must match the existing nodes.

.. caution::

  Make sure that the ScyllaDB version on the new node is identical to the
  version on the other nodes in the cluster. Running nodes with different
  versions is not supported.

#. If using GossipingPropertyFileSnitch, configure /etc/scylla/cassandra-rackdc.properties with the correct datacenter and rack assignment for this node:

.. code-block:: none

  dc = <datacenter-name>
  rack = <rack-name>
  prefer_local = true

.. warning::

  Each node must have the correct rack assignment. Using the same rack for
  all new nodes breaks rack-aware replication topology.

#. Start ScyllaDB on the new node:

.. code-block:: shell

  sudo systemctl start scylla-server

For Docker deployments:

.. code-block:: shell

  docker exec -it <container-name> supervisorctl start scylla

#. Monitor the bootstrap process from an existing node:

.. code-block:: shell

  nodetool status

The new node will appear with UJ (Up, Joining) status while streaming data from existing nodes. Wait until it transitions to UN (Up, Normal).

Example output during bootstrap:

.. code-block:: shell

  Datacenter: dc1
  Status=Up/Down
  State=Normal/Leaving/Joining/Moving
  --  Address        Load       Tokens  Owns   Host ID                               Rack
  UN  192.168.1.10   500 MB     256     33.3%  8d5ed9f4-7764-4dbd-bad8-43fddce94b7c  RACK0
  UN  192.168.1.11   500 MB     256     33.3%  125ed9f4-7777-1dbn-mac8-43fddce9123e  RACK1
  UN  192.168.1.12   500 MB     256     33.3%  675ed9f4-6564-6dbd-can8-43fddce952gy  RACK2
  UJ  192.168.2.10   250 MB     256     ?      a1b2c3d4-5678-90ab-cdef-112233445566  RACK0

Example output after bootstrap completes:

.. code-block:: shell

  Datacenter: dc1
  Status=Up/Down
  State=Normal/Leaving/Joining/Moving
  --  Address        Load       Tokens  Owns   Host ID                               Rack
  UN  192.168.1.10   400 MB     256     25.0%  8d5ed9f4-7764-4dbd-bad8-43fddce94b7c  RACK0
  UN  192.168.1.11   400 MB     256     25.0%  125ed9f4-7777-1dbn-mac8-43fddce9123e  RACK1
  UN  192.168.1.12   400 MB     256     25.0%  675ed9f4-6564-6dbd-can8-43fddce952gy  RACK2
  UN  192.168.2.10   400 MB     256     25.0%  a1b2c3d4-5678-90ab-cdef-112233445566  RACK0

#. For tablets-enabled clusters, wait for tablet load balancing to complete. After the node reaches UN status, verify no streaming is in progress:

.. code-block:: shell

  nodetool netstats

Wait until output shows "Not sending any streams" and no active receiving streams.

#. Repeat steps 1-6 for each new node to be added.

.. note::

You can add multiple nodes in parallel if they are in different datacenters. Within a single datacenter, add nodes one at a time for best results.

Updating seed node configuration

If any of your original nodes are configured as seed nodes, you must update the seed configuration before decommissioning them.

#. Check the current seed configuration on any node:

.. code-block:: shell

  grep -A 4 "seed_provider" /etc/scylla/scylla.yaml

#. If the seeds include nodes you plan to decommission, update scylla.yaml on all new nodes to use the new node IPs as seeds:

.. code-block:: yaml

  seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
        - seeds: "192.168.2.10,192.168.2.11,192.168.2.12"

.. note::

  Updating seed configuration on the **old nodes** (that will be
  decommissioned) is optional. Seeds are only used during node startup
  to discover the cluster. If you don't plan to restart the old nodes
  before decommissioning them, their seed configuration doesn't matter.
  However, updating all nodes is recommended for safety in case an old
  node unexpectedly restarts during the migration.

#. Restart ScyllaDB on each new node (one at a time) to apply the new seed configuration:

.. code-block:: shell

  sudo systemctl restart scylla-server

Wait for the node to fully start before restarting the next node.

#. After restarting the new nodes, verify the cluster is healthy:

.. code-block:: shell

  nodetool status
  nodetool describecluster

.. warning::

Complete this seed list update on all new nodes before decommissioning any old nodes. This ensures the new nodes can reform the cluster after the old nodes are removed.

Decommissioning old nodes

After all new nodes are added and healthy, decommission the old nodes one at a time.

#. Verify all nodes are healthy before starting decommission:

.. code-block:: shell

  nodetool status

All nodes should show UN status.

#. On the node to be decommissioned, run:

.. code-block:: shell

  nodetool decommission

This command blocks until the decommission is complete. The node will stream its data to the remaining nodes.

#. Monitor the decommission progress from another node:

.. code-block:: shell

  nodetool status

The decommissioning node will transition from UN → UL (Up, Leaving) → removed from the cluster.

You can also monitor streaming progress:

.. code-block:: shell

  nodetool netstats

#. After decommission completes, verify the node is no longer in the cluster:

.. code-block:: shell

  nodetool status

The decommissioned node should no longer appear in the output.

#. Run nodetool cleanup on the remaining nodes to remove data that no longer belongs to them after the topology change:

.. code-block:: shell

  nodetool cleanup

.. note::

  ``nodetool cleanup`` can be resource-intensive. Run it on one node at a
  time during low-traffic periods.

#. Wait for the cluster to stabilize before decommissioning the next node. Ensure no streaming operations are in progress.

#. Repeat steps 1-7 for each old node to be decommissioned.

Post-migration verification

After all old nodes are decommissioned, verify the migration was successful.

Verify cluster topology

.. code-block:: shell

nodetool status

Confirm:

All nodes show UN (Up, Normal) status
Only the new instance type nodes are present
Nodes are balanced across racks

Verify schema agreement

.. code-block:: shell

nodetool describecluster

All nodes should report the same schema version.

Verify data connectivity

Connect to the cluster and run a test query:

.. code-block:: shell

cqlsh <node-ip> -e "SELECT count(*) FROM system_schema.keyspaces;"

.. note::

If ScyllaDB is configured with listen_interface, you must use the node's interface IP address (not localhost) for cqlsh connections.

Verify ScyllaDB version

Confirm all nodes are running the same ScyllaDB version:

.. code-block:: shell

scylla --version

Verify data integrity (optional)

Run data validation on each keyspace to verify sstable integrity:

.. code-block:: shell

nodetool scrub --mode=VALIDATE <keyspace_name>

Rollback

If issues occur during the migration, you can roll back by reversing the procedure.

During add phase

If a new node fails to bootstrap:

#. Stop ScyllaDB on the new node:

.. code-block:: shell

  sudo systemctl stop scylla-server

#. From an existing node, remove the failed node:

.. code-block:: shell

  nodetool removenode <host-id-of-failed-node>

During decommission phase

If a decommission operation gets stuck:

#. If the node is still reachable, try stopping and restarting ScyllaDB #. If the node is unresponsive, from another node:

.. code-block:: shell

  nodetool removenode <host-id>

See :doc:Remove a Node from a ScyllaDB Cluster </operating-scylla/procedures/cluster-management/remove-node> for more details.

Full rollback

To roll back after the migration is complete (all nodes on new instance type), apply the same add-and-replace procedure in reverse:

#. Add new nodes on the original instance type #. Wait for data streaming to complete #. Decommission the nodes on the new instance type

Troubleshooting

Node stuck in Joining (UJ) state

If a new node remains in UJ state for an extended period:

Check ScyllaDB logs for streaming errors: journalctl -u scylla-server
Verify network connectivity between nodes
Ensure sufficient disk space on all nodes
Check for any ongoing operations that may be blocking

Decommission taking too long

Decommission duration depends on data size. If it appears stuck:

Check streaming progress: nodetool netstats
Look for errors in ScyllaDB logs
Verify network bandwidth between nodes

Schema disagreement

If nodes report different schema versions:

Wait a few minutes for schema to propagate
If disagreement persists, restart the nodes one by one
Run nodetool describecluster to verify agreement

Additional resources

:doc:Adding a New Node Into an Existing ScyllaDB Cluster </operating-scylla/procedures/cluster-management/add-node-to-cluster>
:doc:Remove a Node from a ScyllaDB Cluster </operating-scylla/procedures/cluster-management/remove-node>
:doc:Replace a Running Node in a ScyllaDB Cluster </operating-scylla/procedures/cluster-management/replace-running-node>
:doc:Upgrade </upgrade/index>