ContextQMD
Back to Scylladb

ScyllaDB CDC Source Connector

docs/using-scylla/integrations/scylla-cdc-source-connector.rst

latest3.0 KB
Original Source

============================= ScyllaDB CDC Source Connector

.. toctree:: :hidden:

scylla-cdc-source-connector-quickstart

ScyllaDB CDC Source Connector is a source connector capturing row-level changes in the tables of a ScyllaDB cluster. It is a Debezium connector, compatible with Kafka Connect (with Kafka 2.6.0+) and built on top of scylla-cdc-java library. The source code of the connector is available at GitHub <https://github.com/scylladb/scylla-cdc-source-connector>_.

The connector reads the CDC log for specified tables and produces Kafka messages for each row-level INSERT, UPDATE or DELETE operation. The connector is able to split reading the CDC log across multiple processes: the connector can start a separate Kafka Connect task for reading each :doc:Vnode of ScyllaDB cluster </architecture/ringarchitecture/index> allowing for high throughput. You can limit the number of started tasks by using tasks.max property.

ScyllaDB CDC Source Connector seamlessly handles schema changes and topology changes (adding, removing nodes from ScyllaDB cluster). The connector is fault-tolerant, retrying reading data from ScyllaDB in case of failure. It periodically saves the current position in ScyllaDB CDC log using Kafka Connect offset tracking (configurable by offset.flush.interval.ms parameter). If the connector is stopped, it is able to resume reading from previously saved offset. ScyllaDB CDC Source Connector has at-least-once semantics.

The connector has the following capabilities:

  • Kafka Connect connector using Debezium framework
  • Replication of row-level changes from ScyllaDB using :doc:ScyllaDB CDC </features/cdc/cdc-intro>. The connector replicates the following operations: INSERT, UPDATE, DELETE (single row deletes)
  • High scalability - able to split work across multiple Kafka Connect workers
  • Fault tolerant - connector periodically saves its progress and can resume from previously saved offset (with at-least-once semantics)
  • Support for many standard Kafka Connect converters, such as JSON and Avro
  • Compatible with standard Kafka Connect transformations
  • Metadata about CDC events - each generated Kafka message contains information about source, such as timestamp and table name
  • Seamless handling of schema changes and topology changes (adding, removing nodes from ScyllaDB cluster)

The connector has the following limitations:

  • Only Kafka 2.6.0+ is supported
  • Only row-level operations are produced (INSERT, UPDATE, DELETE) - partition deletes and row range deletes are not replicated
  • No support for collection types (LIST, SET, MAP) and UDT - columns with those types are omitted from generated messages
  • Preimage and postimage - changes only contain those columns that were modified, not the entire row before/after change

The following documents will help you get started with ScyllaDB CDC Source Connector:

  • :doc:ScyllaDB CDC Source Connector Quickstart <scylla-cdc-source-connector-quickstart>