CDC using YugabyteDB gRPC replication protocol

YugabyteDB CDC captures changes made to data in the database and streams those changes to external processes, applications, or other databases. CDC allows you to track and propagate changes in a YugabyteDB database to downstream consumers based on its Write-Ahead Log (WAL). YugabyteDB CDC uses Debezium to capture row-level changes resulting from INSERT, UPDATE, and DELETE operations in the upstream database, and publishes them as events to Kafka using Kafka Connect-compatible connectors.

<!-- {{<lead link="./cdc-overview">}} To know more about the internals of CDC, see [Overview](./cdc-overview). {{</lead>}} -->

Get started

Get started with Yugabyte gRPC replication.

For tutorials on streaming data to Kafka environments, including Amazon MSK, Azure Event Hubs, and Confluent Cloud, see Kafka environments.

{{<lead link="./cdc-get-started/">}} Get started using the connector. {{</lead>}}

Monitoring

You can monitor the activities and status of the deployed connectors using the http end points provided by YugabyteDB.

{{<lead link="./cdc-monitor/">}} Learn how to monitor your CDC setup. {{</lead>}}

YugabyteDB gRPC Connector

To capture and stream your changes in YugabyteDB to an external system, you need a connector that can read the changes in YugabyteDB and stream it out. For this, you can use the YugabyteDB gRPC connector, which is based on the Debezium platform. The connector is deployed as a set of Kafka Connect-compatible connectors, so you first need to define a YugabyteDB connector configuration and then start the connector by adding it to Kafka Connect.

{{<lead link="./debezium-connector-yugabytedb/">}} For reference documentation, see YugabyteDB gRPC Connector. {{</lead>}}

Known limitations

• A single stream can only be used to stream data from one namespace only.
• There should be a primary key on the table you want to stream the changes from.
• CDC is not supported on tables (both source and target) for xCluster replication. Issues {{<issue 25371>}} and {{<issue 15534>}}.
• Currently, CDC doesn't support schema evolution for changes that require table rewrites (for example, ALTER TYPE), or DROP TABLE and TRUNCATE TABLE operations.
• YCQL tables aren't currently supported. Issue {{<issue 11320>}}.

In addition, CDC support for the following features will be added in upcoming releases:

• Support for point-in-time recovery (PITR) is tracked in issue {{<issue 10938>}}.
• Support for transaction savepoints is tracked in issue {{<issue 10936>}}.
• Support for enabling CDC on Read Replicas is tracked in issue {{<issue 11116>}}.
• Support for schema evolution with before image is tracked in issue {{<issue 15197>}}.

Learn more

• CDC architecture
• Examples of CDC usage and patterns {{<icon/github>}}
• Tutorials to deploy in different Kafka environments {{<icon/tutorial>}}
• Data Streaming Using YugabyteDB CDC, Kafka, and SnowflakeSinkConnector {{<icon/blog>}}
• Unlock Azure Storage Options With YugabyteDB CDC {{<icon/blog>}}
• Change Data Capture From YugabyteDB to Elasticsearch {{<icon/blog>}}
• Snowflake CDC: Publishing Data Using Amazon S3 and YugabyteDB {{<icon/blog>}}
• Streaming Changes From YugabyteDB to Downstream Databases {{<icon/blog>}}
• Change Data Capture from YugabyteDB CDC to ClickHouse {{<icon/blog>}}
• How to Run Debezium Server with Kafka as a Sink {{<icon/blog>}}
• Change Data Capture Using a Spring Data Processing Pipeline {{<icon/blog>}}