metadata-ingestion/docs/sources/kafka/README.md
Apache Kafka is a distributed event streaming platform used for high-throughput, fault-tolerant messaging and real-time data pipelines. Confluent Cloud provides a fully managed Kafka service with an integrated Schema Registry.
The DataHub Kafka connector extracts topic metadata and schemas from Kafka clusters and Confluent Schema Registry. It supports Avro, Protobuf, and JSON schemas, multi-stage schema resolution with automatic fallback and inference for topics not registered in the schema registry, and optional data profiling to generate field-level statistics and sample values from message content. The integration also captures stateful deletion detection.
| Kafka Concept | DataHub Concept | Notes |
|---|---|---|
| Topic | Dataset | Subtype Topic |
| Schema (Subject) | Schema Metadata | Avro, Protobuf, JSON schemas |
| Message Fields | Dataset Fields | Extracted from schemas or inferred |
| Kafka Cluster | Data Platform Instance | When platform_instance is configured |
| Schema metadata | Tags, Glossary Terms, Owners (CorpUser / CorpGroup) | Optional, Avro only: derived from schema properties when enable_meta_mapping is set with meta_mapping / field_meta_mapping directives |