Engine Overview

SeaTunnel supports multiple execution engines, allowing you to choose the best one for your use case. This document provides a comprehensive comparison to help you make the right choice.

Supported Engines

Quick Comparison

Feature Comparison

Performance Comparison

Ease of Use

When to Use Each Engine

SeaTunnel Engine (Zeta) - Recommended

Best for:

• New data integration projects
• Data synchronization and CDC scenarios
• Users without existing big data infrastructure
• Scenarios requiring low resource consumption
• Real-time synchronization of many small tables

Advantages:

• No external dependencies (no Zookeeper, HDFS required)
• Optimized for data synchronization scenarios
• Dynamic thread sharing for efficient resource usage
• Pipeline-level fault tolerance
• Built-in cluster management and HA
• JDBC connection multiplexing

Example use cases:

• MySQL to ClickHouse real-time sync
• Multi-table CDC synchronization
• Database migration projects

Apache Flink

Best for:

• Organizations with existing Flink infrastructure
• Complex stream processing requirements
• Scenarios requiring Flink ecosystem integration

Advantages:

• Mature stream processing capabilities
• Rich ecosystem and community
• Advanced state management
• Integration with Flink SQL

Example use cases:

• Integration with existing Flink pipelines
• Complex event processing
• Scenarios requiring Flink-specific features

Apache Spark

Best for:

• Organizations with existing Spark infrastructure
• Large-scale batch processing
• Integration with Spark ecosystem (MLlib, GraphX)

Advantages:

• Mature batch processing capabilities
• Rich ecosystem
• Integration with Hive, HDFS
• Support for YARN, Kubernetes

Example use cases:

• Large-scale ETL jobs
• Integration with existing Spark workflows
• Batch data warehouse loading

Decision Flowchart

Start
  │
  ▼
Do you have existing Flink/Spark infrastructure?
  │
  ├─ Yes ──► Do you want to reuse it?
  │            │
  │            ├─ Yes (Flink) ──► Use Flink Engine
  │            │
  │            ├─ Yes (Spark) ──► Use Spark Engine
  │            │
  │            └─ No ──► Use SeaTunnel Engine
  │
  └─ No ──► Use SeaTunnel Engine (Recommended)

Configuration Examples

SeaTunnel Engine

env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 10000
}

Flink Engine

env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 10000
  flink.execution.checkpointing.mode = "EXACTLY_ONCE"
  flink.execution.checkpointing.timeout = 600000
}

Spark Engine

env {
  parallelism = 2
  job.mode = "BATCH"
  spark.app.name = "SeaTunnel-Job"
  spark.executor.memory = "2g"
  spark.executor.instances = "2"
}

Connector Compatibility

All SeaTunnel V2 connectors are compatible with all three engines. However, some features may have different behaviors:

Migration Guide

From Flink to SeaTunnel Engine

1. Remove Flink-specific configurations (prefixed with flink.)
2. Keep common configurations (parallelism, checkpoint.interval)
3. Test with SeaTunnel Engine

From Spark to SeaTunnel Engine

1. Remove Spark-specific configurations (prefixed with spark.)
2. Keep common configurations (parallelism, job.mode)
3. Test with SeaTunnel Engine

Summary

Next Steps

• SeaTunnel Engine Quick Start
• Flink Engine Guide
• Spark Engine Guide