Back to Connect

S3 Polling with Bookmarking

.claude-plugin/plugins/redpanda-connect/skills/pipeline-assistant/resources/recipes/s3-polling.md

4.90.2568 B
Original Source

S3 Polling with Bookmarking

Pattern: Cloud Storage - S3 Polling Difficulty: Intermediate Components: aws_s3 input, kafka_franz Use Case: Poll S3 for new files and stream to Kafka

Overview

Continuously poll S3 for new files and stream contents to Kafka. Tracks processed files to avoid re-processing.

Configuration

See s3-polling.yaml

Key Concepts

Scanner

Tracks which files have been processed.

Polling Interval

Balance between latency and S3 API costs.