Introduction

Broadway is a library for building concurrent and multi-stage data ingestion and data processing pipelines with Elixir. Broadway pipelines are concurrent and robust, thanks to the Erlang VM and its actors. It features:

Back-pressure
Automatic acknowledgements at the end of the pipeline
Batching
Fault tolerance
Graceful shutdown
Built-in testing
Custom failure handling
Ordering and partitioning
Rate-limiting
Metrics

Official Producers

Currently we officially support four Broadway producers:

Amazon SQS: Source - Guide
Apache Kafka: Source - Guide
Google Cloud Pub/Sub: Source - Guide
RabbitMQ: Source - Guide

The guides links above will help you get started with your adapter of choice. For API reference, you can check out the Broadway module.

Non-official (Off-Broadway) Producers

For those interested in rolling their own Broadway Producers (which we actively encourage!), we recommend using the OffBroadway namespace, mirroring the Off-Broadway theaters. For example, if you want to publish your own integration with Amazon SQS, you can package it as off_broadway_sqs, which uses the OffBroadway.SQS namespace.

The following Off-Broadway libraries are available (feel free to send a PR adding your own in alphabetical order):