How to scale self-hosted Prefect

Running multiple Prefect server instances enables high availability and distributes load across your infrastructure. This guide covers configuration and deployment patterns for scaling self-hosted Prefect.

Requirements

Multi-server deployments require:

PostgreSQL database version 14.9 or higher (SQLite does not support multi-server synchronization)
Redis for event messaging
Load balancer for API traffic distribution

Architecture

A scaled Prefect deployment typically includes:

Multiple API server instances - Handle UI and API requests
Background services - Runs the scheduler, automation triggers, and other loop services. Can be scaled horizontally by running multiple instances coordinated through Redis
PostgreSQL database - Stores all persistent data and synchronizes state across servers
Redis - Distributes events between services and coordinates background service work
Load balancer - Routes traffic to healthy API instances (e.g. NGINX or Traefik)

mermaid

%%{
  init: {
    'theme': 'neutral',
    'flowchart': {
      'curve' : 'linear',
      'rankSpacing': 120,
      'nodeSpacing': 80
    }
  }
}%%

flowchart TB
    %% Style definitions
    classDef userClass fill:#ede7f6db,stroke:#4527a0db,stroke-width:2px
    classDef lbClass fill:#e3f2fddb,stroke:#1565c0db,stroke-width:2px
    classDef apiClass fill:#1860f2db,stroke:#1860f2db,stroke-width:2px
    classDef bgClass fill:#7c3aeddb,stroke:#7c3aeddb,stroke-width:2px
    classDef dataClass fill:#16a34adb,stroke:#16a34adb,stroke-width:2px
    classDef workerClass fill:#f59e0bdb,stroke:#f59e0bdb,stroke-width:2px

    %% Nodes
    subgraph clients[Client Side]
        direction TB
        Users[Users / UI / API Clients]:::userClass
        Workers[Workers poll any available API server
Process / K8s / Docker / Serverless]:::workerClass
    end

    LB[Load Balancer
NGINX / HAProxy / ALB
Port 4200]:::lbClass

    subgraph servers[Prefect Server Components]
        direction TB
        subgraph api[API Servers - Horizontal Scaling]
            direction LR
            API1[API Server 1
--no-services]:::apiClass
            API2[API Server 2
--no-services]:::apiClass
            API3[API Server N...
--no-services]:::apiClass
        end

        subgraph bg[Background Services - Horizontal Scaling]
            direction LR
            BG1[Background Services 1
prefect server services start]:::bgClass
            BG2[Background Services 2
prefect server services start]:::bgClass
            BG3[Background Services N...
prefect server services start]:::bgClass
        end
    end

    subgraph data[Data Layer]
        direction LR
        PG[(PostgreSQL
• Flow/Task State
• Configuration
• History)]:::dataClass
        Redis[(Redis
• Events
• Automations
• Real-time Updates)]:::dataClass
    end

    %% Connections
    Users --> |HTTPS| LB
    LB --> |Round Robin| api

    api --> |Read/Write| PG
    api --> |Publish| Redis

    bg --> |Read/Write| PG
    bg --> |Subscribe / Coordinate| Redis

    Workers -.-> |Poll Work| api

Configuration

Database setup

Configure PostgreSQL as your database backend:

bash

export PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:password@host:5432/prefect"

<Warning> PostgreSQL version 14.9 or higher is required for multi-server deployments. SQLite does not support the features needed for state synchronization across multiple servers. </Warning>

AWS RDS IAM Authentication

To use AWS IAM authentication for your PostgreSQL database (experimental):

Install the AWS integration:
bash
```
pip install prefect-aws
```
Create an IAM policy with rds-db:connect permission and attach it to your IAM user/role.

Enable experimental plugins and IAM authentication:

bash

export PREFECT_EXPERIMENTS_PLUGINS_ENABLED=true
export PREFECT_INTEGRATIONS_AWS_RDS_IAM_ENABLED=true
# Optional: export PREFECT_INTEGRATIONS_AWS_RDS_IAM_REGION_NAME=us-east-1

Configure your connection URL:

bash

export PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://iam_user@host:5432/prefect"

Redis setup

Configure Redis as your server's message broker, cache, and lease storage:

bash

export PREFECT_MESSAGING_BROKER="prefect_redis.messaging"
export PREFECT_MESSAGING_CACHE="prefect_redis.messaging"
export PREFECT_SERVER_EVENTS_CAUSAL_ORDERING="prefect_redis.ordering"
export PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE="prefect_redis.lease_storage"
export PREFECT_REDIS_MESSAGING_HOST="redis-host"
export PREFECT_REDIS_MESSAGING_PORT="6379"
export PREFECT_REDIS_MESSAGING_DB="0"

If your Redis instance requires authentication, you may configure a username and password:

bash

export PREFECT_REDIS_MESSAGING_USERNAME="marvin"
export PREFECT_REDIS_MESSAGING_PASSWORD="dontpanic!"

For Redis instances that require an encrypted connection, you can enable SSL/TLS:

bash

export PREFECT_REDIS_MESSAGING_SSL="true"

Alternatively, configure the Redis connection with a single URL instead of individual fields. When PREFECT_REDIS_MESSAGING_URL is set, it takes precedence and the individual host, port, db, username, password, and SSL fields are ignored:

bash

export PREFECT_REDIS_MESSAGING_URL="redis://username:password@redis-host:6379/0"

Use rediss:// for TLS connections:

bash

export PREFECT_REDIS_MESSAGING_URL="rediss://redis-host:6379/0"

Docket URL for background services

Prefect uses Docket to coordinate background services like the scheduler, late run detection, and automation triggers. By default, Docket uses in-memory storage (memory://), which only works for single-server deployments.

For high-availability deployments, configure Docket to use Redis:

bash

export PREFECT_SERVER_DOCKET_URL="redis://redis-host:6379/0"

If your Redis instance requires authentication:

bash

export PREFECT_SERVER_DOCKET_URL="redis://username:password@redis-host:6379/0"

For Redis instances that require SSL/TLS:

bash

export PREFECT_SERVER_DOCKET_URL="rediss://redis-host:6379/0"

<Note> The Docket URL can use the same Redis instance as the messaging configuration above, but you may use a different database number (e.g., `/1` instead of `/0`) to keep the data separate. </Note>

Service separation

For optimal performance, run API servers and background services separately:

API servers (multiple instances):

bash

prefect server start --host 0.0.0.0 --port 4200 --no-services

Background services:

bash

prefect server services start

<Tip> For high-volume deployments, consider reducing the event retention period from the default 7 days to prevent rapid database growth. See [database maintenance](/v3/advanced/database-maintenance#configure-event-retention) for configuration details. </Tip>

Running multiple background services

For high availability and throughput, you can run multiple prefect server services start processes in parallel. Prefect uses Docket (backed by Redis) to coordinate work across background service processes so that periodic work (like scheduling, late run detection, and automation trigger evaluation) runs exactly once per interval even when multiple processes are running.

To run multiple background service processes:

Configure PREFECT_SERVER_DOCKET_URL to point at a shared Redis instance (see Docket URL for background services). The default in-memory backend (memory://) is only safe for a single process.
Ensure every background service process connects to the same PostgreSQL database, Redis messaging instance, and Docket URL as the API servers.
Start one prefect server services start process per replica. Each replica runs the same set of enabled services; Docket ensures only one replica picks up each scheduled run.

<Warning> Do not run multiple background service processes without configuring `PREFECT_SERVER_DOCKET_URL` to use Redis. With the default `memory://` backend each process schedules its own work independently, which causes duplicate scheduled runs, duplicate automation actions, and other correctness problems. </Warning>

Running specific services on dedicated processes

All services run together in a single prefect server services start process by default. To dedicate a process to a specific subset of services (for example, to scale a noisy neighbor independently), disable the services you don't want to run on that process with their *_ENABLED environment variable.

List all services and their enable/disable environment variable:

bash

prefect server services ls

For example, to run a process that only handles the scheduler and late run detection:

bash

export PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED=false
export PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED=false
export PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=false
export PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=false
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=false
export PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=false
# ...disable any other services you don't want on this process
prefect server services start

Then run another process with the complementary set of services enabled. As long as every enabled service is running on at least one process (and all processes share the same Docket Redis), every enabled service continues to operate.

<Note> Some services maintain their own at-least-once semantics at the database or Redis level rather than relying on Docket's run-once guarantee. Running multiple processes with the same service enabled is supported, but starting with one process per service (and scaling up only when you observe a specific bottleneck) keeps operations simple. </Note>

Database migrations

Disable automatic migrations in multi-server deployments:

bash

export PREFECT_API_DATABASE_MIGRATE_ON_START="false"

Run migrations separately before deployment:

bash

prefect server database upgrade -y

Load balancer configuration

Configure health checks for your load balancer:

Health endpoint: /api/health
Expected response: HTTP 200 with JSON {"status": "healthy"}
Check interval: 5-10 seconds

Example NGINX configuration:

nginx

upstream prefect_api {
    least_conn;
    server prefect-api-1:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-2:4200 max_fails=3 fail_timeout=30s;
    server prefect-api-3:4200 max_fails=3 fail_timeout=30s;
}

server {
    listen 4200;

    location /api/health {
        proxy_pass http://prefect_api;
        proxy_connect_timeout 1s;
        proxy_read_timeout 1s;
    }

    location / {
        proxy_pass http://prefect_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Reverse proxy configuration

When hosting Prefect behind a reverse proxy, ensure proper header forwarding:

nginx

server {
    listen 80;
    server_name prefect.example.com;

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name prefect.example.com;

    ssl_certificate /path/to/ssl/certificate.pem;
    ssl_certificate_key /path/to/ssl/certificate_key.pem;

    location /api {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;

        # WebSocket support
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Authentication headers
        proxy_set_header Authorization $http_authorization;
        proxy_pass_header Authorization;

        proxy_pass http://prefect_api;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://prefect_api;
    }
}

UI proxy settings

When self-hosting the UI behind a proxy:

PREFECT_UI_API_URL: Connection URL from UI to API
PREFECT_UI_SERVE_BASE: Base URL path to serve the UI
PREFECT_UI_URL: URL for clients to access the UI

SSL certificates

For self-signed certificates:

Add certificate to system bundle and set:
bash
```
export SSL_CERT_FILE=/path/to/certificate.pem
```

Or disable verification (testing only):

bash

export PREFECT_API_TLS_INSECURE_SKIP_VERIFY=True

Environment proxy settings

Prefect respects standard proxy environment variables:

bash

export HTTPS_PROXY=http://proxy.example.com:8080
export HTTP_PROXY=http://proxy.example.com:8080
export NO_PROXY=localhost,127.0.0.1,.internal

Deployment examples

Docker Compose

yaml

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready -h localhost -U $$POSTGRES_USER
      interval: 2s
      timeout: 5s
      retries: 15

  redis:
    image: redis:7

  migrate:
    image: prefecthq/prefect:3-latest
    depends_on:
      postgres:
        condition: service_healthy
    command: prefect server database upgrade -y
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect

  prefect-api:
    image: prefecthq/prefect:3-latest
    depends_on:
      migrate:
        condition: service_completed_successfully
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    deploy:
      replicas: 3
    command: prefect server start --host 0.0.0.0 --no-services
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect
      PREFECT_API_DATABASE_MIGRATE_ON_START: "false"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
      PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: "6379"
      PREFECT_SERVER_DOCKET_URL: redis://redis:6379/1
    ports:
      - "4200-4202:4200"  # Maps to different ports for each replica

  prefect-background:
    image: prefecthq/prefect:3-latest
    depends_on:
      migrate:
        condition: service_completed_successfully
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    deploy:
      replicas: 2
    command: prefect server services start
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect
      PREFECT_API_DATABASE_MIGRATE_ON_START: "false"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
      PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: "6379"
      PREFECT_SERVER_DOCKET_URL: redis://redis:6379/1

volumes:
  postgres_data:

</Accordion> <Tip> Deploying Prefect self-hosted somehow else? Consider [opening a PR](/contribute/docs-contribute) to add your deployment pattern to this guide. </Tip>

Operations

Migration considerations

Handling large databases

When running migrations on large database instances (especially where tables like events, flow_runs, or task_runs can reach millions of rows), the default database timeout of 10 seconds may not be sufficient for creating indexes.

If you encounter a TimeoutError during migrations, increase the database timeout:

bash

# Set timeout to 10 minutes (adjust based on your database size)
export PREFECT_API_DATABASE_TIMEOUT=600

# Then run the migration
prefect server database upgrade -y

For Docker deployments:

bash

docker run -e PREFECT_API_DATABASE_TIMEOUT=600 prefecthq/prefect:latest prefect server database upgrade -y

<Note> Index creation time scales with table size. A database with millions of events may require 30+ minutes for some migrations. If a migration fails due to timeout, you may need to manually clean up any partially created indexes before retrying. </Note>

Recovering from failed migrations

If a migration times out while creating indexes, you may need to manually complete it. For example, if migration 7a73514ca2d6 fails:

First, check which indexes were partially created:

sql

SELECT indexname FROM pg_indexes WHERE tablename = 'events' AND indexname LIKE 'ix_events%';

Manually create the missing indexes using CONCURRENTLY to avoid blocking:

sql

-- Drop any partial indexes from the failed migration
DROP INDEX IF EXISTS ix_events__event_related_occurred;
DROP INDEX IF EXISTS ix_events__related_resource_ids;

-- Create the new indexes
CREATE INDEX CONCURRENTLY ix_events__related_gin ON events USING gin(related);
CREATE INDEX CONCURRENTLY ix_events__event_occurred ON events (event, occurred);
CREATE INDEX CONCURRENTLY ix_events__related_resource_ids_gin ON events USING gin(related_resource_ids);

Mark the migration as complete:

sql

UPDATE alembic_version SET version_num = '7a73514ca2d6';

<Warning> Only use manual recovery if increasing the timeout and retrying the migration doesn't work. Always verify the correct migration version and index definitions from the migration files. </Warning>

Monitoring

Monitor your multi-server deployment:

Database connections: Watch for connection pool exhaustion
Redis memory: Ensure adequate memory for message queues
API response times: Track latency across different endpoints
Background service lag: Monitor time between event creation and processing

Best practices

Start with 2-3 API instances and scale based on load
Use connection pooling to manage database connections efficiently
Monitor extensively before scaling further (e.g. Prometheus + Grafana or Logfire)
Test failover scenarios regularly

Requirements

Architecture

Configuration

Database setup

AWS RDS IAM Authentication

Redis setup

Docket URL for background services

Service separation

Running multiple background services

Running specific services on dedicated processes

Database migrations

Load balancer configuration

Reverse proxy configuration

UI proxy settings

SSL certificates

Environment proxy settings

Deployment examples

Docker Compose

Operations

Migration considerations

Handling large databases

Recovering from failed migrations

Monitoring

Best practices

Further reading