Rust feature flags service

The Rust feature flags service (rust/feature-flags/) handles all runtime feature flag evaluation. It serves the /flags and /decide endpoints that SDKs call. Django remains the admin API for flag CRUD operations (/api/projects/{id}/feature_flags/) and serves the local evaluation endpoint (/api/feature_flag/local_evaluation).

Infrastructure routing

Traffic routing happens at the Kubernetes infrastructure level using Contour HTTPProxy resources (Envoy-based). The Rust service never receives requests through Django -- they are routed directly by Contour.

Client
  │
  ▼
AWS ALB
  │
  ▼
Contour / Envoy (path-based routing)
  │
  ├── /decide/*              ──▶ posthog-feature-flags:3001  (Rust)
  ├── /flags/?               ──▶ posthog-feature-flags:3001  (Rust)
  ├── /api/feature_flag/local_evaluation ──▶ posthog-local-evaluation:8000 (Django, dedicated deployment)
  ├── /api/*                 ──▶ posthog-web-django:8000     (Django, catch-all)
  └── /*                     ──▶ posthog-web-django:8000     (Django, final catch-all)

Key routing details:

• The decide and feature-flags proxy blocks are included before the api block in Contour, so they match first
• /decide adds an X-Original-Endpoint: decide header so the Rust service can adjust response format
• A dedicated subdomain (us-d.i.posthog.com / eu-d.i.posthog.com) routes only to decide + feature-flags with no Django fallback
• All flag routes have a 5-second timeout and 2 retries on reset/cancelled
• Canary rollouts are supported via Argo Rollouts adjusting weights on the HTTPProxy resources

Routing config lives in the charts repo: argocd/contour-ingress/values/values.prod-us.yaml (and prod-eu, dev variants).

Architecture overview

┌─────────────────────────────────────────────────────────────────┐
│                          SDK Request                            │
│                    POST /flags or /decide                       │
└─────────────────────────────────────────────────────────────────┘
                               │
                        Contour / Envoy
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Rust Feature Flags Service                  │
│                     (Axum, port 3001)                           │
├─────────────────────────────────────────────────────────────────┤
│  Rate limiting ──▶ Auth ──▶ Decode ──▶ Evaluate ──▶ Response   │
└─────────────────────────────────────────────────────────────────┘
        │                │                    │
        ▼                ▼                    ▼
  ┌──────────┐   ┌──────────────┐   ┌──────────────────┐
  │  Redis   │   │  PostgreSQL  │   │   S3 (fallback)  │
  │ (cache)  │   │  (source of  │   │   via HyperCache │
  │          │   │   truth)     │   │                  │
  └──────────┘   └──────────────┘   └──────────────────┘

Project structure

HTTP endpoints

All routes are defined in rust/feature-flags/src/router.rs.

All flag routes accept trailing slashes.

/flags request processing

The POST handler follows this pipeline:

1. Rate limiting: IP-based check (DDoS defense), then token-based check (per-project limits)
2. Body decoding: JSON, base64, or gzip-compressed bodies
3. Authentication: Extracts API token from body, query params, or headers
4. Team lookup: HyperCache (Redis -> S3) with PostgreSQL fallback
5. Flag definitions fetch: HyperCache (Redis -> S3) with PostgreSQL fallback
6. Billing check: Verifies the team's feature flag quota hasn't been exceeded
7. Flag evaluation: Core matching logic (see flag-evaluation-engine.md)
8. Config assembly: Session recording settings, error tracking, site apps
9. Response formatting: Version-specific serialization

Response versioning

The response format depends on the v query parameter and the endpoint:

/flags/definitions endpoint (under construction)

Not live in production. This endpoint is under active development and is not routed by Contour. Local evaluation is currently served by Django at /api/feature_flag/local_evaluation (see Django API endpoints), which remains the production endpoint for server-side SDKs.

The goal is for this Rust endpoint to replace the Django local evaluation endpoint. When complete, it will serve flag definitions for SDKs that evaluate flags locally, authenticated via:

• Team secret API token (Authorization: Bearer phs_...), or
• Personal API key with feature_flag:read scope

Current implementation returns flag definitions with cohort data from HyperCache, with PostgreSQL fallback on cache miss. Supports ETag-based conditional requests (If-None-Match header) to avoid re-transferring unchanged definitions. Rate limited per team (default 600/minute, per-team overrides via LOCAL_EVAL_RATE_LIMITS).

Billing quota enforcement matches Django's /api/feature_flag/local_evaluation behavior:

• Quota check: Uses FeatureFlagsLimiter.is_limited(token) to verify the team hasn't exceeded their feature flag request quota. Returns HTTP 402 with a JSON body ({"type": "quota_limited", "code": "payment_required", ...}) when the quota is exceeded.
• Non-billable flag filtering: Usage tracking skips requests where the response contains only non-billable flags — i.e., flags with keys starting with survey-targeting- or product-tour-targeting-. The shared is_billable_flag_key() predicate (in flag_analytics.rs) is used by both this endpoint and the /flags billing handler.
• 304 responses skip billing: Usage is recorded after the ETag/304 path, so conditional responses that return 304 Not Modified are not counted toward billing. This matches Django's behavior.

Request and response types

FlagRequest (POST body)

pub struct FlagRequest {
    pub token: Option<String>,               // aliases: $token, api_key
    pub distinct_id: Option<String>,         // alias: $distinct_id
    pub geoip_disable: Option<bool>,
    pub disable_flags: Option<bool>,
    pub person_properties: Option<HashMap<String, Value>>,
    pub groups: Option<HashMap<String, Value>>,
    pub group_properties: Option<HashMap<String, HashMap<String, Value>>>,
    pub anon_distinct_id: Option<String>,    // alias: $anon_distinct_id
    pub device_id: Option<String>,           // alias: $device_id
    pub flag_keys: Option<Vec<String>>,      // evaluate only these flags
    pub timezone: Option<String>,
    pub evaluation_contexts: Option<Vec<String>>,
    pub evaluation_runtime: Option<EvaluationRuntime>,
}

FlagsResponse (v2 response)

pub struct FlagsResponse {
    pub errors_while_computing_flags: bool,
    pub flags: HashMap<String, FlagDetails>,
    pub quota_limited: Option<Vec<String>>,
    pub request_id: Uuid,
    pub evaluated_at: i64,
    pub config: ConfigResponse,
}

pub struct FlagDetails {
    pub key: String,
    pub enabled: bool,
    pub variant: Option<String>,
    pub reason: FlagEvaluationReason,
    pub metadata: FlagDetailsMetadata,
}

Rate limiting

Three independent rate limiters (IP, token, definitions), all in-process using the governor crate. The /flags IP and token limiters support a warn-then-enforce model with X-PostHog-Rate-Limit-Warning headers and per-token custom overrides. See rate-limiting.md for the full model, configuration modes, and migration path.

Server initialization

The serve() function in rust/feature-flags/src/server.rs orchestrates startup:

1. Redis clients: Shared ReadWriteClient (auto-routes reads to replica). Optional dedicated flags Redis with 3-mode migration: shared-only -> dual-write -> dedicated-only.
2. Database pools: PostgresRouter with 4 pools (persons reader/writer, non-persons reader/writer), plus an optional behavioral cohorts reader pool. See database-interaction-patterns.md.
3. GeoIP: MaxMind database for IP geolocation.
4. Cohort cache: In-memory CohortCacheManager (moka, 256 MB default, 5-minute TTL).
5. HyperCache readers: 4 pre-initialized readers for flags, flags+cohorts, team metadata, and config.
6. Billing limiters: Redis-backed quota enforcement for feature flags and session replay.
7. Cookieless manager: Redis-backed cookieless identity resolution.
8. Background tasks: DB pool monitoring, cohort cache monitoring, rate limiter cleanup, health heartbeat.

Configuration reference

All values come from environment variables via the envconfig crate. Defined in rust/feature-flags/src/config.rs.

Server

PostgreSQL

Behavioral cohorts

The behavioral cohorts pool uses tight limits (max 5 connections, 1s statement timeout) since it only performs simple key lookups against the cohort_membership table. When BEHAVIORAL_COHORTS_READ_DATABASE_URL is not set, a NoOpCohortMembershipProvider is used and all realtime cohort checks return false (graceful degradation).

Redis

S3 / HyperCache

Rate limiting

See rate-limiting.md for the full configuration reference.

Caching

See Behavioral cohorts for cohort membership cache settings.

Observability

Other

Key dependencies

Middleware

Applied in order via Axum layers (defined in router.rs):

1. ConcurrencyLimitLayer: Caps concurrent flag evaluation requests (default 1000)
2. TraceLayer: HTTP request tracing with spans
3. CorsLayer: Permissive CORS (mirrors request origin, allows credentials, exposes x-posthog-rate-limit-warning)
4. track_metrics: Prometheus HTTP request metrics

Related files

See also

• Rate limiting - Warn-then-enforce model, configuration modes, per-token overrides
• Flag evaluation engine - How flags are matched and evaluated
• Database interaction patterns - PostgreSQL connection pooling and query routing
• HyperCache system - Multi-tier caching with Redis, S3, and PostgreSQL
• Experience continuity - Hash key overrides for consistent flag values across identity changes
• Django API endpoints - CRUD and management endpoints served by Django