Onyx Azure modules

Status

These modules are validated against the azurerm provider schema and each one ships a terraform test suite that plans it against a mocked provider. They have not yet been applied to a live subscription. Treat the first deployment as a first deployment.

Overview

This directory contains Terraform modules to provision the core Azure infrastructure for Onyx:

• vnet: a virtual network with subnets sized for AKS, a NAT gateway for stable egress, and optional flow logs
• aks: an AKS cluster with workload identity, node pools for the application and the document index, and optional GPU and sandbox pools
• postgres: a PostgreSQL Flexible Server on a delegated subnet, its private DNS zone, and five metric alerts
• redis: an Azure Cache for Redis behind a private endpoint, with four metric alerts
• storage: a storage account and container for the Onyx file store, with versioning, lifecycle rules and network rules
• waf: a regional Web Application Firewall policy for an Application Gateway
• onyx: a higher-level composition that wires the above together

Use the onyx module for a working cluster with sane defaults. Use the individual modules when you need more control.

These mirror the AWS modules in ../aws. Read Differences from the AWS modules before porting a configuration across.

Consuming these modules from another repository

The quickstart below uses local paths. To consume the modules from somewhere else, point source at this repository and pin a ref:

module "vnet" {
  source              = "git::https://github.com/onyx-dot-app/onyx.git//deployment/terraform/modules/azure/vnet?ref=tf-azure/v1.0.0"
  resource_group_name = "onyx-prod"
  location            = "eastus"
}

Azure releases are tagged tf-azure/vX.Y.Z, versioned independently of both the AWS modules and Onyx product releases. A commit sha works as a ref too, and is the better choice for automated consumers: a sha cannot be moved, where a tag can.

Pin something. Without a ref Terraform tracks the default branch, so an unrelated merge can change your infrastructure.

Quickstart (copy/paste)

Unlike the AWS composition, this module declares no provider block. The azurerm provider needs a features block and a subscription, both of which belong to your root module.

locals {
  location = "eastus"
  # Supply this from a secret store or TF_VAR_ in anything but a scratch stack.
  postgres_password = "your-postgres-password"
}

provider "azurerm" {
  features {}
  subscription_id = "00000000-0000-0000-0000-000000000000"

  # The storage module turns shared access keys off, so Terraform has to reach
  # the account with your Entra identity rather than with a key.
  storage_use_azuread = true
}

module "onyx" {
  source = "./modules/azure/onyx"

  name     = "onyx"
  location = local.location
  size     = "medium"

  postgres_password = local.postgres_password

  # Required. A public API server with no ranges is reachable from every address
  # on the internet, so the module refuses to build one unless you either
  # restrict it, make the cluster private, or say the exposure is intended.
  api_server_authorized_ip_ranges = ["203.0.113.0/24"]
}

# The kubernetes provider needs the cluster, so it reads its credentials back
# out of the module.
provider "kubernetes" {
  host                   = module.onyx.cluster_host
  cluster_ca_certificate = base64decode(module.onyx.cluster_ca_certificate)
  client_certificate     = base64decode(module.onyx.client_certificate)
  client_key             = base64decode(module.onyx.client_key)
}

output "storage_account_name" {
  value = module.onyx.storage_account_name
}

output "postgres_host" {
  value = module.onyx.postgres_host
}

output "redis_host" {
  value = module.onyx.redis_host
}

Then:

terraform init
terraform apply

T-shirt sizing

size sets every compute and data-plane knob together. Any individual sizing variable set to a non-null value overrides its tier default.

Roughly: small suits pilots and small teams, medium a department or company, large an org-wide deployment. The index pool is memory-optimised at every tier because on Azure it carries the document index itself.

Using an existing network

module "onyx" {
  source = "./modules/azure/onyx"

  location               = "eastus"
  create_virtual_network = false

  virtual_network_id         = "/subscriptions/.../virtualNetworks/existing"
  aks_subnet_id              = "/subscriptions/.../subnets/aks"
  postgres_subnet_id         = "/subscriptions/.../subnets/postgres"
  private_endpoint_subnet_id = "/subscriptions/.../subnets/private-endpoints"

  postgres_password               = local.postgres_password
  api_server_authorized_ip_ranges = ["203.0.113.0/24"]
}

The postgres subnet must be delegated to Microsoft.DBforPostgreSQL/flexibleServers, and the private endpoint subnet needs private_endpoint_network_policies set to Disabled. Without a NAT gateway on the AKS subnet the composition falls back to letting AKS manage outbound, and the egress address can then change. If your subnet already has a NAT gateway, set aks_outbound_type = "userAssignedNATGateway" to keep it.

What each module does

onyx

Creates a resource group, then wires the modules below together with t-shirt sizing. Its outputs carry everything the Helm chart needs.

vnet

A virtual network and four subnets, keyed by purpose: aks, private_endpoints, postgres (delegated) and app_gateway. A NAT gateway attaches to the subnets that opt into it, so egress keeps one address.

aks

An AKS cluster with the OIDC issuer and workload identity on, a system pool, a document-index pool, and optional GPU and Craft sandbox pools. Given a list of storage accounts it creates a managed identity, federates it to a Kubernetes service account, and grants it blob data access.

postgres

A Flexible Server on a delegated subnet, so it has no public endpoint, plus the private DNS zone that makes it resolvable. prevent_destroy guards the server and the database.

redis

An Azure Cache for Redis reachable only through a private endpoint, with its plaintext port disabled.

storage

A storage account and a private container. Shared access keys are off by default: Onyx authenticates with DefaultAzureCredential, so no key has to exist.

waf

A regional WAF policy carrying the OWASP Core Rule Set, the Microsoft bot manager set, two rate limits, and optional IP allowlisting and geo blocking. Attach it to an Application Gateway. Front Door uses a different resource, azurerm_cdn_frontdoor_firewall_policy, and cannot take this one.

Differences from the AWS modules

Most of these follow from the platform rather than from taste.

Enabling flow logs

Azure writes flow logs to a storage account and needs a Network Watcher in the region. The composition creates a second storage account for them, so the network never depends on the account holding application data:

module "onyx" {
  # ...
  enable_flow_logs = true
}

Azure creates a Network Watcher named NetworkWatcher_<region> in a resource group called NetworkWatcherRG the first time a virtual network appears in a region. If your subscription has that creation turned off, create one first or point network_watcher_name at yours.

This only applies to a network the composition creates. With create_virtual_network = false it refuses the setting rather than creating a log storage account that nothing would ever write to; configure the flow log against your own network instead.

Installing the Onyx Helm chart (after Terraform)

az aks get-credentials --resource-group "$(terraform output -raw resource_group_name)" \
  --name "$(terraform output -raw cluster_name)"

Install into the onyx namespace. A Kubernetes service account belongs to one namespace, and the aks module creates the federated one in onyx. A release installed anywhere else references an account that does not exist there, and the API and Celery pods never start.

Terraform creates the namespace itself, because the service account cannot be created before it exists and the Helm install happens afterwards. So the release joins that namespace rather than making its own:

helm install onyx onyx/onyx --namespace onyx -f values.yaml

Set create_workload_namespace = false if something else already creates it.

Three pieces of chart configuration are needed. None of them have defaults that work here, so a plain helm install will not pick up the infrastructure Terraform just built.

1. Point the workloads at the federated service account

The aks module creates a service account annotated with the managed identity's client ID. Pods only receive a token if they run under that account and carry the azure.workload.identity/use label, which is what the webhook looks for. Without both, DefaultAzureCredential finds no token and every file-store call fails to authenticate.

serviceAccount:
  create: false
  name: onyx-workload-access   # matches the aks module's default, in namespace onyx

# The label is per pod, so it goes on every component that touches the file
# store: the API server and the Celery workers.
api:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_primary:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_heavy:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_light:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_docfetching:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_docprocessing:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_user_file_processing:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_scheduled_tasks:
  podLabels:
    azure.workload.identity/use: "true"
celery_worker_monitoring:
  podLabels:
    azure.workload.identity/use: "true"
celery_beat:
  podLabels:
    azure.workload.identity/use: "true"

2. Point the file store at the storage account

The outputs line up with the environment the app reads:

Leave AZURE_STORAGE_ACCOUNT_KEY unset. Onyx authenticates with DefaultAzureCredential, which picks up the workload identity from step 1, and the storage account has shared keys turned off anyway.

3. Send the document index to its own node pool

The aks module taints the index pool document-index=true:NoSchedule so nothing else lands on it. The OpenSearch subchart sets no tolerations of its own, so without this it schedules onto the system pool instead and the dedicated pool sits empty:

opensearch:
  nodeSelector:
    onyx.app/workload: document-index
  tolerations:
    - key: document-index
      operator: Equal
      value: "true"
      effect: NoSchedule

Set index_node_pool_enabled = false if you would rather run the index on the system pool and skip this.

Testing

Every module has a terraform test suite that plans it against a mocked provider, so the suites need no Azure subscription and no credentials:

cd deployment/terraform/modules/azure/vnet
terraform init -backend=false
terraform test

They cover the wiring and the input validation, which is where a wrong argument name or an out-of-range value would otherwise only surface at apply. They do not prove the modules work against Azure. Only an apply does that.

Security

• Shared access keys on the storage account are off, and blobs are never anonymously readable.
• The database and cache have no public endpoint.
• The cache refuses its plaintext port, and both refuse TLS below 1.2.
• The API server will not be built open. You must set api_server_authorized_ip_ranges, or private_cluster_enabled, or allow_unrestricted_api_server_access to record that an open control plane is what you meant.
• The database will not be built without a way to log in: supply postgres_password, or turn on entra_database_authentication_only together with a database_administrator_object_id.
• Turn restrict_storage_to_cluster on once Terraform runs from inside the network. From outside, it locks out whoever manages the container.