ContextQMD
Back to Materialize

Install on Azure

doc/user/content/self-managed-deployments/installation/install-on-azure.md

12312.7 KB
Original Source

Materialize provides a set of modular Terraform modules that can be used to deploy all services required for Materialize to run on Azure. The module is intended to provide a simple set of examples on how to deploy Materialize. It can be used as is or modules can be taken from the example and integrated with existing DevOps tooling.

{{% self-managed/materialize-components-sentence %}} The example on this page deploys a complete Materialize environment on Azure using the modular Terraform setup from this repository.

{{< warning >}}

{{< self-managed/terraform-disclaimer >}}

{{< /warning >}}

What Gets Created

This example provisions the following infrastructure:

Resource Group

ResourceDescription
Resource GroupNew resource group to contain all resources

Networking

ResourceDescription
Virtual Network20.0.0.0/16 address space
AKS Subnet20.0.0.0/20 with NAT Gateway association and service endpoints for Storage and SQL
PostgreSQL Subnet20.0.16.0/24 delegated to PostgreSQL Flexible Server
NAT GatewayStandard SKU with static public IP for outbound connectivity
Private DNS ZoneFor PostgreSQL private endpoint resolution with VNet link

Compute

ResourceDescription
AKS ClusterVersion 1.32 with Cilium networking (network plugin: azure, data plane: cilium, policy: cilium)
Default Node PoolStandard_D4pds_v6 VMs, autoscaling 2-5 nodes, labeled for generic workloads
Materialize Node PoolStandard_E4pds_v6 VMs with 100GB disk, autoscaling 2-5 nodes, swap enabled, dedicated taints for Materialize workloads
Managed IdentitiesAKS cluster identity (used by AKS control plane to provision Azure resources like load balancers and network interfaces) and Workload identity (used by Materialize pods for secure, passwordless authentication to Azure Storage)

Database

ResourceDescription
Azure PostgreSQL Flexible ServerVersion 15
SKUGP_Standard_D2s_v3 (2 vCores, 4GB memory)
Storage32GB with 7-day backup retention
Network AccessPublic Network Access is disabled, Private access only (no public endpoint)
Databasematerialize database pre-created

Storage

ResourceDescription
Storage AccountPremium BlockBlobStorage with LRS replication for Materialize persistence
Containermaterialize blob container
Access ControlWorkload Identity federation for Kubernetes service account (passwordless authentication via OIDC)
Network AccessCurrently allows <red>all traffic</red>(production deployments should restrict to AKS subnet only traffic)

Kubernetes Add-ons

ResourceDescription
cert-managerCertificate management controller for Kubernetes that automates TLS certificate provisioning and renewal
Self-signed ClusterIssuerProvides self-signed TLS certificates for Materialize instance internal communication (balancerd, console). Used by the Materialize instance for secure inter-component communication.

Materialize

ResourceDescription
OperatorMaterialize Kubernetes operator in the materialize namespace
InstanceSingle Materialize instance in the materialize-environment namespace
Load BalancersAzure Load Balancers for access to Materialize {{< yaml-table data="self_managed/default_ports" >}}

Prerequisites

Azure Account Requirements

An active Azure subscription with appropriate permissions to create:

  • AKS clusters
  • Azure PostgreSQL Flexible Server instances
  • Storage accounts
  • Virtual networks and networking resources
  • Managed identities and role assignments

Required Tools

License Key

{{< yaml-table data="self_managed/license_key" >}}

Getting started: Simple example

{{< warning >}}

{{< self-managed/terraform-disclaimer >}}

{{< /warning >}}

{{< tip >}}

The simple example used in this tutorial enables Password authentication for the Materialize instance. To use a different authentication method, update authenticator_kind. See Authentication for the supported authentication mechanisms. {{< /tip >}}

Step 1: Set Up the Environment

  1. Open a terminal window.

  2. Clone the Materialize Terraform repository and go to the azure/examples/simple directory.

    bash
    git clone https://github.com/MaterializeInc/materialize-terraform-self-managed.git
cd materialize-terraform-self-managed/azure/examples/simple

  3. Authenticate with Azure.

    bash
    az login

    The command opens a browser window to sign in to Azure. Sign in.

  4. Select the subscription and tenant to use. After you have signed in, back in the terminal, your tenant and subscription information is displayed.

    none
    Retrieving tenants and subscriptions for the selection...

[Tenant and subscription selection]

No     Subscription name    Subscription ID                       Tenant
-----  -------------------  ------------------------------------  ----------------
[1]*   ...                  ...                                   ...

The default is marked with an *; the default tenant is '<Tenant>' and
subscription is '<Subscription Name>' (<Subscription ID>).

    Select the subscription and tenant.

Step 2: Configure Terraform Variables

  1. Create a terraform.tfvars file with the following variables:

    • subscription_id: Azure subscription ID
    • resource_group_name: Name for the resource group to create (e.g. mz-demo-rg)
    • name_prefix: Prefix for all resource names (e.g., simple-demo)
    • location: Azure region for deployment (e.g., westus2)
    • license_key: Materialize license key
    • tags: Map of tags to apply to resources
    hcl
    subscription_id     = "your-subscription-id"
resource_group_name = "mz-demo-rg"
name_prefix         = "simple-demo"
location            = "westus2"
license_key         = "your-materialize-license-key"
tags = {
  environment = "demo"
}
# internal_load_balancer = false   # default = true (internal load balancer). You can set to false = public load balancer.
# ingress_cidr_blocks = ["x.x.x.x/n", ...]
# k8s_apiserver_authorized_networks  = ["x.x.x.x/n", ...]

    {{% include-from-yaml data="self_managed/installation" name="installation-tfvars-variables-optional" %}}

Step 3: Apply the Terraform

  1. Initialize the Terraform directory to download the required providers and modules:

    bash
    terraform init

  2. Apply the Terraform configuration to create the infrastructure.

    bash
    terraform apply

    If you are satisfied with the planned changes, type yes when prompted to proceed.

  3. From the output, you will need the following field(s) to connect:

    • console_load_balancer_ip for the Materialize Console
    • balancerd_load_balancer_ip to connect PostgreSQL-compatible clients/drivers.
    • external_login_password_mz_system.
    bash
    terraform output -raw <field_name>

    {{< tip >}} Your shell may show an ending marker (such as %) because the output did not end with a newline. Do not include the marker when using the value. {{< /tip >}}

  4. Configure kubectl to connect to your cluster, replacing:

    • <your-resource-group-name> with your resource group name; i.e., the resource_group_name in the Terraform output or in the terraform.tfvars file.

    • <your-aks-cluster-name> with your cluster name; i.e., the aks_cluster_name in the Terraform output. For the sample example, your cluster name has the form {prefix_name}-aks; e.g., simple-demo-aks.

    bash
    # az aks get-credentials --resource-group <your-resource-group-name> --name <your-aks-cluster-name>
az aks get-credentials --resource-group $(terraform output -raw resource_group_name) --name $(terraform output -raw aks_cluster_name)

Step 4. Optional. Verify the deployment.

  1. Check the status of your deployment: {{% include-from-yaml data="self_managed/installation" name="installation-verify-status" %}}

Step 5: Connect to Materialize

You can connect to Materialize via the Materialize Console or PostgreSQL-compatible tools/drivers using the following ports:

{{< yaml-table data="self_managed/default_ports" >}}

Connect using the Materialize Console

{{% include-from-yaml data="self_managed/installation" name="installation-access-methods" %}}

Using the console_load_balancer_ip and external_login_password_mz_system from the Terraform output, you can connect to Materialize via the Materialize Console.

  1. To connect to the Materialize Console, open a browser to https://<console_load_balancer_ip>:8080, substituting your <console_load_balancer_ip>.

    From the terminal, you can type:

    sh
    open "https://$(terraform output -raw  console_load_balancer_ip):8080/materialize"

    {{< tip >}}

    {{% include-from-yaml data="self_managed/installation" name="install-uses-self-signed-cluster-issuer" %}}

    {{< /tip >}}

  2. Log in as mz_system, using external_login_password_mz_system as the password.

  3. Create new users and log out.

    In general, other than the initial login to create new users for new deployments, avoid using mz_system since mz_system also used by the Materialize Operator for upgrades and maintenance tasks.

    For more information on authentication and authorization for Self-Managed Materialize, see:

  4. Login as one of the created user.

Connect using psql

{{% include-from-yaml data="self_managed/installation" name="installation-access-methods" %}}

Using the balancerd_load_balancer_ip and external_login_password_mz_system from the Terraform output, you can connect to Materialize via PostgreSQL-compatible clients/drivers, such as psql.

  1. To connect using psql, in the connection string, specify:

    • mz_system as the user
    • balancerd_load_balancer_ip as the host
    • 6875 as the port:
    bash
    psql "postgres://mz_system@$(terraform output -raw balancerd_load_balancer_ip):6875/materialize"

    When prompted for the password, enter the external_login_password_mz_system value.

  2. Create new users and log out.

    In general, other than the initial login to create new users for new deployments, avoid using mz_system since mz_system also used by the Materialize Operator for upgrades and maintenance tasks.

    For more information on authentication and authorization for Self-Managed Materialize, see:

  3. Login as one of the created user.

Customizing Your Deployment

{{< tip >}} To reduce cost in your demo environment, you can tweak VM sizes and database tiers in main.tf. {{< /tip >}}

You can customize each Terraform module independently.

{{< note >}} Autoscaling: Uses Azure's native cluster autoscaler that integrates directly with Azure Virtual Machine Scale Sets for automated node scaling. {{< /note >}}

See also:

Cleanup

{{% self-managed/cleanup-cloud %}}

See Also