ContextQMD
Back to Mattermost

AWS OpenSearch server setup

docs/main/administration-guide/scale/opensearch-setup.mdx

11.10.09.5 KB
Original Source

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

import Inc0_common_configure_mattermost_for_enterprise_search from './common-configure-mattermost-for-enterprise-search.mdx';

<PlanAvailability slug="ent-plus" />

AWS OpenSearch Service allows you to search large volumes of data quickly, in near real-time, by creating and managing an index of post data. The indexing process can be managed from the System Console after setting up and connecting an OpenSearch server. The post index is stored on the OpenSearch server and updated constantly after new posts are made. In order to index existing posts, a bulk index of the entire post database must be generated.

Deploying AWS OpenSearch includes the following two steps: setting up AWS OpenSearch, and configuring Mattermost.

Set up AWS OpenSearch

From Mattermost v9.11, beta support is available for AWS OpenSearch v1.x and v2.x. This document covers both on‑premises and AWS OpenSearch setup, including manual steps and Terraform examples.

We highly recommend that you set up an AWS OpenSearch server on a separate machine from the Mattermost server.

<Tabs> <TabItem value="on-premises-opensearch" label="On-Premises OpenSearch">
  1. To install on-premise OpenSearch, provision a dedicated server (e.g. Ubuntu 22.04 LTS).
  2. Install Java (OpenSearch requires Java 11+):
sh
sudo apt update 
sudo apt install -y openjdk-11-jdk 
java -version
  1. Download & extract OpenSearch 2.x:
sh
wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.9.0/opensearch-2.9.0-linux-x64.tar.gz 
tar -xzf opensearch-2.9.0-linux-x64.tar.gz 
sudo mv opensearch-2.9.0 /usr/share/opensearch
  1. Create a dedicated user & set permissions:
sh
sudo useradd --no-create-home --shell /bin/false opensearch 
sudo chown -R opensearch:opensearch /usr/share/opensearch
  1. Configure systemd:
sh
[Unit] 
  Description=OpenSearch 
  Wants=network-online.target 
  After=network-online.target 

[Service] 
  Type=notify 
  User=opensearch 
  Group=opensearch 
  ExecStart=/usr/share/opensearch/bin/opensearch 
  Restart=on-failure 
  LimitNOFILE=65536 
  LimitNPROC=4096 

[Install] 
  WantedBy=multi-user.target
  1. Edit opensearch.yml to include the following:
yaml
cluster.name: mattermost-cluster
node.name: node-1
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
discovery.seed_hosts: ["&lt;other-node-ip&gt;"]
cluster.initial_master_nodes: ["node-1", "node-2"]
<div class="note"> <div class="title">

Note

</div>

Ensure path.data and path.logs directories exist and are owned by the opensearch user before starting the service:

sh
sudo mkdir -p /var/lib/opensearch /var/log/opensearch
sudo chown -R opensearch:opensearch /var/lib/opensearch /var/log/opensearch
</div>
  1. Enable & start OpenSearch:
sh
sudo systemctl daemon-reload 
sudo systemctl enable opensearch 
sudo systemctl start opensearch 
sudo systemctl status opensearch
  1. Install the icu-analyzer plugin to the /usr/share/opensearch/plugins directory by running the following command:
sh
sudo /usr/share/opensearch/bin/opensearch-plugin install analysis-icu

(Optional) CJK language analyzer plugins: To improve search for Korean, Japanese, or Chinese content, install one or more of the following language-specific analyzer plugins: analysis-nori (Korean), analysis-kuromoji (Japanese), and analysis-smartcn (Chinese).

sh
sudo /usr/share/opensearch/bin/opensearch-plugin install analysis-nori
sudo /usr/share/opensearch/bin/opensearch-plugin install analysis-kuromoji
sudo /usr/share/opensearch/bin/opensearch-plugin install analysis-smartcn

After installing the CJK plugins, restart OpenSearch to load them:

sh
sudo systemctl restart opensearch

Then enable the EnableCJKAnalyzers configuration setting. See Enabling Chinese, Japanese, and Korean Search for additional CJK search configuration options.

If you enable CJK analyzers on a server with existing indexed content, you must purge and rebuild the search index in System Console > Environment > Elasticsearch for the CJK analyzers to take effect on existing posts.

Terraform (Docker) Example

sh
provider "docker" { 
  host = "unix:///var/run/docker.sock" 
}

resource "docker_image" "opensearch" { 
  name = "opensearchproject/opensearch:2.9.0" 
}

resource "docker_container" "opensearch" { 
  name = "opensearch" 
  image = docker_image.opensearch.latest 

  ports { 
    internal = 9200 
    external = 9200 
  }
  ports { 
    internal = 9600 
    external = 9600 
  }

  env = [
    "cluster.name=mattermost-cluster",
    "network.host=0.0.0.0",
    "discovery.type=single-node", # remove for multi-node 
  ]

  restart = "unless-stopped" 
}

resource "null_resource" "install_icu_plugin" {
  depends_on = [docker_container.opensearch]

  provisioner "local-exec" {
    command = "docker exec opensearch /usr/share/opensearch/bin/opensearch-plugin install analysis-icu && docker restart opensearch"
  }
}
</TabItem> <TabItem value="aws-opensearch-console-setup" label="AWS OpenSearch Console Setup">
  1. To install AWS OpenSearch, open the AWS Console > OpenSearch Service.
  2. Create a domain, where:
  • Domain name: mattermost-os
  • Engine version: OpenSearch 2.x.
  1. Configure the cluster, where:
  • instance type: r6g.xlarge.search
  • data nodes: 2
  • master nodes: 2
  • storage: EBS gp3 (1536 GiB, 4608 IOPS, 250 MiB/s)
  1. Specify the network for: VPC with 2 subnets, and a security group allowing Mattermost IPs on port 443.
<Note>

Port 9200 is commonly used for local or on-premise OpenSearch. The AWS OpenSearch domain only supports HTTPS over port 443.

</Note>
  1. Configure the access policy (JSON):
sh
&#123;
  "Version": "2012-10-17", 
  "Statement": [&#123; 
    "Effect": "Allow", 
    "Principal": &#123; "AWS": 
"arn:aws:iam::123456789012:role/MattermostAppRole" &#125;, 
    "Action": "es:*", 
    "Resource": "arn:aws:es:us-east-1:123456789012:domain/mattermost-os/*" &#125;] 
&#125;
  1. Configure the following advanced settings (JSON):
sh
&#123;
  "action.destructive_requires_name": "false", 
  "rest.action.multi.allow_explicit_index": "true", 
  "indices.query.bool.max_clause_count": "1024", 
  "indices.fielddata.cache.size": "20" 
&#125;
  1. Configure the automated snapshot start hour as 23 (UTC), enforce HTTPS, then review & create.
  2. To test, run the following command:
sh
curl https://mattermost-os-xxxxxxxxxxx.us-east-1.es.amazonaws.com

AWS Terraform Example

sh
provider "aws" &#123; 
  region = "us-east-1" 
&#125;

resource "aws_iam_role" "os_service_role" &#123; 
  name = "OSServiceRole" 
  assume_role_policy = &lt;<EOF 
&#123;
  "Version": "2012-10-17", 
  "Statement": [&#123; 
    "Action": "sts:AssumeRole", 
    "Effect": "Allow", 
    "Principal": &#123; "Service": "es.amazonaws.com" &#125; 
  &#125;]
&#125;
EOF
&#125;

resource "aws_opensearch_domain" "mattermost" &#123; 
  domain_name = "mattermost-os" 
  engine_version = "OpenSearch_2.9"
  cluster_config &#123; 
    instance_type = "r6g.xlarge.search" 
    instance_count = 2 
    dedicated_master_enabled = true 
    dedicated_master_type = "r6g.xlarge.search" 
    dedicated_master_count = 2 
    zone_awareness_enabled = true 
  &#125;

  ebs_options &#123; 
    ebs_enabled = true 
      volume_type = "gp3" 
      volume_size = 1536 
      iops = 4608 
  &#125;

  vpc_options &#123; 
    subnet_ids = ["subnet-blah1", "subnet-blah2"] 
    security_group_ids = ["sg-1234567890"] 
  &#125;

  advanced_options = &#123; 
    "rest.action.multi.allow_explicit_index" = "true" 
    "indices.query.bool.max_clause_count" = "1024" 
    "indices.fielddata.cache.size" = "20" 
    "action.destructive_requires_name" = "false" 
  &#125;

  access_policies = &lt;<POLICY 
&#123;
  "Version": "2012-10-17", 
  "Statement": [&#123; 
    "Effect": "Allow", 
    "Principal": &#123; 
      "AWS": "arn:aws:iam::123456789012:role/MattermostAppRole" 
    &#125;,
      "Action": "es:*", 
      "Resource": "arn:aws:es:us-east-1:123456789012:domain/mattermost-os/*" 
    &#125;]
  &#125;
  POLICY
    service_software_options &#123; 
      automated_snapshot_start_hour = 23 
    &#125;

    domain_endpoint_options &#123; 
      enforce_https = true 
    &#125;
&#125;
</TabItem> </Tabs>

Configure Mattermost

Follow these steps to configure Mattermost to use your AWS OpenSearch server and to generate the post index:

  1. Go to System Console > Environment > Elasticsearch.
  2. Set Enable Elasticsearch Indexing to true to enable the other the settings on the page.
  3. Ensure Backend type is set to opensearch.
  4. Set the Server Connection Address to your Elasticsearch or OpenSearch cluster endpoint.
  5. Monitor cluster health: curl https://mattermost-os-xxxxx.us-east-1.es.amazonaws.com/_cluster/health

<Inc0_common_configure_mattermost_for_enterprise_search />