High Availability Workflows

This notebook covers a lot of the core workflows to setup and manage YBA in a High Availability configuration.

Setup

First, import the required packages.

Next, specify some important variables:

platform_api_key: The API key used to authenticate with the active YBA
standby_api_key: The API key used to authenticate with the standby YBA
platform_address: The address of the active YBA API
standby_address: The address of the standby YBA API

Finally, open a HTTP connection to both the active and standby YBA.

python

import http.client
import json
import os
from pprint import pprint

platform_api_key = os.getenv('YB_API_KEY')
standby_api_key = os.getenv('YB_STANDBY_API_KEY')
platform_address = os.getenv('API_BASE_URL')
standby_address = os.getenv('API_STANDBY_URL')
conn = http.client.HTTPSConnection(f"{platform_address}", context=http.client.ssl._create_unverified_context())
standby_conn = http.client.HTTPSConnection(f"{standby_address}", context=http.client.ssl._create_unverified_context())

headers = {
  'Content-Type': "application/json",
  'X-AUTH-YW-API-TOKEN': f"{platform_api_key}"
}
session_info_url = "/api/v1/session_info"

conn.request("GET", session_info_url, headers=headers)
session_info = json.loads(conn.getresponse().read())
pprint(session_info)
customer_uuid = session_info['customerUUID']

standby_headers = {
  'Content-Type': "application/json",
  'X-AUTH-YW-API-TOKEN': f"{standby_api_key}"
}
standby_conn.request("GET", session_info_url, headers=standby_headers)
pprint(json.loads(standby_conn.getresponse().read()))

Certificate Setup

This section details how to add the CA certificate of both the active and standby YBA to the active YBA's trust store. This will be used to make requests from the active to the standby. These certificates will also be transferred on promotion so do NOT need to be set up on the standby.

Variables:

active_cert_path: The path to active YBA's CA certificate
standby_cert_path: The path to standby YBA's CA certificate

python

# Add the certificates of BOTH active and standby to the active YBA
# Either read certs from file or directly paste contents
active_cert_path = os.getenv('YB_ACTIVE_CERT_PATH')
standby_cert_path = os.getenv('YB_STANDBY_CERT_PATH')

try:
  with open(os.path.expanduser(active_cert_path), 'r') as active:
    active_cert = active.read()
  with open(os.path.expanduser(standby_cert_path), 'r') as standby:
    standby_cert = standby.read()
except Exception as e:
  pprint(f"An error occurred: {e}")

json_data_active = json.dumps({"name":"active-cert","contents":active_cert})
json_data_standby = json.dumps({"name": "standby-cert", "contents":standby_cert})

custom_ca_url = f"/api/v1/customers/{customer_uuid}/customCAStore"
conn.request("POST", custom_ca_url, body=json_data_active, headers=headers)
pprint(json.loads(conn.getresponse().read()))

conn.request("POST", custom_ca_url, body=json_data_standby, headers=headers)
pprint(json.loads(conn.getresponse().read()))

HA Config Setup

This section details all the API calls necessary to configure a valid active/standby HA pair. It uses the addresses specified in the first section.

The replication frequency can be changed in the request body frequency. The default is set to 60000 milliseconds (1 minute).

python

# Generate HA authentication key used for both active and standby
conn.request("GET", "/api/v1/settings/ha/generate_key", headers=headers)
cluster_key = json.loads(conn.getresponse().read())
pprint(cluster_key)

# Create config on active
ha_config_url = "/api/v1/settings/ha/config"
conn.request("POST", ha_config_url, body=json.dumps(cluster_key), headers=headers)
config_uuid = json.loads(conn.getresponse().read())['uuid']

# Create config on standby
standby_conn.request("POST", ha_config_url, body=json.dumps(cluster_key), headers=standby_headers)
standby_uuid = json.loads(standby_conn.getresponse().read())['uuid']

# Create active YBA instance on active YBA
active_instance_url = f"/api/v1/settings/ha/config/{config_uuid}/instance"
https_active_address = f"https://{platform_address}"
active_active_body = {"address":https_active_address, "is_leader": True, "is_local": True}
conn.request("POST", active_instance_url, body=json.dumps(active_active_body), headers=headers)
pprint(json.loads(conn.getresponse().read()))

# Create standby YBA instance on active YBA
https_standby_address = f"https://{standby_address}"
active_standby_body = {"address":https_standby_address, "is_leader": False, "is_local": False}
conn.request("POST", active_instance_url, body=json.dumps(active_standby_body), headers=headers)
pprint(json.loads(conn.getresponse().read()))

# Create standby YBA instance on standby YBA
standby_instance_url = f"/api/v1/settings/ha/config/{standby_uuid}/instance"
standby_standby_body = {"address":https_standby_address, "is_leader": False, "is_local": True}
standby_conn.request("POST", standby_instance_url, body=json.dumps(standby_standby_body), headers=standby_headers)
standby_instance = json.loads(standby_conn.getresponse().read())
pprint(standby_instance)

# Create replication schedule
schedule_url = f"/api/v1/settings/ha/config/{config_uuid}/replication_schedule/start"
frequency = {"frequency_milliseconds": 60000}
conn.request("PUT", schedule_url, body=json.dumps(frequency), headers=headers)
pprint(json.loads(conn.getresponse().read()))

HA Promotion

This section describes how to promote a standby to an active (and internally will also demote the current active). If the active demotion does not work please follow the manual steps in the next section to manually demote the active instance. The standby config information is used from the previous section.

python

standby_config_uuid = standby_instance['config_uuid']
standby_instance_uuid = standby_instance['uuid']

# Get the most recent backup
backup_url = f"/api/v1/settings/ha/config/{standby_config_uuid}/backup/list"
standby_conn.request("GET", backup_url, headers=standby_headers)
backup_list = json.loads(standby_conn.getresponse().read())
recent_backup = backup_list[0]

# Promote standby instance to active
promote_url = f"/api/v1/settings/ha/config/{standby_config_uuid}/instance/{standby_instance_uuid}/promote"
standby_conn.request("POST", promote_url, body=json.dumps({"backup_file": recent_backup}), headers=standby_headers)
pprint(standby_conn.getresponse().read())

Manual HA Demotion

This section lists all of the steps necessary to manually demote an active instance. This is useful when the HA pair gets into an incorrect state with two running actives (highly dangerous).

For simplicity sake we will assume that the previously defined standby is the "true" active and was succesfully promoted, but the original active is "stale" and needs to be demoted. The specific scenario may change based on your configuration, but the steps should be the same.

Query the true active for the accurate last failover timestamp.
Manually send demote request to stale active

Note the leader_address body must be the full URL (including HTTPS) of the leader, similar to how the instance addresses were filled in above.

python

# Find the last failover timestamp
standby_conn.request("GET", ha_config_url, headers=standby_headers)
timestamp = json.loads(standby_conn.getresponse().read())["last_failover"]

# Send demote request to stale active identifying the new leader
demote_url = f"/api/v1/settings/ha/internal/config/demote/{timestamp}"
conn.request("POST", demote_url, headers=headers, body=json.dumps({"leader_address": https_standby_address}))
pprint(conn.getresponse().read())