consul - Apisix — ContextQMD

Summary

APACHE APISIX supports Consul as a service discovery

Configuration for discovery client

Configuration for Consul

First of all, we need to add following configuration in conf/config.yaml :

yaml

discovery:
  consul:
    servers:                      # make sure service name is unique in these consul servers
      - "http://127.0.0.1:8500"   # `http://127.0.0.1:8500` and `http://127.0.0.1:8600` are different clusters
      - "http://127.0.0.1:8600"   # `consul` service is default skip service
    token: "..."                  # if your consul cluster has enabled acl access control, you need to specify the token
    skip_services:                # if you need to skip special services
      - "service_a"
    timeout:
      connect: 1000               # default 2000 ms
      read: 1000                  # default 2000 ms
      wait: 60                    # default 60 sec
    weight: 1                     # default 1
    fetch_interval: 5             # default 3 sec, only take effect for keepalive: false way
    keepalive: true               # default true, use the long pull way to query consul servers
    sort_type: "origin"           # default origin
    default_service:              # you can define default service when missing hit
      host: "127.0.0.1"
      port: 20999
      metadata:
        fail_timeout: 1           # default 1 ms
        weight: 1                 # default 1
        max_fails: 1              # default 1
    dump:                         # if you need, when registered nodes updated can dump into file
       path: "logs/consul.dump"
       expire: 2592000            # unit sec, here is 30 day

And you can config it in short by default value:

yaml

discovery:
  consul:
    servers:
      - "http://127.0.0.1:8500"

The keepalive has two optional values:

true, default and recommend value, use the long pull way to query consul servers
false, not recommend, it would use the short pull way to query consul servers, then you can set the fetch_interval for fetch interval

The sort_type has four optional values:

origin, not sorting
host_sort, sort by host
port_sort, sort by port
combine_sort, with the precondition that hosts are ordered, ports are also ordered.

Dump Data

When we need reload apisix online, as the consul module maybe loads data from CONSUL slower than load routes from ETCD, and would get the log at the moment before load successfully from consul:

 http_access_phase(): failed to set upstream: no valid upstream node

So, we import the dump function for consul module. When reload, would load the dump file before from consul; when the registered nodes in consul been updated, would dump the upstream nodes into file automatically.

The dump has three optional values now:

path, the dump file save path
- support relative path, eg: logs/consul.dump
- support absolute path, eg: /tmp/consul.dump
- make sure the dump file's parent path exist
- make sure the apisix has the dump file's read-write access permission,eg: add below config in conf/config.yaml

yaml

nginx_config:                     # config for render the template to generate nginx.conf
  user: root                     # specifies the execution user of the worker process.

load_on_init, default value is true
- if true, just try to load the data from the dump file before loading data from consul when starting, does not care the dump file exists or not
- if false, ignore loading data from the dump file
- Whether true or false, we don't need to prepare a dump file for apisix at anytime
expire, unit sec, avoiding load expired dump data when load
- default 0, it is unexpired forever
- recommend 2592000, which is 30 days(equals 3600 * 24 * 30)

Register Http API Services

Now, register nodes into consul:

shell

curl -X PUT 'http://127.0.0.1:8500/v1/agent/service/register' \
-d '{
  "ID": "service_a1",
  "Name": "service_a",
  "Tags": ["primary", "v1"],
  "Address": "127.0.0.1",
  "Port": 8000,
  "Meta": {
    "service_a_version": "4.0"
  },
  "EnableTagOverride": false,
  "Weights": {
    "Passing": 10,
    "Warning": 1
  }
}'

curl -X PUT 'http://127.0.0.1:8500/v1/agent/service/register' \
-d '{
  "ID": "service_a1",
  "Name": "service_a",
  "Tags": ["primary", "v1"],
  "Address": "127.0.0.1",
  "Port": 8002,
  "Meta": {
    "service_a_version": "4.0"
  },
  "EnableTagOverride": false,
  "Weights": {
    "Passing": 10,
    "Warning": 1
  }
}'

In some cases, same service name might exist in different consul servers. To avoid confusion, use the full consul key url path as service name in practice.

Port Handling

When APISIX retrieves service information from Consul, it handles port values as follows:

If the service registration includes a valid port number, that port will be used.
If the port is nil (not specified) or 0, APISIX will default to port 80 for HTTP services.

Upstream setting

L7

Here is an example of routing a request with a URL of "/*" to a service which named "service_a" and use consul discovery client in the registry :

:::note You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

bash

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

:::

shell

$ curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d '
{
    "uri": "/*",
    "upstream": {
        "service_name": "service_a",
        "type": "roundrobin",
        "discovery_type": "consul"
    }
}'

The format response as below:

json

{
  "key": "/apisix/routes/1",
  "value": {
    "uri": "/*",
    "priority": 0,
    "id": "1",
    "upstream": {
      "scheme": "http",
      "type": "roundrobin",
      "hash_on": "vars",
      "discovery_type": "consul",
      "service_name": "service_a",
      "pass_host": "pass"
    },
    "create_time": 1669267329,
    "status": 1,
    "update_time": 1669267329
  }
}

You could find more usage in the apisix/t/discovery/consul.t file.

L4

Consul service discovery also supports use in L4, the configuration method is similar to L7.

shell

$ curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d '
{
    "remote_addr": "127.0.0.1",
    "upstream": {
      "scheme": "tcp",
      "service_name": "service_a",
      "type": "roundrobin",
      "discovery_type": "consul"
    }
}'

You could find more usage in the apisix/t/discovery/stream/consul.t file.

Debugging API

It also offers control api for debugging.

Memory Dump API

shell

GET /v1/discovery/consul/dump

For example:

shell

# curl http://127.0.0.1:9090/v1/discovery/consul/dump | jq
{
  "config": {
    "fetch_interval": 3,
    "timeout": {
      "wait": 60,
      "connect": 6000,
      "read": 6000
    },
    "weight": 1,
    "servers": [
      "http://172.19.5.30:8500",
      "http://172.19.5.31:8500"
    ],
    "keepalive": true,
    "default_service": {
      "host": "172.19.5.11",
      "port": 8899,
      "metadata": {
        "fail_timeout": 1,
        "weight": 1,
        "max_fails": 1
      }
    },
    "skip_services": [
      "service_d"
    ]
  },
  "services": {
    "service_a": [
      {
        "host": "127.0.0.1",
        "port": 30513,
        "weight": 1
      },
      {
        "host": "127.0.0.1",
        "port": 30514,
        "weight": 1
      }
    ],
    "service_b": [
      {
        "host": "172.19.5.51",
        "port": 50051,
        "weight": 1
      }
    ],
    "service_c": [
      {
        "host": "127.0.0.1",
        "port": 30511,
        "weight": 1
      },
      {
        "host": "127.0.0.1",
        "port": 30512,
        "weight": 1
      }
    ]
  }
}

Show Dump File API

It offers another control api for dump file view now. Maybe would add more api for debugging in future.

shell

GET /v1/discovery/consul/show_dump_file

For example:

shell

curl http://127.0.0.1:9090/v1/discovery/consul/show_dump_file | jq
{
  "services": {
    "service_a": [
      {
        "host": "172.19.5.12",
        "port": 8000,
        "weight": 120
      },
      {
        "host": "172.19.5.13",
        "port": 8000,
        "weight": 120
      }
    ]
  },
  "expire": 0,
  "last_update": 1615877468
}