integrations/Nginx/markdown/README.en_US.md
There are several ways to monitor Nginx; the most recommended one is the vts plugin:
Sample configuration:
[[instances]]
## An array of Nginx stub_status URI to gather stats.
urls = [
# "http://192.168.0.216:8000/nginx_status",
# "https://www.baidu.com/ngx_status"
]
## append some labels for series
# labels = { region="cloud", product="n9e" }
## interval = global.interval * interval_times
# interval_times = 1
## Set response_timeout (default 5 seconds)
response_timeout = "5s"
## Whether to follow redirects from the server (defaults to false)
# follow_redirects = false
## Optional HTTP Basic Auth Credentials
#username = "admin"
#password = "admin"
## Optional headers
# headers = ["X-From", "categraf", "X-Xyz", "abc"]
## Optional TLS Config
# use_tls = false
# tls_ca = "/etc/categraf/ca.pem"
# tls_cert = "/etc/categraf/cert.pem"
# tls_key = "/etc/categraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
Sample configuration:
[[instances]]
targets = [
# "http://127.0.0.1/status?format=json",
# "http://10.2.3.56/status?format=json"
]
# # append some labels for series
# labels = { region="cloud", product="n9e" }
# # interval = global.interval * interval_times
# interval_times = 1
## Set http_proxy (categraf uses the system wide proxy settings if it's is not set)
# http_proxy = "http://localhost:8888"
## Interface to use when dialing an address
# interface = "eth0"
## HTTP Request Method
# method = "GET"
## Set timeout (default 5 seconds)
# timeout = "5s"
## Whether to follow redirects from the server (defaults to false)
# follow_redirects = false
## Optional HTTP Basic Auth Credentials
# username = "username"
# password = "pa$$word"
## Optional headers
# headers = ["X-From", "categraf", "X-Xyz", "abc"]
## Optional TLS Config
# use_tls = false
# tls_ca = "/etc/categraf/ca.pem"
# tls_cert = "/etc/categraf/cert.pem"
# tls_key = "/etc/categraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
nginx_vts already supports exposing data in Prometheus format, so this collection plugin is actually no longer needed. Simply use categraf's prometheus collection plugin to read the Prometheus data exposed by nginx_vts. Sample configuration:
[[instances]]
urls = [
"http://IP:PORT/vts/format/prometheus"
]
labels = {job="nginx-vts"}
Typically used when business systems rely on a proxy service for external routing and mapping; it is one of the most common and most important proxy tools in operations.
This plugin should be enabled on the virtual machine where the nginx service is running.
down; if it succeeds, it is marked as up.down becomes 2, up becomes 1, and other states become 0, represented by the metric nginx_upstream_check_status_code. Therefore, we may need an alert rule like the following:Building the module from source is recommended. If you are not sure which modules to install, you can refer to:
cd /opt/nginx-1.20.1 && ./configure \
--prefix=/usr/share/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib64/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--http-client-body-temp-path=/var/lib/nginx/tmp/client_body \
--http-proxy-temp-path=/var/lib/nginx/tmp/proxy \
--http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi \
--http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi \
--http-scgi-temp-path=/var/lib/nginx/tmp/scgi \
--pid-path=/var/run/nginx.pid \
--lock-path=/run/lock/subsys/nginx \
--user=nginx \
--group=nginx \
--with-compat \
--with-threads \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-mail \
--with-mail_ssl_module \
--with-stream \
--with-stream_realip_module \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-select_module \
--with-poll_module \
--with-file-aio \
--with-http_xslt_module=dynamic \
--with-http_image_filter_module=dynamic \
--with-http_perl_module=dynamic \
--with-stream=dynamic \
--with-mail=dynamic \
--with-http_xslt_module=dynamic \
--add-module=/etc/nginx/third-modules/nginx_upstream_check_module \
--add-module=/etc/nginx/third-modules/ngx_devel_kit-0.3.0 \
--add-module=/etc/nginx/third-modules/lua-nginx-module-0.10.13 \
--add-module=/etc/nginx/third-modules/nginx-module-vts \
--add-module=/etc/nginx/third-modules/ngx-fancyindex-0.5.2
# adjust according to the number of CPU cores
make -j2
make install
Note: the third-party modules nginx_upstream_check_module, lua-nginx-module and nginx-module-vts are required dependencies for the related plugins.
[root@aliyun categraf]# cat /etc/nginx/conf.d/nginx-upstream.domains.com.conf
server {
listen 80;
listen 443 ssl;
server_name nginx-upstream.domains.com;
include /etc/nginx/ssl_conf/domains.com.conf;
location / {
check_status;
include /etc/nginx/ip_whitelist.conf;
}
access_log /var/log/nginx/nginx-upstream.domains.com.access.log main;
error_log /var/log/nginx/nginx-upstream.domains.com.error.log warn;
}
Visiting https://nginx-upstream.domains.com?format=json in a browser shows:
Visiting https://nginx-upstream.domains.com in a browser shows:
For example:
[root@aliyun upstream_conf]# cat upstream_n9e.conf
upstream n9e {
server 127.0.0.1:18000 weight=10 max_fails=2 fail_timeout=5s;
check interval=3000 rise=2 fall=5 timeout=1000 type=tcp default_down=false port=18000;
check_http_send "HEAD / HTTP/1.0\r\n\r\n";
check_http_expect_alive http_2xx http_3xx;
}
[root@aliyun upstream_conf]# cat upstream_n9e_server_api.conf
upstream n9e-server-api {
server 127.0.0.1:19000 weight=10 max_fails=2 fail_timeout=5s;
check interval=3000 rise=2 fall=5 timeout=1000 type=tcp default_down=false port=19000;
check_http_send "HEAD / HTTP/1.0\r\n\r\n";
check_http_expect_alive http_2xx http_3xx;
}
[root@aliyun upstream_conf]# cat upstream_vm.conf
upstream vm {
server 127.0.0.1:8428 weight=10 max_fails=2 fail_timeout=5s;
keepalive 20;
check interval=3000 rise=2 fall=5 timeout=1000 type=tcp default_down=false port=8428;
check_http_send "HEAD / HTTP/1.0\r\n\r\n";
check_http_expect_alive http_2xx http_3xx;
}
This configuration enables or defines the following features:
Add custom labels, so that data can be filtered by custom labels and alerts can be pushed more precisely.
The response timeout is 5 seconds.
Fill the urls field with the domain name defined in Prerequisite 2.
[root@aliyun conf]# cat input.nginx_upstream_check/nginx_upstream_check.toml
# # collect interval
# interval = 15
[[instances]]
# This is the most critical setting: the endpoint URL that exposes the status information
targets = [
"https://nginx-upstream.domains.com/?format=json"
]
# Please pay attention to the labels configuration
# If Categraf and Nginx run on the same machine, the target may be configured as 127.0.0.1
# If Nginx runs on multiple machines, each with its own Categraf collecting the local Nginx status,
# the time series data may end up with identical labels and be hard to distinguish. Of course, Categraf
# automatically attaches the ident label, which identifies the local hostname
# If the ident label is not enough, you can use the labels configuration below to attach labels such as instance, region, etc.
# # append some labels for series
labels = { cloud="my-cloud", region="my-region",azone="az1", product="my-product" }
# # interval = global.interval * interval_times
# interval_times = 1
### Set http_proxy (categraf uses the system wide proxy settings if it's is not set)
# http_proxy = "http://localhost:8888"
### Interface to use when dialing an address
# interface = "eth0"
### HTTP Request Method
# method = "GET"
### Set timeout (default 5 seconds)
# timeout = "5s"
### Whether to follow redirects from the server (defaults to false)
# follow_redirects = false
### Optional HTTP Basic Auth Credentials
# username = "username"
# password = "pa$$word"
### Optional headers
# headers = ["X-From", "categraf", "X-Xyz", "abc"]
### Optional TLS Config
# use_tls = false
# tls_ca = "/etc/categraf/ca.pem"
# tls_cert = "/etc/categraf/cert.pem"
# tls_key = "/etc/categraf/key.pem"
### Use TLS but skip chain & host verification
# insecure_skip_verify = false
./categraf --test --inputs nginx_upstream_check
Restart the categraf service for the changes to take effect
systemctl daemon-reload && systemctl restart categraf && systemctl status categraf
Check the startup logs for errors
journalctl -f -n 500 -u categraf | grep "E\!" | grep "W\!"
After 1-2 minutes, the data will show up in the charts, as shown below:
The key thing to check is whether the backend is abnormal. nginx_upstream_check_status_code returning 1 means healthy and returning 2 means abnormal (as can be seen from the chart above in actual testing).
nginx_upstream_check_status_code!=1 should be treated as abnormal and trigger an alert immediately: severity level 1, evaluation frequency 60 seconds, duration 60 seconds, recovery observation period 2 minutes, repeat notification interval 5 minutes, maximum number of notifications 0 (unlimited), sending the alert via the WeCom app and phone voice channels to the system operations team. This rule applies all day, Monday through Sunday.