manual/english/Integration/Grafana.md
NOTE: Some dashboard panels and metrics require Manticore Buddy. If some panels appear empty, ensure that Buddy is installed.
Grafana is an open-source platform for data visualization and monitoring that enables the creation of interactive dashboards and charts. Manticore Search can integrate with Grafana in two primary ways:
Currently, Grafana versions 10.0-13.0 are tested and supported.
Manticore provides a prebuilt Grafana dashboard and Prometheus alerting rules in the dedicated repository manticoresoftware/grafana-dashboard. The repository also includes an all-in-one Docker image with Grafana and Prometheus for a quick start. These resources can be used independently with your existing Grafana and Prometheus setup, allowing you to monitor Manticore's operational metrics—such as latency, resource usage, and errors—without needing to run a separate instance.
For a fully preconfigured setup, run the all-in-one Docker image with Grafana and Prometheus tailored for Manticore monitoring. This is ideal if you don't have an existing setup or want a quick test environment:
docker run -e MANTICORE_TARGETS=localhost:9308 -p 127.0.0.1:3000:3000 manticoresearch/dashboard:latest
MANTICORE_TARGETS is a comma-separated list of Manticore http endpoints in the host:port format (for example: localhost:9308,10.0.0.5:9308).
If you already have your own Grafana and Prometheus running, you can skip the Docker image and directly import the dashboard and alert rules as described below.
To add the Manticore dashboard to your existing Grafana instance:
job dashboard variable if needed (the default expected label value is manticoresearch).To integrate the Manticore alert rules into your existing Prometheus setup:
rule_files configuration.manticoresearch, update the {job="manticoresearch"} matchers in the rules accordingly.The manticore-alerts.yml file is designed to focus on key signals related to availability, overload, and resource exhaustion—issues that often impact users directly. Thresholds are set as safe defaults; adjust them based on your specific workload.
critical): Prometheus fails to scrape Manticore (up == 0 for 2 minutes). This typically indicates the metrics endpoint is down, unreachable, or the scrape configuration is incorrect.warning): Uptime remains below 5 minutes for 5 minutes. This often signals instability, such as crash loops, OOM kills, or orchestrator restarts.warning): manticore_maxed_out_error_count increases over the last 5 minutes. Manticore is rejecting requests due to hitting capacity, concurrency, or resource limits.warning): P95 search latency exceeds 500 ms for 10 minutes. Many users may perceive searches as slow.critical): P99 search latency exceeds 1,000 ms for 10 minutes. Tail latency is severely elevated, indicating very slow worst-case requests.warning): Work queue length exceeds 100 for 5 minutes. Requests are accumulating, which often leads to increased latency.warning): More than 90% of workers are active for 10 minutes. Worker threads are nearing full capacity, likely causing higher latency and queueing.warning): Query cache usage exceeds 90% of its configured maximum for 10 minutes. This increases the risk of cache churn and evictions, potentially slowing queries; consider expanding the cache or optimizing query patterns.warning): manticore_agent_retry_count increases by more than 10 in 5 minutes. This usually points to connectivity issues with remote agents (e.g., network problems, timeouts, or failures in distributed queries).warning): Current connections exceed 500 for 10 minutes. This could indicate a traffic surge, connection leaks, or slow clients.warning): manticore_slowest_thread_seconds exceeds 30 seconds for 10 minutes. Long-running or stuck queries are present; investigate slow queries and resource contention.warning): manticore_connect_time_seconds exceeds 0.2 seconds for 5 minutes. Connection establishment is delayed (e.g., due to network issues, TLS overhead, or server overload).critical): manticore_searchd_crashes_total increases in the last 10 minutes. A crash has occurred; review logs, core dumps, and OOM events.warning): Binlog file count exceeds 1,000 for 10 minutes. This suggests binlogs are not rotating or cleaning properly; check binlog settings and disk usage.warning): searchd file descriptor count exceeds 4,096 for 10 minutes. This risks hitting OS limits; verify ulimit -n, connections, open files, and potential leaks.warning): Buddy file descriptor count exceeds 4,096 for 10 minutes. Similar to the above, but for the Buddy process.warning): searchd anonymous RSS exceeds 8 GiB for 10 minutes. High non-file-backed memory usage; examine memory growth, query/index patterns, and container limits.warning): Buddy anonymous RSS exceeds 8 GiB for 10 minutes. Elevated Buddy memory usage; review Buddy logs and workload.warning): One or more tables listed in manticore.json are missing from SHOW TABLES for 10 minutes. This typically means tables failed to load or were dropped/moved; check startup logs and table paths.warning): Disk mapped cache ratio falls below 50% for 15 minutes. Increased disk I/O is likely, potentially slowing searches if the working set exceeds RAM.critical): Disk mapped cache ratio falls below 20% for 15 minutes. Severe cache misses; anticipate heavy disk I/O and high latency.The dashboard organizes panels by topic. Key highlights include:
This approach uses Grafana's MySQL connector to directly query Manticore tables, enabling visualizations of your search data, such as time-series trends or aggregations.
Before setting up data visualization:
To connect Manticore Search to Grafana:
localhost:9306)root)After connecting Manticore Search to Grafana, you can create dashboards and visualizations:
Here's a simple example using time-series data. First, create a table and load sample data:
CREATE TABLE btc_usd_trading (
id bigint,
time timestamp,
open float,
high float,
low float,
close float
);
Load the data:
curl -sSL https://gist.githubusercontent.com/donhardman/df109ba6c5e690f73198b95f3768e73f/raw/0fab3aee69d7007fad012f4e97f38901a64831fb/btc_usd_trading.sql | mysql -h0 -P9306
In Grafana, you can create:
Example queries:
-- Time series query
SELECT time, close FROM btc_usd_trading ORDER BY time;
-- Aggregation query
SELECT DATE(time) as date, AVG(close) as avg_price
FROM btc_usd_trading
GROUP BY date
ORDER BY date;
When working with Manticore Search through Grafana, you can:
information_schema.tablesFor more information and detailed tutorials: