docs/api-reference/dynamic-configuration-api.md
import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';
<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, ~ software distributed under the License is distributed on an ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY ~ KIND, either express or implied. See the License for the ~ specific language governing permissions and limitations ~ under the License. -->This document describes the API endpoints to retrieve and manage dynamic configurations for the Coordinator and Overlord in Apache Druid.
In this topic, http://ROUTER_IP:ROUTER_PORT is a placeholder for your Router service address and port.
Replace it with the information for your deployment.
For example, use http://localhost:8888 for quickstart deployments.
The Coordinator has dynamic configurations to tune certain behavior on the fly, without requiring a service restart. For information on the supported properties, see Coordinator dynamic configuration.
Retrieves the current Coordinator dynamic configuration. Returns a JSON object with the dynamic configuration properties.
GET /druid/coordinator/v1/config
Successfully retrieved dynamic configuration
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config"
GET /druid/coordinator/v1/config HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
{
"millisToWaitBeforeDeleting": 900000,
"maxSegmentsToMove": 100,
"replicantLifetime": 15,
"replicationThrottleLimit": 500,
"balancerComputeThreads": 1,
"killDataSourceWhitelist": [],
"killPendingSegmentsSkipList": [],
"maxSegmentsInNodeLoadingQueue": 500,
"decommissioningNodes": [],
"decommissioningMaxPercentOfMaxSegmentsToMove": 70,
"pauseCoordination": false,
"replicateAfterLoadTimeout": false,
"maxNonPrimaryReplicantsToLoad": 2147483647,
"useRoundRobinSegmentAssignment": true,
"smartSegmentLoading": true,
"debugDimensions": null,
"turboLoadingNodes": [],
"cloneServers": {}
}
Submits a JSON-based dynamic configuration spec to the Coordinator. For information on the supported properties, see Dynamic configuration.
POST /druid/coordinator/v1/config
The endpoint supports a set of optional header parameters to populate the author and comment fields in the configuration history.
X-Druid-Author
X-Druid-Comment
Successfully updated dynamic configuration
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config" \
--header 'Content-Type: application/json' \
--data '{
"millisToWaitBeforeDeleting": 900000,
"maxSegmentsToMove": 5,
"percentOfSegmentsToConsiderPerMove": 100,
"useBatchedSegmentSampler": true,
"replicantLifetime": 15,
"replicationThrottleLimit": 10,
"balancerComputeThreads": 1,
"emitBalancingStats": true,
"killDataSourceWhitelist": [],
"killPendingSegmentsSkipList": [],
"maxSegmentsInNodeLoadingQueue": 100,
"decommissioningNodes": [],
"decommissioningMaxPercentOfMaxSegmentsToMove": 70,
"pauseCoordination": false,
"replicateAfterLoadTimeout": false,
"maxNonPrimaryReplicantsToLoad": 2147483647,
"useRoundRobinSegmentAssignment": true,
"turboLoadingNodes": [],
"cloneServers": {}
}'
POST /druid/coordinator/v1/config HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 683
{
"millisToWaitBeforeDeleting": 900000,
"maxSegmentsToMove": 5,
"percentOfSegmentsToConsiderPerMove": 100,
"useBatchedSegmentSampler": true,
"replicantLifetime": 15,
"replicationThrottleLimit": 10,
"balancerComputeThreads": 1,
"emitBalancingStats": true,
"killDataSourceWhitelist": [],
"killPendingSegmentsSkipList": [],
"maxSegmentsInNodeLoadingQueue": 100,
"decommissioningNodes": [],
"decommissioningMaxPercentOfMaxSegmentsToMove": 70,
"pauseCoordination": false,
"replicateAfterLoadTimeout": false,
"maxNonPrimaryReplicantsToLoad": 2147483647,
"useRoundRobinSegmentAssignment": true,
"turboLoadingNodes": [],
"cloneServers": {}
}
A successful request returns an HTTP 200 OK message code and an empty response body.
Retrieves the history of changes to Coordinator dynamic configuration over an interval of time. Returns an empty array if there are no history records available.
GET /druid/coordinator/v1/config/history
The endpoint supports a set of optional query parameters to filter results.
interval
/. For example, 2023-07-13/2023-07-19. The default interval is one week. You can change this period by setting druid.audit.manager.auditHistoryMillis in the runtime.properties file for the Coordinator.count
n entries.Successfully retrieved history
</TabItem> </Tabs>The following example retrieves the dynamic configuration history between 2022-07-13 and 2024-07-19. The response is limited to 10 entries.
curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/history?interval=2022-07-13%2F2024-07-19&count=10"
GET /druid/coordinator/v1/config/history?interval=2022-07-13/2024-07-19&count=10 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
[
{
"key": "coordinator.config",
"type": "coordinator.config",
"auditInfo": {
"author": "",
"comment": "",
"ip": "127.0.0.1"
},
"payload": "{\"millisToWaitBeforeDeleting\":900000,\"maxSegmentsToMove\":5,\"replicantLifetime\":15,\"replicationThrottleLimit\":10,\"balancerComputeThreads\":1,\"killDataSourceWhitelist\":[],\"killPendingSegmentsSkipList\":[],\"maxSegmentsInNodeLoadingQueue\":100,\"decommissioningNodes\":[],\"decommissioningMaxPercentOfMaxSegmentsToMove\":70,\"pauseCoordination\":false,\"replicateAfterLoadTimeout\":false,\"maxNonPrimaryReplicantsToLoad\":2147483647,\"useRoundRobinSegmentAssignment\":true,\"smartSegmentLoading\":true,\"debugDimensions\":null,\"decommissioningNodes\":[]}",
"auditTime": "2023-10-03T20:59:51.622Z"
}
]
Broker dynamic configuration is managed through the Coordinator but consumed by Brokers. These settings control broker behavior such as query blocking rules.
Retrieves the current Broker dynamic configuration. Returns a JSON object with the dynamic configuration properties.
GET /druid/coordinator/v1/broker/config
Successfully retrieved broker dynamic configuration
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/broker/config"
GET /druid/coordinator/v1/broker/config HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
{
"queryBlocklist": [
{
"ruleName": "block-expensive-scans",
"dataSources": ["large_table"],
"queryTypes": ["scan"]
}
]
}
Updates the Broker dynamic configuration.
POST /druid/coordinator/v1/broker/config
The endpoint supports a set of optional header parameters to populate the audit log information.
X-Druid-Author
X-Druid-Comment
Successfully updated configuration
</TabItem> </Tabs>curl -X POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/broker/config" \
-H "Content-Type: application/json" \
-H "X-Druid-Author: admin" \
-H "X-Druid-Comment: Add query blocklist rules" \
-d '{
"queryBlocklist": [
{
"ruleName": "block-expensive-scans",
"dataSources": ["large_table", "huge_dataset"],
"queryTypes": ["scan"]
},
{
"ruleName": "block-debug-queries",
"contextMatches": {
"debug": "true"
}
}
]
}'
POST /druid/coordinator/v1/broker/config HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
X-Druid-Author: admin
X-Druid-Comment: Add query blocklist rules
{
"queryBlocklist": [
{
"ruleName": "block-expensive-scans",
"dataSources": ["large_table", "huge_dataset"],
"queryTypes": ["scan"]
},
{
"ruleName": "block-debug-queries",
"contextMatches": {
"debug": "true"
}
}
]
}
A successful request returns an HTTP 200 OK message code and an empty response body.
The following table shows the dynamic configuration properties for the Broker.
| Property | Description | Default |
|---|---|---|
queryBlocklist | List of rules to block queries based on datasource, query type, and/or query context parameters. Each rule defines criteria that are combined with AND logic. Blocked queries return an HTTP 403 error. See Query blocklist rules for details. | none |
Query blocklist rules allow you to block specific queries based on datasource, query type, and/or query context parameters. This feature is useful for preventing expensive or problematic queries from impacting cluster performance.
Each rule in the queryBlocklist array is a JSON object with the following properties:
| Property | Description | Required | Default |
|---|---|---|---|
ruleName | Unique name identifying this blocklist rule. Used in error messages when queries are blocked. | Yes | N/A |
dataSources | List of datasource names to match. A query matches if it references any datasource in this list. | No | Matches all datasources |
queryTypes | List of query types to match (e.g., scan, timeseries, groupBy, topN). A query matches if its type is in this list. | No | Matches all query types |
contextMatches | Map of query context parameter key-value pairs to match. A query matches if all specified context parameters match the provided values (case-sensitive string comparison). | No | Matches all contexts |
Rule matching behavior:
queryTypes or setting it to null matches all query types)Note: Query blocking is best-effort. Queries may not be blocked in certain cases, such as when a Broker has recently started and hasn't received the config yet, or if the Broker cannot contact the Coordinator. Brokers poll the configuration periodically (default every 1 minute) and also receive push updates from the Coordinator for immediate propagation.
Error response:
When a query is blocked, the Broker returns an HTTP 403 error with a message indicating the query ID and the rule that blocked it:
{
"error": "Forbidden",
"errorMessage": "Query[abc-123-def] blocked by rule[block-expensive-scans]",
"persona": "USER",
"category": "FORBIDDEN"
}
Retrieves the history of changes to Broker dynamic configuration over an interval of time. Returns an empty array if there are no history records available.
GET /druid/coordinator/v1/broker/config/history
The endpoint supports a set of optional query parameters to filter results.
interval
/. For example, 2023-07-13/2023-07-19. The default interval is one week. You can change this period by setting druid.audit.manager.auditHistoryMillis in the runtime.properties file for the Coordinator.count
n entries.Successfully retrieved history
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/broker/config/history?count=10"
GET /druid/coordinator/v1/broker/config/history?count=10 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
[
{
"key": "broker.config",
"type": "broker.config",
"auditInfo": {
"author": "admin",
"comment": "Add query blocklist rules",
"ip": "127.0.0.1"
},
"payload": "{\"queryBlocklist\":[{\"ruleName\":\"block-expensive-scans\",\"dataSources\":[\"large_table\"],\"queryTypes\":[\"scan\"]}]}",
"auditTime": "2024-03-06T12:00:00.000Z"
}
]
The Overlord has dynamic configurations to tune how Druid assigns tasks to workers. For information on the supported properties, see Overlord dynamic configuration.
Retrieves the current Overlord dynamic configuration. Returns a JSON object with the dynamic configuration properties. Returns an empty response body if there is no current Overlord dynamic configuration.
GET /druid/indexer/v1/worker
Successfully retrieved dynamic configuration
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker"
GET /druid/indexer/v1/worker HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
{
"type": "default",
"selectStrategy": {
"type": "fillCapacityWithCategorySpec",
"workerCategorySpec": {
"categoryMap": {},
"strong": true
}
},
"autoScaler": null
}
Submits a JSON-based dynamic configuration spec to the Overlord. For information on the supported properties, see Overlord dynamic configuration.
POST /druid/indexer/v1/worker
The endpoint supports a set of optional header parameters to populate the author and comment fields in the configuration history.
X-Druid-Author
X-Druid-Comment
Successfully updated dynamic configuration
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker" \
--header 'Content-Type: application/json' \
--data '{
"type": "default",
"selectStrategy": {
"type": "fillCapacityWithCategorySpec",
"workerCategorySpec": {
"categoryMap": {},
"strong": true
}
},
"autoScaler": null
}'
POST /druid/indexer/v1/worker HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
Content-Type: application/json
Content-Length: 196
{
"type": "default",
"selectStrategy": {
"type": "fillCapacityWithCategorySpec",
"workerCategorySpec": {
"categoryMap": {},
"strong": true
}
},
"autoScaler": null
}
A successful request returns an HTTP 200 OK message code and an empty response body.
Retrieves the history of changes to Overlord dynamic configuration over an interval of time. Returns an empty array if there are no history records available.
GET /druid/indexer/v1/worker/history
The endpoint supports a set of optional query parameters to filter results.
interval
/. For example, 2023-07-13/2023-07-19. The default interval is one week. You can change this period by setting druid.audit.manager.auditHistoryMillis in the runtime.properties file for the Overlord.count
n entries.Successfully retrieved history
</TabItem> </Tabs>The following example retrieves the dynamic configuration history between 2022-07-13 and 2024-07-19. The response is limited to 10 entries.
curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker/history?interval=2022-07-13%2F2024-07-19&count=10"
GET /druid/indexer/v1/worker/history?interval=2022-07-13%2F2024-07-19&count=10 HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
[
{
"key": "worker.config",
"type": "worker.config",
"auditInfo": {
"author": "",
"comment": "",
"ip": "127.0.0.1"
},
"payload": "{\"type\":\"default\",\"selectStrategy\":{\"type\":\"fillCapacityWithCategorySpec\",\"workerCategorySpec\":{\"categoryMap\":{},\"strong\":true}},\"autoScaler\":null}",
"auditTime": "2023-10-03T21:49:49.991Z"
}
]
Returns an array of all the worker nodes in the cluster along with its corresponding metadata.
GET /druid/indexer/v1/workers
Successfully retrieved worker nodes
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/workers"
GET /druid/indexer/v1/workers HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
[
{
"worker": {
"scheme": "http",
"host": "localhost:8091",
"ip": "198.51.100.0",
"capacity": 2,
"version": "0",
"category": "_default_worker_category"
},
"currCapacityUsed": 0,
"currParallelIndexCapacityUsed": 0,
"availabilityGroups": [],
"runningTasks": [],
"lastCompletedTaskTime": "2023-09-29T19:13:05.505Z",
"blacklistedUntil": null
}
]
Returns Overlord scaling events if autoscaling runners are in use. Returns an empty response body if there are no Overlord scaling events.
GET /druid/indexer/v1/scaling
Successfully retrieved scaling events
</TabItem> </Tabs>curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/scaling"
GET /druid/indexer/v1/scaling HTTP/1.1
Host: http://ROUTER_IP:ROUTER_PORT
A successful request returns a 200 OK response and an array of scaling events.