docs/reference/query-languages/esql/README.md
[!NOTE] This directory contains the source files for the ES|QL documentation.
This README covers how the ES|QL docs are structured, what's hand-written vs. auto-generated, and how to add or update commands and functions correctly.
Adding a new command or function:
Promoting an existing command or function to GA:
Reference:
Learn about the docs system
Each command has its own standalone page under commands/ that includes a layout snippet from _snippets/commands/layout/.
To add a new processing command called <my_command>:
_snippets/commands/layout/<my_command>.md
_snippets/lists/processing-commands.md or _snippets/lists/source-commands.md
commands/<my-command>.md
navigation_title:::{include} of the layout snippetcommands/change-point.md as a templatedocs/reference/query-languages/toc.yml
processing-commands.md or source-commands.mdWhen adding tested examples to a command, for example adding an example to the CHANGE_POINT command, do the following:
x-pack/plugin/esql/qa/testFixtures/src/main/resources/ directory.-result._snippets/generated/x-pack-esql/commands/examples directory. While you could add the content to that file, it is not necessary, merely that the file existsCommandDocsTests in the x-pack/plugin/esql module to generate the content.For example, we tag the following test in change_point.csv-spec:
example for docs
required_capability: change_point
// tag::changePointForDocs[]
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
| MV_EXPAND key
| EVAL value = CASE(key<13, 0, 42)
| CHANGE_POINT value ON key
| WHERE type IS NOT NULL
// end::changePointForDocs[]
;
// tag::changePointForDocs-result[]
key:integer | value:integer | type:keyword | pvalue:double
13 | 42 | step_change | 0.0
// end::changePointForDocs-result[]
;
Then we create the file _snippets/generated/x-pack-esql/commands/examples/change_point.csv-spec/changePointForDocs.md with the content:
This should be overwritten
Then we run the test CommandDocsTests in the x-pack/plugin/esql module to generate the content.
Now the content of the changePointForDocs.md file should have been updated:
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
\```esql
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
| MV_EXPAND key
| EVAL value = CASE(key<13, 0, 42)
| CHANGE_POINT value ON key
| WHERE type IS NOT NULL
\```
| key:integer | value:integer | type:keyword | pvalue:double |
| --- | --- | --- | --- |
| 13 | 42 | step_change | 0.0 |
Finally include this file in the CHANGE_POINT command file _snippets/commands/layout/change_point.md:
## Examples
The following example shows the detection of a step change:
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
:::
Each function has its own standalone page under functions-operators/<group>/ that includes a generated layout snippet from _snippets/generated/<plugin-name>/functions/layout/.
Generated snippets are organized by the Elasticsearch plugin that provides the function. The <plugin-name> matches the name field in the module's esplugin { } block in build.gradle (e.g. x-pack-esql for the core plugin, x-pack-esql-function-math for the math function plugin).
To add a new function called <my_func> to the <group> group (e.g. string-functions):
[!TIP] For a full walkthrough of the Java implementation, see the guide in
package-info.java.
@FunctionInfo and @Param annotationsmv_, st_, to_, date_, is_),
add it to the switch statement in DocsV3Support.functionGroupFor().
This maps function names to groups for cross-reference link generation.
[!WARNING] Without this, the test in the next step will fail with
Docs Generation Error: Unknown function group.
AbstractFunctionTestCase in the same module./gradlew :<module-path>:test -Dtests.class='<MyFunc>Tests'_snippets/generated/<plugin-name>/functions/ (layout, description, parameters, types, examples) and images/generated/<plugin-name>/_snippets/lists/<group>.md
functions-operators/<group>/<my_func>.mdfunctions-operators/<group>/<my_func>.md
navigation_title:::{include} of the layout snippetfunctions-operators/aggregation-functions/avg.md as a templatedocs/reference/query-languages/toc.yml
[!IMPORTANT] Starting with 9.0, we no longer publish separate documentation branches for every minor release (
9.0,9.1,9.2, etc.). This means there won't be a different page for9.1,9.2, and so on. Instead, all changes landing in subsequent minor releases will appear on the same page.
Because we now publish just one docs set off of the main branch, we use the applies_to metadata to differentiate features and their availability across different versions. This is a cumulative approach: instead of creating separate pages for each product and release, we update a single page with product- and version-specific details over time.
applies_to allows us to clearly communicate when features are introduced, when they transition from preview to GA, and which versions support specific functionality.
This metadata accepts a lifecycle and an optional version. The version supports three operators:
=X.x applies to that version only, for example =9.4X.x-Y.y applies to a version range, for example 9.4-9.6X.x+ applies to that version and later, for example 9.5+Use the @FunctionAppliesTo annotation within the @FunctionInfo annotation on function and operator classes to specify the lifecycle and version for functions and operators.
For example, to indicate that a function is in technical preview and applies to version 9.0.0, you would use:
@FunctionInfo(
returnType = "boolean",
preview = true, // this marks the feature as preview in serverless
appliesTo = {
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0")
},
...
)
When a feature evolves from preview in 9.0 to GA in 9.2, add a new entry alongside the existing preview entry, remove the preview = true boolean, and switch the version strings to range operators:
@FunctionInfo(
returnType = "boolean",
preview = false, // the preview boolean can be removed (or flipped to false) when the function becomes GA
appliesTo = {
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "=9.0"),
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2+")
},
...
)
The =9.0 marks that the feature was in preview only in 9.0. You can only state this at GA time, once you know it went GA in the next release. The generator passes these version strings straight through, so the layout snippet renders stack: preview =9.0, ga 9.2+ to match the list entry.
You can also use applies_to for Param, MapParam, and MapParamEntry. In this case applies_to is a plain text field that accepts the same format used in inline applies_to metadata.
For example, to mark a specific parameter as preview:
@Param(name = "query", type = { "keyword" }, description = "The query.", applies_to = "stack: preview 9.4.0")
To mark a whole map parameter as preview:
@MapParam(
name = "options",
description = "Optional parameters.",
optional = true,
applies_to = "stack: preview 9.5.0",
params = { ... }
)
To mark a specific map entry as preview:
@MapParam(
name = "options",
description = "Optional parameters.",
params = {
@MapParamEntry(
name = "my_option",
type = "keyword",
description = "Description of my_option.",
applies_to = "stack: preview 9.2"
)
}
)
We updated DocsV3Support.java to generate the applies_to metadata correctly for functions and operators.
When a function adds support for a new data type in a specific version, the Supported types table in the docs needs applies_to metadata on those type rows. This is done by annotating the type's TypedDataSupplier in the function's *Tests.java class.
Use withAppliesTo(...) to tag which version introduced the type, and withPreview() if the function has preview = true on its @FunctionInfo annotation:
FunctionAppliesTo histogramAppliesTo = appliesTo(FunctionAppliesToLifecycle.PREVIEW, "9.3.0", "", true);
// In the suppliers list:
MultiRowTestCaseSupplier.exponentialHistogramCases(1, 100)
.stream()
.map(s -> s.withAppliesTo(histogramAppliesTo).withPreview()) // withPreview() because the function has preview=true
.toList()
withAppliesTo(...) on TypedDataSupplier entries for any type that was added after the function's initial releasewithPreview() only if the data types function's @FunctionInfo has preview = trueThis ensures the generated Supported types table includes the correct {applies_to} and serverless: preview annotations per type row.
Use inline annotations to specify applies_to metadata in descriptions, parameter lists, etc.
For example, the second item in this list is in technical preview as of version 9.2:
- Item 1
- Item 2 {applies_to}`stack: preview 9.2.`
preview = true boolean for any tech preview feature - this is required for the Kibana inline docs
preview = true only when the feature becomes GA on serverless and is definitely going GA in the next minor releaseappliesTo entries - only add new ones as features evolve from preview to GA9.0.0, 9.1.0) when known, or just PREVIEW without a version if timing is uncertainapplies_to to examples where necessary[!IMPORTANT] We don't use
applies_toin the legacy asciidoc system for 8.x and earlier versions.
PREVIEW - Feature is in technical previewGA - Feature is generally availableDEPRECATED - Feature is deprecated and will be removed in a future releaseUNAVAILABLE - Feature is not available in the current version, but may be available in future releases[!NOTE] Unreleased version information is automatically sanitized in the docs build output. For example, say you specify
preview 9.3.0:
- Before
9.3.0is released, the live documentation will display "Planned for a future release" instead of the specific version number.- This will be updated automatically when the version is released.
List entries in _snippets/lists/ are always maintained by hand — they are never auto-generated. Each entry can carry inline {applies_to} tags.
[!IMPORTANT]
stacktags are cumulative — never remove astack: previewtag when adding astack: gatag. Theserverless: previewtag is different: drop it when the feature becomes GA.
A few formatting rules:
9.x format, not 9.x.0 — strip the patch versionserverless: ga — GA is the default on serverlessserverless: preview only when the feature is GA, then add stack: ga X.YFunctions and operators:
GA entry to appliesTo, remove (or set to false) preview = true, and switch the version strings to range operators:
@FunctionInfo(
- preview = true,
appliesTo = {
- @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0"),
+ @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "=9.0"),
+ @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2+")
},
)
=9.0 on the preview entry to mark that the feature was in preview only in 9.0, and 9.2+ on the GA entry. You can only state this at GA time, once you know the next release makes it GA../gradlew :<module-path>:test -Dtests.class='MyFuncTests'
{applies_to} tag, and drop serverless: preview:
- * [`MY_FUNC`](path/to/my-func.md) {applies_to}`stack: preview 9.0` {applies_to}`serverless: preview`
+ * [`MY_FUNC`](path/to/my-func.md) {applies_to}`stack: preview =9.0, ga 9.2+`
Commands:
applies_to front matter in _snippets/commands/layout/<my_command>.md:
- stack: preview 9.1.0
+ stack: preview 9.1.0, ga 9.3.0
_snippets/lists/processing-commands.md (or source-commands.md):
- * [`MY_COMMAND`](/reference/.../my-command.md) {applies_to}`stack: preview 9.1` {applies_to}`serverless: preview`
+ * [`MY_COMMAND`](/reference/.../my-command.md) {applies_to}`stack: preview 9.1` {applies_to}`stack: ga 9.3`
docs/reference/query-languages/toc.yml is the source of truth for what gets published. A page that exists on disk but isn't listed there won't appear in the navigation or be published.
To merge docs before a feature is publicly available, use hidden instead of file in toc.yml:
- hidden: commands/my-command.md
Also comment out the corresponding entry in _snippets/lists/ with a % prefix:
% * [My Command](../commands/my-command.md)
[!NOTE] The page stays accessible via direct URL but won't appear in navigation or search. You don't need to change any
.mdcontent files. See #142859 for an example.
When you're ready to publish, flip both:
# toc.yml
- - hidden: commands/my-command.md
+ - file: commands/my-command.md
# _snippets/lists/
- % * [My Command](../commands/my-command.md)
+ * [My Command](../commands/my-command.md)
[!TIP] If the feature isn't yet available on stack, use
applies_toversion metadata rather than hiding the page. See Add version metadata.
All generated content is produced by running ESQL tests. The section below covers what to run and when.
Generated snippets are written to _snippets/generated/<plugin-name>/functions/ where <plugin-name> is the name from the module's esplugin { } block. Run the test task for whichever module owns the function.
Run a single function's tests to regenerate its snippets (layout, description, parameters, types, examples, syntax diagrams, and Kibana definitions):
./gradlew :<module-path>:test -Dtests.class='CaseTests'
To regenerate everything for all functions and operators in a module:
./gradlew :<module-path>:test
Query settings (see SET) are documented in _snippets/generated/x-pack-esql/commands/settings/. To regenerate, run QuerySettingsTests in the x-pack/plugin/esql module. Markdown docs and the table of contents only include non-snapshot settings; Kibana definitions are generated for all settings, with snapshot-only ones carrying "snapshotOnly": true.
There are three overlapping mechanisms for generating the content:
AbstractFunctionTestCase class generates the content for all the functions and most operators.
This class makes use of the DocsV3Support class to generate the content.
It uses the @FunctionInfo and @Param annotations on function and operator classes to know what content should be generated.
All tests that extend this class will automatically generate the content for the functions they test.AbstractOperatorTestCase class. See, for example, operators such as Cast ::,
which uses CastOperatorTests to call directly into the DocsV3Support class to generate the content.CommandDocsTests class.
Currently, this only covers tested examples used in the documentation, and all other commands
content is static.
Since there are no annotations to mark which examples to use, the command documentation
relies on the docs author providing the knowledge of which examples to use by creating subdirectories
and examples files that match the csv-spec files and tags to include.To help differentiate between the static and generated content, the generated content is prefixed with a comment:
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
PromQL reference documentation is auto-generated and stored in:
docs/reference/query-languages/promql/_snippets/generated/x-pack-esql/functions/ # function markdown snippets: per-function pages, per-category lists, and the "Not yet supported" list
docs/reference/query-languages/promql/_snippets/generated/x-pack-esql/operators/ # operator markdown snippets: per-operator pages, per-category lists, and the "Not yet supported" list
docs/reference/query-languages/promql/kibana/generated/x-pack-esql/ # Kibana definition JSON files for functions and operators
For functions, the descriptions, examples, and "Differences from Prometheus" notes come from each function's PromqlFunctionDefinition (see PromqlBuiltinFunctionDefinitions and the PROMQL_DEFINITION fields on the individual function classes). For operators, the same metadata (plus the documentation category and per-operator stack availability) comes from OPERATOR_DEFS in PromqlDocsSupport, because operators have no runtime registry. Edit those sources rather than the generated files.
The hand-written hub pages (promql/functions.md, promql/operators.md) and the per-category wrapper pages (promql/functions/<category>.md, promql/operators/<category>.md) include the generated snippets; update them by hand when adding a category.
For PromQL function and operator documentation, see: https://prometheus.io/docs/prometheus/latest/querying/functions/ and https://prometheus.io/docs/prometheus/latest/querying/operators/
To regenerate these files, run:
./gradlew :x-pack:plugin:esql:test --tests "PromqlKibanaDefinitionGeneratorTests"
Locally this writes the generated files straight into the directories above (generateDocs=write). In CI the same test runs with generateDocs=assert and fails if the committed files are out of date, so commit the regenerated files.