docs/reference/query-languages/esql/README.md
[!NOTE] This directory contains the source files for the ES|QL documentation.
This README covers how the ES|QL docs are structured, what's hand-written vs. auto-generated, and how to add or update commands and functions correctly.
Adding a new command or function:
Promoting an existing command or function to GA:
Reference:
Learn about the docs system
Each command has its own standalone page under commands/ that includes a layout snippet from _snippets/commands/layout/.
To add a new processing command called <my_command>:
_snippets/commands/layout/<my_command>.md
_snippets/lists/processing-commands.md or _snippets/lists/source-commands.md
commands/<my-command>.md
navigation_title:::{include} of the layout snippetcommands/change-point.md as a templatedocs/reference/query-languages/toc.yml
processing-commands.md or source-commands.mdWhen adding tested examples to a command, for example adding an example to the CHANGE_POINT command, do the following:
x-pack/plugin/esql/qa/testFixtures/src/main/resources/ directory.-result._snippets/commands/examples directory. While you could add the content to that file, it is not necessary, merely that the file existsCommandDocsTests in the x-pack/plugin/esql module to generate the content.For example, we tag the following test in change_point.csv-spec:
example for docs
required_capability: change_point
// tag::changePointForDocs[]
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
| MV_EXPAND key
| EVAL value = CASE(key<13, 0, 42)
| CHANGE_POINT value ON key
| WHERE type IS NOT NULL
// end::changePointForDocs[]
;
// tag::changePointForDocs-result[]
key:integer | value:integer | type:keyword | pvalue:double
13 | 42 | step_change | 0.0
// end::changePointForDocs-result[]
;
Then we create the file _snippets/commands/examples/change_point.csv-spec/changePointForDocs.md with the content:
This should be overwritten
Then we run the test CommandDocsTests in the x-pack/plugin/esql module to generate the content.
Now the content of the changePointForDocs.md file should have been updated:
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
\```esql
ROW key=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]
| MV_EXPAND key
| EVAL value = CASE(key<13, 0, 42)
| CHANGE_POINT value ON key
| WHERE type IS NOT NULL
\```
| key:integer | value:integer | type:keyword | pvalue:double |
| --- | --- | --- | --- |
| 13 | 42 | step_change | 0.0 |
Finally include this file in the CHANGE_POINT command file _snippets/commands/layout/change_point.md:
**Examples**
The following example shows the detection of a step change:
:::{include} ../examples/change_point.csv-spec/changePointForDocs.md
:::
Each function has its own standalone page under functions-operators/<group>/ that includes a generated layout snippet from _snippets/functions/layout/.
To add a new function called <my_func> to the <group> group (e.g. string-functions):
[!TIP] For a full walkthrough of the Java implementation, see the guide in
package-info.java.
@FunctionInfo and @Param annotationsmv_, st_, to_, date_, is_),
add it to the switch statement in DocsV3Support.functionGroupFor().
This maps function names to groups for cross-reference link generation.
[!WARNING] Without this, the test in the next step will fail with
Docs Generation Error: Unknown function group.
AbstractFunctionTestCase./gradlew :x-pack:plugin:esql:test -Dtests.class='<MyFunc>Tests'_snippets/functions/ (layout, description, parameters, types, examples) and images/functions/_snippets/lists/<group>.md
functions-operators/<group>/<my_func>.mdfunctions-operators/<group>/<my_func>.md
navigation_title:::{include} of the layout snippetfunctions-operators/aggregation-functions/avg.md as a templatedocs/reference/query-languages/toc.yml
[!IMPORTANT] Starting with 9.0, we no longer publish separate documentation branches for every minor release (
9.0,9.1,9.2, etc.). This means there won't be a different page for9.1,9.2, and so on. Instead, all changes landing in subsequent minor releases will appear on the same page.
Because we now publish just one docs set off of the main branch, we use the applies_to metadata to differentiate features and their availability across different versions. This is a cumulative approach: instead of creating separate pages for each product and release, we update a single page with product- and version-specific details over time.
applies_to allows us to clearly communicate when features are introduced, when they transition from preview to GA, and which versions support specific functionality.
This metadata accepts a lifecycle and an optional version.
Use the @FunctionAppliesTo annotation within the @FunctionInfo annotation on function and operator classes to specify the lifecycle and version for functions and operators.
For example, to indicate that a function is in technical preview and applies to version 9.0.0, you would use:
@FunctionInfo(
returnType = "boolean",
preview = true, // this marks the feature as preview in serverless
appliesTo = {
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0")
},
...
)
When a feature evolves from preview in 9.0 to GA in 9.2, add a new entry alongside the existing preview entry and remove the preview = true boolean:
@FunctionInfo(
returnType = "boolean",
preview = false, // the preview boolean can be removed (or flipped to false) when the function becomes GA
appliesTo = {
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0"),
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2.0")
},
...
)
We updated DocsV3Support.java to generate the applies_to metadata correctly for functions and operators.
Use inline annotations to specify applies_to metadata in descriptions, parameter lists, etc.
For example, the second item in this list is in technical preview as of version 9.2:
- Item 1
- Item 2 {applies_to}`stack: preview 9.2.`
preview = true boolean for any tech preview feature - this is required for the Kibana inline docs
preview = true only when the feature becomes GA on serverless and is definitely going GA in the next minor releaseappliesTo entries - only add new ones as features evolve from preview to GA9.0.0, 9.1.0) when known, or just PREVIEW without a version if timing is uncertainapplies_to to examples where necessary[!IMPORTANT] We don't use
applies_toin the legacy asciidoc system for 8.x and earlier versions.
PREVIEW - Feature is in technical previewGA - Feature is generally availableDEPRECATED - Feature is deprecated and will be removed in a future releaseUNAVAILABLE - Feature is not available in the current version, but may be available in future releases[!NOTE] Unreleased version information is automatically sanitized in the docs build output. For example, say you specify
preview 9.3.0:
- Before
9.3.0is released, the live documentation will display "Planned for a future release" instead of the specific version number.- This will be updated automatically when the version is released.
List entries in _snippets/lists/ are always maintained by hand — they are never auto-generated. Each entry can carry inline {applies_to} tags.
[!IMPORTANT]
stacktags are cumulative — never remove astack: previewtag when adding astack: gatag. Theserverless: previewtag is different: drop it when the feature becomes GA.
A few formatting rules:
9.x format, not 9.x.0 — strip the patch versionserverless: ga — GA is the default on serverlessserverless: preview only when the feature is GA, then add stack: ga X.YFunctions and operators:
GA entry to appliesTo and remove (or set to false) preview = true:
@FunctionInfo(
- preview = true,
appliesTo = {
@FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.PREVIEW, version = "9.0.0"),
+ @FunctionAppliesTo(lifeCycle = FunctionAppliesToLifecycle.GA, version = "9.2.0")
},
)
./gradlew :x-pack:plugin:esql:test -Dtests.class='MyFuncTests'
stack: ga X.Y tag and drop serverless: preview:
- * [`MY_FUNC`](path/to/my-func.md) {applies_to}`stack: preview 9.0` {applies_to}`serverless: preview`
+ * [`MY_FUNC`](path/to/my-func.md) {applies_to}`stack: preview 9.0` {applies_to}`stack: ga 9.2`
Commands:
applies_to front matter in _snippets/commands/layout/<my_command>.md:
- stack: preview 9.1.0
+ stack: preview 9.1.0, ga 9.3.0
_snippets/lists/processing-commands.md (or source-commands.md):
- * [`MY_COMMAND`](/reference/.../my-command.md) {applies_to}`stack: preview 9.1` {applies_to}`serverless: preview`
+ * [`MY_COMMAND`](/reference/.../my-command.md) {applies_to}`stack: preview 9.1` {applies_to}`stack: ga 9.3`
docs/reference/query-languages/toc.yml is the source of truth for what gets published. A page that exists on disk but isn't listed there won't appear in the navigation or be published.
To merge docs before a feature is publicly available, use hidden instead of file in toc.yml:
- hidden: commands/my-command.md
Also comment out the corresponding entry in _snippets/lists/ with a % prefix:
% * [My Command](../commands/my-command.md)
[!NOTE] The page stays accessible via direct URL but won't appear in navigation or search. You don't need to change any
.mdcontent files. See #142859 for an example.
When you're ready to publish, flip both:
# toc.yml
- - hidden: commands/my-command.md
+ - file: commands/my-command.md
# _snippets/lists/
- % * [My Command](../commands/my-command.md)
+ * [My Command](../commands/my-command.md)
[!TIP] If the feature isn't yet available on stack, use
applies_toversion metadata rather than hiding the page. See Add version metadata.
All generated content is produced by running ESQL tests. The section below covers what to run and when.
Run a single function's tests to regenerate its snippets (layout, description, parameters, types, examples, syntax diagrams, and Kibana definitions):
./gradlew :x-pack:plugin:esql:test -Dtests.class='CaseTests'
To regenerate everything for all functions and operators:
./gradlew :x-pack:plugin:esql:test
Query settings (see SET) are documented in _snippets/commands/settings/. To regenerate, run QuerySettingsTests in the x-pack/plugin/esql module. Only settings with snapshot=false are included.
There are three overlapping mechanisms for generating the content:
AbstractFunctionTestCase class generates the content for all the functions and most operators.
This class makes use of the DocsV3Support class to generate the content.
It uses the @FunctionInfo and @Param annotations on function and operator classes to know what content should be generated.
All tests that extend this class will automatically generate the content for the functions they test.AbstractOperatorTestCase class. See, for example, operators such as Cast ::,
which uses CastOperatorTests to call directly into the DocsV3Support class to generate the content.CommandDocsTests class.
Currently, this only covers tested examples used in the documentation, and all other commands
content is static.
Since there are no annotations to mark which examples to use, the command documentation
relies on the docs author providing the knowledge of which examples to use by creating subdirectories
and examples files that match the csv-spec files and tags to include.To help differentiate between the static and generated content, the generated content is prefixed with a comment:
% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
PromQL documentation is generated separately and stored in:
docs/reference/query-languages/promql/kibana/definitions/*.json
For PromQL function documentation, see: https://prometheus.io/docs/prometheus/latest/querying/functions/
To generate the PromQL definition files, run:
./gradlew :x-pack:plugin:esql:test --tests "PromqlKibanaDefinitionGeneratorTests"
The result will be in x-pack/plugin/esql/build/testrun/test/temp/promql/kibana/definitions/.