Elasticsearch compatible API

In order to facilitate migrations and integrations with existing tools, Quickwit offers an Elasticsearch/Opensearch compatible API. This API is incomplete. This page lists the available features and endpoints.

Supported endpoints

All the API endpoints start with the api/v1/_elastic/ prefix.

_bulk   Batch ingestion endpoint

POST api/v1/_elastic/_bulk

POST api/v1/_elastic/<index>/_bulk

The _bulk ingestion API makes it possible to index a batch of documents, possibly targeting several indices in the same request.

Request Body example

{ "create" : { "_index" : "wikipedia", "_id" : "1" } }
{"url":"https://en.wikipedia.org/wiki?id=1","title":"foo","body":"foo"}
{ "create" : { "_index" : "wikipedia", "_id" : "2" } }
{"url":"https://en.wikipedia.org/wiki?id=2","title":"bar","body":"bar"}
{ "create" : { "_index" : "wikipedia", "_id" : "3" } }
{"url":"https://en.wikipedia.org/wiki?id=3","title":"baz","body":"baz"}'

Ingest a batch of documents to make them searchable using the Elasticsearch bulk API. This endpoint provides compatibility with tools or systems that already send data to Elasticsearch for indexing. Currently, only the create action of the bulk API is supported, all other actions such as delete or update are ignored.

If an index is specified via the url path, it will act as a default value for the _index properties.

The refresh parameter is supported.

:::caution The quickwit API will not report errors, you need to check the server logs.

In Elasticsearch, the create action has a specific behavior when the ingested documents contain an identifier (the _id field). It only inserts such a document if it was not inserted before. This is extremely handy to achieve At-Most-Once indexing. Quickwit does not have any notion of document id and does not support this feature. :::

:::info The payload size is limited to 10MB as this endpoint is intended to receive documents in batch. :::

Query parameter

Response

The response is a JSON object, and the content type is application/json; charset=UTF-8.

_search   Index search endpoint

POST api/v1/_elastic/<index_id>/_search

GET api/v1/_elastic/<index_id>/_search

Request Body example

{
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "bitpacking"
          }
        },
        {
          "term": {
            "actor.login": {
              "value": "fulmicoton"
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "actor.id": {
        "order": null
      }
    }
  ],
  "aggs": {
    "event_types": {
      "terms": {
        "field": "type",
        "size": 5
      }
    }
  }
}

Search into a specific index using the Elasticsearch search API.

Some of the parameter can be passed as query string parameter, and some via JSON payload. If a parameter appears both as a query string parameter and in the JSON payload, the query string parameter value will take priority.

Supported Query string parameters

Supported Request Body parameters

Sort order

You can define up to two criteria on which to apply sort. The second criterion will only be used in presence of a tie for the first criterion.

A given criterion can either be

• the name of a fast field (explicitly defined in the schema or captured by the dynamic mode)
• _score to sort by BM25.

By default, the sort order is ascending for fast fields and descending for _score.

When sorting by a fast field and this field contains several values in a single document, only the first value is used for sorting.

The sort order can be set as descending/ascending using the following syntax.

{
  // ...
  "sort" : [
    { "timestamp" : {"order" : "asc"}},
    { "serial_number" : "desc" }
  ]
  // ...
}

It is also possible to not supply an order and rely on the default order using the following syntax.

{ //...
  "sort" : ["_score", "timestamp"]
  // ...
}

If no format is provided for timestamps, timestamps are returned with milliseconds precision.

If you need nanosecond precision, you can use the epoch_nanos_int format. Beware this means the resulting JSON may contain high numbers for which there is loss of precision when using languages where all numbers are floats, such as JavaScript.

{
  // ...
  "sort" : [
    { "timestamp" : {"format": "epoch_nanos_int","order" : "asc"}},
    { "serial_number" : "desc" }
  ]
  // ...
}

#### Search after

When sorting results, the answer looks like the following

```json
{
  // ...
  "hits": {
    // ...
    "hits": [
      // ...
      {
        // ...
        "sort": [
          1701962929199
        ]
      }
    ]
  }
}

You can pass the sort value of the last hit in a subsequent request where other fields are kept unchanged:

{
  // keep all fields from the original request
  "search_after": [
    1701962929199
  ]
}

This allows you to paginate your results.

_msearch   Multi search API

POST api/v1/_elastic/_msearch

Request Body example

{"index": "gharchive" }
{"query" : {"match" : { "author.login": "fulmicoton"}}}
{"index": "gharchive"}
{"query" : {"match_all" : {}}}

Multi search endpoint ES API reference

Runs several search requests at once.

The payload is expected to alternate:

• a header json object, containing the targeted index id.
• a search request body as defined in the [_search endpoint section].

_search/scroll   Scroll API

GET api/v1/_elastic/_search/scroll

Supported Request Body parameters

The _search/scroll endpoint, in combination with the _search API makes it possible to request successive pages of search results. First, the client needs to call the search api with a scroll query parameter, and then pass the scroll_id returned in the response payload to _search/scroll endpoint.

Each subsequent call to the _search/scroll endpoint will return a new scroll_id pointing to the next page.

:::tip

Using _search and then _search/scroll is somewhat similar to using _search with the search_after parameter, except that it creates a lightweight snapshot view of the dataset during the initial call to _search. Further calls to _search/scroll only return results from that view, thus ensuring more consistent results.

:::

_cat   Cat API

GET api/v1/_elastic/_cat/indices/<index>

GET api/v1/_elastic/_cat/indices

Supported Query string parameters

Use the cat indices API to get the following information for each index in a cluster:

• Shard count
• Document count
• Deleted document count
• Primary store size
• Total store size

Response

The response is a JSON object, and the content type is application/json; charset=UTF-8.

Example response:

[
  {
    "dataset.size": "0b",
    "docs.count": "0",
    "docs.deleted": "0",
    "health": "green",
    "index": "otel-traces-v0_7",
    "pri": "1",
    "pri.store.size": "0b",
    "rep": "1",
    "status": "open",
    "store.size": "0b",
    "uuid": "otel-traces-v0_7:01HTJC6TQDGM07KBDQZ2KDHW53"
  },
  {
    "dataset.size": "387.5gb",
    "docs.count": "224453081",
    "docs.deleted": "0",
    "health": "green",
    "index": "otel-logs-v0_7",
    "pri": "1",
    "pri.store.size": "37.5gb",
    "rep": "1",
    "status": "open",
    "store.size": "37.5gb",
    "uuid": "otel-logs-v0_7:01HTJC6TME1JGXBFERHZ0FJ860"
  }
]

_field_caps   Field capabilities API

GET api/v1/_elastic/<index>/_field_caps

POST api/v1/_elastic/<index>/_field_caps

GET api/v1/_elastic/_field_caps

POST api/v1/_elastic/_field_caps

The field capabilities API returns information about the capabilities of fields among multiple indices.

Supported Query string parameters

Supported Request Body parameters

index_filter

The index_filter parameter allows you to filter which indices contribute to the field capabilities response. When provided, Quickwit uses the filter query to prune indices (splits) that cannot match the filter, and only returns field capabilities for the remaining ones.

Like Elasticsearch, this is a best-effort approach: Quickwit may return field capabilities from indices that do not actually contain any matching documents. In Quickwit, the filtering is limited to the existing split-pruning based on metadata:

• Time pruning: Range queries on the timestamp field can eliminate splits whose time range does not overlap with the filter.
• Tag pruning: Term queries on tag fields can eliminate splits that do not contain the requested tag value.

Other filter types (e.g. full-text queries or term queries on non-tag fields) are accepted but will not prune any splits — all indices will be returned as if no filter was specified. In particular, Quickwit does not check whether terms are present in the term dictionary.

Request Body example

{
  "index_filter": {
    "range": {
      "timestamp": {
        "gte": "2024-01-01T00:00:00Z",
        "lt": "2024-02-01T00:00:00Z"
      }
    }
  }
}

{
  "index_filter": {
    "term": {
      "status": "active"
    }
  }
}

Query DSL

Elasticsearch Query DSL reference.

The following query types are supported.

query_string

Elasticsearch reference documentation

Example

{
  "query": {
    "query_string": {
      "query": "bitpacking AND author.login:fulmicoton",
      "fields": [
        "payload.description"
      ]
    }
  }
}

Supported parameters

bool

Elasticsearch reference documentation

Example

{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "bitpacking"
          }
        }
      ],
      "must_not": {
        "term": {
          "type": {
            "value": "CommitEvent"
          }
        }
      }
    }
  }
}

Supported parameters

range

Elasticsearch reference documentation

Example

{
  "query": {
    "range": {
      "my_date_field": {
        "lt": "2015-02-01T00:00:13Z",
        "gte": "2015-02-01T00:00:10Z"
      }
    }
  }
}

Supported parameters

match

Elasticsearch reference documentation

Example

{
  "query": {
    "match": {
        "type": {
            "query": "CommitEvent",
            "zero_terms_query": "all"
        }
    }
  }
}

Supported Parameters

match_phrase

Elasticsearch reference documentation

Example

{
  "query": {
    "match_phrase": {
      "title": "search keywords",
      "analyzer": "default"
    }
  }
}

match_phrase_prefix

Elasticsearch reference documentation

Example

{
  "query": {
    "match_phrase_prefix": {
      "payload.commits.message": {
        "query": "automated comm" // This will match "automated commit" for instance.
      }
    }
  }
}

Supported Parameters

match_bool_prefix

Elasticsearch reference documentation

Example

{
  "query": {
    "match_bool_prefix": {
      "payload.commits.message": {
        "query": "automated comm" // This will match "automated commit" for instance.
      }
    }
  }
}

Contrary to ES/Opensearch, in Quickwit, at most 50 terms will be considered when searching the last term of the query as a prefix match_bool_prefix.

Supported Parameters

Multi-match

Elasticsearch reference documentation

Example

{
  "query": {
    "multi_match": {
      "query": "search keywords",
      "fields": [
        "title",
        "body"
      ]
    }
  }
}

{
  "query": {
    "multi_match": {
      "query": "search keywords",
      "type": "most_fields",
      "fields": [
        "title",
        "body"
      ]
    }
  }
}

{
  "query": {
    "multi_match": {
      "query": "search keywords",
      "type": "phrase",
      "fields": [
        "title",
        "body"
      ]
    }
  }
}

{
  "query": {
    "multi_match" : {
      "query":      "search key",
      "type":       "phrase_prefix",
      "fields":     [ "title", "body" ]
    }
  }
}

Supported parameters

Supported types:

:::warning

In phrase, phrase_prefix and bool_prefix modes, Quickwit sums the score of the different fields instead of returning their max.

Moreover, while Quickwit does not support best_fields or cross_fields, it will not return an error when presented a best_fields or cross_fields type. For compatibilility reasons, Quickwit silently accepts these parameters and interprets them as a most_fields type.

:::

term

Elasticsearch reference documentation

:::note

When working on text, it is recommended to only use term queries on fields configured with tokenizer: raw. This is the Quickwit equivalent of the Elasticsearch keyword type.

:::

Example

{
  "query": {
    "term": {
      "payload.commits.message": {
        "value": "automated",
        "boost": 2.0
      }
    }
  }
}

Supported Parameters

match_all / match_none

Elasticsearch reference documentation

Example

{"match_all": {}}

{"match_none": {}}

exists

Elasticsearch reference documentation

Query matching only documents containing a non-null value for a given field.

Example

{
  "query": {
    "exists": {
      "field": "author.login"
    }
  }
}

Supported Parameters

prefix

Elasticsearch reference documentation

Returns documents that contain a specific prefix in a provided field.

Example

{
  "query": {
    "prefix": {
      "author.login" {
        "value": "adm",
      }
    }
  }
}

Supported Parameters

wildcard

Elasticsearch reference documentation

Returns documents that contain terms matching a wildcard pattern:

• ? replaces one and only one term character
• * replaces any number of term characters or an empty string

Example

{
  "query": {
    "wildcard": {
      "author.login" {
        "value": "adm?n*",
      }
    }
  }
}

Supported Parameters

regexp

Elasticsearch reference documentation

Returns documents that contain terms matching a regular expression.

Example

{
  "query": {
    "regexp": {
      "author.login" {
        "value": "adm.*n",
      }
    }
  }
}

Supported Parameters

About the lenient argument

Quickwit and Elasticsearch have different interpretations of the lenient setting:

• In Quickwit, lenient mode allows ignoring parts of the query that reference non-existing columns. This is a behavior that Elasticsearch supports by default.
• In Elasticsearch, lenient mode primarily addresses type errors (such as searching for text in an integer field). Quickwit always supports this behavior, regardless of the lenient setting.

Search multiple indices

Search APIs that accept <index_id> requests path parameter also support multi-target syntax.

Multi-target syntax

In multi-target syntax, you can use a comma or its URL encoded version '%2C' separated list to run a request on multiple indices: test1,test2,test3. You can also sue glob-like wildcard ( * ) expressions to target indices that match a pattern: test* or *test or te*t or *test*.

The multi-target expression has the following constraints:

- It must follow the regex `^[a-zA-Z\*][a-zA-Z0-9-_\.\*]{0,254}$`.
- It cannot contain consecutive asterisks (`*`).
- If it does not contain an asterisk (`*`), the length must be greater than or equal to 3 characters.

Examples

GET api/v1/_elastic/stackoverflow-000001,stackoverflow-000002/_search
{
  "query": {
    "query_string": {
      "query": "search AND engine",
      "fields": [
        "title",
        "body"
      ]
    }
  }
}

GET api/v1/_elastic/stackoverflow*/_search
{
  "query": {
    "query_string": {
      "query": "search AND engine",
      "fields": [
        "title",
        "body"
      ]
    }
  }
}