documentation/ingestion/clients/java.md
import Tabs from "@theme/Tabs"
import TabItem from "@theme/TabItem"
import CodeBlock from "@theme/CodeBlock"
import { RemoteRepoExample } from "@theme/RemoteRepoExample"
:::note
This is the reference for the QuestDB Java Client when QuestDB is used as a server.
For embedded QuestDB, please check our Java Embedded Guide.
:::
The QuestDB Java client is distributed as a separate Maven artifact
(org.questdb:questdb-client).
The client provides the following benefits:
:::info
This page focuses on our high-performance ingestion client, which is optimized for writing data to QuestDB. For retrieving data, we recommend using a PostgreSQL-compatible Java library or our HTTP query endpoint.
:::
Add the QuestDB Java client as a dependency in your project's build configuration file.
<Tabs defaultValue="maven" values={[ { label: "Maven", value: "maven" }, { label: "Gradle", value: "gradle" }, ]}
<TabItem value="maven">
<InterpolateJavaClientVersion renderText={(release) => (
<CodeBlock className="language-xml">
{`<dependency>
<groupId>org.questdb</groupId>
<artifactId>questdb-client</artifactId>
<version>${release.name}</version>
</dependency>} </CodeBlock> )} /> </TabItem> <TabItem value="gradle"> <InterpolateJavaClientVersion renderText={(release) => ( <CodeBlock className="language-text"> {implementation 'org.questdb:questdb-client:${release.name}'`}
</CodeBlock>
)} />
</TabItem>
</Tabs>
The code below creates a client instance configured to use HTTP transport to connect to a QuestDB server running on localhost, port 9000. It then sends two rows, each containing one symbol and two floating-point values. The client asks the server to assign a timestamp to each row based on the server's wall-clock time.
<RemoteRepoExample name="ilp-http" lang="java" header={false} />The client is configured using a configuration string. See Ways to create the client for all configuration methods, and Configuration options for available settings.
This sample configures the client to use HTTP transport with TLS enabled for a connection to a QuestDB server. It also instructs the client to authenticate using HTTP Basic Authentication.
When using QuestDB Enterprise, you can authenticate using a REST bearer token as well. Please check the RBAC docs for more info.
<RemoteRepoExample name="ilp-http-auth" lang="java" header={false} />There are three ways to create a client instance:
From a configuration string. This is the most common way to create a client instance. It describes the entire client configuration in a single string, and allows sharing the same configuration across clients in different languages. The general format is:
<protocol>::<key>=<value>;<key>=<value>;...;
Transport protocol can be one of these:
http — ILP/HTTPhttps — ILP/HTTP with TLS encryptiontcp — ILP/TCPtcps — ILP/TCP with TLS encryptionThe key addr sets the hostname and port of the QuestDB server. Port
defaults to 9000 for HTTP(S) and 9009 for TCP(S). The minimum configuration
includes the transport and the address.
try (Sender sender = Sender.fromConfig("http::addr=localhost:9000;auto_flush_rows=5000;retry_timeout=10000;")) {
// ...
}
For all available options, see Configuration options.
From an environment variable. The QDB_CLIENT_CONF environment variable
is used to set the configuration string. Moving configuration parameters to
an environment variable allows you to avoid hard-coding sensitive information
such as tokens and passwords in your code.
export QDB_CLIENT_CONF="http::addr=localhost:9000;auto_flush_rows=5000;retry_timeout=10000;"
try (Sender sender = Sender.fromEnv()) {
// ...
}
Using the Java builder API. This provides type-safe configuration.
try (Sender sender = Sender.builder(Sender.Transport.HTTP)
.address("localhost:9000")
.autoFlushRows(5000)
.retryTimeoutMillis(10000)
.build()) {
// ...
}
:::note
This feature requires QuestDB OSS 9.1.0+ or Enterprise 3.0.4+.
:::
The ILP client can be configured with multiple possible endpoints to send your data to. Only one endpoint is used at a time.
To configure this feature, simply provide multiple addr entries. For example:
try (Sender sender = Sender.fromConfig("http::addr=localhost:9000;addr=localhost:9999;")) {
// ...
}
On initialisation, if protocol_version=auto, the sender will identify the first instance that is writeable. Then it will stick to this instance and write
any subsequent data to it.
In the event that the instance becomes unavailable for writes, the client will retry the other possible endpoints, and when it finds a new writeable instance, will stick to it instead. This unavailability is characterised by failures to connect or locate the instance, or the instance returning an error code due to it being read-only.
By configuring multiple addresses, you can continue to capture data if your primary instance fails, without having to reconfigure the clients. This backup instance can be hot or cold, and so long as it is assigned a known address, it will be written to as soon as it is started.
Enterprise users can leverage this feature to transparently handle replication failover, without the need to introduce a load-balancer or reconfigure clients.
:::tip
You may wish to increase the value of retry_timeout if you expect your backup instance to take a large amount of time to become writeable.
For example, when performing a primary migration (Enterprise replication), with default settings, you might want to increase this
to 30s or higher.
:::
Create a client instance via Sender.fromConfig().
Use table(CharSequence) to select a table for inserting a new row.
Use symbol(CharSequence, CharSequence) to add all symbols. You must add
symbols before adding other column types.
Use the following options to add all the remaining columns:
stringColumn(CharSequence, CharSequence)longColumn(CharSequence, long)doubleColumn(CharSequence, double)boolColumn(CharSequence, boolean)arrayColumn() -- several variants, see belowtimestampColumn(CharSequence, Instant), or
timestampColumn(CharSequence, long, ChronoUnit)decimalColumn(CharSequence, Decimal256) or
decimalColumn(CharSequence, CharSequence) (string literal):::caution Decimal values require QuestDB version 9.2.0 or later.
Create decimal columns ahead of time with DECIMAL(precision, scale) so QuestDB can ingest the values
with the expected precision. See the
decimal data type page for a refresher on
precision and scale.
:::
at(Instant) or at(long timestamp, ChronoUnit unit) or atNow() to
set a designated timestamp.flush() to send locally buffered data into a
server.close() to dispose the Sender after you no longer need it.To ingest a 1D or 2D array, simply construct a Java array of the appropriate
type (double[], double[][]) and supply it to the arrayColumn() method. In
order to avoid GC overheads, create the array instance once, and then populate
it with the data of each row.
For arrays of higher dimensionality, use the DoubleArray class. Here's a basic
example for a 3D array:
// or "tcp::addr=localhost:9009;protocol_version=2;"
try (Sender sender = Sender.fromConfig("http::addr=localhost:9000;");
DoubleArray ary = new DoubleArray(3, 3, 3);
) {
for (int i = 0; i < ROW_COUNT; i++) {
for (int value = 0; value < 3 * 3 * 3; value++) {
ary.append(value);
}
sender.table("tango")
.doubleArray("array", ary)
.at(getTimestamp(), ChronoUnit.MICROS);
}
}
The ary.append(value) method allows you to populate the array in the row-major
order, without having to compute every coordinate individually. You can also use
ary.set(value, coords...) to set a value at specific coordinates.
:::note Arrays are supported from QuestDB version 9.0.0, and require updated client libraries. :::
The client accumulates the data into an internal buffer and doesn't immediately send it to the server. It can flush the buffer to the server either automatically or on explicit request.
You can configure the client to not use automatic flushing, and issue explicit
flush requests by calling sender.flush():
try (Sender sender = Sender.fromConfig("http::addr=localhost:9000;auto_flush=off")) {
sender.table("trades")
.symbol("symbol", "ETH-USD")
.symbol("side", "sell")
.doubleColumn("price", 2615.54)
.doubleColumn("amount", 0.00044)
.atNow();
sender.table("trades")
.symbol("symbol", "BTC-USD")
.symbol("side", "sell")
.doubleColumn("price", 39269.98)
.doubleColumn("amount", 0.001)
.atNow();
sender.flush();
}
:::note
Calling sender.flush() will flush the buffer even with auto-flushing enabled,
but this isn't a typical way to use the client.
:::
By default, the client automatically flushes the buffer according to a simple policy. With HTTP, it will automatically flush at the time you append a new row, if either of these has become true:
Both parameters can be customized in order to achieve a good tradeoff between throughput (large batches) and latency (small batches).
This configuration string will cause the client to auto-flush every 10 rows or every 10 seconds, whichever comes first:
http::addr=localhost:9000;auto_flush_rows=10;auto_flush_interval=10000;
With TCP, the client flushes its internal buffer whenever it gets full.
The client will also flush automatically when it is being closed and there's still some data in the buffer. However, if the network operation fails at this time, the client won't retry it. Always explicitly flush the buffer before closing the client.
HTTP automatically retries failed, recoverable requests: network errors, some server errors, and timeouts. Non-recoverable errors include invalid data, authentication errors, and other client-side errors.
:::note
If you have configured multiple addresses, retries will be run against different instances.
:::
Retrying is especially useful during transient network issues or when the server
goes offline for a short period. Configure the retrying behavior through the
retry_timeout configuration option or via the builder API with
retryTimeoutMillis(long timeoutMillis). The client continues to retry after
recoverable errors until it either succeeds or the specified timeout expires. If
it hits the timeout without success, the client throws a LineSenderException.
The client won't retry requests while it's being closed and attempting to flush the data left over in the buffer.
The TCP transport has no mechanism to notify the client it encountered an
error; instead it just disconnects. When the client detects this, it throws a
LineSenderException and becomes unusable.
With HTTP transport, the client always prepares a full row in RAM before trying to send it. It also remains usable after an exception has occurred. This allows you to cancel sending a row, for example due to a validation error, and go on with the next row.
With TCP transport, you don't have this option. If you get an exception, you can't continue with the same client instance, and don't have insight into which rows were accepted by the server.
:::caution
Error handling behaviour changed with the release of QuestDB 9.1.0.
Previously, failing all retries would cause an exception and release the buffered data.
Now the buffer will not be released. If you wish to re-use the same sender with fresh data, you must call the
new reset() function.
:::
The concept of designated timestamp is important when ingesting data into QuestDB.
There are two ways to assign a designated timestamp to a row:
User-assigned timestamp: the client assigns a specific timestamp to the row.
java.time.Instant timestamp = Instant.now(); // or any other timestamp
sender.table("trades")
.symbol("symbol", "ETH-USD")
.symbol("side", "sell")
.doubleColumn("price", 2615.54)
.doubleColumn("amount", 0.00044)
.at(timestamp);
The Instant class is part of the java.time package and is used to
represent a specific moment in time. The sender.at() method can accept a
long timestamp representing the elapsed time since the beginning of the
Unix epoch, as well as a
ChronoUnit to specify the time unit. This approach is useful in
high-throughput scenarios where instantiating an Instant object for each
row is not feasible due to performance considerations.
Server-assigned timestamp: the server automatically assigns a timestamp to the row based on the server's wall-clock time at the time of ingesting the row. Example:
sender.table("trades")
.symbol("symbol", "ETH-USD")
.symbol("side", "sell")
.doubleColumn("price", 2615.54)
.doubleColumn("amount", 0.00044)
.atNow();
We recommend using the event's original timestamp when ingesting data into QuestDB. Using ingestion-time timestamps precludes the ability to deduplicate rows, which is important for exactly-once processing.
:::note
QuestDB works best when you send data in chronological order (sorted by timestamp).
:::
To enhance data ingestion performance, QuestDB version 9.0.0 introduced an upgraded version "2" to the text-based InfluxDB Line Protocol which encodes arrays and f64 values in binary form. Arrays are supported only in this upgraded protocol version.
You can select the protocol version with the protocol_version setting in the
configuration string.
HTTP transport automatically negotiates the protocol version by default.
In order to avoid the slight latency cost at connection time, you can explicitly
configure the protocol version by setting protocol_version=2|1;.
TCP transport does not negotiate the protocol version and uses version 1 by
default. You must explicitly set protocol_version=2; in order to ingest
arrays, as in this example:
tcp::addr=localhost:9009;protocol_version=2;
Client can be configured either by using a configuration string as shown in the examples above, or by using the builder API.
The builder API is available via the Sender.builder(Transport transport)
method.
For a breakdown of available options, see the Configuration string page.
The client relies on some JDK internal libraries, which certain specialised JDK offerings may not support.
Here is a list of known incompatible JDKs:
flush() can be called to force sending the internal buffer to a
server, even when the buffer is not full yet.java.lang.AutoCloseable interface, and therefore the
try-with-resource
pattern can be used to ensure that the Sender is closed.