docs/operations/nats-jetstream.md
The guide covers enabling and operating the JetStream publish path for the NATS notify and audit targets in RustFS. It is written for operators who need at-least-once delivery of events to NATS, must pre-provision and size the stream the targets publish to, and need to diagnose delivery failures.
By default the NATS notify and audit targets publish with NATS Core, which returns once the message is written to the socket. A process or server failure in the gap between the socket write and the server persisting the message loses the event. The JetStream path closes that gap. With it enabled, each event is published to a JetStream stream and the local store-and-forward queue entry clears only after the server returns a publish acknowledgement, which means the stream leader has accepted and sequenced the message, or after a terminal rejection has been recorded in the failed-events store. A server restart mid-flight loses nothing, because an unacknowledged event stays in the local queue and replays after the server returns.
The path is opt-in and off by default. With it off, behaviour is the NATS Core path. It applies to both the notify NATS target and the audit NATS target.
RustFS gets each event to the server without losing it before the acknowledgement. The operator owns the stream and its policy. RustFS validates that the stream exists and is writable and reports a validation failure otherwise. A store-backed target keeps running through a failed validation, holding queued events until the stream is repaired. RustFS never creates the stream. Stream retention, storage, and replication policy stay under operator control.
Three keys turn the path on and tune it. Each is available on both the notify NATS target and the audit NATS target, and each has a configuration-key form and an environment-variable form per target.
RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE=true
RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME=RUSTFS_EVENTS
RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS=30
RUSTFS_NOTIFY_NATS_QUEUE_DIR=/var/lib/rustfs/notify-nats
The audit target takes the same keys under the RUSTFS_AUDIT_NATS_ prefix.
A configuration that enables the path without a stream name or without a queue directory is rejected at startup and in the admin validation path. An acknowledgement timeout outside the 10 to 120 second range is rejected the same way.
The 30 second acknowledgement timeout suits production. A server replicating to multiple replicas acknowledges only after the replication and the deferred fsync, which legitimately takes well over 100 milliseconds at the tail, so a short timeout would spuriously fail a publish mid-replication.
After enabling, confirm the target is on the JetStream path before relying on it. Value errors, an out-of-range acknowledgement timeout or a missing stream name while enable is on, are already rejected loudly at startup and in the admin validation path. A mistyped configuration key name is not a value error, so it escapes that check: the key reads as absent, and the target stays on the NATS Core path with no durability. Verify the enable took effect by confirming the target reports its JetStream fields on startup and logs the stream-validation success line naming the configured stream. Absence of that line for a target that was meant to be enabled indicates a key name the server did not recognise.
An event is queued to the local store-and-forward queue first, then published through JetStream. The publish carries a stable Nats-Msg-Id header and the path awaits the real publish acknowledgement.
For durability across a node failure, provision the stream with a replica count of at least 3. An acknowledgement returns after the stream leader commits. A leader that acknowledges and then fails before the message replicates to a quorum can lose that message on failover, so a single-replica stream is durable only against a clean restart, not against the loss of the node holding the data.
The stream is provisioned by the operator before the path is enabled. RustFS reads the stream once at target init and in the admin validation path, asserts the requirements below, and reports a validation failure rather than publishing into a stream where writes would silently fail. A failed validation does not stop a store-backed target. Events keep queueing to the local store. Revalidation runs while the validation verdict is unset and stops once one validation passes, resuming only after the verdict is reset, and the queued events deliver once the stream is repaired. The stream is not auto-created. A missing stream is an error.
The stream must:
Choose retention, storage type, and replica count to match the durability the deployment needs. Use file storage if a server restart must preserve already-persisted events. These are operator decisions and RustFS does not set them. If the stream is provisioned shortly after the path is enabled, the queued events deliver on the next replay rather than being lost, because a stream-not-found error is retryable and keeps the events on the live queue until the stream exists.
The acknowledgement timeout and the stream duplicate window are linked. The duplicate window must cover the worst-case retry span, or a late retry of an event the server already persisted is delivered a second time instead of being recognised as a duplicate.
The worst-case retry span is the retry count times the acknowledgement timeout, plus the realized backoff sleeps across the retries and a final headroom term:
duplicate_window >= (retry_count * ack_timeout_secs) + realized_backoff + headroom
There are 5 retry attempts. The realized backoff sleeps sum to 60 seconds (4 + 8 + 16 + 32), and no sleep follows the last attempt, so a 64 second headroom term is added above the realized span. At the default 30 second acknowledgement timeout the required window is:
(5 * 30) + 60 + 64 = 274 seconds
Set the stream duplicate window to at least 274 seconds when the acknowledgement timeout is at the 30 second default. A configuration whose duplicate window is below this span is rejected at validation, with the configured and required window named in the log line.
The span scales with the acknowledgement timeout. Raising jetstream_ack_timeout_secs requires raising the stream duplicate window in step using the formula above. The 60 second realized backoff and the 64 second headroom stay fixed, so each extra second of acknowledgement timeout adds 5 seconds to the required window. At the 120 second maximum timeout the required window is (5 * 120) + 60 + 64 = 724 seconds. Each publish attempt, including connection establishment, is bounded by a single deadline equal to the acknowledgement timeout. The formula is a conservative upper bound on the retry span, and the final headroom term holds the required window above the realized span.
The validated window covers one retry cycle: the worst-case span from the first publish attempt of a stored entry to its last within a single replay cycle. An entry that exhausts a cycle without delivering stays on the live queue and is retried on a later cycle, so an entry surviving across cycles can spend longer in the queue than the duplicate window and be delivered again, consistent with at-least-once delivery.
An event that cannot be delivered is recorded, not silently dropped. Only one case produces a failed event:
Every other rejection is retryable and never produces a failed event. A timeout, backpressure, no responders, a missing stream, a subject the stream does not capture, or an authorization failure keeps the event on the live queue and retries it until it delivers or an operator intervenes. A wrong subject and wrong credentials surface through the validation and connection path as retryable, so they hold the events on the live queue rather than moving them to the failed store.
A failed event is moved to an on-disk failed store that sits in a failed child directory inside the queue directory, one per target. The directory is created lazily on the first failed write, so a target that never fails terminally leaves no failed directory. The entry preserves the event body, its routing metadata, the deduplication identifier, an error class tag of terminal, the failure time, and the retry count. An error-level log line is written for each failed event, naming the bucket, object, event name, and the error, so a broken integration is visible rather than hidden. A record in the failed store is diagnostic only and is never republished, so a condition repaired later, for example a raised broker payload limit, delivers only events still on the live queue. The NATS Core path without JetStream instead retries such an event until the limit allows it.
The failed store is bounded by count (10000 entries per target) and by age (a 72 hour retention), and is kept separate from the live queue limit so an accumulation of failures cannot crowd out new events. The count is maintained as a cached value, seeded at startup and reconciled to the directory on each maintenance interval, so it stays accurate without a directory scan on the hot path. Failed-store writes and the maintenance scan run under one exclusive guard, so the at-bound check and the write stay atomic and the bound holds against concurrent writers. A change made to the directory outside the store can drift the cached count until the next maintenance interval reconciles it. When the count bound is reached the oldest failed entry is dropped, with a warning naming the trimmed entry, so a newer failure is never lost in favour of an older one. Entries past the retention bound are removed as expired on the replay maintenance tick.
Because a retryable failure keeps events on the live queue, a long outage grows the queue toward its configured queue_limit bound. Once the queue reaches that bound, new events are rejected at ingest with a logged error rather than overwriting queued events, so the backlog is bounded and visible.
Size queue_limit for the longest outage the deployment must survive without rejecting new events. Multiply the peak event rate in events per second by the outage window in seconds. A target that averages 50 events per second and must ride out a one hour broker outage needs a queue_limit of at least 50 * 3600 = 180000 entries. Add headroom above the calculated figure, and provision the queue directory storage for the resulting entry count.
Every publish outcome falls into one of two families. Transient conditions are retried until they deliver and the entry stays on the live queue, so a transient condition never reaches the failed store. Permanent conditions move to the failed store immediately. Transient conditions classify toward retry because a terminal misclassification risks losing an entry once the failed store retention lapses. Permanent conditions move immediately so a poison message cannot block the queue.
Both families cover publish outcomes on an established connection. A failure to establish the connection at all, a refused connection or unreadable TLS material, is handled before either family applies: the entry retries with backoff and stays queued until the connection recovers. A publish-level failure on an established connection, including a connection that drops during the stream-validation lookup, is retried on the live queue when it is transient and moved to the failed store only when it is a permanent rejection.
| Outcome family | Examples | Handling |
|---|---|---|
| Connectivity | connection lost mid-publish, broken pipe | Retried until delivered, live queue |
| Timeouts | no acknowledgement within the timeout, attempt deadline reached | Retried until delivered, live queue |
| Cluster in transition | no leader elected, peer membership changing | Retried until delivered, live queue |
| Stream offline | stream or JetStream subsystem temporarily offline, stream not found | Retried until delivered, live queue |
| Resource and quota exhaustion | insufficient server resources, storage, memory, or account quota reached | Retried until delivered, live queue |
| Server errors | any rejection reporting a 5xx status, with or without a specific error code | Retried until delivered, live queue |
| Permanent rejections | payload too large, wrong expected sequence or message identifier, sealed stream | Failed store immediately |
A sealed stream is classified at three points, with three outcomes. Startup validation and the admin validation path reject a sealed stream before the target serves traffic, so init fails and no event is queued against it. A stream sealed while the target runs is caught by the next validation on the publish path, which classifies it retryable and keeps the entry on the live queue while validation keeps failing. A publish that reaches an already-cached validation pass and is then rejected with the sealed-stream code is terminal and moves to the failed store immediately. The permanent-rejections row lists that last outcome.
The failed_store_length gauge reports the number of entries in the failed-events store per target, next to the existing failed_messages and queue_length gauges, on both the notify and audit metric paths. A rising failed_store_length points to terminal rejections accumulating for a target. An exhaustion warning marks each cycle where an entry spends its full retry budget without delivering, after which the entry stays queued and retries on the next scan. The warning repeats once per retry cycle while the entry stays queued, so a persistent delivery problem is visible at warn level without per-attempt noise. The failed_messages count advances only on terminal and dropped events, never on a retry or an exhaustion, so it counts entries that left the queue for good rather than entries still retrying.
Set jetstream_enable off. The target reverts to the NATS Core path. Events already in the queue are delivered by the standard replay. No failed-store entries are created while the path is off.