docs/release_notes/v1.18.3.md
This update contains the following bug fixes:
@every job schedulescontinue_as_new iterations sharing a single unbounded traceWith hot reloading enabled, daprd refused to apply any change to a state store component that is, or would become, the actor state store.
Creating, updating, or deleting such a component logged Aborting to hot-reload a state store component that is used as an actor state store and the change was silently ignored until daprd was restarted.
In particular, a daprd which started without an actor state store could never gain working actor hosting or Workflow APIs: adding actorStateStore: "true" to a state store component had no effect, and every workflow operation kept failing with the state store is not configured to use the actor runtime.
You were affected if you relied on hot reloading to manage components and needed to introduce, replace, or remove the actor state store without restarting daprd; for example enabling workflows on a running sidecar by promoting an existing state store to the actor state store.
The restriction was added when the actors subsystem had no support for its state store changing at runtime: the store name was captured once at startup, the decision to enable actor hosting was made once at startup, and nothing reacted to the store later appearing, changing, or disappearing.
The actor state store can now be hot reloaded, and the actors subsystem reconciles hosting with the store at runtime:
the state store is not configured to use the actor runtime until a store is configured again. daprd itself keeps running, and hosting resumes automatically when a store returns, with no app or workflow-worker reconnection required.Two further consistency fixes ride along with this change:
actor hosting disabled log.Periodically, the Placement server closed every sidecar stream in a namespace within the same millisecond, and all sidecars reconnected at once. On reconnect each sidecar halts its local actors, so actors mid-call were interrupted on hosts that were perfectly healthy. The dissemination version restarted from scratch after each occurrence.
You were affected if you ran actor-hosting sidecars, most visibly in namespaces with many sidecars and frequent deployment rollouts. Each sidecar disconnecting while a dissemination round was in flight brought the namespace one step closer to a mass disconnect, so large, frequently rolled clusters could see every sidecar in a namespace dropped every few minutes. Actors mid-invocation on unaffected hosts were interrupted each time.
A single failing Placement stream could report its closure twice: once from its receive loop unwinding, and once from a failed dissemination send on the same connection. The per-namespace connection counter was decremented for both reports, leaving it one below the real number of live connections after each such event. Once the accumulated drift reached zero, the namespace's dissemination loop was shut down while live streams were still attached, closing all of them and resetting the dissemination version.
A stream closure is now reported exactly once: a failed dissemination send cancels the stream's context, preserving the send error as the close cause, and the receive loop unwinding is the single place a closure is reported from. The namespace connection counter now always matches the number of live streams, so a healthy namespace is never torn down by one sidecar's disconnect.
After upgrading to 1.18, all embedded etcd metrics (etcd_*, such as etcd_server_has_leader, etcd_mvcc_db_total_size_in_bytes, and etcd_disk_backend_commit_duration_seconds_bucket) disappeared from the Scheduler's metrics endpoint (default port 9090).
Dapr's own dapr_scheduler_* metrics were unaffected.
You were affected if you scraped the Scheduler metrics endpoint for embedded etcd metrics, for example to populate the embedded etcd panels of the dapr-scheduler Grafana dashboard or to alert on etcd database size and disk commit latency.
Those series stopped being emitted after upgrading from 1.17 to 1.18.
In 1.18 the metrics exporter shared by all Dapr binaries switched from serving the process-global default Prometheus registry to a fresh private registry, to avoid duplicate collector registration panics when the exporter is restarted within the same process. Embedded etcd registers all of its collectors on the global default registry during package initialization, so its metrics were never written into the private registry the endpoint served, and were exposed nowhere.
The metrics endpoint now gathers from both the private registry and the global default registry.
Nothing is ever registered on the default registry by the exporter itself, so in-process restarts remain safe and the original panic cannot reoccur.
All etcd_* metrics are emitted on the Scheduler metrics endpoint again, and other metrics registered on the default registry by linked libraries (such as grpc_server_* and grpc_client_*) are restored to their 1.17 behavior across Dapr binaries.
Two related failure modes in the Scheduler:
A Scheduler restart or crash under load could escalate into a cluster-wide outage window: job and actor reminder delivery stalled until pods were manually restarted or clients reconnected and re-registered. You were affected if you run the Scheduler in HA with actively connected sidecars, most visibly under high job or workflow throughput.
Shutdown used an unbounded gRPC GracefulStop(), which waits for every open stream.
A WatchJobs stream whose client had connected but never sent, or had stopped reading, left its handler blocked in its initial Recv (or a flow-control blocked Send), so the drain never completed and the process never exited.
Readiness was only failed after the drain completed, so health probes never detected the zombie.
Separately, a dead WatchJobs stream emitted duplicate close events (one from its receive loop plus one per failed send), while the per-namespace connection counter was decremented once per event.
Duplicates could drive the count to zero while live streams remained, deleting the namespace and dropping every stream and deliverable job prefix in it.
Failed sends also held their job's completion callback until the stream was fully reaped, so the scheduling engine could not promptly redeliver those jobs to healthy streams.
A data race in the connection event-loop object recycling could additionally corrupt loop state under connection churn.
Scheduler shutdown now fails readiness before draining and bounds the graceful drain to 5 seconds before forcing the gRPC server to stop, guaranteeing process exit. Dead streams are now closed exactly once: the close cancels the stream at detection time, promptly deregisters its deliverable prefixes, and resolves in-flight jobs as undeliverable so the engine immediately redelivers them to healthy streams. Namespace deletion is now confirmed by the connection tracking loop that owns the authoritative stream set, so duplicate or stray close events can no longer tear down a namespace that still has live streams. The event-loop object recycling race was removed.
Scheduling a job whose schedule carried a timezone prefix (TZ= or CRON_TZ=) with no schedule after it, such as TZ=UTC, crashed the Scheduler process instead of returning an error.
Any client permitted to schedule a job could bring down a shared Scheduler with a single malformed schedule string. daprd does not validate the schedule; it forwards it verbatim to the Scheduler, which parses it, and the Scheduler's gRPC server has no panic-recovery interceptor, so the parse panic terminated the process. In a highly available deployment this manifested as a Scheduler pod crash-looping whenever the offending job was (re)loaded.
The cron parser located the boundary between the timezone prefix and the schedule with strings.Index(spec, " ") and sliced the string on the result without checking for -1.
A prefix with no following schedule produced an index of -1, and the resulting out-of-range slice panicked.
The cron parser (from github.com/dapr/kit) now finds the boundary using any whitespace and returns a descriptive error when no schedule follows the timezone prefix.
The malformed schedule is now rejected when the job is scheduled, and the Scheduler no longer crashes.
@every job schedulesA schedule that combined a timezone prefix with an @every interval, such as CRON_TZ=Europe/Rome @every 1h, was accepted but silently dropped the timezone.
The prefix looked like it pinned the job to a timezone, but an @every schedule fires at a fixed interval and has no wall-clock time for a timezone to apply to.
A user who set CRON_TZ=Europe/Rome @every 24h intending "every day at the same local time" would see the fire time drift by an hour across daylight saving transitions — the exact problem the timezone prefix appears to prevent.
An @every schedule parses to a fixed constant-delay schedule that carries no location, so the parsed timezone was discarded rather than applied.
The cron parser (from github.com/dapr/kit) now rejects a timezone prefix on an @every schedule when the job is scheduled, so the misconfiguration surfaces immediately instead of silently producing wrong fire times.
To pin a recurring job to a wall-clock time in a timezone, use a cron expression instead, for example CRON_TZ=Europe/Rome 0 0 9 * * *.
A read-only workflow status query (GetInstance / GetWorkflowMetadata, or the equivalent SDK calls such as GetWorkflowStateAsync) could intermittently fail with a gRPC Unknown error while the queried workflow was running normally:
Status(StatusCode="Unknown", Detail="workflow '<id>': inbox key 'inbox-000000' declared in metadata (inboxLength=1) but missing from state store (transient store read failure or partial save?)")
The failure was transient and self-healing: the next poll for the same instance succeeded, and the workflow itself completed successfully.
You were affected if you polled workflow status while workflows were making progress, most visibly under high activity concurrency (for example a fan-out of many parallel activities polled by several concurrent clients), although a single activity transition could also trigger it. Callers received a terminal-looking error for a healthy workflow, so clients without their own retry logic surfaced spurious failures. The persisted workflow state was never actually inconsistent.
Loading workflow state reads the metadata row and the inbox-*/history-* entry rows in two separate state store calls, and the status query path performs these reads without holding the workflow actor's lock.
A workflow actor save is a single atomic transaction that deletes consumed inbox entries and writes updated metadata, so a save committing between the reader's two calls produced a torn read: the old metadata still declared inbox entries that the second read no longer found.
This torn read was reported as a hard error to the caller.
The workflow state load now detects this mismatch and retries the whole load with freshly read metadata, up to 5 attempts spaced 15 ms apart.
The metadata ETag is compared between attempts: a changed ETag proves a concurrent save landed between the reads (retry), while an unchanged ETag proves the entries are genuinely missing from the store, in which case the original error is still returned.
Status queries racing an active workflow now return consistent results instead of transient Unknown errors.
Before it starts reading from an input binding, daprd asks the application whether it subscribes to that binding: an HTTP OPTIONS request to the binding's route, or a gRPC ListInputBindings call.
This request was given a hardcoded 3 second budget with no way to change it.
An application that had not finished warming up within those 3 seconds never answered in time, so daprd treated the binding as unsubscribed and never activated it.
You were affected if your application is slow to serve its first request after startup — JVM or JIT warmup, large dependency-injection graphs, or resource-constrained nodes — and you declared an input binding without an explicit direction: input metadata entry.
The binding component itself initialized correctly and appeared in the sidecar's metadata endpoint, so the component looked healthy while no events were ever delivered.
On the HTTP channel a failed probe also aborted the remaining bindings, leaving every input binding on that sidecar inactive; the only trace was a failed to read from bindings warning in the sidecar log.
On the gRPC channel the probe failure was silent.
The subscription discovery deadline was hardcoded to 3 seconds in the binding processor and built from a background context. It could neither be tuned for applications with slow startup nor cancelled when the runtime shut down while a probe was still in flight.
The timeout is now configurable through the new daprd --app-binding-options-timeout flag, which applies to both the HTTP OPTIONS probe and the gRPC ListInputBindings probe.
The default remains 3 seconds, so existing deployments are unchanged, and non-positive values fall back to that default.
The probe is now derived from the runtime's context, so it is cancelled promptly on shutdown instead of running to its full deadline.
Setting direction: input on the binding component continues to skip the probe entirely.
Loading an MCPServer resource connects to the MCP server and lists its tools, which is what installs that server's dapr.internal.mcp.<name>.ListTools and .CallTool.<tool> workflows and registers the workflow actor host they run on.
If that connection failed, the failure was final: registration was never re-attempted for the life of the sidecar process.
You were affected if an MCPServer's endpoint was not serving at the moment daprd loaded it, for example an origin still starting up, a proxy or tunnel in front of it returning 502 while its backend came up, or DNS that had not yet propagated.
The resource still appeared in the metadata API, so it looked loaded, but none of its workflows existed. A client that read the metadata and scheduled one of them found no host for the workflow actor type, and workflow creation retries a missing host indefinitely, so the call hung until the caller's context expired instead of failing. With an unbounded context it did not return at all. Recovery required editing the resource to trigger a hot reload, or restarting the sidecar.
Registration was invoked exactly once per resource load with no retry, so a momentary connection failure was indistinguishable from a permanently unreachable endpoint. Both left the server listed in metadata and unusable.
MCPServer registration now runs under Dapr's built-in initialization retry policy, the same one used elsewhere for resource initialization: exponential backoff starting at 500ms, up to 3 retries within a 10 second budget.
An endpoint that becomes reachable inside that window is registered normally and its tools work as expected.
A genuinely unreachable endpoint behaves as before once the retries are exhausted, and ignoreErrors: true continues to keep daprd running.
Delivering a pub/sub message to a subscriber over HTTP crashed the daprd process when the message's CloudEvent carried a non-string traceparent or traceid field, for example a JSON number, boolean, or object instead of a string.
Any client permitted to publish to a subscribed topic could crash the sidecar with a single message.
A publisher controls the CloudEvent trace fields: publishing with content type application/cloudevents+json and a body such as {"specversion":"1.0", ..., "traceid":12345} preserves the non-string value all the way through to delivery.
The HTTP delivery path runs in a background goroutine with no panic recovery, so the failure terminated the whole process rather than dropping the single message.
The gRPC delivery path was not affected.
In a highly available deployment this manifested as sidecars crash-looping whenever the offending message was redelivered.
The HTTP pub/sub delivery path (Deliver and DeliverBulk in pkg/runtime/subscription/postman/http) read the trace field from the CloudEvent and performed an unchecked type assertion to string.
The CloudEvent is deserialized from publisher-controlled bytes into a map[string]any, so a non-string trace field became a float64, bool, or map, and the assertion panicked.
The HTTP delivery path now uses a checked type assertion, matching the gRPC delivery path: a non-string trace field is ignored (tracing is skipped for that message) instead of crashing the process. Messages carrying a malformed trace field are now delivered to the subscriber normally.
continue_as_new iterations sharing a single unbounded traceWhen a workflow restarted itself with continue_as_new, the new iteration inherited the previous iteration's trace context instead of starting a trace of its own.
Every iteration of an eternal workflow therefore joined the trace of the very first iteration, sharing one trace ID indefinitely.
You were affected if you ran eternal or polling workflows built on continue_as_new with distributed tracing enabled (OTLP or Zipkin exporter to any backend, such as Jaeger, Azure Monitor, or Geneva).
A workflow iterating over hours or days produced a single trace accumulating thousands of spans.
Workflows running without tracing enabled were unaffected.
On a continue_as_new completion, the durable task engine built the new iteration's execution-started event by copying the previous iteration's parent trace context verbatim.
Each iteration's orchestration span was therefore parented off the original trace, and the trace grew without bound.
When the prior iteration was traced, each continue_as_new transition now generates a fresh W3C root trace context (new random trace and span IDs) for the new iteration, so every iteration produces its own bounded, independently queryable trace.
Activity spans and spans created inside the application continue to join their own iteration's trace as before.
A workflow that was not being traced stays untraced across the transition.
A workflow that entered the STALLED state (version not available, patch mismatch, or payload size exceeded) could become permanently stuck if the last connected workflow worker disconnected while the workflow was stalled.
Reconnecting workers, including workers registering the exact workflow version the stall was waiting for, did not resume it: status queries kept reporting STALLED and the only recovery was restarting daprd.
When it occurred, the sidecar logged:
error while disconnecting work item stream: failed to deactivate workflow '<id>': actor is stalled
Stalling exists so that a workflow survives its workers going away and resumes once capable workers return, most commonly a rolling upgrade where the old application version disconnects and the new version reconnects.
You were affected if you use workflow versioning, patching, or a configured --max-body-size, and all workflow workers of an application disconnected while a workflow was stalled; for example during application restarts, rolling upgrades, scale-to-zero.
A stalled workflow actor parks its execution in-process, holding the execution reminder in flight until its context is cancelled, and marks its lock as stalled.
When the last workflow worker disconnects, daprd unregisters the workflow actor types and deactivates all workflow actors, but deactivation begins by acquiring the actor's lock, and the lock rejects acquisition while the actor is stalled.
Whether the workflow could later recover came down to a race: if the scheduler's reminder stream teardown cancelled the parked execution before deactivation reached the actor, deactivation succeeded and the unacknowledged reminder was redelivered when workers reconnected.
If deactivation won the race, it failed with actor is stalled, leaving the actor activated and holding the reminder in flight, so reconnecting workers had nothing to redeliver and the workflow never re-executed.
Deactivating a stalled workflow actor now wakes the parked execution instead of failing: the held execution returns immediately, leaving the execution reminder unacknowledged, and deactivation completes.
When workflow workers reconnect, the reminder is redelivered and the workflow re-executes, resuming and completing once the connected workers satisfy the stall condition (for example the required workflow version is registered, or daprd was restarted with a larger --max-body-size).
Recovery from a stall no longer depends on timing, and a daprd restart is no longer required.
When a workflow activity completes but its parent workflow actor is unreachable (for example during placement rebalancing or a host restart), the activity actor durably queues the result as an activity-result reminder on the workflow actor so the outcome is delivered once the workflow is reachable again.
If the workflow instance is purged before that reminder fires, the reminder becomes an orphan: it targets an instance that no longer exists.
The issue impacts users on Dapr 1.18.0-1.18.2 running workflows with activities where instances are purged (explicitly, or via a state retention policy) while activity results are still in flight, particularly across placement churn such as rolling restarts or scale events.
Each orphaned reminder adds a permanent one-invocation-per-second load on the daprd hosting the workflow actor type.
The workflow actor's activity-result reminder handler forwarded the instance-not-found error to the reminder system without classifying it as terminal.
The scheduler treats any error as a failed invocation and applies the reminder's failure policy, which for this reminder type is a constant one-second retry with no retry limit.
The activity-result reminder handler now treats instance-not-found as a successful delivery outcome:
At-least-once delivery for live instances is unchanged as the fix only affects reminders whose target instance has been purged.
Delivering a pub/sub message to an application connected over gRPC intermittently failed before the application ever saw the message, and the sidecar logged:
error returned from app while processing pub/sub event <id>: retriable error occurred: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read tcp 127.0.0.1:56264->127.0.0.1:14208: use of closed network connection"
The application was healthy and listening throughout, and the same message was delivered successfully on a later attempt or after a sidecar restart.
You were affected if you ran an application with --app-protocol grpc (or grpcs) subscribing to topics, and had upgraded to 1.18.
The failures were most visible on subscriptions receiving sporadic traffic, and on hosts where the application is slow to accept a new connection, for example a busy thread pool or a garbage collection pause.
Every affected delivery was reported to the broker as a retriable failure, so the message was either redelivered by the broker (RabbitMQ, Kafka, and other components with redelivery) or dropped for components without it.
The same connection handling is used for service invocation into a gRPC application, input binding delivery, and job triggers, so those paths could fail the same way.
Applications using --app-protocol http were not affected.
daprd dialed the application with a MinConnectTimeout of one second.
In gRPC that value is the budget for an entire connection attempt, covering the TCP connect and the HTTP/2 handshake, not just the TCP connect: when it expires, gRPC hard-closes the socket, and a request already riding on that connection fails with error reading server preface: use of closed network connection.
Up to 1.17 this was almost never reachable, because daprd held a single application connection for the lifetime of the process and only ever dialed at startup. In 1.18 the application connection moved into the same pool used for sidecar-to-sidecar connections, so connections are established on demand while messages are being delivered, and each new connection had one second to complete its handshake before the delivery riding on it failed.
The pool made this far more frequent than it needed to be. The pool is configured to keep one warm connection to the application, but once that connection had been idle for longer than the pool's three minute idle window it was neither handed out again (it was treated as expired) nor closed (it was the connection being kept warm). It was stuck in the pool, unusable and open, so every request arriving after an idle period dialed a brand new connection rather than reusing the warm one.
A connection attempt to the application is now given gRPC's default budget of 20 seconds instead of one second, matching what daprd already used for sidecar-to-sidecar connections. This does not change how long a request waits for an unreachable application: a refused connection still fails immediately, and how long a request waits remains governed by the caller's context and resiliency policy.
Connections the pool holds to satisfy its warm connection minimum are also no longer expired out. The application connection established at startup now stays in use for the life of the sidecar, so a subscription that receives one message an hour no longer re-dials the application for every message.
Terminating a workflow intermittently had no effect: the workflow's status stayed RUNNING and it kept executing activities and timers as if the terminate had never been issued.
Blocking terminate calls (such as TerminateWorkflowAsync in the .NET SDK) never returned, because they wait for the instance to reach a terminal status that never came.
With debug logging enabled, a dropped terminate showed the ExecutionTerminated event being delivered and consumed with no effect:
received work item with 2 new event(s): [ExecutionTerminated, TaskCompleted#1]
workflow execution returned with status 'ORCHESTRATION_STATUS_RUNNING'
A workflow terminated while it was suspended was affected the same way, deterministically: the status stayed SUSPENDED and every terminate sent to it was lost.
You were affected if you terminated workflows that were actively making progress, regardless of SDK language.
The window depends on inbox pressure: the terminate is dropped when it arrives in the same work item batch as another event and is not the last event in that batch, so long-running workflows that loop over short activities and timers (endless pollers, monitors, continue_as_new loops) were the most exposed, while a workflow idling on a single slow activity or timer almost always received its terminate alone and was unaffected.
A dropped terminate is consumed with its batch and never redelivered, so the instance kept running; a retried terminate raced the same window again. A recursive terminate that was dropped also never cascaded, leaving child workflows running as orphans. For suspended workflows there was no window: every terminate was lost until the workflow was resumed.
The workflow engine hands an instance's pending events to the workflow executor as one batch, and relied entirely on the SDK executor to turn an ExecutionTerminated event into a termination outcome.
SDK executors registered the termination when they processed the event but kept feeding the remaining events of the batch into the workflow code, which resumed past its own termination and produced a competing outcome: scheduling more work, completing normally, or restarting via continue_as_new.
A continue_as_new outcome always discarded the termination; other outcomes raced it nondeterministically.
A terminate delivered while the workflow was suspended produced no outcome at all, because suspension suppressed every action the executor would have returned.
Since the engine trusted the executor's result and the event was consumed with the batch, the terminate was lost permanently.
When a delivered batch contains an ExecutionTerminated event and the executor does not complete the workflow, daprd discards the doomed execution's pending work and completes the instance as TERMINATED.
A continue_as_new returned alongside a terminate no longer starts a new iteration, and terminating a suspended workflow now terminates it without requiring a resume.
Azure (Microsoft Entra ID) components authenticate by trying a chain of credentials in order until one succeeds.
When azureClientId and azureTenantId were set, the chain included the SPIFFE workload identity credential, and if no SPIFFE JWT SVID source was available the chain stopped at that step:
ChainedTokenCredential: failed to acquire a token.
Attempted credentials:
ClientAssertionCredential: failed to get JWT SVID source from context
Credentials later in the chain, such as managed identity or the Azure CLI, were never attempted, so the component failed to authenticate even though a working credential was available.
You were affected on Dapr 1.16.0 or later (where the SPIFFE credential joined the default chain) in either of these configurations:
azureClientId and azureTenantId set but no client secret or certificate, relying on a later credential in the default chain (for example managed identity or the Azure CLI).azureAuthMethods list placing spiffeworkloadidentity before another method (for example spiffeworkloadidentity,managedidentity), expecting fallback when SPIFFE is not configured.ChainedTokenCredential only continues past a credential that reports a credentialUnavailableError; any other error is treated as fatal and ends the chain.
The SPIFFE credential returned a plain error when the context carried no JWT SVID source, so a missing prerequisite was treated as a fatal authentication failure rather than a signal to try the next credential.
The SPIFFE credential now reports itself as unavailable when no JWT SVID source is present, before any token request is made, which is exactly what ChainedTokenCredential requires to continue to the next credential in the chain.
Both the default chain and explicitly ordered azureAuthMethods lists now fall through as expected, and behavior when a SPIFFE source is configured is unchanged.