Dapr 1.10.5

This patch release contains fixes for 6 bugs.

• Fixed large goroutine leak when connection to placement service fails
• Fixed a number of race conditions when saving actor reminders
• Fixed Dapr keeping only one HTTP header with the same name during HTTP service invocation
• Fixed actors not working when Dapr sidecars are configured with app-ssl due to failing health checks
• Fixed delete operations failing in three state store components when using a custom table name
• Fixed five state store components not allowing saving values that are empty strings

Fixed large goroutine leak when connection to placement service fails

Problem

A regression introduced in Dapr 1.10.0 caused a severe goroutine leak in sidecars when the connection to the placement service failed, causing Dapr to use very large amounts of CPU and memory, as well as creating many TCP connections (which could manifest in issues such as port exhaustion).

Impact

The issue impacts all users on Dapr 1.10.0-1.10.4.

Root cause

A change introduced in Dapr 1.10.0 in the client that Dapr sidecars use to connect to the Placement service to improperly handle connection errors. Because of that, new gRPC clients were created on very short intervals and never released, eventually causing background goroutines to leak and consume very large amounts of CPU and memory.

Solution

We have corrected the error handling code, so gRPC clients and goroutines are not leaked in case of network errors while connecting to the Placement service.

Fixed a number of race conditions when saving actor reminders

Problem

When saving actor reminders, we identified the possibility race conditions that could cause, in the worst scenarios, for reminders to be persisted in an inconsistent way, when multiple Dapr sidecars were trying to create or update reminders at the same time.

Impact

The issue impacts users on Dapr 1.10.4 and earlier that use actor reminders.

Root cause

We identified two possible situations that could cause race conditions when saving (creating, updating, or deleting) actor reminders:

• When multiple actor reminders are saved in parallel (by the same Dapr sidecar or multiple ones), the write operations saving the metadata document and each modified data partition were not happening in the same transaction, possibly causing the state persisted in the state store to be inconsistent.
• When multiple reminders are created in parallel, users may have experienced errors such as "possible etag mismatch" after enabling Resiliency; this was due to not correctly honoring context cancellation.

Solution

We addressed the race conditions and made operations that save reminders and their metadata document happen in the same transaction, guaranteeing consistency of the data. This and other small improvements included in this change should also allow users to notice slightly improved efficiency when performing operations on reminders.

Fixed Dapr keeping only one HTTP header with the same name during HTTP service invocation

Problem

Although the HTTP protocol specs allow passing more than one header with the same name (e.g. multiple Set-Cookie headers in responses), Dapr was only maintaining one header of each kind while performing HTTP service invocation.

Impact

This issue impacts users on Dapr 1.10.4 and earlier that use HTTP service invocation.

Root cause

While processing headers in HTTP service invocation calls, Dapr was only persisting one value for each header type–the last one. This behavior was inconsistent with the specs of the HTTP protocol.

Solution

We changed the code in Dapr to allow keeping all HTTP headers, even when they have the same name, to be aligned with the specs of the HTTP protocol. We've updated our end-to-end tests to prevent future regressions.

Fixed actors not working when Dapr sidecars are configured with app-ssl due to failing health checks

Problem

The Dapr actors runtime includes built-in checks to ensure that the app is in a healthy state and thus capable of running actors' code. A bug prevented the health checks from working when Dapr sidecars were configured with app-ssl, i.e. when the app's HTTP server was using HTTPS.

Impact

The issue impacts users on Dapr 1.10.4 and earlier that use actors and use HTTPS for their app's server.

Root cause

The issue was caused by the actors' health check subsystem not being configured to use HTTPS when app-ssl is enabled for the Dapr sidecar.

Solution

We fixed the actors' health check subsystem to use HTTPS when app-ssl is enabled, just like for the HTTP(S) app channel (used by service invocation, pubsub messages, etc).

As part of the process, we also migrated the health check subsystem from the fasthttp library to the Go's standard library, sharing the same HTTP client with the HTTP(S) channel. This also allows for re-using the same TCP sockets if keep-alives are enabled, and includes support HTTP/2 (with multiplexing) if the app's server supports that.

Fixed delete operations failing in three state store components when using a custom table name

Problem

We identified a regression in Dapr 1.10.0 causing delete operations to fail in three state store components when the components were initialized with a custom table name:

• Cockroach DB
• Oracle Database
• PostgreSQL

Impact

The issue impacts users on Dapr 1.10.0-1.10.4 that use the impacted state store components.

Root cause

The default table name in "DELETE" queries was mistakenly hardcoded in the three impacted state store components. Because of that, users would not have been able to perform delete operations when the components were configured with a custom table name

Solution

We fixed the impacted components to make sure that the table names are never hardcoded in any query sent to the databases.

Fixed five state store components not allowing saving values that are empty strings

Problem

Five state store components were returning an error while trying to save a value that was an empty string. This made them unusable as state stores for Dapr Workflow. The impacted components were:

• Cockroach DB
• Oracle
• MySQL
• PostgreSQL
• SQLite

Impact

The issue impacts users on Dapr 1.10.4 and earlier that use the state stores listed above.

Root cause

The five listed state store components included checks that returned an error if the application was trying to save an empty string as value. Although the behavior was added intentionally at the time (because otherwise there'd be no way in get operations to distinguish an empty value from a missing key), it was inconsistently implemented across all components and caused issues such as not being able to use those components to store Dapr Workflow state.

Solution

We removed the checks that forbid storing values that are empty strings, bringing the listed state stores in line with the other components. Our conformance tests have been updated to enforce this behavior on all state store components.

In the next version of Dapr we are planning to add a new property in the response object from get operations to inform the application if the value was not found, allowing it to distinguish between empty values and inexistent keys.