Rdi 1 15 0

RDI's mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

• Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
• Save resources and time when building pipelines and coding data transformations.
• Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a Change Data Capture (CDC) mechanism. It also lets you transform the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

What's New in 1.15.0

{{<warning>}} Breaking change when using JSON with json_update_strategy: merge

RDI now uses the native JSON.MERGE command instead of Lua scripts for JSON merge operations. While this provides significant performance improvements (2x faster), there is a functional difference in how null values are handled:

• Previous behavior (Lua merge): When merging {"field1": "value1", "field2": "value2"} with {"field2": null, "field3": "value3"}, the result was {"field1": "value1", "field2": null, "field3": "value3"} (null value is preserved)
• New behavior (JSON.MERGE): The same merge produces {"field1": "value1", "field3": "value3"} (null value removes the field, following RFC 7396)

Impact: If your application logic distinguishes between a field with a null value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation.

Configuration: You can control this behavior using the use_native_json_merge property in the processors section of your configuration. Set it to false to revert to the previous Lua-based merge behavior if needed. {{</warning>}}

• Native JSON merge for improved performance: RDI now automatically uses the native JSON.MERGE command from RedisJSON 2.6.0+ instead of Lua scripts for JSON merge operations, providing 2x performance improvement. This feature is enabled by default and can be controlled via the use_native_json_merge property in the processors section of the configuration. Note: If the target Redis database has RedisJSON version lower than 2.6.0, the processor will automatically revert to using the Lua-based merge implementation.

• Support for sharded Redis databases: RDI now supports writing to multi-sharded Redis Enterprise databases for the RDI database, resolving cross-slot violations when reading from streams.

• Enhanced processor performance metrics: Detailed performance metrics are now exposed through the metrics exporter and statistics endpoint, with separate tracking for transformation time and write time.

• Resource management improvements: Collector and processor pods now support configurable resource requests, limits, and node affinity/tolerations for better cluster resource utilization.

  • The collector defaults to 1 CPU and 1024Mi memory (requests), with limits of 4 CPUs and 4096Mi memory.
  • The processor defaults to 1 CPU and 512Mi memory (requests), with limits of 4 CPUs and 3072Mi memory.

• Leadership status monitoring: New metrics expose leadership status and pipeline phase information for better monitoring of HA deployments.

  • The rdi_operator_is_leader metric tracks the current leadership status of the operator: 1 indicates the instance is the leader, 0 indicates it is not the leader.
  • The rdi_operator_pipeline_phase metric tracks the current phase of the pipeline. Phase indicates the current pipeline phase, must be one of Active, Inactive, Resetting, Pending, or Error.

• Improved configuration validation: More rigid validation for config.yaml and jobs.yaml files helps catch configuration errors earlier in the deployment process.

• Custom K3s installation options: The installer now supports passing custom arguments to K3s installation for more flexible on-premises deployments.

  • Example: sudo INSTALL_K3S_EXEC='--write-kubeconfig-mode=644' ./install.sh

• Workload Identity authentication: Added support for Google Cloud Workload Identity authentication to Google Cloud Storage (GCS), eliminating the need for service account JSON files if using GSC-based leader election.

• Exposed new processor performance metrics showing transformation and write times separately

  • {namespace}_processor_process_time_ms_total - Total time spent in the processor (transform + write)
  • {namespace}_processor_transform_time_ms_total - Time spent transforming data
  • {namespace}_processor_write_time_ms_total - Time spent writing data to Redis

• Exposed all existing processor metrics that were only available through the RDI CLI status and the API statistics endpoint.

• Enhanced statistics endpoint with new metrics for transform and process time

Bug Fixes and Stability Improvements

• Fixed task reconciliation errors: Resolved "The ID argument cannot be a complete ID because xadd-id-uniqueness-mode is strict" errors during task reconciliation when using an RDI Redis database with strict XADD id uniqueness mode.
• Fixed Debezium unavailable values: Addressed issues where __debezium_unavailable_value was appearing in Redis data.
• Improved operator stability: Disabled operator webhooks by default to simplify deployments and reduce potential issues.

Limitations

RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.