ContextQMD
Back to Redis

Rdi 1 16 0

content/integrate/redis-data-integration/release-notes/rdi-1-16-0.md

latest5.3 KB
Original Source

RDI's mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

  • Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
  • Save resources and time when building pipelines and coding data transformations.
  • Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a Change Data Capture (CDC) mechanism. It also lets you transform the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

What's New in 1.16.0

  • External secrets mount status tracking: The operator tracks and surfaces the status of external secrets mounts, helping you diagnose issues with secrets providers more quickly.
  • Improved metrics exposure for VM installs: RDI annotates ingresses during VM-based installations so that metrics are automatically exposed for monitoring systems.
  • Debezium upgrade: Upgraded Debezium to version 3.3.1, bringing connector fixes and performance improvements from the upstream project.
  • Installer preflight disk space checks: The installer now checks for sufficient disk space and noexe permissions on the /tmp directory before starting the installation, reducing failures due to insufficient capacity.

RDI API Changes

  • Dry-run support in API v2: RDI API v2 endpoints now supports dry-run mode, allowing you to validate pipeline configuration and connectivity without applying changes. They also now return validation errors instead of generic failures when the request body is missing or invalid.
  • Expanded API v2 observability and metadata: RDI API v2 now exposes additional information and metrics:
    • New /info endpoint that returns the RDI version. This endpoint does not require authentication and can be used to check RDI availability.
    • New /metric-collections endpoints that describe available metric collections.
    • Pipeline /api/v2/pipelines/{name} and status /api/v2/pipelines/{name}/status responses now include component-level information.
    • Metric collection selection is now based on component type instead of name.
  • API correctness and compatibility improvements: Ensured backwards compatibility between API v1 and v2 and fixed issues where API v2 source/target dry-run endpoints could return HTTP 500 for certain invalid requests.
  • Environment variable handling for monitoring statistics: Fixed an issue where /api/v1/monitoring/statistics could return HTTP 500 when the target database port was specified as an environment variable.
  • Enhanced Spanner disaster recovery with Redis Active-Active: When using Cloud Spanner as a CDC source with a Redis Active-Active database, the Flink collector now handles long-running workloads and failover scenarios more reliably, improving the overall ability to handle disaster recovery scenarios.
  • Spanner client resource management: Spanner clients are now reused per database instead of being created repeatedly, eliminating a memory leak and reducing connection churn on Spanner.
  • Flink collector memory tuning: Increased the default process and heap size for the Flink collector and tuned memory fractions for better performance and stability.
  • Resilient Redis operations by the Flink collector: The Flink collector now uses configurable connection and socket timeouts, as well as retries with exponential backoff for Redis operations, for improved resilience. It also supports waiting for a configurable number of replicas when writing for improved durability, and handles Redis out-of-memory errors as backpressure instead of failures.

Bug Fixes and Stability Improvements

  • Improved support package generation: Fixed several issues in the support package dump process so that diagnostic bundles are generated reliably, including rotated logs. Added a new --log-days option to specify the number of days of logs to collect.
  • Fixed log level configuration: The redis-di configure command now correctly applies the rdi-log-level setting.
  • Enhanced processor error handling: The processor now fails with a clear error when it cannot connect to the target database after a timeout and writes such errors to the pod termination log to expose them.
  • Improved pipeline error detection: Pipelines now transition to an Error status when deployments are not available after 2 minutes, exposing failures sooner.
  • Fixed empty password handling: Resolved issues with RDI commands when using empty passwords.
  • Operator namespace isolation: The RDI operator now only reconciles resources within the RDI namespace, improving performance and isolation. It also no longer requires cluster-level permissions for any resources, reducing security concerns for enterprise Kubernetes deployments.

Limitations

RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.