ContextQMD
Back to Redis

Rdi 1 8 0

content/integrate/redis-data-integration/release-notes/rdi-1-8-0.md

latest5.6 KB
Original Source

{{< note >}}This minor release replaces the 1.6.7 release.{{< /note >}}

RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to:

  • Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
  • Save resources and time when building pipelines and coding data transformations.
  • Reduce the total cost of ownership by saving money on expensive database read replicas.

RDI keeps the Redis cache up to date with changes in the primary database, using a Change Data Capture (CDC) mechanism. It also lets you transform the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required.

Headlines

  • Enhanced RDI operator for better pipeline orchestration, resiliency, observability, and flexibility. It will also enable many new features in the near future.
  • You can now use an external collector that is not managed by RDI but writes into RDI streams (Debezium compatible).
  • You can now add labels and annotations to RDI data plane pods, for example to control service mesh features.
  • RDI now uses a custom image of Debezium (based on 3.0.8.Final) to address known vulnerabilities.
  • Added support for calculated TTL for target database keys via expire expressions.

Detailed changes

Helm chart changes

  • All collector and processor values that were previously under collector, collectorSourceMetricsExporter, and processor have been moved to operator.dataPlane.collector and operator.dataPlane.processor.
  • global.collectorApiEnabled has been moved to operator.dataPlane.collectorApi.enabled, and is now a boolean value (true or false), not "0" or "1".
  • api.authEnabled is also now a boolean value, not "0" or "1".
  • The following values have been deprecated: rdiMetricsExporter.service.protocol, rdiMetricsExporter.service.port, rdiMetricsExporter.serviceMonitor.path, api.service.name
  • You can now add custom labels and annotations to all RDI components.
  • You can now disable the creation of the RDI system secrets.

Operator Improvements

The RDI operator has been significantly enhanced in the following areas:

  • Resilience: The operator now always maintains the desired pipeline state. Manual changes or random disruptions are reverted automatically.
  • Automatic recovery: When a configuration issue is resolved, the entire pipeline starts automatically, eliminating the need for manual redeployment.
  • Consistency: A pipeline that has been stopped with stop will remain stopped after deploy or reset, until explicitly started again.
  • Enhanced configuration: You can now configure data plane components in ways that were previously not supported, such as adding labels and annotations.
  • External collector support: No collector resources are created for sources of type external.
  • Enhanced troubleshooting: You can now gain extra insight into the pipeline state by examining the Pipeline and PipelineRelease custom K8s resources.

Other Features, Improvements and Enhancements

  • Added expire expression for target output in transformation jobs.
  • Addressed security vulnerabilities: TLS certificate hostname verification is now ON by default.
  • Improved Helm default values while preserving values.yaml formatting.
  • Enhanced Helm values and templates for better configuration.
  • Added a script to create or update secrets when using Helm (rdi-secret.sh in the Helm zip file).
  • Improved validation schema and ensured backward compatibility.
  • Fixed compatibility issues with newer versions of requests and urllib3.
  • Improved error messages for JSON schema validation.
  • Improved PostgreSQL documentation for mTLS.
  • Added timestamps to the status command.
  • Fixed issues with primary_key and unique_constraint attributes in Oracle metadata.
  • Added capture.mode to MongoDB scaffolding.
  • Improved Helm TLS setup for RDI database connections.
  • Enhanced error handling and validation for transformation jobs.
  • Improved documentation for supported platforms and configurations.

Fixes

  • Fixed HTTP 500 error when querying columns with tables parameter.
  • Improved Helm TLS setup for RDI database connections.
  • Fixed keystore overwrite when using mTLS on both source and RDI DBs in the collector.

Limitations

RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.

Known Issues

High Availability upgrade issue

When upgrading from RDI < 1.8.0 to RDI >= 1.8.0 in an HA setup, both RDI instances may incorrectly consider themselves active after the upgrade. This occurs because the upgrade process doesn't update the rdi:ha:lock value from the legacy cluster-1 identifier, causing both clusters to assume they are the active cluster.

Symptoms:

  • The upgraded passive node will start collector and processor components
  • Collector may enter a crash loop as it fails to connect to the source
  • Both clusters will restart in a loop

Workaround:

After upgrading, manually set a unique cluster ID for one of the installations by editing the configmap:

bash
kubectl edit cm -n rdi rdi-sys-config

Then add the following line to distinguish between the clusters:

bash
RDI_CLUSTER_ID: cluster-2