ContextQMD
Back to Rxdb

RxDB 13.0.0 - A New Era of Replication

docs-src/docs/releases/13.0.0.md

17.1.07.7 KB
Original Source

13.0.0

So in the last major RxDB versions, the focus was set to improvements of the storage engine. This is done. RxDB has now multiple RxStorage implementations, a better query planner and an improved test suite to ensure everything works correct. This let to huge improvements in write and query performance and decreased the initial pageload of RxDB based applications.

In the new major version 13.0.0, the focus was set to improvements to the replication protocol. When I first implemented the GraphQL replication a few years ago, I had a specific use case in mind and designed the whole protocol and replication plugins around that use case.

But the time has shown, that the old replication protocol is a big downside of RxDB:

  • The replication relied on the backend to solve all conflicts. This was easy to implement into RxDB because the whole responsibility was given away to the person that has to implement a compatible backend.
  • In each point in time, the replication did either push or pull documents, but never in parallel. This slows done the whole replication process and makes RxDB not usable for the implementation of features like multi-user-real-time-collaboration or when many read- and write operations have to happen in a short timespan.
  • After each push, a pull had to be run to check if the backend had changed the state to solve a conflict.
  • The replication protocol did not support attachments and was not designed to ever support them.

So in version 13.0.0 I replaced the whole replication plugins with a new replication protocol. The main goals have been:

  • Push- and Pull in parallel.
  • Use the data in the changestream (optional) to decrease replication latency.
  • Implement the conflict resolution into RxDB so that the client resolves its own conflicts and does not rely on the backend.
  • Decrease the complexity for a compatible backend implementation. The new protocol relies on a dumb backend. This will open compatibility with many other use cases like implementing Offline-First in Supabase or using CouchDB but having a faster replication compared to the native CouchDB replication.
  • Make it possible to use CRDTs instead of a conflict resolution.
  • Design a the protocol in a way to make it possible to add attachments replication in the future.

On the RxDocument level, the replication works like git, where the fork/client contains all new writes and must be merged with the master/server before it can push its new state to the master/server.

A---B1---C1---X    master/server state
     \       /
      B1---C2      fork/client state

For more details, read the documentation about the new RxDB Sync Engine.

Backends that have been compatible with the previous RxDB versions 12 and older, will not work with the new replication protocol. To learn how to do that, either read the docs or check out the GraphQL example.

Other breaking changes

  • RENAMED the ajv-validate plugin to validate-ajv to be in equal with the other validation plugins.

  • The is-my-json-valid validation is no longer supported until this bug is fixed.

  • REFACTORED the schema validation plugins, they are no longer plugins but now they get wrapped around any other RxStorage.

    • It allows us to run the validation inside of a Worker RxStorage instead of running it in the main JavaScript process.
    • It allows us to configure which RxDatabase instance must use the validation and which does not. In production it often makes sense to validate user data, but you might not need the validation for data that is only replicated from the backend.

  • REFACTORED the key compression plugin, it is no longer a plugin but now a wrapper around any other RxStorage.

    • It allows to run the key-compression inside of a Worker RxStorage instead of running it in the main JavaScript process.

  • REFACTORED the encryption plugin, it is no longer a plugin but now a wrapper around any other RxStorage.

    • It allows to run the encryption inside of a Worker RxStorage instead of running it in the main JavaScript process.
    • It allows do use asynchronous crypto function like WebCrypto

  • Store the password hash in the same write request as the database token to improve performance.

  • REMOVED support for temporary documents see here

  • REMOVED RxDatabase.broadcastChannel The broadcast channel has been moved out of the RxDatabase and is part of the RxStorage. So it is no longer exposed via RxDatabase.broadcastChannel.

  • Removed the liveInterval option of the replication plugins. It was an edge case feature with wrong defaults. If you want to run the pull replication on interval, you can send a RESYNC event manually in a loop.

  • REPLACED RxReplicationPullError and RxReplicationPushError with normal RxError like in the rest of the RxDB code.

  • REMOVED the option to filter out replication documents with the push/pull modifiers #2552 because this does not work with the new replication protocol.

  • CHANGE default of replication live to be set to true. Because most people want to do a live replication, not a one time replication.

  • RENAMED the server plugin is now called server-couchdb and RxDatabase.server() is now RxDatabase.serverCouchDB()

  • CHANGED Attachment data is now always handled as Blob because Node.js does support Blob since version 18.0.0 so we no longer have to use a Buffer but instead can use Blob for browsers and Node.js

  • REFACTORED the layout of RxChangeEvent to better match the RxDB requirements and to fix the 'deleted-document-is-modified-but-still-deleted' bug.

When used with Node.js, RxDB now requires Node.js version 18.0.0 or higher.

Other non breaking or internal changes

  • REMOVED many unused plugin hooks because they decreased the performance.

  • REMOVE RxStorageStatics .hash and .hashKey

  • CHANGE removed default usage of md5 as default hashing. Use a faster non-cryptographic hash instead.

    • ADD option to pass a custom hash function when calling createRxDatabase.

  • CHANGE use Float instead of Int to represent timestamps in GraphQL.

  • FIXED multiple problems with encoding attachments data. We now use the js-base64 library which properly handles utf-8/binary/ascii transformations.

  • In the RxDB internal _meta.lwt field, we now use 2 decimals number of the unix timestamp in milliseconds.

  • ADDED checkpointSchema to the RxStorage.statics interface.

New Features

Migration to the new version

Stored data of the previous RxDB versions is not compatible with RxDB 13.0.0. So if you want to keep that data, you have to migrate it in a way. For most use cases you might want to just drop the data from the client and re-sync it again from the backend. To keep the data locally, you might want to use the storage migration plugin.