ContextQMD
Back to Pulsar

PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before.

pip/pip-373.md

4.2.13.8 KB
Original Source

PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before.

Background knowledge

Transaction Buffer

  • It maintains aborted messages indexes and ongoing TXN messages indexes in memory, we call them Two Indexes in the following sections.
  • It helps to filter out the messages that have been aborted or are still binding with an on-going TXN, to avoid consuming aborted messages or TXN in-progress messages.

Transaction Buffer initializes when a topic is loading up

  • It reads messages that contain TXN state from the original topic, to re-build Two Indexes into memory.

Transaction Buffer closing when a topic is unloading

  • It takes a snapshot for Two Indexes to rebuild the states faster next loading up.

Motivation

  • Since TXN is a feature that across multiple namespaces, it can be enabled or disabled at the cluster level.
  • Transaction Buffer will try to re-build Two Indexes for every topic if you enable TXN.
  • If you have a huge number of topics in a namespace, the task that re-build Two Indexes costs huge resources(CPU, Memory).

We'd better skip Transaction Buffer re-building Two Indexes if the topic it is related to does not contain TXN messages to save resources usage.

Goals

Skip Transaction Buffer re-building Two Indexes if the topic it is related to does not contain TXN messages to save resources usage.

In Scope

This PIP only focuses on improving the scenario in which users have never published TXN messages on a topic.

Out of Scope

This PIP does not focus on the scenario that follows.

  • Published TXN messages.
  • Consumed all TXN messages that were sent before.
  • Transaction Buffer keeps re-building Two Indexes even if there are no TXN messages in the topic anymore, and it costs resources.

Detailed Design

Design & Implementation Details

  • Add a topic-level system property named __contains_txn_messages, the default value is false, and it will be set to true when the first TXN message is publishing.
  • Transaction Buffer skips re-building Two Indexes if the property is false.

Public-facing Changes

The topic property __contains_txn_messages becomes to a system property, it can never be used by users anymore.

Public API

  • You will get a NotAllowedException when you try to set/remove a topic property named __contains_txn_messages by the API pulsar-admin topics update-properties/remove-properties.
  • The value of the property __contains_txn_messages can be queried by pulsar-admin topics get-properties.

Metrics

NameDescriptionAttributesUnits
pulsar_broker_using_txn_topics_countCounter. The number of topics contains TXN messages.cluster-

Backward & Forward Compatibility

Upgrade

There are 3 scenarios that the topic's property __contains_txn_messages is when users try to upgrade.

  • __contains_txn_messages is empty: broker initializes it by confirming whether there is TXN messages or not.
  • (Highlight) __contains_txn_messages is not empty and is not typed boolean, rollback to the original behavior that always re-building Two Indexes, but you can never modify it anymore.
  • (Highlight) __contains_txn_messages is not empty and is typed boolean, but it is a users' property, broker assumed that it is a system property.

Downgrade / Rollback

You can downgrade or roll back gracefully.

Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations

The PIP does not affect Geo-Replication.

Links