pip/pip-444.md
Apache Pulsar uses Apache BookKeeper as its storage layer. Each topic consists of multiple ledgers, which are created periodically based on the configuration. Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a ledger is deleted, the corresponding ZK node will also be deleted. Retention policies determine how long messages are retained in a topic. When the retention period is reached, the corresponding ledgers will be deleted.
When the retention of a large topic is reduced, a significant number of ledgers need to be deleted. Since this deletion operation is not rate-limited, it could results in ZooKeeper (ZK) latency of several minutes, which threatens the stability of the entire Pulsar cluster.
Add rate limit feature for deleting ledgers.
Introduce a new configuration parameter:
@FieldContext(
category = CATEGORY_STORAGE_ML,
doc = "Max number of concurrent requests for deleting ledgers at broker level"
)
private int managedLedgerDeleteMaxConcurrentRequests = 1000;
These parameters can be configured in the broker.conf file.
When the managedLedgerDeleteMaxConcurrentRequests parameter is set to a value greater than 0,
the concurrency of ledger deletions will be limited to the specified value.
Fully backward compatible and forward compatible.
New feature is enabled by default with a rate limit of 1000 deletions per second and concurrency of 1.
To downgrade or rollback, simply revert the configuration changes in the broker.conf file and restart the broker.