Back to Pulsar

PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.

pip/pip-444.md

4.2.12.0 KB
Original Source

PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.

Background knowledge

Apache Pulsar uses Apache BookKeeper as its storage layer. Each topic consists of multiple ledgers, which are created periodically based on the configuration. Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a ledger is deleted, the corresponding ZK node will also be deleted. Retention policies determine how long messages are retained in a topic. When the retention period is reached, the corresponding ledgers will be deleted.

Motivation

When the retention of a large topic is reduced, a significant number of ledgers need to be deleted. Since this deletion operation is not rate-limited, it could results in ZooKeeper (ZK) latency of several minutes, which threatens the stability of the entire Pulsar cluster.

Goals

Add rate limit feature for deleting ledgers.

Detailed Design

Introduce a new configuration parameter:

    @FieldContext(
            category = CATEGORY_STORAGE_ML,
            doc = "Max number of concurrent requests for deleting ledgers at broker level"
    )
    private int managedLedgerDeleteMaxConcurrentRequests = 1000;

These parameters can be configured in the broker.conf file.

When the managedLedgerDeleteMaxConcurrentRequests parameter is set to a value greater than 0, the concurrency of ledger deletions will be limited to the specified value.

Backward & Forward Compatibility

Fully backward compatible and forward compatible.

Upgrade

New feature is enabled by default with a rate limit of 1000 deletions per second and concurrency of 1.

Downgrade / Rollback

To downgrade or rollback, simply revert the configuration changes in the broker.conf file and restart the broker.

Links

<!-- Updated afterwards -->