PIP-367: Propose a Contributor Repository for Pulsar

Background knowledge

There are now many non-core, controversial features being added to Pulsar. The demand for these features does exist, but continuously adding too many features to Pulsar can make it more bloated and blur the semantics of some functionalities provided by Pulsar.

Here are some examples I encountered while participating in the community.

Some are questioned for potentially affecting the semantics of Pulsar, but they are ultimately added to Pulsar after thorough discussions and proper processes;

Pulsar is being used by many legacy systems and changing its semantics for specific usecases without considering consequences are creating a lot of pain and incompatibility problems for other existing systems and let's avoid doing it as we are struggling with such changes and breaking compatibility or changing semantics are just not acceptable. [0]

As for PIP-321 getting in without a consensus, I was one who had concerns with it (and still think poorly of it), but I don't think it was decided in violation to the rules. [1]

Some, after discussion, were abstracted and simplified, leaving the implementation details to the users. [2]

Additionally, when contributors proposed to contribute more features to the Pulsar Client, they encountered some skepticism. Although these concerns were addressed after discussions, they also reflected the ongoing dissatisfaction with the bloated API of Pulsar.

IMO the surface area the client already exposes is way too big. It feels any person that wanted to scratch their own itch just added their feature without holding back. This needs to be maintained. In OTel they have a contrib repo where people can contribute their plugin . In your case it's contributing a listener implementation. [3]

The suggestion of the OpenTelemetry contrib repository as a final proposal might be a good solution to this problem. It can prevent too many new features from being added to Pulsar, which would make Pulsar's semantics vague, giving it a kind of hodgepodge feeling. At the same time, it can also enrich the ecosystem of Pulsar. Only vital and critical features will be included in the main repository of Pulsar, while other personalized requirements should be placed in contributor repositories, similar to the OpenTelemetry contributor repository approach. [4]

Motivation

Enrich the Pulsar ecosystem by adding a contrib repository for maintaining non-core requirements.

The main differences between Pulsar's main repository and contributor repositories typically include the following points:

Main Repository

Core Components: The main repository houses the core functionalities and critical components of the Pulsar project, which form the foundation and are essential for all users.
Stability: The code within the main repository is usually subjected to rigorous testing and review to ensure its stability and reliability.
Maintenance: The main repository is maintained and updated by the project's core maintenance team.
Integration: The code and features from the main repository are typically integrated into Pulsar's official release versions.
Documentation: The main repository is complemented by comprehensive documentation and examples on the official website, aiding users in understanding and utilizing the project.

Contributor Repository

In contrast, contributor repositories might include:

Experimental Features: These could be in the early stages of development and not yet mature enough to be incorporated into the main repository.
Non-Core Features: Features that are useful for some users but not necessary for all, thus not fitting into the main repository.
Optimization for Specific Scenarios: Tailored optimizations for particular use cases or environments, which may not be universally applicable.
Community-Driven: Typically maintained by community members, these repositories may encompass a variety of perspectives and implementation approaches.
Diversity: Contributor repositories can offer multiple implementations, allowing users to select based on their specific needs.
Risk: The use of code from contributor repositories might carry higher risks due to potentially less stringent testing compared to the main repository.
Flexibility: These repositories allow for faster iteration and experimentation, not being bound by the release cycle and stability requirements of the main repository.
Gradual Integration: Mature features from contributor repositories are allowed to be gradually integrated into the main repository. This usually necessitates a series of reviews and tests to ensure they meet the quality standards of the main repository.
Community Participation: Encourages community members to contribute and expresses gratitude to all those who have contributed.
Documentation: Focuses on maintaining and showcasing newly introduced features to enhance transparency and encourage community involvement and contribution. The documentation will exclusively contain information on new features, including feature descriptions, development progress, and a list of contributors, to acknowledge their contributions to the project.

Goals

In Scope

Requirement Assessment:

Discuss the necessity of establishing a contributor repository and analyze its potential benefits to project development and community engagement.

Project Affiliation:

Evaluate and decide whether to make this repository an official subproject of Apache Pulsar for better management and promotion.

Process Design:

If the community decides to establish a contributor repository, we will collaboratively design and determine the contribution process to ensure it is efficient and easy for new contributors to follow.

Maintenance Strategy:

Develop maintenance processes and strategies, clarifying the responsibilities of maintainers, code review standards, and quality assurance measures.

Out of Scope

Core Feature Decisions: Deciding which features are core and which are non-core contributions is typically determined by project maintainers and community leaders based on the project roadmap and vision, and is not within the direct discussion scope of the contributor repository.
Feature Migration: Migrating non-core or semantically inconsistent features of Pulsar out of the core repository involves in-depth analysis and decision-making of the existing codebase, usually handled by the core maintenance team.

High Level Design

Provide a non-core code maintenance repository to collect plugin implementations, personalized features, experimental features, and user best practices.

This repository will not be released
After iteration, mature features or plugins will be gradually merged into the main branch.

Detailed Design

Design & Implementation Details

The contributor repository is different from Pulsar's main contribution process. The contributor repository should take a more flexible approach to handle the integration of new features:

Custom interface implementation:

For newly added custom interface implementations, if they do not cause functional conflicts, they can be directly merged into the main branch after completing the code review.

Potentially conflicting features:

For new features or optimizations that may conflict with other features, we recommend developing them in a private repository and maintaining a curated list in the contributor repository.

Branch review and merge:

Maintainers and community committers review the code and decide whether to allow the plugin to be merged into the main branch or add experimental features to the curated list. After multiple iterations, widely used and mature features can be tried to be merged into the Pulsar main repository.

User-defined integration:

Users can select the required functions according to the description in the function document and integrate them into their own fork for customized release.

Version release

Contributor repositories will not be released, users need to release them by themselves.

For specific repository design, please refer to: https://github.com/StevenLuMT/pulsar-java-contrib/blob/stable/README.md?plain=1

Alternatives

PIP-367: Propose a Contributor Repository for Pulsar

PIP-367: Propose a Contributor Repository for Pulsar

Background knowledge

Motivation

Goals

In Scope

Requirement Assessment:

Project Affiliation:

Process Design:

Maintenance Strategy:

Out of Scope

High Level Design

Detailed Design

Design & Implementation Details

Alternatives

General Notes

Links