pip/pip-463.md
Apache Pulsar currently uses Maven as its build system. The project has grown to over 100 modules with complex dependency relationships, shaded JARs, NAR packaging, and Docker image builds. Maven's sequential execution model and limited caching capabilities result in long build times that impact developer productivity and CI throughput.
Gradle is a modern build system used by large-scale Java projects (e.g., Spring Boot, Micronaut, Apache Kafka). It provides parallel task execution, fine-grained caching, and incremental compilation out of the box.
The current Maven build has several pain points that affect developer velocity and CI efficiency:
Slow local builds. A full mvn install -DskipTests takes 5-8 minutes on a modern machine.
Developers frequently wait for unrelated modules to rebuild when iterating on a single component.
Maven has no built-in mechanism to skip unchanged modules — it rebuilds everything in the reactor.
Slow CI. The CI pipeline takes 50-60 minutes end-to-end. Maven's lack of caching means each CI run starts from scratch. Test jobs must either rebuild everything or rely on fragile artifact-sharing workarounds.
Imprecise dependency tracking. Maven treats the entire module as the unit of rebuild. Changing a test resource file triggers a full recompile of the module. There is no way to run "only the tests affected by my change" — developers must run the entire test suite for a module or manually specify test classes.
Limited parallelism. Maven's -T flag enables module-level parallelism, but tasks within
a module still run sequentially. The Pulsar build has several bottleneck modules (e.g.,
pulsar-broker) where compilation, resource processing, and test execution could overlap
with other modules but don't.
Complex shading and packaging. The project uses Maven Shade plugin, NAR plugin, and
custom Ant tasks for packaging. These configurations are verbose, hard to maintain, and
have subtle interactions (e.g., the ahc-default.properties content replacement for
AsyncHttpClient requires an Ant <replace> task in Maven but is a single filesMatching
call in Gradle).
Poor IDE integration for multi-module builds. IntelliJ IDEA's Maven import for a project of Pulsar's size is slow and memory-intensive. Gradle's tooling API provides faster, more reliable IDE synchronization.
1:1 functional equivalence with Maven. The Gradle build produces identical artifacts:
apache-pulsar-X.Y.Z-bin.tar.gz) with the same JARspulsar, pulsar-all, java-test-image, pulsar-test-latest-version)pulsar-client, pulsar-client-admin, pulsar-client-all)
verified to contain the same classes and relocations as Maven outputAll CI tests passing. Unit tests, integration tests, system tests, shade tests (Java 17/21/24), and backward compatibility tests all pass on the Gradle build.
Enforced dependency management. A pulsar-dependencies platform module (Gradle's equivalent
of Maven's dependencyManagement) ensures consistent dependency versions across all modules.
Version catalog. A single gradle/libs.versions.toml file defines all dependency coordinates
and versions, replacing scattered version properties across 100+ POM files.
CI workflow migration. All GitHub Actions workflows converted from Maven to Gradle commands.
The migration introduces Gradle build scripts alongside (and eventually replacing) the existing Maven POM files. The approach is:
build.gradle.kts, settings.gradle.kts,
gradle/libs.versions.toml)pom.xml, mvnw, .mvn/)The Gradle build is structured as:
settings.gradle.kts # Module includes and plugin repositories
build.gradle.kts # Root build: common config, enforced platform
gradle/libs.versions.toml # Version catalog (single source of truth for versions)
pulsar-dependencies/ # Enforced platform module (replaces dependencyManagement)
<module>/build.gradle.kts # Per-module build script
Key design decisions:
filesMatching for
property file content relocationio.github.merlimat.nar) for connector packagingpulsar-dependencies) for version pinning across all modules| Aspect | Maven | Gradle |
|---|---|---|
| Incremental compilation | No | Yes — only recompiles changed files |
| Task-level caching | No | Yes — skips tasks whose inputs haven't changed |
| Parallel execution | Module-level only (-T) | Task-level (automatic dependency graph) |
| Configuration caching | No | Yes — reuses build configuration across runs |
| Local build cache | No | Yes — persists across builds |
| Remote build cache | No | Yes — shared across CI and developers (future) |
Expected impact:
./gradlew :pulsar-broker:test builds only
the broker and its dependencies, skipping unrelated modules entirelyGradle provides native integration with Develocity
(formerly Gradle Enterprise), hosted by the ASF at develocity.apache.org. Every CI
build automatically publishes a build scan that provides:
Example build scan from the PoC CI run: https://develocity.apache.org/s/h6ckzn3nn4w2s
This level of observability is not available with the Maven build today.
Maven's dependencyManagement in the root POM is replaced by:
Version catalog (gradle/libs.versions.toml): Defines all dependency coordinates
and version numbers in one file. Modules reference dependencies as libs.netty.buffer
instead of hardcoded group:artifact:version strings.
Enforced platform (pulsar-dependencies): A java-platform module that creates
version constraints from the catalog. Applied to all subprojects via
implementation(enforcedPlatform(project(":pulsar-dependencies"))). This ensures
transitive dependencies are pinned to the same versions Maven would resolve.
The Shadow plugin replaces Maven Shade. Key differences handled:
<replace> to fix property name prefixes
in ahc-default.properties. Gradle uses filesMatching { filter { } }.dependencies { include/exclude } DSL replaces
Maven Shade's <includes>/<excludes>.relocate() is functionally identical to Maven Shade's.A custom NAR Gradle plugin (io.github.merlimat.nar) handles connector packaging.
Global exclusions for platform modules (provided by java-instance.jar at runtime)
are configured in the root build.gradle.kts.
Some modules require version overrides that differ from the enforced platform:
kinesis-kpl-shaded: Forces protobuf-java:4.29.0 (KPL requires protobuf 4.x,
while Pulsar uses 3.x). The protobuf is relocated so no runtime conflict.jclouds-shaded: Forces Guice 7.0.0, jakarta.annotation-api:3.0.0,
jakarta.ws.rs-api:3.1.0, jakarta.inject-api:2.0.1 (jclouds 2.6.0 requires
Jakarta EE 10+ APIs). All are bundled in the shadow JAR.No new broker/client configuration options. The build system change is transparent to users.
mvn commands replaced by ./gradlew commands in documentation and scriptssrc/set-project-version.sh updated to modify gradle/libs.versions.tomlArtifacts are functionally identical. Minor differences:
package-info.class files (no runtime impact)No security implications. The build system change does not affect Pulsar's runtime security model, authentication, or authorization.
The Gradle wrapper (gradlew) is committed to the repository with a checksum-verified
distribution URL, following the same security model as the Maven wrapper.
The implementation PR demonstrates full CI green status across all test suites, confirming functional equivalence with the Maven build.