Packages / Internal Dependency Management

Summary

The size of the Kibana repository has surpassed almost all other Typescript projects on Github, and the Javascript tooling ecosystem is inadequate. AppEx Operations team has done a lot of work over the years to close these gaps and keep up with our codebase's growth. Still, significant steps are necessary to provide a more efficient development experience. The AppEx Operations team is leading an effort to migrate to Bazel, which among other things, will provide remote caching and incremental builds. This initiative should drastically improve the productivity of Kibana contributors by minimizing what needs to be built, both locally and in CI, and providing faster and more thorough feedback.

Goals

• Use packages as a core unit of code used throughout the repository
• Packages have well-defined boundaries, a single responsibility, and are easily reusable and shareable across the repository when desired
• Support organizing the repository by domain
• Allow developers and CI to benefit from a remote cache and incremental builds for all development/validation/build tasks

Definitions

These are some of the terms we are using to describe parts of this initiative:

Build Tasks/Tasks : We refer to any task/command that is executed as part of development or CI as a "build task" or just "task".

Package : Packages can be installed from NPM, or implemented in the repository. Local packages are structured in a specific way detailed in the Package Structure section.

Incremental tasks : The ability to execute the minimal set of build tasks necessary by inspecting the files which have changed and utilizing local+remote caches.

Package Structure

Every package has:

• a globally unique module ID in the @kbn/ namespace
• a type, see Package Types
• dependencies on other packages in the repo or from NPM declared in their kibana.jsonc file and updated automatically
• configuration files like jest.config.js, package.json, etc. which are automatically generated when necessary from the kibana.jsonc file.
• a single interface. When you import a package you get its one (and only) interface. Packages don't have sub-imports or sub-modules.
• the ability to include jest tests alongside the code
• an optional devOnly: true | false property. This flag allows excluding development packages (tooling and processes) from the production builds.

Package Types

Package types allow us to have many different packages, pre-defined build tasks, restrictions, and unique configurations for each package type. Every package is one of the following types:

shared-common : These packages can be imported from all other packages.

shared-browser : These packages can be imported from shared-browser and public directories within plugin packages. shared-browser packages may include Storybooks.

shared-server : These packages can be imported from shared-server and server directories within plugin packages.

shared-scss : These packages can be imported by shared-browser and public directories within plugin packages, and expose an index.scss file to consumers instead of an index.ts file.

plugin : These packages were automatically created from the existing plugins at the time we switched everything over to packages. Module IDs must end with -plugin. Consumers must use import type statements.

functional-test : These packages expose one or more functional testing configurations, including API integration tests, and can not be imported by other packages. Separating functional and integration tests allows us to iterate on tests without rebuilding the application. Similarly, iterating and updating the application should mostly mean the tests don't need to rebuild.

There is currently a proposal to add a new test-helpers package type, and we expect more package types to be defined in the future if and when the need arises.

Phases

We're planning to implement the full package system in phases. Currently, those phases look like this:

Phase 1: Ground preparation

status: ✅ complete

This phase is about identifying issues that would prevent a migration for teams to provide adequate time for them before they need to migrate.

• Migrate all plugins to use module IDs
• Find instances where ESLint is disabled and illegal cross-boundary imports are used; create issues for teams to address
• Prevent naked eslint-disable
• Prevent disabling specific ESLint rules
• Windows development is not supported
• Rewrite @kbn/pm using native Node.js to remove its build step. Anything outside of bootstrap, like clean and reset commands, will move to different packages.
• Document how to break up packages into smaller units of code

Phase 2: Legacy packages migration

status: in progress

This phase is about migrating the existing legacy packages to one of the package types.

• Add kibana.json files to existing packages
• Auto-generate configuration files based on kibana.jsonc manifests
• Automate package dependency discovery

Phase 3: Double down on DX

This phase is all about making it easier for teams to start breaking their plugin up into packages.

• Rebuilding packages when running the Kibana Development CLI is automatic
• Pause requests when packages are being re-built
• Ensure the server restarts when files in a shared-server package are rebuilt
• Package linting to validate rules
• Start with a package rule to validate dependencies are all used and included
• Document how to create, build and watch packages.

Phase 4: Plugins everywhere

This phase is all about supporting the creation of plugin-browser and plugin-server packages anywhere in the repository.

• Extracting Webpack config from @kbn/optimizer to work on a single bundle inside Bazel
• Update plugin discovery to find and differentiate between legacy plugins and plugin-browser/server packages
• ESLint rules to validate imports into and out of plugin-browser and plugin-server packages
• Finish the migration of core to packages and reflect with the core team (and possibly the Shared UX team) on what we should recommend for plugin authors before we start to migrate legacy plugins to the package system
• Documentation, documentation, documentation
• How do 3rd party plugins migrate to the new system?

Phase 5: Legacy plugins migration

This phase is all about having the solution teams migrate their legacy plugins into packages.

• Identify the order of plugins that can be migrated
• Identify and communicate what needs to be done by teams
• Provide migration consultations
• Deprecate legacy 3rd party plugin styles and communicate 18 month migration period

Phase 6: Cleanup

This phase is about finalizing the rough edges and making sure every piece of code is on Bazel.

• No code lives outside a Bazel package
• Extend the package development tooling to support 3rd party package development and allow packages to participate in the benefits of Bazel within the plugins directory.
• Ability for 3rd party packages to require specific versions of specific packages from NPM
• Automatically build the components that need changes
• Build package artifacts that can be installed in a Kibana distributable

Sustainable Kibana Architecture

Introduction

The tools and processes around the Kibana repository (linting, building, TS compiler, testing, CI, ...) are put under a lot of load, and execution times are becoming more and more of a bottleneck. The Sustainable Kibana Architecture initative aims at breaking down the Kibana monolith, creating independent builds and CI pipelines for each solution.

Modules belong to groups

To that end, the different modules (packages and plugins) have been categorised into one of the following groups:

group: 'platform' : These modules are used by more than one solution, and thus they belong to the platform layer, which is built / tested / loaded systematically. Platform modules cannot depend on solution modules.

group: 'observability' | 'security' | 'search' | 'workplaceai' : These modules belong to a particular solution, and their code/services are private (aka they cannot be accessed from platform or from other solutions). This allows building / testing / loading them in the context of the solution they belong to.

Platform modules have visibility

Some of the platform code contains logic and services that we do not want solutions to access / consume. To that end, the kibana.jsonc files support the visibility attribute:

visibility: 'private' : Services and logic in these modules (packages and plugins) can only be accessed from modules within the same group. This allows e.g. abstracting away some implementations and keeping some internal logic and services private.

visibility: 'shared' : Services and logic in these modules (packages and plugins) can be accessed from modules from any group. This configuration can only be applied to platform modules, as solution modules are private by definition.

Sustainable folder structure

To make module categories clear and reinforce our commitment to splitting the Kibana monolith into platform and solution components, we've restructured the Kibana repository as follows:

src/platform/[plugins|packages]/[private|shared]/... : SSPL licensed platform modules that are used at test-time (mocks) and/or runtime.

x-pack/platform/[plugins|packages]/[private|shared]/... : Elastic licensed platform modules that are used at test-time (mocks) and/or runtime.

x-pack/[solution]/[plugins|packages]/... : Solution-specific modules.

src/platform/test and x-pack/platform/test : Platform testing modules.

src/core/packages/... : Low-level platform packages that contain Kibana's essential building blocks.

packages/... : Mainly devOnly: true packages and CLI tools that are only used at development / CI time.

FAQ

Is it time for me to start creating packages?

Probably not. The Shared UX and Core teams are currently our "Guinea Pig" teams and they're experiencing the pain of living on the bleeding edge. If you want to create a single package you are welcome to, but for now, it's probably best that you wait until Operations reached out to your team.

How do circular dependencies work?

By breaking the repository into packages we can't support cross-package circular dependencies.

For example, imagine trying to build the types for @kbn/a that depend on the types for @kbn/b. If @kbn/b also depends on the types for @kbn/a there's a circular dependency meaning there isn't a way to build the types for either.

If you cause a circular dependency in the task graph, Bazel will produce a pretty great error message explaining where the cycle is.

The solution to resolving a circular dependency has, thus far, been to break out the component causing it, into a separate package. Packages are lightweight and are very easy to create (work in progress) so please feel comfortable creating more packages as you need to.

How do we name packages?

There are a few package naming rules:

• all packages must use the @kbn/ namespace
• plugin-type packages must end with -plugin
• considering that we operate in a global namespace, avoid overly generic names

Other than these rules, it's up to you and your team to decide on an appropriate name for your package.

The shared-ux team makes a lot of packages, each containing a single, widely shared component along with helpers and types, for other packages to consume. The shared-ux team has used the following naming scheme:

/{domain}/{componentName}
  impl/  :: `@kbn/shared-ux-{domain}-{componentName}`
  mocks/ :: `@kbn/shared-ux-{domain}-{componentName}-mocks`
  types/ :: `@kbn/shared-ux-{domain}-{componentName}-types`

The @kbn/{team}-{domain}-{component}(-{type})? style naming scheme is also followed by the core team in their packages:

• @kbn/core-analytics-browser
• @kbn/core-analytics-browser-internal
• @kbn/core-analytics-browser-mocks
• @kbn/core-analytics-server
• @kbn/core-analytics-server-mocks
• @kbn/core-analytics-server-internal
• etc.

What works now?

Today we have a simplistic package generator that produces legacy package definitions. We've finished laying the groundwork and have both the Core and Shared UX teams experimenting with it. You can use the legacy package generator with node scripts/generate package and create a package, however, we don’t recommend other teams to start migrating large portions of their code to packages just yet.

We're now entering Phase 2 of the plan, more details about the phases of our plan can be found above