Bootstrap MetadataChangeProposals (MCPs)

Bootstrap MCPs are templated MCPs which are loaded when the system-update job runs. This allows adding entities and aspects to DataHub at install time with the ability to customize them via environment variable overrides.

The built-in bootstrap MCP process can also be extended with custom MCPs. This can streamline deployment scenarios where a set of standard ingestion recipes, data platforms, users groups, or other configuration can be applied without the need for developing custom scripts.

Process Overview

When DataHub is installed or upgraded, a job runs called system-update, this job is responsible for data migration (particularly Elasticsearch indices) and ensuring the data is prepared for the next version of DataHub. This is the job which will also apply the bootstrap MCPs.

The system-update job, depending on configuration, can be split into two sequences of steps. If they are not split, then all steps are blocking.

1. An initial blocking sequence which is run prior to the new version of GMS and other components
2. Second sequence of steps where GMS and other components are allowed to run while additional data migration steps are continued in the background

When applying bootstrap MCPs system-update will perform the following steps:

1. The bootstrap_mcps.yaml file is read, either from a default classpath location, bootstrap_mcps.yaml, or a filesystem location provided by an environment variable, SYSTEM_UPDATE_BOOTSTRAP_MCP_CONFIG.
2. Depending on the mode of blocking or non-blocking each entry in the configuration file will be executed in sequence.
3. The template MCP file is loaded either from the classpath, or a filesystem location, and the template values are applied.
4. The rendered template MCPs are executed with the options specified in the bootstrap_mcps.yaml.

bootstrap_mcps.yaml Configuration

The bootstrap_mcps.yaml file has the following format.

bootstrap:
  templates:
    - name: <name>
      version: <version>
      force: false
      blocking: false
      async: true
      optional: false
      mcps_location: <classpath or file location>
      values_env: <environment variable>

Each entry in the list of templates points to a single yaml file which can contain one or more MCP objects. The execution of the template MCPs is tracked by name and version to prevent re-execution. The MCP objects are executed once unless force=true for each name/version combination.

See the following table of options for descriptions of each field in the template configuration.

Template MCPs

Template MCPs are stored in a yaml file which uses the mustache templating library to populate values from an optional environment variable. Defaults can be provided inline making override only necessary when providing install/upgrade time configuration.

In general the file contains a list of MCPs which follow the schema definition for MCPs exactly. Any valid field for an MCP is accepted, including optional fields such as headers.

Example: Native Group

An example template MCP collection, configuration, and values environment variable is shown below which would create a native group.

- entityUrn: urn:li:corpGroup:{{group.id}}
  entityType: corpGroup
  aspectName: corpGroupInfo
  changeType: UPSERT
  aspect:
   description: {{group.description}}{{^group.description}}Default description{{/group.description}}
   displayName: {{group.displayName}}
   created: {{&auditStamp}}
   members: [] # required as part of the aspect's schema definition
   groups: [] # required as part of the aspect's schema definition
   admins: [] # required as part of the aspect's schema definition
- entityUrn: urn:li:corpGroup:{{group.id}}
  entityType: corpGroup
  aspectName: origin
  changeType: UPSERT
  aspect:
     type: NATIVE

Creating an entry in the bootstrap_mcps.yaml to populate the values from the environment variable DATAHUB_TEST_GROUP_VALUES

- name: test-group
  version: v1
  mcps_location: "bootstrap_mcps/test-group.yaml"
  values_env: "DATAHUB_TEST_GROUP_VALUES"

An example json values are loaded from environment variable in DATAHUB_TEST_GROUP_VALUES might look like the following.

{
  "group": {
    "id": "mygroup",
    "displayName": "My Group",
    "description": "Description of the group"
  }
}

Using standard mustache template semantics the values in the environment would be inserted into the yaml structure and ingested when the system-update runs.

Default values

In the example above, the group's description if not provided would default to Default description if not specified in the values contain in the environment variable override following the standard mustache template semantics.

AuditStamp

A special template reference, {{&auditStamp}} can be used to inject an auditStamp into the aspect. This can be used to populate required fields of type auditStamp calculated from when the MCP is applied. This will insert an inline json representation of the auditStamp into the location and avoid escaping html characters per standard mustache template indicated by the & character.

Ingestion Template MCPs

Ingestion template MCPs are slightly more complicated since the ingestion recipe is stored as a json string within the aspect. For ingestion recipes, special handling was added so that they can be described naturally in yaml instead of the normally encoded json string.

This means that in the example below, the structure beneath the aspect.config.recipe path will be automatically converted to the required json structure and stored as a string.

- entityType: dataHubIngestionSource
  entityUrn: urn:li:dataHubIngestionSource:demo-data
  aspectName: dataHubIngestionSourceInfo
  changeType: UPSERT
  aspect:
    type: "demo-data"
    name: "demo-data"
    config:
      recipe:
        source:
          type: "datahub-gc"
          config: {}
      executorId: default

bootstrap_mcps.yaml Override

Additionally, the bootstrap_mcps.yaml can be overridden. This might be useful for applying changes to the version when using helm defined template values.

bootstrap:
  templates:
    - name: myMCPTemplate
      version: v1
      mcps_location: <classpath or file location>
      values_env: <value environment variable>
      revision_env: REVISION_ENV

In the above example, we've added a revision_env which allows overriding the MCP bootstrap definition itself (excluding revision_env).

In this example we could configure REVISION_ENV to contain a timestamp or hash: {"version":"2024060600"} This value can be changed/incremented each time the helm supplied template values change. This ensures the MCP is updated with the latest values during deployment.

Known Limitations

• Supported change types:
  • UPSERT
  • CREATE
  • CREATE_ENTITY