Option for origin and transformer annotations

Authors:

• natasha41575

Reviewers:

• monopole
• KnVerey
• yuwenma

Status: implementable

Summary

This proposal is to extend the buildMetadata.originAnnotations option in the kustomization to annotate generated resources with information about the source of their generators, and to add a new option buildMetadata.transformerAnnotations that will add annotations to each resource with information about all the transformers that have processed it. These options will help users understand how their output was created and debug it accordingly.

Motivation

When a user is managing a large number of resources with kustomize, the output of kustomize build will be a long stream of yaml resources, and it can be extremely tedious for the user to understand how each resource was created and processed. However, we can provide the user with an option to annotate each resource with origin and transformation data. This information will help kustomize users debug their large kustomize configurations. Because these annotations can be used for debugging purposes, we would like them to be verbose and communicate as much information as possible to the user.

Kustomize previously had a feature request to add an option to kustomize build to retain the origin of a resource in an annotation, and a final design was settled on after some discussion. It was implemented by this pull request and released in kustomize v4.3.0. This feature added a new field to the kustomization file, buildMetadata, an array of boolean options such as originAnnotations. If originAnnotations is set, kustomize build will add annotations to each resource describing the resource's origin.

While the above feature gives us valuable information about where many resources originated from, there are two pieces missing:

• Origin annotations for generated resources. Resources can either be generated using a builtin generator such as ConfigMapGenerator or SecretGenerator, or can be generated from a custom generator invoked from either the generators or transformers field.

• Everything in kustomize is a transformer, so we would like to have annotations about which transformers, builtin or custom, touched each resource.

For a user attempting to debug a large stream of output from kustomize build, knowing how each resource has been transformed through various layers will make it much easier to understand how the output was rendered, easing debugging and making changes to kustomization files.

Builtin Transformers

Builtin transformers include those that are invoked through various kustomization fields. Some of these transformers are:

• AnnotationsTransformer
• HashTransformer
• ImageTagTransformer
• LabelTransformer
• LegacyOrderTransformer
• NamespaceTransformer
• PatchJson6902Transformer
• PatchStrategicMergeTransformer
• PatchTransformer
• PrefixSuffixTransformer
• ReplacementTransformer
• ReplicaCountTransformer
• ValueAddTransformer

Custom transformers are invoked via the transformers field in the kustomization file. Although KRM-style plugins can be invoked via the generators and validators fields, it is impossible for plugins in these fields to transform existing resources, as generators do not take input and validators are not permitted to change input. Therefore, all custom transformers will appear in the transformers field.

It is, however, possible to put a generator or validator in the transformers field, so we should try to differentiate the plugins in the transformers field where we can, and only store data about actual transformers.

Goals:

1. When originAnnotations is set, add annotations to generated resources that describe the source of the generator that created them.
2. Add a new option, transformerAnnotations, to buildMetadata that, when set, will annotate each resource with information about which transformers, builtin or custom, touched each resource.

Non-goals:

1. Change the syntax of the existing originAnnotations option.
2. Add these annotations to the output of kustomize by default.

Proposal

Origin Annotation

To retain the information about the origin of a resource, the user can specify the originAnnotations option in the buildMetadata field of the kustomization:

buildMetadata: [originAnnotations]

When this option is set, generated resources should receive an annotation with key config.kubernetes.io/origin, containing data about the generator that produced it. If the resource is from the resources field, this annotation contains data about the file it originated from.

The possible fields of these annotations are:

config.kubernetes.io/origin: |
  path: path.yaml # The path to the resource file itself
  ref: v0.0.0 # If from a remote file or generator, the ref of the repo URL
  repo: http://github.com/examplerepo # If from a remote file or generator, the repo source
  configuredIn: path/to/generatorconfig # If a generated resource, the path to the generator config
  configuredBy: # If a generated resource, the ObjectReference of the generator config
    kind: Generator
    apiVersion: builtin 
    name: foo
    namespace: default

All local file paths are relative to the top-level kustomization, i.e. the kustomization file in the directory upon which kustomize build was invoked. For example, if someone were to run kustomize build foo, all file paths in the annotation output would be relative tofoo/kustomization.yaml.

All remote file paths are relative to the root of the remote repository.

Any fields that are not applicable would be omitted from the final output. If a generator is invoked via a field in the kustomization file, configuredIn would point to the kustomization file itself.

Examples

Resource declared from resources

A kustomization such as the following:

resources:
- deployment.yaml

buildMetadata: [originAnnotations]

would produce a resource with an annotation like the following:

config.kubernetes.io/origin: |
  path: deployment.yaml

Local custom generator

A kustomization such as the following:

generators:
- generator.yaml

buildMetadata: [originAnnotations]

would produce a resource with an annotation like the following:

config.kubernetes.io/origin: |
  configuredIn: generator.yaml
  configuredBy: 
    kind: MyGenerator
    apiVersion: v1
    name: generator

Remote builtin generator

We have a top-level kustomization such as the following:

resources:
- github.com/examplerepo/?ref=v1.0.6

buildMetadata: [originAnnotations]

which uses github.com/examplerepo/?ref=v1.0.6 as a remote base. This remote base has the following kustomization defined in github.com/examplerepo/kustomization.yaml:

configMapGenerator:
- name: my-java-server-env-vars
  literals:
  - JAVA_HOME=/opt/java/jdk
  - JAVA_TOOL_OPTIONS=-agentlib:hprof

Running kustomize build on the top-level kustomization would produce the following output:

apiVersion: v1
data:
  JAVA_HOME: /opt/java/jdk
  JAVA_TOOL_OPTIONS: -agentlib:hprof
kind: ConfigMap
metadata:
  name: my-java-server-env-vars-44k658k8gk
  annotations:
    config.kubernetes.io/origin: |
      ref: v1.0.6 
      repo: github.com/examplerepo 
      configuredIn: kustomization.yaml
      configuredBy: 
        kind: ConfigMapGenerator
        apiVersion: builtin 

Transformer Annotations

To retain the information about what transformers have acted on each resource, we can propose a new option transformerAnnotations in the buildMetadata field of the kustomization:

buildMetadata: [transformerAnnotations]

It is possible for the user to set both transformerAnnotations and originAnnotations:

buildMetadata: [originAnnotations, transformerAnnotations]

When the transformerAnnotations option is set, kustomize will add annotations with information about what transformers have acted on each resource. Transformers can be invoked either through various fields in the kustomization file (e.g. the replacements field will invoke the ReplacementTransformer), or through the transformers field.

The annotation key for transformer annotatinos will be config.kubernetes.io/transformations, which will contain a list of transformer data:

config.kubernetes.io/transformations: |
- ref: v0.0.0 # If from a remote transformer, the ref of the repo URL
  repo: http://github.com/examplerepo # If from a remote transformer, the repo source
  configuredIn: path/to/transformerconfig # The path to the transformer config
  configuredBy: # The ObjectReference of the transformer config
    kind: Transformer
    apiVersion: builtin 
    name: foo
    namespace: default

The possible fields for each item in the list is identical to the possible fields in config.kubernetes.io/origin, except that the transformer annotation does not have a path field for the path to the resource file itself.

All local file paths are relative to the top-level kustomization, i.e. the kustomization file in the directory upon which kustomize build was invoked. For example, if someone were to run kustomize build foo, all file paths in the annotation output would be relative tofoo/kustomization.yaml.

All remote file paths are relative to the root of the remote repository.

Any fields that are not applicable would be omitted from the final output. If a transformer is invoked via a field in the kustomization file, configuredIn would point to the kustomization file itself.

Examples

Local custom transformer

A kustomization such as the following:

transformers:
- transformer.yaml

buildMetadata: [transformerAnnotations]

would produce a resource with an annotation like the following:

config.kubernetes.io/transformations: |
- configuredIn: transformer.yaml
  configuredBy: 
    kind: MyTransformer
    apiVersion: v1
    name: transformer

Remote builtin transformer + local builtin transformer

We have a top-level kustomization such as the following:

resources:
- github.com/examplerepo/?ref=v1.0.6

buildMetadata: [transformerAnnotations]

namespace: my-ns

which uses github.com/examplerepo/?ref=v1.0.6 as a remote base. This remote base has the following kustomization defined in github.com/examplerepo/kustomization.yaml:

resources:
- deployment.yaml

namePrefix: pre-

deployment.yaml contains the following:

apiVersion: v1
kind: Deployment
metadata:
  name: deploy

Running kustomize build on the top-level kustomization would produce the following output:

apiVersion: v1
kind: Deployment
metadata:
  name: pre-deploy
  annotations:
    config.kubernetes.io/transformations: |
    - ref: v1.0.6 
      repo: github.com/examplerepo 
      configuredIn: kustomization.yaml
      configuredBy: 
        kind: PrefixSuffixTransformer
        apiVersion: builtin 
    - configuredIn: kustomization.yaml
      configuredBy:
        kind: NamespaceTransformer
        apiVersion: builtin

Kustomize edit

We will want to provide convenient commands for users to edit the buildMetadata field in their kustomization. We should support the following commands:

kustomize edit add buildMetadata originAnnotations

kustomize edit add buildMetadata transformerAnnotations

kustomize edit add buildMetadata *

kustomize edit remove buildMetadata originAnnotaitons

kustomize edit remove buildMetadata transformerAnnotations

kustomize edit remove buildMetadata *

The wildcard match * should add/remove all possible options for buildMetadata.

Implementation

If there is a large tree of kustomization files, the data stored for origin and transformer data could be potentially very large, so we should be careful about implementation. We should:

1. Ensure that we are only accumulating the data if the option is set.
2. Account for generators that may appear in the transformers field. When we are processing each plugin invoked in the transformers field, we should check to see if the resources have an origin annotation. If not, we can assume that it was generated by the transformer and we can add the corresponding origin annotation. If it already has an origin annotation, we can add the corresponding transformation annotation.
3. A ConfigMapGenerator with merge set to true actually transforms instead of generates, so we should take care to account for this special case. In this case, we should add a transformer annotation.

User Stories

Story 1

I am running kustomize build on some configuration that has already been created. This configuration has a large number of resources and patches through multiple overlays in different directories. After running kustomize build, I get a stream of 100+ resources, but I notice that a few of them have a typo, likely originating from the same base or patch file. However, because my directory structure is large and complex, it is difficult to narrow down which base these resources originated from and which patches have been applied to it. In the top-level kustomization file, I can add a new buildMetadata field:

buildMetadata: [originAnnotations, transformerAnnotations]

The output of kustomize build will contain patched resources with annotations similar to the following:

annotations:
  config.kubernetes.io/origin: |
    path: deployment.yaml
  config.kubernetes.io/transformations:
  - configuredIn: ../base/kustomization.yaml
    configuredBy: 
      kind: PatchTransformer
      apiVersion: builtin 
  - configuredIn: ../dev/kustomization.yaml
    configuredBy: 
      kind: PatchStrategicMergeTransformer
      apiVersion: builtin 
  - configuredIn: ../dev/kustomization.yaml
    configuredBy: 
      kind: PatchesJson6902Transformer
      apiVersion: builtin 

From these annotations, I can narrow down the set of files that contributed to the resource's final output and debug it accordingly.

Story 2

I am using a tool that automatically runs kustomize build and deploys the output to the cluster, such as skaffold deploy, kubectl apply -k, or Google's Config Sync. I am unfamiliar with kustomize CLI and do not wish to install it to my local machine or learn how to use it beyond using tools that run kustomize build. I add the originAnnotations and transformationAnnotations options to my top-level kustomization file.

With one of these tools, I apply my kustomize configuration to my cluster, but later discover that something in my cluster is not configured correctly. Because the annotations have persisted to the cluster, I can take a look at the resources in my cluster and understand how each of them was created, helping me figure out what went wrong.

Risks and Mitigations

N/A

Dependencies

N/A

Scalability

With a large input, the data collected here could be very large, so we should take care to only accumulate data when necessary.

Drawbacks

N/A

Alternatives

1. Putting this data in the comments of the rendered resources, but this would be a large project because kustomize doesn't currently support comments.

2. Making this a flag to kustomize build rather than additional field, but we would like to align with the kustomize way of avoiding build-time side effects and have everything declared explicitly in the kustomization.

3. Having a separate kustomize command that prints out only the origin and/or transformer data. While this could be very useful as a debugging feature, there are use cases for tools that automatically run kustomize and deploy, where it would be more useful to the user to have the annotations persist to the cluster. See User Story 2.

Rollout Plan

This is a fairly lightweight, minor and optional feature that can be implemented and released without needing a rollout plan, much like the originAnnotations option that we are extending.

The origin annotation is simply being extended, so there is no need to mark this feature as alpha. The rollout plan for the transformation annotation is outlined below:

Alpha

• Will the feature be gated by an "alpha" flag? Which one?

The feature will not be gated by an alpha flag, but we will start the annotation with the prefix "alpha", so the annotation will key be alpha.config.kubernetes.io/transformations

• Will the feature be available in kubectl kustomize during alpha? Why or why not?

Yes, the feature will be part of kubectl kustomize as an option in the buildMetadata field, and will likewise start with the annotation key alpha.config.kubernetes.io/transformations

GA

We will wait for two kubectl releases before promoting the feature. We will iterate the feature based on feedback from users of Config Sync, Skaffold, kustomize CLI, and kustomize-in-kubectl prior to promotion.