ko builder

Author(s): Halvard Skogsrud (@halvards)
Design Shepherd: <skaffold-core-team-member>
Date: 2021-03-29
Status: Draft

Objectives

Fast, standardized, reproducible, configuration-less, Docker-less, and secure-by-default container image builds for Go apps.

Background

ko is a container image builder for Go. It's fast, doesn't use a Dockerfile or rely on the Docker daemon, and uses distroless base images by default. ko is to Go apps what Jib is to JVM-based apps (approximately).

The Knative and Tekton open source projects use ko.

Proposal

This proposal adds a new ko builder to Skaffold, based on the ko build command (a.k.a. ko publish). The integration does not include other ko functionality related to rendering manifests, deploying to Kubernetes clusters, and file watching.

Compared to ...

the Cloud Native buildpacks builder, the ko builder is fast, doesn't require Docker, and uses a default base image that has a small attack surface (distroless).
the Docker builder, the ko builder standardizes builds, avoiding artisanal snowflake Dockerfiles. It also doesn't require the Docker daemon, so builds can run in security-constrained environments.
the Kaniko builder, the ko builder doesn't need a Kubernetes cluster, and avoids the previously-mentioned artisanal Dockerfiles.
the Bazel builder, the ko builder doesn't require users to adopt Bazel. However, users who already use Bazel for their Go app should use the Bazel builder.
the custom builder, the ko builder is portable:
1. The Skaffold config can be shared with other developers and ops teams, and used in CI/CD pipelines, without requiring users to install additional tools such as Docker Engine or crane (or even ko, depending on how the builder is implemented). This eases the path to adoption of Skaffold and reduces friction for users, both for local development, and for anyone using Skaffold in CI/CD pipelines.
2. The ko builder doesn't require running custom shell scripts. This means more standardized builds, a desirable trait for enterprise users.

The ko builder supports and enhances these Skaffold features:

fast local workflow: building with ko is fast.
share with other developers: no additional tools are required to skaffold run with the ko builder, not even Docker. Though if we don't embed ko in Skaffold, ko will be a tool that all developers in a team would have to install.
works great as a CI/CD building block: when using the ko builder, pipeline steps can run using the default Skaffold container image, without installing additional tools or keeping toolchain versions in sync across local development and CI/CD.

Background: ko image names and Go import paths

Ko uses Go import paths to build images. The ko build command is similar to go build and takes a positional argument, which can be either a local file path or a Go import path. If the argument is a local file path (as per go/build.IsLocalImport()) then, ko resolves the local file path to a Go import path (see github.com/google/ko/pkg/build).

The import path must be of the package than contains the main() function. For instance, to build Skaffold using ko, from the repository root directory:

ko build ./cmd/skaffold

ko build github.com/GoogleContainerTools/skaffold/cmd/skaffold

When the ko CLI is used to populate the image name in templated Kubernetes resource files, only the Go import path option can be used, and the import path must be prefixed by the ko:// scheme, e.g., ko://github.com/GoogleContainerTools/skaffold/cmd/skaffold.

Ko determines the image name from the container image registry (provided by the KO_DOCKER_REPO environment variable) and the Go import path. The Go import path is appended in one of these ways:

The last path segment (e.g., skaffold), followed by a hyphen and a MD5 hash. This is the default behavior of the ko build command.
The last path segment (e.g., skaffold) only, if ko build is invoked with the -B or --base-import-paths flag.
The full import path, lowercased (e.g., github.com/googlecontainertools/skaffold/cmd/skaffold), if ko build is invoked with the -P or --preserve-import-paths flag. This is the option used by projects such as Knative (see the release.sh script) and Tekton (see the pipeline in publish.yaml).
No import path (just KO_DOCKER_REPO), if ko build is invoked with the --bare flag.

Supporting existing Skaffold users

The Skaffold ko builder follows the existing Skaffold image naming logic. This means that the image naming behavior doesn't change for existing Skaffold users who migrate from other builders to the ko builder.

The ko builder achieves this by using ko's Bare naming option.

By using this option, the image name is not tied to the Go import path. If the Skaffold default repo value is gcr.io/k8s-skaffold and the value of the image field in skaffold.yaml is skaffold, the resulting image name will be gcr.io/k8s-skaffold/skaffold.

It is still necessary to resolve the Go import path for the underlying ko implementation. To do so, the ko builder determines the import path based on the value of the main config field. The main config field refers to the location of a main package and corresponds to a go build pattern, e.g., go build ./cmd/skaffold. Using the main field results in deterministic behavior even in cases where there are multiple main packages in different directories.

If main is a relative path (and it will be most of the time), it is relative to the current context (a.k.a. Workspace) directory.

For example, to build Skaffold itself, with package main in the ./cmd/skaffold/ subdirectory, the config would be as follows:

yaml

apiVersion: skaffold/v2beta26
kind: Config
build:
  artifacts:
  - image: skaffold
    context: .
    ko:
      main: ./cmd/skaffold

Users can specify a main value using a pattern with the ... wildcard, such as ./.... In this case, ko locates the main package. If there are multiple main packages, ko fails.

Implementation note: The value of main will be the input when invoking QualifyImport().

If the Go sources and the go.mod file are in a subdirectory of the context directory, users can use the dir config field to specify the path where the ko builder runs the go tool.

Supporting existing ko users

To support existing ko users moving to Skaffold, the ko builder also supports image names in skaffold.yaml that use the Go import path, prefixed by the ko:// scheme. Examples of such image references in Kubernetes manifest files can be seen in projects such as Knative and Tekton.

In the case of ko://-prefixed image names, the Skaffold ko builder constructs the image name by:

Removing the ko:// scheme prefix.
Transforming the import path to a valid image name using the function SanitizeImageName() (from the package github.com/GoogleContainerTools/skaffold/pkg/skaffold/docker).
Combining the Skaffold default repo with the transformed import path as per existing Skaffold image naming logic.

This will result in image names that match those produced by the ko CLI when using the -P or --preserve-import-paths flag. For example, if the Skaffold default repo is gcr.io/k8s-skaffold and the image name in skaffold.yaml is ko://github.com/GoogleContainerTools/skaffold/cmd/skaffold, the resulting image name will be gcr.io/k8s-skaffold/github.com/googlecontainertools/skaffold/cmd/skaffold.

Real-world examples of image names that follow this naming convention can be found in the Tekton and Knative release manifests. For instance, view the images in the Knative Serving release YAMLs:

curl -sL https://github.com/knative/serving/releases/download/v0.24.0/serving-core.yaml | grep 'image: '

If the image field in skaffold.yaml starts with the ko:// scheme prefix, the Skaffold ko builder uses the Go import path that follows the prefix. For example, to build Skaffold itself, with package main in the ./cmd/skaffold/ subdirectory, the config would be as follows:

yaml

apiVersion: skaffold/v2beta26
kind: Config
build:
  artifacts:
  - image: ko://github.com/GoogleContainerTools/skaffold/cmd/skaffold
    context: .
    ko: {}

The main field is ignored if the image field starts with the ko:// scheme prefix.

Debugging ko images using Skaffold

Ko provides the DisableOptimizations build option to set gcflags to disable optimizations and inlining. The ko builder sets DisableOptimizations to true when the Skaffold runMode is Debug.

Skaffold can recognize Go-based container images built by ko by the presence of the KO_DATA_PATH environment variable. This allows Skaffold to transform Pod specifications to enable remote debugging.

The ko builder implementation work will add KO_DATA_PATH to the set of environment variables used to detect Go-based applications and updating the associated unit tests and documentation.

Design

Adding the ko builder requires making config changes to the Skaffold schema.

Add a KoArtifact type:

// KoArtifact builds images using [ko](https://github.com/google/ko).
type KoArtifact struct {
    // Asmflags are assembler flags passed to the builder.
    Asmflags []string `yaml:"asmflags,omitempty"`

    // BaseImage overrides the default ko base image.
    // Corresponds to, and overrides, the `defaultBaseImage` in `.ko.yaml`.
    BaseImage string `yaml:"fromImage,omitempty"`

    // Dependencies are the file dependencies that Skaffold should watch for both
    // rebuilding and file syncing for this artifact.
    Dependencies *KoDependencies `yaml:"dependencies,omitempty"`

    // Dir is the directory where the `go` tool will be run.
    // The value is a directory path relative to the `context` directory.
    // If empty, the `go` tool will run in the `context` directory.
    // Examples: `live-at-head`, `compat-go114`
    Dir string `yaml:"dir,omitempty"`

    // Env are environment variables, in the `key=value` form, passed to the build.
    // These environment variables are only used at build time.
    // They are _not_ set in the resulting container image.
    // For example: `["GOPRIVATE=source.developers.google.com", "GOCACHE=/workspace/.gocache"]`.
    Env []string `yaml:"env,omitempty"`

    // Flags are additional build flags passed to the builder.
    // For example: `["-trimpath", "-v"]`.
    Flags []string `yaml:"flags,omitempty"`

    // Gcflags are Go compiler flags passed to the builder.
    // For example: `["-m"]`.
    Gcflags []string `yaml:"gcflags,omitempty"`

    // Labels are key-value string pairs to add to the image config.
    // For example: `{"org.opencontainers.image.source":"https://github.com/GoogleContainerTools/skaffold"}`.
    Labels map[string]string `yaml:"labels,omitempty"`

    // Ldflags are linker flags passed to the builder.
    // For example: `["-buildid=", "-s", "-w"]`.
    Ldflags []string `yaml:"ldflags,omitempty"`

    // Main is the location of the main package. It is the pattern passed to `go build`.
    // If main is specified as a relative path, it is relative to the `context` directory.
    // If main is empty, the ko builder uses a default value of `.`.
    // If main is a pattern with wildcards, such as `./...`,
    // the expansion must contain only one main package, otherwise ko fails.
    // Main is ignored if the `ImageName` starts with `ko://`.
    // Example: `./cmd/foo`
    Main string `yaml:"main,omitempty"`

    // Platforms is the list of platforms to build images for. Each platform
    // is of the format `os[/arch[/variant]]`, e.g., `linux/amd64`.
    // By default, the ko builder builds for `all` platforms supported by the
    // base image.
    Platforms []string `yaml:"platforms,omitempty"`

    // SourceDateEpoch is the `created` time of the container image.
    // Specify as the number of seconds since January 1st 1970, 00:00 UTC.
    // You can override this value by setting the `SOURCE_DATE_EPOCH`
    // environment variable.
    SourceDateEpoch uint64 `yaml:"sourceDateEpoch,omitempty"`
}

Add a KoArtifact field to the ArtifactType struct:

type ArtifactType struct {
  [...]
    // KoArtifact builds images using [ko](https://github.com/google/ko).
    KoArtifact *KoArtifact `yaml:"ko,omitempty" yamltags:"oneOf=artifact"`
}

Define KoDependencies:

// KoDependencies is used to specify dependencies for an artifact built by ko.
type KoDependencies struct {
    // Paths should be set to the file dependencies for this artifact,
    // so that the Skaffold file watcher knows when to rebuild and perform file synchronization.
    // Defaults to ["**/*.go"].
    Paths []string `yaml:"paths,omitempty" yamltags:"oneOf=dependency"`

    // Ignore specifies the paths that should be ignored by Skaffold's file watcher.
    // If a file exists in both `paths` and in `ignore`, it will be ignored,
    // and will be excluded from both rebuilds and file synchronization.
    Ignore []string `yaml:"ignore,omitempty"`
}

Add KO to the BuilderType enum in proto/enums/enums.proto:

proto

enum BuilderType {
    // Could not determine builder type
    UNKNOWN_BUILDER_TYPE = 0;
    // JIB Builder
    JIB = 1;
    // Bazel Builder
    BAZEL = 2;
    // Buildpacks Builder
    BUILDPACKS = 3;
    // Custom Builder
    CUSTOM = 4;
    // Kaniko Builder
    KANIKO = 5;
    // Docker Builder
    DOCKER = 6;
    // Ko Builder
    KO = 7;
}

Builder config schema

Example basic config, this will be sufficient for many users:

yaml

apiVersion: skaffold/v2beta26
kind: Config
build:
  artifacts:
  - image: skaffold-example-ko
    ko: {}

The value of the image field is the Go import path of the app entry point, prefixed by ko://.

A more comprehensive example config:

yaml

apiVersion: skaffold/v2beta26
kind: Config
build:
  artifacts:
  - image: skaffold-example-ko-comprehensive
    ko:
      fromImage: gcr.io/distroless/base:nonroot
      dependencies:
        paths:
        - go.mod
        - "**.go"
      dir: '.'
      env:
      - GOPRIVATE=source.developers.google.com
      flags:
      - -trimpath
      - -v
      gcflags:
      - -m
      labels:
        foo: bar
        baz: frob
      ldflags:
      - -buildid=
      - -s
      - -w
      main: ./cmd/foo
      platforms:
      - linux/amd64
      - linux/arm64

ko requires setting a KO_DOCKER_REPO environment variable to specify where container images are pushed. The Skaffold default repo maps directly to this value.

Resolved questions

Should Skaffold embed ko as a Go module, or shell out?

Resolved: Embed as a Go module

Benefits of embedding:
- Skaffold can pin the ko version it supports in its go.mod file. Users wouldn't raise bugs/issues for incompatible version pairings of Skaffold and ko.
- Reduce toolchain maintenance toil for users. Skaffold users wouldn't need to synchronize ko versions used by different team members or in their CI build, since the Skaffold version determines the ko version.
- Portability. Skaffold+ko users only need one tool for their container image building needs: the skaffold binary. (Plus the Go distribution, of course.) The current gcr.io/k8s-skaffold/skaffold container image could serve as a build and deploy image for CI/CD pipeline steps.
Embedding ko would require some level of documented behavioural stability guarantees for the most ko interfaces that Skaffold would use, such as build.Interface and publish.Interface, or others?

Benefits of shelling out:
- It's an established pattern used by other Skaffold builders.
- It would allow Skaffold to support a range of ko versions. On the other hand, these versions would need to be tracked and documented.
- No need to resolve dependency version differences between Skaffold and ko.
- If a new ko version provided a significant bug fix, there would be no need to release a new version of Skaffold for this fix.
Shelling out to ko would require some stability guarantees for the ko build subcommand.

Suggest embedding as a Go module.
Should Skaffold use base image settings from .ko.yaml if the ko builder definition in skaffold.yaml doesn't specify a base image?

Resolved: Yes, to simplify adoption of Skaffold for existing ko users.
If a config value is set both as an environment variable, and as a config value, which takes precedence? E.g., ko.sourceDateEpoch vs SOURCE_DATE_EPOCH.

Resolved: Follow existing Skaffold patterns.
Should the ko builder have a config option for SOURCE_DATE_EPOCH, or should users specify the value via an environment variable?

Resolved: Specify via the reproducible builds spec environment variable SOURCE_DATE_EPOCH, see https://github.com/google/ko#why-are-my-images-all-created-in-1970 and https://reproducible-builds.org/docs/source-date-epoch/.
Should we default dependency paths to ["go.mod", "**.go"] instead of ["."].?

The former is a useful default for many (most?) Go apps, and it's used in the custom example. The latter is the default for some other builders.

Resolved: Default to ["**/*.go"], see #6617.
Add a Google Cloud Build (gcb) support for the ko builder?

By embedding ko as a module, there is no need for a ko-specific Skaffold builder image.

Resolved: Add remote builder support.
File sync support: Should we limit this to ko static assets only?

This is the only way to include additional files in a container image built by ko.

Resolved: Implement file sync (for the Beta stage).

Open questions

Should the ko builder be the default for skaffold init, instead of buildpacks, for Go apps, when there's no Dockerfile and no Bazel workspace file?

Suggest yes, to make Skaffold a compelling choice for Go developers.

If no, we can still consider configuring the ko builder if skaffold init finds a .ko.yaml configuration file.

Not Yet Resolved

Approach

Implement the ko builder as a series of small PRs that can be merged one by one. The PRs should not surface any new user-visible behavior until the feature is ready.

This approach has a lower risk than implementing the entire feature on a separate branch before merging all at once.

The steps roughly outlined:

Add dependency on the github.com/google/ko module.
Implement the core ko build and publish logic, including unit tests in the package github.com/GoogleContainerTools/skaffold/pkg/skaffold/build/ko.

Support this implementation with "dummy" schema definitions for types such as KoArtifact in a separate "temporary" package. This package only exists until the definitions are merged into the latest v1 schema, and it allows for evolution of the types until they are committed to the schema.
Add integration test for the ko builder to the integration package, and an example app + config to a new integration/examples/ko directory.

To avoid failures in schema unit tests from the new example in the integration directory, add an if statement to TestParseExamples in the package github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema that skips directories called ko.
Add the ko builder schema types (KoArtifact and KoDependency) to the latest unreleased schema.

For the ArtifactType struct, add a KoArtifact field, but set its YAML flag key to "-" (full syntax `yaml:"-,omitempty" yamltags:"oneOf=artifact"`).

Do not yet add the KO = 7; entry to the BuilderType enum in proto/enums/enums.proto.
Plumb the code in the package pkg/skaffold/build/ko into the other parts of the Skaffold codebase that interacts with the config schema.

E.g., a case statement for a.KoArtifact in the newPerArtifactBuilder function in pkg/skaffold/build/local (file types.go).
When we are ready to add the ko builder as an alpha feature to an upcoming Skaffold release, set the KoArtifact YAML flag key to ko and add KO to the BuilderType enum.

Implementation plan

[Done] Define integration points in the ko codebase that allows ko to be used from Skaffold without duplicating existing ko CLI code.

In the package github.com/google/ko/pkg/commands:

resolver.go

// NewBuilder creates a ko builder
func NewBuilder(ctx context.Context, bo *options.BuildOptions) (build.Interface, error)

resolver.go

// NewPublisher creates a ko publisher
func NewPublisher(po *options.PublishOptions) (publish.Interface, error)

publisher.go

// PublishImages publishes images
func PublishImages(ctx context.Context, importpaths []string, pub publish.Interface, b build.Interface) (map[string]name.Reference, error)

Add build and publish options to support Skaffold config propagating to ko. In the package github.com/google/ko/pkg/commands/options:

build.go

type BuildOptions struct {
      // BaseImage enables setting the default base image programmatically.
      // If non-empty, this takes precedence over the value in `.ko.yaml`.
      BaseImage string

      // WorkingDirectory allows for setting the working directory for invocations of the `go` tool.
      // Empty string means the current working directory.
      WorkingDirectory string

    // UserAgent enables overriding the default value of the `User-Agent` HTTP
      // request header used when retrieving the base image.
      UserAgent string

    [...]
}

publish.go

type PublishOptions struct {
      // DockerRepo configures the destination image repository.
      // In normal ko usage, this is populated with the value of $KO_DOCKER_REPO.
      DockerRepo string

      // LocalDomain overrides the default domain for images loaded into the local Docker daemon.
      // Use with Local=true.
      LocalDomain string

      // UserAgent enables overriding the default value of the `User-Agent` HTTP
      // request header used when pushing the built image to an image registry.
      UserAgent string

    [...]
}

Add ko builder with support for existing ko config options. Provide this as an Alpha feature in an upcoming Skaffold release.

Config options supported, all are optional:

fromImage, to override the default distroless base image
dependencies, for Skaffold file watching
dir, if Go sources are not in the context directory
env, to support ko CLI users who currently set environment variables such as GOFLAGS when running ko.
labels, e.g., to link an image to a Git repository
platforms
sourceDateEpoch
flags, e.g., -v, -trimpath
ldflags, e.g., -s

Example skaffold.yaml supported at this stage:

yaml

apiVersion: skaffold/v2beta26
kind: Config
build:
  artifacts:
  - image: skaffold-ko
    ko:
      fromImage: gcr.io/distroless/base:nonroot
      dependencies:
        paths:
        - go.mod
        - "**.go"
      dir: '.'
      env:
      - GOPRIVATE=source.developers.google.com
      labels:
        foo: bar
        baz: frob
      ldflags:
      - -s
      main: ./cmd/foo
      platforms:
      - linux/amd64
      - linux/arm64

Implement Skaffold config support for additional ko config options not currently supported by ko:
- asmflags
- gcflags
Provide this as a feature in an upcoming Skaffold release.

Release plan

The ko builder will go through the release stages Alpha -> Beta -> Stable.

The following features will be released at each stage:

Alpha

Support for the Skaffold lifecycle subcommands that build images:
- build
- debug
- dev
- run
Images built using the ko builder are automatically detected as Go images for debugging support.
File watch support for dev mode.
Local builder only (no gcb or cluster builder at this stage).
Multi-platform images support, including support for all, which builds images for all platforms supported by the base image).
Image names following standard Skaffold naming, for existing Skaffold users.
Support for ko://-prefixed image names, for existing ko users.

Beta

File sync support.
Remote builders support.
skaffold init support, behind a --enableKoInit flag.

Stable

Cloud Code integration

Integration test plan

Please describe what new test cases you are going to consider.

Unit and integration tests for ko builder, similar to other builders.

The integration tests should be written to catch situations such as where changes to ko interfaces break the Skaffold ko builder.
Test that the ko flag --disable-optimization is added for debugging.
Add basic and comprehensive ko examples to the integration/examples directory.