Auditing dockerfile

Because of the nature of GitHub cache, and the time it takes to build the dockerfile for testing, it is desirable to be able to audit what is going on there.

This document provides a few pointers on how to do that, and some results as of 2025-02-26 (run inside lima, nerdctl main, on a macbook pro M1).

Intercept network traffic

On macOS

Use Charles:

• start SSL proxying
• enable SOCKS proxy
• export the root certificate

On linux

Left as an exercise to the reader.

If using lima

• restart your lima instance with HTTP_PROXY=http://X.Y.Z.W:8888 HTTPS_PROXY=socks5://X.Y.Z.W:8888 limactl start instance - where XYZW is the local ip of the Charles proxy (non-localhost)

On the host where you are running containerd

• copy the root certificate from above into /usr/local/share/ca-certificates/charles-ssl-proxying-certificate.crt
• update your host: sudo update-ca-certificates
• now copy the root certificate again to your current nerdctl clone

Hack the dockerfile to insert our certificate

Add the following stages in the dockerfile:

FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-trixie AS hack-build-base-debian
RUN apt-get update -qq; apt-get -qq install ca-certificates
COPY charles-ssl-proxying-certificate.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS hack-build-base
RUN apk add --no-cache ca-certificates
COPY charles-ssl-proxying-certificate.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

FROM ubuntu:${UBUNTU_VERSION} AS hack-base
RUN apt-get update -qq; apt-get -qq install ca-certificates
COPY charles-ssl-proxying-certificate.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

Then replace any later "FROM" with our modified bases:

golang:${GO_VERSION}-trixie => hack-build-base-debian
golang:${GO_VERSION}-alpine => hack-build-base
ubuntu:${UBUNTU_VERSION} => hack-base

Mimicking what the CI is doing

A quick helper:

run(){
  local no_cache="${1:-}"
  local platform="${2:-arm64}"
  local dockerfile="${3:-Dockerfile}"
  local target="${4:-test-integration}"

  local cache_shard="$CONTAINERD_VERSION"-"$platform"
  local shard="$cache_shard"-"$target"-"$UBUNTU_VERSION"-"$no_cache"-"$dockerfile"

  local cache_location=$HOME/bk-cache-"$cache_shard"
  local destination=$HOME/bk-output-"$shard"
  local logs="$HOME"/bk-debug-"$shard"

  if [ "$no_cache" != "" ]; then
    nerdctl system prune -af
    nerdctl builder prune -af
    rm -Rf "$cache_location"
  fi

  nerdctl build \
    --build-arg UBUNTU_VERSION="$UBUNTU_VERSION" \
    --build-arg CONTAINERD_VERSION="$CONTAINERD_VERSION" \
    --platform="$platform" \
    --output type=tar,dest="$destination" \
    --progress plain \
    --build-arg HTTP_PROXY=$HTTP_PROXY \
    --build-arg HTTPS_PROXY=$HTTPS_PROXY \
    --cache-to type=local,dest="$cache_location",mode=max \
    --cache-from type=local,src="$cache_location" \
    --target "$target" \
    -f "$dockerfile" . 2>&1 | tee "$logs"
}

And here is what the CI is doing:

ci_run(){
  local no_cache="${1:-}"
  export UBUNTU_VERSION=24.04

  # The actual version may differ
  CONTAINERD_VERSION=v1.7.25 run "$no_cache"  arm64 Dockerfile.origin build-dependencies
  UBUNTU_VERSION=22.04 CONTAINERD_VERSION=v1.7.25 run "" arm64 Dockerfile.origin test-integration

  CONTAINERD_VERSION=v2.0.3 run "$no_cache"  arm64 Dockerfile.origin build-dependencies
  UBUNTU_VERSION=24.04 CONTAINERD_VERSION=v2.0.3 run "" arm64 Dockerfile.origin test-integration

  CONTAINERD_VERSION=v2.0.3 run "$no_cache"  amd64 Dockerfile.origin build-dependencies
  UBUNTU_VERSION=24.04 CONTAINERD_VERSION=v2.0.3 run "" amd64 Dockerfile.origin test-integration
}

# To simulate what happens when there is no cache, go with:
ci_run no_cache

# Once you have a cached run, you can simulate what happens with cache
# First modify something in the nerdctl tree
# Then run it
touch mimick_nerdctl_change
ci_run

Analyzing results

Network

Full CI run, cold cache (the first three pipelines, and part of the fourth)

The following numbers are based on the above script, with cold cache.

Unfortunately golang did segfault on me during the last (cross-run targetting amd), so, these numbers should be taken as (slightly) underestimated.

Total number of requests: 7190

Total network duration: 13 minutes 11 seconds

Outbound: 1.31MB

Inbound: 5202MB

Breakdown per domain

Full CI run, warm cache (only the first three pipelines)

Analysis

Docker Hub

Images from Docker Hub are clearly a source of concern (made even worse by the fact they apply strict limitations on the number of requests permitted).

When the cache is cold, this is about 1GB per run, for 200 requests and 3 minutes.

Actions:

• reduce the number of images
  • we currently use 2 golang images, which does not make sense
• reduce the round trips
  • there is no reason why any of the images should be queried more than once per build
• move away from Hub golang image, and instead use a raw distro + golang download
  • Hub golang is a source of pain and issues (diverging version scheme forces ugly shell contorsions, delay in availability creates broken situations)
  • we are already downloading the go release tarball anyhow, so, this is just wasted bandwidth with no added value

Success criteria:

• on a cold cache, reduce the total number of requests against Docker properties by 50% or more
• on a cold cache, cut the data transfer and time in half

Distro packages

On a WARM cache, close to 1 minute is spent fetching Ubuntu packages. This should not happen, and distro downloads should always be cached.

On a cold cache, distro packages download near 3 minutes. Very likely there is stage duplication that could be reduced and some of that could be cut of.

Actions:

• ensure distro package downloading is staged in a way we can cache it
• review stages to reduce package installation duplication

Success criteria:

• 0 package installation on a warm cache
• cut cold cache package install time by 50% (XXX not realistic?)

GitHub repositories

Clones from GitHub do clock in at 1GB on a cold cache. Containerd alone counts for more than half of it (at 160MB+ x4).

Hopefully, on a warm cache it is practically non-existent.

But then, this is ridiculous.

Actions:

• shallow clone

Success criteria:

• reduce network traffic from cloning by 80%

Go modules

At 750+MB and over 4 minutes, this is the number one speed bottleneck on a cold cache.

On a warm cache, it is still over 150MB and 30+ seconds.

In and of itself, this is hard to reduce, as we need these...

Actions:

• we could cache the module download location to reduce round-trips on modules that are shared across different projects
• we are likely installing nerdctl modules six times - (once per architecture during the build phase, then once per ubuntu version and architecture during the tests runs (this is not even accounted for in the audit above)) - it should only happen twice (once per architecture)

Success criteria:

• achieve 20% reduction of total time spent downloading go modules

Other downloads

1. At 500MB+ and 30 seconds, storage.googleapis.com is serving a SINGLE go module that gets special treatment: klauspost/compress. This module is very small, but does serve along with it a very large testdata folder. The fact that nerdctl downloads its module multiple times is further compounding the effect.

2. the golang archive is downloaded multiple times - it should be downloaded only once per run, and only on a cold cache

3. some of the binary releases we are retrieving are also being retrieved with a warm cache, and they are generally quite large. We could consider building certain things from source instead, and in all cases ensure that we are only downloading with a cold cache.

Success criteria:

• 0 static downloads on a warm cache
• cut extra downloads by 20%

Duration

Unscientific numbers, per pipeline

dependencies, no cache:

• 224 seconds total
• 53 seconds exporting cache

dependencies, with cache:

• 12 seconds

test-integration, no cache:

• 282 seconds

Caching

Number of layers in cache:

after dependencies stage: 78
intermediate size: 1.5G
after test-integration stage: 118
total size: 2.8G

Generic considerations

Caching compression

This is obviously heavily dependent on the runner properties.

With local cache, on high-performance IO (laptop SSD), zstd is definitely considerably better (about twice as fast).

With GHA, the impact is minimal, since network IO is heavily dominant, but zstd still has the upper hand with regard to cache size.

Output

Loading image into the Docker store comes at a somewhat significant cost. It is quite possible that a significant performance boost could be achieved by using buildkit containerd worker and nerdctl instead.