Back to Dagger

LLB -> Dagger ID Whiteboard

util/llbtodagger/WHITEBOARD.md

0.21.073.1 KB
Original Source

LLB -> Dagger ID Whiteboard

Last updated: 2026-02-28

Explicit Goal and Hard Requirements (Do Not Forget)

  • Goal: implement a utility library under ./util/llbtodagger that converts BuildKit LLB + Docker image metadata into a Dagger *call.ID.
  • Required input/output:
    • Input: *pb.Definition (BuildKit LLB definition).
    • Input: *dockerspec.DockerOCIImage (Docker OCI image metadata/config, e.g. from dockerfile2llb).
    • Output: *call.ID (must reference Dagger API calls/fields).
    • Output object type should be Container for Docker-build-derived LLB.
  • Primary parsing utility to use: engine/buildkit/llbdefinition.go (DefToDAG / OpDAG).
  • We should support "almost all" LLB operations where there is a meaningful Dagger API representation.
  • If an LLB construct cannot be represented faithfully in Dagger API ID form, return an error immediately.
  • This file (WHITEBOARD.md) is the persistent task log for:
    • requirements,
    • plan/checklist,
    • gotchas,
    • progress updates,
    • unresolved mismatch cases.

Current Working Assumptions

  • Primary target is LLB produced in Dagger engine workflows; support for arbitrary external LLB is best-effort.
  • We should prioritize deterministic and explainable mappings over trying to "guess" hidden runtime state.

Conversion Principles (Do Not Violate)

  • Fail fast on unsupported or imperfect mappings.
    • If an op or field cannot be represented faithfully yet, return an error.
    • Do not add fallback behavior yet.
    • Do not silently skip unsupported data.
  • Do not implement any custom-op parsing/decoding path.
    • dagger.customOp handling is intentionally out of scope.
    • Custom ops are being removed in parallel work; this library should not depend on them.
  • Focus scope: convert LLB ops to Dagger IDs.
    • Do not parse Dockerfiles in this package.
    • This package exists to map already-produced LLB into Dagger IDs.
  • For Docker-build-derived LLB, prefer returning a Container-typed ID and preserve container metadata where representable from LLB.

Progress Snapshot

  • Read cache-expert core references and debugging guidance.
  • Reviewed engine/buildkit/llbdefinition.go (OpDAG, op type helpers).
  • Reviewed dagql/call/id.go construction/digest rules.
  • Reviewed schema entry points for container, directory, file, git, http, host.
  • Initialized this whiteboard with explicit goal/requirements/input-output.
  • Implement package scaffolding in util/llbtodagger.
  • Implement first conversion library pass + initial tests.
  • Expanded test matrix for source/exec/file happy paths + unsupported/imperfect mappings.
  • Added package docs with explicit non-goals.
  • Added Dockerfile-driven integration tests (dockerfile2llb -> pb.Definition -> DefinitionToID) for simple and complex supported cases.
  • Updated converter/tests so final IDs are container-typed for Docker-build-derived outputs.
  • Updated converter/tests so Dockerfile metadata is applied via image-config input where representable.
  • Added core/integration end-to-end suite validating Dockerfile -> dockerfile2llb -> DefinitionToID -> LoadContainerFromID execution path.
  • Added readonly bind-mount support + non-sticky exec mount cleanup semantics for converted ExecOp mount handling.
  • Added support for copy to explicit file destination paths by mapping single-file copies to withFile.
  • Added support for group-only chown (--chown=:gid) in converted file ops.
  • Added support for named user/group chown on copy actions when container context is available.
  • Added internal container metadata mapping for Docker image config fields: healthcheck, onbuild, shell, volumes, stop signal.

Proposed Library Shape (Planning Draft)

  • Public entrypoint (exact naming TBD):
    • func DefinitionToID(def *pb.Definition, img *dockerspec.DockerOCIImage) (*call.ID, error)
      • pass img=nil when metadata input is unavailable.
  • Optional debug API (if needed):
    • func DefinitionToIDWithTrace(def *pb.Definition) (*call.ID, *Trace, error)
    • Trace is diagnostic only, not a fallback mechanism.
  • Internal core:
    • DAG parse: buildkit.DefToDAG(def)
    • Recursive mapper: mapOp(dag *buildkit.OpDAG) (*call.ID, error) with memoization by (opDigest, outputIndex).

Detailed Implementation Plan (Checklist)

Phase 1: Package and Contracts

  • Create package files:
    • util/llbtodagger/convert.go
    • util/llbtodagger/types.go
    • util/llbtodagger/ops_*.go (split by op class)
    • util/llbtodagger/convert_test.go
  • Finalize API surface:
    • strict primary API is (*call.ID, error)
    • optional trace API may exist for debugging only
  • Add package-level docs and explicit non-goals.

Phase 2: Mapping Strategy Core

  • Parse *pb.Definition into *buildkit.OpDAG.
  • Normalize root wrapper ops (top-level synthetic selector op with Op == nil case).
  • Add memoized recursive conversion of op DAG to call ID DAG.
  • Implement deterministic argument builder helpers using call.NewArgument + call.WithArgs.
  • Decide digest policy for synthesized IDs:
    • default recipe digest unless a custom digest is still semantically faithful.
    • if semantics would be lossy/imperfect, return error (do not synthesize).

Phase 3: Structural Op Mapping

  • Implement source op mapping:
    • docker-image:// -> query.container(...).from(address: ...) chain.
    • git:// -> query.git(url: ...).<ref/head/...>.tree(...) chain.
    • local:// -> query.host().directory(...) chain.
    • http:///https:// -> query.http(url: ...).
    • oci-layout:// currently returns explicit unsupported error.
    • blob:// is explicitly unsupported; returns error.
  • Implement ExecOp mapping to container API chain:
    • base/rootfs source mapping
    • mount mapping (withMountedDirectory, withMountedCache, withMountedTemp) for supported cases
    • env/user/cwd argument projection
    • final withExec(args: ...) and output mount projection
  • Implement FileOp mapping:
    • copy -> withDirectory mapping for supported cases
    • mkfile -> withNewFile
    • mkdir -> withNewDirectory
    • rm -> withoutFile
  • Implement MergeOp mapping (directory composition chain).
  • Implement DiffOp mapping (directory.diff(other: ...) style chain).
  • Implement BuildOp behavior:
    • explicitly unsupported; return error immediately.

Phase 4: Error Handling Framework (Fail-Fast)

  • Add stable error categories for unsupported/unfaithful mappings.
  • Include op digest + op type in returned errors for diagnosis.
  • Ensure converter aborts on first unsupported/imperfect mapping (no fallback path yet).

Phase 5: Test Matrix

  • Unit tests for empty/nil definition behavior.
  • Unit tests for each op family using synthetic llb definitions.
  • Tests that unsupported ops (BuildOp, blob://) return deterministic errors.
  • Tests that imperfect/unfaithful mappings return errors (no fallback).
  • Golden tests for deterministic ID encoding across runs.

Phase 6: Documentation and Developer UX

  • Add package README or doc comments explaining:
    • supported mappings,
    • explicit unsupported ops,
    • fail-fast behavior.
  • Keep this whiteboard updated after each implementation chunk.

Phase 6 Nice-To-Have Backlog

  • Add "debug mode" helper to print op->ID mapping trace.

Phase 7: Dockerfile->LLB Integration Test Matrix

  • Add a new dedicated unit test file for Dockerfile-driven conversion coverage.
  • Build a helper that runs:
    • Dockerfile bytes -> dockerfile2llb.Dockerfile2LLB
    • LLB state -> Marshal(...).ToPB()
    • PB definition + Docker image config -> DefinitionToID
  • Add simple Dockerfile cases (one capability at a time):
    • image source (FROM ...)
    • local source (COPY ... from main context)
    • exec (RUN ...)
    • file ops via Dockerfile instructions (COPY, WORKDIR, ENV)
    • remote HTTP source (ADD https://...)
    • remote git source (ADD https://...git)
  • Add complex Dockerfile cases combining supported features across multiple stages.
  • Add deterministic encoding assertions for Dockerfile-driven outputs.
  • Keep this whiteboard updated after each implementation chunk.

Phase 8: Container-Typed Output and Metadata Preservation

  • Update conversion flow so final DefinitionToID output is container-typed for Docker-build LLB.
  • Introduce two-input conversion API (pb.Definition + DockerOCIImage) for metadata-complete conversion.
  • Preserve metadata where representable from Docker OCI config (entrypoint/cmd/env/user/workdir/labels/exposed ports).
  • Update unit tests to assert container IDs and metadata-sensitive fields.
  • Document metadata not inferable from pb.Definition alone.

Phase 9: End-to-End Integration Tests (Core Integration Suite)

  • Add a new integration test file under ./core/integration with suite name LLBToDaggerSuite.
  • Add helper pipeline in integration tests:
    • Dockerfile string -> dockerfile2llb.Dockerfile2LLB
    • LLB state -> Marshal(...).ToPB()
    • PB definition + Docker image config -> llbtodagger.DefinitionToID
    • ID encode -> SDK load path
  • Add execution checks that exercise loaded objects (not only IDs):
    • call Sync and/or WithExec/Stdout
    • inspect files/contents produced by Dockerfile instructions
    • validate expected side effects from RUN, COPY, ADD, WORKDIR, ENV
  • Add at least one complex multi-stage Dockerfile end-to-end case.
  • Add deterministic checks for ID encoding in integration path where stable.
  • Add export-based metadata verification in e2e coverage:
    • export resulting container image (local temp dir or test registry path),
    • inspect OCI image config (config JSON),
    • assert expected metadata fields are present/unchanged (especially metadata not directly observable via runtime exec behavior).
  • Keep this whiteboard updated after each implementation chunk.

Phase 10: withDirectory Permissions Support (copy mode override)

  • Extend engine API to support explicit permissions on directory copy operations.
    • Add permissions argument to Directory.withDirectory.
    • Add permissions argument to Container.withDirectory for API parity.
    • Thread the new argument through schema args -> core methods -> copy implementation.
  • Add dedicated integration coverage in core/integration/directory_test.go:
    • Verify Directory.withDirectory(..., permissions: ...) applies mode recursively to copied tree.
    • Keep coverage separate from LLBToDaggerSuite.
  • Add LLBToDaggerSuite integration coverage for COPY --chmod:
    • dedicated COPY --chmod case.
    • complex multi-op Dockerfile case also validates chmod-preserved output permissions.
  • Update util/llbtodagger copy mapping:
    • Map pb.FileActionCopy.Mode to the new withDirectory(permissions: ...) API instead of erroring.
  • Regenerate Go SDK after schema API changes.
    • Required command: dagger -y call -m ./toolchains/go-sdk-dev generate
  • Update unsupported catalogs/checklists to remove copy mode override once implemented.

Phase 11: Read-Only Bind Mount Support + Non-Sticky Exec Mount Semantics

  • Extend container API for read-only directory mounts.
    • Add optional readOnly (or readonly, schema-consistent naming) argument to Container.withMountedDirectory.
    • Thread arg through schema -> core WithMountedDirectory(..., readonly bool) call.
    • Regenerate Go SDK after schema change (dagger -y call -m ./toolchains/go-sdk-dev generate; use --workspace=. in this worktree if needed).
  • Update llbtodagger ExecOp bind-mount conversion for read-only mounts.
    • Remove fail-fast rejection for m.Readonly bind mounts.
    • Map pb.Mount{MountType=BIND, Readonly=true} to withMountedDirectory(..., readOnly: true).
  • Enforce BuildKit-style mount lifetime semantics in converter output.
    • Ensure mounts introduced for an ExecOp do not leak/stick into subsequent execs.
    • Add withoutMount(path: ...) cleanup in the generated chain where needed so post-exec state matches BuildKit behavior.
    • Keep ordering deterministic (mount application, exec, cleanup, and output projection ordering).
  • Add/adjust unit tests in util/llbtodagger.
    • Replace readonly-bind "unsupported" expectation with positive mapping assertions.
    • Add a multi-exec synthetic LLB test that fails if mount stickiness leaks across exec boundaries.
    • Assert generated ID chain contains expected withMountedDirectory + cleanup structure for sticky/non-sticky semantics.
  • Add integration coverage.
    • Add llbtodagger Dockerfile-driven integration case for RUN --mount=type=bind,readonly conversion/runtime behavior.
    • Add llbtodagger Dockerfile-driven integration case with two RUN steps: mount used in first RUN, then explicitly absent in second RUN (non-sticky check).
    • Add direct container API integration coverage (outside llbtodagger) validating withMountedDirectory(readOnly: true) behavior.
  • Validation + whiteboard bookkeeping.
    • Run focused unit tests: go test ./util/llbtodagger.
    • Run focused integration tests using the debugging.md-prescribed workflow.
    • Remove "readonly bind mounts unsupported" entries from unsupported catalogs after implementation lands.

Phase 12: Group-Only chown Support

  • Support group-only ownership mapping in llbtodagger file actions.
    • Map group-only chown to Dagger owner string using explicit UID (0:<gid> baseline) instead of erroring.
    • Preserve current fail-fast behavior for unsupported named-user/group variants that still cannot be represented.
  • Add unit coverage.
    • Add focused tests for chown-owner normalization helper behavior for group-only inputs.
    • Add FileOp conversion unit assertions for group-only chown on supported copy paths.
  • Add Dockerfile-driven conversion unit coverage.
    • Add COPY --chown=:<gid> conversion test and assert owner field in emitted ID chain.
  • Add integration coverage.
    • Add llbtodagger integration test using Dockerfile with group-only chown and assert resulting uid:gid on copied artifact.
  • Validation + catalog updates.
    • Run go test ./util/llbtodagger.
    • Run focused core/integration llbtodagger tests via debugging.md command shape.
    • Remove/update "group-only chown unsupported" entries after implementation lands.

Phase 13: Named User/Group chown via Container-Aware FileOp Mapping

  • Planning + guardrails.
    • Keep fail-fast behavior for any named-ownership case that cannot be represented faithfully.
    • Scope to Dockerfile-relevant COPY/ADD --chown=<name> path first; do not broaden with fallback heuristics.
  • Converter support.
    • Accept non-empty UserOpt_ByName values in chownOwnerString normalization.
    • Detect owner strings that require name resolution (user/group names vs numeric ids).
    • For copy actions with named ownership, emit container-level withFile / withDirectory calls (with rootfs sync) so Dagger resolves names through container /etc/passwd and /etc/group.
    • Keep numeric/group-only paths on the existing directory-level fast path.
    • Return explicit unsupported errors when named ownership is requested but no container context is available.
  • Tests.
    • Add unit tests for owner normalization that include named-user and named-group cases.
    • Add FileOp conversion tests asserting named-chown copy emits container-level calls.
    • Add Dockerfile-driven conversion tests for COPY --chown=<name>.
    • Add integration tests that create users/groups in image and assert final file ownership after LoadContainerFromID.
  • Validation + bookkeeping.
    • Run go test ./util/llbtodagger.
    • Run focused core/integration llbtodagger tests following skills/cache-expert/references/debugging.md.
    • Update unsupported catalogs to remove named-chown entries that are now supported and keep any still-unsupported nuances explicit.

Phase 14: Internal Container API for OCI Metadata Fields

  • Add internal-only container schema API (underscore-prefixed) for setting OCI config metadata not currently exposed in public SDK methods.
    • Confirm API naming/shape (_...) so it is callable from raw ID construction but intentionally not codegen'd for SDKs.
    • Keep scope limited to llbtodagger conversion needs; avoid broad public-surface changes.
  • Add arguments covering the currently unsupported Dockerfile-relevant metadata:
    • healthcheck
    • onBuild
    • shell
    • volumes
    • stopSignal
  • Define GraphQL-friendly argument encodings for non-scalar fields.
    • Representation for healthcheck: JSON-encoded dockerspec.HealthcheckConfig.
    • Representation for volumes: sorted []string of volume paths (re-hydrated to map/set in schema resolver).
  • Thread schema args into core container mutation logic and ensure Container.Config is updated deterministically.
  • Keep existing behavior unchanged unless internal API is explicitly used.
  • Regenerate Go SDK if schema generation is impacted (even though underscore APIs are internal-only).
    • Command: dagger -y call -m ./toolchains/go-sdk-dev generate
  • Update llbtodagger metadata conversion to use the new internal API instead of fail-fast for these fields.
  • Add/expand tests:
    • unit tests for the internal schema resolver path and config mutation behavior.
    • llbtodagger unit tests that verify IDs include internal metadata call when these fields are present.
    • integration test coverage (llbtodagger and/or container-focused) validating loaded container behavior/config where observable.
    • include export-and-inspect assertions for OCI config fields (healthcheck/onbuild/shell/volumes/stopSignal) so metadata is validated directly, not only via runtime behavior.
  • Update unsupported catalogs after implementation lands, removing items no longer unsupported.

Phase 15: Hard-Cutover dockerBuild Integration

  • Scope and entrypoints.
    • Cut over directory.dockerBuild to the new LLB->ID pipeline while keeping API shape unchanged.
    • Cut over deprecated container.build via the same Container.Build implementation path.
  • Replace Container.Build internals with llbtodagger pipeline.
    • Read Dockerfile bytes from dockerfileDir.
    • Convert contextDir.LLB to llb.State and call dockerfile2llb.Dockerfile2LLB.
    • Marshal returned state to *pb.Definition.
    • Convert definition+image metadata to *call.ID via llbtodagger.DefinitionToID.
    • Load resulting ContainerID through DAGQL/server load path and return resulting *core.Container.
  • Preserve old implementation as commented reference blocks in Container.Build for this transition only.
    • Comment out legacy bk.Solve path.
    • Comment out legacy WithSecretTranslator / WithSSHTranslator setup.
    • Comment out legacy local DefToDAG walk/marshal mutation path.
    • Put this exact TODO above each commented block:
      • TODO: remove commented code once fully replaced, just a reference on how it used to work for now
  • Deliberate first-iteration behavior (accepted regressions for now).
    • Secret/SSH dockerBuild tests are expected to fail in this phase; do not block cutover on them.
    • Do not implement secret handling in this phase.
    • Do not implement SSH handling in this phase.
    • Do not support remote frontend syntax pragma behavior (#syntax=...) in this phase.
  • Follow-up TODOs to keep visible for next iteration.
    • TODO: restore/port noInit behavior in cutover path.
    • TODO: implement secret mount/env support in llbtodagger exec conversion.
    • TODO: implement SSH mount support in llbtodagger exec conversion.
    • TODO: decide and implement SSH ID mapping semantics.
  • Validation and bookkeeping.
    • Run focused dockerBuild integration tests (expect secret/ssh failures in this phase).
    • Keep WHITEBOARD.md updated with exactly which dockerBuild tests fail post-cutover.
    • 2026-02-28 focused runs:
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestDockerBuild' -> fail
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestBuildMergesWithParent' -> pass
    • Failures observed in TestDockerfile/TestDockerBuild:
      • with_syntax_pragma, with_old_syntax_pragma: expected for Phase 15 (#syntax=... unsupported in cutover path).
      • secret/ssh cases (with_build_secrets*, with_unknown_build_secrets*, TestDockerBuildSSH/*): expected for Phase 15 (secret/ssh unsupported).
      • many baseline copy-path cases (default_Dockerfile_location, custom/subdirectory Dockerfile location, .dockerignore compatibility, with_build_args, prevent_duplicate_secret_transform) failed due:
        • llbtodagger: unsupported op "file.copy": copy without createDestPath is unsupported.
      • This file.copy createDestPath=false gap was the major non-secret/non-ssh blocker for broad dockerBuild cutover coverage (resolved in Phase 16).

Phase 16: file.copy createDestPath=false Support

  • Goal:

    • Support BuildKit FileActionCopy.CreateDestPath=false semantics in llbtodagger conversion.
    • Unblock dockerBuild cutover tests currently failing on this unsupported copy variant.
  • Semantics to preserve (BuildKit reference behavior):

    • When createDestPath=false, copy should fail if destination parent path does not exist.
    • BuildKit checks destination parent existence before copy (internal/buildkit/solver/llbsolver/file/backend.go, docopy).
  • Initial gap (before this phase):

    • llbtodagger errored on all createDestPath=false.
    • Existing Dagger Directory.withDirectory / Directory.withFile implementations always created parent directories (MkdirAll path), so they could not express createDestPath=false faithfully.
  • Decision:

    • Selected approach: Option B (full-fidelity, internal hidden args).
  • Option B (selected):

    • Add hidden internal args (internal:"true") to existing copy schema args using inverted naming: doNotCreateDestPath.
    • Keep default doNotCreateDestPath=false so current Dagger behavior (create destination parent paths) remains unchanged.
    • Wire llbtodagger to set doNotCreateDestPath=true via raw ID construction when LLB has createDestPath=false.
    • Keep public SDK surface unchanged (internal APIs callable via raw ID only).
    • Rationale: faithful semantics and broad dockerBuild compatibility without changing public SDK behavior.
  • Option B execution plan:

    • 16.1 Add hidden internal args (internal:"true") to existing directory/container copy schema args:
      • doNotCreateDestPath bool with default false.
    • 16.2 Implement core behavior:
      • when doNotCreateDestPath=false, keep current behavior (create destination parent path as today).
      • when doNotCreateDestPath=true, verify destination parent exists and return error if missing.
    • 16.3 Update llbtodagger applyCopy:
      • map CreateDestPath=true as today.
      • map CreateDestPath=false by setting hidden internal arg doNotCreateDestPath=true in call IDs.
    • 16.4 Tests:
      • unit: ID construction for createDestPath=false copy path.
      • integration: successful copy when parent exists.
      • integration: expected error when parent missing.
      • rerun focused dockerBuild integration tests from debugging.md command shape.
    • 16.5 Bookkeeping:
      • update unsupported catalog entry for createDestPath=false after landing.
    • 2026-02-28 validation runs:
      • dagger -y call -m ./toolchains/go-sdk-dev generate -> pass (no changes to apply)
      • go test ./util/llbtodagger ./core ./core/schema -> pass
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestLLBToDagger/TestLoadContainerFromConvertedIDCopyDoNotCreateDestPath' -> pass
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestLLBToDagger' -> pass
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestDockerBuild' -> fail (expected for known unsupported areas); createDestPath=false unsupported error is gone.
        • Remaining failures include syntax pragma, secret/ssh, and several context-copy cases now failing with missing-path errors (for example stat /src: no such file or directory, stat /subcontext: no such file or directory).
        • Empirical debug (TestDockerfile/TestDockerBuild/default_Dockerfile_location):
          • failure now occurs while loading converted ID at directory(path: "/src") inside source resolution (stat /src: no such file or directory).
          • this directory(path: "/src") comes from Directory.StateWithSourcePath() copy-shim (llb.Copy(dirSt, dir.Dir, ".", CopyDirContentsOnly:true) with dir.Dir=/src).
          • indicates current local/context conversion path is still mismatched for these dockerBuild contexts; after removing prior createDestPath=false blocker, this is the next concrete failure to address.

Phase 17: Sentinel MainContext + Structural Context Rebinding (No Directory.State* Dependency)

  • Goal:

    • Remove dockerBuild reliance on Directory.State() / Directory.StateWithSourcePath() for Dockerfile context injection.
    • Preserve Dockerfile2LLB's path/selector semantics while binding context reads to the real Dagger directory ID.
    • Fix current stat /src / stat /subcontext class failures without text-based ID hacks.
  • Approach:

    • Feed dockerfile2llb a synthetic, deterministic sentinel MainContext local source state.
    • Extend llbtodagger conversion so sentinel local-source ops are rebound to the actual Dagger context directory ID provided by caller.
    • Perform replacement structurally in ID construction/conversion logic; never do string replacement on encoded/display IDs.
  • Checklist:

    • Define sentinel local-source identity constants (name/shared-key marker) in the dockerBuild conversion path.
    • Build sentinel llb.State for ConvertOpt.MainContext (valid non-nil output graph; deterministic marker).
    • Update core.Container.Build to stop calling contextDir.State* for Dockerfile2LLB MainContext.
    • Extend llbtodagger API to accept context-rebinding input (actual Dagger context Directory ID/call ID) alongside def + img.
    • Add strict validation: if sentinel local source appears in LLB and no context rebinding value is provided, return fail-fast error.
    • Implement sentinel detection in local-source conversion (identifier + attrs match), and map it to provided context directory ID.
    • Keep non-sentinel local-source behavior unchanged.
    • Ensure selector/include/exclude/copy semantics emitted by Dockerfile2LLB still apply on top of rebound context directory.
    • Add llbtodagger unit tests:
      • sentinel local source maps to provided directory ID.
      • missing rebinding input for sentinel returns explicit error.
      • sentinel marker does not leak into final emitted ID display/encoding.
    • Add/adjust dockerBuild integration coverage:
      • TestDockerfile/TestDockerBuild/default_Dockerfile_location passes.
      • subdirectory_with_default_Dockerfile_location passes.
      • custom Dockerfile location variants pass.
    • Re-run focused integration suite using debugging.md command shape and log outcomes in this whiteboard.
  • 2026-02-28 validation runs:

    • go test ./util/llbtodagger ./core ./core/schema -> pass.
    • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestDockerBuild/default_Dockerfile_location' -> pass.
    • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestDockerBuild/(custom_Dockerfile_location|subdirectory_with_default_Dockerfile_location|subdirectory_with_custom_Dockerfile_location)' -> pass.
    • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestDockerBuild' -> fail (expected+known unsupported plus one newly surfaced copy-semantics issue):
      • expected unsupported: ssh mounts, secret mounts, remote syntax pragma, builtin secret-mount exec path.
      • newly surfaced non-context issue (resolved): onbuild_command_is_published exposed COPY source-missing behavior mismatch (include-filter no-op vs BuildKit error).
    • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestDockerBuild/onbuild_command_is_published' -> pass after strict source-path existence wiring (requiredSourcePath internal arg on withDirectory path).
  • Follow-up tie-in (future branch):

    • This phase is the prerequisite for removing Directory.State() usage from dockerBuild path in the branch where LLB is no longer a general Dagger runtime dependency.

Phase 18: Secret Env + Secret Mount Support (dockerBuild Cutover)

  • Goal:

    • Support Dockerfile/LLB secret environment and secret mount semantics in hard-cutover dockerBuild through llbtodagger conversion.
    • Keep strict fail-fast behavior for unsupported/imperfect cases.
  • Scope:

    • In scope: ExecOp.Secretenv and ExecOp mounts with MountType_SECRET.
    • Out of scope for this phase: MountType_SSH (stays unsupported until a dedicated follow-up phase).
  • Design checklist:

    • Extend llbtodagger conversion options with secret resolution input.
      • Add DefinitionToIDOptions field for mapping LLB secret IDs -> Dagger Secret call IDs.
      • Keep converter independent from core package types; use *call.ID mapping data only.
    • Wire dockerBuild secret inputs into llbtodagger options in core.Container.Build.
      • Build name->secret-ID mapping from secrets []dagql.ObjectResult[*Secret] + secretStore.GetSecretName(...).
      • Pass map into DefinitionToIDWithOptions(...).
    • Implement ExecOp.Secretenv conversion.
      • Map to container.withSecretVariable(name: ..., secret: ...).
      • Required/optional behavior:
        • If secret ID is mapped: emit withSecretVariable.
        • If secret ID missing and optional=true: skip.
        • If secret ID missing and optional=false: return explicit error.
    • Implement MountType_SECRET conversion.
      • Map to container.withMountedSecret(path: ..., source: ..., owner: ..., mode: ...).
      • Map owner from uid/gid to owner string (<uid>:<gid>).
      • Preserve file mode semantics via mode arg.
      • Required/optional behavior:
        • If secret ID is mapped: emit withMountedSecret.
        • If secret ID missing and optional=true: skip.
        • If secret ID missing and optional=false: return explicit error.
    • Preserve BuildKit per-RUN non-sticky behavior inside converted exec chains.
      • Secret envs are cleaned with withoutSecretVariable.
      • Secret mounts are cleaned with withoutMount(path).
      • core.Container.WithoutMount also removes secret mounts with matching MountPath.
    • Keep deterministic cleanup ordering and dedupe cleanup entries.
      • Stable order for secret env + mount cleanup calls.
      • Duplicate cleanup emission avoided.
    • Preserve legacy dockerBuild returned-container secret behavior.
      • After loading converted container ID, clone and append named secret mounts on container.Secrets (legacy-compatible persistence on returned container object).
      • Clone-before-mutate to avoid cache object corruption.
  • Test checklist:

    • llbtodagger unit tests (util/llbtodagger/convert_test.go):
      • Secret env mapping emits withSecretVariable + cleanup.
      • Secret mount mapping emits withMountedSecret + cleanup.
      • Missing required secret returns explicit error.
      • Missing optional secret is skipped (no emitted secret calls).
      • Multiple secrets in one exec preserve deterministic ordering.
    • Dockerfile-driven converter tests (util/llbtodagger/dockerfile_convert_test.go):
      • RUN --mount=type=secret,id=...,env=... converts as expected.
      • RUN --mount=type=secret,id=... mount path/mode/owner mapping.
      • Optional unknown secret does not fail conversion path.
      • Required unknown secret yields explicit conversion error.
    • Core integration tests (core/integration):
      • Add/extend llbtodagger e2e test in llbtodagger_test.go for secret env + secret mount behavior.
      • Re-run TestDockerfile/TestDockerBuild secret cases and make builtin frontend path pass:
        • with_build_secrets
        • with_unknown_build_secrets
      • Add explicit non-sticky validation across two RUNs for secret mount/env.
      • Keep #syntax=... remote frontend expectations unchanged for now (still unsupported by cutover path).
    • Validation command shape:
      • Follow skills/cache-expert/references/debugging.md for integration test runs.
      • Use sub-agents for longer integration test execution/log parsing when iterating.
  • Completion notes:

    • with_build_secrets and with_unknown_build_secrets now pass on dockerBuild hard-cutover path.
    • Converted RUN --mount=type=secret/env=... behavior is non-sticky per-run via cleanup calls.
    • Returned container keeps legacy-compatible named secret entries for dockerBuild output behavior.
  • Post-landing bookkeeping:

    • Remove secret-env/secret-mount entries from Unsupported and relevant to Dockerfile-generated LLB.
    • Keep SSH unsupported entry in place until SSH phase lands.
    • Update Current Explicit Unsupported Cases and nuanced catalogs accordingly.

Phase 19: SSH Socket Mount Support (dockerBuild Cutover)

  • Goal:

    • Support Dockerfile/LLB SSH socket mount semantics (RUN --mount=type=ssh) in hard-cutover dockerBuild through llbtodagger conversion.
    • Keep fail-fast behavior for unsupported/imperfect cases.
  • Scope:

    • In scope: ExecOp mounts with MountType_SSH.
    • Out of scope for this phase: git source custom SSH socket naming nuances beyond current Dockerfile path requirements.
  • Design checklist:

    • Extend llbtodagger conversion options with SSH socket resolution input.
      • Add DefinitionToIDOptions field mapping LLB SSH IDs -> Dagger Socket call IDs.
      • Keep converter independent from core package types; mapping stays *call.ID.
    • Wire dockerBuild SSH input into llbtodagger options in core.Container.Build.
      • Remove current hard error dockerBuild SSH mounts are not supported....
      • Map dockerBuild SSH argument to LLB SSH IDs expected by Dockerfile frontend (including default/empty ID path via fallback mapping).
      • Define and implement mapping semantics for non-default SSH IDs with single SSH input: empty-key mapping acts as default for any unmatched SSH ID (legacy translator-compatible behavior).
    • Implement MountType_SSH conversion in util/llbtodagger/exec.go.
      • Map to container.withUnixSocket(path: ..., source: ..., owner: ...).
      • Map owner from uid/gid to owner string (<uid>:<gid>).
      • Required/optional behavior:
        • If socket ID is mapped: emit withUnixSocket.
        • If socket ID missing and optional=true: skip.
        • If socket ID missing and optional=false: return explicit error.
    • Preserve BuildKit per-RUN non-sticky behavior for SSH mounts.
      • Cleanup converted exec chain with withoutUnixSocket(path) after withExec.
      • Keep cleanup deterministic and deduped.
    • Decide on SSH mount mode handling.
      • Current withUnixSocket API has no explicit mode control.
      • For now, fail fast on unsupported non-default mode combinations.
    • Preserve cache safety when mutating loaded container objects (clone before mutation where needed).
  • Test checklist:

    • llbtodagger unit tests (util/llbtodagger/convert_test.go):
      • SSH mount mapping emits withUnixSocket + cleanup.
      • Missing required SSH socket returns explicit error.
      • Missing optional SSH socket is skipped.
      • Deterministic ordering/dedupe for repeated SSH mounts.
    • Dockerfile-driven converter tests (util/llbtodagger/dockerfile_convert_test.go):
      • RUN --mount=type=ssh converts as expected.
      • Optional unknown SSH ID does not fail conversion path.
      • Required unknown SSH ID yields explicit conversion error.
    • Core integration tests (core/integration):
      • Re-enable/target TestDockerBuildSSH/* coverage on cutover path.
      • Add llbtodagger integration coverage in llbtodagger_test.go for SSH mount behavior + non-sticky cleanup.
      • Validate mounted SSH socket behavior functionally (socket available during mounted RUN, absent in next RUN).
    • Validation command shape:
      • Follow skills/cache-expert/references/debugging.md exactly.
      • Use sub-agents for longer integration test runs/log parsing.
  • Completion notes:

    • Converter now supports MountType_SSH via withUnixSocket + withoutUnixSocket cleanup.
    • dockerBuild hard-cutover path now passes SSH socket mappings into llbtodagger and supports Dockerfile RUN --mount=type=ssh.
    • Empty-key SSH mapping fallback allows one provided dockerBuild SSH socket to satisfy default or named SSH IDs (matching legacy translator behavior).
    • Remote syntax frontend behavior for known Dockerfile pragmas is now handled in Phase 20.

Phase 20: Syntax Pragma Relaxation

  • Goal:

    • Make dockerBuild cutover path laxer for Dockerfile syntax pragmas.
    • Allow known Dockerfile syntax pragma values and ignore them (continue conversion/build normally).
    • Fail fast only when pragma specifies an unknown/unsupported frontend reference.
  • Scope:

    • In scope: #syntax=... handling in core.Container.Build preflight checks around Dockerfile bytes.
    • Out of scope: actually delegating to remote frontends or supporting non-Dockerfile frontend behavior.
  • Design checklist:

    • Replace current blanket hard-error on detected syntax pragma with allowlist behavior.
    • Define "known Dockerfile syntax pragma" matcher.
      • Accept canonical Dockerfile frontend references we explicitly recognize (Docker/Moby Dockerfile frontend refs).
      • Treat pinned tag/digest variants of recognized Dockerfile frontends as allowed.
    • On allowed syntax pragma, do not alter conversion flow:
      • Continue using local dockerfile2llb + llbtodagger conversion path.
      • Do not fetch or dispatch remote frontend.
    • On unknown syntax pragma:
      • Return explicit error that includes the pragma value and states it's unsupported in cutover mode.
    • Keep behavior deterministic and obvious in logs/errors (no silent fallback beyond allowed-ignore path).
  • Test checklist:

    • Unit/functional coverage around syntax detection helper:
      • known Dockerfile syntax pragma -> allowed
      • unknown syntax pragma -> explicit error
    • Integration coverage (core/integration/dockerfile_test.go):
      • with_syntax_pragma passes on cutover path.
      • with_old_syntax_pragma passes on cutover path.
      • Add/keep a negative case for unknown syntax pragma that still errors clearly.
    • Validation command shape:
      • Follow skills/cache-expert/references/debugging.md command conventions.
  • Completion notes:

    • Known Dockerfile syntax pragmas (for example docker/dockerfile:*) are now accepted and ignored by the hard-cutover path.
    • Unknown syntax pragmas now fail with explicit unsupported-frontend errors.
    • Existing syntax pragma integration coverage is passing again, including SSH remote-frontend subtests that use Dockerfile syntax pragma.
  • Post-landing bookkeeping:

    • Remove syntax-pragma item from current unsupported catalog.
    • Update related historical notes in this whiteboard to reflect the new allowed-ignore policy.

Phase 21: Public query.http Checksum Enforcement

  • Goal:

    • Add checksum enforcement support to Dagger's public query.http API.
    • Allow callers to provide an expected digest and fail if downloaded content does not match.
  • Scope:

    • In scope: query.http schema/API and HTTP download execution path in core.
    • Out of scope: llbtodagger ADD --checksum support (separate track).
  • Design checklist:

    • Add a new public optional string arg checksum to query.http.
      • Add it in core/schema/http.go argument docs + httpArgs.
      • Keep it public (not internal-only).
    • Parse and validate checksum input.
      • Parse with digest.Parse(...).
      • Return explicit error for invalid checksum format.
    • Enforce checksum match in download path.
      • Compare expected checksum vs actual downloaded content digest.
      • Return explicit mismatch error containing expected + actual digest values.
      • Ensure enforcement applies for both fresh download and cache-hit/304 paths.
    • Include checksum in query.http resolver identity digest.
      • Add checksum argument into DagQL digest mixin / cache-key hash composition in core/schema/http.go (hashutil.HashStrings(...) path).
    • Regenerate Go SDK after schema API change.
      • Run: dagger -y call -m ./toolchains/go-sdk-dev generate
      • Worktree override required here: dagger -y call -m ./toolchains/go-sdk-dev --workspace=. generate
  • Test checklist:

    • Core unit/functional coverage:
      • valid matching checksum succeeds.
      • valid mismatched checksum fails with explicit mismatch error.
      • invalid checksum string fails early with parse/validation error.
    • Integration coverage (core/integration/http_test.go):
      • add happy-path checksum test with deterministic content.
      • add mismatch test.
      • add invalid-checksum input test.
    • Validation command shape:
      • follow skills/cache-expert/references/debugging.md format for integration runs.
  • Completion notes:

    • query.http now supports an optional public checksum argument and enforces digest match against downloaded content.
    • Resolver identity digest now mixes in the expected checksum value.
    • Go SDK regen required the whiteboard-documented worktree override (--workspace=.) to pick up local schema changes.
    • Focused integration coverage is passing with command shape from skills/cache-expert/references/debugging.md.

Phase 22: Local Source followPaths Support (Internal-Only Path)

  • Goal:

    • Support BuildKit local.followpaths emitted by Dockerfile->LLB conversion, without exposing this knob in public Dagger APIs.
    • Preserve current fail-fast posture for malformed attrs while adding faithful behavior for supported followPaths inputs.
  • Design principles:

    • Keep filesync core behavior simple and unchanged for all existing call sites.
    • Add an explicit, narrow, internal-only path that is only used when IDs include followPaths (llbtodagger conversion path).
    • No fallback behavior; malformed/invalid followPaths still return explicit conversion/runtime errors.
  • Implementation checklist:

    • llbtodagger mapping:
      • Parse pb.AttrFollowPaths JSON in local source conversion.
      • Remove current hard-error on non-empty followPaths.
      • Emit internal-only followPaths arg on host.directory(...) when present.
      • Keep strict errors for invalid JSON / unknown attrs.
    • Schema/internal API surface:
      • Add internal-only followPaths: [String!] arg to host.directory args struct (internal:"true").
      • Keep it hidden from generated SDK/public callers.
    • Host/filesync threading (explicit side path):
      • Thread followPaths through core.Host.Directory into filesync.SnapshotOpts.
      • Thread SnapshotOpts.FollowPaths through filesync snapshot/sync into:
        • remote import opts (engine.LocalImportOpts.FollowPaths)
        • local mirror filter construction (fsutil.FilterOpt.FollowPaths)
      • Keep existing include/exclude/gitignore behavior unchanged when followPaths is empty.
    • Cache identity parity:
      • Ensure followPaths participates in identity where relevant (import metadata + dagql call args) so cache correctness matches semantics.
    • Testing:
      • Update llbtodagger unit tests:
        • positive local source followPaths conversion case (ID contains internal arg)
        • malformed followPaths JSON fails fast
      • Add focused integration coverage for behavior:
        • symlink target inclusion case that requires followPaths expansion
        • command shape per skills/cache-expert/references/debugging.md
      • Keep existing non-followPaths imports unchanged (regression check).
  • Non-goals for this phase:

    • Do not expose followPaths in public SDK APIs.
    • Do not redesign filesync transport/protocol.
    • Do not broaden to unrelated local source attrs beyond current scope.
  • Completion notes:

    • local.followpaths now converts to hidden host.directory(followPaths: [...]) in llbtodagger IDs.
    • Host import path now threads followPaths explicitly through SnapshotOpts into remote and local filesync filters.
    • Filesync transport/protocol stayed unchanged; this is a narrow additive side path.
    • Focused coverage:
      • go test ./util/llbtodagger -run 'TestDefinitionToIDLocal(Source|FollowPaths|FollowPathsInvalidUnsupported)$' -count=1
      • go test ./engine/filesync -count=1
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestLLBToDagger/TestLoadContainerFromConvertedIDLocalFollowPaths' --parallel=1
  • Expected touchpoints (planning index):

    • util/llbtodagger/source.go
    • core/schema/host.go
    • core/host.go
    • engine/filesync/filesyncer.go
    • engine/filesync/remotefs.go
    • engine/filesync/localfs.go
    • tests in util/llbtodagger/* and core/integration/*

Phase 23: Hidden Archive Auto-Unpack Compatibility for File Copy

  • Goal:

    • Support BuildKit FileActionCopy.attemptUnpackDockerCompatibility semantics (Dockerfile ADD local archive behavior) without exposing new public SDK surface.
    • Preserve current user-facing API while enabling llbtodagger fidelity for Dockerfile-generated LLB.
  • Design principles:

    • Keep this as a narrow, explicit compatibility path behind internal-only args.
    • Match BuildKit behavior: attempt unpack first; if input is not an archive, fall back to normal copy.
    • No fallback to other unrelated behavior; malformed inputs still error.
  • Implementation checklist:

    • Schema/internal args:
      • Add hidden attemptUnpackDockerCompatibility bool arg to WithDirectoryArgs (internal:"true" default:"false").
      • Add hidden attemptUnpackDockerCompatibility bool arg to WithFileArgs (internal:"true" default:"false").
      • Keep these args hidden from public SDK callers (internal-only pattern).
    • Core directory/file copy implementation:
      • Thread internal arg through directorySchema.withDirectory -> core.Directory.WithDirectory.
      • Thread internal arg through directorySchema.withFile -> core.Directory.WithFile.
      • Implement unpack-attempt helper in core/directory.go copy path:
        • detect archive stream (gzip/bzip/xz/etc via decompressor + tar first header probe).
        • if archive: untar into destination with existing owner/permissions handling semantics.
        • if not archive: fall back to existing copy behavior.
      • Preserve existing semantics for doNotCreateDestPath and requiredSourcePath.
    • llbtodagger mapping:
      • Remove current hard-error on cp.AttemptUnpackDockerCompatibility.
      • Emit hidden attemptUnpackDockerCompatibility: true on generated withDirectory/withFile call IDs when LLB copy action sets it.
      • Keep strict errors for unrelated unsupported copy attrs.
    • Behavior and safety notes:
      • Ensure unpack path cannot escape destination root (path traversal hardening).
      • Ensure archive extraction path follows same ownership/chmod intent used by copy path where applicable.
      • Keep the feature Linux-first if platform nuances require; fail clearly where unsupported.
    • Tests:
      • Unit tests in util/llbtodagger/convert_test.go:
        • verify AttemptUnpackDockerCompatibility maps to hidden arg (instead of unsupported error).
        • keep non-attempt copy behavior unchanged.
      • Unit/integration tests in core:
        • positive: local tar archive is unpacked when hidden arg true.
        • fallback: non-archive file is copied as-is when hidden arg true.
        • regression: hidden arg false keeps current copy semantics.
      • Integration coverage in core/integration/llbtodagger_test.go:
        • Dockerfile case using local ADD archive produces expected extracted tree via converted ID.
        • command shape per skills/cache-expert/references/debugging.md.
  • Non-goals for this phase:

    • Do not expose a public API flag for unpack behavior.
    • Do not rework all copy semantics; only compatibility path for AttemptUnpackDockerCompatibility.
    • Do not add Dockerfile parsing logic in llbtodagger (LLB-driven only).

Phase 24: Conversion-Only Named Ownership for mkdir (Dockerfile WORKDIR path)

  • Goal:

    • Support named-owner FileActionMkDir in llbtodagger without adding new Dagger schema APIs.
    • Unblock Dockerfile cases where USER <name> precedes WORKDIR and BuildKit emits named-owner mkdir actions.
  • Approach (conversion-only):

    • Keep existing direct directory path for numeric/no-owner mkdir.
    • For named owner mkdir, route through container withDirectory so name resolution uses existing container /etc/passwd and /etc/group behavior.
  • Implementation checklist:

    • Converter mkdir path update:
      • Update applyMkdir to accept/return container context like copy path.
      • If owner is named and container context exists:
        • rebind container to current directory (withRootfs(directory: baseID)).
        • create synthetic single-empty-directory source (directory().withNewDirectory(...).directory(...)).
        • call container withDirectory(path, source, owner, permissions).
        • return updated container rootfs as next directory output and updated container context.
      • If owner is named and no container context exists, keep strict unsupported error.
    • Keep semantics for:
      • makeParents=true required (no behavior change for unsupported makeParents=false).
      • timestamp override remains unsupported.
    • Tests:
      • unit: named-owner mkdir without container context -> unsupported.
      • unit: named-owner mkdir with container context -> emits container withDirectory (not unsupported).
      • dockerfile unit: USER <name> + WORKDIR converts successfully via container-level path.
      • integration: Dockerfile with named user + WORKDIR yields expected directory ownership after load/sync.
      • integration command shape follows skills/cache-expert/references/debugging.md.
  • Non-goals:

    • Do not add new public or internal schema APIs for this phase.
    • Do not attempt to support non-canonical malformed LLB variants beyond Dockerfile-relevant behavior.

Phase 25: Exec Security Mode Mapping (RUN --security=insecure)

  • Goal:
    • Support Dockerfile/LLB exec security mode INSECURE by mapping it to Dagger withExec(insecureRootCapabilities: true).
  • Checklist:
    • Update exec conversion guard to allow pb.SecurityMode_INSECURE in addition to sandbox/default.
    • Emit withExec arg insecureRootCapabilities: true when exec security mode is insecure.
    • Keep unsupported error for any unknown/unhandled security mode enum values.
    • Add unit test coverage for insecure security mapping in util/llbtodagger/convert_test.go.
  • Notes:
    • This is a conversion-layer mapping only; entitlement/runtime enforcement remains the responsibility of the execution environment (same as normal withExec(insecureRootCapabilities: true)).

Phase 26: Internal-Only Exec Network none Mapping (RUN --network=none)

  • Goal:

    • Support Dockerfile/LLB exec network mode NONE by mapping to a hidden/internal-only withExec argument.
    • Keep public SDK/API surface unchanged while preserving Dockerfile fidelity for RUN --network=none.
  • Design:

    • Add an internal-only boolean arg on withExec for this narrow behavior (proposed: noNetwork).
    • Plumb it through schema -> core exec opts -> executor meta (pb.NetMode_NONE).
    • Keep HOST network mode out of scope for this phase only; track as a medium-priority follow-up.
  • Implementation checklist:

    • Schema + opts plumbing:
      • Add internal-only noNetwork bool to containerExecArgs / ContainerExecOpts path.
      • Ensure it is hidden from generated SDK/public callers (internal:"true").
      • Wire withExec resolver to pass this hidden arg into core exec opts.
    • Core exec behavior:
      • In core/container_exec.go metaSpec, set metaSpec.NetMode = pb.NetMode_NONE when noNetwork is true.
      • Preserve default behavior when noNetwork is false/unset.
    • llbtodagger mapping:
      • In util/llbtodagger/exec.go, accept pb.NetMode_UNSET and pb.NetMode_NONE.
      • Emit withExec(noNetwork: true) only for pb.NetMode_NONE.
      • Keep pb.NetMode_HOST unsupported (explicit error).
    • Tests:
      • Unit: exec with NetMode_NONE maps to withExec hidden noNetwork: true.
      • Unit: existing NetMode_HOST unsupported behavior remains.
      • Optional dockerfile conversion unit: RUN --network=none ... includes hidden arg.
      • Integration (focused): Dockerfile RUN --network=none behavior validates successfully through converted ID.
      • Run integration command using skills/cache-expert/references/debugging.md shape.
    • Whiteboard updates:
      • Remove/update RUN --network unsupported item once none is implemented.
      • Keep explicit note that host remains unsupported (and why, entitlement/runtime constraints).
  • Status note:

    • Initial focused integration run hit an engine panic in engine/buildkit/resources/netstat.go:113 due to nil network samples from none/host providers.
    • Fixed by normalizing nil network samples to zero-value samples in engine/buildkit/resources/netstat.go.
    • Re-ran focused integration command successfully:
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestLLBToDagger/TestLoadContainerFromConvertedIDRunNetworkNone'
  • Non-goals:

    • Do not implement RUN --network=host in this phase (tracked as medium-priority follow-up).
    • Do not expose network-mode knobs in public SDK withExec options.

Phase 27: Internal-Only Exec Network host Mapping (RUN --network=host) With Security-Setting Gate

  • Goal:

    • Support Dockerfile/LLB exec network mode HOST by mapping to a hidden/internal-only withExec argument.
    • Gate successful execution behind the same engine security setting used for insecureRootCapabilities (security.insecureRootCapabilities), via entitlement enforcement.
    • Keep public SDK/API surface unchanged.
  • Design:

    • Add a second internal-only exec arg (proposed: hostNetwork) in the withExec/ContainerExecOpts path.
    • Map hostNetwork: true to pb.NetMode_HOST in exec metadata.
    • Keep entitlement checks authoritative: host network only works when the engine grants the network-host entitlement, and entitlement grant is derived from the same security setting as insecure root capabilities.
    • Preserve noNetwork support from Phase 26 and reject contradictory combinations (hostNetwork && noNetwork).
  • Implementation checklist:

    • Schema + opts plumbing:
      • Add internal-only hostNetwork bool to ContainerExecOpts/withExec arg path (internal:"true").
      • Keep it hidden from generated SDK/public callers.
      • Validate hostNetwork and noNetwork are not both true (fail fast with clear error).
    • Core exec behavior:
      • In core/container_exec.go metaSpec, map hostNetwork: true -> metaSpec.NetMode = pb.NetMode_HOST.
      • Preserve existing insecure-root-capabilities mapping and noNetwork mapping behavior.
    • Entitlement/config gating:
      • Update engine entitlement setup to derive both security.insecure and network.host from security.insecureRootCapabilities policy.
      • Ensure security.insecureRootCapabilities=false denies host-network execs with entitlement error.
      • Keep existing execution-time entitlement check path (validateEntitlements) as enforcement point.
    • llbtodagger mapping:
      • In util/llbtodagger/exec.go, accept pb.NetMode_HOST in addition to UNSET/NONE.
      • Emit withExec(hostNetwork: true) for pb.NetMode_HOST.
      • Keep unknown network modes unsupported.
    • Tests:
      • Unit: exec conversion NetMode_HOST maps to hidden hostNetwork: true.
      • Unit: conflicting hostNetwork && noNetwork fails.
      • Dockerfile conversion unit: RUN --network=host ... includes hidden hostNetwork.
      • Integration: converted Dockerfile with RUN --network=host succeeds when security setting allows.
      • Integration: same Dockerfile fails with entitlement error when security.insecureRootCapabilities=false (use engineWithConfig in core integration suite).
      • Run integration command(s) using skills/cache-expert/references/debugging.md shape.
    • Whiteboard updates:
      • Remove/adjust medium-priority unsupported entry for RUN --network=host once implemented and gated.
      • Record explicit behavior: host-network support is policy-gated by security.insecureRootCapabilities.
  • Status note:

    • RUN --network=host now converts to hidden withExec(hostNetwork: true) and executes when policy allows.
    • Host network execution is policy-gated by security.insecureRootCapabilities through network.host entitlement setup.
    • Focused integration validation passed with expected deny-path error text when disabled (network.host is not allowed).
  • Non-goals:

    • Do not expose hostNetwork publicly in SDK-generated APIs.
    • Do not broaden support for other Dockerfile network modes in this phase.

Phase 28: dockerBuild(noInit) Support in Hard-Cutover Path

  • Goal:

    • Restore dockerBuild(noInit: true) behavior in the hard-cutover path (currently hard-erroring).
    • Preserve prior semantics: Dockerfile RUN execs run without injected init when noInit is set.
    • Keep default behavior unchanged when noInit is unset/false.
  • Design:

    • Thread a conversion option through llbtodagger.DefinitionToIDWithOptions that means: set withExec(noInit: true) on converted ExecOps.
    • Use that option only from Container.Build(..., noInit bool, ...) when caller requested noInit.
    • Do not add new public schema fields; reuse existing directory.dockerBuild / container.build noInit arg.
  • Implementation checklist:

    • llbtodagger options + mapping:
      • Add NoInit bool (or equivalently named) to DefinitionToIDOptions.
      • Thread option into converter state.
      • In convertExec, append withExec(noInit: true) when option is enabled.
      • Keep all existing exec mappings (insecureRootCapabilities, noNetwork, hostNetwork) intact and composable.
    • hard-cutover Build wiring:
      • Remove the current early hard-error: dockerBuild noInit is not supported in hard-cutover path yet.
      • Pass noInit from Container.Build into llbtodagger.DefinitionToIDWithOptions.
    • Tests:
      • Unit: llbtodagger exec conversion includes withExec(noInit: true) when option is set.
      • Unit: same conversion omits noInit when option is unset.
      • Dockerfile conversion unit: with option enabled, converted ID includes withExec(noInit: true) for RUN.
      • Integration: ContainerSuite/TestExecInit/disable automatic init in dockerfile build passes in hard-cutover mode.
      • Optional integration: llbtodagger E2E test asserts encoded ID includes withExec.noInit == true when converter option is enabled.
      • Run focused integration command(s) using skills/cache-expert/references/debugging.md command shape.
    • Whiteboard bookkeeping:
      • Mark Phase 15 follow-up TODO (restore/port noInit behavior in cutover path) done after landing.
      • Record any behavior differences vs legacy path if discovered.
  • Status note:

    • dockerBuild(noInit: true) is now supported in hard-cutover by passing conversion option NoInit.
    • Converted ExecOps now include withExec(noInit: true) when requested.
    • Focused validation passed:
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestContainer/TestExecInit'
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestLLBToDagger/TestLoadContainerFromConvertedIDRunNoInitOption'
  • Non-goals:

    • Do not change default init behavior for plain withExec unless noInit is explicitly requested.
    • Do not change Dockerfile frontend lowering behavior; this phase is conversion/plumbing only.

Phase 29: Explicit-File COPY Fallback for Directory Sources (SHA256SUMS.d regression)

  • Goal:

    • Fix Dockerfile/LLB conversion bug where an explicit-destination COPY can be lowered to withFile(...) even when source is actually a directory, causing runtime error: path ... is a directory, not a file.
    • Keep existing fast path for real file sources.
    • Avoid widening public API surface.
  • Design:

    • Add hidden/internal withFile arg to allow directory-source fallback when file load fails.
    • In llbtodagger explicit-file copy mapping, set that hidden arg so runtime can recover when source is directory.
    • Resolver fallback rewrites source from <dir>.file(path: X) to <dir>.directory(path: X) and performs withDirectory(...) with equivalent options.
    • Keep fallback disabled by default so non-llbtodagger callers keep existing behavior.
  • Implementation checklist:

    • Schema/API plumbing:
      • Add hidden allowDirectorySourceFallback bool arg to WithFileArgs (internal:"true", default false).
      • Implement fallback in directorySchema.withFile: on source-file load failure and hidden flag set, try directory-source rewrite and call Directory.WithDirectory(...).
      • Extend WithFileArgs.Inputs to apply the same fallback during dependency loading (pre-resolver path), otherwise fallback never triggers.
      • Implement the same fallback in containerSchema.withFile (for named-owner/container-path copy mapping).
    • Converter wiring:
      • In llbtodagger explicit-file copy emission (withFile path), include hidden arg allowDirectorySourceFallback: true.
      • Apply for both plain and container-owner copy paths.
    • Tests:
      • Unit: explicit-file copy conversion includes hidden fallback arg.
      • Integration repro passes:
        • TestLLBToDagger/TestLoadContainerFromConvertedIDCopyDirectoryToExplicitDestinationPath
        • TestDockerfile/TestDockerBuild/copy-directory-to-explicit-destination-path
    • Keep behavior constraints:
      • Do not change public SDK-visible API.
      • Do not remove explicit-file fast path for real file sources.
  • Status note:

    • Repro regression fixed for both llbtodagger and dockerBuild integration paths.
    • Hidden fallback arg is now emitted for explicit-file copy fast path and consumed by schema fallback logic.
    • Focused validation passed:
      • go test ./util/llbtodagger -run 'TestDefinitionToIDFileCopyExplicitDestPathUsesWithFile|TestDefinitionToIDFileCopyModeOverride|TestDefinitionToIDDockerfileCopyToExplicitFileDestWithChmod|TestDefinitionToIDDockerfileCopyGroupOnlyChown|TestDefinitionToIDDockerfileCopyNamedChown' -count=1
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestLLBToDagger/TestLoadContainerFromConvertedIDCopyDirectoryToExplicitDestinationPath'
      • dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestDockerfile/TestDockerBuild/copy-directory-to-explicit-destination-path'
  • Non-goals:

    • Do not redesign deriveCopySelection semantics in this phase.
    • Do not add source-type probing to converter at conversion-time.

Initial Op Coverage Matrix (Planning Draft)

LLB op kindIntended Dagger API representationConfidenceStatus
SourceOp docker-imagequery.container().from(address) (possibly followed by rootfs projection)HighImplemented
SourceOp gitquery.git(url...).{head/ref/branch/tag/commit}.tree(...)MediumImplemented
SourceOp localquery.host().directory(path, include/exclude/...)Low-MediumImplemented (strict attrs)
SourceOp http/httpsquery.http(url, ...)Low-MediumImplemented (strict attrs)
SourceOp oci-layoutlikely partial (host.containerImage or unsupported)LowUnsupported (explicit error)
SourceOp blobexplicitly unsupported -> errorN/AUnsupported-by-design
ExecOpcontainer.* + withExec(...) chainHighImplemented (strict subset)
FileOpdirectory.with*/without* / file.with* chainHighImplemented (strict subset)
MergeOpdirectory merge chainMediumImplemented
DiffOpdirectory.diff(other)MediumImplemented
BuildOpexplicitly unsupported -> errorN/AUnsupported-by-design

Known Snags / Mismatch Log

  • BuildOp nested build semantics are not supported here; immediate error by policy.
  • local:// source identifiers include session-specific attributes that may not round-trip to a user-level host path.
  • blob:// source is explicitly unsupported here; immediate error by policy.
  • Some ExecOp mount/metadata details (certain cache/secret/socket internals) may be partially representable only.
  • If type/view/module details are missing from LLB context and we cannot represent faithfully, return error (no fallback).
  • dockerfile2llb coverage note: COPY --link currently lowered to FileOp in this test path (not MergeOp), so MergeOp integration is still primarily covered by synthetic LLB tests.
  • Container output nuance is resolved:
    • DefinitionToID now returns a container-typed ID for Dockerfile-derived paths.
    • Metadata calls may be appended after withRootfs/withExec, so tests should not assume withRootfs is the terminal field.
  • Metadata inference limit resolved by API change:
    • dockerfile2llb returns final image config (img) separately from *pb.Definition.
    • Converter should accept this as a second input for metadata-complete output.
  • Image config nuance:
    • Config.ArgsEscaped is ignored on non-Windows images (no Dagger API equivalent needed for Linux behavior).
    • Config.ArgsEscaped on Windows images is still unsupported and returns error.
  • COPY semantics nuance (resolved):
    • TestDockerfile/TestDockerBuild/onbuild_command_is_published failed because file-op COPY lowered to withDirectory(include: [...]) silently no-oped when source was missing.
    • Added internal requiredSourcePath propagation to enforce BuildKit-like missing-source error for literal single-path COPY cases.

Current Explicit Unsupported Cases (Implemented)

  • All BuildOp vertices.
  • All blob:// sources.
  • All oci-layout:// sources (currently unsupported).
  • ExecOp with unsupported network modes (unknown/invalid enum values), non-default mount content cache, or unsupported metadata fields.
  • FileOp copy actions with alwaysReplaceExistingDestPaths.
  • FileOp mkdir without makeParents=true.
  • FileOp mkfile with non-UTF8 content.
  • Named ownership on mkdir actions when no container context is available.
  • Named ownership on mkfile file actions.
  • Named ownership on copy actions when no container context is available.

Detailed Unsupported Nuance Catalog (Exhaustive, Dockerfile-Classified)

Unsupported and relevant to Dockerfile-generated LLB

HIGH

MEDIUM

LOW

  • Platform OSVersion and OSFeatures are unsupported.

  • ArgsEscaped on Windows images is unsupported.

Unsupported but outside Dockerfile instruction support (or malformed/non-canonical LLB)

  • Synthetic root op with more than one input is rejected.
  • Unknown/non-classified op types are rejected.
  • Top-level non-Directory/non-Container result types are rejected.
  • Any non-Directory/non-Container input where a Directory is required is rejected.
  • Source identifiers with wrong or empty scheme payload are rejected.
  • Platform with missing OS or Architecture is rejected.
  • BuildOp is explicitly unsupported.
  • blob:// source ops are explicitly unsupported.
  • oci-layout:// source ops are currently unsupported.
  • Unknown source schemes are unsupported.
  • Git source missing remote URL is rejected.
  • Git custom auth token secret names are unsupported (only default secret key accepted).
  • Git custom auth header secret names are unsupported (only default secret key accepted).
  • Git known-hosts override is unsupported.
  • Git custom SSH socket mount names are unsupported (only default/empty accepted).
  • Any unrecognized git source attr is unsupported.
  • Local source with empty resolved path is rejected.
  • Local source invalid include/exclude JSON attrs are rejected.
  • Local source non-metadata differ mode is unsupported.
  • Any unrecognized local source attr is unsupported.
  • HTTP source with non-HTTP(S) identifier is rejected.
  • Any unrecognized HTTP source attr is unsupported.
  • HTTP uid/gid override attrs are unsupported.
  • HTTP invalid URL parsing is rejected.
  • HTTP invalid permission attr parsing is rejected.
  • Missing exec op or missing exec meta is rejected.
  • Hostname override is unsupported.
  • Extra hosts are unsupported.
  • Ulimit settings are unsupported.
  • Cgroup parent is unsupported.
  • Valid-exit-code overrides are unsupported.
  • Missing root bind mount (dest="/", bind type) is unsupported.
  • Mount resultID usage is unsupported.
  • Non-default mount content cache mode is unsupported.
  • Cache mounts without cache ID are unsupported.
  • Cache sharing modes outside {SHARED, PRIVATE, LOCKED} are unsupported.
  • Any unknown mount type is unsupported.
  • Invalid env entries without name=value are rejected.
  • Missing output mount for selected output index is unsupported.
  • Mount input indices out of range are unsupported.
  • Missing merge/diff/file op payloads are rejected.
  • Diff input indices out of range are unsupported.
  • Missing output mapping for selected file output index is unsupported.
  • File action input indices out of range are unsupported.
  • File action references to unresolved prior action outputs are unsupported.
  • Primary file action input must be a Directory; other types are unsupported.
  • File copy secondary input must be a Directory; other types are unsupported.
  • File actions other than mkdir, mkfile, rm, copy are unsupported.
  • mkdir without makeParents=true is unsupported.
  • mkdir timestamp override is unsupported.
  • Named ownership for mkdir actions without container context is unsupported.
  • Named ownership for mkfile actions is unsupported.
  • mkfile timestamp override is unsupported.
  • mkfile non-UTF8/binary payload is unsupported.
  • copy alwaysReplaceExistingDestPaths=true is unsupported.
  • Named ownership for copy actions without container context is unsupported (non-Dockerfile/non-canonical LLB path).
  • Unknown UserOpt discriminator in chown is unsupported.
  • Invalid env entries (without name=value) are rejected.
  • Exposed ports with invalid format are rejected.
  • Exposed ports outside 1..65535 are rejected.
  • Exposed ports using protocols other than TCP/UDP are unsupported.

Crucial Notes To Not Forget

  • skills/cache-expert/references/debugging.md is the authoritative source for how to run integration tests; follow it exactly.
    • This applies to both primary agent commands and any subagent that runs/tests/parses integration output.
  • In core/schema, APIs prefixed with _ are internal-only:
    • they can be called via raw ID construction,
    • they are intentionally not codegen'd into SDK clients.
    • Also: args/fields tagged with internal:"true" are hidden from callers/codegen even without an _ prefix on the field/function name.
  • After any engine schema API change, regenerate the Go SDK used by integration tests.
    • Required command: dagger -y call -m ./toolchains/go-sdk-dev generate
    • Worktree gotcha: if module resolution fails, pass --workspace=. explicitly:
      • dagger -y call -m ./toolchains/go-sdk-dev --workspace=. generate
  • No custom-op handling in this package.
    • Ignore dagger.customOp.
    • Do not decode/convert dagop.fs, dagop.raw, dagop.ctr.
  • Unsupported/unfaithful mapping policy is strict fail-fast for now.
    • First unsupported/imperfect mapping returns error immediately.

Whiteboard Usage Rules (For This Task)

  • Every time scope, assumptions, or mapping behavior changes, update this file in the same change.
  • Keep section boundaries strict:
    • Unsupported and relevant to Dockerfile-generated LLB must contain only Dockerfile-relevant items.
    • Non-Dockerfile/non-canonical items must go in Unsupported but outside Dockerfile instruction support (or malformed/non-canonical LLB).
  • For each implemented op mapper, update:
    • checklist box,
    • coverage matrix status,
    • mismatch log (if new gaps found).
  • Keep this file current across context compaction; treat it as the source-of-truth progress ledger.