ContextQMD
Back to Claude Scientific Skills

Nextflow and Snakemake Integration

skills/latchbio-integration/references/nextflow-snakemake.md

2.55.08.4 KB
Original Source

Nextflow and Snakemake Integration

Latch supports Python, Nextflow, and Snakemake workflows, but their packaging paths are not interchangeable. Confirm the SDK version and choose one track before generating files.

Nextflow: Documented SDK Wrapper Path

Prerequisites

  • A runnable Nextflow pipeline
  • A nextflow_schema.json
  • Containers or a documented execution profile for every process
  • A pinned Latch SDK

Install:

bash
uv pip install "latch==2.76.8"

Generate metadata

bash
latch generate-metadata nextflow_schema.json --nextflow

Current generation creates:

text
latch_metadata/
├── __init__.py
└── generated.py
  • generated.py is regenerated from the schema. Do not edit it.
  • Put persistent custom metadata and flow changes in latch_metadata/__init__.py.
  • Re-run generation after changing nextflow_schema.json.
  • Review inferred file, enum, default, and required types before registration.

SDK 2.67.0 changed Nextflow generation to use one generated dataclass containing all parameters and a generated base flow. Older parameters.py examples may no longer match the generated layout.

Register

The currently documented wrapper command is:

bash
latch login
latch register . \
  --nf-script main.nf \
  --nf-execution-profile docker,test

Registration generates a Latch workflow wrapper and latch.config. The exact entrypoint location depends on the generation path and SDK version.

Important:

  • If a root Dockerfile exists, registration uses it.
  • Otherwise Latch can generate a Dockerfile under .latch/.
  • Passing --nf-script again can regenerate and overwrite wrapper code.
  • After intentionally customizing a generated entrypoint, re-register without --nf-script to preserve it.
  • Keep custom code in a separate module when possible.

Generate an entrypoint explicitly

SDK 2.76.8 also exposes:

bash
latch nextflow generate-entrypoint . \
  --nf-script main.nf \
  --execution-profile docker,test \
  --output wf/custom_entrypoint.py

This requires a valid NextflowMetadata object in the metadata root.

Experimental Forch-only registration

The CLI also has:

bash
latch nextflow register . --script-path main.nf

The official CLI guide marks this command experimental and only applicable to Forch, Latch's architecture for running Nextflow in the user's own AWS account. It is not a general alternative to latch register --nf-script. Use it only for a configured Forch/BYOC project with current Latch guidance.

Nextflow configuration rules

  • Define every process container.
  • Use profiles for environment-specific settings rather than editing the pipeline for Latch.
  • Keep Latch's generated latch.config in the effective config chain.
  • Use the official Latch shared-storage guidance for processes that need a common work directory.
  • Configure private registries through Latch's supported credentials path.
  • Follow the Latch GPU guide for process accelerators; do not translate Python task decorators into Nextflow resource syntax.
  • Set NextflowRuntimeResources.storage_gib for the runtime/shared filesystem, not just final outputs.
  • Understand storage-retention cost before increasing storage_expiration_hours.

Debugging

bash
latch register --staging .
latch develop .
latch nextflow attach --execution-id <execution-id>

Within latch develop, the production storage initializer is unavailable. Follow the official debug-mode guidance: use the local executor, bypass the initializer in debug mode, use a compatible Latch Nextflow base image, and reduce process resources.

Snakemake: Choose a Compatibility Track

The current docs and current stable package expose different Snakemake tracks. Do not mix them.

Track A: legacy flags in 2.76.8 source (documentation conflict)

The 2.76.8 wheel still exposes the snakemake extra, legacy metadata classes, generate-metadata --snakemake, and register --snakefile. However, the official CLI guide marks the Snakemake metadata and registration flags deprecated and says metadata generation no longer works for latch >= 2.55.0.a6.

Treat this as an unresolved source/documentation conflict. The commands below are useful when maintaining a legacy project already known to use this path, but should not be presented as a supported new-project flow without confirming with current Latch documentation or support.

Use Python 3.11 for the broadest compatibility with the pinned Snakemake 7.x dependency:

bash
uv venv --python 3.11
source .venv/bin/activate
uv pip install "latch[snakemake]==2.76.8"

Generate metadata from the workflow config:

bash
latch generate-metadata config.yaml --snakemake

Register:

bash
latch register . --snakefile Snakefile

Relevant options:

bash
latch register . \
  --snakefile Snakefile \
  --metadata-root latch_metadata \
  --cache-tasks

Use the current SnakemakeMetadata, SnakemakeParameter, FileMetadata, EnvironmentConfig, and DockerMetadata APIs from latch.types.metadata. Inspect their signatures before generating hand-written metadata.

Track B: official Snakemake v2 tutorial

The current Snakemake v2 tutorial is a compatibility-specific path. At the time of this refresh, it explicitly requires:

bash
uv venv --python 3.11
source .venv/bin/activate
uv pip install "latch==2.62.1a2"

It uses imports such as:

python
from latch.types.metadata.snakemake_v2 import SnakemakeV2Metadata

and commands such as:

bash
latch snakemake generate-entrypoint .
latch dockerfile --snakemake -c environment.yaml . -f
latch register -y .

Those v2 metadata modules and the latch snakemake command group are not present in the stable 2.76.8 source tree. The tutorial's generated Dockerfile also pins the workflow runtime separately to latch[snakemake]==2.55.0.a6.

Therefore:

  • Re-check the official tutorial's exact pin before starting.
  • Use an isolated environment.
  • Preserve and review both the local CLI pin and generated runtime pin.
  • Do not upgrade that environment to stable 2.76.8 without a migration plan.
  • Do not copy v2 imports into a stable-track project.
  • Treat the alpha pin as pre-release software and validate end to end.

Snakemake resource and environment rules

  • Give every rule CPU and memory resources or define safe defaults in profiles/default/config.yaml.
  • Pin Conda and container environments.
  • Keep the Latch executor/storage plugin configuration from the chosen track.
  • Use LatchOutputDir for output destinations exposed in the form.
  • Avoid embedding credentials in environment.yaml, Dockerfiles, profiles, or generated metadata.

Generated-File Discipline

Before editing a generated file:

  1. Find the command and source file that generated it.
  2. Determine whether regeneration overwrites the file.
  3. Prefer the documented extension file.
  4. If customization of generated code is unavoidable, record the generation command and stop passing flags that regenerate it.
  5. Diff generated output after every SDK upgrade.

Release Checklist

  • SDK and pipeline-engine versions are pinned.
  • Schema/config and generated metadata are in sync.
  • Every parameter has the expected type and default.
  • Containers and package environments are immutable enough to reproduce.
  • Small test data succeeds.
  • Resume/cache behavior has been exercised.
  • Shared storage has enough capacity and an intentional retention period.
  • Result paths and execution reports appear in the Console.
  • A clean registration from a fresh checkout works.

Official Sources