ContextQMD
Back to Claude Scientific Skills

Training, Evaluation, Configuration, and Logging

skills/pufferlib/references/training.md

2.55.010.2 KB
Original Source

Training, Evaluation, Configuration, and Logging

Research snapshot: 2026-07-23.

Choose a version profile first

Published stable package

PyPI's latest stable pufferlib release is 3.0.0, published 2025-06-23. It declares Python >=3.9 and is distributed only as a 60.7 MB source archive:

text
pufferlib-3.0.0.tar.gz
sha256: 7df3a3e3f5f894d78d2a1f5374097890aec01473183e748abefe4f3faa10eaa9

The uploaded metadata depends on NumPy <2.0, Gym <=0.23, Gymnasium <=0.29.1, PettingZoo <=1.24.1, Shimmy, Torch, Neptune, W&B, and other packages without a complete transitive lock. It does not declare a CUDA version or a minimum Torch version. Do not invent compatibility guarantees.

Current source line

The upstream default branch is 4.0; its pyproject.toml says version 4.0.0, Python >=3.10, and Torch >=2.9. As of the research date, this source line is not the latest stable PyPI artifact.

The current PufferTank Dockerfile uses:

  • Ubuntu 24.04
  • NVIDIA CUDA 13.0.2 cuDNN development image
  • Python 3.12
  • the CUDA 13.0 PyTorch wheel index
  • Nsight Systems 2025.6.3

The Dockerfile does not pin an exact Torch wheel, uv version, PufferLib commit, or every apt package. It is an upstream convenience environment, not a complete reproducibility lock.

Reproducible uv workflow

Do not use an unpinned uv pip install pufferlib. Work in a disposable, project-specific environment and commit pyproject.toml plus uv.lock.

For the published profile, after reviewing the source archive and build:

bash
uv venv --python 3.11
uv add --exact --no-sync "pufferlib==3.0.0"
uv lock
uv sync --frozen

Confirm the lock records the published SHA-256 above and review every resolved dependency. The 3.0.0 source build can compile native code and may fetch build assets. Resolve and build in a sandbox with no credentials or sensitive mounts. Do not treat a successful resolver run as a security review.

For 4.0 source work, pin an immutable revision rather than branch 4.0:

bash
uv add --no-sync \
  "pufferlib @ git+https://github.com/PufferAI/PufferLib.git@25647630e1b15330bb3153a5a0d3ff8d234c3acf"
uv lock

The commit above is the reviewed 4.0 branch head on 2026-07-23. Re-review before updating it. Native training still requires an audited build of a specific environment; uv locking does not lock compilers, CUDA, NCCL, cuDNN, Raylib, or system libraries.

Never run remote install scripts directly from a pipe. Download, inspect, pin, verify, and execute only in an appropriate sandbox.

Published 3.0.0 training

CLI

The 3.0 console entry point is puffer = pufferlib.pufferl:main:

bash
puffer train ENV_NAME [OPTIONS]
puffer eval ENV_NAME [OPTIONS]
puffer sweep ENV_NAME [OPTIONS]
puffer autotune ENV_NAME [OPTIONS]
puffer profile ENV_NAME [OPTIONS]
puffer export ENV_NAME [OPTIONS]

Environment, vector, policy, recurrent, training, and sweep values come from INI sections. Overrides use section-qualified flags:

bash
puffer train puffer_breakout \
  --train.device cpu \
  --train.total-timesteps 100000 \
  --vec.backend Serial \
  --vec.num-envs 2

Run puffer train ENV_NAME --help against the exact locked environment because available options are generated from merged INI files.

Python API

The stable trainer is pufferlib.pufferl.PuffeRL, not a top-level pufferlib.PuffeRL:

python
from pufferlib import pufferl

args = pufferl.load_config("puffer_breakout")
vecenv = pufferl.load_env("puffer_breakout", args)
policy = pufferl.load_policy(args, vecenv, "puffer_breakout")
trainer = pufferl.PuffeRL(args["train"], vecenv, policy)

try:
    while trainer.epoch < trainer.total_epochs:
        trainer.evaluate()
        trainer.train()
        trainer.mean_and_log()
finally:
    trainer.close()

The exact public methods include evaluate, train, mean_and_log, save_checkpoint, print_dashboard, and close. Use the CLI when possible; the Python trainer is a relatively low-level implementation surface.

Stable configuration checks

  • Make rollout/batch relationships explicit; do not rely on auto in a published experiment.
  • Record environment, vector, policy, recurrent, and train sections verbatim.
  • Fix seed in both [vec] and [train], then run multiple independent seeds.
  • Record torch_deterministic, precision, compile settings, optimizer, horizon, minibatch, and total timesteps.
  • Keep evaluation seeds, instances, and metrics separate from training.

Current 4.0 training

Build one audited environment, then use:

bash
puffer train breakout
puffer eval breakout --load-model-path checkpoints/.../weights.bin
puffer sweep breakout
puffer match breakout \
  --load-model-path trusted-a.bin \
  --load-enemy-model-path trusted-b.bin

Current modes are train, eval, sweep, paretosweep, and match. Native training is the default. --slowly selects the Torch fallback. Configuration uses sections such as:

ini
[vec]
total_agents = 4096
num_buffers = 2
num_threads = 16

[train]
total_timesteps = 10_000_000
minibatch_size = 8192
horizon = 64

[torch]
network = MinGRU
encoder = DefaultEncoder
decoder = DefaultDecoder

Current source validates that minibatch_size is divisible by horizon and does not exceed horizon * total_agents. Multi-GPU launch uses spawn. Do a small CPU/local build and contract test before CUDA training.

Held-out evaluation

Training rollouts are not evaluation. For every reported result:

  1. Freeze one checkpoint-selection rule before inspecting held-out scores.
  2. Construct fresh evaluation environment instances.
  3. Use evaluation seeds disjoint from training seeds.
  4. Disable optimizer updates, exploration noise unless explicitly measuring it, curriculum updates, normalization-stat updates, and reward shaping used only for training.
  5. Report deterministic and stochastic policy protocols separately.
  6. Run enough episodes for uncertainty; report per-seed results and aggregate intervals, not only a best run.
  7. Preserve terminated versus truncated semantics in return/length accounting.
  8. Record wrappers, frame skip, autoreset mode, opponent pool, policy state reset, and rendering state.

Generate a starting plan:

bash
python3 scripts/repro_plan.py --environment synthetic

Checkpoints

PufferLib 3.0 saves a policy state_dict with torch.save and a separate trainer state containing optimizer state, global step, epoch, and run ID. Its loading paths call torch.load. The 4.0 native backend writes .bin weight files; the 4.0 Torch fallback also uses torch.save/torch.load.

Rules:

  • Never load an untrusted checkpoint, even to “inspect” it.
  • Record SHA-256, size, source URL, immutable revision, license, environment, policy architecture, package lock, and training config in a strict JSON sidecar.
  • Do not use latest in a reproducible run; resolve and record the exact path and digest.
  • Do not auto-download a run artifact by ID.
  • Test restore and evaluation in a disposable environment before a long resume.
  • A model-only checkpoint is not a bitwise resume; optimizer, scheduler, normalizer, RNG, environment, and recurrent state may also matter.

Safe metadata inspection:

bash
python3 scripts/inspect_checkpoint.py trusted/model.pt \
  --expected-sha256 EXPECTED_DIGEST

The helper hashes and classifies bytes only. It never imports Torch, invokes pickle, opens archive members, or extracts files.

External logging

Local logging is the default. W&B and Neptune are optional network services that may transmit configuration, metrics, source metadata, hardware telemetry, stdout/stderr, and explicitly uploaded checkpoints/artifacts. They can create storage, seat, compute, or retention costs and are subject to vendor privacy, access, and retention policies.

Credential rules:

  • W&B: use the named environment variable WANDB_API_KEY or an approved secret manager.
  • Neptune: use NEPTUNE_API_TOKEN or an approved secret manager.
  • Never pass either secret as a CLI argument, INI/JSON value, logger config, tag, run name, or chat/tool input.
  • Never print the value or include it in a broad environment dump.
  • Do not recursively search for .env files. If policy permits a local secret file, read only the explicitly named key from the explicitly named file.
  • Sanitize configuration before logging; reject keys containing token, secret, password, credential, authorization, private key, or API key.
  • Disable checkpoint/source upload unless separately approved.

PufferLib 3.0 supports both --wandb and --neptune; its sweep mode requires one. Current 4.0 source exposes W&B but no Neptune CLI integration. In either profile, require explicit logging opt-in and disclosure acknowledgment. The bundled training planner enforces this without reading credential values:

bash
python3 scripts/train_template.py \
  --logger wandb \
  --enable-external-logging \
  --acknowledge-external-disclosure

Sources