ContextQMD
Back to Claude Scientific Skills

Data management, h5path, manifests, datasets, and provenance

skills/pathml/references/data_management.md

2.55.011.2 KB
Original Source

Data management, h5path, manifests, datasets, and provenance

This reference targets PathML 3.0.5 stable and a local, de-identified research workflow.

Data boundaries

Separate four classes of data:

  1. Source slides — immutable, access-controlled originals.
  2. Linkage data — direct identifiers and the pseudonym mapping, held outside the analysis workspace by an authorized custodian.
  3. Analysis data — pseudonymous manifests, tiles, masks, counts, graphs, and features.
  4. Reports/models — potentially identifying derived artifacts that still require governance.

Do not assume derived images or embeddings are anonymous. Rare morphology, scanner metadata, dates, or cohort combinations can re-identify a participant. Apply the minimum-necessary principle and institutional retention policy.

Manifest first

Use one row per slide. Recommended columns:

text
slide_id,patient_id,specimen_id,path,split,stain,backend,site,scanner

Rules:

  • IDs are pseudonyms, not MRNs, accessions, initials, dates, or names.
  • slide_id is unique.
  • one patient_id maps to exactly one split;
  • one slide path maps to one slide ID;
  • paths are local, relative to a declared root where possible;
  • URLs and symlinks are rejected;
  • split values are a fixed allowlist such as train, validation, test;
  • serial sections, rescans, and multiple blocks from one patient remain together.

Validate before PathML:

bash
python scripts/slide_manifest.py validate \
  --manifest metadata/manifest.csv \
  --root .

The validator checks strict CSV structure, duplicate IDs/paths, missing local files, unsafe paths, supported suffixes, and patient/slide leakage. It does not upload data or inspect arbitrary clinical fields.

h5path format

PathML processes slides into an HDF5-based .h5path file. Stable documentation describes:

text
root/
├── fields/
│   ├── labels/          # slide-level attributes
│   └── slide_type/      # stain/platform flags
├── masks/               # slide-level masks
├── counts/              # AnnData-like counts storage
└── tiles/
    ├── attributes       # tile_shape, tile_stride
    └── "(i, j)"/
        ├── array
        ├── masks/
        ├── labels/
        └── attributes   # coords, name

Write and reopen through public APIs:

python
from pathml.core import SlideData

slide.write("derived/slide-001.h5path")
reopened = SlideData("derived/slide-001.h5path")

There is no stable to_hdf5(), from_hdf5(), or load_tiles_from_hdf5() API. SlideDataset.write(directory, filenames=None) calls each slide's write().

Stable documentation states HDF5 datasets are stored as float16; confirm dtype for the exact arrays your workflow writes. Quantitative marker intensities can lose precision if silently cast. Record and test expected dtype, range, NaN/Inf, compression, and round-trip tolerance.

h5path trust boundary

Treat .h5path as a structured binary input, not harmless data:

  • HDF5 parsers have a large attack surface; open third-party files in isolation.
  • PathML 3.0.5 TileDataset dynamically interprets the stored tile_shape attribute as a Python expression. Never open an untrusted .h5path.
  • Labels can contain sensitive values. Do not copy direct identifiers into HDF5.
  • A malformed file can request large allocations. Check file size and schema before loading.
  • Do not edit HDF5 concurrently from multiple processes unless the access pattern is explicitly designed and tested.

Use a sidecar JSON manifest for provenance rather than relying on arbitrary HDF5 labels. Keep the JSON strict, bounded, pseudonymous, and versioned.

PyTorch tile dataset

The canonical stable import is:

python
from pathml.datasets import TileDataset
from torch.utils.data import DataLoader

tiles = TileDataset("derived/slide-001.h5path")
loader = DataLoader(
    tiles,
    batch_size=8,
    shuffle=False,
    num_workers=0,
)

Each item is:

text
(tile_image, tile_masks, tile_labels, slide_labels)

Shapes:

  • RGB/multichannel 3-D input becomes (C, H, W).
  • 5-D PathML input (i, j, z, c, t) becomes (T, C, Z, W, H) in stable source; verify axis semantics before use.
  • masks are stacked as (n_masks, tile_height, tile_width) when present.
  • label dictionaries are user-defined and may need a custom collate_fn.

Do not assume mask dictionary order carries semantics. Persist ordered mask names in a separate schema and assert them when loading.

pathml.ml.TileDataset is also exported in 3.0.5, but pathml.datasets.TileDataset is the documented dataset API.

SlideDataset

SlideDataset(slides) accepts a list of already constructed SlideData objects:

python
from pathml.core import HESlide, SlideDataset

slides = [
    HESlide("data/slide-001.svs", backend="openslide"),
    HESlide("data/slide-002.svs", backend="openslide"),
]
cohort = SlideDataset(slides)
cohort.run(pipeline, distributed=False, tile_size=512, level=0)
cohort.write("derived")

It does not accept a glob/path list plus tiling arguments as a constructor. Preserve a deterministic manifest order and map output filenames explicitly.

Public data modules

Stable pathml.datasets exports:

python
from pathml.datasets import DeepFocusDataModule, PanNukeDataModule

PanNuke

python
pannuke = PanNukeDataModule(
    data_dir="approved_data/pannuke",
    download=False,
    shuffle=True,
    nucleus_type_labels=True,
    split=1,
    batch_size=8,
    hovernet_preprocess=True,
)
  • 7,901 256-pixel patches, 19 tissue types, five nucleus categories plus background.
  • download=False is the safe default.
  • download=True downloads three ZIPs from Warwick and extracts them.
  • split must be 1, 2, 3, or None; each integer rotates the three published folds across train/validation/test.
  • split=None exposes the whole dataset; do not use it for performance estimation.

Published folds are not a substitute for verifying patient/source-slide independence for the intended claim.

DeepFocus

python
deepfocus = DeepFocusDataModule(
    data_dir="approved_data/deepfocus",
    download=False,
    shuffle=True,
    batch_size=8,
)
  • focus classification patches derived from four slides/patients and four stains;
  • download=True contacts Zenodo;
  • stable code checks the downloaded HDF5 file against a fixed MD5 value.

MD5 here is an upstream integrity check, not a modern provenance guarantee. Record a SHA-256 and dataset license/source separately.

PathML 3.0.5 does not export TCGADataModule. Use a separately governed data acquisition process for TCGA/GDC and document its API/version/consent terms.

Before changing any download flag to True, tell the user:

  • exact host and expected dataset;
  • approximate size (stable docs report PanNuke ~37.33 GB and DeepFocus ~10 GB);
  • destination and available disk;
  • dataset license/terms and citation;
  • whether the environment logs outbound IP/account metadata; and
  • that no local slide or clinical data will be uploaded.

Require explicit opt-in. Never place downloaded archives inside the repository.

Graph datasets and unsafe .pt files

pathml.datasets.EntityDataset assembles cell graphs, tissue graphs, and assignment matrices. Stable source opens .pt files using PyTorch object deserialization with unrestricted object loading.

Consequences:

  • only load artifacts created by the trusted project;
  • never load an emailed/downloaded .pt file merely to inspect it;
  • verify SHA-256, producer, code revision, PyTorch/PyG versions, and schema;
  • prefer non-executable interchange formats for exchange;
  • run legacy artifacts in a disposable, network-disabled environment if review is unavoidable.

The bundled inference planner and graph validator never load .pt, .pth, .ckpt, pickle, ONNX, or other model/graph binaries.

Split design and leakage

Create the split column once, before tiling:

text
patient → specimen/block → slide/rescan/serial section → region → tile

Everything below a patient follows the patient's split unless the scientific design explicitly requires a stricter grouping.

Common leakage paths:

  • overlapping tiles from one slide in different splits;
  • serial sections or rescans assigned separately;
  • stain reference fitted on all slides;
  • QC threshold chosen after viewing test failures;
  • normalization/scaling fit before split;
  • graph neighborhoods crossing a split boundary;
  • duplicated public patches;
  • institution/scanner confounding;
  • selecting a checkpoint on the test metric.

The manifest validator reports patient and slide leakage, but it cannot discover unknown biological relatedness. Document grouping assumptions.

Provenance sidecar

Recommended strict JSON fields:

json
{
  "schema_version": "1.0",
  "pathml_version": "3.0.5",
  "source_sha256": "hex-digest",
  "slide_id": "slide-001",
  "patient_id": "patient-001",
  "split": "train",
  "backend": "openslide",
  "level": 0,
  "downsample": 1.0,
  "mpp_x": null,
  "mpp_y": null,
  "tile_size_ij": [512, 512],
  "tile_stride_ij": [512, 512],
  "tile_pad": false,
  "pipeline_id": "he-v1",
  "code_revision": "project-commit",
  "created_utc": "RFC3339 timestamp"
}

Do not put a direct identifier in these fields. Add:

  • ordered transform parameters and fitted stain arrays;
  • mask/label schema;
  • QC counts and exclusion reasons;
  • dependency lock hash;
  • model artifact SHA-256 and license;
  • random seed manifest;
  • coordinate units and conversion;
  • output hashes and software/hardware details.

Use SHA-256 for provenance:

python
import hashlib
from pathlib import Path

def sha256_file(path: Path, chunk_bytes: int = 1024 * 1024) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(chunk_bytes), b""):
            digest.update(chunk)
    return digest.hexdigest()

Hash only authorized local files and expect full-slide hashing to be I/O-heavy. Do not print paths containing identifiers.

Storage and lifecycle checklist

  • Estimate raw, temporary, .h5path, mask, count, graph, and model storage.
  • Write to a same-filesystem temporary destination, validate, then atomically rename where possible.
  • Do not overwrite source slides.
  • Use private permissions and encrypted storage/backups.
  • Verify output counts, shapes, dtypes, coordinates, and hashes.
  • Record partial failures and retry policy.
  • Test disaster recovery and retention/deletion.
  • Do not commit slide data, model binaries, linkage files, or manifests with PHI.

Sources, accessed 2026-07-23