skills/pathml/references/preprocessing.md
This reference describes PathML 3.0.5 stable. It corrects older examples that
called nonexistent Pipeline.run(), omitted required mask/label names, or passed
unsupported transform arguments.
A Pipeline is an ordered list of Transform objects. Pipeline.apply(tile)
modifies one Tile in place and returns it. A slide or dataset owns execution:
from pathml.core import HESlide
from pathml.preprocessing import BoxBlur, Pipeline, TissueDetectionHE
slide = HESlide("data/slide-001.svs", backend="openslide")
pipeline = Pipeline(
[
BoxBlur(kernel_size=5),
TissueDetectionHE(mask_name="tissue"),
]
)
slide.run(
pipeline,
distributed=False,
tile_size=512,
tile_stride=512,
level=0,
tile_pad=False,
)
Stable facts:
Pipeline(transform_sequence=None) and Pipeline.apply(tile) are public.Pipeline has no run() method.SlideData.run() and SlideDataset.run() apply pipelines.distributed=True is the default and may create a local Dask cluster using
available cores. Start with distributed=False.overwrite_existing_tiles=True.write_dir causes a <slide.name>.h5path write after processing.Pipeline.save() writes a pickle. A pickle is executable on load; never use a
pipeline file from an untrusted source.from pathml.preprocessing import (
AdaptiveHistogramEqualization,
BinaryThreshold,
BoxBlur,
CollapseRunsCODEX,
CollapseRunsVectra,
ForegroundDetection,
GaussianBlur,
HistogramEqualization,
LabelArtifactTileHE,
LabelWhiteSpaceHE,
MedianBlur,
MorphClose,
MorphOpen,
NucleusDetectionHE,
Pipeline,
QuantifyMIF,
RescaleIntensity,
SegmentMIF,
SegmentMIFRemote,
StainNormalizationHE,
SuperpixelInterpolation,
TissueDetectionHE,
)
There is no stable transform='...' registry and no safe reason to construct
transforms from arbitrary Python expressions. Parse a strict allowlisted config,
then instantiate known classes explicitly.
from pathml.preprocessing import TissueDetectionHE
tissue = TissueDetectionHE(
mask_name="tissue",
use_saturation=True,
blur_ksize=17,
threshold=None, # Otsu when None
morph_n_iter=3,
morph_k_size=7,
min_region_size=5000,
max_hole_size=1500,
outer_contours_only=False,
)
The transform expects an H&E uint8 tile. It:
tile.masks["tissue"].mask_name is required in practice; None fails when apply() runs.
min_region_size, max_hole_size, and morphology kernels are measured in pixels
at the processing level. Re-tune if level or MPP changes.
Tissue detection is tile-local. It can disagree at tile edges, and PathML does not automatically remove background tiles from the pipeline. Compute and record tissue coverage after the mask exists:
coverage = float((tile.masks["tissue"] > 0).mean())
keep = coverage >= 0.50
Choose the coverage rule on training data and preserve rejected-tile counts.
These transforms write tile labels, not pixel masks:
from pathml.preprocessing import LabelArtifactTileHE, LabelWhiteSpaceHE
whitespace = LabelWhiteSpaceHE(
label_name="mostly_white",
greyscale_threshold=230,
proportion_threshold=0.5,
)
artifact = LabelArtifactTileHE(label_name="artifact")
LabelWhiteSpaceHE labels a tile when the proportion of greyscale pixels above
the threshold exceeds proportion_threshold.LabelArtifactTileHE is a fixed rule-based HSI heuristic for whitespace,
dark regions, and pen-like colors. It exposes only label_name; older examples
with pen_threshold or bubble_threshold are invalid.Neither is a complete slide-quality system. Review representative overlays and track blur, folds, bubbles, pen, tissue coverage, clipping, color drift, missing channels, and focus separately. Do not convert a heuristic QC flag into a clinical quality judgment.
The dependency-free helper provides a deliberately simple synthetic/local check, not a replacement for PathML:
python scripts/image_qc.py synthetic --width 256 --height 256
python scripts/image_qc.py inspect --image tests/fixtures/synthetic.ppm --root .
It reports brightness/saturation and a coarse tissue-like mask using bounded pixels. PNG/JPEG/TIFF input needs Pillow, imported only after argument validation.
from pathml.preprocessing import StainNormalizationHE
normalizer = StainNormalizationHE(
target="normalize", # normalize | hematoxylin | eosin
stain_estimation_method="macenko", # macenko | vahadane
optical_density_threshold=0.15,
regularizer=0.1,
angular_percentile=0.01,
background_intensity=245,
)
Stable StainNormalizationHE does not accept tissue_mask_name,
target_od, or target_concentrations. It accepts stain_matrix_target_od and
max_c_target, and supplies fixed defaults.
To fit a reference:
normalizer.fit_to_reference(training_reference_rgb)
normalized_rgb = normalizer.F(source_rgb)
Leakage controls:
Macenko and Vahadane are model-based color standardization methods, not guarantees that biological staining becomes comparable. Preserve raw inputs and assess whether normalization removes task-relevant signal or amplifies artifacts.
from pathml.preprocessing import NucleusDetectionHE
nuclei = NucleusDetectionHE(
mask_name="nuclei",
stain_estimation_method="vahadane",
superpixel_region_size=10,
n_iter=30,
)
This is a simple transform: hematoxylin separation, superpixel interpolation, and Otsu thresholding. It writes a binary tile mask. It is not HoVer-Net, does not assign nucleus classes, and should not be treated as a validated cell count. Inspect touching objects, fragments, necrosis, stain failure, and tile boundaries.
Useful stable signatures:
from pathml.preprocessing import BinaryThreshold, MorphClose, MorphOpen
threshold = BinaryThreshold(
mask_name="foreground",
use_otsu=True,
threshold=0,
inverse=False,
)
opened = MorphOpen(mask_name="foreground", kernel_size=5, n_iterations=1)
closed = MorphClose(mask_name="foreground", kernel_size=5, n_iterations=1)
BinaryThreshold.apply() creates a named mask. MorphOpen and MorphClose
modify the named mask. Match dtype, polarity, and dimensions explicitly.
from pathml.preprocessing import (
BoxBlur,
LabelArtifactTileHE,
LabelWhiteSpaceHE,
Pipeline,
StainNormalizationHE,
TissueDetectionHE,
)
pipeline = Pipeline(
[
LabelWhiteSpaceHE(
label_name="mostly_white",
greyscale_threshold=230,
proportion_threshold=0.8,
),
LabelArtifactTileHE(label_name="artifact"),
BoxBlur(kernel_size=3),
TissueDetectionHE(
mask_name="tissue",
min_region_size=5000,
outer_contours_only=False,
),
StainNormalizationHE(
target="normalize",
stain_estimation_method="macenko",
),
]
)
QC labels do not short-circuit later transforms. If expensive stages should run only on accepted tiles, write an explicit custom transform or bounded manual loop with a documented policy. Keep the logic deterministic and test it.
PathML has no max_tiles argument. Use islice for a local pilot:
from itertools import islice
sampled = []
for tile in islice(
slide.generate_tiles(shape=512, stride=512, pad=False, level=0),
16,
):
pipeline.apply(tile)
sampled.append(
{
"coords_ij": tile.coords,
"shape": tuple(tile.image.shape),
"tissue_fraction": float((tile.masks["tissue"] > 0).mean()),
"mostly_white": bool(tile.labels["mostly_white"]),
"artifact": bool(tile.labels["artifact"]),
}
)
Plan first:
python scripts/plan_pipeline.py \
--width 100000 --height 80000 \
--tile-size 512 --stride 512 \
--pipeline TissueDetectionHE,LabelWhiteSpaceHE,StainNormalizationHE \
--max-tiles 1000000
The planner never opens the slide or imports PathML. Supply dimensions from a trusted technical metadata inspection.
tissue, nuclei, cell_segmentation) and record
whether each mask is binary, semantic, or instance-labeled.tile_pad=True introduces zeros. Document whether padded pixels are ignored in
QC, stain fitting, loss, and stitching.SlideData.generate_tiles() cannot slice slide-level masks into padded
tiles.Do all splitting before:
All tiles, regions, serial sections, and repeat scans from one patient belong to one split. If site/scanner generalization is the target, reserve entire sites or scanners as designed. Record exclusions before viewing test outcomes.
For each run, retain: