Train

A Rerun catalog can feed training pipelines two ways: export recordings to a standard format, or stream them directly into a PyTorch DataLoader.

Export to a training format

The catalog exposes recordings as queryable DataFrames via DataFusion. Multi-rate sensor streams can be time-aligned and columns of interest extracted, with the result written to whatever format a training pipeline expects.

See Export recordings to LeRobot datasets for a worked example.

Train directly from the catalog

The experimental rerun.experimental.dataloader module wraps a catalog as iterable or map-style PyTorch datasets, with no intermediate export step.

Sample space

Three things describe a dataset (see reference):

• DataSource — a catalog DatasetEntry with an optional segment filter; each registered RRD is one segment, typically one episode or trajectory
• index — the timeline that defines what "one sample" means (e.g. "frame_index" or "real_time")
• fields — a dict of Fields, each mapping a source column (an entity:Archetype:component triple) to a decoder

SampleIndex pre-computes the full sample space from lightweight per-segment index-range metadata — one query per segment, not a scan of the data. For timestamp timelines, FixedRateSampling defines the sampling grid and the server handles drift between grid and real row positions via fill_latest_at.

Decoders

Each Field has a ColumnDecoder that converts a raw Arrow column to a training value:

• NumericDecoder — scalars and numeric lists
• ImageDecoder — JPEG/PNG blobs
• VideoFrameDecoder — compressed video (h264/h265/av1) as RGB tensors or compact YUV420 planes

Windows

Field(window=(offset, …)) returns one value per explicit relative offset rather than a single value. Offsets must use the unit of the dataset's selected index timeline: integral index units for integer timelines, and seconds for timestamp or duration timelines. This is how action chunks and observation history are expressed.

Dataset styles

• RerunIterableDataset — streaming with automatic shuffling and cross-worker and DDP partitioning
• RerunMapDataset — random access by global index; works with PyTorch samplers like DistributedSampler and WeightedRandomSampler

See Train PyTorch models with Rerun for usage.