Genomic Interval Operations

Overview

polars-bio provides 8 core operations for genomic interval arithmetic. All operations work on Polars DataFrames or LazyFrames containing genomic intervals (columns: chrom, start, end by default) and return a LazyFrame by default. Pass output_type="polars.DataFrame" for eager results.

Operations Summary

overlap

Find pairs of overlapping intervals between two DataFrames.

Functional API

import polars as pl
import polars_bio as pb

df1 = pl.DataFrame({
    "chrom": ["chr1", "chr1", "chr1"],
    "start": [1, 5, 22],
    "end":   [6, 9, 30],
})

df2 = pl.DataFrame({
    "chrom": ["chr1", "chr1"],
    "start": [3, 25],
    "end":   [8, 28],
})

# Returns LazyFrame by default
result_lf = pb.overlap(df1, df2, suffixes=("_1", "_2"))
result_df = result_lf.collect()

# Or get DataFrame directly
result_df = pb.overlap(df1, df2, suffixes=("_1", "_2"), output_type="polars.DataFrame")

Method-Chaining API (LazyFrame only)

result = df1.lazy().pb.overlap(df2, suffixes=("_1", "_2")).collect()

Parameters

Output Schema

Returns columns from both inputs with suffixes applied:

• chrom_1, start_1, end_1 (from df1)
• chrom_2, start_2, end_2 (from df2)
• Any additional columns from df1 and df2

Column dtypes are String for chrom and Int64 for start/end.

count_overlaps

Count the number of overlapping intervals from df2 for each interval in df1.

# Functional
counts = pb.count_overlaps(df1, df2)

# Method-chaining (LazyFrame)
counts = df1.lazy().pb.count_overlaps(df2)

Parameters

Output Schema

Returns df1 columns with an additional count column (Int64).

nearest

Find the nearest interval in df2 for each interval in df1.

# Find nearest (default: k=1, any direction)
nearest = pb.nearest(df1, df2, output_type="polars.DataFrame")

# Find k nearest
nearest = pb.nearest(df1, df2, k=3)

# Exclude overlapping intervals from results
nearest = pb.nearest(df1, df2, overlap=False)

# Without distance column
nearest = pb.nearest(df1, df2, distance=False)

Parameters

Output Schema

Returns columns from both DataFrames (with suffixes) plus a distance column (Int64) with the distance to the nearest interval (0 if overlapping). Distance column is omitted if distance=False.

merge

Merge overlapping and bookended intervals within a single DataFrame.

import polars as pl
import polars_bio as pb

df = pl.DataFrame({
    "chrom": ["chr1", "chr1", "chr1", "chr2"],
    "start": [1, 4, 20, 1],
    "end":   [6, 9, 30, 10],
})

# Functional
merged = pb.merge(df, output_type="polars.DataFrame")

# Method-chaining (LazyFrame)
merged = df.lazy().pb.merge().collect()

# Merge intervals within a minimum distance
merged = pb.merge(df, min_dist=10)

Parameters

Output Schema

cluster

Assign cluster IDs to overlapping intervals. Intervals that overlap are assigned the same cluster ID.

# Functional
clustered = pb.cluster(df, output_type="polars.DataFrame")

# Method-chaining (LazyFrame)
clustered = df.lazy().pb.cluster().collect()

# With minimum distance
clustered = pb.cluster(df, min_dist=5)

Parameters

Output Schema

Returns the original columns plus:

coverage

Compute per-interval coverage counts. This is a two-input operation: for each interval in df1, count the coverage from df2.

# Functional
cov = pb.coverage(df1, df2, output_type="polars.DataFrame")

# Method-chaining (LazyFrame)
cov = df1.lazy().pb.coverage(df2).collect()

Parameters

Output Schema

Returns columns from df1 plus a coverage column (Int64).

complement

Find gaps between intervals within a genome. Requires a genome definition specifying chromosome sizes.

import polars as pl
import polars_bio as pb

df = pl.DataFrame({
    "chrom": ["chr1", "chr1"],
    "start": [100, 500],
    "end":   [200, 600],
})

genome = pl.DataFrame({
    "chrom": ["chr1"],
    "start": [0],
    "end":   [1000],
})

# Functional
gaps = pb.complement(df, view_df=genome, output_type="polars.DataFrame")

# Method-chaining (LazyFrame)
gaps = df.lazy().pb.complement(genome).collect()

Parameters

Output Schema

Returns a DataFrame with chrom (String), start (Int64), end (Int64) columns representing gaps between intervals.

subtract

Remove portions of intervals in df1 that overlap with intervals in df2.

# Functional
result = pb.subtract(df1, df2, output_type="polars.DataFrame")

# Method-chaining (LazyFrame)
result = df1.lazy().pb.subtract(df2).collect()

Parameters

Output Schema

Returns chrom (String), start (Int64), end (Int64) representing the remaining portions of df1 intervals after subtraction.

Performance Considerations

Probe-Build Architecture

Two-input operations (overlap, nearest, count_overlaps, coverage, subtract) use a probe-build join:

• Probe (first DataFrame): Iterated over, row by row
• Build (second DataFrame): Indexed into an interval tree for fast lookup

For best performance, pass the larger DataFrame as the probe (first argument) and the smaller one as the build (second argument).

Parallelism

By default, polars-bio uses a single execution partition. For large datasets, enable parallel execution:

import os
import polars_bio as pb

pb.set_option("datafusion.execution.target_partitions", os.cpu_count())

Streaming Execution

DataFusion streaming is enabled by default for interval operations. Data is processed in batches, enabling out-of-core computation for datasets larger than available RAM.

When to Use Lazy Evaluation

Use scan_* functions and lazy DataFrames for:

• Files larger than available RAM
• When only a subset of results is needed
• Pipeline operations where intermediate results can be optimized away

# Lazy pipeline
lf1 = pb.scan_bed("large1.bed")
lf2 = pb.scan_bed("large2.bed")
result = pb.overlap(lf1, lf2).collect()