Video Pipeline

This notebook demonstrates docling's video processing pipeline, which:

Transcribes audio using Whisper ASR
Samples representative frames using scene-change detection
Optionally assigns speaker labels via diarization
Exports to HTML, Markdown, JSON, or WebVTT

Use cases:

Business meeting recordings → searchable transcript with frames
Lecture videos → structured notes with slide captures
Any video → subtitle file (.vtt) played back with captions in a browser

Setup

Install docling with video support (ASR + speaker diarization):

bash

pip install 'docling-slim[format-video]'

FFmpeg must also be installed on your system:

bash

# macOS
brew install ffmpeg
# Ubuntu/Debian
sudo apt-get install ffmpeg

python

from pathlib import Path

from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import VideoPipelineOptions
from docling.document_converter import DocumentConverter, VideoFormatOption
from docling.utils.video_frame_sampling import VideoFrameSamplingMode


def print_transcript(document):
    """Print each transcript segment as [mm:ss] [speaker] text."""
    for item, _ in document.iterate_items():
        if not (hasattr(item, "text") and item.text):
            continue
        track = item.source[0] if item.source else None
        ts = track.start_time if track else 0.0
        speaker = f" [{track.voice}]" if track and track.voice else ""
        m, s = divmod(int(ts), 60)
        print(f"[{m:02d}:{s:02d}]{speaker} {item.text.strip()}")

Basic Usage

Convert a video using fixed-interval frame sampling (one frame every 10 seconds):

python

VIDEO_PATH = "path/to/your/video.mp4"  # replace with your video

dc = DocumentConverter(allowed_formats=[InputFormat.VIDEO])
result = dc.convert(VIDEO_PATH)

print(f"Status: {result.status}")
print(f"Transcript segments: {len(result.document.texts)}")
print(f"Frames captured: {len(result.document.pictures)}")

Scene-Change Detection

For meeting recordings, scene-change sampling captures frames at meaningful transitions rather than at fixed intervals. The prominence parameter controls sensitivity — lower values detect more subtle scene changes.

Recommended settings:

Business meetings: prominence=0.03
Lectures with slides: cuts_per_minute=2
Dynamic content: prominence=0.01

python

opts = VideoPipelineOptions(
    frame_sampling_mode=VideoFrameSamplingMode.SCENE_CHANGE,
    scene_change_prominence=0.03,  # recommended for meetings
    min_scene_duration_seconds=2.0,
)

dc = DocumentConverter(
    allowed_formats=[InputFormat.VIDEO],
    format_options={InputFormat.VIDEO: VideoFormatOption(pipeline_options=opts)},
)
result = dc.convert(VIDEO_PATH)

print_transcript(result.document)

Speaker Diarization

Enable speaker diarization to identify who is speaking in each segment. Requires resemblyzer and scikit-learn. Speaker count is automatically detected.

python

opts = VideoPipelineOptions(
    frame_sampling_mode=VideoFrameSamplingMode.SCENE_CHANGE,
    scene_change_prominence=0.03,
    enable_diarization=True,  # requires resemblyzer
)

dc = DocumentConverter(
    allowed_formats=[InputFormat.VIDEO],
    format_options={InputFormat.VIDEO: VideoFormatOption(pipeline_options=opts)},
)
result = dc.convert(VIDEO_PATH)

print_transcript(result.document)

Export Formats

The DoclingDocument produced by the video pipeline can be exported to any format that docling supports.

python

output_dir = Path("output")
output_dir.mkdir(exist_ok=True)

# HTML — transcript with timestamps and embedded frames
result.document.save_as_html(output_dir / "video.html")

# Markdown — plain transcript
result.document.save_as_markdown(output_dir / "video.md")

# JSON — full structured document
result.document.save_as_json(output_dir / "video.json")

# WebVTT — subtitle file
result.document.save_as_vtt(output_dir / "video.vtt")

print("Exported to:", list(output_dir.iterdir()))

Use Case: Play the Video with Captions in a Browser

Build a minimal HTML page that plays the video with the exported WebVTT captions overlaid, using the standard <video> + <track> elements:

python

video_name = Path(VIDEO_PATH).name
player_html = output_dir / "player.html"
player_html.write_text(f"""<!DOCTYPE html>
<html>
<video controls src="{video_name}">
  <track default kind="captions" srclang="en" label="English" src="video.vtt" />
  Video not supported.
</video>
</html>
""")

print(f"Player page saved to: {player_html}")

Note: Browsers block <track> (caption) loading for pages opened directly from disk via file://. Copy your video into output/ next to player.html, then serve the directory and open it over HTTP:
bash
python -m http.server 8080 --directory output
then open http://localhost:8080/player.html.

Firefox: captions may not appear until you enable them from the video's CC/subtitles menu — unlike some browsers, Firefox does not always turn on a default track automatically.