pj_scene3D/docs/REQUIREMENTS.md
pj_scene3D — Requirements| Status | Shipped (v1 feature-complete: TF, pointclouds, occupancy grids, dense voxel grids, depth-image back-projection, markers, pose arrays, URDF/mesh, HDR/SSAO/EDL, live streaming) |
| Date | 2026-05-17 (rev. 2026-06-20) |
| Scope | What, not how |
| Supersedes | PJ4_PLAN.md §5.5 (refinement) |
This document is the source of truth for the intent of pj_scene3D. It deliberately avoids design-level prescriptions (API signatures, class layouts, CMake details, file structures). Those belong in a subsequent design document. When this document conflicts with PJ4_PLAN.md §5.5, this document wins.
Current architecture (supersedes the Phase-1 framing below). The module ships integrated into
pj_app(not as a standalone demo), and it does not decode wire formats. DataSource plugins (e.g.parser_ros, pj-official-plugins#122) decode ROS/CDR messages into canonicalpj_base/builtinobjects (PointCloud,FrameTransforms,OccupancyGrid,OccupancyGridUpdate) and publish them to theObjectStore;pj_scene3Dconsumes those objects from the store on tracker change and renders them. Pointclouds, TF, and occupancy grids / costmaps are implemented. 3D markers (SceneEntitiesLayer— visualization_msgs/MarkerArray-equivalentkSceneEntitiestopics), URDF/mesh robot models (RobotModelLayer), and pose arrays (PosesInFrameLayer—geometry_msgs/PoseArray/foxglove.PosesInFramedrawn as per-pose coordinate-triad gizmos) are also implemented. The Phase-1 sections below are retained as historical intent — read "lazy host-side CDR decode ofPointCloud2" as "consume the canonicalPointCloudobject the plugin already decoded".
PJ4's 3D visualization module — the sibling family to pj_scene2D, focused on the robotics-data subset: TF trees, rigid bodies via URDF, pointclouds, gridmaps, 3D markers and primitives, paths, laserscans.
Target user: roboticists who use PlotJuggler 4 to debug recorded or live data alongside their time-series plots, scrubbing back and forth through replay time. Inspirations: RViz2, Foxglove, Rerun.
map vs odom vs base_link, etc.).Full v1 target:
nav_msgs/OccupancyGrid + incremental OccupancyGridUpdate, with stateful time-travel reconstruction). (implemented — OccupancyGridLayer)grid_map_msgs/GridMap elevation-map sense (textured/height planes in a source frame) remain future work — distinct from the occupancy grids above.sensor_msgs/PointCloud2 and equivalents). (implemented)foxglove_msgs/CompressedPointCloud + point_cloud_interfaces/CompressedPointCloud2), formats Draco and Cloudini — decoded into a PointCloud and rendered identically (see §3a). (implemented)SceneEntitiesLayer)geometry_msgs/PoseArray, foxglove.PosesInFrame → canonical PosesInFrame), drawn as per-pose coordinate-triad gizmos with customizable arrow length + opacity, an X-arrow-only geometry mode, and an orthogonal color override (one shared color across all arms, in either mode). (implemented — PosesInFrameLayer)foxglove.VoxelGrid → canonical VoxelGrid), drawn as GPU-instanced cubes; the per-voxel value is generic (occupancy/cost/ESDF/semantic via fields, or a direct RGBA channel) and a viewer-side draw predicate + colormap decide which voxels are shown. The dense→cubes expansion is entirely GPU-side (one glDrawElementsInstanced, gl_InstanceID→texel), so the CPU / draw-call cost is independent of voxel count (the GPU vertex shader still evaluates the draw predicate per voxel). (implemented — VoxelGridLayer)sdk::Image with a depth encoding joined to a CameraInfo by frame_id, back-projected one point per valid pixel and colored by depth. (implemented — DepthCloudLayer)nav_msgs/Path).sensor_msgs/LaserScan).Phase 1 subset (two types):
sensor_msgs/PointCloud2).Image+Pinhole (camera frustum + textured near-plane) from the original PJ4_PLAN.md §5.5 list is dropped from v1.
A CompressedPointCloud canonical object carries {timestamp, frame_id, format, data} where
data is a self-describing codec blob. pj_scene3D decodes it into a PointCloud and renders it
through the same PointCloudLayer and convertCanonical() path as a raw cloud — one dual-mode
layer, no separate widget.
cloudini (header-embedded schema; via cloudini/1.2.2) and draco
(draco/1.5.6 — pinned to 1.5.6 to match the assimp/5.4.3 glTF importer requirement;
see conanfile.txt). Plain zstd_point_cloud_transport is out of scope — its blob is
not self-describing (it relies on layout fields the canonical object does not carry).foxglove_msgs/CompressedPointCloud (Foxglove ROS2 schema; its pose is read but dropped — clouds
are placed via TF on frame_id) and point_cloud_interfaces/CompressedPointCloud2
(ros-perception/point_cloud_transport). The parser only repackages bytes; it never decodes.core/pointcloud_codecs.cpp (decodeCompressedPointCloud), draco + cloudini
linked PRIVATE to pj_scene3d_core — mirrors pj_scene2d_core decoding JPEG/PNG.QtConcurrent + QFutureWatcher) with latest-wins coalescing; the decoded cloud is
cached per sample — identity is (store timestamp, payload byte size) — so repaints and color-field
changes don't re-decode, and a tracker tick that resolves to the already-pushed sample skips the
re-conversion/upload entirely. A sample whose decode fails clears the view (matching the raw
path's malformed-cloud behavior) and is never retried. Fast scrubbing of very large clouds may
show transient lag (the in-flight sample lands a frame late); async decode + the per-sample cache
keep the UI responsive.INT64/UINT64 fields (no PJ datatype) are dropped (their bytes
still occupy point_step, so surviving fields keep their offsets). Draco field names are
recovered from Draco attribute metadata when present (Foxglove / draco_point_cloud_transport store
the original name, e.g. intensity/ring), else inferred from attribute type (POSITION→x/y/z,
COLOR→red/green/blue/alpha, NORMAL→nx/ny/nz, else generic_N).frame_locked=false toggle is intentionally not adopted).SceneEntitiesLayer additionally honors each SceneEntity's lifetime_ns (0 = never expires): the entity is dropped once the tracker passes its expiry. Expiry is anchored on the ingest (ObjectStore entry) timestamp the entity was folded from — the host/tracker clock — not the entity's embedded sensor timestamp, which under live streaming rides a different epoch. There is still no hold-and-fade: an expired entity disappears, it does not decay.Full v1: per-widget fixed-frame + display-frame + follow-mode, with Foxglove-parity follow modes (Pose / Heading / Position / Off). Each widget independent — two widgets side-by-side can render the same scene against different frames with different cameras.
Phase 1 subset: per-widget fixed-frame dropdown + a camera "Follow frame" picker (the "Camera" section atop the scene-controls panel). Follow is Position-only so far: the active camera's anchor tracks the chosen frame's origin (in the fixed frame) each tracker tick, so the view pans to keep the frame in place while orbit/zoom stay world-referenced ("None" = off). A recenter button beside the picker snaps the camera onto the followed frame on demand (the on-demand counterpart to the no-jump-on-enable policy). The remaining follow modes (Heading = + yaw, Pose = + full orientation) and the separate display-frame + RViz-style Frames panel are Phase 2+. Both dropdowns list the whole TF buffer (every frame ever ingested, time-independent — getFrameHierarchy() takes no time argument) and are refreshed as transforms are folded in (on datasetTransformsReady / live ingest), so the tree fills in during a file load without needing to press play.
A newly-created widget does not blindly default to the map/world/odom/… heuristic root: if the user has already picked a fixed frame by hand for the same dataset's TransformBuffer, the new widget defaults to that remembered frame (when it is still present in the tree), falling back to the heuristic otherwise. The choice is shared in-session across sibling widgets on the dataset and persisted across restarts (see §10). It stays an auto-mode preference — picking it does not flip the widget to explicit, so it still self-heals if the frame later disappears. TransformService::rememberFixedFrame/rememberedFixedFrame own the memory; Scene3DDockWidget::resolveAutoFixedFrame consumes it.
Phase 1 — RViz Orbit subset:
| Gesture | Action |
|---|---|
| Left-drag | Rotate around focal point (azimuth + elevation) |
| Middle-drag (or shift+left-drag) | Pan focal point in screen plane |
| Wheel scroll | Zoom toward focal point — shrink radius, target fixed (not a dolly) |
| Right-drag | Axis-based zoom (RViz parity) |
| Toolbar reset | Restore default view |
Coordinate convention: ROS Z-up (per PJ4_PLAN.md §5.5).
Implemented: Orbit, XYOrbit, TopDownOrtho, and Fly (WASD / FPS) — all four selectable via the scene-controls combo. Zoom-to-cursor and Home control are also implemented. Position-only "follow a frame" composes with every model via the ICamera::followShift(world_delta) seam (Orbit/TopDown shift the focal, Fly the eye, XYOrbit XY-only to stay ground-locked); SceneViewWidget::applyFollow drives it per tracker tick using the delta of the followed origin so user orbit/zoom is preserved and enabling causes no jump.
Still deferred: Heading/Pose follow modes, bookmark / saved views, animated transitions, F-recenter-on-selected-drawable.
Multiple SceneViewWidget instances cohabit in a PJ4 layout (via the existing ADS docking system). Each widget is fully independent:
No camera sync, no fixed-frame sync, no shared selection. Foxglove-style "sync camera across panels" is explicitly deferred from v1.
Performance is benchmark-driven, not pre-committed at the requirements stage. The pointcloud rendering pipeline in particular is to be discovered through the demo / benchmark phases, modeled on pj_scene2D's mpv-vs-ffmpeg shootout.
Headline target (post-Phase-1): aggregate ~10 million points snappy across N concurrent pointcloud displays in a single PJ4 process, characterized empirically against a stress-test MCAP set.
Render loop policy (Phase 1 forward):
QSurfaceFormat::setSwapInterval(1) (~60 Hz on standard monitors).This matches the pattern pj_scene2D already uses for video playback.
The TF buffer is the central runtime data structure that makes scrub-replay work for 3D data. Its requirements:
TransformBuffer per loaded dataset. Ownership lives in
TransformService (pj_scene3d_widgets), not in pj_runtime::SessionManager — the
runtime must stay domain-neutral and must not depend on 3D-specific types.
TransformService exposes the buffer as a shared_ptr so every 3D dock attached to the
same dataset shares the already-populated buffer without re-walking the store.FrameTransforms objects and publish them to the
ObjectStore; TransformService consumes those objects.TransformService probes every
object topic via SessionManager::parserBindingForObjectTopic(), classifies TF topics at
most once, and ingests entries using a per-topic timestamp cursor (TfCursor) that guards
against double-ingest. Two ingest paths share the same cursor logic:
ingestFrameTransformsForDataset() sweeps the whole topic history
in one call on a worker thread; emits datasetTransformsReady when done.ingestNewTransforms() is called on each
SessionManager::samplesIngested signal; it ingests only entries newer than the cursor
and returns true if the buffer changed. Scene3DDockWidget::driveVisibleLayersToLiveEdge()
then advances the visible layers to the new live edge.Behavioral contract of TransformBuffer (core/include/pj_scene3d_core/tf/tf_buffer.h):
T_target_from_source convention (same as ROS tf2): lookupTransform(target, source, t) returns the transform T such that p_target = T * p_source.
Tree, not DAG: each child frame has exactly one parent. Two forms of malformed input are rejected: a reparent conflict (child already claimed under a different parent) returns SetTransformError::ReparentConflict; a self-loop edge (child == parent) returns SetTransformError::SelfLoop to prevent chainToRoot from hanging on corrupt TF data. Both are non-throwing — a bulk ingest continues past dropped edges.
Zero-order hold (ZOH) sample lookup, not linear interpolation:
t equal to a stored stamp → that sample.t strictly between two stamps → the earlier sample (sample-and-hold).t > latest stamp → the latest sample (forward extrapolation by hold).t < earliest stamp on any edge → no result (NoSampleAtTime).Transforms with a single sample (e.g. /tf_static entries) resolve for all
t >= their stamp and hold forward indefinitely; for t strictly before
that stamp, NoSampleAtTime is returned. In practice /tf_static is stamped
at recording start, so this slice never occurs during normal scrub replay.
This is a deliberate departure from tf2's default linear interpolation. The ZOH semantic answers "what was the system's belief at time t?" rather than "what would a smooth interpolation of the recorded data look like at time t?". For scrub replay over recorded data, the first question is the correct one — the system never actually held an interpolated pose.
Thread-safe: concurrent writers (ingest thread) and readers (render thread + future Frames panel) must be supported without external locking.
Introspection: callers must be able to enumerate all known frames, query the parent of any frame, and query the latest sample stamp for any edge. This is what the future Frames panel and the per-widget fixed-frame dropdown consume.
Non-throwing query variant for the hot render path. A throwing variant remains available for explicit user code where a missing transform is a programmer error.
Per-widget fixed-frame, camera follow target, per-layer configuration + visibility, and the active camera model + pose are serialized into the PJ4 layout XML and restored on load. Scene3DDockWidget::xmlSaveState/xmlLoadState writes a versioned <Scene3DDock> (fixed-frame; follow_frame — empty = off; camera model as a stable string id + CameraState JSON; dataset re-resolution by source), and every layer implements its own xmlSaveState/xmlLoadState. A restored follow target absent from the current TF tree stays inert until it appears. The as-built schema (camera persistence + the per-layer round-trips) lives in ARCHITECTURE.md.
Separately from the layout XML, the last fixed frame a user picked by hand for a dataset is remembered across app restarts in QSettings under pj_scene3d/fixed_frame_by_source, keyed by the dataset's stable source name (DatasetInfo::source_name, the same identity layout restore matches datasets by — not the per-session DatasetId). This is the cross-restart half of the §5 "new widget defaults to the remembered frame" behavior; a layout's explicitly-saved fixed frame still wins for the specific widget it was saved on, and a dataset with no source name (e.g. an unnamed live stream) is remembered for the session only.
Third-party drawable type registration by plugins. Out of v1 scope. Built-in drawable types only in v1.
CLAUDE.md).frame_locked opt-in — chose always-locked (§4).lifetime_ns expiry itself is implemented — see §4.)VoxelGrid. (Dense voxel grids — costmap-3D / ESDF / semantic — are supported via VoxelGridLayer; see §3.)Phase 1 ships an end-to-end MCAP-driven demo binary that exercises the substrate, the adapted TF buffer, the lazy pointcloud pipeline, and the per-widget UI. It is not a substrate-only proof; it is the smallest demo that proves the foundation works on real data.
A ROS2 MCAP file (provided by the user) containing at minimum:
tf2_msgs/TFMessage transforms.sensor_msgs/PointCloud2 pointcloud(s).TFMessage sample pushed into the per-dataset TF buffer.pj_scene2D's MCAP lazy-media pattern. Raw CDR bytes remain indexed in MCAP at open; decoding occurs on demand through a small LRU cache for recently decoded clouds.core/tf/tf_connections.h helper; the GL shell is TfConnectionsRenderPass.GL_POINTS (single VBO, no LOD), in its source frame, re-transformed to the widget's fixed-frame each frame via the TF buffer.{ X, Y, Z, intensity, ring }. The dropdown is populated by inspecting the first decoded cloud's fields schema (the timestamp field is explicitly ignored). The selected field drives a colormap — Phase 1 shipped Turbo only; the current build offers the shared PJ::Colormap set (turbo / viridis / plasma / grayscale, from pj_widgets/Colormap.h, the same colors as the 2D depth view). For a spatial axis (X/Y/Z) the colormap input is the coordinate in the fixed frame, computed on the GPU from the same source→fixed transform that places the geometry — so sensors at different mounts agree on world height (the raw x/y/z field is sensor-local). Its auto-range is derived per frame from the cloud's source-AABB transformed by that live model, so colour and range track TF motion without a re-decode. Non-spatial fields (intensity, ring, …) colour by the raw per-point value and auto-fit to the cloud's min/max; both spatial and non-spatial ranges are held across tracker scrub so colors stay stable.
rgba field of datatype kUint32 whose 4 bytes are R, G, B, A in increasing-address order; the DataSource parsers normalize to it (parser_protobuf collapses foxglove's separate red/green/blue/alpha uint8 channels; parser_ros repacks a PCL-packed rgb/rgba 0x00RRGGBB field), so the host renders one canonical colour and never sees per-source byte order. The host (detectColorLayout in pointcloud_convert) also defensively recognizes raw channels, so an un-normalized source still renders true colour. alpha is treated as opaque padding (ignored for blending); see pointcloud_layer.cpp (mode selection / persistence) and passes/pointcloud_render_pass.cpp (ColorType::kRgb).The demo passes Phase 1 when, on the user-provided MCAP, all of the following hold:
timestamp).Scene is implemented with hardcoded axes_ and pointcloud_ members; no premature IDrawable polymorphism. (Polymorphism arrives in Phase 2 when the third drawable type lands.)GL_DEBUG_OUTPUT is clean.assimp, URDF, tinyply, PCD reader, RGB-direct color mode, additional colormaps beyond Turbo, display-frame, follow-mode, TF Frames panel, picking, measurement, click-to-publish, gridmaps, paths, laserscans, integration into pj_app (the Phase 1 demo is a standalone binary, mirroring pj_scene2D/tools/).
Note (post–Phase 1): several of the above have since shipped — URDF/mesh models (assimp), occupancy grids, the additional colormaps, depth-image back-projection (
DepthCloudLayer), the RGB-direct color mode (see Pointcloud coloring above), and fullpj_appintegration. This list is the historical Phase-1 boundary; for the current as-built feature set seeCLAUDE.mdandARCHITECTURE.md.
Note (post–Phase 1): several of the above have since shipped — URDF/mesh models (assimp), occupancy grids, the additional colormaps, depth-image back-projection (
DepthCloudLayer), and fullpj_appintegration. This list is the historical Phase-1 boundary; for the current as-built feature set seeCLAUDE.mdandARCHITECTURE.md.
Note (post–Phase 1): several of the above have since shipped — URDF/mesh models (assimp), occupancy grids, the additional colormaps, depth-image back-projection (
DepthCloudLayer), the RGB-direct color mode (see Pointcloud coloring above), and fullpj_appintegration. This list is the historical Phase-1 boundary; for the current as-built feature set seeCLAUDE.mdandARCHITECTURE.md.
QOpenGLWidget.pj_base/builtin schemas read from the ObjectStore
(no wire/CDR decoding in this module — see the architecture note at the top).gtest (matches pj_scene2D).pj_scene3D (OpenGL) and pj_scene2D (QRhi) form a mixed GPU stack inside the same PJ4 process. They are sibling widget families that never share GPU state. An IRenderPass invariant in the substrate keeps GL calls confined so a future QRhi swap would be a contained refactor rather than a rewrite.
The TF prototype has been promoted into pj_scene3D/core/include/pj_scene3d_core/tf/ as the TransformBuffer class (tf/tf_buffer.h, tf/transform.h) — single source of truth, no wrapper layer. The edits applied during promotion were:
sampleAt rewritten to zero-order hold (nearest sample at or before t), replacing the prototype's lerp+slerp interpolation. The free-function interpolate() becomes unused inside the buffer (may be deleted or kept exposed as a free utility).TimePoint retyped from std::chrono::steady_clock::time_point to a replay-time-friendly type compatible with PlaybackEngine's tracker time. (Final concrete type chosen at integration time.)std::shared_mutex guarding the buffer: setTransform exclusive; lookupTransform, tryLookupTransform, canTransform, latestCommonTime, and the introspection getters take shared locks.tryLookupTransform(target, source, t) → PJ::Expected<Transform, LookupError> — the primary accessor for the render hot path; the typed LookupError enum (UnknownSource/UnknownTarget/Disconnected/NoSampleAtTime) keeps render-loop misses allocation-free. The throwing lookupTransform is a thin wrapper over it.setTransform normalizes the quaternion on insert to defend against ingest noise accumulating through composed transforms.Unchanged from the prototype:
T_target_from_source convention.setTransform returns PJ::unexpected(SetTransformError::ReparentConflict) or SetTransformError::SelfLoop respectively, dropping that one edge (non-throwing) so a bulk ingest continues. Self-loop rejection fires first to prevent chainToRoot from hanging on malformed TF data.std::deque sample storage with 10-second cache window pruning.Transform struct (translation + unit quaternion).