PJ4_PLAN.md
This section captures the architectural decisions that supersede portions of this document. The body below remains as planning context; where it conflicts with this section, this section wins.
pj_plotting (Qwt, lifted wholesale from PJ3), pj_scene2D/widgets via the pj_scene2d_widgets target (QRhi, wraps pj_scene2D/core), and pj_scene3D/widgets via the pj_scene3d_widgets target (hand-rolled OpenGL 4.5 Core, wraps pj_scene3d_core). Heterogeneity is a feature, not a problem. Shared reusable Qt controls/helpers live in pj_widgets — including the reusable Source Timeline control (see §5.6); shared runtime state flows through the small IDataWidget contract exposed by pj_runtime.pj_scripting is its own module. Language-agnostic engine (Luau today — a hand-written Luau binding behind a ScriptEngine seam; NOT sol2, whose API is incompatible with Luau's C++ linkage. Python pluggable later) decoupled from both the GUI and pj_runtime's services layer. (The plan body below still uses the older IScriptingEngine/LuaScriptingEngine/sol2 planning names; the as-built names are ScriptEngine / LuauEngine / makeLuauEngine.) Depends only on pj_base + pj_datastore. Custom Lua transforms reach DerivedEngine through a thin transform_adapter. Reactive scripts live in a Toolbox plugin that links pj_scripting directly.pj_runtime Qt boundary is relaxed. Previously "no Qt"; now Qt is allowed (QObject, QTimer, QSettings, signals), but no concrete QWidget/QDialog implementation and no Qt6::Widgets link. IDataWidget may forward-declare QWidget as the shell contract. Services remain headlessly testable via QCoreApplication. This trades a small amount of purity for much cheaper timer/settings/reactive plumbing.PlotWidgetBase, PlotWidget, PlotDocker, TabbedPlotWidget, zoomers, axis-time, drag-drop, per-curve display transform UI all move into pj_plotting/; their data reads are rebound to pj_datastore via a DatastoreCurveAdapter. No IPlotBackend abstraction.pj_datastore, pj_scene2D, pj_scene3D, pj_scene_common, pj_marketplace, pj_dialog_host, pj_scripting, pj_runtime, pj_widgets, pj_plotting, pj_app) live inside this repository. pj_datastore (the columnar storage engine) was moved out of the plotjuggler_sdk submodule into the app repo, because plugins reach storage only through the pj_base C ABI and never link the engine — so the plotjuggler_sdk submodule is purely the plugin SDK (pj_base + pj_plugins). The long-term intent is still a separate plotjuggler_app repo with the plotjuggler_sdk SDK as a submodule; module boundaries are designed so that split is a mechanical move later.pj_scripting), reactive scripts (via Toolbox + onTimeChanged), multi-tab workspace, marketplace install UI, all toolboxes.onTimeChanged, ScatterXY outputs) are being addressed in parallel and are assumed solved.PlaybackEngine in pj_runtime. Plots redraw a vertical marker line (cheap); 2D/3D widgets snap to the frame at tracker time (via MediaSource::setTimestamp / equivalent).Where the sections below refer to pj_app_shell_qt, read that as pj_app.
This document defines the implementation plan for building a full PlotJuggler 4.x desktop application with feature coverage comparable to PlotJuggler 3.x, while modernizing the internal architecture.
The goal is not to recreate the 3.x code structure. The goal is to recreate the 3.x product capability set on top of the foundation libraries:
pj_base (in the plotjuggler_sdk SDK submodule)pj_plugins (in the plotjuggler_sdk SDK submodule)pj_datastore (top-level app module; moved out of the submodule)The main design objective is a strict separation between:
The removed prototype app was a useful proof of concept, but it was not a sufficient foundation for the final 4.x app architecture. The final app is organized around dedicated modules instead of growing the prototype window into the final application.
The 4.x app is considered feature-complete with respect to 3.x when it provides:
These items are not required for the first feature-complete target:
StatePublisher parity4.x targets modernized parity:
MainWindow designThe current repository already provides substantial infrastructure:
pj_datastore
ObjectStore for media blobs (images, video frames, scene primitives) with lazy fetch and retentionpj_plugins
DataSource, MessageParser, Dialog, and Toolboxpj_scene2D
ObjectStoreMediaSource pull-based abstraction (ImagePipelineSource, StreamingVideoSource)pj_scene2d_widgets QRhi widget with BT.709 YUV→RGB shaderpj_marketplace
The missing work is the full application layer that 3.x had:
The final app should be built in three concrete levels:
These already exist and remain the substrate:
pj_base (in the plotjuggler_sdk SDK submodule)pj_plugins (in the plotjuggler_sdk SDK submodule)pj_datastore (top-level app module — ObjectStore + DerivedEngine; moved out of the submodule, since plugins reach it only via the pj_base C ABI)pj_scene2D (2D/video visualization)pj_scene_common (backend-agnostic shared scene-dock framework consumed by the 2D/3D widget families)pj_marketplacepj_scripting (new — language-agnostic engine, Lua today)They provide the vocabulary types, columnar + object stores, extension ABI/runtime, media pipeline, marketplace services, and scripting runtime.
pj_runtime)This is the authoritative business layer for the 4.x app.
Responsibilities:
Constraints (revised):
QObject, QTimer, QSettings, signalsQWidget / QDialog in this layerQCoreApplicationpj_widgetsReusable Qt building blocks that are not tied to PlotJuggler's app shell live in pj_widgets. This is for controls and UI utilities such as color pickers, SVG icon loading, numeric sliders, and the reusable Source Timeline control (§5.6). It is not itself a widget family and it must not own application state.
The concrete desktop UI layer is three independent widget families, each with its own rendering world. They are siblings: they do not depend on each other.
pj_plotting — Qwt-based plot widgets (lifted wholesale from PlotJuggler 3.x). Time-series, XY, tracker, zoomers, per-curve display transforms.pj_scene2D/widgets — 2D viewer widgets built as the pj_scene2d_widgets target and wrapping pj_scene2D/core. Image, video, depth, annotation display via QRhi.pj_scene3D/widgets — 3D viewer widgets built as the pj_scene3d_widgets target, wrapping pj_scene3d_core. Hand-rolled OpenGL 4.5 Core renderer (QOpenGLWidget + QOpenGLFunctions_4_5_Core), with TF, pointcloud, occupancy-grid, axis/grid render passes; integrated into pj_app via Scene3DDockWidget + TransformService.All three implement the small IDataWidget contract from pj_runtime (tracker callback, save/load state, subscribed topics, Qt widget accessor). They may use reusable Qt helpers from pj_widgets, but they do not depend on each other. The variability of these three rendering worlds is by design.
pj_app)This is the main-window shell.
Responsibilities:
pj_runtime services to widgets from all three familiesConstraints:
The source of truth for each concern must be explicit:
pj_datastoreAll important application changes should be performed by app-core services instead of direct widget manipulation:
This keeps the GUI thin and ensures the workspace, playback, and session state remain coherent.
Use concrete service classes as the internal API by default.
Examples:
AppSessionSessionManager (formerly DataSessionManager)WorkspaceManagerPlaybackEngineTransformRegistryToolboxManagerExtensionCatalogServiceUndoManagerUserSettingsServiceIntroduce abstract interfaces only when there is a real boundary or more than one plausible implementation, such as:
pj_plugins)IDataWidget (the small contract bridging app_core to the three widget families)IScriptingEngine (the boundary in pj_scripting allowing Lua/Python/other)The GUI should not reach into DataEngine, DerivedEngine, ObjectStore, or extension handles directly unless the operation is purely read-only and already mediated by a service.
The final app is organized as several sibling modules on top of the existing foundation libraries, all living inside this repository (with a future split into plotjuggler_app planned but not required).
pj_scripting — scripting engine (new foundation module)Purpose:
pj_runtime and by Toolbox plugins that provide script editors (e.g. Lua editor, reactive scripts)Depends on: pj_base, pj_datastore (for transform interfaces).
Contents:
IScriptingEngine — abstract interface (compile, eval, registerFunction, lastError, languageName)Value — neutral data exchange type (scalar/vector/timestamp variant)LuaScriptingEngine — Lua implementation (sol2 or port of PJ3's Lua wrapper)transform_adapter — wraps a compiled ScriptHandle as ISISOTransform / IMIMOTransform for DerivedEnginePythonScriptingEngine as an additional source file, no other changesRule:
pj_runtime — business servicesPurpose:
Main owned objects:
SessionManager — owns the datastore + ObjectStore, DataSource plugin lifecyclePlaybackEngine — global tracker, play/pause, rate, follow-live (QTimer-driven)CatalogModel — Qt model wrapping topics/fields; drag-mime source for all widget familiesWorkspaceManager — JSON layout save/load, schema versionedWidgetRegistry — maps widgetType string → factory for layout restoreTransformRegistry — wraps DerivedEngine, registers built-ins + script-backed transformsToolboxManager — toolbox plugin lifecycle, onTimeChanged dispatchUserSettingsServiceUndoManager — snapshot-basedExtensionCatalogService — facade over pj_marketplaceNotificationCenterSmall contracts defined here:
IDataWidget — the tiny contract the three widget families implementstd::function or Qt signals)Constraints:
QWidget / QDialog implementation and no Qt6::Widgets linkQCoreApplicationpj_plotting — Qwt plot widgets (lifted from PJ3)Purpose:
Approach:
PlotWidgetBase, PlotWidget, PlotDocker, TabbedPlotWidget, zoomers, AxisTimeOffset, custom tracker, drag-drop, context menus, curve list helpers (per-curve display transform UI ports later — see deferral note below)PlotDataMapRef reads with DatastoreCurveAdapter — a QwtSeriesData<QPointF>-derived class that pull-throughs to pj_datastore via SessionManager (no copy of timeseries data; see §8.2 for the chunk-index design)QwtPlotCurve::FilterPointsAggressive + ClipPolygons) for large datasets — no adapter-side decimation; pre-decimating breaks on zoomIDataWidget on the plot widget; register the plot factory directly in pj_app (no WidgetRegistry service in v1; introduce one only when 2D / 3D widget families need symmetric registration)Deferred to a follow-up phase (still planned, not in v1 scope): PlotWidgetEditor (full edit-curves dialog with line width / Y limits / style toggles), PlotWidgetTransforms (per-curve display-transform dialog), StatisticsDialog, PlotBackground colored-zone item, colormap selector / editor. The four v1 demo features (drag-drop, color via stock QColorDialog from a context menu, sync-zoom, datastore plumbing) are the acceptance bar.
Constraints:
pj_scene2d_widgets or pj_scene3d_widgetspj_scene2D/widgets — 2D media widgetsPurpose:
Approach:
pj_scene2D/core using MediaViewerWidget and QRhi renderingPlaybackEngine::trackerChanged → call MediaSource::setTimestamp → request repaintIDataWidget; register factoryConstraints:
pj_plotting or pj_scene3d_widgetspj_scene3D — 3D widget family (robotics viz)Implemented and wired into pj_app (built via add_subdirectory(pj_scene3D/core) + add_subdirectory(pj_scene3D/widgets); MainWindow instantiates Scene3DDockWidget + TransformService). See pj_scene3D/docs/REQUIREMENTS.md for the full spec (it supersedes this section).
Data types (v1 target, 6 total): TF2 (infrastructure), URDF/mesh, Pointcloud, Markers (arrows/boxes/spheres/cylinders/line strips/text), Image+Pinhole (frustum + optional textured near-plane), OccupancyGrid (textured plane in 3D).
Storage: all via pj_datastore::ObjectStore. Pointcloud is lazy-fetch (like MCAP images in pj_scene2D); everything else lives in memory. DataSource plugins write via existing object_write_host.push(topic, encoding, bytes, t) using per-type encoding strings — no new write hosts.
Stack (locked):
QOpenGLFunctions_4_5_Core)QOpenGLWidget subclass (SceneViewWidget), hand-rolled scene (no Qt 3D, no Qt 3D scene graph)glm::slerp for TF interpolation, header-only conan dep)QDomDocument (see pj_scene3D/docs/ARCHITECTURE.md)PointCloud2)QPainter + QFont → QImage → GL texturepj_scene2DScene organization: flat per-widget drawable list (Foxglove-lean; RViz-style per-type config knobs can be added incrementally). Scenes in robotics contexts are small enough that a deep scene graph is unnecessary.
Caching: minimal. Pointcloud drawables cache GPU buffers across tracker changes; everything else re-decodes on change (cheap).
TF interpretation layer (stateful: per-edge ring buffer + slerp/lerp interpolation + tree traversal over ObjectStore TF samples) lives inside pj_scene3D for v1. To be reviewed later: if 2D widgets or pj_runtime transforms need TF too, promote to a sibling module pj_tf.
Interaction: orbit camera (drag rotate / wheel zoom / middle-drag pan), drag-drop topics from CatalogModel, per-drawable context menu (visibility / color / delete), click-to-select picking.
Deferred (post-v1): shared GPU resources across multiple 3D widgets, advanced pointcloud LOD, costmap 3D / octomap, richer per-display config UI.
Inspiration source (not a dependency): threepp (MIT-licensed C++20 Three.js port) is read as a reference for specific rendering patterns (URDF traversal, OrbitControls math, Raycaster algorithm, material abstractions). Logic may be ported with attribution. threepp is never linked as a runtime dep — the single-GPU-stack property is preserved.
Rule: implements IDataWidget from pj_runtime; register directly in pj_app for v1. Introduce a WidgetRegistry service only when 2D / 3D widget families need symmetric registration.
pj_widgets control (ADR)Status: implemented. Lives in pj_widgets as Timeline.{h,cpp}; see
pj_widgets/CLAUDE.md. (Originally prototyped as a standalone pj_timeline
module with a core/ + widget/ split; dissolved into a single reusable
pj_widgets control once it had exactly one consumer and Qt-only deps — the
math/render separation is preserved as a Qt-free class inside the same file,
not as a separate library.)
Purpose: Reusable multi-track Source Timeline that drives PJ4's dormant
per-source TimeDomain.display_offset. Each loaded dataset gets one
horizontal bar on a shared display-time ruler; dragging a bar shifts only
that source's display offset live (display_time = raw_time − display_offset).
One Align button snaps all displayed starts to the global earliest displayed
start.
One file pair, two classes (pj_widgets/include/pj_widgets/Timeline.h +
pj_widgets/src/Timeline.cpp, namespace PJ):
TimelineScene — a genuinely Qt-free timing/geometry/alignment math class (no Q* in any method signature). Plain value types (TimelineSpanInput, PxSpan, TimeSpan, TimelineViewport, TimelineRuler), plain quint64 ids / qint64 ns / double pixels. Holds displayWindow/sceneExtent/ns↔px/barSpan/zoom/ruler/alignStartsToCommonOrigin.Timeline : QWidget — the QGraphicsView-backed view. Public input struct TimelineTrack (Qt-OK: carries QString name + QColor color). Runtime-agnostic: plain-typed slots (setTracks/setPlayhead/setDisplayRange) and intent signals (offsetChangeRequested(quint64,qint64)/alignRequested/playheadSeeked(double)); never includes a pj_runtime header. The bar/ruler/playhead QGraphicsItems are folded into the .cpp (a private timeline_detail namespace).The SourceTimelineController lives in pj_app — the only place that
may know both the widget and pj_runtime. It populates TimelineTracks from
CatalogModel + AppSession::datasetRawTimeRange + SessionManager::displayOffset
(converting DatasetId→quint64, Timestamp→qint64), maps widget intent
signals to SessionManager::setDisplayOffset / PlaybackEngine::setCurrentTime,
runs TimelineScene::alignStartsToCommonOrigin for the Align intent, and
coalesces AppSession::recomputeRange calls during live drags (~16 ms QTimer).
Dependency rules:
pj_widgets stays Qt-only — the Timeline added no pj_base/PJ-module dependency (hence the plain-typed surface).pj_runtime signals (SessionManager::displayOffsetChanged), to which each widget family subscribes independently.Key architectural decision — single global cursor, per-source offsets:
Multi-source alignment is implemented by folding per-source display_offset
into the read path (the curve adapter, and future scene-layer query sites),
not by creating per-source clocks. The PlaybackEngine / IDataWidget::onTrackerTime(double)
fleet contract is unchanged. This is simpler and keeps the existing DatastoreCurveAdapter
display-offset mechanism (which was already designed but had no production driver) as the
sole locus of offset application.
Per-source TimeDomain: Before this feature every dataset shared one
"default" TimeDomain so an offset would shift all sources. The loaders
(FileLoader, StreamingSourceManager) now mint one TimeDomain per
dataset so per-source isolation is guaranteed.
v1 scope: plots only. 2D/3D scene widgets are offset-blind (known limitation; no crash). Deferred: 2D/3D per-layer offset, offset persistence, undo/redo, richer alignment (leading source, per-bar pins), snapping, derived-series following their parent offset.
pj_app — desktop shellPurpose:
Responsibilities:
pj_runtime services to widgets from all three familiesRule:
pj_runtimeAppSessionAppSession is the central runtime object for the 4.x app.
It should:
Suggested composition (all from pj_runtime, plus one from pj_scripting):
SessionManagerPlaybackEngineWorkspaceManagerCatalogModelWidgetRegistryTransformRegistryToolboxManagerExtensionCatalogServiceUserSettingsServiceUndoManagerNotificationCenterIScriptingEngine instance from pj_scripting (the default LuaScriptingEngine)Represent data-producing activities explicitly:
ImportedDataSessionStreamingDataSessionToolboxSessionEach session tracks:
A dataset being removed from the visible catalog must not imply that the underlying data disappears immediately.
The app must support:
This prevents UI operations from becoming destructive without explicit user intent.
The workspace must support:
The layout model must be serializable without referring to concrete widget IDs.
Each plot panel must store:
Each tool panel must store:
Use a dedicated workspace file format with:
Do not serialize widget trees directly.
Undo/redo should follow the 3.x spirit and remain simple.
Use snapshot-based undo for workspace-level state:
This keeps the implementation compact and avoids a large hierarchy of command classes.
Snapshot-based undo should cover:
Data samples themselves are not part of undo/redo. Undo is for workspace and app configuration state, not for reverting imported or streamed data.
The plotting subsystem lives in pj_plotting. The approach is wholesale lift from PlotJuggler 3.x: the plot widget code has been refined over years and is good; the rot was in PJ3's MainWindow coupling, not in the plot widgets.
The plotting subsystem must support:
DataReader is the public read API of pj_datastore and the canonical query surface.
pj_plotting adds only what the lift needs to cleanly consume the new datastore:
DatastoreCurveAdapter — a QwtSeriesData<QPointF>-derived class that holds a SessionManager* + CurveDescriptor and pull-throughs to pj_datastore via TopicStorage::sealedChunks(). No copy of timeseries data — only a small chunk-index (per-chunk (chunk*, row_start, row_end, cumulative_begin, cumulative_end)) with a last-slot cache for sequential sample(i) from QwtPointMapper. ROI narrowing via QwtSeriesData::setRectOfInterest. Cross-chunk boundary guards keep line segments crossing the viewport boundary visible. boundingRect() returns full-data bounds (X from TopicMetadata::time_range_min/max, Y from union of ColumnStats::min_value/max_value across chunks) so Qwt's autoscale path does not lag a frame.No adapter-side decimation. Qwt's paint-time FilterPointsAggressive + ClipPolygons own per-pixel collapse; pre-decimating in the data layer breaks zoom correctness because the cached array is sized for the previous viewport.
No IPlotBackend abstraction. Qwt is the rendering library, full stop.
Lift from PJ3 as-is:
pj_runtime state (a link_group per plot widget, stored in layout)PlaybackEngine and redraws a vertical lineDisplay transforms (smoothing, derivative, etc.) stay plot-local: they do not create global topics in the datastore; they are serialized as part of plot panel state.
Per-plot display transforms must be modeled separately from persisted derived series.
Properties:
Examples:
Create a registry with descriptors containing:
The UI uses the descriptor metadata. The app-core uses the factories to bind to DerivedEngine.
Derived series definitions should contain:
These definitions are owned by app-core and persisted in the workspace.
The formula editor should be built as a UI client of app-core definitions, not as a thin text box over arbitrary runtime code.
Responsibilities:
Reactive scripting is required for parity. The architecture makes it cheap because the pieces already exist:
pj_scripting (separate module; Lua today, Python later)onTimeChanged(timestamp) and re-evaluates its script on tracker moveNeeded parts:
pj_scripting::IScriptingEngine (language-agnostic) with LuaScriptingEngine defaultonTimeChangedThe runtime exposes controlled access via native functions registered with the engine:
catalogSnapshot() + readSeries()No special pj_runtime wiring is required — reactive scripts are a toolbox like any other.
DerivedEngineUse DerivedEngine for:
Use app-side runtime logic for:
Introduce one authoritative tracker/time service.
It owns:
Support at least these modes:
Playback service must:
For streams, the service must define:
The following default rules should be implemented:
These rules must live in app-core, not inside the plot widget.
Add a catalog service that tracks:
This catalog is the source of truth for import/start-stream/toolbox availability.
The data source runtime must support:
The parser runtime must support:
Toolboxes are long-lived and UI-driven. The host must support:
When loading a workspace that references a missing extension:
Workspace documents should include:
User settings should include:
Provide:
The Qt shell should provide:
The full app should eventually include:
Widgets should:
Widgets should not:
DataEngineDeliver:
AppSessionAcceptance:
Deliver:
Acceptance:
Deliver:
Acceptance:
Deliver:
pj_plotting (wholesale; see §5.3)DatastoreCurveAdapter rebinding data reads to pj_datastore via TopicStorage::sealedChunks (pull-through; see §8.2). No adapter-side downsampler — Qwt's paint-time filtering owns decimation.IDataWidget implementation; plot factory wired directly in pj_appMainWindow owns the global Link toggle (buttonLink); plots emit rectChanged; broadcast updates peer plots via setZoomRectangle(rect, /*emit=*/false) to avoid echo recursion. XY plots are excluded from sync.PlaybackEngine::currentTimeChanged (both directions: timeline → tracker line; user-dragged tracker → timeline)QColorDialog from a right-click context menuDeferred to a follow-up phase: PlotWidgetEditor, PlotWidgetTransforms, StatisticsDialog, PlotBackground, colormap selector / editor.
Acceptance:
Deliver:
DerivedEngine bindingsAcceptance:
Deliver:
Acceptance:
Deliver:
Acceptance:
Deliver:
Acceptance:
Add unit tests for:
Add headless tests for:
Add Qt integration tests for:
Track manual scenarios for:
Recommended stable internal API surface:
Core types:
AppSessionWorkspaceDocumentPlotPanelStateToolPanelStateCurveBindingDerivedSeriesDefinitionReactiveScriptDefinitionPlaybackStateTrackerStateDataSessionDescriptorExtensionDescriptorConcrete services (in pj_runtime unless noted):
AppSessionSessionManagerWorkspaceManagerPlaybackEngineCatalogModelWidgetRegistryTransformRegistryToolboxManagerUndoManagerExtensionCatalogServiceUserSettingsServiceDialogHostLuaScriptingEngine (in pj_scripting)StatePublisher parity remains explicitly deferred.