DataProducts Subtopology — Software Design Document (SDD)

The DataProducts subtopology packages the standard F´ data product services — product management, product writing, product cataloging, and the buffer infrastructure that connects them — into a single pre-wired subtopology. Client components that produce data products connect to the exposed topology ports; the subtopology handles buffer allocation, accumulation, writing products to the filesystem, and cataloging written products for prioritized downlink.

1. Requirements

2. Design & Core Functions

2.1 Instance Summary

2.2 Internal Wiring

• dpMgr.bufferGetOut -> dpBufferManager.bufferGetCallee — product buffer allocation.
• dpMgr.productSendOut -> dpBufferAccumulator.bufferSendInFill — filled products enter the accumulator.
• dpBufferAccumulator.bufferSendOutDrain -> dpWriter.bufferSendIn — accumulated products drain to the writer.
• dpWriter.deallocBufferSendOut -> dpBufferAccumulator.bufferSendInReturn and dpBufferAccumulator.bufferSendOutReturn -> dpBufferManager.bufferSendIn — written buffers return to the buffer manager.
• dpWriter.dpWrittenOut -> dpCat.addToCat — written products are added to the catalog.

2.3 Configuration Hooks inside the Subtopology

• dpCat: The data product directory (DataProductsConfig::Paths::dpDir) is created at startup and dpCat.configure() is called with the directory, state file (DataProductsConfig::Paths::dpState), and the subtopology memory allocator.
• dpWriter: dpWriter.configure() is called with the data product directory.
• dpBufferAccumulator: The queue is allocated at startup (drain mode DRAIN) and deallocated at teardown, sized by DataProductsConfig::BufferAccumulator::maxNumBuffers.
• dpBufferManager: A single buffer bin is set up from DataProductsConfig::BuffMgr (buffer size and count); the manager is cleaned up (and dpCat.shutdown() called) at teardown.

2.4 Required Inputs for Operation

The DataProducts subtopology is not a stand-alone application. It requires connections from the including deployment topology:

• Client components: Connect data-product-producing components to productGetIn / productRequestIn / productSendIn / productResponseOut.
• Rate groups: Connect scheduler outputs to dpMgrSchedIn, dpWriterSchedIn, and dpBufferManagerSchedIn for telemetry output.
• File downlink: Connect dpCatFileOut / dpCatFileDone to a file downlink component (e.g., via the FileHandling subtopology) so cataloged products can be downlinked.
• Optional processing: dpWriterProcOut provides a hook for processing product buffers before they are written (e.g., compression via the DpCompression subtopology).

3. Usage

3.1 Exposed Topology Ports

3.2 Example Usage

topology Flight {
  instance DataProducts.Subtopology

  connections RateGroups {
    rg1Hz.RateGroupMemberOut[0] -> DataProducts.Subtopology.dpMgrSchedIn
    rg1Hz.RateGroupMemberOut[1] -> DataProducts.Subtopology.dpWriterSchedIn
    rg1Hz.RateGroupMemberOut[2] -> DataProducts.Subtopology.dpBufferManagerSchedIn
  }

  connections Producers {
    myProducer.productGetOut -> DataProducts.Subtopology.productGetIn
    myProducer.productSendOut -> DataProducts.Subtopology.productSendIn
  }

  connections Downlink {
    DataProducts.Subtopology.dpCatFileOut -> FileHandling.Subtopology.fileDownlinkSendIn
    FileHandling.Subtopology.fileDownlinkCompleteOut -> DataProducts.Subtopology.dpCatFileDone
  }
}

4. Configuration

Configure the instance properties for the static instances owned by the subtopology. All knobs live under Svc/Subtopologies/DataProducts/DataProductsConfig/DataProductsConfig.fpp:

• Base ID — Base identifier for the subtopology; component IDs are offset from this base.
• Queue sizes / stack sizes / priorities — For the active instances (dpCat, dpMgr, dpWriter, dpBufferAccumulator).
• BufferAccumulator — Allocator ID and maximum number of accumulated buffers.
• BuffMgr — Data product buffer store size, count, and buffer manager ID.
• Paths — Data product directory and catalog state file path.

The memory allocator used by the subtopology is provided by DataProductsSubtopologyConfig.cpp (DataProducts::Allocation::memAllocator).

5. Traceability Matrix