docs/user-manual/framework/data-products.md
[!TIP] The following document is a technical guide to the Data Products system in F´. For a quick introduction to get started with using data products, see the How To: Generate Data Products.
A data product is any data that is produced by an embedded system, stored on board the system, and transmitted to the ground, typically in priority order. F Prime provides several features for managing the generation, storage, and downlink of data products. In this section, we document those features.
First we explain some basic concepts.
F Prime data products are based on records and containers.
A record is a basic unit of data.
For example, it may be a struct, an array of typed objects of
statically known size, or an array of bytes of statically unknown size.
A container has an identifier and a priority and stores records.
In C++, a container is represented as a class object with member fields that
(1) store header data and (2) store an Fw::Buffer object pointing
to the memory that stores the records.
The set of all container specifications forms the data product dictionary. To manage the data product dictionary, F Prime uses the same general approach as for commands, telemetry, events, and parameters:
Each component C specifies records and containers. The container IDs are local to C. Typically they have the values 0, 1, 2, ... .
Each instance I of C contributes one container I.c to the dictionary for each container c defined in C. The global identifier for I.c is the base identifier of I plus the local identifier for c. For example, if the base identifier is 0x1000, then the global identifiers might be 0x1000, 0x1001, 0x1002, ... .
For any topology T, the global identifiers I.c for all the component instances of T form the data product dictionary for T.
Typically a data product system in an F Prime application consists of the following components:
One or more data product producers. These components produce data products and are typically mission-specific. For example, they may produce science data.
Standard F Prime components for managing data products.
A data product manager.
This component allocates memory for empty containers.
It also forwards filled containers to the data product writer.
See Svc::DpManager.
A data product writer.
This component receives filled containers from data product
producers. It writes the contents of the containers to non-volatile
storage. See Svc::DpWriter.
A data product catalog.
This component maintains a database of available data
products. By command, it downlinks and deletes data products.
See Svc::DpCatalog.
A data product processor. This component is not yet developed. When it is developed, it will perform in-memory processing on data product containers.
Note that when using data products, you need to develop only the producer components. The other components are provided by F Prime.
In this section we provide more detail about producer components.
A producer component typically repeats the following activities, as often as necessary:
Request a container from a data manager component.
When the container is received, serialize records into the container. This action fills the container with data.
When the container is full, send the container to the data product manager, which forwards it to the data product writer.
The FPP model and the autocoded C++ have several features that support these activities. We discuss these features in the following sections.
In this section we summarize the features of the FPP modeling language used in constructing data product producer components. Each of these features is fully documented in The FPP User's Guide and The FPP Language Specification.
FPP provides the following special ports for managing data products:
A product get port of type Fw::DpGet.
This is an output port for synchronously requesting
memory from a buffer manager.
The request is served on the thread that invokes the port
and causes a mutex lock to be taken on that thread.
Example syntax:
product get port productGetOut
A product request port of type Fw::DpRequest.
This is an output port for asynchronously requesting memory
from a data product manager.
The request is served on the thread of the data product manager.
This approach incurs the overhead of a separate thread, but it
does not require the requesting thread to take a lock.
Example syntax:
product request port productRequestOut
A product receive port of type Fw::DpResponse.
This is an input port for receiving an empty container in response
to an asynchronous request. Example syntax:
async product recv port productRecvIn
A product send port of type Fw::DpSend.
This is an output port for sending a filled container
to a data product writer. Example syntax:
product send port productSendOut
Each data product producer component must have the following ports in its component model:
One or both of a product get port and a product request port.
A product send port.
A component that has a product request port must also have
a product receive port.
A record is a unit of data. When defining a producer component, you can specify one or more records. A record specification consists of a name, a type specifier, and an optional identifier. The type specifier may be one of the following:
An FPP type T. In this case, the record contains a single value of type T. T may be any FPP type, including a struct or array type.
An FPP type T followed by the keyword array.
In this case, the record is an array of values of type T
of statically unknown size.
The size of the array is stored in the record.
In either case, T may be any FPP type, including a struct or array type.
Example syntax:
@ A struct with a fixed-size member array
struct FixedSizeData {
data: [1024] F32
}
@ A record containing fixed-size data
product record FixedSizeDataRecord: FixedSizeData id 0x00
@ A record containing a variable-size array
product record F32ArrayRecord: F32 array id 0x01
A container is a data structure that stores records. When defining a producer component, you can specify one or more containers. Each container specified in a component can store any of the records specified in the component.
A container specification consists of a name, an optional identifier, and an optional default priority. The default priority is the priority to use if no other priority is specified for the container during operations. Example syntax:
product container C1
product container C2 id 0x01 default priority 10
The autocoded C++ base class for a producer component C provides the following API elements:
Enumerations defining the available container IDs, container priorities, and record IDs.
A member class C ::DpContainer. This class is derived from
Fw::DpContainer and represents a container
specialized to the data products defined in C.
Each instance of C ::DpContainer is a wrapper for an Fw::Buffer B,
which points to allocated memory.
The class provides operations for serializing the records
defined in C into the memory pointed to by B.
There is one operation C ::DpContainer::serialize_ R
for each record R defined in C.
For the serialized format of each record, see the documentation
for Fw::DpContainer.
If C has a product get port, a member function dpGet_
c for each container c defined in C.
This function takes a data size and a reference
to a data product container D.
It invokes productGetOut, which is typically connected
to a data product manager component.
In the nominal case, the invocation returns an Fw::Buffer B large enough
to store a data product packet with the requested data size.
The dpGet function then uses the ID and B to initialize D.
It returns a status value indicating whether the buffer
allocation succeeded.
If C has a product request port, a member function
dpRequest_ c for each container c defined in C.
This function takes a data size.
It sends out a request on productRequestOut, which is
typically connected to a data product manager component.
The request is for a buffer large enough to store a data
product packet with the requested data size.
If C has a product recv port, a pure virtual
member function dpRecv_ c _handler for each container c
defined in C.
When a fresh container arrives in response to a
dpRequest invocation, the autocoded C++ uses the container ID to
select and invoke the appropriate dpRecv handler.
The implementation of C must override each handler
to provide the mission-specific behavior for filling
in the corresponding container.
The arguments to dpRecv_ c _handler provide
(1) a reference to the container, which the implementation can fill in;
and (2) a status value indicating whether the container
is valid. An invalid container can result if the buffer
allocation fails.
A member function dpSend for sending a filled
data product container.
This function takes a reference to a container c and an
optional time tag.
It does the following:
If no time tag is provided, then invoke timeGetOut
to get the system time and use it to set the time tag.
Store the time tag into c.
Send c on productSendOut.
Constant expressions representing the sizes of the records.
If a record R holds a single value, then
the expression SIZE_OF_ R _RECORD
evaluates to the size of that record.
Otherwise R is an array record. In this case
the expression SIZE_OF_ R _RECORD( size )
evaluates to the size of an array record R with
size array elements.
You can use these expressions to compute data sizes
when requesting data product buffers. For example,
if a component specifies a record Image,
then inside the component implementation the expression
10 * SIZE_OF_Image_RECORD represents the size of the
storage necessary to hold 10 Image records.
In F Prime, each component C comes with auto-generated
classes C TesterBase and C GTestBase for writing
unit tests against C.
C GTestBase is derived from C TesterBase; it
provides test macros based on the Google Test framework.
To write unit tests, you construct a class C Tester.
Typically C Tester is derived from C GTestBase and
uses the Google Test framework macros.
If for some reason you can't use the Google Test framework
(e.g., because you are running on a platform that does not support it),
then your C Tester class can be derived from C TesterBase.
This section documents the unit test support for producer components.
History data structures:
The class C TesterBase provides the following histories:
If C has a product get port,
then C TesterBase has a history called productGetHistory.
Each element in the history is of type DpGet.
DpGet is a struct with fields storing the container ID and the
size emitted on the product get port.
If C has a product request port, then C TesterBase has a
corresponding history called productRequestHistory.
Each element in the history is of type DpRequest.
DpRequest is a struct with fields storing the container ID and the
size emitted on the product request port.
C TesterBase has a history called productSendHistory.
Each element in the history is of type DpSend.
DpSend is a struct with fields storing the container ID and
a shallow copy of the buffer emitted on the product send port.
History functions:
The class C TesterBase provides the following functions
for managing the histories:
If C has a product get port, then C TesterBase provides
the following functions:
pushProductGetEntry: This function takes a container ID and
a size. It constructs the corresponding DpGet history object
and pushes it on productGetHistory. Typically this function is
called by productGet_handler (see below).
productGet_handler: This function is called when the tester
component receives data emitted on the product get port of the
component under test. It takes a container ID, a size, and a
mutable reference to a buffer B. By default it calls
pushProductGetEntry with the ID and size and returns FAILURE,
indicating that no memory was allocated and B was not updated.
This function is virtual, so you can override it with your own
behavior. For example, your function could call pushProductGetEntry,
allocate a buffer, store the allocated buffer into B, and return
SUCCESS.
If C has a product request port, then C TesterBase provides
the following functions:
pushProductRequestEntry: This function takes a container ID and
a size. It constructs the corresponding DpRequest history object
and pushes it on productRequestHistory. Typically this function is
called by productRequest_handler (see below).
productRequest_handler: This function is called when the tester
component receives data emitted on the product request port of the
component under test. It takes a container ID and a size. By default
it calls pushProductRequestEntry with the ID and size. This function
is virtual, so you can override it with your own behavior.
C TesterBase provides the following functions:
pushProductSendEntry: This function takes a container ID and a
const reference to a buffer. It constructs the corresponding
DpSend history object and pushes it on productSendHistory.
Typically this function is called by productSend_handler (see below).
productSend_handler: This function is called when the tester
component receives data emitted on the product send port of the
component under test. It takes a container ID and a const reference
to a buffer. By default it calls pushProductSendEntry with the
ID and buffer. This function is virtual, so you can override it
with your own behavior.
Testing macros:
The class C GTestBase provides the following macros for
verifying the histories managed by C TesterBase.
If C defines data products and has a product get port, then C
GTestBase provides the following macros:
ASSERT_PRODUCT_GET_SIZE(size): This macro checks that productGetHistory
has the specified size (number of entries).
ASSERT_PRODUCT_GET(index, id, size): This macro checks that
productGetHistory has the specified container ID and size
at the specified history index.
If C defines data products and has a product request port,
then C GTestBase provides the following macros:
ASSERT_PRODUCT_REQUEST_SIZE(size): This macro checks that
productRequestHistory has the specified size (number of entries).
ASSERT_PRODUCT_REQUEST(index, id, size): This macro checks that
productRequestHistory has the specified container ID and size
at the specified history index.
If C defines data products, then C GTestBase provides
the following macros:
ASSERT_PRODUCT_SEND_SIZE(size): This macro checks that
productSendHistory has the specified size (number of entries).
ASSERT_PRODUCT_SEND(index, id, priority, timeTag, procType, userData, dataSize, buffer):
All the arguments of this macro are inputs (read-only) except buffer, which is
a by-reference output and must be a variable of type Fw::Buffer&.
This macro verifies the entry entry stored at the specified
index of productSendHistory. It does the following:
Check that entry.id matches the specified ID.
Deserialize the data product header stored in entry.buffer.
Check that the container ID, priority, time tag, processor type, user data, and data size stored in the deserialized header match the specified values.
Assign entry.buffer to buffer. After this macro runs,
the deserialization pointer of buffer points into the start
of the data payload of entry.buffer. You can write additional
code to deserialize and check the data payload.
Container IDs:
The container IDs emitted by the component under test are global
IDs.
Therefore, when constructing specified IDs you must add
the ID base specified in the tester component to the local
ID specified in the component under test.
For example, for container CONTAINER in component Component,
you would write
ID_BASE + Component::ContainerId::CONTAINER
ID_BASE is a standard constant defined in each Tester implementation
and provided to the Tester base classes in their constructors.
In this section we discuss several common use cases involving data products.
Requesting and sending data products:
See the example uses in the documentation for
Svc::DpManager.
The component referred to as producer in that document
is a data product producer.
Writing data products to non-volatile storage:
See the example uses in the documentation for
Svc::DpWriter.
The component referred to as producer in that document
is a data product producer.
Cataloging and downlinking data products:
For a preliminary implementation of the data product catalog,
see Svc::DpCatalog.
Processing data products:
Data product binary files (typically with the .fdp extension) can be decoded on the ground with the fprime-dp command, which is a CLI tool shipped with the F Prime GDS. See fprime-dp decode -h for options on how to use the tool.