skills/flowio/references/fcs_semantics.md
Use this reference when the task depends on what values or metadata mean, not just how to call the API.
FlowIO reads FCS 2.0, 3.0, and 3.1 and writes a constrained FCS 3.1 representation. It is a file-format library, not a complete flow-cytometry analysis system.
Before downstream biological interpretation, decide whether the workflow also requires:
FlowIO does none of these. Its optional preprocessing is limited to scaling defined by FCS acquisition metadata.
An FCS dataset can contain:
FlowIO exposes these through header, text, events, and analysis.
FCS 3.1 deprecated storing multiple datasets in one file through $NEXTDATA,
but FlowIO can read legacy multi-dataset files with
read_multiple_data_sets().
The FCS standard treats keyword names as case-insensitive. Standard keywords
are written with $, but FlowIO normalizes parsed keys:
$.Examples:
$DATE becomes text["date"]$P1N becomes text["p1n"]$SPILLOVER becomes text["spillover"]$NEXTDATA becomes text["nextdata"]Use:
date = flow.text.get("date")
spill = flow.text.get("spillover", flow.text.get("spill"))
nextdata = int(flow.text.get("nextdata", "0"))
Avoid:
date = flow.text.get("$DATE") # Always misses in FlowIO's normalized mapping.
Custom, non-standard keys are also lowercased. The normalized mapping may contain subject, sample, operator, institution, instrument serial-number, and free-text fields. Treat an unrestricted metadata dump as potentially identifying.
FlowIO 1.4.0 implements the first step by removing every $ character from the
decoded segment before splitting keys and values. A literal $ inside a value
is therefore lost ("a$b" becomes "ab"). Do not use the parsed mapping as a
lossless metadata archive.
FCS parameter numbers begin at 1:
FlowIO presents these through two index systems:
flow.channels[1], flow.channels[2], ... use one-based parameter numbers.*_indices attributes use zero-based indices.When reporting a channel, label both values if ambiguity matters:
for array_index, pnn in enumerate(flow.pnn_labels):
parameter_number = array_index + 1
pns = flow.pns_labels[array_index]
print(parameter_number, array_index, pnn, pns)
PnS is optional. FlowIO inserts "" for missing PnS entries so that
pns_labels and pnn_labels have the same length.
FlowIO infers scatter, fluorescence, and time indices from channel labels. Treat these as conveniences, not a substitute for checking the instrument panel. Vendor-specific labels may not classify as expected.
null_channel_list accepts PnN labels and excludes matching channels from the
derived fluorescence/scatter/time index lists. It does not remove columns from
the event data. flow.null_channels stores the supplied labels unchanged; it
does not contain zero-based indices and can retain labels that did not match a
channel.
flow.events is a flattened one-dimensional sequence, ordered by event and
then channel. It is usually an array.array; mixed-width integer channels use
a Python list:
event_1_channel_1, event_1_channel_2, ..., event_2_channel_1, ...
flow.as_array(preprocess=False) reshapes that representation to:
(event_count, channel_count)
The returned NumPy array is float64, even when the encoded file uses integer
or single-precision event data.
flow.as_array(preprocess=True) performs the following operations.
When a time channel and nonempty timestep keyword exist:
time_scaled = time_encoded * timestep
An empty or whitespace-only timestep is treated as 1.0 in FlowIO 1.4.0.
For PnE (decades, log_zero) and PnR range:
linear_value = 10 ** (decades * encoded_value / range) * log_zero
The conversion is applied when decades > 0.
For PnG gain:
gain_scaled = value / gain
FlowIO skips division when gain is zero or one.
The operations are metadata-driven. Bad metadata can therefore produce bad scaled values even when the DATA bytes were parsed correctly.
FlowIO does not:
$SPILL/$SPILLOVER compensationFor compensation/transformation/gating, use a higher-level package such as FlowKit and preserve a record of the matrix and transform parameters.
FlowIO can expose or write a spillover keyword as a string. It does not validate the matrix scientifically or apply it.
A standard spillover string begins with:
Items are comma-delimited with no newline characters.
Before passing the matrix downstream:
FCS 3.0/3.1 DATA offsets can appear in HEADER and TEXT. FlowIO normally uses TEXT offsets and checks that HEADER agrees.
Some files have known defects:
FlowIO accounts for the FCS 3.1 large-file rule. Do not use relaxed flags merely because a file is large.
Recovery options:
ignore_offset_error=True: tolerate the documented off-by-one case.ignore_offset_discrepancy=True: use TEXT despite a HEADER/TEXT mismatch.use_header_offsets=True: use HEADER and suppress the mismatch error.Each option changes which bytes are interpreted as events. Use one only when the file's provenance or vendor behavior justifies it, and validate event count, channel distributions, and known controls afterward.
Normal FlowData construction reads the full DATA segment into memory.
as_array() then allocates a second float64 representation.
Approximate additional memory for the 2-D array is:
event_count * channel_count * 8 bytes
This excludes the original event array, Python objects, temporary arrays, and downstream DataFrames.
Use only_text=True for inventory. FlowIO 1.4.0 does not expose chunked,
streaming, lazy, or memory-mapped event reads. If a file does not fit safely in
memory, use a different parser/workflow or process it in a resource-limited
environment; do not claim FlowIO can chunk it.
only_text=True skips DATA loading but still parses ANALYSIS. Also prefer paths
over open handles: FlowData closes caller-provided handles, and the
multi-dataset helper cannot reuse a closed handle for its second dataset in
FlowIO 1.4.0.
create_fcs() and write_fcs() produce:
The float32 representation has roughly 6-7 decimal digits of precision. Round-trip comparisons should use tolerances rather than exact equality.
Passing a NumPy array makes create_fcs() allocate an array('f') copy. An
existing array('f') is written directly, so retain that representation when
large writer memory is a concern.
create_fcs() needs flattened event data. Flatten a two-dimensional array in
C order:
flat = events_2d.astype("float32", copy=False).ravel(order="C")
Validate:
if events_2d.ndim != 2:
raise ValueError("expected events x channels")
if events_2d.shape[1] != len(channel_names):
raise ValueError("channel label count mismatch")
Required keywords and channel interpretation fields are generated by FlowIO.
Do not rely on metadata_dict to override $PAR, $TOT, $MODE,
$DATATYPE, PnB, or PnN.
For floating-point sources, write_fcs() can keep encoded values while
dropping PnG and timestep. This changes the interpretation returned by
as_array(preprocess=True). Validate both encoded and metadata-scaled values,
not just event counts.
For any converted or rewritten file, record:
$DATATYPEpreprocess was true or false