IO module

This document describes the overall design of this module.

Rules:

• Each directory in this module corresponds to a specific format such as csv and json.

• directories that depend on external dependencies MUST be feature gated, with a feature named with a prefix io_.

• modules MUST re-export any API of external dependencies they require as part of their public API. E.g.

  • if a module as an API write(writer: &mut csv:Writer<W>, ...), it MUST contain pub use csv::Writer;.

  The rational is that adding this crate to cargo.toml must be sufficient to use it.

• Each directory SHOULD contain two directories, read and write, corresponding to functionality about reading from the format and writing to the format respectively.

• The base module SHOULD contain use pub read; and use pub write;.

• Implementations SHOULD separate reading of "data" from reading of "metadata". Examples:

  • schema read or inference SHOULD be a separate function
  • functions that read "data" SHOULD consume a schema typically pre-read.

• Implementations SHOULD separate IO-bounded operations from CPU-bounded operations. I.e. implementations SHOULD:

  • contain functions that consume a Read implementor and output a "raw" struct, i.e. a struct that is e.g. compressed and serialized
  • contain functions that consume a "raw" struct and convert it into Arrow.
  • offer each of these functions as independent public APIs, so that consumers can decide how to balance CPU-bounds and IO-bounds.