Back to Lz4

1.10.0 Manual

doc/lz4frame_manual.html

1.10.023.9 KB
Original Source

1.10.0 Manual


Contents

  1. Introduction
  2. Compiler specifics
  3. Error management
  4. Frame compression types
  5. Simple compression function
  6. Advanced compression functions
  7. Resource Management
  8. Compression
  9. Decompression functions
  10. Streaming decompression functions
  11. Dictionary compression API
  12. Bulk processing dictionary compression
  13. Advanced compression operations
  14. Custom memory allocation

Introduction

lz4frame.h implements LZ4 frame specification: see doc/lz4_Frame_format.md .
 LZ4 Frames are compatible with `lz4` CLI,
 and designed to be interoperable with any system.

Compiler specifics

Error management

**unsigned LZ4F\_isError(LZ4F\_errorCode\_t code);** /**< tells when a function result is an error code */
**const char\* LZ4F\_getErrorName(LZ4F\_errorCode\_t code);** /**< return error code string; for debugging */

Frame compression types

**typedef enum { LZ4F\_default=0, LZ4F\_max64KB=4, LZ4F\_max256KB=5, LZ4F\_max1MB=6, LZ4F\_max4MB=7 LZ4F\_OBSOLETE\_ENUM(max64KB) LZ4F\_OBSOLETE\_ENUM(max256KB) LZ4F\_OBSOLETE\_ENUM(max1MB) LZ4F\_OBSOLETE\_ENUM(max4MB) } LZ4F\_blockSizeID\_t;**
**typedef enum { LZ4F\_blockLinked=0, LZ4F\_blockIndependent LZ4F\_OBSOLETE\_ENUM(blockLinked) LZ4F\_OBSOLETE\_ENUM(blockIndependent) } LZ4F\_blockMode\_t;**
**typedef enum { LZ4F\_noContentChecksum=0, LZ4F\_contentChecksumEnabled LZ4F\_OBSOLETE\_ENUM(noContentChecksum) LZ4F\_OBSOLETE\_ENUM(contentChecksumEnabled) } LZ4F\_contentChecksum\_t;**
**typedef enum { LZ4F\_noBlockChecksum=0, LZ4F\_blockChecksumEnabled } LZ4F\_blockChecksum\_t;**
**typedef enum { LZ4F\_frame=0, LZ4F\_skippableFrame LZ4F\_OBSOLETE\_ENUM(skippableFrame) } LZ4F\_frameType\_t;**
**typedef struct { LZ4F\_blockSizeID\_t blockSizeID;** /* max64KB, max256KB, max1MB, max4MB; 0 == default (LZ4F_max64KB) */ **LZ4F\_blockMode\_t blockMode;** /* LZ4F_blockLinked, LZ4F_blockIndependent; 0 == default (LZ4F_blockLinked) */ **LZ4F\_contentChecksum\_t contentChecksumFlag;** /* 1: add a 32-bit checksum of frame's decompressed data; 0 == default (disabled) */ **LZ4F\_frameType\_t frameType;** /* read-only field : LZ4F_frame or LZ4F_skippableFrame */ **unsigned long long contentSize;** /* Size of uncompressed content ; 0 == unknown */ **unsigned dictID;** /* Dictionary ID, sent by compressor to help decoder select correct dictionary; 0 == no dictID provided */ **LZ4F\_blockChecksum\_t blockChecksumFlag;** /* 1: each block followed by a checksum of block's compressed data; 0 == default (disabled) */ **} LZ4F\_frameInfo\_t;**

makes it possible to set or read frame parameters. Structure must be first init to 0, using memset() or LZ4F\_INIT\_FRAMEINFO, setting all parameters to default. It's then possible to update selectively some parameters
**typedef struct { LZ4F\_frameInfo\_t frameInfo; int compressionLevel;** /* 0: default (fast mode); values > LZ4HC_CLEVEL_MAX count as LZ4HC_CLEVEL_MAX; values < 0 trigger "fast acceleration" */ **unsigned autoFlush;** /* 1: always flush; reduces usage of internal buffers */ **unsigned favorDecSpeed;** /* 1: parser favors decompression speed vs compression ratio. Only works for high compression modes (>= LZ4HC_CLEVEL_OPT_MIN) */ /* v1.8.2+ */ **unsigned reserved[3];** /* must be zero for forward compatibility */ **} LZ4F\_preferences\_t;**

makes it possible to supply advanced compression instructions to streaming interface. Structure must be first init to 0, using memset() or LZ4F\_INIT\_PREFERENCES, setting all parameters to default. All reserved fields must be set to zero.

Simple compression function

**size\_t LZ4F\_compressFrame(void\* dstBuffer, size\_t dstCapacity, const void\* srcBuffer, size\_t srcSize, const LZ4F\_preferences\_t\* preferencesPtr);**

Compress srcBuffer content into an LZ4-compressed frame. It's a one shot operation, all input content is consumed, and all output is generated. Note : it's a stateless operation (no LZ4F\_cctx state needed). In order to reduce load on the allocator, LZ4F\_compressFrame(), by default, uses the stack to allocate space for the compression state and some table. If this usage of the stack is too much for your application, consider compiling `lz4frame.c` with compile-time macro LZ4F\_HEAPMODE set to 1 instead. All state allocations will use the Heap. It also means each invocation of LZ4F\_compressFrame() will trigger several internal alloc/free invocations. @dstCapacity MUST be \>= LZ4F\_compressFrameBound(srcSize, preferencesPtr). @preferencesPtr is optional : one can provide NULL, in which case all preferences are set to default. @return : number of bytes written into dstBuffer. or an error code if it fails (can be tested using LZ4F\_isError())
**size\_t LZ4F\_compressFrameBound(size\_t srcSize, const LZ4F\_preferences\_t\* preferencesPtr);**

Returns the maximum possible compressed size with LZ4F\_compressFrame() given srcSize and preferences. `preferencesPtr` is optional. It can be replaced by NULL, in which case, the function will assume default preferences. Note : this result is only usable with LZ4F\_compressFrame(). It may also be relevant to LZ4F\_compressUpdate() \_only if\_ no flush() operation is ever performed.
**int LZ4F\_compressionLevel\_max(void);** /* v1.8.0+ */ 

@return maximum allowed compression level (currently: 12)

Advanced compression functions

**typedef struct { unsigned stableSrc;** /* 1 == src content will remain present on future calls to LZ4F_compress(); skip copying src content within tmp buffer */ **unsigned reserved[3]; } LZ4F\_compressOptions\_t;**

Resource Management

**LZ4F\_errorCode\_t LZ4F\_createCompressionContext(LZ4F\_cctx\*\* cctxPtr, unsigned version); LZ4F\_errorCode\_t LZ4F\_freeCompressionContext(LZ4F\_cctx\* cctx);**

The first thing to do is to create a compressionContext object, which will keep track of operation state during streaming compression. This is achieved using LZ4F\_createCompressionContext(), which takes as argument a version, and a pointer to LZ4F\_cctx\*, to write the resulting pointer into. @version provided MUST be LZ4F\_VERSION. It is intended to track potential version mismatch, notably when using DLL. The function provides a pointer to a fully allocated LZ4F\_cctx object. @cctxPtr MUST be != NULL. If @return != zero, context creation failed. A created compression context can be employed multiple times for consecutive streaming operations. Once all streaming compression jobs are completed, the state object can be released using LZ4F\_freeCompressionContext(). Note1 : LZ4F\_freeCompressionContext() is always successful. Its return value can be ignored. Note2 : LZ4F\_freeCompressionContext() works fine with NULL input pointers (do nothing).

Compression

**size\_t LZ4F\_compressBegin(LZ4F\_cctx\* cctx, void\* dstBuffer, size\_t dstCapacity, const LZ4F\_preferences\_t\* prefsPtr);**

will write the frame header into dstBuffer. dstCapacity must be \>= LZ4F\_HEADER\_SIZE\_MAX bytes. `prefsPtr` is optional : NULL can be provided to set all preferences to default. @return : number of bytes written into dstBuffer for the header or an error code (which can be tested using LZ4F\_isError())
**size\_t LZ4F\_compressBound(size\_t srcSize, const LZ4F\_preferences\_t\* prefsPtr);**

Provides minimum dstCapacity required to guarantee success of LZ4F\_compressUpdate(), given a srcSize and preferences, for a worst case scenario. When srcSize==0, LZ4F\_compressBound() provides an upper bound for LZ4F\_flush() and LZ4F\_compressEnd() instead. Note that the result is only valid for a single invocation of LZ4F\_compressUpdate(). When invoking LZ4F\_compressUpdate() multiple times, if the output buffer is gradually filled up instead of emptied and re-used from its start, one must check if there is enough remaining capacity before each invocation, using LZ4F\_compressBound(). @return is always the same for a srcSize and prefsPtr. prefsPtr is optional : when NULL is provided, preferences will be set to cover worst case scenario. tech details : @return if automatic flushing is not enabled, includes the possibility that internal buffer might already be filled by up to (blockSize-1) bytes. It also includes frame footer (ending + checksum), since it might be generated by LZ4F\_compressEnd(). @return doesn't include frame header, as it was already generated by LZ4F\_compressBegin().
**size\_t LZ4F\_compressUpdate(LZ4F\_cctx\* cctx, void\* dstBuffer, size\_t dstCapacity, const void\* srcBuffer, size\_t srcSize, const LZ4F\_compressOptions\_t\* cOptPtr);**

LZ4F\_compressUpdate() can be called repetitively to compress as much data as necessary. Important rule: dstCapacity MUST be large enough to ensure operation success even in worst case situations. This value is provided by LZ4F\_compressBound(). If this condition is not respected, LZ4F\_compress() will fail (result is an errorCode). After an error, the state is left in a UB state, and must be re-initialized or freed. If previously an uncompressed block was written, buffered data is flushed before appending compressed data is continued. `cOptPtr` is optional : NULL can be provided, in which case all options are set to default. @return : number of bytes written into `dstBuffer` (it can be zero, meaning input data was just buffered). or an error code if it fails (which can be tested using LZ4F\_isError())
**size\_t LZ4F\_flush(LZ4F\_cctx\* cctx, void\* dstBuffer, size\_t dstCapacity, const LZ4F\_compressOptions\_t\* cOptPtr);**

When data must be generated and sent immediately, without waiting for a block to be completely filled, it's possible to call LZ4\_flush(). It will immediately compress any data buffered within cctx. `dstCapacity` must be large enough to ensure the operation will be successful. `cOptPtr` is optional : it's possible to provide NULL, all options will be set to default. @return : nb of bytes written into dstBuffer (can be zero, when there is no data stored within cctx) or an error code if it fails (which can be tested using LZ4F\_isError()) Note : LZ4F\_flush() is guaranteed to be successful when dstCapacity \>= LZ4F\_compressBound(0, prefsPtr).
**size\_t LZ4F\_compressEnd(LZ4F\_cctx\* cctx, void\* dstBuffer, size\_t dstCapacity, const LZ4F\_compressOptions\_t\* cOptPtr);**

To properly finish an LZ4 frame, invoke LZ4F\_compressEnd(). It will flush whatever data remained within `cctx` (like LZ4\_flush()) and properly finalize the frame, with an endMark and a checksum. `cOptPtr` is optional : NULL can be provided, in which case all options will be set to default. @return : nb of bytes written into dstBuffer, necessarily \>= 4 (endMark), or an error code if it fails (which can be tested using LZ4F\_isError()) Note : LZ4F\_compressEnd() is guaranteed to be successful when dstCapacity \>= LZ4F\_compressBound(0, prefsPtr). A successful call to LZ4F\_compressEnd() makes `cctx` available again for another compression task.

Decompression functions

**typedef struct { unsigned stableDst; /\* pledges that last 64KB decompressed data is present right before @dstBuffer pointer. \* This optimization skips internal storage operations. \* Once set, this pledge must remain valid up to the end of current frame. \*/ unsigned skipChecksums; /\* disable checksum calculation and verification, even when one is present in frame, to save CPU time. \* Setting this option to 1 once disables all checksums for the rest of the frame. \*/ unsigned reserved1;** /* must be set to zero for forward compatibility */ **unsigned reserved0;** /* idem */ **} LZ4F\_decompressOptions\_t;**
**LZ4F\_errorCode\_t LZ4F\_createDecompressionContext(LZ4F\_dctx\*\* dctxPtr, unsigned version); LZ4F\_errorCode\_t LZ4F\_freeDecompressionContext(LZ4F\_dctx\* dctx);**

Create an LZ4F\_dctx object, to track all decompression operations. @version provided MUST be LZ4F\_VERSION. @dctxPtr MUST be valid. The function fills @dctxPtr with the value of a pointer to an allocated and initialized LZ4F\_dctx object. The @return is an errorCode, which can be tested using LZ4F\_isError(). dctx memory can be released using LZ4F\_freeDecompressionContext(); Result of LZ4F\_freeDecompressionContext() indicates current state of decompressionContext when being released. That is, it should be == 0 if decompression has been completed fully and correctly.

Streaming decompression functions

**size\_t LZ4F\_headerSize(const void\* src, size\_t srcSize);**

Provide the header size of a frame starting at `src`. `srcSize` must be \>= LZ4F\_MIN\_SIZE\_TO\_KNOW\_HEADER\_LENGTH, which is enough to decode the header length. @return : size of frame header or an error code, which can be tested using LZ4F\_isError() note : Frame header size is variable, but is guaranteed to be \>= LZ4F\_HEADER\_SIZE\_MIN bytes, and \<= LZ4F\_HEADER\_SIZE\_MAX bytes.
**size\_t LZ4F\_getFrameInfo(LZ4F\_dctx\* dctx, LZ4F\_frameInfo\_t\* frameInfoPtr, const void\* srcBuffer, size\_t\* srcSizePtr);**

This function extracts frame parameters (max blockSize, dictID, etc.). Its usage is optional: user can also invoke LZ4F\_decompress() directly. Extracted information will fill an existing LZ4F\_frameInfo\_t structure. This can be useful for allocation and dictionary identification purposes. LZ4F\_getFrameInfo() can work in the following situations : 1) At the beginning of a new frame, before any invocation of LZ4F\_decompress(). It will decode header from `srcBuffer`, consuming the header and starting the decoding process. Input size must be large enough to contain the full frame header. Frame header size can be known beforehand by LZ4F\_headerSize(). Frame header size is variable, but is guaranteed to be \>= LZ4F\_HEADER\_SIZE\_MIN bytes, and not more than \<= LZ4F\_HEADER\_SIZE\_MAX bytes. Hence, blindly providing LZ4F\_HEADER\_SIZE\_MAX bytes or more will always work. It's allowed to provide more input data than the header size, LZ4F\_getFrameInfo() will only consume the header. If input size is not large enough, aka if it's smaller than header size, function will fail and return an error code. 2) After decoding has been started, it's possible to invoke LZ4F\_getFrameInfo() anytime to extract already decoded frame parameters stored within dctx. Note that, if decoding has barely started, and not yet read enough information to decode the header, LZ4F\_getFrameInfo() will fail. The number of bytes consumed from srcBuffer will be updated in \*srcSizePtr (necessarily \<= original value). LZ4F\_getFrameInfo() only consumes bytes when decoding has not yet started, and when decoding the header has been successful. Decompression must then resume from (srcBuffer + \*srcSizePtr). @return : a hint about how many srcSize bytes LZ4F\_decompress() expects for next call, or an error code which can be tested using LZ4F\_isError(). note 1 : in case of error, dctx is not modified. Decoding operation can resume from beginning safely. note 2 : frame parameters are \*copied into\* an already allocated LZ4F\_frameInfo\_t structure.
**size\_t LZ4F\_decompress(LZ4F\_dctx\* dctx, void\* dstBuffer, size\_t\* dstSizePtr, const void\* srcBuffer, size\_t\* srcSizePtr, const LZ4F\_decompressOptions\_t\* dOptPtr);**

Call this function repetitively to regenerate data compressed in `srcBuffer`. The function requires a valid dctx state. It will read up to \*srcSizePtr bytes from srcBuffer, and decompress data into dstBuffer, of capacity \*dstSizePtr. The nb of bytes consumed from srcBuffer will be written into \*srcSizePtr (necessarily \<= original value). The nb of bytes decompressed into dstBuffer will be written into \*dstSizePtr (necessarily \<= original value). The function does not necessarily read all input bytes, so always check value in \*srcSizePtr. Unconsumed source data must be presented again in subsequent invocations. `dstBuffer` can freely change between each consecutive function invocation. `dstBuffer` content will be overwritten. Note: if `LZ4F_getFrameInfo()` is called before `LZ4F_decompress()`, srcBuffer must be updated to reflect the number of bytes consumed after reading the frame header. Failure to update srcBuffer before calling `LZ4F_decompress()` will cause decompression failure or, even worse, successful but incorrect decompression. See the `LZ4F_getFrameInfo()` docs for details. @return : an hint of how many `srcSize` bytes LZ4F\_decompress() expects for next call. Schematically, it's the size of the current (or remaining) compressed block + header of next block. Respecting the hint provides some small speed benefit, because it skips intermediate buffers. This is just a hint though, it's always possible to provide any srcSize. When a frame is fully decoded, @return will be 0 (no more data expected). When provided with more bytes than necessary to decode a frame, LZ4F\_decompress() will stop reading exactly at end of current frame, and @return 0. If decompression failed, @return is an error code, which can be tested using LZ4F\_isError(). After a decompression error, the `dctx` context is not resumable. Use LZ4F\_resetDecompressionContext() to return to clean state. After a frame is fully decoded, dctx can be used again to decompress another frame.
**void LZ4F\_resetDecompressionContext(LZ4F\_dctx\* dctx);** /* always successful */ 

In case of an error, the context is left in "undefined" state. In which case, it's necessary to reset it, before re-using it. This method can also be used to abruptly stop any unfinished decompression, and start a new one using same context resources.

Dictionary compression API

**size\_t LZ4F\_compressBegin\_usingDict(LZ4F\_cctx\* cctx, void\* dstBuffer, size\_t dstCapacity, const void\* dictBuffer, size\_t dictSize, const LZ4F\_preferences\_t\* prefsPtr);**

Inits dictionary compression streaming, and writes the frame header into dstBuffer. @dstCapacity must be \>= LZ4F\_HEADER\_SIZE\_MAX bytes. @prefsPtr is optional : one may provide NULL as argument, however, it's the only way to provide dictID in the frame header. @dictBuffer must outlive the compression session. @return : number of bytes written into dstBuffer for the header, or an error code (which can be tested using LZ4F\_isError()) NOTE: The LZ4Frame spec allows each independent block to be compressed with the dictionary, but this entry supports a more limited scenario, where only the first block uses the dictionary. This is still useful for small data, which only need one block anyway. For larger inputs, one may be more interested in LZ4F\_compressFrame\_usingCDict() below.
**size\_t LZ4F\_decompress\_usingDict(LZ4F\_dctx\* dctxPtr, void\* dstBuffer, size\_t\* dstSizePtr, const void\* srcBuffer, size\_t\* srcSizePtr, const void\* dict, size\_t dictSize, const LZ4F\_decompressOptions\_t\* decompressOptionsPtr);**

Same as LZ4F\_decompress(), using a predefined dictionary. Dictionary is used "in place", without any preprocessing. It must remain accessible throughout the entire frame decoding.

Bulk processing dictionary compression

**LZ4F\_CDict\* LZ4F\_createCDict(const void\* dictBuffer, size\_t dictSize); void LZ4F\_freeCDict(LZ4F\_CDict\* CDict);**

When compressing multiple messages / blocks using the same dictionary, it's recommended to initialize it just once. LZ4\_createCDict() will create a digested dictionary, ready to start future compression operations without startup delay. LZ4\_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. @dictBuffer can be released after LZ4\_CDict creation, since its content is copied within CDict.
**size\_t LZ4F\_compressFrame\_usingCDict(LZ4F\_cctx\* cctx, void\* dst, size\_t dstCapacity, const void\* src, size\_t srcSize, const LZ4F\_CDict\* cdict, const LZ4F\_preferences\_t\* preferencesPtr);**

Compress an entire srcBuffer into a valid LZ4 frame using a digested Dictionary. @cctx must point to a context created by LZ4F\_createCompressionContext(). If @cdict==NULL, compress without a dictionary. @dstBuffer MUST be \>= LZ4F\_compressFrameBound(srcSize, preferencesPtr). If this condition is not respected, function will fail (@return an errorCode). The LZ4F\_preferences\_t structure is optional : one may provide NULL as argument, but it's not recommended, as it's the only way to provide @dictID in the frame header. @return : number of bytes written into dstBuffer. or an error code if it fails (can be tested using LZ4F\_isError()) Note: for larger inputs generating multiple independent blocks, this entry point uses the dictionary for each block.
**size\_t LZ4F\_compressBegin\_usingCDict(LZ4F\_cctx\* cctx, void\* dstBuffer, size\_t dstCapacity, const LZ4F\_CDict\* cdict, const LZ4F\_preferences\_t\* prefsPtr);**

Inits streaming dictionary compression, and writes the frame header into dstBuffer. @dstCapacity must be \>= LZ4F\_HEADER\_SIZE\_MAX bytes. @prefsPtr is optional : one may provide NULL as argument, note however that it's the only way to insert a @dictID in the frame header. @cdict must outlive the compression session. @return : number of bytes written into dstBuffer for the header, or an error code, which can be tested using LZ4F\_isError().
**typedef enum { LZ4F\_LIST\_ERRORS(LZ4F\_GENERATE\_ENUM) \_LZ4F\_dummy\_error\_enum\_for\_c89\_never\_used } LZ4F\_errorCodes;**

Advanced compression operations

**LZ4FLIB\_STATIC\_API size\_t LZ4F\_getBlockSize(LZ4F\_blockSizeID\_t blockSizeID);**

@return, in scalar format (size\_t), the maximum block size associated with @blockSizeID, or an error code (can be tested using LZ4F\_isError()) if @blockSizeID is invalid.
**LZ4FLIB\_STATIC\_API size\_t LZ4F\_uncompressedUpdate(LZ4F\_cctx\* cctx, void\* dstBuffer, size\_t dstCapacity, const void\* srcBuffer, size\_t srcSize, const LZ4F\_compressOptions\_t\* cOptPtr);**

LZ4F\_uncompressedUpdate() can be called repetitively to add data stored as uncompressed blocks. Important rule: dstCapacity MUST be large enough to store the entire source buffer as no compression is done for this operation If this condition is not respected, LZ4F\_uncompressedUpdate() will fail (result is an errorCode). After an error, the state is left in a UB state, and must be re-initialized or freed. If previously a compressed block was written, buffered data is flushed first, before appending uncompressed data is continued. This operation is only supported when LZ4F\_blockIndependent is used. `cOptPtr` is optional : NULL can be provided, in which case all options are set to default. @return : number of bytes written into `dstBuffer` (it can be zero, meaning input data was just buffered). or an error code if it fails (which can be tested using LZ4F\_isError())

Custom memory allocation

**typedef void\* (\*LZ4F\_AllocFunction) (void\* opaqueState, size\_t size); typedef void\* (\*LZ4F\_CallocFunction) (void\* opaqueState, size\_t size); typedef void (\*LZ4F\_FreeFunction) (void\* opaqueState, void\* address); typedef struct { LZ4F\_AllocFunction customAlloc; LZ4F\_CallocFunction customCalloc;** /* optional; when not defined, uses customAlloc + memset */ **LZ4F\_FreeFunction customFree; void\* opaqueState; } LZ4F\_CustomMem; static #ifdef \_\_GNUC\_\_ \_\_attribute\_\_((\_\_unused\_\_)) #endif LZ4F\_CustomMem const LZ4F\_defaultCMem = { NULL, NULL, NULL, NULL };** /**< this constant defers to stdlib's functions */ 

These prototypes make it possible to pass custom allocation/free functions. LZ4F\_customMem is provided at state creation time, using LZ4F\_create\*\_advanced() listed below. All allocation/free operations will be completed using these custom variants instead of regular ones.
**LZ4FLIB\_STATIC\_API size\_t LZ4F\_cctx\_size(const LZ4F\_cctx\* cctx); LZ4FLIB\_STATIC\_API size\_t LZ4F\_dctx\_size(const LZ4F\_dctx\* dctx);**

These functions return the total memory footprint of the provided context.