Back to Lance

Format Versioning

docs/src/format/table/versioning.md

11.0.02.9 KB
Original Source

Format Versioning

Feature Flags

As the table format evolves, new feature flags are added to the format. There are two separate fields for checking for feature flags, depending on whether you are trying to read or write the table. Readers should check the reader_feature_flags to see if there are any flag it is not aware of. Writers should check writer_feature_flags. If either sees a flag they don't know, they should return an "unsupported" error on any read or write operation.

Current Feature Flags

<style> .feature-flags-table th:nth-child(2), .feature-flags-table td:nth-child(2) { white-space: nowrap; min-width: 250px; } </style> <div class="feature-flags-table" markdown="1">
Flag BitFlag NameReader RequiredWriter RequiredDescription
1FLAG_DELETION_FILESYesYesFragments may contain deletion files, which record the tombstones of soft-deleted rows.
2FLAG_STABLE_ROW_IDSYesYesRow IDs are stable for both moves and updates. Fragments contain an index mapping row IDs to row addresses.
4FLAG_USE_V2_FORMAT_DEPRECATEDNoNoFiles are written with the new v2 format. This flag is deprecated and no longer used.
8FLAG_TABLE_CONFIGNoYesTable config is present in the manifest.
16FLAG_BASE_PATHSYesYesDataset uses multiple base paths (for shallow clones or multi-base datasets).
32FLAG_DISABLE_TRANSACTION_FILENoYesTransactions are recorded in the manifest rather than in a separate transaction file.
64FLAG_UNSTABLE_DATA_OVERLAY_FILESYesYesFragments may carry data overlay files. Unstable: release builds reject it unless explicitly opted in.
128FLAG_COVERED_INDEX_METADATAYesYesSome index declares covering columns (IndexMetadata.covering_fields), so fields means keyed columns followed by carried ones. An implementation without this flag selects an index by membership of fields and would answer a query on a merely-carried column with an index keyed on a different one.
</div>

Flags with bit values 256 and above are unknown and will cause implementations to reject the dataset with an "unsupported" error.