In an erasure-coded pool with k data shards and m parity shards, the
primary-capable shards are:

- The first data shard
- All m parity shards

This means there are **m + 1 primary-capable shards** in total. For example,
in a k=4, m=2 configuration, shards 0, 4, and 5 are primary-capable.

.. ditaa::

   EC Pool (k=4, m=2)

   +--------+  +--------+  +--------+  +--------+  +--------+  +--------+
   | Shard  |  | Shard  |  | Shard  |  | Shard  |  | Shard  |  | Shard  |
   |   0    |  |   1    |  |   2    |  |   3    |  |   4    |  |   5    |
   | (Data) |  | (Data) |  | (Data) |  | (Data) |  | (Parity|  | (Parity|
   |PRIMARY |  |        |  |        |  |        |  |PRIMARY |  |PRIMARY |
   |CAPABLE |  |        |  |        |  |        |  |CAPABLE |  |CAPABLE |
   +--------+  +--------+  +--------+  +--------+  +--------+  +--------+
       |                                               |           |
       v                                               v           v
   +--------+                                      +--------+  +--------+
   |  Omap  |                                      |  Omap  |  |  Omap  |
   | Replica|                                      | Replica|  | Replica|
   +--------+                                      +--------+  +--------+

   Primary-capable shards (0, 4, 5) maintain omap replicas

Each primary-capable shard maintains a complete copy of the omap data for
objects in the PG. This replication ensures that omap data remains available
even if some shards fail, and allows any primary-capable shard to serve omap
read requests when acting as primary.

Journal Implementation
~~~~~~~~~~~~~~~~~~~~~~

The ECOmapJournal is maintained on the primary OSD. The journal:

- Records all omap updates before they are applied
- Ensures atomic application of updates across all primary-capable replicas
- Enables recovery in case of failures during update operations
- Provides a consistent view of omap state during recovery
- Handles object deletion and recreation

Object State Map
^^^^^^^^^^^^^^^^

The journal maintains an ``object_state_map`` to track objects that are in the
process of being deleted. This map is critical for ensuring that omap updates
are written to the correct object generation when objects are deleted and
recreated.

The object_state_map:

- **Tracks outstanding deletes**: When a delete operation is appended to the
  journal, the object's version number is added to the map along with a boolean
  indicating whether it's a lost delete
- **Manages version lifecycle**: The version number remains in the map until
  the delete is trimmed from the PG log
- **Determines generation numbers**: The version number is used to calculate
  the generation number for any outstanding omap updates, ensuring updates are
  applied to the correct object generation
- **Handles object recreation**: If an object is deleted and then recreated
  before the delete is trimmed, the map ensures omap updates target the
  appropriate generation

When ``get_generation()`` is called for an object, it returns:

- The lowest version number from the object_state_map if any deletes are
  outstanding
- A boolean indicating whether the delete was lost
- ``NO_GEN`` if no deletes are outstanding for the object

This mechanism prevents omap updates from being applied to the wrong generation
of an object, which could occur if an object is deleted and recreated while
omap updates are still in flight.

**Operations Using append_delete/trim_delete:**

The object_state_map's ``append_delete`` and ``trim_delete`` sequence is used
by several operations that involve object deletion and recreation:

- **Explicit deletes**: Direct object deletion operations
- **REPLACE operations**: ``copy_from`` operations that atomically delete and
  recreate objects (see below)
- **Clone operations to non-snapshot objects**: When cloning to a non-snapshot
  object, the target object is effectively deleted and recreated with the
  cloned content, requiring the same generation tracking as other
  delete-and-recreate operations

All of these operations follow the same pattern: a delete is appended to the
object_state_map when the operation is logged, and the version is trimmed when
the PG log entry is eventually removed. This ensures consistent generation
tracking regardless of which operation causes the object lifecycle transition.

**Clone Operations and Outstanding Omap Updates:**

Clone operations require special handling to ensure that outstanding omap
updates are properly applied to the cloned object. When a clone operation is
performed:

#. A visitor pattern is used to traverse the PG log and accumulate all
   outstanding omap updates for the source object
#. These accumulated omap updates are collected from journal entries that have
   not yet been applied to the object store
#. All accumulated updates are then applied to the clone transaction, ensuring
   the cloned object receives the complete, up-to-date omap state
#. This process ensures that the clone includes not just the omap data from the
   object store, but also any in-flight updates that exist only in the journal

This visitor-based approach is necessary because the journal may contain omap
updates that have been logged but not yet applied to the source object's
persistent storage. Without accumulating these updates, the clone would have
stale omap data, missing recent modifications. By applying all outstanding
updates to the clone transaction, the system ensures that clones are created
with a consistent and complete view of the object's omap state, including both
persisted data and in-flight journal entries.

**Trimmer Architecture for Post-Removal Scenarios:**

The EC omap implementation uses a trimmer to determine when to apply journal
updates to the underlying BlueStore. However, a special case arises when an
object has just been deleted: we must avoid applying omap updates to an object
that no longer exists. To handle this, a specialized trimmer architecture has
been implemented:

- **TrimmerPostRemove**: A base class that performs all standard trimming
  actions except EC omap operations. This trimmer handles PG log cleanup
  without attempting to apply omap updates to the object store.

- **Trimmer**: Inherits from TrimmerPostRemove and overrides the ``ec_omap``
  method to add EC omap update application. This is the standard trimmer used
  during normal PG log maintenance.

- **trim_after_remove()**: A function that uses TrimmerPostRemove to trim the
  PG log immediately after an object has been removed. This is called in
  PGLog.h after a remove operation (``rollbacker->trim(i)``).

The key insight is that when trimming after a remove operation, we want to
clean up the PG log entries but we must not apply any omap updates from those
entries to BlueStore, since the object has just been deleted. By using
TrimmerPostRemove (which skips the ``ec_omap`` step), the system ensures that:

#. PG log entries are properly trimmed after object removal
#. Omap updates from those entries are not applied to the now-deleted object
#. The journal's object_state_map is updated appropriately via trim_delete
#. Normal trimming (using the full Trimmer class) continues to apply omap
   updates for objects that still exist

This design prevents attempting to write omap data to deleted objects while
maintaining proper journal cleanup and state tracking.

REPLACE Operation Type
^^^^^^^^^^^^^^^^^^^^^^

To properly support the object_state_map mechanism, a new ``REPLACE`` operation
type has been added to ``pg_log_entry_t``. This operation type is used in place
of the ``MODIFY`` operation type for ``copy_from`` operations where an object
is deleted and recreated.

**Why REPLACE is Necessary:**

The ``copy_from`` operation atomically deletes an existing object and recreates
it with new content. Without the REPLACE operation type, this would be logged
as a simple MODIFY operation, which would not trigger the journal to track the
deletion in the object_state_map. This creates a critical problem:

- If omap updates are in flight when a ``copy_from`` occurs, they could be
  applied to the wrong generation of the object
- The journal would not know that the object was deleted and recreated
- Outstanding omap updates would target the old generation, causing data
  corruption or inconsistency

**How REPLACE Works:**

When a REPLACE operation is logged:

#. The journal appends a delete to the object_state_map, recording the version
   number
#. This ensures that any outstanding omap updates will use the correct
   generation number
#. The object is then recreated with new content
#. When the delete is eventually trimmed from the PG log, the version is
   removed from the object_state_map

This approach ensures that ``copy_from`` operations, which are commonly used
for object cloning and migration, correctly interact with the omap journal's
generation tracking mechanism. Without REPLACE, the object_state_map would not
be aware of the implicit delete, leading to potential data corruption when omap
updates are applied to recreated objects.

Two-Phase Update Design
^^^^^^^^^^^^^^^^^^^^^^^

For EC pools, omap updates are persisted in PG log entries first and are then
applied to the object store once all copies have been updated and the transaction
can no longer be rolled back. This two-phase update approach is more efficient
than reading and saving the old omap data in case the transaction has to be
rolled back.

To avoid omap reads having to search PG log entries for recent updates, the
ECOmapJournal tracks updates that have not yet been applied to the object store
in memory. The ECOmapJournal provides a fast way of locating recent omap updates,
ensuring efficient read operations while updates are in flight.

Journal entries contain:

- List of omap Updates:
    - Operation type (set, remove, clear)
    - Bufferlist containing details about the operation (e.g. key/value pairs)
- An optional omap header
- A 'clear omap' boolean
- The object version

The journals on primary-capable shards that are not the primary shard store
the object deletion information, but not the omap updates. This allows for
updates to be committed to the correct object generation.

Journal Persistence and Peering
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The ECOmapJournal does not need to be persistent because the updates are also
stored in the PG log entries. The journal is short-lived and volatile, containing
only entries for in-flight writes that are updating an omap. Whenever a new peering
interval starts, the journal is discarded. After any disruption, the Peering process
will roll forward or backward each outstanding entry in the PG log so the object
store will be up to date, eliminating the need for complicated reconciliation of the
log and journal.

Commit Protocol
~~~~~~~~~~~~~~~

The omap commit protocol ensures that updates are applied consistently across
all primary-capable shards. The protocol is divided into two phases: Apply
Update and Complete Update.

Apply Update Phase
^^^^^^^^^^^^^^^^^^

This is entered when the PG needs to apply omap updates. The primary adds the updates
to its journal and replicates them via the PG log:

.. ditaa::

   Primary                    Primary-Capable           Primary-Capable
                                 Shard 1                   Shard 2
      |                             |                          |
      | Store in PG Log             |                          |
      |------+                      |                          |
      |      |                      |                          |
      |<-----+                      |                          |
      |                             |                          |
      | Add to Journal              |                          |
      |------+                      |                          |
      |      |                      |                          |
      |<-----+                      |                          |
      |                             |                          |
      | Send PG Log Entry           |                          |
      |---------------------------->|                          |
      |                             |                          |
      | Send PG Log Entry           |                          |
      |------------------------------------------------------->|
      |                             |                          |
      |                             | Store in PG Log          |
      |                             |------+                   |
      |                             |      |                   |
      |                             |<-----+                   | Store in PG Log
      |                             |                          |------+
      |                         ACK |                          |      |
      |<----------------------------|                          |<-----+
      |                             |                          |
      |                             |                      ACK |
      |<-------------------------------------------------------|
      |                             |                          |

Complete Update Phase
^^^^^^^^^^^^^^^^^^^^^

Triggered later (during PG log trim). The primary applies updates to
object stores and coordinates completion across all shards:

.. ditaa::

   Primary                    Primary-Capable           Primary-Capable
                                 Shard 1                   Shard 2
      |                             |                          |
      | Apply to Object Store       |                          |
      |------+                      |                          |
      |      |                      |                          |
      |<-----+                      |                          |
      |                             |                          |
      | Remove Journal Entry        |                          |
      |------+                      |                          |
      |      |                      |                          |
      |<-----+                      |                          |
      |                             |                          |
      | Complete Update             |                          |
      |---------------------------->|                          |
      |                             |                          |
      | Complete Update             |                          |
      |------------------------------------------------------->|
      |                             |                          |
      |                             | Apply to Store           |
      |                             |------+                   |
      |                             |      |                   |
      |                             |<-----+                   | Apply to Store
      |                             |                          |------+
      |                         ACK |                          |      |
      |<----------------------------|                          |<-----+
      |                             |                          |
      |                             |                      ACK |
      |<-------------------------------------------------------|
      |                             |                          |

Protocol Steps
^^^^^^^^^^^^^^

**Apply Update Phase:**

#. Primary adds the omap updates to its local ECOmapJournal as an ECOmapJournalEntry
#. Primary encodes the omap updates into the PG log as a PG log entry
#. Primary sends PG log entry to all other primary-capable shards
#. Each shard stores the PG log entry in its local PG log

**Complete Update Phase:**

#. Primary applies the omap updates from a PG log entry to its own object store
#. Primary removes the corresponding journal entry
#. Primary sends "Complete Update" messages to all primary-capable shards
#. Each shard applies the update from its PG log to its object store
#. Each shard sends an ACK back to the primary

This journal approach means that if a write fails before it is completed,
there is nothing to rollback in the object stores. This means that it is
not necessary to read and store old omap data just incase an update needs
to be undone.

Recovery and Consistency
~~~~~~~~~~~~~~~~~~~~~~~~~

Omap recovery is integrated into the existing EC recovery loop. When a
primary-capable shard recovers:

- The recovering shard receives omap data from the current primary or another
  primary-capable shard
- Omap data is transferred as part of the normal EC recovery process

During primary failover:

- The new primary (which must be a primary-capable shard) already has a
  complete copy of the omap data
- A new journal is initialized on the new primary
- Operations can continue without data loss

This integration with existing recovery mechanisms simplifies the
implementation and ensures consistency with EC pool recovery behavior.

Omap Operations
---------------

Supported Operations
~~~~~~~~~~~~~~~~~~~~

The following omap operations are supported in EC pools:

- ``omap_get_keys``: Retrieve all keys in the omap
- ``omap_get_vals``: Retrieve all key-value pairs in the omap
- ``omap_get_vals_by_keys``: Retrieve specific key-value pairs
- ``omap_set``: Set one or more key-value pairs
- ``omap_rm_keys``: Remove one or more keys
- ``omap_clear``: Remove all key-value pairs
- ``omap_get_header``: Retrieve the omap header
- ``omap_set_header``: Set the omap header
- ``omap_cmp``: Compare omap values with other values

These operations provide the same semantics as in replicated pools, ensuring
compatibility with existing applications.

Read Operation Flow
~~~~~~~~~~~~~~

Read operations follow a simple flow:

.. ditaa::

   Client                Primary
      |                     |
      | Omap Read           |
      |-------------------->|
      |                     |
      |                     | Read Local Omap
      |                     |------+
      |                     |      |
      |                     |<-----+
      |                     |
      |                     | Apply Journal Updates
      |                     |------+
      |                     |      |
      |                     |<-----+
      |                     |
      | Return Data         |
      |<--------------------|

Read operations are served from the primary OSD by:

#. Reading the stored omap data from the local replica
#. Applying any pending updates from the ECOmapJournal on top of the stored
   omap
#. Returning the combined result to the client

Using a journal means that there is a lag between an omap update and the
update being applied to the object store. Therefore, it is important that
modifications in the journal are considered during client omap reads, to
ensure that the correct data is returned.

The journal updates are applied in-memory during the read
operation, providing low-latency access to the omap data while maintaining
consistency.

Journal Overhead
~~~~~~~~~~~~~~~~

The journal introduces some performance overhead:

- **Journal Updates**: Each omap update requires the journal to be updated
- **Latency**: Omap operations require the primary osd to check the journal for
  updates
- **Storage**: Journal entries consume memory on the primary osd

However, this overhead is acceptable given the consistency guarantees provided.
Performance testing will quantify the impact and guide optimisation efforts.

Replication Impact
~~~~~~~~~~~~~~~~~~

Replicating omap data across primary-capable shards has performance
implications:

- **Network Traffic**: Updates generate network traffic to multiple shards
- **Storage**: Each primary-capable shard stores a complete omap replica
- **CPU**: Applying updates on multiple shards consumes CPU

These costs are offset by the benefits of high availability and fast reads.

Crimson-Specific Considerations
--------------------------------

The Crimson implementation will need to:

- Implement omap support using Crimson's asynchronous architecture
- Integrate with Crimson's seastar-based I/O framework
- Adapt the journal mechanism to Crimson's storage backend
- Ensure compatibility with the classic OSD implementation


Synchronous Reads
=================

Motivation
----------

Synchronous read operations are required to support Cls operations in EC pools.
Cls methods must execute synchronously, meaning they must complete a read
operation and receive the data before proceeding with their logic. The
traditional asynchronous read path in the EC backend does not provide this
capability.

Additionally, synchronous reads are beneficial for:

- Simplifying certain code paths that require sequential operations
- Enabling synchronous semantics without blocking threads
- Supporting future features that require synchronous data access

The key challenge is implementing synchronous semantics without blocking
threads, which would harm performance and scalability.

Implementation Design
---------------------

The synchronous read implementation uses Boost pull-type coroutines to provide
synchronous semantics without blocking threads:

- A Boost coroutine is created for the synchronous read operation
- The coroutine initiates an asynchronous read and yields control
- When the asynchronous read completes, the coroutine is resumed
- The coroutine returns the read data to the caller

This approach provides the synchronous semantics required by Cls operations
while maintaining the performance benefits of asynchronous I/O and avoiding
thread blocking.

Boost Pull-Type Coroutines
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The implementation uses Boost.Coroutine2 pull-type coroutines to bridge
asynchronous and synchronous code. The coroutine:

- Yields control back to the caller when waiting for I/O
- Allows the thread to process other work while I/O is in progress
- Resumes execution when the asynchronous operation completes
- Provides synchronous semantics to the Cls operation

This mechanism allows Cls operations to be written in a straightforward,
synchronous style while the underlying I/O remains asynchronous and
non-blocking.

Integration with EC Backend
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The synchronous read path integrates with the existing EC backend:

- The ECSwitch routes synchronous reads to the EC Backend
- The handler uses the existing asynchronous read infrastructure
- Results are returned synchronously to the caller using the coroutine

This integration minimizes code duplication and leverages existing, well-tested
read logic.

Synchronous Operation Ordering Guarantees
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Synchronous operations provide strong ordering guarantees:

- Synchronous reads block concurrent operations to the same object
- Multiple synchronous reads to the same object are serialized


Performance Impact
------------------

Latency Considerations
~~~~~~~~~~~~~~~~~~~~~~

Synchronous reads introduce some latency overhead compared to pure asynchronous
operations:

- **Coroutine Overhead**: Creating and resuming coroutines has a small CPU cost
- **Context Switching**: Yielding and resuming coroutines involves context
  switching overhead

However, these overheads are minimal compared to the I/O latency itself. In
practice, the latency impact will be negligible for most workloads.

Throughput Implications
~~~~~~~~~~~~~~~~~~~~~~~~

The throughput impact of synchronous reads depends on the workload:

- **Cls-Heavy Workloads**: Workloads with many Cls operations may see some
  impact, but the coroutine approach minimizes this
- **Mixed Workloads**: Workloads with a mix of synchronous and asynchronous
  operations will see minimal impact
- **Pure Data Workloads**: Workloads without Cls operations will see very
  minimal impact

Performance testing will quantify these impacts and guide optimisation efforts.

Comparison with Async Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Synchronous reads are not intended to replace asynchronous operations. They
serve a specific purpose for Cls operations and other use cases requiring
synchronous semantics. The EC backend continues to use asynchronous reads for
all other operations, maintaining optimal performance for the common case.


Cls, RBD, RGW and CephFS Support
================================

Cls (Class) Support
-------------------

Background
~~~~~~~~~~

Cls (class) operations are server-side methods that execute on OSDs, enabling
efficient data processing without client round-trips. Examples include:

- **RBD Operations**: Image metadata management, snapshot operations
- **RGW Operations**: Bucket index operations, object tagging
- **Custom Operations**: User-defined server-side processing

Cls operations are inherently synchronous - they must read data, process it,
and potentially write results, all within a single operation context. This
synchronous nature is why they require synchronous read support in the EC
backend.

Current Limitations in EC Pools
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Without synchronous read support, Cls operations could not be implemented in
EC pools. This prevented:

- RBD from using EC pools for image storage
- RGW from using EC pools for certain bucket operations
- Custom Cls methods from working with EC pools

Enabling Cls in EC Pools
~~~~~~~~~~~~~~~~~~~~~~~~~

Technical Requirements
^^^^^^^^^^^^^^^^^^^^^^

Enabling Cls in EC pools requires:

#. **Synchronous Read Support**: Implemented via coroutines as described above
#. **Omap Support**: Many Cls operations require omap for metadata storage

All of these requirements are met by the features described in this document.

Integration with Synchronous Reads
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cls operations use synchronous reads through a straightforward integration:

.. code-block:: cpp

   // Simplified Cls operation using synchronous reads
   int cls_method(cls_method_context_t hctx)
   {
       bufferlist bl;

       // Synchronous read - blocks until data is available
       int r = cls_cxx_read(hctx, 0, 1024, &bl);
       if (r < 0)
           return r;

       // Process data
       process_data(bl);

       // Write results
       return cls_cxx_write(hctx, 0, bl);
   }

The ``cls_cxx_read`` function internally uses the synchronous read path,
suspending the coroutine until data is available.

Integration with Omap Support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Many Cls operations require omap access for metadata:

.. code-block:: cpp

   // Cls operation using omap
   int cls_method_with_omap(cls_method_context_t hctx)
   {
       map<string, bufferlist> vals;

       // Read omap values
       int r = cls_cxx_map_get_vals(hctx, "", "", 100, &vals);
       if (r < 0)
           return r;

       // Process metadata
       process_metadata(vals);

       // Update omap
       return cls_cxx_map_set_vals(hctx, &vals);
   }

The omap operations work seamlessly with Cls, providing the metadata storage
required for complex operations.

RBD Support
-----------

RBD (RADOS Block Device) is a primary beneficiary of Cls support in EC pools.
RBD uses Cls operations extensively for:

- Image metadata management
- Snapshot operations
- Clone operations
- Exclusive lock management

With Cls support, RBD can now use EC pools for metadata. This gives the user more flexibility
about how they use RBD, including the option to use a single EC pool for data and metadata.

RGW Support
-----------

RGW (RADOS Gateway) benefits immensely from omap and Cls support in EC pools,
as it heavily relies on these features for S3 and Swift object storage semantics.
Traditionally, RGW required separate replicated pools for metadata and bucket indices.
RGW uses omap and Cls operations extensively for:

- Bucket index management (tracking objects within buckets)
- Multipart upload state tracking and assembly
- User quota and usage tracking
- Object extended attributes and custom metadata

With omap and synchronous read support natively in EC pools, as well as a few tweaks to remove
current restrictions, users will be able to use EC pools as metadata pools in RGW.

CephFS Support
--------------

CephFS (Ceph File System) requires robust omap support and strictly consistent reads
to maintain POSIX-compliant file system semantics.
Historically, the MDS (Metadata Server) required a dedicated replicated pool for its
metadata backing store. CephFS relies on omap and synchronous operations for:

- Directory object management (storing dentries as omap key-value pairs)
- MDS journal and log storage
- File extended attributes (xattrs) and layout metadata
- Inode state management and lock tracking

By bringing omap and synchronous reads to EC pools, as well as a few tweaks to remove 
current retrictions, users will be able to use an EC pool as the metadata pool in CephFS.


Testing
=======

Test Strategy Overview
----------------------

The testing strategy for these features is comprehensive and multi-layered:

- **Omap Journal Unit Tests**: Test the functionality of the journal in isolation
- **Omap Integration Tests**: Test omap operations and recovery in a full rados cluster
- **Cls Integration Tests**: Test cls method calls in a full rados cluster
- **EC Omap in Teuthology Tests**: Allow omap operations in EC pools with ceph_test_rados
- **RBD in Teuthology Tests**: Test RBD in teuthology with an EC metadata pool, and a single EC pool for data and metadata
- **RGW in Teuthology Tests**: Test RGW in teuthology using EC metadata pools
- **CephFS in Teuthology Tests**: Test CephFS in teuthology using an EC metadata pool

A key aspect of the testing strategy is the use of common test fixtures that
enable running existing tests on both replicated and fast EC pools.

Common Test Class Approach
~~~~~~~~~~~~~~~~~~~~~~~~~~~

A common test class has been implemented that:

- Provides a unified interface for test cases
- Supports both replicated and FastEC pools
- Allows existing test suites to run on EC pools with minimal modifications
- Ensures consistent test coverage across pool types
- Reduces code duplication

This approach significantly increases test coverage while minimizing test
development effort.

Existing Test Suite Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Integration with existing test suites includes:

- Running existing Cls tests on EC pools
- Enabling omap operations in EC pools for ceph_test_rados
- Changing all uses of RBD, RGW and CephFS to use just an EC pool in Teuthology

This integration ensures comprehensive coverage with minimal new test
development.

Migration and Compatibility
===========================

Release Requirements
--------------------

Umbrella Release Requirement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ability to enable omap support on EC pools will be tied to the **Umbrella
release**. This introduces important version requirements:

- **All OSDs must be running at least the Umbrella release** before omap
  support can be enabled on any EC pool
- **Any OSDs added to the cluster in the future** must also be running at least
  the Umbrella release
- Attempting to enable omap support on a cluster with pre-Umbrella OSDs will
  fail with an error

This requirement ensures that all OSDs in the cluster have the necessary code
to support omap operations on EC pools, preventing data corruption or
inconsistencies that could arise from version mismatches.

Upgrade Path
------------

Enabling Features on Existing Pools
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Existing EC pools can be upgraded to support the new features:

#. Upgrade all OSDs to the Umbrella release or later
#. Verify that all OSDs in the cluster are at the required version
#. Enable EC overwrites on the pool (if not already enabled)
#. Enable EC optimisations on the pool (if not already enabled)
#. Enable omap support on the pool (if desired)
#. Existing data remains accessible throughout the upgrade

The upgrade process is designed to be non-disruptive, but the version
requirement must be strictly enforced.

Backward Compatibility
~~~~~~~~~~~~~~~~~~~~~~

Backward compatibility is maintained with important caveats:

- Pools without omap support continue to work as before on any version
- Clients that don't use the new features are unaffected
- **Downgrade is not supported** once omap has been enabled on an EC pool, as
  pre-Umbrella OSDs cannot handle omap data on EC pools
- Pools with omap enabled require all OSDs to remain at Umbrella or later

This compatibility ensures that upgrades are safe, but downgrades are
restricted to protect data integrity.

Configuration
-------------

Required Settings
~~~~~~~~~~~~~~~~~

To enable the new features, the following OSDMap pool settings are required:

- ``allows_ec_overwrites = true`
- ``allows_ec_optimizations = true``
- ``supports_omap = true``

These settings can be configured per-pool. The cluster will enforce that all
OSDs are at the Umbrella release before allowing omap support to be enabled.