Internode Transport Buffer Contract

Status: design note only. This document defines a backend-neutral buffer ownership and lifecycle contract for the InternodeDataTransport adapter. It does not implement a new backend and does not change production behavior.

Open-source Scope

The open-source RustFS path keeps tcp-http as the default internode data transport. This document defines adapter contracts only:

no additional production backend is introduced;
no dependency is added;
no new accepted production backend value is added;
RustFS core data-plane logic remains independent of the concrete transport implementation.

Current Adapter Surface

The current data-plane surface is byte-stream based:

Current path	Current API shape	Current ownership
Remote read stream	`InternodeDataTransport::open_read(...) -> FileReader`	Backend returns boxed `AsyncRead`; callers provide temporary `ReadBuf` storage per poll.
Remote write stream	`InternodeDataTransport::open_write(...) -> FileWriter`	Callers pass borrowed `&[u8]` slices into boxed `AsyncWrite`; the backend owns any async body staging.
Walk-dir stream	`InternodeDataTransport::open_walk_dir(...) -> FileReader`	Same boxed stream model as read, with a small serialized request body.

This API is correct for the current TCP/HTTP backend. The adapter contract describes current ownership boundaries without assuming implementation details outside TcpHttpInternodeDataTransport.

Buffer Ownership Model

Buffer role	Allocator	Lifetime owner	Transport state	TCP/HTTP behavior
Send buffer	Caller or RustFS-owned pool	Caller until the writer copies or accepts bytes for the HTTP body	Existing HTTP body staging	Copy into the existing `AsyncWrite` path when the writer cannot use the borrowed slice directly.
Receive buffer	Caller-provided storage	Reader while filling; caller after `poll_read` returns	Existing HTTP response body chunks	Copy from `AsyncRead` into caller storage as today.
Control metadata	RustFS caller	Caller/request object	Not buffer-managed by the data-plane backend	Serialize into HTTP/gRPC/control-plane messages.
Fallback staging	TCP/HTTP backend	TCP/HTTP backend	Existing `HttpReader`/`HttpWriter` buffers	Existing buffering semantics.

The current writer must not retain borrowed caller slices beyond the write call. When bytes must outlive the call, they are copied into owned HTTP body chunks.

This contract does not claim zero-copy behavior. The current TCP/HTTP path documents where copies occur.

Compatibility Contract

The current stream API remains the OSS compatibility contract:

rust

#[async_trait::async_trait]
pub trait InternodeDataTransport {
    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
}

Current Adapter Contract

Area	Required contract
Ownership	Define when caller-owned bytes are copied or accepted by the transport.
Completion	Return from stream operations only when bytes are accepted or an error is reported.
Staging	Keep staging behavior inside the TCP/HTTP implementation.
Size limits	Report any RustFS-visible `max_transfer_size`; TCP/HTTP currently reports none.
Ordering	Preserve ordered byte-stream semantics.
Copy accounting	Document known copy boundaries and avoid unmeasured zero-copy claims.

Current API Limitations

Current API	Current limitation
`FileReader = Box<dyn AsyncRead + Send + Sync + Unpin>`	`AsyncRead` exposes temporary caller `ReadBuf` storage.
`FileWriter = Box<dyn AsyncWrite + Send + Sync + Unpin>`	`AsyncWrite::poll_write` receives borrowed `&[u8]` that cannot outlive the poll.
`HttpWriter`	The async HTTP body must own `Bytes`, so borrowed write buffers are copied into `BytesMut` or `Bytes`.
`write_body_chunks_to_writer`	Server-side HTTP body chunks are copied into `BytesMut` before local disk write.
Erasure encode output	Encoded shards are represented as `Vec<Bytes>` and written through `AsyncWrite`.
Erasure decode input	Shard reads allocate `Vec<u8>` buffers before decode.

These limitations do not block the current tcp-http backend.

Adapter Stability

InternodeDataTransport should keep RustFS core data-plane logic separate from the concrete transport implementation. The trait and tcp-http backend remain inside ecstore.

This PR does not perform a crate split, add runtime loading, introduce a plugin system, add a backend value, or implement a new transport backend.