Large File Streaming

Large file streaming lets a piece process a big file by passing it through as a stream — transferring the bytes a chunk at a time — instead of loading the whole file into memory first. It is useful when a file is too large to hold in RAM at either the source or the destination. Because the file is never fully buffered, a flow can move files far larger than the worker's memory budget.

Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB, minus overhead — see Limits). Reading a large file into a Buffer holds the entire file in that budget at once, so a big enough file exhausts the memory and the worker is OOM-killed.

Streaming avoids this: the bytes flow through as a Node Readable, roughly 5 MB at a time, so no process ever holds the whole file. The transfer is transparent — there are no extra "chunk" steps in your flow; a streaming action looks and behaves like any other.

When to use it

• Buffer (a Buffer, the default) — small files of known size where holding the whole file in memory is cheap and simple.
• Stream (a Readable) — large files, files of unknown size, or app-to-app transfers (e.g. downloading a large object from one service and uploading it to another) where buffering would risk exhausting memory. </Note>

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and storage-to-storage transfers.

Pieces that support streaming

Streaming is enabled per action. Today:

More pieces are being enabled over time. Actions not listed here still work — they buffer the file in memory, which is fine within the size limit.

Building streaming actions

If you are building a piece, you can stream on both sides: read an input file as a stream, and write an output file as a stream.

Writing a file as a stream

ctx.files.write accepts a Buffer or a Readable. Pass a Readable — such as an S3 object body or a streaming HTTP response — and it streams straight to storage instead of being buffered. It returns a file reference string you return from the action, exactly like the buffered form (see Files).

async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable — hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}

Reading a file as a stream

Add streaming: true to a Property.File. The property then resolves to an ApStreamingFile instead of an ApFile:

type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};

Consume body directly. Prefer an uploader that accepts a stream of unknown length — for S3 that is Upload from @aws-sdk/lib-storage, which splits the body into ~5 MB parts and needs no content length up front.

props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { file } = context.propsValue;
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: file.body,
    },
  }).done();
}

size is informational and best-effort — it is undefined when the source reports no Content-Length, and it is also dropped when the response is compressed (Content-Encoding: gzip/br/deflate), because the decompressed body no longer matches the advertised length. Don't require it: only pass it to an API that needs an explicit content length, and make sure that path has a buffered fallback for when it is missing.

Limits & storage

• Writing into storage is capped. A stream passed to ctx.files.write is counted against AP_MAX_FILE_SIZE_MB (Cloud: 10 MB, self-hosted default: 25 MB) while the bytes flow; exceeding it aborts the transfer and fails the step. See Limits.
• Reading a streamed input is not capped. A streaming: true file input has no AP_MAX_FILE_SIZE_MB ceiling — that is deliberate, since the point of the feature is to move files larger than the cap out to an external service.
• Storage backend. Streaming end-to-end requires S3 file storage — see the callout at the top of this page.