Large File Streaming

Large file streaming lets a piece process a big file by passing it through as a stream — transferring the bytes a chunk at a time — instead of loading the whole file into memory first. It is useful when a file is too large to hold in RAM at either the source or the destination. Because the file is never fully buffered, a flow can move files far larger than the worker's memory budget.

Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB, minus overhead — see Limits). Reading a large file into a Buffer holds the entire file in that budget at once, so a big enough file exhausts the memory and the worker is OOM-killed.

Streaming avoids this: the bytes flow through as a Node Readable, roughly 5 MB at a time, so no process ever holds the whole file. The transfer is transparent — there are no extra "chunk" steps in your flow; a streaming action looks and behaves like any other.

When to use it

• Buffer (a Buffer, the default) — small files of known size where holding the whole file in memory is cheap and simple.
• Stream (a Readable) — large files, files of unknown size, or app-to-app transfers (e.g. downloading a large object from one service and uploading it to another) where buffering would risk exhausting memory. </Note>

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and storage-to-storage transfers.

Pieces that support streaming

Streaming is enabled per action. "In" means the action reads its input file as a stream; "out" means it writes the file it produces out as a stream.

More pieces are being enabled over time. Actions not listed here still work — they buffer the file in memory, which is fine within the size limit.

Per-service upload ceilings

Streaming removes the memory ceiling, not the destination API's own limit. Where a service caps what a single request can carry, the action switches to a chunked upload session above that cap:

These are pre-existing API limits rather than streaming limits, and each action now handles its own. One difference matters if your source doesn't report a size: Dropbox's session is offset-based, so it just streams the chunks as they arrive, while Graph wants the file's total length in every fragment's Content-Range header — so the two Microsoft actions buffer once to learn the length before they can chunk. Give those a source that reports Content-Length when you can.

Building streaming actions

If you are building a piece, you can stream on both sides: read an input file as a stream, and write an output file as a stream.

Writing a file as a stream

ctx.files.write accepts a Buffer or a Readable. Pass a Readable — such as an S3 object body or a streaming HTTP response — and it streams straight to storage instead of being buffered. It returns a file reference string you return from the action, exactly like the buffered form (see Files).

async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable — hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}

Reading a file as a stream

Add streaming: true to a Property.File. The property then resolves to an ApStreamingFile instead of an ApFile:

type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};

Consume body directly. How you hand it to the destination depends on what that destination's client accepts — in order of preference:

1. A chunking uploader (best). Accepts a stream of unknown length and buffers each part before sending it, so no content length is needed and parts are individually replayable. For S3 that is Upload from @aws-sdk/lib-storage; for Azure Blob Storage, blockBlobClient.uploadStream.

props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { file } = context.propsValue;
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: file.body,
    },
  }).done();
}

2. An SDK that takes a stream directly. Some clients accept a Readable as-is — Google Drive's media.body, SFTP's client.put. Just pass file.body.

3. A single-request HTTP upload. If the destination is a plain PUT/POST that needs an explicit Content-Length, you have to use file.size — and size is best-effort, so this path needs a buffered fallback for when it is missing. Dropbox, SharePoint and OneDrive all look like this:

import { buffer as readableToBuffer } from 'node:stream/consumers';

const headers: Record<string, string> = { 'Content-Type': 'application/octet-stream' };
let body;
if (file.size != null) {
  headers['Content-Length'] = String(file.size);
  body = file.body;
} else {
  body = await readableToBuffer(file.body);
}

size is informational and best-effort — it is undefined when the source reports no Content-Length, and it is also dropped when the response is compressed (Content-Encoding: gzip/br/deflate), because the decompressed body no longer matches the advertised length. Don't require it: prefer pattern 1 or 2, which never need it.

Limits & storage

• Writing into storage is capped. A stream passed to ctx.files.write is counted against AP_MAX_FILE_SIZE_MB (Cloud: 10 MB, self-hosted default: 25 MB) while the bytes flow; exceeding it aborts the transfer and fails the step. See Limits.
• Reading a streamed input is not capped. A streaming: true file input has no AP_MAX_FILE_SIZE_MB ceiling — that is deliberate, since the point of the feature is to move files larger than the cap out to an external service.
• Storage backend. Streaming end-to-end requires S3 file storage — see the callout at the top of this page.