Large File Streaming

Large file streaming lets a piece process a big file by passing it through as a stream, transferring the bytes a chunk at a time, instead of loading the whole file into memory first. It is useful when a file is too large to hold in RAM at either the source or the destination. Because the file is never fully buffered, a flow can move files far larger than the worker's memory budget.

Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB, minus overhead, see Limits). Reading a large file into a Buffer holds the entire file in that budget at once, so a big enough file exhausts the memory and the worker is OOM-killed.

Streaming avoids this: the bytes flow through as a Node Readable, roughly 5 MB at a time, so no process ever holds the whole file. The transfer is transparent: there are no extra "chunk" steps in your flow; a streaming action looks and behaves like any other.

When to use it

• Buffer (a Buffer, the default): small files of known size where holding the whole file in memory is cheap and simple.
• Stream (a Readable): large files, files of unknown size, or app-to-app transfers (e.g. downloading a large object from one service and uploading it to another) where buffering would risk exhausting memory. </Note>

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and storage-to-storage transfers.

Pieces that support streaming

Streaming is enabled per action. "In" means the action reads its input file as a stream; "out" means it writes the file it produces out as a stream.

More pieces are being enabled over time. Actions not listed here still work: they buffer the file in memory, which is fine within the size limit.

Per-service upload ceilings

Streaming removes the memory ceiling, not the destination API's own limit. Where a service caps what a single request can carry, the action switches to a chunked upload session above that cap:

These are pre-existing API limits rather than streaming limits, and each action now handles its own. One difference matters if your source doesn't report a size: Dropbox's session is offset-based, so it just streams the chunks as they arrive, while Graph wants the file's total length in every fragment's Content-Range header, so the two Microsoft actions buffer once to learn the length before they can chunk. Give those a source that reports Content-Length when you can.

Building streaming actions

If you are building a piece, you can stream on both sides: read an input file as a stream, and write an output file as a stream.

Writing a file as a stream

ctx.files.write accepts a Buffer or a Readable. Pass a Readable, such as an S3 object body or a streaming HTTP response, and the engine drains it for you, counting the bytes against AP_MAX_FILE_SIZE_MB as they arrive. Your action never builds a Buffer itself, though the write is not streamed end-to-end: that cap sits far below the sandbox's memory budget, so the upload declares a real Content-Length — which is what makes it retryable. It returns a file reference string you return from the action, exactly like the buffered form (see Files).

async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable: hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}

Reading a file as a stream

Add streaming: true to a Property.File. On platforms 0.87.0 and later the property resolves to an ApStreamingFile; older engines still deliver a buffered ApFile, so the value is typed as ApStreamingFile | ApFile:

type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};

Never read .body or .size off the property directly — normalize it first with streamUtils.toStreamingBody from @activepieces/pieces-common. It passes an ApStreamingFile through untouched and wraps an ApFile's buffer into a Readable with an exact size:

import { streamUtils } from '@activepieces/pieces-common';

const { body, size } = streamUtils.toStreamingBody(context.propsValue.file);

Consume body directly. How you hand it to the destination depends on what that destination's client accepts, in order of preference:

1. A chunking uploader (best). Accepts a stream of unknown length and buffers each part before sending it, so no content length is needed and parts are individually replayable. For S3 that is Upload from @aws-sdk/lib-storage; for Azure Blob Storage, blockBlobClient.uploadStream.

props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { body } = streamUtils.toStreamingBody(context.propsValue.file);
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: body,
    },
  }).done();
}

2. An SDK that takes a stream directly. Some clients accept a Readable as-is: Google Drive's media.body, SFTP's client.put. Just pass body.

3. A single-request HTTP upload. If the destination is a plain PUT/POST that needs an explicit Content-Length, you have to use size, and size is best-effort, so this path needs a buffered fallback for when it is missing. Dropbox, SharePoint and OneDrive all look like this:

import { buffer as readableToBuffer } from 'node:stream/consumers';

const { body: streamedBody, size } = streamUtils.toStreamingBody(context.propsValue.file);
const headers: Record<string, string> = { 'Content-Type': 'application/octet-stream' };
let body;
if (size != null) {
  headers['Content-Length'] = String(size);
  body = streamedBody;
} else {
  body = await readableToBuffer(streamedBody);
}

size is informational and best-effort: it is undefined when the source reports no Content-Length, and it is also dropped when the response is compressed (Content-Encoding: gzip/br/deflate), because the decompressed body no longer matches the advertised length. Don't require it: prefer pattern 1 or 2, which never need it.

Limits & storage

• Writing into storage is capped, and buffered by the engine. A stream passed to ctx.files.write is counted against AP_MAX_FILE_SIZE_MB (Cloud: 10 MB, self-hosted default: 25 MB) while the bytes flow; crossing it aborts the transfer and fails the step before anything is uploaded. See Limits.
• Reading a streamed input is not capped. A streaming: true file input has no AP_MAX_FILE_SIZE_MB ceiling; that is deliberate, since the point of the feature is to move files larger than the cap out to an external service.
• Storage backend. Streaming a file out of Activepieces storage requires S3; see the callout at the top of this page. Writes are capped and engine-buffered on every backend.