> ## Documentation Index
> Fetch the complete documentation index at: https://www.activepieces.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Large File Streaming

> Move large files through a flow without buffering them in memory

Large file streaming lets a piece process a big file by passing it through as a stream,
transferring the bytes a chunk at a time, instead of loading the whole file into memory
first. It is useful when a file is too large to hold in RAM at either the source or the
destination. Because the file is never fully buffered, a flow can move files far larger
than the worker's memory budget.

## Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB,
minus overhead, see [Limits](https://www.activepieces.com/docs/install/reference/limits.md)). Reading a large file into a
`Buffer` holds the entire file in that budget at once, so a big enough file exhausts the
memory and the worker is OOM-killed.

Streaming avoids this: the bytes flow through as a Node `Readable`, roughly 5 MB at a
time, so no process ever holds the whole file. The transfer is transparent: there are no
extra "chunk" steps in your flow; a streaming action looks and behaves like any other.

> [!WARNING]
>
> **Reading a *stored* file as a stream requires S3 file storage.** Moving bytes from one
> third-party service to another — the common case — never touches Activepieces storage and
> works on any backend. But when the file being read already lives in Activepieces storage,
> only S3 (`AP_FILE_STORAGE_LOCATION=S3`) serves it incrementally: with the default database
> (`DB`) storage the bytes cannot be read out of a column a chunk at a time, so the server
> loads the whole row into memory before sending it to your action, which defeats the memory
> savings. Self-hosted installs default to `DB`; set S3 to get the benefit. See
> [Set up S3](https://www.activepieces.com/docs/install/configure-operate/setup-s3.md).

## When to use it

> [!NOTE]
>
> **Which approach should you use?**
>
> * **Buffer** (a `Buffer`, the default): small files of known size where holding the whole
>   file in memory is cheap and simple.
> * **Stream** (a `Readable`): large files, files of unknown size, or app-to-app transfers
>   (e.g. downloading a large object from one service and uploading it to another) where
>   buffering would risk exhausting memory.

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and
storage-to-storage transfers.

## Pieces that support streaming

Streaming is enabled per action. "In" means the action reads its **input** file as a stream;
"out" means it writes the file it produces **out** as a stream.

| Piece | Streams in | Streams out |
| - | - | - |
| Amazon S3 | Upload File | Read File |
| Azure Blob Storage | Create Blob | Read Blob |
| Dropbox | Upload file | Download File |
| Google Drive | Upload file | Read File Content, List files, Set public access, New File (trigger) |
| Microsoft OneDrive | Upload file | Get File |
| Microsoft SharePoint | Upload File | N/A |
| FTP/SFTP | Upload File | Read File Content |
| Subflows | Stream CSV to Subflows | N/A |

More pieces are being enabled over time. Actions not listed here still work: they buffer
the file in memory, which is fine within the [size limit](https://www.activepieces.com/docs/install/reference/limits.md).

> [!NOTE]
>
> Stream CSV to Subflows streams a CSV straight from the URL you give it and never writes the
> file into storage, so it is the one entry above that does **not** need S3 file storage.

### Per-service upload ceilings

Streaming removes the *memory* ceiling, not the destination API's own limit. Where a service caps
what a single request can carry, the action switches to a chunked upload session above that cap:

| Piece | One-request cap | Above it |
| - | - | - |
| Dropbox | [150 MB](https://developers.dropbox.com/dbx-performance-guide) (`/2/files/upload` answers `409 payload_too_large`) | `/2/files/upload_session/*`, 8 MiB chunks |
| Microsoft SharePoint | [250 MB](https://learn.microsoft.com/en-us/graph/api/driveitem-put-content) (Graph's simple `PUT …/content`) | Graph upload session, 10 MiB chunks |
| Microsoft OneDrive | 4 MiB | Graph upload session, 10 MiB chunks |

These are pre-existing API limits rather than streaming limits, and each action now handles its own.
One difference matters if your source doesn't report a size: Dropbox's session is offset-based, so it
just streams the chunks as they arrive, while Graph wants the file's total length in every fragment's
`Content-Range` header, so the two Microsoft actions buffer once to learn the length before they can
chunk. Give those a source that reports `Content-Length` when you can.

## Building streaming actions

If you are [building a piece](https://www.activepieces.com/docs/build-pieces/building-pieces/overview.md), you can stream on both sides: read an
input file as a stream, and write an output file as a stream.

### Writing a file as a stream

`ctx.files.write` accepts a `Buffer` **or** a `Readable`. Pass a `Readable`, such as an S3
object body or a streaming HTTP response, and the engine drains it for you, counting the
bytes against `AP_MAX_FILE_SIZE_MB` as they arrive. Your action never builds a `Buffer`
itself, though the write is not streamed end-to-end: that cap sits far below the sandbox's
memory budget, so the upload declares a real `Content-Length` — which is what makes it
retryable. It returns a file reference string you return from the action, exactly like the
buffered form (see [Files](https://www.activepieces.com/docs/build-pieces/piece-reference/files.md)).

```ts theme={null}
async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable: hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}
```

### Reading a file as a stream

Add `streaming: true` to a `Property.File`. On platforms 0.87.0 and later the property
resolves to an `ApStreamingFile`; older engines still deliver a buffered `ApFile`, so the
value is typed as `ApStreamingFile | ApFile`:

```ts theme={null}
type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};
```

Never read `.body` or `.size` off the property directly — normalize it first with
`streamUtils.toStreamingBody` from `@activepieces/pieces-common`. It passes an
`ApStreamingFile` through untouched and wraps an `ApFile`'s buffer into a `Readable` with an
exact `size`:

```ts theme={null}
import { streamUtils } from '@activepieces/pieces-common';

const { body, size } = streamUtils.toStreamingBody(context.propsValue.file);
```

Consume `body` directly. How you hand it to the destination depends on what that destination's
client accepts, in order of preference:

**1. A chunking uploader (best).** Accepts a stream of unknown length and buffers each part
before sending it, so no content length is needed and parts are individually replayable. For S3
that is `Upload` from `@aws-sdk/lib-storage`; for Azure Blob Storage,
`blockBlobClient.uploadStream`.

```ts theme={null}
props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { body } = streamUtils.toStreamingBody(context.propsValue.file);
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: body,
    },
  }).done();
}
```

**2. An SDK that takes a stream directly.** Some clients accept a `Readable` as-is: Google
Drive's `media.body`, SFTP's `client.put`. Just pass `body`.

**3. A single-request HTTP upload.** If the destination is a plain `PUT`/`POST` that needs an
explicit `Content-Length`, you have to use `size`, and `size` is best-effort, so this
path needs a buffered fallback for when it is missing. Dropbox, SharePoint and OneDrive all
look like this:

```ts theme={null}
import { buffer as readableToBuffer } from 'node:stream/consumers';

const { body: streamedBody, size } = streamUtils.toStreamingBody(context.propsValue.file);
const headers: Record<string, string> = { 'Content-Type': 'application/octet-stream' };
let body;
if (size != null) {
  headers['Content-Length'] = String(size);
  body = streamedBody;
} else {
  body = await readableToBuffer(streamedBody);
}
```

`size` is informational and best-effort: it is `undefined` when the source reports no
`Content-Length`, and it is also dropped when the response is compressed
(`Content-Encoding: gzip`/`br`/`deflate`), because the decompressed body no longer matches the
advertised length. Don't require it: prefer pattern 1 or 2, which never need it.

> [!TIP]
>
> `Property.File()` without `streaming` is unchanged: it still resolves to an `ApFile` with a
> `data` buffer, so existing actions keep working.

## Limits & storage

* **Writing into storage is capped, and buffered by the engine.** A stream passed to
  `ctx.files.write` is counted against `AP_MAX_FILE_SIZE_MB` (Cloud: 10 MB, self-hosted
  default: 25 MB) *while* the bytes flow; crossing it aborts the transfer and fails the
  step before anything is uploaded. See [Limits](https://www.activepieces.com/docs/install/reference/limits.md).
* **Reading a streamed input is not capped.** A `streaming: true` file input has no
  `AP_MAX_FILE_SIZE_MB` ceiling; that is deliberate, since the point of the feature is to move
  files larger than the cap out to an external service.
* **Storage backend.** Streaming a file *out of* Activepieces storage requires S3; see the
  callout at the top of this page. Writes are capped and engine-buffered on every backend.

> [!WARNING]
>
> * **Individual parts retry, the whole transfer doesn't.** A multipart uploader buffers each
>   \~5 MB part before sending it, so it can replay *that part* on a transient error. But the
>   source `Readable` can be read only once, so there is no retry of the transfer as a whole,
>   and an action that fails after its stream is drained cannot be re-run against the same
>   stream — it has to fetch the source again.
> * **`httpClient` does not retry a stream body at all.** The retry loop in `pieces-common`
>   reuses the body it serialized before the first attempt, and a stream is one-shot, so retrying
>   would replay a drained stream and send a truncated body. So a request whose body is a
>   `Readable` (or a `form-data` payload, which is streamed too) runs its `retries` setting as
>   `0`. This applies to **any** piece sending a stream through `httpClient`, not just file
>   actions. If you need real retries, buffer the body instead.
> * **Multipart webhook signatures aren't verified.** Verifying an HMAC signature over an
>   uploaded file needs the raw bytes held in memory, which is exactly what streaming avoids.
>   Streamed multipart webhook uploads therefore skip signature verification. JSON, XML, form,
>   and text webhook bodies are unaffected.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.