Pattern: File Upload

17. Pattern: File Upload

The File Upload pattern governs how binary objects, large documents, and media assets are transferred into a RESTful system safely, efficiently, and scalability.

Instead of forcing binary files into standard JSON payloads, APIs use dedicated transport mechanisms—ranging from simple multipart form requests to decoupled, presigned cloud storage URLs.

17.1. Overview

Handling file uploads in REST requires selecting the right upload strategy based on file size, client environment, network reliability, and backend architecture.

The four primary approaches for API file uploads are:

1. Multipart Requests (multipart/form-data): Uploading metadata and binary content together in a single HTTP request.
2. Raw Binary Uploads (application/octet-stream): Streaming binary data directly in the HTTP body with metadata passed via headers or path variables.
3. Presigned Storage URLs: Decoupling the upload flow by granting clients direct, short-lived write access to object storage (e.g., AWS S3, Google Cloud Storage).
4. Chunked / Resumable Uploads: Breaking large files into distinct chunks sent across a session to allow pausing, resuming, and fault recovery.

17.2. When to Use File Upload Patterns

Use File Upload patterns when:

• Your API accepts non-textual assets such as images, PDFs, audio/video streams, or archives.
• You are processing large datasets (e.g., CSV or Parquet files) that exceed typical JSON request body limits.
• You need to offload I/O bottleneck risks from application servers by uploading binary files directly to object storage.
• Network instability requires upload resilience, making chunked and resumable upload mechanics necessary.

17.3. When NOT to Use File Upload Patterns

Avoid File Upload patterns when:

• The asset is already accessible on the web. Pass a URL reference ({ "sourceUrl": "https://..." }) and let the backend fetch it asynchronously instead of requiring the client to re-upload bytes.
• The data is structured JSON/XML. Structured tabular data should generally be submitted as JSON unless payload size mandates bulk file submission.

17.4. What the Pattern Looks Like

The strategy you choose dictates the HTTP request layout and resource lifecycle. Below are HTTP request and response examples for each acceptable upload strategy.

Approach 1: Direct Multipart (multipart/form-data)

Combines binary data and structured metadata in a single payload divided by boundary strings.

• Best for: Small-to-medium files (e.g., profile photos, small attachments) from web forms.

Request

POST /documents/multipart HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOi...
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Length: 5432

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="category"

invoices
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="invoice_2026.pdf"
Content-Type: application/pdf

%PDF-1.4 ... [binary payload bytes] ...
------WebKitFormBoundary7MA4YWxkTrZu0gW--

Response

HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": "doc_99182",
  "category": "invoices",
  "status": "READY"
}

---

Approach 2: Raw Binary Stream (application/octet-stream or MIME type)

Streams raw bytes directly in the request body. Metadata is provided via headers or path parameters.

• Best for: Single file updates where dedicated endpoints exist and boundary parsing overhead should be avoided.

Request

PUT /users/usr_123/avatar HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOi...
Content-Type: image/png
Content-Length: 204800

[raw binary PNG bytes...]

Response

HTTP/1.1 200 OK
Content-Type: application/json

{
  "userId": "usr_123",
  "avatarUrl": "https://cdn.example.com/avatars/usr_123.png",
  "updatedAt": "2026-07-28T18:00:00Z"
}

---

Approach 3: Presigned Storage URLs (Decoupled Pattern)

A three-step flow that completely bypasses application server bandwidth.

• Best for: Medium-to-large files, serverless architectures, and high-throughput APIs.

Step 1: Request Presigned Upload URL

Request

POST /documents/upload-url HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOi...
Content-Type: application/json

{
  "fileName": "report.pdf",
  "contentType": "application/pdf",
  "sizeBytes": 5242880
}

Response

HTTP/1.1 200 OK
Content-Type: application/json

{
  "documentId": "doc_99182",
  "uploadUrl": "https://storage.example.com/uploads/doc_99182?signature=abc123xyz",
  "expiresAt": "2026-07-28T18:00:00Z"
}

Step 2: Upload Directly to Object Storage

Request

PUT /uploads/doc_99182?signature=abc123xyz HTTP/1.1
Host: storage.example.com
Content-Type: application/pdf
Content-Length: 5242880

[binary PDF bytes...]

Response

HTTP/1.1 200 OK

Step 3: Confirm Upload Completion

Request

POST /documents/doc_99182/complete HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOi...
Content-Type: application/json

{
  "uploadStatus": "SUCCESS"
}

Response

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "documentId": "doc_99182",
  "status": "PROCESSING"
}

---

Approach 4: Chunked / Resumable Uploads

A session-driven model using chunked transfers.

• Best for: Multi-gigabyte files, video processing pipelines, and uploads over shaky connections.

Step 1: Create Upload Session

Request

POST /upload-sessions HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOi...
Content-Type: application/json

{
  "fileName": "video_recording.mp4",
  "totalBytes": 104857600
}

Response

HTTP/1.1 201 Created
Content-Type: application/json

{
  "sessionId": "sess_88319",
  "chunkSizeBytes": 52428800,
  "nextOffset": 0
}

Step 2: Upload Chunk

Request

PATCH /upload-sessions/sess_88319 HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOi...
Content-Type: application/offset+octet-stream
Upload-Offset: 0
Content-Length: 52428800

[50MB binary chunk bytes...]

Response

HTTP/1.1 200 OK
Upload-Offset: 52428800
Content-Type: application/json

{
  "sessionId": "sess_88319",
  "nextOffset": 52428800,
  "bytesReceived": 52428800,
  "status": "IN_PROGRESS"
}

17.5. Anti-Patterns to Avoid

1. Base64-encoding binary files in JSON payloads

{
  "fileName": "avatar.png",
  "fileData": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
}

• Why it’s bad: Base64 inflates payload size by ~33%, drastically increases memory usage during JSON serialization/deserialization, spikes CPU overhead, and frequently hits HTTP body limits. Avoid Base64 encoding binary data unless handling tiny inline thumbnails (under 50 KB).

2. Proxying large binary streams through application servers

• Routing gigabytes of binary data through application web workers saturates network bandwidth and exhausts server memory threads. Use Presigned Storage URLs to let clients upload directly to cloud storage.

3. Synchronous file processing during the upload HTTP request

• Executing virus scanning, video transcoding, thumbnail generation, or parsing synchronously before responding causes request timeouts. Accept the upload, save the file, return a 202 Accepted or 201 Created status, and process asynchronously using background workers.

4. Missing early header and payload size validation

• Accepting binary chunks without first checking Content-Length or Content-Type headers allows malicious or oversized files to consume backend resources before being rejected. Validate payload constraints before reading body bytes.

5. Saving uploaded files directly to public static web directories

• Writing uploaded assets into publicly accessible web server paths opens critical security vulnerabilities (such as arbitrary file execution). Store uploads in isolated, non-executable storage buckets with secure access controls.

17.6. OpenAPI Examples

Below are distinct OpenAPI 3.0.3 paths illustrating each acceptable upload strategy individually.

1. Direct Multipart Upload (POST /documents/multipart)

paths:
  /documents/multipart:
    post:
      summary: Direct multipart file upload
      description: Uploads document metadata and binary payload in a single multipart request.
      tags: [Multipart Upload]
      requestBody:
        required: true
        content:
          multipart/form-data:
            schema:
              type: object
              required:
                - file
                - category
              properties:
                category:
                  type: string
                  example: "invoices"
                file:
                  type: string
                  format: binary
                  description: The document file binary stream.
      responses:
        '201':
          description: File uploaded successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Document'

2. Raw Binary Stream Upload (PUT /users/{userId}/avatar)

paths:
  /users/{userId}/avatar:
    put:
      summary: Upload user avatar as raw binary stream
      description: Replaces the user's avatar image using raw octet-stream bytes.
      tags: [Raw Binary Upload]
      parameters:
        - in: path
          name: userId
          required: true
          schema:
            type: string
            example: usr_123
      requestBody:
        required: true
        description: Raw PNG or JPEG image bytes
        content:
          image/png:
            schema:
              type: string
              format: binary
          image/jpeg:
            schema:
              type: string
              format: binary
      responses:
        '200':
          description: Avatar updated successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/AvatarResponse'

3. Presigned Storage URL Upload (POST /documents/upload-url & POST /documents/{documentId}/complete)

paths:
  /documents/upload-url:
    post:
      summary: Request a presigned URL for direct upload
      description: Generates a short-lived presigned URL to upload binary files directly to cloud storage.
      tags: [Presigned Upload]
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/UploadUrlRequest'
      responses:
        '200':
          description: Presigned URL generated successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/UploadUrlResponse'

  /documents/{documentId}/complete:
    post:
      summary: Confirm presigned upload completion
      description: Notifies the backend that the file was successfully uploaded to cloud storage.
      tags: [Presigned Upload]
      parameters:
        - in: path
          name: documentId
          required: true
          schema:
            type: string
            example: doc_99182
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - uploadStatus
              properties:
                uploadStatus:
                  type: string
                  enum: [SUCCESS, FAILED]
                  example: SUCCESS
      responses:
        '202':
          description: Upload confirmed; processing initiated
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Document'

4. Resumable / Chunked Upload (POST /upload-sessions & PATCH /upload-sessions/{sessionId})

paths:
  /upload-sessions:
    post:
      summary: Initiate a chunked upload session
      description: Creates a new session for uploading large files in sequential chunks.
      tags: [Chunked Upload]
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateSessionRequest'
      responses:
        '201':
          description: Session created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/UploadSession'

  /upload-sessions/{sessionId}:
    patch:
      summary: Upload a binary chunk to an active session
      description: Appends binary bytes at a specified offset.
      tags: [Chunked Upload]
      parameters:
        - in: path
          name: sessionId
          required: true
          schema:
            type: string
            example: sess_88319
        - in: header
          name: Upload-Offset
          required: true
          schema:
            type: integer
            example: 0
      requestBody:
        required: true
        content:
          application/offset+octet-stream:
            schema:
              type: string
              format: binary
      responses:
        '200':
          description: Chunk received
          headers:
            Upload-Offset:
              schema:
                type: integer
                example: 52428800
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/UploadSession'

Shared Components and Schemas

components:
  schemas:
    Document:
      type: object
      properties:
        id:
          type: string
          example: doc_99182
        category:
          type: string
          example: invoices
        status:
          type: string
          example: PROCESSING

    AvatarResponse:
      type: object
      properties:
        userId:
          type: string
          example: usr_123
        avatarUrl:
          type: string
          format: uri
          example: "https://cdn.example.com/avatars/usr_123.png"
        updatedAt:
          type: string
          format: date-time

    UploadUrlRequest:
      type: object
      required:
        - fileName
        - contentType
        - sizeBytes
      properties:
        fileName:
          type: string
          example: report.pdf
        contentType:
          type: string
          example: application/pdf
        sizeBytes:
          type: integer
          example: 5242880

    UploadUrlResponse:
      type: object
      properties:
        documentId:
          type: string
          example: doc_99182
        uploadUrl:
          type: string
          format: uri
          example: "https://storage.example.com/uploads/doc_99182?signature=abc123xyz"
        expiresAt:
          type: string
          format: date-time

    CreateSessionRequest:
      type: object
      required:
        - fileName
        - totalBytes
      properties:
        fileName:
          type: string
          example: video_recording.mp4
        totalBytes:
          type: integer
          example: 104857600

    UploadSession:
      type: object
      properties:
        sessionId:
          type: string
          example: sess_88319
        chunkSizeBytes:
          type: integer
          example: 52428800
        nextOffset:
          type: integer
          example: 52428800
        bytesReceived:
          type: integer
          example: 52428800
        status:
          type: string
          example: IN_PROGRESS

17.7. Visualizing File Upload Patterns (Mermaid Diagram)

sequenceDiagram
    autonumber
    actor Client
    participant API as API Server
    participant Storage as Object Storage
    participant Worker as Async Worker

    rect rgb(240, 248, 255)
    Note over Client,API: Approach 1: Direct Multipart Upload
    Client->>API: POST /documents/multipart (form-data: metadata + file)
    API->>API: Validate & parse multipart boundary
    API->>Storage: Store binary payload
    API-->>Client: 201 Created { id: "doc_99182", status: "READY" }
    end

    rect rgb(255, 245, 238)
    Note over Client,API: Approach 2: Raw Binary Stream Upload
    Client->>API: PUT /users/usr_123/avatar (Content-Type: image/png, raw body)
    API->>Storage: Stream raw bytes directly to storage
    API-->>Client: 200 OK { avatarUrl: "https://..." }
    end

    rect rgb(240, 255, 240)
    Note over Client,Worker: Approach 3: Presigned Storage URL Upload
    Client->>API: POST /documents/upload-url { fileName, sizeBytes }
    API-->>Client: 200 OK { documentId, uploadUrl, expiresAt }
    Client->>Storage: PUT binary payload directly to uploadUrl
    Storage-->>Client: 200 OK
    Client->>API: POST /documents/doc_99182/complete
    API->>Worker: Trigger async virus scan / processing
    API-->>Client: 202 Accepted { status: "PROCESSING" }
    end

    rect rgb(255, 250, 205)
    Note over Client,API: Approach 4: Chunked / Resumable Upload
    Client->>API: POST /upload-sessions { fileName, totalBytes }
    API-->>Client: 201 Created { sessionId, chunkSizeBytes, nextOffset: 0 }
    loop For each file chunk
        Client->>API: PATCH /upload-sessions/sess_88319 (Upload-Offset: X, binary chunk)
        API-->>Client: 200 OK { Upload-Offset: Y, nextOffset: Y }
    end
    end