# How to Migrate from Amazon S3 to Fastio API

Migrating from Amazon S3 to Fastio API replaces raw object storage with intelligent workspaces for agentic teams. Fastio unifies file storage, semantic search, RAG chat, and MCP tools in one API, eliminating separate vector databases and relational layers. This guide delivers a 5-step zero-downtime plan with scripts for syncing petabyte-scale buckets, code examples, and verification steps.


Source: https://fast.io/resources/migrating-amazon-s3-fastio-api/
Last reviewed: 2026-02-24

## Why Migrate from Amazon S3 to Fastio

Amazon S3 handles object storage well for basic needs. Developers often pair it with vector databases like Pinecone for RAG, relational DBs for metadata, and custom sharing logic. This splits your stack across services, increases costs, and complicates agent workflows.

Fastio provides object-like storage through workspaces but adds native intelligence. Upload files and they auto-index for semantic search and RAG chat. Agents access a consolidated MCP toolset for full CRUD, versioning, and ownership transfer.

Benefits include scalable cloud storage, no infrastructure management, and unified APIs for storage, AI, and collaboration. For agentic teams, this streamlines architecture without custom integrations.

## Fastio vs Amazon S3 Comparison

| Feature | Amazon S3 | Fastio API |
|---------|-----------|-------------|
| Storage Model | Buckets/objects | Workspaces/files/folders |
| Max File Size | Unlimited | 1GB per upload |
| Pricing | $0.023/GB/mo storage | Starter $29/mo, 14-day trial |
| AI/RAG | None (add Pinecone) | Built-in semantic search + RAG |
| Agent Tools | Custom boto3 | Consolidated toolset via HTTP/SSE |
| Sharing | Presigned URLs | Branded portals, passwords, expiration |
| Collaboration | None | Real-time presence, comments, shares |
| Ownership Transfer | N/A | Agent-to-human handoff |

Fastio suits agentic workflows where S3 feels like commodity storage. Use S3 for pure archival; Fastio for active team/agent use.

### Cost Breakdown

S3 charges for storage, API calls, and data egress. Fastio offers predictable usage-based plans starting at $29/mo and a 14-day Business Trial requiring a credit card.

## Migration Prerequisites

Review your S3 bucket size, object count, and access patterns. Ensure files <1GB or plan splitting. Get AWS credentials with read access. Sign up for a Fastio account and start a 14-day Business Trial requiring a credit card.

Install tools:
- AWS CLI v2
- Python 3.10+ with boto3, requests
- jq for JSON processing

Test Fastio API access with curl after auth.

## Step 1: Set Up Fastio Organization and Workspace

Create agent account via API:

```bash
curl -X POST https://api.fast.io/current/user/ \\
  -u email:password \\
  -d name="Migration Admin"
```

Create org and workspace:

```bash
ORG_ID=$(curl -s -X POST https://api.fast.io/current/org/ \\
  -H "Authorization: Bearer $JWT" \\
  -d "domain=my-migration-org" -d "name=My Org" | jq -r .response.id)

WS_ID=$(curl -s -X POST "https://api.fast.io/current/org/$ORG_ID/workspace/" \\
  -H "Authorization: Bearer $JWT" \\
  -d "folder_name=migrated-bucket" -d "name=Migrated Bucket" | jq -r .response.id)
```

Enable intelligence for auto-indexing.

## Step 2: Inventory S3 Bucket

List objects and metadata:

```bash
aws s3 ls s3://your-bucket --recursive --summarize \\
  --page-size 1000 > s3-inventory.json
```

Parse for total objects, size, last modified. Identify large files >1GB to split or skip.

## Step 3: Zero-Downtime Data Transfer

Use presigned S3 URLs + Fastio web-import for server-side copy (no local bandwidth).

Python script (migrate_s3.py):

```python
import boto3
import requests
import json
import time
from concurrent.futures import ThreadPoolExecutor

s3 = boto3.client('s3')
FASTIO_BASE = 'https://api.fast.io/current'
headers = {'Authorization': f'Bearer {jwt_token}'}
ws_id = 'your_ws_id'
parent_id = 'root'

def presign_and_import(key):
    presign_url = s3.generate_presigned_url('get_object',
        Params={'Bucket': 'your-bucket', 'Key': key},
        ExpiresIn=3600)
    resp = requests.post(f'{FASTIO_BASE}/upload/web-import/',
        headers=headers,
        data={'profile_type': 'workspace', 'profile_id': ws_id,
              'parent_id': parent_id, 'url': presign_url})
    if resp.json().get('result'):
        print(f'Migrated {key}')
    else:
        print(f'Failed {key}: {resp.text}')

# List keys
paginator = s3.get_paginator('list_objects_v2')
keys = []
for page in paginator.paginate(Bucket='your-bucket'):
    keys.extend(obj['Key'] for obj in page.get('Contents', []))

# Batch import (parallel 10)
with ThreadPoolExecutor(max_workers=10) as executor:
    executor.map(presign_and_import, keys)

# Poll import status if needed
```

Run in batches for PB scale. Monitor with /upload/web-list/. Incremental: import only modified since last sync.

## Step 4: Update Code from S3 SDK to Fastio API

Replace boto3 put_object with Fastio upload.

S3 example:
```python
s3.put_object(Bucket='bucket', Key='file.txt', Body=data)
```

Fastio:
```python
# Init upload
init_resp = requests.post(f'{FASTIO_BASE}/upload/init/',
    headers=headers,
    data={'profile_type': 'workspace', 'profile_id': ws_id,
          'parent_id': 'root', 'name': 'file.txt', 'size': len(data)})
upload_id = init_resp.json()['response']['id']

# Upload chunk (parallel for large)
requests.put(init_resp.json()['response']['upload_url'], data=data)

# Complete
requests.post(f'{FASTIO_BASE}/upload/{upload_id}/complete/', headers=headers)
```

Use MCP for agents.

## Step 5: Verify and Cut Over

Compare object count/size S3 vs Fastio storage list.

Test reads from Fastio preview URLs.

Update app config to point to Fastio endpoints.

Monitor activity logs. Go live with dual-write during transition.

## Petabyte-Scale Tips

- Parallelize with 100+ threads
- Use S3 inventory reports for delta sync
- Split >1GB files with ffmpeg or zip
- Throttle requests (Fastio rate limits)
- Dual-read until confident

## Frequently asked questions

### Is Fastio API compatible with S3?

No direct S3 compatibility, but object operations map closely. Use scripts to sync data and update SDK calls to Fastio REST API.

### How do I move data from AWS S3 to Fastio?

Generate presigned S3 URLs and use Fastio /upload/web-import endpoint. Python script above handles batching and parallelism.

### What plans does Fastio offer for migration?

Fast.io offers Starter ($29/mo for 1 TB), Business ($99/mo for 10 TB), and Growth ($299/mo for 50 TB), plus a 14-day Business Trial requiring a credit card. See /pricing/.

### Does Fastio support S3 multipart uploads?

Chunked uploads up to 1GB with init/upload/complete flow.

### Can agents automate the migration?

Yes, use MCP server with auth/upload/storage tools.

### How can agents automate file lifecycle in Fastio?

Agents can poll the activity feed or monitor WebSocket events, triggering file management operations directly via the REST API or MCP server.

## About Fast.io

Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server at https://mcp.fast.io/mcp, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli.