About half the price of S3 Standard · Audit-ready integrity reports · Encrypted off-site copies
Storage built for AI training datasets.
Keep training data for about half the price of S3 Standard and spend the difference on compute. Upload terabytes with resumable transfers, freeze read-only versions for reproducible runs, and hand auditors an integrity report for any dataset — with every file encrypted and replicated off-site.
$ pip install datasetsync$ export DATASETSYNC_API_KEY=dsk_live_...$ datasetsync sync ./shards ds://my-dataset/train/uploaded 1024, skipped 0 identical# Interrupted? Re-run it: identical files are skipped$ datasetsync sync ./shards ds://my-dataset/train/uploaded 3, skipped 1021 identical$ datasetsync versions create my-dataset v1Created version v1 (1024 files)Why teams switch
Lower storage bills, audit-ready data, fewer ways to lose it
Training data is written once and read in bursts. Paying hot-storage prices to keep it is money that could buy GPUs.
Spend on GPUs, not storage
Storage from $0.012 per GB-month, about half of Amazon S3 Standard's $0.023¹, with generous download allowances included.
- Pro: 4 TB and 10 TB of downloads for $49 a month
- Large downloads with the desktop app don't count as egress
- Invoices in USD, paid by card or, on Enterprise, by invoice
Audit-ready data provenance
Show exactly which data a model was trained on, and that it hasn't changed since, without engineering work.
- SHA-256 checksum on every file, verified end to end
- Read-only versions pin each training run to its data
- One-click integrity report for any dataset or version
- Archived copies are held by independent providers that must prove every day they still hold them
Resilient by design
Archived data lives off-site, encrypted, with independent providers — outside the network you run day to day.
- Versions can't be modified; files a version uses survive deletes in the live dataset
- Scoped API keys and roles limit who can delete anything
- Every deletion, key change and share is in the audit log
¹ Amazon S3 Standard in US East (N. Virginia), first 50 TB per month, list price as of September 2026, excluding request fees. Our rate is the Pro and Scale price for storage beyond the plan's included amount.
How it works
From local disk to data loader
DatasetSync keeps a fast copy for training and a verifiable copy for durability, with one API over both.
- Step 1
Upload
Large files go straight to our backend file servers with the desktop app or SDK, encrypted on your machine and resumable. Small files upload from the browser too.
- Step 2
Hot NVMe tier
Files are readable the moment an upload completes. Range requests, ETags and conditional GETs for random access.
- Step 3
Durable archival
Data is split into segments, encrypted and archived to replicated cold storage, independent of the hot tier.
- Step 4
Stream to training
Presigned URLs, manifests and tar streaming feed PyTorch data loaders, aria2 or any HTTP client directly.
One namespace. Hot and archived files share the same paths, versions and URLs.
Verifiable. SHA-256 on every file and part; archival status exposed per segment.
Lifecycle you can see. Processing → Hot → Archiving → Archived, per file.
Features
Everything a training data pipeline needs
Built for datasets measured in terabytes and files measured in millions.
Resumable multi-TB uploads
The desktop app and SDK send large files in parallel, encrypted segments directly to our backend file servers. Interrupted uploads pick up where they left off.
Versioned, immutable datasets
Snapshot the file tree into named versions and read any file, listing or manifest with ?version= for reproducible runs.
S3-compatible API
Point aws-cli, boto3 or rclone at the gateway. Datasets are buckets; multipart, Range and batch delete included.
Bulk download for training
Presigned manifests in JSON, plain text or aria2 format, plus streaming tar archives for curl | tar x.
Share links
Scoped, expiring links to a dataset, version or path prefix with optional download limits. Revoke any time.
Team roles & scoped keys
Owner, admin, member and viewer roles. API keys carry explicit scopes and can be restricted to specific datasets.
Audit log
Every sign-in, key change, deletion and share is recorded with actor, IP and timestamp.
Webhooks
Signed HMAC-SHA256 events for uploads, versions, deletions and archival so pipelines react automatically.
Transparent quotas & usage
Daily storage, ingress and egress, hot vs. archived bytes and per-file archival status, visible in the dashboard and API.
Developers
Use the tools you already have
A plain REST API, a Python SDK and CLI, and an S3-compatible gateway. No proprietary client required.
- Checksums verified server-side with
X-Content-SHA256 - Range reads for random access into large shards
- Presigned manifests for
aria2cand PyTorch
# Upload a small file (≤ 64 MB) with an end-to-end checksumcurl -X PUT https://datasetsync.com/api/v1/datasets/my-dataset/files/train/shard-00001.parquet \ -H "Authorization: Bearer $DATASETSYNC_API_KEY" \ -H "X-Content-SHA256: $(sha256sum shard-00001.parquet | cut -d' ' -f1)" \ --data-binary @shard-00001.parquet# Read the first 1 MiB (206 Partial Content)curl https://datasetsync.com/api/v1/datasets/my-dataset/files/train/shard-00001.parquet \ -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Range: bytes=0-1048575" -o head.binPricing
Simple, usage-based pricing
Start free. Pay for what you store beyond your plan, with archival included on every tier.
Free
$0/ month
- Storage
- 5 GB cap
- Egress / month
- 10 GB
Large files (64 MB+) downloaded with the desktop app don't count as egress.
Pro
Popular$49/ month
- Storage
- 4 TB
- Egress / month
- 10 TB
Large files (64 MB+) downloaded with the desktop app don't count as egress.
Scale
$499/ month
- Storage
- 40 TB
- Egress / month
- 200 TB
Large files (64 MB+) downloaded with the desktop app don't count as egress.
Enterprise
Custom
- Storage
- Custom
- Egress / month
- Custom
Large files (64 MB+) downloaded with the desktop app don't count as egress.
FAQ
Frequently asked questions
How durable is my data?
Large files uploaded with the desktop app or SDK are encrypted on your machine and sent straight to durable, replicated storage held by independent storage providers. Small files land on the hot tier first and are then encrypted and archived the same way. Every file and upload part is checksummed with SHA-256 end to end.
A file is marked Archived only once every one of its segments is safely stored in the archival tier.
What's the difference between hot and archived retrieval?
Files on the hot tier are served immediately, with Range requests. Archived files that have been evicted from the hot tier can be restored with POST /api/v1/files/{fileId}/restore; they re-hydrate from the archival tier and are then served from the hot tier again. The API reports each file's status and whether a hot copy exists.
How is egress billed?
Each plan includes a monthly egress allowance covering downloads, presigned URLs, share links and S3 GETs. Paid plans bill usage beyond the allowance per GB; the Free plan has a fixed allowance. Uploads (ingress) are free. See pricing for details.
Is it really S3-compatible?
The gateway at https://datasetsync.com/s3 speaks SigV4 with path-style addressing and supports the operations training pipelines rely on: listing, Get/Head (with Range), Put, Copy, Delete, batch delete and the full multipart upload API. Bucket management maps to datasets. See the S3 gateway docs for the exact list.
What's the maximum file size?
A single PUT accepts up to 5 GiB. Multipart uploads accept up to 10,000 parts of 5 MiB to 5 GiB each, so a single file can be many terabytes.
What happens when I delete data?
Deletion removes the file from the hot tier and your account immediately, and stops renewal of its archival storage. Copies held by third-party archival storage providers may persist until their existing storage contracts expire, so encrypt sensitive data client-side before uploading. Details are in our privacy policy.
How is my data secured?
All traffic uses TLS. API keys are scoped, can be limited to specific datasets and are stored hashed. Accounts support TOTP two-factor authentication, and every sensitive action is written to your organization's audit log. For confidential datasets we recommend client-side encryption before upload.
Put your next dataset on DatasetSync
Durable storage for AI training data.