Quickstart
Create an API key in the dashboard (API keys, requires the admin role) with the read and write scopes, then export it:
export DATASETSYNC_API_KEY=dsk_live_...- Create a dataset:
curl -X POST https://datasetsync.com/api/v1/datasets \ -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"name": "ImageNet shards", "slug": "imagenet-shards"}'- Upload a file (up to 5 GiB in a single request):
curl -X PUT https://datasetsync.com/api/v1/datasets/imagenet-shards/files/train/shard-00000.tar \ -H "Authorization: Bearer $DATASETSYNC_API_KEY" --data-binary @shard-00000.tar- List and download it:
curl "https://datasetsync.com/api/v1/datasets/imagenet-shards/files?prefix=train/" -H "Authorization: Bearer $DATASETSYNC_API_KEY"curl -o shard-00000.tar https://datasetsync.com/api/v1/datasets/imagenet-shards/files/train/shard-00000.tar -H "Authorization: Bearer $DATASETSYNC_API_KEY"For large files use multipart uploads or the CLI, which handles chunking, parallelism and resumption for you.
Authentication & API keys
Send an API key as a bearer token on every request. Keys start with dsk_live_; the full secret is shown once when the key is created and only a hash is stored.
Authorization: Bearer dsk_live_...The web app authenticates with a session cookie instead; cookie-authenticated mutating requests must be same-origin.
Scopes
Members have one of four roles: owner (everything, plus billing and deleting the organization), admin (all scopes), member (read/write/delete) and viewer (read).
Dataset-restricted keys
Keys carry explicit scopes and can be limited to specific datasets, which is ideal for training jobs that should only read one corpus. Create them in the dashboard or via the API (admin):
curl -X POST https://datasetsync.com/api/org/api-keys -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"name": "trainer-readonly", "scopes": ["read"], "datasetIds": ["<dataset id>"], "expiresInDays": 90}'# → { "apiKey": { ... }, "secret": "dsk_live_..." } (secret shown once)Datasets
A dataset is a named collection of files inside your organization. Anywhere a path contains {dataset} you can use its id or its slug.
curl -X POST https://datasetsync.com/api/v1/datasets -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" -d '{ "name": "Common Crawl 2026-09", "slug": "cc-2026-09", "description": "Deduplicated WET shards", "license": "CC-BY-4.0", "tags": ["text", "web"], "visibility": "private"}'# → 201 DatasetFields: name (required), slug, description, category, license, tags, visibility (private or public).
Slug rules
If you omit slug it is derived from the name. Slugs follow S3 bucket naming and must match ^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$: 3–63 characters, lowercase letters, digits, dots and hyphens, starting and ending with a letter or digit.
List files with GET /api/v1/datasets/{dataset}/files?prefix=&delimiter=/. With the default delimiter=/ you get one directory level ({ directories, files, nextCursor }); pass an empty delimiter= to list every file under prefix sorted by path. q does a substring search on the path and version lists a snapshot.
Uploading
Single request (≤ 5 GiB)
PUT the raw bytes to the file path. Content-Length is required. Send X-Content-SHA256 (hex) and the server verifies it before committing. An existing file at the same path is overwritten (the previous object is kept if a version references it).
curl -X PUT https://datasetsync.com/api/v1/datasets/my-dataset/files/train/shard-00001.parquet \ -H "Authorization: Bearer $DATASETSYNC_API_KEY" \ -H "Content-Type: application/vnd.apache.parquet" \ -H "X-Content-SHA256: $(sha256sum shard-00001.parquet | cut -d' ' -f1)" \ --data-binary @shard-00001.parquet# → 201 FileResumable multipart uploads
Use multipart for anything larger than a few hundred MB. Parts can be uploaded in parallel, in any order, and retried; an interrupted upload is resumed by listing the parts the server already has.
curl -X POST https://datasetsync.com/api/v1/datasets/my-dataset/uploads -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"path": "train/big.tar", "size": 1099511627776, "sha256": "<optional hex>"}'# → 201 { "id": "<upload id>", "partSize": 110100480, "partCount": 9987, "expiresAt": "...", ... }# export UPLOAD_ID=<id> PART_SIZE=<partSize> from the responseBody: path, size, and optionally partSize, contentType, sha256, metadata. If you omit partSize the server picks one (default 64 MiB, min 5 MiB, max 5 GiB, at most 10,000 parts). If size would exceed your organization's quota the request fails with 402 quota_exceeded.
# Every part is exactly partSize bytes except the last. Re-uploading a part replaces it.split -b $PART_SIZE -d -a 5 big.tar part-for f in part-*; do n=$((10#${f#part-} + 1)) curl -sf -X PUT https://datasetsync.com/api/v1/uploads/$UPLOAD_ID/parts/$n -H "Authorization: Bearer $DATASETSYNC_API_KEY" \ -H "X-Content-SHA256: $(sha256sum "$f" | cut -d' ' -f1)" --data-binary @"$f" &done; wait# each → { "partNumber", "sizeBytes", "sha256", "etag" }curl -X POST https://datasetsync.com/api/v1/uploads/$UPLOAD_ID/complete -H "Authorization: Bearer $DATASETSYNC_API_KEY"# → 201 File# → 400 incomplete_upload with details.missingParts if any part is absentResume or abort
# Parts already received (upload any that are missing, then complete)curl https://datasetsync.com/api/v1/uploads/$UPLOAD_ID -H "Authorization: Bearer $DATASETSYNC_API_KEY" # → Upload with parts: [{ partNumber, sizeBytes, sha256 }]# Open uploads for a datasetcurl "https://datasetsync.com/api/v1/uploads?dataset=my-dataset&status=ACTIVE" -H "Authorization: Bearer $DATASETSYNC_API_KEY"# Abort and discard received partscurl -X DELETE https://datasetsync.com/api/v1/uploads/$UPLOAD_ID -H "Authorization: Bearer $DATASETSYNC_API_KEY"Uploads expire after 7 days of inactivity and are cleaned up.
Recommended part sizes
8–16 concurrent part uploads is a good starting point on a fast link. Larger parts mean fewer requests; smaller parts mean less to retry when a connection drops.
Downloading
GET and HEAD on a file stream its content with ETag, Accept-Ranges and, when known, X-Checksum-SHA256. Range requests return 206; If-None-Match returns 304 when unchanged. Add download=1 for Content-Disposition: attachment and version= to read a snapshot.
# Metadata onlycurl -I https://datasetsync.com/api/v1/datasets/my-dataset/files/train/shard-00001.parquet -H "Authorization: Bearer $DATASETSYNC_API_KEY"# Byte range (Parquet footer, random access into tar shards, ...)curl https://datasetsync.com/api/v1/datasets/my-dataset/files/train/shard-00001.parquet -H "Authorization: Bearer $DATASETSYNC_API_KEY" \ -H "Range: bytes=-65536" -o footer.bin# Conditional GETcurl https://datasetsync.com/api/v1/datasets/my-dataset/files/train/shard-00001.parquet -H "Authorization: Bearer $DATASETSYNC_API_KEY" \ -H 'If-None-Match: "<etag>"' -o shard.parquetPresigned URLs
Hand a single file to a machine without credentials. The returned URL (/dl/{token}) needs no auth and supports Range. expiresIn is in seconds (default 3600, max 604800).
curl -X POST https://datasetsync.com/api/v1/datasets/my-dataset/presign -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"path": "train/shard-00001.parquet", "version": "v1", "expiresIn": 86400}'# → { "url": "https://datasetsync.com/dl/...", "expiresAt": "..." }Manifests for bulk download
GET /api/v1/datasets/{dataset}/manifest?prefix=&version=&format=json|txt|aria2&expiresIn= returns every selected file with a presigned URL:
json→{ dataset, version, files: [{ path, sizeBytes, sha256, url }] }txt→ one URL per linearia2→ anaria2c -iinput file without=andchecksum=sha-256=per entry
curl -o manifest.txt -H "Authorization: Bearer $DATASETSYNC_API_KEY" \ "https://datasetsync.com/api/v1/datasets/my-dataset/manifest?prefix=train/&version=v1&format=aria2&expiresIn=86400"aria2c -i manifest.txt -j 16 -x 4 # 16 files at once, 4 connections each, checksums verifiedStreaming tar archives
GET /api/v1/datasets/{dataset}/archive?prefix=&version= streams an uncompressed POSIX tar of the selection, so you can extract without staging to disk:
curl -sf -H "Authorization: Bearer $DATASETSYNC_API_KEY" "https://datasetsync.com/api/v1/datasets/my-dataset/archive?prefix=val/&version=v1" | tar xVersions
A version is an immutable snapshot of a dataset's file tree. Pin training runs to a version and later uploads or deletions will not change what they read. Names must match ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$.
# Createcurl -X POST https://datasetsync.com/api/v1/datasets/my-dataset/versions -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"name": "v1", "description": "Dedup pass 2"}'# List (newest first) / get onecurl https://datasetsync.com/api/v1/datasets/my-dataset/versions -H "Authorization: Bearer $DATASETSYNC_API_KEY"curl https://datasetsync.com/api/v1/datasets/my-dataset/versions/v1 -H "Authorization: Bearer $DATASETSYNC_API_KEY"# Read anything at a versioncurl "https://datasetsync.com/api/v1/datasets/my-dataset/files/train/shard-00001.parquet?version=v1" -H "Authorization: Bearer $DATASETSYNC_API_KEY" -o shard.parquet# Delete (204); objects no longer referenced become eligible for garbage collectioncurl -X DELETE https://datasetsync.com/api/v1/datasets/my-dataset/versions/v1 -H "Authorization: Bearer $DATASETSYNC_API_KEY"?version= works on file downloads, listings, manifests and archives.
Share links & public datasets
Share links give read-only, unauthenticated access to a dataset, optionally narrowed to a version or path prefix, with an expiry and download limit.
curl -X POST https://datasetsync.com/api/v1/datasets/my-dataset/share-links -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"label": "eval partners", "version": "v1", "pathPrefix": "val/", "expiresInDays": 14, "maxDownloads": 500}'# → 201 ShareLink with "url": "https://datasetsync.com/s/..." (shown once)curl -X DELETE https://datasetsync.com/api/v1/share-links/$SHARE_LINK_ID -H "Authorization: Bearer $DATASETSYNC_API_KEY" # revokeRecipients can browse and download through the public endpoints (no auth, Range supported):
Datasets with visibility: "public" are readable by anyone at GET /api/public/datasets/{org}/{dataset}, plus /files and /files/{path...} beneath it.
S3 gateway
Existing S3 tooling works against https://datasetsync.com/s3. Each dataset is a bucket named by its slug.
curl -X POST https://datasetsync.com/api/org/s3-credentials -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"name": "training-cluster", "scopes": ["read", "write"]}'# → { "credential": { ... }, "accessKeyId": "...", "secretAccessKey": "..." } (secret shown once)Supported operations: ListBuckets, CreateBucket, DeleteBucket, HeadBucket, ListObjectsV2 (and V1), HeadObject, GetObject (Range), PutObject, CopyObject, DeleteObject, DeleteObjects, CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts, ListMultipartUploads.
Client configuration
# accessKeyId / secretAccessKey from POST /api/org/s3-credentialsaws configure set aws_access_key_id DSAK... --profile dsaws configure set aws_secret_access_key ... --profile dsaws configure set region us-east-1 --profile dsaws configure set s3.addressing_style path --profile dsaws s3 ls --endpoint-url https://datasetsync.com/s3 --profile dsaws s3 cp ./shards s3://my-dataset/train/ --recursive \ --endpoint-url https://datasetsync.com/s3 --profile dsaws s3 sync s3://my-dataset/val/ ./val \ --endpoint-url https://datasetsync.com/s3 --profile dsDirect uploads
The Python SDK and CLI upload files of 64 MiB and more directly to our backend file servers: the server plans the file as segments of up to ~1 GiB, issues a random AES-256 key for each, and names the file server. Your machine encrypts every segment and sends the ciphertext to that file server, so file bytes never pass through our web servers, and your API key is never sent to the file server. The file is readable as soon as the upload completes and becomes ARCHIVED once the file servers have stored it durably. Large downloads of archived files are fetched from the file servers and decrypted locally in the same way. Pass --no-direct (CLI) or direct=False (Python) to send bytes through our web servers instead. The endpoints are described in the API reference (/api/v1/datasets/{dataset}/direct-uploads).
Python SDK & CLI
The SDK wraps the REST API with automatic direct and multipart uploads, resumption and checksum verification. It reads credentials from DATASETSYNC_API_KEY and DATASETSYNC_URL, or from the config file written by datasetsync login. Remote locations are written ds://<dataset>/<path>.
pip install datasetsyncdatasetsync login --url https://datasetsync.com # prompts for an API key; or export DATASETSYNC_API_KEYdatasetsync mb my-datasetdatasetsync sync ./shards ds://my-dataset/train/ # parallel, resumable, skips identical filesdatasetsync ls ds://my-dataset/train/datasetsync versions create my-dataset v1datasetsync cp -r ds://my-dataset/train/ ./out --version v1 # verifies SHA-256datasetsync manifest ds://my-dataset/train/ --version v1 --format aria2 -o train.aria2aria2c -x8 -j8 -i train.aria2datasetsync tar ds://my-dataset/val/ --version v1 | tar ximport osfrom datasetsync import Clientc = Client(api_key=os.environ["DATASETSYNC_API_KEY"], base_url="https://datasetsync.com")c.datasets.create("My dataset", slug="my-dataset")c.upload_dir("my-dataset", "./local_dir", prefix="train/") # parallel, resumable, skips identical filesc.versions.create("my-dataset", "v1", description="first snapshot")for f in c.iter_files("my-dataset", "train/", version="v1"): print(f["path"], f["sizeBytes"], f["status"])c.download_dir("my-dataset", "./out", prefix="train/", version="v1") # verifies SHA-256url = c.presign("my-dataset", "train/shard-00000.parquet", expires_in=3600)["url"]with c.open("my-dataset", "train/shard-00000.parquet", version="v1") as fh: # seekable Range reads header = fh.read(1024)Streaming into PyTorch
datasetsync.torch.StreamingDataset streams the files under a prefix (optionally pinned to a version) and shards them across DataLoader workers and distributed ranks, so each file is read exactly once per epoch. decode(path, f) receives a seekable file backed by Range requests; a generator may yield many samples per file.
# pip install "datasetsync[torch]" pyarrowimport pyarrow.parquet as pqfrom torch.utils.data import DataLoaderfrom datasetsync.torch import StreamingDatasetdef decode(path, f): # f is a seekable, lazily fetched RemoteFile for batch in pq.ParquetFile(f).iter_batches(batch_size=1024): yield from batch.to_pylist()train = StreamingDataset( "my-dataset", "train/", version="v1", pattern="*.parquet", decode=decode, shuffle=True, base_url="https://datasetsync.com", # api_key defaults to $DATASETSYNC_API_KEY)loader = DataLoader(train, batch_size=None, num_workers=8)for epoch in range(3): train.set_epoch(epoch) for row in loader: ...Webhooks
Register an HTTPS endpoint (admin) to receive events as JSON POST requests. The signing secret is returned once at creation.
curl -X POST https://datasetsync.com/api/org/webhooks -H "Authorization: Bearer $DATASETSYNC_API_KEY" -H "Content-Type: application/json" \ -d '{"url": "https://ci.example.com/hooks/datasets", "events": ["upload.completed", "version.created"]}'# → { "webhook": { "id": "...", ... }, "secret": "..." }curl -X POST https://datasetsync.com/api/org/webhooks/$WEBHOOK_ID/test -H "Authorization: Bearer $DATASETSYNC_API_KEY" # send a test deliveryEvents
Verifying signatures
Every delivery carries:
X-DatasetSync-Signature: t=<unix seconds>,v1=<hex hmac>where v1 = hex(hmac_sha256(secret, t + "." + body)) over the raw request body. Reject deliveries whose timestamp is more than a few minutes old, and compare signatures in constant time.
import crypto from "node:crypto";// rawBody: the exact request body as a string or Buffer (not re-serialized JSON)export function verifyWebhook(secret, header, rawBody, toleranceSec = 300) { const parts = Object.fromEntries( header.split(",").map((kv) => { const i = kv.indexOf("="); return [kv.slice(0, i).trim(), kv.slice(i + 1).trim()]; }) ); const t = Number(parts.t); if (!Number.isFinite(t) || Math.abs(Date.now() / 1000 - t) > toleranceSec) return false; const expected = crypto .createHmac("sha256", secret) .update(parts.t + ".") .update(rawBody) .digest(); const given = Buffer.from(parts.v1 || "", "hex"); return given.length === expected.length && crypto.timingSafeEqual(given, expected);}// Express: app.post("/hooks", express.raw({ type: "*/*" }), (req, res) => {// if (!verifyWebhook(SECRET, req.get("X-DatasetSync-Signature") || "", req.body)) return res.sendStatus(400);// const event = JSON.parse(req.body.toString("utf8"));// res.sendStatus(204);// });Respond with a 2xx status quickly and do heavy work asynchronously.
Storage lifecycle
Every file moves through the same states, reported as status on the File object:
Processing → Hot → Archiving → Archived(or Failed)
Archival splits files into segments of at most 4 GiB, each encrypted and tracked with its own archival status. Inspect them with GET /api/v1/files/{fileId}, which returns the File plus its segments (index, offset, size and status) and the versions that reference it. Organization-wide totals are at GET /api/v1/storage/summary and recent segments at GET /api/v1/storage/segments.
Restoring to the hot tier
Archived files may be evicted from the hot tier (hotStored: false). Re-hydrate one before a training run with:
curl -X POST https://datasetsync.com/api/v1/files/$FILE_ID/restore -H "Authorization: Bearer $DATASETSYNC_API_KEY" # → 202 AcceptedErrors & rate limits
Errors always use the same envelope. Include requestId when contacting support.
{ "error": { "code": "not_found", "message": "Dataset not found", "details": null }, "requestId": "..."}Rate limits
When you exceed a rate limit the API responds 429 with a Retry-After header (seconds). Wait at least that long, and use exponential backoff with jitter for retries of 5xx responses and network errors. Part uploads and PUTs are idempotent and safe to retry.
Limits
Need help?