Spaces:
Running
Download README.md from Wauplin/s3-importer: direct link, hf CLI and curl.
- Browser
- Download file 4.28 kB
-
https://huggingface.co/spaces/Wauplin/s3-importer/resolve/main/README.md
- Command line
-
hf download hf://spaces/Wauplin/s3-importer/README.md
-
curl -L -o README.md https://huggingface.co/spaces/Wauplin/s3-importer/resolve/main/README.md
title: S3 to HF Bucket Importer
emoji: 🪣
colorFrom: gray
colorTo: green
sdk: static
pinned: false
hf_oauth: true
hf_oauth_scopes:
- manage-repos
- jobs
hf_oauth_expiration_minutes: 480
S3 to HF Bucket Importer
A static one-page web app that helps users import data from AWS S3 (or any S3-compatible storage) into Hugging Face Storage Buckets.
Live: huggingface.co/spaces/wauplin/s3-importer
How it works
- Sign in with your Hugging Face account (OAuth)
- Connect to your S3 bucket by providing credentials
- Browse & select files/folders to import (requires CORS on your S3 bucket, see below)
- Choose destination HF bucket (auto-created if needed) and optional prefix
- Import - launches an HF Job that runs
hf buckets importserver-side
Everything runs in the browser. No data is stored locally. The actual data transfer happens in an HF Job (server-side), so your browser doesn't need to stay open.
Architecture
- Static HTML Space - no backend, no build step
- OAuth via
@huggingface/hub(CDN) for authentication - AWS SDK v3 via CDN for browser-side S3 listing
- HF Jobs API for server-side data transfer using
hf buckets import
Files
| File | Purpose |
|---|---|
index.html |
HTML structure, import maps, meta tags |
style.css |
Dark theme stylesheet |
app.js |
All application logic (ES module) |
README.md |
This file + Space YAML config |
AGENTS.md |
Requirements and architecture decisions |
S3 CORS Configuration
To browse your S3 bucket from the browser, CORS must be enabled on the bucket. Add this CORS configuration:
[
{
"AllowedOrigins": ["https://wauplin-s3-importer.static.hf.space"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"MaxAgeSeconds": 3600
}
]
Without CORS: The app gracefully degrades to a "manual mode" where you can still import everything (or specify include/exclude patterns) without browsing.
How to set CORS on AWS S3
- Go to your S3 bucket in the AWS Console
- Navigate to Permissions > Cross-origin resource sharing (CORS)
- Paste the JSON above and save
HF Job Details
The import Job runs:
pip install 'huggingface_hub[s3] @ git+https://github.com/huggingface/huggingface_hub.git@cursor/s3-to-hf-bucket-ingestion-f144'
hf buckets import 's3://bucket/prefix' 'namespace/bucket-name' [--include ...] [--exclude ...] --workers 4 --verbose
- Docker image:
python:3.12 - Hardware:
cpu-basic(I/O-bound task, no GPU needed) - Secrets:
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,HF_TOKEN(encrypted server-side) - Environment:
AWS_DEFAULT_REGION, optionallyAWS_ENDPOINT_URL
OAuth Scopes
| Scope | Reason |
|---|---|
manage-repos |
Create destination buckets |
jobs |
Create and monitor import Jobs |
openid + profile |
Always included, for user identity |
Maintenance Notes
Updating huggingface_hub version
Once the hf buckets import command is released in a stable version, update the pip install command in app.js (search for git+https://github.com/huggingface/huggingface_hub). Replace with:
pip install 'huggingface_hub[s3]>=X.Y.Z'
CDN Dependencies
Dependencies are loaded from esm.sh via import maps in index.html. To update versions, edit the import map URLs. Current deps:
@huggingface/hub- OAuth helpers@aws-sdk/client-s3- S3 listing
Token Lifetime
OAuth tokens expire after 8 hours (configurable in YAML frontmatter via hf_oauth_expiration_minutes). The token is also passed to the Job as a secret. For very long imports (>8h), the Job's token may expire. Consider increasing hf_oauth_expiration_minutes if needed.
Development
This is a static site with no build step. To develop locally:
- Serve files with any HTTP server:
python -m http.server 8000 - OAuth won't work locally (needs HF Space environment variables). Test OAuth flow on the deployed Space.
- S3 listing works locally if CORS is configured for
http://localhost:8000.
License
MIT