--- title: S3 to HF Bucket Importer emoji: "\U0001FAA3" colorFrom: gray colorTo: green sdk: static pinned: false hf_oauth: true hf_oauth_scopes: - manage-repos - jobs hf_oauth_expiration_minutes: 480 --- # S3 to HF Bucket Importer A static one-page web app that helps users import data from AWS S3 (or any S3-compatible storage) into [Hugging Face Storage Buckets](https://huggingface.co/docs/hub/storage-buckets). **Live**: [huggingface.co/spaces/wauplin/s3-importer](https://huggingface.co/spaces/wauplin/s3-importer) ## How it works 1. **Sign in** with your Hugging Face account (OAuth) 2. **Connect** to your S3 bucket by providing credentials 3. **Browse & select** files/folders to import (requires CORS on your S3 bucket, see below) 4. **Choose destination** HF bucket (auto-created if needed) and optional prefix 5. **Import** - launches an HF Job that runs `hf buckets import` server-side Everything runs in the browser. No data is stored locally. The actual data transfer happens in an [HF Job](https://huggingface.co/docs/hub/jobs) (server-side), so your browser doesn't need to stay open. ## Architecture - **Static HTML Space** - no backend, no build step - **OAuth** via `@huggingface/hub` (CDN) for authentication - **AWS SDK v3** via CDN for browser-side S3 listing - **HF Jobs API** for server-side data transfer using `hf buckets import` ### Files | File | Purpose | |------|---------| | `index.html` | HTML structure, import maps, meta tags | | `style.css` | Dark theme stylesheet | | `app.js` | All application logic (ES module) | | `README.md` | This file + Space YAML config | | `AGENTS.md` | Requirements and architecture decisions | ## S3 CORS Configuration To browse your S3 bucket from the browser, CORS must be enabled on the bucket. Add this CORS configuration: ```json [ { "AllowedOrigins": ["https://wauplin-s3-importer.static.hf.space"], "AllowedMethods": ["GET", "HEAD"], "AllowedHeaders": ["*"], "MaxAgeSeconds": 3600 } ] ``` **Without CORS**: The app gracefully degrades to a "manual mode" where you can still import everything (or specify include/exclude patterns) without browsing. ### How to set CORS on AWS S3 1. Go to your S3 bucket in the AWS Console 2. Navigate to **Permissions** > **Cross-origin resource sharing (CORS)** 3. Paste the JSON above and save ## HF Job Details The import Job runs: ```bash pip install 'huggingface_hub[s3] @ git+https://github.com/huggingface/huggingface_hub.git@cursor/s3-to-hf-bucket-ingestion-f144' hf buckets import 's3://bucket/prefix' 'namespace/bucket-name' [--include ...] [--exclude ...] --workers 4 --verbose ``` - **Docker image**: `python:3.12` - **Hardware**: `cpu-basic` (I/O-bound task, no GPU needed) - **Secrets**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `HF_TOKEN` (encrypted server-side) - **Environment**: `AWS_DEFAULT_REGION`, optionally `AWS_ENDPOINT_URL` ## OAuth Scopes | Scope | Reason | |-------|--------| | `manage-repos` | Create destination buckets | | `jobs` | Create and monitor import Jobs | | `openid` + `profile` | Always included, for user identity | ## Maintenance Notes ### Updating `huggingface_hub` version Once the `hf buckets import` command is released in a stable version, update the pip install command in `app.js` (search for `git+https://github.com/huggingface/huggingface_hub`). Replace with: ``` pip install 'huggingface_hub[s3]>=X.Y.Z' ``` ### CDN Dependencies Dependencies are loaded from `esm.sh` via import maps in `index.html`. To update versions, edit the import map URLs. Current deps: - `@huggingface/hub` - OAuth helpers - `@aws-sdk/client-s3` - S3 listing ### Token Lifetime OAuth tokens expire after 8 hours (configurable in YAML frontmatter via `hf_oauth_expiration_minutes`). The token is also passed to the Job as a secret. For very long imports (>8h), the Job's token may expire. Consider increasing `hf_oauth_expiration_minutes` if needed. ## Development This is a static site with no build step. To develop locally: 1. Serve files with any HTTP server: `python -m http.server 8000` 2. OAuth won't work locally (needs HF Space environment variables). Test OAuth flow on the deployed Space. 3. S3 listing works locally if CORS is configured for `http://localhost:8000`. ## License MIT