s3-importer / README.md
Wauplin's picture
Wauplin HF Staff
Upload folder using huggingface_hub
5bafb68 verified
|
Raw History Blame Contribute Delete
4.28 kB
metadata
title: S3 to HF Bucket Importer
emoji: 🪣
colorFrom: gray
colorTo: green
sdk: static
pinned: false
hf_oauth: true
hf_oauth_scopes:
  - manage-repos
  - jobs
hf_oauth_expiration_minutes: 480

S3 to HF Bucket Importer

A static one-page web app that helps users import data from AWS S3 (or any S3-compatible storage) into Hugging Face Storage Buckets.

Live: huggingface.co/spaces/wauplin/s3-importer

How it works

  1. Sign in with your Hugging Face account (OAuth)
  2. Connect to your S3 bucket by providing credentials
  3. Browse & select files/folders to import (requires CORS on your S3 bucket, see below)
  4. Choose destination HF bucket (auto-created if needed) and optional prefix
  5. Import - launches an HF Job that runs hf buckets import server-side

Everything runs in the browser. No data is stored locally. The actual data transfer happens in an HF Job (server-side), so your browser doesn't need to stay open.

Architecture

  • Static HTML Space - no backend, no build step
  • OAuth via @huggingface/hub (CDN) for authentication
  • AWS SDK v3 via CDN for browser-side S3 listing
  • HF Jobs API for server-side data transfer using hf buckets import

Files

File Purpose
index.html HTML structure, import maps, meta tags
style.css Dark theme stylesheet
app.js All application logic (ES module)
README.md This file + Space YAML config
AGENTS.md Requirements and architecture decisions

S3 CORS Configuration

To browse your S3 bucket from the browser, CORS must be enabled on the bucket. Add this CORS configuration:

[
  {
    "AllowedOrigins": ["https://wauplin-s3-importer.static.hf.space"],
    "AllowedMethods": ["GET", "HEAD"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3600
  }
]

Without CORS: The app gracefully degrades to a "manual mode" where you can still import everything (or specify include/exclude patterns) without browsing.

How to set CORS on AWS S3

  1. Go to your S3 bucket in the AWS Console
  2. Navigate to Permissions > Cross-origin resource sharing (CORS)
  3. Paste the JSON above and save

HF Job Details

The import Job runs:

pip install 'huggingface_hub[s3] @ git+https://github.com/huggingface/huggingface_hub.git@cursor/s3-to-hf-bucket-ingestion-f144'
hf buckets import 's3://bucket/prefix' 'namespace/bucket-name' [--include ...] [--exclude ...] --workers 4 --verbose
  • Docker image: python:3.12
  • Hardware: cpu-basic (I/O-bound task, no GPU needed)
  • Secrets: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, HF_TOKEN (encrypted server-side)
  • Environment: AWS_DEFAULT_REGION, optionally AWS_ENDPOINT_URL

OAuth Scopes

Scope Reason
manage-repos Create destination buckets
jobs Create and monitor import Jobs
openid + profile Always included, for user identity

Maintenance Notes

Updating huggingface_hub version

Once the hf buckets import command is released in a stable version, update the pip install command in app.js (search for git+https://github.com/huggingface/huggingface_hub). Replace with:

pip install 'huggingface_hub[s3]>=X.Y.Z'

CDN Dependencies

Dependencies are loaded from esm.sh via import maps in index.html. To update versions, edit the import map URLs. Current deps:

  • @huggingface/hub - OAuth helpers
  • @aws-sdk/client-s3 - S3 listing

Token Lifetime

OAuth tokens expire after 8 hours (configurable in YAML frontmatter via hf_oauth_expiration_minutes). The token is also passed to the Job as a secret. For very long imports (>8h), the Job's token may expire. Consider increasing hf_oauth_expiration_minutes if needed.

Development

This is a static site with no build step. To develop locally:

  1. Serve files with any HTTP server: python -m http.server 8000
  2. OAuth won't work locally (needs HF Space environment variables). Test OAuth flow on the deployed Space.
  3. S3 listing works locally if CORS is configured for http://localhost:8000.

License

MIT