Huggingface.js documentation
🤗 Hugging Face Hub API
🤗 Hugging Face Hub API
用于使用 Hugging Face Hub API 的官方实用工具。
安装
pnpm add @huggingface/hub npm add @huggingface/hub yarn add @huggingface/hub
Deno
// esm.sh
import { uploadFiles, listModels } from "https://esm.sh/@huggingface/hub"
// or npm:
import { uploadFiles, listModels } from "npm:@huggingface/hub"用法
对于某些调用,您需要创建一个账户并生成一个访问令牌。
在本交互式教程中,学习如何使用 hub 包查找免费模型。
import * as hub from "@huggingface/hub";
import type { RepoDesignation } from "@huggingface/hub";
const repo: RepoDesignation = { type: "model", name: "myname/some-model" };
const {name: username} = await hub.whoAmI({accessToken: "hf_..."});
for await (const model of hub.listModels({search: {owner: username}, accessToken: "hf_..."})) {
console.log("My model:", model);
}
const specificModel = await hub.modelInfo({name: "openai-community/gpt2"});
await hub.checkRepoAccess({repo, accessToken: "hf_..."});
await hub.createRepo({ repo, accessToken: "hf_...", license: "mit" });
await hub.uploadFiles({
repo,
accessToken: "hf_...",
files: [
// path + blob content
{
path: "file.txt",
content: new Blob(["Hello World"]),
},
// Local file URL
pathToFileURL("./pytorch-model.bin"),
// Local folder URL
pathToFileURL("./models"),
// Web URL
new URL("https://huggingface.co/xlm-roberta-base/resolve/main/tokenizer.json"),
// Path + Web URL
{
path: "myfile.bin",
content: new URL("https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin")
}
// Can also work with native File in browsers
],
});
// or
for await (const progressEvent of await hub.uploadFilesWithProgress({
repo,
accessToken: "hf_...",
files: [
...
],
})) {
console.log(progressEvent);
}
// Edit a file in place, without downloading/uploading its unchanged data
const file = await hub.downloadFile({ repo, path: "myfile.bin", accessToken: "hf_..." });
await hub.uploadFile({
repo,
accessToken: "hf_...",
file: {
path: "myfile.bin",
originalContent: file,
edits: [{
// Replace bytes [0, 4)
start: 0,
end: 4,
content: new Blob(["new prefix"])
}, {
// Append at the end
start: file.size,
end: file.size,
content: new Blob(["suffix"])
}]
}
});
await hub.deleteFile({repo, accessToken: "hf_...", path: "myfile.bin"});
await (await hub.downloadFile({ repo, path: "README.md" })).text();
for await (const fileInfo of hub.listFiles({repo})) {
console.log(fileInfo);
}
await hub.deleteRepo({ repo, accessToken: "hf_..." });桶
存储桶是 Hub 上可变的、非版本化的存储。使用 buckets/ 前缀(或 { type: "bucket", name })配合相同的函数:
const bucket = "buckets/myname/my-bucket";
await hub.createRepo({ repo: bucket, accessToken: "hf_..." });
await hub.uploadFiles({ repo: bucket, accessToken: "hf_...", files: [pathToFileURL("./checkpoints")] });
for await (const fileInfo of hub.listFiles({ repo: bucket, recursive: true, accessToken: "hf_..." })) {
console.log(fileInfo.path, fileInfo.uploadedAt);
}
// Copy files from a model or dataset repo into a bucket server-side, without downloading them
await hub.copyFolder({
source: { repo: { type: "dataset", name: "myname/my-dataset" }, path: "data" },
destination: { repo: bucket, path: "data" },
accessToken: "hf_...",
});CLI 用法
您可以在 CLI 模式下使用 @huggingface/hub 将文件和文件夹上传到或从您的仓库下载。
npx @huggingface/hub upload coyotte508/test-model .
npx @huggingface/hub upload datasets/coyotte508/test-dataset .
# Same thing
npx @huggingface/hub upload --repo-type dataset coyotte508/test-dataset .
# Upload new data with 0 history in a separate branch
npx @huggingface/hub branch create coyotte508/test-model release --empty
npx @huggingface/hub upload coyotte508/test-model . --revision release
# Download files to the current directory (or --local-dir)
npx @huggingface/hub download coyotte508/test-model config.json
npx @huggingface/hub download coyotte508/test-model config.json README.md --local-dir ./test-model
# Download an entire repo, optionally filtered with --include / --exclude
npx @huggingface/hub download coyotte508/test-model --local-dir ./test-model
npx @huggingface/hub download --repo-type dataset coyotte508/test-dataset --include "*.json"
npx @huggingface/hub --help
npx @huggingface/hub upload --help
npx @huggingface/hub download --help您也可以使用 npm install -g @huggingface/hub 进行全局安装。然后您可以执行以下操作:
hfjs upload coyotte508/test-model . hfjs branch create --repo-type dataset coyotte508/test-dataset release --empty hfjs upload --repo-type dataset coyotte508/test-dataset . --revision release hfjs download coyotte508/test-model config.json hfjs download coyotte508/test-model --local-dir ./test-model hfjs --help hfjs upload --help hfjs help jobs hfjs download --help
OAuth 登录
可以使用 OAuth(“使用 HF 登录”)进行登录。
这将使您能够获取访问令牌,以便使用部分 API,具体取决于 Space 或 OAuth App 中设置的权限范围。
import { oauthLoginUrl, oauthHandleRedirectIfPresent } from "@huggingface/hub";
const oauthResult = await oauthHandleRedirectIfPresent();
if (!oauthResult) {
// If the user is not logged in, redirect to the login page
window.location.href = await oauthLoginUrl();
}
// You can use oauthResult.accessToken, oauthResult.accessTokenExpiresAt and oauthResult.userInfo
console.log(oauthResult);查看演示:https://huggingface.co/spaces/huggingfacejs/client-side-oauth
Hugging face 缓存
@huggingface/hub 包提供了扫描缓存目录的基本功能。有关 管理 huggingface_hub 缓存系统 的更多信息,请参阅相关文档。
scanCacheDir
您可以使用 scanCacheDir 函数获取缓存仓库的列表。
import { scanCacheDir } from "@huggingface/hub";
const result = await scanCacheDir();
console.log(result);注意:此功能在浏览器中无法使用
downloadFileToCacheDir
您可以使用 downloadFileToCacheDir 函数来缓存仓库中的文件。
import { downloadFileToCacheDir } from "@huggingface/hub";
const file = await downloadFileToCacheDir({
repo: 'foo/bar',
path: 'README.md'
});
console.log(file);注意:此功能在浏览器中无法使用
snapshotDownload
您可以使用 snapshotDownload 函数在缓存目录中下载指定修订版本的整个仓库。
import { snapshotDownload } from "@huggingface/hub";
const directory = await snapshotDownload({
repo: 'foo/bar',
});
console.log(directory);该代码内部使用了 downloadFileToCacheDir 函数。
注意:此功能在浏览器中无法使用
性能考量
在上传大文件时,您可能希望在 worker 中运行 commit 调用,以将 sha256 计算任务卸载出去。
在可能的情况下,远程资源和本地文件应作为 URL 传入,以便按块进行延迟加载,从而降低内存占用。在浏览器上下文中传入 File 也是可行的,因为它原生表现为 Blob。
在底层,@huggingface/hub 使用惰性 blob 实现来加载文件。
若要编辑现有文件,请优先使用 edit 提交操作,并将 downloadFile 返回的 blob 作为 originalContent:这样只会下载、重新分块并计算哈希值被编辑的区域。对于向同一文件进行多次追加操作,请在 commit 调用中传入共享的 rangeEditCache: new Map(),以同时跳过存储元数据的往返请求。
依赖
@huggingface/tasks: 仅类型定义