# Too many requests for URL

**URL:** <https://discuss.huggingface.co/t/too-many-requests-for-url/139743>\
**Category:** 🤗Hub\
**Created:** [February 8, 2025, 6:26pm UTC](https://discuss.huggingface.co/t/too-many-requests-for-url/139743 "2025-02-08T18:26:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nishadsinghi](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nishadsinghi](https://discuss.huggingface.co/u/nishadsinghi)\
**Post date:** [February 8, 2025, 6:26pm UTC](https://discuss.huggingface.co/t/too-many-requests-for-url/139743/1 "2025-02-08T18:26:43Z")

</div>

I am using a finetuned llama model for inference (using vllm) and keep getting this error:

```auto
Traceback (most recent call last):
  File "/pfss/mlde/workspaces/mlde_wsp_Rohrbach/users/ns94feza/.conda/envs/llmonk2/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
    response.raise_for_status()
  File "/pfss/mlde/workspaces/mlde_wsp_Rohrbach/users/ns94feza/.conda/envs/llmonk2/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/hbXNov/llama3.1-8b_train_gpt_4o_verifications_e3_lr5e-7-31389-merged/resolve/main/sentence_bert_config.json

```

I have been using this script for weeks and only got this error recently. For context, I got rate limited earlier today while trying to upload a large dataset. I wonder if that caused this issue, and if it’s possible to lift the limit. Also, I have this model locally, so I’m able to successfully load it this way:

```auto
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

```

but when I go the vllm way, everything crashes.

Any help would be much appreciated!  
Tagging @Wauplin in case you could help 🙂

---

<div class="post-metadata">

**Author:** ![Wauplin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wauplin/32/40815_2.png) [@Wauplin](https://discuss.huggingface.co/u/Wauplin)\
**Post date:** [February 10, 2025, 11:42am UTC](https://discuss.huggingface.co/t/too-many-requests-for-url/139743/2 "2025-02-10T11:42:53Z")

</div>

Hi @nishadsinghi it is very probable that yes, the rate limit for the dataset upload and this one are related. Since yesterday rate limits should have been lifted again.

---

<div class="post-metadata">

**Author:** ![nishadsinghi](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nishadsinghi](https://discuss.huggingface.co/u/nishadsinghi)\
**Post date:** [February 10, 2025, 7:11pm UTC](https://discuss.huggingface.co/t/too-many-requests-for-url/139743/3 "2025-02-10T19:11:51Z")

</div>

> [@Wauplin](#):
>
> probable that yes, the rate

Is there a way to check if the rate limit has been lifted? I have not been able to log into HF hub (see: [Can't login to Huggingface CLI - #11 by Wauplin](https://discuss.huggingface.co/t/cant-login-to-huggingface-cli/139741/11)), so I can’t upload a dataset as a check.

---

<div class="post-metadata">

**Author:** ![nishadsinghi](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nishadsinghi](https://discuss.huggingface.co/u/nishadsinghi)\
**Post date:** [March 5, 2025, 12:52am UTC](https://discuss.huggingface.co/t/too-many-requests-for-url/139743/4 "2025-03-05T00:52:22Z")

</div>

Hey @Wauplin, thanks for your reply. I was just trying to upload another large dataset using `huggingface-cli upload-large-folder`, and again got rate-limited. I thought the whole point of `huggingface-cli upload-large-folder` was to avoid being rate-limited. What’s the best way to upload large datasets (with lots of files) to HF without facing this issue? Also, the last time this happened to me, it took more than a day for the limit to be lifted. Is it possible to get it lifted sooner? Thanks 🙂

---

<div class="post-metadata">

**Author:** ![Wauplin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wauplin/32/40815_2.png) [@Wauplin](https://discuss.huggingface.co/u/Wauplin)\
**Post date:** [March 6, 2025, 11:28am UTC](https://discuss.huggingface.co/t/too-many-requests-for-url/139743/5 "2025-03-06T11:28:58Z")

</div>

Hi @nishadsinghi sorry about that. `upload_large_folder`still has to be improved, especially for datasets with lots of smaller files. We are planning to work on it soon. In the meantime, the best you can do is to use the `---include` parameter to upload only subparts of the dataset (per folder for instance). Sorry for the inconvenience!

---

<div class="post-metadata">

**Author:** ![Norod78](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/norod78/32/48228_2.png) [@Norod78](https://discuss.huggingface.co/u/Norod78)\
**Post date:** [May 25, 2025, 10:32am UTC](https://discuss.huggingface.co/t/too-many-requests-for-url/139743/6 "2025-05-25T10:32:48Z")

</div>

I just found this thread after getting rate-limited under these circumstances (`huggingface-cli upload-large-folder`on a dataset with lots of small files). I understand it’s still a feature-under-construction and all and I just wondered when should I try to resume (I assume running `huggingface-cli upload-large-folder` again will resume, right?). The error message said to wait about 1h, is this indeed correct?

UPDATE: I just switched it to use git-lfs directly and so far it looks like it’s working (as in uploading). Also, it might be that upload-large-folder failed because I had \> 10k file in a single folder? ([Says here](https://huggingface.co/docs/hub/en/storage-limits) it’s a bad idea)

Thank you @Wauplin