# Issue with Deploying LoRA-adapted Model on Hugging Face Endpoint **URL:** \ **Category:** Beginners\ **Created:** [April 9, 2025, 1:27pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608 "2025-04-09T13:27:41Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![tmestiri](https://avatars.discourse-cdn.com/v4/letter/t/dbc845/32.png) [@tmestiri](https://discuss.huggingface.co/u/tmestiri)\ **Post date:** [April 9, 2025, 1:27pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/1 "2025-04-09T13:27:41Z")
I am encountering issues when trying to deploy my model with LoRA (Low-Rank Adaptation) adapters on Hugging Face Endpoints. Initially, I uploaded only the LoRA adapters to the Hugging Face model hub. However, when I attempt to deploy the model on the Endpoint, it keeps failing. **What I Have Tried:** 1. **Pushing LoRA Adapters to Hugging Face Hub:** I successfully uploaded only the LoRA adapters to Hugging Face. They work fine when I load the model locally in my development environment using the appropriate library and configuration. 2. **Merging the Base Model with LoRA Adapters:** In an attempt to ensure compatibility for deployment, I merged the base model with the LoRA adapter and then pushed the merged model to Hugging Face. Unfortunately, the deployment still fails. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 9, 2025, 1:57pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/2 "2025-04-09T13:57:19Z")
I’m not sure whether they’re Transformers or Diffusers, but in any case, LoRA is under the jurisdiction of PEFT. The Endpoint library uses a somewhat old stable version, and many of the libraries related to quantization are not included. Therefore, it may be easy to fail with QLoRA or new models. Why not try writing the new library version in the **requirements.txt**? > **[Inference Endpoints Version](https://huggingface.co/docs/inference-endpoints/others/runtime)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Add custom Dependencies](https://huggingface.co/docs/inference-endpoints/guides/custom_dependencies)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 9, 2025, 1:58pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/3 "2025-04-09T13:58:35Z")
Oh… If that’s the case, then 2 is strange… I wonder what’s going on. Does it work locally, or is it failing to apply LoRA? ---
**Author:** ![tmestiri](https://avatars.discourse-cdn.com/v4/letter/t/dbc845/32.png) [@tmestiri](https://discuss.huggingface.co/u/tmestiri)\ **Post date:** [April 9, 2025, 2:17pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/4 "2025-04-09T14:17:24Z")
Thanks for the reply! Just to clarify: - The base model I’m using is: **`unsloth/Llama-3.2-11B-Vision-Instruct`** - I fine-tuned it using **Unsloth** with **LoRA adapters** - I pushed **only the LoRA adapters** to Hugging Face initially - The LoRA adapters **work perfectly on Colab** when I load them with the base model manually - The issue only happens when I try to deploy it as an **Endpoint on Hugging Face** — it keeps failing to start, either with the LoRA-only upload or even when I later **merged the LoRA with the base model** and pushed that So I’m a bit stuck figuring out what’s going wrong during deployment. Any ideas or suggestions are super appreciated! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 9, 2025, 3:01pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/5 "2025-04-09T15:01:23Z")
> I fine-tuned it using **Unsloth** with **LoRA adapters** Hugging Face TGI (body of Endpoint) supports only **safetensors** basically. So possibly this issue? > [@LORA Adapated Deepseek R1 not working with inference endpoints](https://discuss.huggingface.co/t/lora-adapated-deepseek-r1-not-working-with-inference-endpoints/148536): > > I used a standard LORA adapter training pipeline to train the deepseek-ai/DeepSeek-R1-Distill-Llama-8B model. I then stored the model on the huggingface hub using the model.push\_to\_hub\_merged() method. I then wanted to use huggingface inference endpoints to load the model and I kept getting an error. I shall link my model as well as the error I am getting, any help would be appreciated, thanks! Name of model - bhaskars113/DeepSeek-R1-Entity-8B-V1.1 Error: Apr 02, 16:20:29 WARN police_car_lightpolice_car_lightBREAKING CHA… ---
**Author:** ![tmestiri](https://avatars.discourse-cdn.com/v4/letter/t/dbc845/32.png) [@tmestiri](https://discuss.huggingface.co/u/tmestiri)\ **Post date:** [April 9, 2025, 3:31pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/6 "2025-04-09T15:31:35Z")
I fine-tuned the vision model `unsloth/Llama-3.2-11B-Vision-Instruct` using **Unsloth** , and initially pushed only the **LoRA adapters** to Hugging Face — it worked fine locally on Colab. Later, I **merged the LoRA adapters with the base model** and pushed the full model to the Hub. However, the deployment on Hugging Face Endpoints still fails. Do you suggest that I need to push the **merged fine-tuned model in 16-bit `.safetensors` format** for it to work properly with TGI? Thanks in advance! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 10, 2025, 1:53am UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/7 "2025-04-10T01:53:19Z")
> Do you suggest that I need to push the **merged fine-tuned model in 16-bit `.safetensors` format** for it to work properly with TGI? That’s right. There is no particular need for it to be 16-bit, but I personally recommend **torch.bfloat16**. Otherwise it looks like you need to set the environment variables explicitly to use the Pickle files (.bin). Alternatively, you could use a custom handler or custom container to change the Endpoint software or OS itself, but that would be quite a struggle. Alternatively, you could use Transformers’ **AutoModelForCausalLM.from\_pretrained** and **save\_pretrained**. Well, it’s the same thing… ---
**Author:** ![tmestiri](https://avatars.discourse-cdn.com/v4/letter/t/dbc845/32.png) [@tmestiri](https://discuss.huggingface.co/u/tmestiri)\ **Post date:** [April 10, 2025, 8:23am UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/8 "2025-04-10T08:23:22Z")
Thank you for your answer. So basically, after I fine-tune a **vision model using Unsloth** , I need to push it back to Hugging Face as a **16-bit model** , not just the LoRA adapters. To ensure compatibility with Hugging Face Endpoints, should I save my fine-tuned vision model in float16 format by selecting merged\_16bit and then upload the full model (not just LoRA adapters) using push\_to\_hub\_merged()? This avoids deployment issues, correct? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 10, 2025, 8:57am UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/9 "2025-04-10T08:57:06Z")
> should I save my fine-tuned vision model in float16 format by selecting merged\_16bit and then upload the full model (not just LoRA adapters) using push\_to\_hub\_merged()? This avoids deployment issues, correct? Correct. If you save it in the safetensors file format using this method, it will be ideal in terms of compatibility. LoRA alone may work if you set the base model appropriately, but it should be less troublesome if you have the whole thing. ---
**Author:** ![tmestiri](https://avatars.discourse-cdn.com/v4/letter/t/dbc845/32.png) [@tmestiri](https://discuss.huggingface.co/u/tmestiri)\ **Post date:** [April 10, 2025, 9:14am UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/10 "2025-04-10T09:14:58Z")
Thank you for your answer. i really appreciated , it was helpful ---
**Author:** ![Neobozrim](https://avatars.discourse-cdn.com/v4/letter/n/ecb155/32.png) [@Neobozrim](https://discuss.huggingface.co/u/Neobozrim)\ **Post date:** [April 26, 2025, 7:40pm UTC](https://discuss.huggingface.co/t/issue-with-deploying-lora-adapted-model-on-hugging-face-endpoint/149608/11 "2025-04-26T19:40:25Z")
I had the same issue a couple of days ago and wanted to say thank you to both of you guys - to @tmestiri for sharing his issue (I had absolutely the same case) and for @John6666 for the guidance and support! Thank you both once again!