I am trying to run a large DeepSeek-R1-Distill-Qwen-32B-Uncensored-Q8_0-GGUF language model (~34.8 GB) on the Hugging Face Spaces platform using an Nvidia L40S GPU (48 GB VRAM). The model successfully loads on VRAM, but an error (runtime error) occurs while attempting to initialize, after which the …

import gradio as gr import os from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig import torch import time HF_TOKEN = os.getenv(“HF_TOKEN”, None) REPO_ID = “nicoboss/DeepSeek-R1-Distill-Qwen-32B-Uncensored” DESCRIPTION = f"“” {REPO_ID} """ PLACEHOLDER = f"“” …

Unfortunately no, I tried to disable quantization but then the model does not fit in memory, I tried to increase quantization to 8 bits, but it did not change significantly

I tried adding a system promt, but it doesn’t affect the result either.

That’s strange… I wonder if it’s different from the model I’m using for testing… I’m testing it again now. BTW, that’s normal for quantization-related things. I quantized it because I didn’t have enough VRAM.

Yes, I saw in the code that you applied quantization to 4 bits, and I’m trying a different model now, I’ll report back soon.

I can not find in search Original Model: DeepSeek-R1-Distill-Qwen-32B-Uncensored I see only versions after quantization of this model, but there is no original file. or it is not available on huggingface and should be taken elsewhere ?

This one. nicoboss/DeepSeek-R1-Distill-Qwen-32B-Uncensored · Hugging Face I’ve figured out the cause, but it’s a problem with the VRAM. The standard Transformers cache implementation is easy to use, but it eats up VRAM… I think I’ll try to implement a better version tomorrow. For now, I’ve upload…

I’m running using Nvidia 1x L40S vCPU: 8 RAM (RAM): ~62GB VRAM (GPU memory): 48 GB and the model responds much faster, and always responds to the first message, but it is not stable and after the first message it hangs and does not respond to the next messages.

import spaces # Импорт spaces обязательно должен быть до импорта torch import gradio as gr import os import torch from transformers import ( AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer, BitsAndBytesConfig ) from threading import Thread from torch.nn.attention import SDPBack…

inputs = tokenizer.apply_chat_template( #history + messages, # this line causes hang messages, Edit: might be fixed. input_tensors = tokenizer.apply_chat_template([{"role": x["role"], "content": x["content"]} for x in history] + messages, add_generation_prompt=True, ret…

import spaces import gradio as gr import os from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig import torch import torch.cuda import time HF_TOKEN = os.getenv(“HF_TOKEN”, None) REPO_ID = “nicoboss/DeepSeek-R1-Distill-Qwen-32B-Uncensored” DESCRIPTION = f"“” {…

Problem with launching DeepSeek-R1-Distill-Qwen-32B-Uncensored-Q8_0-GGUF

Models

John6666 March 17, 2025, 4:17pm 29

I see, so the model doesn’t recognize its own statements as its own.

https://medium.com/@vaibkumr/llm-chatcompletion-system-prompt-inconsistency-issue-how-to-improve-conversational-rag-systems-e3a391252629

In LangChain

https://medium.com/@vinayakdeshpande111/how-to-make-llm-remember-conversation-with-langchain-924083079d95

Topic		Replies	Views
Come utilizzare DeepSeek-V3 tramite API remota? Models	6	183	January 5, 2025
Triage `Gateway Time-out` Intermediate	1	166	November 20, 2025
Cannot create new endpoints: WebserverFailed Inference Endpoints on the Hub	1	797	November 30, 2023
Never ending inference until it reaches max context length Intermediate	4	136	February 12, 2026
Why the model provide an error response ever time Beginners	2	106	March 4, 2025

Problem with launching DeepSeek-R1-Distill-Qwen-32B-Uncensored-Q8_0-GGUF

In LangChain

Related topics