# QLoRA Fine-tuning is Too Slow on LLaMA-based Model Despite BitsAndBytes Optimization

**URL:** <https://discuss.huggingface.co/t/qlora-fine-tuning-is-too-slow-on-llama-based-model-despite-bitsandbytes-optimization/166964>\
**Category:** Intermediate\
**Created:** [August 16, 2025, 10:05am UTC](https://discuss.huggingface.co/t/qlora-fine-tuning-is-too-slow-on-llama-based-model-despite-bitsandbytes-optimization/166964 "2025-08-16T10:05:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![AylinNaebzadeh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aylinnaebzadeh/32/52343_2.png) [@AylinNaebzadeh](https://discuss.huggingface.co/u/AylinNaebzadeh)\
**Post date:** [August 16, 2025, 10:05am UTC](https://discuss.huggingface.co/t/qlora-fine-tuning-is-too-slow-on-llama-based-model-despite-bitsandbytes-optimization/166964/1 "2025-08-16T10:05:35Z")

</div>

Hi everyone,

I’m fine-tuning a LLaMA-based model ([universitytehran/PersianMind-v1.0](https://huggingface.co/universitytehran/PersianMind-v1.0)) using **QLoRA** and **BitsAndBytes** in 4-bit precision. I am working with Kaggle GPU T4, and it takes about 75 hours to be fine-tuned using [ParsMap](https://www.kaggle.com/datasets/zahrarazaghi/parsmap/versions/1) dataset with 40,000 records for training related to converting informal to formal text.  
Here is my code:

```auto
base_model_id = "universitytehran/PersianMind-v1.0"
compute_dtype = torch.bfloat16 if torch.cuda.get_device_capability(0)[0] >= 8 else torch.float16

print("Compute dtype:", compute_dtype)

```

```auto
def safe_str(x):
    return "" if x is None or (isinstance(x, float) and np.isnan(x)) else str(x)

df = df_parsmap.copy()
df = df.dropna(subset=["inFormalForm","formalForm"]) # keep only rows with both sides

def make_text(row):
    informal = safe_str(row["inFormalForm"])
    formal = safe_str(row["formalForm"])
    return f"<s><|startoftext|>[Informal]{informal}[Formal]{formal}<|endoftext|>"

df["text"] = df.apply(make_text, axis=1)

```

```auto
perm = np.random.permutation(len(df))
cut = int(0.9*len(df))
train_df = df.iloc[perm[:cut]].reset_index(drop=True)
val_df = df.iloc[perm[cut:]].reset_index(drop=True)

ds = DatasetDict({
    "train": Dataset.from_pandas(train_df[["text"]]),
    "validation": Dataset.from_pandas(val_df[["text"]]),
})
len(ds["train"]), len(ds["validation"])

```

```auto

tokenizer = AutoTokenizer.from_pretrained(base_model_id, use_fast=True, trust_remote_code=True)

specials = {
    "bos_token": "<s>",
    "eos_token": "</s>",
    "pad_token": "<pad>",
}

for k,v in specials.items():
    if getattr(tokenizer, k, None) != v:
        tokenizer.add_special_tokens({k: v})

added = tokenizer.add_tokens(["<|startoftext|>", "<|endoftext|>", "[Informal]", "[Formal]", "<sep>"], special_tokens=True)
print("Added new tokens:", added)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

```

```auto
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    trust_remote_code=True,
    quantization_config=bnb_config,
    device_map="auto",
)

model.resize_token_embeddings(len(tokenizer))

model = prepare_model_for_kbit_training(model)
model.config.use_cache = False

```

```auto
lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.1, bias="none", task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
)
model = get_peft_model(model, lora_config)

model.gradient_checkpointing_enable()

# quick param report
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Trainable: {trainable:,} / Total: {total:,} ({100*trainable/total:.2f}%)")

```

```auto
max_length = 128

def tokenize_batch(batch):
    return tokenizer(
        batch["text"],
        truncation=True,
        max_length=max_length,
        padding="max_length",
    )

tokenized = ds.map(tokenize_batch, batched=True, remove_columns=ds["train"].column_names)

```

`collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)`

```auto
effective_bs = 16  
per_device_train_bs = 2
per_device_eval_bs = 2
grad_accum = max(1, effective_bs // per_device_train_bs)
epochs = 3

args = TrainingArguments(
    output_dir="./persianmind-formalizer-lora",
    num_train_epochs=epochs,
    per_device_train_batch_size=per_device_train_bs,
    per_device_eval_batch_size=per_device_eval_bs,
    gradient_accumulation_steps=grad_accum,
    learning_rate=1e-5,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    weight_decay=0.0,
    logging_steps=50,

    eva_strategy="steps",
    eval_steps=2000,                   
    save_strategy="epoch",             
    save_total_limit=2,
    load_best_model_at_end=True,

    bf16=(compute_dtype==torch.bfloat16),
    fp16=(compute_dtype==torch.float16),

    optim="paged_adamw_8bit",          
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},

    dataloader_num_workers=4,
    dataloader_pin_memory=True,
    dataloader_persistent_workers=True,

    group_by_length=True,              
    tf32=True,
    report_to="none",
)

```

```auto
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["validation"],
    data_collator=collator,
    tokenizer=tokenizer,
)

trainer.train()

```

Any insights or references to similar cases would be greatly appreciated!

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 16, 2025, 11:58am UTC](https://discuss.huggingface.co/t/qlora-fine-tuning-is-too-slow-on-llama-based-model-despite-bitsandbytes-optimization/166964/2 "2025-08-16T11:58:54Z")

</div>

> `tf32=True`

This wouldn’t work with T4 generation (Maybe Turing) GPUs. Using fp16 will allow you to take advantage of the hardware.

> `gradient_checkpointing=True,`  
> `gradient_checkpointing_kwargs={"use_reentrant": False},`

It saves VRAM but [slows down the training speed](https://huggingface.co/docs/transformers/v4.53.3/en/perf_train_gpu_one).

> `target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],`

As the number of layers to be trained increases, the amount of computation will likely increase, causing the process to slow down.

[With shorter sentences, `packing=True` may be effective](https://huggingface.co/docs/trl/en/sft_trainer#packing). If you want faster trainer, [try optimized version](https://huggingface.co/blog/unsloth-trl).

---

<div class="post-metadata">

**Author:** ![AylinNaebzadeh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aylinnaebzadeh/32/52343_2.png) [@AylinNaebzadeh](https://discuss.huggingface.co/u/AylinNaebzadeh)\
**Post date:** [August 16, 2025, 1:52pm UTC](https://discuss.huggingface.co/t/qlora-fine-tuning-is-too-slow-on-llama-based-model-despite-bitsandbytes-optimization/166964/3 "2025-08-16T13:52:20Z")

</div>

Thank you!  
I was able to decrease the time to 23 hours instead of 75 hours!  
Which `target_modules` do you suggest to train?  
I’ve tried a lot to use `SFTTrainer` but all the time it raise an error due to versioning and then `CUDA out of memory..`

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 16, 2025, 11:29pm UTC](https://discuss.huggingface.co/t/qlora-fine-tuning-is-too-slow-on-llama-based-model-despite-bitsandbytes-optimization/166964/4 "2025-08-16T23:29:04Z")

</div>

I don’t know [Kaggle’s etiquette](https://www.kaggle.com/code/charankancheti/fine-tuning)…  
Is it like this?

```py
pip install -U --no-cache-dir \
  "trl==0.18.2" \
  "transformers==4.52.3" \
  "datasets>=2.20.0" \
  "accelerate>=1.2.0" \
  "peft>=0.16.0" \
  "huggingface_hub>=0.23.0" \
  "safetensors>=0.4.3" \
  "bitsandbytes==0.43.1"
python - <<'PY'
import IPython; IPython.Application.instance().kernel.do_shutdown(True)
PY

```

> Which `target_modules` do you suggest to train?

`target_modules=["q_proj","k_proj","v_proj","o_proj"],`  
I think many people do this. Is it the attention module? It feels like fine-tuning only that part.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [August 17, 2025, 11:29am UTC](https://discuss.huggingface.co/t/qlora-fine-tuning-is-too-slow-on-llama-based-model-despite-bitsandbytes-optimization/166964/5 "2025-08-17T11:29:35Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.