# Help with merging LoRA to base model

**URL:** <https://discuss.huggingface.co/t/help-with-merging-lora-to-base-model/151842>\
**Category:** Beginners\
**Created:** [April 22, 2025, 6:56pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-to-base-model/151842 "2025-04-22T18:56:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Unnati24](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/unnati24/32/39158_2.png) [@Unnati24](https://discuss.huggingface.co/u/Unnati24)\
**Post date:** [April 22, 2025, 6:56pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-to-base-model/151842/1 "2025-04-22T18:56:55Z")

</div>

Hi, i have fine tuned llama 8b instruct model by QLoRA (loaded base model in 4bit followed by Lora) and saved trained model. im using google colab T4(15gb) gpu. when i load base model completely on gpu (device\_map=“cuda:0”) in float 16 for merging then i get _Out Of Memory error_  
if i use following code i get _NotImplementedError: Cannot copy out of meta tensor; no data!_  
Is there any solution?

`tokenizer = AutoTokenizer.from\_pretrained(NEW\_MODEL)

model = AutoModelForCausalLM.from\_pretrained(  
MODEL\_NAME,  
torch\_dtype=torch.float16,  
device\_map=“auto”,  
offload\_folder=OFFLOAD\_DIR,  
token = HF\_TOKEN  
)

model.resize\_token\_embeddings(len(tokenizer), pad\_to\_multiple\_of=8)  
model = PeftModel.from\_pretrained(model,NEW\_MODEL,offload\_folder=OFFLOAD\_DIR)  
model = model.merge\_and\_unload()`

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 23, 2025, 7:08am UTC](https://discuss.huggingface.co/t/help-with-merging-lora-to-base-model/151842/2 "2025-04-23T07:08:10Z")

</div>

When using bitsandbytes, I think it is better not to use “auto” (or rather, accelerate) to avoid errors.

```py
#device_map=“auto”,

```

> <https://github.com/huggingface/transformers/issues/26700>
>
> \### System Info
> 
> \- \`transformers\` version: 4.34.0
> \- Platform: Linux-5.15.0-86-g…eneric-x86\_64-with-glibc2.31
> \- Python version: 3.11.6
> \- Huggingface\_hub version: 0.17.3
> \- Safetensors version: 0.4.0
> \- Accelerate version: 0.24.0.dev0
> \- Accelerate config: not found
> \- PyTorch version (GPU?): 2.1.0+cu121 (True)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- GPU: RTX2060 6G
> 
> \### Who can help?
> 
> @Narsil
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[x\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[x\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> here is my code below:
> \`\`\`
> def ndarray\_to\_image(ndarray):
> return Image.fromarray(np.uint8(ndarray))
> import cv2
> from transformers import pipeline
> from PIL import Image
> import numpy as np
> cap = cv2.VideoCapture(0)
> 
> while True:
> ret, frame = cap.read()
> 
> cv2.imshow('frame', frame)
> 	
> image = ndarray\_to\_image(frame)
> 
> pipe = pipeline("object-detection", model="facebook/detr-resnet-50", device\_map="auto")
> result = pipe(image)
> print(result)
> 
> if cv2.waitKey(1) & 0xFF == ord('q'):
> break
> 
> cap.release()
> cv2.destroyAllWindows()
> \`\`\`
> when set pipeline(device\_map="auto") will raise an error:
> \`\`\`
> {
> "name": "NotImplementedError",
> "message": "Cannot copy out of meta tensor; no data!",
> "stack": "---------------------------------------------------------------------------
> NotImplementedError Traceback (most recent call last)
> /home/yongjer/程式/object detection/main.ipynb 儲存格 1 line 1
> \<a href='vscode-notebook-cell:/home/yongjer/%E7%A8%8B%E5%BC%8F/object%20detection/main.ipynb#W6sZmlsZQ%3D%3D?line=11'\>12\</a\> cv2.imshow('frame', frame)
> \<a href='vscode-notebook-cell:/home/yongjer/%E7%A8%8B%E5%BC%8F/object%20detection/main.ipynb#W6sZmlsZQ%3D%3D?line=13'\>14\</a\> image = ndarray\_to\_image(frame)
> \---\> \<a href='vscode-notebook-cell:/home/yongjer/%E7%A8%8B%E5%BC%8F/object%20detection/main.ipynb#W6sZmlsZQ%3D%3D?line=15'\>16\</a\> pipe = pipeline(\\"object-detection\\", model=\\"facebook/detr-resnet-50\\", device\_map=\\"auto\\")
> \<a href='vscode-notebook-cell:/home/yongjer/%E7%A8%8B%E5%BC%8F/object%20detection/main.ipynb#W6sZmlsZQ%3D%3D?line=16'\>17\</a\> result = pipe(image)
> \<a href='vscode-notebook-cell:/home/yongjer/%E7%A8%8B%E5%BC%8F/object%20detection/main.ipynb#W6sZmlsZQ%3D%3D?line=17'\>18\</a\> print(result)
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/pipelines/\_\_init\_\_.py:834, in pipeline(task, model, config, tokenizer, feature\_extractor, image\_processor, framework, revision, use\_fast, token, device, device\_map, torch\_dtype, trust\_remote\_code, model\_kwargs, pipeline\_class, \*\*kwargs)
> 832 if isinstance(model, str) or framework is None:
> 833 model\_classes = {\\"tf\\": targeted\_task\[\\"tf\\"\], \\"pt\\": targeted\_task\[\\"pt\\"\]}
> \--\> 834 framework, model = infer\_framework\_load\_model(
> 835 model,
> 836 model\_classes=model\_classes,
> 837 config=config,
> 838 framework=framework,
> 839 task=task,
> 840 \*\*hub\_kwargs,
> 841 \*\*model\_kwargs,
> 842 )
> 844 model\_config = model.config
> 845 hub\_kwargs\[\\"\_commit\_hash\\"\] = model.config.\_commit\_hash
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/pipelines/base.py:269, in infer\_framework\_load\_model(model, config, model\_classes, task, framework, \*\*model\_kwargs)
> 263 logger.warning(
> 264 \\"Model might be a PyTorch model (ending with \`.bin\`) but PyTorch is not available. \\"
> 265 \\"Trying to load the model with Tensorflow.\\"
> 266 )
> 268 try:
> \--\> 269 model = model\_class.from\_pretrained(model, \*\*kwargs)
> 270 if hasattr(model, \\"eval\\"):
> 271 model = model.eval()
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/models/auto/auto\_factory.py:565, in \_BaseAutoModelClass.from\_pretrained(cls, pretrained\_model\_name\_or\_path, \*model\_args, \*\*kwargs)
> 563 elif type(config) in cls.\_model\_mapping.keys():
> 564 model\_class = \_get\_model\_class(config, cls.\_model\_mapping)
> \--\> 565 return model\_class.from\_pretrained(
> 566 pretrained\_model\_name\_or\_path, \*model\_args, config=config, \*\*hub\_kwargs, \*\*kwargs
> 567 )
> 568 raise ValueError(
> 569 f\\"Unrecognized configuration class {config.\_\_class\_\_} for this kind of AutoModel: {cls.\_\_name\_\_}.\\
> \\"
> 570 f\\"Model type should be one of {', '.join(c.\_\_name\_\_ for c in cls.\_model\_mapping.keys())}.\\"
> 571 )
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/modeling\_utils.py:3085, in PreTrainedModel.from\_pretrained(cls, pretrained\_model\_name\_or\_path, config, cache\_dir, ignore\_mismatched\_sizes, force\_download, local\_files\_only, token, revision, use\_safetensors, \*model\_args, \*\*kwargs)
> 3082 config = cls.\_check\_and\_enable\_flash\_attn\_2(config, torch\_dtype=torch\_dtype, device\_map=device\_map)
> 3084 with ContextManagers(init\_contexts):
> \-\> 3085 model = cls(config, \*model\_args, \*\*model\_kwargs)
> 3087 # Check first if we are \`from\_pt\`
> 3088 if use\_keep\_in\_fp32\_modules:
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/models/detr/modeling\_detr.py:1498, in DetrForObjectDetection.\_\_init\_\_(self, config)
> 1495 super().\_\_init\_\_(config)
> 1497 # DETR encoder-decoder model
> \-\> 1498 self.model = DetrModel(config)
> 1500 # Object detection heads
> 1501 self.class\_labels\_classifier = nn.Linear(
> 1502 config.d\_model, config.num\_labels + 1
> 1503 ) # We add one for the \\"no object\\" class
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/models/detr/modeling\_detr.py:1330, in DetrModel.\_\_init\_\_(self, config)
> 1327 super().\_\_init\_\_(config)
> 1329 # Create backbone + positional encoding
> \-\> 1330 backbone = DetrConvEncoder(config)
> 1331 object\_queries = build\_position\_encoding(config)
> 1332 self.backbone = DetrConvModel(backbone, object\_queries)
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/models/detr/modeling\_detr.py:361, in DetrConvEncoder.\_\_init\_\_(self, config)
> 359 # replace batch norm by frozen batch norm
> 360 with torch.no\_grad():
> \--\> 361 replace\_batch\_norm(backbone)
> 362 self.model = backbone
> 363 self.intermediate\_channel\_sizes = (
> 364 self.model.feature\_info.channels() if config.use\_timm\_backbone else self.model.channels
> 365 )
> 
> File ~/miniforge3/envs/od/lib/python3.11/site-packages/transformers/models/detr/modeling\_detr.py:319, in replace\_batch\_norm(model)
> 316 if isinstance(module, nn.BatchNorm2d):
> 317 new\_module = DetrFrozenBatchNorm2d(module.num\_features)
> \--\> 319 new\_module.weight.data.copy\_(module.weight)
> 320 new\_module.bias.data.copy\_(module.bias)
> 321 new\_module.running\_mean.data.copy\_(module.running\_mean)
> 
> NotImplementedError: Cannot copy out of meta tensor; no data!"
> }
> \`\`\`
> 
> \### Expected behavior
> 
> when set device=0 rather than device\_map = "auto", it works

> <https://github.com/huggingface/transformers/issues/31104>
>
> When I fine tuning llama2 with deepspeed and qlora on one node and multi GPUs, I… used zero3 to partition the model paramters, but it always first load the whole params on each GPU and partition params just before training instead load params after partition it. After I check the huggingface document, I find it need to put \`TrainingArguments\` before \`from\_pretrained\`. I did it and zero3\_init indeed wored, but the confusing problem arised: NotImplementedError: Cannot copy out of meta tensor; no data! 
> 
> Here is my code:
> \`\`\`python
> from datasets import load\_dataset
> import torch
> from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer, TrainingArguments
> import bitsandbytes as bnb
> from peft import LoraConfig
> from trl import SFTTrainer
> 
> from accelerate import Accelerator
> accelerator = Accelerator()
> 
> import deepspeed
> 
> dataset = load\_dataset("json",data\_files="Belle\_open\_source\_0.5M\_changed.json",split="train")
> 
> result\_dir = "tmp"
> training\_args = TrainingArguments(
> report\_to="none",
> output\_dir=result\_dir, 
> # per\_device\_train\_batch\_size \* gradient\_accumulation\_steps = batch\_size
> per\_device\_train\_batch\_size=1, 
> gradient\_accumulation\_steps=16,
> learning\_rate=2e-4, 
> logging\_steps=10,
> # max\_steps=520,
> num\_train\_epochs=0.016, 
> save\_steps=500, # 65
> bf16 = True, # set bf16 to True with an A100
> # optim='paged\_adamw\_32bit',
> gradient\_checkpointing=True,
> # group\_by\_length=True,
> # remove\_unused\_columns=False,
> # warmup\_ratio=0.03,
> # lr\_scheduler\_type='constant',
> # max\_grad\_norm=0.3
> )
> 
> current\_device = Accelerator().process\_index
> print("current\_device:", current\_device)
> \# print(type(current\_device))
> 
> base\_model\_name ="/home/yangtong/data/llama2-hf/llama2-13b-chat\_hf"
> bnb\_config = BitsAndBytesConfig(
> load\_in\_4bit=True, 
> bnb\_4bit\_use\_double\_quant=True,
> bnb\_4bit\_quant\_type="nf4", 
> bnb\_4bit\_compute\_dtype=torch.bfloat16, 
> # bnb\_4bit\_quant\_storage=torch.bfloat16
> )
> 
> base\_model = AutoModelForCausalLM.from\_pretrained(
> base\_model\_name, 
> quantization\_config=bnb\_config, 
> torch\_dtype=torch.bfloat16,
> load\_in\_4bit=True
> )
> base\_model.tie\_weights()
> base\_model.config.use\_cache = False
> base\_model.config.pretraining\_tp = 1
> 
> 
> def find\_all\_linear\_names(model):
> cls = bnb.nn.Linear4bit
> lora\_module\_names = set()
> for name, module in model.named\_modules():
> if isinstance(module, cls):
> names = name.split('.')
> lora\_module\_names.add(names\[0\] if len(names) == 1 else names\[-1\])
> if 'lm\_head' in lora\_module\_names: # needed for 16-bit
> lora\_module\_names.remove('lm\_head')
> return list(lora\_module\_names)
> models=find\_all\_linear\_names(base\_model)
> \# print(models)
> 
> peft\_config = LoraConfig(
> lora\_alpha=16,
> lora\_dropout=0.1,
> r=64,
> bias="none",
> task\_type="CAUSAL\_LM",
> target\_modules=models
> )
> 
> tokenizer = AutoTokenizer.from\_pretrained(base\_model\_name, trust\_remote\_code=True)
> tokenizer.deprecation\_warnings\["Asking-to-pad-a-fast-tokenizer"\] = True
> tokenizer.pad\_token = tokenizer.eos\_token
> 
> max\_seq\_length = 512
>           
> trainer = SFTTrainer(
> model=base\_model,
> train\_dataset=dataset,
> peft\_config=peft\_config,
> dataset\_text\_field="text",
> max\_seq\_length=max\_seq\_length,
> tokenizer=tokenizer,
> args=training\_args
> )
> 
> trainer.train()
> 
> output\_dir = os.path.join(result\_dir, "final\_checkpoint")
> trainer.model.save\_pretrained(output\_dir)
> \`\`\` 
> 
> Here is my accelerate config:
> \`\`\`
> compute\_environment: LOCAL\_MACHINE
> debug: false
> deepspeed\_config:
> deepspeed\_config\_file: /home/yangtong/ft\_dis/ds\_config/3.json
> zero3\_init\_flag: true
> distributed\_type: DEEPSPEED
> downcast\_bf16: 'no'
> enable\_cpu\_affinity: false
> machine\_rank: 0
> main\_training\_function: main
> num\_machines: 1
> num\_processes: 4
> rdzv\_backend: 'c10d'
> same\_network: true
> tpu\_env: \[\]
> tpu\_use\_cluster: false
> tpu\_use\_sudo: false
> use\_cpu: false
> 
> \`\`\`
> 
> And here is error:
> \`\`\`
> Traceback (most recent call last):
> File "/home/yangtong/ft\_dis/ft\_acc\_new.py", line 58, in \<module\>
> base\_model = AutoModelForCausalLM.from\_pretrained(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/transformers/models/auto/auto\_factory.py", line 565, in from\_pretrained
> return model\_class.from\_pretrained(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/transformers/modeling\_utils.py", line 3307, in from\_pretrained
> ) = cls.\_load\_pretrained\_model(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/transformers/modeling\_utils.py", line 3695, in \_load\_pretrained\_model
> new\_error\_msgs, offload\_index, state\_dict\_index = \_load\_state\_dict\_into\_meta\_model(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/transformers/modeling\_utils.py", line 749, in \_load\_state\_dict\_into\_meta\_model
> set\_module\_quantized\_tensor\_to\_device(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/transformers/integrations/bitsandbytes.py", line 108, in set\_module\_quantized\_tensor\_to\_device
> new\_value = value.to(device)
> NotImplementedError: Cannot copy out of meta tensor; no data!
> \`\`\` 
> 
> After I set \`low\_cpu\_mem\_usage=False\` in \`from\_pretrained\`, here is another error:
> \`\`\` 
> Traceback (most recent call last):
> File "/home/yangtong/ft\_dis/ft\_acc\_new.py", line 58, in \<module\>
> base\_model = AutoModelForCausalLM.from\_pretrained(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/transformers/models/auto/auto\_factory.py", line 565, in from\_pretrained
> return model\_class.from\_pretrained(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/transformers/modeling\_utils.py", line 3366, in from\_pretrained
> dispatch\_model(model, \*\*device\_map\_kwargs)
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/accelerate/big\_modeling.py", line 419, in dispatch\_model
> attach\_align\_device\_hook\_on\_blocks(
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/accelerate/hooks.py", line 608, in attach\_align\_device\_hook\_on\_blocks
> add\_hook\_to\_module(module, hook)
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/accelerate/hooks.py", line 157, in add\_hook\_to\_module
> module = hook.init\_hook(module)
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/accelerate/hooks.py", line 275, in init\_hook
> set\_module\_tensor\_to\_device(module, name, self.execution\_device, tied\_params\_map=self.tied\_params\_map)
> File "/home/yangtong/anaconda3/envs/llama2/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 354, in set\_module\_tensor\_to\_device
> raise ValueError(f"{tensor\_name} is on the meta device, we need a \`value\` to put in on {device}.")
> ValueError: weight is on the meta device, we need a \`value\` to put in on 0.
> \`\`\` 
> 
> I also try to set \`empty\_init=False\` , but the error is LlamaForCausalLM.from\_pretrained doesn't has this paramter.
> 
> I will truly appreciate if anyone can help me solve it !