# LLama2 with accelerate issues

**URL:** <https://discuss.huggingface.co/t/llama2-with-accelerate-issues/56512>\
**Category:** 🤗Accelerate\
**Created:** [September 27, 2023, 4:35am UTC](https://discuss.huggingface.co/t/llama2-with-accelerate-issues/56512 "2023-09-27T04:35:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajash](https://avatars.discourse-cdn.com/v4/letter/a/53a042/32.png) [@ajash](https://discuss.huggingface.co/u/ajash)\
**Post date:** [September 27, 2023, 4:35am UTC](https://discuss.huggingface.co/t/llama2-with-accelerate-issues/56512/1 "2023-09-27T04:35:26Z")

</div>

MY machine has.4 A100 gpus & I am trying to train llama2-7b-hf using LORA. Here are some machine details  
nvcc --version (cuda version)  
nvcc: NVIDIA (R) Cuda compiler driver  
Copyright (c) 2005-2022 NVIDIA Corporation  
Built on Wed\_Jun\_\_8\_16:49:14\_PDT\_2022  
Cuda compilation tools, release 11.7, V11.7.99  
Build cuda\_11.7.r11.7/compiler.31442593\_0

python3 -c “import torch; print(torch. **version** )”. (pytorch version)  
2.0.1+cu117

When I run my trainer using accelerate (runs fine on single gpu) I get this error:  
Traceback (most recent call last):  
Traceback (most recent call last):  
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 81, in   
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 81, in   
run\_model()  
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 77, in run\_model  
run\_model()  
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 77, in run\_model  
model.train\_model(gradient\_accum\_steps=args.batch\_size,  
File “/home/paperspace/DigitalSynapse/models/reviews\_model.py”, line 97, in train\_model  
model.train\_model(gradient\_accum\_steps=args.batch\_size,  
File “/home/paperspace/DigitalSynapse/models/reviews\_model.py”, line 97, in train\_model  
model, optimizer, dataloader, lr\_scheduler = accelerator.prepare(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in prepare  
model, optimizer, dataloader, lr\_scheduler = accelerator.prepare(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in prepare  
result = tuple(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, in   
result = tuple(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, in   
self.\_prepare\_one(obj, first\_pass=True, device\_placement=d) for obj, d in zip(args, device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in \_prepare\_one  
self.\_prepare\_one(obj, first\_pass=True, device\_placement=d) for obj, d in zip(args, device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in \_prepare\_one  
return self.prepare\_model(obj, device\_placement=device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare\_model  
return self.prepare\_model(obj, device\_placement=device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare\_model  
model = torch.nn.parallel.DistributedDataParallel(model, \*\*kwargs)Traceback (most recent call last):

File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, in **init**  
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 81, in   
model = torch.nn.parallel.DistributedDataParallel(model, \*\*kwargs)  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, in **init**  
run\_model()  
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 77, in run\_model  
model.train\_model(gradient\_accum\_steps=args.batch\_size,  
File “/home/paperspace/DigitalSynapse/models/reviews\_model.py”, line 97, in train\_model  
model, optimizer, dataloader, lr\_scheduler = accelerator.prepare(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in prepare  
result = tuple(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, in   
self.\_prepare\_one(obj, first\_pass=True, device\_placement=d) for obj, d in zip(args, device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in \_prepare\_one  
\_sync\_module\_states(  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in \_sync\_module\_states  
\_sync\_module\_states(  
return self.prepare\_model(obj, device\_placement=device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in \_sync\_module\_states  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare\_model  
model = torch.nn.parallel.DistributedDataParallel(model, \*\*kwargs)  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, in **init**  
\_sync\_module\_states(  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in \_sync\_module\_states  
\_sync\_params\_and\_buffers(  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in \_sync\_params\_and\_buffers  
\_sync\_params\_and\_buffers(\_sync\_params\_and\_buffers(

File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in \_sync\_params\_and\_buffers  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in \_sync\_params\_and\_buffers  
dist.\_broadcast\_coalesced(  
dist.\_broadcast\_coalesced(dist.\_broadcast\_coalesced(

RuntimeErrorRuntimeErrorRuntimeError: : Invalid scalar type: Invalid scalar type  
Invalid scalar type

Traceback (most recent call last):  
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 81, in   
run\_model()  
File “/home/paperspace/DigitalSynapse/models/run\_models.py”, line 77, in run\_model  
model.train\_model(gradient\_accum\_steps=args.batch\_size,  
File “/home/paperspace/DigitalSynapse/models/reviews\_model.py”, line 97, in train\_model  
model, optimizer, dataloader, lr\_scheduler = accelerator.prepare(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in prepare  
result = tuple(  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, in   
self.\_prepare\_one(obj, first\_pass=True, device\_placement=d) for obj, d in zip(args, device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in \_prepare\_one  
return self.prepare\_model(obj, device\_placement=device\_placement)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare\_model  
model = torch.nn.parallel.DistributedDataParallel(model, \*\*kwargs)  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, in **init**  
\_sync\_module\_states(  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in \_sync\_module\_states  
\_sync\_params\_and\_buffers(  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in \_sync\_params\_and\_buffers  
dist.\_broadcast\_coalesced(  
RuntimeError: Invalid scalar type  
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local\_rank: 0 (pid: 3824) of binary: /usr/bin/python3.9  
Traceback (most recent call last):  
File “/home/paperspace/.local/bin/accelerate”, line 8, in   
sys.exit(main())  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/commands/accelerate\_cli.py”, line 45, in main  
args.func(args)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/commands/launch.py”, line 970, in launch\_command  
multi\_gpu\_launcher(args)  
File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/commands/launch.py”, line 646, in multi\_gpu\_launcher  
distrib\_run.run(args)  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/run.py”, line 785, in run  
elastic\_launch(  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 134, in **call**  
return launch\_agent(self.\_config, self.\_entrypoint, list(args))  
File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 250, in launch\_agent  
raise ChildFailedError(  
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

Please help. If this is not the correct place to ask questions it would be awesome if someone can point me in the correct direction

---

<div class="post-metadata">

**Author:** ![PaulQ1](https://avatars.discourse-cdn.com/v4/letter/p/dbc845/32.png) [@PaulQ1](https://discuss.huggingface.co/u/PaulQ1)\
**Post date:** [October 12, 2023, 2:16pm UTC](https://discuss.huggingface.co/t/llama2-with-accelerate-issues/56512/2 "2023-10-12T14:16:48Z")

</div>

Did you find a solution for this error?

---

<div class="post-metadata">

**Author:** ![ajash](https://avatars.discourse-cdn.com/v4/letter/a/53a042/32.png) [@ajash](https://discuss.huggingface.co/u/ajash)\
**Post date:** [October 16, 2023, 4:40am UTC](https://discuss.huggingface.co/t/llama2-with-accelerate-issues/56512/3 "2023-10-16T04:40:33Z")

</div>

not yet… do you have any ideas?

---

<div class="post-metadata">

**Author:** ![ajash](https://avatars.discourse-cdn.com/v4/letter/a/53a042/32.png) [@ajash](https://discuss.huggingface.co/u/ajash)\
**Post date:** [October 29, 2023, 7:11pm UTC](https://discuss.huggingface.co/t/llama2-with-accelerate-issues/56512/4 "2023-10-29T19:11:43Z")

</div>

I think I have finally figured this out. The error is because (this is my guess) the layer\_norm layer is getting sharded which makes it hard to compute the norms. Fixed this with a few lines of code:

```auto
from accelerate import Accelerator, dispatch_model, infer_auto_device_map
from accelerate.utils import get_balanced_memory
# These are the layer class names in the model code. Had to dig these up.
no_split_module_classes =
  ["LlamaDecoderLayer", "LlamaAttention", "Linear", "LlamaMLP", "LlamaRMSNorm"]
max_memory = get_balanced_memory(
      model,
      max_memory=None,
      no_split_module_classes=no_split_module_classes,
      dtype=torch.bfloat16,
    )
device_map = infer_auto_device_map(
      model=model,
      max_memory=max_memory,
      no_split_module_classes=no_split_module_classes,
      dtype=torch.bfloat16,
    )
model = dispatch_model(model, device_map=device_map)
# Also when calling accelerator.prepare do not pass model to it, like so:
optimizer, dataloader, lr_scheduler = accelerator.prepare(
      optimizer, dataloader, lr_scheduler
    )

```