# Initializing a big model on GPU with random weights

**URL:** <https://discuss.huggingface.co/t/initializing-a-big-model-on-gpu-with-random-weights/135634>\
**Category:** 🤗Transformers\
**Created:** [January 14, 2025, 10:16am UTC](https://discuss.huggingface.co/t/initializing-a-big-model-on-gpu-with-random-weights/135634 "2025-01-14T10:16:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![j-klesen](https://avatars.discourse-cdn.com/v4/letter/j/df788c/32.png) [@j-klesen](https://discuss.huggingface.co/u/j-klesen)\
**Post date:** [January 14, 2025, 10:16am UTC](https://discuss.huggingface.co/t/initializing-a-big-model-on-gpu-with-random-weights/135634/1 "2025-01-14T10:16:01Z")

</div>

Hi all 🤗

to test and debug our PEFT script for very large models (405B), we want to circumvent downloading the whole model on every rerun, in order to reduce experimentation time (and thus, cost). Downloading once and storing is not an option, unfortunately.

To this end, we want to have the model loaded as if it was properly initialized (using e.g. from\_pretrained), i.e. with the correct dtype and distributed across the gpus/nodes, and so on, just with random weights for now, and skipping the download.

When using something like

```auto
mconfig = AutoConfig.from_pretrained(train_config.model_name)
model = LlamaForCausalLM(mconfig)

```

the model gets loaded to CPU and the RAM runs out and we get an OOMKilled error.

So, ideally, we would want to use something like

```auto
model = LlamaForCausalLM.from_pretrained(
    train_config.model_name, \
    device_map= "auto", \
    torch_dtype=torch.float16, # or torch.bfloat16,
    random_weights=True
)

```

since the `from_pretrained` function seems to handle all the complexities we need.

Is there any easy way to do this? We checked the parameters of from\_pretrained, but did not find something applicable.

Thank you for any ideas,  
Jonas

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 14, 2025, 10:37am UTC](https://discuss.huggingface.co/t/initializing-a-big-model-on-gpu-with-random-weights/135634/2 "2025-01-14T10:37:43Z")

</div>

device=“meta”?

> **[How 🤗 Accelerate runs very large models thanks to PyTorch](https://huggingface.co/blog/accelerate-large-models)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![j-klesen](https://avatars.discourse-cdn.com/v4/letter/j/df788c/32.png) [@j-klesen](https://discuss.huggingface.co/u/j-klesen)\
**Post date:** [January 14, 2025, 10:57am UTC](https://discuss.huggingface.co/t/initializing-a-big-model-on-gpu-with-random-weights/135634/3 "2025-01-14T10:57:28Z")

</div>

Hi @John6666 ,  
thanks for the suggestion! But the meta device doesn’t quite cut it for us.

We need to have real weights on the devices, because we e.g. also want to do forward/backward passes and measure time taken, memory load, etc.

Any other ideas?

Best,  
Jonas