# 2B Model Fill Up Memory Usage on 4xA100s

**URL:** <https://discuss.huggingface.co/t/2b-model-fill-up-memory-usage-on-4xa100s/149680>\
**Category:** 🤗Transformers\
**Created:** [April 9, 2025, 10:29pm UTC](https://discuss.huggingface.co/t/2b-model-fill-up-memory-usage-on-4xa100s/149680 "2025-04-09T22:29:26Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenatCambridge](https://avatars.discourse-cdn.com/v4/letter/b/e19adc/32.png) [@BenatCambridge](https://discuss.huggingface.co/u/BenatCambridge)\
**Post date:** [April 9, 2025, 10:29pm UTC](https://discuss.huggingface.co/t/2b-model-fill-up-memory-usage-on-4xa100s/149680/1 "2025-04-09T22:29:26Z")

</div>

With 4xA100s, I am training a 2B model (Gemma-2-2b-it) to perform SequenceClassification on a custom dataset. My code is quite boilerplate, but what I don’t understand is why would model training fill up the GPU memory? As I see it, With a model like gemma-2b, the total number of parameters is 2B. If we assume full precision training, i.e. float32, then the total GPU memory required to hold the model is 2B\*(32/8=4 bytes/parameter) = 8GB of memory. When the model undergoes training, there is also optimizer states, gradients, and optimizer intermediates that add into the total memory consumption. Generally, the GPU memory required to train a 2B model at full precision is ~5\*8 = 40GB of GPU memory. If my understanding is correct, then how can the model nearly take up ~250GB of GPU memory? FYI I use `accelerate` with `DeepSpeed ZeRO 3`

```auto
with open("all.json", 'r') as f:
    all_data = json.load(f)
dataset = load_dataset('json', data_files={'train': 'debug.json', 'test': 'test.json'})
classes = list(set(d['label'] for d in all_data))
id2label = {i: str(c) for i, c in enumerate(classes)}
label2id = {str(c): i for i, c in enumerate(classes)}

model_path = "gemma2"

tokenizer = AutoTokenizer.from_pretrained(model_path)

def preprocess_function(example):

   labels = [1. if classes[i] == str(example["label"]) else 0. for i in range(len(classes))]

   example = tokenizer(example["text"], truncation=True)
   example['label'] = labels
   return example

tokenized_dataset = dataset.map(preprocess_function)

data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

model = AutoModelForSequenceClassification.from_pretrained(
    model_path,
    num_labels=len(classes),
    id2label=id2label,
    label2id=label2id,
    problem_type = "multi_label_classification"
)

training_args = TrainingArguments(

   output_dir="model",
   per_device_train_batch_size=1,
   per_device_eval_batch_size=1,
   gradient_accumulation_steps=4,
   num_train_epochs=5,
   weight_decay=0.01,
   eval_strategy="epoch",
   save_strategy="epoch",
   logging_strategy="steps",
   logging_steps=100
)

trainer = Trainer(

   model=model,
   args=training_args,
   train_dataset=tokenized_dataset["train"],
   eval_dataset=tokenized_dataset["test"],
   tokenizer=tokenizer,
   data_collator=data_collator
)

trainer.train()

```

```auto
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100 80GB PCIe On | 00000001:00:00.0 Off | 0 |
| N/A 42C P0 79W / 300W | 74815MiB / 81920MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A100 80GB PCIe On | 00000002:00:00.0 Off | 0 |
| N/A 38C P0 69W / 300W | 81031MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA A100 80GB PCIe On | 00000003:00:00.0 Off | 0 |
| N/A 39C P0 81W / 300W | 54311MiB / 81920MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA A100 80GB PCIe On | 00000004:00:00.0 Off | 0 |
| N/A 40C P0 81W / 300W | 54311MiB / 81920MiB | 100% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 561923 C /home/azureuser/miniconda/bin/python 74806MiB |
| 1 N/A N/A 561924 C /home/azureuser/miniconda/bin/python 81022MiB |
| 2 N/A N/A 561925 C /home/azureuser/miniconda/bin/python 54302MiB |
| 3 N/A N/A 561926 C /home/azureuser/miniconda/bin/python 54302MiB |
+-----------------------------------------------------------------------------------------+

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 10, 2025, 3:48am UTC](https://discuss.huggingface.co/t/2b-model-fill-up-memory-usage-on-4xa100s/149680/2 "2025-04-10T03:48:04Z")

</div>

Could it be that DeepSpeed has a bug…?

> [@DeepSpeed Zero causes intermittent GPU usage](https://discuss.huggingface.co/t/deepspeed-zero-causes-intermittent-gpu-usage/132138/2):
>
> I am not sure how to explain that but I am finally getting much better results with DeepSpeed. The recipe that worked for me: use the example configs provided in [transformers/tests/deepspeed at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/main/tests/deepspeed) use fused AdamW (I am using Pytorch implem.) use gradient checkpointing together with DeepSpeed Zero3 updating the config to the one I linked didn’t bring any improvement alone adding fused AdamW brought some improvement adding gradient checkpointing made the…

> **[Poor GPU utilization when using multi-node ZeRO-3 on small model ·...](https://github.com/deepspeedai/DeepSpeed/discussions/5488)**
>
> After publishing Fietje, a continue pretrain of phi-2 (2.7B parameters), I got some questions about training efficiency. People rightly pointed out that GPU utilization is low of around 50%, despit...

> <https://github.com/deepspeedai/DeepSpeed/issues/4527>
>
> \*\*Describe the bug\*\*
> I run deepspeed.runtime.zero.stage3.estimate\_zero3\_model\_s…tates\_mem\_needs\_all\_live(). It says in stage 3, with offload\_optimizer and offload\_param enabled, it requires 0.49GB of GPU memory for params, optim states and gradients
> 
> \`\`\`
> Estimated memory needed for params, optim states and gradients for a:
> HW: Setup with 1 node, 4 GPUs per node.
> SW: Model with 6805M total params, 131M largest layer params.
> per CPU | per GPU | Options
> 171.13GB | 0.49GB | offload\_param=OffloadDeviceEnum.cpu, offload\_optimizer=OffloadDeviceEnum.cpu, zero\_init=1
> 171.13GB | 0.49GB | offload\_param=OffloadDeviceEnum.cpu, offload\_optimizer=OffloadDeviceEnum.cpu, zero\_init=0
> 152.11GB | 3.66GB | offload\_param=none, offload\_optimizer=OffloadDeviceEnum.cpu, zero\_init=1
> 152.11GB | 3.66GB | offload\_param=none, offload\_optimizer=OffloadDeviceEnum.cpu, zero\_init=0
> 2.93GB | 29.01GB | offload\_param=none, offload\_optimizer=none, zero\_init=1
> 152.11GB | 29.01GB | offload\_param=none, offload\_optimizer=none, zero\_init=0
> \`\`\`
> 
> But in training, the GPU memory usage is 14,678MiB after \`deepspeed.initialize()\`, 19,890MiB after the first forward \`engine(input\_token)\`, 20,148MiB after the first backward \`engine.backward(loss)\` and \`engine.step()\`, 1,838MiB before the second forward, 21,272MiB after the second forward. And crashes during the second backward for backwarding loss for twice, I report this in another issue. \[https://github.com/microsoft/DeepSpeed/issues/4528\](url)
> 
> I obtained the memory usage through \`nvidia-smi\`. The above results are obtained on 1x RTX 3090ti(24G).
> 
> My question is:
> 1. Why deepspeed uses 14678MiB memory after \`deepspeed.initialize()\`. It casues oom in the first forward on 4x P100(16G). I primarily train my model on 4x P100, RTX 3090ti is just for testing.
> 2. How does DeepSpeed utilize GPU memory? I have enabled cpu offload for parameters and optimizer, the 1,838MiB of memory usage between the first backward and the second forward, I speculate, is due to the parameters and optimizer being offloaded to the CPU after which the parameters and optimizer are accounted for. Then, is the additional approximately 20,000MiB after the first forward all activations? If so, I have already enabled activation checkpointing, but the memory usage is still high.
> 3. What is the difference between \`zero\_init=1\` and \`zero\_init=0\`. Why is there such a big difference in CPU memory usage between setting \`zero\_init\` to 1 and 0 when not using \`offload\_optimizer\` and \`offload\_param\`? How to set \`zero\_init\`=1 or 0? I did not find any description regarding \`zero\_init\` in the documentation.
> 4. Is there any chance I could continue my training on 4 or 8 P100 with 16384MiB memory? I'm finetuning llama2 with 6,700,000,000 of parameter with LoRA. Num of trainable parameters is 67,329,792 (1% of llama2).
> 
> \*\*To Reproduce\*\*
> Code of llama2 model can be found here \[https://github.com/facebookresearch/llama/blob/main/llama/model.py\](url), I implemented LoRA myself, each lora linear has different LoRA rank. Each linear in my model is replaced to 
> \`\`\`
> class Adaptive\_Linear(torch.nn.Module):
> def \_\_init\_\_(
> self,
> model\_arg: dict, # batch size, max seqlen, vocab size, etc.
> name: str, # example: layer\_list.0.attention.key
> lora\_rank\_dict: dict, # lora rank of each linear
> in\_features: int,
> out\_features: int 
> ):
> super().\_\_init\_\_()
> 
> self.name = name
> self.model\_arg = model\_arg
> 
> self.linear = torch.nn.Linear(in\_features, out\_features, bias = False)
> 
> self.lora\_rank = lora\_rank\_dict\[self.name\]
> if self.lora\_rank \> 0:
> self.lora\_dropout = torch.nn.Dropout(model\_arg.lora\_dropout)
> self.lora\_a\_linear = torch.nn.Linear(in\_features, self.lora\_rank, bias = False)
> self.lora\_b\_linear = torch.nn.Linear(self.lora\_rank, out\_features, bias = False)
> 
> self.lora\_a\_linear.weight.data = torch.rand\_like(self.lora\_a\_linear.weight.data) - 1 \* 2
> self.lora\_b\_linear.weight.data = torch.zeros\_like(self.lora\_b\_linear.weight.data)
> 
> def trainable\_parameters(self):
> # only train lora matrix
> # the trainable parameters of my model are trainable\_parameters
> # of all Adaptive\_Linear connected together using itertools.chain.
> return iter((self.lora\_a\_linear.weight, self.lora\_b\_linear.weight))
> 
> def forward(self, x):
> hidden = self.linear(x)
> 
> if self.lora\_rank \> 0:
> input = self.lora\_dropout(x)
> lora\_a = self.lora\_a\_linear(input)
> lora\_b = self.lora\_b\_linear(lora\_a)
> output = hidden + lora\_b
> else:
> output = hidden
> 
> return output
> \`\`\`
> 
> My dataset is
> \`\`\`
> class Dataset(torch.utils.data.Dataset):
> def \_\_init\_\_(self, model\_arg, dataset\_path):
> self.model\_arg = model\_arg
> 
> self.dataset\_path = dataset\_path
> self.dataset = torch.load(self.dataset\_path) # List\[Tuple\] list of tuples of (input token list, output token list, label) token and label are int
> 
> self.dataset = \[data\_item for data\_item in self.dataset if len(data\_item\[0\]) \>= MIN\_SEQ\_LEN and len(data\_item\[0\]) \<= MAX\_SEQ\_LEN\]
> 
> def \_\_len\_\_(self):
> return len(self.dataset)
> 
> def \_\_getitem\_\_(self, index):
> input\_token, output\_token, label = self.dataset
> 
> return torch.tensor(input\_token), torch.tensor(output\_token), torch.tensor(label)
> \`\`\`
> 
> Collate function to pad tokens of a batch to same length
> \`\`\`
> collate\_fn(batch):
> input\_token\_list, output\_token\_list, label\_list = zip(\*batch)
> 
> padded\_input\_token = torch.nn.utils.rnn.pad\_sequence(\[seq for seq in input\_token\_list\], batch\_first = True)
> padded\_output\_token = torch.nn.utils.rnn.pad\_sequence(\[seq for seq in output\_token\_list\], batch\_first = True)
> 
> return padded\_input\_token, padded\_output\_token, label\_list
> \`\`\`
> 
> Initialize model and deepspeed
> \`\`\`
> model = Llama\_2\_With\_Adaptive\_Lora(model\_arg, lora\_rank\_dict)
> model.load\_state\_dict(state\_dict)
> for name, parameter in model.named\_parameters():
> if not ('lora' in name):
> parameter.requires\_grad = False
> 
> engine, optimizer, train\_dataloader, \_= deepspeed.initialize(
> model = model,
> model\_parameters = model.trainable\_parameters(),
> training\_data = train\_dataset,
> args = args, # local\_rank only
> collate\_fn = collate\_fn,
> config = deepspeed\_config
> )
> \`\`\`
> 
> Deepspeed config
> \`\`\`
> {
> "train\_micro\_batch\_size\_per\_gpu": 1,
> "gradient\_accumulation\_steps": 8,
> "gradient\_clipping": 1.0,
> "steps\_per\_print": 10,
> "wall\_clock\_breakdown": false,
> "memory\_breakdown": false,
> "optimizer": {
> "type": "AdamW",
> "params": {
> "lr": 2e-5,
> "betas": \[
> 0.9,
> 0.95
> \],
> "eps": 1e-8,
> "weight\_decay": 1e-4
> }
> },
> "scheduler": {
> "type": "WarmupDecayLR",
> "params": {
> "warmup\_min\_lr": 0,
> "warmup\_max\_lr": 2e-5,
> "warmup\_num\_steps": 1000,
> "total\_num\_steps": 10000
> }
> },
> "bf16": {
> "enabled": true,
> "loss\_scale": 0,
> "loss\_scale\_window": 1000,
> "initial\_scale\_power": 16,
> "hysteresis": 2,
> "min\_loss\_scale": 1
> },
> "zero\_optimization": {
> "stage": 3,
> "offload\_param": {
> "device": "cpu",
> "pin\_memory": true
> },
> "offload\_optimizer": {
> "device": "cpu",
> "pin\_memory": true
> },
> 
> "allgather\_partitions": true,
> "allgather\_bucket\_size": 1e7,
> "reduce\_scatter": true,
> "reduce\_bucket\_size": 1e7,
> 
> "overlap\_comm": true,
> "contiguous\_gradients": true,
> 
> "sub\_group\_size": 1e9,
> "stage3\_prefetch\_bucket\_size": 1e7,
> "stage3\_param\_persistence\_threshold": 1e5,
> "stage3\_max\_live\_parameters": 1e8,
> "stage3\_max\_reuse\_distance": 1e8,
> "stage3\_gather\_16bit\_weights\_on\_model\_save": true
> },
> "activation\_checkpointing": {
> "partition\_activations": true,
> "number\_checkpoints": 100,
> 
> "cpu\_checkpointing": true,
> "contiguous\_memory\_optimization": true,
> "synchronize\_checkpoint\_boundary": false,
> "profile": true
> },
> "data\_types": {
> "grad\_accum\_dtype":"bf16"
> }
> }
> \`\`\`
> 
> Main code of training
> \`\`\`
> engine.train()
> for input\_token, target\_token, \_ in train\_dataloader:
> input\_token = input\_token. to(engine.device)
> target\_token = target\_token.to(engine.device)
> 
> pred\_logits = engine(input\_token)
> 
> target\_pos = (target\_token != PAD\_ID) # only calculate loss on non-pad token
> pred\_logits = pred\_logits\[target\_pos\].view(-1, model\_arg.vocab\_size)
> target\_token = target\_token\[target\_pos\].view(-1)
> loss = torch.nn.functional.cross\_entropy(pred\_logits, target\_token)
> 
> engine.backward(loss)
> engine.step()
> \`\`\`
> 
> \*\*Expected behavior\*\*
> Finetun my model on P100 without oom
> 
> \*\*ds\_report output\*\*
> ds\_report of machine one with 1x RTX3090ti(24G). The above GPU usage information was obtained on this machine.
> \`\`\`
> \[2023-10-17 22:28:50,665\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \--------------------------------------------------
> DeepSpeed C++/CUDA extension op report
> \--------------------------------------------------
> NOTE: Ops not installed will be just-in-time (JIT) compiled at
> runtime if needed. Op compatibility means that your system
> meet the required dependencies to JIT install the op.
> \--------------------------------------------------
> JIT compiled ops requires ninja
> ninja .................. \[OKAY\]
> \--------------------------------------------------
> op name ................ installed .. compatible
> \--------------------------------------------------
> \[WARNING\] async\_io requires the dev libaio .so object and headers but these were not found.
> \[WARNING\] async\_io: please install the libaio-dev package with apt
> \[WARNING\] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
> async\_io ............... \[NO\] ....... \[NO\]
> fused\_adam ............. \[NO\] ....... \[OKAY\]
> cpu\_adam ............... \[NO\] ....... \[OKAY\]
> cpu\_adagrad ............ \[NO\] ....... \[OKAY\]
> cpu\_lion ............... \[NO\] ....... \[OKAY\]
> \[WARNING\] Please specify the CUTLASS repo directory as environment variable $CUTLASS\_PATH
> evoformer\_attn ......... \[NO\] ....... \[NO\]
> fused\_lamb ............. \[NO\] ....... \[OKAY\]
> fused\_lion ............. \[NO\] ....... \[OKAY\]
> quantizer .............. \[NO\] ....... \[OKAY\]
> random\_ltd ............. \[NO\] ....... \[OKAY\]
> \[WARNING\] sparse\_attn requires a torch version \>= 1.5 and \< 2.0 but detected 2.1
> \[WARNING\] using untested triton version (2.1.0), only 1.0.0 is known to be compatible
> sparse\_attn ............ \[NO\] ....... \[NO\]
> spatial\_inference ...... \[NO\] ....... \[OKAY\]
> transformer ............ \[NO\] ....... \[OKAY\]
> stochastic\_transformer . \[NO\] ....... \[OKAY\]
> transformer\_inference .. \[NO\] ....... \[OKAY\]
> \--------------------------------------------------
> DeepSpeed general environment info:
> torch install path ............... \['/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch'\]
> torch version .................... 2.1.0+cu121
> deepspeed install path ........... \['/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed'\]
> deepspeed info ................... 0.11.1, unknown, unknown
> torch cuda version ............... 12.1
> torch hip version ................ None
> nvcc version ..................... 11.6
> deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7
> shared memory (/dev/shm) size .... 125.77 GB
> \`\`\`
> 
> ds\_report of machine 2 with 8x P100(16G). I finetune my model on this machine.
> \`\`\`
> \[2023-10-17 22:28:40,428\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \--------------------------------------------------
> DeepSpeed C++/CUDA extension op report
> \--------------------------------------------------
> NOTE: Ops not installed will be just-in-time (JIT) compiled at
> runtime if needed. Op compatibility means that your system
> meet the required dependencies to JIT install the op.
> \--------------------------------------------------
> JIT compiled ops requires ninja
> ninja .................. \[OKAY\]
> \--------------------------------------------------
> op name ................ installed .. compatible
> \--------------------------------------------------
> \[WARNING\] async\_io requires the dev libaio .so object and headers but these were not found.
> \[WARNING\] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
> async\_io ............... \[NO\] ....... \[NO\]
> fused\_adam ............. \[YES\] ...... \[OKAY\]
> cpu\_adam ............... \[YES\] ...... \[OKAY\]
> cpu\_adagrad ............ \[YES\] ...... \[OKAY\]
> cpu\_lion ............... \[NO\] ....... \[OKAY\]
> \[WARNING\] Please specify the CUTLASS repo directory as environment variable $CUTLASS\_PATH
> evoformer\_attn ......... \[NO\] ....... \[NO\]
> fused\_lamb ............. \[YES\] ...... \[OKAY\]
> fused\_lion ............. \[NO\] ....... \[OKAY\]
> quantizer .............. \[NO\] ....... \[OKAY\]
> random\_ltd ............. \[NO\] ....... \[OKAY\]
> \[WARNING\] sparse\_attn requires a torch version \>= 1.5 and \< 2.0 but detected 2.1
> \[WARNING\] using untested triton version (2.1.0), only 1.0.0 is known to be compatible
> sparse\_attn ............ \[NO\] ....... \[NO\]
> spatial\_inference ...... \[NO\] ....... \[OKAY\]
> transformer ............ \[NO\] ....... \[OKAY\]
> stochastic\_transformer . \[NO\] ....... \[OKAY\]
> transformer\_inference .. \[NO\] ....... \[OKAY\]
> \--------------------------------------------------
> DeepSpeed general environment info:
> torch install path ............... \['/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch'\]
> torch version .................... 2.1.0+cu118
> deepspeed install path ........... \['/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed'\]
> deepspeed info ................... 0.11.2+unknown, unknown, unknown
> torch cuda version ............... 11.8
> torch hip version ................ None
> nvcc version ..................... 11.8
> deepspeed wheel compiled w. ...... torch 2.1, cuda 11.8
> shared memory (/dev/shm) size .... 125.78 GB
> \`\`\`
> 
> \*\*Screenshots\*\*
> No Screenshots
> 
> \*\*System info (please complete the following information):\*\*
> Machine 1:
> OS: Ubuntu 20.04.4
> GPU: 1x RTX 3090ti
> Interconnects: No
> Python version: 3.11.4
> 
> Machine 2:
> OS: Ubuntu 16.04.6
> GPU: 8x Tesla P100
> Interconnects: No
> Python version: 3.11.5
> 
> \*\*Launcher context\*\*
> Machine 1:
> \`PYTORCH\_CUDA\_ALLOC\_CONF=max\_split\_size\_mb:64 deepspeed --include localhost:0 train.py\`
> 
> Machine 2:
> \`PYTORCH\_CUDA\_ALLOC\_CONF=max\_split\_size\_mb:64 deepspeed --include localhost:0,1,2,3 train.py\`
> 
> \*\*Docker context\*\*
> No docker
> 
> \*\*Additional context\*\*
> In Reproduce, I simplified some of the code, so the error traceback is slightly different from Reproduce.
> 
> Complete deepspeed output on machine 1 (1x RTX 3090)
> The error is trying to backward through the graph a second time. I report it in another issue. \[https://github.com/microsoft/DeepSpeed/issues/4528\](url)
> The issue is too long to submit, I cannot attach the output log here.
> Please check the output in the issue 4528. I apologize for any inconvenience caused.
> 
> Complete deepspeed output on machine 2 (8x P100)
> The error is oom
> \`\`\`
> \[2023-10-17 23:10:48,978\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \[2023-10-17 23:10:50,117\] \[WARNING\] \[runner.py:203:fetch\_hostfile\] Unable to find hostfile, will proceed with training with local resources only.
> \[2023-10-17 23:10:50,117\] \[INFO\] \[runner.py:570:main\] cmd = /home/user/miniconda3/envs/llama2/bin/python -u -m deepspeed.launcher.launch --world\_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNF19 --master\_addr=127.0.0.1 --master\_port=29500 --enable\_each\_rank\_log=None train.py
> \[2023-10-17 23:10:52,522\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \[2023-10-17 23:10:53,468\] \[INFO\] \[launch.py:145:main\] WORLD INFO DICT: {'localhost': \[0, 1, 2, 3, 4\]}
> \[2023-10-17 23:10:53,468\] \[INFO\] \[launch.py:151:main\] nnodes=1, num\_local\_procs=5, node\_rank=0
> \[2023-10-17 23:10:53,468\] \[INFO\] \[launch.py:162:main\] global\_rank\_mapping=defaultdict(\<class 'list'\>, {'localhost': \[0, 1, 2, 3, 4\]})
> \[2023-10-17 23:10:53,468\] \[INFO\] \[launch.py:163:main\] dist\_world\_size=5
> \[2023-10-17 23:10:53,468\] \[INFO\] \[launch.py:165:main\] Setting CUDA\_VISIBLE\_DEVICES=0,1,2,3,4
> \[2023-10-17 23:10:55,910\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \[2023-10-17 23:10:55,948\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \[2023-10-17 23:10:55,955\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \[2023-10-17 23:10:55,955\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \[2023-10-17 23:10:55,994\] \[INFO\] \[real\_accelerator.py:158:get\_accelerator\] Setting ds\_accelerator to cuda (auto detect)
> \[2023-10-17 23:10:57,398\] \[INFO\] \[comm.py:637:init\_distributed\] cdb=None
> \[2023-10-17 23:10:57,767\] \[INFO\] \[comm.py:637:init\_distributed\] cdb=None
> \[2023-10-17 23:10:57,816\] \[INFO\] \[comm.py:637:init\_distributed\] cdb=None
> \[2023-10-17 23:10:57,841\] \[INFO\] \[comm.py:637:init\_distributed\] cdb=None
> \[2023-10-17 23:10:57,842\] \[INFO\] \[comm.py:637:init\_distributed\] cdb=None
> \[2023-10-17 23:10:57,843\] \[INFO\] \[comm.py:668:init\_distributed\] Initializing TorchBackend in DeepSpeed with backend nccl
> \[2023-10-17 23:13:00,132\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] DeepSpeed info: version=0.11.2+unknown, git-hash=unknown, git-branch=unknown
> \[2023-10-17 23:13:12,414\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] DeepSpeed Flops Profiler Enabled: False
> Adam Optimizer #0 is created with AVX512 arithmetic capability.
> Config: alpha=0.000020, betas=(0.900000, 0.950000), weight\_decay=0.000100, adam\_w=1
> Adam Optimizer #0 is created with AVX512 arithmetic capability.
> Config: alpha=0.000020, betas=(0.900000, 0.950000), weight\_decay=0.000100, adam\_w=1
> \[2023-10-17 23:13:14,520\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] Using DeepSpeed Optimizer param name adamw as basic optimizer
> \[2023-10-17 23:13:14,520\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] Removing param\_group that has no 'params' in the basic Optimizer
> Adam Optimizer #0 is created with AVX512 arithmetic capability.
> Config: alpha=0.000020, betas=(0.900000, 0.950000), weight\_decay=0.000100, adam\_w=1
> Adam Optimizer #0 is created with AVX512 arithmetic capability.
> Config: alpha=0.000020, betas=(0.900000, 0.950000), weight\_decay=0.000100, adam\_w=1
> Adam Optimizer #0 is created with AVX512 arithmetic capability.
> Config: alpha=0.000020, betas=(0.900000, 0.950000), weight\_decay=0.000100, adam\_w=1
> \[2023-10-17 23:13:14,561\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
> \[2023-10-17 23:13:14,561\] \[INFO\] \[utils.py:56:is\_zero\_supported\_optimizer\] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=\<class 'deepspeed.ops.adam.cpu\_adam.DeepSpeedCPUAdam'\>
> \[2023-10-17 23:13:14,561\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
> \[2023-10-17 23:13:14,561\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] Creating torch.bfloat16 ZeRO stage 3 optimizer
> \[2023-10-17 23:13:15,405\] \[INFO\] \[utils.py:802:see\_memory\_usage\] Stage 3 initialize beginning
> \[2023-10-17 23:13:15,406\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 12.99 GB Max\_MA 12.99 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:15,406\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 121.56 GB, percent = 48.3%
> \[2023-10-17 23:13:15,412\] \[INFO\] \[stage3.py:126:\_\_init\_\_\] Reduce bucket size 10000000
> \[2023-10-17 23:13:15,412\] \[INFO\] \[stage3.py:127:\_\_init\_\_\] Prefetch bucket size 10000000
> \[2023-10-17 23:13:15,841\] \[INFO\] \[utils.py:802:see\_memory\_usage\] DeepSpeedZeRoOffload initialize \[begin\]
> \[2023-10-17 23:13:15,841\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 12.99 GB Max\_MA 12.99 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:15,841\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 121.56 GB, percent = 48.3%
> Parameter Offload: Total persistent parameters: 10247936 in 260 params
> \[2023-10-17 23:13:28,903\] \[INFO\] \[utils.py:802:see\_memory\_usage\] DeepSpeedZeRoOffload initialize \[end\]
> \[2023-10-17 23:13:28,904\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.31 GB Max\_MA 12.99 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:28,904\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 143.14 GB, percent = 56.9%
> \[2023-10-17 23:13:29,558\] \[INFO\] \[utils.py:802:see\_memory\_usage\] Before creating fp16 partitions
> \[2023-10-17 23:13:29,558\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.31 GB Max\_MA 0.31 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:29,558\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 143.04 GB, percent = 56.9%
> \[2023-10-17 23:13:35,210\] \[INFO\] \[utils.py:802:see\_memory\_usage\] After creating fp16 partitions: 1
> \[2023-10-17 23:13:35,211\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.31 GB Max\_MA 0.31 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:35,212\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 143.57 GB, percent = 57.1%
> \[2023-10-17 23:13:35,692\] \[INFO\] \[utils.py:802:see\_memory\_usage\] Before creating fp32 partitions
> \[2023-10-17 23:13:35,693\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.31 GB Max\_MA 0.31 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:35,695\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 143.57 GB, percent = 57.1%
> \[2023-10-17 23:13:36,267\] \[INFO\] \[utils.py:802:see\_memory\_usage\] After creating fp32 partitions
> \[2023-10-17 23:13:36,268\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.31 GB Max\_MA 0.31 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:36,268\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 143.68 GB, percent = 57.1%
> \[2023-10-17 23:13:37,129\] \[INFO\] \[utils.py:802:see\_memory\_usage\] Before initializing optimizer states
> \[2023-10-17 23:13:37,129\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.31 GB Max\_MA 0.31 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:37,129\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 144.35 GB, percent = 57.4%
> \[2023-10-17 23:13:37,776\] \[INFO\] \[utils.py:802:see\_memory\_usage\] After initializing optimizer states
> \[2023-10-17 23:13:37,777\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.31 GB Max\_MA 0.31 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:37,777\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 144.51 GB, percent = 57.4%
> \[2023-10-17 23:13:37,778\] \[INFO\] \[stage3.py:459:\_setup\_for\_real\_optimizer\] optimizer state initialized
> \[2023-10-17 23:13:38,669\] \[INFO\] \[utils.py:802:see\_memory\_usage\] After initializing ZeRO optimizer
> \[2023-10-17 23:13:38,670\] \[INFO\] \[utils.py:803:see\_memory\_usage\] MA 0.33 GB Max\_MA 0.34 GB CA 13.01 GB Max\_CA 13 GB 
> \[2023-10-17 23:13:38,670\] \[INFO\] \[utils.py:810:see\_memory\_usage\] CPU Virtual Memory: used = 114.13 GB, percent = 45.4%
> \[2023-10-17 23:13:38,671\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] DeepSpeed Final Optimizer = adamw
> \[2023-10-17 23:13:38,671\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] DeepSpeed using configured LR scheduler = WarmupDecayLR
> \[2023-10-17 23:13:38,671\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] DeepSpeed LR Scheduler = \<deepspeed.runtime.lr\_schedules.WarmupDecayLR object at 0x7fa76bb5fe10\>
> \[2023-10-17 23:13:38,671\] \[INFO\] \[logging.py:96:log\_dist\] \[Rank 0\] step=0, skipped=0, lr=\[2e-05\], mom=\[\[0.9, 0.95\]\]
> \[2023-10-17 23:13:38,673\] \[INFO\] \[config.py:968:print\] DeepSpeedEngine configuration:
> \[2023-10-17 23:13:38,673\] \[INFO\] \[config.py:972:print\] activation\_checkpointing\_config {
> "partition\_activations": true, 
> "contiguous\_memory\_optimization": true, 
> "cpu\_checkpointing": true, 
> "number\_checkpoints": 100, 
> "synchronize\_checkpoint\_boundary": false, 
> "profile": true
> }
> \[2023-10-17 23:13:38,673\] \[INFO\] \[config.py:972:print\] aio\_config ................... {'block\_size': 1048576, 'queue\_depth': 8, 'thread\_count': 1, 'single\_submit': False, 'overlap\_events': True}
> \[2023-10-17 23:13:38,673\] \[INFO\] \[config.py:972:print\] amp\_enabled .................. False
> \[2023-10-17 23:13:38,673\] \[INFO\] \[config.py:972:print\] amp\_params ................... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] autotuning\_config ............ {
> "enabled": false, 
> "start\_step": null, 
> "end\_step": null, 
> "metric\_path": null, 
> "arg\_mappings": null, 
> "metric": "throughput", 
> "model\_info": null, 
> "results\_dir": "autotuning\_results", 
> "exps\_dir": "autotuning\_exps", 
> "overwrite": true, 
> "fast": true, 
> "start\_profile\_step": 3, 
> "end\_profile\_step": 5, 
> "tuner\_type": "gridsearch", 
> "tuner\_early\_stopping": 5, 
> "tuner\_num\_trials": 50, 
> "model\_info\_path": null, 
> "mp\_size": 1, 
> "max\_train\_batch\_size": null, 
> "min\_train\_batch\_size": 1, 
> "max\_train\_micro\_batch\_size\_per\_gpu": 1.024000e+03, 
> "min\_train\_micro\_batch\_size\_per\_gpu": 1, 
> "num\_tuning\_micro\_batch\_sizes": 3
> }
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] bfloat16\_enabled ............. True
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] checkpoint\_parallel\_write\_pipeline False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] checkpoint\_tag\_validation\_enabled True
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] checkpoint\_tag\_validation\_fail False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] comms\_config ................. \<deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7faac78d7510\>
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] communication\_data\_type ...... None
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] compression\_config ........... {'weight\_quantization': {'shared\_parameters': {'enabled': False, 'quantizer\_kernel': False, 'schedule\_offset': 0, 'quantize\_groups': 1, 'quantize\_verbose': False, 'quantization\_type': 'symmetric', 'quantize\_weight\_in\_forward': False, 'rounding': 'nearest', 'fp16\_mixed\_quantize': False, 'quantize\_change\_ratio': 0.001}, 'different\_groups': {}}, 'activation\_quantization': {'shared\_parameters': {'enabled': False, 'quantization\_type': 'symmetric', 'range\_calibration': 'dynamic', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'sparse\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'row\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'head\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'topk', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'channel\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'layer\_reduction': {'enabled': False}}
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] curriculum\_enabled\_legacy .... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] curriculum\_params\_legacy ..... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] data\_efficiency\_config ....... {'enabled': False, 'seed': 1234, 'data\_sampling': {'enabled': False, 'num\_epochs': 1000, 'num\_workers': 0, 'curriculum\_learning': {'enabled': False}}, 'data\_routing': {'enabled': False, 'random\_ltd': {'enabled': False, 'layer\_token\_lr\_schedule': {'enabled': False}}}}
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] data\_efficiency\_enabled ...... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] dataloader\_drop\_last ......... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] disable\_allgather ............ False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] dump\_state ................... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] dynamic\_loss\_scale\_args ...... None
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_enabled ........... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_gas\_boundary\_resolution 1
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_layer\_name ........ bert.encoder.layer
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_layer\_num ......... 0
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_max\_iter .......... 100
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_stability ......... 1e-06
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_tol ............... 0.01
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] eigenvalue\_verbose ........... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] elasticity\_enabled ........... False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] flops\_profiler\_config ........ {
> "enabled": false, 
> "recompute\_fwd\_factor": 0.0, 
> "profile\_step": 1, 
> "module\_depth": -1, 
> "top\_modules": 1, 
> "detailed": true, 
> "output\_file": null
> }
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] fp16\_auto\_cast ............... None
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] fp16\_enabled ................. False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] fp16\_master\_weights\_and\_gradients False
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] global\_rank .................. 0
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] grad\_accum\_dtype ............. bf16
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] gradient\_accumulation\_steps .. 8
> \[2023-10-17 23:13:38,674\] \[INFO\] \[config.py:972:print\] gradient\_clipping ............ 1.0
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] gradient\_predivide\_factor .... 1.0
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] hybrid\_engine ................ enabled=False max\_out\_tokens=512 inference\_tp\_size=1 release\_inference\_cache=False pin\_parameters=True tp\_gather\_partition\_size=8
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] initial\_dynamic\_scale ........ 1
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] load\_universal\_checkpoint .... False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] loss\_scale ................... 1.0
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] memory\_breakdown ............. False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] mics\_hierarchial\_params\_gather False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] mics\_shard\_size .............. -1
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] monitor\_config ............... tensorboard=TensorBoardConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv\_monitor=CSVConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') enabled=False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] nebula\_config ................ {
> "enabled": false, 
> "persistent\_storage\_path": null, 
> "persistent\_time\_interval": 100, 
> "num\_of\_version\_in\_retention": 2, 
> "enable\_nebula\_load": true, 
> "load\_path": null
> }
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] optimizer\_legacy\_fusion ...... False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] optimizer\_name ............... adamw
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] optimizer\_params ............. {'lr': 2e-05, 'betas': \[0.9, 0.95\], 'eps': 1e-08, 'weight\_decay': 0.0001}
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed\_layers': False, 'activation\_checkpoint\_interval': 0}
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] pld\_enabled .................. False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] pld\_params ................... False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] prescale\_gradients ........... False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] scheduler\_name ............... WarmupDecayLR
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] scheduler\_params ............. {'warmup\_min\_lr': 0, 'warmup\_max\_lr': 2e-05, 'warmup\_num\_steps': 1000, 'total\_num\_steps': 10000}
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] sparse\_attention ............. None
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] sparse\_gradients\_enabled ..... False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] steps\_per\_print .............. 10
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] train\_batch\_size ............. 40
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] train\_micro\_batch\_size\_per\_gpu 1
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] use\_node\_local\_storage ....... False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] wall\_clock\_breakdown ......... False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] weight\_quantization\_config ... None
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] world\_size ................... 5
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] zero\_allow\_untested\_optimizer False
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] zero\_config .................. stage=3 contiguous\_gradients=True reduce\_scatter=True reduce\_bucket\_size=10000000 allgather\_partitions=True allgather\_bucket\_size=10000000 overlap\_comm=True load\_from\_fp32\_weights=True elastic\_checkpoint=False offload\_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme\_path=None, buffer\_count=5, buffer\_size=100,000,000, max\_in\_cpu=1,000,000,000, pin\_memory=True) offload\_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme\_path=None, buffer\_count=4, pin\_memory=True, pipeline=False, pipeline\_read=False, pipeline\_write=False, fast\_init=False) sub\_group\_size=1000000000 cpu\_offload\_param=None cpu\_offload\_use\_pin\_memory=None cpu\_offload=None prefetch\_bucket\_size=10000000 param\_persistence\_threshold=100000 model\_persistence\_threshold=sys.maxsize max\_live\_parameters=100000000 max\_reuse\_distance=100000000 gather\_16bit\_weights\_on\_model\_save=True stage3\_gather\_fp16\_weights\_on\_model\_save=False ignore\_unused\_parameters=True legacy\_stage1=False round\_robin\_gradients=False zero\_hpz\_partition\_size=1 zero\_quantized\_weights=False zero\_quantized\_nontrainable\_weights=False zero\_quantized\_gradients=False mics\_shard\_size=-1 mics\_hierarchical\_params\_gather=False memory\_efficient\_linear=True pipeline\_loading\_checkpoint=False override\_module\_apply=True
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] zero\_enabled ................. True
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] zero\_force\_ds\_cpu\_optimizer .. True
> \[2023-10-17 23:13:38,675\] \[INFO\] \[config.py:972:print\] zero\_optimization\_stage ...... 3
> \[2023-10-17 23:13:38,676\] \[INFO\] \[config.py:958:print\_user\_config\] json = {
> "train\_micro\_batch\_size\_per\_gpu": 1, 
> "gradient\_accumulation\_steps": 8, 
> "gradient\_clipping": 1.0, 
> "steps\_per\_print": 10, 
> "wall\_clock\_breakdown": false, 
> "memory\_breakdown": false, 
> "optimizer": {
> "type": "AdamW", 
> "params": {
> "lr": 2e-05, 
> "betas": \[0.9, 0.95\], 
> "eps": 1e-08, 
> "weight\_decay": 0.0001
> }
> }, 
> "scheduler": {
> "type": "WarmupDecayLR", 
> "params": {
> "warmup\_min\_lr": 0, 
> "warmup\_max\_lr": 2e-05, 
> "warmup\_num\_steps": 1000, 
> "total\_num\_steps": 1.000000e+04
> }
> }, 
> "bf16": {
> "enabled": true, 
> "loss\_scale": 0, 
> "loss\_scale\_window": 1000, 
> "initial\_scale\_power": 16, 
> "hysteresis": 2, 
> "min\_loss\_scale": 1
> }, 
> "zero\_optimization": {
> "stage": 3, 
> "offload\_param": {
> "device": "cpu", 
> "pin\_memory": true
> }, 
> "offload\_optimizer": {
> "device": "cpu", 
> "pin\_memory": true
> }, 
> "allgather\_partitions": true, 
> "allgather\_bucket\_size": 1.000000e+07, 
> "reduce\_scatter": true, 
> "reduce\_bucket\_size": 1.000000e+07, 
> "overlap\_comm": true, 
> "contiguous\_gradients": true, 
> "sub\_group\_size": 1.000000e+09, 
> "stage3\_prefetch\_bucket\_size": 1.000000e+07, 
> "stage3\_param\_persistence\_threshold": 1.000000e+05, 
> "stage3\_max\_live\_parameters": 1.000000e+08, 
> "stage3\_max\_reuse\_distance": 1.000000e+08, 
> "stage3\_gather\_16bit\_weights\_on\_model\_save": true
> }, 
> "activation\_checkpointing": {
> "partition\_activations": true, 
> "number\_checkpoints": 100, 
> "cpu\_checkpointing": true, 
> "contiguous\_memory\_optimization": true, 
> "synchronize\_checkpoint\_boundary": false, 
> "profile": true
> }, 
> "data\_types": {
> "grad\_accum\_dtype": "bf16"
> }
> }
> 
> Traceback (most recent call last):
> File "/data/user/llama2/src/7\_train/train.py", line 449, in \<module\>
> trainer.train\_train()
> File "/data/user/llama2/src/7\_train/train.py", line 408, in train\_train
> self.train\_epoch(epoch)
> File "/data/user/llama2/src/7\_train/train.py", line 430, in train\_epoch
> pred\_logits = self.model\_engine(input\_token)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in \_wrapped\_call\_impl
> return self.\_call\_impl(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in \_call\_impl
> return forward\_call(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn
> ret\_val = func(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1807, in forward
> loss = self.module(\*inputs, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in \_wrapped\_call\_impl
> return self.\_call\_impl(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in \_call\_impl
> result = forward\_call(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/data/user/llama2/src/7\_train/../0\_all\_types\_of\_llama2/Llama\_2\_Adaptive\_Lora.py", line 275, in forward
> hidden = layer(hidden, start\_pos, freqs\_cis, mask)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in \_wrapped\_call\_impl
> return self.\_call\_impl(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in \_call\_impl
> result = forward\_call(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/data/user/llama2/src/7\_train/../0\_all\_types\_of\_llama2/Llama\_2\_Adaptive\_Lora.py", line 230, in forward
> hidden = hidden + self.feed\_forward(self.ffn\_norm(hidden))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in \_wrapped\_call\_impl
> return self.\_call\_impl(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in \_call\_impl
> result = forward\_call(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/data/user/llama2/src/7\_train/../0\_all\_types\_of\_llama2/Llama\_2\_Adaptive\_Lora.py", line 198, in forward
> hidden = self.w2(hidden)
> ^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in \_wrapped\_call\_impl
> return self.\_call\_impl(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in \_call\_impl
> result = forward\_call(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/data/user/llama2/src/7\_train/../0\_all\_types\_of\_llama2/Llama\_2\_Adaptive\_Lora.py", line 30, in forward
> hidden = self.linear(x)
> ^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in \_wrapped\_call\_impl
> return self.\_call\_impl(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1557, in \_call\_impl
> args\_result = hook(self, args)
> ^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn
> ret\_val = func(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/runtime/zero/parameter\_offload.py", line 392, in \_pre\_forward\_module\_hook
> self.pre\_sub\_module\_forward\_function(module)
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/runtime/zero/parameter\_offload.py", line 505, in pre\_sub\_module\_forward\_function
> param\_coordinator.fetch\_sub\_module(sub\_module, forward=prev\_grad\_state)
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn
> ret\_val = func(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/torch/utils/\_contextlib.py", line 115, in decorate\_context
> return func(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/runtime/zero/partitioned\_param\_coordinator.py", line 284, in fetch\_sub\_module
> self.\_\_all\_gather\_params(params\_to\_fetch, forward)
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn
> ret\_val = func(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/runtime/zero/partitioned\_param\_coordinator.py", line 428, in \_\_all\_gather\_params
> self.\_\_all\_gather\_params\_(nonquantized\_params, forward, quantize=self.zero\_quantized\_weights)
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/runtime/zero/partitioned\_param\_coordinator.py", line 446, in \_\_all\_gather\_params\_
> handle = partitioned\_params\[0\].all\_gather\_coalesced(partitioned\_params,
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn
> ret\_val = func(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^
> File "/home/user/miniconda3/envs/llama2/lib/python3.11/site-packages/deepspeed/runtime/zero/partition\_parameters.py", line 1109, in all\_gather\_coalesced
> param\_buffer = torch.empty(
> ^^^^^^^^^^^^
> torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 88.00 MiB. GPU 3 has a total capacty of 15.89 GiB of which 61.88 MiB is free. Including non-PyTorch memory, this process has 15.82 GiB memory in use. Of the allocated memory 14.22 GiB is allocated by PyTorch, and 387.59 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF
> \[2023-10-17 23:13:48,521\] \[INFO\] \[launch.py:315:sigkill\_handler\] Killing subprocess 136199
> \[2023-10-17 23:13:51,768\] \[INFO\] \[launch.py:315:sigkill\_handler\] Killing subprocess 136200
> \[2023-10-17 23:13:55,047\] \[INFO\] \[launch.py:315:sigkill\_handler\] Killing subprocess 136201
> \[2023-10-17 23:13:58,292\] \[INFO\] \[launch.py:315:sigkill\_handler\] Killing subprocess 136202
> \[2023-10-17 23:13:58,292\] \[INFO\] \[launch.py:315:sigkill\_handler\] Killing subprocess 136203
> \[2023-10-17 23:14:01,610\] \[ERROR\] \[launch.py:321:sigkill\_handler\] \['/home/user/miniconda3/envs/llama2/bin/python', '-u', 'train.py', '--local\_rank=4'\] exits with return code = 1
> \`\`\`
> 
> I would appreciate any help.
> 
> \`\`\`\[tasklist\]
> \### Tasks
> \- \[\] Add a draft title or issue reference here
> \`\`\`