# Multi-node training fails Proxy Call to rank 0 failed (Connect)

**URL:** <https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877>\
**Category:** 🤗Accelerate\
**Created:** [December 15, 2022, 2:57pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877 "2022-12-15T14:57:10Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Noman](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@Noman](https://discuss.huggingface.co/u/Noman)\
**Post date:** [December 15, 2022, 2:57pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/1 "2022-12-15T14:57:10Z")

</div>

I am trying to run multi-node training with two nodes with one GPU in each:  
This is my configuration:

```auto
compute_environment: LOCAL_MACHINE
deepspeed_config:
  deepspeed_multinode_launcher: standard
  gradient_accumulation_steps: 1
  gradient_clipping: 1.0
  offload_optimizer_device: none
  offload_param_device: none
  zero3_init_flag: false
  zero_stage: 2
distributed_type: DEEPSPEED
fsdp_config: {}
machine_rank: 0 # 1 in the second node
main_process_ip: 10.3.40.125 # Same on both nodes
main_process_port: 29500
main_training_function: main
mixed_precision: fp16
num_machines: 2
num_processes: 2
use_cpu: false

```

I get the following logs on running `accelerate test --config_file accelerate_config.yaml` on the slave node where master node just keeps waiting for synchronization.

```auto
Running: accelerate-launch --config_file=accelerate_config.yaml /home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py
stdout: [2022-12-15 19:47:59,266] [INFO] [comm.py:654:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
stdout: **Initialization**
stdout: Testing, testing. 1, 2, 3.
stdout: Distributed environment: DEEPSPEED Backend: nccl
stdout: Num processes: 2
stdout: Process index: 0
stdout: Local process index: 0
stdout: Device: cuda:0
stdout: ds_config: {'train_batch_size': 'auto', 'train_micro_batch_size_per_gpu': 'auto', 'gradient_accumulation_steps': 1, 'zero_optimization': {'stage': 2, 'offload_optimizer': {'device': 'none'}, 'offload_param': {'device': 'none'}, 'stage3_gather_16bit_weights_on_model_save': False}, 'gradient_clipping': 1.0, 'steps_per_print': inf, 'fp16': {'enabled': True, 'auto_cast': True}}
stdout: 
stdout: 
stdout: **Test random number generator synchronization**
stderr: Traceback (most recent call last):
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py", line 359, in <module>
stderr: main()
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py", line 336, in main
stderr: rng_sync_check()
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py", line 44, in rng_sync_check
stderr: synchronize_rng_states(["torch"])
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/utils/random.py", line 88, in synchronize_rng_states
stderr: synchronize_rng_state(RNGType(rng_type), generator=generator)
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/utils/random.py", line 70, in synchronize_rng_state
stderr: torch.distributed.broadcast(rng_state, 0)
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1400, in broadcast
stderr: work = default_pg.broadcast([tensor], opts)
stderr: RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1666642975993/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, internal error, NCCL version 2.14.3
stderr: ncclInternalError: Internal check failed.
stderr: Last error:
stderr: Proxy Call to rank 0 failed (Connect)
stderr: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6239) of binary: /home4/nouman_tanveer/anaconda3/envs/ldm/bin/pythonRunning: accelerate-launch --config_file=accelerate_config.yaml /home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py
stdout: [2022-12-15 19:47:59,266] [INFO] [comm.py:654:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
stdout: **Initialization**
stdout: Testing, testing. 1, 2, 3.
stdout: Distributed environment: DEEPSPEED Backend: nccl
stdout: Num processes: 2
stdout: Process index: 0
stdout: Local process index: 0
stdout: Device: cuda:0
stdout: ds_config: {'train_batch_size': 'auto', 'train_micro_batch_size_per_gpu': 'auto', 'gradient_accumulation_steps': 1, 'zero_optimization': {'stage': 2, 'offload_optimizer': {'device': 'none'}, 'offload_param': {'device': 'none'}, 'stage3_gather_16bit_weights_on_model_save': False}, 'gradient_clipping': 1.0, 'steps_per_print': inf, 'fp16': {'enabled': True, 'auto_cast': True}}
stdout: 
stdout: 
stdout: **Test random number generator synchronization**
stderr: Traceback (most recent call last):
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py", line 359, in <module>
stderr: main()
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py", line 336, in main
stderr: rng_sync_check()
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py", line 44, in rng_sync_check
stderr: synchronize_rng_states(["torch"])
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/utils/random.py", line 88, in synchronize_rng_states
stderr: synchronize_rng_state(RNGType(rng_type), generator=generator)
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/accelerate/utils/random.py", line 70, in synchronize_rng_state
stderr: torch.distributed.broadcast(rng_state, 0)
stderr: File "/home4/nouman_tanveer/anaconda3/envs/ldm/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1400, in broadcast
stderr: work = default_pg.broadcast([tensor], opts)
stderr: RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1666642975993/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, internal error, NCCL version 2.14.3
stderr: ncclInternalError: Internal check failed.
stderr: Last error:
stderr: Proxy Call to rank 0 failed (Connect)
stderr: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6239) of binary: /home4/nouman_tanveer/anaconda3/envs/ldm/bin/python

```

Your help would be appreciated.  
Also if anyone can point to sources for beginners in multi-node training that would also be helpful. Accelerate documentation assumes you already know how it works.

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [December 15, 2022, 2:58pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/2 "2022-12-15T14:58:56Z")

</div>

> [@Noman](#):
>
> `num_processes: 2`

I think you want this to be `num_processes: 1` because it’s the number of GPUs _per machine_ not in total

---

<div class="post-metadata">

**Author:** ![Noman](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@Noman](https://discuss.huggingface.co/u/Noman)\
**Post date:** [December 15, 2022, 3:01pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/3 "2022-12-15T15:01:13Z")

</div>

Thank you, that fixed it.

---

<div class="post-metadata">

**Author:** ![Noman](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@Noman](https://discuss.huggingface.co/u/Noman)\
**Post date:** [December 15, 2022, 3:10pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/4 "2022-12-15T15:10:17Z")

</div>

@muellerzr but when I try to run the [nlp\_example.py](https://github.com/huggingface/accelerate/blob/main/examples/nlp_example.py) with the updated config from the repo it doesn’t process that. I tried prepending `NCCL_DEBUG=INFO` still it doesn’t return anything. However, it does work when I launch it with the default config on a single node.

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [December 15, 2022, 3:25pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/5 "2022-12-15T15:25:21Z")

</div>

> [@Noman](#):
>
> However, it does work when I launch it with the default config on a single node.

Can you print the full config file and I can try to take a look at this before the team is on vacation 🙂

Also the contents of `accelerate env` and your `deepspeed` version preferably.

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [December 15, 2022, 3:28pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/6 "2022-12-15T15:28:39Z")

</div>

cc @smangrul as well 🙂

---

<div class="post-metadata">

**Author:** ![Noman](https://avatars.discourse-cdn.com/v4/letter/n/d6d6ee/32.png) [@Noman](https://discuss.huggingface.co/u/Noman)\
**Post date:** [December 15, 2022, 3:39pm UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/7 "2022-12-15T15:39:55Z")

</div>

This is the default config I’m overwriting:

```auto
- `Accelerate` version: 0.15.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.9.15
- Numpy version: 1.23.4
- PyTorch version (GPU?): 1.13.0 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: fp16
        - use_cpu: False
        - dynamo_backend: NO
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - main_process_ip: None
        - main_process_port: None
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {}
        - fsdp_config: {}
        - megatron_lm_config: {}
        - downcast_bf16: no
        - tpu_name: None
        - tpu_zone: None
        - command_file: None
        - commands: None

```

with:

```auto
compute_environment: LOCAL_MACHINE
deepspeed_config:
  deepspeed_multinode_launcher: standard
  gradient_accumulation_steps: 1
  gradient_clipping: 1.0
  offload_optimizer_device: none
  offload_param_device: none
  zero3_init_flag: false
  zero_stage: 2
distributed_type: DEEPSPEED
fsdp_config: {}
machine_rank: 0 # 1 in the second node
main_process_ip: 10.3.40.125 # Same on both nodes
main_process_port: 29500
main_training_function: main
mixed_precision: fp16
num_machines: 2
num_processes: 2
use_cpu: false

```

Deepspeed version:

```auto
deepspeed 0.7.7 pypi_0 pypi

```

Thank you for your help.

---

<div class="post-metadata">

**Author:** ![smangrul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smangrul/32/3791_2.png) [@smangrul](https://discuss.huggingface.co/u/smangrul)\
**Post date:** [January 2, 2023, 10:02am UTC](https://discuss.huggingface.co/t/multi-node-training-fails-proxy-call-to-rank-0-failed-connect/27877/8 "2023-01-02T10:02:03Z")

</div>

Hello @Noman, does simple Multi-GPU training across the 2 nodes work?