# NCCL Watchdog Timeout error while using Deepspeed and accelerate

**URL:** <https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195>\
**Category:** Beginners\
**Created:** [July 16, 2025, 3:10am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195 "2025-07-16T03:10:31Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaydeepb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jaydeepb/32/14906_2.png) [@jaydeepb](https://discuss.huggingface.co/u/jaydeepb)\
**Post date:** [July 16, 2025, 3:10am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/1 "2025-07-16T03:10:31Z")

</div>

Hi all! I have a 12B model that is distributed on 4 GPUs and a 2.8B model that is also distributed. I’m performing inference on 12B followed by training 2.8B. However, after a first few thousand steps of training, I’m getting the NCCL timeout error. I even tried increasing the default timeout to 90 minutes using this: `os.environ["NCCL_TIMEOUT"] = "5400" os.environ["TORCH_NCCL_BLOCKING_WAIT"] = "1"` but looks like it’s not getting applied and the default value is still 10 minutes. Here is my error:

```auto
rank1]:[E715 17:27:51.148884976 ProcessGroupNCCL.cpp:632] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1518933, OpType=ALLREDUCE, NumelIn=10490880, NumelOut=10490880, Timeout(ms)=600000) ran for 600000 milliseconds before timing out.

[rank2]:[E715 19:31:08.566950534 ProcessGroupNCCL.cpp:756] [Rank 2] Work WorkNCCL(SeqNum=1518933, OpType=ALLREDUCE, NumelIn=10490880, NumelOut=10490880, Timeout(ms)=600000) timed out in blocking wait.

[rank2]:[E715 19:31:09.873338639 ProcessGroupNCCL.cpp:684] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E715 19:31:09.873366470 ProcessGroupNCCL.cpp:698] [Rank 2] To avoid data inconsistency, we are taking the entire process down.

```

I would appreciate any pointers! I’m running the script using `accelerate launch script.py` and have a deepspeed config file setup already.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 16, 2025, 3:32am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/2 "2025-07-16T03:32:44Z")

</div>

I think there might be a bug…  
[Setting .`deepspeed_env` directly](https://www.deepspeed.ai/getting-started/) might be a quicker workaround.

> <https://github.com/huggingface/accelerate/issues/3680>
>
> \### System Info
> 
> \`\`\`Shell
> accelerator 1.8.1
> Ubuntu 22.04
> \`\`\`
> 
> \### Information
> 
> \-… \[x\] The official example scripts
> \- \[\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] One of the scripts in the examples/ folder of Accelerate or an officially supported \`no\_trainer\` script in the \`examples\` folder of the \`transformers\` repo (such as \`run\_no\_trainer\_glue.py\`)
> \- \[x\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> In version 1.8.1, when using the following code to set NCCL timeout and Deepspeed is used at the same time, the timeout setting is invalid.
> \`\`\`
> deepspeed\_cfg = DeepSpeedPlugin(
> zero\_stage=2,
> gradient\_accumulation\_steps=cfg.policy.gradient\_accumulation\_steps, 
> gradient\_clipping=1.0) 
> 
> kwargs = InitProcessGroupKwargs(backend="nccl", timeout=timedelta(seconds=5000))
> 
> accelerator = Accelerator(
> deepspeed\_plugin=deepspeed\_cfg,
> kwargs\_handlers=\[kwargs\],
> )
> \`\`\`
> 
> Through the above code, we hope that the timeout value is 5000 seconds, but this will not take effect, and the timeout value is still the default 600 seconds.
> 
> I haven't studied the parameter passing logic of accelerate carefully, so I provide the author with the following ideas, hoping to help the author modify the code of accelerate:
> 
> When using deepspeed, the first time that the code call \`dist.init\_distributed\` function of deepspeed is in \[line355 of accelerator.py\](https://github.com/huggingface/accelerate/blob/524e5f9828b106b054ffd20d5b007008ab427544/src/accelerate/accelerator.py#L335) and \[line216 of state.py\](https://github.com/huggingface/accelerate/blob/524e5f9828b106b054ffd20d5b007008ab427544/src/accelerate/state.py#L216), if I pass param \`timeout = timedelta(seconds=5000)\` in \`PartialState\` function, the timeout setting will work.
> 
> Therefore, I guess there is a problem with the accelerator's timeout parameter passing somewhere. I hope the author can check and modify it.
> 
> \### Expected behavior
> 
> Setting nccl timeout when using deepspeed works fine.

> <https://github.com/huggingface/accelerate/issues/1129>
>
> \### System Info
> 
> \`\`\`Shell
> \- \`Accelerate\` version: 0.16.0
> \- Platform: Linux-5….4.0-1100-gcp-x86\_64-with-glibc2.17
> \- Python version: 3.8.13
> \- Numpy version: 1.23.3
> \- PyTorch version (GPU?): 1.12.1+cu102 (True)
> \- \`Accelerate\` default config:
> Not found
> \`\`\`
> 
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] One of the scripts in the examples/ folder of Accelerate or an officially supported \`no\_trainer\` script in the \`examples\` folder of the \`transformers\` repo (such as \`run\_no\_trainer\_glue.py\`)
> \- \[X\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> Essentially, the problem is that this line 
> https://github.com/huggingface/accelerate/blob/639c1da8df8cdc0dfa4dd057c7f3e7e01b40f3f5/src/accelerate/state.py#L129
> does not use kwargs. So even if I tune the timeout with a \`InitProcessGroupKwargs(timeout=...)\`, deepspeed which initialize the dist backend, doesn't have the new timeout.
> I believe the line should pass the \`timeout\` arg conditionally on the fact that \`kwargs\` has it.
> 
> in my \`main.py\`:
> \`\`\`
> kwargs\_handlers = \[InitProcessGroupKwargs(timeout=timedelta(seconds=3\_600))\]
> accelerator = Accelerator(
> log\_with="all",
> rng\_types=\["torch", "cuda", "generator"\],
> gradient\_accumulation\_steps=GRAD\_ACC\_SIZE,
> kwargs\_handlers=kwargs\_handlers,
> )
> \`\`\`    
> 
> \`accelerate\_config.yaml\`:
> \`\`\`compute\_environment: LOCAL\_MACHINE
> deepspeed\_config:
> deepspeed\_multinode\_launcher: standard
> deepspeed\_config\_file: /home/victor/m4/experiments/pretraining/vloom/tr\_victordebug/ds\_config.json
> zero3\_init\_flag: true
> distributed\_type: DEEPSPEED
> fsdp\_config: {}
> machine\_rank: 0
> main\_process\_ip: null
> main\_process\_port: 29399
> main\_training\_function: main
> num\_machines: 1
> num\_processes: 1
> use\_cpu: false
> gpu\_ids: 0,
> \`\`\`
> \`ds\_config.json\`:
> \`\`\`
> {
> "bf16": {
> "enabled": false
> },
> "fp16": {
> "enabled": false,
> "auto\_cast": true,
> "loss\_scale": 0.0,
> "initial\_scale\_power": 1,
> "loss\_scale\_window": 1000,
> "hysteresis": 2,
> "min\_loss\_scale": 1
> },
> "zero\_optimization": {
> "stage": 3,
> "allgather\_partitions": true,
> "allgather\_bucket\_size": 5e8,
> "overlap\_comm": false,
> "reduce\_scatter": true,
> "reduce\_bucket\_size": "auto",
> "contiguous\_gradients": true,
> "stage3\_gather\_16bit\_weights\_on\_model\_save": true,
> "offload\_optimizer": {
> "device": "cpu"
> },
> "offload\_param": {
> "device": "cpu"
> }
> },
> "gradient\_clipping": "auto",
> "train\_batch\_size": "auto",
> "train\_micro\_batch\_size\_per\_gpu": "auto",
> "steps\_per\_print": 2000000
> }
> \`\`\`
> 
> Deepspeed version: \`Version: 0.8.2+9bfee830\`
> 
> \### Expected behavior
> 
> \`\`\`Shell
> \-
> \`\`\`

---

<div class="post-metadata">

**Author:** ![jaydeepb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jaydeepb/32/14906_2.png) [@jaydeepb](https://discuss.huggingface.co/u/jaydeepb)\
**Post date:** [July 16, 2025, 3:52am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/4 "2025-07-16T03:52:49Z")

</div>

thanks! let me try that and see if it helps.

---

<div class="post-metadata">

**Author:** ![jaydeepb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jaydeepb/32/14906_2.png) [@jaydeepb](https://discuss.huggingface.co/u/jaydeepb)\
**Post date:** [July 16, 2025, 4:45am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/5 "2025-07-16T04:45:29Z")

</div>

@John6666 do you mean only setting up DS env directly but still using accelerate for training? or do you mean completely switching to DS for everything (moving away from HF)?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 16, 2025, 5:25am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/6 "2025-07-16T05:25:35Z")

</div>

> DS env directly but still using accelerate for training?

this one.

---

<div class="post-metadata">

**Author:** ![jaydeepb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jaydeepb/32/14906_2.png) [@jaydeepb](https://discuss.huggingface.co/u/jaydeepb)\
**Post date:** [July 16, 2025, 10:56pm UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/7 "2025-07-16T22:56:55Z")

</div>

@John6666 it looks like `export NCCL_P2P_LEVEL=NVL` did the job!

---

<div class="post-metadata">

**Author:** ![jaydeepb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jaydeepb/32/14906_2.png) [@jaydeepb](https://discuss.huggingface.co/u/jaydeepb)\
**Post date:** [July 21, 2025, 6:25am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/8 "2025-07-21T06:25:38Z")

</div>

@John6666 looks like my previous solution didn’t work, so I’m back at it ☹

Do you mean creating a `config.json` myself and then passing it in then training args using `deepseed=config.json`? And then do I run the script using the usual `accelerate launch script.py`?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 21, 2025, 7:22am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/9 "2025-07-21T07:22:06Z")

</div>

Yeah. I haven’t actually used DeepSpeed, but I think [that method above](https://huggingface.co/docs/accelerate/usage_guides/deepspeed) is the most moderate way to work around it. If you handle DeepSpeed manually, it should be fine.

As for the compatibility issue with Accelerate, that’s probably the case, but fixing it with a patch would be difficult…

Or it might be possible to [replace DeepSpeed with a different framework](https://huggingface.co/docs/accelerate/concept_guides/fsdp_and_deepspeed).

---

<div class="post-metadata">

**Author:** ![alexge233](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alexge233/32/52093_2.png) [@alexge233](https://discuss.huggingface.co/u/alexge233)\
**Post date:** [August 4, 2025, 5:05pm UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/10 "2025-08-04T17:05:19Z")

</div>

Hi, this is pretty much still an issue.  
DeepSpeed with accelerate just hangs at the end of an epoch, sitting there doing nothing. No errors thrown, until a time out occurs.

I’ve tried:

- `NCCL_P2P_DISABLE`
- `NCCL_IB_DISABLE=1`
- `NCCL_CUMEM_ENABLE=0`
- `NCCL_SHM_DISABLE=1`

And various debug approaches, such as:

- `CUDA_LAUNCH_BLOCKING=1`
- `TORCH_NCCL_ASYNC_ERROR_HANDLING=1`

What really puzzles me is that DeepSpeed on a single GPU, aka when I disable the rest via `CUDA_VISIBLE_DEVICES="0"` works fine. As soon as I start enabling them back, it hangs at the end of the epoch. All 8 GPUs are using RAM and show 100% usage.

---

<div class="post-metadata">

**Author:** ![jaydeepb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jaydeepb/32/14906_2.png) [@jaydeepb](https://discuss.huggingface.co/u/jaydeepb)\
**Post date:** [August 7, 2025, 4:49pm UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/11 "2025-08-07T16:49:49Z")

</div>

I still have the same issue, haven’t found any solution yet ☹

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 7, 2025, 10:09pm UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/12 "2025-08-07T22:09:32Z")

</div>

> `os.environ["NCCL_TIMEOUT"] = "5400" `

A bug that caused [this environment variable to be overwritten and ignored by `accelerate` seems to have been fixed a few weeks ago](https://github.com/huggingface/accelerate/issues/3680).😅

```bash
pip install git+https://github.com/huggingface/accelerate

```

---

<div class="post-metadata">

**Author:** ![xor-s](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xor-s/32/55080_2.png) [@xor-s](https://discuss.huggingface.co/u/xor-s)\
**Post date:** [November 4, 2025, 3:35am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/13 "2025-11-04T03:35:19Z")

</div>

NCCL can modify timeout in 1.11.0, i set NCCL\_TIMEOUT=1800000 (30 minutes), the timeout still occurs.  
if you fixed it, tell me the method, please!!!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 4, 2025, 5:07am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/14 "2025-11-04T05:07:38Z")

</div>

I think we need to set timeouts **other than NCCL as well**.

* * *

Your timeout still fires because you changed the wrong knob and you likely have a rank that is stalled. In PyTorch+DeepSpeed the watchdog that aborts jobs is owned by **ProcessGroupNCCL** , not by bare NCCL. Its default “Timeout(ms)” in logs is **600000** (10 minutes). DeepSpeed also has its own 30-minute default unless you override it. If Accelerate initializes the process group via DeepSpeed and ignores your override, you still hit 10 minutes. Logs from many reports show exactly this pattern. ([GitHub](https://github.com/NVIDIA/nccl/issues/1409 "NCCL timeout issue · Issue #1409 · NVIDIA/nccl"))

Below is a detailed, practical map: background → common causes → concrete fixes.

* * *

## Background in one page

- **There are multiple timeouts.**

- **Timeouts are usually symptoms of a stalled rank** , not an inherently too-short limit: save/eval pauses on one rank, uneven dataloader lengths, an OOM or exception on a single worker, or network setup issues. The watchdog times out the collective after N minutes. ([PyTorch Forums](https://discuss.pytorch.org/t/multi-gpu-training-hangs-watchdog-caught-collective-operation-timeout/188702 "Watchdog caught collective operation timeout - distributed"))

* * *

## Why your `NCCL_TIMEOUT=1800000` did nothing

- The abort you see is from **ProcessGroupNCCL’s watchdog** , which looks at the **PG timeout** , not a raw `NCCL_TIMEOUT` knob. Your logs show the watchdog honoring 600000 ms. Multiple users report that setting an env alone leaves the watchdog at 10 minutes. You must raise the PG timeout explicitly or via DeepSpeed. ([GitHub](https://github.com/NVIDIA/nccl/issues/1409 "NCCL timeout issue · Issue #1409 · NVIDIA/nccl"))
- If **Accelerate+DeepSpeed** path ignores your kwargs, you still run with the default and hit the 10-minute abort. Recent Accelerate issues resolve this. ([GitHub](https://github.com/huggingface/accelerate/issues/3680 "When using deepspeed, setting timeout is invalid #3680"))

* * *

## Root causes → fixes

### A) Wrong timeout knob or it isn’t propagating

**Cause.** PG timeout wasn’t raised where DeepSpeed/Torch actually read it, or Accelerate dropped it.  
**Fix.** Upgrade Accelerate. Set **`DEEPSPEED_TIMEOUT`** and pass an explicit PG timeout via kwargs.

```bash
# Versions
pip install -U accelerate deepspeed # keep these current
# DeepSpeed consumes this (seconds). Docs confirm override behavior.
# https://deepspeed.readthedocs.io/en/stable/initialize.html
export DEEPSPEED_TIMEOUT=5400

```

```python
# Set the ProcessGroupNCCL timeout explicitly via Accelerate.
# https://huggingface.co/docs/accelerate/en/package_reference/kwargs
from datetime import timedelta
from accelerate import Accelerator, InitProcessGroupKwargs
from accelerate.utils import DeepSpeedPlugin

pg_kwargs = InitProcessGroupKwargs(timeout=timedelta(seconds=5400))
accel = Accelerator(deepspeed_plugin=DeepSpeedPlugin(), kwargs_handlers=[pg_kwargs])

```

DeepSpeed docs: default 30 min, overridable by `DEEPSPEED_TIMEOUT`. HF reports: older Accelerate builds ignored kwargs with DS; newer builds fix it. ([deepspeed.readthedocs.io](https://deepspeed.readthedocs.io/en/stable/initialize.html "Training Setup — DeepSpeed 0.17.7 documentation"))

**Propagate to all ranks.** DeepSpeed will forward `NCCL_*`/`PYTHON*` automatically and lets you pin extra envs with a `.deepspeed_env` file. Do this for multi-node. ([DeepSpeed](https://www.deepspeed.ai/getting-started/ "Getting Started - DeepSpeed"))

* * *

### B) Rank desync during save/eval or data iteration

**Symptoms.** Timeout on `ALLREDUCE`, `ALLGATHER` right after eval or `accelerator.save_state`. One rank is slower or never reaches the collective. Many reports match this. ([GitHub](https://github.com/huggingface/accelerate/issues/3512 "NCCL timeout when try to save · Issue #3512"))

**Fixes.**

- Make loaders **even per rank**. Use `drop_last=True` or ensure `len(dataset) % world_size == 0`. Mismatched steps per rank cause hangs. ([PyTorch Forums](https://discuss.pytorch.org/t/multi-gpu-training-hangs-watchdog-caught-collective-operation-timeout/188702 "Watchdog caught collective operation timeout - distributed"))
- **Guard checkpointing**. Save from rank-0 only. Surround with barriers (or `accelerator.wait_for_everyone()`). Users hit timeouts precisely on save. ([GitHub](https://github.com/huggingface/accelerate/issues/3512 "NCCL timeout when try to save · Issue #3512"))
- Treat any **OOM or exception** on a single worker as a desync root cause. Recent issues propose better behavior because a single crash leaves peers waiting until the watchdog fires. ([GitHub](https://github.com/pytorch/pytorch/issues/163546 "Better nccl timeout handling · Issue #163546 · pytorch ..."))

* * *

### C) Networking and container friction

**Causes.** Wrong NIC, slow/blocked path, IB tuning, small shared memory in Docker, or multi-node rendezvous issues. These often manifest as “works for small scale or single node, times out at scale.” ([GitHub](https://github.com/microsoft/DeepSpeed/issues/4845 "Connection timed out when using deepspeed multi-node ..."))

**Fixes.**

- Pick the NIC: `export NCCL_SOCKET_IFNAME=eth0` (or your interface). Tune or disable IB as needed. See NCCL env reference. ([NVIDIA Docs](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html "Environment Variables — NCCL 2.28.6 documentation"))
- On NVLink boxes, try `export NCCL_P2P_LEVEL=NVL`. Several users report stability improvements. ([Stack Overflow](https://stackoverflow.com/questions/69693950/error-some-nccl-operations-have-failed-or-timed-out "Error: Some NCCL operations have failed or timed out"))
- In containers, lift IPC/SHM/memlock: `--ipc=host --shm-size=8g --ulimit memlock=-1`. This resolves several “timeout” patterns that were actually IPC pressure. ([Stack Overflow](https://stackoverflow.com/questions/69693950/error-some-nccl-operations-have-failed-or-timed-out "Error: Some NCCL operations have failed or timed out"))
- Validate the fabric with **nccl-tests** before training. If rings fail under load, fix network first. ([GitHub](https://github.com/NVIDIA/nccl-tests "NVIDIA/nccl-tests"))

* * *

### D) PyTorch NCCL diagnostics

Turn on Torch’s NCCL diagnostics to find the **offending collective and rank** instead of guessing.

```bash
# PyTorch ProcessGroupNCCL diagnostics
# Docs: https://docs.pytorch.org/docs/stable/torch_nccl_environment_variables.html
export TORCH_NCCL_BLOCKING_WAIT=1
export TORCH_NCCL_DUMP_ON_TIMEOUT=1
export TORCH_NCCL_TRACE_BUFFER_SIZE=1048576
export NCCL_DEBUG=INFO

```

These envs print the timed-out op, sequence numbers, and often the stuck rank. Then you fix the exact point of desync. ([docs.pytorch.org](https://docs.pytorch.org/docs/stable/torch_nccl_environment_variables.html "PYTORCH ProcessGroupNCCL Environment Variables"))

* * *

## End-to-end “works in practice” recipe

1. **Upgrade** and set timeouts where they are read.

```bash
pip install -U accelerate deepspeed
# https://deepspeed.readthedocs.io/en/stable/initialize.html
export DEEPSPEED_TIMEOUT=5400
# Optional: keep NCCL timers consistent. This alone won't raise PG timeout.
export NCCL_DEBUG=INFO

```

```python
# https://huggingface.co/docs/accelerate/en/package_reference/kwargs
from datetime import timedelta
from accelerate import Accelerator, InitProcessGroupKwargs
from accelerate.utils import DeepSpeedPlugin
accel = Accelerator(
    deepspeed_plugin=DeepSpeedPlugin(),
    kwargs_handlers=[InitProcessGroupKwargs(timeout=timedelta(seconds=5400))],
)

```

1. **Propagate envs** across nodes with `.deepspeed_env` when launching multi-node. ([DeepSpeed](https://www.deepspeed.ai/getting-started/ "Getting Started - DeepSpeed"))

2. **Harden the loop** :

- `drop_last=True` on train/eval samplers.
- Only rank-0 saves. Add `accelerator.wait_for_everyone()` around checkpoint I/O.  
Evidence: timeouts cluster around save/eval in reports. ([GitHub](https://github.com/huggingface/accelerate/issues/3512 "NCCL timeout when try to save · Issue #3512"))

1. **Harden the runtime** :

- Container flags: `--ipc=host --shm-size=8g --ulimit memlock=-1`.
- Network hints: `NCCL_SOCKET_IFNAME=...`, `NCCL_P2P_LEVEL=NVL` on NVLink hosts.
- Multi-node: verify rendezvous and that all ranks see identical envs. ([Stack Overflow](https://stackoverflow.com/questions/69693950/error-some-nccl-operations-have-failed-or-timed-out "Error: Some NCCL operations have failed or timed out"))

1. **Debug with signal, not guesses** : enable the `TORCH_NCCL_*` variables, reproduce, read which collective and rank timed out, then fix that call-site. ([docs.pytorch.org](https://docs.pytorch.org/docs/stable/torch_nccl_environment_variables.html "PYTORCH ProcessGroupNCCL Environment Variables"))

* * *

## Quick sanity checks you can run

- **Confirm the PG timeout actually changed.** Newer Accelerate honors `InitProcessGroupKwargs` with DeepSpeed; older builds did not. If your logs still show `Timeout(ms)=600000)`, your override is not wired. ([GitHub](https://github.com/huggingface/accelerate/issues/3680 "When using deepspeed, setting timeout is invalid #3680"))
- **Repro on a single node with `nccl-tests`**. If fabric is flaky, you will see it there too. ([GitHub](https://github.com/NVIDIA/nccl-tests "NVIDIA/nccl-tests"))

* * *

## Curated references (grouped)

### Raise the correct timeout

- DeepSpeed `timeout` and `DEEPSPEED_TIMEOUT` documented. Shows 30-min default and env override. ([deepspeed.readthedocs.io](https://deepspeed.readthedocs.io/en/stable/initialize.html "Training Setup — DeepSpeed 0.17.7 documentation"))
- Accelerate kwargs and examples for `InitProcessGroupKwargs(timeout=...)`. Use recent docs. ([Hugging Face](https://huggingface.co/docs/accelerate/en/package_reference/kwargs "Kwargs handlers - Accelerate documentation"))
- Accelerate bug threads where timeout was ignored with DeepSpeed. Upgrade to fix. ([GitHub](https://github.com/huggingface/accelerate/issues/3680 "When using deepspeed, setting timeout is invalid #3680"))

### Evidence of the 10-minute watchdog and typical failure logs

- GitHub issues and HF threads with `Timeout(ms)=600000)` in `ProcessGroupNCCL` logs. ([GitHub](https://github.com/NVIDIA/nccl/issues/1409 "NCCL timeout issue · Issue #1409 · NVIDIA/nccl"))

### Rank-desync during save/eval or load

- HF issue: timeout “when try to save.”
- HF forum: `accelerator.save_state` timeout.
- HF forum: resume/load then timeout on first backward. ([GitHub](https://github.com/huggingface/accelerate/issues/3512 "NCCL timeout when try to save · Issue #3512"))

### Diagnostics and NCCL envs

- PyTorch ProcessGroupNCCL env reference: `TORCH_NCCL_BLOCKING_WAIT`, dump and tracing knobs. ([docs.pytorch.org](https://docs.pytorch.org/docs/stable/torch_nccl_environment_variables.html "PYTORCH ProcessGroupNCCL Environment Variables"))
- NCCL env catalogue for networking knobs. ([NVIDIA Docs](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html "Environment Variables — NCCL 2.28.6 documentation"))

### Networking and container hygiene

- SHM/memlock and `NCCL_P2P_LEVEL=NVL` fixes reported by practitioners. ([Stack Overflow](https://stackoverflow.com/questions/69693950/error-some-nccl-operations-have-failed-or-timed-out "Error: Some NCCL operations have failed or timed out"))
- DeepSpeed env propagation with `.deepspeed_env`. ([DeepSpeed](https://www.deepspeed.ai/getting-started/ "Getting Started - DeepSpeed"))
- `nccl-tests` to validate topology. ([GitHub](https://github.com/NVIDIA/nccl-tests "NVIDIA/nccl-tests"))

* * *

### Bottom line

Raise the **PG timeout where it is consumed** (DeepSpeed `timeout` or `DEEPSPEED_TIMEOUT` plus `InitProcessGroupKwargs`). **Upgrade Accelerate** so those settings apply. Then **remove rank desyncs** in save/eval and **fix the network/container**. This combination, not `NCCL_TIMEOUT` alone, stops the watchdog aborts. ([deepspeed.readthedocs.io](https://deepspeed.readthedocs.io/en/stable/initialize.html "Training Setup — DeepSpeed 0.17.7 documentation"))

---

<div class="post-metadata">

**Author:** ![xor-s](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xor-s/32/55080_2.png) [@xor-s](https://discuss.huggingface.co/u/xor-s)\
**Post date:** [November 5, 2025, 9:48am UTC](https://discuss.huggingface.co/t/nccl-watchdog-timeout-error-while-using-deepspeed-and-accelerate/163195/15 "2025-11-05T09:48:46Z")

</div>

i think it is OOM. using another framework throws an OOM error after running for a while.

thank you for providing such great think 🙂