# Accelerator.save\_state errors out due to timeout. Unable to increase timeout through kwargs\_handlers

**URL:** <https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747>\
**Category:** 🤗Accelerate\
**Created:** [April 15, 2024, 1:27am UTC](https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747 "2024-04-15T01:27:54Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rranjan1](https://avatars.discourse-cdn.com/v4/letter/r/df788c/32.png) [@rranjan1](https://discuss.huggingface.co/u/rranjan1)\
**Post date:** [April 15, 2024, 1:27am UTC](https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747/1 "2024-04-15T01:27:54Z")

</div>

I am using accelerate with deepspeed stage 1. I get the following error while saving the model:

NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(Seq  
Num=913, OpType=ALLREDUCE, NumelIn=499133440, NumelOut=499133440, Timeout(ms)=600000) ran for 600489 milliseconds before timing out.

I am not sure why the timeout is 600000ms. I tried to set the timeout in the code using InitProcessGroupKwargs

kwargs = InitProcessGroupKwargs(timeout=timedelta(seconds=3600))  
accelerator = Accelerator(  
gradient\_accumulation\_steps=args.gradient\_accumulation\_steps,  
mixed\_precision=args.mixed\_precision,  
log\_with=args.report\_to,  
project\_config=accelerator\_project\_config,  
kwargs\_handlers=[kwargs]  
)

But this did not change the timeout at all. How can I increase the current timeout from 600s to 3600s ?

accelerate verison: 0.29.2

---

<div class="post-metadata">

**Author:** ![marcsun13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcsun13/32/19059_2.png) [@marcsun13](https://discuss.huggingface.co/u/marcsun13)\
**Post date:** [April 15, 2024, 11:58am UTC](https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747/2 "2024-04-15T11:58:00Z")

</div>

Hi @rranjan1, could you check if the following ?

```auto
import time
from datetime import timedelta
from accelerate import Accelerator, InitProcessGroupKwargs
from torch import tensor

kwargs = [InitProcessGroupKwargs(timeout=timedelta(seconds=10))]
accelerator = Accelerator(kwargs_handlers=kwargs)

if accelerator.is_main_process:
    t = tensor(0).to(accelerator.device)
    time.sleep(8)
else:
    t = tensor(0).to(accelerator.device)
accelerator.wait_for_everyone()

print("All called!")

```

It should fail since the timeout is 4. If you change it to 10, it should works. I just want to see if the timeout is indeed changed as expected.

---

<div class="post-metadata">

**Author:** ![rranjan1](https://avatars.discourse-cdn.com/v4/letter/r/df788c/32.png) [@rranjan1](https://discuss.huggingface.co/u/rranjan1)\
**Post date:** [April 15, 2024, 4:02pm UTC](https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747/3 "2024-04-15T16:02:05Z")

</div>

I checked this with different timeout=timedelta(seconds=TK) value ranging from 1 to 10. The code never fails for any of these values.

---

<div class="post-metadata">

**Author:** ![JiaxuZ](https://avatars.discourse-cdn.com/v4/letter/j/8edcca/32.png) [@JiaxuZ](https://discuss.huggingface.co/u/JiaxuZ)\
**Post date:** [February 24, 2025, 1:47am UTC](https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747/4 "2025-02-24T01:47:12Z")

</div>

I meet the same issue. Have you solved it?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 24, 2025, 5:28pm UTC](https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747/5 "2025-02-24T17:28:51Z")

</div>

Maybe ongoing issue.

> <https://github.com/NVIDIA/nccl/issues/1409>
>
> Hi, I'm having this issue: Watchdog caught collective operation timeout: WorkNCC…L(SeqNum=80078...) ran for 600026 milliseconds before timing out
> 
> The code I'm running is a VQGAN training script. Parallelism is done with accelerate. This issue happens at the end of each epoch. We believe it's the problem of the environment setup, as the code could run on another machine without any problem
> 
> environment: Ubuntu 20
> nvcc: NVIDIA (R) Cuda compiler driver
> Copyright (c) 2005-2024 NVIDIA Corporation
> Built on Fri\_Jun\_14\_16:34:21\_PDT\_2024
> Cuda compilation tools, release 12.6, V12.6.20
> Build cuda\_12.6.r12.6/compiler.34431801\_0
> 
> full log
> \`\`\`
> \[rank7\]:\[E816 01:45:40.183009389 ProcessGroupNCCL.cpp:607\] \[Rank 7\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=BROADCAST, NumelIn=5056, NumelOut=5056, Timeout(ms)=600000) ran for 600026 milliseconds before timing out.
> \[rank7\]:\[E816 01:45:40.184373931 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 7\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80078, last completed NCCL work: 80077.
> \[rank3\]:\[E816 01:45:40.212150935 ProcessGroupNCCL.cpp:607\] \[Rank 3\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600013 milliseconds before timing out.
> \[rank9\]:\[E816 01:45:40.212150385 ProcessGroupNCCL.cpp:607\] \[Rank 9\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600013 milliseconds before timing out.
> \[rank9\]:\[E816 01:45:40.212958430 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 9\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> \[rank3\]:\[E816 01:45:40.212958690 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 3\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> \[rank8\]:\[E816 01:45:40.265372190 ProcessGroupNCCL.cpp:607\] \[Rank 8\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600066 milliseconds before timing out.
> \[rank8\]:\[E816 01:45:40.266489823 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 8\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> \[rank1\]:\[E816 01:45:40.278487367 ProcessGroupNCCL.cpp:607\] \[Rank 1\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600079 milliseconds before timing out.
> \[rank1\]:\[E816 01:45:40.279310302 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 1\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> \[rank5\]:\[E816 01:45:40.285038686 ProcessGroupNCCL.cpp:607\] \[Rank 5\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600085 milliseconds before timing out.
> \[rank5\]:\[E816 01:45:40.286293758 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 5\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> \[rank2\]:\[E816 01:45:40.292277610 ProcessGroupNCCL.cpp:607\] \[Rank 2\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600093 milliseconds before timing out.
> \[rank0\]:\[E816 01:45:40.292277580 ProcessGroupNCCL.cpp:607\] \[Rank 0\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600090 milliseconds before timing out.
> \[rank0\]:\[E816 01:45:40.293076425 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 0\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> \[rank2\]:\[E816 01:45:40.293076395 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 2\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> \[rank6\]:\[E816 01:45:40.296233705 ProcessGroupNCCL.cpp:607\] \[Rank 6\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=ALLREDUCE, NumelIn=2106881, NumelOut=2106881, Timeout(ms)=600000) ran for 600097 milliseconds before timing out.
> \[rank6\]:\[E816 01:45:40.298295572 ProcessGroupNCCL.cpp:1664\] \[PG 0 (default\_pg) Rank 6\] Exception (either an error or timeout) detected by watchdog at work: 80078, last enqueued NCCL work: 80087, last completed NCCL work: 80077.
> 0%| | 0/8008 \[10:00\<?, ?it/s\]
> \[rank7\]: Traceback (most recent call last):
> \[rank7\]: File "/home/duanwei/ML/magvit\_imagenet.py", line 480, in \<module\>
> \[rank7\]: for batch\_idx, (real, \_) in zip(pbar, dataloader):
> \[rank7\]: File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/accelerate/data\_loader.py", line 447, in \_\_iter\_\_
> \[rank7\]: synchronize\_rng\_states(self.rng\_types, self.synchronized\_generator)
> \[rank7\]: File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/accelerate/utils/random.py", line 132, in synchronize\_rng\_states
> \[rank7\]: synchronize\_rng\_state(RNGType(rng\_type), generator=generator)
> \[rank7\]: File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/accelerate/utils/random.py", line 127, in synchronize\_rng\_state
> \[rank7\]: generator.set\_state(rng\_state)
> \[rank7\]: RuntimeError: Invalid mt19937 state
> \[rank7\]:\[E816 01:45:41.635484056 ProcessGroupNCCL.cpp:1709\] \[PG 0 (default\_pg) Rank 7\] Timeout at NCCL work: 80078, last enqueued NCCL work: 80078, last completed NCCL work: 80077.
> \[rank7\]:\[E816 01:45:41.635510825 ProcessGroupNCCL.cpp:621\] \[Rank 7\] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
> \[rank7\]:\[E816 01:45:41.635516125 ProcessGroupNCCL.cpp:627\] \[Rank 7\] To avoid data inconsistency, we are taking the entire process down.
> \[rank7\]:\[E816 01:45:41.636462549 ProcessGroupNCCL.cpp:1515\] \[PG 0 (default\_pg) Rank 7\] Process group watchdog thread terminated with exception: \[Rank 7\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=BROADCAST, NumelIn=5056, NumelOut=5056, Timeout(ms)=600000) ran for 600026 milliseconds before timing out.
> Exception raised from checkTimeout at /opt/conda/conda-bld/pytorch\_1720538439675/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first):
> frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8b9912af86 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libc10.so)
> frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional\<std::chrono::duration\<long, std::ratio\<1l, 1000l\> \> \>) + 0x1d2 (0x7f8b9a40e0b2 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libtorch\_cuda.so)
> frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f8b9a414af3 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libtorch\_cuda.so)
> frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f8b9a416edc in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libtorch\_cuda.so)
> frame #4: \<unknown function\> + 0xd3b55 (0x7f8bea0f0b55 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/../../../.././libstdc++.so.6)
> frame #5: \<unknown function\> + 0x94ac3 (0x7f8bfaf3eac3 in /lib/x86\_64-linux-gnu/libc.so.6)
> frame #6: \<unknown function\> + 0x126850 (0x7f8bfafd0850 in /lib/x86\_64-linux-gnu/libc.so.6)
>     
> terminate called after throwing an instance of 'c10::DistBackendError'
> what(): \[PG 0 (default\_pg) Rank 7\] Process group watchdog thread terminated with exception: \[Rank 7\] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=80078, OpType=BROADCAST, NumelIn=5056, NumelOut=5056, Timeout(ms)=600000) ran for 600026 milliseconds before timing out.
> Exception raised from checkTimeout at /opt/conda/conda-bld/pytorch\_1720538439675/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first):
> frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8b9912af86 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libc10.so)
> frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional\<std::chrono::duration\<long, std::ratio\<1l, 1000l\> \> \>) + 0x1d2 (0x7f8b9a40e0b2 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libtorch\_cuda.so)
> frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f8b9a414af3 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libtorch\_cuda.so)
> frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f8b9a416edc in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libtorch\_cuda.so)
> frame #4: \<unknown function\> + 0xd3b55 (0x7f8bea0f0b55 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/../../../.././libstdc++.so.6)
> frame #5: \<unknown function\> + 0x94ac3 (0x7f8bfaf3eac3 in /lib/x86\_64-linux-gnu/libc.so.6)
> frame #6: \<unknown function\> + 0x126850 (0x7f8bfafd0850 in /lib/x86\_64-linux-gnu/libc.so.6)
>     
> Exception raised from ncclCommWatchdog at /opt/conda/conda-bld/pytorch\_1720538439675/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1521 (most recent call first):
> frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8b9912af86 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libc10.so)
> frame #1: \<unknown function\> + 0xe3ec34 (0x7f8b9a096c34 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/libtorch\_cuda.so)
> frame #2: \<unknown function\> + 0xd3b55 (0x7f8bea0f0b55 in /root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/lib/../../../.././libstdc++.so.6)
> frame #3: \<unknown function\> + 0x94ac3 (0x7f8bfaf3eac3 in /lib/x86\_64-linux-gnu/libc.so.6)
> frame #4: \<unknown function\> + 0x126850 (0x7f8bfafd0850 in /lib/x86\_64-linux-gnu/libc.so.6)
>     
> W0816 01:45:41.681000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340257 closing signal SIGTERM
> W0816 01:45:41.682000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340258 closing signal SIGTERM
> W0816 01:45:41.682000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340259 closing signal SIGTERM
> W0816 01:45:41.683000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340260 closing signal SIGTERM
> W0816 01:45:41.684000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340261 closing signal SIGTERM
> W0816 01:45:41.684000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340262 closing signal SIGTERM
> W0816 01:45:41.684000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340263 closing signal SIGTERM
> W0816 01:45:41.684000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340265 closing signal SIGTERM
> W0816 01:45:41.685000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:858\] Sending process 340266 closing signal SIGTERM
> E0816 01:45:43.466000 140650667333440 torch/distributed/elastic/multiprocessing/api.py:833\] failed (exitcode: -6) local\_rank: 7 (pid: 340264) of binary: /root/miniconda3/envs/magvit/bin/python3.12
> Traceback (most recent call last):
> File "/root/miniconda3/envs/magvit/bin/accelerate", line 10, in \<module\>
> sys.exit(main())
> ^^^^^^
> File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/accelerate/commands/accelerate\_cli.py", line 48, in main
> args.func(args)
> File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/accelerate/commands/launch.py", line 1097, in launch\_command
> multi\_gpu\_launcher(args)
> File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/accelerate/commands/launch.py", line 734, in multi\_gpu\_launcher
> distrib\_run.run(args)
> File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/distributed/run.py", line 892, in run
> elastic\_launch(
> File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 133, in \_\_call\_\_
> return launch\_agent(self.\_config, self.\_entrypoint, list(args))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/root/miniconda3/envs/magvit/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 264, in launch\_agent
> raise ChildFailedError(
> torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
> =======================================================
> magvit\_imagenet.py FAILED
> 
> \---
> Failures:
> \<NO\_OTHER\_FAILURES\>
> 
> \---
> Root Cause (first observed failure):
> \[0\]:
> time : 2024-08-16\_01:45:41
> host : kansas3
> rank : 7 (local\_rank: 7)
> exitcode : -6 (pid: 340264)
> error\_file: \<N/A\>
> traceback : Signal 6 (SIGABRT) received by PID 340264
> =======================================================
> \`\`\`
> 
> Can anyone help me with it?

---

<div class="post-metadata">

**Author:** ![nasos10](https://avatars.discourse-cdn.com/v4/letter/n/f0a364/32.png) [@nasos10](https://discuss.huggingface.co/u/nasos10)\
**Post date:** [March 3, 2025, 10:40am UTC](https://discuss.huggingface.co/t/accelerator-save-state-errors-out-due-to-timeout-unable-to-increase-timeout-through-kwargs-handlers/81747/6 "2025-03-03T10:40:23Z")

</div>

I am facing the same problem, trying to do accelerator.save\_state() (i call it with all processes, as I have seen this is the way to do this), with FSDP wrapping. It timeouts. Have you guys found any workarounds?