# Synchronizing State, Trainer and Accelerate

**URL:** <https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255>\
**Category:** 🤗Accelerate\
**Created:** [May 22, 2025, 1:25am UTC](https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255 "2025-05-22T01:25:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![donb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/donb/32/3744_2.png) [@donb](https://discuss.huggingface.co/u/donb)\
**Post date:** [May 22, 2025, 1:25am UTC](https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255/1 "2025-05-22T01:25:10Z")

</div>

Using Trainer, and it appears that if I load any class from accelerate, the Trainer doesn’t perform its accelerate magic behind the scenes, meaning I get an error like this:

```auto
[rank1]: File "/opt/code/repos/MyProject/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5779, in caching_allocator_warmup
[rank1]: re.compile("|".join([re.escape(plan) for plan in model._tp_plan]))
[rank1]: ^^^^^^^^^^^^^^
[rank1]: TypeError: 'NoneType' object is not iterable

```

I have two use cases where I’d like slightly more control:

1. My script creates a directory with a timestamp, and there is a synchronization issue that creates two checkpoint directories, one for each GPU.

2. I load two models, the second attempt to load it always fails with this error. It appears that once the Trainer/TrainingArguments go out of scope, the accelerate process is torn down and doesn’t get reinitialized.

How can I take more control of the process? Is there a way to manually manage accelerate with the Trainer and TrainingArguments objects? How about synchronization primitives: something that allows a function to run on the main process before forking to the subprocesses? I tried the decorators, but they cause the Trainer code to crash with the same error.

---

<div class="post-metadata">

**Author:** ![donb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/donb/32/3744_2.png) [@donb](https://discuss.huggingface.co/u/donb)\
**Post date:** [May 22, 2025, 4:45pm UTC](https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255/2 "2025-05-22T16:45:23Z")

</div>

I have worked around this issue by modifying caching\_allocator\_warmup to set the tp\_plan\_regex to None if in addition to `if _torch_distributed_available and torch.distributed.is_initialized()` it checks if `model._tp_plan` is valid:  
`if _torch_distributed_available and torch.distributed.is_initialized() and hasattr(model, '_tp_plan') and model._tp_plan is not None`.

This prevents the failure and ddp is working correctly across multiple invocations inside the Trainers.

I don’t know the implications of this \_tp\_plan modification, but my AI pair programmer suggests that when using accelerate launch and ddp, model.\_tp\_plan should be None. (my pair programmer was not helpful in fixing this naturally - no impactful suggestions). If I understood it better I would create an issue and submit a pull request. For now, I will just monkeypatch it.

---

<div class="post-metadata">

**Author:** ![donb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/donb/32/3744_2.png) [@donb](https://discuss.huggingface.co/u/donb)\
**Post date:** [May 22, 2025, 4:47pm UTC](https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255/3 "2025-05-22T16:47:29Z")

</div>

Also noting that the few issues I’ve found related to the iteration over a None \_tp\_plan is the model’s fault and addressable through proper \_post\_init usage. This seems like a brittle solution and one that won’t scale across all the sources for custom models.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [May 23, 2025, 4:48am UTC](https://discuss.huggingface.co/t/synchronizing-state-trainer-and-accelerate/156255/4 "2025-05-23T04:48:23Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.