# Multi-gpu training - condition to stop the training computed in the main process - broadcast?

**URL:** <https://discuss.huggingface.co/t/multi-gpu-training-condition-to-stop-the-training-computed-in-the-main-process-broadcast/20577>\
**Category:** 🤗Accelerate\
**Created:** [July 19, 2022, 3:25pm UTC](https://discuss.huggingface.co/t/multi-gpu-training-condition-to-stop-the-training-computed-in-the-main-process-broadcast/20577 "2022-07-19T15:25:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![VictorSanh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/victorsanh/32/21405_2.png) [@VictorSanh](https://discuss.huggingface.co/u/VictorSanh)\
**Post date:** [July 19, 2022, 3:25pm UTC](https://discuss.huggingface.co/t/multi-gpu-training-condition-to-stop-the-training-computed-in-the-main-process-broadcast/20577/1 "2022-07-19T15:25:22Z")

</div>

Hi!

I am using accelerate for multi-gpu training.  
The main process computes a boolean condition on whether we need to stop the training (to be precise, whether a certain time limit is exceeded). I only want the main process to compute that condition so that all processes are on the same page.

How can I do that?

Should I broadcast a boolean tensor with `accelerator.broadcast` once the condition is computed in the main process? Are there alternatives to that approach?

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [July 19, 2022, 3:46pm UTC](https://discuss.huggingface.co/t/multi-gpu-training-condition-to-stop-the-training-computed-in-the-main-process-broadcast/20577/2 "2022-07-19T15:46:37Z")

</div>

> [@VictorSanh](#):
>
> How can I do that?

You could use the Accelerator.main\_process\_first() context manager to do the calculation, followed then by the broadcast to ensure all devices get it if on the main process otherwise do nothing.

Since you’d use main\_process\_first this also then uses wait\_for\_everyone so nothing continues until everyone has reached the state of completion.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [July 19, 2022, 4:13pm UTC](https://discuss.huggingface.co/t/multi-gpu-training-condition-to-stop-the-training-computed-in-the-main-process-broadcast/20577/3 "2022-07-19T16:13:57Z")

</div>

To complete a bit the answer, here is how to broadcast:

```nohighlight
from accelerate.utils import broadcast_object_list

end_of_training = [False]
if accelerator.is_main_process():
    end_of_training = [my_condition_on_main_process()]
accelerator.wait_for_everyone()
# This is in place which is why end_of_training needs to be a list
broadcast_object_list(end_of_training)
if end_of_training[0]:
    break_loop

```

I’m not entirely sure `accelerator.wait_for_everyone()` is needed since the broadcast should wait for all processes to get there, but it doesn’t hurt either.

---

<div class="post-metadata">

**Author:** ![VictorSanh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/victorsanh/32/21405_2.png) [@VictorSanh](https://discuss.huggingface.co/u/VictorSanh)\
**Post date:** [July 19, 2022, 5:02pm UTC](https://discuss.huggingface.co/t/multi-gpu-training-condition-to-stop-the-training-computed-in-the-main-process-broadcast/20577/4 "2022-07-19T17:02:04Z")

</div>

works perfectly, thank you @muellerzr and @sgugger