# Wav2Vec2 Loss Function Question

**URL:** <https://discuss.huggingface.co/t/wav2vec2-loss-function-question/99221>\
**Category:** 🤗Transformers\
**Created:** [July 24, 2024, 7:18pm UTC](https://discuss.huggingface.co/t/wav2vec2-loss-function-question/99221 "2024-07-24T19:18:29Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![priyammaz](https://avatars.discourse-cdn.com/v4/letter/p/a5b964/32.png) [@priyammaz](https://discuss.huggingface.co/u/priyammaz)\
**Post date:** [July 24, 2024, 7:18pm UTC](https://discuss.huggingface.co/t/wav2vec2-loss-function-question/99221/1 "2024-07-24T19:18:29Z")

</div>

So I am working Wav2Vec2 so I can apply it to a new dataset. To do this, I have mainly been looking at the example trainer provided:

> **[transformers/examples/pytorch/speech-pretraining at main ·...](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-pretraining)**
>
> 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - huggingface/transformers

My main question is this:

In the model, when it computes the loss, it adds together all the losses across all the samples. This is seen in both the cross entropy loss for contrastive loss (reduction=“sum”) and the diversity loss, where it sums across the the codevectors and the codebooks.

So the actual loss that we are doing loss.backward() on is pretty large (in the thousands)!

On the other hand, when we the loss is printed out to the console it starts out around 4. Upon further inspection, I see that they divide the loss by the number of masked tokens (basically the average at this point). So if we are printing out the loss with a value around 4, but it is actually training on a loss that is in the thousands, is this correct? Is it ok to have such a large loss value to train on, and why arent we just taking the average before doing loss.backward()?

This might be a dumb question but thanks!!

---

<div class="post-metadata">

**Author:** ![priyammaz](https://avatars.discourse-cdn.com/v4/letter/p/a5b964/32.png) [@priyammaz](https://discuss.huggingface.co/u/priyammaz)\
**Post date:** [July 24, 2024, 9:11pm UTC](https://discuss.huggingface.co/t/wav2vec2-loss-function-question/99221/2 "2024-07-24T21:11:12Z")

</div>

So now that i am looking more carefully, there is also this line of code:

```auto
if accelerator.state.num_processes > 1:
                num_losses = accelerator.gather_for_metrics(num_losses).sum()
                gradient_multiplier = accelerator.state.num_processes / num_losses
                multiply_grads(model.module.parameters(), gradient_multiplier)
else:
         multiply_grads(model.parameters(), 1 / num_losses)

```

Now it makes sense to me. We dont know the number of losses ahead of time across all GPUs because we randomly mask, i guess it effectively is doing the same thing then