# Does masked language model training script does random shuffle on the dataset?

**URL:** <https://discuss.huggingface.co/t/does-masked-language-model-training-script-does-random-shuffle-on-the-dataset/11197>\
**Category:** 🤗Datasets\
**Created:** [October 29, 2021, 11:08am UTC](https://discuss.huggingface.co/t/does-masked-language-model-training-script-does-random-shuffle-on-the-dataset/11197 "2021-10-29T11:08:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Emanuel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/emanuel/32/20406_2.png) [@Emanuel](https://discuss.huggingface.co/u/Emanuel)\
**Post date:** [October 29, 2021, 11:08am UTC](https://discuss.huggingface.co/t/does-masked-language-model-training-script-does-random-shuffle-on-the-dataset/11197/1 "2021-10-29T11:08:14Z")

</div>

I haven’t seen this been explicitly defined in the [code](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py), although in the “no trainer” [script](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm_no_trainer.py) the shuffling is defined. Internally in `datasets` the shuffling is done or is it missing in the MLM with `trainer`?  
If it’s missing, I can contribute with a PR 🙂

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [October 29, 2021, 11:29am UTC](https://discuss.huggingface.co/t/does-masked-language-model-training-script-does-random-shuffle-on-the-dataset/11197/2 "2021-10-29T11:29:41Z")

</div>

~~Shuffle is not enabled in the default dataloaders in the trainer. If you want to add a PR, you need to add an argument to the TrainingArguments and update the Trainer dataloaders.~~

See comment by @sgugger below.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [October 29, 2021, 11:44am UTC](https://discuss.huggingface.co/t/does-masked-language-model-training-script-does-random-shuffle-on-the-dataset/11197/3 "2021-10-29T11:44:53Z")

</div>

Hi @Emanuel,

I’m not sure about your setup, so could you run the following after the `trainer` is initialized and copy-and-paste the value here:

```python
trainer.get_train_dataloader().sampler

```

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 29, 2021, 11:49am UTC](https://discuss.huggingface.co/t/does-masked-language-model-training-script-does-random-shuffle-on-the-dataset/11197/4 "2021-10-29T11:49:47Z")

</div>

> [@BramVanroy](#):
>
> Shuffle is not enabled in the default dataloaders in the trainer.

That is incorrect. The training dataloader is always defined with shuffle=True (more precisely with a random sampler because we have to handle distributed training, but that’s the same as not passing a sampler and pass `shuffle=True`).

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [October 29, 2021, 12:22pm UTC](https://discuss.huggingface.co/t/does-masked-language-model-training-script-does-random-shuffle-on-the-dataset/11197/5 "2021-10-29T12:22:49Z")

</div>

My bad, I only checked the dataloader definition but missed the sampler.