# Using Datasets, DataCollators and DataLoaders to create an NLP data pipeline

**URL:** <https://discuss.huggingface.co/t/using-datasets-datacollators-and-dataloaders-to-create-an-nlp-data-pipeline/43550>\
**Category:** 🤗Datasets\
**Created:** [June 16, 2023, 5:49pm UTC](https://discuss.huggingface.co/t/using-datasets-datacollators-and-dataloaders-to-create-an-nlp-data-pipeline/43550 "2023-06-16T17:49:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vikalex](https://avatars.discourse-cdn.com/v4/letter/v/0ea827/32.png) [@vikalex](https://discuss.huggingface.co/u/vikalex)\
**Post date:** [June 16, 2023, 5:49pm UTC](https://discuss.huggingface.co/t/using-datasets-datacollators-and-dataloaders-to-create-an-nlp-data-pipeline/43550/1 "2023-06-16T17:49:03Z")

</div>

It’s hard to have a good grasp of how various libraries and their components interact.

Here are my requirements.

1. I want to pad my texts to maximum length in a batch.
2. I want to shuffle my data after each epoch.

Right now I use the datasets library and use datasets map function to pre tokenize the data and convert it into torch format like

`ds.map(...).with_format("torch")`

But the issue is if I call the dataloader like this with suffle on (I am using Pytorch Lightning)

```auto
    def train_dataloader(self):
        """The training data loader."""
        return DataLoader(
            self.dataset["train"], # type: ignore
            shuffle=True,
            collate_fn=self.data_collator, # Default data collator
            batch_size=self.batch_sizes["train"],
        )

```

Then the original padding is useless and you will get batches with unequal number of tokens.

Maybe I can address this using a different data collator? What is the purpose of a data collator, can I use it somehow to do the padding? The Transformers library warns that if you use a fast tokenizer, it is much faster to pad with the original call instead of separately tokenizing and padding.

What is the most efficient way of doing this pipeline keeping in mind DDP scenarios. If someone has any experience in Pytorch Lightning, I would appreciate if the answer is in that context. But even a general purpose answer should be helpful.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 21, 2023, 1:05pm UTC](https://discuss.huggingface.co/t/using-datasets-datacollators-and-dataloaders-to-create-an-nlp-data-pipeline/43550/2 "2023-06-21T13:05:10Z")

</div>

These concepts are explained in the [course](https://huggingface.co/learn/nlp-course/chapter3/2?fw=pt#dynamic-padding) or Transformers’ task pages/[examples](https://github.com/huggingface/transformers/tree/main/examples/pytorch). The idea is to tokenize the samples without padding in `map` and then pass a task’s data collator to `Trainer`/`DataLoader` to pad the batches.