# Data collation: cannot understand the logics of the API

**URL:** <https://discuss.huggingface.co/t/data-collation-cannot-understand-the-logics-of-the-api/104932>\
**Category:** 🤗Transformers\
**Created:** [September 2, 2024, 10:07am UTC](https://discuss.huggingface.co/t/data-collation-cannot-understand-the-logics-of-the-api/104932 "2024-09-02T10:07:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![thistlillo](https://avatars.discourse-cdn.com/v4/letter/t/51bf81/32.png) [@thistlillo](https://discuss.huggingface.co/u/thistlillo)\
**Post date:** [September 2, 2024, 10:07am UTC](https://discuss.huggingface.co/t/data-collation-cannot-understand-the-logics-of-the-api/104932/1 "2024-09-02T10:07:01Z")

</div>

I am a bit confused about the logics behind the data collators for seq2seq models in the Hugging Face library.

My training set is composed by N subsets of pairs (source\_text, target\_text) with the goal of training a sequence-to-sequence model (specifically in the T5 family) to generate the target\_text from the input source\_text.

The experiments are based on a cross-validation where N-1 subsets are used for training and the remaining one for test.

I do not “see” how to correctly perform the encoding (tokenisation + truncation + padding) for training the model.

I want the model to be trained on input sequences of length N\_in\_train to output sequence of length N\_out\_train, i.e., I want to compute two lengths N\_in\_train and N\_out\_train rom the data in the training set.

Since the encoding (tokenisation) takes a long time (several minutes), this is what I did:

- pre-encode the entire dataset
- save the encoded dataset.

When training the model, for each iteration of the cross-validation I want to compute the length of the input and output sequences:

- compute the max length of the sources and the max length of the targets.
- From these two values, I then set N\_in\_train, N\_out\_train.
- Then I want to train and test the model using the two values above. I need to truncate and pad sequences based on N\_in\_train and N\_out\_train.

Now, I would like to use a data collator to truncate and pad the batches in both the training and the test set. For example I wrote this (where max\_source\_length and max\_target\_length correspond to, respectively, N\_in\_train and N\_out\_train):

```auto
class MyCollator(DataCollatorForSeq2Seq):
    def __init__ (self, tokenizer, max_source_length, max_target_length, model=None, padding=True, ignore_label=-100):
        super(). __init__ (tokenizer, model, padding, max_length)
        self.max_source_len = max_source_length
        self.max_target_len = max_target_length
        self.ignore_label = ignore_label
    #<

    def __call__ (self, features):
        for feature in features:
            feature['input_ids'] = feature['input_ids'][:self.max_source_len]
            feature['attention_mask'] = feature['attention_mask'][:self.max_source_len]
            feature['labels'] = feature['labels'][:self.max_target_len]

            feature['input_ids'] = torch.tensor(
                feature['input_ids'] + [self.tokenizer.pad_token_id] * (self.max_source_len - len(feature['input_ids'])),
                dtype=torch.long
            )
            feature['attention_mask'] = torch.tensor(
                feature['attention_mask'] + [0] * (self.max_source_len - len(feature['attention_mask'])),
                dtype=torch.long
            )

            feature['labels'] = torch.tensor(
                feature['labels'] + [self.ignore_label] * (self.max_target_len - len(feature['labels'])),
                dtype=torch.long
            )
        
        return super(). __call__ (features)

```

However, it seems too clumsy for a very basic pre-processing step. Additionally, looking at the docs here: [Data Collator](https://huggingface.co/docs/transformers/main_classes/data_collator#transformers.DataCollatorForSeq2Seq.max_length)

I see only one parameters `max_length` and not two for, respectively, the input and the output sequence. Why?

What is the best way to pre-tokenise a dataset (without truncation and padding), then specify only at training time the length of the various sequences?