# Problem with data collator

**URL:** <https://discuss.huggingface.co/t/problem-with-data-collator/84961>\
**Category:** Beginners\
**Created:** [May 3, 2024, 7:49am UTC](https://discuss.huggingface.co/t/problem-with-data-collator/84961 "2024-05-03T07:49:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rychlik](https://avatars.discourse-cdn.com/v4/letter/r/7c8e57/32.png) [@rychlik](https://discuss.huggingface.co/u/rychlik)\
**Post date:** [May 3, 2024, 7:49am UTC](https://discuss.huggingface.co/t/problem-with-data-collator/84961/1 "2024-05-03T07:49:36Z")

</div>

I have a dataset used to train a token classification model.  
This dataset contains 3 columns: tokens, labels and frequencies (float numbers).  
These float values will be concatenated to last\_hidden\_state, just before calling classifier.  
Unfortunately, the DataCollatorForTokenClassification does not accept “frequencies” column.  
I am receiving the following error:

File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/tokenization\_utils\_base.py”, line 748, in convert\_to\_tensors  
tensor = as\_tensor(value)  
^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/tokenization\_utils\_base.py”, line 720, in as\_tensor  
return torch.tensor(value)  
^^^^^^^^^^^^^^^^^^^  
ValueError: expected sequence of length 5 at dim 1 (got 14)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):  
File “/Users/chopin/PycharmProjects/term-extractor/train.py”, line 136, in   
trainer.train()  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/trainer.py”, line 1591, in train  
return inner\_training\_loop(  
^^^^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/trainer.py”, line 1870, in \_inner\_training\_loop  
for step, inputs in enumerate(epoch\_iterator):  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/accelerate/data\_loader.py”, line 384, in **iter**  
current\_batch = next(dataloader\_iter)  
^^^^^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/torch/utils/data/dataloader.py”, line 630, in **next**  
data = self.\_next\_data()  
^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/torch/utils/data/dataloader.py”, line 674, in \_next\_data  
data = self.\_dataset\_fetcher.fetch(index) # may raise StopIteration  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/torch/utils/data/\_utils/fetch.py”, line 54, in fetch  
return self.collate\_fn(data)  
^^^^^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/data/data\_collator.py”, line 45, in **call**  
return self.torch\_call(features)  
^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/data/data\_collator.py”, line 310, in torch\_call  
batch = self.tokenizer.pad(  
^^^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/tokenization\_utils\_base.py”, line 3303, in pad  
return BatchEncoding(batch\_outputs, tensor\_type=return\_tensors)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/tokenization\_utils\_base.py”, line 223, in **init**  
self.convert\_to\_tensors(tensor\_type=tensor\_type, prepend\_batch\_axis=prepend\_batch\_axis)  
File “/Users/chopin/anaconda3/envs/learn/lib/python3.11/site-packages/transformers/tokenization\_utils\_base.py”, line 764, in convert\_to\_tensors  
raise ValueError(  
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length. Perhaps your features (`frequencies` in this case) have excessive nesting (inputs type `list` where type `int` is expected).  
0%| | 0/27360 [00:00\<?, ?it/s]

What should I do?

---

<div class="post-metadata">

**Author:** ![KateWinslet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/katewinslet/32/26764_2.png) [@KateWinslet](https://discuss.huggingface.co/u/KateWinslet)\
**Post date:** [May 8, 2024, 12:50pm UTC](https://discuss.huggingface.co/t/problem-with-data-collator/84961/2 "2024-05-08T12:50:17Z")

</div>

To resolve the error, preprocess the dataset by padding or truncating the “frequencies” column to ensure consistent lengths. Modify the DataCollatorForTokenClassification to handle the “frequencies” column or write a custom collation function to include it appropriately[.](http://myolsd.top)