# How to use Data Collator?

**URL:** <https://discuss.huggingface.co/t/how-to-use-data-collator/5794>\
**Category:** Beginners\
**Created:** [April 26, 2021, 8:42pm UTC](https://discuss.huggingface.co/t/how-to-use-data-collator/5794 "2021-04-26T20:42:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Constantin](https://avatars.discourse-cdn.com/v4/letter/c/74df32/32.png) [@Constantin](https://discuss.huggingface.co/u/Constantin)\
**Post date:** [April 26, 2021, 8:42pm UTC](https://discuss.huggingface.co/t/how-to-use-data-collator/5794/1 "2021-04-26T20:42:09Z")

</div>

I want to train transformer TF model for NER with my pipeline. I have a problem with alignment of labels. As I understand for this task one uses DataCollatorForTokenClassification. But I can’t figure out how to use it outside of Trainer to get aligned labels.

Just to clearify what do I mean:

tokens: [‘Europe’,‘is’,‘international’]  
labels: [‘1’,‘0’.‘0’]  
input\_ids: [‘545’,‘43’,‘6343’,‘2334’,‘2’]

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 26, 2021, 8:56pm UTC](https://discuss.huggingface.co/t/how-to-use-data-collator/5794/2 "2021-04-26T20:56:44Z")

</div>

hey @Constantin you should be able to use the `tokenize_and_align_labels` function from here: [transformers/run\_ner\_no\_trainer.py at bc2571e61c985ec82819cf01ad038342771c94d0 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/bc2571e61c985ec82819cf01ad038342771c94d0/examples/pytorch/token-classification/run_ner_no_trainer.py#L324)

you could also try adapting the pytorch code to tensorflow for the training loop 🙂