# Pretrain longT5

**URL:** <https://discuss.huggingface.co/t/pretrain-longt5/49541>\
**Category:** Beginners\
**Created:** [August 5, 2023, 12:44am UTC](https://discuss.huggingface.co/t/pretrain-longt5/49541 "2023-08-05T00:44:22Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![enpassant](https://avatars.discourse-cdn.com/v4/letter/e/e95f7d/32.png) [@enpassant](https://discuss.huggingface.co/u/enpassant)\
**Post date:** [August 5, 2023, 12:44am UTC](https://discuss.huggingface.co/t/pretrain-longt5/49541/1 "2023-08-05T00:44:22Z")

</div>

I would like to pretrain and then fine tune on longT5 on a custom dataset. Ideally, I would train a tokenizer on my data, then merge tokenizers with longT5 tokenizer and then pretrain from the published longT5 checkpoint. has anyone tried this or is aware of any good resources?