# Any reference on how to train (unsupervised learning) longformer model from scratch (not fine-tuning) using Tensorflow?

**URL:** <https://discuss.huggingface.co/t/any-reference-on-how-to-train-unsupervised-learning-longformer-model-from-scratch-not-fine-tuning-using-tensorflow/3998>\
**Category:** 🤗Transformers\
**Created:** [February 25, 2021, 9:16am UTC](https://discuss.huggingface.co/t/any-reference-on-how-to-train-unsupervised-learning-longformer-model-from-scratch-not-fine-tuning-using-tensorflow/3998 "2021-02-25T09:16:22Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sakinaljana](https://avatars.discourse-cdn.com/v4/letter/s/3ab097/32.png) [@sakinaljana](https://discuss.huggingface.co/u/sakinaljana)\
**Post date:** [February 25, 2021, 9:16am UTC](https://discuss.huggingface.co/t/any-reference-on-how-to-train-unsupervised-learning-longformer-model-from-scratch-not-fine-tuning-using-tensorflow/3998/1 "2021-02-25T09:16:23Z")

</div>

Hello folks, i wanna give it a try to use my own dataset to build pre-trained model for longformer, because current pre-trained model only available in English while my task targeting for non-english.

Any reference on how to train longformer model from scratch (not fine-tuning)? Specifically using Tensorflow since i plan to use it for training model for the downstream tasks later.

Thank you in advance!