# How pretrain ELECTRA on custom dataset? **URL:** \ **Category:** Beginners\ **Created:** [September 2, 2020, 6:27pm UTC](https://discuss.huggingface.co/t/how-pretrain-electra-on-custom-dataset/968 "2020-09-02T18:27:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![FeryET](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/feryet/32/2520_2.png) [@FeryET](https://discuss.huggingface.co/u/FeryET)\ **Post date:** [September 2, 2020, 6:28pm UTC](https://discuss.huggingface.co/t/how-pretrain-electra-on-custom-dataset/968/1 "2020-09-02T18:28:00Z")
I have a 1GB raw text dataset in a niche domain. I want to train an ELECTRA model but I couldn’t find any tutorial/examples to do so. Can anyone help me? I tried using the `simpletransformers` package, but it has memory issues at the moment and after a few epochs my colab session crashes. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 3, 2020, 4:15am UTC](https://discuss.huggingface.co/t/how-pretrain-electra-on-custom-dataset/968/2 "2020-09-03T04:15:04Z")
pinging @lysandre ---
**Author:** ![lysandre](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lysandre/32/20820_2.png) [@lysandre](https://discuss.huggingface.co/u/lysandre)\ **Post date:** [September 3, 2020, 7:31am UTC](https://discuss.huggingface.co/t/how-pretrain-electra-on-custom-dataset/968/3 "2020-09-03T07:31:47Z")
Do you want to fine-tune or pre-train an ELECTRA model? If you want to fine-tune it, you can leverage the `examples/run_language_modeling.py` script. If you want to pre-train it, your best bet is to use the original implementation (in TF1) and then convert it to our library using our conversion script which is [here](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py). [Here’s a PR](https://github.com/huggingface/transformers/pull/4656) for pre-training ELECTRA with our library but it’s not working right now and we don’t have the bandwidth to get back into it right now. ---
**Author:** ![FeryET](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/feryet/32/2520_2.png) [@FeryET](https://discuss.huggingface.co/u/FeryET)\ **Post date:** [September 3, 2020, 8:36am UTC](https://discuss.huggingface.co/t/how-pretrain-electra-on-custom-dataset/968/4 "2020-09-03T08:36:30Z")
@lysandre I am planning to pretrain the model, but I would like to also give a shot to `examples/run_language_modeling.py` script. How can I finetune the base Electra model on MLM task on my data? The script is a bit ambigious. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 6, 2020, 7:50am UTC](https://discuss.huggingface.co/t/how-pretrain-electra-on-custom-dataset/968/5 "2020-09-06T07:50:22Z")
I’m not sure if fine-tuning ELECTRA with MLM is good idea since the main idea behind ELECTRA was to train it as a discriminator rather than generator and overcome the problems of MLM’s Any thoughts @lysandre ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 6, 2020, 10:53am UTC](https://discuss.huggingface.co/t/how-pretrain-electra-on-custom-dataset/968/6 "2020-09-06T10:53:40Z")
> [@ELECTRA training reimplementation and discussion](https://discuss.huggingface.co/t/electra-training-reimplementation-and-discussion/1004): > > After months of development and debugging, I finally successfully train a model from scratch and replicate the official results. [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/abs/2003.10555) by Kevin Clark. Minh-Thang Luong. Quoc V. Le. Christopher D. Manning computer Code: [electra\_pytorch](https://github.com/richarddwang/electra_pytorch) AFAIK, the closest reimplementation to the original one, taking care of many easily overlooked details (described below). AFAIK, the only one successfully validate itself by replica…