# Training GPT-2 from scratch

**URL:** <https://discuss.huggingface.co/t/training-gpt-2-from-scratch/562>\
**Category:** Beginners\
**Created:** [August 2, 2020, 7:22am UTC](https://discuss.huggingface.co/t/training-gpt-2-from-scratch/562 "2020-08-02T07:22:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![miguelvictor](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/miguelvictor/32/231_2.png) [@miguelvictor](https://discuss.huggingface.co/u/miguelvictor)\
**Post date:** [August 2, 2020, 7:22am UTC](https://discuss.huggingface.co/t/training-gpt-2-from-scratch/562/1 "2020-08-02T07:22:58Z")

</div>

Hello!

I’m currently working on a toy project that uses GPT-2 (smallest variant but only 6 layers, from scratch) to predict next tokens in the context of programming languages. So my dataset are all source codes and I am using a custom tokenizer and i have the following questions:

1. If my sample is longer than 1024 tokens (supposing the model’s max length is 1024), is the past tokens automatically fed back to the model during training? or should I do it myself?
2. My custom tokenizer works well (in my opinion) but i want to use the huggingface API to take advantage of the “fast” tokenizers. How do I go about subclassing the Tokenizer class so that my tokenizer is compatible with huggingface’s tokenizer api ?

Thank you very much!!!

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 2, 2020, 2:31pm UTC](https://discuss.huggingface.co/t/training-gpt-2-from-scratch/562/2 "2020-08-02T14:31:33Z")

</div>

Hi @miguelvictor,

1. You can train you tokenizer using the`[tokenizers](https://github.com/huggingface/tokenizers) library. These are fast rust tokenizers with python API.

---

<div class="post-metadata">

**Author:** ![miguelvictor](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/miguelvictor/32/231_2.png) [@miguelvictor](https://discuss.huggingface.co/u/miguelvictor)\
**Post date:** [August 3, 2020, 3:41pm UTC](https://discuss.huggingface.co/t/training-gpt-2-from-scratch/562/3 "2020-08-03T15:41:04Z")

</div>

Hello. Thank you for replying.

Do you have any ideas about my question#1? I tried looking at the source code and it seems no “past” is given to the model during training for longer sequences (or maybe I’m wrong)