# How to train a transformer from scratch

**URL:** <https://discuss.huggingface.co/t/how-to-train-a-transformer-from-scratch/84901>\
**Category:** Beginners\
**Created:** [May 2, 2024, 9:41pm UTC](https://discuss.huggingface.co/t/how-to-train-a-transformer-from-scratch/84901 "2024-05-02T21:41:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![IParraMartin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iparramartin/32/26521_2.png) [@IParraMartin](https://discuss.huggingface.co/u/IParraMartin)\
**Post date:** [May 2, 2024, 9:41pm UTC](https://discuss.huggingface.co/t/how-to-train-a-transformer-from-scratch/84901/1 "2024-05-02T21:41:11Z")

</div>

Hi all!

I am trying to train a transformer from scratch using either HuggingFace or the NanoGPT repository. The task will be to generate questions from declarative sentences (e.g., _the dog does bark → Does the dog bark?_).

My problems are:

1. Where can I find a good and straightforward way to train a transformer from scratch?
2. How do I structure the dataset? I was thinking of something like this:  
_The dog does bark \n Does the dog bark?[EOS]_

Thank you all!🤗

---

<div class="post-metadata">

**Author:** ![nimatov](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimatov/32/18720_2.png) [@nimatov](https://discuss.huggingface.co/u/nimatov)\
**Post date:** [May 2, 2024, 9:59pm UTC](https://discuss.huggingface.co/t/how-to-train-a-transformer-from-scratch/84901/2 "2024-05-02T21:59:14Z")

</div>

I found this one useful when I tried to understand how transformers work.  
[https://nlp.seas.harvard.edu/annotated-transformer/#encoder-and-decoder-stacks](https://nlp.seas.harvard.edu/annotated-transformer/#encoder-and-decoder-stacks)