# GPT-J training from scratch error

**URL:** <https://discuss.huggingface.co/t/gpt-j-training-from-scratch-error/34103>\
**Category:** Beginners\
**Created:** [March 18, 2023, 2:47pm UTC](https://discuss.huggingface.co/t/gpt-j-training-from-scratch-error/34103 "2023-03-18T14:47:29Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lokiii](https://avatars.discourse-cdn.com/v4/letter/l/e47774/32.png) [@Lokiii](https://discuss.huggingface.co/u/Lokiii)\
**Post date:** [March 18, 2023, 2:47pm UTC](https://discuss.huggingface.co/t/gpt-j-training-from-scratch-error/34103/1 "2023-03-18T14:47:29Z")

</div>

> ValueError: The model did not return a loss from the inputs, only the following keys: last\_hidden\_state,past\_key\_values. For reference, the inputs it received are input\_ids,attention\_mask.

```auto
from datasets import load_dataset
dataset = load_dataset("text", data_files="Filosofia Teoretica - Dataset - GPT-J.txt")

from transformers import AutoTokenizer
Tokenizer = AutoTokenizer.from_pretrained("Filosofia Teoretica - Tokenizer - GPT-J")

def TokenizeDataset( inputs ):
    return Tokenizer(inputs['text'], padding = "max_length", max_length = 1024, truncation = True)

TDataset = dataset.map(TokenizeDataset, batched = True)

from transformers import GPTJModel, GPTJConfig
configuration = GPTJConfig(n_embd = 128, vocab_size = 20000)
model = GPTJModel(configuration)

from transformers import Trainer
from transformers import TrainingArguments

training_args = TrainingArguments(output_dir = "Filosofia Teoretica - Model - GPT-J", evaluation_strategy = "epoch", num_train_epochs = 3)

model_trainer = Trainer(
    model = model,
    args = training_args,
    train_dataset = TDataset['train'],
)

model_trainer.train()

```

What I’m doing wrong? Can someone help?