# Evaluate Model on Test dataset (PPL)

**URL:** <https://discuss.huggingface.co/t/evaluate-model-on-test-dataset-ppl/6528>\
**Category:** Beginners\
**Created:** [June 2, 2021, 1:42pm UTC](https://discuss.huggingface.co/t/evaluate-model-on-test-dataset-ppl/6528 "2021-06-02T13:42:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChrisDelClea](https://avatars.discourse-cdn.com/v4/letter/c/e480ec/32.png) [@ChrisDelClea](https://discuss.huggingface.co/u/ChrisDelClea)\
**Post date:** [June 2, 2021, 1:42pm UTC](https://discuss.huggingface.co/t/evaluate-model-on-test-dataset-ppl/6528/1 "2021-06-02T13:42:43Z")

</div>

Hi guys,

i am kinda new to hugginface and have a question regarding the PPL.  
So what i have is, i fine-tuned a model and at the end of the traning i get the PPL  
for the dev dataset by doing:

```auto
eval_results = trainer.evaluate()
print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")

```

However, in my DatasetsDict I have train, dev and test and also want to get the PPL  
for the test set. Is there any simple way do get it done?

Best regards  
Chris

---

<div class="post-metadata">

**Author:** ![Felipehonorato](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/felipehonorato/32/48418_2.png) [@Felipehonorato](https://discuss.huggingface.co/u/Felipehonorato)\
**Post date:** [June 2, 2021, 8:23pm UTC](https://discuss.huggingface.co/t/evaluate-model-on-test-dataset-ppl/6528/2 "2021-06-02T20:23:06Z")

</div>

Hey Chris, i’m a begginer as well but i think you could try using [EvalPrediction](https://huggingface.co/transformers/internal/trainer_utils.html#transformers.EvalPrediction). You pass your test\_set and it’s labels and then you should get your loss at the specific set.

---

<div class="post-metadata">

**Author:** ![ChrisDelClea](https://avatars.discourse-cdn.com/v4/letter/c/e480ec/32.png) [@ChrisDelClea](https://discuss.huggingface.co/u/ChrisDelClea)\
**Post date:** [June 4, 2021, 9:54am UTC](https://discuss.huggingface.co/t/evaluate-model-on-test-dataset-ppl/6528/3 "2021-06-04T09:54:32Z")

</div>

Hi @Felipehonorato ,

thanks for your response. I am not sure if i understood it correct.  
In the doc you send it says it’s used to compute the metrics.  
So if i get it right, it just returns an object of the class `EvalPrediction`.

However, what i would like to have is the loss itself.  
I found i could use trainer.predict(test\_dataset) but it always gives me and error:

```auto
untimeError: CUDA out of memory. Tried to allocate

```

So i don’t really get why trainer.evaluate() works but predict does not. Any ideas?

Also it would be useful if anyone could send me link where the difference between  
`trainer.predict()` and `trainer.prediction_loop()` , `.trainer.prediction_step()` is explained in detail.

---

<div class="post-metadata">

**Author:** ![Namrata](https://avatars.discourse-cdn.com/v4/letter/n/41988e/32.png) [@Namrata](https://discuss.huggingface.co/u/Namrata)\
**Post date:** [June 10, 2021, 2:36pm UTC](https://discuss.huggingface.co/t/evaluate-model-on-test-dataset-ppl/6528/4 "2021-06-10T14:36:08Z")

</div>

Did you try decreasing the batch size?