# Evaluating your model on more than one dataset

**URL:** <https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544>\
**Category:** Beginners\
**Created:** [October 15, 2020, 8:03am UTC](https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544 "2020-10-15T08:03:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![konradboguslav](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/konradboguslav/32/473_2.png) [@konradboguslav](https://discuss.huggingface.co/u/konradboguslav)\
**Post date:** [October 15, 2020, 8:03am UTC](https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544/1 "2020-10-15T08:03:05Z")

</div>

Hi,

Transformer’s Trainer and Trainingarguments classes allow for only one dataset to use for evaluation. Is there a simple way of adding another one? So, after after an epoch of training my model I could evaluate it on both training and developmental datasets and get metrics for both of them as one output? I know I could alter the training\_args.py or trainer.py but I am pretty sure I would only mess things up…

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 15, 2020, 1:29pm UTC](https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544/2 "2020-10-15T13:29:32Z")

</div>

I think the easiest way to do this is to use the new system of [`TrainerCallback`](https://huggingface.co/transformers/master/main_classes/callback.html#trainercallback) and write a callback that performs a new evaluation on your other datasets during the event `on_validate`.

---

<div class="post-metadata">

**Author:** ![aswinsson](https://avatars.discourse-cdn.com/v4/letter/a/a9a28c/32.png) [@aswinsson](https://discuss.huggingface.co/u/aswinsson)\
**Post date:** [January 4, 2021, 10:28am UTC](https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544/3 "2021-01-04T10:28:30Z")

</div>

Is it possible to provide an example?

---

<div class="post-metadata">

**Author:** ![deathcrush](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@deathcrush](https://discuss.huggingface.co/u/deathcrush)\
**Post date:** [February 28, 2022, 4:58pm UTC](https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544/4 "2022-02-28T16:58:25Z")

</div>

@sgugger, I had a brief look at the interfaces provided to achieve this, but I don’t see how this is possible. As I understand, the `TrainerCallback` class receives the `TrainerArguments` which are used to initialise the `Trainer` but this class does not allow us to pass additional `eval_dataloaders`. Therefore, I am not sure how to pass the additional datasets to the `Trainer` in the first place.