# How does Hugging Face Hub jointly versions models and their training data?

**URL:** <https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059>\
**Category:** 🤗Hub\
**Created:** [January 6, 2023, 10:45am UTC](https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059 "2023-01-06T10:45:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\
**Post date:** [January 6, 2023, 10:45am UTC](https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059/1 "2023-01-06T10:45:43Z")

</div>

Hi,

I see HF Models are versioned, and HF Datasets are versioned too.  
How do I know which version of a dataset went into training a given version of a model? (interested in answer both for public and private hubs)

---

<div class="post-metadata">

**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](https://discuss.huggingface.co/u/julien-c)\
**Post date:** [January 8, 2023, 2:49pm UTC](https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059/2 "2023-01-08T14:49:45Z")

</div>

Hi Olivier,

in the model metadata spec, you’ll see we suggest encoding the version (= git-revision) of the dataset inside the model-index dict, i.E. the place where you encode your eval results in a structured way:

> <https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1#L31>

models that are pushed by the hf trainer should include that metadata most of the time

cc @sgugger @osanseviero

---

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\
**Post date:** [January 9, 2023, 10:29am UTC](https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059/3 "2023-01-09T10:29:16Z")

</div>

> [@julien-c](#):
>
> models that are pushed by the hf trainer should include that metadata most of the time

→ Does the HF Trainer automatically copy the git hash of the training dataset (if it detects it’s an HF Dataset) to the model metadata?

thanks

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [January 9, 2023, 5:47pm UTC](https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059/4 "2023-01-09T17:47:19Z")

</div>

We don’t as I’m not sure we have that information in the dataset metadata (maybe it was added recently in which case we can make a PR to add support for this).

---

<div class="post-metadata">

**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](https://discuss.huggingface.co/u/julien-c)\
**Post date:** [January 9, 2023, 6:10pm UTC](https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059/5 "2023-01-09T18:10:58Z")

</div>

yes i think that should be possible, cc @lhoestq

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 13, 2023, 4:11pm UTC](https://discuss.huggingface.co/t/how-does-hugging-face-hub-jointly-versions-models-and-their-training-data/29059/6 "2023-01-13T16:11:55Z")

</div>

Yup that will definitely be possible, there’s an issue on this here but we haven’t started working on this yet: [dataset metadata for reproducibility · Issue #4129 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/4129) Contributions are welcome ^^
