# Specifying K-fold splits in a dataset

**URL:** <https://discuss.huggingface.co/t/specifying-k-fold-splits-in-a-dataset/78162>\
**Category:** 🤗Datasets\
**Created:** [March 19, 2024, 9:12pm UTC](https://discuss.huggingface.co/t/specifying-k-fold-splits-in-a-dataset/78162 "2024-03-19T21:12:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![joshvm](https://avatars.discourse-cdn.com/v4/letter/j/d6d6ee/32.png) [@joshvm](https://discuss.huggingface.co/u/joshvm)\
**Post date:** [March 19, 2024, 9:12pm UTC](https://discuss.huggingface.co/t/specifying-k-fold-splits-in-a-dataset/78162/1 "2024-03-19T21:12:56Z")

</div>

Hi,

We have a dataset where our main evaluation metrics are reported via a k-fold cross evaluation + a small, fixed, holdout set. For example, we have 5-10% of the data that is hand selected as a “gold standard” for testing, and we do a 5-fold split of the remaining 90% as the dataset isn’t that large, so that we can train on more images. What is the best/canonical way to share this sort of split using `datasets`? We’d like to distribute the exact splits we used because they’re stratified based on certain attributes of the sample and people ought to be able to replicate results (while also having the option to define their own splits).

Ideally it would be nice to have separation between e.g. labels (which are large) and filename lists (which are small) without having to write a custom load function. It seems like custom loaders are semi-deprecated due to the risk of malicious code execution (i.e. users must opt in), so a purely config-based setup would be best.

The brute force solution would seem to be a bunch of label metadata files `train_kfold_x.jsonl`, but then we have to duplicate the annotation files 10 or more times.

Perhaps a more generic question is ‘How can one specify that a sample is in multiple splits, without duplicating annotation metadata’?

It seems like the `select` function might be a good way to do this, if we provide lists of indices for each split? And then, what’s the best way to distribute those indices with the dataset?

Thanks!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [March 20, 2024, 4:50pm UTC](https://discuss.huggingface.co/t/specifying-k-fold-splits-in-a-dataset/78162/2 "2024-03-20T16:50:18Z")

</div>

You can indeed use `.select()` with the train/validation sets indices 🙂

You can define one configuration of your dataset that would contain the data

```yaml
configs:
- config_name: default
  data_files: train.jsonl

```

and one config for the indices

```yaml
- config_name: "kfold_indices"
  data_files: indices.jsonl

```

This YAML configuration can be placed in the YAML header at the top of the README.md.

See the documentation on Data Files Configuration here: [Manual Configuration](https://huggingface.co/docs/hub/datasets-manual-configuration)