# Does saving a shuffled dataset to arrow format eliminate the indirection?

**URL:** <https://discuss.huggingface.co/t/does-saving-a-shuffled-dataset-to-arrow-format-eliminate-the-indirection/127561>\
**Category:** 🤗Datasets\
**Created:** [November 25, 2024, 5:38am UTC](https://discuss.huggingface.co/t/does-saving-a-shuffled-dataset-to-arrow-format-eliminate-the-indirection/127561 "2024-11-25T05:38:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dhruvgrammarly](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dhruvgrammarly](https://discuss.huggingface.co/u/dhruvgrammarly)\
**Post date:** [November 25, 2024, 5:38am UTC](https://discuss.huggingface.co/t/does-saving-a-shuffled-dataset-to-arrow-format-eliminate-the-indirection/127561/1 "2024-11-25T05:38:49Z")

</div>

According to [Differences between Dataset and IterableDataset](https://huggingface.co/docs/datasets/en/about_mapstyle_vs_iterable)

> However as soon as your [Dataset](https://huggingface.co/docs/datasets/v3.1.0/en/package_reference/main_classes#datasets.Dataset) has an indices mapping (via [Dataset.shuffle()](https://huggingface.co/docs/datasets/v3.1.0/en/package_reference/main_classes#datasets.Dataset.shuffle) for example), the speed can become 10x slower. This is because there is an extra step to get the row index to read using the indices mapping, and most importantly, you aren’t reading contiguous chunks of data anymore. To restore the speed, you’d need to rewrite the entire dataset on your disk again using [Dataset.flatten\_indices()](https://huggingface.co/docs/datasets/v3.1.0/en/package_reference/main_classes#datasets.Dataset.flatten_indices), which removes the indices mapping. This may take a lot of time depending on the size of your dataset though:

However, if I save the dataset to disk using `save_to_disk()`, will it save the shuffled dataset in the shuffled order or will it save the dataset in the original order along with the redirection sequence? i.e. will doing random access on the saved dataset be fast or 10x slower?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [December 4, 2024, 10:50am UTC](https://discuss.huggingface.co/t/does-saving-a-shuffled-dataset-to-arrow-format-eliminate-the-indirection/127561/2 "2024-12-04T10:50:47Z")

</div>

Hi ! Save\_to\_disk() re-writes the dataset in the right order, and therefore reloading from there gives you max speed 🙂

---

<div class="post-metadata">

**Author:** ![Spela40](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/spela40/32/36940_2.png) [@Spela40](https://discuss.huggingface.co/u/Spela40)\
**Post date:** [December 4, 2024, 3:25pm UTC](https://discuss.huggingface.co/t/does-saving-a-shuffled-dataset-to-arrow-format-eliminate-the-indirection/127561/3 "2024-12-04T15:25:29Z")

</div>

I’m Spela and I’m here for the first time

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [December 5, 2024, 3:26am UTC](https://discuss.huggingface.co/t/does-saving-a-shuffled-dataset-to-arrow-format-eliminate-the-indirection/127561/4 "2024-12-05T03:26:02Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.