# Performance tips for shuffle and flatten\_indices **URL:** \ **Category:** 🤗Datasets\ **Created:** [November 18, 2020, 5:25pm UTC](https://discuss.huggingface.co/t/performance-tips-for-shuffle-and-flatten-indices/2117 "2020-11-18T17:25:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 18, 2020, 5:25pm UTC](https://discuss.huggingface.co/t/performance-tips-for-shuffle-and-flatten-indices/2117/1 "2020-11-18T17:25:06Z")
Hey @lhoestq, How can I speed up shuffle+flatten on a dataset with millions of instances? It’s painfully slow for whatever setting I tried. TIA ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 18, 2020, 5:40pm UTC](https://discuss.huggingface.co/t/performance-tips-for-shuffle-and-flatten-indices/2117/2 "2020-11-18T17:40:45Z")
Hi ! By flatten you mean flattend\_indices ? Is your dataset made of strings ? If so, then the speed bottleneck is the I/O: read from the current dataset arrow file, and write to a new file. The new file is written when doing flatten\_indices. To speed up things you can use a SSD or distribute the writing (using shard on the shuffled dataset for example). ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 18, 2020, 6:12pm UTC](https://discuss.huggingface.co/t/performance-tips-for-shuffle-and-flatten-indices/2117/3 "2020-11-18T18:12:48Z")
Yes, flatten\_indices. Not strings, int arrays of various precisions. Let me backtrack. I need to concatenate two large datasets, all features are int arrays. Now, concatenate works fast except the dataset is not shuffled. I need a shuffled dataset and I don’t want it to use indices, caches etc. Just clean dataset (dataset.arrow, dataset\_info.json, and state.json). I found those to be the fastest when loading and processing. How could I do that? ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 20, 2020, 2:30pm UTC](https://discuss.huggingface.co/t/performance-tips-for-shuffle-and-flatten-indices/2117/4 "2020-11-20T14:30:31Z")
I tried flatten\_indices on a dataset with 30 million examples, the progress bar indicated a running time of 28 hours. How can we speed it up? ---
**Author:** ![jxm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jxm/32/29614_2.png) [@jxm](https://discuss.huggingface.co/u/jxm)\ **Post date:** [December 5, 2024, 4:20pm UTC](https://discuss.huggingface.co/t/performance-tips-for-shuffle-and-flatten-indices/2117/5 "2024-12-05T16:20:33Z")
I have the same question! A performance guide for these kinds of simple tasks with Datasets would be great. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [December 11, 2024, 3:38pm UTC](https://discuss.huggingface.co/t/performance-tips-for-shuffle-and-flatten-indices/2117/6 "2024-12-11T15:38:08Z")
My guide is passing `num_proc=` to make it faster ^^’ We also did various optimizations since 2020 so make sure to use a recent version of `datasets`