# How to save datasets as distributed with save\_to\_disk?

**URL:** <https://discuss.huggingface.co/t/how-to-save-datasets-as-distributed-with-save-to-disk/25674>\
**Category:** 🤗Datasets\
**Created:** [November 7, 2022, 10:46am UTC](https://discuss.huggingface.co/t/how-to-save-datasets-as-distributed-with-save-to-disk/25674 "2022-11-07T10:46:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![fujindemi](https://avatars.discourse-cdn.com/v4/letter/f/7cd45c/32.png) [@fujindemi](https://discuss.huggingface.co/u/fujindemi)\
**Post date:** [November 7, 2022, 10:46am UTC](https://discuss.huggingface.co/t/how-to-save-datasets-as-distributed-with-save-to-disk/25674/1 "2022-11-07T10:46:07Z")

</div>

I have a lot of data to save. Is it possible to save data into multiple files and load multiple files together?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [November 15, 2022, 5:54pm UTC](https://discuss.huggingface.co/t/how-to-save-datasets-as-distributed-with-save-to-disk/25674/2 "2022-11-15T17:54:21Z")

</div>

Hi ! Right now you have to shard the dataset yourself to save multiple files, but I’m working on supporting saving into multiple files, it will be available soon 😉

In the meantime you can do:

```python
ds = load_dataset(...)
num_shards = 32
for shard_idx in range(num_shards):
    shard = ds.shard(num_shards=num_shards, index=shard_idx, contiguous=True)
    shard.save_to_disk(f"path/to/shard_{shard_idx}")

# reload later
from datasets import load_from_disk, concatenate_datasets
ds = concatenate_datasets([
    load_from_disk(f"path/to/shard_{shard_idx}")
    for shard_idx in range(num_shards)
])
```