# Compressing, saving, and loading datasets

**URL:** <https://discuss.huggingface.co/t/compressing-saving-and-loading-datasets/1818>\
**Category:** 🤗Datasets\
**Created:** [October 30, 2020, 12:15pm UTC](https://discuss.huggingface.co/t/compressing-saving-and-loading-datasets/1818 "2020-10-30T12:15:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [October 30, 2020, 12:15pm UTC](https://discuss.huggingface.co/t/compressing-saving-and-loading-datasets/1818/1 "2020-10-30T12:15:00Z")

</div>

Hello everyone,

I am working with large datasets (Wikipedia), and use map transform to create new datasets. The workflow involves creating new datasets that are saved using save\_to\_disk, and subsequently, I use terminal compression utils to compress the dataset folder. Then I decompress these files and the use load\_from\_disk to load them on other machines. These manual steps are pita.

1. It would be great to use compression within datasets and have one compressed file as a result of save\_to\_disk if so desired.

2. If I could save these datasets to a remote location immediatelly bypassing save\_to\_disk, compression, copy manual steps that would be amazing.

3. Loading these created datasets via URL from s3, gs, etc. via a single load\_dataset call would be a killer.

All the best,  
Vladimir

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 30, 2020, 2:24pm UTC](https://discuss.huggingface.co/t/compressing-saving-and-loading-datasets/1818/2 "2020-10-30T14:24:20Z")

</div>

I agree that would be super cool to be able to archive and save/load archived dataset from/to a cloud storage. We’re thinking about this actively. Do you think that some dataset versioning logic could be interesting as well ?

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [October 30, 2020, 4:09pm UTC](https://discuss.huggingface.co/t/compressing-saving-and-loading-datasets/1818/3 "2020-10-30T16:09:01Z")

</div>

Not important for me. But maybe I am overlooking it. How would you use versioning? Like git for datasets?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [November 10, 2020, 3:58pm UTC](https://discuss.huggingface.co/t/compressing-saving-and-loading-datasets/1818/4 "2020-11-10T15:58:34Z")

</div>

Yes maybe something similar to git lfs, directly integrated in the library. We’re already doing that for models in the transformers library (since today’s migration).