# \`save\_to\_disk\` saving ALL data, even items I filtered out

**URL:** <https://discuss.huggingface.co/t/save-to-disk-saving-all-data-even-items-i-filtered-out/167575>\
**Category:** 🤗Datasets\
**Created:** [August 21, 2025, 4:13pm UTC](https://discuss.huggingface.co/t/save-to-disk-saving-all-data-even-items-i-filtered-out/167575 "2025-08-21T16:13:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![treehugg3](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/treehugg3/32/52887_2.png) [@treehugg3](https://discuss.huggingface.co/u/treehugg3)\
**Post date:** [August 21, 2025, 4:13pm UTC](https://discuss.huggingface.co/t/save-to-disk-saving-all-data-even-items-i-filtered-out/167575/1 "2025-08-21T16:13:28Z")

</div>

When loading a dataset and saving via `save_to_disk` like this:

```py
from datasets import load_dataset

# Load the dataset
dataset = load_dataset(input_dir)

# Filter the dataset
user_wins_dataset = dataset["train"].filter(lambda x: x["winner"] == "user")

# Save the filtered dataset... saves the filtered-out rows, too :(
user_wins_dataset.save_to_disk(output_dir)

```

The written pyarrow files contain the _entire_ dataset, even all the rows I filtered out. This is severely slowing down my data pipeline because the dataset is massive. Why is `datasets` not saving only the filtered rows, as I would expect, and how can I make it do that?

When I load the saved dataset, the `len` is correct (24680), much reduced, but I notice the `dataset_info.json` file contains the original size (28608139).

`flatten_indices` doesn’t appear to help:

```py
d2 = d.flatten_indices(keep_in_memory=True)
d2.save_to_disk("flattened") # Same size, all rows saved :(

```

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [August 21, 2025, 4:34pm UTC](https://discuss.huggingface.co/t/save-to-disk-saving-all-data-even-items-i-filtered-out/167575/2 "2025-08-21T16:34:15Z")

</div>

Hi ! which version of python and `datasets` are you using ? Does it also happen with `datasets==4.0.0` ?

---

<div class="post-metadata">

**Author:** ![treehugg3](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/treehugg3/32/52887_2.png) [@treehugg3](https://discuss.huggingface.co/u/treehugg3)\
**Post date:** [August 21, 2025, 4:42pm UTC](https://discuss.huggingface.co/t/save-to-disk-saving-all-data-even-items-i-filtered-out/167575/3 "2025-08-21T16:42:13Z")

</div>

Hi! I was on `datasets==2.21.0`, so after reading your reply (thank you!) I updated to `datasets==4.0.0` as you suggested, and tried it again, including using the `flatten_indices` function. Unfortunately even on this version the size of the saved dataset is still the same as the size of the original, even though `len(d2) == 24680` (meaning the filter was in place for this object), so there is still an issue.