# How to filter samples on-the-fly?

**URL:** <https://discuss.huggingface.co/t/how-to-filter-samples-on-the-fly/14573>\
**Category:** 🤗Datasets\
**Created:** [February 10, 2022, 9:35am UTC](https://discuss.huggingface.co/t/how-to-filter-samples-on-the-fly/14573 "2022-02-10T09:35:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sfalk](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sfalk](https://discuss.huggingface.co/u/sfalk)\
**Post date:** [February 10, 2022, 9:35am UTC](https://discuss.huggingface.co/t/how-to-filter-samples-on-the-fly/14573/1 "2022-02-10T09:35:34Z")

</div>

My question is similar to [this issue](https://discuss.huggingface.co/t/preprocessing-augmentation-during-training/14550/) but this time it’s about filtering.

How can I filter samples on-the-fly without going through the process of writing another cache file. I was able to preprocess samples using `datasets.Dataset.set_transform()`

```auto
train_dataset = make_dataset("my_dataset")
train_dataset = datasets.Dataset(train_dataset.data)
train_dataset.set_transform(preprocess)

```

but how can I do something similar as I want to filter?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 15, 2022, 4:05pm UTC](https://discuss.huggingface.co/t/how-to-filter-samples-on-the-fly/14573/2 "2022-02-15T16:05:19Z")

</div>

I think you would have to use `.filter()` before-hand. I don’t think you can do that on-the-fly on a “map-style dataset” or you can create uintuitive situations. For example what if the first example has to be filtered out and your data loader asks for `train_dataset[0]` ? Should it raise an error ?

There is another type of dataset that can actually filter on-the-fly: “iterable datasets” (or “[streaming datasets](https://huggingface.co/docs/datasets/stream.html)”). Currently you can load a dataset in streaming mode with `load_dataset(..., streaming=True)`. The `filter` method is still to be implemented, though it should be possible to filter-out examples using a batched [map](https://huggingface.co/docs/datasets/stream.html#map), e.g.

```auto
def process_and_filter(batch):
    """This function passed to map can return less examples that the input batches"""
    return {"text": text for text in batch["text"] if len(text) > 0}

ds = load_dataset("path/to/dataset", streaming=True, split="train")
ds = ds.map(process_and_filter, batched=True)

```