# IterableDataset.from\_generator with iterator

**URL:** <https://discuss.huggingface.co/t/iterabledataset-from-generator-with-iterator/62049>\
**Category:** 🤗Datasets\
**Created:** [November 13, 2023, 9:48am UTC](https://discuss.huggingface.co/t/iterabledataset-from-generator-with-iterator/62049 "2023-11-13T09:48:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cwallenwein](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cwallenwein/32/21752_2.png) [@cwallenwein](https://discuss.huggingface.co/u/cwallenwein)\
**Post date:** [November 13, 2023, 9:48am UTC](https://discuss.huggingface.co/t/iterabledataset-from-generator-with-iterator/62049/1 "2023-11-13T09:48:08Z")

</div>

Hi Friends 👋  
I want to create an IterableDataset from an iterator like this

```auto
def gen():
     sample = next(iterator)
     yield sample
ds = Dataset.from_generator(gen)

```

but it throws an error  
`TypeError: cannot pickle 'generator' object`

Is there a way to create a dataset like this?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [November 14, 2023, 12:56pm UTC](https://discuss.huggingface.co/t/iterabledataset-from-generator-with-iterator/62049/2 "2023-11-14T12:56:39Z")

</div>

Currently `from_generator` requires picklable generator functions (this is because we hash the function using pickle to be able to cache the dataset on disk)

A workaround is to use a callable picklable class (i.e. that implements ` __reduce__ ` to be picklable and ` __call__ ` to yield the samples)

In the future we might allow non-picklable generators (e.g. generators based on iterators) and throw a warning to say that the cache is not used in this case.

---

<div class="post-metadata">

**Author:** ![cwallenwein](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cwallenwein/32/21752_2.png) [@cwallenwein](https://discuss.huggingface.co/u/cwallenwein)\
**Post date:** [November 18, 2023, 5:55pm UTC](https://discuss.huggingface.co/t/iterabledataset-from-generator-with-iterator/62049/3 "2023-11-18T17:55:23Z")

</div>

Thank you 🙂 I ended up using this hack [Support custom fingerprinting with `Dataset.from_generator` · Issue #6194 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/6194#issuecomment-1708080653)