# Making multiple samples from single samples using HuggingFace Datasets

**URL:** <https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819>\
**Category:** 🤗Datasets\
**Created:** [June 14, 2021, 7:49pm UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819 "2021-06-14T19:49:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![vasudevgupta](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vasudevgupta/32/1251_2.png) [@vasudevgupta](https://discuss.huggingface.co/u/vasudevgupta)\
**Post date:** [June 14, 2021, 7:49pm UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819/1 "2021-06-14T19:49:34Z")

</div>

My use case involved building multiple samples from a single sample. Is there any way I can do that with `Datasets.map()`.

Just a view of what I need to do:

```auto
# this is how my dataset looks like
dataset = [(1, 2, 3), (5, 7, 8)] 

# this is how my output dataset should look like
output = [1, 2, 3, 5, 7, 8]

** these list & tuple are just for making my point more clear, in actual its a Huggingface dataset 😉.

```

Can I do something like that with Datasets.map() or any other way to do this??

cc @lhoestq

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [June 15, 2021, 8:32am UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819/2 "2021-06-15T08:32:39Z")

</div>

Hi ! Yes you can definitely do that using a batched `map` call.  
Basically you can return more examples than the number of examples in per batch:

```python
from datasets import Dataset

d = Dataset.from_dict({"foo": [(1, 2, 3), (5, 7, 8)]})
d = d.map(lambda x: {"foo": [i for row in x["foo"] for i in row]}, batched=True)

print(d["foo"])
# [1, 2, 3, 5, 7, 8]

```

---

<div class="post-metadata">

**Author:** ![vasudevgupta](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vasudevgupta/32/1251_2.png) [@vasudevgupta](https://discuss.huggingface.co/u/vasudevgupta)\
**Post date:** [June 15, 2021, 8:51am UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819/3 "2021-06-15T08:51:36Z")

</div>

Woww!! Thankyou so much.

---

<div class="post-metadata">

**Author:** ![oliversn](https://avatars.discourse-cdn.com/v4/letter/o/c77e96/32.png) [@oliversn](https://discuss.huggingface.co/u/oliversn)\
**Post date:** [July 15, 2022, 9:50pm UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819/4 "2022-07-15T21:50:12Z")

</div>

Hi @lhoestq ,

Thanks for the insight. I have been rethinking this solution, wondering is there any way to do the same job without using “batched=True”?

Thanks!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 18, 2022, 1:11pm UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819/5 "2022-07-18T13:11:04Z")

</div>

Hi @oliversn , to apply a function that makes several examples out of one, you still have to use `batched=True` because the output is a batch. Note that you can set the batch\_size to 1 if you only want to give your function a batch of one example at a time.

---

<div class="post-metadata">

**Author:** ![oliversn](https://avatars.discourse-cdn.com/v4/letter/o/c77e96/32.png) [@oliversn](https://discuss.huggingface.co/u/oliversn)\
**Post date:** [July 18, 2022, 8:23pm UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819/6 "2022-07-18T20:23:06Z")

</div>

Much appreciated @lhoestq! That makes totally sense. 🙂

---

<div class="post-metadata">

**Author:** ![austindavis](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@austindavis](https://discuss.huggingface.co/u/austindavis)\
**Post date:** [March 3, 2024, 4:53pm UTC](https://discuss.huggingface.co/t/making-multiple-samples-from-single-samples-using-huggingface-datasets/6819/7 "2024-03-03T16:53:57Z")

</div>

One important note: If your’re doing this, you can’t map to a new column name. You must overwrite the original column name or use `remove_columns` as in:

```auto
from datasets import Dataset

d = Dataset.from_dict({"foo": [(1, 2, 3), (5, 7, 8)]})
d = d.map(lambda x: {"bar": [i for row in x["foo"] for i in row]}, batched=True, remove_columns="foo")

print(d)
# [1, 2, 3, 5, 7, 8]

```