# Preprocessing & augmentation during training

**URL:** <https://discuss.huggingface.co/t/preprocessing-augmentation-during-training/14550>\
**Category:** Beginners\
**Created:** [February 9, 2022, 1:32pm UTC](https://discuss.huggingface.co/t/preprocessing-augmentation-during-training/14550 "2022-02-09T13:32:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sfalk](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sfalk](https://discuss.huggingface.co/u/sfalk)\
**Post date:** [February 9, 2022, 1:32pm UTC](https://discuss.huggingface.co/t/preprocessing-augmentation-during-training/14550/1 "2022-02-09T13:32:14Z")

</div>

Hi!

I am currently fine-tuning a Wav2Vec2 model and I would like to know how I could modify training samples.

In Tensorflow it’s possible to do this by calling `dataset.map()` but doing this with a `datasets.arrow_dataset.Dataset` will write a cache file. Besides that, it’s static and unnecessary.

So, this here is not an option:

```auto
train_dataset = train_dataset.map(preprocess_and_augment)

```

There is the `data_collator` and one could do it here. However, this is very likely not very efficient and I would have to take care of parallelization myself.

I’m looking for a similar way to process input examples like [`Dataset.map()`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map).

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [February 9, 2022, 4:14pm UTC](https://discuss.huggingface.co/t/preprocessing-augmentation-during-training/14550/2 "2022-02-09T16:14:04Z")

</div>

Hi! I think [`Dataset.set_transform`](https://huggingface.co/docs/datasets/process.html?highlight=set_transform#format-transform) is what you are looking for.

---

<div class="post-metadata">

**Author:** ![sfalk](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sfalk](https://discuss.huggingface.co/u/sfalk)\
**Post date:** [February 10, 2022, 8:08am UTC](https://discuss.huggingface.co/t/preprocessing-augmentation-during-training/14550/3 "2022-02-10T08:08:37Z")

</div>

Yes, this would be something I’d like to do. The problem is that I am having a `datasets.arrow_dataset.Dataset`

```auto
$ print(train_dataset)
<class 'dataset.arrow_dataset.Dataset'>

```

which is what I get after calling `builder_instance.as_dataset(..)` following this description: [Writing a dataset loading script — datasets 1.11.0 documentation](https://huggingface.co/docs/datasets/add_dataset.html)

I have tried to call:

```auto
datasets.Dataset(train_dataset)`

```

giving me

```auto
TypeError: Expected a pyarrow.Table or a `datasets.table.Table` object

```

and

```auto
datasets.table.Table(train_dataset)

```

throws

```auto
AttributeError: 'Dataset' has no attribute 'schema'

```

I guess my question is: How can I get a `datasets.Dataset` from a `dataset.arrow_dataset.Dataset`?

---

<div class="post-metadata">

**Author:** ![sfalk](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sfalk](https://discuss.huggingface.co/u/sfalk)\
**Post date:** [February 10, 2022, 8:29am UTC](https://discuss.huggingface.co/t/preprocessing-augmentation-during-training/14550/4 "2022-02-10T08:29:22Z")

</div>

I figured that I can do this:

```auto
train_dataset = datasets.Dataset(train_dataset.data)
train_dataset.set_transform(example_adapter.preprocess)

```

It’s a bit weird and I am not sure how efficient this is e.g. if I’d have to do more expensive computations though.