# \[Bug?\] Datasets map and concatenation after sharding OOM

**URL:** <https://discuss.huggingface.co/t/bug-datasets-map-and-concatenation-after-sharding-oom/105182>\
**Category:** 🤗Datasets\
**Created:** [September 4, 2024, 12:58am UTC](https://discuss.huggingface.co/t/bug-datasets-map-and-concatenation-after-sharding-oom/105182 "2024-09-04T00:58:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [September 4, 2024, 12:58am UTC](https://discuss.huggingface.co/t/bug-datasets-map-and-concatenation-after-sharding-oom/105182/1 "2024-09-04T00:58:33Z")

</div>

When using

```auto
dataset.map(myfunc, num_proc=16,  
    keep_in_memory=False, 
    cache_file_name='parts.arrow', 
    batch_size=16, writer_batch_size=16
)

```

Due to the size of my dataset, it results in:

```auto
/site-packages/datasets/table.py:1421:
  table = cls._concat_blocks(blocks, axis=0)
Killed

```

Looking carefully at the code in `.map`, I see that the shards are being created and the `.arrow` files exists after the progress bar goes to 100%, then at it goes OOM at [datasets/src/datasets/arrow\_dataset.py at 2.21.0 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/blob/2.21.0/src/datasets/arrow_dataset.py#L3276)

```auto
            logger.info(f"Concatenating {num_proc} shards")
            result = _concatenate_map_style_datasets(transformed_shards)

```

Not sure if it’s a bug that writing to individual arrow files work but the combination of the shard fails.

Any way to resolve this and avoid the OOM in the `_concat_blocks` function?

---

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [September 4, 2024, 1:01am UTC](https://discuss.huggingface.co/t/bug-datasets-map-and-concatenation-after-sharding-oom/105182/2 "2024-09-04T01:01:43Z")

</div>

Also asked on [concatenation - How to resolve OOM when .map concatenate the sharded parts? - Stack Overflow](https://stackoverflow.com/questions/78946478/how-to-resolve-oom-when-map-concatenate-the-sharded-parts)