# Datasets mapper hanging issue

**URL:** <https://discuss.huggingface.co/t/datasets-mapper-hanging-issue/32995>\
**Category:** 🤗Datasets\
**Created:** [March 4, 2023, 1:37am UTC](https://discuss.huggingface.co/t/datasets-mapper-hanging-issue/32995 "2023-03-04T01:37:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![plamb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/plamb/32/4741_2.png) [@plamb](https://discuss.huggingface.co/u/plamb)\
**Post date:** [March 4, 2023, 1:37am UTC](https://discuss.huggingface.co/t/datasets-mapper-hanging-issue/32995/1 "2023-03-04T01:37:56Z")

</div>

Lots to detail here because what is happening is not obvious. I’ll try to simplify it:

- Using Datasets.map to encode a dataset that has Sequence, Array3d and Array2d features. Note that the Array2D features have a dynamic first dimension.

- I can successfully encode and save to disk locally on a fraction of the data, I can also encode and save to S3 running on sagemaker on a 1/3rd of the data (~3000 rows).

- However, when I run on the full dataset (~9,000 rows) on sagemaker with 16 processors and a batch size of 1000, the mapper gets to the end of mapping across all processors and hangs indefinitely.

- Oddly when I manually press “stop” on the sagemaker processing instance, whatever “stop” does jostles the CPU(s) out of their hanging state and the mapper proceeds to the next step (where it appears to be flattening the indices) before the instance dies

- In an attempt to debug i ran the same dataset and same code but with `num_proc=1`. This time mapping fully completed, but I got an error on the `save_to_disk` call:

```auto
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1348, in save_to_disk
    dataset = self.flatten_indices(num_proc=num_proc) if self._indices is not None else self
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 528, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/fingerprint.py", line 511, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3541, in flatten_indices
    return self.map(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 563, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 528, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2953, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3346, in _map_single
    writer.write_batch(batch)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py", line 555, in write_batch
    self.write_table(pa_table, writer_batch_size)
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py", line 567, in write_table
    pa_table = pa_table.combine_chunks()
  File "pyarrow/table.pxi", line 3241, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: concatenation of extension<arrow.py_extension_type<Array2DExtensionType>>

```

Which is odd because I’ve used Array2D successfully multiple times in the past.

I tried this with num\_proc=1 with a batch\_size=1000 and batch\_size=100 and got the same error.

---

<div class="post-metadata">

**Author:** ![plamb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/plamb/32/4741_2.png) [@plamb](https://discuss.huggingface.co/u/plamb)\
**Post date:** [March 5, 2023, 9:08pm UTC](https://discuss.huggingface.co/t/datasets-mapper-hanging-issue/32995/2 "2023-03-05T21:08:38Z")

</div>

> [@plamb](#):
>
> `ArrowNotImplementedError: concatenation of extension<arrow.py_extension_type<Array2DExtensionType>>`

This problem was caused by datasets==2.10.0. Downgrading to 2.9.0 fixed it

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [March 8, 2023, 12:30pm UTC](https://discuss.huggingface.co/t/datasets-mapper-hanging-issue/32995/3 "2023-03-08T12:30:40Z")

</div>

Hi ! This error comes from Arrow which doesn’t implement concatenation for extension types yet: [[C++] Can't concatenate extension arrays · Issue #31868 · apache/arrow · GitHub](https://github.com/apache/arrow/issues/31868)

It was working in `datasets` 2.9 because we missed the `.combine_chunks()` call which is necessary to not end up with record batches of 1 row which cause performance issues.

Unfortunately I don’t have any workaround for now except using an old version of `datasets` - though we should definitely see with the Arrow community what we can do about this