# Creating HuggingFace Dataset from PyArrow table is slow

**URL:** <https://discuss.huggingface.co/t/creating-huggingface-dataset-from-pyarrow-table-is-slow/130510>\
**Category:** 🤗Datasets\
**Created:** [December 11, 2024, 6:12am UTC](https://discuss.huggingface.co/t/creating-huggingface-dataset-from-pyarrow-table-is-slow/130510 "2024-12-11T06:12:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![m4nhb4nhq](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@m4nhb4nhq](https://discuss.huggingface.co/u/m4nhb4nhq)\
**Post date:** [December 11, 2024, 6:12am UTC](https://discuss.huggingface.co/t/creating-huggingface-dataset-from-pyarrow-table-is-slow/130510/1 "2024-12-11T06:12:43Z")

</div>

I tried to convert a pyarrow table to a hugging face dataset with the following code

```python
arrow_tbl = deltalake.DeltaTable(
    table_uri=table_uri
).to_pyarrow_table()

print("Converting to HF ...")
start_time = time.time()
hf_ds = HFDataset(arrow_tbl)

print(f"Took {time.time() - start_time} seconds")

```

The Delta Lake table contains 1 image binary type column and 1 string type column.  
The binary column’s size on average is round 500kb and the string column’s size is much more smaller in comparison so I think it doesn’t matter.  
The number of rows is 5000.

It took around 14 seconds to convert pyarrow table to huggingface dataset which doesn’t make sense to me because I think we can zero-copy the underlying pyarrow data. Did I miss anything?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [December 11, 2024, 3:43pm UTC](https://discuss.huggingface.co/t/creating-huggingface-dataset-from-pyarrow-table-is-slow/130510/2 "2024-12-11T15:43:02Z")

</div>

Hi ! The arrow table is hashed to create a unique Dataset fingerprint (typically a hash string) used for caching 🙂

But you can provide the fingerprint yourself to make it faster:

```python
ds = Dataset(arrow_tbl, fingerprint=fingerprint)

```