# Error, dataset could not be generated

**URL:** <https://discuss.huggingface.co/t/error-dataset-could-not-be-generated/42939>\
**Category:** 🤗Datasets\
**Created:** [June 12, 2023, 1:57pm UTC](https://discuss.huggingface.co/t/error-dataset-could-not-be-generated/42939 "2023-06-12T13:57:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![murraybent](https://avatars.discourse-cdn.com/v4/letter/m/e5b9ba/32.png) [@murraybent](https://discuss.huggingface.co/u/murraybent)\
**Post date:** [June 12, 2023, 1:57pm UTC](https://discuss.huggingface.co/t/error-dataset-could-not-be-generated/42939/1 "2023-06-12T13:57:49Z")

</div>

In file “/hface1/env/lib/python3.11/site-packages/datasets/builder.py”, line 1746, in \_prepare\_split  
for job\_id, done, content in self.\_prepare\_split\_single(  
File “/hface1/env/lib/python3.11/site-packages/datasets/builder.py”, line 1891, in \_prepare\_split\_single  
raise DatasetGenerationError(“An error occurred while generating the dataset”) from e  
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset  
What to do next , the data is sownloaded but there’s a bug in builder.py to generate training set?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 12, 2023, 7:19pm UTC](https://discuss.huggingface.co/t/error-dataset-could-not-be-generated/42939/2 "2023-06-12T19:19:29Z")

</div>

Hi! Can you share the entire error stack trace? Also, feel free to share a link to the problematic dataset if it’s public.

---

<div class="post-metadata">

**Author:** ![murraybent](https://avatars.discourse-cdn.com/v4/letter/m/e5b9ba/32.png) [@murraybent](https://discuss.huggingface.co/u/murraybent)\
**Post date:** [June 17, 2023, 3:36pm UTC](https://discuss.huggingface.co/t/error-dataset-could-not-be-generated/42939/3 "2023-06-17T15:36:29Z")

</div>

Tried again, same problem.  
Tried different datasets like the-stack-dedup and the-stack .  
After spending past 5 days downloading and extracting I get a bug:

Extracting data files: 100%|███████████████| 1/1 [03:32\<00:00, 212.39s/it]  
Traceback (most recent call last):  
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 1858, in \_prepare\_split\_single  
for \_, table in generator:  
File “/media/env/lib/python3.11/site-packages/datasets/packaged\_modules/parquet/parquet.py”, line 67, in \_generate\_tables  
parquet\_file = pq.ParquetFile(f)  
^^^^^^^^^^^^^^^^^  
File “/media/env/lib/python3.11/site-packages/pyarrow/parquet/core.py”, line 334, in **init**  
self.reader.open(  
File “pyarrow/\_parquet.pyx”, line 1220, in pyarrow.\_parquet.ParquetReader.open  
File “pyarrow/error.pxi”, line 100, in pyarrow.lib.check\_status  
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):  
File “”, line 1, in   
File “/media/env/lib/python3.11/site-packages/datasets/load.py”, line 1797, in load\_dataset  
builder\_instance.download\_and\_prepare(  
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 890, in download\_and\_prepare  
self.\_download\_and\_prepare(  
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 985, in \_download\_and\_prepare  
self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)  
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 1746, in \_prepare\_split  
for job\_id, done, content in self.\_prepare\_split\_single(  
File “/media/env/lib/python3.11/site-packages/datasets/builder.py”, line 1891, in \_prepare\_split\_single  
raise DatasetGenerationError(“An error occurred while generating the dataset”) from e  
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset