# Error when downloading own dataset with git lfs files

**URL:** <https://discuss.huggingface.co/t/error-when-downloading-own-dataset-with-git-lfs-files/19245>\
**Category:** 🤗Datasets\
**Created:** [June 16, 2022, 2:11pm UTC](https://discuss.huggingface.co/t/error-when-downloading-own-dataset-with-git-lfs-files/19245 "2022-06-16T14:11:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkon](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mkon](https://discuss.huggingface.co/u/mkon)\
**Post date:** [June 16, 2022, 2:11pm UTC](https://discuss.huggingface.co/t/error-when-downloading-own-dataset-with-git-lfs-files/19245/1 "2022-06-16T14:11:41Z")

</div>

I’ve made a couple of data loaders for different stance detection datasets. For this I’ve stored the .csv files used with git lfs because one of them surpassed the 10MB maximum. [which can be seen here](https://huggingface.co/datasets/strombergnlp/x-stance). When trying to download the dataset in my own code using `load_dataset("strombergnlp/x-stance")` I get this error and I’m unsure why. It’s worth noting that the one dataset I’ve imported to huggingface which doesn’t use git lfs works perfectly fine when downloaded.

> Traceback (most recent call last):  
> File “”, line 1, in   
> File “C:\Users\Arche\miniconda3\lib\site-packages\datasets\load.py”, line 1679, in load\_dataset  
> builder\_instance.download\_and\_prepare(  
> File “C:\Users\Arche\miniconda3\lib\site-packages\datasets\builder.py”, line 704, in download\_and\_prepare  
> self.\_download\_and\_prepare(  
> File “C:\Users\Arche\miniconda3\lib\site-packages\datasets\builder.py”, line 775, in \_download\_and\_prepare  
> verify\_checksums(  
> File “C:\Users\Arche\miniconda3\lib\site-packages\datasets\utils\info\_utils.py”, line 33, in verify\_checksums  
> raise ExpectedMoreDownloadedFiles(str(set(expected\_checksums) - set(recorded\_checksums)))  
> datasets.utils.info\_utils.ExpectedMoreDownloadedFiles: {‘x-stance-valid-de.csv’, ‘x-stance-test-de.csv’, ‘x-stance-train-de.csv’}

Any pointers as to what could be wrong are greatly appreciated 😊

---

<div class="post-metadata">

**Author:** ![polinaeterna](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/polinaeterna/32/19055_2.png) [@polinaeterna](https://discuss.huggingface.co/u/polinaeterna)\
**Post date:** [June 21, 2022, 10:41am UTC](https://discuss.huggingface.co/t/error-when-downloading-own-dataset-with-git-lfs-files/19245/2 "2022-06-21T10:41:38Z")

</div>

Hi! Can you try using `load_dataset` with `ignore_verifications=True` option?

---

<div class="post-metadata">

**Author:** ![mkon](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mkon](https://discuss.huggingface.co/u/mkon)\
**Post date:** [June 21, 2022, 12:41pm UTC](https://discuss.huggingface.co/t/error-when-downloading-own-dataset-with-git-lfs-files/19245/3 "2022-06-21T12:41:40Z")

</div>

Hi! Thanks for the response. With `ignore_verifications=True` I get this new error:

> Traceback (most recent call last):  
> File “stance\_detection.py”, line 167, in run  
> dataset\_de = load\_dataset(‘strombergnlp/x-stance’, ‘de’, ignore\_verifications=True)  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/load.py”, line 1736, in load\_dataset  
> use\_auth\_token=use\_auth\_token,  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/builder.py”, line 614, in download\_and\_prepare  
> dl\_manager=dl\_manager, verify\_infos=verify\_infos, \*\*download\_and\_prepare\_kwargs  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/builder.py”, line 702, in \_download\_and\_prepare  
> self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/builder.py”, line 1167, in \_prepare\_split  
> writer.write\_table(table)  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/arrow\_writer.py”, line 523, in write\_table  
> pa\_table = table\_cast(pa\_table, self.\_schema)  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1895, in table\_cast  
> return cast\_table\_to\_schema(table, schema)  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1878, in cast\_table\_to\_schema  
> arrays = [cast\_array\_to\_feature(table[name], feature) for name, feature in features.items()]  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1878, in   
> arrays = [cast\_array\_to\_feature(table[name], feature) for name, feature in features.items()]  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1673, in wrapper  
> return pa.chunked\_array([func(chunk, \*args, \*\*kwargs) for chunk in array.chunks])  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1673, in   
> return pa.chunked\_array([func(chunk, \*args, \*\*kwargs) for chunk in array.chunks])  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1843, in cast\_array\_to\_feature  
> return array\_cast(array, feature(), allow\_number\_to\_str=allow\_number\_to\_str)  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1675, in wrapper  
> return func(array, \*args, \*\*kwargs)  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/datasets/table.py”, line 1752, in array\_cast  
> return array.cast(pa\_type)  
> File “pyarrow/array.pxi”, line 915, in pyarrow.lib.Array.cast  
> File “/home/mkon/miniconda3/envs/stance/lib/python3.7/site-packages/pyarrow/compute.py”, line 376, in cast  
> return call\_function(“cast”, [arr], options)  
> File “pyarrow/\_compute.pyx”, line 542, in pyarrow.\_compute.call\_function  
> File “pyarrow/\_compute.pyx”, line 341, in pyarrow.\_compute.Function.call  
> File “pyarrow/error.pxi”, line 144, in pyarrow.lib.pyarrow\_internal\_check\_status  
> File “pyarrow/error.pxi”, line 100, in pyarrow.lib.check\_status  
> pyarrow.lib.ArrowInvalid: Failed to parse string: ‘AGAINST’ as a scalar of type int64

Here, `AGAINST` is one of the two class labels for the dataloader, the other being `FAVOR`. The dataloader passes the test suite so unsure why it crashes here.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 22, 2022, 10:32am UTC](https://discuss.huggingface.co/t/error-when-downloading-own-dataset-with-git-lfs-files/19245/4 "2022-06-22T10:32:33Z")

</div>

Hi! The script name should match the repo name, so can you please rename the script to `x-stance.py` and regenerate the `dataset_infos.json` file? The loading machinery currently loads it as a CSV module due to the name mismatch:

```python
>>> from datasets import load_dataset_builder
>>> load_dataset_builder("strombergnlp/x-stance")
Using custom data configuration de-468d65e3f7ce1e40
<datasets.packaged_modules.csv.csv.Csv object at 0x000001E3A74F1248>

```

(It first checks if `x-stance.py` is present and, if not, infers the most common data file type, which is CSV)

---

<div class="post-metadata">

**Author:** ![mkon](https://avatars.discourse-cdn.com/v4/letter/m/9d8465/32.png) [@mkon](https://discuss.huggingface.co/u/mkon)\
**Post date:** [June 22, 2022, 9:32pm UTC](https://discuss.huggingface.co/t/error-when-downloading-own-dataset-with-git-lfs-files/19245/5 "2022-06-22T21:32:45Z")

</div>

Thank you! That seemed to fix the error. I’ve made two other dataloaders with the exact same problem, who all use git lfs, so I thought that was where the problem was. But they’re all also incorrectly named with an underscore and not a dash, so I’ll go ahead and fix those as well.

Thank you for all the help! 😊