# Json dump format for load\_dataset

**URL:** <https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974>\
**Category:** 🤗Datasets\
**Created:** [September 14, 2021, 10:10am UTC](https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974 "2021-09-14T10:10:31Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [September 14, 2021, 10:10am UTC](https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974/1 "2021-09-14T10:10:31Z")

</div>

Hey guys,

How do I properly encode/format json file dump (or use any other approach for creating JSON files) so that the created JSON file is easily digested by load\_dataset JSON variant as described in the [docs](https://huggingface.co/docs/datasets/loading.html#json)?

TIA,  
Vladimir

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 14, 2021, 2:37pm UTC](https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974/2 "2021-09-14T14:37:04Z")

</div>

Hi ! You can simply use .to\_json() - see documentation [here](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.to_json)

Here is an example using SQuAD:

```python
from datasets import load_dataset                   

squad = load_dataset("squad", split="train")        
squad.to_json("squad.json")            

data_files = {"train": "squad.json"}
re_squad = load_dataset("json", data_files=data_files, split="train")

```

This creates a JSON Lines file, then it reloads it using the JSON dataset loader 🙂

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [September 14, 2021, 2:49pm UTC](https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974/3 "2021-09-14T14:49:33Z")

</div>

Yes, of course, @lhoestq , but I wondered about a JSON file I made myself in some processing, not an already prepared JSON file. If I attempt to load the newly created JSON file (created using JSON dump), it needs to be one record per line, cannot have square brackets at the beginning/end, etc. How can I format any JSON holding data records so it could be consumed by datasets easily?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 14, 2021, 3:09pm UTC](https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974/4 "2021-09-14T15:09:17Z")

</div>

You can format your data as JSON Lines, so as you said:

- one record per line (they can be created via `json.dumps` from the standard lib for example)
- no square brackets at the beginning/end

Moreover:

- use “\n” for end of lines in string data - so that each record is on one single line
- nested fields are supported

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [September 14, 2021, 3:51pm UTC](https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974/5 "2021-09-14T15:51:15Z")

</div>

Yeah, this works. I manually wrote out the JSON records one line at a time with “\n” at the end of the line using for loop. I thought there might be some sort of HF datasets encoder that could do this for me.

---

<div class="post-metadata">

**Author:** ![Rasputin312](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@Rasputin312](https://discuss.huggingface.co/u/Rasputin312)\
**Post date:** [September 5, 2024, 6:27pm UTC](https://discuss.huggingface.co/t/json-dump-format-for-load-dataset/9974/6 "2024-09-05T18:27:20Z")

</div>

Can you show an example of this? I have the exact same issue – trying to create my own dataset using .json but it doesn’t like anything I do. I know how to create a json in python. But the question is what exactly is the format of that json? I know to create Features and feed that to load\_dataset.