# Error Handling in IterableDataset?

**URL:** <https://discuss.huggingface.co/t/error-handling-in-iterabledataset/72827>\
**Category:** 🤗Datasets\
**Created:** [February 12, 2024, 12:53pm UTC](https://discuss.huggingface.co/t/error-handling-in-iterabledataset/72827 "2024-02-12T12:53:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![FelixLabelle](https://avatars.discourse-cdn.com/v4/letter/f/49beb7/32.png) [@FelixLabelle](https://discuss.huggingface.co/u/FelixLabelle)\
**Post date:** [February 12, 2024, 12:53pm UTC](https://discuss.huggingface.co/t/error-handling-in-iterabledataset/72827/1 "2024-02-12T12:53:30Z")

</div>

Is there a built in way of handling errors when streaming Datasets? I’ve been trying to stream the [RedPajama-1T dataset](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) and have gotten errors across most subsets on multiple occasions (see screenshots for Github and C4 below):

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/9/a/9ac0157a7b47760473b729accf074bcc86910986.png)

I don’t need all the data, most of it is fine.

If there isn’t a built in way, it’s all good. I’ll look at writing a class that inherits the IterableDataset and handles the issue (unless there is a better way, I’m all ears).

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 12, 2024, 2:20pm UTC](https://discuss.huggingface.co/t/error-handling-in-iterabledataset/72827/2 "2024-02-12T14:20:43Z")

</div>

RedPajama-1T uses a custom dataset loading script to download files outside of HF which can lead to unexpected failures. Maybe you can ask authors about why they’re hosting the files on HF directly by opening a discussion: [togethercomputer/RedPajama-Data-1T · Discussions](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T/discussions)

There are retry mechanisms in `datasets` / `huggingface_hub` already when streaming files from HF

---

<div class="post-metadata">

**Author:** ![FelixLabelle](https://avatars.discourse-cdn.com/v4/letter/f/49beb7/32.png) [@FelixLabelle](https://discuss.huggingface.co/u/FelixLabelle)\
**Post date:** [February 12, 2024, 4:07pm UTC](https://discuss.huggingface.co/t/error-handling-in-iterabledataset/72827/3 "2024-02-12T16:07:09Z")

</div>

Merci Quentin, I’ll do that.

For anyone looking for an immediate fix, I added error handling to their custom DatasetBuilder class and it looks like a viable workaround (still testing).

I used GitLFS to download their dataset/loader in the directory where my code was. Then modified the RedPajama1T class, specifically the " \_generate\_examples" functions. It should look like this:

```auto
    def _generate_examples(self, files):                                                                                                                                                        
        """This function returns the examples in the raw (text) form."""                                                                                                                        
        key = 0                                                                                                                                                                                 
        errors = []                                                                                                                                                                             
        for subset in files:                                                                                                                                                                    
            if subset == "common_crawl":                                                                                                                                                        
                import zstandard as zstd                                                                                                                                                        
                try:                                                                                                                                                                            
                    for path in files[subset]:                                                                                                                                                  
                        with zstd.open(open(path, "rb"), "rt", encoding="utf-8") as f:                                                                                                          
                            for i, row in enumerate(f):                                                                                                                                         
                                try:                                                                                                                                                            
                                    data = json.loads(row)                                                                                                                                      
                                    text = data["text"]                                                                                                                                         
                                    del data["text"]  
                                    yield key, {                                                                                                                                         
                                        "text": text,                                           
                                        "meta": json.dumps(data),                                                                                                                               
                                        "red_pajama_subset": subset,                                                                                                                            
                                    }                                                           
                                    key += 1                                                    
                                except Exception as e:                                          
                                    print(f'Subset: {subset}')                                                                                                                                  
                                    print(f'Path: {path}')                                                                                                                                      
                                    print(f'Row: {row}')                                                                                                                                        
                                    print(e)                                                    
                except Exception as e:                                                          
                    errors.append(e)                                                            
            else:                               
                for path in files[subset]:                                                      
                    try:                        
                        with open(path, encoding="utf-8") as f:                                                                                                                                 
                            for i, row in enumerate(f):                                                                                                                                         
                                try:                                                            
                                    data = json.loads(row)                                                                                                                                      
                                    if "meta" not in data:                                                                                                                                      
                                        text = data["text"]                                                                                                                                     
                                        del data["text"]                                                                                                                                        
                                        yield key, {                                            
                                            "text": text,                                                                                                                                       
                                            "meta": json.dumps(data),                                                                                                                           
                                            "red_pajama_subset": subset,                                                                                                                        
                                        }                                                       
                                    else:                                                       
                                        yield key, {                                            
                                            "text": data["text"],                                                                                                                               
                                            "meta": data["meta"],                                                                                                                               
                                            "red_pajama_subset": subset,                                                                                                                        
                                        }                                                       
                                    key += 1                                                    
                                except Exception as e:                                          
                                    print(f'Subset: {subset}')                                                                                                                                  
                                    print(f'Path: {path}')                                                                                                                                      
                                    print(f'Row: {row}')                                                                                                                                        
                                    print(e)                                                    
                    except Exception as e:                                                      
                        errors.append(e) 

```

You can then use “load\_dataset” to read the modified dataset. The name of dataset’s directory is RedPajama-Data-1T and was in the same directory as my code (you’ll need to change the path passed in to load\_datasets otherwise)

```auto
import datasets

rpj_arxiv_dataset= datasets.load_dataset('./RedPajama-Data-1T','arxiv', streaming=True)

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [February 13, 2024, 4:07am UTC](https://discuss.huggingface.co/t/error-handling-in-iterabledataset/72827/4 "2024-02-13T04:07:28Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.