# Handling decoding errors such as UnidentifiedImageError

**URL:** <https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301>\
**Category:** 🤗Datasets\
**Created:** [December 2, 2023, 12:04am UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301 "2023-12-02T00:04:31Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![keturn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/keturn/32/18215_2.png) [@keturn](https://discuss.huggingface.co/u/keturn)\
**Post date:** [December 2, 2023, 12:04am UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/1 "2023-12-02T00:04:31Z")

</div>

I’m using the datasets library to load [`gigant/oldbookillustrations`](https://huggingface.co/datasets/gigant/oldbookillustrations). It’s stored in parquet files, with a couple Image fields plus a handful of other short text fields with assorted metadata.

It loads okay, and many records seem to work just fine, but when I try to `filter` or `map` over the whole thing, it throws an `UnidentifiedImageError` from within the decoder:

> **traceback**
>
> ```auto
> File datasets/features/image.py:184, in Image.decode_example(self, value, token_per_repo_id)
> 182 image = PIL.Image.open(bytes_)
> 183 else:
> --> 184 image = PIL.Image.open(BytesIO(bytes_))
> 185 image.load() # to avoid "Too many open files" errors
> 186 return image
> 
> File PIL/Image.py:3280, in open(fp, mode, formats)
> 3278 warnings.warn(message)
> 3279 msg = "cannot identify image file %r" % (filename if filename else fp)
> -> 3280 raise UnidentifiedImageError(msg)
> 
> ```

> cannot identify image file \<\_io.BytesIO object at 0x7fa50f2822a0\>

How can I fix this or work around it?

Is there a way to install some kind of error handler on the decoder?

How can I identify _which_ records are failing to decode, so that I can troubleshoot further or repair them?

Can I tell my operation to skip the failing record and move on?

> **env versions**
>
> - `datasets` version: 2.15.0
> - Platform: Linux-5.15.0-89-generic-x86\_64-with-glibc2.35
> - Python version: 3.11.5
> - `huggingface_hub` version: 0.19.4
> - PyArrow version: 13.0.0
> - Pandas version: 2.1.0
> - `fsspec` version: 2023.6.0

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [December 4, 2023, 10:19am UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/2 "2023-12-04T10:19:00Z")

</div>

What version of pillow are you using ? Can you try to update pillow ?

---

<div class="post-metadata">

**Author:** ![keturn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/keturn/32/18215_2.png) [@keturn](https://discuss.huggingface.co/u/keturn)\
**Post date:** [December 4, 2023, 3:55pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/3 "2023-12-04T15:55:05Z")

</div>

Pillow version 10.1.0, the current version.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [December 4, 2023, 5:40pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/4 "2023-12-04T17:40:57Z")

</div>

You can disable decoding and apply your own transform to decode the images:

```python
def decode_images(batch):
    batch["rawscan"] = [decode_image(raw_data) for raw_data in batch["rawscan"]]
    return batch
ds = ds.cast_column("rawscan", Image(decode=False)
ds = ds.with_transform(decode_images)
ds[0]["rawscan"] # transformed using decode_images

```

Then you can also iterate on your dataset to print the list of invalid images (e.g. using a try/except in decode\_images)

---

<div class="post-metadata">

**Author:** ![keturn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/keturn/32/18215_2.png) [@keturn](https://discuss.huggingface.co/u/keturn)\
**Post date:** [December 13, 2023, 9:47pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/5 "2023-12-13T21:47:14Z")

</div>

Thank you, re-casting the type with `decode=False` worked to get access to that data without having to re-specify all of that type’s internal fields. I managed to track down the problem from there. 👍

(For a few records, the HTML of the source’s 404 page had been saved to that Image field.)

---

<div class="post-metadata">

**Author:** ![finnAndTheSharks](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/finnandthesharks/32/40301_2.png) [@finnAndTheSharks](https://discuss.huggingface.co/u/finnAndTheSharks)\
**Post date:** [February 3, 2025, 12:03pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/6 "2025-02-03T12:03:33Z")

</div>

Which function `decode_image()` are you using in this example? I could not find a suitable import from the `datasets` library.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 3, 2025, 2:52pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/7 "2025-02-03T14:52:28Z")

</div>

> Which function `decode_image()` are you using in this example? I could not find a suitable import from the `datasets` library.

this would be your own custom decode function

---

<div class="post-metadata">

**Author:** ![finnAndTheSharks](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/finnandthesharks/32/40301_2.png) [@finnAndTheSharks](https://discuss.huggingface.co/u/finnAndTheSharks)\
**Post date:** [February 5, 2025, 1:11pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/8 "2025-02-05T13:11:20Z")

</div>

Thank you @lhoestq!

I was trying to pinpoint which rows exactly fail, but am a bit lost in the process. When iterating over the `Dataset`, I can access each row just fine. But when iterating over the `IterableDataset`, a warning about the EXIF data is thrown.

`/pfss/mlde/workspaces/mlde_wsp_KIServiceCenter/finngu/envs/dataset_download/lib/python3.10/site-packages/PIL/TiffImagePlugin.py:949: UserWarning: Corrupt EXIF data. Expecting to read 2 bytes but only got 0.`

Do you have any idea, how I could find all rows with faulty EXIF data efficiently?

```auto
def validate_download():
    dataset = load_dataset("ILSVRC/imagenet-1k")
    splits = ["train", "validation", "test"]

    logging.basicConfig(
        filename='/pfss/mlde/workspaces/mlde_wsp_KIServiceCenter/finngu/LlavaGuard/src/experiments/datasets/imagenet/validate_download.log', 
        level=logging.INFO,
        format='%(asctime)s %(levelname)s %(name)s %(message)s'
    )
    logger = logging.getLogger( __name__ )

    for split in splits:
        logger.info(f"Validating all images of split '{split}'...")

        iterable_ds = dataset[split].to_iterable_dataset()

        for idx, example in enumerate(iterable_ds):
            try:
                img = example['image']
            except Exception as e:
                logger.error(f"{idx}: {e}")
            
            if idx % 10000 == 0:
                logger.info(f"\t{idx} images done.")

        # ds = dataset[split]
        # ds_len = len(ds)

        # for idx in range(ds_len):
        # try:
        # _ = ds[idx]['image']
        # except Exception as e:
        # logger.error(f"{idx}: {e}")
            
        # if idx % 10000 == 0:
        # print(f"\t{idx} images done.")

```

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 5, 2025, 2:20pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/9 "2025-02-05T14:20:07Z")

</div>

Is the corrupted EXIF really an issue ? I think `PIL` is still able to load the image no ?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 5, 2025, 2:44pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/10 "2025-02-05T14:44:59Z")

</div>

I found a similar case.

> **[aharley/rvl\_cdip · UserWarning: Corrupt EXIF data. Expecting to read 2 bytes...](https://huggingface.co/datasets/aharley/rvl_cdip/discussions/2)**
>
> When I used the test set, I found that there was an error when I got the test set sample， code: dataset = load\_dataset(dataset\_name, split='test') print(dataset\[33669\]) output: lib/python3.7/site-p...

---

<div class="post-metadata">

**Author:** ![finnAndTheSharks](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/finnandthesharks/32/40301_2.png) [@finnAndTheSharks](https://discuss.huggingface.co/u/finnAndTheSharks)\
**Post date:** [February 5, 2025, 4:18pm UTC](https://discuss.huggingface.co/t/handling-decoding-errors-such-as-unidentifiedimageerror/64301/11 "2025-02-05T16:18:34Z")

</div>

Yeah, it is an issue. Iterating with the `IterableDataset` oftentimes fails alltogether and if it doesn’t fail, it becomes extreeeemely slow.

Edit: Ok, I finally found a solution and will attach it below if anyone else needs it in the future:

```auto
import logging
import warnings
from datasets import load_dataset

def validate_download():
    dataset = load_dataset("ILSVRC/imagenet-1k")
    splits = ["train", "validation", "test"]

    logging.basicConfig(
        filename='/src/experiments/datasets/imagenet/validate_download.log', 
        level=logging.INFO,
        format='%(asctime)s %(levelname)s %(name)s %(message)s'
    )
    logger = logging.getLogger( __name__ )

    # Treat all warnings as errors
    warnings.filterwarnings("error")

    for split in splits:
        logger.info(f"Validating all images of split '{split}'...")
        ds = dataset[split]

        for idx in range(len(ds)):
            try:
                ds[idx]['image'].load()
                ds[idx]['image'].close()
            except Exception as e:
                logger.error(f"{idx}: {e}")

    # No longer treat warnings as errors
    warnings.resetwarnings()

if __name__ == ' __main__':
    validate_download()

```