Access to MNIST examples is 7 times slower than torchvision: import torchvision import datasets mnist_hf = datasets.load_dataset("mnist", split="train") mnist_hf_inmem = datasets.load_dataset("mnist", split="train", keep_in_memory=True) mnist_tv = torchvision.datasets.MNIST("~/home", train=True, d…

Hi ! I think this is because the torchvision dataset stores the array of pixel values, while the HF dataset stores encoded images in PNG. Therefore the HF dataset uses significantly less disk space, but there is an extra decoding step to get the image.

Accessing dataset is very slow compared to torchvision

🤗Datasets

CarloLucibello May 15, 2022, 5:45am 2

related (same?) topic: Why is simply accessing dataset features so slow?

Topic		Replies	Views
Hugging face datasets -- reading image shape takes very long time Beginners	1	303	April 4, 2023
Slow Iteration speed (with and without keep_in_memory=True) 🤗Datasets	3	1520	March 14, 2023
Dataset slow during model training 🤗Datasets	1	2085	June 13, 2022
Why is it so slow to access data through iteration with hugginface dataset? Intermediate	2	2930	July 21, 2022
Why is simply accessing dataset features so slow? 🤗Datasets	3	3915	November 22, 2021

Accessing dataset is very slow compared to torchvision

Related topics