# Dataset Info: question about dataset sizes **URL:** \ **Category:** 🤗Datasets\ **Created:** [November 11, 2025, 8:55am UTC](https://discuss.huggingface.co/t/dataset-info-question-about-dataset-sizes/170278 "2025-11-11T08:55:16Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ldr934](https://avatars.discourse-cdn.com/v4/letter/l/919ad9/32.png) [@ldr934](https://discuss.huggingface.co/u/ldr934)\ **Post date:** [November 11, 2025, 8:55am UTC](https://discuss.huggingface.co/t/dataset-info-question-about-dataset-sizes/170278/1 "2025-11-11T08:55:16Z")
Hello! I am exploring trends across the datasets stored on the Hub. Can someone explain me the difference between the dataset\_size, download\_size, and usedStorage? I assume usedStorage is the storage space occupied by the dataset on the Hub, and that download\_size also includes metadata etc. Is this correct? Also, does these sizes reflect compressed or decompressed sizes? References: [https://huggingface.co/docs/datasets/package\_reference/main\_classes](https://huggingface.co/docs/datasets/package_reference/main_classes "https://huggingface.co/docs/datasets/package\_reference/main\_classes") [https://huggingface.co/spaces/hugging-science/dataset-insight-portal](https://huggingface.co/spaces/hugging-science/dataset-insight-portal "https://huggingface.co/spaces/hugging-science/dataset-insight-portal") Thank you! /Sophia ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [November 11, 2025, 3:28pm UTC](https://discuss.huggingface.co/t/dataset-info-question-about-dataset-sizes/170278/2 "2025-11-11T15:28:06Z")
The summary I got from repeated searches looks like this, but I’m not confident it’s correct… Since it’s about HF specifications. @lhoestq * * * Definitions first: - `download_size` = bytes the dataset builder downloads to create the dataset. It is the sum recorded by the download manager, not docs/README. If sources are compressed, this is the compressed on-wire size. ([Hugging Face](https://huggingface.co/docs/datasets/package_reference/main_classes "Main classes")) - `dataset_size` = bytes of the generated Arrow tables on disk across all splits after preparation. Think local, post-extraction, columnar footprint. ([Hugging Face](https://huggingface.co/docs/datasets/package_reference/main_classes "Main classes")) - `size_in_bytes` = total the library uses for disk checks and progress messages. Computed as `download_size + dataset_size` and, when present, `+ post_processing_size`. ([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/_modules/datasets/builder.html "datasets.builder — datasets 1.12.0 documentation")) - `usedStorage` = storage the dataset repo consumes on the Hub itself. The Insight Portal labels it “Storage used by the dataset (bytes).” It reflects files stored in the repo (CSV/JSON/Parquet shards, images, README, etc.). ([Hugging Face](https://huggingface.co/spaces/hugging-science/dataset-insight-portal/blob/main/app.py "app.py · hugging-science/dataset-insight-portal at main")) Background and mechanics: - The builder checks free disk using these fields and prints them in one line: “download …, generated …, post-processed …, total …”. That “total” is `size_in_bytes`. After building it also sets `size_in_bytes = dataset_size + download_size`. ([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/_modules/datasets/builder.html "datasets.builder — datasets 1.12.0 documentation")) - `download_size` comes from the download manager’s accounting of files actually fetched by the script, then extraction happens separately. It does not count repo metadata unless the script downloads it as a resource. ([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/_modules/datasets/builder.html "datasets.builder — datasets 1.12.0 documentation")) - Extraction/processing writes Arrow shards to the cache; those bytes are what `dataset_size` measures. This step happens after downloads. ([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/_modules/datasets/builder.html "datasets.builder — datasets 1.12.0 documentation")) - The Hub reports file sizes for stored blobs. For LFS files, the recorded size is the real blob size, not the small pointer. That is why `usedStorage` matches what’s physically stored server-side. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/package_reference/file_download "Downloading files")) Compressed vs. decompressed: - `download_size`: compressed if the upstream artifact is compressed; otherwise raw file size. It describes what traverses the network. ([Hugging Face](https://huggingface.co/docs/datasets/package_reference/main_classes "Main classes")) - `dataset_size`: the size of the produced Arrow data on disk. Post-extraction and typically larger than compressed downloads. ([Hugging Face](https://huggingface.co/docs/datasets/package_reference/main_classes "Main classes")) - `usedStorage`: size of whatever files live in the Hub repo. If you pushed compressed Parquet or zipped assets, it reflects those compressed files; if you pushed raw CSV, it reflects raw CSV bytes. ([Hugging Face](https://huggingface.co/spaces/hugging-science/dataset-insight-portal/blob/main/app.py "app.py · hugging-science/dataset-insight-portal at main")) What your assumptions got right and wrong: - Right: `usedStorage` is the space the dataset takes on the Hub. It is independent of your local cache. ([Hugging Face](https://huggingface.co/spaces/hugging-science/dataset-insight-portal/blob/main/app.py "app.py · hugging-science/dataset-insight-portal at main")) - Partially wrong: `download_size` is not “metadata + everything.” It is only the files the builder downloads to build the dataset; README and similar metadata are not included unless the script fetches them as inputs. ([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/_modules/datasets/builder.html "datasets.builder — datasets 1.12.0 documentation")) Quick ways to inspect the numbers: - Use the Dataset Viewer API to read `dataset_info` without downloading data; the JSON mirrors `DatasetInfo` and includes `download_size` and `dataset_size`. ([Hugging Face](https://huggingface.co/docs/dataset-viewer/en/info "Get dataset information")) - In the Python docs, `Dataset.info` examples show these fields live on common datasets so you can sanity-check behavior. ([Hugging Face](https://huggingface.co/docs/datasets/v2.1.0/access "The Dataset object")) Why the three numbers often differ: - Scripted datasets that pull from external URLs may have small `usedStorage` (little or nothing stored in the repo) but large `download_size`/`dataset_size` when you build locally. Conversely, pushing prepared Parquet/Arrow shards inflates `usedStorage` even if `download_size` is minimal. The library’s uploader shards datasets to the Hub, which is exactly what `usedStorage` counts. ([Hugging Face](https://huggingface.co/docs/datasets/package_reference/main_classes "Main classes")) Bottom line: - Treat `download_size` as the network payload. Treat `dataset_size` as the local Arrow footprint. Treat `usedStorage` as the Hub repo footprint. The library’s `size_in_bytes` equals the sum it prints in the builder logs and uses for disk checks. ([Hugging Face](https://huggingface.co/docs/datasets/package_reference/main_classes "Main classes")) ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 13, 2025, 1:03pm UTC](https://discuss.huggingface.co/t/dataset-info-question-about-dataset-sizes/170278/3 "2025-11-13T13:03:05Z")
Good summary ! Note that scripted datasets are no longer supported, so you shoudln’t see cases with small userStorage and big download\_size / dataset\_size. And post\_processing\_size is legacy, isn’t used anywhere and is probably getting removed soon ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [November 13, 2025, 1:20pm UTC](https://discuss.huggingface.co/t/dataset-info-question-about-dataset-sizes/170278/4 "2025-11-13T13:20:15Z")
Thank you! ---
**Author:** ![ldr934](https://avatars.discourse-cdn.com/v4/letter/l/919ad9/32.png) [@ldr934](https://discuss.huggingface.co/u/ldr934)\ **Post date:** [November 14, 2025, 1:34pm UTC](https://discuss.huggingface.co/t/dataset-info-question-about-dataset-sizes/170278/5 "2025-11-14T13:34:54Z")
Hi @John6666 Thank you so much for the big effort and thorough summary. It is very much appreciated. Best Sophia