# How to handle the cache system properly?

**URL:** <https://discuss.huggingface.co/t/how-to-handle-the-cache-system-properly/165186>\
**Category:** 🤗Datasets\
**Created:** [July 31, 2025, 2:20pm UTC](https://discuss.huggingface.co/t/how-to-handle-the-cache-system-properly/165186 "2025-07-31T14:20:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jtourille](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jtourille/32/51989_2.png) [@jtourille](https://discuss.huggingface.co/u/jtourille)\
**Post date:** [July 31, 2025, 2:20pm UTC](https://discuss.huggingface.co/t/how-to-handle-the-cache-system-properly/165186/1 "2025-07-31T14:20:56Z")

</div>

Hello,

I’ve been struggling with large datasets on our HPC cluster and I think I don’t understand fully how to use the datasets library.

I want to use fineweb-edu, so I downloaded the dataset using `snapshot_download` as presented below. I skip the sample parts. The dataset is stored on a NFS server.

```python
...

_ = snapshot_download(
    "HuggingFaceFW/fineweb-edu",
    repo_type="dataset",
    allow_patterns=["data/*"],
    local_dir=os.path.join(target_dir, "HuggingFaceFW/fineweb-edu"),
)

...

```

Now I want to load this dataset. As I understand, `load_dataset` will create an optimized cache version of this dataset. So I specify the location of the cache using the `HF_HOME` variable and load the dataset.

```python
...

_ = load_dataset(os.path.join(target_dir, "HuggingFaceFW/fineweb-edu", "data"), num_proc=12)

...

```

The cached version is correctly built, so far so good. Now I want to load this cached version without having access to the original parquet files. How can I do that ? I’ve read the documentation several times and the best way seems to use the `arrow` builder with `load_dataset("arrow", data_files=data_files)` but the process is building another cached version of this dataset. I’ve tried to disable caching without success.

I am sure I am missing something, could someone point me in the right direction ?

Thanks,

Julien

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 31, 2025, 3:29pm UTC](https://discuss.huggingface.co/t/how-to-handle-the-cache-system-properly/165186/2 "2025-07-31T15:29:46Z")

</div>

Some resources for now:

> **[Cache management](https://huggingface.co/docs/datasets/en/cache)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> [@Best way to access the cached transformation arrow file](https://discuss.huggingface.co/t/best-way-to-access-the-cached-transformation-arrow-file/29422):
>
> I was experimenting with the cached arrow dataset but ran into an error The steps followed Loaded the dataset from HF using load\_dataset() I created a simple transformation by shuffling the data dataset = dataset["test"].shuffle(seed=42) This step produced a new cache file in the hugging face cache directory - ~/.cache/huggingface/datasets/bt-tech\_\_\_ofac/dataset/1.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559/cache-c7f4daa392c5ed6a.arrow After I restart the sessio…

> [@Load dataset from a specific cache file](https://discuss.huggingface.co/t/load-dataset-from-a-specific-cache-file/24994):
>
> Dear community, Does anyone know if there is a way to load dataset from a specific cached file? I applied very costly transformations using a map function to do some data augmentation and finally, I had my dataset. However, now in my main script, I would like to use the cached file built from another script. Is there any way to call a specific cache file from ~/.cache/huggingface/datasets/ ?

---

<div class="post-metadata">

**Author:** ![jtourille](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jtourille/32/51989_2.png) [@jtourille](https://discuss.huggingface.co/u/jtourille)\
**Post date:** [August 1, 2025, 3:43pm UTC](https://discuss.huggingface.co/t/how-to-handle-the-cache-system-properly/165186/3 "2025-08-01T15:43:36Z")

</div>

Thanks for the pointers, I’ve read them already.

I think my question could be rephrased: **how can we manage large datasets in large companies ?** More precisely, I am looking at a way to avoid to have multiple copies of the same dataset at different locations and minimize the number of download (ideally only one).

I’ve been experimenting since my initial question and I’ve come up with this workflow:

1. Download the dataset with `snapshot_download()` or `git`.
2. One person load the dataset with `load_dataset()` and export it with `save_to_disk()`
3. Other people that want to use the dataset can make a local copy, do their stuff and remove everything afterwards.

Does that make sense ?

One thing I don’t understand is that `load_dataset()` and `save_to_disk()` both save the dataset in arrow format. However, as far as I understand, they do not perform the same optimizations, and therefore `load_from_disk` cannot load the dataset directly from the cache. Is there any particular reason for that ?

Bests

Julien

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 2, 2025, 2:16am UTC](https://discuss.huggingface.co/t/how-to-handle-the-cache-system-properly/165186/4 "2025-08-02T02:16:05Z")

</div>

> `load_dataset()` and `save_to_disk()`

Even though they are saved in the same Arrow format, [there is a difference in purpose between long-term storage and internal caching for speed optimization](https://github.com/huggingface/datasets/issues/6703#issuecomment-1974761165). Some people seem to be [trying to reuse them](https://discuss.huggingface.co/t/best-way-to-access-the-cached-transformation-arrow-file/29422).

Additionally, while I am unsure how robust it is in multi-process or multi-user environments, [one possible solution to avoid re-downloading the cache is to simply set `HF_HOME` to a relatively fast _shared folder_ on the network](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache). For remote environments, services like S3 seem to be available by default. However, this does come at a cost…

By the way, for enterprise use cases, [there is an option to consult dedicated support on Hugging Face](https://huggingface.co/enterprise). Whether this is suitable or not will depend on the scale of the project.