# Resuming .map Transform with Intermediate Caching in Dataset Generation

**URL:** <https://discuss.huggingface.co/t/resuming-map-transform-with-intermediate-caching-in-dataset-generation/73721>\
**Category:** 🤗Datasets\
**Created:** [February 18, 2024, 5:38pm UTC](https://discuss.huggingface.co/t/resuming-map-transform-with-intermediate-caching-in-dataset-generation/73721 "2024-02-18T17:38:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kdcyberdude](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kdcyberdude/32/27478_2.png) [@kdcyberdude](https://discuss.huggingface.co/u/kdcyberdude)\
**Post date:** [February 18, 2024, 5:38pm UTC](https://discuss.huggingface.co/t/resuming-map-transform-with-intermediate-caching-in-dataset-generation/73721/1 "2024-02-18T17:38:04Z")

</div>

I’m currently utilizing an LLM to generate a synthetic dataset, a process that could extend over several weeks on a single GPU. I am using the `.map` function for the transformation of the existing dataset into a synthetic one. My query revolves around efficient data caching strategies that enable process resumption from the last checkpoint if the operation is interrupted or terminated unexpectedly.

Although I’ve tried using the `cache_file_name` parameter with the `.map` function, it doesn’t seem to resume from where it left off; instead, it restarts from the beginning. Is there a more effective method or workaround to achieve resumable processing, ensuring that progress isn’t lost and computation time isn’t wasted?

Any advice or experiences shared would be greatly appreciated.

cc: @lhoestq

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 19, 2024, 1:31pm UTC](https://discuss.huggingface.co/t/resuming-map-transform-with-intermediate-caching-in-dataset-generation/73721/2 "2024-02-19T13:31:19Z")

</div>

Hi ! The cache is able to reload results once they’re completely processed, it’s not able to reload and resume a partial `map()` call.

There no native resuming in `datasets`, though you can e.g. chunk your dataset and process it by chunk
