# batched I/O from disk when load\_dataset API is used? **URL:** \ **Category:** 🤗Datasets\ **Created:** [January 26, 2025, 12:06pm UTC](https://discuss.huggingface.co/t/batched-i-o-from-disk-when-load-dataset-api-is-used/137391 "2025-01-26T12:06:03Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [January 26, 2025, 1:24pm UTC](https://discuss.huggingface.co/t/batched-i-o-from-disk-when-load-dataset-api-is-used/137391/2 "2025-01-26T13:24:59Z")
Is this it? It’s probably a little different… > [@Streaming dataset and cache](https://discuss.huggingface.co/t/streaming-dataset-and-cache/32751): > > Hello! I was wondering about the inner working of dataset streaming. Is there any form of caching of the samples that will be needed soon or are they downloaded and processed only when the model requires them for training ? In the later case, we could expect a higher latency when going from a batch to another, especially when the samples require some heavy processing. In this case it’s probably better to process the dataset locally beforehand. Cheers --- _[View the full topic](https://discuss.huggingface.co/t/batched-i-o-from-disk-when-load-dataset-api-is-used/137391)._