# OOM issue with large dataset streaming

**URL:** <https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488>\
**Category:** 🤗Datasets\
**Created:** [March 6, 2025, 3:56pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488 "2025-03-06T15:56:42Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 6, 2025, 4:06pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/2 "2025-03-06T16:06:22Z")

</div>

Is the data set file itself too large…?  
If it seems like a bug, let’s call lhonestq.

> [@How to load a large hf dataset efficiently?](https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288):
>
> I am trying to load a dataset [axiong/pmc\_oa · Datasets at Hugging Face](https://huggingface.co/datasets/axiong/pmc_oa) The dataset size is around 22 gb and I have ram ~10 GB, the dataset object is stuck at extracting file point I also tried streaming mode but that’s giving another error. from datasets import load\_dataset dataset = load\_dataset("axiong/pmc\_oa", 'pmc\_oa', split='train', streaming=True) print(next(iter(dataset))) Any suggestion on how to deal with…

---

_[View the full topic](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488)._