# LoadDataSet pyarrow.lib.ArrowCapacityError

**URL:** <https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328>\
**Category:** 🤗Datasets\
**Created:** [January 12, 2025, 4:14am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328 "2025-01-12T04:14:59Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 12, 2025, 5:18am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/4 "2025-01-12T05:18:00Z")

</div>

How about trying .shard()?

> [@How to load a large hf dataset efficiently?](https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288/3):
>
> Thank you for the reply, following your suggestion is there any way in hf dataset, split into multiple shards to be loaded in parallel?

---

_[View the full topic](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328)._