# Map with tokenize function stuck in the beginning

**URL:** <https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047>\
**Category:** 🤗Datasets\
**Created:** [December 26, 2024, 2:21am UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047 "2024-12-26T02:21:00Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [December 27, 2024, 6:41am UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047/2 "2024-12-27T06:41:27Z")

</div>

Maybe try IterableDataset?

> [@Training a Tokenizer on a Streamed Dataset](https://discuss.huggingface.co/t/training-a-tokenizer-on-a-streamed-dataset/8026):
>
> Hi, I’m trying to train a tokenizer on a dataset that uses streaming. I followed the instructions provided [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer), with the addition of streaming=True during the dataset loading step. However, it quickly failed as the IterableDataset class does not have a length property (unlike the normal Dataset class). How can I work around this issue without having to download the dataset file entirely, i.e. purely relying on dataset streaming? Here’s the [Colab link](https://colab.research.google.com/drive/1rOdAzmfdp2S1tGDk4iaiM2d4pai4QGES?usp=sharing) to reproduce the error. Any help would be …

---

_[View the full topic](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047)._