# The datasets.map function does not load cached dataset **URL:** \ **Category:** Beginners\ **Created:** [August 2, 2021, 3:31am UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905 "2021-08-02T03:31:52Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ezio98](https://avatars.discourse-cdn.com/v4/letter/e/b9e5f3/32.png) [@ezio98](https://discuss.huggingface.co/u/ezio98)\ **Post date:** [August 2, 2021, 3:31am UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/1 "2021-08-02T03:31:52Z")
I am using the run\_mlm.py provided in the transformers repository to pretrain bert. The dataset is of version 1.8.0. Since the used dataset Wikipedia is large, I hope the processing is one time and can be reused later. However, I find it always re-computing instead of load from the disk. I don’t think I changed any parameters to the map function. I notice the description of the parameter **new\_fingerprint** might influence. The original codes of run\_mlm.py do not specify it, should I give it a value? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [August 16, 2021, 8:42am UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/2 "2021-08-16T08:42:14Z")
Hi ! `new_fingerprint` is computed automatically by taking into account: - the previous dataset fingerprint - a hash of your map function - a hash of the parameters passed to map So as long as you don’t change your code and you keep the same parameters, the fingerprint will stay the same and the dataset will be reloaded from the disk. Can you make sure you didn’t change your function or the parameters passed to map ? Note that `preprocessing_num_workers ` is part of the parameters passed to map and you must make sure it doesn’t change either. ---
**Author:** ![ezio98](https://avatars.discourse-cdn.com/v4/letter/e/b9e5f3/32.png) [@ezio98](https://discuss.huggingface.co/u/ezio98)\ **Post date:** [August 17, 2021, 7:24am UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/3 "2021-08-17T07:24:01Z")
Thanks for your reply! I am sure that all of the parameters are not changed. Actually, I have tried to figure out the reason and found an interesting thing. The map function indeed loads the processed datasets if I changed nothing. However, if I copy the codes to another .py file and run it, the datasets are processed again. But I suppose the second processing should not happen. The following is the code snippet, I changed nothing but run it in different files. ```auto if __name__ == ' __main__': from datasets import load_dataset from transformers import AutoTokenizer raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1") tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") def tokenize_function(examples): return tokenizer(examples["text"], padding="max_length", truncation=True) tokenized_datasets = raw_datasets.map(tokenize_function, batched=True) ``` ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [August 18, 2021, 3:36pm UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/4 "2021-08-18T15:36:46Z")
Thanks for reporting, indeed we should make sure the cached results get reloaded even if you move your script. Can you open an issue on [github](https://github.com/huggingface/datasets/issues) so we can work on a solution ? ---
**Author:** ![ezio98](https://avatars.discourse-cdn.com/v4/letter/e/b9e5f3/32.png) [@ezio98](https://discuss.huggingface.co/u/ezio98)\ **Post date:** [August 20, 2021, 1:50am UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/5 "2021-08-20T01:50:33Z")
Yes, I am glad to do that. 😀 ---
**Author:** ![ezio98](https://avatars.discourse-cdn.com/v4/letter/e/b9e5f3/32.png) [@ezio98](https://discuss.huggingface.co/u/ezio98)\ **Post date:** [August 23, 2021, 3:24am UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/6 "2021-08-23T03:24:30Z")
The link to the issue is [The datasets.map function does not load cached dataset · Issue #2825 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/2825) ---
**Author:** ![kopyl](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kopyl](https://discuss.huggingface.co/u/kopyl)\ **Post date:** [November 17, 2023, 6:58pm UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/7 "2023-11-17T18:58:22Z")
@lhoestq Why it is impossible to have a cached dataset when changing arguments for text to image SD training? Whenever I change my batch size, the model cache resets and spends another 5 hours mapping my dataset. This is so annoying. I would love to play around with my batch size while not having to wait 5 hours each time I change something in training args. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 21, 2023, 10:55am UTC](https://discuss.huggingface.co/t/the-datasets-map-function-does-not-load-cached-dataset/8905/8 "2023-11-21T10:55:28Z")
Changing the batch size invalidates the cache because the resulting dataset after processing is not necessarily the same depending on the batch size. For example if your processing tokenizes text with max-length padding, then your processed dataset is not the same on batch\_size 1 or batch\_size 8 (one has no padding and the other has padding with fixed length every 8 examples). I guess your problem is that you use the same batch\_size for data processing and for your model training. You should probably set a fixed value for your data processing step.