# Is the huggingface run\_mlm Script dynamically masked? **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 20, 2021, 9:51am UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370 "2021-08-20T09:51:36Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![moma1820](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moma1820/32/4141_2.png) [@moma1820](https://discuss.huggingface.co/u/moma1820)\ **Post date:** [August 20, 2021, 9:51am UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/1 "2021-08-20T09:51:36Z")
Hi, In the roberta paper, the model is trained by dynamic masking of sentences. If a roberta model is further pre-trained using the `run_mlm.py` , is the sentences going to be dynamically masked during pre-training using this file or is it statically masked like vanilla BERT? The script is found here > **[transformers/examples/pytorch/language-modeling at master ·...](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling)** > > master/examples/pytorch/language-modeling ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [August 21, 2021, 8:33am UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/2 "2021-08-21T08:33:33Z")
Apparently, dynamic masking is always used, no matter the model. See this: [dynamic masking for RoBERTa model · Issue #5979 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/5979) ---
**Author:** ![moma1820](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moma1820/32/4141_2.png) [@moma1820](https://discuss.huggingface.co/u/moma1820)\ **Post date:** [August 21, 2021, 9:53am UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/3 "2021-08-21T09:53:08Z")
Hello 😃 Thats actually cool Have an awesome day Neils! ---
**Author:** ![moma1820](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moma1820/32/4141_2.png) [@moma1820](https://discuss.huggingface.co/u/moma1820)\ **Post date:** [August 25, 2021, 12:29pm UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/4 "2021-08-25T12:29:05Z")
Hi again, I have one small question, What does the checkpoint folders that are created during the pre training. There folders are eating up my har drive space during training large text files(or when i was pre training line by line). ---
**Author:** ![anon81579828](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@anon81579828](https://discuss.huggingface.co/u/anon81579828)\ **Post date:** [May 27, 2022, 6:06am UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/5 "2022-05-27T06:06:15Z")
Hey I don’t quite understand what is being said in the discussion linked by @nielsr Can someone please confirm the claim and explain it? ---
**Author:** ![lezakkaz](https://avatars.discourse-cdn.com/v4/letter/l/bc8723/32.png) [@lezakkaz](https://discuss.huggingface.co/u/lezakkaz)\ **Post date:** [May 27, 2022, 4:47pm UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/6 "2022-05-27T16:47:10Z")
They are talking about dynamic masking that was mentioned in the [RoBERTa paper](https://arxiv.org/pdf/1907.11692.pdf). The [original BERT paper](https://arxiv.org/pdf/1810.04805.pdf) did not use dynamic masking, but rather static masking meaning they masked all sentences before training and used those same masks across multiple epochs. Dynamic masking is better because it masks sentences during training, meaning that each epoch would have different masks. Since the masks are different each time, the learning is more challenging and thus converges slower than the original BERT approach. All masked language models (including BertModel) implementations on Huggingface use [Data Collators](https://huggingface.co/docs/transformers/v4.19.2/en/main_classes/data_collator) to mask sentences. Collators are inherently dynamic because they are applied during training. If you want to use static masking then you will have to code that yourself. I hope that clarifies it for you. ---
**Author:** ![anon81579828](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@anon81579828](https://discuss.huggingface.co/u/anon81579828)\ **Post date:** [May 29, 2022, 6:12am UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/7 "2022-05-29T06:12:14Z")
Thanks for the explanaton! I still don’t quite get the part where the paper says each sentence is duplicated 10 times and passed into the masking function though. The DataCollator does not do that right? ---
**Author:** ![lezakkaz](https://avatars.discourse-cdn.com/v4/letter/l/bc8723/32.png) [@lezakkaz](https://discuss.huggingface.co/u/lezakkaz)\ **Post date:** [May 29, 2022, 3:17pm UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/8 "2022-05-29T15:17:23Z")
It is explained in the Roberta paper, section 4.1 static vs dynamic masking. So basically they duplicated their corpus 10 times and masked each sentence in it. That way every sentence gets new masks. They did so over 40 epochs (duplicated corpus 10 \* 4), so each sentence with the same masks is trained 4 times. As for the DataCollator class (DataCollatorForLanguageModeling), you have to understand that their code doesn’t follow the original papers. The collator masks a batch of sentences in real-time. So no duplication, but still dynamic masking. The only difference is that the masks might be different each epoch across many epochs. Also, at least in the case of the [torch implementation](https://github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/data/data_collator.py#L748), masking is done on the batch, and not on each sentence, which means that sometimes sentences might have no masks whatsoever, and some other times sentences might end up with more than 15% masks. Took me a while to figure that out… ---
**Author:** ![anon81579828](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@anon81579828](https://discuss.huggingface.co/u/anon81579828)\ **Post date:** [June 1, 2022, 11:46pm UTC](https://discuss.huggingface.co/t/is-the-huggingface-run-mlm-script-dynamically-masked/9370/9 "2022-06-01T23:46:31Z")
Thanks a lot for your reply. Really helped clear things up