# Continue Pre-Training Roberta

**URL:** <https://discuss.huggingface.co/t/continue-pre-training-roberta/34888>\
**Category:** Intermediate\
**Created:** [March 28, 2023, 9:23am UTC](https://discuss.huggingface.co/t/continue-pre-training-roberta/34888 "2023-03-28T09:23:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![parthb](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@parthb](https://discuss.huggingface.co/u/parthb)\
**Post date:** [March 28, 2023, 9:23am UTC](https://discuss.huggingface.co/t/continue-pre-training-roberta/34888/1 "2023-03-28T09:23:57Z")

</div>

I want to further train the Roberta Model on my own dataset.

**General Idea :**

I want to resume the training from the last point where Roberta was trained and continue that training on my dataset.

I have a fairly small dataset - 60K Sentences mostly in English (Domain Specific). I want to leverage transfer learning in Transformers.

Is it even possible? If so how?

---

<div class="post-metadata">

**Author:** ![LuckiestOne](https://avatars.discourse-cdn.com/v4/letter/l/2bfe46/32.png) [@LuckiestOne](https://discuss.huggingface.co/u/LuckiestOne)\
**Post date:** [March 29, 2023, 9:39am UTC](https://discuss.huggingface.co/t/continue-pre-training-roberta/34888/2 "2023-03-29T09:39:29Z")

</div>

Yes, it surely is.  
Roberta is not the easiest model for finetuning since it can be used for many tasks. I am sure you will find a finetuned version of a roberta checkpoint for your specific task in [here](https://huggingface.co/models?other=roberta). You can simply copy a lot of the hyperparameters and tweak the others like batch size, learning rate and maybe even some of the adam values.  
All of this can be done using HFs `Trainer` class, even though the training itselfs depends on the task you want to use it for.  
For ex. Text classification can be done easily following [this notebook](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb)

For additional resources, you might want to take a look at plain BERT. There are way more example scripts for training and the architecture is the same except for the tokenizer.  
See the [BERT](https://huggingface.co/docs/transformers/v4.27.2/en/model_doc/bert) Model page, it will be helpful - many guides await you.

Good luck finetuning!

---

<div class="post-metadata">

**Author:** ![Aung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aung/32/4217_2.png) [@Aung](https://discuss.huggingface.co/u/Aung)\
**Post date:** [April 14, 2023, 7:21am UTC](https://discuss.huggingface.co/t/continue-pre-training-roberta/34888/3 "2023-04-14T07:21:46Z")

</div>

> [@LuckiestOne](#):
>
> same except for the tokenizer.  
> See the [BERT](https://huggingface.co/docs/transformers/v4.27.2/en/model_doc/bert) Model page, it will be helpful - many guides await you.
> 
> Good luck finetuning!

The original question is to continue pre-training the model, so I’m not sure if fine-tuning is the right answer here. It also depends on what sort of dataset they have, whether it is a general text corpus for pre-training or a task specific dataset for fine-tuning. Are there examples on how to continue pre-training the model with the smaller domain specific corpus?

---

<div class="post-metadata">

**Author:** ![parthb](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@parthb](https://discuss.huggingface.co/u/parthb)\
**Post date:** [May 18, 2023, 2:14pm UTC](https://discuss.huggingface.co/t/continue-pre-training-roberta/34888/4 "2023-05-18T14:14:00Z")

</div>

Hey, there is a script I came across called run\_mlm.py. This script helped me in fine-tuning, do take a look.

([transformers/examples/pytorch/language-modeling at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling))

Hope this helps,  
Parth