CuBERT
CuBERT
**CuBERT**, or **Code Understanding BERT**, is a [BERT](https://paperswithcode.com/method/bert) based model for code understanding. In order to achieve this, the authors curate a massive corpus of Python programs collected from GitHub. GitHub projects are known to contain a large amount of duplicate code. To avoid biasing the model to such duplicated code, authors perform deduplication using the method of [Allamanis (2018)](https://arxiv.org/abs/1812.06469). The resulting corpus has 7.4 million files with a total of 9.3 billion tokens (16 million unique).
Learning and Evaluating Contextual Embedding of Source Code
https://paperswithcode.com/paper/pre-trained-contextual-embedding-of-source-1
Learning and Evaluating Contextual Embedding of Source Code