XLM
XLM
**XLM** is a [Transformer](https://paperswithcode.com/method/transformer) based architecture that is pre-trained using one of three language modelling objectives:

1. Causal Language Modeling - models the probability of a word given the previous words in a sentence.
2. Masked Language Modeling - the masked language modeling objective of [BERT](https://paperswithcode.com/method/bert).
3. Translation Language Modeling - a (new) translation language modeling objective for improving cross-lingual pre-training.

The authors find that both the CLM and MLM approaches provide strong cross-lingual features that can be used for pretraining models.
Cross-lingual Language Model Pretraining
https://paperswithcode.com/paper/cross-lingual-language-model-pretraining
Cross-lingual Language Model Pretraining