CoBERL
Contrastive BERT
**Contrastive BERT** is a reinforcement learning agent that combines a new contrastive loss and a hybrid [LSTM](https://paperswithcode.com/method/lstm)-[transformer](https://paperswithcode.com/method/transformer) architecture to tackle the challenge of improving data efficiency for RL. It uses bidirectional masked prediction in combination with a generalization of recent contrastive methods to learn better representations for transformers in RL, without the need of hand engineered data augmentations.

For the architecture, a residual network is used to encode observations into embeddings $Y\_{t}$. $Y_{t}$  is fed through a causally masked [GTrXL transformer](https://www.paperswithcode.com/method/gtrxl), which computes the predicted masked inputs $X\_{t}$ and passes those together with $Y\_{t}$ to a learnt gate. The output of the gate is passed through a single [LSTM](https://www.paperswithcode.com/method/lstm) layer to produce the values that we use for computing the RL loss. A contrastive loss is computed using predicted masked inputs $X_{t}$ and $Y_{t}$ as targets. For this, we do not use the causal mask of the Transformer.
CoBERL: Contrastive BERT for Reinforcement Learning
https://paperswithcode.com/paper/coberl-contrastive-bert-for-reinforcement
CoBERL: Contrastive BERT for Reinforcement Learning