Embedding Dropout
Embedding Dropout
**Embedding Dropout** is equivalent to performing [dropout](https://paperswithcode.com/method/dropout) on the embedding matrix at a word level, where the dropout is broadcast across all the word vector’s embedding. The remaining non-dropped-out word embeddings are scaled by $\frac{1}{1-p\_{e}}$ where $p\_{e}$ is the probability of embedding dropout. As the dropout occurs on the embedding matrix that is used for a full forward and backward pass, this means that all occurrences of a specific word will disappear within that pass, equivalent to performing [variational dropout](https://paperswithcode.com/method/variational-dropout) on the connection between the one-hot embedding and the embedding lookup.

Source: Merity et al, Regularizing and Optimizing [LSTM](https://paperswithcode.com/method/lstm) Language Models
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
https://paperswithcode.com/paper/a-theoretically-grounded-application-of
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks