Gated Linear Unit
Gated Linear Unit
A Gated Linear Unit, or GLU computes:

$$
\mathrm{GLU}(a, b) = a \otimes \sigma(b)
$$

It is used in natural language processing architectures, for example the Gated CNN, because here $\sigma(b)$ is the gate that control what information from $a$ is passed up to the following layer. Intuitively, for a language modeling task, the gating mechanism allows selection of words or features that are important for predicting the next word. The GLU also has non-linear capabilities, but has a linear path for the gradient so diminishes the vanishing gradient problem.
Language Modeling with Gated Convolutional Networks
https://paperswithcode.com/paper/language-modeling-with-gated-convolutional
Language Modeling with Gated Convolutional Networks