SwiGLU
SwiGLU
**SwiGLU** is an activation function which is a variant of [GLU](https://paperswithcode.com/method/glu). The definition is as follows:

$$ \text{SwiGLU}\left(x, W, V, b, c, \beta\right) = \text{Swish}\_{\beta}\left(xW + b\right) \otimes \left(xV + c\right) $$
GLU Variants Improve Transformer
https://paperswithcode.com/paper/glu-variants-improve-transformer
GLU Variants Improve Transformer