modReLU
modReLU
**modReLU** is an activation that is a modification of a [ReLU](https://paperswithcode.com/method/relu). It is a pointwise nonlinearity, $\sigma\_{modReLU}\left(z\right) : C \rightarrow C$, which affects only the absolute value of a complex number, defined as:

$$ \sigma\_{modReLU}\left(z\right) = \left(|z| + b\right)\frac{z}{|z|} \text{ if } |z| + b \geq 0 $$
$$ \sigma\_{modReLU}\left(z\right) = 0 \text{ if } |z| + b \leq 0 $$

where $b \in \mathbb{R}$ is a bias parameter of the nonlinearity. For a $n\_{h}$ dimensional hidden space we learn $n\_{h}$ nonlinearity bias parameters, one per dimension.
Unitary Evolution Recurrent Neural Networks
https://paperswithcode.com/paper/unitary-evolution-recurrent-neural-networks
Unitary Evolution Recurrent Neural Networks