ELiSH
Exponential Linear Squashing Activation
The **Exponential Linear Squashing Activation Function**, or **ELiSH**, is an activation function used for neural networks. It shares common properties with [Swish](https://paperswithcode.com/method/swish), being made up of an [ELU](https://paperswithcode.com/method/elu) and a [Sigmoid](https://paperswithcode.com/method/sigmoid-activation):

$$f\left(x\right) = \frac{x}{1+e^{-x}} \text{ if } x \geq 0 $$
$$f\left(x\right) = \frac{e^{x} - 1}{1+e^{-x}} \text{ if } x < 0 $$

The Sigmoid part of **ELiSH** improves information flow, while the linear parts solve issues of vanishing gradients.
The Quest for the Golden Activation Function
https://paperswithcode.com/paper/the-quest-for-the-golden-activation-function
The Quest for the Golden Activation Function