Spectral Normalization
Spectral Normalization
**Spectral Normalization** is a normalization technique used for generative adversarial networks, used to stabilize training of the discriminator. Spectral normalization has the convenient property that the Lipschitz constant is the only hyper-parameter to be tuned.

It controls the Lipschitz constant of the discriminator $f$ by constraining the spectral norm of each layer $g : \textbf{h}\_{in} \rightarrow \textbf{h}_{out}$. The Lipschitz norm $\Vert{g}\Vert\_{\text{Lip}}$ is equal to $\sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right)$, where $\sigma\left(a\right)$ is the spectral norm of the matrix $A$ ($L\_{2}$ matrix norm of $A$):

$$ \sigma\left(a\right) = \max\_{\textbf{h}:\textbf{h}\neq{0}}\frac{\Vert{A\textbf{h}}\Vert\_{2}}{\Vert\textbf{h}\Vert\_{2}} = \max\_{\Vert\textbf{h}\Vert\_{2}\leq{1}}{\Vert{A\textbf{h}}\Vert\_{2}} $$

which is equivalent to the largest singular value of $A$. Therefore for a [linear layer](https://paperswithcode.com/method/linear-layer) $g\left(\textbf{h}\right) = W\textbf{h}$ the norm is given by $\Vert{g}\Vert\_{\text{Lip}} = \sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right) = \sup\_{\textbf{h}}\sigma\left(W\right) = \sigma\left(W\right) $. Spectral normalization normalizes the spectral norm of the weight matrix $W$ so it satisfies the Lipschitz constraint $\sigma\left(W\right) = 1$:

$$ \bar{W}\_{\text{SN}}\left(W\right) = W / \sigma\left(W\right) $$
Spectral Normalization for Generative Adversarial Networks
https://paperswithcode.com/paper/spectral-normalization-for-generative
Spectral Normalization for Generative Adversarial Networks