Switch FFN
Switch FFN
A **Switch FFN** is a sparse layer that operates independently on tokens within an input sequence. It is shown in the blue block in the figure. We diagram two tokens ($x\_{1}$ = “More” and $x\_{2}$ = “Parameters” below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
https://paperswithcode.com/paper/switch-transformers-scaling-to-trillion
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity