Adaptive Softmax
Adaptive Softmax
**Adaptive Softmax** is a speedup technique for the computation of probability distributions over words. The adaptive [softmax](https://paperswithcode.com/method/softmax) is inspired by the class-based [hierarchical softmax](https://paperswithcode.com/method/hierarchical-softmax), where the word classes are built to minimize the computation time. Adaptive softmax achieves efficiency by explicitly taking into account the computation time of matrix-multiplication on parallel systems and combining it with a few important observations, namely keeping a shortlist of frequent words in the root node
and reducing the capacity of rare words.
Efficient softmax approximation for GPUs
https://paperswithcode.com/paper/efficient-softmax-approximation-for-gpus
Efficient softmax approximation for GPUs