TernaryBERT
TernaryBERT
**TernaryBERT** is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based model which ternarizes the weights of a pretrained [BERT](https://paperswithcode.com/method/bert) model to $\{-1,0,+1\}$, with different granularities for word embedding and weights in the Transformer layer. Instead of directly using knowledge distillation to compress a model, it is used to improve the performance of ternarized student model with the same size as the teacher model. In this way, we transfer the knowledge from the highly-accurate teacher model to the ternarized student model with smaller capacity.
TernaryBERT: Distillation-aware Ultra-low Bit BERT
https://paperswithcode.com/paper/ternarybert-distillation-aware-ultra-low-bit
TernaryBERT: Distillation-aware Ultra-low Bit BERT