STD
Spatial-Channel Token Distillation
The **Spatial-Channel Token Distillation** method is proposed to improve the spatial and channel mixing from a novel knowledge distillation (KD) perspective. To be specific, we design a special KD mechanism for MLP-like Vision Models called Spatial-channel Token Distillation (STD), which improves the information mixing in both the spatial and channel dimensions of MLP blocks. Instead of modifying the mixing operations themselves, STD adds spatial and channel tokens to image patches. After forward propagation, the tokens are concatenated for distillation with the teachers’ responses as targets. Each token works as an aggregator of its dimension. The objective of them is to encourage each mixing operation to extract maximal task-related information from their specific dimension.
Spatial-Channel Token Distillation for Vision MLPs
https://paperswithcode.com/paper/spatial-channel-token-distillation-for-vision
Spatial-Channel Token Distillation for Vision MLPs