Mixed Attention Block
Mixed Attention Block
**Mixed Attention Block** is an attention module used in the [ConvBERT](https://paperswithcode.com/method/convbert) architecture. It is a mixture of [self-attention](https://paperswithcode.com/method/scaled) and [span-based dynamic convolution](https://paperswithcode.com/method/span-based-dynamic-convolution) (highlighted in pink). They share the same Query but use different Key to generate the attention map and [convolution](https://paperswithcode.com/method/convolution) kernel respectively. The number of attention heads is reducing by directly projecting the input to a smaller embedding space to form a bottleneck structure for self-attention and span-based dynamic convolution. Dimensions of the input and output of some blocks are labeled on the left top corner to illustrate the overall framework, where $d$ is the embedding size of the input and $\gamma$ is the reduction ratio.
ConvBERT: Improving BERT with Span-based Dynamic Convolution
https://paperswithcode.com/paper/convbert-improving-bert-with-span-based
ConvBERT: Improving BERT with Span-based Dynamic Convolution