ECANet
efficient channel attention
An ECA block has similar formulation to an SE block including a squeeze module for aggregating global spatial information and an efficient excitation module for modeling cross-channel interaction. Instead of indirect correspondence, an ECA block only considers direct interaction between each channel and its k-nearest neighbors to control model complexity. Overall, the formulation of an ECA block is:
\begin{align}
    s = F_\text{eca}(X, \theta) & = \sigma (\text{Conv1D}(\text{GAP}(X))) 
\end{align}
\begin{align}
    Y & = s  X
\end{align}
where $\text{Conv1D}(\cdot)$ denotes 1D convolution with a kernel of shape $k$ across the channel domain, to model local cross-channel interaction. The parameter $k$ decides the coverage of interaction, and in ECA the kernel size $k$ is adaptively determined from the channel dimensionality $C$ instead of by manual tuning, using cross-validation:
\begin{equation}
    k = \psi(C) = \left | \frac{\log_2(C)}{\gamma}+\frac{b}{\gamma}\right |_\text{odd}
\end{equation}

where $\gamma$ and $b$ are hyperparameters. $|x|_\text{odd}$ indicates the nearest odd function of $x$. 

Compared to SENet, ECANet has an 
improved excitation module, and provides an efficient and effective block which can readily be 
 incorporated into various
CNNs.
ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
https://paperswithcode.com/paper/eca-net-efficient-channel-attention-for-deep
ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks