Improving Sample Quality of Diffusion Models Using Self-Attention Guidance

Susung Hong
Gyuseong Lee
Wooseok Jang
Seungryong Kim
Korea University, Seoul, Korea

{susung1999, jpl358, jws1997<EMAIL_ADDRESS>
(a) ADM [7] without (top) and with (bottom) SAG

(b) Stable Diffusion [31] without (top) and with (bottom) SAG

Qualitative comparisons between unguided (top) and self-attention-guided (bottom) samples. Unlike classifier guidance (CG) [7] or classifier-free guidance (CFG) [16], self-attention guidance (SAG) does not necessarily require an external condition, , a class label or text prompt, nor additional training, improving the details of the images generated by pre-trained diffusion models such as (a) unconditional ADM [7] and (b) Stable Diffusion [31] with an empty prompt.

Denoising diffusion models (DDMs) have attracted attention for their exceptional generation quality and diversity. This success is largely attributed to the use of class- or text-conditional diffusion guidance methods, such as classifier and classifier-free guidance. In this paper, we present a more comprehensive perspective that goes beyond the traditional guidance methods. From this generalized perspective, we introduce novel condition- and training-free strategies to enhance the quality of generated images. As a simple solution, blur guidance improves the suitability of intermediate samples for their fine-scale information and structures, enabling diffusion models to generate higher quality samples with a moderate guidance scale. Improving upon this, Self-Attention Guidance (SAG) uses the intermediate self-attention maps of diffusion models to enhance their stability and efficacy. Specifically, SAG adversarially blurs only the regions that diffusion models attend to at each iteration and guides them accordingly. Our experimental results show that our SAG improves the performance of various diffusion models, including ADM, IDDPM, Stable Diffusion, and DiT. Moreover, combining SAG with conventional guidance methods leads to further improvement.

[The project page and code can be accessed at:

§ INTRODUCTION

Recently, denoising diffusion models (DDMs) [35, 37, 14, 7, 15, 31], which synthesize images from noise through an iterative denoising process, have been actively researched and attracted attention due to their exceptional performance in synthesizing high-quality and diverse images.

Behind this remarkable success lies the introduction of diffusion guidance methods [7, 23, 16]. Several studies have revealed that to improve the quality of image samples generated by diffusion models, guidance techniques using class labels [7, 16] or captions [23] are essential. However, despite the significant improvement provided by these guidance methods, they are bounded within the limits of using external conditions. For example, classifier guidance (CG) [7] requires the training of an additional classifier, and classifier-free guidance (CFG) [16] adds complexity to the training process through label-dropping. In addition, both methods are limited by their need for hard-earned external conditions, which binds them to conditional settings.

In light of the limitations mentioned above, in this work, we present a more general formulation of diffusion guidance that can make use of information within the intermediate samples of diffusion models. This formulation detaches the necessary condition of traditional approaches [16, 7, 23], , the requirement for external information, from diffusion guidance, and facilitates a flexible and condition-free approach to guide diffusion models. This broadens the applicability of diffusion guidance to cases with or without external conditions.

Based on the generalized formulation and the intuition that any internal information within intermediate samples can also serve as guidance, we firstly propose blur guidance as a straightforward solution to improve sample quality. Blur guidance uses the eliminated information resulting from Gaussian blur to guide intermediate samples, exploiting the benign property of Gaussian blur that it naturally removes fine-scale details [17, 20, 30]. While our results show that this method improves sample quality with a moderate guidance scale, it becomes problematic with a large guidance scale, since it may introduce structural ambiguity in entire regions, which makes it difficult to align the prediction of the degraded input with that of the original one.

To improve the effectiveness and stability of blur guidance with a larger guidance scale, we explore the self-attention mechanism of diffusion models. Generally, recent diffusion models [14, 7, 24, 31, 15, 27] are equipped with a self-attention module [40, 8] within their architecture. Claiming that the self-attention is a key to capture salient information during generation process [18, 45, 46, 12], we present Self-Attention Guidance (SAG), which adversarially blurs the region that contains salient information using the self-attention map of diffusion models and guides diffusion models with the residual information. Leveraging the attention maps during the reverse process of diffusion models, it can encouragingly boost the quality and reduce the artifacts through self-conditioning without requiring external information nor additional training, as shown in Fig. <ref>. The pseudocode and pipeline are provided in Alg. <ref> and Fig. <ref>(b), respectively.

In experiments, we evaluate the effectiveness of the proposed approach by plugging it into various diffusion models including ADM [7], IDDPM [24], Stable Diffusion [31], and DiT [27], which demonstrates our method's broad applicability. We also show that in addition to the increased sample quality when using SAG alone, performance further improves when using it on top of existing guidance schemes, , classifier [7] or classifier-free [16] guidance, demonstrating the orthogonality with the existing methods. Finally, we present ablation studies to validate our choices.

To sum up, our work has the following contributions:

* Generalizing conditional guidance methods [7, 16, 23] into a condition-free method that can be applied to any diffusion model without external conditions, expanding the applicability of guidance.

* Introducing novel guidance, dubbed Self-Attention Guidance (SAG), that uses the internal self-attention maps of diffusion models, improving sample quality without external conditions or additional fine-tuning.

* Demonstrating the orthogonality of SAG to existing conditional models and methods, enabling its flexible combination with others to achieve higher performance.

* Presenting extensive ablation studies to justify the design choices and demonstrate the effectiveness of the proposed method.

§ RELATED WORK

Denoising diffusion models.
Diffusion models [35], which are closely related to score-based models [37, 38], have attracted much attention owing to their superior sampling quality and diversity. As a pioneering work, DDPM [14] generates an image through an iterative process that progressively performs denoising to recover an image. Following this work, there have been several approaches to improve the sampling process, in terms of quality and speed [36, 24, 31, 15, 7]. Notably, IDDPM [24] additionally predicts the variance of the reverse process of the diffusion model. DDIM [36] accelerates the sampling speed by introducing the non-Markovian diffusion process. LDM [31] reduces the computational cost by processing the diffusion process in the latent space.

Self-Attention Guidance (SAG) Sampling

$\textrm{Model}(\mathbf{x}_t)$: a diffusion model that outputs the predicted noise $\epsilon_t$, variance $\Sigma_t$, and self-attention map $A_t$ given the input $\mathbf{x}_t$.

$\textrm{Gaussian-Blur}(\hat{\mathbf{x}}_0)$: a Gaussian blurring function.

[0]
$\mathbf{x}_T \sim \mathcal{N}(0, \mathbf{I})$
$t$ in $T, T-1, ..., 1$
$\epsilon_t, \Sigma_t, A_t \leftarrow \textrm{Model}(\mathbf{x}_t)$
$M_t \leftarrow \mathbbm{1}(A_t > \psi)$
$\hat{\mathbf{x}}_0 \leftarrow (\mathbf{x}_t-\sqrt{1-\bar{\alpha}_t}\epsilon_t)/\sqrt{\bar{\alpha}_t}$Eq. <ref>
$\tilde{\mathbf{x}}_0 \leftarrow \textrm{Gaussian\text-Blur}(\hat{\mathbf{x}}_0)$
$\tilde{\mathbf{x}}_t \leftarrow \sqrt{\bar{\alpha}_t}\tilde{\mathbf{x}}_0 + \sqrt{1-\bar{\alpha}_t}\epsilon_t $
$\widehat{{\mathbf{x}}}_t \leftarrow (1-M_t)\odot {\mathbf{x}}_t+M_t\odot \tilde{\mathbf{x}}_t$Eq. <ref>
$\widehat{\epsilon}_t \leftarrow \textrm{Model}(\widehat{{\mathbf{x}}}_t)$
$\tilde\epsilon_t \leftarrow \widehat{\epsilon}_t + (1+s)(\epsilon_t - \widehat{\epsilon}_t)$Eq. <ref>
$\mathbf{x}_{t-1} \sim \mathcal{N}(\frac{1}{\sqrt{\bar{\alpha}_t}}(\mathbf{x}_t-\frac{1-\alpha_t}{\sqrt{1 - \bar{\alpha}_t}}\tilde{\epsilon}_t), \Sigma_t)$Eq. <ref>
return $\mathbf{x}_0$

Sampling guidance for diffusion models.
Recent works have proposed diffusion guidance methods based on class labels to generate images with higher quality [7, 16]. Classifier guidance (CG) [7] is an approach that uses a trained classifier that guides the reverse process toward a specific class distribution. As an alternative strategy without an additional classifier, Ho and Salimans [16] propose classifier-free guidance (CFG). Due to its simplicity of implementation and effectiveness, the guidance has been used in various high-quality diffusion models [29, 31, 39, 41, 23, 33]. Adopting the concepts of the guidance methods above, Nichol et al. [23] propose text-to-image generation with CLIP [28] guidance and CFG. However, these approaches have limitations since they do not apply to unlabeled datasets and require additional training procedures [7, 16].

Self-attention in generative models.
A self-attention mechanism is the key ingredient of Transformer-based models [40]. Notably, it has become a de facto method in natural language processing tasks [40] for its expressive power and capability to encode global context, which has inspired many works to incorporate this mechanism into computer vision [8, 18, 45, 46]. Among those, Jiang et al. [18] and Zhang et al. [45, 46] attempt to bring self-attention into generative adversarial networks (GANs) for better image quality. Following this, diffusion models have also brought self-attention into their model architectures. DDPM [14] initiates this trend by introducing a self-attention layer at a coarse resolution of the U-Net [32]. Inspired by this work, Dhariwal and Nichol [7] measure the boost performance according to the varying number of self-attention heads and resolutions. Concurrently, DiT [27] even accomplishes high performance leveraging Transformer-based backbones.

Internal representations of diffusion models.
Motivated by the success of diffusion models in generation tasks, some works have tried to utilize the representations of diffusion models to do other tasks, such as semantic segmentation. Brempong et al. [2] show that the denoising pre-training boosts the performance on semantic segmentation, and Baranchuk et al. [1] propose a label-efficient strategy for semantic segmentation using the U-Net [32] representations of diffusion models. While specific tasks such as text-driven manipulation using cross-attention has been researched concurrently [12], these are inherently different from improving and self-conditioning general diffusion models in a condition-free way leveraging the internal self-attention maps, which is mainly discussed in this paper.

Classifier-free guidance

Self-attention guidance

Comparison of classifier-free guidance [16] and self-attention guidance (SAG). Compared to classifier-free guidance that uses external class information, SAG extracts the internal information with the self-attention to guide the models, making it training- and condition-free.

§ PRELIMINARIES

Denoising diffusion probabilistic models.
DDPM [14] is a model that recovers an image from white noise through an iterative denoising process. Formally, given an image $\mathbf{x}_0$ and a variance schedule $\beta_t$ at a timestep $t\in \{T, T-1, \ldots, 1\}$, we can obtain $\mathbf{x}_t$ through the forward process which is defined as a Markovian process. Similarly, given a trained diffusion model parameterized by $\epsilon_\theta(\mathbf{x}_t, t)$ and $\Sigma_\theta(\mathbf{x}_t, t)$, we can define the reverse process. In this case, we set $\Sigma_\theta(\mathbf{x}_t, t)$ to $\sigma_t^2 = \beta_t$ [14] although it is possible to predict the variance [24, 7]. Specifically, given $\mathbf{x}_T \sim \mathcal{N}(0, \mathbf{I})$ and $\Sigma_\theta(\mathbf{x}_t, t)$, DDPM samples $\mathbf{x}_{T-1}, \mathbf{x}_{T-2}, \ldots, \mathbf{x}_{0}$ by computing:

\begin{equation}
\mathbf{x}_{t-1} = \frac{1}{\sqrt{\bar{\alpha}_t}}(\mathbf{x}_t-\frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_{\theta}(\mathbf{x}_t, t))+\sigma_t\mathbf{z},
\label{eq:xt-1_pred}
\end{equation}

where $\alpha_t = 1 - \beta_t$, $\bar{\alpha}_t=\prod_{i=1}^{t}\alpha_i$, $\mathbf{z} \sim \mathcal{N}(0, \mathbf{I})$, and $\epsilon_\theta$ denotes a neural network parameterized by $\theta$. Note that for simplicity, we define $\epsilon_{\theta}(\mathbf{x}_t) := \epsilon_{\theta}(\mathbf{x}_t, t)$ for the rest of the paper. Using the reparameterization trick, we can obtain $\hat{\mathbf{x}}_0$,  an intermediate reconstruction of $\mathbf{x}_0$ at a timestep $t$, using the following equation:

\begin{equation}
\hat{\mathbf{x}}_0 = (\mathbf{x}_t-\sqrt{1-\bar{\alpha}_t}\epsilon_{\theta}(\mathbf{x}_t, t))/\sqrt{\bar{\alpha}_t}.
\label{eq:x0_pred}\vspace{-5pt}
\end{equation}

Classifier guidance and classifier-free guidance.
To bring the capability of GANs' trading diversity for fidelity to diffusion models, Dhariwal and Nichol [7] propose the classifier guidance that uses an additional classifier $p(c|\mathbf x_t)$, where $c$ is a class label. The guidance can be formulated as the following with a guidance scale $s>0$:

\begin{equation}
\tilde{\epsilon}(\mathbf x_t,c) = \epsilon_\theta(\mathbf x_t,c)-s\sigma_t\nabla_{\mathbf x_t}\log p(c|\mathbf x_t),
\label{eq:classifier-guidance}
\end{equation}

where $\epsilon_\theta(\mathbf x_t,c)$ is a conditional diffusion model, and $\tilde{\epsilon}(\mathbf x_t,c)$ is the guided output by the classifier. On the other hand, Ho and Salimans [16] present a classifier-free guidance strategy that achieves the similar effect as classifier guidance without the use of an additional classifier:

\begin{align}
\tilde{\epsilon}(\mathbf x_t,c)&={\epsilon_\theta}(\mathbf x_t,c)+s ({\epsilon_\theta}(\mathbf x_t,c)-\epsilon_\theta(\mathbf x_t))\\&=\epsilon_\theta(\mathbf x_t)+(1+s)(\epsilon_\theta(\mathbf x_t,c)-\epsilon_\theta(\mathbf x_t)).
\label{eq:classifier-free}
\end{align}

However, this method still demands hard-earned labels and confines the application to conditional diffusion models that use external conditions such as class or text [23, 31, 33] conditions. Moreover, it requires additional training detail that occasionally zero-outs the class embedding in the training phase [16], thus imposing extra complexity.

Self-attention in diffusion models.

Several works of diffusion models use the U-Net structure [32] with self-attention [40] at one or some of the intermediate layers [14, 7]. Moreover, very recently, diffusion models using Transformers [40] as the backbone has also been proposed [27]. Specifically, for the height $H$ and width $W$, given any feature map $X_t\in\mathbb{R}^{(HW) \times C}$ at a timestep $t$, the $N$-head self-attention is defined as:

\begin{equation}
Q^{h}_t = X_tW^{h}_Q,\quad K^{h}_t = X_tW^{h}_K,
\end{equation}

\begin{equation}
A^{h}_t = \textrm{softmax}(Q^{h}_t (K^{h}_t)^T/\sqrt{d}),
\label{eq:attn-map}
\end{equation}

where $W^{h}_Q, W^{h}_K \in \mathbb{R}^{C \times d}$ for $h=0,1,...,N-1$. Each $A^{h}_t$ is then right multiplied by $V^{h}_t = X_tW^{h}_V$, where $W^{h}_V \in \mathbb{R}^{C \times d}$.

§ GENERALIZING DIFFUSION GUIDANCE

Although classifier guidance and classifier-free guidance have largely contributed to the conditional generation of diffusion models [7, 16, 23], they depend on external inputs. In this work, we broaden our perspective by extending them to handle both cases: with or without external inputs. We also show how CFG [16] can be integrated into our framework at the end of this section.

At a given timestep $t$, the entire input for a diffusion model comprises a generalized condition represented as $\mathbf{h}_t$, and a perturbed sample $\bar{\mathbf{x}}_t$ that lacks $\mathbf{h}_t$. More specifically, the condition $\mathbf{h}_t$ can encompass internal information within $\mathbf{x}_t$, an external condition, or both. With this definition, the resulting guidance is formulated through the utilization of an imaginary regressor, $p_{\textrm{im}}(\mathbf{h}_t|\bar{\mathbf x}_t)$, which is assumed to predict $\mathbf{h}_t$ given $\bar{\mathbf{x}}_t$. Modifying guidance proposed in prior works [38, 7], we present:

\begin{equation}
\tilde{\epsilon}(\bar{\mathbf x}_t, \mathbf{h}_t) = \epsilon_\theta(\bar{\mathbf x}_t, \mathbf{h}_t)-s\sigma_t\nabla_{\bar{\mathbf x}_t}\log p_{\textrm{im}}(\mathbf{h}_t|\bar{\mathbf x}_t),
\label{eq:general-guidance}
\end{equation}

where we slightly abuse the notation for ${\epsilon_\theta}(\bar{\mathbf x}_t,\mathbf{h}_t)$ since we assume that the inputs are simply aggregated to match the original whole input. Intuitively, the gradient of the regressor, $\nabla_{\bar{\mathbf x}_t}\log p_{\textrm{im}}(\mathbf{h}_t|\bar{\mathbf x}_t)$, guides generated samples to be more suitable with that information.

With Bayes' rule, $p_{\textrm{im}}(\mathbf{h}|\bar{\mathbf x}_t)\propto p(\bar{\mathbf{x}}_t|\mathbf{h})/p(\bar{\mathbf{x}}_t)$, and the score of an imaginary regressor $p_{\textrm{im}}(\mathbf{h}_t|\bar{\mathbf x}_t)$ is derived:

\begin{equation}
\nabla_{\bar{\mathbf x}_t}\log p_{\textrm{im}}(\mathbf{h}_t|\bar{\mathbf x}_t) = {-\frac{1}{\sigma_t} }(\epsilon^*(\bar{\mathbf x}_t,\mathbf{h}_t)-\epsilon^*(\bar{\mathbf x}_t)),
\end{equation}

where $\epsilon^*$ denotes the true score of the regressor. Eventually, this term is plugged into Eq. <ref> and produces:

\begin{align}
\tilde{\epsilon}(\bar{\mathbf x}_t, \mathbf{h}_t) &= \epsilon_\theta(\bar{\mathbf x}_t, \mathbf{h}_t)-s\sigma_t\nabla_{\bar{\mathbf x}_t}\log p_{\textrm{im}}(\mathbf{h}_t|\bar{\mathbf x}_t)\\
&={\epsilon_\theta}(\bar{\mathbf x}_t,\mathbf{h}_t)+s ({\epsilon_\theta}(\bar{\mathbf x}_t,\mathbf{h}_t)-\epsilon_\theta(\bar{\mathbf x}_t))
\label{eq:gg-prev}\\
&={\epsilon_\theta}(\bar{\mathbf x}_t)+(1+s) ({\epsilon_\theta}(\bar{\mathbf x}_t,\mathbf{h}_t)-\epsilon_\theta(\bar{\mathbf x}_t)).
\label{eq:gg}
\end{align}

Note that Eq. <ref> induces a constraint that $\bar{\mathbf x}_t$ be in-manifold that the diffusion model $\epsilon_\theta$ defines. Also note that CFG [16] is a special case of Eq. <ref> where $\bar{\mathbf x}_t = {\mathbf x}_t$, ${\mathbf h}_t = c$, and the imaginary regressor $p_{\textrm{im}}(\mathbf{h}_t|\bar{\mathbf x}_t)$ is reduced into the implicit classifier in [16].

Benefiting from this formulation, we can also define diffusion guidance on unconditional models, which have a sole noised image $\mathbf x_t$ as an input and no external label [16, 7], by making it self-conditional on visual information within the intermediate samples of the reverse process. In this light, we present comprehensive discussions on how to find appropriate ${\mathbf h}_t$ for unconditional models and according $\bar{\mathbf x}_t$, and subsequently propose guidance in Section <ref>.

Comparison of blur guidance with self-attention guidance (SAG) under a large guidance scale. Given an extreme guidance scale ($s=5.0$), blur guidance generates relatively noisy images (top) compared to those generated with SAG (bottom).

High-frequency masks (top) and the self-attention masks (bottom) of the finally generated images. Note that the frequency masks are calculated after the generation process, while the self-attention masks are accumulated during the entire reverse process.

Dataset     Input     # of steps     SAG     FID ($\downarrow$)     sFID ($\downarrow$)     IS ($\uparrow$)     Precision ($\uparrow$)     Recall ($\uparrow$)

2*ImageNet 256$\times$256     2*Uncond.     2*250        26.21     6.35     39.70     0.61     0.63

20.08     5.77     45.56     0.68     0.59

2*ImageNet 256$\times$256     2*Cond.     2*250        10.94     6.02     100.98     0.69     0.63

9.41     5.28     104.79     0.70     0.62

2*LSUN Cat 256$\times$256     2*Uncond.     2*250        7.03     8.24     -     0.60     0.53

6.87     8.21     -     0.60     0.50

2*LSUN Horse 256$\times$256     2*Uncond.     2*250        3.45     7.55     -     0.68     0.56

3.43     7.51     -     0.68     0.55

50K results of self-attention guidance on ADM [7] pre-trained on 256$\times$256 images. The best values are in bold.

§ UTILIZING THE SELF-ATTENTION MAP TO IMPROVE SAMPLE QUALITY

The derivation presented in Section <ref> implies that by extracting salient information ${\mathbf h}_t$ contained in ${\mathbf x}_t$, it is possible to provide guidance to the reverse process of diffusion models. Inspired by this implication, we propose an innovative guidance technique, Self-Attention Guidance (SAG), which effectively capture the salient information for reverse process while mitigating the risk of out-of-distribution issues of $\bar{\mathbf x}_t$ in pre-trained diffusion models. We first explain blur guidance, which is a primitive form of SAG, in Section <ref>, and then we introduce SAG in Section <ref>.

§.§ Blur Guidance for Diffusion Models

Gaussian blur is a linear filtering technique that involves convolving an input signal $\hat{\mathbf{x}}_0$ with a Gaussian filter $G_\sigma$ to produce an output $\tilde{\mathbf{x}}_0$. Formally, $\tilde{\mathbf{x}}_0 = \hat{\mathbf{x}}_0 \ast G_\sigma$, where $\ast$ represents the convolution operator. As the standard deviation $\sigma$ increases, Gaussian blur reduces the fine-scale details within the input signals and smooths them towards constant [30], resulting in locally indistinguishable ones.

It is evident that there is an information imbalance between $\tilde{\mathbf{x}}_0$ and $\hat{\mathbf{x}}_0$, where $\hat{\mathbf{x}}_0$ contains more fine-scale information. Based on this fundamental insight, we introduce a specialized version of Eq. <ref>, which we refer to as blur guidance in this paper. In essence, blur guidance intentionally excludes the information from intermediate reconstructions (Eq. <ref>) during the diffusion process, using this information to guide our predictions towards enhancing the relevance of the images to the information. In detail, blur guidance makes the original prediction deviate more from the prediction of the blurred input. Moreover, we note that Gaussian blur has a benign property in that it prevents the resulting signals from deviating significantly from the original manifold with a moderate $\sigma$, , blurring occurs naturally in images [17, 20, 30], which makes Gaussian blur particularly suitable for its application to pre-trained diffusion models. These models generally include latent diffusion models [31], given that the spatial latents also contains low-level information such as local structures [9, 25].

Schedule     Objective     Input     SAG     FID ($\downarrow$)

2*cosine     2*$L_{\textrm{hybrid}}$     2*Uncond.         19.2

A 50K result of self-attention guidance on IDDPM [24] pre-trained on ImageNet 64$\times$64.

A result of human evaluation on self-attention guidance with pairs sampled from Stable Diffusion [31].

To be specific, we first blur $\hat{\mathbf{x}}_0$ at Eq. <ref> with a Gaussian filter $G_\sigma$. Subsequently, we diffuse it again with the noise $\epsilon_{\theta}(\mathbf{x}_t)$ to produce $\tilde{\mathbf{x}}_t$. It is important to note that by doing this, we bypass the side effect of blur that reduces Gaussian noise, making the guidance depend on the intermediate content rather than the random noise. For brevity and to incorporate diffusion models in latent space [31], we let $\mathbf{x}_t$ represent either noised images or the spatial latents [9, 31].

The blur guidance is then incorporated into Eq. <ref> by setting $\bar{\mathbf x}_t=\tilde{\mathbf x}_t$ and ${\mathbf h}_t={\mathbf{x}}_t - {\tilde{\mathbf{x}}_t}$. In practice, the joint input $(\tilde{\mathbf x}_t,\mathbf{h}_t)$ is simply computed as the summation $\mathbf{x}_t=\tilde{\mathbf x}_t+\mathbf{h}_t$. The term ${\mathbf{x}}_t - {\tilde{\mathbf{x}}_t}$ retains the information present before the blurring process, thus guiding the diffusion process to be more appropriate to the removed salient information in the original input. Our results, as shown in Table <ref> “Global", demonstrate the effect of blur guidance in improving the baseline in terms of the quality metrics.

Despite its benefit with moderate guidance scales, the application of blur guidance on existing models with large guidance scales ($s>5.0$) produces noisy results, as shown in the top row of Fig. <ref>. We assume that this is because global blur introduces structural ambiguity across entire regions. This makes it difficult to align the prediction of the degraded input with that of the original, contributing to the noisy outcome accumulated over $t$. This issue highlights the need for a more adaptive approach that can capture finer and more relevant information during the reverse process than the global blurring.

Results from ADM [7].

Results from ADM [7] with our method (SAG).

Uncurated samples from ADM [7] without and with our method (SAG). Both results are sampled from unconditional ADM pre-trained on ImageNet 256$\times$256 [6], and share the same random seed. The samples guided by SAG typically show fewer artifacts, benefiting from the self-conditioning of the internal conditions.

Results from Stable Diffusion [31].

Results from Stable Diffusion [31] with our method (SAG).

Uncurated samples from Stable Diffusion [31] without and with our method (SAG). Both results are sampled from Stable Diffusion, and share the same random seed. The prompt is set to an empty prompt with a single space (“ ”).

§.§ Self-Attention Guidance for Diffusion Models

The self-attention mechanism [8, 40] has been shown to be a key component of diffusion models [7, 14]. This mechanism, which is implemented in the backbones of diffusion models, allows the model to attend to salient parts of the input during the generative process [18, 45, 46, 12]. A particular example of the information capture is illustrated in Fig. <ref>, which shows that the region of the self-attention masks from ADM [7] overlaps with the high-frequency details that diffusion models ought to elaborate on and that are one of essential factors of image generation [4, 42, 43] and human perception [5]. See the appendix for more examples and analyses.

Building upon this intuition, we propose SAG, which leverages the self-attention maps of diffusion models. In essence, we adversarially blur self-attended patches of ${\mathbf x}_t$, , conceal the information of patches that diffusion models attend to. We then use the concealed information to guide diffusion models. In addition, it can be shown that $\bar{\mathbf{x}}_t$ of self-attention guidance contains intact regions of ${\mathbf{x}}_t$, which means that it does not cause the structural ambiguity of the inputs and thus mitigates the problem of global blur.

To obtain the aggregated self-attention map from Eq. <ref>, we conduct global average pooling (GAP) to aggregate the stacked self-attention maps $A^{S}_t \in \mathbb{R}^{N\times(HW)\times(HW)}$ to the dimension $ \mathbb{R}^{HW}$, followed by reshaping to $\mathbb{R}^{H \times W}$ and subsequent nearest-neighbor upsampling to match the resolution of $\mathbf{x}_t$:

\begin{equation}
{A}_t = \textrm{Upsample}(\textrm{Reshape}(\textrm{GAP}(A^{S}_t))).
\label{eq:real-attn-map}
\end{equation}

Generalizing blur guidance, given a masking threshold $\psi$, which is practically set to the mean value of $A_t$, SAG blurs only the masked patches of $\mathbf{x}_t$ according to the self-attention map and is formulated as follows:

\begin{align}
&M_t = \mathbbm{1}(A_t > \psi),\\
&{\widehat{\mathbf{x}}}_t=(1-M_t)\odot {\mathbf{x}}_t+M_t\odot \tilde{\mathbf{x}}_t,~\label{eq:sel-blur}\\
&\tilde\epsilon(\mathbf{x}_t) = \epsilon_\theta(\widehat{{\mathbf{x}}}_t) + (1+s) (\epsilon_\theta(\mathbf{x}_t) - \epsilon_\theta({\widehat{\mathbf{x}}}_t)),~\label{eq:sag-ori}
\end{align}

where $\odot$ denotes the Hadamard product and $\tilde{\mathbf{x}}_t$ is obtained in the same manner as that in Sec. <ref>. Note that Eq. <ref> is also a special case of Eq. <ref> where ${\mathbf{h}}_t = M_t\odot {\mathbf{x}}_t - M_t\odot \tilde{\mathbf{x}}_t$, $\bar{\mathbf x}_t={\widehat{\mathbf x}}_t$, and the joint input undergoes the simple summation as in Sec. <ref>. Unlike blur guidance, ${\widehat{\mathbf{x}}}_t$ explicitly contains intact patches of ${\mathbf{x}}_t$, preventing the output $\epsilon_\theta(\widehat{{\mathbf{x}}}_t)$ from deviating too far from the original with even a large scale (Fig. <ref>) as well as effectively concealing the information critical for the reverse process in an adversarial manner.

“A girl
showing a
smiling face."

“A living area with a television and a table."

“A Scottish Fold playing with a ball."

“ "

Text-to-image results of Stable Diffusion [31], where the top row is sampled with only CFG and the bottom row with CFG and SAG. SAG helps the model generate a high-quality image that is more self-conditioned and has fewer artifacts even with an empty prompt (4th column), exhibiting independence from external information.

§ EXPERIMENTS

§.§ Experimental Settings

For the experiments, we use two servers with 8 NVIDIA GeForce RTX 3090 GPUs each to sample from. We build upon the pre-trained models of ADM [7], IDDPM [24], Stable Diffusion [31], and DiT [27]. We take all the weights for our experiments from their publicly available repositories, and use the same evaluation metrics as [7], including FID [13], sFID [22], IS [34], and Improved Precision and Recall [19].

§.§ Experimental Results

Unconditional generation with SAG.
We show the effectiveness of SAG on the unconditional models, which demonstrates our condition-free property that CG and CFG do not possess. We use unconditionally pre-trained ADM [7] and IDDPM [24] for this experiment, and evaluate 50k samples for the metrics.

We evaluate pre-trained ADM [7] on ImageNet [6] 256$\times$256, LSUN Cat [44], and LSUN Horse [44]. As shown in Table <ref>, we observe that SAG consistently improves the FID, sFID and IS of unconditional, while it lowers the recall. As explained in recent studies [7, 16], we suspect for the lower recall that there also exists a trade-off relationship between sample fidelity and diversity. Nevertheless, the qualitative improvement is made due to the self-conditioning of our method, as we can see the comparison of unselected samples in Fig. <ref>.

Subsequently, we include the results of the unconditional model of IDDPM [24] equipped with the proposed method, which is trained on ImageNet at resolution 64$\times$64. The result is in Table <ref>, which also shows an improvement in terms of FID by applying SAG.

CG [7]     SAG     FID ($\downarrow$)     sFID ($\downarrow$)     Precision ($\uparrow$)     Recall ($\uparrow$)

5.91     5.09     0.70     0.65

2.97     5.09     0.78     0.59

5.11     4.09     0.72     0.65

2.58     4.35     0.79     0.59

Compatibility of SAG with CG [7]. The results are from ADM trained on ImageNet 128$\times$128.

Model     CFG [16]     SAG     FID ($\downarrow$)

2*DiT-XL/2 [27]             2.27

Compatibility of SAG with CFG [16]. The results are from DiT-XL/2 trained on ImageNet 256$\times$256.

Masking strategy     FID ($\downarrow$)     IS ($\uparrow$)

Baseline     5.98     141.72

Global (blur guidance in Sec. <ref>)     5.82     143.15

High-frequency     5.74     148.87

Random     5.68     148.99

Square     5.68     146.50

Self-attention (SAG in Sec. <ref>)     5.47     151.12

DINO [3]-attention     5.63     146.18

Ablation study of the masking strategy. The results are from ADM trained on ImageNet 128$\times$128.

Conditional generation with SAG.
While our method is effective on unconditional models, Eq. <ref> implies the condition-agnosticity, meaning that SAG can also be applied to conditional models. To evaluate SAG on conditional models, we perform an experiment on ADM [7] that is conditionally trained on ImageNet 256x256. The results are presented in Table <ref>, which demonstrates a similar effect on conditional models as on unconditional ones.

Stable Diffusion with SAG.

We compare our results with Stable Diffusion [31] using human evaluation (see the appendix for the protocol) on 500 pairs of images with and without SAG. We use empty prompt for Stable Diffusion and the same random seed for each pair. The results show samples with SAG are more visually favorable or realistic to human. See Fig. <ref> and Fig. <ref>.

In addition, we also broaden the range to text-to-image (T2I) generation by utilizing Stable Diffusion and fusing CFG with SAG, although SAG is not intended for the specific T2I task. Notably, in Fig. <ref>, the image samples generated from the model with SAG show higher quality and fewer artifacts due to the self-conditioning effect of SAG. Interestingly, even with an empty prompt (Fig. <ref> and Fig. <ref> 4th column), we observe an obviously improved quality. This corroborates the independence of SAG with an external condition.

Inception Score

Precision & Recall

Ablation study of the guidance scale. The x-axis is guidance scale, and the dotted line denotes the performance of the baseline, , the model without SAG. The results are from ADM trained on ImageNet 128$\times$128.

§.§ Ablation Studies and Analyses

Orthogonality with CG and CFG.
Although designing SAG for unguided models, we can combine it with CG [7] that utilizes external conditions to further improve the performance. To this end, we test four cases to use the guidance, with or without CG and SAG. The metrics are evaluated on 50k samples generated by the ImageNet 128$\times$128 model [7]. As shown in Table <ref>, we observe additional improvements in FID and precision when using both of them, yet in terms of sFID only giving SAG is the best. This implies that SAG have an orthogonal component with and can be used simultaneously with traditional guidance.

Moreover, CFG [16] is another method of providing class-conditional guidance. However, it requires diffusion models to be trained in a specific manner. Therefore, we use DiT-XL/2 [27], a Transformer [40]-based model which has self-attention layers as well. The 50k results are presented in Table <ref>. They show that samples guided by CFG also benefit from the self-conditioning effect of SAG. Note that the combined effect of SAG and CFG is also corroborated by text-to-image samples in Fig. <ref>.

Masking strategy.
We test various masking strategies to verify the effectiveness of our self-attention masking with 10k samples on ADM [7]. Those strategies replace the masking function of SAG at each timestep. For a fair comparison, we mask 40% of the pixels of the image for the other masking schemes, which is the equivalent portion of the masked area when the threshold of the self-attention masking is $1.0$. The results are in Table <ref>. We find that the self-attention masking strategy outperforms other masking strategies. Notably, applying global masking, blur guidance, shows the worst performance among the schemes, which validates the motivation for SAG. In addition, we applied the high-frequency mask using FFT on $\hat{\mathbf{x}}_0$, as well as the self-attention mask of DINO [3]. However, these methods demonstrated worse performance than ours in terms of FID and IS metrics. Therefore, this result indicates that the self-attention masking is a sufficiently effective method.

2*$\sigma$     Baseline     2*$\sigma=1$     2*$\sigma=3$     2*$\sigma=9$     2*$\sigma=27$     Avg. pixel

($\sigma\rightarrow 0$)                     ($\sigma\rightarrow \infty$)

FID ($\downarrow$)    5.98     5.58     5.47     5.70     5.80     5.84

IS  ($\uparrow$)     141.72     145.85     151.12     148.70     147.83     147.52

Ablation study of the sigma ($\sigma$) of Gaussian blur. The results are from ADM trained on ImageNet 128$\times$128.

Guidance scale.
We also evaluate the performance changes as the guidance scale changes with 10k samples on ADM [7]. As shown in Fig. <ref>, we test the scales of $-0.1$, $0.1$, $0.2$, $0.3$, and $0.4$ to ADM and obtain the best FID, sFID, and Inception Score at the guidance scale $s=0.1$. The precision metric shows the best results when the guidance scale is $s=0.3$. We also find out that applying self-attention guidance with a negative scale ($s=-0.1$) or a scale that is too large ($s\ge{}0.4$) harms the sample quality.

Gaussian blur.
We examine the effect of changes on $\sigma$ using 10k samples, testing for $\sigma \in \{1, 3, 9, 27\}$ and the extreme cases. As $\sigma\rightarrow\infty$, the filter gradually blurs the signal content, reducing every pixel to the average value. Conversely, if $\sigma\rightarrow0$, the signal remains unchanged. The results are in Table <ref>. SAG is robust against linear changes in $\sigma$, while there still exists an optimal $\sigma$ that yields the best performance. Note that the impact also depends on the input resolution; for instance, a higher input resolution generally requires a larger $\sigma$.

No guidance     SAG     CFG [16]

GPU memory     12,167MB     12,209MB     12,218MB

Run-time     108.27s     186.60s     190.27s

Computational cost.

Computational cost.

We report the computational cost of SAG and CFG [16] in Table <ref>. The memory and time consumption of SAG is almost the same as CFG, which indicates that the overhead due to the operations in SAG (, blurring and masking) is negligible. However, due to the additional step, the cost is high compared to no guidance.

§ CONCLUSION

We present a novel and general formulation of guidance that utilizes internal information within diffusion models for synthesizing high-quality images. Our method, self-attention guidance, is condition- and training-free, and can be applied to various diffusion models, such as ADM, IDDPM, Stable Diffusion, and DiT, improving their quality and reducing the artifacts via self-conditioning. The results of our experiments demonstrate the effectiveness of our proposed method and the orthogonality of self-attention guidance to existing guidance methods. With the findings and the generalization of guidance, we believe that our work opens new avenues for further research in the field of denoising diffusion models and their guidance.

§ ACKNOWLEDGEMENTS

This research was supported by the MSIT, Korea (IITP-2022-2020-0-01819, ICT Creative Consilience program), and National Research Foundation of Korea (NRF-2021R1C1C1006897). This research was also supported by Samsung Mobile eXperience Business department.

[1]
Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem
Label-efficient semantic segmentation with diffusion models.
In ICLR, 2021.

[2]
Emmanuel Asiedu Brempong, Simon Kornblith, Ting Chen, Niki Parmar, Matthias
Minderer, and Mohammad Norouzi.
Denoising pretraining for semantic segmentation.
In CVPR, pages 4175–4186, 2022.

[3]
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal,
Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers.
In ICCV, pages 9650–9660, 2021.

[4]
Yuanqi Chen, Ge Li, Cece Jin, Shan Liu, and Thomas Li.
Ssd-gan: measuring the realness in the spatial and spectral domains.
In AAAI, volume 35, pages 1105–1112, 2021.

[5]
Kanjar De and V Masilamani.
Image sharpness measure for blurred images in frequency domain.
Procedia Engineering, 64:149–158, 2013.

[6]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In CVPR, pages 248–255. Ieee, 2009.

[7]
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat gans on image synthesis.
NeurIPS, 34:8780–8794, 2021.

[8]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at
In ICLR, 2020.

[9]
Patrick Esser, Robin Rombach, and Bjorn Ommer.
Taming transformers for high-resolution image synthesis.
In CVPR, pages 12873–12883, 2021.

[10]
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan,
and Baining Guo.
Vector quantized diffusion model for text-to-image synthesis.
In CVPR, pages 10696–10706, 2022.

[11]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.
Mask r-cnn.
In ICCV, pages 2961–2969, 2017.

[12]
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel
Prompt-to-prompt image editing with cross attention control.
arXiv preprint arXiv:2208.01626, 2022.

[13]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
Gans trained by a two time-scale update rule converge to a local nash
NeurIPS, 30, 2017.

[14]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
NeurIPS, 33:6840–6851, 2020.

[15]
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi,
and Tim Salimans.
Cascaded diffusion models for high fidelity image generation.
Journal of Machine Learning Research, 23(47):1–33, 2022.

[16]
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
In NeurIPS 2021 Workshop on Deep Generative Models and
Downstream Applications, 2021.

[17]
Emiel Hoogeboom and Tim Salimans.
Blurring diffusion models.
arXiv preprint arXiv:2209.05557, 2022.

[18]
Yifan Jiang, Shiyu Chang, and Zhangyang Wang.
Transgan: Two pure transformers can make one strong gan, and that can
scale up.
NeurIPS, 34:14745–14758, 2021.

[19]
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and
Timo Aila.
Improved precision and recall metric for assessing generative models.
NeurIPS, 32, 2019.

[20]
Sangyun Lee, Hyungjin Chung, Jaehyeon Kim, and Jong Chul Ye.
Progressive deblurring of diffusion models for coarse-to-fine image
In NeurIPS 2022 Workshop on Score-Based Methods, 2022.

[21]
Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim
On distillation of guided diffusion models.
arXiv preprint arXiv:2210.03142, 2022.

[22]
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter Battaglia.
Generating images with sparse representations.
In ICML, pages 7958–7968. PMLR, 2021.

[23]
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin,
Bob McGrew, Ilya Sutskever, and Mark Chen.
Glide: Towards photorealistic image generation and editing with
text-guided diffusion models.
arXiv preprint arXiv:2112.10741, 2021.

[24]
Alexander Quinn Nichol and Prafulla Dhariwal.
Improved denoising diffusion probabilistic models.
In ICML, pages 8162–8171. PMLR, 2021.

[25]
Bjorn Ommer and Joachim M Buhmann.
Learning the compositional nature of visual objects.
In CVPR, pages 1–8. IEEE, 2007.

[26]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
Pytorch: An imperative style, high-performance deep learning library.
NeurIPS, 32, 2019.

[27]
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
arXiv preprint arXiv:2212.09748, 2022.

[28]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
In ICML, pages 8748–8763. PMLR, 2021.

[29]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
arXiv preprint arXiv:2204.06125, 2022.

[30]
Severi Rissanen, Markus Heinonen, and Arno Solin.
Generative modelling with inverse heat dissipation.
arXiv preprint arXiv:2206.13397, 2022.

[31]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
High-resolution image synthesis with latent diffusion models.
In CVPR, pages 10684–10695, 2022.

[32]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In International Conference on Medical image computing and
computer-assisted intervention, pages 234–241. Springer, 2015.

[33]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily
Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi,
Rapha Gontijo Lopes, et al.
Photorealistic text-to-image diffusion models with deep language
arXiv preprint arXiv:2205.11487, 2022.

[34]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and
Xi Chen.
Improved techniques for training gans.
NeurIPS, 29, 2016.

[35]
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In ICML, pages 2256–2265. PMLR, 2015.

[36]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
In ICLR, 2021.

[37]
Yang Song and Stefano Ermon.
Generative modeling by estimating gradients of the data distribution.
NeurIPS, 32, 2019.

[38]
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano
Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential
In ICLR, 2020.

[39]
Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen.
Improved vector quantized diffusion models.
arXiv preprint arXiv:2205.16007, 2022.

[40]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
NeurIPS, 30, 2017.

[41]
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and
Fang Wen.
Pretraining is all you need for image-to-image translation.
arXiv preprint arXiv:2205.12952, 2022.

[42]
Yiwen Xu, Maurice Pagnucco, and Yang Song.
Dhg-gan: Diverse image outpainting via decoupled high frequency
In ACCV, pages 3977–3993, 2022.

[43]
Mengping Yang, Zhe Wang, Ziqiu Chi, and Wenyi Feng.
Wavegan: Frequency-aware gan for high-fidelity few-shot image
In ECCV, pages 1–17. Springer, 2022.

[44]
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong
Lsun: Construction of a large-scale image dataset using deep learning
with humans in the loop.
arXiv preprint arXiv:1506.03365, 2015.

[45]
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang,
and Baining Guo.
Styleswin: Transformer-based gan for high-resolution image
In CVPR, pages 11304–11314, 2022.

[46]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.
Self-attention generative adversarial networks.
In ICML, pages 7354–7363. PMLR, 2019.


# TeliNet: Classifying CT scan images for COVID-19 diagnosis

Mohammad Nayeem Teli
Department of Computer Science
University of Maryland
College Park, MD 20742
<EMAIL_ADDRESS>
###### Abstract

COVID-19 has led to hundreds of millions of cases and millions of deaths
worldwide since its onset. The fight against this pandemic is on-going on
multiple fronts. While vaccinations are picking up speed, there are still
billions of unvaccinated people. In this fight against the virus, diagnosis of
the disease and isolation of the patients to prevent any spread play a huge
role. Machine Learning approaches have assisted in the diagnosis of COVID-19
cases by analyzing chest X-rays and CT-scan images of patients. To push
algorithm development and research in this direction of radiological
diagnosis, a challenge to classify CT-scan series was organized in conjunction
with ICCV, 2021. In this research we present a simple and shallow
Convolutional Neural Network based approach, TeliNet, to classify these CT-
scan images of COVID-19 patients presented as part of this competition. Our
results outperform the F1 ‘macro’ score of the competition benchmark and
VGGNet approaches. Our proposed solution is also more lightweight in
comparison to the other methods.

##  1 Introduction

On March 11, 2020, the World Health Organization(WHO) declared COVID-19,
caused by a novel coronavirus, a global pandemic [5]. Soon after that the
global lock downs and restrictions on travel brought life to a stand still as
we know it. Since then there have been over 208 Million cases with over 4.3
Million deaths caused by this virus worldwide, as of the middle of August,
2021. In an effort to combat this virus our main defenses were isolation,
quarantining, face masks and frequent washing of hands. Some of these
approaches continue till date as we transition into in-person work and other
activities.

In addition to these simple measures, frequent testing and tracing back helps
identify and isolate spreaders and super spreaders of the virus. The real game
changer in preventing hospitalizations and deaths in this yet to be over fight
against this virus have been the vaccines. However, with so many cases, and
slower rate of vaccinations the researchers have relied on many types of
diagnostic techniques and tests to identify a COVID-19 positive patient.

There are three common types of tests that are carried out to identify a COVID
positive patient. First type of testing, Reverse-Transcriptase Polymerase
Chain Reaction (RT-PCR) testing, identifies the presence of actual viral
genetic material by taking a nasal pharyngeal swab. The second type of testing
is antigen testing. It detects the presence of one of the outer proteins of
the viral shell or envelope. The third type of testing is to detect the
presence of antibodies inside a person that identify the presence of the
virus. The RT-PCR testing is the most popular and more accurate than antigen
testing. However, it is more sensitive than antigen testing and needs to be
done in a laboratory setting. As a result RT-PCR testing has a turnaround time
of several days.

In addition to these tests it is a common practice to supplement diagnostic
tests for detection and accurate prognosis of different diseases by applying
machine learning approaches [24, 1, 6, 8, 20, 4, 19, 21]. The most commonly
used imaging diagnostic tools include, X-ray imaging, Computed Tomography
(CT), ultrasound imaging and Magnetic Resonance Imaging (MRI).

Since the beginning of the Coronavirus pandemic, many machine learning
approaches have been applied to help clinicians detect and diagnose COVID-19.
Many of these approaches specifically use images obtained from Chest
Radiographs (CXR) and chest Computed Tomography(CT) [26]. In order to further
push Machine Learning research for better analysis and classification of
COVID-19 diagnostic image data, a workshop is organized in conjunction with
International Conference on Computer Vision (ICCV) ’21 conference. In this
workshop a competition is organized to classify CT-scan image series into
covid and non-covid classes [18]. This paper presents our work on this data
set in response to this COVID-19 CT scan series classification challenge.

We introduce TeliNet, a very simple and lightweight but effective
Convolutional Neural Network (CNN) based approach to classify chest CT-scan
series images of patients as COVID-19 positive or not. We compare our results
with VGGNet-16 [29] and the benchmark classification approach presented by
Dimitrios et al. [18] on the same data set. The results indicate that our
TeliNet method performs significantly better than both these approaches. Our
F1 ‘macro’ score on the validation data set is 0.81 in comparison to the
benchmark score of 0.70 and a VGGNet-16 score of 0.72. Our method is also
significantly faster than VGGNet-16 when run on the same hardware. The main
contributions of this paper include, 1) TeliNet, a lightweight and simple but
very effective CNN based approach to classify CT-scan series, and 2) low
computational requirements. Our approach was run on a MacBook Pro with 2.3 Ghz
dual core i5 processor with 8 GB of RAM. This makes it easily accessible to
users who may not have access to high-end computational resources.

In the next few sections we present the related work, followed by the data set
used in this research. Next, we discuss the methodology, including the
architectural details of TeliNet and VGGNet. Finally, we present results
followed by the conclusions.

##  2 Related work

Computer vision has been a very useful technique in assisting clinicians with
diagnosis. Due to the advancement of pattern recognition and classification
algorithms, computer aided diagnostic tools have greatly supplemented the
ability to detect different diseases [24]. In order to classify the images
obtained using X-rays, MRI and Computer Tomography, Convolution Neural
Networks (CNN) have been the most popular tool [33]. Different types of CNN
architectures have been used for image diagnosis since 2012, such as, AlexNet
[22] [16], VGGNet [29], ResNet [11], InceptionNet [30], etc.

Bressem et al. [3] studied and compared 16 different CNN architectures on two
datasets, CheXpert and COVID-19 image data collection. Bressem et al.
concluded that it is not necessary to use complex architectures and that
simpler CNN’s perform great in classification on both data sets. Sahlol [27]
combined Inception CNN as a feature extractor with Fractional-order Calculus -
Marine Predators Algorithm (FO-MPA) to detect COVID-19 in images using the
COVID-19 dataset available on Kaggle. Jia et al. [15] used a modified
MobileNet and modified ResNet architectures to classify COVID-19 and non-
COVID-19 CT scan images data set [31]. The results indicate almost perfect
accuracy on that data set. Ibrahim et al. [12] compared four different
architectures to classify chest X-ray and CT images to detect COVID-19,
pneumonia and lung cancer using a single model. Many other Convolutional
Neural Network based architectures [25, 2, 9, 14, 23, 10] have been similarly
proposed since the onset of COVID-19 to help diagnose this disease and assist
clinicians.

##  3 Data set

The data set used for the this research is part of “AI-enabled Medical Images
Workshop and COVID-19 Diagnosis Competition (MIA-COV19D)”, organized in
conjunction with the International Conference on Computer Vision (ICCV) 2021
in Montreal, Canada, October 11- 17, 2021. The data is a part of a chest CT
scan database manually annotated for COVID-19 / nonCOVID-19 diagnosis. The
data set consists of 5000 CT scan series’ with each series consisting of
50-700 2D-CT slices. Some of the representative slices of covid and non-covid
CT scan slices are presented in Figure 1.

Figure 1: Samples of 2D CT scan slices. The top row represents covid slices
and the bottom row, non-covid slices.

Each of these slices is a $512\times 512$ pixels 2D image. There are three
different subsets that are provided, training, validation and test. The
training set has 335672 images belonging to 2 classes and the validation data
set has 75534 images belonging to 2 classes. The test set consists of 450
folders each representing a series containing a total of 609057 2D slices. For
a detailed description of this data a curious reader is encouraged to read the
benchmark paper by Dimitrios et al. [18]. This paper describes the competition
data set and the benchmark approach applied on it in great detail.

##  4 Methods

In this section we present our CNN architecture, TeliNet, and compare its
results with VGGNet-16 and the benchmark results. Both, TeliNet and VGGNet-16
approaches were run on the same hardware. The following sections describe our
experiments using the two architectures and their details. However, we begin
with data pre-processing, followed by TeliNet and VGGNet-16 architectures’
description.

###  4.1 Preprocessing

In order to better organize the data for our experiments we use data
generators. These data generator inputs are applied to both TeliNet as well as
VGGNet architectures. For the benchmark approach results we rely on the
experiments and the results provided in the benchmark paper by Dimitrios et
al.

Data generators require that the data is organized in folders representing
each class. Since we have two main classes, COVID-19 and non-COVID-19 the data
is stored in two folders named as such. At the top level we have two main
folders train and validation. Each of these two folders contain covid and non-
covid folders representing the two classes for binary classification. All
images belonging to their specific class were stored in their respective
folders. A separate test folder contains eight different subsets. Each of
those subsets contains images for predictions with 450 folders, each
representing a CT-scan series of test slices for which labels were not
provided.

###  4.2 TeliNet

It is a four-layer shallow Convolutional Neural Network. It consists of 2D
convolutions, batch normalizations, LeakyReLU activation functions and max
pooling layers. This architecture is represented in Figure 2 with its
corresponding summary in Figure 3.

Figure 2: TeliNet architecture. Figure 3: Description of the TeliNet
architecture with Convolution layers and the tunable parameters.

As seen, it is a very shallow network with less than 8.5 Million trainable
parameters. That is much smaller than over 134 Million trainable parameters in
VGGNet-16. The input images are resized from the original $512\times 512$
pixels to $256\times 256$ images and re-scaled to values between 0 and 1. 2D
convolution is carried out on these images using 16 filters of size, $3\times
3$ followed by LeakyReLU activations. After the initial convolutions, there
are 3, 2D convolutions layers, each followed by batch normalization, LeakyReLU
activation functions and maxpooling. The padding is same in all layers, with a
unit stride for each convolution. The max pooling for each layers consists of
$2\times 2$ pixels with a stride of 2.

After the convolution layers, two dense layers, each of 256 units is used,
followed by batch normalization and LeakyReLU activation functions. Unlike the
convolutions layers, we use a 10% dropout for this dense layer. Finally, we
have a Sigmoid unit for binary classification as covid or non-covid.

Batch normalization smoothes the optimization problem by normalizing the
inputs to each layer. This helps fight the covariant shift problem [13] and
also enables a quicker and more stable training [28] of the data set using
TeliNet. It leads to faster convergence and takes advantage of a wider range
of learning rates. To further benefit from these features we use a learning
rate scheduler that reduces the learning rate by a factor of 70% with each
epoch. This values was chosen after a series of pilot experiments. The
advantages of using LeakyReLU as an activation function over other activation
functions are well known [32]. It also facilitates a faster convergence,
complimenting the speed-up due to fewer trainable parameters and batch
normalizations.

###  4.3 VGGNet

Simonyan and Zisserman [29] presented the VGGNet architecture for a large
scale image recognition. This work presented the use of the small $3\times 3$
convolution filters that helped extend the depth of the deep convolution
network to 16-19 network layers. Using this architecture Simonyan et al. were
able to secure both the first and the second positions in the ImageNet [7]
challenge of 2014.

For this research we used VGGNet-16 that consists of 16 layers and over 134
Million trainable parameters. The architectural summary is presented in Figure
4.

Figure 4: Description of the VGGNet-16 architecture with Convolution layers
and the tunable parameters.

Each layer of convolutions involves filters of sizes $3\times 3$, ReLU
activation function,and padding to retain the size with a stride of 1. The
maxpooling in this architecture is a $2\times 2$ filter with a stride of 2.
The simplicity of this approach is the uniformity of the filter sizes, padding
and maxpooling across all layers.

The input to this architecture is a $224\times 224$ pixel image convolved with
64 filters. Same convolutions are repeated in the next layer. This is followed
by maxpooling and two layers of convolutions using 128 filters and maxpooling.
Next, there are three convolutions using 256 filters and maxpooling. It is
followed by 3 convolutions with 512 filters and maxpooling. The next two
layers are fully connected layers with 4096 units each. The final layer is a
Sigmoid unit for the binary classifcation into covid and non-covid images. For
a detailed description of the VGGNet-16 architecture, please refer to the
original work by Simonyan and Zisserman [29].

##  5 Results

We present a comparison of the results of TeliNet, VGGNet-16 and the benchmark
approaches. In the training set there are 153681 covid slices and 181991 non-
covid slices. The validation set contains 35016 covid slices and 40517 non-
covid slices. Since the data sets are not balanced we use an F1 ‘macro’ score
metric to measure the efficacy of our approach just like the one used in the
Benchmark paper [18]. These results are shown in Table 1.

Method | F1 ‘macro’ score
---|---
benchmark [18] | 0.7
VGGNet-16 | 0.72
TeliNet(ours) | 0.81
Table 1: Validation set F1 score results using the three approaches.

The results of TeliNet and VGGNet-16 were obtained on the same hardware,
comprising of a MacBook Pro, with a 2.3 GHz Dual-Core Intel Core i5 processor
and 8 Gb of RAM. Both these approaches were trained using the RMSprop
optimizer, an adaptive learning method, proposed by Geoffrey Hinton, after
empircally comparing with the Adam optimizer [17]. We also tried experiments
using different sized batches ranging from 4 to 128 and finally settled on 32
for the final results due to better performance.

Since it is a binary classification problem, we used binary cross-entropy loss
function to train these networks. Accuracy, precision and recall metrics were
computed. Precision and recall were used to determine the F1 score by taking
their harmonic mean.

Even though our solution has done better than the benchmark method and the
VGGNet-16 architecture, there is a plenty of room for improvement. Our F1
‘macro‘ score on validation data set is 81% in comparison to 72% and 71% on
the same data set by VGGNet-16 and the benchmark approach. On the test set
TeliNet gives an F1 ‘macro’ score of 70.86% in comparison to 67% for the
benchmark score. The labels for the test set were not given to us and
therefore these results are based on the outcomes of the experiments performed
on the test sets by the competition organizing team.

It is important to note that these results are for 3D CT-scan series’. While
it may seem that the reported results on the validation and the test sets have
been reported on 2D slices, it is in fact a 3D series of these slices. Each of
these series’ provides a sequences of 2D slices. Our results were obtained on
these series’ with the label assigned based on the classification outcomes on
these slices. Although our goal was to use a majority voting to determine the
label for a series in these slices, it was seen that majority of these slices
were classified to be belonging to a single class for each series anyway and
so it was a very straightforward decision.

##  6 Conclusion

In this research we introduce a shallow and lightweight but very effective CNN
architecture, TeliNet, for binary classification of CT-scan series’ of
COVID-19 and non-COVID19 slices. We compare our results with VGGNet-16 and the
benchmark results on this difficult data set. The results indicate that our
approach does significantly better than the other two methods on the
validation data set and better than the benchmark approach on the test set.
The results on the test set using the VGGNet have not been evaluated since the
test set labels were not provided to us.

Our contributions include, 1) a very simple CNN architecture trained to do
better than more complex architectures, and 2) a lightweight architecture that
can be run on a personal computer. These advantages make this architecture
accessible to clinicians and researchers who may not always have a high end
hardware.

However, this paper would not be complete without discussing the pitfalls of
the Machine Learning research conducted in the diagnosis using chest X-rays
and CT scan images. Roberts et al. [26] conducted a review of 62 published
papers after a thorough screening of 2,212 studies conducted between Jan. 1,
2020 to October 3, 2020 that presented new Machine Learning models for the
diagnosis or prognosis of COVID-19 from chest X-ray and CT images. In each and
every paper they found either methodological flaws and/or underlying biases
and lack of reproducibility which render these proposed approaches useless of
any potential clinical use. Keepng this in mind it is worthwhile to know that
we have presented not only detailed architectures and the methodologies
applied, we also make our code available on
Github111https://github.com/nayeemmz/TeliNet for easy reproducibility. The
dataset has been validated and the results of the competition have been
uploaded to a Leaderboard for easy evaluation of the approaches and the
validation of the results.

## References

  * [1] An overview of deep learning in medical imaging focusing on mri.  Zeitschrift für Medizinische Physik, 29(2):102–127, 2019.  Special Issue: Deep Learning in Medical Physics.
  * [2] Nur-A Alam, Mominul Ahsan, Md. Abdul Based, Julfikar Haider, and Marcin Kowalski.  Covid-19 detection from chest x-ray images using feature fusion and deep learning.  Sensors, 21(4), 2021.
  * [3] Keno K. Bressem, Lisa C.Adams, ZChristoph Erxleben, Bernd Hamm, Stefan M. Niehues, and Janis L.Vahldiek.  Comparing diferent deep learning architectures for classifcation of chest radiographs.  Scientific Reports (Nature Research), 10(13590), Aug. 12, 2020.
  * [4] Lawrence Carin and Michael J. Pencina.  On Deep Learning for Medical Image Analysis.  JAMA, 320(11):1192–1193, 09 2018.
  * [5] Domenico Cucinotta and Maurizio Vanelli.  WHO declares covid-19 a pandemic.  Acta Biomed, 91(1):157–160, March 19, 2020.
  * [6] Marleen de Bruijne.  Machine learning approaches in medical image analysis: From detection to diagnosis.  Medical Image Analysis, 33:94–97, 2016.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [8] Bradley J. Erickson, Panagiotis Korfiatis, Zeynettin Akkus, and Timothy L. Kline.  Machine learning for medical imaging.  RadioGraphics, 37:505–515, Feb. 17, 2017.
  * [9] Mustafa Ghaderzadeh, Farkhondeh Asadi, Ramezan Jafari, Davood Bashash, Hassan Abolghasemi, and Mehrad Aria.  Deep convolutional neural network–based computer-aided detection system for covid-19 using multiple lung scans: Design and implementation study.  J Med Internet Res, 23(4):e27468, Apr 2021.
  * [10] Ghulam Gilanie, Usama Ijaz Bajwa, Mustansar Mahmood Waraich, Mutyyba Asghar, Rehana Kousar, Adnan Kashif, Rabab Shereen Aslam, Muhammad Mohsin Qasim, and Hamza Rafique.  Coronavirus (covid-19) detection from chest radiology images using convolutional neural networks.  Biomed Signal Process Control, 66(102490), Feb. 10, 2021.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [12] Dina M. Ibrahim, Nada M. Elshennawy, and Amany M. Sarhan.  Deep-chest: Multi-classification deep learning model for diagnosing covid-19, pneumonia, and lung cancer chest diseases.  Computers in Biology and Medicine, 132:104348, 2021.
  * [13] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 448–456. JMLR.org, 2015.
  * [14] Ronaldus Morgan James, Kusrini, and M. Rudyanto Arief.  Classification of x-ray covid-19 image using convolutional neural network.  In 2020 2nd International Conference on Cybernetics and Intelligent System (ICORIS), pages 1–6, 2020.
  * [15] Guangyu Jia, Hak-Keung Lam, and Yujia Xu.  Classification of covid-19 chest x-ray and ct images using a type of dynamic cnn modification method.  Comput Biol Med., 134(104425), April 29, 2021.
  * [16] Sameer Khan and Suet-Peng Yong.  A deep learning architecture for classifying medical images of anatomy object.  In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1661–1668, 2017\.
  * [17] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [18] Dimitrios Kollias, Anastasios Arsenos, Levon Soukissian, and Stefanos Kollias.  Mia-cov19d: Covid-19 detection through 3-d chest ct image analysis.  arXiv preprint arXiv:2106.07524, 2021.
  * [19] Dimitrios Kollias, N Bouas, Y Vlaxos, V Brillakis, M Seferis, Ilianna Kollia, Levon Sukissian, James Wingate, and S Kollias.  Deep transparent prediction through latent representation analysis.  arXiv preprint arXiv:2009.07044, 2020.
  * [20] Dimitrios Kollias, Athanasios Tagaris, Andreas Stafylopatis, Stefanos Kollias, and Georgios Tagaris.  Deep neural architectures for prediction in healthcare.  Complex & Intelligent Systems, 4(2):119–131, 2018.
  * [21] Dimitris Kollias, Y Vlaxos, M Seferis, Ilianna Kollia, Levon Sukissian, James Wingate, and Stefanos D Kollias.  Transparent adaptation in deep medical image diagnosis.  In TAILOR, pages 251–267, 2020.
  * [22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
  * [23] Shuang Liang, Huixiang Liu, Yu Gu, Xiuhua Guo, Hongjun Li, Li Li, Zhiyuan Wu, Mengyang Liu, and Lixin Tao.  Fast automated detection of covid-19 from medical images using convolutional neural networks.  Communications Biology, 4(35), Jan. 4, 2021.
  * [24] Xiaoqing Liu, Kunlun Gao, Bo Liu, Chengwei Pan, Kongming Liang, Lifeng Yan, Jiechao Ma, Fujin He, Shu Zhang, Siyuan Pan, and Yizhou Yu.  Advances in deep learning-based medical image analysis.  In Health Data Science, page 14 pages, June 16, 2021.
  * [25] Caio B. S. Maior, João M. M. Santana, Isis D. Lins, and Márcio J. C. Moura.  Convolutional neural network model based on radiological images to support covid-19 diagnosis: Evaluating database biases.  PLoS ONE, 16(3), March 1, 2021.
  * [26] Michael Roberts, Derek Driggs , Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, Angelica I. Aviles-Rivero, Christian Etmann, Cathal McCague, Lucian Beer, Jonathan R. Weir-McCall , Zhongzhao Teng, Effrossyni Gkrania-Klotsas, AIX-COVNET, James H. F. Rudd, Evis Sala, and Carola-Bibiane Schönlieb.  Common pitfalls and recommendations for using machine learning to detect and prognosticate for covid-19 using chest radiographs and CT scans.  Nature Machine Intelligence, 3:199–217, March 15, 2021.
  * [27] Ahmed T. Sahlol, Dalia Yousri, Ahmed A. Ewees, Mohammed A. A. Al‑qaness, Robertas Damasevicius, and Mohamed Abd Elaziz.  Covid‑19 image classifcation using deep features and fractional‑order marine predators algorithm.  Scientific Reports (Nature Research), 10(1364), Sep. 21, 2020.
  * [28] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [29] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [30] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Computer Vision and Pattern Recognition (CVPR), 2015.
  * [31] Linda Wang, Zhong Qiu Lin, and Alexander Wong.  Covid‑net: a tailored deep convolutional neural network design for detection of covid‑19 cases from chest x‑ray images.  Scientific Reports (Nature Research), 10(19549), Nov. 11, 2020.
  * [32] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.  Empirical evaluation of rectified activations in convolutional network.  CoRR, abs/1505.00853, 2015.
  * [33] Samir S. Yadav and Shivajirao M. Jadhav.  Deep convolutional neural network based medical image classification for disease diagnosis.  Journal of Big Data, 6(113), December 17, 2019.