# A Novel Approach to Industrial Defect Generation through Blended Latent
Diffusion Model with Online Adaptation

Hanxi Li This work was conducted during Hanxi Li’s visit to Zhejiang
University. Jiangxi Normal University, Jiangxi, China  Zhejiang University,
Zhejiang, China  Zhengxun Zhang  Jiangxi Normal University, Jiangxi, China
Hao Chen  Zhejiang University, Zhejiang, China  Lin Wu  Department of Computer
Science, Swansea University, SA1 8EN, United Kingdom  Bo Li  Northwestern
Polytechnical University, Shannxi, China  Deyin Liu  Zhengzhou University,
Henan, China  Mingwen Wang  Jiangxi Normal University, Jiangxi, China

###### Abstract

Effectively addressing the challenge of industrial Anomaly Detection (AD)
necessitates an ample supply of defective samples, a constraint often hindered
by their scarcity in industrial contexts. This paper introduces a novel
algorithm designed to augment defective samples, thereby enhancing AD
performance. The proposed method tailors the blended latent diffusion model
for defect sample generation, employing a diffusion model to generate
defective samples in the latent space. A feature editing process, controlled
by a ”trimap” mask and text prompts, refines the generated samples. The image
generation inference process is structured into three stages: a free diffusion
stage, an editing diffusion stage, and an online decoder adaptation stage.
This sophisticated inference strategy yields high-quality synthetic defective
samples with diverse pattern variations, leading to significantly improved AD
accuracies based on the augmented training set. Specifically, on the widely
recognized MVTec AD dataset, the proposed method elevates the state-of-the-art
(SOTA) performance of AD with augmented data by 1.5%, 1.9%, and 3.1% for AD
metrics AP, IAP, and IAP90, respectively. The implementation code of this work
can be found at the GitHub repository
https://github.com/GrandpaXun242/AdaBLDM.git

###### Index Terms:

Anomaly detection, Blended latent diffusion model, Online adaption.

##  I Introduction

In a practical manufacturing workflow, obtaining defective samples is
considerably more challenging compared to acquiring defect-free ones.
Consequently, the majority of recently proposed algorithms for industrial
defect inspection, such as [1, 2, 3, 4, 5, 6, 7], address the issue as an
Anomaly Detection (AD) problem—an established machine learning challenge [8,
9].

In these industrial AD algorithms, the assumption is made that all anomaly-
free samples, be they complete images or image patches, belong to a single
distribution. Meanwhile, defective samples are identified as “outliers”. The
strategic choice of adopting an Anomaly Detection approach for industrial
defect inspection is evident: the absence of the need for defective samples
during the training stage makes these AD-based algorithms inherently
compatible with practical manufacturing scenarios.

Figure 1: Illustration of three defect generation styles. From top to bottom:
conventional approaches, GAN-based algorithms, and the proposed method.

On the flip side, achieving this compatibility comes at the expense of an
extremely imbalanced data distribution, a characteristic that is deemed
unacceptable in the context of most discriminative algorithms. Consequently,
some researchers suggest the generation of artificial defect patterns using
conventional approaches [1, 3, 5] to facilitate subsequent discriminative
learning processes.

More recently, state-of-the-art (SOTA) algorithms [10, 11, 12] highlight that,
in addition to synthetic defects, incorporating a few real anomalous samples
during the training stage can significantly enhance Anomaly Detection (AD)
accuracy. This approach holds the promise of achieving even higher performance
when superior simulated defective patterns are integrated into training.
Pioneering works [13, 14] have already demonstrated the benefits of generating
these ”lifelike” defects using more sophisticated methods. The first two rows
of Fig. 1 illustrate the existing two styles of defect generation.

In this paper, we propose the integration of the cutting-edge AI generation
algorithm, namely, the Diffusion Probabilistic Model (DPM) [15], into the
realm of defect generation. Specifically, we leverage the Blended Latent
Diffusion Model [16, 17, 18] as our foundational framework. To enhance the
generation of defective samples, we implement three key modifications to the
original Blended Latent Diffusion Model (BLDM). Firstly, we design a novel
“defect trimap” to delineate the target object mask and the defective regions
on the generated image, incorporating it as a new form of controlling
information alongside the language prompts within the stable diffusion model.
Secondly, to ensure the authenticity of the generated samples, a cascade
“editing” stage is introduced for the BLDM, operating in both the latent and
pixel space. Finally, an innovative online adaptation of the image encoder is
proposed to refine the quality of the generated images. Consequently, the
highly customized BLDM algorithm, referred to as AdaBLDM in this paper,
exhibits the ability to generate defective samples with significantly higher
image quality and pattern variations compared to those produced by state-of-
the-art (SOTA) methods. The third row of Fig. 1 provides a brief illustration
of the proposed BLDM-based method. In summary, the contributions of this paper
are threefold, as outlined below.

  * •

To the best of our knowledge, it is the first time that the Diffusion Model
has been adopted for the task of industrial defect generation. The advantages
of the BLDM such as the generation stability, the high image quality, and the
controllable content could all contribute to achieving a better defect
generator compared with the SOTA methods.

  * •

To tailor the BLDM algorithm to the defect generation scenario, we bake BLDM
in the aspects of controlling information, preheating stage, and online
adaptation, respectively. This endeavor leads to the proposed algorithm,
termed AdaBLDM, which achieves notably improved defective samples compared to
synthetic ones generated by existing SOTA methods.

  * •

Given the defective samples generated by the proposed method, we achieved new
SOTA performance on the MVTec AD dataset [19]. In specific, the proposed
AdaBLDM outperforms the best-to-date method DeSTseg [6] by
$\boldsymbol{1.5\%}$, $\boldsymbol{1.9\%}$, $\boldsymbol{3.1\%}$ on the AD
metrics AP, IAP and IAP$\bf 90$, respectively.

The remainder of this paper is organized as follows. In Sec. II, we discussed
the related work in literature. The section III presents the proposed method
in detail. Experimental results and the ablation study are addressed in Sec.
IV. We conclude this paper in Sec. V.

##  II Related work

###  II-A Synthetic Defect Generation

As introduced in Sec. I, due to a scarcity of defective samples, many
researchers [1, 2, 3, 5, 6] are “compelled” to generate synthetic defective
samples for better AD performance. The most straightforward way to mimic
defects is to “paste” anomaly pixels on the normal images as a special case of
data augmentation. Baseline methods like [20], CutPaste [21], and [1] randomly
cut regions from normal images and paste them to the “incorrect” places as
artificial defects. Crop&Paste [22] and PRN[23] crop defect areas from genuine
defective images and paste them onto defect-free images. Those algorithms,
though achieve better AD performance compared with the standard one-class AD
approaches, can not create new defect patterns and thus likely lead to
overfitting problems. To increase the variations of anomalies, DRAEM [3],
DeSTseg [6], MegSeg [5] and ReSynthDetect [24] employ extra datasets combined
with Berlin noise to fabricate defects. However, in these cases, the
distribution of fabricated defects differs from the real one and thus the
performance gain can not be ensured.

Encouraged by the great success of AIGC (AI Generated Contents) methods
including GAN (Generative Adversarial Network) algorithms [25] and Diffusion
Models [26], some efforts have been made to generate more natural-looking and
various simulated defects. In particular, SDGAN [27] employs two generators to
switch the defective and non-defective status for sample images. SDGAN can
generate a high-quality, diverse dataset of steel surface with both defective
and defect-free images, thus higher classification performance is obtained.
Similarly, Defect-GAN [28] can convert non-defective images into defective
ones to enhance the learning process of the defect classifier. However, the
above two GAN-based generation algorithms can not generate well-defined
defective regions. Consequently, they could boost the performance of defect
classification while can hardly lift the defect segmentation accuracy which is
crucial in many real-world applications.

To the best of our knowledge, [13] and [14] are the only two exiting works
that generate both the synthetic defective images as well as the corresponding
pixel-level labels. In specific, the DCDGANc algorithm proposed in [13] design
a novel GAN-based method to mimic real defects and the defects are fused with
defect-free images via an improved Poisson blending algorithm. On the other
hand, [14] introduced DFMGAN, stems from StyleGANv2 [29], to generate defect
images and defect masks by using the proposed defect-aware residual blocks.
Although the above two GAN-based methods can generate high-quality images with
defects, they fall short in generating the defect mask strictly aligned to the
generated defective pixels on the image. The misalignment stems from the
disparate nature of generative methods which is a blessing in image diversity
whereas it could significantly mislead the defect segmentation model in the
scenario of AD. In contrast, in this paper, we propose a novel defect
generation method based on Diffusion Models. We tame this AIGC algorithm to
generate diverse, realistic but accurately-controlled defective image samples
via a series of novel modifications.

###  II-B Diffusion Probabilistic Models for Image Editing

Compared with the more traditional generation methods like Generative
Adversarial Networks (GANs) [30], Variational AutoEncoders (VAEs) [31] and
flow-based algorithms [32], Diffusion Probabilistic Models (DPMs) [15, 33, 34,
35] enjoy a higher quality of the synthetic images and impressive stability of
training. Accordingly, during the past few years, DPM-based approaches, such
as Stable Diffusion (Stability AI) [36], DALL-E2 (OpenAI) [37], and Imagen
(Google) [38], have achieved state-of-the-art performances in the task of
content generation.

In the subdomain of image editing, DPM-based methods also play the most
important role nowadays. In particular, the Latent Diffusion Model [16]
conducts the image generation or editing in the latent space with a much less
feature dimensionality and thus achieve much higher editing efficiency.
RePaint [39] fully leverages a pre-trained DDPM [34] to perform image editing
by sampling given pixels during the reverse diffusion stage. DCFace [40]
employs two feature encoders to control the Diffusion Model, facilitating
facial image editing. [41] proposes the ControlNet to govern existing
Diffusion models to achieve a better-controlled synthesis. [17, 18] enables
image editing by making modifications in either the pixel space or feature
space. [42] introduces sparse text optimization to achieve better image
editing.

Those methods, though achieve SOTA performance in general image editing tasks,
the fuzzily governed synthesis easily fails to generate the image regions
aligned with pixel-level accuracy. This can potentially decrease segmentation
model performance. In this paper, we highly customize the Blended Latent
Diffusion Model (BLDM) [18] algorithm for generating the precisely located
defect regions for industrial defect detection and localization.

Note that this work employs the diffusion model for generating synthetic
datasets, just like the pioneering work [15, 43] do for more general purposes.
Our method is orthogonal to those algorithms that use DPM to predict anomaly
regions [44, 45, 46, 47].

##  III The proposed method

###  III-A Task Definition and Preliminaries

Following the conventional setting of anomaly detection, an anomaly-free (OK)
sample set $\mathcal{X}_{OK}=\\{\mathbf{x}^{i}_{OK}\in\mathbb{R}^{H_{x}\times
W_{x}\times 3}\mid i=1,2,\dots,N_{OK}\\}$ is available for training.
Furthermore, as introduced in Sec. I, a few defective (NG) samples
$\mathcal{X}_{NG}=\\{\mathbf{x}^{i}_{NG}\in\mathbb{R}^{H_{x}\times W_{x}\times
3}\mid i=1,2,\dots,N_{NG}\\}$ and the corresponding anomaly masks
$\mathcal{M}_{NG}=\\{\mathbf{M}^{i}_{NG}\in\mathbb{B}^{H_{x}\times W_{x}}\mid
i=1,2,\dots,N_{NG}\\}$ are given as the “seeds” of defect generation and
usually $N_{NG}\ll N_{OK}$. The task of this paper is to generate the
synthetic training set of defective samples as

$\\{\mathcal{X}_{OK},\mathcal{X}_{NG},\mathcal{M}_{NG}\\}\xrightarrow{\texttt{Defect
Generator}}\\{\mathcal{X}^{\ast}_{NG},\mathcal{M}^{\ast}_{NG}\\},$ (1)

where $\mathcal{X}^{\ast}_{NG}=\\{\mathbf{x}^{\ast
i}_{NG}\in\mathbb{R}^{H_{x}\times W_{x}\times 3}\mid
i=1,2,\dots,N^{\ast}_{NG}\\}$ and $\mathcal{M}^{\ast}_{NG}=\\{\mathbf{M}^{\ast
i}_{NG}\in\mathbb{B}^{H_{x}\times W_{x}}\mid i=1,2,\dots,N^{\ast}_{NG}\\}$
comprise the obtained synthetic sample images and the corresponding label
masks, respectively. Furthermore, the defect generation algorithm usually sets
$N^{\ast}_{NG}\gg N_{NG}$ to ensure higher sample variations.

In this paper, we propose to generate the synthetic samples using a diffusion
model. It is common for diffusion models [15, 33] to approach generation
target as a noise-free variable $\mathbf{x}_{0}$ and define the “forward
diffusion process” as follows

$\begin{split}&q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\text{I})\\\
&q(\mathbf{x}_{1:T}\mid\mathbf{x}_{0})=\prod^{T}_{t=1}q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1}),\end{split}$
(2)

where $\\{\beta_{t}\in(0,1)\\}^{T}_{t=1}$ denotes the predefined variance
schedule and I is a identity matrix. When $T$ is large enough, the diffusion
process could convert a normal image into a random noise $\mathbf{x}_{T}$. On
the contrary, the “reverse diffusion process” is defined as

$\begin{split}&p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t},t),\boldsymbol{\Sigma}_{\theta}(\mathbf{x}_{t},t))\\\
&p_{\theta}(\mathbf{x}_{0:T})=p(\mathbf{x}_{T})\prod^{T}_{t=1}p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}),\end{split}$
(3)

where $\boldsymbol{\mu}_{\theta}(\cdot)$ and
$\boldsymbol{\Sigma}_{\theta}(\cdot)$ are the two unknown functions that are
usually estimated as deep models. In the test phase, given different random
Gaussian noise samples $\mathbf{x}^{i}_{T},i=1,\dots,N_{tst}$, one could
obtain high quality and various samples $\mathbf{x}^{i}_{0},\forall i$
(usually RGB images) from the same domain of the training samples. Akin to the
above diffusion process, we could also generate lifelike anomalous images in
$\mathcal{X}^{\ast}_{NG}$ based on a diffusion model which has been well
learned on the samples sets $\mathcal{X}_{OK}$ and $\mathcal{X}_{NG}$.

###  III-B Latent Diffusion Model and Cross-Modal Prompts

Figure 2: The network structure of the proposed BLDM-based method for
generating defective regions on a image. One can see besides the noise inupt,
the model is governed by a a text prompt and a trimap that indicates the
locations of the object and defect

####  III-B1 Latent diffusion model with controlling information

In this paper, we employ the Latent Diffusion Model (LDM) as the backbone of
our method encouraged by its praiseworthy advantages [16]. As the name
indicating, the forward diffusion and reverse diffusion processes of LDM are
all performed in the latent space. In general, given the input image
$\mathbf{x}\in\mathbb{R}^{H_{x}\times W_{x}\times 3}$, its corresponding
latent feature $\mathbf{z}\in\mathbb{R}^{H_{z}\times W_{z}\times C_{z}}$ is
obtained using the encoder function $\Omega(\cdot)$ as
$\mathbf{z}=\Omega(\mathbf{x})$ while the image can also be recovered via the
decoder function $\Phi(\cdot)$ as $\mathbf{x}=\Phi(\mathbf{z})$. Based on the
latent feature $\mathbf{z}$ and some simplifications 111Readers are referred
to [16] and [33] for details from Eq. 3, the reverse conditional probability
in LDM is defined as

$\begin{split}&p_{\theta}(\mathbf{z}_{t-1}\mid\mathbf{z}_{t},t,\mathcal{C})=\\\
&\mathcal{N}\left(\mathbf{z}_{t-1};\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{z}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\right){\epsilon}_{\theta}(\mathbf{z}_{t},t,\mathcal{C}),\sigma_{t}\text{I}\right),\end{split}$
(4)

where ${\epsilon}_{\theta}(\cdot)$ stands for the $\theta$-parameterized deep
model to be trained; $\mathcal{C}$ denotes the controlling prompts fed into
LDM for stabilizing the generation process [16]; $\alpha_{t}$,
$\bar{\alpha}_{t}$ and $\sigma_{t}$ the parameters which can be calculated in
a deterministic way at each $t$-th step. In our method, we propose to generate
cross-modal controlling features to enhance the quality of the synthetic
defect.

####  III-B2 Linguistic prompts

First, a linguistic prompt is obtained via changing the keywords of a
following template sentence

$y\coloneqq``\\{obj\\},\text{a }\\{obj\\}\text{ with}\text{ a }\\{def\\}",$
(5)

where $obj$ and $def$ denote to the category of the current object (as the
capsule in Fig. 2) and defect type (as the black contamination on the
capsule), respectively. One can see this template design is straightforward
and thus the prompt is always easy to obtain. In this work, the sentence $y$
is then processed by the parameter-frozen text encoder
$\tau(\cdot):\mathbb{R}^{d_{y}}\rightarrow\mathbb{R}^{d_{lang}}$ which is
proposed in the CLIP algorithm [48]. In particular, as shown in Fig. 2, the
linguistic controlling information is fed into the generation model
$\epsilon_{\theta}(\cdot)$ and the generation process is influenced by the
K-Q-V attention operations [16].

####  III-B3 Defect trimaps

Additionally, recall that in practical anomaly detection tasks, the defect-
free samples are usually sufficient in terms of both amount and variation, we
thus propose to generate defective samples under the guidance of the randomly
selected defect-free samples. In a nutshell, a “trimap”
$\Gamma\in\mathbb{R}^{H_{x}\times W_{x}}$ is designed to specify the desired
location of the generated object and the defect, as shown in Fig. 2. To obtain
a proper trimap, we firstly estimate the foreground region _i.e._ , the
interested object, by using the method proposed in [11] and store the
foreground mask into $\mathbf{F}\in\mathbb{B}^{H_{x}\times W_{x}}$ with
$\mathbf{F}(x,y)=1$ indicating a foreground at the coordinate $[x,y]$.
Secondly, a synthetic defect mask is generated based on the genuine anomaly
mask set $\mathcal{M}_{NG}$ as

$\mathcal{M}_{NG}\xrightarrow{\text{rand.}~{}i}\mathbf{M}^{i}_{NG}\xrightarrow{\text{rand.
affine}}\mathbf{M}^{A}_{NG}\xrightarrow{\text{fit}~{}\mathbf{F}}\mathbf{M}^{\ast}_{NG},$
(6)

where
$\mathbf{M}^{i}_{NG},\mathbf{M}^{A}_{NG},\mathbf{M}^{\ast}_{NG}\in\mathbb{B}^{H_{x}\times
W_{x}}$; “rand. $i$” refers to a random index selection; “rand. affine”
denotes a random affine transform while “fit $\mathbf{F}$” stands for
adjusting the location and scale of $\mathbf{M}^{A}_{NG}$ so that the adjusted
mask is entirely inside the foreground region defined by $\mathbf{F}$.
Finally, the value of each pixel on $\Gamma$ are defined as follows

$\Gamma(x,y)=\begin{cases}1\quad&\text{if}~{}\mathbf{M}^{\ast}_{NG}(x,y)=1,\\\
0.5\quad&\text{else if}~{}\mathbf{F}(x,y)=1,\\\
0\quad&\text{else}.\end{cases}$ (7)

As shown in Fig. 2, in the proposed AdaBLDM algorithm, the controlling trimap
$\Gamma$ is firstly embedded by a convolutional block as
$\zeta(\Gamma)\in\mathbb{R}^{H_{z}\times W_{z}\times C_{z}}$. The embedding
feature is then fed into an encoder network $\hat{\epsilon}_{tri}(\cdot)$ that
shares the same structure and initial parameters with the denoising encoder of
$\epsilon_{\theta}(\cdot)$, except that the original “spatial attention”
module is replaced with a self-attention process over the embedded trimap
features. The deep features extracted from different layers of the denoising
encoder are merged with the intermediate features of the denoising decoder of
LDM via element-wise addition (see Fig. 2).

###  III-C Training Scheme

Formally, our cross-modal controlling features is defined as
$\mathcal{C}=\\{\tau(y),\Gamma\\}$ and the learning objective of LDM is
defined as:

$L_{LDM}\coloneqq\mathbb{E}_{\mathbf{z}_{t},\mathcal{C},\epsilon\thicksim\mathcal{N}(0,1),t}\left[\|\epsilon-\epsilon_{\theta}(\mathbf{z}_{t},t,\mathcal{C})\|^{2}_{2}\right].$
(8)

In this work, the denoising model ${\epsilon}_{\theta}(\cdot)$, the embedding
block $\zeta(\cdot)$ of trimap prompts and the encoder $\hat{\epsilon}_{tri}$
for the latent trimap feature are all learned via optimizing the loss function
defined in Eq. 8. However, different training schemes are applied for
different modules. In specific, ${\epsilon}_{\theta}(\cdot)$ is firstly pre-
trained by using the domain dataset (e.g. all the training images of all the
subcategories of MVTec AD [19] for the experiments on this dataset) and then
frozen during model learning for each specific subcategory; the latent feature
encoder $\hat{\epsilon}_{tri}(\cdot)$ is initialized by the corresponding
parameters of the original LDM model [16] and then fine-tuned for each
subcategory based on the real defective samples; the convolutional block
$\zeta(\cdot)$ is only learned based on the genuine anomaly samples with a
random initialization.

Apart from the reverse diffusion model, the text prompts $y$ are obtained by
using the CLIP [48] and BLIP [49] algorithm for sufficiently various
descriptions. The text encoder is copied from CLIP and frozen during training.
As to the image encoder $\Omega(\cdot)$ and decoder $\Phi(\cdot)$, we follow
[16] to inherit the deep models of the VQ-VAE algorithm [50] and fix them
during the whole training stage.

###  III-D Multi-Stage Denoising with Content Editing

Inspired the seminal work, namely Blended Latent Diffusion (BLD) [17], we
propose to perform feature editing in the inference phase. However, due to the
task difference, we extensively modify the editing scheme of BLD for defect
generation. First of all, given a defect-free image $\mathbf{x}_{OK}$ and a
trimap $\Gamma$ defined before, the corresponding defect mask
$\mathbf{M}^{\ast}_{NG}$ is firstly calculated as introduced in Sec. III-B.
Then the multi-stage denoising algorithm with content editing is performed and
the workflow of this algorithm is illustrated in Alg. 1 where $\odot$
indicates element-wise multiplication and $\neg$ stands for the element-wise
“Not” operation on the binary map.

1

Input: Defect-Free Image $\mathbf{x}_{OK}$, Trimap $\Gamma$, Defect Mask
$\mathbf{M}^{\ast}_{NG}$, Language Prompt $y$, Decoder $\Psi(\cdot)$, Encoder
$\Omega(\cdot)$, Range Numbers $T_{1}$, $T_{2}$ and $T_{3}$, Predefined Normal
Distribution $\mathcal{N}(\mu_{z},\sigma^{2}_{z})$

2

Output: Blended Latent Feature $\mathbf{z}^{\ast}_{NG}$

3

4$\mathbf{z}_{OK}\in\mathbb{R}^{H_{z}\times W_{z}\times
C_{z}}\leftarrow\Omega(\mathbf{x}_{OK})$

5 $\mathbf{M}^{z}_{NG}\in\mathbb{B}^{H_{z}\times W_{z}}\leftarrow
Dilate\&DownSample(\mathbf{M}^{\ast}_{NG})$

6 $\mathbf{z}_{t}\leftarrow RandSapmle(\mathcal{N}(\mu_{z},\sigma^{2}_{z}))$

7 $t=T_{1}+T_{2}+T_{3}$

8 /* _Free Diffusion Stage_ */

9 while  _$t >T_{2}+T_{3}$_ do

10

11 $\mathbf{z}_{t-1}\leftarrow Denoise(\mathbf{z}_{t},\Gamma,y)$

12 $t=t-1$

13 end while

14/* _Latent Editing Stage_ */

15 while  _$t >T_{3}$_ do

16

17
$\mathbf{z}_{t}=\mathbf{z}_{t}\odot\mathbf{M}^{z}_{NG}+\mathbf{z}_{OK}\odot\neg\mathbf{M}^{z}_{NG}$

18 $\mathbf{z}_{t-1}\leftarrow Denoise(\mathbf{z}_{t},\Gamma,y)$

19 $t=t-1$

20 end while

21/* _Image Editing Stage_ */

22 while  _$t >=0$_ do

23 $\mathbf{x}_{t}=\Psi(\mathbf{z}_{t})$

24
$\mathbf{z}_{t}\leftarrow\Omega(\mathbf{x}_{t}\odot\mathbf{M}_{NG}+\mathbf{x}_{OK}\odot\neg\mathbf{M}_{NG})$

25 $\mathbf{z}_{t-1}\leftarrow Denoise(\mathbf{z}_{t},\Gamma,y)$

26 $t=t-1$

27 end while

28

29return _$\mathbf{z}^{\ast}_{NG}=\mathbf{z}_{0}$_

Algorithm 1 Multi-Stage Denoising with Editing

As shown in Fig. 2, the pipeline consistes of three stages in the denoising
phase. The first stage is the conventional denosing process without content
editing and takes $T_{1}$ steps. The second stage takes $T_{2}$ steps and the
input feature $\mathbf{z}_{t}$ is merged with the feature of
$\mathbf{x}_{OK}$, under the guidance of the dilated defect mask
$\mathbf{M}^{\ast}_{NG}$. In the third stage, a novel content blending method
proposed in this paper is conducted. In specific, the blending operation is
performed in the pixel space rather than the latent space and the frozen
encoder $\Omega(\cdot)$ and decoder $\Phi(\cdot)$ are used to switch the
information between the two spaces.

Figure 3: The inference scheme of the proposed AdaBLDM algorithm. One can see
that the whole procedure can be mainly divided into $4$ stages, namely the
pure denoising stage without editing; the latent editing stage; the image
editing stage, and the decoder adaptation stage.

###  III-E Online Decoder Adaptation

Recall that the aim of our algorithm is to generate better synthetic defective
images which are actually reconstructed by the decoder $\Phi(\cdot)$, we thus
propose to fine tune the decoder for each generated sample to achieve higher
image quality.

Given the blended latent feature
$\mathbf{z}^{\ast}_{NG}\in\mathbb{R}^{H_{z}\times W_{z}\times C_{z}}$ obtained
in Alg. 1, the demanding defect mask
$\mathbf{M}^{\ast}_{NG}\in\mathbb{B}^{H_{x}\times W_{x}}$ and the source image
$\mathbf{x}_{OK}\in\mathbb{R}^{H_{x}\times W_{x}\times 3}$, one can fine tune
the decoder model $\Phi(\cdot)$ by conducting the online update algorithm
described in Alg. 2

1

Input: Blended Latent Feature $\mathbf{z}^{\ast}_{NG}$, Defect-Free Image
$\mathbf{x}_{OK}$, Maximum Step $T_{ft}$, Conservative Ratio $\lambda_{con}$,
Anomaly Mask $\mathbf{M}^{\ast}_{NG}$, Decoder $\Phi(\cdot)$

2

Output: Refined Image $\mathbf{x}^{\ast}_{NG}$

3

4$\mathbf{x}^{\ast}_{NG}\leftarrow\Phi(\mathbf{z}^{\ast}_{NG}),~{}\bar{\mathbf{M}}^{\ast}_{NG}(i,j)\leftarrow\neg\mathbf{M}^{\ast}_{NG}(i,j)$

5 $t=1$

6 while  _$t <=T_{ft}$_ do

7

8 $\tilde{\mathbf{x}}\leftarrow\Phi(\mathbf{z}^{\ast}_{NG})$

9

10
$L_{i}=\sum_{i,j}^{H_{x}W_{x}}\left(\tilde{\mathbf{x}}(i,j)-\mathbf{x}_{OK}(i,j)\right)^{2}\odot\bar{\mathbf{M}}^{\ast}_{NG}(i,j)$

11

12
$L_{d}=\sum_{i,j}^{H_{x}W_{x}}\left(\tilde{\mathbf{x}}(i,j)-\mathbf{x}^{\ast}_{NG}(i,j)\right)^{2}\odot\mathbf{M}^{\ast}_{NG}(i,j)$

13

14 $\Phi(\cdot)\leftarrow AdamW(L_{i}+\lambda_{con}L_{d},\Phi(\cdot))$

15

16 $t=t+1$

17 end while

18$\mathbf{x}^{\ast}_{NG}\leftarrow\Phi(\mathbf{z}^{\ast}_{NG})$

19

20return _$\mathbf{x}^{\ast}_{NG}$_

Algorithm 2 Online Decoder Adaptation

One can see that the online adaptation strategy encourages the decoder
$\Phi(\cdot)$ generating similar defect-free pixels as $\mathbf{x}_{OK}$ while
outputting the defective pixels as the original decoder suggests. The
conservative ratio $\lambda_{con}$ balance the two requirements. In the
experiments of this work, we prove that the individually updated decoder can
yield better samples and lead to higher AD performance.

###  III-F Implementation Details

In this paper, all the images are resize to $256\times 256$ following the
setting of DeSTseg [6] which is a SOTA anomaly detection algorithm and adopted
by this paper for evaluating the quality of generated samples. In this way, we
set $H_{x}=W_{x}=256$ and $H_{z}=W_{z}=32$. The latent feature channel number
$C_{z}$ is $4$ in this paper for efficient denoising operation. As to the
trimap prompts, the object’s region is estimated by using the foreground
estimation method proposed in [11] and the anomalous mask is obtained based on
the randomly selected $10$ genuine defective samples for each subcategory of
the dataset. The affine transformation matrix mentioned in Sec. III-B3 is
determined by a random rotation angle $\gamma\in[0^{\circ},360^{\circ}]$ and a
random scaling factor $s\in[0.85,1.15]$. For the AD tasks on texture
subcategories, such as a wooden or a fabric surface, all the pixels belong to
the foreground region except the defective ones.

We employ the AdamW optimizer [51] for updating the model parameters, with a
default learning rate of $1\times 10^{-5}$, and default betas are set to $0.9$
and $0.999$, respectively. During training, there is a $10\%$ chance that
empty text prompt is input to the model.

During the inference stage, we adopt the DDIM [34] sampling method with the
normal denoising step $T_{1}=50$, the latent blended step $T_{2}=30$, and
pixel blended step $T_{3}=5$. In the online decoder adaptation stage,
$T_{ft}=200$ and the conservative ratio $\lambda_{con}=100$ to avoid
overfitting to the anomaly-free region. The same optimizer as offline learning
is used for this online adaptation except the learning rate is set to $1\times
10^{-4}$.

##  IV Experiments

###  IV-A Experiment Setting

####  IV-A1 Evaluation Methodology

In this section, we evaluate the proposed method for generating defective
samples practically and extensively. In most AIGC applications, the appearance
quality of the generated image, measured in either quantitative or qualitative
ways, is the most important criterion to evaluate a generation algorithm.
However, in the scenario of defective sample generation, we claim that a more
important measuring standard is the improvement in AD accuracy. Consequently,
a state-of-the-art anomaly detection algorithm, namely DeSTseg [6] is learned
based on the augmented dataset, and the corresponding performance gain (if
any) directly reflects the effectiveness of the synthetic samples. In
addition, the classical Support Vector Machine (SVM) [52] is also employed as
a complement to DeSTseg. Specifically, the SVM model is learned to classify
the pixels into normal and anomalous, and in this way, a vanilla anomaly
detection/localization method is obtained. The combination of the
sophisticated (DeSTseg) and the naive (SVM) AD algorithms increases the
comprehensiveness and thus the reliability of our experiment.

####  IV-A2 Comparing Generation Methods

We compare our defect generation algorithm with the cutting-edge generative
models including DFM [14] and DCDGANc [13] which is built based on StarGAN
[53] or StyleGANv2 [29] algorithms. However, as the original StyelGANv2 in
DCDGANc can easily lead to training crash according to our experiment, in this
work, the StyleGANv2 is replaced by StyleGANv2-Ada [54] to ensure the
stability of training. Furthermore, the results of the original DeSTseg
algorithm, which also generates synthetic defects in the pasting fashion, are
also compared. The training samples of the compared methods are strictly
identical to those of AdaBLDM.

####  IV-A3 Datasets

To achieve a more general study, we perform the comparison on three well-known
datasets, namely MVTEC-AD [19] ($7$ subcategories over $15$), BTAD [55] (full
dataset), and KSDD2 [56] which are introduced below.

  * •

MVTec AD[19] is an open dataset containing ten object categories and five
texture categories commonly seen, with up to eight defect categories for each
object/texture category. All the images are accompanied by pixel-level masks
showing the defect regions. Among the available defect categories, we opted to
utilize defect categories that contain more than $10$ defect images as the
training dataset for our model. Specifically, we randomly selected $10$ defect
images per defect category from the test set and total defect-free images from
the train set for model training and defect data generation. Due to time
constraints, within the MVTec AD dataset, we focused on evaluating our model
using data from $2$ Object categories (Hazelnut and Capsule) and $5$ Texture
categories.

  * •

BTAD [55] comprises $2540$ images from three categories of real-world
industrial products with different body and surface defects. It is usually
considered as a complementary dataset to the MVTec AD when evaluating an AD
algorithm. Consistent with MVTec AD, we similarly opt for the selection of 10
instances of defect data along with defect-free data for the generation and
training of the model.

  * •

KSDD2 [56] with over $3000$ images obtained while addressing a real-world
industrial problem. This dataset comprises only one category, focusing on
surface defects. For model training and data generation, we exclusively
utilize the defect data and defect-free data from the training set.

To the best of our knowledge, the experiment of this paper involves much more
subcategories, both in terms of number and variations, than any others in the
current literature.

####  IV-A4 Evaluation metrics for Anomaly Detection and Localization

  * •

Pixel-AUC is the area under the receiver operating characteristic curve at the
pixel level. It is the most popular AD measuring method while failing to
reflect the real performance difference between algorithms when a serious
class imbalance exists.

  * •

PRO [57] on the contrary, focuses on the anomaly pixels and treats the AD
performance on each anomaly region equally. Consequently, the PRO metric is
more robust to the class imbalance which is a common situation in most AD
benchmarks.

  * •

AP [58] metric as a conventional metric for semantic segmentation, is
frequently adopted in recently proposed AD algorithms. It reflects the anomaly
detection performance from a pixel-level perspective.

  * •

IAP [6] focus on instance recall as a more straightforward metric.

  * •

IAP90 For those applications requiring an extremely high recall, the precision
at recall = k% is also computed and denoted as IAP@k, and k we follow [6] by
setting k = $90$ in our experiments.

####  IV-A5 Evaluation metrics for generation quality

Following the conventional evaluation manner for image generation algorithms,
we here also compare the scores related to the quality of the generated
images. In particular, the Kernel Inception Distance (KID) [59] as well as the
Learned Perceptual Image Patch Similarity (LPIPS) [60] are involved in the
comparison. In simple words, the former criterion reflects the generation’s
authenticity while the latter one indicates the generation’s diversity.

####  IV-A6 Training Settings for the Involved AD Models

In this paper, we propose to evaluate the generated samples by using the
simplest AD method and a sophisticated AD method. In this way, one can obtain
a thorough examination of the generated samples.

As one end of the spectrum, the SVM-based anomaly detection algorithm is
extremely simple and thus can directly reflect the patch-wise quality of the
generated images. In specific, the SVM model based on the RBF kernel [61] is
employed as the patch classifier. The patch features are extracted by using
the PatchCore algorithm [4] and each $768$-D feature is assigned a positive
label if the corresponding patch is normal while negative labels are assigned
to the anomalous ones. In each trial, $100$ images are randomly sampled from
the synthetic defective images and then $5000$ positive patch features and
$5000$ negative patch features are extracted for training the SVM model. The
final performance on each subcategory is calculated by averaging the results
of $10$ random trails with the same setting. Besides, the performance of the
SVM model trained merely on the genuine defective data is also reported. As to
the parameter setting, we by default set the $\gamma$ of the RBF kernel as
$1\times 10^{-4}$ while the regularization parameter $C$ is set to $1$.

As a SOTA anomaly detection method, DeSTseg [6] employs a discriminative model
to detect anomalous regions based on comparing them and their reconstructed
defect-free counterparts. Before training, we first generate $5000$ defective
samples by using AdaBLDM and another $5000$ defective samples by using the
data augmentation method of DeSTseg. We store the defective samples generated
by AdaBLDM in $\mathcal{D}_{gen}$ and store those augmented by DeSTseg in
$\mathcal{D}_{raw}$. The training data is then sampled from the defect-free
samples, $\mathcal{D}_{raw}$ and $\mathcal{D}_{gen}$ in the ratio of $5:4:1$.
Note that according to the methodology of DeSTseg, the genuine defective
samples, which are the seeds of the synthetic data generation, can not be used
for training as the anomaly-free version of them is unknown. For each
subcategory, the DeSTseg is learned for $10000$ iterations and we pick the
DeSTseg model with a UNIQUE iteration number for all the subcategories in one
dataset.

###  IV-B Quantitative Results

####  IV-B1 Results on MVTec AD

We first conduct the comparison of the SVM-based anomaly detection on the
well-known MVTec AD dataset and report the results in Tab. I. From the table,
we can see that the SVM achieves consistently best accuracies based on the
samples generated by AdaBLDM. The genuine samples achieve the second-best
performance while the samples obtained by comparing SOTA generation algorithms
perform worse than the two aforementioned competitors.

TABLE I: The Anomaly Detection and Localization Performance of SVM on the
MVTec AD dataset, Pixel-Auc/PRO/AP/IAP/IAP90, the best and the second-best
numbers are shown in red and blue, respectively.

Category | Genuine |  DFM[14] (AAAI2023) |  DCDGANc-StarGAN[13] (CVPRW2023) |  DCDGANc-StyleGAN[13] (CVPRW2023) |  AdaBLDM (ours)
---|---|---|---|---|---
Hazelnut |  98.07/70.03/47.07/94.85/63.03 | 97.65/66.05/38.7/93.87/57.6 | 82.41/54.24/19.19/71.69/52.3 | 96.79/67.63/41.27/94.18/59.98 |  98.23/71.08/47.94/94.88/66.65
Wood | 95.22/63.53/36.95/91.22/55.33 | 39.91/3.9/2.89/12.99/5.12 |  96.79/67.63/41.27/94.18/59.98 | 89.83/51.97/25.08/81.6/44.96 |  95.96/71.02/48.83/93.05/63.41
Capsule | 92.28/13.91/6.65/84.01/12.31 | 91.72/37.37/9.01/87.36/25.69 |  99.51/71.09/52.11/99.14/61.98 | 94.05/37.13/10.47/89.09/28.15 |  94.17/45.26/13.12/91.77/32.86
Leather | 99.07/62.88/40.86/98.52/51.21 | 45.15/3.05/0.66/29.62/2.25 | 89.83/51.97/25.08/81.6/44.96 |  99.51/71.09/52.11/99.14/61.98 |  99.23/68.5/60.52/99.56/61.49
Grid | 89.02/28.68/2.87/75.29/23.78 | 73.75/3.53/0.78/47.6/3.7 | 93.66/46.29/19.09/83.09/45.71 |  95.82/46.91/12.81/92.49/42.47 |  95.22/35.31/7.55/89.01/32.86
Tile |  93.94/74.81/49.55/87.72/68.71 | 73.54/38.69/7.85/50.4/43.48 |  94.05/37.13/10.47/89.09/28.15 | 82.41/54.24/19.19/71.69/52.3 | 92.74/71.86/47.4/85.9/68.46
Carpet | 95.22/63.53/36.95/91.22/55.33 | 39.91/3.9/2.89/12.99/5.12 |  95.54/44.8/13.12/91.49/40.75 | 89.83/51.97/25.08/81.6/44.96 |  95.96/71.02/48.83/93.05/63.41
Average |  95.06/54.48/32.97/89.49/48.11 | 72.09/24.93/9.76/54.7/22.52 | 93.11/53.31/25.76/87.18/47.69 | 93.23/54.0/26.34/87.62/48.1 |  95.95/57.69/35.51/91.57/52.53

On the other hand, Tab. II illustrates the AD performances of DeSTseg based on
the MVTec AD dataset. As shown in the table, the proposed method ranks the
first for all the $5$ AD metrics. In particular, the performance gains brought
by the synthetic samples generated by AdaBLDM are $1.00\%$, $1.98\%$ $6.68\%$,
$6.35\%$ and $7.93\%$ on Pixel-AUC, PRO, AP, IAP, and IAP90, respectively. The
significant improvements prove the validness of the proposed method.

TABLE II: The Anomaly Detection and Localization Performance of DeSTseg on the
MVTec AD dataset, Pixel-Auc/PRO/AP/IAP/IAP90, the best and the second-best
numbers are shown in red and blue, respectively.

Category |  DESTSEG[6] (CVPR2023) |  DFM[14] (AAAI2023) |  DCDGANc-StarGAN[13] (CVPRW2023) |  DCDGANc-StyleGAN[13] (CVPRW2023) |  AdaBLDM (ours)
---|---|---|---|---|---
Hazelnut |  99.65/97.46/88.22/89.06/75.58 | 99.45/98.01/88.5/90.48/82.32 | 99.53/97.69/87.47/87.78/74.17 | 99.62/98.1/90.31/90.69/80.78 |  99.65/98.65/91.27/91.47/81.96
Wood | 95.72/93.0/73.62/79.77/63.6 | 97.91/97.09/83.78/92.49/85.63 | 97.83/96.99/84.14/92.63/85.05 |  98.34/97.17/84.67/91.98/85.24 |  98.23/97.23/84.59/90.85/82.47
Capsule |  99.21/96.29/57.79/52.7/33.32 | 98.43/95.39/62.89/58.5/31.82 | 98.44/94.78/61.07/56.63/30.49 |  98.85/95.77/62.08/57.28/34.16 | 98.61/96.14/63.39/61.04/39.0
Leather | 99.76/99.06/75.0/74.96/64.92 | 99.81/99.42/80.08/83.28/72.91 |  99.84/99.56/82.53/86.47/76.82 |  99.84/99.5/81.7/83.64/74.32 |  99.84/99.56/82.62/85.89/73.51
Grid |  98.82/94.3/57.74/57.11/35.25 | 98.24/94.38/56.0/57.38/26.34 | 98.21/93.52/57.2/57.52/20.91 | 97.88/91.47/56.4/55.95/13.72 |  98.95/96.21/60.65/62.87/31.11
Tile | 98.7/97.28/93.12/96.38/92.33 | 98.68/96.77/92.87/95.58/88.65 | 99.04/97.54/94.1/96.41/90.38 |  99.1/97.69/93.97/96.27/90.24 |  99.17/97.86/94.57/96.61/90.14
Carpet | 94.86/92.99/69.74/84.79/58.4 | 97.89/96.64/78.99/88.6/74.26 | 98.18/97.08/81.45/89.28/76.53 |  98.55/97.5/82.07/90.34/78.81 |  99.33/98.47/84.87/90.53/80.77
Average | 98.1/95.77/73.6/76.4/60.49 | 98.63/96.81/77.59/80.9/65.99 | 98.72/96.73/78.28/80.96/64.91 |  98.88/96.74/78.74/80.88/65.32 |  99.11/97.73/80.28/82.75/68.42

####  IV-B2 Results on BTAD and KSDD2

Similarly, we compare the proposed method with other generation algorithms in
terms of the AD performance on the other two well-known AD datasets. The
accuracies obtained by DeSTseg are summarized in Tab. III and Tab. IV for BTAD
[55] and KSDD2 [56], respectively. One can again observe the remarkable
superiority of the proposed method in the two tables. The artificial defects
simulated by AdaBLDM lead to the best performances for all the involved AD
metrics, on both two datasets.

TABLE III: The Anomaly Detection and Localization Performance of DeSTseg on
the BTAD dataset, Pixel-Auc/PRO/AP/IAP/IAP90, the best and the second-best
numbers are shown in red and blue, respectively.

Category |  DESTSEG[6] (CVPR2023) |  DFM[14] (AAAI2023) |  DCDGANc-StarGAN[13] (CVPRW2023) |  DCDGANc-StyleGAN[13] (CVPRW2023) |  AdaBLDM (ours)
---|---|---|---|---|---
01 |  96.25/82.5/45.74/39.93/26.95 | 95.97/74.17/37.13/31.71/14.92 | 95.33/71.39/35.37/31.2/11.32 | 95.64/82.34/38.15/36.24/26.78 |  96.84/84.25/43.22/39.04/27.93
02 | 95.73/57.99/60.58/40.82/7.96 |  96.15/64.78/62.73/47.46/12.61 | 94.67/60.65/55.6/49.6/10.63 | 94.39/61.32/64.72/45.09/8.46 |  96.55/65.46/70.16/47.99/13.24
03 |  99.47/98.01/44.29/35.87/17.54 | 98.94/95.45/39.13/34.18/19.22 | 98.92/95.32/37.26/35.82/18.02 | 99.27/97.43/43.5/36.58/16.2 |  99.47/97.97/45.91/36.59/17.95
Average |  97.15/79.5/50.2/38.87/17.48 | 97.02/78.13/46.33/37.78/15.58 | 96.31/75.79/42.74/38.87/13.32 | 96.43/80.36/48.79/39.3/17.15 |  97.62/82.56/53.1/41.21/19.71

TABLE IV: The Anomaly Detection and Localization Performance of DeSTseg on the
KSDD2 dataset, Pixel-Auc/PRO/AP/IAP/IAP90, the best and the second-best
numbers are shown in red and blue, respectively.

Method | Pixel AUC | PRO | AP | IAP | IAP90
---|---|---|---|---|---
DESTSEG[6] (CVPR2023) | 89.45 | 83.42 | 52.3 | 57.59 | 1.29
DFM[14] (AAAI2023) | 86.69 | 85.53 | 47.38 | 54.39 | 1.26
DCDGANc-StarGAN[13] (CVPRW2023) | 93.08 | 85.3 | 46.53 | 49.96 | 1.69
DCDGANc-StyleGAN[13] (CVPRW2023) | 92.38 | 84.83 | 51.72 | 55.65 | 1.56
AdaBLDM (ours) | 96.42 | 92.93 | 70.62 | 73.88 | 7.6

####  IV-B3 Image Quality

Tab. V demonstrates the scores on the image quality of the comparing
generation algorithms. From the table, we can see that the proposed method
achieves the best KID performances for most subcategories (except Grid and
Wood) as well as the overall comparison. This is consistent with the AD
performances reported in Tab. II as the KID metric reflects the “reality” of
the generated images.

TABLE V: The KID$\times 10^{3}$@5k $\downarrow$ and LPIPS@5k $\uparrow$ of the
generated images on MVTec AD, the best and the second-best numbers are shown
in red and blue, respectively

Category |  DFM[14] (AAAI2023) |  DCDGANc-StarGAN[13] (CVPRW2023) |  DCDGANc-StyleGAN[13] (CVPRW2023) |  AdaBLDM (ours)
---|---|---|---|---
Hazelnut | 4.74/0.188 |  1.76/0.16 | 1.97/0.167 |  1.72/0.182
Wood | 48.47/0.172 |  7.91/0.046 | 8.74/0.048 |  2.74/0.076
Capsule | 3.66/0.018 |  1.87/0.026 |  1.08/0.026 | 4.14/0.027
Leather | 61.35/0.199 |  9.23/0.088 | 13.6/0.078 |  5.85/0.032
Grid |  4.00/0.061 |  2.04/0.073 | 7.21/0.087 | 5.38/0.07
Tile |  16.74/0.015 | 23.16/0.036 | 21.5/0.05 |  12.83/0.046
Carpet | 3.64/0.082 | 4.29/0.095 |  3.61/0.088 |  3.55/0.073
Average | 20.37/0.105 |  7.18/0.075 | 8.25/0.078 |  5.17/0.072

In contrast, it is interesting to see that our method lead to poor LPIPS
scores compared with other generation algorithms. Recall that either the
editing process in the denoising stage or the online decoder adaptation
prevents the generated image from being too diverse from the source image and
the desired defect. _In this way, we claim that the poor image diversity might
be a disadvantage of the proposed algorithm from the traditional perspective
while it also implies a better-controlled generation process that can benefit
the following training stage of AD models._

###  IV-C Qualitative Results

To offer readers a more direct impression of the proposed image generation
algorithm, We hereby depict the generated defects by using the comparing
methods in Fig. 4. One can see the proposed method, either with or without the
online adaptation stage, can achieve significantly more lifelike images
compared with other methods. In particular, only our method can generate the
“crack” defects on the hazelnut subcategory. On the contrary, the DFM
algorithm, though can generate synthetic defective samples with good quality
in some scenarios, could fail in generation for some other cases.

Figure 4: The genuine and synthetic defective samples on MVTec AD dataset.
From top to bottom, the samples with genuine defects; the samples generated by
using DCDGANc-StarGAN; the samples generated by using DCDGANc-StyleGAN; the
samples generated with DFM and the samples generated by using our method.

We further illustrate the comparison in a more fine-grained level in Fig. 5.
From the figure we can see that the proposed method is more well-controlled in
generating the defect regions with the specific category and location.

Figure 5: The fine-grained comparison in the generation quality on the
Hazelnut subcategory of MVTec AD. Different kinds of defects are mimicked by
the involved generation algorithms and their performances can be compared on a
more detailed level.

In addition, the generated images with synthetic defects on BTAD and KSDD2 are
also shown in Fig. 6. One can see that besides MVTec AD, the image quality of
AdaBLDM is maintained for the two different datasets.

Figure 6: The generation outputs of AdaBLDM on BTAD and KSDDV2 datasests.

Finally, as a qualitative complement to Tab. II, Fig. 7 shows the heat map of
anomaly predicted by the DeSTseg algorithm. We can see that thanks to the
higher quality of the simulated synthetics, the AdaBLDM-equipped AD algorithm
can output the best heat maps for anomaly detection and localization.

Figure 7: Test results of different DeSTseg models learned based on the
training sets obtained by using different generation methods. All the involved
subcategories in this work ($7$ from MVTec AD, $3$ from BTAD, and the only one
in KSDDV2) are shown here.

###  IV-D Ablation Study

To understand the success of the proposed AdaBLDM algorithm, we utilize
ablation experiments to evaluate the specific contributions of the primary
components which include:

  * •

Pretrain with domain knowledge
As introduced in Sec. III-C, the denoising model ${\epsilon}_{\theta}(\cdot)$
is pretrained on the domain data. If this module is removed, AdaBLDM will rely
on the prior knowledge from the LAION-5B dataset [62], which might not highly
correlate with the domain of industrial AD. Note that the pretraining is
strictly independent to the current subcategory to maintain the fair
comparison.

  * •

Image Editing
Different from the conventional blended latent diffusion model, we propose to
perform the intent editing in both latent space and pixel space (see Alg. 1).
If this module is removed, the editing is only performed in the latent space.

  * •

Online Decoder Adaptation
The online decoder adaptation is introduced in Alg. 2. The absence of this
module results in a fixed image decoder.

The ablation results are summarized in Tab. VI. It is clear that as each
module mentioned above is added to the algorithm, the corresponding AD
performance increases steadily. In particular, the total performance
improvements are $9.78\%$ in AP, $14.46\%$ in IAP, and $13.98\%$ in IAP90,
respectively.

TABLE VI: Ablation Study of AdaBLDM

Modules | Performance
---|---
Pretrain | Image Editing | Adaptation | Pixel auc | Pro | AP | IAP | IAP90
✗ | ✗ | ✗ | 96.51 | 92.18 | 70.50 | 68.29 | 54.44
✓ | ✗ | ✗ | 97.23 | 93.14 | 71.21 | 69.30 | 58.32
✓ | ✓ | ✗ | 99.12 | 97.49 | 80.25 | 82.36 | 67.65
✓ | ✓ | ✓ | 99.11 | 97.73 | 80.28 | 82.75 | 68.42

##  V Conclusion & future work

In this paper, we propose a sophisticated method to generate realistic
defective samples for industrial anomaly detection and localization. Motivated
by the great success of the diffusion model, we highly customize the Blended
Latent Diffusion Model (BLDM) with several novel modifications for this
specific task. The yielded algorithm is termed AdaBLDM and can generate better
synthetic samples compared with the existing methods based on conventional
“cut-and-paste” scheme or Generative Adversarial Networks (GANs). By applying
the AdaBLDM for AD data augmentation, one can observe significant increase on
the AD performances of SOTA algorithms. The AIGC algorithm proposed in this
paper can be directly used as an effective data augmentation method in real-
world applications and also sheds light on the influential factors for solving
defect generalization task. In the future, one could obtain even better
synthetic defective samples by introducing finer controlling signals such as
the language prompts describing the details (shape, size etc.) about the
demanding defects. Another appealing direction of future work is to speed up
the denoising process of the diffusion model so that more defective samples
can be generated given a limited time budget, which is a crucial constraint in
most practical scenarios.

## References

  * [1] H. M. Schlüter, J. Tan, B. Hou, and B. Kainz, “Natural synthetic anomalies for self-supervised anomaly detection and localization,” in _Eur. Conf. Comput. Vis._ , S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds. Cham: Springer Nature Switzerland, 2022, pp. 474–489.
  * [2] T. Defard, A. Setkov, A. Loesch, and R. Audigier, “Padim: a patch distribution modeling framework for anomaly detection and localization,” in _International Conference on Pattern Recognition_. Springer, 2021, pp. 475–489.
  * [3] V. Zavrtanik, M. Kristan, and D. Skočaj, “DrÆm – a discriminatively trained reconstruction embedding for surface anomaly detection,” in _Int. Conf. Comput. Vis._ , 2021, pp. 8310–8319.
  * [4] K. Roth, L. Pemula, J. Zepeda, B. Schölkopf, T. Brox, and P. Gehler, “Towards total recall in industrial anomaly detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , June 2022, pp. 14 318–14 328.
  * [5] M. Yang, P. Wu, and H. Feng, “Memseg: A semi-supervised method for image surface defect detection using differences and commonalities,” _Engineering Applications of Artificial Intelligence_ , vol. 119, p. 105835, 2023.
  * [6] X. Zhang, S. Li, X. Li, P. Huang, J. Shan, and T. Chen, “Destseg: Segmentation guided denoising student-teacher for anomaly detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2023, pp. 3914–3923.
  * [7] W. Liu, H. Chang, B. Ma, S. Shan, and X. Chen, “Diversity-measurable anomaly detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2023, pp. 12 147–12 156.
  * [8] B. Scholkopf, R. Williamson, A. Smola, J. Shawe-Taylor, J. Platt _et al._ , “Support vector method for novelty detection,” _Adv. Neural Inform. Process. Syst._ , vol. 12, no. 3, pp. 582–588, 2000.
  * [9] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” _ACM computing surveys (CSUR)_ , vol. 41, no. 3, pp. 1–58, 2009.
  * [10] C. Huang, Z. Kang, and H. Wu, “A prototype-based neural network for image anomaly detection and localization,” _arXiv preprint arXiv:2310.02576_ , 2023.
  * [11] H. Li, J. Hu, B. Li, H. Chen, Y. Zheng, and C. Shen, “Target before shooting: Accurate anomaly detection and localization under one millisecond via cascade patch retrieval,” _arXiv preprint arXiv:2308.06748_ , 2023.
  * [12] H. Li, J. Wu, H. Chen, M. Wang, and C. Shen, “Efficient anomaly detection with budget annotation using semi-supervised residual transformer,” _arXiv preprint arXiv:2306.03492_ , 2023.
  * [13] J. Wei, F. Shen, C. Lv, Z. Zhang, F. Zhang, and H. Yang, “Diversified and multi-class controllable industrial defect synthesis for data augmentation and transfer,” in _IEEE Conf. Comput. Vis. Pattern Recog. Worksh._ , 2023, pp. 4444–4452.
  * [14] Y. Duan, Y. Hong, L. Niu, and L. Zhang, “Few-shot defect image generation via defect-aware feature manipulation,” in _AAAI_ , vol. 37, no. 1, 2023, pp. 571–578.
  * [15] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _AAAI Conference on Artificial Intelligence_. PMLR, 2015, pp. 2256–2265.
  * [16] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 10 684–10 695.
  * [17] O. Avrahami, D. Lischinski, and O. Fried, “Blended diffusion for text-driven editing of natural images,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 18 208–18 218.
  * [18] O. Avrahami, O. Fried, and D. Lischinski, “Blended latent diffusion,” _ACM Transactions on Graphics_ , vol. 42, no. 4, pp. 1–11, 2023.
  * [19] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, “Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 9592–9600.
  * [20] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [21] C.-L. Li, K. Sohn, J. Yoon, and T. Pfister, “Cutpaste: Self-supervised learning for anomaly detection and localization,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 9664–9674.
  * [22] D. Lin, Y. Cao, W. Zhu, and Y. Li, “Few-shot defect segmentation leveraging abundant defect-free training samples through normal background regularization and crop-and-paste operation,” in _Int. Conf. Multimedia and Expo_ , 2021, pp. 1–6.
  * [23] H. Zhang, Z. Wu, Z. Wang, Z. Chen, and Y.-G. Jiang, “Prototypical residual networks for anomaly detection and localization,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2023, pp. 16 281–16 291.
  * [24] J. Niu, Q. Yu, S. Dong, Z. Wang, K. Dang, and xiaowei ding, “Resynthdetect: A fundus anomaly detection network with reconstruction and synthetic features,” in _Brit. Mach. Vis. Conf._ BMVA, 2023. [Online]. Available: https://papers.bmvc2023.org/0099.pdf
  * [25] W. Xia, Y. Zhang, Y. Yang, J.-H. Xue, B. Zhou, and M.-H. Yang, “Gan inversion: A survey,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 45, no. 3, pp. 3121–3138, 2022.
  * [26] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” _ACM Computing Surveys_ , vol. 56, no. 4, pp. 1–39, 2023.
  * [27] S. Niu, B. Li, X. Wang, and H. Lin, “Defect image sample generation with gan for improving defect recognition,” _IEEE Transactions on Automation Science and Engineering_ , vol. 17, no. 3, pp. 1611–1622, 2020.
  * [28] G. Zhang, K. Cui, T.-Y. Hung, and S. Lu, “Defect-gan: High-fidelity defect synthesis for automated defect inspection,” in _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 2523–2533.
  * [29] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” _IEEE Conf. Comput. Vis. Pattern Recog._ , pp. 8107–8116, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:209202273
  * [30] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [31] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [32] L. Dinh, D. Krueger, and Y. Bengio, “Nice: Non-linear independent components estimation,” _arXiv preprint arXiv:1410.8516_ , 2014.
  * [33] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [34] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” _arXiv preprint arXiv:2010.02502_ , 2020.
  * [35] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” _Advances in neural information processing systems_ , vol. 34, pp. 8780–8794, 2021.
  * [36] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , vol. 1, no. 2, p. 3, 2022.
  * [37] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [38] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [39] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 11 461–11 471.
  * [40] M. Kim, F. Liu, A. Jain, and X. Liu, “Dcface: Synthetic face generation with dual condition diffusion model,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2023, pp. 12 715–12 725.
  * [41] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” _arXiv preprint arXiv:2302.05543_ , 2023.
  * [42] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, “Null-text inversion for editing real images using guided diffusion models,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2023, pp. 6038–6047.
  * [43] W. Wu, Y. Zhao, H. Chen, Y. Gu, R. Zhao, Y. He, H. Zhou, M. Z. Shou, and C. Shen, “Datasetdm: Synthesizing data with perception annotations using diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [44] J. Wyatt, A. Leach, S. M. Schmon, and C. G. Willcocks, “Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 650–656.
  * [45] X. Zhang, N. Li, J. Li, T. Dai, Y. Jiang, and S.-T. Xia, “Unsupervised surface anomaly detection with diffusion probabilistic model,” in _Int. Conf. Comput. Vis._ , 2023, pp. 6782–6791.
  * [46] J. Wolleb, F. Bieder, R. Sandkühler, and P. C. Cattin, “Diffusion models for medical anomaly detection,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2022, pp. 35–45.
  * [47] H. Xu, S. Xu, and W. Yang, “Unsupervised industrial anomaly detection with diffusion models,” _Journal of Visual Communication and Image Representation_ , vol. 97, p. 103983, 2023.
  * [48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in _AAAI Conference on Artificial Intelligence_ , 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:231591445
  * [49] J. Li, D. Li, C. Xiong, and S. C. H. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in _AAAI Conference on Artificial Intelligence_ , 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:246411402
  * [50] A. Van Den Oord, O. Vinyals _et al._ , “Neural discrete representation learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [51] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _Int. Conf. Learn. Represent._ , 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:53592270
  * [52] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine learning_ , vol. 20, pp. 273–297, 1995.
  * [53] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, “Stargan v2: Diverse image synthesis for multiple domains,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 8188–8197.
  * [54] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, “Training generative adversarial networks with limited data,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 12 104–12 114. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/8d30aa96e72440759f74bd2306c1fa3d-Paper.pdf
  * [55] P. Mishra, R. Verk, D. Fornasier, C. Piciarelli, and G. L. Foresti, “Vt-adl: A vision transformer network for image anomaly detection and localization,” in _International Symposium on Industrial Electronics_. IEEE, 2021, pp. 01–06.
  * [56] J. Božič, D. Tabernik, and D. Skočaj, “Mixed supervision for surface-defect detection: From weakly to fully supervised learning,” _Computers in Industry_ , vol. 129, p. 103459, 2021.
  * [57] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, “Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 4183–4192.
  * [58] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets,” _PloS one_ , vol. 10, no. 3, p. e0118432, 2015.
  * [59] M. Bińkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Demystifying mmd gans,” 2018.
  * [60] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 586–595.
  * [61] K.-M. Chung, W.-C. Kao, C.-L. Sun, L.-L. Wang, and C.-J. Lin, “Radius margin bounds for support vector machines with the rbf kernel,” _Neural computation_ , vol. 15, no. 11, pp. 2643–2681, 2003.
  * [62] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman _et al._ , “Laion-5b: An open large-scale dataset for training next generation image-text models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 278–25 294, 2022.