#  AM2-EmoJE:Adaptive Missing-Modality Emotion Recognition in Conversation via
Joint Embedding Learning

###### Abstract

Human emotion can be presented in different modes i.e., audio, video, and
text. However, the contribution of each mode in exhibiting each emotion is not
uniform. Furthermore, the availability of complete mode-specific details may
not always be guaranteed in the test time. In this work, we propose AM2-EmoJE,
a model for Adaptive Missing-Modality Emotion Recognition in Conversation via
Joint Embedding Learning model that is grounded on two-fold contributions:
First, a query adaptive fusion that can automatically learn the relative
importance of its mode-specific representations in a query-specific manner. By
this the model aims to prioritize the mode-invariant spatial (within
utterance) query details of the emotion patterns, while also retaining its
mode-exclusive aspects within the learned multimodal query descriptor. Second
the multimodal joint embedding learning module that explicitly addresses
various missing modality scenarios in test-time. By this, the model learns to
emphasize on the correlated patterns across modalities, which may help align
the cross-attended mode-specific descriptors pairwise within a joint-embedding
space and thereby compensate for missing modalities during inference. By
leveraging the spatio-temporal details at the dialogue level, the proposed
AM2-EmoJE not only demonstrates superior performance (around $2-4\%$
improvement in Weighted-F1 scores) compared to the best-performing state-of-
the-art multimodal methods, by effectively leveraging body language in place
of face expression, it also exhibits an enhanced privacy feature. By reporting
around $2-5\%$ improvement in the weighted-F1 score, the proposed multimodal
joint embedding module facilitates an impressive performance gain in a variety
of missing-modality query scenarios during test time.

Index Terms—  multimodal emotion recognition, joint embedding learning,
adaptive fusion, missing modality

##  1 Introduction

Fig. 1: Proposed AM2-EmoJE framework

Emotion analysis and its dynamic evolution in humans during conversations is
pivotal to various critical tasks ranging from sentiment analysis in social
media to affect-aware human-robot interactions. The problem is particularly
complex due to the involvement of multi-party stakeholders, their mutual
interactions described via different modalities like text, video, and audio,
as well as their body language, which influence the speaker’s emotions and its
spatio-temporal progression expressed in an utterance.

In particular, a speaker’s emotion depends on various intra (e.g. personal
details, behavioral patterns, and habits) and inter (e.g. audience behavior,
their interpreting conducts) personal contexts and other environmental
circumstances. At one end, most multimodal methods assume data completeness,
which may not hold in practice due to privacy, device, or security
constraints. We note that the mode-specific descriptor components exhibiting
an individual’s emotion have some correlation, which may be instrumental to
handling the missing modality information. However, this aspect has not yet
been sufficiently explored by the existing literature. At the other end, the
significance of this variety of heterogeneous mode information may not be
uniform in evaluating emotions for all speakers. For example, a speaker’s
facial expression may have only limited information about their true emotion,
while their body language may still be more semantically rich. Thus the
challenges related to the presence of strong heterogeneity both within and
across multiple modalities and complex interplay and the influence of a
missing modality information toward learning comprehensive multimodal
interaction patterns in a real-life conversation setting are critical yet
under-studied.

Toward these, we propose AM2-EmoJE, an Adaptive Missing-Modality Emotion
Recognition in Conversation via Joint Embedding Learning model that is
grounded on the following contributions.

  1. 1.

Query Adaptive Fusion (QAF) that derives the relative importance of the cross-
attended mode descriptors on the fly to deliver a robust multimodal query
descriptor preserving both instance-specific and category-specific utterance-
level spatial patterns in parallel.

  2. 2.

Multimodal joint embedding that may leverage an auxiliary boolean mask vector
as user input to turn on an effective mode switching mechanism, which proves
instrumental in delivering a competitive decision ability for queries with
incomplete mode information.

  3. 3.

Extensive Evaluation Analysis using publicly available (MELD [1], IEMOCAP [2])
datasets not only demonstrate an impressive classification performance
($2-4\%$ improvement in Weighted-F1 of AM2-EmoJE in complete multimodal test
scenarios, but also exhibit a competitive decision making for queries with
missing modality information.

##  2 Related Work

Multimodal solutions toward motion Recognition in Conversations (ERC) have
recently demonstrated significant performance improvements compared to their
unimodal counterparts. While natural language transcriptions serve as strong
emotion indicators for unimodal ER pipelines, leveraging the advantages of
cross-modal interactions demonstrates significant performance gain[3, 4, 5,
6].

While exhibiting promising performances, traditional multimodal methods assume
data completeness, which may not be a feasible constraint for various practice
issues related to privacy, device, or security constraints and missing
modality in a test environment appears to be challenging to these models. In a
recent work, Ma et al. [7] investigate the behavior of Transformer-based
multimodal frameworks in the presence of modal-incomplete data. Lee et al. [8]
propose a prompt learning framework to address the issues of missing
modalities during training or testing. However, the limitation of
computational resources still persists as a bottleneck to such training-heavy
transformer models.

While a few recent works [9, 10, 11, 12] attempt to address the missing
modality challenge in specific application settings, a straightforward
extension to these preliminary models for the complex task of ERC, may not be
feasible. Evaluating the relative contribution of multiple mode-specific
components describing a speaker’s emotion dynamics on the fly, while
estimating their missing-mode description in parallel, is challenging. Toward
these, we propose AM2-EmoJE that is motivated by two-fold research
objectives:First, a query adaptive fusion that can automatically learn the
relative importance of its mode-specific representations in a query-specific
manner; Second multimodal joint embedding learning framework that learns a
mode-switching mechanism to align the cross-attended mode-specific descriptors
pairwise within a joint-embedding space and thereby allowing the model to
compensate for missing modalities during inference.

##  3 Proposed Method

Problem Definition: Given a multi-party dialogue
$d:=\\{u_{j}\\}_{j}\in\mathcal{D}$ represented as a sequence of utterances
$\\{u_{j}\\}_{j}$, the objective is to evaluate the dominant emotional states
of the speaker as expressed in each utterance $u_{j}$. For brevity, now onward
we will omit the suffix $j$, and an arbitrary utterance $u_{j}$ will be
represented as $u$ unless the suffix is specifically required. Each
$u\in\mathcal{D}$ contains $(u_{v},u_{a},u_{t})$, where $u_{v}$ is the video,
$u_{a}$ is the audio, and $u_{t}$ is the text transcription component of the
utterance. In this work, we propose a continual emotion evaluation model that
can evaluate the series of dialogues by single or multiple speakers in a
conversational environment. Unlike existing multimodal literature, which
primarily relies on the availability of complete mode-specific components
representing a multimode utterance, in this work we allow incomplete query
videos, i.e., some of the mode-specific information related to the query may
be missing. Therefore, the availability information of each mode-specific
query component is passed by a boolean mask vector
$\mathbf{w}:=(w_{v},w_{a},w_{t})$. For example, a query that only has an audio
and transcript will is passed with a mask vector $\mathbf{w}$ with $w_{v}=0$.
In another query environment, if a user is only interested in analyzing the
visual component of a multimode query, then the mask vector $\mathbf{w}$ with
$w_{a}=0$ and $w_{t}=0$ is passed with the query that may enable the system to
perform only the visual analysis.

###  3.1 Mode-Specific Feature Representation

To capture the spatio-temporal evolution of information within each utterance,
the first-level mode-specific feature representation scheme as described
below:

####  3.1.1 Text Representation

To derive a compact descriptor for the text component $u_{t}$ represented as a
sequence of $w$ words, i.e.
$u_{t}=\\{\omega_{1},\omega_{2},...,\omega_{w}\\}$, we employ the pretrained
model SBERT [13] to obtain the fixed language embedding
$\mathbf{f}^{T}\in\mathbb{R}^{w\times d_{t}}$ for the text component $u_{t}$.

####  3.1.2 Video Representation

For the visual component $u_{v}$ of each utterance $u\in d$, FFmpeg is used to
identify $n$ key frames and MTCNN [zhang2016joint] is applied to extract the
aligned faces from each key frame. To deliver a comprehensive analysis of the
subjects’ emotional evolution to appear in an utterance, we analyze their
facial expression in contrast to their body language, each $u_{v}$ is split
into two sequences, which cover the visuals of distinct regions in the
utterance: ‘face sequence’ that contains the sequence of derived frames
containing only the subjects’ face regions; ‘body sequence’ that exclusively
contains the subjects’ body segment in the utterance. YOLOv7 [14] fine-tuned
for human detection, is employed to localize the human body segment in a
frame, To analyze the body language in exclusion, we subtract the face region
from the identified body segment in a frame to design the derived frames in
the ‘body sequence’. Thus, the visual content $u_{v}$ of $u$ is represented in
terms of two equal-sized derived frame sequences:
$\mathbf{v}^{face}=\\{\mathbf{au}_{1},\mathbf{au}_{2},...,\mathbf{au}_{n}\\}$
and
$\mathbf{v}^{body}=\\{\mathbf{b}_{1},\mathbf{b}_{2},...,\mathbf{b}_{n}\\}$,
where each $\mathbf{au}_{j}$ and $\mathbf{b}_{j}$ represent a learned
descriptor describing the $j^{th}$ element in $\mathbf{v}^{face}$ and
$\mathbf{v}^{body}$ respectively. Two identical Bi-LSTM-based sequence
representation modules with a hidden embedding size of $d_{v}$, which take
$\mathbf{v}^{face}$ or $\mathbf{v}^{body}$ as inputs, are employed to obtain
the initial regional descriptors $\mathbf{f}^{face}\in\mathbb{R}^{n\times
d_{v}}$ or $\mathbf{f}^{body}\in\mathbb{R}^{n\times d_{v}}$. These two
independent LSTM networks are then merged via a stacked self-attention layer
that attends to the pair of these network-specific inputs to derive a self-
attended visual descriptor $\mathbf{f}^{v}\in\mathbb{R}^{2n\times d_{v}}$ for
$u_{v}$.

####  3.1.3 Audio Representation

We use the Pathout Fast 2D Spectrogram model (PASST) [15], which is
initialized from a vision transformer trained on ImageNet, to fine-tuned on
10s audio clips from AudioSet. To derive a fine-grained understanding of the
audio signal evolution, the audio component of each utterance is divided into
smaller $e$ overlapped segments and each segment is represented using their
corresponding PASST descriptor $\mathbf{a}_{i}\in\mathbb{R}^{d_{passt}}$ such
that $u_{a}:==\\{\mathbf{a}_{1},\mathbf{a}_{2},...,\mathbf{a}_{e}\\}$. Similar
to the video representation, we use a BiLSTM to derive an overall audio
feature for the utterance $\mathbf{f}^{a}\in\mathbb{R}^{d_{a}}$.

###  3.2 Weighted Multimodal Attention

Toward integrating the mode-specific contents across modalities, we note that
the expression captured by these uni-mode components may not solely reflect
their intra-mode data pattern. They may also represent the influence of
information depicted by other modalities. For example, a speaker’s audio may
reflect certain emotions, which may not be sufficiently supported by their
body language or an utterance transcript may be continually modulated by the
expressions shown by the audience in the previous utterance segments.
Furthermore, the availability of a quality mode-specific representation of an
utterance may also not be guaranteed. For instance, the availability of visual
cues may not be assured in every query scenario. Otherwise, noisy audio may
also deteriorate the overall data quality occasionally.

While existing multimodal emotion analysis models do not allow such incomplete
query information, we design an Adaptive Cross-attended Network (ACN) that
enables a flexible yet robust feature representation technique to capture the
effects of same- and different-modality cues at various levels of details
(including missing modality cues). As observed in Figure 1, each layer of ACN
is administered by multiple cross-modal cues from multiple outer networks into
a single mode-specific inner network. The merging layer of ACN incorporates a
simple yet effective masking mechanism toward facilitating the learning of
several variants of mode combinations, wherein the model is explicitly trained
to control the contribution of each ’outer’ network including a variety of
missing-modality scenarios. The keys and values are generated as
$\mathbf{K}^{l}_{m_{i}}=linear(w_{m_{i}}\mathbf{(}W^{l,K}_{m_{i}})^{T}\mathbf{e}_{m_{i}})$,
$\mathbf{V}^{l}_{m_{i}}=linear(w_{m_{i}}\mathbf{(}W^{l,V}_{m_{i}})^{T}\mathbf{e}_{m_{i}})$,
where $W^{l,K}\in\mathbb{R}^{d_{m_{i}}\times d_{l}}$ and
$W^{l,V}\in\mathbb{R}^{d_{m_{i}}\times d_{l}}$ are key and value weight
matrices in the $l^{th}$ layer of the attention network. The output of each
attention head in the $l^{th}$ layer is then computed as:

$\scriptsize\mathbf{g}^{l}_{m}=\mathbf{g}^{l-1}_{m}+\frac{1}{|\mathcal{M}|}\mathrm{softmax}(linear(\frac{\mathbf{g}^{l-1}_{m}(\sum_{m_{i}\in\mathcal{M}\setminus\\{m\\}}\mathbf{K}^{l-1}_{m_{i}})^{T}}{\sqrt{d_{m_{i}}}}))\mathbf{V}^{l-1}_{m_{i}}$
(1)

Thus, for each modality $m$ in $\mathcal{M}$, the set of all modalities
leveraged in the model, the learned descriptor $\mathbf{g}^{m}$ from multiple
heads are mean-pooled to capture the weighted aggregated details within the
final output of the Central network.

As shown in Figure 1, each mode-specific Central query network for each
$m\in\mathcal{M}$ of ACN thus produces an average pooled cross-attended mode-
specific descriptor $\mathbf{f}^{m}_{ACN}\in\mathbb{R}^{d}$ for the uni-mode
components $t$, $v$, and $a$ for $u$. We will discuss the learning algorithm
later in Section 3.5.

###  3.3 Query Adaptive Fusion

In contrast to the existing fusion techniques, which assume that the mode-
specific contributions in the resulting multimodal descriptor for a query
input should be uniform, in reality, the quality and availability of the
complete mode-specific components may not be feasible in general. Therefore,
we propose a Query Adaptive Fusion (QAF) mechanism that designs a linear
combination of the learned cross-attended mode descriptors $\mathbf{g}^{m}$ to
define a comprehensive multimode utterance-level descriptor as follows:

$\mathcal{A}(u)=\sum_{m\in\mathcal{M}}\left(\frac{w_{m}}{|\mathcal{M}|}\sum_{m_{i}\in\mathcal{M}\setminus
m}\alpha^{m}_{m_{i}}\mathbf{f}^{m_{i}}+(1-\alpha^{m}_{m_{i}})\mathbf{f}^{m_{i}}\right)$
(2)

where $0\leq\alpha^{m}_{m_{i}}\forall m,m_{i}\in\mathcal{M}\leq 1$ are
learnable parameters.

Thus, the proposed fusion function $\mathcal{A}$ provides a flexible
multimodal representation mechanism, by which the resulting multimodal
descriptor $\mathcal{A}(u)$ for an utterance $u$ can retain category-specific
discriminative data patterns, however not completely disregarding the unique
instance-specific data patterns observed in the utterance. Furthermore, given
an utterance $u$, in the absence of its complete mode-specific details (as
provided by the user-specified mask vector $\mathbf{w}$) related to $m$, the
model automatically learns to adopt its learnable parameters
$\\{\alpha^{m}_{m_{i}}\\}_{m_{i}}$ to derive an adjusted multimodal descriptor
$\mathcal{A}(u)$. We adopt the optimization approach of [16] to learn the
interpolation parameters $\alpha^{m}_{m_{i}}$.

###  3.4 Classification

Given a conversational dialogue represented using a sequence of $n$ utterances
$\\{u_{j}\\}^{n}_{j}\in\mathcal{D}$, the emotion of the speaker $s$ is
estimated by leveraging two parallel utterance sequences: Dialogue Context
that describes the sequence $\\{\mathcal{A}(u_{j})\\}_{j}$; Speaker Context
that describes a sub-sequence $\\{\mathcal{A}(u_{{s}_{j}})\\}_{j}$, where the
sub-sequence $\\{u_{{s}_{j}}\\}_{{s_{j}\in[1,n]}}$ is generated from the
dialogue and includes only those utterances, in which $s$ vocally contributes
to the conversation. Two parallel Bi-LSTMs are trained to capture the spatio-
temporal contexts independently of these contexts’ perspectives:
$\mathbf{s}_{l}\in\mathbb{R}^{s}$ representing the Speaker Context and
$\mathbf{d}_{l}\in\mathbb{R}^{s}$ representing the Dialogue Context. These
representations are passed through a classification head to arrive at the
classification decision for the emotion for the utterance $u$.

###  3.5 Training objectives

The learning of AM2-EmoJE includes two independent learning objectives: a
multi-component loss objective ($\mathcal{L}$) that combinedly optimizes the
guided NCE loss ($\mathcal{L}_{ACE}$) and the focal loss ($\mathcal{L}_{fl}$)
for the classifier; a CLIP-like loss function $\mathcal{L}_{joint}$ to learn
the pairwise joint embedding spaces. We discuss them below.

####  3.5.1 Guided NCE

The proposed Adaptive Cross-attended Network (ACN) jointly learns the cross-
attended representations $\mathbf{f}^{t}_{ACN,j}$, $\mathbf{f}^{v}_{ACN,j}$,
and $\mathbf{f}^{a}_{ACN,j}$ with twofold contributions: 1) preserving
instance specific discriminability by identifying more reliable modes on the
fly. We intuitively expect a speaker’s face and body language to display
similar emotions, and a significant deviation from this would require us to
rely more on the other modalities like audio or transcript to accurately
estimate their true emotions. Guided by this observation, we leverage an
aggregated noise contrastive estimation ($\mathcal{L}_{ACE}$) as below:

$\mathcal{L}_{ACE}=\frac{1}{|\mathcal{D}|}\sum_{u_{j}\in\mathcal{D}}\frac{1}{|\mathcal{M}|}\sum_{\begin{subarray}{c}m\neq
m_{i}\\\
m,m_{i}\in\mathcal{M}\end{subarray}}{\mathcal{L}_{\mathrm{NCE}}(\mathbf{f}^{m}_{ACN,j},\mathbf{f}^{m_{i}}_{ACN,j})}\vspace{-2mm}$
(3)

with

$\mathcal{L}_{\mathrm{NCE}}(\mathbf{f}^{m}_{ACN,j},\mathbf{f}^{m_{i}}_{ACN,j})=\\\
\Bigg{[}-\log\Bigg{(}\frac{P(\mathbf{f}^{m}_{ACN,j}|\mathbf{f}^{m_{i}}_{ACN,j})}{P(\mathbf{f}^{m}_{ACN,j}|\mathbf{f}^{m_{i}}_{ACN,j})+\frac{|\mathcal{N}_{j}|}{|\mathcal{N}|}}\Bigg{)}\\\
+\sum_{k\in\mathcal{N}_{j}}\log\Bigg{(}\frac{P(\mathbf{f}^{m}_{ACN,k}|\mathbf{f}^{m_{i}}_{ACN,j})}{P(\mathbf{f}^{body}_{ACN,k}|\mathbf{f}^{face}_{ACN,j})+\frac{|\mathcal{N}_{j}|}{|\mathcal{N}|}}\Bigg{)}-1\Bigg{]}$
(4)

that computes the probability of both features $\mathbf{f}^{m}_{ACN,j}$ and
$\mathbf{f}^{m_{i}}_{ACN,j}$ representing the same instance $u_{j}$ compared
to other elements in a uniformly sampled negative set $\mathcal{N}_{j}$.
$\mathcal{N}$ is the sample batch. The averaged Focal Loss $\mathcal{L}_{fl}$
[17], specifically effective for an imbalanced dataset like ours, is employed
to preserve the category details and a combined loss function
$\mathcal{L}=\mathcal{L}_{ACE}+\mathcal{L}_{fl}$ to jointly learn its mode-
specific Central query networks.

####  3.5.2 Multimodal Joint Embedding

To address the potential missing modality scenarios in a test environment, we
introduce an effective joint latent representation learning model that follows
a similar approach as proposed by Radford et al. [18]: the latent descriptor
for a positive pair (i.e. a pair of mode-specific representatives describing
the same sample using two different modes) are mapped closely, while a
negative pair (i.e. a pair of mode-specific representatives describing the two
different samples using two different modes) samples will be mapped father
from each other in the joint embedding space. Formally, given a minibatch of
$\mathcal{N}$ of samples represented using their cross-attended descriptors in
a two-mode space (i.e.
$\mathcal{N}:=\\{(\mathbf{f}^{m_{k}}_{i},\mathbf{f}^{m_{l}}_{i})\\}_{i,m_{k}\neq
m_{l}}$), we use Jensen-Shannon Divergence, which is a symmetric version of
the KL-Divergence to learn the pair-wise invertible linear mappings
$S_{m_{k}\rightarrow
m_{joint}}:\mathcal{R}^{d_{m_{k}}}\rightarrow\mathcal{R}^{d_{joint}}$,
$\forall m_{k}\in\mathcal{M}$ to a single joint embedding space
$\mathcal{D}_{joint}\subset\mathcal{R}^{d_{joint}}$ and the corresponding loss
function is defined as:

$\mathcal{L}_{m_{k}\rightarrow
m_{joint}}=\frac{-1}{|\mathcal{N}|}\big{[}\sum_{x\in\mathcal{N}}\frac{1}{2}\log(\mathrm{KL}(S_{m_{k}\rightarrow
m_{joint}}(\mathbf{f}^{m_{k}}_{i})||\mathbf{t}^{kl}_{i})\\\
+\log(\mathrm{KL}(S_{m_{l}\rightarrow
m_{joint}}(\mathbf{f}^{m_{l}}_{i})||\mathbf{t}^{kl}_{i})\big{]}$ (5)

where $x:=(\mathbf{f}^{m_{k}}_{i},\mathbf{f}^{m_{l}}_{i})$ and
$\mathbf{t}^{kl}_{i}=\frac{S_{m_{k}\rightarrow
m_{joint}}(\mathbf{f}^{m_{k}}_{i}+\mathbf{f}^{m_{l}}_{i})}{2}$. The model
jointly optimizes a single loss function
$\mathcal{L}_{joint}:=\underset{\begin{subarray}{c}m_{k},m_{l}\in\mathcal{M}\\\
m_{k}\neq m_{l}\end{subarray}}{\sum}\mathcal{L}_{m_{k}\rightarrow m_{joint}}$
to learn the collection of invertible mapping functions:
$\\{S_{m_{k}\rightarrow m_{joint}}\\}_{m,k\in\mathcal{M}}$.

##  4 Experiments

Table 1: Performance Comparison of different methods using the weighted average F1 measure (W-Avg F1) on the MELD dataset with uni modal (T-Text, A-Audio, and V- Video) and multi-modal representation. Due to the imbalanced class distribution of the dataset, the ‘Fear’ and ‘disgust’ classes are represented as the minority classes, the proposed method was also compared against other $5$ majority classes (‘Neutral’, ‘Surprise’, ‘Sadness’, ‘Joy’, and ‘Anger’) in the dataset and the results are reported in column ‘w-avg F1 5 CLS’. More details on emotion-specific comparison are provided in the supplementary material. Method | Mode | w-Avg F1 | w-avg F1
---|---|---|---
5-CLS
MFN [19] | T + A | 0.547 | 0.5732
ICON [20] | T | 0.546 | 0.5718
A | 0.377 | 0.3947
T + A | 0.563 | 0.5897
DialogueRNN [21] | T | 0.551 | 0.5759
A | 0.34 | 0.3542
T + A | 0.57 | 0.5971
ConGCN [22] | T | 0.574 | 0.5969
A | 0.422 | 0.44
T + A | 0.594 | 0.6175
DialogueCRN [23] | T + A | 0.6073 | -
EmoCaps [24] | T + A + V | 0.6400 | -
M2FNet [3] | T + A + V | 0.6785 | -
Cross-Modal Distribution Matching [25] | T + A | 0.571 | -
Transformer Based Cross-modality Fusion [26] | T + A + V | 0.64 | -
Hierarchical Uncertainty [27] | T + A + V | 0.59 | -
Shape of Emotion [28] | T + A + V | 0.63 | -
UniMSE [29] | T + A + V | 0.66 | -
EmotionCLIP [30] | T + A + V | 0.3459 | -
AM2-EmoJE | T + A | 0.6263 | 0.6845
T + V | 0.6196 | 0.6782
A + V | 0.5285 | 0.5825
T + A (JE) | 0.6836 | 0.7051
T + V (JE) | 0.6897 | 0.7106
A + V (JE) | 0.6085 | 0.6572
No Face | 0.6914 | 0.7944
No Body | 0.7089 | 0.8117
| T + A + V | 0.7198 | 0.8205

###  4.1 Datasets

MELD [1] is a multimodal group conversational dataset that sources clips from
the TV Series FRIENDS and consists of the 6 basic emotions. IEMOCAP [24] is a
dual-party conversational dataset acted out by professional actors and
features conversations that are far longer on average than those in MELD.

###  4.2 Results

Tables 1 and 2 report the comparative performance of the proposed method
against several state-of-the-art methods [20, 19, 21, 23, 24, 3, 25, 31, 26,
27, 28, 29] in the MELD and IEMOCAP datasets, respectively. While text appears
as the most reliable unimodal feature, combining multiple features has been
undoubtedly helpful to deliver a competitive $71.98\%$ weighted F1-score
(w-avg F1), which is an improvement of $\approx 4\%$ across all classes, as
compared to the best-performing baseline M2FNet[3]. MELD is an unbalanced
dataset that has ‘fear’ and ‘disgust’ identified as the minority classes, with
less than $50$ samples as representatives. AM2-EmoJE reports $82.05\%$ w-avg
F1, which is a significant $20\%$ improvement in the w-avg F1 score compared
to the best-performing baseline using the five majority classes. Compared to
the existing works, which offer equal emphasis on all modes, adaptively
identifying the more reliable modes to design the relative weight assignments
in a query-specific manner has been proven to be effective for delivering a
more accurate estimate of a speaker’s emotional state. Furthermore, from the
performance reported in the sub-row described using the mode ‘T+A+V(no face)’
(that uses cues from audio, transcript, and visual capturing subjects’ body
regions only), we observe that the proposed AM2-EmoJE still remains
competitive (i.e., reporting $\approx 2\%$ improved w-avg F1 score) against
the best-performing baseline. In fact, as we compare the result reported in
another sub-row described using mode ‘T+A+V(no body)’ (that uses cues from
audio, transcript, and visual capturing subjects’ face regions only), we
observe that AM2-EmoJE’s performance remains nearly equivalent as in two
different multi-mode data scenarios. Finally, to evaluate the contribution of
the proposed multimodal joint learning (Section 3.5.2), in the table we also
report the performance of the proposed AM2-EmoJE in multiple missing modality
scenarios. For example, comparing the performance between the sub-row
described using mode ‘T+A’ (that uses only text and audio) and ‘T+A(JE)’ (that
uses only text and audio and uses multimodal joint learning to compensate for
the visual modality) we note that the proposed multimodal joint learning
module enables around $5\%$ (and $2\%$) gain in the w-avg F1 score overall
(and five majority) classes. A similar observation can also be made by
comparing the subrows ‘A+V’ and ‘A+V(JE)’ (and subrows ‘T+V’ and ‘T+V(JE)’ )
which reports around $8\%$ (and $7\%$) performance gain in the w-avg F1 score.
Augmented with the proposed multimodal joint learning module, AM2-EmoJE
attains a state-of-the-art performance in two-mode data scenarios, where the
queries are presented with a variety of missing modality scenarios (e.g.
‘video’ or ‘audio’).

A similar performance was also observed in Table 2 which reports the
comparative analysis using the IEMOCAP dataset. The proposed AM2-EmoJE reports
$74.91\%$ w-avg F1, which is around $4.9\%$ improvement compared to the best-
performing baseline LIGHT-SERNET[32]. Being equipped with the proposed
multimodal joint learning module, the model again demonstrates a competitive
performance in two-mode data scenarios, where the queries are represented
using ‘text and audio’ or ‘text and video’.

Table 3 reports an ablation study. The first two rows in the Table demonstrate
the superiority of the weighted multimodal attention (Section 3.2) and
weighted fusion techniques (Section 3.3) compared to vanilla cross-attention
and static weights (where we use equal weight for all modes). The subrows in
the third row report the performance of the proposed AM2-EmoJE where only
classification focal loss $\mathcal{L}_{fl}$ is used, but we did not use
$\mathcal{L}_{ACE}$ in Eqn (3). As observed in the table, while the
introduction of the Guided NCE (Section 3.5.1) with focal loss delivers around
$1-2\%$ improvement in w-F1 score, by enabling a query adaptive multimodal
fusion scheme (Section 3.3) AM2-EmoJE delivers a robust ($2-4\%$ improved) the
w-F1 score.

Table 2: Performance comparison of difference methods using the weighted average F1 measure (W-Avg F1) on the IEMOCAP dataset with uni (T:=Text, A:=Audio, and V:= Video) and multi-modal Data Representations. ‘Feature Concat’ in row 13 and row 14 describe the concatenation of multiple uni-mode descriptors to define a multimodal descriptor. More details on emotion-specific comparison are provided in the supplementary material Method | Mode | w-Avg F1
---|---|---
MFN[19] | T + A | 0.3490
ICON[20] | T + A + V | 0.6350
DialogueRNN[21] | T + A + V | 0.6275
MMGCN[33] | T + A + V | 0.6622
DialogueCRN[23] | T + A | 0.6620
Hierarchical Uncertainty [27] | T + A + V | 0.6598
DAG-ERC+HCL[31] | T | 0.6803
M2FNet[3] | T + A + V | 0.6986
LIGHT-SERNET[32] | T + A + V | 0.7020
AM2-EmoJE | T + A | 0.6162
T + V | 0.6343
A + V | 0.5379
T + A (JE) | 0.6919
T + V (JE) | 0.7094
A + V (JE) | 0.6580
No Face | 0.7175
No Body | 0.7286
| T + A + V | 0.7491
Table 3: Ablation Study using w-F1 Score Model | Modalities | MELD | IEMOCAP
---|---|---|---
No Multimodal Attention | T+A+V | 0.6160 | 0.7527
Equal Fusion Weights | T+A+V | 0.6934 | 0.7165
No Guided NCE | No Body | 0.6343 | 0.6491
No Face | 0.6021 | 0.6182
T+A+V | 0.7041 | 0.7286
AM2-EmoJE | T+A+V | 0.7198 | 0.7491

##  5 Conclusion

We present AM2-EMOJE with our weighted multimodal attention and query adaptive
fusion, allowing us to effectively combine the information from various modes
to make better decision about the emotional state of subjects in a group
conversation. The model is also trained with our proposed Guided NCE loss that
allows the model to learn representation of the subjects with only their
facial features or body language, allowing us to better preserve the privacy
of participants and still achieve performance that is very close to the state-
of-the-art. Furthermore, we also propose an effective Multimodal Joint-
Embedding scheme that allows the model to effectively compensate for missing
modalities during inference, allowing its performance on a subset of
modalities to be close to its performance on the full set of modalities as
shown in our comparison with other state-of-the-art models and in the ablation
studies in the absence of Joint-Embedding.

## References

  * [1] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea,  “Meld: A multimodal multi-party dataset for emotion recognition in conversations,” 2019.
  * [2] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan,  “Iemocap: interactive emotional dyadic motion capture database,”  Language Resources and Evaluation, vol. 42, no. 4, pp. 335, Nov 2008.
  * [3] Vishal Chudasama, Purbayan Kar, Ashish Gudmalwar, Nirmesh Shah, Pankaj Wasnik, and Naoyuki Onoe,  “M2fnet: Multi-modal fusion network for emotion recognition in conversation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4652–4661.
  * [4] Zaijing Li, Fengxiao Tang, Ming Zhao, and Yusen Zhu,  “EmoCaps: Emotion capsule based model for conversational emotion recognition,”  in Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 2022, pp. 1610–1618, Association for Computational Linguistics.
  * [5] Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov,  “Efficient large-scale multi-modal classification,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  * [6] Piao Shi, Min Hu, Fuji Ren, Xuefeng Shi, and Liangfeng Xu,  “Learning modality-fused representation based on transformer for emotion analysis,”  Journal of Electronic Imaging, vol. 31, no. 6, pp. 063032, 2022.
  * [7] Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng,  “Are multimodal transformers robust to missing modality?,”  in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18156–18165.
  * [8] Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee,  “Multimodal prompting with missing modalities for visual recognition,”  2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14943–14952, 2023.
  * [9] Zhizhong Liu, Bin Zhou, Dianhui Chu, Yuhang Sun, and Lingqiang Meng,  “Modality translation-based multimodal sentiment analysis under uncertain missing modalities,”  Information Fusion, vol. 101, pp. 101973, 2024.
  * [10] Jiandian Zeng, Tianyi Liu, and Jiantao Zhou,  “Tag-assisted multimodal sentiment analysis under uncertain missing modalities,”  in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA, 2022, SIGIR ’22, p. 1545–1554, Association for Computing Machinery.
  * [11] Hong Liu, Dong Wei, Donghuan Lu, Jinghan Sun, Liansheng Wang, and Yefeng Zheng,  “M3ae: Multimodal representation learning for brain tumor segmentation with missing modalities,”  in Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence. 2023, AAAI’23/IAAI’23/EAAI’23, AAAI Press.
  * [12] Chaohe Zhang, Xu Chu, Liantao Ma, Yinghao Zhu, Yasha Wang, Jiangtao Wang, and Junfeng Zhao,  “M3care: Learning with missing modalities in multimodal healthcare data,”  in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 2022, KDD ’22, p. 2418–2428, Association for Computing Machinery.
  * [13] Nils Reimers and Iryna Gurevych,  “Sentence-bert: Sentence embeddings using siamese bert-networks,”  CoRR, vol. abs/1908.10084, 2019.
  * [14] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao,  “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [15] Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer,  “Efficient training of audio transformers with patchout,”  in Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022. 2022, pp. 2753–2757, ISCA.
  * [16] Amin Parvaneh, Ehsan Abbasnejad, Damien Teney, Reza Haffari, Anton van den Hengel, and Javen Qinfeng Shi,  “Active learning by feature mixing,” 2022.
  * [17] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár,  “Focal loss for dense object detection,”  CoRR, vol. abs/1708.02002, 2017.
  * [18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in International conference on machine learning. PMLR, 2021, pp. 8748–8763.
  * [19] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency,  “Memory fusion network for multi-view sequential learning,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  * [20] Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann,  “ICON: Interactive conversational memory network for multimodal emotion detection,”  in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Oct.-Nov. 2018, pp. 2594–2604, Association for Computational Linguistics.
  * [21] Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria,  “Dialoguernn: An attentive rnn for emotion detection in conversations,” 2019.
  * [22] Dong Zhang, Liangqing Wu, Changlong Sun, Shoushan Li, Qiaoming Zhu, and Guodong Zhou,  “Modeling both context- and speaker-sensitive dependence for emotion detection in multi-speaker conversations,”  in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. 7 2019, pp. 5415–5421, International Joint Conferences on Artificial Intelligence Organization.
  * [23] Dou Hu, Lingwei Wei, and Xiaoyong Huai,  “Dialoguecrn: Contextual reasoning networks for emotion recognition in conversations,”  arXiv preprint arXiv:2106.01978, 2021.
  * [24] Zaijing Li, Fengxiao Tang, Ming Zhao, and Yusen Zhu,  “Emocaps: Emotion capsule based model for conversational emotion recognition,”  arXiv preprint arXiv:2203.13504, 2022.
  * [25] Jingjun Liang, Ruichen Li, and Qin Jin,  “Semi-supervised multi-modal emotion recognition with cross-modal distribution matching,”  in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 2852–2861.
  * [26] Baijun Xie, Mariia Sidulova, and Chung Hyuk Park,  “Robust multimodal emotion recognition from conversation with transformer-based crossmodality fusion,”  Sensors, vol. 21, no. 14, pp. 4913, 2021.
  * [27] Feiyu Chen, Jie Shao, Anjie Zhu, Deqiang Ouyang, Xueliang Liu, and Heng Tao Shen,  “Modeling hierarchical uncertainty for multimodal emotion recognition in conversation,”  IEEE Transactions on Cybernetics, 2022.
  * [28] Harsh Agarwal, Keshav Bansal, Abhinav Joshi, and Ashutosh Modi,  “Shapes of emotions: Multimodal emotion recognition in conversations via emotion shifts,”  arXiv preprint arXiv:2112.01938, 2021.
  * [29] Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li,  “Unimse: Towards unified multimodal sentiment analysis and emotion recognition,”  arXiv preprint arXiv:2211.11256, 2022.
  * [30] Sitao Zhang, Yimu Pan, and James Z. Wang,  “Learning emotion representations from verbal and nonverbal communication,” 2023.
  * [31] Lin Yang, Yi Shen, Yue Mao, and Longjun Cai,  “Hybrid curriculum learning for emotion recognition in conversation,”  in AAAI Conference on Artificial Intelligence, 2021.
  * [32] Arya Aftab, Alireza Morsali, Shahrokh Ghaemmaghami, and Benoit Champagne,  “Light-sernet: A lightweight fully convolutional neural network for speech emotion recognition,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6912–6916.
  * [33] Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua,  “Mmgcn: Multi-modal graph convolution network for personalized recommendation of micro-video,”  in Proceedings of the 27th ACM International Conference on Multimedia, New York, NY, USA, 2019, MM ’19, p. 1437–1445, Association for Computing Machinery.


# [Reproducibility Report] Explainable Deep One-Class Classification

João P C Bertoldo
Center for Mathematical Morphology
MINES Paris - PSL University
Fontainebleau, France
<EMAIL_ADDRESS>
&Etienne Decencière
Center for Mathematical Morphology
MINES Paris - PSL University
Fontainebleau, France
<EMAIL_ADDRESS>

## Reproducibility Summary

### Scope of Reproducibility

Liznerski et al. [23] proposed Fully Convolutional Data Description (FCDD), an
explainable version of the Hypersphere Classifier (HSC) to directly address
image anomaly detection (AD) and pixel-wise AD without any post-hoc explainer
methods. The authors claim that FCDD achieves results comparable with the
state-of-the-art in sample-wise AD on Fashion-MNIST and CIFAR-10 and exceeds
the state-of-the-art on the pixel-wise task on MVTec-AD. They also give
evidence to show a clear improvement by using few (1 up to 8) real anomalous
images in MVTec-AD for supervision at the pixel level. Finally, a qualitative
study with horse images on PASCAL-VOC shows that FCDD can intrinsically reveal
spurious model decisions by providing built-in anomaly score heatmaps.

### Methodology

We have reproduced the quantitative results in the main text of [23] except
for the performance on ImageNet: sample-wise AD on Fashion-MNIST and CIFAR-10,
and pixel-wise AD on MVTec-AD. We used the author’s code with GPUs NVIDIA
TITAN X and NVIDIA TITAN Xp. A more detailed look into FCDD’s performance
variability is presented, and a Critical Difference (CD) diagram is proposed
as a more appropriate tool to compare methods over the datasets in MVTec-AD.
Finally, we study the generalization power of the unsupervised FCDD during
training.

### Results

All per-class performances (in terms of Area Under the ROC Curve (ROC-AUC)
[31]) announced in the paper were replicated with absolute difference of at
most 2% and below 1% on average, confirming the paper’s claims. We report the
experiments’ GPU and CPU memory requirements and their average training time.
Our analyses beyond the paper’s scope show that claiming to exceed the state-
of-the-art should be considered with care, and evidence is given to argue that
the pixel-wise unsupervised FCDD could narrow the gap with its semi-supervised
version.

### What was easy

The paper was clear and explicitly gave many training and hyperparameters
details, which were conveniently set as default in the author’s scripts. Their
code was well organized and easy to interact with.

### What was difficult

Using ImageNet proved to be challenging due to its size and need to manually
set it up; we could not complete the experiments on this dataset.

### Communication with original authors

We reached the main author by e-mail to ask for help with ImageNet and discuss
a few practical details. He promptly replied with useful information.

##  1 Introduction

Liznerski et al. [23] proposed a deep learning based AD method capable of
doing pixel-wise AD (also known as anomaly segmentation) by directly
generating anomaly score maps with a loss function based on the Hypersphere
Classifier (HSC) [29], a successor of Deep Support Vector Data Description
(DSVDD) [28], using a fully-convolutional neural network – hence the name
Fully Convolutional Data Description (FCDD).

By only using convolutions, down-samplings, and batch normalization (no
attention mechanism, nor fully connected layers), an image of dimensions
$C\times H\times W$ (respectively, the number of channels, the height, and the
width) is transformed into a latent representation $C^{\prime}\times U\times
V$, where $U<H$ and $V<W$. This low-resolution representation is a $U\times V$
grid of $C^{\prime}$-dimensional vectors, from which the pseudo-Huber loss
function yields a $U\times V$ heatmap of anomaly scores.

Each of these $C^{\prime}$-dimensional vectors contains information from a
corresponding receptive field within the full resolution ($H\times W$) image.
Evidence [24] suggests that the effective influence of the input pixels
decreases Gaussian-ly as their position is further away from the center of the
receptive field. FCDD uses this principle to up-sample the obtained heatmap
back to the original resolution ($H\times W$), therefore directly obtaining a
visual, explainable anomaly score map.

Finally, FCDD is also adapted to perform anomaly detection at the sample
(image) level by taking the average score on the low resolution anomaly
heatmap.

#### Vocabulary: Sample v.s. Pixel-wise Anomaly Detection

The authors refer to anomaly detection (AD) at the image level (e.g. given an
unseen image, a model trained on horse images should infer if there is a horse
present in it or, otherwise, the image is anomalous) simply as detection,
while anomaly segmentation/localization (i.e. finding regions, sets of pixels
where there exists anomalous characteristics) is referred as pixel-wise AD.
Analogously, for the sake of clarity, we refer to the former as sample-wise
AD. Both setups are further explained in Section 3.3.

##  2 Scope of reproducibility

We aimed to reproduce the results announced in [23] to verify the
effectiveness of the proposed method both in sample-wise and pixel-wise
anomaly detection. Specifically, we tested the following claims from the
original paper:

  1. 1.

Claim 1: FCDD is comparable with state-of-the-art methods in terms of ROC-AUC
in sample-wise anomaly detection on standard benchmarks (namely, Fashion-
MNIST, CIFAR-10, and ImageNet);

  2. 2.

Claim 2: FCDD exceeds the state-of-the-art on MVTec-AD in anomaly segmentation
in the unsupervised setting in terms of pixel-wise ROC-AUC;

  3. 3.

Claim 3: FCDD can incorporate real anomalies, and including only a few
annotated images ($\approx$ 5) containing real, segmented anomalies, the
performance consistently improves;

  4. 4.

Claim 4: FCDD can reveal spurious model decisions without any extra
explanation method on top of it.

The experiments on supporting Claim 1 on Fashion-MNIST and CIFAR-10 have been
replicated, as well as all the tests on MVTec-AD, supporting Claims 2 and 3,
and the qualitative analysis on PASCAL-VOC, supporting Claim 4. We provide
details about computational requirements (CPU memory, GPU memory, and training
time) necessary to run these experiments.

#### Beyond the paper

Other analyses are proposed on the results obtained from the experiments
corresponding to Claims 2 and 3, which further confirm Claim 3 but show that
Claim 2 should be taken with consideration. We also investigate the evolution
of the test performance during the optimization in MVTec-AD’s unsupervised
setting (see Section 3.3), revealing opportunity for improvement that could
narrow down the gap with the semi-supervised setting.

##  3 Methodology

We used the author’s code (PyTorch 1.9.1 and Torchvision 0.10.1), publicly
available on GitHub [4], to reproduce the quantitative experiments presented
in the main text. It required no external documentation, and the whole
reproduction took roughly one month-person of work.

###  3.1 Datasets

The proposed method was originally tested [23] on Fashion-MNIST [32], CIFAR-10
[19], ImageNet1k [13], MVTec-AD [9], and PASCAL VOC [14]. Besides, EMNIST
[11], CIFAR-100 [19], and ImageNet21k111Also known as ImageNet22k or full
ImageNet. (version fall 2011) were used as Outlier Exposure (OE) [16]
datasets. All the datasets except for ImageNet were publicly available and
automatically downloaded.

#### ImageNet

We requested access and download ImageNet1k (version ILSVRC 2012) from its
official website [5]. ImageNet21k (a.k.a. ImageNet22k) was downloaded from
academictorrents.com [1] because the version used in the original paper was
not available in the official website anymore.

###  3.2 Models

We used the same neural networks as the original paper, which depend on the
dataset:

  * •

Fashion-MNIST: three convolutional layers separated by two max-pool layers,
where the first convolution is followed by a batch normalization and a leaky
ReLU;

  * •

CIFAR-10: two convolutions preceded by three blocks, each, composed of a
convolution, a batch normalization, a leaky ReLU, and a max-pool layer;

  * •

MVTec-AD and PASCAL-VOC (Clever Hans): the first 10 (frozen) layers from VGG11
pre-trained on ImageNet followed by two convolutional layers.

###  3.3 Experimental setup

The paper presents two quantitative experiments: sample-wise (section 4.1
Standard Anomaly Detection Benchmarks in [23]) and pixel-wise AD (section 4.2
Explaining Defects in Manufacturing in [23]), as well as a qualitative
experiment.

We followed the same experimental procedure used in [23]: each experiment –
i.e. given a dataset, its OE when applicable, a normal class, and all
hyperparameters – was repeated five times, and the reported values are the
average over them unless stated otherwise (e.g. Figure 1).

#### Sample-wise

Standard one-vs-rest setup, where one class of the given database is chosen as
normal and all the others are used as anomalous. Each image has a binary
ground truth signal – logically derived from its label – and the model assigns
an anomaly score to it (therefore sample-wise). The metric used is the ROC-AUC
on the test split, and all the classes are evaluated as normal. The datasets
used in this experiment and their respective OE dataset is summarized in Table
1, and its results support Claim 1.

Table 1:  Sample-wise experiments: tested datasets and their respective OE sources. From the dataset in the column "one-vs-rest", one class is used as normal at training and test time while all others are considered anomalies at test time only. The column Outlier Exposure (OE) is the dataset used as a source of anomalies at training time. "Experiment reference" will further be used to reference these configurations.  One-vs-rest dataset | OE dataset | Experiment reference
---|---|---
Fashion-MNIST | EMNIST | F-MNIST (OE-EMNIST)
CIFAR-100 | F-MNIST (OE-CIFAR-100)
CIFAR-10 | CIFAR-100 | CIFAR-10
ImageNet1k | ImageNet21k | ImageNet

#### Pixel-wise

Anomalies are defined at the pixel level (binary segmentation mask where 1
means anomalous and 0 means normal) and an image is considered anomalous if it
_contains_ anomalous pixels although normal pixels are also present in the
image. In each experiment a single class in MVTec-AD is fixed, its normal
images are used both for training and test, and anomalous ones are used for
test. As for the anomalous samples at training time, two settings were tested:

  * •

Unsupervised: synthetic random anomalies are generated using a confetti noise
(colored blobs added to the image);

  * •

Semi-supervised: one image per anomaly group (1 up to 8 types depending on the
class) is removed from the test set and used for training.

Neither of these settings require an OE dataset because the anomalous samples
are either synthetic or real on images of the nominal class. The performance
metric is the ROC-AUC of the anomaly scores at the pixel level. MVTec-AD is
the only dataset used in this case, and the results of these experiment
support Claims 2 and 3.

#### Clever Hans (PASCAL VOC)

About one fifth of the images in the class horse in PASCAL VOC [14] contain a
watermark [20], which may cause models to learn spurious features. This is
known as the Clever Hans effect as a reference to Hans, a horse claimed to be
capable of performing arithmetic operations while it, in fact, read his
master’s reactions [26] – analogously, a model making decisions based on the
watermarks would be cheating the real problem. In this experiment, a model is
trained using all the classes in PASCAL VOC as normal, only the class horse as
anomalous (a swapped one-vs-rest setting), and ImageNet1k as OE dataset. The
goal is to qualitatively observe if one class classifiers are also vulnerable
to the Clever Hans effect and show that FCDD transparently reveals such
weaknesses as it intrinsically provides explanations (score heatmaps). This
experiment has no quantitative metric but it supports Claim 4.

###  3.4 Hyperparameters

Running the author’s code with the default parameters, as described in the
original paper, did not require any hyperparameter tuning to achieve the
reported results (differences detailed in Section 4) and confirm the authors’
claims. We underline that the results on MVTec-AD were obtained using the same
hyperparameters on both settings, unsupervised and semi-supervised.

###  3.5 Computational requirements

We used the NVIDIA GPUs TITAN X [6] and TITAN Xp [7] to run our experiments.
The two GPUs were used indistinctly as they have similar characteristics, and
only one GPU was used at a time. The GPU and CPU memory requirements and the
average training duration of our experiments are listed in the Table 2 below.

CPU memory was recorded with an in-house python script using the library
psutil [3] at 1Hz. GPU memory was recorded using gpustat [2] at 1Hz. Both
memory values are the maximum recorded during the experiments, including
training and inference time. The training duration is an average over all the
experiments.

On F-MNIST (OE-CIFAR-100) and CIFAR-10, the range of duration did not vary
more than two minutes, on MVTec-AD unsupervised it ranged from 15 minutes up
to one hour, and on MVTec-AD semi-supervised it ranged from 22 minutes up to
one hour and 56 minutes depending on the class.

Table 2:  Memory requirements and training time using NVIDIA GPUs TITAN X and TITAN Xp (one at a time, indistinctly).  Experiment | CPU memory (Gb) | GPU memory (Gb) | Training duration
---|---|---|---
F-MNIST (OE-CIFAR-100) | 2 | 1.3 | 12 min
CIFAR-10 | 3 | 1.9 | 34 min
MVTec-AD unsupervised | 38 | 5.5 | 1h 13 min
MVTec-AD semi-supervised | 33 | 5.5 | 41 min
PASCAL VOC (Clever Hans) | 5 | 11.8 | 21 min

###  3.6 Beyond the paper

We propose a more detailed visualization of the distribution of performances
(due to random effects, all hyperparameters held constant) of the two settings
(unsupervised and semi-supervised) evaluated on MVTec-AD, and a critical
difference diagram as alternative evaluation of performance across several
datasets (the individual classes in MVTec-AD).

The network architectures used for the experiments on MVTec-AD were pre-
trained on ImageNet and most of the weights are kept frozen, therefore raising
the question of how much of FCDD’s performance is due to the pre-training. We
took snapshots of the unsupervised model’s weights in order to visualize the
evolution of the performance on the test set during training.

##  4 Results

###  4.1 Reproducing the original paper

We reproduced the unsupervised and semi-supervised settings for MVTec-AD, and
all experiments on Table 1 except for ImageNet – due to resource limitations,
this experiment could not be completed in time.

The results of F-MNIST (OE-EMNIST) were not detailed in the original paper but
it is claimed to have a class-mean ROC-AUC of $\sim 3\%$ below F-MNIST (OE-
CIFAR-100), and we observed a difference of $2.7\%$.

We summarize the differences between our results and those from the original
paper [23] in Table 3. The error margins presented are in absolute differences
and refer to the ROC-AUC (which is expressed in %) from each individual
class’s experiment (recall: the mean over five iterations).

Table 3:  Differences between the original paper results and ours. All the values are in absolute difference of ROC-AUC, expressed in % (it is not a relative error). The columns in "Difference per class" show statistics of the absolute difference of each individual class performance, while the column "Mean ROC-AUC diff." corresponds to the difference measured after the mean is taken over all the classes.  Experiment |  ROC-AUC  type | N. classes | Diff. per class |  |  Mean ROC-AUC diff.
---|---|---|---|---|---
Max | Mean |
F-MNIST (OE-CIFAR-100) | sample-wise | 10 | 1% | 0.6% |  | 0.01%
CIFAR-10 | sample-wise | 10 | 0.5% | 0.3% |  | 0.4%
MVTec-AD unsupervised | pixel-wise | 15 | 2% | 0.6% |  | 0.2%
MVTec-AD semi-supervised | pixel-wise | 15 | 2% | 0.7% |  | 0.4%

#### Clever Hans (PASCAL VOC)

The experiment on PASCAL VOC (Clever Hans Effect) has been manually verified,
and similar (flawed) explanations on horse images have been observed. Two
examples are shown in Figure 5 in Appendix A.

###  4.2 Beyond the paper

Figure 1 further details the performance comparison between the unsupervised
and semi-supervised settings on MVTec-AD on each class.

Figure 2 compares the methods in Table 2 in [23] with a CD diagram using the
Wilcoxon-Holm procedure implemented by [17]. We replaced the results for FCDD
from [23] by our own and copied the others from the literature [10, 30, 25,
22, 21, 12, 35]. For each class, the methods are sorted by their respective
ROC-AUC and assigned a ranking from 1 to 10 according to their position; then,
every pair of methods is compared with the Wilcoxon signed rank test with the
confidence level $\alpha=5\%$. The CD diagram shows the average ranks of the
methods on the horizontal scale, and the red bars group methods where each
pair of methods are not significantly different according to the Wilcoxon
signed rank test. The ranks from one to five (six to ten omitted for the sake
of brevity) are shown in Table 4 in the Appendix A.

Figure 3 shows the test pixel-wise ROC-AUC scores during the optimization of
the model used for MVTec-AD with the unsupervised setting. Due to time and
resources constraints, we ran this experiment on 7 out of the 15 classes in
MVTec-AD, each of them being evaluated 6 times (a few of them could not finish
in time).

Figure 1:  Our experiments on MVTec-AD: unsupervised and semi-supervised
settings compared. We display a box plot of the performances (in terms of
pixel-wise ROC-AUC on the test set) achieved in different runs along with
their individual performances scattered on the $x$-axis.  Figure 2:  MVTec-AD
Critical Difference diagram. Using the Table 2 from [23] with the results for
FCDD replaced by our own, we build a critical difference diagram using the
Wilcoxon-Holm method. Values on the scale are average rankings, and each red
line groups a set of methods that are not significantly different in terms of
ranking with a confidence level of $\alpha=5\%$. The per-class ROC-AUC values
used for FCDD are from our own experiments, and those marked with "*" were
taken from the literature. References: Scores for Self-Similarity (AE-SS)
[10], L2 Autoencoder (AE-L2) [10], AnoGAN [30], and CNN Feature Dictionaries
(CNNFD) [25] were taken from Table 3 in [10]. Other scores were taken from
their respective papers: Visually Explained Variational Autoencoder (VEVAE)
from Table 2 in [22], Superpixel Masking and Inpainting (SMAI) from Table 2 in
[21], Gradient Descent Reconstruction with VAEs (GDR) from Table 1 in [12],
Encoding Structure-Texture Relation with P-Net for AD (P-NET) from Table 6 in
[35].  Figure 3: MVTec-AD test performance history.

##  5 Discussion

Our reproduction of the experiments closely agree with the quantitative
results published in the original paper. The proposed setup is adapted to
support the claims announced in the paper hold, and the results corroborate
it. We obtained results consistently close to the published ones without any
further tuning of the parameters or modification of the authors’ code.

###  5.1 What was easy

The paper is clear, and it was easy to grasp the core ideas presented in the
main text. It also provided enough details about the experimental setup,
including training hyper parameters and network architecture in the
appendices.

The code was overall well organized and the instructions to use it were direct
and easy to use. Conveniently, the experiments were well encapsulated scripts
and default parameters matched those described in the text. In particular, the
experiments are self-documenting (i.e. they keep record of the configurations,
logs, and results, etc) and flexible, allowing the user to change (many)
parameters without modifying the code.

###  5.2 What was difficult

#### ImageNet

Using ImageNet was the hardest part. At first, it took about a month to get
access to it on the official website. Then we had to find an alternative
source [1] to find the correct version of ImageNet21k (fall 2011) because it
was not available on the official website anymore. Basic operations (e.g.
decompressing data, moving files) proved challenging due to its size (1.2 TB
compressed), and the instructions to manually prepare this dataset could be
more explicit – we wasted several hours of work because of a few mistakes we
made.

We could not run the experiments on that dataset with the same hyperparameters
because the GPU we dispose of did not have enough memory (16GB). Although, we
note that some solutions like using multiple GPUs or decreasing the batch size
were possible but could not be tried in time.

#### Minor code issues

There were a few minor bugs, which we corrected without considerable
difficulty. They were mostly related with the script add_exp_to_base.py, which
automatically configures and launches baseline experiments based on a
previously executed one. Finally, the code structure was slightly
overcomplicated; e.g. the levels of abstraction/indirection, specially
heritage, could be simpler. Although, we stress that this negative point is
minor and did not cause any critical issues.

###  5.3 Communication with original authors

We exchanged e-mails with the main author, mostly to ask for help with getting
access to the right versions of ImageNet and executing the experiments on it.
He replied promptly, his answers were certainly helpful, and we would like to
express our sincere appreciation.

###  5.4 Beyond the paper

#### MVTec-AD: supervision effect

The visualization proposed in Figure 1 further demonstrates that, with only a
few images of real anomalies added to the training, the model’s performance
consistently improves. Only 4/15 classes have performance overlap, all others
show a clear shift in the performance distribution.

However, it must be mentioned that the synthetic anomalies ignore the local
supervision, therefore making its training sub-optimal. Figure 4 illustrates
this with training images and their respective masks from the class Pill: in
4(a) we see that the the semi-supervised setting provides pixel-level
annotations on the anomalies (the ground truth mask), while in 4(b) we see
that the entire image is considered anomalous in the unsupervised setting.
This is a source of sub-optimality because, in the anomalous images, most
pixels are, in fact, normal. In other words, similar images patches, free of
synthetic anomaly, can be found both in normal and anomalous images.

Ultimately, this a clear opportunity of improvement that can bring the
unsupervised setting’s performance closer to the semi-supervised setting.

#### Test performance history

Figure 3 reveals another issue with the method. Take, for instance, the purple
and blue lines in the row Carpet; they reach a maximum point at the beginning
of the gradient descent then converge to a point with less and less
generalization power. These performance histories are evaluated on the test
set, which is assumed to be unavailable at training time, so this information
could not be used to stop the training or reject it. However, this reveals
another opportunity of improvement because the training setting does not push
the model to generalize well enough. Note, in Figure 4(b), that the confetti
noise also "stains" the background, creating synthetic anomalies out of
context, so using more realistic ones could be a solution?

We also see that some (good) performances are likely due to the pre-training
(on ImageNet). For instance, the class Hazelnut often reaches its maximum test
performance (or almost) with few epochs.

####  Critical Difference (CD) diagram

We propose a CD diagram in Figure 2 as a more appropriate methodology to
aggregate results of all the classes. As a potential user is choosing an AD
method for a new dataset, he or she is looking for the method(s) that will
most likely be the best on his or her own problem. Therefore, the specific
ROC-AUC scores on standard datasets have little importance but their relative
performances is essential. In other words, what matters for a potential user
is how the comparison of methods generalizes over specific datasets.

The experiments on MVTec-AD do not interact from one class (set as nominal) to
another, making them essentially independent datasets; therefore, taking the
average score over the classes may mislead the analysis. For instance, in
[23], FCDD unsupervised is claimed to beat the state-of-the-art, although
Figure 2 shows that GDR [12] has a better average ranking.

Note that the red bars in the diagram may give the impression that there is no
relevant difference at all; although, it is important to observe that it was
built considering only 15 datasets (therefore 15 rankings), making the
statistical test hard, so using more datasets could refine these groups and
provide better understanding. Finally, it is worth noting that the CD diagram
is capable of incorporating new datasets, while the mean score over them would
be too much affected if some cases are much easier or much harder than the
others.

#### State-of-the-art

It is worth mentioning that more recent methods have claimed better results on
the same benchmarks used in this work. For instance, at least 5 papers [27,
18, 34, 33, 15] claim to have a mean ROC-AUC above $98\%$ on the leaderboard
for anomaly segmentation (pixel-wise AD) on MVTec-AD in Papers with Code [8].
Unfortunately, we did not have the time to fully verify the experimental
conditions in the sources but this serves as proxy evidence to take these
results with consideration.

(a) Semi-supervised

(b) Unsupervised

Figure 4: MVTec-AD training images: unsupervised vs. semi-supervised.

## References

  * [1] Academic torrents, download page for ImageNet22k version fall 2011.  https://academictorrents.com/details/564a77c1e1119da199ff32622a1609431b9f1c47, 2022\.
  * [2] Documentation page of gpustat.  https://github.com/wookayin/gpustat, 2022.
  * [3] Documentation page of the python library psutil.  https://psutil.readthedocs.io/en/latest/, 2022.
  * [4] GitHub repository liznerski/fcdd.  https://github.com/liznerski/fcdd, 2022.  Downloaded: 2021-12-20. Forked on commit 7af3d8eadabee81ab8f7db5dea7f8389ef090213.
  * [5] ImageNet official page, challenge ILSVRC 2012.  https://image-net.org/challenges/LSVRC/2012/2012-downloads.php, 2022\.
  * [6] NVIDIA’s TITAN X product page.  https://www.nvidia.com/en-us/geforce/products/10series/titan-x-pascal/, 2022\.  Accessed: 2022-01-24.
  * [7] NVIDIA’s TITAN Xp product page.  https://www.nvidia.com/en-us/titan/titan-xp/, 2022.  Accessed: 2022-01-24.
  * [8] Papers with code’s leaderboard on Anomaly Detection on MVTec AD.  https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad?metric=Segmentation, 2022\.
  * [9] P. Bergmann, K. Batzner, M. Fauser, D. Sattlegger, and C. Steger.  The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection.  International Journal of Computer Vision, 129(4):1038–1059, Apr. 2021.
  * [10] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger.  MVTec AD – A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [11] G. Cohen, S. Afshar, J. Tapson, and A. van Schaik.  EMNIST: Extending MNIST to handwritten letters.  In 2017 International Joint Conference on Neural Networks (IJCNN), pages 2921–2926, 2017.
  * [12] D. Dehaene, O. Frigo, S. Combrexelle, and P. Eline.  Iterative energy-based projection on a normal data manifold for anomaly localization.  page 17, 2020.
  * [13] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [14] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The Pascal Visual Object Classes (VOC) Challenge.  International Journal of Computer Vision, 88(2):303–338, June 2010\.
  * [15] D. Gudovskiy, S. Ishizaka, and K. Kozuka.  CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows.  arXiv:2107.12571 [cs], July 2021.  arXiv: 2107.12571 version: 1.
  * [16] D. Hendrycks, M. Mazeika, and T. Dietterich.  Deep Anomaly Detection with Outlier Exposure.  In International Conference on Learning Representations, 2019\.
  * [17] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller.  Deep learning for time series classification: a review.  Data Mining and Knowledge Discovery, 33(4):917–963, 2019.
  * [18] J.-H. Kim, D.-H. Kim, S. Yi, and T. Lee.  Semi-orthogonal Embedding for Efficient Unsupervised Anomaly Segmentation.  arXiv:2105.14737 [cs], May 2021.  arXiv: 2105.14737 version: 1.
  * [19] A. Krizhevsky.  Learning Multiple Layers of Features from Tiny Images, 2009\.  Technical report.
  * [20] S. Lapuschkin, A. Binder, G. Montavon, K.-R. Muller, and W. Samek.  Analyzing Classifiers: Fisher Vectors and Deep Neural Networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [21] Z. Li, N. Li, K. Jiang, Z. Ma, X. Wei, X. Hong, and Y. Gong.  Superpixel Masking and Inpainting for Self-Supervised Anomaly Detection.  page 12, 2020.
  * [22] W. Liu, R. Li, M. Zheng, S. Karanam, Z. Wu, B. Bhanu, R. J. Radke, and O. Camps.  Towards Visually Explaining Variational Autoencoders.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [23] P. Liznerski, L. Ruff, R. A. Vandermeulen, B. J. Franks, M. Kloft, and K. R. Muller.  Explainable Deep One-Class Classification.  In International Conference on Learning Representations, 2021\.
  * [24] W. Luo, Y. Li, R. Urtasun, and R. Zemel.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 4905–4913, Red Hook, NY, USA, 2016. Curran Associates Inc.  event-place: Barcelona, Spain.
  * [25] P. Napoletano, F. Piccoli, and R. Schettini.  Anomaly Detection in Nanofibrous Materials by CNN-Based Self-Similarity.  Sensors (Basel, Switzerland), 18(1):209, Jan. 2018.
  * [26] O. Pfungst.  Clever Hans (The Horse of Mr. Von Osten) A contribution to experimental animal and human psychology.  Oct. 2010.
  * [27] K. Roth, L. Pemula, J. Zepeda, B. Schölkopf, T. Brox, and P. Gehler.  Towards Total Recall in Industrial Anomaly Detection.  arXiv:2106.08265 [cs], June 2021.  arXiv: 2106.08265 version: 1.
  * [28] L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft.  Deep one-class classification.  In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4393–4402. PMLR, 10–15 Jul 2018.
  * [29] L. Ruff, R. A. Vandermeulen, B. J. Franks, K.-R. Müller, and M. Kloft.  Rethinking Assumptions in Deep Anomaly Detection.  arXiv:2006.00339 [cs, stat], July 2021.  arXiv: 2006.00339.
  * [30] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery.  In M. Niethammer, M. Styner, S. Aylward, H. Zhu, I. Oguz, P.-T. Yap, and D. Shen, editors, Information Processing in Medical Imaging, pages 146–157, Cham, 2017. Springer International Publishing.
  * [31] K. A. Spackman.  Signal Detection Theory: Valuable Tools for Evaluating Inductive Learning.  In Proceedings of the Sixth International Workshop on Machine Learning, pages 160–163, Ithaca, New York, USA, 1989. Morgan Kaufmann Publishers Inc., 340 Pine Street, Sixth Floor, San Francisco, CA, USA.
  * [32] H. Xiao, K. Rasul, and R. Vollgraf.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, 2017.  arXiv: 1708.07747.
  * [33] J. Yu, Y. Zheng, X. Wang, W. Li, Y. Wu, R. Zhao, and L. Wu.  FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flows.  arXiv:2111.07677 [cs], Nov. 2021.  arXiv: 2111.07677 version: 2.
  * [34] Y. Zheng, X. Wang, R. Deng, T. Bao, R. Zhao, and L. Wu.  Focus Your Distribution: Coarse-to-Fine Non-Contrastive Learning for Anomaly Detection and Localization.  arXiv:2110.04538 [cs], Oct. 2021.  arXiv: 2110.04538 version: 1.
  * [35] K. Zhou, Y. Xiao, J. Yang, J. Cheng, W. Liu, W. Luo, Z. Gu, J. Liu, and S. Gao.  Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images.  In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision – ECCV 2020, pages 360–377, Cham, 2020. Springer International Publishing.

##  Appendix A Supplementary details

Normal Class | 5 | 4 | 3 | 2 | 1
---|---|---|---|---|---
Bottle | GDR* | AE-SS* | FCDD (SS) | FCDD (U) | P-NET*
| 92.0 | 93.0 | 96.7 | 97.0 | 99.0
Cable | VEVAE* | FCDD (U) | GDR* | SMAI* | FCDD (SS)
| 90.0 | 90.5 | 91.0 | 92.0 | 94.1
Capsule | GDR* | FCDD (SS) | SMAI* | FCDD (U) | AE-SS*
| 92.0 | 92.9 | 93.0 | 93.0 | 94.0
Carpet | VEVAE* | AE-SS* | SMAI* | FCDD (U) | FCDD (SS)
| 78.0 | 87.0 | 88.0 | 96.4 | 98.7
Grid | AE-SS* | FCDD (SS) | GDR* | SMAI* | P-NET*
| 94.0 | 95.2 | 96.0 | 97.0 | 98.0
Hazelnut | P-NET* | SMAI* | FCDD (SS) | GDR* | VEVAE*
| 97.0 | 97.0 | 97.1 | 98.0 | 98.0
Leather | P-NET* | GDR* | VEVAE* | FCDD (U) | FCDD (SS)
| 89.0 | 93.0 | 95.0 | 98.4 | 98.7
Metal nut | GDR* | SMAI* | FCDD (U) | VEVAE* | FCDD (SS)
| 91.0 | 92.0 | 94.0 | 94.0 | 97.5
Pill | AE-SS* | P-NET* | SMAI* | GDR* | FCDD (SS)
| 91.0 | 91.0 | 92.0 | 93.0 | 96.8
Screw | AE-L2* | AE-SS* | SMAI* | VEVAE* | P-NET*
| 96.0 | 96.0 | 96.0 | 97.0 | 100.0
Tile | VEVAE* | FCDD (U) | CNNFD* | P-NET* | FCDD (SS)
| 80.0 | 91.4 | 93.0 | 97.0 | 98.5
Toothbrush | VEVAE* | FCDD (SS) | SMAI* | GDR* | P-NET*
| 94.0 | 94.7 | 96.0 | 99.0 | 99.0
Transistor | FCDD (U) | AE-SS* | FCDD (SS) | GDR* | VEVAE*
| 87.6 | 90.0 | 91.3 | 92.0 | 93.0
Wood | GDR* | FCDD (U) | CNNFD* | FCDD (SS) | P-NET*
| 84.0 | 86.9 | 91.0 | 92.0 | 98.0
Zipper | AE-SS* | P-NET* | SMAI* | FCDD (U) | FCDD (SS)
| 88.0 | 90.0 | 90.0 | 92.2 | 98.1
Table 4:  Method rankings on MVTec-AD based on pixel-wise ROC-AUC. Using the
Table 2 from [23], we compare the methods by normal class individually sorting
the performances by pixel-wise ROC-AUC. The numbers in column names indicate
the ranking (from 1 to 10); only the first 5 are displayed for the sake
brevity. FCDD "unsupervised" and "semi-supervised" versions are respectively
indicated by "(U)" and "(SS)", and their original values have been replaced by
our own experiments’ results.  Figure 5: Heatmaps from the experiment on
PASCAL-VOC where the Clever Hans effect can be observed.