# Are all outliers alike?
A Taxonomy of Out of Distribution Samples

Ramneet Kaur , Susmit Jha , Anirban Roy††footnotemark:  , Oleg Sokolsky &
Insup Lee††footnotemark:
ramneetk<EMAIL_ADDRESS>
{sokolsky<EMAIL_ADDRESS>Ramneet Kaur is a graduate student at the
Department of Computer and Information Science, University of Pennsylvania,
Philadelphia, USA. This work was done when she was a summer intern at
SRI.Computer Science Laboratory, SRI International, Menlo Park, CA 94025,
USADepartment of Computer and Information Science, University of Pennsylvania,
Philadelphia, USA.

# Are all outliers alike?
On Understanding the Diversity of Outliers
for Detecting OODs

Ramneet Kaur , Susmit Jha , Anirban Roy††footnotemark:  , Oleg Sokolsky &
Insup Lee††footnotemark:
ramneetk<EMAIL_ADDRESS>
{sokolsky<EMAIL_ADDRESS>Ramneet Kaur is a graduate student at the
Department of Computer and Information Science, University of Pennsylvania,
Philadelphia, USA. This work was done when she was a summer intern at
SRI.Computer Science Laboratory, SRI International, Menlo Park, CA 94025,
USADepartment of Computer and Information Science, University of Pennsylvania,
Philadelphia, USA.

###### Abstract

Deep neural networks (DNNs) are known to produce incorrect predictions with
very high confidence on out-of-distribution (OOD) inputs. This limitation is
one of the key challenges in the adoption of deep learning models in high-
assurance systems such as autonomous driving, air traffic management, and
medical diagnosis. This challenge has received significant attention recently,
and several techniques have been developed to detect inputs where the model’s
prediction cannot be trusted. These techniques use different statistical,
geometric, or topological signatures. This paper presents a taxonomy of OOD
outlier inputs based on their source and nature of uncertainty. We demonstrate
how different existing detection approaches fail to detect certain types of
outliers. We utilize these insights to develop a novel integrated detection
approach that uses multiple attributes corresponding to different types of
outliers. Our results include experiments on CIFAR10, SVHN and MNIST as in-
distribution data and Imagenet, LSUN, SVHN (for CIFAR10), CIFAR10 (for SVHN),
KMNIST, and F-MNIST as OOD data across different DNN architectures such as
ResNet34, WideResNet, DenseNet, and LeNet5.

##  1 Introduction

Deep neural networks (DNNs) have achieved remarkable performance-levels in
many areas such as computer vision (Gkioxari et al., 2015), speech recognition
(Hannun et al., 2014), and text analysis (Majumder et al., 2017). But their
deployment in the safety-critical systems such as self-driving vehicles
(Bojarski et al., 2016), aircraft collision avoidance (Julian & Kochenderfer,
2017), and medical diagnoses (De Fauw et al., 2018) is hindered by their
brittleness. One major challenge is the inability of DNNs to be self-aware of
when new inputs are outside the training distribution and likely to produce
incorrect predictions. It has been widely reported in literature (Guo et al.,
2017a; Hendrycks & Gimpel, 2016) that deep neural networks exhibit
overconfident incorrect predictions on inputs which are outside the training
distribution. The responsible deployment of deep neural network models in
high-assurance applications necessitates detection of out-of-distribution
(OOD) data so that DNNs can abstain from making decisions on those.

Recent approaches for OOD detection consider different statistical, geometric
or topological signatures in data that differentiate OODs from the training
distribution. For example, the changes in the softmax scores due to input
perturbations and temperature scaling have been used to detect OODs (Hendrycks
& Gimpel, 2016; Liang et al., 2017; Guo et al., 2017b). Papernot & McDaniel
(2018) use the conformance among the labels of the nearest neighbors while
Tack et al. (2020) use cosine similarity (modulated by the norm of the feature
vector) to the nearest training sample for the detection of OODs. Lee et al.
(2018) consider the Mahalanobis distance of an input from the in-distribution
data to detect OODs. Several other metrics such as reconstruction error (An &
Cho, 2015), likelihood-ratio between the in-distribution and OOD samples (Ren
et al., 2019), trust scores (ratio of the distance to the nearest class
different from the predicted class and the distance to the predicted class)
(Jiang et al., 2018), density function (Liu et al., 2020; Hendrycks et al.,
2019a), probability distribution of the softmax scores (Lee et al., 2017;
Hendrycks et al., 2019b; Tack et al., 2020; Hendrycks et al., 2019a) have also
been used to detect OODs. All these methods attempt to develop a uniform
approach with a single signature to detect all OODs accompanied by empirical
evaluations that use datasets such as CIFAR10 as in-distribution data and
other datasets such as SVHN as OOD.

Our study shows that OODs can be of diverse types with different defining
characteristics. Consequently, an integrated approach that takes into account
the diversity of these outliers is needed for effective OOD detection. We make
the following three contributions in this paper:

  * •

Taxonomy of OODs. We define a taxonomy of OOD samples that classify OODs into
different types based on aleatoric vs epistemic uncertainty (Hüllermeier &
Waegeman, 2019), distance from the predicted class vs the distance from the
tied training distribution, and uncertainty in the principal components vs
uncertainty in non-principal components with low variance.

  * •

Incompleteness of existing uniform OOD detection approaches. We examine the
limitations of the state-of-the-art approaches to detect various types of OOD
samples. We observe that not all outliers are alike and existing approaches
fail to detect particular types of OODs. We use a toy dataset comprising two
halfmoons as two different classes to demonstrate these limitations.

  * •

An integrated OOD detection approach. We propose an integrated approach that
can detect different types of OOD inputs. We demonstrate the effectiveness of
our approach on several benchmarks, and compare against state-of-the-art OOD
detection approaches such as the ODIN (Liang et al., 2017) and Mahalanobis
distance method (Lee et al., 2018).

##  2 OOD Taxonomy and Existing Detection Methods

DNNs predict the class of a new input based on the classification boundaries
learned from the samples of the training distribution. Aleatory uncertainty is
high for inputs which are close to the classification boundaries, and
epistemic uncertainty is high when the input is far from the learned
distributions of all classes (Hora, 1996; Hüllermeier & Waegeman, 2019). Given
the predicted class of a DNN model on a given input, we can observe the
distance of the input from the distribution of this particular class and
identify it as an OOD if this distance is high. We use this top-down inference
approach to detect this type of OODs which are characterized by an
inconsistency in model’s prediction and input’s distance from the distribution
of the predicted class. Further, typical inputs to DNNs are high-dimensional
and can be decomposed into principal and non-principal components based on the
direction of high variation; this yields another dimension for classification
of OODs. We, thus, categorize an OOD using the following three criteria.

  1. 1.

Is the OOD associated with higher epistemic or aleatoric uncertainty, i.e., is
the input away from in-distribution data or can it be confused between
multiple classes?

  2. 2.

Is the epistemic uncertainty of an OOD sample unconditional or is it
conditioned on the class predicted by the DNN model?

  3. 3.

Is the OOD an outlier due to unusually high deviation in the principal
components of the data or due to small deviation in the non-principal (and
hence, statistically invariant) components?

Figure 1: The different types of OODs in a 2D space with three different
classes. The class distributions are represented as Gaussians with black
boundaries and the tied distribution of all training data is a Gaussian with
red boundary.

Figure 1 demonstrates different types of OODs which differ along these
criteria. Type 1 OODs have high epistemic uncertainty and are away from the
in-distribution data. Type 2 OODs have high epistemic uncertainty with respect
to each of the 3 classes even though approximating all in-distribution (ID)
data using a single Guassian distribution will miss these outliers. Type 3
OODs have high aleatoric uncertainty as they are close to the decision
boundary between class 0 and class 1. Type 4 and 5 have high epistemic
uncertainty with respect to their closest classes. While Type 4 OODs are far
from the distribution along the principal axis, Type 5 OODs vary along a
relatively invariant axis where even a small deviation indicates that the
sample is an OOD.

Limitations of Existing Detection Methods. We empirically demonstrate the
limitations of existing OOD detection methods on a two-dimensional (2D) half-
moon dataset with two classes. As shown in Figure 2, we consider three
clusters of OOD samples: cluster A (black), B (brown) and C(red). Figure 2
(right) shows the 2D penultimate features of the classifier.

Figure 2:  The cluster A (black), cluster B (brown), and cluster C (red)
clusters represent the OOD due to epistemic uncertainty in the tied training
distribution, epistemic uncertainty in the class-conditional training
distribution, and the aleatoric uncertainty in the class-conditional
distribution, respectively. (Left) shows the training data of the 2 half-moon
classes and the 3 OOD clusters in the input space along with the trained
classifier’s boundary and its softmax scores. (Right) shows the ID samples and
the OODs after projection to the 2D feature space (penultimate layer) of the
DNN.

(a) Estimating distance from the tied in-distribution fails to detect OOD
clusters B and C. (b) Techniques based on softmax score fail to detect OODs in
cluster A. (c) Estimating distance from the class-wise in-distribution fails
to detect OODs in cluster C. (d) Non-conformance among the nearest neighbors
fails to detect OODs in cluster A and B.

Figure 3: Detected OODs are shown in blue and undetected OODs are in red.
Different techniques fail to detect different types of OODs.

Different approaches differ in their ability to detect different OOD types as
illustrated in Figure 3.

  * •

Figure 3(a) shows that the Mahalanobis distance (Lee et al., 2018) from the
mean and tied covariance of all the training data in the feature space cannot
detect OODs in the clusters B and C corresponding to class-conditional
epistemic uncertainty and aleatoric uncertainty, respectively. It attains the
overall true negative rate (TNR) of 39.09% at the 95% true positive rate
(TPR).

  * •

Figure 3(b) shows that the softmax prediction probability (SPB) (Hendrycks &
Gimpel, 2016) cannot detect the OODs in cluster A corresponding to high
epsitemic uncertainty. The TNR ( at 95% TPR) reported by the SPB technique is
60.91%.

  * •

Figure 3(c) shows that class-wise Principal Component Analysis (PCA)
(Hoffmann, 2007) cannot detect OODs in cluster C corresponding to high
aleatoric uncertainty. We performed PCA of the two classes separately in the
feature space and used the minimum reconstruction error to detect OODs. This
obtained overall TNR of 80.91% (at 95% TPR).

  * •

Figure 3(d) shows that K-Nearest Neighbor (kNN) (Papernot & McDaniel, 2018)
non-conformance in the labels of the nearest neighbors cannot detect OODs in
clusters A and B with high epistemic uncertainty. The overall TNR (at 95% TPR)
reported by this technique is 15%.

These observations can be explained by the focus of different detection
techniques on measuring different forms of uncertainty. This motivates our
integrated OOD detection method.

##  3 Integrated OOD Detection Method

Complementary information about different OOD types can be used to detect a
wider range of OODs. Figure 4 shows the improvement in the TNR of the OOD
detector composed with information about different classes of OODs on the two
half-moons dataset. Non-conformity in the labels of the nearest neighbors
captures OODs in cluster C. Mahalanobis distance from the tied in-distribution
detects OODs in cluster A. Reconstruction error from the PCA of the 2 class
distributions captures OODs in cluster B. Softmax scores further strengthens
the OOD detection by reporting OODs in cluster C that are undetected by the
other three methods.

Figure 4: Complementary information about different types of OODs improves
detection. (Top-left) 15% TNR with non-conformance among the labels of the
nearest neighbors. (Top-right) Adding Mahalanobis distance over the tied in-
distribution improves TNR to 54.09%. (Bottom-left) Adding Class-wise PCA
further improves TNR to 95.91% TNR. (Bottom-right) Adding softmax score
further improves TNR to 99.55%. TPR is 95% in all the cases.

The integrated OOD detection approach, thus, uses the following attributes,
each specialized in detecting a specific type (or a combination of types) of
OODs:

  1. 1.

Mahalanobis distance from the in-distribution density estimate that considers
either tied (Lee et al., 2018) or class-wise covariance estimate. This
attribute captures the overall or class-conditional epistemic uncertainty of
an OOD. Our refinement to also use class-wise covariance significantly
improves detection of OODs when coupled with PCA approach described below.

  2. 2.

Conformance measure among the variance of the Annoy (Bernhardsson, 2018)
nearest neighbors calculated as the Mahalanobis distance of the input’s
conformance to the closest class conformance. Our experiments found this to be
very effective in capturing aleatoric uncertainty. This new attribute is a
fusion of nearest-neighbor and Mahalanobis distance methods in literature.

  3. 3.

Prediction confidence of the classifier as the maximum softmax score on the
perturbed input where the perturbation used is the same as ODIN approach
(Liang et al., 2017). This boosts the detection of high aleatoric uncertainty
by sharpening the class-wise distributions.

  4. 4.

Reconstruction error using top 40% of PCA components where the components are
obtained via class conditional PCA of the training data. This boosts the
detection of high class-wise epistemic uncertainty by eliminating irrelevant
features.

This fusion of attributes from existing state-of-the-art detection methods and
new attributes was found to be the most effective integrated appraoch capable
of detecting the different types of OODs. We evaluated it on several
benchmarks as discussed in Section 4 with ablation study in Appendix.

##  4 Experimental Results

Attributes forming the signature of the OOD detector used in the experiments
The signature of the OOD detector used in the experiments is the weighted sum
of four attributes, one from each of the following four categories:

  1. 1.

Distance from the in-distribution density estimate: We use mahalanobis
distance of the input with respect to the closest class conditional
distribution. The parameters of this distance are chosen from one of the
following two categories:

     * •

empirical class means and tied empirical covariance of training samples

     * •

empirical class means and empirical class covariance of training samples

  2. 2.

Reconstruction error: We perform class conditional PCA empirically from the
training samples. We use minimum reconstruction error of the input from the
top 40% eigen vectors of the class conditional eigen spaces.

  3. 3.

Prediction confidence of the classifier: We use maximum value of the
temperature scaled softmax scores ($S$) on the perturbed input. Perturbations
to the input ($x$) are made according to the following equation (Liang et al.,
2017)

$\widetilde{x}=x-\epsilon\text{sign}(-\nabla_{x}\text{log}S_{\hat{y}(x;T)})$
(1)

The values of the magnitude of noise ($\epsilon$) and the temperature scaling
parameter ($T$) are chosen from one of the following three categories:

     * •

$\epsilon=0$ and $T=0$

     * •

$\epsilon=0$ and $T=10$

     * •

$\epsilon=0.005$ and $T=10$

  4. 4.

Conformance measure among the nearest neighbors: We compute an m-dimensional
feature vector to capture the conformance among the input’s nearest neighbors
in the training samples, where m is the dimension of the input. We call this
m-dimensional feature vector as the conformance vector. The conformance vector
is calculated by taking the mean deviation along each dimension of the nearest
neighbors from the input. We hypothesize that this deviation for the in-
distribution samples would vary from the OODs due to aleatory uncertainty.

The value of the conformance measure is calculated by computing mahalanobis
distance of the input’s conformance vector to the closest class conformance
distribution. Similar to the distance for the in-distribution density
estimate, the parameters of this mahalanobis distance are chosen from the
following two categories:

     * •

empirical class means and tied empirical covariance on the conformance vectors
of the training samples

     * •

empirical class means and empirical class covariance on the conformance
vectors of the training samples

The value of the number of the nearest neighbors is chosen from the set
$\\{10,20,30,40,50\\}$ via validation. We used Annoy (Approximate Nearest
Neighbors Oh Yeah) (Bernhardsson, 2018) to compute the nearest neighbors.

The weights of the four attributes forming the signature of the OOD detector
are generated in the following manner. We use a small subset (1000 samples) of
both the in-distribution and the generated OOD data to train a binary
classifier using the logistic loss. The OOD data used to train the classifier
is generated by perturbing the in-distribution data using the Fast Gradient
Sign attack (FGSM) (Goodfellow et al., 2014). The trained classifier (or OOD
detector) is then evaluated on the real OOD dataset at the True Positive Rate
of 95%. The best result, in terms of the highest TNR on the validation dataset
(from the training phase of the OOD detector), from the twelve combinations of
the aforementioned sub-categories (one from each of the four attributes) are
then reported on the test (or real) OOD datasets.

Datasets and metrics. We evaluate the proposed integrated OOD detection on
benchmarks such as CIFAR10 (Krizhevsky et al., 2009) and SVHN (Netzer et al.,
2011). We consider standard metrics (Hendrycks & Gimpel, 2016; Liang et al.,
2017; Lee et al., 2018) such as the true negative rate (TNR) at 95% true
positive rate (TPR), the area under the receiver operating characteristic
curve (AUROC), area under precision recall curve (AUPR), and the detection
accuracy (DTACC) to evaluate our performance.

DNN-based classifier architectures. To demonstrate that the proposed approach
generalizes across various network architectures, we consider a wide range of
DNN models such as , ResNet (He et al., 2016), WideResNet (Zagoruyko &
Komodakis, 2016), and DenseNet (Huang et al., 2017).

Comparison with the state-of-the-art. We compare our approach with the three
state-of-the-art approaches: SPB (Hendrycks & Gimpel, 2016), ODIN (Liang et
al., 2017), and Mahalanobis (Lee et al., 2018). For the ODIN method, the
perturbation noise is chosen from the set {0, 0.0005, 0.001, 0.0014, 0.002,
0.0024, 0.005, 0.01, 0.05, 0.1, 0.2}, and the temperature $T$ is chosen from
the set {1, 10, 100, 1000}. These values are chosen from the validation set of
the adversarial samples of the in-distribution data generated by the FGSM
attack. For the Mahalanobis method, we consider their best results obtained
after feature ensemble and input preprocessing with the hyperparameters of
their OOD detector tuned on the in-distribution and adversarial samples
generated by the FGSM attack. The magnitude of the noise used in pre-
processing of the inputs is chosen from the set {0.0, 0.01, 0.005, 0.002,
0.0014, 0.001, 0.0005}.

CIFAR10. With CIFAR10 as in-distribution, we consider SVHN (Netzer et al.,
2011), Tiny-Imagenet (Deng et al., 2009), and LSUN (Yu et al., 2015) as the
OOD datasets. For CIFAR10, we consider two DNNs: ResNet50, and WideResNet.
Table 1 shows the results.

SVHN. With SVHN as in-distribution, we consider CIFAR10, Imagenet, and LSUN
and as the OOD datasets. For SVHN, we use the DenseNet classifier. Table 1
shows the results.

Table 1: Comparison of TNR, AUROC, DTACC, AUPR with SPB, ODIN and Mahalanobis methods In-dist | OOD Dataset | Method | TNR | AUROC | DTACC | AUPR
---|---|---|---|---|---|---
(model) |  |  |  |  |  |
CIFAR10 |  |  |  |  |  |
(ResNet50) |  |  |  |  |  |
| SVHN | SPB | 44.69 | 97.31 | 86.36 | 87.78
|  | ODIN | 63.57 | 93.53 | 86.36 | 87.58
|  | Mahalanobis | 72.89 | 91.53 | 85.39 | 73.80
|  | Ours | 85.90 | 95.14 | 90.66 | 80.01
| Imagenet | SPB | 42.06 | 90.8 | 84.36 | 92.6
|  | ODIN | 79.48 | 96.25 | 90.07 |  96.45
|  | Mahalanobis | 94.26 | 97.41 | 95.16 | 93.11
|  | Ours | 95.19 | 97.00 | 96.02 | 90.92
| LSUN | SPB | 48.37 | 92.78 | 86.97 | 94.45
|  | ODIN | 87.29 | 97.77 | 92.65 | 97.96
|  | Mahalanobis | 98.17 | 99.38 | 97.38 | 98.69
|  | Ours | 99.36 | 99.65 | 98.57 | 98.96
CIFAR10 |  |  |  |  |  |
(WideResNet) |  |  |  |  |  |
| SVHN | SPB | 45.46 | 90.10 | 82.91 | 82.52
|  | ODIN | 57.14 | 89.30 | 81.14 | 75.48
|  | Mahalanobis | 85.86 | 97.21 | 91.87 | 94.69
|  | Ours | 88.95 | 97.61 | 92.46 | 92.84
| LSUN | SPB | 52.64 | 92.89 | 86.81 | 94.13
|  | ODIN | 79.60 | 96.08 | 89.74 | 96.23
|  | Mahalanobis | 95.69 | 98.93 | 95.41 | 98.99
|  | Ours | 98.84 | 99.63 | 97.72 | 99.25
SVHN |  |  |  |  |  |
(DenseNet) |  |  |  |  |  |
| Imagenet | SPB | 79.79 | 94.78 | 90.21 | 97.2
|  | ODIN | 79.8 | 94.8 | 90.2 | 97.2
|  | Mahalanobis | 99.85 | 99.88 | 98.87 | 99.95
|  | Ours | 98.02 | 98.34 | 98.00 | 97.05
| LSUN | SPB | 77.12 | 94.13 | 89.14 | 96.96
|  | ODIN | 77.1 | 94.1 | 89.1 | 97.0
|  | Mahalanobis | 99.99 | 99.91 | 99.23 | 99.97
|  | Ours | 99.74 | 99.79 | 99.08 | 99.65
| CIFAR10 | SPB | 69.31 | 91.9 | 86.61 | 95.7
|  | ODIN | 69.3 | 91.9 | 86.6 | 95.7
|  | Mahalanobis | 97.03 | 98.92 | 96.11 | 99.61
|  | Ours | 94.87 | 98.41 | 94.97 | 98.76

Key observations. We do not consider pre-processing of the inputs in our
integrated OOD detector. Even without input pre-processing and with the
exception of CIFAR10 OOD dataset for SVHN in-distribution trained on DenseNet,
we could perform equally well (and even out-perform in most of the cases) as
the Mahalanobis method on its best results generated after pre-processing the
input.

We also consider a Subset-CIFAR100 as OODs for CIFAR10. Specifically, from the
CIFAR100 classes, we select sea, road, bee, and butterfly as OODs which are
visually similar to the ship, automobile, and bird classes in the CIFAR10,
respectively. Thus, there can be numerous OOD samples due to aleatoric and
class-conditional epistemic uncertainty which makes the OOD detection
challenging. Figure 5 shows the t-SNE (Maaten & Hinton, 2008) plot of the
penultimate features from the ResNet50 model trained on CIFAR10. We show 4
examples of OODs (2 due to epistemic and 2 due to aleatoric uncertainty) from
Subset-CIFAR100. These OODs were detected by our integrated approach but
missed by the Mahalanobis approach.

These observations justify the effectiveness of integrating multiple
attributes to detect OOD samples.

Additional experimental results in the appendix. We also compare the
performance of the integrated OOD detector with the SPB, ODIN and Mahalanobis
detector in supervised settings, as reported by the Mahalanobis method for OOD
detection (Lee et al., 2018). These results include experiments on CIFAR10,
SVHN and MNIST as in-distribution data and Imagenet,LSUN, SVHN (for CIFAR10),
CIFAR10 (for SVHN), KMNIST, and F-MNISTas OOD data across different DNN
architectures such as ResNet34, WideResNet,DenseNet, and LeNet5. All these
results, along with the ablation studies on OOD detectors with single
attributes are included in the Appendix. In almost all of the reported results
in the Appendix, our OOD detector could outperform the compared state-of-the-
art methods with improvements of even 2X higher TNR at 95% TPR in some cases.

Figure 5: t-SNE plot of the penultimate layer feature space of ResNet50
trained on CIFAR10. We show four OOD images from the SCIFAR100. OOD 1 and OOD
2 are far from the distributions of all classes and thus represent OODs due to
epistemic uncertainty. OOD 3 and OOD 4 are OODs due to aleatoric uncertainty
as they lie closer to two class distributions. Third OOD is closer to the cat
and frog classes of the ID and forth OOD is closer to the airplane and
automobile classes of the ID. Mahalanobis distance cannot detect these OODs
but our integrated approach can detect them.

##  5 Discussion and future work

Recent techniques propose refinement in the training process of the
classifiers for OOD detection. Some of these techniques include fine-tuning
the classifier’s training with an auxiliary cost function for OOD detection
(Hendrycks et al., 2019a; Liu et al., 2020). Other techniques make use of
self-supervised models for OOD detection (Tack et al., 2020; Hendrycks et al.,
2019b). We perform preliminary experiments to compare the performance of these
techniques with our integrated OOD detector that makes use of the feature
space of the pre-trained classifiers to distinguish in-distribution samples
from OODs. Our approach does not require modification of the training cost
function of the original task. These results are reported in the Appendix. We
consider making use of the feature space of in our OOD detection technique as
a promising prospective future work. Another direction of the future work is
to explore the score functions used in these refined training processes for
OOD detection (Liu et al., 2020; Hendrycks et al., 2019a; Tack et al., 2020;
Hendrycks et al., 2019b) as attributes (or categories of attributes) forming
the signature of the integrated OOD detector. Another avenue of future work is
to explore OOD generation techniques other than adversarial examples generated
by the FGSM attack for training of the integrated OOD detector.

##  6 Conclusion

We introduced a taxonomy of OODs and proposed an integrated approach to detect
different types of OODs. Our taxonomy classifies OOD on the nature of their
uncertainty and we demonstrated that no single state-of-the-art approach
detects all these OOD types. Motivated by this observation, we formulated an
integrated approach that fuses multiple attributes to target different types
of OODs. We have performed extensive experiments on a synthetic dataset and
several benchmark datasets (e.g., MNIST, CIFAR10, SVHN). Our experiments show
that our approach can accurately detect various types of OODs coming from a
wide range of OOD datasets such as KMNIST, Fashion-MNIST, SVHN, LSUN, and
Imagenet. We have shown that our approach generalizes over multiple DNN
architectures and performs robustly when the OOD samples are similar to in-
distribution data.

#### Acknowledgments

This work was supported by the Air Force Research Laboratory and the Defense
Advanced Research Projects Agency under DARPA Assured Autonomy under Contract
No. FA8750-19-C-0089 and Contract No. FA8750-18-C-0090, U.S. Army Research
Laboratory Cooperative Research Agreement W911NF-17-2-0196, U.S. National
Science Foundation(NSF) grants #1740079 and #1750009, and the Army Research
Office under Grant Number W911NF-20-1-0080. Any opinions, findings and
conclusions or recommendations expressed in this material are those of the
authors and do not necessarily reflect the views of the Air Force Research
Laboratory (AFRL), the Army Research Office (ARO), the Defense Advanced
Research Projects Agency (DARPA), or the Department of Defense, or the United
States Government.

## References

  * An & Cho (2015) Jinwon An and Sungzoon Cho.  Variational autoencoder based anomaly detection using reconstruction probability.  _Special Lecture on IE_ , 2(1):1–18, 2015.
  * Bernhardsson (2018) Erik Bernhardsson.  Annoy, 2018.  URL https://github.com/spotify/annoy.
  * Bojarski et al. (2016) Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_ , 2016.
  * Clanuwat et al. (2018) Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha.  Deep learning for classical japanese literature.  _arXiv preprint arXiv:1812.01718_ , 2018.
  * Coates et al. (2011) Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pp. 215–223, 2011.
  * De Fauw et al. (2018) Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O’Donoghue, Daniel Visentin, et al.  Clinically applicable deep learning for diagnosis and referral in retinal disease.  _Nature medicine_ , 24(9):1342–1350, 2018.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Gkioxari et al. (2015) Georgia Gkioxari, Ross Girshick, and Jitendra Malik.  Contextual action recognition with r* cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1080–1088, 2015.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Guo et al. (2017a) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  _arXiv preprint arXiv:1706.04599_ , 2017a.
  * Guo et al. (2017b) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  _arXiv preprint arXiv:1706.04599_ , 2017b.
  * Hannun et al. (2014) Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al.  Deep speech: Scaling up end-to-end speech recognition.  _arXiv preprint arXiv:1412.5567_ , 2014.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hendrycks & Gimpel (2016) Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_ , 2016.
  * Hendrycks et al. (2019a) Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich.  Deep anomaly detection with outlier exposure.  _In International Conference on Learning Representations_ , 2019a.
  * Hendrycks et al. (2019b) Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song.  Using self-supervised learning can improve model robustness and uncertainty.  In _Advances in Neural Information Processing Systems_ , pp. 15663–15674, 2019b.
  * Hoffmann (2007) Heiko Hoffmann.  Kernel pca for novelty detection.  _Pattern recognition_ , 40(3):863–874, 2007.
  * Hora (1996) Stephen C Hora.  Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management.  _Reliability Engineering & System Safety_, 54(2-3):217–223, 1996.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4700–4708, 2017.
  * Hüllermeier & Waegeman (2019) Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: A tutorial introduction.  _arXiv preprint arXiv:1910.09457_ , 2019.
  * Jiang et al. (2018) Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta.  To trust or not to trust a classifier.  In _Advances in neural information processing systems_ , pp. 5541–5552, 2018.
  * Julian & Kochenderfer (2017) Kyle D Julian and Mykel J Kochenderfer.  Neural network guidance for uavs.  In _AIAA Guidance, Navigation, and Control Conference_ , pp. 1743, 2017.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lee et al. (2017) Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  _arXiv preprint arXiv:1711.09325_ , 2017.
  * Lee et al. (2018) Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In _Advances in Neural Information Processing Systems_ , pp. 7167–7177, 2018.
  * Liang et al. (2017) Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _arXiv preprint arXiv:1706.02690_ , 2017.
  * Liu et al. (2020) Weitang Liu, Xiaoyun Wang, John Owens, and Sharon Yixuan Li.  Energy-based out-of-distribution detection.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Maaten & Hinton (2008) Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(Nov):2579–2605, 2008.
  * Majumder et al. (2017) Navonil Majumder, Soujanya Poria, Alexander Gelbukh, and Erik Cambria.  Deep learning-based document modeling for personality detection from text.  _IEEE Intelligent Systems_ , 32(2):74–79, 2017\.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Papernot & McDaniel (2018) Nicolas Papernot and Patrick McDaniel.  Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning.  _arXiv preprint arXiv:1803.04765_ , 2018.
  * Ren et al. (2019) Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan.  Likelihood ratios for out-of-distribution detection.  In _Advances in Neural Information Processing Systems_ , pp. 14707–14718, 2019.
  * Tack et al. (2020) Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * Yu et al. (2015) Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Zagoruyko & Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ , 2016.

##  Appendix A Appendix

###  A.1 Defining OODs due to epistemic and aleatoric uncertainty

In general, let there be $k$ classes $c_{1},c_{2},\ldots,c_{k}$ and the
distribution of training data for each class is $p(x|c_{i})$. The overall
training distribution is denoted by $p(x)$. Now, given a new input $\hat{x}$
to the trained DNN model $M$, let $\hat{c}=M(\hat{x})$ denote the predicted
class. The flowchart in Figure 6 shows different sources of uncertainty that
could make $\hat{x}$ an OOD.

Figure 6: OODs due to High Epistemic and Aleatoric Uncertainty

###  A.2 Additional experimental results

We first present preliminary results for comparison with the OOD detection
techniques based on fine-tuning of the classifiers (Hendrycks et al., 2019a;
Liu et al., 2020; Tack et al., 2020; Hendrycks et al., 2019b). We then present
our results on various vision datasets and different architectures of the pre-
trained DNN based classifiers for these datasets in comparison to the ODIN,
the Mahalanobis and the SPB methods in supervised settings. Finally, we then
report results from the ablation study on OOD detection with individual
attributes and compare it with our integrated approach on OOD detection.

####  A.2.1 Comparison with the OOD detection techniques based on refinement
of the training process for classifiers

Recent techniques propose refinement in the training process of the
classifiers for OOD detection. Some of these techniques include fine-tuning
the training of classifiers with a trainable cost function for OOD detection
(Hendrycks et al., 2019a; Liu et al., 2020), self-supervised training of the
classifiers to enhance OOD detection (Tack et al., 2020; Hendrycks et al.,
2019b) etc.

We perform preliminary experiments to compare the performance of these
techniques with our integrated OOD detector that uses features of the pre-
trained classifiers to distinguish in-distribution samples from OODs. Table 2
compares TNR (at 95% TPR), AUROC and AUPR for the energy based OOD detector
(Liu et al., 2020) and our integrated OOD detector on CIFAR10 with pre-trained
WideResNet model. The integrated OOD detector was trained on in-distribution
and adversarial samples generated by FGSM attack. Table 3 compares the results
of the WideResNet model trained on CIFAR10 and fine-tuned with the outlier
exposure from the 80 Million Tiny Images with our OOD detector that uses
features from the pre-trained WideResNet model trained on CIFAR10. Since 80
Million Tiny Images dataset is no longer available for use, we used a small
subset of ImageNet (treated as OOD dataset for CIFAR10 and SVHN datasets (Lee
et al., 2018)) for generating OODs for training of the integrated OOD
detector. Table 4 compares the OOD detection performance of the self-
supervised training based OOD detector with our method. We trained our OOD
detector with the in-distribution CIFAR10 as in-distribution samples and
adversarial samples generated by FGSM attack from the test dataset of CIFAR10
as OODs. The trained OOD detector was then tested on LSUN as OODs and the
results are reported in Table 4. With ResNet-50 as the classifier for CIFAR10,
we trained our OOD detector with the in-distribution CIFAR10 as in-
distribution samples and adversarial samples generated by FGSM from the test
dataset of CIFAR10 as OODs. The trained OOD detector was then tested on SVHN
as OODs and these results are compared with the contrastive based learning for
OOD detection (Tack et al., 2020) in table 5.

Table 2: Results with Energy based OOD detector (Liu et al., 2020) / Our method. In-dist |  OOD |  TNR |  AUROC |  AUPR
---|---|---|---|---
(model) |  dataset |  (TPR=95%) |  |
CIFAR10 |  SVHN |  64.41 / 88.95 |  90.96 / 97.61 |  97.64 / 92.84
(WideResNet) |  LSUN |  72.42 / 98.84 |  94.24 / 99.63 |  98.67 / 99.25
Table 3: Results with Outlier Exposure based OOD detector (Hendrycks et al., 2019a) / Our method. In-dist |  OOD |  TNR |  AUROC |  AUPR
---|---|---|---|---
(model) |  dataset |  (TPR=95%) |  |
CIFAR10 |  SVHN |  95.64 / 92.53 |  98.63 / 98.56 |  99.74 / 96.62
(WideResNet) |  |  |  |
Table 4: Results with self-supervised learning based OOD detector (Hendrycks et al., 2019b) / Our method. In-dist |  OOD |  TNR |  AUROC |  DTACC
---|---|---|---|---
(model) |  dataset |  (TPR=95%) |  |
CIFAR10 |  LSUN |  71.3 / 98.84 |  93.2 / 99.63 |  71.0 / 97.72
(WideResNet) |  |  |  |
Table 5: Results with contrastive learning based OOD detector (Tack et al., 2020) / Our method. In-dist |  OOD |  TNR |  AUROC |  DTACC |  AUPR
---|---|---|---|---|---
(model) |  dataset |  (TPR=95%) |  |  |
CIFAR10 |  SVHN |  97.2 / 82.88 |  99.5 /96.98 |  96.7 / 91.74 |  99.6 / 94.71
(ResNet50) |  |  |  |  |

####  A.2.2 Comparison with the state-of-the-art OOD detection methods in
supervised settings on pre-trained classifiers

We compare our results with the state-of-the-art methods in supervised
settings, as reported by the Mahalanobis method for OOD detection (Lee et al.,
2018). In supervised settings, a small subset of the real OOD dataset is used
in the training of the OOD detector.

Datasets and metrics. We evaluate the proposed integrated OOD detection on
benchmarks such as MNIST (LeCun et al., 1998), CIFAR10 (Krizhevsky et al.,
2009), and SVHN (Netzer et al., 2011). We consider standard metrics (Hendrycks
& Gimpel, 2016; Liang et al., 2017; Lee et al., 2018) such as the true
negative rate (TNR) at 95% true positive rate (TPR), the area under the
receiver operating characteristic curve (AUROC), area under precision recall
curve (AUPR) with both in-distribution and OODs as positive samples (AUPR IN
and AUPR OUT respectively), and the detection accuracy (DTACC) to evaluate our
performance.

DNN-based classifier architectures. To demonstrate that the proposed approach
generalizes across various network architectures, we consider a wide range of
DNN models such as Lenet (LeCun et al., 1998) , ResNet (He et al., 2016) , and
DenseNet (Huang et al., 2017).

Comparison with the state-of-the-art. We compare our approach with the three
state-of-the-art approaches: SPB (Hendrycks & Gimpel, 2016), ODIN (Liang et
al., 2017) and Mahalanobis (Lee et al., 2018). Since, these experiments are
performed in supervised settings, we fix $T=10$ and $\epsilon=0.005$ for
generating results from the ODIN method. For Mahalanobis distance, we consider
the distance in the penultimate layer feature space as well as features from
all the layers of the DNN without preprocessing of the input in either
settings.

MNIST. With MNIST as in-distribution, we consider KMNIST (Clanuwat et al.,
2018) and Fashion-MNIST(F-MNIST) (Xiao et al., 2017) as OOD datasets. For
MNIST, we use the LeNet5 (LeCun et al., 1998) DNN. Results in terms of TNR (at
95% TPR), AUROC, and DTACC are reported in tables 6, 7, and
LABEL:table:comp_baseline. Table 6 shows the results with the features from
the penultimate layer in comparison to the ODIN and Mahalanobis methods. Table
7 shows the results with the features from all the layers in comparison to the
Mahalanobis method. Table LABEL:table:comp_baseline shows the results with the
features from the penultimate layer in comparison to the SPB method. In all
these settings, our approach outperforms the state-of-the-art approaches for
both the OOD datasets. Results in comparison to AUPR IN and AUPR OUT are shown
in table 9. Here also, our technique out-performs all the three OOD detectors
on all the test cases.

CIFAR10. With CIFAR10 as in-distribution, we consider STL10 (Coates et al.,
2011), SVHN (Netzer et al., 2011), Imagenet (Deng et al., 2009), LSUN (Yu et
al., 2015), and a subset of CIFAR100 (SCIFAR100) (Krizhevsky et al., 2009) as
OOD datasets. For CIFAR10, we consider three DNNs: DenseNet, ResNet34, and
ResNet50. Results in terms of TNR (at 95% TPR), AUROC, and DTACC are reported
in tables 6, 7, and LABEL:table:comp_baseline. Table 6 shows the results with
the features from the penultimate layer in comparison to the ODIN and
Mahalanobis methods. Table 7 shows the results with the features from all the
layers in comparison to the Mahalanobis method. Table
LABEL:table:comp_baseline shows the results with the features from the
penultimate layer in comparison to the SPB method. Results in comparison to
AUPR IN and AUPR OUT are shown in tables 10, 11, and 12. Here also, the
integrated OOD detection technique could out-perform the other three detectors
on most of the test cases.

Note that images from STL10 and the subset of CIFAR100 are quite similar to
CIFAR10 images. Furthermore, from the CIFAR100 classes, we select sea, road,
bee, and butterfly as OODs which are visually similar to the ship, automobile,
and bird classes in the CIFAR10, respectively.

SVHN. With SVHN as in-distribution, we consider STL10, CIFAR10, Imagenet, LSUN
and, SCIFAR100 as OOD datasets. For SVHN, we consider two DNNs: DenseNet and
ResNet34. Results in terms of TNR (at 95% TPR), AUROC, and DTACC are reported
in tables 6, 7, and LABEL:table:comp_baseline. Table 6 shows the results with
the features from the penultimate layer in comparison to the ODIN and
Mahalanobis methods. Table 7 shows the results with the features from all the
layers in comparison to the Mahalanobis method. Table
LABEL:table:comp_baseline shows the results with the features from the
penultimate layer in comparison to the SPB method. Results in comparison to
AUPR IN and AUPR OUT are shown in tables 13, and 14. Here also, the integrated
OOD detection technique could out-perform the other three detectors on most of
the test cases.

Key observations. As shown in Table 6 and Table 7, and Table
LABEL:table:comp_baseline, our approach outperforms the state-of-the-art on
all three datasets and with various DNN architectures. On CIFAR10, in terms of
the TNR metric, our approach with Resnet50 outperforms Mahalanobis by 56% when
SVHN is OOD and our approach with Resnet34 outperforms ODIN by 36% when LSUN
is OOD.

While considering STL10 and Subset-CIFAR100 as OODs for CIFAR10, the images
from both these datasets are quite similar to CIFAR-10 images. Thus, there can
be numerous OOD samples due to aleatoric and class-conditional epistemic
uncertainty which makes detection challenging. Although our performance is low
on the STL10 dataset, it still outperforms the state-of-the-art. For instance,
the proposed approach achieves a 27% better TNR score than the Mahalanobis
using ResNet50. On SVHN, in terms of the TNR metric, our approach outperforms
ODIN and Mahalanobis by 63% and 13%, respectively on SCIFAR100 using ResNet34.
The above observations justify the effectiveness of integrating multiple
attributes to detect OOD samples.

Table 6: Results with ODIN/Mahalanobis/Our method. The best results are highlighted. In-dist |  OOD |  TNR |  AUROC |  DTACC
---|---|---|---|---
(model) |  dataset |  (TPR=95%) |  |
MNIST |  KMNIST |  67.72 / 80.52 / 91.82 |  92.98 / 96.53 / 98.3 |  85.99 / 90.82 / 94.01
(LeNet5) |  F-MNIST |  58.47 / 63.33 / 74.49 |  90.76 / 94.11 / 95.55 |  83.21 / 87.76 / 90.98
CIFAR10 |  STL10 |  8.89 / 9.23 / 15.29 |  56.31 / 62.16 / 63.96 |  55.38 / 59.57 / 61.02
(DenseNet) |  SVHN |  69.96 / 83.63 / 91.29 |  92.02 / 97.1 / 98.38 |  84.1 / 91.26 / 93.28
|  Imagenet |  61.03 / 49.33 / 77.81 |  91.4 / 90.32 / 95.98 |  83.85 / 83.08 / 89.74
|  LSUN |  71.89 / 46.63 / 84.34 |  94.37 / 91.18 / 97.27 |  87.72 / 84.93 / 92.1
|  SCIFAR100 |  35.06 / 20.33 / 38.78 |  80.18 / 80.4 / 90.58 |  72.58 / 74.15 / 85.35
CIFAR10 |  STL10 |  10.63 / 13.9 / 17.4 |  61.56 / 66.47 / 67.52 |  59.22 / 62.75 / 63.7
(ResNet34) |  SVHN |  72.85 / 53.16 / 88.2 |  93.85 / 93.85/ 97.69 |  85.4 / 89.173 / 92.14
|  Imagenet |  46.54 / 68.41 / 74.53 |  90.45 / 95.02 / 95.73 |  83.06 / 88.63 / 89.73
|  LSUN |  45.16 / 77.53 / 81.23 |  89.63 / 96.51 / 96.87 |  81.83 / 90.64 / 91.19
|  SCIFAR100 |  37 / 38.39 / 61.11 |  86.13 / 88.86 / 94.74 |  78.5 / 82.51 / 90.53
CIFAR10 |  STL10 |  12.19 / 10.33 / 16 |  60.29 / 61.95 / 66.39 |  58.57 / 59.36 / 62.28
(ResNet50) |  SVHN |  86.61 / 34.49/ 91.06 |  84.41 / 98.19 / 91.98 |  91.25 / 76.72 / 93.2
|  Imagenet |  73.23 / 29.48 / 75.96 |  94.91 / 84.3 / 95.79 |  88.23 / 77.19 / 89.26
|  LSUN |  80.72 / 32.18 / 81.38 |  96.51 / 87.09 / 96.93 |  90.59 / 80.07 / 91.79
|  SCIFAR100 |  47.44 / 21.06 / 48.33 |  86.16 / 77.42/ 92.98 |  78.69 / 71.43 / 88.27
SVHN |  STL10 |  45.91 / 81.66 / 87.76 |  77.6 / 96.97 / 97.63 |  72.62 / 92.29 / 93.35
(DenseNet) |  CIFAR10 |  37.23 / 80.82 / 86.42 |  73.14 / 96.8 / 97.37 |  68.92 / 92.27 / 92.86
|  Imagenet |  62.76 / 85.44 / 93.44 |  85.41 / 97.29 / 98.38 |  79.94 / 93.39 / 94.53
|  LSUN |  62.91 / 76.87 / 89.73 |  86.06 / 96.37 / 97.73 |  80.04 / 92.43 / 93.55
|  SCIFAR100 |  48.17 / 86.06 / 96.72 |  78.94 / 97.43 / 98.24 |  73.72 / 93.02 / 96.26
SVHN |  STL10 |  35.14 / 85.3 / 90.9 |  67.05 / 97.19 / 97.76 |  66.19 / 93.41 / 94.34
(ResNet34) |  CIFAR10 |  32.6 / 85.03 / 90.34 |  66.75 / 97.05 / 97.64 |  65.37 / 93.15 / 94.29
|  Imagenet |  41.8 / 84.46 / 89.82 |  73 / 96.95 / 97.59 |  69.84 / 93.14 / 94.32
|  LSUN |  35.92 / 78.38 / 85.46 |  68.6 / 96.17 / 97.09 |  66.75 / 91.98 / 93.17
|  SCIFAR100 |  36.67 / 86.61 / 99.61 |  68.01 / 97.3 / 98.47 |  67.26 / 93.6 / 97.36
Table 7: Results with Mahalanobis/Our method with feature ensemble. The best results are highlighted. In-dist |  OOD |  TNR |  AUROC |  DTACC
---|---|---|---|---
(model) |  dataset |  (TPR=95%) |  |
MNIST |  KMNIST |  96 / 98.8 |  99.19 / 99.65 |  95.56 / 97.3
(LeNet5) |  F-MNIST |  99.9 / 99.98 |  99.95 / 99.96 |  98.98 / 99.17
CIFAR10 |  STL10 |  16.44 / 22.94 |  72.4 / 75.23 |  66.69 / 69.31
(DenseNet) |  SVHN |  92.4 / 98.23 |  98.41 / 99.49 |  93.97 / 97.02
|  Imagenet |  96.46 / 98.8 |  99.16/ 99.63 |  95.74 / 97.55
|  LSUN |  98.09 / 99.64 |  99.47 / 99.85 |  96.76 / 98.45
|  SCIFAR100 |  27.33 / 46.5 |  83.7 / 92.17 |  77.08 / 86.69
CIFAR10 |  STL10 |  26.14 / 29.8 |  76.23 / 76.46 |  70.33 / 70.94
(ResNet34) |  SVHN |  91.53 / 97.07 |  98.4 / 99.32 |  93.63 / 96.27
|  Imagenet |  97.09 / 98.11 |  99.47 / 99.58 |  96.31 / 96.91
|  LSUN |  98.67 / 99.41 |  99.71 / 99.81 |  97.56 / 98.14
|  SCIFAR100 |  38.89 / 62.78 |  88.8 / 94.23 |  82.14 / 90.16
CIFAR10 |  STL10 |  26.36 / 30.83 |  73.74 / 76.73 |  67.37 / 70.4
(ResNet50) |  SVHN |  84.44 / 98.59 |  96.56 / 99.65 |  90.63 / 97.43
|  Imagenet |  97.87 / 99.46 |  99.58 / 99.84 |  97.09 / 98.22
|  LSUN |  99.21 / 99.83 |  99.64 / 99.91 |  98.39 /99.21
|  SCIFAR100 |  29.33 / 55 |  80.26 / 91.48 |  74.51 / 86.42
SVHN |  STL10 |  97.31 / 98.76 |  99.14 / 99.47 |  96.23 / 97.24
(DenseNet) |  CIFAR10 |  96.36 / 97.64 |  98.8 / 99.16 |  95.7 / 96.34
|  Imagenet |  99.89 / 99.82 |  99.88 / 99.9 |  98.85 / 98.95
|  LSUN |  99.99 / 99.97 |  99.91 / 99.91 |  99.26 /99.18
|  SCIFAR100 |  99.33 / 100 |  99.53 / 99.78 |  97.89 / 98.95
SVHN |  STL10 |  98.44 / 98.88 |  99.31 / 99.52 |  96.91 / 97.4
(ResNet34) |  CIFAR10 |  98.44 / 98.88 |  99.31 / 99.52 |  96.91 / 97.4
|  Imagenet |  99.83 / 99.87 |  99.85 / 99.91 |  99.07 / 99.07
|  LSUN |  99.87 / 99.99 |  99.83 / 99.95 |  99.5 / 99.47
|  SCIFAR100 |  99.83 / 100 |  99.72 / 99.91 |  98.33/ 99.56
Table 8: Experimental results with SPB/Our method. The best results are highlighted. In-dist | OOD | TNR | AUROC | DTACC
---|---|---|---|---
(model) | dataset | (TPR=95%) |  |
MNIST | KMNIST | 69.33 / 91.82 | 93.24 / 98.3 | 86.88 / 94.01
(LeNet5) | F-MNIST | 52.69 / 74.49 | 89.19 / 95.55 | 82.77 / 90.98
CIFAR10 | STL10 |  15.64 / 15.29 |  64.15 / 63.96 |  62.12 / 61.02
(DenseNet) | SVHN | 39.22 / 91.29 | 88.24 / 98.38 | 82.41 / 93.28
| Imagenet | 40.13 / 77.81 | 89.3 / 95.98 | 82.67 / 89.74
| LSUN | 48.38 / 84.34 | 92.14 / 97.27 | 86.22 / 92.1
| SCIFAR100 | 34.11 / 38.78 | 85.53 / 90.58 | 79.18 / 85.35
CIFAR10 | STL10 | 14.9 / 17.4 | 65.88 / 67.52 | 62.85 / 63.7
(ResNet34) | SVHN | 32.47 / 88.2 | 89.88 / 97.69 | 85.06 / 92.14
| Imagenet | 44.72 / 74.53 | 91.02 / 95.73 | 85.05 / 89.73
| LSUN | 45.44 / 81.23 | 91.04 / 96.87 | 85.26 / 91.19
| SCIFAR100 | 38.17 / 61.11 | 88.91 / 94.74 | 82.34 / 90.53
CIFAR10 | STL10 | 15.33 / 16 | 66.68 / 66.39 | 63.47 / 62.28
(ResNet50) | SVHN | 44.69 / 91.06 | 97.31 / 91.98 | 86.36 /93.2
| Imagenet | 42.06 / 75.96 | 90.8 /95.79 | 84.36 / 89.26
| LSUN | 48.37 / 81.38 | 92.78 / 96.93 | 86.97 / 91.79
| SCIFAR100 | 36.39 / 48.33 | 89.09 / 92.98 | 83.37 / 88.27
SVHN | STL10 | 72.87 / 87.76 | 92.79 / 97.63 | 87.76 / 93.35
(DenseNet) | CIFAR10 | 69.31 / 86.42 | 91.9 / 97.37 | 86.61 / 92.86
| Imagenet | 79.79 / 93.44 | 94.78 / 98.38 | 90.21 / 94.53
| LSUN | 77.12 / 89.73 | 94.13 / 97.73 | 89.14 / 93.55
| SCIFAR100 | 76.94 / 96.72 | 94.18 / 98.24 | 89.57 / 96.26
SVHN | STL10 | 79.57 / 99.59 | 93.84 / 99.72 | 90.83 / 98.06
(ResNet34) | CIFAR10 | 78.26 / 90.34 | 92.92 / 97.64 | 90.03 / 94.29
| Imagenet | 79.02 / 89.82 | 93.51 / 97.59 | 90.44 / 94.32
| LSUN | 74.29 / 85.46 | 91.58 / 97.09 | 88.96 / 93.17
| SCIFAR100 | 81.28 / 99.61 | 94.62 /98.47 | 91.48 / 97.36
Table 9: Experimental Results with MNIST on Lenet5 for AUPR IN and AUPR OUT.
The best results are highlighted. OOD Dataset | Layer | Method | AUPR IN | AUPR OUT
---|---|---|---|---
| Penultimate |  |  |
KMNIST |  | Baseline | 92.47 | 92.41
|  | ODIN | 92.65 | 92.69
|  | Mahalanobis | 96.69 | 96.2
|  | Ours | 98.48 | 98.13
Fashion-MNIST |  | Baseline | 87.98 | 87.89
|  | ODIN | 90.94 | 89.99
|  | Mahalanobis | 95.24 | 91.94
|  | Ours | 96.64 | 92.96
| All |  |  |
KMNIST |  | Mahalanobis | 99.22 | 99.18
|  | Ours | 99.67 | 99.64
Fashion-MNIST |  | Mahalanobis | 99.95 | 99.94
|  | Ours | 99.96 | 99.96
Table 10: Experimental Results with CIFAR10 on DenseNet for AUPR IN and AUPR
OUT.
The best results are highlighted. OOD Dataset | Layer | Method | AUPR IN | AUPR OUT
---|---|---|---|---
| Penultimate |  |  |
STL10 |  | Baseline | 64.55 | 59.37
|  | ODIN | 60.24 | 50.17
|  | Mahalanobis | 65.86 | 53.87
|  | Ours | 66.01 | 58.5
SVHN |  | Baseline | 74.53 | 94.09
|  | ODIN | 80.49 | 97.05
|  | Mahalanobis | 94.13 | 98.78
|  | Ours | 96.26 | 99.37
Imagenet |  | Baseline | 90.88 | 86.74
|  | ODIN | 91.32 | 90.55
|  | Mahalanobis | 91.32 | 88.6
|  | Ours | 96.22 | 95.68
LSUN |  | Baseline | 93.68 | 89.83
|  | ODIN | 94.65 | 93.39
|  | Mahalanobis | 92.71 | 87.74
|  | Ours | 97.6 | 96.74
Subset CIFAR100 |  | Baseline | 96.65 | 50.08
|  | ODIN | 95.14 | 47.64
|  | Mahalanobis | 95.68 | 37.86
|  | Ours | 98.18 | 54.83
| All |  |  |
STL10 |  | Mahalanobis | 77.21 | 63.45
|  | Ours | 78.29 | 68.14
SVHN |  | Mahalanobis | 96.72 | 99.31
|  | Ours | 98.57 | 99.81
Imagenet |  | Mahalanobis | 99.19 | 99.13
|  | Ours | 99.62 | 99.54
LSUN |  | Mahalanobis | 99.49 | 99.45
|  | Ours | 99.82 | 99.85
Subset CIFAR100 |  | Mahalanobis | 96.58 | 42.41
|  | Ours | 98.53 | 59.7
Table 11: Experimental Results with CIFAR10 on ResNet34 for AUPR IN and AUPR
OUT.
The best results are highlighted. OOD Dataset | Layer | Method | AUIN | AUOUT
---|---|---|---|---
| Penultimate |  |  |
STL10 |  | Baseline | 67.17 | 59.74
|  | ODIN | 64.22 | 53.83
|  | Mahalanobis | 68.48 | 59.47
|  | Ours | 68.78 | 61.52
SVHN |  | Baseline | 85.4 | 93.96
|  | ODIN | 86.46 | 97.55
|  | Mahalanobis | 91.19 | 96.14
|  | Ours | 94.7 | 99.1
Imagenet |  | Baseline | 92.49 | 88.4
|  | ODIN | 92.11 | 87.46
|  | Mahalanobis | 95.77 | 94.02
|  | Ours | 96.32 | 94.99
LSUN |  | Baseline | 92.45 | 88.55
|  | ODIN | 91.58 | 86.5
|  | Mahalanobis | 97.08 | 95.78
|  | Ours | 97.36 | 96.29
Subset CIFAR100 |  | Baseline | 97.77 | 55.62
|  | ODIN | 97.05 | 51.57
|  | Mahalanobis | 97.71 | 54.11
|  | Ours | 99.06 | 64.53
| All |  |  |
STL10 |  | Mahalanobis | 77.59 | 71.15
|  | Ours | 77.32 | 72.38
SVHN |  | Mahalanobis | 96.46 | 99.37
|  | Ours | 98.37 | 99.73
Imagenet |  | Mahalanobis | 99.48 | 99.48
|  | Ours | 99.59 | 99.58
LSUN |  | Mahalanobis | 99.71 | 99.71
|  | Ours | 99.8 | 99.82
Subset CIFAR100 |  | Mahalanobis | 97.75 | 52.28
|  | Ours | 98.74 | 65.99
Table 12: Experimental Results with CIFAR10 as on ResNet50 for AUPR IN and
AUPR OUT.
The best results are highlighted. OOD Dataset | Layer | Method | AUIN | AUOUT
---|---|---|---|---
| Penultimate |  |  |
STL10 |  | Baseline | 67.47 | 60.83
|  | ODIN | 62.79 | 55.04
|  | Mahalanobis | 65.14 | 54.43
|  | Ours | 68.54 | 59.75
SVHN |  | Baseline | 87.78 | 95.61
|  | ODIN | 93.17 | 99.03
|  | Mahalanobis | 71.88 | 92.54
|  | Ours | 95.38 | 99.34
Imagenet |  | Baseline | 92.6 | 87.98
|  | ODIN | 95.16 | 94.45
|  | Mahalanobis | 86.14 | 80.6
|  | Ours | 96.22 | 95.23
LSUN |  | Baseline | 94.45 | 90.41
|  | ODIN | 96.9 | 96.01
|  | Mahalanobis | 89.34 | 82.87
|  | Ours | 97.53 | 96.03
Subset CIFAR100 |  | Baseline | 97.72 | 55.29
|  | ODIN | 96.67 | 60.62
|  | Mahalanobis | 94.49 | 36.12
|  | Ours | 98.72 | 59.3
| All |  |  |
STL10 |  | Mahalanobis | 75.6 | 69.32
|  | Ours | 77.79 | 73.22
SVHN |  | Mahalanobis | 91.89 | 98.58
|  | Ours | 99.12 | 99.86
Imagenet |  | Mahalanobis | 99.56 | 99.6
|  | Ours | 99.84 | 99.84
LSUN |  | Mahalanobis | 98.91 | 99.75
|  | Ours | 99.72 | 99.93
Subset CIFAR100 |  | Mahalanobis | 94.1 | 42.96
|  | Ours | 97.54 | 64.12
Table 13: Experimental Results with SVHN as on DenseNet for AUPR IN and AUPR
OUT.
The best results are highlighted. OOD Dataset | Layer | Method | AUIN | AUOUT
---|---|---|---|---
| Penultimate |  |  |
STL10 |  | Baseline | 97.01 | 82.02
|  | ODIN | 89.18 | 63.71
|  | Mahalanobis | 99.2 | 87.32
|  | Ours | 99.36 | 90.31
CIFAR10 |  | Baseline | 95.7 | 82.8
|  | ODIN | 84.32 | 60.32
|  | Mahalanobis | 98.94 | 88.91
|  | Ours | 99.09 | 91.59
Imagenet |  | Baseline | 97.2 | 88.42
|  | ODIN | 90.95 | 79.59
|  | Mahalanobis | 99.12 | 90.22
|  | Ours | 99.4 | 95.15
LSUN |  | Baseline | 96.96 | 87.44
|  | ODIN | 92.03 | 79.98
|  | Mahalanobis | 98.84 | 85.79
|  | Ours | 99.17 | 92.92
Subset CIFAR100 |  | Baseline | 99.39 | 63.21
|  | ODIN | 97.24 | 45.23
|  | Mahalanobis | 99.82 | 72.35
|  | Ours | 99.88 | 68.25
| All |  |  |
STL10 |  | Mahalanobis | 99.75 | 96.51
|  | Ours | 99.77 | 98.18
CIFAR10 |  | Mahalanobis | 99.6 | 95.39
|  | Ours | 99.69 | 97.21
Imagenet |  | Mahalanobis | 99.96 | 99.59
|  | Ours | 99.96 | 99.74
LSUN |  | Mahalanobis | 99.97 | 99.7
|  | Ours | 99.95 | 99.74
Subset CIFAR100 |  | Mahalanobis | 99.97 | 91.41
|  | Ours | 99.98 | 94.54
Table 14: Experimental Results with SVHN as on ResNet34 for AUPR IN and AUPR
OUT.
The best results are highlighted. OOD Dataset | Layer | Method | AUIN | AUOUT
---|---|---|---|---
| Penultimate |  |  |
STL10 |  | Baseline | 96.63 | 84.15
|  | ODIN | 84.03 | 47.26
|  | Mahalanobis | 99.89 | 97.5
|  | Ours | 99.93 | 98.2
CIFAR10 |  | Baseline | 95.06 | 85.66
|  | ODIN | 80.69 | 50.49
|  | Mahalanobis | 99.04 | 88.62
|  | Ours | 99.17 | 91.17
Imagenet |  | Baseline | 95.68 | 86.18
|  | ODIN | 84.62 | 58.28
|  | Mahalanobis | 99 | 88.39
|  | Ours | 99.19 | 90.77
LSUN |  | Baseline | 94.19 | 83.95
|  | ODIN | 82.37 | 53.12
|  | Mahalanobis | 98.73 | 85.11
|  | Ours | 99.03 | 89.03
Subset CIFAR100 |  | Baseline | 99.35 | 64.38
|  | ODIN | 95.57 | 23.04
|  | Mahalanobis | 99.81 | 64.4
|  | Ours | 99.89 | 67.9
| All |  |  |
STL10 |  | Mahalanobis | 99.7 | 97.03
|  | Ours | 99.84 | 97.86
CIFAR10 |  | Mahalanobis | 99.7 | 97.03
|  | Ours | 99.84 | 97.86
Imagenet |  | Mahalanobis | 99.86 | 99.14
|  | Ours | 99.93 | 99.62
LSUN |  | Mahalanobis | 99.82 | 98.85
|  | Ours | 99.98 | 99.64
Subset CIFAR100 |  | Mahalanobis | 99.98 | 93.59
|  | Ours | 99.99 | 95.75

####  A.2.3 Ablation study

We report ablation study on OOD detection with individual attributes and
compare it with our integrated approach on the penultimate feature space of
the classifier in the supervised settings as described in the previous
section. We call the OOD detector with Mahalanobis distance estimated on class
mean and tied covariance (Lee et al., 2018) as Mahala-Tied. Detector based on
Mahalanobis distance estimated on class mean and class covariance is referred
as Mahala-Class. Similarly conformance among the K-nearest neighbors (KNN)
measured by Mahala-Tied and Mahala-Class is referred as KNN-Tied and KNN-Class
respectively in these experiments. Results for this study on CIFAR10 with
DenseNet architecture, SVHN with DenseNet and ResNet34 architectures are shown
in Tables LABEL:table:ablation-cifar, 16 and 17 respectively.

The integrated approach could out-perform all the single attribute based OOD
detector in all the tested cases due to detection of diverse OODs. An
important observation made from these experiments is that the performance of
the single attribute based methods could depend on the architecture of the
classifier. For example, while the performance of PCA was really bad in case
of DenseNet (for both CIFAR10 and SVHN) as compared to all other methods, it
could out-perform all but the integrated approach for SVHN on ResNet34.

Table 15: Ablation study with CIFAR10 on DenseNet.
The best results are highlighted. OOD | Method | TNR | AUROC | DTACC | AUPR | AUPR
---|---|---|---|---|---|---
dataset |  | (TPR=95%) |  |  | IN | OUT
SVHN | Mahala-Tied | 83.63 | 97.1 | 91.26 | 94.13 | 98.78
| Mahala-Class | 71.73 | 95.16 | 87.92 | 90.77 | 97.98
| KNN-Tied | 84.07 | 97.18 | 91.32 | 94.2 | 98.84
| KNN-Class | 77.95 | 96.19 | 89.68 | 92.06 | 98.45
| SPB | 39.22 | 88.24 | 82.41 | 74.53 | 94.09
| ODIN | 69.96 | 92.02 | 84.1 | 80.49 | 97.05
| PCA | 2.46 | 55.89 | 56.36 | 35.42 | 74.12
| Integrated(Our) | 91.29 | 98.38 | 93.28 | 96.26 | 99.37
Imagenet | Mahala-Tied | 49.33 | 90.32 | 83.08 | 91.32 | 88.6
| Mahala-Class | 53.11 | 92.16 | 85.3 | 93.42 | 90.29
| KNN-Tied | 51.36 | 90.73 | 83.31 | 91.75 | 88.87
| KNN-Class | 57.94 | 92.74 | 86.01 | 93.67 | 91.28
| SPB | 40.13 | 89.3 | 82.67 | 90.88 | 86.74
| ODIN | 61.03 | 91.4 | 83.85 | 91.32 | 90.55
| PCA | 4.66 | 58.68 | 57.19 | 60.66 | 54.42
| Integrated(Our) | 77.81 | 95.98 | 89.74 | 96.22 | 95.68
LSUN | Mahala-Tied | 46.63 | 91.18 | 84.93 | 92.71 | 87.74
| Mahala-Class | 58.53 | 93.82 | 88.16 | 95.15 | 91.43
| KNN-Tied | 51.48 | 92.25 | 85.96 | 93.75 | 89.13
| KNN-Class | 65.17 | 94.57 | 88.6 | 95.61 | 92.61
| SPB | 48.38 | 92.14 | 86.22 | 93.68 | 89.83
| ODIN | 71.89 | 94.37 | 87.72 | 94.65 | 93.39
| PCA | 2.06 | 53.26 | 54.88 | 57.08 | 49.33
| Integrated(Our) | 84.34 | 97.27 | 92.1 | 97.6 | 96.74
Table 16: Ablation study with SVHN on DenseNet.
The best results are highlighted. OOD | Method | TNR | AUROC | DTACC | AUPR | AUPR
---|---|---|---|---|---|---
dataset |  | (TPR=95%) |  |  | IN | OUT
CIFAR10 | Mahala-Tied | 80.82 | 96.8 | 92.27 | 98.94 | 88.91
| Mahala-Class | 82.99 | 97.11 | 92.83 | 99.05 | 89.71
| KNN-Tied | 69.99 | 95.58 | 90.77 | 98.52 | 84.3
| KNN-Class | 74.52 | 96.01 | 91.21 | 98.64 | 85.99
| SPB | 69.31 | 91.9 | 86.61 | 95.7 | 82.8
| ODIN | 37.23 | 73.14 | 68.92 | 84.32 | 60.32
| PCA | 5.27 | 65.82 | 64.83 | 86.62 | 33.51
| Integrated(Our) | 86.6 | 97.41 | 92.88 | 99.11 | 91.76
Imagenet | Mahala-Tied | 85.44 | 97.29 | 93.39 | 99.12 | 90.22
| Mahala-Class | 77.66 | 96.83 | 93.17 | 98.98 | 88.59
| KNN-Tied | 65.76 | 94.67 | 89.59 | 98.18 | 80.16
| KNN-Class | 73.44 | 95.69 | 90.68 | 98.55 | 84.28
| SPB | 79.79 | 94.78 | 90.21 | 97.2 | 88.42
| ODIN | 62.76 | 85.41 | 79.94 | 90.95 | 79.59
| PCA | 5.16 | 65.08 | 65.39 | 86.65 | 32.83
| Integrated(Our) | 93.46 | 98.39 | 94.54 | 99.41 | 95.16
LSUN | Mahala-Tied | 76.87 | 96.37 | 92.43 | 98.84 | 85.79
| Mahala-Class | 69.44 | 96.05 | 92.4 | 98.74 | 84.89
| KNN-Tied | 59.64 | 93.71 | 88.22 | 97.83 | 77.17
| KNN-Class | 66.96 | 94.77 | 89.45 | 98.21 | 81.27
| SPB | 77.12 | 94.13 | 89.14 | 96.96 | 87.44
| ODIN | 62.91 | 86.06 | 80.04 | 92.03 | 79.98
| PCA | 3.19 | 62.66 | 64.7 | 85.72 | 30.37
| Integrated(Our) | 89.73 | 97.73 | 93.55 | 99.17 | 92.92
Table 17: Ablation study with SVHN on ResNet34.
The best results are highlighted. OOD | Method | TNR | AUROC | DTACC | AUPR | AUPR
---|---|---|---|---|---|---
dataset |  | (TPR=95%) |  |  | IN | OUT
SCIFAR100 | Mahala-Tied | 86.61 | 97.3 | 93.6 | 99.81 | 64.4
| Mahala-Class | 88.44 | 97.7 | 94.19 | 99.84 | 69.59
| KNN-Tied | 84.67 | 96.82 | 92.83 | 99.76 | 61.08
| KNN-Class | 83.72 | 96.83 | 93.05 | 99.77 | 57.58
| SPB | 81.28 | 94.62 | 91.48 | 99.35 | 64.38
| ODIN | 36.67 | 68.01 | 67.26 | 95.57 | 23.04
| PCA | 89.94 | 97.81 | 94.52 | 99.84 | 70.83
| Integrated(Our) | 99.61 | 98.47 | 97.36 | 99.89 | 67.9
LSUN | Mahala-Tied | 78.38 | 96.17 | 91.98 | 98.73 | 85.11
| Mahala-Class | 81.51 | 96.71 | 92.44 | 98.91 | 87.63
| KNN-Tied | 77.61 | 95.98 | 91.34 | 98.61 | 85.56
| KNN-Class | 78.77 | 96.05 | 91.45 | 98.62 | 85.71
| SPB | 74.29 | 91.58 | 88.96 | 94.19 | 83.95
| ODIN | 35.92 | 68.6 | 66.75 | 82.37 | 53.12
| PCA | 82.93 | 96.88 | 92.74 | 98.97 | 88.27
| Integrated(Our) | 85.46 | 97.09 | 93.17 | 99.03 | 89.03
CIFAR10 | Mahala-Tied | 85.03 | 97.05 | 93.15 | 99.04 | 88.62
| Mahala-Class | 86.84 | 97.41 | 93.48 | 99.15 | 90.37
| KNN-Tied | 82.17 | 96.65 | 92.24 | 98.87 | 87.63
| KNN-Class | 83.24 | 96.73 | 92.38 | 98.9 | 87.67
| SPB | 78.26 | 92.92 | 90.03 | 95.06 | 85.66
| ODIN | 32.67 | 66.75 | 65.37 | 80.69 | 50.49
| PCA | 88.18 | 97.55 | 93.83 | 99.2 | 90.77
| Integrated(Our) | 90.34 | 97.64 | 94.29 | 99.17 | 91.17


# Accurate solar-power integration: Solar-weighted Gaussian quadrature

Steven G. Johnson, MIT Applied Mathematics  Department of Mathematics,
Massachusetts Institute of Technology, Cambridge MA 02139.

(Date: Created December 2019; updated .)

###### Abstract.

In this technical note, we explain how to construct Gaussian quadrature rules
for efficiently and accurately computing integrals of the form $\int
S(\lambda)f(\lambda)d\lambda$ where $S(\lambda)$ is the solar irradiance
function tabulated in the ASTM standard and $f(\lambda)$ is an arbitary
application-specific smooth function. This allows the integral to be computed
accurately with a relatively small number of $f(\lambda)$ evaluations despite
the fact that $S(\lambda)$ is non-smooth and wildly oscillatory. Julia
software is provided to compute solar-weighted quadrature rules for an
arbitrary bandwidth or number of points. We expect that this technique will be
useful in solar-energy calculations, where $f(\lambda)$ is often a
computationally expensive function such as an absorbance calculated by solving
Maxwell’s equations.

##  1\. Introduction

For many problems involving solar energy, such as simulating the efficiency of
solar cells, it is necessary to compute integrals over wavelength $\lambda$ of
the form

(1.1) $\int_{a}^{b}S(\lambda)f(\lambda)d\lambda,$

where $S(\lambda)\geq 0$ is the _solar irradiance_ spectrum (the intensity of
sunlight on the surface of the Earth), $(a,b)$ is some bandwidth of interest,
and $f(\lambda)$ is an application-specific integrand. For example, in solar
photovoltaic devices, $f(\lambda)$ is computed from the optical absorption
spectrum of a device in order to quantify the photonic efficiency [1]. The
basic challenge in computing eq. (1.1) is that $S(\lambda)$ is _extremely
oscillatory_ and _non-smooth_ tabulated data (given by the ASTM standard [2],
shown in Fig. 1.1), while $f(\lambda)$ is typically smooth but computationally
expensive (e.g. requiring a solution of Maxwell’s equations for each $\lambda$
[1]).

Figure 1.1. ASTM AM1.5 solar-irradiance spectrum [2], given by 2002 tabulated
data points for wavelengths $\lambda$ from 280–4000nm. The spectrum is
extremely oscillatory, with many sharp dips corresponding to absorption by
various molecules.

Worse, for device design and optimization the integral (1.1) must be computed
many times for different $f(\lambda)$ [1, 3, 4, 5, 6]. In these notes, we
explain how eq. (1.1) can be accurately approximated with only a small number
of $f(\lambda)$ evaluations by a _Gaussian quadrature_ rule [7]

(1.2)
$\int_{a}^{b}S(\lambda)f(\lambda)d\lambda\approx\sum_{i=1}^{N}w_{i}f(\lambda_{i}),$

where $\lambda_{i}$ are the quadrature points (nodes), $w_{i}$ are the
quadrature weights, and $N$ is the _order_ of the rule. The key fact is that
we can construct a Gaussian quadrature rule in which the effect of the
complicated irradiance function $S(\lambda)$ is _precomputed_ in the
$(w_{i},\lambda_{i})$ data, so that any polynomial $f(\lambda)$ of degree up
to $2N-1$ is integrated _exactly_ by eq. (1.2) and smooth $f(\lambda)$ are
integrated with an error that decreases _exponentially_ with $N$ [7, 8].

In the following notes, we first briefly review the numerical construction of
Gaussian quadrature rules (Sec. 2), then present the resulting solar-weighted
quadrature rules and demonstrate their accuracy on some examples (Sec. 3), and
finally comment on various possible extensions and variations (Sec. 4). We
provide a selection of precomputed rules (1.2) as well as a program in the
Julia language [9] (in the form of a Jupyter notebook [10] tutorial) to
compute a rule (1.2) for any desired $N$ and bandwidth $(a,b)$ [11].

##  2\. Constructing Gaussian-quadrature rules

The construction of Gaussian quadrature rules, by which polynomial
$f(\lambda)$ are integrated exactly up to a given degree, is closely tied to
the theory of orthogonal polynomials [7]. For a given weight function
$S(x)\geq 0$ and interval $(a,b)$, we define an inner product

(2.1) $\langle p_{1},p_{2}\rangle=\int_{a}^{b}S(x)p_{1}(x)p_{2}(x)dx$

of any polynomial functions $p_{1,2}(x)$. Then, by a Gram–Schmidt
orthogonalization process applied to $\\{1,x,x^{2},\ldots\\}$, one can obtain
a sequence of orthonormal polynomials $\\{q_{0}(x),q_{1}(x),\ldots\\}$ with
respect to this inner product, where $q_{n}(x)$ has degree $n$. Remarkably, it
turns out that the quadrature points $\lambda_{i}$ in eq. (1.2) are exactly
the roots of $q_{N}(x)$, and the weights can also be computed from these
polynomials [7].

Numerically, there are a variety of schemes for computing the quadrature
points $\lambda_{i}$ and weights $w_{i}$. One of the simplest accurate methods
is the Golub–Welsch algorithm [12], which constructs a tridiagonal “Jacobi”
matrix corresponding to a three-term recurrence for the $q_{k}$ polynomials,
and then obtains the nodes $\lambda_{i}$ and weights $w_{i}$ from the
eigenvalues and eigenvectors of this matrix, respectively. It turns out that
this procedure is equivalent to a Lanczos iteration for the symmetric linear
operator corresponding to multiplication by $x$ [13]. The only input that is
required is a way to compute the functional

(2.2) $S\\{p\\}=\int_{a}^{b}S(x)p(x)dx,$

which is just the integral (1.1) computed for any given polynomial $p(x)$ (up
to degree $2N$). For the most famous applications of Gaussian quadrature to
simple weight functions like $1$ on $(-1,1)$ (Gauss–Legendre quadrature),
$e^{-x^{2}}$ on $(-\infty,+\infty)$ (Gauss–Hermite quadrature), or $e^{-x}$ on
$(0,\infty)$ (Gauss–Laguerre quadrature), these integrals and the resulting
three-term recurrences are known analytically. For an arbitrary weight
function $S(x)$ like the solar-irradiance spectrum, we must perform integral
(2.2) numerically (for $2N$ polynomials in total).

This process may at first seem fruitless: isn’t eq (2.2) equivalent to our
original problem? There are two critical differences, however. First,
polynomials $p(x)$ are very _cheap to evaluate_ , so we can afford to use
brute-force numerical integration methods that require $\sim 10^{4}$ function
evaluations to evaluate (2.2), unlike solar-cell applications of eq. (1.1)
where the integrand $f(\lambda)$ is very expensive. Second, we only need to
perform these polynomial integrations _once_ for a given $N$: we can then _re-
use_ the resulting quadrature rule $(\lambda_{i},w_{i})$ over and over for
many different $f(\lambda)$.

In the specific case of our solar-irradiance spectrum $S(\lambda)$, the data
are supplied in the form of 2002 data points from 280nm to 4000nm by the ASTM
standard [2]. To define $S(\lambda)$ continuously for all $\lambda$ in this
range, we perform a cubic-spline interpolation of the data using a library by
Dierckx  _et al._[14]. More precisely, since we require an interpolant
$S(\lambda)\geq 0$ everywhere in order for eq. (2.1) to define a proper inner
product, we compute a cubic-spline interpolation $C(\lambda)$ of
$\sqrt{S(\lambda)}$ at the ASTM data points, and then approximate $S(\lambda)$
by $C(\lambda)^{2}$ everywhere. This spline interpolant is shown for a portion
of the spectrum from 400–500nm in Fig. 2.1.

Figure 2.1. Cubic-spline interpolation [14] of the ASTM AM1.5 solar-irradiance
spectrum [2], shown for a portion of the spectrum. To ensure positivity of the
interpolant for all $\lambda$, we actually fit a spline $C(\lambda)$ to the
square root of the irradiance, and approximate $S(\lambda)$ by
$C(\lambda)^{2}$ for all $\lambda\in(280,4000)$nm.

A cubic spline has the property that not only does it pass through all of the
data points, but its first and second derivatives are also continuous; the
data points where the spline switches from one cubic curve to another (where
the third derivative is discontinuous) are known as the _knots_ of the spline
[14]. In order to perform the $S\\{p\\}$ integrals (2.2) of polynomials
$p(\lambda)$ against $C(\lambda)^{2}$, we simply break the integral into
$\approx 2000$ segments at the knots (to avoid integrating through
discontinuities) and apply a globally adaptive [15] Gauss–Kronrod [7]
quadrature scheme implemented in Julia [16] to high accuracy ($>$ 9 digits).

Given an arbitrary weight function (and optionally a specialized routine to
compute the $S\\{p\\}$ integrals), the QuadGK package in Julia [16] provides a
subroutine to execute the Golub–Welsch algorithm and compute the quadrature
points and weights. The only addition to the textbook description of the
algorithm [13], besides the numerical computation of $S\\{p\\}$, is that the
integration domain $(a,b)$ is internally rescaled to $(-1,1)$ so that the
polynomials $p(x)$ can be represented in the basis of the Chebyshev
polynomials $\\{T_{0},T_{1},\ldots\\}$ [17]. Given the coefficients $c_{k}$ of
a Chebyshev series $p(x)=\sum_{j}a_{k}T_{k}(x)$, the polynomial $p(x)$ is
evaluated by a Clenshaw recurrence [17], and another recurrence is used to
compute the coefficients of the $xp(x)$ polynomial required for the
Golub–Welsch algorithm [13]. This approach111A similar Chebyshev-based Lanczos
recurrence was implemented by S. Olver in 2014 for the ApproxFun Julia package
[18] following a suggestion by A. Edelman. to representing and evaluating
polynomials avoids the numerical problems (ill conditioning) that arise in the
familiar monomial basis $\\{1,x,x^{2},\ldots\\}$—Chebyshev polynomials
$T_{k}(x)=\cos(k\cos^{-1}x)$ are well-behaved numerically since they are
“cosines in disguise” [17]. (A related, but more complicated, algorithm in
which one first computes the “moments” $S\\{T_{k}(x)\\}$, is reviewed by
Gautschi [19], and various other methods have also been proposed [19, 20, 21,
22].)

##  3\. Solar integration rules and convergence

Figure 3.1. Solar-weighted Gaussian quadrature rules (ASTM AM1.5 spectrum [2])
in a bandwidth of 280–4000nm, for orders $N=15$ and $N=99$.

Given the computational methods in Sec. 2, we can then construct solar-
weighted Gaussian quadrature rules for any desired order $N$ and any given
bandwidth $\lambda\in(a,b)\subseteq(280,4000)\text{nm}$ using the code
provided in the attached Julia notebook [11]. The resulting weights $w_{i}$
are plotted versus the corresponding quadrature points $\lambda_{i}$ for
$N=15$ and $N=99$. Not surprisingly, they roughly follow the peaks and dips of
the solar spectrum $S(\lambda)$ from Fig. 1.1.

Figure 3.2. Test integrand $f_{1}(\lambda)=\sin(2\pi\lambda/500)$,
superimposed on the solar-irradiance spectrum $S(\lambda)$. The blue dots show
$f_{1}(\lambda_{i})$ at the quadrature points $\lambda_{i}$ of the $N=15$
point solar-weighted Gaussian quadrature rule from Fig. 3.1.

To validate these quadrature schemes and evaluate their accuracy, let us
consider two test integrands, $f_{1}(\lambda)=\sin(2\pi\lambda/500)$ and
$f_{2}(\lambda)=\sin(2\pi\lambda/50)$. The first integrand $f_{1}$ is shown in
Fig. 3.2: it oscillates 8 times over the solar bandwidth, and seems to be
barely sampled adequately by the $N=15$ point quadrature rule. To compute the
“exact” integrals of $f_{1}$ and $f_{2}$, we apply the same method that we
used for numerical polynomial integration $S\\{p\\}$ in Sec. 2: we partition
the integral into $\approx 2000$ segments at the knots of the cubic-spline
interpolant for $S(\lambda)$ and then apply an adaptive Gauss–Kronrod
integration scheme to nearly machine precision ($>12$ significant digits).
Comparing this brute-force result to that of the quadrature rules (1.2)
applied to $f_{1}(\lambda)$ from Fig. 3.2, we find that the $N=15$ quadrature
rule yields the correct result with an error of only $0.7\%$, while the $N=99$
quadrature rule yields the correct result to at least 13 significant digits.

Figure 3.3. Test integrand $f_{2}(\lambda)=\sin(2\pi\lambda/50)$, superimposed
on the solar-irradiance spectrum $S(\lambda)$. The blue dots show
$f_{2}(\lambda_{i})$ at the quadrature points $\lambda_{i}$ of the $N=99$
point solar-weighted Gaussian quadrature rule from Fig. 3.1. This 99-point
sampling is too coarse to capture the rapid oscillation of $f_{2}$, and
results in an aliasing phenomenon [8] in which the sampled function appears to
oscillate at a lower frequency.

For the test function $f_{2}(\lambda)$, which oscillates $10\times$ more
rapidly than $f_{1}$, even the $N=99$ quadrature rule is not enough to obtain
an accurate result: the sampling of $f_{2}$ is simply not fine enough to
detect its rapid variations. This can be seen in Fig. 3.3, which depicts the
function $f_{2}$ sampled at the quadrature points of the $N=99$ rule. The
sampling clearly exhibits the phenomenon of _aliasing_ , which is central to
understanding the convergence rates of these quadrature schemes [8]: the
sampled function $f_{2}(\lambda_{i})$ appears to oscillate at a much lower
frequency. Since aliasing means that the quadrture rule cannot “perceive” the
actual rapid oscillation of $f_{2}$, the quadrature formula (1.2)
unsurprisingly yields a completely incorrect result ($>100$% error). A larger
$N$ must be employed to integrate such a rapidly oscillating $f(\lambda)$. In
the case of this $f_{2}(\lambda)$, an $N=140$ solar-weighted Gaussian
quadrature rule is sufficient to compute $\int S(\lambda)f_{2}(\lambda)$ to 9
significant digits. The key point is that the required order of the quadrature
rule is completely determined by the features of the application-supplied
$f(\lambda)$, and is _independent_ of the preprocessed complexity of the solar
irradiance $S(\lambda)$.

Figure 3.4. Convergence of solar-weighted Gaussian quadrature: fraction error
in the integral $\int S(\lambda)f(\lambda)d\lambda$ versus the number $N$ of
quadrature points for the two test integrands
$f_{1}(\lambda)=\sin(2\pi\lambda/500)$ and
$f_{2}(\lambda)=\sin(2\pi\lambda/50)$. In both cases, once $N$ becomes
sufficiently large, the error falls exponentially with $N$. (However, in the
case of the rapidly oscillating $f_{2}$ from Fig. 3.3, $N$ must first become
$>100$ in order to adequately sample $f_{2}$’s oscillations.) The error in the
$f_{1}$ does not drop below a “noise floor” of $\approx 10^{-13}$ due to the
limitations of double-precision floating-point arithmetic [13].

In general, when Gaussian quadrature is applied to any smooth (infinitely
differentiable) function $f(\lambda)$, such as an absorption spectrum, the
error converges to zero faster than any power of $1/N$; typically (for
analytic functions), the error converges _exponentially_ with $N$ [8]. This
rapid convergence222In fact, since $f_{1}$ and $f_{2}$ are “entire” functions
analytic everywhere in the complex plane, the convergence should
asymptotically be _faster_ than exponential in $N$. In practice, this is hard
to detect because the accuracy is quickly constrained by machine precision. is
demonstrated in Fig. 3.4 for both $f_{1}$ and $f_{2}$. The accuracy of the
$f_{1}$ integrand, in fact, nearly hits the limits of machine precision around
$N\approx 30$ (and might be able to go slightly lower than $10^{-13}$ if we
reduced the tolerances in the brute-force integration used in this section and
in Sec. 2). The $f_{2}$ integration accuracy also exhibits exponential
convergence, but the error does not begin its asymptotic decline until
$N\gtrsim 100$, at which point the quadrature rule begins to adequately sample
$f_{2}$’s rapid oscillation without destructive aliasing.

##  4\. Extensions and variations

Figure 4.1. Solar-weighted Gaussian quadrature rules (ASTM AM1.5 spectrum [2])
in a bandwidth of 280–1100nm, for orders $N=15$ and $N=99$.

To apply the solar-weighted quadrature rule to specific applications, one may
wish to consider some variations. For example, with silicon solar cells one
rarely considers wavelengths beyond 1100nm, since longer wavelengths fall
short of the bandgap of silicon and cannot easily generate electron–hole pairs
[1, 4]. So, for these devices one might want a quadrature rule for
$\int_{280\text{nm}}^{1100\text{nm}}S(\lambda)f(\lambda)d\lambda$. Two such
quadrature rules, for $N=15$ and $N=99$ points, are shown in Fig. 4.1. It is
advantageous to specialize the quadrature rule to the bandwidth of interest
because one can then resolve much finer features in $f(\lambda)$ for the same
number of points, and optimized solar cells typically exhibit many sharp
resonant peaks [1, 3, 4]. As another example, the figure of merit for a
photovoltaic cell is typically proportional to $\int S(\lambda)\lambda
A(\lambda)d\lambda$ where $A(\lambda)$ is the device’s absorption spectrum
[1], so one could also include the factor of $\lambda$ in the precomputed
weight when constructing the Gaussian-quadrature schemes. It might also be
useful to rescale the weight by the absorption coefficient of the photovoltaic
material, to further skew the quadrature points towards the wavelengths where
more absorption occurs.

Naturally, the same techniques could also be applied to completely different
weight functions. Although weighted Gaussian quadrature schemes have been well
known for decades in the numerical-integration literature, they seem to have
been rarely applied for the complicated weighting functions that often arise
in optical physics. For example, the CIE color-projection functions (used to
model the perception of a human eye) [23] are described by tabulated data
(though there are also analytical fits [24]), leading to integrals that
qualitatively resemble Gauss–Hermite quadrature but require a quantitatively
distinct quadrature scheme for high accuracy. Thermal emission from heated
bodies is another good example: the emitted power is an integral of a
surface’s absorption spectrum (found by solving Maxwell’s equations) weighted
by a black-body “Planck” spectrum $\sim\omega^{3}/(e^{\hbar\omega/kT}-1)$
[25], suggesting a need for a Gauss–Laguerre-like Planck-weighted quadrature
scheme.

## Acknowledgements

This work was supported in part by the U. S. Army Research Office through the
Institute for Soldier Nanotechnologies under grant W911NF-13-D-0001.

## References

  * [1] P. Bermel, C. Luo, L. Zeng, L. C. Kimerling, and J. D. Joannopoulos, “Improving thin-film crystalline silicon solar cell efficiencies with photonic crystals,” _Optics Express_ , vol. 15, no. 25, p. 16986, 2007.
  * [2] ASTMG173-03, “Standard tables for reference solar spectral irradiances: Direct normal and hemispherical on 37 degree tilted surface,” ASTM International, West Conshohocken, PA, Tech. Rep., 2005.
  * [3] X. Sheng, S. G. Johnson, J. Michel, and L. C. Kimerling, “Optimization-based design of surface textures for thin-film Si solar cells,” _Optics Express_ , vol. 19, pp. A841–A850, June 2011.
  * [4] A. Oskooi, P. A. Favuzzi, Y. Tanaka, H. Shigeta, Y. Kawakami, and S. Noda, “Partially disordered photonic-crystal thin films for enhanced and robust photovoltaics,” _Applied Physics Letters_ , vol. 100, no. 18, p. 181110, 2012\.
  * [5] C. Lin, L. J. Martínez, and M. L. Povinelli, “Experimental broadband absorption enhancement in silicon nanohole structures with optimized complex unit cells,” _Optics Express_ , vol. 21, no. S5, p. A872, 2013.
  * [6] S. Wiesendanger, M. Zilk, T. Pertsch, F. Lederer, and C. Rockstuhl, “A path to implement optimized randomly textured surfaces for solar cells,” _Applied Physics Letters_ , vol. 103, no. 13, p. 131115, 2013.
  * [7] H. Brass and K. Petras, _Quadrature Theory: The Theory of Numerical Integration on a Compact Interval_ , ser. Mathematical Surveys and Monographs. Providence, RI: American Mathematical Society, 2011, vol. 178.
  * [8] L. N. Trefethen, “Is Gauss quadrature better than Clenshaw–Curtis?” _SIAM Review_ , vol. 50, no. 1, pp. 67–87, 2008.
  * [9] J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, “Julia: A fresh approach to numerical computing,” _SIAM review_ , vol. 59, no. 1, pp. 65–98, 2017.
  * [10] T. Kluyver, B. Ragan-Kelley, F. Pérez, B. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. Hamrick, J. Grout, S. Corlay, P. Ivanov, D. Avila, S. Abdalla, and C. Willing, “Jupyter notebooks—a publishing format for reproducible computational workflows,” in _Positioning and Power in Academic Publishing: Players, Agents and Agendas_ , F. Loizides and B. Schmidt, Eds. IOS Press, 2016, pp. 87–90.
  * [11] S. G. Johnson, “Solar-weighted Gaussian quadrature,” https://nbviewer.jupyter.org/urls/math.mit.edu/~stevenj/Solar-Quadrature.ipynb, accessed: 2019-12-13.
  * [12] G. H. Golub and J. H. Welsch, “Calculation of Gauss quadrature rules,” _Mathematics of Computation_ , vol. 23, no. 106, pp. 221–221, 1969.
  * [13] L. N. Trefethen and D. Bau, _Numerical Linear Algebra_. SIAM, 1997.
  * [14] P. Dierckx, _Curve and Surface Fitting with Splines_. Oxford University Press, 1993.
  * [15] M. A. Malcolm and R. B. Simpson, “Local versus global strategies for adaptive quadrature,” _ACM Transactions on Mathematical Software_ , vol. 1, no. 2, pp. 129–146, 1975.
  * [16] S. G. Johnson, “QuadGK.jl: Gauss–Kronrod integration in Julia,” https://github.com/JuliaMath/QuadGK.jl, accessed: 2019-12-13.
  * [17] L. N. Trefethen, _Approximation Theory and Approximation Practice_. SIAM, 2012.
  * [18] S. Olver and A. Townsend, “A practical framework for infinite-dimensional linear algebra,” in _Proceedings of the 1st Workshop for High Performance Technical Computing in Dynamic Languages – HPTCDL ‘14_. IEEE, 2014.
  * [19] W. Gautschi, “Algorithm 726: ORTHPOL—a package of routines for generating orthogonal polynomials and Gauss-type quadrature rules,” _ACM Transactions on Mathematical Software_ , vol. 20, no. 1, pp. 21–62, 1994.
  * [20] G. Mantica, “A stable Stieltjes technique for computing orthogonal polynomials and jacobi matrices associated with a class of singular measures,” _Constructive Approximation_ , vol. 12, no. 4, pp. 509–530, 1996\.
  * [21] M. J. Gander and A. H. Karp, “Stable computation of high order Gauss quadrature rules using discretization for measures in radiation transfer,” _Journal of Quantitative Spectroscopy and Radiative Transfer_ , vol. 68, no. 2, pp. 213–223, 2001.
  * [22] A. D. Fernandes and W. R. Atchley, “Gaussian quadrature formulae for arbitrary positive measures,” _Evolutionary Bioinformatics Online_ , vol. 2, pp. 251–259, Jan. 2006.
  * [23] J. Schanda, _Colorimetry: Understanding the CIE System._ New York, NY: Wiley, 2007.
  * [24] C. Wyman, P.-P. Sloan, and P. Shirley, “Simple analytic approximations to the CIE XYZ color matching functions,” _Journal of Computer Graphics Techniques (JCGT)_ , vol. 2, no. 2, pp. 1–11, July 2013.
  * [25] C. M. Cornelius and J. P. Dowling, “Modification of Planck blackbody radiation by photonic band-gap structures,” _Physical Review A_ , vol. 59, no. 6, pp. 4736–4746, 1999.