# On the relation between Stratonovich and Itô integrals with functional
integrands of conditional measure flows

Gonçalo dos Reis***G. dos Reis acknowledges support from the _Fundaç
$\tilde{\text{a}}$o para a Ci$\hat{e}$ncia e a Tecnologia_ (Portuguese
Foundation for Science and Technology) through the project [UIDB/00297/2020]
(Centro de Matemática e Aplicaç$\tilde{\text{o}}$es CMA/FCT/UNL).
University of Edinburgh
School of Mathematics
Edinburgh, EH9 3FD, UK
and
Centro de Matemática e Aplicaç$\tilde{\text{o}}$es
(CMA), FCT, UNL, Portugal
<EMAIL_ADDRESS>Vadim Platonov
University of Edinburgh
School of Mathematics
Edinburgh, EH9 3FD, UK

<EMAIL_ADDRESS>
( , (File: main.tex))

###### Abstract

In this small note we explicit the relation between Itô and Stratonovich
integrals when conditional measure flow components are present in the
integrands. The ‘correction’ term involves Lions-type measure derivatives and
clarifies which cross-correlations need to be taken into account. We cast the
framework in relation to SDEs of mean-field type depending on conditional
flows of measure. The result being trivial under full flows of measure.

Keywords: Stratonovich integral, Itô integral, Stochastic Differential
Equations, conditional measure flows, Lions derivative, mean-field

2010 AMS subject classifications:
Primary: 60H05 Secondary: 60H10, 60H15

###### Contents

  1. 1 Introduction
  2. 2 Notation and auxiliary results
    1. 2.1 Notation and Spaces
    2. 2.2 The Lions derivative
  3. 3 Stratonovich - Itô equivalence under conditional flows of measures
    1. 3.1 Stratonovich-Itô correspondence for measure-dependent integrands
    2. 3.2 Stratonovich-Itô correspondence for McKean-Vlasov SDEs

##  1 Introduction

Itô and Stratonovich integrals are ubiquitous in stochastic analysis
[KaratzasShreve1991BMandStochCalc, kunita1997stochastic,
kloedenplaten1992NumericalSDEs] and each approach with its own advantages and
disadvantages depending on usage [vankampen1981itovsStrat,
holm2020itovsStrat]. For instance, while the Stratonovich form has a simple
chain rule, its drawback lies in the fact that expectations of Stratonovich
integrals are more difficult to control (their expectation is non-zero in
strict opposition to the Itô integral). Examples of this Itô Vs. Stratonovich
dichotomy can be found in the motivation for numerical methods
[Foster2020polynomialapproximation], regularization by noise
[Maurelli2020Nonexplosion, Kupferman2004ItoStrat] or the recent development of
probabilistic rough paths [delarue2021probabilistic]: each approach leads to
different forms of iterated integrals (in Taylor expansions) and, depending on
the context, one is more convenient to use than the other.

Recall the context of Stochastic Differential Equations (SDE) with $W$
denoting a standard Brownian motion and, for sake of argument, let $b$ and
$\sigma$ be ‘nice’ maps. Denote the Stratonovich integral with its usual
symbol $\circ\text{d}W$ with $\text{d}W$ referring to the usual Itô integral.

A key result in stochastic analysis is the conversion rule between the Itô and
Stratonovich integrals in the context of SDEs, here in one dimension, namely

$\displaystyle\mathrm{d}X_{t}=b(t,X_{t})\mathrm{d}t+\sigma(t,X_{t})\circ\mathrm{d}W_{t}\
\ \Leftrightarrow\ \
\mathrm{d}X_{t}=\big{(}\,b(t,X_{t})+\frac{1}{2}\sigma(t,X_{t})\partial_{x}\sigma(t,X_{t})\,\big{)}\mathrm{d}s+\sigma(t,X_{t})\mathrm{d}W_{t},$

and, conversely, that

$\displaystyle\mathrm{d}X_{t}=b(t,X_{t})\mathrm{d}t+\sigma(t,X_{t})\mathrm{d}W_{t}\
\ \Leftrightarrow\ \
\mathrm{d}X_{t}=\big{(}\,b(t,X_{t})-\frac{1}{2}\sigma(t,X_{t})\partial_{x}\sigma(t,X_{t})\,\big{)}\mathrm{d}t+\sigma(t,X_{t})\circ\mathrm{d}W_{t}.$

The moral behind the correction term $\sigma\partial_{x}\sigma$ is the same as
in the Itô-Wentzell formula [kunita1997stochastic, Platonov2019ito]: it
emerges from the cross variation between the randomness in the integrand
$\sigma(t,X_{t})$ and the driving noise when one moves from the Stratonovich
mid-point integration rule to the Itô left-point integration rule (and vice-
versa). If $\sigma$ has no random elements (or existing, they are independent
of the driving noise) then the Itô and Stratonovich integrals coincide.

The goal of this work is to clarify the relation between Itô and Stratonovich
integrals when conditional measure-flow components, in the style of mean-field
games, are present in the integrands [Platonov2019ito,
CarmonaDelarue2017book2]. Namely, when $\sigma$ is not just a function of time
and space but also a measure functional and the measure input is a conditional
measure flow – if the measure is not random then the problem is trivial as
explained above. Our contribution is motivated by developments in mean-field
games with common noise [CarmonaDelarue2017book2, delarue2021probabilistic,
cardaliaguet2015master], mean-field SDEs and conditional propagation of chaos
[Erny2021ConditoinalPoC], open problems in regularization by noise (for
conditional measures flows) or numerical methods among others. The
contributions within the framework of conditional measure flows is slowly
developing and these results are a step in supporting this literature.

In terms of the results themselves there are two comments to be made. Firstly,
the ‘correction’ term involves a Lions-type measure derivative
[CarmonaDelarue2017book1, CarmonaDelarue2017book2] in a way that is
intuitively comparable to the $\sigma\partial_{x}\sigma$ term appearing above.
The result is intuitive but critically clarifies that no cross noise variation
appears between the (possible) multiple sources of randomness in the
integrand. The second comment relates to the employed methodology: we follow
is the “limit of particles systems” one. This approach was initiated in
[chassagneux2014classical, CarmonaDelarue2017book1, CarmonaDelarue2017book2]
and employed in [Platonov2019ito] to prove an Itô-Wentzell-Lions formula. Our
result follows the arguments in the latter. Critically, we point the reader to
the introduction of [Platonov2019ito] for a review on alternative proof
methodologies [buckdahn2017mean, cardaliaguet2015master, cavallazzi2021krylov,
guo2020s, talbi2021dynamic]. On the latter two, [guo2020s, talbi2021dynamic],
the approach for conditional flows is not developed and this is the critical
case for this manuscript. Nonetheless, we believe their approach, under the
assumption that the diffusion map $\sigma$ is deterministic, would deliver the
same results under the same assumptions. For simplicity we follow
[Platonov2019ito].

Organisation of the paper. In Section 2 we set notation and review a few
concepts necessary for the main constructions. The main result is presented in
Section 3.

Acknowledgements. The authors would like to thank William Salkeld (Université
de Nice Sophia-Antipolis, FR) for the helpful discussions.

##  2 Notation and auxiliary results

###  2.1 Notation and Spaces

Let $\mathbb{N}$ be the set of natural numbers starting at $1$, $\mathbb{R}$
denotes the real numbers. For collections of vectors in
$\\{x^{l}\\}_{l}\in\mathbb{R}^{d}$, let the upper index $l$ denotes the
distinct vectors, whereas the lower index the vector components, i.e.
$x^{l}=(x^{l}_{1},\cdots,x^{l}_{d})\in\mathbb{R}^{d}$ namely $x^{l}_{j}$
denotes the $j$-th component of $l$-th vector. For $x,y\in\mathbb{R}^{d}$
denote the scalar product by $x\cdot y=\sum_{j=1}^{d}x_{j}y_{j}$; and
$|x|=(\sum_{j=1}^{d}x_{j}^{2})^{1/2}$ the usual Euclidean distance; and
$x\otimes y$ denotes the tensor product of vectors $x,y\in\mathbb{R}^{d}$. Let
$\mathbbm{1}_{A}$ be the indicator function of set $A\subset\mathbb{R}^{d}$.
For a matrix $A\in\mathbb{R}^{d\times m}$ we denote by $A^{\intercal}$ its
transpose and its Frobenius norm by
$|A|=\textrm{Trace}\\{AA^{\intercal}\\}^{1/2}$. Let
$I_{d}:\mathbb{R}^{d}\to\mathbb{R}^{d}$ be the identity map.

We denote by $\mathcal{C}(A,B)$ for $A,B\subseteq\mathbb{R}^{d}$,
$d\in\mathbb{N}$, the space of continuous functions $f:A\to B$. In terms of
derivative operators and differentiable functions, $\partial_{t}$ denotes the
partial differential in the time parameter $t\in[0,T]$; $\partial_{y}$ and
$\partial_{yy}^{2}$ denote the gradient and the Hessian operator in
$y\in\mathbb{R}^{d}$ respectively.

We say that the function is locally bounded, when its restriction to the
compact set is bounded.

### Spaces

We introduce over $\mathbb{R}^{d}$ the space of probability measures
$\mathcal{P}(\mathbb{R}^{d})$ and its subset $\mathcal{P}_{2}(\mathbb{R}^{d})$
of those with finite second moment. The space
$\mathcal{P}_{2}(\mathbb{R}^{d})$ is Polish under the Wasserstein distance

$\displaystyle
W_{2}(\mu,\nu)=\inf_{\pi\in\Pi(\mu,\nu)}\Big{(}\int_{\mathbb{R}^{d}\times\mathbb{R}^{d}}|x-y|^{2}\pi(dx,dy)\Big{)}^{\frac{1}{2}},\quad\mu,\nu\in\mathcal{P}_{2}(\mathbb{R}^{d}),$

where $\Pi(\mu,\nu)$ is the set of couplings for $\mu$ and $\nu$ such that
$\pi\in\Pi(\mu,\nu)$ is a probability measure on
$\mathbb{R}^{d}\times\mathbb{R}^{d}$ such that
$\pi(\cdot\times\mathbb{R}^{d})=\mu$ and $\pi(\mathbb{R}^{d}\times\cdot)=\nu$.
Let $\textrm{Supp}(\mu)$ denote the support of
$\mu\in\mathcal{P}(\mathbb{R}^{d})$.

Throughout set some $0<T<+\infty$ and we work the finite time interval
$[0,T]$. We consider
$(\Omega^{0},\mathcal{F}^{0},\mathbb{F}^{0}=(\mathcal{F}^{0}_{t})_{t\in[0,T]},\mathbb{P}^{0})$
and
$(\Omega^{1},\mathcal{F}^{1},\mathbb{F}^{1}=(\mathcal{F}^{1}_{t})_{t\in[0,T]},\mathbb{P}^{1})$
atomless Polish probability spaces to be the respective completions of
$(\Omega^{0},\mathbb{F}^{0},\mathbb{P}^{0})$ and
$(\Omega^{1},\mathbb{F}^{1},\mathbb{P}^{1})$ carrying a respective
$m$-dimensional Brownian motions $W^{0}=(W_{t}^{0})_{t\in[0,T]}$ and
$W^{1}=(W_{t}^{1})_{t\in[0,T]}$ generating the probability space’s filtration,
augmented by all $\mathbb{P}^{0}$\- and $\mathbb{P}^{1}$-null sets
respectively. We augment
$(\Omega^{0},\mathcal{F}^{0},\mathbb{F}^{0}=(\mathcal{F}^{0}_{t})_{t\in[0,T]},\mathbb{P}^{0})$
with a sufficiently rich sub $\sigma$-algebra $\mathcal{F}^{0}_{0}$
independent of $W^{0}$ and $W^{1}$. We denote by
$(\Omega,\mathbb{F},\mathbb{P})$ the completion of the product space
$(\Omega^{0}\times\Omega^{1},\mathbb{F}^{0}\otimes\mathbb{F}^{1},\mathbb{P}^{0}\otimes\mathbb{P}^{1})$
equipped with the filtration $\mathbb{F}$ obtained by augmenting the product
filtration $\mathbb{F}^{0}\otimes\mathbb{F}^{1}$ in a right-continuous way and
by completing it. We let $\mathbb{E}^{0}$ and $\mathbb{E}^{1}$ taking the
expectation on the first and second space respectively. We adopt the following
convention, that for $d$-dimensional random vector $Y=(Y_{1},\cdots,Y_{d})$ we
understand denote $\mathbb{E}[Y]$ by the $d$-dimensional vector
$(\mathbb{E}[Y_{1}],\cdots,\mathbb{E}[Y_{d}])$.

We define $L^{2}(\Omega,\mathcal{F}_{0},\mathbb{P},\mathbb{R}^{d})$ as the
space of $\mathcal{F}_{0}$-measurable random variables
$\xi:\Omega\to\mathbb{R}^{d}$ that are square integrable
$\mathbb{E}^{\mathbb{P}}[|\xi|^{2}]<\infty$. Given two processes
$(X_{t})_{t\in[0,T]}$ and $(Y_{t})_{t\in[0,T]}$ let $\langle
X_{t},Y_{t}\rangle$ denote their cross-variation up to time $t\in[0,T]$.

Lastly, for convenience we choose to work over $1$-, $d$\- and $d\times
m$-dimensional spaces. The generalisation to different dimensions is
straightforward from our text.

###  2.2 The Lions derivative

####  2.2.1 The Lions derivative and notational conventions

To consider the calculus for the mean-field setting one requires to build a
suitable differentiation operator on the $2$-Wasserstein space. Among the
several notions of differentiability of a functional $u$ defined over
$\mathcal{P}_{2}(\mathbb{R}^{d})$ we follow the approach introduced by Lions
in his lectures at Collège de France [lions2007cours] and further developed in
[cardaliaguet2010notes]. A comprehensive presentation can be found in the
joint monograph of Carmona and Delarue
[CarmonaDelarue2017book1],[CarmonaDelarue2017book2].

We consider a canonical lifting of the function
$u:\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}$ to
$\tilde{u}:L^{2}(\Omega,\mathcal{F},\mathbb{P};\mathbb{R}^{d})\ni
Y\to\tilde{u}(Y)=u(Law(Y))\in\mathbb{R}$, where
$L^{2}(\Omega,\mathcal{F},\mathbb{P};\mathbb{R}^{d})$ is a space of square
integrable random variables. We can say that $u$ is $L$-differentiable at
$\mu$, if $\tilde{u}$ is Frechèt differentiable (in $L^{2}$) at some $Y$, such
that $\mu=\mathbb{P}\circ Y^{(-1)}$. Denoting the gradient by $D\tilde{u}$ and
using a Hilbert structure of the $L^{2}$ space, we can identify $D\tilde{u}$
as an element its dual, $L^{2}$ itself. It was shown in
[cardaliaguet2010notes] that $D\tilde{u}$ is a $\sigma(Y)$-measurable random
variable and given by the function
$Du(\mu,\cdot):\mathbb{R}^{d}\to\mathbb{R}^{d}$, depending on the law of $Y$
and satisfying $Du(\mu,\cdot)\in
L^{2}(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}),\mu;\mathbb{R}^{d})$.
Hereinafter the $L$-derivative of $u$ at $\mu$ is the map
$\partial_{\mu}u(\mu,\cdot):\mathbb{R}^{d}\ni
v\to\partial_{\mu}u(\mu,v)\in\mathbb{R}^{d}$, satisfying
$D\tilde{u}(Y)=\partial_{\mu}u(\mu,Y)$. We always denote $\partial_{\mu}u$ as
the version of the $L$-derivative that is continuous in the product topology
of all components of $u$.

While working with a common noise setting, we reconstruct the space by adding
the copy spaces, this procedure is well-described in the proof of Proposition
3.2.

####  2.2.2 The Empirical projection map

We recall the concept of _empirical projection map_ given in
[chassagneux2014classical] which will be one of the main workhorses throughout
our work.

######  Definition 2.1 (Empirical projection of a map).

Given $u:\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}$ and $N\in\mathbb{N}$,
define the empirical projection $u^{N}$ of $u$ via
$u^{N}:(\mathbb{R}^{d})^{N}\to\mathbb{R}$, such that

$u^{N}(y^{1},\dots,y^{N}):=u\big{(}\bar{\mu}^{N}\big{)},\quad\text{with}\quad\bar{\mu}^{N}:=\frac{1}{N}\sum\limits_{l=1}^{N}\delta_{y^{l}}\quad\textrm{and}\quad
y^{l}\in\mathbb{R}^{d},\leavevmode\nobreak\ l=1,\dots,N.$

We recall [CarmonaDelarue2017book1]*Proposition 5.35 which relates the spatial
derivative of $u^{N}$ with the $L$-derivative of $u$.

######  Proposition 2.2.

Let $u:\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}$ be
Fully-$\mathcal{C}^{2}(\mathcal{P}_{2}(\mathbb{R}^{d}))$, then, for any $N>1$,
the empirical projection $u^{N}$ is $\mathcal{C}^{2}$ on
$(\mathbb{R}^{d})^{N}$ and for all $y^{1},\cdots,y^{N}\in\mathbb{R}^{d}$ we
have the following differentiation rule

$\displaystyle\partial_{y^{j}}u^{N}(y^{1},\dots,y^{N})$
$\displaystyle=\frac{1}{N}\>\partial_{\mu}u\Big{(}\frac{1}{N}\sum_{l=1}^{N}\delta_{y^{l}},y^{j}\Big{)}.$

##  3 Stratonovich - Itô equivalence under conditional flows of measures

The setting discussed in this section is inspired by the developments in the
theory of mean-field games with common noise, [cardaliaguet2015master] and
[CarmonaDelarue2017book2].

Take measurable
$(b,\sigma^{0},\sigma^{1}):[0,T]\times\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}^{d}\times\mathbb{R}^{d\times
m}\times\mathbb{R}^{d\times m}$ and define the following process

$\displaystyle\text{d}Y_{t}=b(t,\mu_{t})\text{d}t+\sigma^{0}(t,\mu_{t})\text{d}W^{0}_{t}+\sigma^{1}(t,\mu_{t})\text{d}W^{1}_{t},\text{
and initial condition }Y_{0}\in L^{2}(\Omega,\mathcal{F}_{0},\mathbb{P}),$
(3.1)

and $\mu_{t}:=\textrm{Law}(Y_{t}(\omega_{0},\cdot))$ for
$\mathbb{P}^{0}$-almost any $\omega_{0}$. Here
$\textrm{Law}(Y_{t}(\omega_{0},\cdot))$ can be understood as RV from
$(\Omega^{0},\mathcal{F}^{0},\mathbb{P}^{0})$ into
$\mathcal{P}(\mathbb{R}^{d})$ (for further details see discussion in
[CarmonaDelarue2017book2]*Section 4.3).

The components of $Y=(Y_{1},\dots,Y_{d})\in\mathbb{R}^{d}$, satisfy the
following 1-dimensional SDE

$\displaystyle\text{d}Y_{i,t}=b_{i}(t,\mu_{t})\text{d}t+\sum_{j=1}^{m}\sigma_{ij}^{0}(t,\mu_{t})\text{d}W^{0}_{j,t}+\sum_{j=1}^{m}\sigma_{ij}^{1}(t,\mu_{t})\text{d}W^{1}_{j,t},\>\>Y_{0,i}\in
L^{2}(\Omega,\mathcal{F}_{0},\mathbb{P}),$ (3.2)

for all $i=1,\dots,d$.

Moreover, the involved coefficients will be assumed to satisfy the next
condition.

######  Assumption 3.1.

Let $Y_{0}\in L^{2}(\Omega,\mathcal{F}_{0},\mathbb{P})$ ($Y_{0}$ is
$\mathcal{F}_{0}$-measurable and independent of
$W_{t}^{0},\leavevmode\nobreak\ W_{t}^{1},\leavevmode\nobreak\ t\in[0,T]$).
Take $b:[0,T]\times\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}^{d}$ and
$\sigma^{0},\leavevmode\nobreak\
\sigma^{1}:[0,T]\times\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}^{d\times
m}$ such that $(b_{t})_{t\in[0,T]},(\sigma^{0}_{t})_{t\in[0,T]}$ and
$(\sigma^{1}_{t})_{t\in[0,T]}$ are $\mathbb{F}$-progressively measurable
processes and satisfy

  1. i)

For any $t\in[0,T]$, the maps
$\mu\mapsto\sigma^{0}(t,\mu),\leavevmode\nobreak\ \mu\mapsto\sigma^{1}(t,\mu)$
are continuous in topology induced by the Wasserstein metric for any
$\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})$;

  2. ii)

For any $t\in[0,T],\leavevmode\nobreak\ i=1,\dots,d$ and $j=1,\dots,m$, the
map $\mu\mapsto\sigma^{0}_{ij}(t,\mu)$ continuously L-differentiable at every
point $\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})$. Moreover, for all $i=1,\dots,d$
and $j=1,\dots,m,\leavevmode\nobreak\
\partial_{\mu}\sigma_{ij}^{0}:[0,T]\times\mathcal{P}_{2}(\mathbb{R}^{d})$ is
joint-continuous and locally bounded at every triple $(t,\mu,v)$, with
$(t,\mu)\in[0,T]\times\mathcal{P}_{2}(\mathbb{R}^{d}),\leavevmode\nobreak\
v\in\textrm{Supp}(\mu)$;

  3. iii)

The involved coefficients satisfy for any compact
$K\subset\mathcal{P}_{2}(\mathbb{R}^{d})$

$\displaystyle\sup_{\mu\in
K}\sup_{i\in\\{1,\dots,d\\}}\int_{0}^{t}\Big{[}|b_{i}(s,\mu)|+|\sigma_{i}^{0}(s,\mu)|^{2}+|\sigma_{i}^{1}(s,\mu)|^{2}+\big{|}\int_{\mathbb{R}^{d}}\partial_{\mu}\sigma_{i}^{0}(s,\mu,y)\mu(\text{d}y)\big{|}^{2}\Big{]}\text{d}s<\infty.$
(3.3)

We name $(W_{t}^{0})_{t\in[0,T]}$ as a common noise affecting the whole
setting, whilst $(W^{1})_{t\in[0,T]}$ is the idiosyncratic chaos for the
process $Y$. For the purposes of the present section we fix the common noise
and make all the transforms by conditioning on $W^{0}$. The measurability of
involved measure-derivative terms is discussed in
[CarmonaDelarue2017book1]*Remarks 5.101 and 5.103. Throughout the text we
exploit the procedure of conditioning on the common noise that is widely
covered by [CarmonaDelarue2017book2]*Section 4.3.

###  3.1 Stratonovich-Itô correspondence for measure-dependent integrands

We are now ready to introduce the main result of the manuscript.

######  Proposition 3.2.

For almost all $\omega^{0}\in\Omega^{0}$ take
$(\mu_{t})_{t\in[0,T]}:=\big{(}Law(Y_{t}(\omega_{0},\cdot))\big{)}_{t\in[0,T]},$
with $Y=(Y_{1},\dots Y_{d})$ solution to (3.1) under Assumption 3.1.

Then for all $i=1,\dots,d$ and $t\in[0,T]$ the Stratonovich SDE for $Y_{i}$
transforms $\mathbb{P}$-a.s. to an Itô SDE according to

$\displaystyle Y_{i,t}$
$\displaystyle=Y_{i,0}+\int_{0}^{t}b_{i}(s,\mu_{s})\leavevmode\nobreak\
\text{d}s+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{0}(s,\mu_{s})\leavevmode\nobreak\
\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{1}(s,\mu_{s})\leavevmode\nobreak\
\text{d}W^{1}_{j,s}$ $\displaystyle\Leftrightarrow\quad Y_{i,t}$
$\displaystyle=Y_{i,0}+\int_{0}^{t}b_{i}(s,\mu_{s})\leavevmode\nobreak\
\text{d}t+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{0}_{ij}(s,\mu_{s})\circ\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{1}_{ij}(s,\mu_{t})\circ\text{d}W^{1}_{j,s}$
(3.4)
$\displaystyle\qquad-\frac{1}{2}\sum_{k=1}^{d}\sum_{j=1}^{m}\int_{0}^{t}\mathbb{E}^{1}\Big{[}(\partial_{\mu}\sigma^{0}_{ij})_{k}(s,\mu_{s},Y^{1}_{s})\Big{]}\sigma^{0}_{kj}(s,\mu_{s})\leavevmode\nobreak\
\text{d}s,$

where $Y^{1}=Y^{1}(\omega_{0},\cdot)$ is an independent copy process of $Y$
satisfying (3.6) for $\mathbb{P}^{0}$-almost any $\omega^{0}\in\Omega^{0}$.

######  Remark 3.3.

Equation (3.2) provides a Stratonovich-to-Itô stochastic integral
correspondence in $1$-dimensional component of $Y$. Under relevant structural
changes to the regularity involved in Assumption 3.1 the general form of this
rule in multidimensional case will be $\mathbb{P}$-a.s. given by

$\displaystyle Y_{t}$
$\displaystyle=Y_{0}+\int_{0}^{t}b(s,\mu_{s})\leavevmode\nobreak\
\text{d}s+\int_{0}^{t}\sigma^{0}(s,\mu_{s})\leavevmode\nobreak\
\text{d}W^{0}_{s}+\int_{0}^{t}\sigma^{1}(s,\mu_{s})\leavevmode\nobreak\
\text{d}W^{1}_{s}$ $\displaystyle\Leftrightarrow\quad Y_{t}$
$\displaystyle=Y_{0}+\int_{0}^{t}b(s,\mu_{s})\leavevmode\nobreak\
\text{d}s+\int_{0}^{t}\sigma^{0}(s,\mu_{s})\circ\text{d}W^{0}_{s}+\int_{0}^{t}\sigma^{1}(s,\mu_{s})\circ\text{d}W^{1}_{s}$
(3.5)
$\displaystyle\qquad-\frac{1}{2}\int_{0}^{t}\mathbb{E}^{1}\Big{[}\big{(}\partial_{\mu}(\sigma^{0})_{i}^{j}\big{)}_{k}(s,\mu_{s},Y^{1}_{s})\Big{]}\otimes(\sigma^{0})^{k}_{j}(s,\mu_{s})\leavevmode\nobreak\
\text{d}s,$

where we use the common Einstein notation for summation across alternating
indices.

###### Proof of Proposition 3.2.

_Step 1. Mollification and compactification_ We carry out mollification in two
steps - firstly we construct the mollifying sequence and later show its
convergence. As in the [Platonov2019ito]*Theorem 3.4, we pick a smooth
function $\rho:\mathbb{R}^{d}\to\mathbb{R}^{d}$ with compact support, letting
for any $t\in[0,T]$,
$(\sigma^{0}\star\rho)_{t}(\mu):=\sigma^{0}_{t}(\mu\circ\rho^{-1})$ and for
any $t\in[0,T]$ having $\sigma^{0}\leavevmode\nobreak\ \mathbb{P}$-a.s.
bounded and continuous at every pair
$(t,\mu)\in\mathcal{P}_{2}(\mathbb{R}^{d}),\leavevmode\nobreak\
\partial_{\mu}\sigma^{0}\leavevmode\nobreak\ \mathbb{P}$-a.s. bounded and
continuous at every triple $(t,\mu,v)$ for $v\in\textrm{Supp}(\mu)$ and what
follows from local boundedness of $\sigma^{0},\sigma^{1}$ and
$\partial_{\mu}\sigma^{0}$. Now picking the sequence $(\rho_{n})_{n\geqslant
1}$ in a way that
$(\rho_{n},\partial_{x}\rho_{n},\partial^{2}_{xx}\rho_{n})(x)\to(x,I_{d},0)$
as $n\to\infty$, we can conclude that
$(\sigma^{0}\star\rho_{n})_{t}(\mu),\leavevmode\nobreak\
(\sigma^{1}\star\rho_{n})_{t}(\mu),\leavevmode\nobreak\
\partial_{\mu}(\sigma^{0}\star\rho_{n})_{t}(\mu,v)$ converge $\mathbb{P}$-a.s.
to $\sigma^{0}_{t}(\mu),\leavevmode\nobreak\
\sigma^{1}_{t}(\mu),\leavevmode\nobreak\ \partial_{\mu}\sigma^{0}_{t}(\mu,v)$
respectively. Thus we can assume $\sigma^{0},\sigma^{1}$ and its
$\partial_{\mu}\sigma^{0}$ $\mathbb{P}$-a.s. bounded.

Again as in [Platonov2019ito]*Theorem 3.4 we consider
$\mu\mapsto(\sigma^{0}\star\rho)(\mu*\phi_{G})$ instead of
$\mu\mapsto(\sigma^{0}\star\rho)(\mu)$ with $\phi_{G}$ \- density of standard
$d$-dimensional Gaussian distribution $N(0,I_{d})$ on $\mathbb{R}^{d}$ and
$(\mu*\phi_{G})(x):=\int_{\mathbb{R}^{d}}\phi_{G}(x-y)d\mu(y)$. Now the
support of $\mu*\phi_{G}$ is the whole $\mathbb{R}^{d}$ and
$\partial_{\mu}\sigma^{0}$ is $\mathbb{P}$-a.s. continuous at every triple
$(t,\mu\circ\phi_{G},v),t\in[0,T],v\in\mathbb{R}^{d}$. Installing
$\phi_{\varepsilon,G}$ \- $d$-dimensional Gaussian distribution
$N(0,\varepsilon I_{d})$ and letting $\varepsilon\searrow 0$, we conclude the
$\mathbb{P}$-a.s. convergence of
$\partial_{\mu}\sigma^{0}_{t}(\mu*\phi_{\varepsilon,G},v)$ to
$\partial_{\mu}\sigma^{0}_{t}(\mu,v)$ respectively. Thus we can assume
$\mathbb{P}$-a.s. uniform continuity of measure expansion terms for the whole
$\mathbb{R}^{d}$.

Now we are to show that mollification procedure is well-posed. It is
straightforward to verify that $\sigma^{0}_{n}:=\sigma^{0}\star\rho_{n}$
satisfies $\mathbb{P}$-a.s. (3.3) uniformly in $n\geqslant 1$. Applying twice
the dominated convergence theorem we conclude the $\mathbb{P}$-a.s.
convergence for all the $\text{d}t$ terms. To handle the convergence of the
stochastic integrals one additionally requires an argument across the
quadratic variation, as written in [Platonov2019ito]*Theorem 2.3 and
localisation. We copy the procedure above to conclude that
$\sigma^{0},\sigma^{1},\partial_{\mu}\sigma^{1}\leavevmode\nobreak\
\mathbb{P}$-a.s. have compact support.

_Step 2. Approximation._ By our mollification argument one can assume the
$\sigma^{0},\leavevmode\nobreak\ \sigma^{1},\partial_{\mu}\sigma^{0}$ to be
$\mathbb{P}$-a.s. bounded and $\mathbb{P}$-a.s. uniformly continuous in
respective topology spaces. Furthermore, one can assume $Y_{t}$ to be
$\mathbb{P}$-a.s. bounded, as its distribution has compact support. We
construct twin processes $(Y^{l}_{t})_{t\in[0,T]},\leavevmode\nobreak\
l=1,\dots,N$ of $(Y_{t})_{t\in[0,T]}$ each supporting its own independent
$m$-dimensional Brownian motion $(W^{1,l}_{t})_{t\in[0,T]}$ that generate
$(\Omega^{1,l},\mathcal{F}^{1,l},\mathbb{F}^{1,l},\mathbb{P}^{1,l})$ alongside
with $\mathcal{F}_{0}^{l}$, altogether forming a copy of
$(\Omega^{1},\mathcal{F}^{1},\mathbb{F}^{1},\mathbb{P}^{1})$. Since the
stochastic basis $(\Omega,\mathcal{F},\mathbb{F},\mathbb{P})$ of our initial
space is constructed as a completion of
$(\Omega^{0}\times\Omega^{1},\mathcal{F}^{0}\otimes\mathcal{F}^{1},\mathbb{F}^{0}\otimes\mathbb{F}^{1},\mathbb{P}^{0}\otimes\mathbb{P}^{1})$
augmented in a right-continuous way and then completed, we introduce a new
product basis $(\Omega^{l},\mathcal{F}^{l},\mathbb{F}^{l},\mathbb{P}^{l})$ to
be completion of
$(\Omega^{0}\times\Omega^{1,l},\mathcal{F}^{0}\otimes\mathcal{F}^{1,l},\mathbb{F}^{0}\otimes\mathbb{F}^{1,l},\mathbb{P}^{0}\otimes\mathbb{P}^{1,l})$
augmented in right-continuous way and then completed. Now we copy the dynamics
of $(Y_{t})_{t\in[0,T]}$, as

$\displaystyle\text{d}Y_{t}^{l}=b(t,\mu_{t}^{l})\leavevmode\nobreak\
\text{d}t+\sigma(t,\mu_{t}^{l})\leavevmode\nobreak\
\text{d}W_{t}^{0}+\sigma(t,\mu_{t}^{l})\leavevmode\nobreak\
\text{d}W^{1,l}_{t},\quad Y^{l}_{0}=Y_{0}^{l},$ (3.6)

where $Y_{0}^{l}$ are $\mathbb{P}^{0}$ i.i.d. copies of $Y^{0}$. The
components of $Y^{l}=(Y^{l}_{1},\dots,Y^{l}_{d})\in\mathbb{R}^{d}$ satisfy the
following 1-dimensional SDE

$\displaystyle\text{d}Y_{i,t}^{l}=b_{i}(t,\mu_{t}^{l})\leavevmode\nobreak\
\text{d}t+\sum_{j=1}^{m}\sigma_{ij}(t,\mu_{t}^{l})\leavevmode\nobreak\
\text{d}W_{j,t}^{0}+\sum_{j=1}^{m}\sigma_{ij}(t,\mu_{t}^{l})\leavevmode\nobreak\
\text{d}W^{1,l}_{j,t},\quad Y^{l}_{0}=Y_{0}^{l},$ (3.7)

for all $i\in 1,\dots,d$. Now we construct a total stochastic basis
$(\Omega^{1,\dots,N},\mathcal{F}^{1,\dots,N},\mathbb{F}^{1,\dots,N},\mathbb{P}^{1,\dots,N})$,
where

$\displaystyle\Omega^{1,\dots,N}=\Omega^{0}\times\Omega^{1}\times\prod_{l=1}^{N}\Omega^{1,l}$
$\displaystyle,\quad\mathcal{F}^{1,\dots,N}=\mathcal{F}^{0}\otimes\mathcal{F}^{1}\otimes\bigotimes_{l=1}^{N}\mathcal{F}^{1,l},$
$\displaystyle\mathbb{F}^{1,\dots,N}=\mathbb{F}^{0}\otimes\mathbb{F}^{1}\otimes\bigotimes_{l=1}^{N}\mathbb{F}^{1,l}$
$\displaystyle,\quad\mathbb{P}^{1,\dots,N}=\mathbb{P}^{0}\otimes\mathbb{P}^{1}\otimes\bigotimes_{l=1}^{N}\mathbb{P}^{1,l},$

where we again and finally augment the filtration in a right-continuous way
and complete. We underline that processes
$(Y^{l}_{t})(\omega^{0},\cdot),\leavevmode\nobreak\ l=1,\dots N$ are i.i.d.
$\mathbb{P}^{0}$-a.s.

Hereinafter while fixing the $\omega^{0}\in\Omega^{0}$, and for the sake of
simplicity we will omit adding the $(\omega^{0},\cdot)$ to the processes
$Y_{t}$ and its copies to highlight the respective relation to $\omega^{0}$,
but will leave in after $\bar{\mu}^{N}_{t}$ as to underline the nature of this
dependency.

We denote the flow of marginals for almost all $\omega^{0}\in\Omega^{0}$ as
$\bar{\mu}_{t}^{N}(\omega^{0},\cdot):=\frac{1}{N}\sum_{l=1}^{N}\delta_{Y_{t}^{l}(\omega^{0},\cdot)}$
for $t\in[0,T]$ and the empirical projection of $\sigma^{0},\sigma^{1}$ as
$\sigma^{0,N},\sigma^{1,N}$.

We consider the components of the vector $Y$ given by (3.2) and the components
of its copies (3.6) given by (3.7).

Now the standard Stratonovich-Itô relation [kunita1997stochastic]*Theorem
2.3.5. for stochastic integrals applied to
$\sigma^{0,N}_{ij},\leavevmode\nobreak\ \sigma^{1,N}_{ij},\leavevmode\nobreak\
\mathbb{P}$-a.s satisfies for all $i=1,\dots,d$,

$\displaystyle\sum_{j=1}^{m}\int_{0}^{t}\sigma^{0,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\leavevmode\nobreak\
\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{1,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\leavevmode\nobreak\
\text{d}W^{1}_{j,s}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{m}\int_{0}^{t}\sigma^{0,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\circ\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{1,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\circ\text{d}W^{1}_{j,s}$
(3.8)
$\displaystyle\qquad-\frac{1}{2}\sum_{j=1}^{m}\int_{0}^{t}\text{d}\big{\langle}\sigma^{0,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s}),W_{j,s}^{0}\big{\rangle}-\frac{1}{2}\sum_{j=1}^{m}\int_{0}^{t}\text{d}\big{\langle}\sigma^{1,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s}),W_{j,s}^{1}\big{\rangle}.$

Using the conditional expectations one can show that mutual independence of
the covariates we argue that the last term of (3.8) is $0\leavevmode\nobreak\
\mathbb{P}$-a.s..

We transform further as in [kloedenplaten1992NumericalSDEs]*Section 4.9. to
have $\mathbb{P}$-a.s. for all $i=1,\dots,d$,

$\displaystyle\sum_{j=1}^{m}\int_{0}^{t}\sigma^{0,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\leavevmode\nobreak\
\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{1,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\leavevmode\nobreak\
\text{d}W^{1}_{j,s}$
$\displaystyle=\sum_{j=1}^{m}\int_{0}^{t}\sigma^{0,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\circ\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{1,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\circ\text{d}W^{1}_{j,s}$
(3.9)
$\displaystyle\qquad-\frac{1}{2}\sum_{j=1}^{m}\sum_{l=1}^{N}\int_{0}^{t}\sum_{k=1}^{d}\partial_{y^{l}_{k}}\sigma^{0,N}_{ij}(s,Y^{1}_{s},\dots,Y^{N}_{s})\leavevmode\nobreak\
\text{d}\big{\langle}Y_{k,s}^{l},W_{j,s}^{0}\big{\rangle}.$

Going back to $\sigma^{0}$ and $\sigma^{1}$, transforming the first bracket
according to (3.6) since the dynamics of the copy processes is independent of
$W^{1}$, we write that $\mathbb{P}$-a.s. for all $i=1,\dots,d$,

$\displaystyle\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{0}(\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\leavevmode\nobreak\
\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{1}(\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\leavevmode\nobreak\
\text{d}W^{1}_{j,s}$
$\displaystyle=\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{0}(\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\circ\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{1}(\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\circ\text{d}W^{1}_{j,s}$
(3.10)
$\displaystyle\qquad-\sum_{k=1}^{d}\sum_{j=1}^{m}\int_{0}^{t}\frac{1}{2N}\sum_{l=1}^{N}(\partial_{\mu}\sigma^{0}_{ij})_{k}(s,\bar{\mu}_{s}^{N}(\omega^{0},\cdot),Y^{l}_{s})\leavevmode\nobreak\
\sigma^{0}_{kj}(s,\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\leavevmode\nobreak\
\text{d}s.$

Taking conditional expectations on the above formula
$\mathbb{E}^{1,1,\dots,N}\big{[}\cdot\big{]}:=\mathbb{E}^{\mathbb{P}^{1,\dots,N}}\big{[}\cdot|\>\mathcal{F}^{0}\otimes\mathcal{F}^{1}\big{]}$
we have $\mathbb{P}$-a.s. by the stochastic Fubini theorem (see
[Veraar2010FubiniRevisited]*Theorem 3.5) for all $i=1,\dots,d$,

$\displaystyle\sum_{j=1}^{m}\int_{0}^{t}\mathbb{E}^{1,1,\dots,N}\Big{[}\sigma_{ij}^{0}(s,\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\leavevmode\nobreak\
\text{d}W^{0}_{j,s}\Big{]}+\sum_{j=1}^{m}\int_{0}^{t}\mathbb{E}^{1,1,\dots,N}\Big{[}\sigma_{ij}^{1}(s,\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\leavevmode\nobreak\
\text{d}W^{1}_{j,s}\Big{]}$
$\displaystyle=\sum_{j=1}^{m}\int_{0}^{t}\mathbb{E}^{1,1,\dots,N}\Big{[}\sigma_{ij}^{0}(s,\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\circ\text{d}W^{0}_{j,s}\Big{]}+\sum_{j=1}^{m}\int_{0}^{t}\mathbb{E}^{1,1,\dots,N}\Big{[}\sigma_{ij}^{1}(s,\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\circ\text{d}W^{1}_{j,s}\Big{]}$
(3.11)
$\displaystyle\qquad-\sum_{k=1}^{d}\sum_{j=1}^{m}\int_{0}^{t}\frac{1}{2}\mathbb{E}^{1,1,\dots,N}\Big{[}(\partial_{\mu}\sigma_{ij}^{0})_{k}(s,\bar{\mu}_{s}^{N}(\omega^{0},\cdot),Y^{1}_{s})\leavevmode\nobreak\
\sigma_{kj}^{0}(\bar{\mu}_{s}^{N}(\omega^{0},\cdot))\Big{]}\text{d}s.$

Letting $n\to\infty$, (3.11) converges to (3.2) $\mathbb{P}$ a.s. by the
dominated convergence theorem for stochastic integrals (applying localisation
when necessary) and continuity of $\sigma^{0},\sigma^{1}$ and
$\partial_{\mu}\sigma^{0}$ (see [Platonov2019ito]*Theorem 2.3 for detailed
arguments). ∎

We are able to establish a more general but less transparent result under
weaker conditions.

######  Proposition 3.4.

For almost all $\omega^{0}\in\Omega^{0}$ take
$(\mu_{t})_{t\in[0,T]}:=\big{(}Law(Y_{t}(\omega_{0},\cdot))\big{)}_{t\in[0,T]},$
with $Y=(Y_{1},\dots Y_{d})$ solution to (3.1) under Assumption 3.1, where we
exclude all conditions on the existence and regularity of
$\partial_{\mu}\sigma_{ij}^{0}$.

Then for all $i=1,\dots d$ and $t\in[0,T]$ the Stratonovich SDE for $Y_{i}$
transforms $\mathbb{P}$-a.s. to Itô SDE according to

$\displaystyle Y_{i,t}$
$\displaystyle=Y_{i,0}+\int_{0}^{t}b_{i}(s,\mu_{s})\leavevmode\nobreak\
\text{d}s+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{0}(s,\mu_{s})\leavevmode\nobreak\
\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{1}(s,\mu_{s})\leavevmode\nobreak\
\text{d}W^{1}_{j,s}$ $\displaystyle\Leftrightarrow\quad Y_{i,t}$
$\displaystyle=Y_{i,0}+\int_{0}^{t}b_{i}(s,\mu_{s})\leavevmode\nobreak\
\text{d}s+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{0}(s,\mu_{s})\circ\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{1}(s,\mu_{s})\circ\text{d}W^{1}_{j,s}$
$\displaystyle\qquad-\frac{1}{2}\sum_{j=1}^{m}\big{\langle}\sigma^{0,N}_{ij}(t,\mu_{t}),W_{j,t}^{0}\big{\rangle}.$
(3.12)

###### Proof of Proposition 3.4.

We follow the same arguments as in a proof of (3.2) up to establishing
equation (3.8). Again noticing that the last term is $\mathbb{P}$-a.s. $0$, we
pass to the limit as $N\to\infty$ in (3.8) yo obtain (3.4) by dominated
convergence theorem, continuity of $\sigma^{0}$ and $\sigma^{1}$ and the limit
property of cross-variation (see [kunita1981some]*Corollary 2.2.16). ∎

In case of $\sigma^{0}$ being a progressively-measurable random process, the
last term of the Stratonovich form of (3.4) will capture not only the cross-
variance between $\sigma^{0}(t,\mu_{\cdot})$ and $W^{0}_{\cdot}$ but also that
between $\sigma^{0}(\cdot,\mu_{t})$ and $W^{0}_{\cdot}$. Nonetheless, without
the differentiability assumption on $\sigma^{0}$ one cannot expand the term
further and recover Proposition 3.2.

###  3.2 Stratonovich-Itô correspondence for McKean-Vlasov SDEs

Enriching the drift and diffusion functions by adding explicit dependence on
the process itself, we write the equivalence between formulation under the
framework of general McKean-Vlasov SDEs.

Take a measurable
$(b,\sigma^{0},\sigma^{1}):[0,T]\times\mathbb{R}^{d}\times\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}^{d}\times\mathbb{R}^{d\times
m}\times\mathbb{R}^{d\times m}$ and define the following process

$\displaystyle\text{d}Y_{t}=b(t,Y_{t},\mu_{t})\leavevmode\nobreak\
\text{d}t+\sigma^{0}(t,Y_{t},\mu_{t})\leavevmode\nobreak\
\text{d}W^{0}_{t}+\sigma^{1}(t,Y_{t},\mu_{t})\leavevmode\nobreak\
\text{d}W^{1}_{t},\quad Y_{0}\in L^{2}(\Omega,\mathcal{F}_{0},\mathbb{P}),$
(3.13)

and $\mu_{t}:=\textrm{Law}(Y_{t}(\omega_{0},\cdot))$ for
$\mathbb{P}^{0}$-almost any $\omega_{0}$. The components of
$Y=(Y_{1},\dots,Y_{d})\in\mathbb{R}^{d}$, satisfy the following 1-dimensional
SDE for all $i=1,\dots,d$,

$\displaystyle\text{d}Y_{i,t}=b_{i}(t,Y_{t},\mu_{t})\leavevmode\nobreak\
\text{d}t+\sum_{j=1}^{m}\sigma_{ij}^{0}(t,Y_{t},\mu_{t})\leavevmode\nobreak\
\text{d}W^{0}_{j,t}+\sum_{j=1}^{m}\sigma_{ij}^{1}(t,Y_{t},\mu_{t})\leavevmode\nobreak\
\text{d}W^{1}_{j,t},\>\>Y_{0,i}\in L^{2}(\Omega,\mathcal{F}_{0},\mathbb{P}).$

Moreover, the involved coefficients satisfy the next condition.

######  Assumption 3.5.

Let $Y_{0}\in L^{2}(\Omega,\mathcal{F}_{0},\mathbb{P})$ ($Y_{0}$ is
$\mathcal{F}_{0}$-measurable and independent of
$W_{t}^{0},\leavevmode\nobreak\ W_{t}^{1},\leavevmode\nobreak\ t\in[0,T]$).
Take
$b:[0,T]\times\mathbb{R}^{d}\times\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}^{d}$
and $\sigma^{0},\leavevmode\nobreak\
\sigma^{1}:[0,T]\times\mathbb{R}^{d}\times\mathcal{P}_{2}(\mathbb{R}^{d})\to\mathbb{R}^{d\times
m}$ such that it satisfies

  1. i)

For any $(t,y)\in[0,T]\times\mathbb{R}^{d}$, the maps
$\mu\mapsto\sigma^{0}(t,y,\mu),\leavevmode\nobreak\
\mu\mapsto\sigma^{1}(t,y,\mu)$ are continuous in topology induced by the
Wasserstein metric for any $\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})$;

  2. ii)

For any $(t,y)\in[0,T]\times\mathbb{R}^{d},\leavevmode\nobreak\ i=1,\dots,d$
and $j=1,\dots,m$, the map $\mu\mapsto(\sigma^{0})_{ij}(t,y,\mu)$ continuously
L-differentiable $\mathbb{P}$-a.s. at every point
$\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})$. Moreover, for all $i=1,\dots,d$ and
$j=1,\dots,m,\leavevmode\nobreak\
\partial_{\mu}\sigma_{ij}^{0}:[0,T]\times\mathcal{P}_{2}(\mathbb{R}^{d})$ is
$\mathbb{P}$-a.s. joint-continuous and locally bounded at every quadruple
$(t,y,\mu,v)$, with
$(t,y,\mu)\in[0,T]\times\mathbb{R}^{d}\times\mathcal{P}_{2}(\mathbb{R}^{d}),\leavevmode\nobreak\
v\in\textrm{Supp}(\mu)$;

  3. iii)

For any $(t,\mu)\in[0,T]\times\mathcal{P}_{2}(\mathbb{R}^{d})$, the maps
$y\mapsto\sigma^{0}(t,y,\mu),\leavevmode\nobreak\ y\mapsto\sigma^{1}(t,y,\mu)$
are $\mathcal{C}^{1}(\mathbb{R}^{d}),\leavevmode\nobreak\ \mathbb{P}$-a.s. at
every $y\in\mathbb{R}^{d}$, with
$\partial_{y}\sigma^{0},\partial_{y}\sigma^{1}$ being $\mathbb{P}$-a.s. joint
continuous at every triple
$(t,y,\mu)\in[0,T]\times\mathbb{R}^{d}\times\mathcal{P}_{2}(\mathbb{R}^{d}),\leavevmode\nobreak\
\mathbb{P}$-a.s.;

  4. iv)

The involved coefficients satisfy for any compact
$K\subset\mathbb{R}^{d}\times\mathcal{P}_{2}(\mathbb{R}^{d})$

$\displaystyle\sup_{(y,\mu)\in K}\sup_{i\in\\{1,\dots,d\\}}$
$\displaystyle\int_{0}^{T}\Big{[}|b_{i}(s,y,\mu)|+|\sigma_{i}^{0}(s,y,\mu)|^{2}+|\sigma_{i}^{1}(s,y,\mu)|^{2}+|\partial_{y}\sigma_{i}^{0}(s,y,\mu)|^{2}$
$\displaystyle\qquad+|\partial_{y}\sigma_{i}^{1}(s,y,\mu)|^{2}+\Big{|}\int_{\mathbb{R}^{d}}\partial_{\mu}\sigma_{i}^{0}(s,y,\mu,v)\mu(\text{d}v)\Big{|}^{2}\Big{]}\text{d}s<\infty.$
(3.14)

Following the arguments of Proposition 3.2 with the classical result of
[kunita1981some, kloedenplaten1992NumericalSDEs] one can prove the following
result.

######  Theorem 3.6.

For almost all $\omega^{0}\in\Omega^{0}$ take
$(\mu_{t})_{t\in[0,T]}:=\big{(}Law(Y_{t}(\omega_{0},\cdot))\big{)}_{t\in[0,T]},$
with $Y$ solution to (3.13) under Assumption 3.5.

Then for all $i=1,\dots,d$ and $t\in[0,T]$ the Stratonovich SDE for $Y_{i}$
transforms $\mathbb{P}$-a.s. to Itô SDE according to

$\displaystyle Y_{i,t}$
$\displaystyle=Y_{i,0}+\int_{0}^{t}b_{i}(s,Y_{s},\mu_{s})\leavevmode\nobreak\
\text{d}s+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{0}(s,Y_{s},\mu_{s})\leavevmode\nobreak\
\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma_{ij}^{1}(s,Y_{s},\mu_{s})\leavevmode\nobreak\
\text{d}W^{1}_{j,s}$ $\displaystyle\Leftrightarrow\quad Y_{i,t}$
$\displaystyle=Y_{i,0}+\int_{0}^{t}b_{i}(s,Y_{s},\mu_{s})\leavevmode\nobreak\
\text{d}s+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{0}_{ij}(s,Y_{s},\mu_{s})\circ\text{d}W^{0}_{j,s}+\sum_{j=1}^{m}\int_{0}^{t}\sigma^{1}_{ij}(s,Y_{s},\mu_{s})\circ\text{d}W^{1}_{j,s}$
$\displaystyle\qquad-\frac{1}{2}\sum_{k=1}^{d}\sum_{j=1}^{m}\int_{0}^{t}\mathbb{E}^{1}\Big{[}(\partial_{\mu}\sigma^{0}_{ij})_{k}(s,Y_{s},\mu_{s},Y^{1}_{s})\Big{]}\sigma^{0}_{jk}(s,Y_{s},\mu_{s})\leavevmode\nobreak\
\text{d}s$ (3.15)
$\displaystyle\qquad-\frac{1}{2}\sum_{k=1}^{d}\sum_{j=1}^{m}\int_{0}^{t}\partial_{y_{k}}\sigma_{ij}^{0}(s,Y_{s},\mu_{s})\leavevmode\nobreak\
\sigma_{jk}^{0}(s,Y_{s},\mu_{s})\leavevmode\nobreak\ \text{d}s$
$\displaystyle\qquad-\frac{1}{2}\sum_{k=1}^{d}\sum_{j=1}^{m}\int_{0}^{t}\partial_{y_{k}}\sigma_{ij}^{1}(s,Y_{s},\mu_{s})\leavevmode\nobreak\
\sigma_{jk}^{1}(s,Y_{s},\mu_{s})\leavevmode\nobreak\ \text{d}s,$

where $Y^{1}=Y^{1}(\omega_{0},\cdot)$ is an independent copy process of $Y$
satisfying (3.13) for $\mathbb{P}^{0}$-almost any $\omega^{0}\in\Omega^{0}$.

## References


# XOOD: Extreme Value Based Out-Of-Distribution Detection For Image
Classification

Frej Berglind
Louisiana State University
Baton Rouge, LA 70803
<EMAIL_ADDRESS>
&Haron Temam
Louisiana State University
Baton Rouge, LA 70803
&Supratik Mukhopadhyay
Louisiana State University
Baton Rouge, LA 70803
&Kamalika Das
Intuit Inc.
2700 Coast Ave
Mountain View CA 94043
&Md Saiful Islam Sajol
Louisiana State University
Baton Rouge, LA 70803
&Sricharan Kumar
Intuit Inc.
2700 Coast Ave
Mountain View CA 94043
&Kumar Kallurupalli
Intuit Inc.
2700 Coast Ave
Mountain View CA 94043

###### Abstract

Detecting out-of-distribution (OOD) data at inference time is crucial for many
applications of machine learning. We present XOOD: a novel extreme value-based
OOD detection framework for image classification that consists of two
algorithms. The first, XOOD-M, is completely unsupervised, while the second
XOOD-L is self-supervised. Both algorithms rely on the signals captured by the
extreme values of the data in the activation layers of the neural network in
order to distinguish between in-distribution and OOD instances. We show
experimentally that both XOOD-M and XOOD-L outperform state-of-the-art OOD
detection methods on many benchmark data sets in both efficiency and accuracy,
reducing false-positive rate (FPR95) by 50%, while improving the inferencing
time by an order of magnitude.

##  1 Introduction

Deep neural networks are known to be opaque in their decision-making process
[1]. This becomes problematic when decisions need to be made on inputs whose
salient characteristics are different than what the neural network has been
trained to identify. Such situations frequently arise when dealing with out-
of-distribution (OOD) data [2], i.e., test data at inference time that does
not come from the training distribution. For example, a neural network that
has been trained to classify images of horses and giraffes, when provided with
an image of an elephant, will classify it either as a horse or a giraffe,
rather than determining that it does not belong to either of the classes.
Indeed, the fundamental assumption in supervised machine learning that both
the training and the test data come from the same probability distribution [3,
4], is violated in many real-world applications like text extraction from
documents, medical diagnosis, autonomous driving, etc. In such situations,
neural networks can make erroneous decisions rather than issuing a warning
that they have encountered OOD data on which their decisions cannot be
trusted. When document text extraction models are deployed in real life
applications, they may encounter a crumpled document that is damaged beyond
recognition. Instead of informing the users of the situation, the model will
attempt to extract a (possibly incorrect) set of texts from it. In safety
critical decision making such as medicine, finance, and autonomous driving,
for machine learning models to be trusted, it is important for the model to
identify when to abstain or require human intervention [5].

It is well known [6, 7] that the “class probabilities" output by the softmax
layer of a neural network are only weakly correlated with how confident the
model should be about the prediction, even in the relatively simple case of
distinguishing Gaussian noise from in-distribution data [6]. Measuring
uncertainty associated with the decisions of a neural network during inference
time is an active area of research [8]. For uncertainties stemming from
encountering OOD data at test time, various supervised and unsupervised
methods have been proposed in the literature [9], [10], [11], [12] that
provide varying degree of accuracy and come with varying degrees of
computational overhead. Since convolutional layers have linear complexity, OOD
detection algorithms for convolutional networks should ideally have linear
complexity in order to be useful and scalable in real life applications. In
this paper, we propose a class of extreme value based OOD detection algorithms
that are linear in complexity with respect to the size of the embedding space,
and are, therefore, highly efficient and scalable, while matching or beating
state of the art results in accuracy metrics.

Our XOOD framework (Extreme Value-based OOD Detection) that comprises two
efficient and accurate algorithms for OOD detection _for image classification_
: the first, XOOD-M, being completely unsupervised (see Section 2.2.2), while
the second XOOD-L being self-supervised (see Section 2.2.3). Neither of the
two algorithms require prior knowledge of the OOD data, but instead rely on
the computation of global extrema of the input features. Figure 1(a) presents
the softmax output of a ResNet34 with CIFAR-10 as in-distribution and SVHN as
OOD. As one can see, the softmax output is unable to separate between the in-
distribution and the OOD images. Figures 1(b) and 1(c) respectively show that
both the unsupervised as well as the self-supervised algorithms of XOOD
(XOOD-M and XOOD-L) are able to separate the in-distribution from the OOD
images. In addition, both algorithms are robust to both uniform and Gaussian
noise (see Table LABEL:reacttable in Appendix A).

(a) Baseline

(b) XOOD-M

(c) XOOD-L

Figure 1: Comparison of the baseline result from the softmax output of a
ResNet34 to our out-of-distribution detection algorithms. In this example,
cifar-10 is in-distribution (ID) and SVHN is out-of-distribution (OOD). Note
how confident the baseline is even though the images in SVHN do not belong to
any of the classes in cifar-10.

##### Contributions:

This paper makes the following contributions

  * •

Based on the observation that the extreme value distribution of the feature
space in any activation layer is unimodal on the in-distribution data, we
propose a highly effective method for OOD detection using these extrema.

  * •

We introduce a regularized Mahalanobis distance on the extracted extreme
values for unsupervised OOD detection (XOOD-M).

  * •

We present a self-supervised method using logistic regression on the extracted
extreme values for OOD detection (XOOD-L).

  * •

We show experimentally that both XOOD-M and XOOD-L outperform state-of-the-art
OOD detection methods on many benchmark data sets across several detection
metrics. In fact, it improves the average false positive rate at 95% true
positive rate by as much as 50% compared to the best performing existing
approach [9].

  * •

Both algorithms have an order-of-magnitude lower computational overhead
compared to several other state-of-the-art OOD detection methods and scale
linearly with the size of the embedding space, making them ideal for many
real-life applications.

##  2 Proposed Approach

In this section, we formally define the problem statement, and discuss the
theoretical background of our proposed methods before describing the two
algorithms for OOD detection.

###  2.1 Problem Formulation

A neural network based image classifier is a function $f_{\theta}:X\rightarrow
Y$, where $\theta$ represents the weights of the network, $X$ is the input
domain defined by a set of images and $Y=\\{0,1,\ldots,K-1\\}$, where $K$ is
the number of classes in the input distribution. In the traditional image
classification setting, the trained model $f_{\theta}$ outputs a decision
$\hat{y}\in Y$ for every test image $\tilde{X}$. However, if $\tilde{X}$ is
out of distribution with respect to the input distribution of $X$, we would
want the model $f_{\theta}$ to return, along with the class label $\hat{y}$, a
score akin to model confidence that can indicate whether $\tilde{X}$ is out of
distribution with respect to the marginal probability distribution $p_{X}$ for
the joint distribution $p(X,Y)$. In other words, we want to learn a function
$g$ that satisfies the following condition

$g(\tilde{X})=\begin{cases*}1&\text{if $\tilde{X}\sim p_{X}$}\\\
0&\text{otherwise}\end{cases*}.$ (1)

###  2.2 Extreme Value Based Out-Of-Distribution Detection

We propose two new extreme value based out-of-distribution detection
algorithms that rely on the signals captured by the extreme values of the data
in the activation layers of a neural network to distinguish between in
distribution and OOD instances. Neither of the two algorithms require prior
knowledge of the OOD data. The first algorithm, called XOOD-M, is completely
unsupervised. It only computes some statistics on the extreme value
distribution of the training data and uses the notion of distance from those
statistics to identify OOD data points. This method is extremely efficient and
accurate and produces state-of-the-art results on all benchmark data sets for
which the extreme value distribution of the training data is compact and well
separated from the OOD data. For situations where the OOD data may not be
significantly different than the in-distribution data, we propose a modified
version of the XOOD-M algorithm called XOOD-L. This is a self-supervised
algorithm in that it does not need to be trained on labeled OOD detection
data. Instead, it learns a logistic regression model on the extreme values of
the in-distribution data along with a range of distortions of that data as a
surrogate for the OOD data. XOOD-L also beats state-of-the-art OOD detection
methods on many benchmark data sets in both efficiency and accuracy.

We describe the details of both the algorithms in the next few sections.

Figure 2: Distribution of maximum values before an activation layer inside
ResNet34. Green represents CIFAR-10 (ID) and Red represents LSUN-Resize(OOD)
(left figure) or SVHN(OOD) (right figure). Note how in-distribution data lies
in a single mode, while out-of-distribution data can be on either side.
Complete examples of extreme values can be found in Appendix C

####  2.2.1 Extreme Value Extraction

The essence of both the algorithms is in the nature of the distribution of the
extreme values of the data as it enters the activation layers in a neural
network. We observe that the per activation layer distribution of maximum or
minimum value is consistently unimodal on in-distribution data, as can be seen
in Figure 2 and Appendix C. Based on this observation, we compute the global
maximum and minimum of the input images before each activation layer in the
architecture. To improve the symmetry of this distribution, we apply a Yeo-
Johnson transform [13] fitted on the extreme values of the training data. The
output of this transform is standardized to zero mean and unit variance. This
computation does not make use of class labels or image topology, yet it is
highly effective for OOD detection.

Algorithm 1 Function: ExtremeValueExtraction

Data set $X_{t}$

for each image $X_{t,i}$ in training set $X_{t}$ do

$X_{min,i}$, $X_{max,i}$ = MinMax($X_{t,i}$)

end for

$X_{min}$ = $[X_{min,1},X_{min,2},X_{min,3},X_{min,4},\dots]$

$X_{max}$ = $[X_{max,1},X_{max,2},X_{max,3},X_{max,4},\dots]$

return $X_{min}$, $X_{max}$

####  2.2.2 Unsupervised Out-of-Distribution Detection: XOOD-M

Once the extreme values are computed for each activation layer of the neural
network, a regularized Mahalanobis distance can be used for OOD detection. Let
$\mu$ and $M$ be the mean and covariance of the extreme values on the training
set. We modify $M$ by adding a constant to the diagonal:

$M^{\prime}=M+C\cdot I,$ (2)

where $C\in[0,\inf)$ is a regularization constant and $I$ is the identity
matrix. Since the extreme values are standardized, the diagonal of $M$ is 1
and we can expect $C$ to have a consistent effect on each instance. The
Mahalanobis distance $D_{M}$ is computed using

$D_{M}(x)=\sqrt{(x-\mu)^{\mathsf{T}}(M+C\cdot I)^{-1}(x-\mu)}.$ (3)

The value $C=0$, corresponds to the standard Mahalanobis distance, that
represents the likelihood of a multivariate normal distribution. If $C$ is
large, the regularized covariance matrix $M^{\prime}$ and the corresponding
inverse covariance matrix $(M^{\prime})^{-1}$ are dominated by the constant on
the diagonal and $D_{M}$ is approximately proportional to the $L^{2}$
distance. This corresponds to the likelihood of a normal distribution where
each of the extreme values are independent. The regularizer $C$ allows us to
control the definition of in-distribution data. Algorithm 1 presents the
pseudo-code of the XOOD-M algorithm. As can be seen from the pseudo-code,
XOOD-M has minimal computational overhead. The only features that are computed
on the training data are mean and covariance on the extreme value feature set,
which is low dimensional with only $2\times r$ features, where $r$ is the
number of activation layers in the network.

Algorithm 2 Algorithm 1: XOOD-M

Training image and label set $<X_{t},y_{t}>$ , Pretrained classifier
$f_{\theta}$, Regularization $C$

Mean $\mu$ and Covariance $K$ of OOD features

Predict $\hat{y}_{t}=f_{\theta}(X_{t})$

$X_{t}$ = $X_{t,\hat{y}_{t}==y_{t}}$

for each input $X^{\prime}_{t,j}$ to activation layer $j$ in $f_{\theta}$ do

$X_{min,j}$, $X_{max,j}$ = FeatureExtraction($X^{\prime}_{t,j}$)

$X_{j}^{P}$, $T_{j}^{P}$ = fit.PowerTransform($X_{min,j}$, $X_{max,j}$)

end for

$\mu$ = Mean($X_{j}^{P}$)

$M$ = Covariance($X_{j}^{P}$)

####  2.2.3 Self-supervised Out-of-Distribution Detection: XOOD-L

The extreme value feature set extracted from the training data can be used in
a self-supervised setting for OOD detection. While the Mahalanobis distance-
based method produces a set of distances from the in-distribution features,
using a logisitic regression-based method it is possible to generate labels to
classify points as out of OOD, as shown in Equation 1. Additionally, the
calibrated probability scores of the logistic regression model can directly be
interpreted as the amount of confidence of the model based on the similarity
of the test instance with instances seen during training. The logistic
regression model predicts the likelihood of an instance being in-distribution
based on the distribution of the extreme values in each activation layer. It
uses the extreme value features obtained from the training data as well as
_distorted_ calibration data (see Section 2.2.4) for training the model. The
labels of the instances are generated based on whether the pre-trained neural
network model can correctly predict each instance or not: all correctly
classified instances are assigned label 1 in the logistic regression training,
while all incorrectly classified instances have label 0. The intuition behind
this is that, OOD data should be further away from the in-distribution data
than images where the model makes a meaningful prediction. Therefore, this
logistic regression should generalize to OOD images and assign them
predictions close to 0. As shown in Figure 2, sometimes the distorted data is
mapped to both sides of the in-distribution mode of the extreme value
distribution. To enable the logistic regression to handle this, we split up
the extreme values for each layer around the mean of the in-distribution data:

$\begin{cases}m_{i}^{+}=\text{relu}(m_{i}-\bar{m}_{i})\\\
m_{i}^{-}=\text{relu}(-m_{i}+\bar{m}_{i})\\\ \end{cases}$ (4)

where $m_{i}$ is maximum or minimum for activation layer $i$ and $\bar{m}_{i}$
is the mean of on the in-distribution data. Lastly, before applying the
logistic regression, each of the split up extreme values are scaled to zero
mean and unit variance to ensure consistent regularization. To avoid
overfitting to the logistic regression training data, we apply
$\ell_{2}$-regularization which is tuned through k-fold cross validation where
each fold excludes one of the distortion types. Algorithm 2 describes the
pseudo-code of the XOOD-L algorithm.

Algorithm 3 Algorithm 2: XOOD-L

Training image and label set $<X_{t},y_{t}>$, Calibration image and label set
$<X_{c},y_{c}>$,

Number of distortions to be applied $k$, Distortion parameters $d$, Pretrained
classifier $f_{\theta}$

Logistic regression model $g(w,\lambda)$ for training module

Predict $\hat{y}_{t}=f_{\theta}(X_{t})$

$X_{t}$ = $X_{t,\hat{y}_{t}==y_{t}}$

for each input $X^{\prime}_{t,j}$ to activation layer $j$ in $f_{\theta}$ do

$X_{min,j}$, $X_{max,j}$ = FeatureExtraction($X^{\prime}_{t,j}$)

$X_{j}^{P}$, $T_{j}^{P}$ = fit.PowerTransform($X_{min,j}$, $X_{max,j}$)

end for

for $i\in$ $1,\dots,k$ do

$X_{D_{i}}$ = Distortion($X_{c},d_{i}$)

end for

$Z$ = $[X_{C},X_{D_{1}},X_{D_{2}},\dots,X_{D_{k}}]$

for each input $Z^{\prime}_{j}$ to activation layer $j$ in $f_{\theta}$ do

$Z_{min,j}$, $Z_{max,j}$ = ExtremeValueExtraction($Z^{\prime}_{j}$)

$Z_{j}^{P}$ = $T_{j}^{P}$.PowerTransform($Z_{min,j}$, $Z_{max,j}$)

end for

Predict $\hat{y}_{c}=f_{\theta}(Z)$

$LR_{x}$ = $[Z_{min,1}$ $Z_{max,1}Z_{min,2}$ $Z_{max,2}Z_{min,3}$
$Z_{max,3}\dots]$

if $\hat{y}_{c}==y_{c}$ then

$LR_{y}$ = 1

else

$LR_{y}$ = 0

end if

$\lambda$ = crossValidate.LogisticRegression($LR_{x},LR_{y}$) using each
distorted data subset $X_{D_{i}}$ and $X_{C}$ as validation set to a
$(k+1)$-fold cross validation

$w$ = fit.LogisticRegression($LR_{x},LR_{y}$, $\lambda$)

####  2.2.4 Design Choices in Models

In both XOOD-M and XOOD-L, we make various choices in terms of the feature
set, the distortions, and the regularization. In this section we discuss the
rationale behind these choices.

#####

Distortions The distortions were designed to create a wide range of colors and
textures, challenge the classifier in many different ways and produce wide and
smooth distributions of extreme values. For satisfying these requirements, we
used the following distortions in XOOD-L:

  1. 1.

Geometric: A combination of common geometric image augmentations: up to 90
degree rotation, up to 0.2 width and height shift, 50% chance of horizontal
flip, brightness range between 0.2 and 2, zoom range from 0.9 to 1.1.

  2. 2.

Mixup: A convex combination of two images in the data set. The label is taken
from the image with the highest weight. This produces images with less
distinct features which lie near a decision boundary.

  3. 3.

Gaussian Noise: Additive Gaussian noise with mean 0 and variance ranging from
0 to 2. After adding noise we apply an affine transformation to each pixel
value to create a wide range of brightness and contrast $x=ax+b$, where
$a\in[1/8,8]$ and $b\in[\min(0,1-a),\max(0,1-a)]$. $a$ and $b$ are constant
for every image.

  4. 4.

Gaussian Blur: Gaussian blur with mean 0 and variance ranging from 0.2 to 5.
After blurring the images we apply the same affine transformation as for the
Gaussian noise.

All resulting images are clipped to have pixel values between 0 and 1.

#####

Regularization While the $\ell_{2}$ regularization in the XOOD-L method is
tuned through cross validation, the regularization parameter $C$ in XOOD-M
cannot be learnt. So we experimented with a wide range of values for $C$. We
found that a regularization $C=10$ worked well across all architectures and
data sets. Figure 3 shows a comparison of detection performance with $C$
ranging from 0 to $\infty$, where $\infty$ corresponds to $L_{2}$-distance
instead of Mahalanobis distance.

Figure 3: AUROC-score of XOOD-M for various regularization values. The score
was computed on the test set and the union of TinyImageNet (Crop),
TinyImageNet (Resize), LSUN (Crop), LSUN (Resize), iSUN for CIFAR-10,
CIFAR-100 and SVHN. For CIFAR-10 and CIFAR-100, SVHN is included as OOD as
well. For IMAGENET-1000, we used Places, SUN, iNaturalist and DTD, as curated
by [12].

#####

Feature Set Although we based our algorithm on the extreme value distribution
of the data in the activation layers, we experimented with a number of
different features: only using the minimum or maximum, the sum; the percentage
of positive values; $L_{p}$-norms; $L_{p}$-norms of $ReLU(x)$ and $ReLU(-x)$.
As one can see in Appendix B, neither replacing the extreme values by these
features, nor adding these features while still using the extreme values
improves the detection performance.

##  3 Experiments

In this section, we report the results obtained from running XOOD-M and XOOD-L
on a variety of data sets on a number of performance metrics. The code is
available at [14]. We track performance with respect to the quality of the
results as well as the efficiency of the methods. For quality of results, we
measure performance based on the following metrics:

  * •

AUROC: The area under plot of true positive rate (TPR) versus false positive
rate (FPR). A random detector has AUROC = 50% and an ideal detector has AUROC
= 100%.

  * •

TNR (95% TPR): The probability that an OOD instance is correctly detected with
a threshold that achieves a TPR of 95%.

  * •

Detection Accuracy: The maximum detection accuracy over all thresholds,
assuming equal amounts of in-distribution and OOD instances. That is,
$\max_{T}\\{\frac{p\left(f(x)>T|x\in\text{ID}\right)+p\left(f(x)\leq
T|x\in\text{OOD}\right)}{2}\\}$, where $f$ is the confidence score.

###  3.1 Results

We have tested the XOOD algorithms on ResNet and DenseNet for Cifar-10,
Cifar-100, and SVHN using the same pretrained models as several other papers
on OOD-detection. In Tables 2 and 1, we compare our algorithms with the
baseline [6], ODIN [10], Mahalanobis [15] and Gram [9]. Even though ODIN and
Mahalanobis are trained on a sample of the OOD-data, XOOD outperforms them in
most cases. And even though Gram is far more computationally complex, XOOD
outperforms it in most experiments. The one case where XOOD struggles is
CIFAR-100 vs CIFAR-10. However, this is not necessarily bad, because the
datasets are very similar. For example, CIFAR-10 includes cars, trucks and
dogs, while CIFAR-100 contains pickup trucks, busses and wolfs. If all of
these are detected as OOD, it limits the generalization ability of the model.

Table 1: DenseNet detection scores for
Baseline/Odin/Mahalanobis/Gram/XOOD-L/XOOD-M. Results for other algorithms are
from Sastry and Oore [16].

|  | TNR (95% TPR) | AUROC | Detection Acc.
---|---|---|---|---
In Dist | Out Dist |  |  |
CIFAR-10 | CIFAR-100 | 40.3/53.1/14.5/26.7/47.6/19.9 | 89.3/90.2/58.5/72.0/87.0/70.7 |  82.9/82.7/57.2/67.3/78.6/64.8
| LSUN-C | 51.8/70.6/48.2/88.4/96.9/91.7 | 92.9/93.6/80.2/97.5/99.3/98.4 | 86.9/86.4/75.6/92.0/96.1/93.6
| LSUN-R | 66.6/96.2/97.2/99.5/99.4/99.7 | 95.4/99.2/99.3/99.9/99.9/99.9 | 90.3/95.7/96.3/98.6/98.5/98.6
| SVHN | 40.2/86.2/90.8/96.1/96.5/97.2 | 89.9/95.5/98.1/99.1/99.2/99.3 | 83.2/91.4/93.9/95.9/96.2/96.7
| TinyImgNet-C | 56.7/87.0/84.2/96.7/98.6/97.8 | 93.8/97.6/95.3/99.3/99.7/99.5 | 88.1/92.3/89.9/96.1/97.7/96.6
| TinyImgNet-R | 58.9/92.4/95.0/98.8/99.1/99.2 | 94.1/98.5/98.8/99.7/99.8/99.8 | 88.5/93.9/95.0/97.9/98.2/97.8
| iSUN | 62.5/93.2/95.3/99.0/99.5/99.4 | 94.7/98.7/98.9/99.8/99.9/99.8 | 89.2/94.3/95.2/97.9/98.5/98.1
CIFAR-100 | CIFAR-10 |  18.9/16.8/ 7.7/10.6/10.0/ 1.7 |  75.9/74.2/60.1/64.2/65.0/43.1 |  69.7/68.6/57.8/60.4/61.8/50.0
| LSUN-C | 28.6/57.8/42.1/65.5/84.9/68.4 | 80.2/91.4/81.7/91.4/97.4/92.6 | 72.7/83.3/74.0/83.6/91.3/85.0
| LSUN-R | 17.6/41.2/91.4/97.2/94.3/97.9 | 70.8/85.5/98.0/99.3/99.0/99.3 | 64.9/77.1/93.9/96.4/94.8/96.7
| SVHN | 26.7/70.6/82.5/89.3/88.8/87.9 | 82.7/93.8/97.2/97.3/97.7/97.1 | 75.6/86.6/91.5/92.4/92.3/91.9
| TinyImgNet-C | 24.6/51.0/60.1/89.0/95.4/89.2 | 76.2/88.3/88.8/97.7/99.1/97.5 | 69.0/80.2/81.6/92.5/95.2/92.7
| TinyImgNet-R | 17.6/42.6/86.6/95.7/95.7/96.1 | 71.7/85.2/97.4/99.0/99.2/98.9 | 65.7/77.0/92.2/95.5/95.4/95.6
| iSUN | 14.9/37.4/87.0/95.9/94.0/96.3 | 69.5/84.5/97.4/99.0/98.9/99.0 | 63.8/76.4/92.4/95.6/94.6/95.7
SVHN | CIFAR-10 | 69.3/71.7/96.8/80.4/99.7/98.4 | 91.9/91.4/98.9/95.5/99.8/99.5 | 86.6/85.8/95.9/89.1/98.3/97.0
| LSUN-R | 77.1/81.1/99.9/99.5/100./100. | 94.1/94.5/99.9/99.8/100./100. | 89.1/89.2/99.3/98.6/99.8/99.8
| TinyImgNet-R | 79.8/84.1/99.9/99.1/100./100. | 94.8/95.1/99.9/99.7/100./100. | 90.2/90.4/98.9/97.9/99.5/99.6
| iSUN | 78.3/82.2/99.9/99.4/100./100. | 94.4/94.7/99.9/99.8/100./100. | 89.6/89.7/99.2/98.3/99.7/99.8

Table 2: ResNet34 detection scores for
Baseline/Odin/Mahalanobis/Gram/XOOD-L/XOOD-M. Results for other algorithms are
from Sastry and Oore [16].

|  | TNR (95% TPR) | AUROC | Detection Acc.
---|---|---|---|---
In Dist | Out Dist |  |  |
CIFAR-10 | CIFAR-100 | 33.3/42.0/41.6/32.9/43.6/32.0 | 86.4/85.8/88.2/79.0/88.8/80.1 | 80.4/78.6/81.2/71.7/81.7/73.7
| LSUN-C | 48.6/62.0/81.3/89.8/97.0/94.6 | 91.9/91.2/96.7/97.8/99.4/99.0 | 86.3/82.4/90.5/92.6/96.0/94.8
| LSUN-R | 49.8/82.1/98.8/99.6/99.1/99.9 | 91.0/94.1/99.7/99.9/99.8/100. | 85.3/86.7/97.7/98.6/97.8/99.2
| SVHN | 50.5/70.3/87.8/97.6/96.4/98.4 | 89.9/96.7/99.1/99.5/99.3/99.7 | 85.1/91.1/95.8/96.7/96.1/97.6
| TinyImgNet-C | 46.4/68.7/92.0/96.7/98.0/99.1 | 91.4/93.1/98.6/99.2/99.5/99.8 | 85.4/85.2/93.9/96.1/96.7/98.0
| TinyImgNet-R | 41.0/67.9/97.1/98.7/97.9/99.5 | 91.0/94.0/99.5/99.7/99.6/99.9 | 85.1/86.5/96.3/97.8/96.9/98.8
| iSUN | 44.6/73.2/97.8/99.3/99.1/99.7 | 91.0/94.0/99.5/99.8/99.8/99.9 | 85.0/86.5/96.7/98.1/97.8/98.8
CIFAR-100 | CIFAR-10 | 19.1/18.7/20.2/12.2/11.7/ 9.3 | 77.1/77.2/77.5/67.9/71.0/63.9 | 71.0/71.2/72.1/63.4/67.1/61.6
| LSUN-C | 18.7/44.1/64.8/64.8/85.3/76.2 | 75.5/82.7/92.0/92.1/97.5/95.3 | 69.2/75.9/84.0/84.2/91.4/88.2
| LSUN-R | 18.8/23.2/90.9/96.6/90.2/98.5 | 75.8/85.6/98.2/99.2/98.3/99.6 | 69.9/78.3/93.5/96.7/92.9/97.4
| SVHN | 20.3/62.7/91.9/80.8/87.2/92.1 | 79.5/93.9/98.4/96.0/97.6/98.2 | 73.2/88.0/93.7/89.6/91.5/93.6
| TinyImgNet-C | 24.3/44.3/80.9/88.5/90.8/95.0 | 79.7/85.4/96.3/97.7/98.4/98.9 | 72.5/78.3/89.9/92.2/93.2/95.0
| TinyImgNet-R | 20.4/36.1/90.9/94.8/89.4/97.9 | 77.2/87.6/98.2/98.9/98.1/99.5 | 70.8/80.1/93.3/95.0/92.6/96.7
| iSUN | 16.9/45.2/89.9/94.8/89.6/97.0 | 75.8/85.5/97.9/98.8/98.3/99.3 | 70.1/78.5/93.1/95.6/92.8/96.2
SVHN | CIFAR-10 | 78.3/79.8/98.4/85.8/98.1/96.1 | 92.9/92.1/99.3/97.3/99.5/98.9 | 90.0/89.4/96.9/92.0/96.8/95.7
| LSUN-R | 74.3/77.3/99.9/99.6/99.5/99.9 | 91.6/89.4/99.9/99.8/99.9/99.9 | 89.0/87.2/99.5/98.5/98.3/98.9
| TinyImgNet-R | 79.0/82.0/99.9/99.3/99.7/99.8 | 93.5/92.0/99.9/99.7/99.9/99.8 | 90.4/89.4/99.1/97.9/98.4/98.6
| iSUN | 77.1/79.1/99.7/99.4/99.8/99.9 | 92.2/91.4/99.8/99.8/99.9/99.9 | 89.7/89.2/98.3/98.1/98.7/98.9

To summarize the performance of the various methods, we computed the mean
scores across all metrics for each OOD detection task, except the
CIFAR-10$-$CIFAR-100 in-distribution-OOD combination, given their overlap. We
observe that XOOD-L and XOOD-M have the best FPR (95% TPR) across all
architectures, beating the state-of-the-art Gram method by approximately 50%.
For details, please refer to the Appendix D (Table 8).

###  3.2 Inference Time

In real-world applications of OOD-detection, the inference time is often
crucial, yet this has largely been neglected in research. We measured the
inference time of a ResNet and DenseNet on cifar10 and cifar100 for the
baseline [6], Mahalanobis [15], Gram [9] and XOOD . We chose to compare with
Mahalanobis and Gram since they have good detection performance and use
similar feature extraction schemes as XOOD . The results are shown in Tables 3
and 4. While XOOD has an overhead of about 30%, Mahalanobis and Gram impose
more than a 10-fold increase in inference time. Mohseni et al. [17] compare
the inference time of Outlier Exposure [18], Geometric OOD [19], shifting
tranformation learning [17], SSD [20], CSI [21], CSI-ens [21], and Gram [9].
All these algorithms have overhead above 100% except Outlier Exposure, which
doesn’t have any overhead at all, but it is not really comparable to XOOD
since it requires fitting on OOD data.

The tables 2 and 1 display the time it takes to compute the classification and
OOD detection of 10000 images consisting of Gaussian noise on a single NVIDIA
Tesla V100 GPU and one Intel Xeon Silver 4216 CPU with a batch size of 128,
averaged over 10 runs. We define overhead as $\frac{T-T_{B}}{T_{B}}$, where
$T$ is the average inference time and $T_{B}$ is the average inference time of
the baseline. Code for Mahalanobis [15] and Gram [9] was modified to enable
this comparison. The computations in these algorithms do not depend on the
content of the images and Gaussian noise was simply used for convenience.

Table 3:  Average inference time measured in seconds of various OOD-detection
algorithms on ResNet34. ± indicates the 99%-confidence interval of the mean.

(a) CIFAR-10 | Inference Time | Overhead
---|---|---
Baseline | 1.45 ± 0.0041 | 0%
XOOD-L | 1.79 ± 0.0575 | 24%
XOOD-M | 1.99 ± 0.0163 | 37%
Mahalanobis | 20.25 ± 0.0413 | 1298%
Gram | 28.70 ± 0.6036 | 1880%

(b) CIFAR-100 | Inference Time | Overhead
---|---|---
Baseline | 1.45 ± 0.0069 | 0%
XOOD-L | 1.83 ± 0.0913 | 26%
XOOD-M | 2.0 ± 0.0177 | 38%
Mahalanobis | 28.02 ± 0.4577 | 1834%
Gram | 28.49 ± 0.1467 | 1866%

Table 4:  Average inference time measured in seconds of various OOD-detection
algorithms on DenseNet. ± indicates the 99%-confidence interval of the mean.

(a) CIFAR-10 | Inference Time | Overhead
---|---|---
Baseline | 2.24 ± 0.0102 | 0%
XOOD-L | 2.90 ± 0.0497 | 29%
XOOD-M | 3.12 ± 0.0480 | 39%
Mahalanobis | 30.11 ± 0.2802 | 1246%
Gram | 63.01 ± 0.7629 | 2717%

(b) CIFAR-100 | Inference Time | Overhead
---|---|---
Baseline | 2.23 ± 0.0056 | 0%
XOOD-L | 2.91 ± 0.0849 | 31%
XOOD-M | 3.08 ± 0.0188 | 39%
Mahalanobis | 36.88 ± 0.6617 | 1557%
Gram | 61.46 ± 0.6788 | 2662%

###  3.3 Further Experiments

Additionally, we compare the performance of XOOD with Energy [22] and ReAct
[12] on DenseNet and ResNet34 for CIFAR-10, CIFAR-100 and SVHN, as well as
ResNet18, ResNet34, ResNet50 and ResNet101 for Imagenet1000. A table with
these results can be found in Appendix A. XOOD outperforms the other
algorithms on CIFAR-10, CIFAR-100 and SVHN. ReAct performs the best on
Imagenet1000, but does not significantly improve over the baseline on the
other data sets. It also struggles with the Gaussian noise data set on
Imagenet1000 and gets 0% TPR at 95% TNR for ResNet34 and ResNet50. XOOD-L
outperforms the baseline and Energy on almost all combinations in in-
distribution and OOD data sets.

##  4 Related Work

The last few years have seen a tremendous amount of work in OOD detection [2,
10, 9, 12, 23, 10, 15, 23, 11, 22, 24, 25, 26, 27, 28, 29, 30]. However, as
argued in [31], no single algorithm can be considered the state-of-the-art. In
[31], the authors tested the algorithms in [10, 6, 15] on three benchmark in-
distribution datasets (CIFAR-10, CIFAR-100, SVHN) and seven benchmark OOD
datasets under standardized conditions. They found inconsistent performance of
these algorithms across all the datasets; in fact, there was no algorithm that
was consistently outperforming others across all the datasets. As shown in
Tables 1, 2, for DenseNet and ResNet34 architectures, both XOOD-L and XOOD-M
outperform [10, 6, 15] for CIFAR-10 (with LSUN-C, LSUN-R, SVHN,
TinyImageNet-C, TinyImageNet-R, and iSun as OOD). Similar is the case with
most other in-distribution-OOD combinations. Unlike algorithms like [10, 15,
22], neither XOOD-M nor XOOD-L require access to the OOD data during training
time. In addition, for the architectures considered, both XOOD-L and XOOD-M
outperform [9] for almost all in-distribution-OOD combinations. Mohseni et al.
[17] compare different OOD detection algorithms in terms of inference time.
All the algorithms considered, that are not allowed to train on OOD data, have
overhead above 100%. Both the algorithms in [9, 15] involve matrix
multiplication which has at least quadratic complexity. Since, the number of
features within a neural network is large, matrix multiplication operations as
used in [9, 15] are prohibitively expensive. In case of XOOD-M, Mahalanobis
distance is computed on the extreme values which has dimension twice the depth
of the model (typically at most 1000); hence Mahalanobis distance can be
computed efficiently. This explains why both our algorithms XOOD-L and XOOD-M
outperform [9] and [15] in terms of inference time by an order of magnitude.
In Table LABEL:reacttable in Appendix A, we show that ReAct [12] is having
difficulty distinguishing Gaussian noise from in-distribution ImageNet (Resnet
34 and ResNet 50). Table LABEL:reacttable also shows that both XOOD-L and
XOOD-M are robust to Gaussian and uniform noise. Additionally, Table
LABEL:reacttable shows that the XOOD framework (in particular XOOD-L)
outperforms [22] on all but one (in-distribution CIFAR-10, OOD CIFAR-100) in-
distribution-OOD data set combinations for all the architectures considered.
ReAct performs the best on Imagenet1000, but does not significantly improve
over the baseline on the other data sets.

##  5 Conclusions

We presented the XOOD framework (Extreme Value-based OOD Detection) that
comprises two new, efficient and accurate algorithms for OOD detection for the
image classification: the unsupervised XOOD-M, and the self-supervised XOOD-L.
They rely on the signals captured by the extreme values of the data in the
activation layers of the neural network to distinguish between in distribution
and out of distribution instances. On most benchmark in-distribution-OOD
dataset combinations, both XOOD-L and XOOD-M outperform state-of-the-art
algorithms for standard architectures.

##  6 Limitations of the Work

The current work is focused on image classification. In future, we would like
to extend the work to language models as well as graph neural networks, as
well as test the XOOD algorithms in a more general confidence and calibration
context.

##  7 Impact

When encountered with OOD examples at inference time, neural networks can make
erroneous decisions rather than issuing a warning that their decisions in
these cases cannot be trusted. Such behavior prevents their deployment in
mission-critical application. The XOOD framework can help improve the
trustworthiness of deep neural networks through efficient and accurate OOD
detection. We do not anticipate any negative social consequences of our work.

## References

  * [1] Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava.  How can i explain this to you? an empirical study of deep neural network explanation methods.  Advances in Neural Information Processing Systems, 33:4211–4222, 2020.
  * [2] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.  Generalized out-of-distribution detection: A survey.  arXiv preprint arXiv:2110.11334, 2021.
  * [3] Christopher M Bishop et al.  Neural networks for pattern recognition.  Oxford university press, 1995.
  * [4] Christopher M Bishop and Nasser M Nasrabadi.  Pattern recognition and machine learning, volume 4.  Springer, 2006.
  * [5] David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang.  Xai—explainable artificial intelligence.  Science Robotics, 4(37):eaay7120, 2019.
  * [6] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [7] Anh Mai Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 427–436. IEEE Computer Society, 2015.
  * [8] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  Advances in neural information processing systems, 32, 2019.
  * [9] Chandramouli Shama Sastry and Sageev Oore.  Detecting out-of-distribution examples with gram matrices.  In International Conference on Machine Learning, pages 8491–8501. PMLR, 2020.
  * [10] Shiyu Liang, Yixuan Li, and R. Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [11] Terrance DeVries and Graham W Taylor.  Learning confidence for out-of-distribution detection in neural networks.  arXiv preprint arXiv:1802.04865, 2018.
  * [12] Yiyou Sun, Chuan Guo, and Yixuan Li.  React: Out-of-distribution detection with rectified activations.  Advances in Neural Information Processing Systems, 34, 2021.
  * [13] In-Kwon Yeo and Richard A Johnson.  A new family of power transformations to improve normality or symmetry.  Biometrika, 87(4):954–959, 2000.
  * [14] https://github.com/FrejBerglind/xood.
  * [15] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  Advances in neural information processing systems, 31, 2018.
  * [16] Chandramouli Shama Sastry and Sageev Oore.  Detecting out-of-distribution examples with Gram matrices.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8491–8501. PMLR, 13–18 Jul 2020\.
  * [17] Sina Mohseni, Arash Vahdat, and Jay Yadawa.  Shifting transformation learning for out-of-distribution detection, 2021\.
  * [18] Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich.  Deep anomaly detection with outlier exposure.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [19] Izhak Golan and Ran El-Yaniv.  Deep anomaly detection using geometric transformations.  Advances in neural information processing systems, 31, 2018.
  * [20] Vikash Sehwag, Mung Chiang, and Prateek Mittal.  Ssd: A unified framework for self-supervised outlier detection.  arXiv preprint arXiv:2103.12051, 2021.
  * [21] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  Advances in neural information processing systems, 33:11839–11852, 2020.
  * [22] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  Advances in Neural Information Processing Systems, 33:21464–21475, 2020.
  * [23] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan.  Likelihood ratios for out-of-distribution detection.  Advances in Neural Information Processing Systems, 32, 2019.
  * [24] Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke.  Out-of-distribution detection using an ensemble of self supervised leave-out classifiers.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 550–564, 2018.
  * [25] Qing Yu and Kiyoharu Aizawa.  Unsupervised out-of-distribution detection by maximum classifier discrepancy.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9518–9526, 2019.
  * [26] Ziqian Lin, Sreya Dutta Roy, and Yixuan Li.  Mood: Multi-level out-of-distribution detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15313–15323, 2021.
  * [27] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang.  Self-supervised learning for generalizable out-of-distribution detection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5216–5223, 2020.
  * [28] Rui Huang and Yixuan Li.  Mos: Towards scaling out-of-distribution detection for large semantic space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710–8719, 2021.
  * [29] Ananya Kumar, Aditi Raghunathan, Tengyu Ma, and Percy Liang.  Calibrated ensembles: A simple way to mitigate id-ood accuracy tradeoffs.  In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  * [30] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song.  Scaling out-of-distribution detection for real-world settings.  arXiv preprint arXiv:1911.11132, 2019.
  * [31] Fahim Tajwar, Ananya Kumar, Sang Michael Xie, and Percy Liang.  No true state-of-the-art? ood detection methods are inconsistent across datasets.  arXiv preprint arXiv:2109.05554, 2021.

## A: Additional Experiments

Table 5: Baseline/Energy/React/XOOD-L/XOOD-M.

|  |  | TNR (95% TPR) | Detection Acc. | AUROC
---|---|---|---|---|---
imagenet | resnet18 | Uniform | 2.5/100./100./100./100. | 86.9/99.1/99.7/100./99.9 | 86.9/99.7/99.9/100./100.
|  | Gaussian | 0.0/80.4/72.1/100./100. | 76.0/95.3/94.4/100./99.9 | 65.1/96.2/95.8/100./100.
|  | Places | 23.9/34.8/57.2/37.1/34.2 | 70.7/75.9/82.3/76.7/70.7 | 78.0/83.6/89.7/84.4/77.1
|  | SUN | 26.4/40.0/65.5/46.9/48.6 | 71.6/78.4/85.3/79.9/77.9 | 78.9/86.1/92.7/88.0/85.2
|  | iNaturalist | 41.6/43.2/69.9/60.9/38.9 | 78.9/82.5/87.4/84.8/74.4 | 87.0/89.8/94.6/92.8/82.0
|  | DTD | 28.6/46.7/60.8/56.9/61.8 | 71.3/79.1/83.6/81.4/81.8 | 78.4/86.3/91.4/88.9/89.5
| resnet34 | Uniform | 86.4/100./99.6/100./100. | 94.6/98.1/97.3/100./100. | 97.2/98.7/98.4/100./100.
|  | Gaussian | 48.6/ 0.0/ 0.0/100./100. | 92.5/89.8/89.1/100./100. | 94.3/85.9/85.2/100./100.
|  | Places | 26.1/37.2/64.1/38.6/34.2 | 71.4/77.0/84.2/77.8/69.8 | 79.2/84.6/91.4/85.6/75.9
|  | SUN | 27.6/42.6/73.6/48.4/48.1 | 72.3/79.2/87.5/81.0/76.9 | 79.8/86.6/94.0/88.6/83.8
|  | iNaturalist | 40.5/46.0/75.9/65.1/40.2 | 78.3/82.2/88.8/86.1/74.1 | 86.6/89.7/95.7/93.7/81.5
|  | DTD | 29.9/44.9/55.3/61.5/70.5 | 71.7/79.3/83.5/83.7/85.1 | 79.1/86.5/91.2/91.0/92.4
| resnet50 | Uniform | 3.6/100./100./100./100. | 89.7/98.3/99.6/100./100. | 89.8/98.9/99.8/100./100.
|  | Gaussian | 0.0/ 0.0/ 0.0/100./100. | 69.7/81.6/87.6/100./100. | 53.2/75.5/85.8/100./100.
|  | Places | 28.0/34.8/66.6/43.5/32.0 | 72.8/76.9/85.0/79.6/69.9 | 80.6/84.2/92.0/87.2/75.9
|  | SUN | 31.0/41.9/76.0/54.4/47.8 | 74.0/79.4/87.9/82.7/77.6 | 81.7/86.8/94.4/90.4/84.8
|  | iNaturalist | 46.8/46.3/80.5/71.2/44.8 | 79.9/83.9/90.0/88.3/76.9 | 88.4/90.7/96.4/95.1/84.6
|  | DTD | 33.5/47.9/54.3/64.8/71.0 | 73.0/79.7/82.8/84.1/85.4 | 80.4/86.8/90.5/91.3/92.9
| resnet101 | Uniform | 27.1/96.7/100./100./100. | 87.3/96.5/99.3/100./100. | 90.8/97.2/99.6/100./100.
|  | Gaussian | 0.0/ 0.0/100./100./100. | 79.0/88.0/98.3/100./100. | 70.6/83.9/98.2/100./100.
|  | Places | 28.8/39.4/63.7/42.7/26.7 | 72.9/78.2/84.2/79.7/71.2 | 80.6/85.5/91.3/87.2/77.0
|  | SUN | 32.1/46.8/73.4/51.0/41.3 | 73.9/80.6/87.2/82.6/78.3 | 81.4/87.7/94.0/89.9/85.2
|  | iNaturalist | 42.9/41.6/79.1/61.9/32.5 | 77.4/81.1/89.5/86.3/75.7 | 86.2/88.3/96.1/93.4/82.8
|  | DTD | 37.4/52.6/59.6/67.5/68.6 | 73.4/80.8/83.5/85.4/87.2 | 81.5/87.9/91.4/92.4/94.0
cifar10 | resnet | Uniform | 72.7/82.0/90.2/100./100. | 93.6/94.2/94.6/100./100. | 96.1/96.2/96.8/100./100.
|  | Gaussian | 90.8/98.8/98.1/100./100. | 95.7/97.0/96.6/100./100. | 97.5/98.0/98.2/100./100.
|  | TinyImageNet (Crop) | 46.9/62.1/65.1/98.0/99.1 | 85.5/86.5/86.5/96.7/98.0 | 91.6/93.1/92.6/99.5/99.8
|  | TinyImageNet (Resize) | 45.1/59.5/63.9/97.9/99.5 | 85.1/85.8/86.3/96.9/98.8 | 91.1/92.5/92.4/99.6/99.9
|  | LSUN (Crop) | 49.1/65.7/62.5/97.0/94.6 | 86.4/87.5/85.1/96.0/94.8 | 92.0/93.9/90.9/99.4/99.0
|  | LSUN (Resize) | 46.3/62.9/69.4/99.1/99.9 | 85.5/86.7/88.2/97.8/99.2 | 91.3/92.9/94.4/99.8/100.
|  | iSUN | 45.6/62.0/68.0/99.1/99.7 | 85.2/86.3/87.9/97.8/98.8 | 91.2/92.8/94.2/99.8/99.9
|  | SVHN | 33.3/47.9/37.9/96.4/98.4 | 85.4/85.5/77.5/96.1/97.6 | 90.1/91.3/83.3/99.3/99.7
|  | Cifar100 | 34.2/44.0/44.0/43.6/32.0 | 80.5/80.7/78.7/81.7/73.7 | 86.6/87.3/84.4/88.8/80.1
| densenet | Uniform | 78.0/96.7/100./100./100. | 94.8/96.3/98.1/100./100. | 96.7/97.4/99.2/100./100.
|  | Gaussian | 89.3/100./100./100./100. | 95.4/97.9/98.4/100./100. | 97.7/98.9/99.5/100./100.
|  | TinyImageNet (Crop) | 57.4/84.1/68.6/98.6/97.8 | 88.2/91.6/87.6/97.7/96.6 | 94.0/97.1/94.1/99.7/99.5
|  | TinyImageNet (Resize) | 60.1/86.2/68.7/99.1/99.2 | 88.8/92.3/87.3/98.2/97.8 | 94.3/97.5/94.0/99.8/99.8
|  | LSUN (Crop) | 52.4/76.1/66.4/96.9/91.7 | 87.2/89.8/88.4/96.1/93.6 | 93.2/96.0/94.4/99.3/98.4
|  | LSUN (Resize) | 67.3/92.3/83.1/99.4/99.7 | 90.5/93.9/91.4/98.5/98.6 | 95.6/98.4/96.9/99.9/99.9
|  | iSUN | 63.8/89.3/78.4/99.5/99.4 | 89.5/93.1/90.0/98.5/98.1 | 95.0/98.0/96.1/99.9/99.8
|  | SVHN | 40.9/52.9/65.3/96.5/97.2 | 83.4/83.5/88.1/96.2/96.7 | 90.1/91.1/93.8/99.2/99.3
|  | Cifar100 | 41.7/55.3/49.3/47.6/19.9 | 83.2/83.6/82.2/78.6/64.8 | 89.6/90.7/89.2/87.0/70.7
svhn | resnet | Uniform | 85.9/86.7/71.9/100./100. | 93.0/92.5/84.7/99.5/99.7 | 96.1/95.6/89.4/100./100.
|  | Gaussian | 86.1/87.3/75.2/100./100. | 93.0/92.7/86.1/99.7/99.9 | 96.3/96.0/90.5/100./100.
|  | TinyImageNet (Crop) | 81.3/83.0/74.4/99.7/99.7 | 91.1/90.7/85.5/98.7/98.3 | 94.2/93.8/88.7/99.9/99.8
|  | TinyImageNet (Resize) | 79.8/81.5/74.1/99.7/99.8 | 90.6/90.1/85.2/98.4/98.6 | 93.6/93.0/88.9/99.9/99.8
|  | LSUN (Crop) | 77.2/79.0/71.5/99.2/98.6 | 89.9/89.4/84.0/97.7/97.0 | 92.9/92.5/86.2/99.7/99.5
|  | LSUN (Resize) | 75.5/77.2/67.2/99.5/99.9 | 89.1/88.3/82.0/98.3/98.9 | 91.6/90.7/84.9/99.9/99.9
|  | iSUN | 78.1/79.8/69.0/99.8/99.9 | 89.9/89.4/82.7/98.7/98.9 | 92.3/91.5/85.6/99.9/99.9
|  | Cifar100 | 77.7/78.9/74.0/97.8/95.5 | 89.5/88.9/85.2/96.4/95.5 | 92.4/91.5/89.1/99.3/98.9
| densenet | Uniform | 66.3/64.0/37.9/100./100. | 87.7/83.6/79.0/100./100. | 93.2/90.1/87.2/100./100.
|  | Gaussian | 88.2/90.8/55.9/100./100. | 93.5/93.2/84.1/100./100. | 97.4/97.8/91.2/100./100.
|  | TinyImageNet (Crop) | 79.1/77.1/72.4/100./99.9 | 89.9/87.4/86.6/99.4/99.3 | 94.7/92.8/93.2/100./100.
|  | TinyImageNet (Resize) | 80.1/78.7/73.1/100./100. | 90.3/88.1/87.0/99.5/99.6 | 94.9/93.4/93.5/100./100.
|  | LSUN (Crop) | 74.0/67.8/71.6/99.7/99.2 | 88.1/83.2/85.5/98.7/97.8 | 93.0/88.5/92.1/99.9/99.8
|  | LSUN (Resize) | 77.4/76.8/70.6/100./100. | 89.3/87.1/85.4/99.8/99.8 | 94.2/92.6/92.3/100./100.
|  | iSUN | 78.7/78.2/70.2/100./100. | 89.8/87.6/85.8/99.7/99.8 | 94.5/92.9/92.6/100./100.
|  | Cifar100 | 68.5/64.6/74.8/98.1/97.9 | 86.6/83.0/87.3/96.8/96.7 | 91.4/88.2/93.9/99.5/99.4
cifar100 | resnet | Uniform | 12.1/ 1.6/ 1.3/100./100. | 81.5/84.7/80.6/100./100. | 85.2/86.5/81.6/100./100.
|  | Gaussian | 0.0/ 0.0/ 0.0/100./100. | 60.7/60.8/81.7/100./100. | 45.1/46.1/76.6/100./100.
|  | TinyImageNet (Crop) | 25.0/27.1/28.5/90.8/95.0 | 72.6/74.4/71.2/93.2/95.0 | 79.7/81.6/78.9/98.4/98.9
|  | TinyImageNet (Resize) | 21.3/25.3/25.6/89.4/97.9 | 71.0/73.0/68.6/92.6/96.7 | 77.2/80.1/76.2/98.1/99.5
|  | LSUN (Crop) | 19.2/18.0/28.0/85.3/76.2 | 69.6/69.2/77.7/91.4/88.2 | 75.6/75.1/84.3/97.5/95.3
|  | LSUN (Resize) | 19.4/23.3/23.0/90.2/98.5 | 70.0/71.9/68.5/92.9/97.4 | 75.7/78.5/75.5/98.3/99.6
|  | iSUN | 17.7/20.9/21.5/89.6/97.0 | 70.3/72.1/68.5/92.8/96.2 | 75.8/78.2/75.3/98.3/99.3
|  | SVHN | 21.3/18.8/26.3/87.2/92.1 | 73.4/73.5/77.3/91.5/93.6 | 79.5/79.5/84.2/97.6/98.2
| densenet | Uniform | 0.0/ 0.0/ 0.0/100./100. | 64.2/66.7/76.1/100./100. | 43.3/50.3/74.0/100./100.
|  | Gaussian | 0.0/ 0.0/ 0.0/100./100. | 58.9/53.0/69.8/100./100. | 30.8/15.6/58.2/100./100.
|  | TinyImageNet (Crop) | 23.5/39.3/65.1/95.4/89.2 | 68.8/76.9/86.4/95.2/92.7 | 75.9/84.8/93.7/99.1/97.5
|  | TinyImageNet (Resize) | 16.6/22.8/64.4/95.7/96.1 | 65.9/71.4/86.8/95.4/95.6 | 71.5/78.2/93.8/99.2/98.9
|  | LSUN (Crop) | 27.5/51.9/43.8/84.9/68.4 | 72.5/82.2/79.0/91.3/85.0 | 79.8/90.1/87.2/97.4/92.6
|  | LSUN (Resize) | 15.8/22.0/72.8/94.3/97.9 | 65.0/72.4/88.8/94.8/96.7 | 70.6/79.5/95.3/99.0/99.3
|  | iSUN | 14.2/18.2/68.4/94.0/96.3 | 64.0/70.5/86.9/94.6/95.7 | 69.5/77.2/93.9/98.9/99.0
|  | SVHN | 25.2/33.2/33.1/88.8/87.9 | 75.5/80.5/79.5/92.3/91.9 | 82.4/87.7/86.9/97.7/97.1

## B: Feature Selection

In this section, we report experimentation with a number of different
features: only using the minimum or maximum, the percentage of positive values
(positivity); the sum; $L_{p}$-norms; $L_{p}$-norms of $\text{ReLU}(x)$ and
$\text{ReLU}(-x)$ (Split Lp). As one can see in Tables 6 and 7, neither
replacing the extreme values by these features, nor adding these features
while still using the extreme values improves the detection performance.

| cifar10 | svhn | cifar100
---|---|---|---
| resnet | densenet | resnet | densenet | resnet | densenet
| XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L
Min & Max | 99.7 | 99.47 | 99.4 | 99.55 | 99.77 | 99.85 | 99.94 | 99.97 | 98.42 | 97.86 | 97.3 | 98.29
Min | 99.63 | 99.4 | 99.23 | 99.41 | 99.43 | 99.77 | 99.87 | 99.93 | 98.17 | 98.04 | 96.63 | 96.84
Max | 99.59 | 99.32 | 99.43 | 99.54 | 99.6 | 99.69 | 99.91 | 99.97 | 97.4 | 97.13 | 97.37 | 97.48
Positivity | 97.86 | 98.76 | 95.56 | 98.22 | 99.43 | 99.82 | 99.45 | 99.81 | 89.76 | 90.47 | 91.38 | 92.06
Sum | 97.67 | 98.92 | 96.06 | 97.1 | 99.54 | 99.81 | 99.64 | 99.75 | 92.2 | 96.41 | 92.32 | 92.82
L1 | 98.15 | 98.75 | 98.11 | 98.2 | 98.87 | 99.53 | 99.69 | 99.74 | 93.55 | 95.77 | 92.22 | 94.18
L2 | 98.5 | 98.78 | 98.57 | 98.68 | 98.98 | 99.51 | 99.78 | 99.82 | 93.71 | 96.01 | 93.19 | 94.78
L3 | 98.83 | 98.9 | 98.83 | 98.99 | 99.16 | 99.54 | 99.8 | 99.87 | 94.74 | 95.96 | 94.01 | 95.37
Split L1 | 98.59 | 98.96 | 98.2 | 98.19 | 99.69 | 99.78 | 99.82 | 99.88 | 95.36 | 96.23 | 93.03 | 93.13
Split L2 | 98.94 | 99.08 | 98.61 | 98.89 | 99.61 | 99.72 | 99.86 | 99.89 | 95.94 | 97.36 | 93.6 | 94.74
Split L3 | 99.23 | 99.18 | 98.92 | 99.24 | 99.69 | 99.82 | 99.88 | 99.92 | 96.8 | 97.5 | 94.33 | 95.66

Table 6: AUROC-score when replacing the extreme values with other features
which have linear computational complexity.

| cifar10 | svhn | cifar100
---|---|---|---
| resnet | densenet | resnet | densenet | resnet | densenet
| XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L | XOOD-M | XOOD-L
Min & Max | 99.7 | 99.46 | 99.4 | 99.5 | 99.77 | 99.85 | 99.94 | 99.97 | 98.42 | 97.8 | 97.3 | 98.18
Positivity | 99.68 | 99.54 | 99.27 | 99.34 | 99.88 | 99.92 | 99.95 | 99.98 | 98.14 | 97.21 | 97.0 | 97.52
Sum | 99.64 | 99.5 | 99.28 | 99.19 | 99.9 | 99.93 | 99.94 | 99.96 | 98.22 | 97.66 | 96.85 | 97.45
L1 | 99.62 | 99.43 | 99.33 | 99.38 | 99.79 | 99.87 | 99.93 | 99.92 | 98.3 | 97.5 | 96.63 | 97.74
L2 | 99.63 | 99.42 | 99.37 | 99.39 | 99.79 | 99.78 | 99.94 | 99.92 | 98.18 | 97.61 | 96.69 | 97.86
L3 | 99.63 | 99.43 | 99.39 | 99.39 | 99.8 | 99.78 | 99.94 | 99.93 | 98.18 | 97.58 | 96.76 | 97.97
Split L1 | 99.57 | 99.43 | 99.26 | 99.17 | 99.9 | 99.91 | 99.93 | 99.95 | 98.21 | 97.5 | 96.33 | 97.11
Split L2 | 99.59 | 99.41 | 99.29 | 99.22 | 99.86 | 99.87 | 99.94 | 99.94 | 98.13 | 97.82 | 96.28 | 97.51
Split L3 | 99.62 | 99.44 | 99.33 | 99.25 | 99.87 | 99.87 | 99.93 | 99.94 | 98.21 | 97.85 | 96.36 | 97.74

Table 7: AUROC-score when extracting additional extreme values before each
activation layer.

## C: Extreme Value Distributions

Figure 4: Distribution of maximum values before an activation layer inside
ResNet34. Green represents CIFAR-10 (In-distribution (ID)) and Orange
represents OOD. Note how in-distribution data lies in a single mode, while
out-of-distribution data can be on either side. Figure 5: Distribution of
minimum values before an activation layer inside ResNet34. Green represents
CIFAR-10 (In-distribution (ID)) and Orange represents OOD. Note how in-
distribution data lies in a single mode, while out-of-distribution data can be
on either side.

## D: False Positive Rate

To summarize the performance of the various methods in Table 1 and 2, we
computed the mean FPR (95% TPR) for each OOD detection task, except the
CIFAR-10$-$CIFAR-100 and CIFAR-100$-$CIFAR-10 in-distribution-OOD
combinations, given their overlap. These averages are shown in Table 8. Note
that XOOD-L and XOOD-M have the best average scores for both DenseNet and
ResNet. For DenseNet, XOOD-L reduced the FPR by 48% compared to the best other
method (Gram), and for ResNet34 XOOD-M reduced the FPR by 51% compared to the
best other method (Gram).

| Baseline | ODIN | Mahalanobis | Gram | XOOD-L | XOOD-M
---|---|---|---|---|---|---
ResNet34 | 55.7 | 37.6 | 8.6 | 7.1 | 5.2 | 3.5
DenseNet | 51.8 | 28.4 | 15.2 | 6.9 | 3.6 | 5.1
Table 8: Average FPR at 95% TPR. Lower score is better.