# Supplementary Information for “An Axiomatisation of Error Intolerant
Estimation”

Michael<EMAIL_ADDRESS>[ School of
Computing Technologies, RMIT Universitypresep=, ]e1

(2024)

††volume: TBA††issue: TBA

2010.00000

##  Appendix A Supplementary proofs

We provide in this section complete proofs for the claims of the main paper.

###  A.1 The EIC estimator

######  Theorem A.1 (Theorem 2.7 of the main text).

Let $(\bm{x},\bm{\theta})$ be an elementary $\bm{\theta}$-continuous
estimation problem over an open set $\Theta$, let $L$ be a smooth loss
function discriminative for it, such that $\hat{\theta}^{L}_{\text{EIC}}$ is a
well-defined set estimator on $(\bm{x},\bm{\theta})$, and let
$\hat{\theta}_{L}$ be a point estimator satisfying AIA with respect to $L$.
For all $x\in X$,

$\hat{\theta}_{L}(x)\in\hat{\theta}^{L}_{\text{EIC}}(x).$

###### Proof.

For a choice of a risk attitude spectrum, $\mathcal{T}=\\{T_{\epsilon}\\}$,
define $\mathcal{A}=\\{A_{\epsilon}=S_{\epsilon}\circ T_{\epsilon}\\}$ such
that each $S_{\epsilon}$ is an affine function mapping $T_{\epsilon}(0)$ to
$1$ and the maximum of $T_{\epsilon}$ to $0$. Minimising the expected loss
$\mathbf{E}[T_{\epsilon}(L(\bm{\theta},\theta))|\bm{x}=x]$ for choosing
$\hat{\theta}_{L}(x)=\theta$ at a given $\epsilon$ and a given $x$ is
equivalent to maximising the expected utility
$\mathbf{E}[A_{\epsilon}(L(\bm{\theta},\theta))|\bm{x}=x]$.

When computing this expected utility,

$\mathbf{E}[A_{\epsilon}(L(\bm{\theta},\theta))|x]=\int_{\Theta}f(\theta^{\prime}|x)A_{\epsilon}(L(\theta^{\prime},\theta))\text{d}\theta^{\prime},$
(A.1)

for $\epsilon$ values tending to zero, one only needs to consider the integral
over $B(\theta,\delta)$, the (closed set) ball of radius $\delta$ around
$\theta$, for any $\delta>0$, as for a sufficiently small $\epsilon$, the rest
of the integral values will be zero by the discriminativity assumption. We
will, in particular, assume that any chosen $\delta$ is small enough so that
$B(\theta,\delta)\subseteq\Theta$.

Because $L$ is smooth, the second derivative (i.e., the Hessian)
$H_{L}^{\theta}$ is defined and finite everywhere, and its own derivative
(i.e., the third derivative) is continuous. Thus, this third derivative is
bounded inside the closed set $B(\theta,\delta)$. Let $m$ be its maximum
absolute value in any direction.

Using a second-order Taylor approximation, we can therefore write, for
$\theta^{\prime}\in B(\theta,\delta)$,

$L(\theta^{\prime},\theta)=\frac{1}{2}(\theta^{\prime}-\theta)^{T}H_{L}^{\theta}(\theta^{\prime}-\theta)\pm\frac{m}{6}|\theta^{\prime}-\theta|^{3}.$
(A.2)

Let $H^{\theta}_{\text{min}}=H^{\theta}_{L}-\frac{m\delta}{3}I_{M}$ and
$H^{\theta}_{\text{max}}=H^{\theta}_{L}+\frac{m\delta}{3}I_{M}$, where $I_{M}$
is the $M\times M$ identity matrix. For all $\theta^{\prime}$ values in
$B(\theta,\delta)$,

$\frac{1}{2}(\theta^{\prime}-\theta)^{T}H^{\theta}_{\text{min}}(\theta^{\prime}-\theta)\leq
L(\theta^{\prime},\theta)\leq\frac{1}{2}(\theta^{\prime}-\theta)^{T}H^{\theta}_{\text{max}}(\theta^{\prime}-\theta).$
(A.3)

Because we also assumed that the posterior distribution $f(\theta|x)$ is
continuous, inside $B(\theta,\delta)$ it is also bounded from above and below.
Let its bounds be $f_{\text{max}}$ and $f_{\text{min}}$.

From (A.3), we know that the value of (A.1) can be bounded by

$\begin{split}f_{\text{min}}&\int_{B(\theta,\delta)}A_{\epsilon}\left(\frac{1}{2}(\theta^{\prime}-\theta)^{T}H^{\theta}_{\text{max}}(\theta^{\prime}-\theta)\right)\text{d}\theta^{\prime}\leq\int_{\Theta}f(\theta^{\prime}|x)A_{\epsilon}(L(\theta^{\prime},\theta))\text{d}\theta^{\prime}\\\
&\leq
f_{\text{max}}\int_{B(\theta,\delta)}A_{\epsilon}\left(\frac{1}{2}(\theta^{\prime}-\theta)^{T}H^{\theta}_{\text{min}}(\theta^{\prime}-\theta)\right)\text{d}\theta^{\prime}.\end{split}$
(A.4)

Because we assumed that $\hat{\theta}^{L}_{\text{EIC}}$ is well-defined, we
know that $H^{\theta}_{L}$ is positive definite, so for a small enough
$\delta$ $H^{\theta}_{\text{min}}$ and $H^{\theta}_{\text{max}}$ will be, too.
Thus, for $H$ being either $H^{\theta}_{\text{min}}$ or
$H^{\theta}_{\text{max}}$ there is a positive infimum to the value of
$\frac{1}{2}(\theta^{\prime}-\theta)^{T}H(\theta^{\prime}-\theta)$ outside
$B(\theta,\delta)$.

When $\epsilon$ is smaller than this infimum, the integration bounds in (A.4)
can be switched from $B(\theta,\delta)$ to $\mathbb{R}^{M}$, without this
impacting the truth of the inequality, because all $\theta^{\prime}$ values
outside $B(\theta,\delta)$ will contribute zero to the integral.

Values of the form

$f\int_{\mathbb{R}^{M}}A_{\epsilon}\left(\frac{1}{2}(\theta^{\prime}-\theta)^{T}H(\theta^{\prime}-\theta)\right)\text{d}\theta^{\prime}$

can be computed via a Jacobian transformation as

$\frac{f}{\sqrt{|H|}}\int_{\mathbb{R}^{M}}A_{\epsilon}\left(\frac{1}{2}|\omega|^{2}\right)\text{d}\omega,$
(A.5)

where the integral is a multiplicative factor independent of $\theta$.

As $\epsilon$, and therefore also $\delta$, tends to zero, both
$|H^{\theta}_{\text{min}}|$ and $|H^{\theta}_{\text{max}}|$ tend to
$|H^{\theta}_{L}|$, and both $f_{\text{min}}$ and $f_{\text{max}}$ converge to
$f(\theta|x)$.

Noting that $f(\theta|x)$ is positive by our assumption on estimation
problems, and that $|H^{\theta}_{L}|$ is positive by our assumption that
$\hat{\theta}^{L}_{\text{EIC}}$ is well-defined, we reach

$\lim_{\epsilon\to
0}\frac{\mathbf{E}[A_{\epsilon}(L(\bm{\theta},\theta))|x]}{\int_{\mathbb{R}^{M}}A_{\epsilon}\left(\frac{1}{2}|\omega|^{2}\right)\text{d}\omega}=\frac{f(\theta|x)}{\sqrt{|H^{\theta}_{L}|}}>0.$

Suppose, contrary to the claim, that
$\hat{\theta}_{L}(x)\notin\hat{\theta}^{L}_{\text{EIC}}(x)$. Because
$\hat{\theta}^{L}_{\text{EIC}}$ was assumed to be a well-defined set
estimator, we know that $\hat{\theta}^{L}_{\text{EIC}}(x)$ is not empty.
Choose $\theta^{*}\in\hat{\theta}^{L}_{\text{EIC}}(x)$.

By its definition, EIC maximises the metric
$f(\theta|x)/\sqrt{|H^{\theta}_{L}|}$, so

$\lim_{\epsilon\to
0}\frac{\mathbf{E}[A_{\epsilon}(L(\bm{\theta},\hat{\theta}_{L}(x)))|x]}{\mathbf{E}[A_{\epsilon}(L(\bm{\theta},\theta^{*}))|x]}=\frac{f(\hat{\theta}_{L}(x)|x)/\sqrt{\left|H^{\hat{\theta}_{L}(x)}_{L}\right|}}{f(\theta^{*}|x)/\sqrt{\left|H^{\theta^{*}}_{L}\right|}}<1.$

This proves that for every low enough $\epsilon$,

$\mathbf{E}[A_{\epsilon}(L(\bm{\theta},\hat{\theta}_{L}(x)))|x]<\mathbf{E}[A_{\epsilon}(L(\bm{\theta},\theta^{*}))|x],$

contradicting our assumption that $\hat{\theta}_{L}$ satisfies AIA. ∎

###  A.2 EIC estimation over reasonable loss

Our main theorem for continuous problems is as follows.

######  Theorem A.2 (Theorem 3.1 of the main text).

If $(\bm{x},\bm{\theta})$ is an elementary continuous estimation problem for
which $\hat{\theta}_{\text{WF}}$ is a well-defined set estimator, and if $L$
is a smooth and problem continuous loss function, sensitive on
$(\bm{x},\bm{\theta})$’s type, that satisfies all of IIA, IRP and IRO, then

$\hat{\theta}^{L}_{\text{EIC}}=\hat{\theta}_{\text{WF}}.$

In the main text, we prove the theorem through a progression of lemmas. While
the proof of the main theorem exists in the main text, we supplement it here
with the complete proofs of its underlying lemmas.

Recall that, throughout, $M$ and $N$ always represent the dimensions of the
parameter space and the observation space, respectively.

######  Lemma A.3 (Lemma 3.2 of the main text).

For $\bm{\theta}$-continuous estimation problems $(\bm{x},\bm{\theta})$, if
$L$ satisfies both IIA and IRP then
$L_{(\bm{x},\bm{\theta})}(\theta_{1},\theta_{2})$ is a function only of the
data distributions $P_{\theta_{1}}$ and $P_{\theta_{2}}$ and of $M$, the
dimension of the parameter space.

###### Proof.

The IIA axiom is tantamount to stating that
$L_{(\bm{x},\bm{\theta})}(\theta_{1},\theta_{2})$ is dependent only on the
following:

  1. 1.

The function’s inputs $\theta_{1}$ and $\theta_{2}$,

  2. 2.

The data distributions $P_{\theta_{1}}$ and $P_{\theta_{2}}$,

  3. 3.

The priors $f(\theta_{1})$ and $f(\theta_{2})$, and

  4. 4.

The problem’s parameter space dimension $M$ (noting that all other elements of
the problem type are already fixed by specifying the data distributions).

We can assume without loss of generality that $\theta_{1}\neq\theta_{2}$, or
else the value of $L(\theta_{1},\theta_{2})$ can be determined to be zero by
(1.1).

Our first claim is that, due to IRP, $L$ can also not depend on the problem’s
prior probability densities $f(\theta_{1})$ and $f(\theta_{2})$. To show this,
construct a diffeomorphism $F:\mathbb{R}^{M}\to\mathbb{R}^{M}$ in the
following way.

Let $(\vec{v}_{1},\ldots,\vec{v}_{M})$ be an orthogonal basis for
$\mathbb{R}^{M}$ wherein $\vec{v}_{1}=\theta_{2}-\theta_{1}$. We design $F$ as

$F\left(\theta_{1}+\sum_{i=1}^{M}b_{i}\vec{v}_{i}\right)=\theta_{1}+F_{1}(b_{1})\vec{v}_{1}+\sum_{i=2}^{M}b_{i}\vec{v}_{i},$

where $F_{1}:\mathbb{R}\to\mathbb{R}$ is a continuous, differentiable function
onto $\mathbb{R}$, with a derivative that is positive everywhere, satisfying

  1. 1.

$F_{1}(0)=0$ and $F_{1}(1)=1$, and

  2. 2.

$F_{1}^{\prime}(0)=d_{0}$ and $F_{1}^{\prime}(1)=d_{1}$, for some arbitrary
positive values $d_{0}$ and $d_{1}$ to be chosen later.

Such a function is straightforward to construct for any values of $d_{0}$ and
$d_{1}$, and by an appropriate choice of these values, it is possible to map
$(\bm{x},\bm{\theta})$ into $(\bm{x},F(\bm{\theta}))$ in a way that does not
change $P_{\theta_{1}}$ or $P_{\theta_{2}}$, but adjusts $f(\theta_{1})$ and
$f(\theta_{2})$ to any desired positive values.

Lastly, we show that $L(\theta_{1},\theta_{2})$ can also not depend on the
values of $\theta_{1}$ and $\theta_{2}$ other than through $P_{\theta_{1}}$
and $P_{\theta_{2}}$. For this we once again invoke IRP: by applying a
similarity transform on $\Theta$, we can map any $\theta_{1}$ and $\theta_{2}$
values into arbitrary new values, again without this affecting their
respective conditional data distributions. ∎

######  Lemma A.4 (Lemma 3.3 of the main text).

The function $c$ is $\mathcal{M}$-continuous for continuous distributions, in
the sense that if both
$\left(f_{i}\right)_{i\in\mathbb{N}}\xrightarrow{\mathcal{M}}f$ and
$\left(g_{i}\right)_{i\in\mathbb{N}}\xrightarrow{\mathcal{M}}g$, where $f$ and
$g$ are pdfs and $\left(f_{i}\right)_{i\in\mathbb{N}}$ and
$\left(g_{i}\right)_{i\in\mathbb{N}}$ are pdf sequences, then
$\left(c[f_{i},g_{i}]\right)_{i\in\mathbb{N}}\xrightarrow{\mathcal{M}}c[f,g]$.

###### Proof.

For any $t_{0}\in[0,1]$, let $r_{0}=c[f,g](t_{0})$, and let $t_{\text{min}}$
and $t_{\text{max}}$ be the infimum $t$ and the supremum $t$, respectively,
for which $c[f,g](t)=r_{0}$.

Because $c[f,g]$ is a monotone increasing function, $t_{\text{min}}$ is also
the supremum $t$ for which $c[f,g](t)<r_{0}$ (unless no such $t$ exists, in
which case $t_{\text{min}}=0$), so by definition $t_{\text{min}}$ is the
supremum of $\mathbf{P}_{\bm{x}\sim g}(r_{f,g}(\bm{x})\leq r)$, for all
$r<r_{0}$, from which we conclude that $t_{\text{min}}=\mathbf{P}_{\bm{x}\sim
g}(r_{f,g}(\bm{x})<r_{0})$.

Because $\left(f_{i}\right)\xrightarrow{\mathcal{M}}f$ and
$\left(g_{i}\right)\xrightarrow{\mathcal{M}}g$, we can use (1.3) to determine
that using a large enough $i$ both $f_{i}(x)/f(x)$ and $g_{i}(x)/g(x)$ are
arbitrarily close to $1$ in all but a diminishing measure of $X$. Hence,

$\lim_{i\to\infty}\mathbf{P}_{\bm{x}\sim
g_{i}}(r_{f_{i},g_{i}}(\bm{x})<r_{0})=\lim_{i\to\infty}\mathbf{P}_{\bm{x}\sim
g}(r_{f_{i},g_{i}}(\bm{x})<r_{0})=\mathbf{P}_{\bm{x}\sim
g}(r_{f,g}(\bm{x})<r_{0})=t_{\text{min}}.$

We conclude that for any $t^{+}>t_{\text{min}}$ a large enough $i$ will
satisfy $\mathbf{P}_{\bm{x}\sim g_{i}}(r_{f_{i},g_{i}}(\bm{x})<r_{0})<t^{+}$,
and hence $c[f_{i},g_{i}](t^{+})\geq r_{0}$. For all such $t^{+}$, and in
particular for all $t^{+}>t_{0}$,

$\liminf_{i\to\infty}c[f_{i},g_{i}](t^{+})\geq r_{0}=c[f,g](t_{0}).$ (A.6)

A symmetrical analysis on $t_{\text{max}}$ yields that for all $t^{-}<t_{0}$,

$\limsup_{i\to\infty}c[f_{i},g_{i}](t^{-})\leq c[f,g](t_{0}).$ (A.7)

Consider, now, the functions

$c_{\text{sup}}(t)\stackrel{{\scriptstyle\text{def}}}{{=}}\limsup_{i\to\infty}c[f_{i},g_{i}](t)$

and

$c_{\text{inf}}(t)\stackrel{{\scriptstyle\text{def}}}{{=}}\liminf_{i\to\infty}c[f_{i},g_{i}](t).$

Because each $c[f_{i},g_{i}]$ is monotone increasing, so are $c_{\text{sup}}$
and $c_{\text{inf}}$. Monotone functions can only have countably many
discontinuity points (for a total of measure zero). For any $t_{0}$ that is
not a discontinuity point of either function, we have from (A.6) and (A.7)
that $\lim_{i\to\infty}c[f_{i},g_{i}](t_{0})$ exists and equals
$c[f,g](t_{0})$, so the conditions of convergence in measure hold. ∎

######  Lemma A.5 (Lemma 3.4 of the main text).

If $L$ is a problem continuous conditional-distribution-based loss function
that satisfies IRO, and $p$ and $q$ are piecewise-continuous probability
density functions that are data distributions in an estimation problem
$(\bm{x},\bm{\theta})$, then $L_{(\bm{x},\bm{\theta})}(p,q)$ depends only on
$c[p,q]$ and on the type of $(\bm{x},\bm{\theta})$.

###### Proof.

Fix the problem type.

Let us first note that the following conditions are equivalent.

  1. 1.

$c[p,q]$ equals the indicator function on $(0,1]$ in all but a measure zero of
values,

  2. 2.

$p$ equals $q$ in all but a measure zero of $X$, the joint support of $p$ and
$q$,

  3. 3.

$p$ and $q$ are $\mathcal{M}$-equivalent, in the sense that a sequence of
elements all equal to $p$ nevertheless satisfies the condition of
$\mathcal{M}$-convergence to $q$, and

  4. 4.

$L(p,q)=0$.

The equivalence of the first and second conditions stems from the definition
of $c[p,q]$, the equivalence of the second and third conditions stems from the
definition of $\mathcal{M}$-convergence, and the equivalence of the last two
conditions follows from problem continuity, together with (1.1).

Hence, if $c[p,q]$ equals the indicator function on $(0,1]$, this uniquely
determines the value of $L(p,q)$ to be $0$, in accordance with the lemma.

It remains to be proved that such a functional relationship from $c[p,q]$ to
$L(p,q)$ holds for all other $c[p,q]$ as well, but for this, in accordance
with the second condition stated, we can safely assume that $p$ and $q$ differ
in a positive measure of $X$. Because both $p$ and $q$ integrate to $1$, we
can consequently also assume that the two are not linearly dependent. The
remainder of this proof follows under this assumption.

Note first that because $L$ is known to be conditional-distribution-based, the
value of $L(p,q)$ is not dependent on the full details of the estimation
problem: it will be the same in any estimation problem of the same type that
contains the conditional data distributions $p$ and $q$. Let us therefore
design an estimation problem that is easy to analyse but contains these two
conditional data distributions.

Let $(\bm{x},\bm{\theta})$ be an estimation problem with $\Theta=[0,1]^{M}$
and a uniform prior on $\bm{\theta}$. Its conditional data distribution at
$\theta_{0}=(0,\ldots,0)$ will be $p$, at $\theta_{1}=(1,0,\ldots,0)$ will be
$q$, and we will choose piecewise continuous conditional data distributions,
$f_{\theta}$, over the rest of the $\theta\in\\{0,1\\}^{M}$ so all $2^{M}$ are
linearly independent, share the same support, and differ from each other over
a positive measure of $X$, and so that their respective $r_{f_{\theta},q}$
values are all monotone weakly increasing with $r_{p,q}$ and with each other.

If $M>1$, we further choose $f_{\theta_{2}}$ at $\theta_{2}=(0,1,0,\ldots,0)$
to satisfy that $c[f_{\theta_{2}},q]$ is monotone strictly increasing. If
$M=1$, this is not necessary and we, instead, choose $\theta_{2}=\theta_{0}$.

We then extend this description of $f^{(\bm{x},\bm{\theta})}_{\theta}$ at
$\\{0,1\\}^{M}$ into a full characterisation of all the problem’s conditional
data distributions by setting these to be multilinear functions of the
coordinates of $\theta$.

We now create a sequence of estimation problems, $(\bm{x}_{i},\bm{\theta})$ to
satisfy the conditions of $L$’s problem-continuity assumption. We do this by
constructing a sequence $\left(S_{i}\right)_{i\in\mathbb{N}}$ of subsets of
$\mathbb{R}^{N}$ such that for all $\theta\in\\{0,1\\}^{M}$,
$\mathbf{P}(\bm{x}\in S_{i}|\bm{\theta}=\theta)$ tends to $1$, and for every
$x\in S_{i}$,

$\left|f^{(\bm{x},\bm{\theta})}_{\theta}(x)-f^{(\bm{x}_{i},\bm{\theta})}_{\theta}(x)\right|<\epsilon_{i},$
(A.8)

for an arbitrarily-chosen sequence
$\left(\epsilon_{i}\right)_{i\in\mathbb{N}}$ tending to zero. By setting the
remaining conditional data distribution values as multilinear functions of the
coordinates of $\theta$, as above, the sequence $(\bm{x}_{i},\bm{\theta})$
will satisfy the problem-continuity condition and will guarantee

$\lim_{i\to\infty}L_{(\bm{x}_{i},\bm{\theta})}(\theta_{0},\theta_{1})=L_{(\bm{x},\bm{\theta})}(\theta_{0},\theta_{1})=L(p,q).$

We now construct the sequence $\left(S_{i}\right)_{i\in\mathbb{N}}$
explicitly.

Each $S_{i}$ will be describable by the positive parameters $(a,b,d,r)$ as
follows. Let $C^{N}_{d}=\\{x\in\mathbb{R}^{N}:|x|_{\infty}\leq d/2\\}$, i.e.
the axis-parallel, origin-centred, $N$-dimensional cube of side length $d$.
$S_{i}$ will be chosen to contain all $x\in C^{N}_{d}$ such that for all
$\theta\in\\{0,1\\}^{M}$, $a\leq f^{(\bm{x},\bm{\theta})}_{\theta}(x)\leq b$
and $x$ is at least a distance of $r$ away from the nearest discontinuity
point of $f^{(\bm{x},\bm{\theta})}_{\theta}$, as well as from the origin. By
choosing small enough $a$ and $r$ and large enough $b$ and $d$, it is always
possible to make $\mathbf{P}(\bm{x}\in S_{i}|\bm{\theta}=\theta)$ arbitrarily
close to $1$, so the sequence can be made to satisfy its requirements.

We will choose $d$ to be a natural.

We now describe how to construct each $f^{(\bm{x}_{i},\bm{\theta})}_{\theta}$
from its respective $f^{(\bm{x},\bm{\theta})}_{\theta}$. We first describe for
each $\theta\in\\{0,1\\}^{M}$ a new function
$g^{i}_{\theta}:\mathbb{R}^{N}\to\mathbb{R}^{\geq 0}$ as follows. Begin by
setting $g^{i}_{\theta}(x)=f^{(\bm{x},\bm{\theta})}_{\theta}(x)$ for all $x\in
S_{i}$. If $x\notin C^{N}_{d}$, set $g^{i}_{\theta}(x)$ to zero. Otherwise,
complete the $g^{i}_{\theta}$ functions so that all are linearly independent
and so that each is positive and continuous inside $C^{N}_{d}$, and integrates
to $1$. Note that because a neighbourhood around the origin is known to not be
in $S_{i}$, it is never the case that $S_{i}=C^{N}_{d}$. This allows enough
degrees of freedom in completing the functions $g$ in order to meet all their
requirements.

As all $g^{i}_{\theta}$ are continuous functions over the compact domain
$C^{N}_{d}$, by the Heine-Cantor Theorem (Rudin, 1964) they are uniformly
continuous. There must therefore exist a natural $n$, such that we can tile
$C^{N}_{d}$ into sub-cubes of side length $1/n$ such that by setting each
$f^{(\bm{x}_{i},\bm{\theta})}_{\theta}$ value in each sub-cube to a constant
for the sub-cube equal to the mean over the entire sub-cube tile of
$g^{i}_{\theta}$, the result will satisfy for all $x\in C^{N}_{d}$ and all
$\theta\in\\{0,1\\}^{M}$,
$\left|f^{(\bm{x}_{i},\bm{\theta})}_{\theta}(x)-g^{i}_{\theta}(x)\right|<\epsilon_{i}$.
Because $f^{(\bm{x}_{i},\bm{\theta})}$ is by design multi-linear in $\theta$,
this implies that for all $\theta\in[0,1]^{M}$ and all $x\in S_{i}$, condition
(A.8) is attained. Furthermore, by choosing a large enough $n$, we can always
ensure, because the $g$ functions are continuous and linearly independent,
that also the $f^{(\bm{x}_{i},\bm{\theta})}_{\theta}$ functions, for
$\theta\in\\{0,1\\}^{M}$ are linearly independent and differ in more than a
measure zero of $\mathbb{R}^{N}$. Together, these properties ensure that the
new problems constructed are well-defined and elementary.

We have therefore constructed $(\bm{x}_{i},\bm{\theta})$ as a sequence of
elementary estimation problems that $\mathcal{M}$-approximate
$(\bm{x},\bm{\theta})$ arbitrarily well, while being entirely composed of
$f^{(\bm{x}_{i},\bm{\theta})}_{\theta}$ functions whose support is
$C^{N}_{d_{i}}$ for some natural $d_{i}$ and whose values within their support
are piecewise-constant inside cubic tiles of side-length $1/n_{i}$, for some
natural $n_{i}$.

We now use IRO to reshape the observation space of the estimation problems in
the constructed sequence by a piecewise-continuous transform.

Namely, we take each constant-valued cube of side length $1/n_{i}$ and
transform it using a scaling transformation in each coordinate, as follows.
Consider a single cubic tile, and let the value of
$f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{1}}(x)$ at points $x$ that are within
it be $G_{i}$. We scale the first coordinate of the tile to be of length
$G_{i}/n_{i}^{N}$, and all other coordinates to be of length $1$. Notably,
this transformation increases the volume of the cube by a factor of $G_{i}$,
so the probability density inside the cube, for each $f_{\theta}$, will drop
by a corresponding factor of $G_{i}$.

We now place the transformed cubes by stacking them along the first
coordinate, sorted by increasing value of
$f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{2}}(x)/f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{1}}(x)$.

Notably, because the probability density $f_{\theta_{1}}$ in all transformed
cubes is $G_{i}/G_{i}=1$, it is possible to arrange all transformed cubes in
this way so that, together, they fill exactly the unit cube in
$\mathbb{R}^{N}$. Let the new estimation problems created in this way be
$(\bm{x}^{\prime}_{i},\bm{\theta})$, let
$t_{i}:\mathbb{R}^{N}\to\mathbb{R}^{N}$ be the transformation,
$t_{i}(\bm{x}_{i})=\bm{x}^{\prime}_{i}$, applied on the observation space and
let $t^{1}_{i}(x)$ be the first coordinate value of $t_{i}(x)$.

By IRO,
$L(f^{(\bm{x}^{\prime}_{i},\bm{\theta})}_{\theta_{0}},f^{(\bm{x}^{\prime}_{i},\bm{\theta})}_{\theta_{1}})=L(f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{0}},f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{1}})$,
which we know tends to $L(p,q)$.

Consider the probability density of each
$f^{(\bm{x}^{\prime}_{i},\bm{\theta})}_{\theta}$ over its support $[0,1]^{N}$.
This is a probability density that is uniform along all axes except the first,
but has some marginal, $s=s^{i}_{\theta}$, along the first axis. We denote
such a distribution by $D_{N}(s)$. Specifically, for $\theta=\theta_{2}$,
because of our choice of sorting order, we have
$s^{i}_{\theta_{2}}=c[f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{2}},f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{1}}]$,
so by Lemma A.4, this is known to $\mathcal{M}$-converge to
$c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}]$.

If $M=1$, the above is enough to show that the $\mathcal{M}$-limit problem of
$(\bm{x}^{\prime}_{i},\bm{\theta})$ exists. If $M>1$, consider the following.

Let $t:X\to[0,1]$ be the transformation mapping each $x\in X$ to the supremum
of all $\tilde{t}$ for which
$c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}](\tilde{t})\leq
f^{(\bm{x},\bm{\theta})}_{\theta_{2}}(x)/f^{(\bm{x},\bm{\theta})}_{\theta_{1}}(x)$.
By design,
$c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}]$
is left continuous, so
$c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}](t(x))=f^{(\bm{x},\bm{\theta})}_{\theta_{2}}(x)/f^{(\bm{x},\bm{\theta})}_{\theta_{1}}(x)$.

Summing up the entire construction, in all but a diminishing measure of $x$ we
have that
$f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{2}}(x)/f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{1}}(x)$
approaches
$f^{(\bm{x},\bm{\theta})}_{\theta_{2}}(x)/f^{(\bm{x},\bm{\theta})}_{\theta_{1}}(x)$,
which in turn equals the value
$c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}](t(x))$.
On the other hand, we have that
$s^{i}_{\theta_{2}}=c[f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{2}},f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{1}}]$
also $\mathcal{M}$-approaches
$c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}]$
by Lemma A.4, and satisfies
$f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{2}}(x)/f^{(\bm{x}_{i},\bm{\theta})}_{\theta_{1}}(x)=s^{i}_{\theta_{2}}(t^{1}_{i}(x))$.

Together, this indicates
$\left(c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}](t^{1}_{i}(x))\right)\xrightarrow{\mathcal{M}}c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}](t(x))$.

Because
$c[f^{(\bm{x},\bm{\theta})}_{\theta_{2}},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}]$
is monotone strictly increasing, it follows that $t^{1}_{i}(x)$
$\mathcal{M}$-converges to $t(x)$. For all other $\theta\in[0,1]^{M}$ this
then implies that $s^{i}_{\theta}$ $\mathcal{M}$-converges to
$c[f^{(\bm{x},\bm{\theta})}_{\theta},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}]$,
because by construction all the problem’s $r_{f_{\theta},q}$ are monotone
increasing with each other.

For all $\theta$, the $f^{(\bm{x}^{\prime}_{i},\bm{\theta})}_{\theta}$
sequence therefore has a limit, that limit being the distribution
$D_{N}(c[f^{(\bm{x},\bm{\theta})}_{\theta},f^{(\bm{x},\bm{\theta})}_{\theta_{1}}])$.

In particular, the limit at $\theta=\theta_{0}$ is $D_{N}(c[p,q])$ and the
limit at $\theta=\theta_{1}$ is $U([0,1]^{N})$, the uniform distribution over
the unit cube.

By problem-continuity of $L$, $L(D_{N}(c[p,q]),U([0,1]^{N}))=L(p,q)$. Hence,
$L(p,q)$ is a function of only $c[p,q]$. ∎

######  Lemma A.6 (Lemma 3.5 of the main text).

Let $L$ be a smooth conditional-distribution-based loss function satisfying
that $L(P,Q)$ is a function only of $c[P,Q]$ and of the problem type, and let
$(\bm{x},\bm{\theta})$ be an elementary $\bm{\theta}$-continuous estimation
problem of a type on which $L$ is sensitive.

If one of the following conditions holds true:

  1. 1.

The problem $(\bm{x},\bm{\theta})$ is a continuous estimation problem, or,

  2. 2.

The problem $(\bm{x},\bm{\theta})$ is a semi-continuous estimation problem and
$L$ satisfies ISI,

then there exists a nonzero constant $\gamma$, dependent only on the choice of
$L$ and the problem type, such that for every $\theta\in\Theta$ the Hessian
matrix $H_{L}^{\theta}$ equals $\gamma$ times the Fisher information matrix
$\mathcal{I}_{\theta}$.

###### Proof.

Fix the problem type.

We wish to calculate the derivatives of $L(P_{\theta_{1}},P_{\theta_{2}})$
according to $\theta_{1}$. For convenience, let us define a new function,
$\tilde{L}$, which describes $L$ in a one-parameter form, by

$\tilde{L}(r_{P,Q})=L(P,Q).$

This is possible by our assumption that $L(P,Q)$ is only a function of
$c[P,Q]$.

We differentiate $\tilde{L}(r_{P,Q})$ as we would any composition of
functions. The derivatives of $r_{P_{\theta_{1}},P_{\theta_{2}}}$ in
$\theta_{1}$ are straightforward to compute, so we concentrate on the question
of how minute perturbations of $r$ affect $\tilde{L}(r)$.

For this, we first extend the domain of $\tilde{L}(r)$. Natively,
$\tilde{L}(r)$ is only defined when $\mathbf{E}_{\bm{x}\sim Q}[r(\bm{x})]=1$.
However, to be able to perturb $r$ more freely, we define, for finite
expectation $r(\bm{x})$, $\tilde{L}(r)=\tilde{L}(r/\mathbf{E}_{\bm{x}\sim
Q}[r(\bm{x})])$.

Let $Y\subseteq X$ be a set with $\mathbf{P}_{\bm{x}\sim Q}(\bm{x}\in
Y)=\epsilon>0$ such that for all $x\in Y$, $r(x)$ is a constant, $r_{0}$. The
derivative of $\tilde{L}(r)$ in $Y$ is defined as

$\left[\nabla_{Y}(\tilde{L})\right](r)=\lim_{\Delta\to
0}\frac{\tilde{L}(r+\Delta\cdot\chi_{Y})-\tilde{L}(r)}{\Delta},$

where for any $S\subseteq\mathbb{R}^{s}$, we denote by $\chi_{S}$ the function
over $\mathbb{R}^{s}$ that yields $1$ when the input is in $S$ and $0$
otherwise.

By our smoothness assumption on $L$, this derivative is known to exist,
because it is straightforward to construct an elementary, continuous
estimation problem for which these would be (up to normalisation) the
$r_{P_{\theta_{0}},P_{\theta_{0}+\Delta e_{1}}}$ values for some $\theta_{0}$
and basis vector $e_{1}$.

Consider, now, what this derivative’s value can be. By assumption that
$L(P,Q)$ only depends on $c[P,Q]$, we know that the derivative’s value can
depend on $\epsilon$, $r_{0}$ and on the distribution of $r(\bm{x})$ for
$\bm{x}\sim Q$, a distribution we will name $D_{Q}(r)$, but it cannot depend
on any other properties of $Y$ or $r$, because any changes to $Y$ and $r$ that
do not affect $\epsilon$, $r_{0}$ and $D_{Q}(r)$ will yield the same $c[P,Q]$
values throughout the calculation of $\left[\nabla_{Y}(\tilde{L})\right](r)$.
Hence, we can describe the derivative as
$\Delta_{\text{x}}(\epsilon,r_{0}|D_{Q}(r))$.

As a next step, consider what would happen if we were to partition $Y$ into
$2$ sets, each of measure $\epsilon/2$ in $Q$. (Such a partition is
straightforward to produce in the continuous case. If $X$ is discrete, one can
create it by utilising ISI: translate $\bm{x}$ to $(\bm{x},\bm{y})$, where
$\bm{y}$ is a Bernoulli random variable with $p=1/2$.)

The marginal impact of each set on the value of $L$ is
$\Delta_{\text{x}}(\epsilon/2,r_{0}|D_{Q}(r))$, but their total impact is
$\Delta_{\text{x}}(\epsilon,r_{0}|D_{Q}(r))$. More generally, either using the
problem’s continuity or using ISI and $L$’s smoothness, we can describe
$\Delta_{\text{x}}(\epsilon,r_{0}|D_{Q}(r))$ as
$\epsilon\cdot\Delta_{\text{x}}(r_{0}|D_{Q}(r))$.

Utilising $L(\theta_{1},\theta_{2})$’s representation as $\tilde{L}(r)$, where
$Q=P_{\theta_{2}}$ and $r=r_{P_{\theta_{1}},P_{\theta_{2}}}$, and noting that
$\epsilon$ was, in the calculation above, the measure of $Y$ in
$Q=P_{\theta_{2}}$, we can now write the first derivative of $L$ in some
direction $i$ of $\theta_{1}$ explicitly as an integral in this measure, i.e.
in “$\text{d}P_{\theta_{2}}$”.

For clarity of presentation, we will write this as an integral in
“$f_{\theta_{2}}(x)\text{d}x$”, using here and throughout the remainder of the
proof pdf notation, as would be appropriate when $\bm{x}$ is known to be
continuous. This change is meant merely to simplify the notation, and in no
way restricts the proof. Readers are welcome to verify that all steps are
equally valid for any $P_{\theta_{1}}$ and $P_{\theta_{2}}$ distributions.

Let $R_{\theta_{1},\theta_{2}}$ be
$D_{f_{\theta_{2}}}(r_{f_{\theta_{1}},f_{\theta_{2}}})$. If
$c[f_{\theta_{1}},f_{\theta_{2}}]$ is a piecewise-constant function, the
derivative of $L$ can be written as follows.

$\frac{\partial
L(\theta_{1},\theta_{2})}{\partial\theta_{1}(i)}=\int_{X}\Delta_{\text{x}}(r_{f_{\theta_{1}},f_{\theta_{2}}}(x)|R_{{\theta_{1}},{\theta_{2}}})\frac{\partial
r_{f_{\theta_{1}},f_{\theta_{2}}}(x)}{\partial\theta_{1}(i)}f_{\theta_{2}}(x)\text{d}x.$

The same reasoning can be used to describe the second derivative of $L$ (this
time in the directions $i$ and $j$ of $\theta_{1}$). The second derivative of
$\tilde{L}(r)$ when perturbing $r$ relative to two subsets $Y_{1}$ and $Y_{2}$
is defined as

$\lim_{\Delta\to
0}\frac{\left[\nabla_{Y_{1}}(\tilde{L})\right](r+\Delta\cdot\chi_{Y_{2}})-\left[\nabla_{Y_{1}}(\tilde{L})\right](r)}{\Delta},$

and once again using the assumption that $L(P,Q)$ only depends on $c[P,Q]$, we
can see that if $Y_{1}$ and $Y_{2}$ are disjoint, if the measures of $Y_{1}$
and $Y_{2}$ in $Q$ are, respectively, $\epsilon_{1}$ and $\epsilon_{2}$, both
positive, and if the value of $r_{P,Q}(x)$ for $x$ values in each subset is a
constant, respectively $r_{1}$ and $r_{2}$, then any such $Y_{1}$ and $Y_{2}$
will perturb $\tilde{L}(r)$ in exactly the same amount over any $r$ with the
same $D_{Q}(r)$. We name the second derivative coefficient in this case
$\Delta_{\text{xy}}(r_{1},r_{2}|D_{Q}(r))$.

The caveat that $Y_{1}$ and $Y_{2}$ must be disjoint is important, because if
$Y=Y_{1}=Y_{2}$ the symmetry no longer holds. This is a second case, and for
it we must define a different coefficient
$\Delta_{\text{xx}}(r_{0}|D_{Q}(r))$, where $r_{0}=r_{1}=r_{2}$.

In the case where $c[f_{\theta_{1}},f_{\theta_{2}}]$ is a piecewise-constant
function, the second derivative can therefore be written as

$\begin{split}\frac{\partial^{2}L(\theta_{1},\theta_{2})}{\partial\theta_{1}(i)\partial\theta_{1}(j)}=&\int_{X}\Delta_{\text{x}}(r_{f_{\theta_{1}},f_{\theta_{2}}}(x)|R_{{\theta_{1}},{\theta_{2}}})\frac{\partial^{2}r_{f_{\theta_{1}},f_{\theta_{2}}}(x)}{\partial\theta_{1}(i)\partial\theta_{1}(j)}f_{\theta_{2}}(x)\text{d}x\\\
&\quad+\int_{X}\int_{X}\Delta_{\text{xy}}(r_{f_{\theta_{1}},f_{\theta_{2}}}(x_{1}),r_{f_{\theta_{1}},f_{\theta_{2}}}(x_{2})|R_{{\theta_{1}},{\theta_{2}}})\\\
&\quad\quad\quad\quad\quad\frac{\partial
r_{f_{\theta_{1}},f_{\theta_{2}}}(x_{1})}{\partial\theta_{1}(i)}\frac{\partial
r_{f_{\theta_{1}},f_{\theta_{2}}}(x_{2})}{\partial\theta_{1}(j)}f_{\theta_{2}}(x_{2})\text{d}x_{2}f_{\theta_{2}}(x_{1})\text{d}x_{1}\\\
&\quad+\int_{X}\Delta_{\text{xx}}(r_{f_{\theta_{1}},f_{\theta_{2}}}(x)|R_{{\theta_{1}},{\theta_{2}}})\frac{\partial
r_{f_{\theta_{1}},f_{\theta_{2}}}(x)}{\partial\theta_{1}(i)}\frac{\partial
r_{f_{\theta_{1}},f_{\theta_{2}}}(x)}{\partial\theta_{1}(j)}f_{\theta_{2}}(x)\text{d}x.\end{split}$
(A.9)

In order to compute the Hessian matrix $H_{L}^{\theta}$, we consider each
matrix element $H_{L}^{\theta}(i,j)$ in turn.

This equals
${\partial^{2}L(\theta_{1},\theta_{2})}/{\partial\theta_{1}(i)\partial\theta_{1}(j)}$
where $\theta_{1}=\theta_{2}=\theta$. In particular,
$c[f_{\theta_{1}},f_{\theta_{2}}]$ is $\chi_{(0,1]}$ and
$R_{{\theta_{1}},{\theta_{2}}}$ is the distribution $\mathbf{1}$, which yields
the constant value $1$.

The value of (A.9) in this case becomes

$\displaystyle\Delta_{\text{x}}\left(1\middle|\mathbf{1}\right)\int_{X}\left.\frac{\partial^{2}r_{f_{\theta_{1}},f_{\theta}}(x)}{\partial\theta_{1}(i)\partial\theta_{1}(j)}\right\rvert_{\theta_{1}=\theta}f_{\theta}(x)\text{d}x$
(A.10)
$\displaystyle\quad+\Delta_{\text{xy}}\left(1,1\middle|\mathbf{1}\right)\left(\int_{X}\left.\frac{\partial
r_{f_{\theta_{1}},f_{\theta}}(x)}{\partial\theta_{1}(i)}\right\rvert_{\theta_{1}=\theta}f_{\theta}(x)\text{d}x\right)\left(\int_{X}\left.\frac{\partial
r_{f_{\theta_{1}},f_{\theta}}(x)}{\partial\theta_{1}(j)}\right\rvert_{\theta_{1}=\theta}f_{\theta}(x)\text{d}x\right)$
$\displaystyle\quad+\Delta_{\text{xx}}\left(1\middle|\mathbf{1}\right)\int_{X}\left(\left.\frac{\partial
r_{f_{\theta_{1}},f_{\theta}}(x)}{\partial\theta_{1}(i)}\right\rvert_{\theta_{1}=\theta}\right)\left(\left.\frac{\partial
r_{f_{\theta_{1}},f_{\theta}}(x)}{\partial\theta_{1}(j)}\right\rvert_{\theta_{1}=\theta}\right)f_{\theta}(x)\text{d}x.$

Note, however, that because $(\bm{x},\bm{\theta})$ is an estimation problem,
i.e. all its conditional data distributions are probability measures, not
general measures, it is the case that

$\int_{X}r_{f_{\theta_{1}},f_{\theta}}(x)f_{\theta}(x)\text{d}x=\int_{X}f_{\theta_{1}}(x)\text{d}x=1,$

and is therefore a constant independent of either $\theta_{1}$ or $\theta$.
Its various derivatives in $\theta_{1}$ are accordingly all zero. This makes
the first two summands in (A.10) zero. What is left, when setting
$\gamma=\Delta_{\text{xx}}(1|\mathbf{1})$, is

$\displaystyle H_{L}^{\theta}(i,j)$
$\displaystyle=\gamma\int_{X}\left(\left.\frac{\partial
r_{f_{\theta_{1}},f_{\theta}}(x)}{\partial\theta_{1}(i)}\right\rvert_{\theta_{1}=\theta}\right)\left(\left.\frac{\partial
r_{f_{\theta_{1}},f_{\theta}}(x)}{\partial\theta_{1}(j)}\right\rvert_{\theta_{1}=\theta}\right)f_{\theta}(x)\text{d}x$
$\displaystyle=\gamma\int_{X}\left(\left.\frac{\partial
f_{\theta_{1}}(x)/f_{\theta}(x)}{\partial\theta_{1}(i)}\right\rvert_{\theta_{1}=\theta}\right)\left(\left.\frac{\partial
f_{\theta_{1}}(x)/f_{\theta}(x)}{\partial\theta_{1}(j)}\right\rvert_{\theta_{1}=\theta}\right)f_{\theta}(x)\text{d}x$
$\displaystyle=\gamma\int_{X}\left(\frac{\partial\log
f_{\theta}(x)}{\partial\theta(i)}\right)\left(\frac{\partial\log
f_{\theta}(x)}{\partial\theta(j)}\right)f_{\theta}(x)\text{d}x$
$\displaystyle=\gamma\mathbf{E}_{\bm{x}\sim
f_{\theta}}\left[\left(\frac{\partial\log
f_{\theta}(\bm{x})}{\partial\theta(i)}\right)\left(\frac{\partial\log
f_{\theta}(\bm{x})}{\partial\theta(j)}\right)\right]$
$\displaystyle=\gamma\mathcal{I}_{\theta}(i,j).$

Hence, $H_{L}^{\theta}=\gamma\mathcal{I}_{\theta}$.

The only difference in this derivation for the case where $\bm{x}$ is
discrete, is that in this case the final result would have used probabilities
rather than probability densities. This is consistent, however, with the way
Fisher information is defined in this more general case. In fact, some sources
(e.g., Bobkov et al., 2014) define the Fisher information directly from Radon-
Nikodym derivatives.

As a final point in the proof, we remark that $\gamma$ must be nonzero,
because had it been zero, $H_{L}^{\theta}$ would have been zero for every
$\theta$ in every elementary $\bm{\theta}$-continuous estimation problem of
the same type, contrary to our sensitivity assumption on the loss function. ∎

######  Theorem A.7 (Theorem 3.6 of the main text).

If $(\bm{x},\bm{\theta})$ is an elementary, semi-continuous estimation problem
for which $\hat{\theta}_{\text{WF}}$ is a well-defined set estimator, and if
$L$ is a smooth loss function, sensitive on $(\bm{x},\bm{\theta})$’s type,
that satisfies all of IIA, IRP, IRO and ISI, then

$\hat{\theta}^{L}_{\text{EIC}}=\hat{\theta}_{\text{WF}}.$

###### Proof.

The proof is essentially identical to that of Theorem A.2. The only change is
that we can no longer apply Lemma A.5. However, we claim that $L(P,Q)$ only
depends on $c[P,Q]$ and $M$ despite this, for which reason we can still apply
Lemma A.6, as before, to complete the proof.

Fix $M$, the dimension of $\Theta$.

To prove that $L(P,Q)$ only depends on $c[P,Q]$, let $(\bm{x},\bm{\theta})$ be
an elementary, semi-continuous estimation problem, and let $L$ be any smooth
loss function, sensitive on its type, that satisfies all of IIA, IRP, IRO and
ISI. Furthermore, fix $\theta_{1},\theta_{2}\in\Theta$, and let $P$ be the
distribution $P_{\theta_{1}}$ and $Q$ be the distribution $P_{\theta_{2}}$. We
will show that $L(P,Q)$ can only depend on $c[P,Q]$.

We note that we can freely assume $P\neq Q$, because

$c[P,Q]=\chi_{(0,1]}\Leftrightarrow P=Q\Leftrightarrow L(P,Q)=0,$

where the first equivalence follows from the definition of $c[P,Q]$ (for
discrete distributions $P$ and $Q$) and the second equivalence follows from
(1.1).

The assumption $P\neq Q$ also implies that the two distributions are not
linearly dependent.

We begin by describing a new estimation problem
$((\bm{x},\bm{y},\bm{z}),\bm{\theta})$, where $\bm{y}$ is a random variable
independent of $\bm{x}$ and of $\bm{\theta}$, which is uniformly distributed
in $\\{1,\ldots,2^{M}\\}$, and where $\bm{z}=r_{P,Q}(\bm{x})$.

We know that
$L_{(\bm{x},\bm{\theta})}(\theta_{1},\theta_{2})=L_{((\bm{x},\bm{y},\bm{z}),\bm{\theta})}(\theta_{1},\theta_{2})=L_{((\bm{z},\bm{y},\bm{x}),\bm{\theta})}(\theta_{1},\theta_{2})$,
because the first equality stems from ISI and the second from IRO.

In principle, we now wish to apply ISI again, in order to show that

$L_{((\bm{z},\bm{y},\bm{x}),\bm{\theta})}(\theta_{1},\theta_{2})=L_{((\bm{z},\bm{y}),\bm{\theta})}(\theta_{1},\theta_{2}),$
(A.11)

because if we can establish (A.11) then we are done. The reason for this is
that from Lemma A.3 we know that the value of
$L_{((\bm{z},\bm{y}),\bm{\theta})}(\theta_{1},\theta_{2})$ can only depend on
the conditional data distributions at $\theta_{1}$ and $\theta_{2}$, and by
construction these conditional data distributions depend only on the
distribution of $\bm{z}=r_{P,Q}(\bm{x})$ at $\bm{x}\sim P$ and at $\bm{x}\sim
Q$, in both of which the distribution of $\bm{z}$ is fully constructible from
knowledge of $c[P,Q]$.

Unfortunately, it is not possible for us to apply ISI directly to prove
(A.11). The reason for this is that even though (as we will demonstrate)
$\bm{x}$ is independent of the choice of $\theta_{1}$ versus $\theta_{2}$
given $(\bm{z},\bm{y})$, this independence may not extend to all other
parameter choices in $\Theta$.

We therefore first apply Lemma A.3 in order to show that
$L_{((\bm{z},\bm{y},\bm{x}),\bm{\theta})}(\theta_{1},\theta_{2})=L_{((\bm{z},\bm{y},\bm{x}),\bm{\theta^{\prime}})}(\theta^{\prime}_{1},\theta^{\prime}_{2})$,
for any elementary estimation problem
$((\bm{z},\bm{y},\bm{x}),\bm{\theta^{\prime}})$ over any parameter domain
$\Theta^{\prime}$ of the same dimension, as long as the distribution of
$(\bm{z},\bm{y},\bm{x})$ given $\bm{\theta^{\prime}}=\theta^{\prime}_{1}$ is
the same as its distribution given $\bm{\theta}=\theta_{1}$, and its
distribution given $\bm{\theta^{\prime}}=\theta^{\prime}_{2}$ is the same as
given $\bm{\theta}=\theta_{2}$.

We will construct such an estimation problem where $\bm{x}$ is independent of
$\bm{\theta^{\prime}}$ given $(\bm{z},\bm{y})$ across the entire range of
$\bm{\theta^{\prime}}$.

To do this, let $\Theta^{\prime}$ be the unit cube of dimension $M$, and let
$\theta^{\prime}_{1}=(1,0,\ldots,0)$ and $\theta^{\prime}_{2}=(0,\ldots,0)$.
The conditional data distributions at $\theta^{\prime}_{1}$ and
$\theta^{\prime}_{2}$ will be the same as in $\theta_{1}$ and $\theta_{2}$ in
the original problem, so as to enable us to apply Lemma A.3.

We will retain the definition of $\bm{z}$ as $r_{P,Q}(\bm{x})$. The variable
$\bm{y}$ will remain independent of $\bm{x}$, but in the new problem it will
depend on $\bm{\theta^{\prime}}$. Specifically, it will remain uniformly
distributed over $\\{1,\ldots,2^{M}\\}$ only for $\bm{\theta^{\prime}}$ values
of the form $(\alpha,0,\ldots,0)$. In the remaining $2^{M}-2$ corners of the
unit cube, $\bm{y}$ will have other distributions, arbitrarily chosen from all
distributions over the same support, such that all $2^{M}-1$ such
distributions of $\bm{y}$ are linearly independent (which is clearly possible
to do).

The distribution of $\bm{x}$ will be $P$ at all corners of the unit cube other
than in the origin.

Elsewhere in the unit cube, away from its corners, the distribution of
$(\bm{z},\bm{y},\bm{x})$ will be the multilinear continuation of their
distributions at the corners. Because, by the construction of $\bm{y}$, we
know the corner distributions to all be linearly independent other than at
$\theta^{\prime}_{1}$ and $\theta^{\prime}_{2}$, the constructed problem is
well-defined (in that it satisfies (1.1)), and is elementary.

We will now show that $\bm{x}$ is independent of $\bm{\theta^{\prime}}$ given
$(\bm{z},\bm{y})$. Consider any specific value $x$ of $\bm{x}$ and $y$ of
$\bm{y}$, and let $z=r_{P,Q}(x)$ be the $\bm{z}$ value corresponding to
$\bm{x}=x$. Furthermore, let $X_{z}$ be the set of all values of $\bm{x}$ that
share the same $\bm{z}$ value. The variable $\bm{x}$ is independent of
$\bm{\theta^{\prime}}$ given $(\bm{z},\bm{y})$ if for every such $x$, the
conditional probability
$\mathbf{P}(\bm{x}=x|(\bm{z},\bm{y})=(z,y),\bm{\theta^{\prime}}=\theta^{\prime})$
is not a function of $\theta^{\prime}$. This conditional probability can be
computed as

$\displaystyle\mathbf{P}(\bm{x}=x|(\bm{z},\bm{y})=(z,y),\bm{\theta^{\prime}}=\theta^{\prime})$
$\displaystyle=\frac{\mathbf{P}(\bm{x}=x,\bm{y}=y|\bm{\theta^{\prime}}=\theta^{\prime})}{\sum_{x^{\prime}\in
X_{z}}\mathbf{P}(\bm{x}=x^{\prime},\bm{y}=y|\bm{\theta^{\prime}}=\theta^{\prime})}$
(A.12)
$\displaystyle=\frac{\mathbf{P}(\bm{x}=x|\bm{\theta^{\prime}}=\theta^{\prime})\mathbf{P}(\bm{y}=y|\bm{\theta^{\prime}}=\theta^{\prime})}{\sum_{x^{\prime}\in
X_{z}}\mathbf{P}(\bm{x}=x^{\prime}|\bm{\theta^{\prime}}=\theta^{\prime})\mathbf{P}(\bm{y}=y|\bm{\theta^{\prime}}=\theta^{\prime})}$
$\displaystyle=\frac{\mathbf{P}(\bm{x}=x|\bm{\theta^{\prime}}=\theta^{\prime})}{\sum_{x^{\prime}\in
X_{z}}\mathbf{P}(\bm{x}=x^{\prime}|\bm{\theta^{\prime}}=\theta^{\prime})}.$

Consider the distribution of $\bm{x}$ at
$\bm{\theta^{\prime}}=\theta^{\prime}$. By construction, it is $\gamma
P+(1-\gamma)Q$, for some $0\leq\gamma\leq 1$. Substituting this into (A.12),
we get

$\displaystyle\mathbf{P}(\bm{x}=x|(\bm{z},\bm{y})=(z,y),\bm{\theta^{\prime}}=\theta^{\prime})$
$\displaystyle=\frac{\gamma\mathbf{P}_{\bm{x}\sim
P}(x)+(1-\gamma)\mathbf{P}_{\bm{x}\sim Q}(x)}{\sum_{x^{\prime}\in
X_{z}}\gamma\mathbf{P}_{\bm{x}\sim
P}(x^{\prime})+(1-\gamma)\mathbf{P}_{\bm{x}\sim Q}(x^{\prime})}$
$\displaystyle=\frac{\gamma r_{P,Q}(x)\mathbf{P}_{\bm{x}\sim
Q}(x)+(1-\gamma)\mathbf{P}_{\bm{x}\sim Q}(x)}{\sum_{x^{\prime}\in X_{z}}\gamma
r_{P,Q}(x^{\prime})\mathbf{P}_{\bm{x}\sim
Q}(x^{\prime})+(1-\gamma)\mathbf{P}_{\bm{x}\sim Q}(x^{\prime})}$
$\displaystyle=\frac{\gamma z\mathbf{P}_{\bm{x}\sim
Q}(x)+(1-\gamma)\mathbf{P}_{\bm{x}\sim Q}(x)}{\sum_{x^{\prime}\in X_{z}}\gamma
z\mathbf{P}_{\bm{x}\sim Q}(x^{\prime})+(1-\gamma)\mathbf{P}_{\bm{x}\sim
Q}(x^{\prime})}$ $\displaystyle=\frac{(\gamma
z+(1-\gamma))\mathbf{P}_{\bm{x}\sim Q}(x)}{\sum_{x^{\prime}\in X_{z}}(\gamma
z+(1-\gamma))\mathbf{P}_{\bm{x}\sim Q}(x^{\prime})}$
$\displaystyle=\frac{\mathbf{P}_{\bm{x}\sim Q}(x)}{\sum_{x^{\prime}\in
X_{z}}\mathbf{P}_{\bm{x}\sim Q}(x^{\prime})}.$

This is clearly not a function of $\theta^{\prime}$, and hence the
independence of $\bm{x}$ is proved, and ISI can be employed to show

$L_{((\bm{z},\bm{y},\bm{x}),\bm{\theta^{\prime}})}(\theta^{\prime}_{1},\theta^{\prime}_{2})=L_{((\bm{z},\bm{y}),\bm{\theta^{\prime}})}(\theta^{\prime}_{1},\theta^{\prime}_{2})=L_{((\bm{z},\bm{y}),\bm{\theta})}(\theta_{1},\theta_{2}),$

where the final equality is given, once again, by Lemma A.3.

Thus, $L(P,Q)$ depends only on $c[P,Q]$ and $M$, and we can use Lemma A.6, as
in Theorem A.2, to complete the proof. ∎

###  A.3 Feasibility and necessity

######  Theorem A.8 (Theorem 4.3 of the main text).

Our system of axioms is feasible, in the sense that there exist smooth and
problem continuous loss functions, $L$, sensitive on all
$\bm{\theta}$-continuous problem types, that satisfy all of IRP, IRO, IIA and
ISI.

While the existence of such a loss function $L$ may not seem trivial, there
are, in fact, many loss functions that satisfy the necessary requirements. We
will show how to construct an infinite family of such $L$.

Our family of loss functions will be a subset of the $f$-divergences (Ali and
Silvey, 1966).

An $f$-divergence is a loss function $L$ that can be computed, most generally,
as

$L(P,Q)=\mathbf{E}_{Q}\left[F(r_{P,Q})\right].$

This is usually expressed as

$L(P,Q)=\int_{X}F(f_{P}(x)/f_{Q}(x))f_{Q}(x)\text{d}x,$

which represents the case where $P$ and $Q$ are continuous distributions.

We call $F:\mathbb{R}^{>0}\to\mathbb{R}$ the $F$-function of the
$f$-divergence (refraining from using the more common term “$f$-function” so
as to avoid unnecessary confusion with our probability density functions). It
should be convex and satisfy $F(1)=0$.

######  Lemma A.9.

All $f$-divergences $L$ satisfy all of IRP, IRO, IIA and ISI.

###### Proof.

As we have already established in Lemma A.3, satisfying IRP and IIA, our two
parameter-space axioms, is tantamount to requiring that $L$ is a conditional-
distribution-based loss function. The two observation-space axioms, IRO and
ISI, add that the loss must depend only on a sufficient statistic for the
observation. The former is proved directly from the definition of
$f$-divergences. The latter is among the most basic properties of
$f$-divergences (See, e.g., Amari and Nagaoka (2000)). ∎

While all $f$-divergences satisfy IRP, IRO, IIA and ISI, not all of them
satisfy smoothness, sensitivity and problem continuity.

######  Lemma A.10.

Any $f$-divergence whose $F$-function satisfies the following conditions

  1. 1.

$F$ has 3 continuous derivatives, and

  2. 2.

$F^{\prime\prime}(1)>0$

is smooth and is sensitive on all $\bm{\theta}$-continuous problem types. If
it also satisfies

  1. 1.

$\lim_{x\to 0}F(x)<\infty$, and

  2. 2.

$\lim_{x\to\infty}F^{\prime}(x)<\infty$,

it is also problem continuous.

Here, $F^{\prime}$ and $F^{\prime\prime}$ are the first two derivatives of
$F$, which we assumed exist.

The main difficulty in Lemma A.10 is proving problem continuity. We will build
up to this using the following two lemmas.

######  Lemma A.11.

If $F:\mathbb{R}^{>0}\to\mathbb{R}$ is continuously differentiable and convex,
and the limit $\lim_{x\to 0}F(x)$ is bounded, then there is a value $x_{0}>0$,
such that for every $x$ in $(0,x_{0}]$, $F^{\prime}(x)>-1/x$, where
$F^{\prime}$ is the derivative of $F$.

###### Proof.

Because we know that $F$ is continuously differentiable, we know that
$F^{\prime}$ exists and is continuous in this range. Furthermore, because $F$
is convex, $F^{\prime}$ is monotone increasing.

Let us assume, to the contrary, that no $x_{0}$ as in the claim exists. Then
there exists an infinite decreasing sequence $x_{1},\ldots$, converging to
zero, for which $F^{\prime}(x_{i})\leq-1/x_{i}$.

Because we know that $F^{\prime}$ is monotone increasing, an upper bound for
it in the range $(x_{i+1},x_{i}]$ is $F^{\prime}(x_{i})$ and therefore also
$-1/x_{i}$.

Integrating this, we see that a lower bound for $F(x_{i+1})$ is

$F(x_{i+1})\geq F(x_{1})+\sum_{j=1}^{i}(x_{j}-x_{j+1})(1/x_{j}).$

We will show, however, that as $i$ goes to infinity, this lower bound also
diverges to infinity, contradicting our assumption that $\lim_{x\to 0}F(x)$ is
bounded.

To see this, consider the series summand. This equals $1-x_{j+1}/x_{j}$. For
the series to converge, this summand must converge to $0$, so for all $j$ at
least as large as some $j_{0}$ we have that there is some $r$ for which $r\leq
x_{j+1}/x_{j}\leq 1$. In this range, we know that

$\log(x_{j+1}/x_{j})\geq(x_{j+1}/x_{j}-1)\frac{\log(r)}{r-1},$

by the concavity of the $\log$ function.

We can therefore conclude the following regarding the series partial sum.

$\sum_{j=j_{0}}^{\infty}1-x_{j+1}/x_{j}\geq-\frac{r-1}{\log(r)}\sum_{j=j_{0}}^{\infty}\log(x_{j+1}/x_{j})=-\frac{r-1}{\log(r)}\lim_{j\to\infty}\log(x_{j}/x_{j_{0}})=\infty,$

proving the claim. ∎

######  Lemma A.12.

If $F:\mathbb{R}^{>0}\to\mathbb{R}$ is the $F$-function of an $f$-divergence,
$F$ is continuously differentiable, $\lim_{x\to 0}F(x)<\infty$ and
$\lim_{x\to\infty}F^{\prime}(x)<\infty$, where $F^{\prime}$ is the derivative
of $F$, then the $f$-divergence is a problem-continuous loss function.

###### Proof.

Let $P$ and $Q$ be probability distributions over a ground set $X$, and let
$\\{P_{i}\\}$ and $\\{Q_{i}\\}$ be sequences of distributions over ground sets
$X_{i}$, converging in measure to $P$ and $Q$, respectively.

The condition of convergence in measure is that for any $\epsilon>0$ and any
$\delta>0$ there is an $i_{0}$, such that for every $i\geq i_{0}$,

$\mathbf{P}_{\bm{x}\sim
P}\left(1-\epsilon\leq\frac{\text{d}P_{i}}{\text{d}P}(\bm{x})\leq
1+\epsilon\right)\geq 1-\delta,$

and

$\mathbf{P}_{\bm{x}\sim
Q}\left(1-\epsilon\leq\frac{\text{d}Q_{i}}{\text{d}Q}(\bm{x})\leq
1+\epsilon\right)\geq 1-\delta.$

We wish to prove that if these conditions hold, then the $f$-divergences
between $P_{i}$ and $Q_{i}$ converge to the $f$-divergence between $P$ and
$Q$, where the $f$-divergence between $P$ and $Q$ is calculated as

$D(P,Q)=\mathbf{E}_{\bm{x}\sim
P}\left[F\left(\frac{\text{d}Q}{\text{d}P}(\bm{x})\right)\right].$

While these formulations, using the Radon-Nikodym derivative
$\text{d}Q/\text{d}P$, are the most general, for convenience of presentation,
we will consider all distributions as continuous, describable by their
probability density functions (pdfs). We will use $f_{P}$ to describe the pdf
of $P$. Our proof will remain entirely general, but the use of pdfs to
describe distributions will make their presentations easiest.

Using pdfs, the claims can be reworded as follows. We know that for all
$\epsilon>0$ and $\delta>0$ a sufficiently large $i$ will satisfy

$\mathbf{P}_{\bm{x}\sim
P}\left(1-\epsilon\leq\frac{f_{P_{i}}(\bm{x})}{f_{P}(\bm{x})}\leq
1+\epsilon\right)\geq 1-\delta$

and

$\mathbf{P}_{\bm{x}\sim
Q}\left(1-\epsilon\leq\frac{f_{Q_{i}}(\bm{x})}{f_{Q}(\bm{x})}\leq
1+\epsilon\right)\geq 1-\delta,$

and wish to prove that

$\lim_{i\to\infty}\int_{X_{i}}F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)f_{P_{i}}(x)\text{d}x=\int_{X}F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)f_{P}(x)\text{d}x.$
(A.13)

For a given $\epsilon$ and a given $i$, let $X^{\epsilon}_{i}$ be the subset
of $X$ wherein

$1-\epsilon\leq\frac{f_{P_{i}}(x)}{f_{P}(x)}\leq 1+\epsilon$ (A.14)

and

$1-\epsilon\leq\frac{f_{Q_{i}}(x)}{f_{Q}(x)}\leq 1+\epsilon$ (A.15)

are both satisfied.

To prove (A.13), we will separate the difference between the left-hand side
and the right-hand side to four elements, then show that all four must equal
zero. The four sub-differences we will look at are:

$\lim_{i\to\infty}\int_{X\setminus
X^{\epsilon}_{i}}F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)f_{P}(x)\text{d}x,$
(A.16)

$\lim_{i\to\infty}\int_{X_{i}\setminus
X^{\epsilon}_{i}}F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)f_{P_{i}}(x)\text{d}x,$
(A.17)

$\lim_{i\to\infty}\int_{X^{\epsilon}_{i}}\left|F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)\right||f_{P_{i}}(x)-f_{P}(x)|\text{d}x$
(A.18)

and

$\lim_{i\to\infty}\int_{X^{\epsilon}_{i}}\left|F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)-F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)\right|f_{P}(x)\text{d}x.$
(A.19)

Regarding $X\setminus X^{\epsilon}_{i}$, we know that it has some measure
$\mu_{P}=\mu_{P}(\epsilon,i)$ in $P$ and some measure
$\mu_{Q}=\mu_{Q}(\epsilon,i)$ in $Q$, both tending to $0$ as $i$ goes to
infinity, for any $\epsilon$. Correspondingly, $X^{\epsilon}_{i}$ has measure
$1-\mu_{P}$ in $P$ and $1-\mu_{Q}$ in $Q$.

Because within $X^{\epsilon}_{i}$ (A.14) and (A.15) hold, the measure of
$X^{\epsilon}_{i}$ in $P_{i}$ is at least $(1-\mu_{P})(1-\epsilon)$ and its
measure in $Q_{i}$ is at least $(1-\mu_{Q})(1-\epsilon)$. Correspondingly, the
measure of $X_{i}\setminus X^{\epsilon}_{i}$ in $P_{i}$ is at most
$\mu_{P}+\epsilon-\mu_{P}\epsilon$, and its measure in $Q_{i}$ is at most
$\mu_{Q}+\epsilon-\mu_{Q}\epsilon$.

We have assumed that $F:\mathbb{R}^{>0}\to\mathbb{R}$ is continuously
differentiable and that it is convex (because it is an $F$-function for an
$f$-divergence). Under these preconditions, the conditions assumed,
$\lim_{x\to 0}F(x)<\infty$ and $\lim_{x\to\infty}F^{\prime}(x)<\infty$, can
easily be shown to be equivalent to the condition that there exists $A\geq 0$
and $B\geq 0$ such that for all $x$, $|F(x)|\leq Ax+B$.

Using $A$ and $B$, we can now bound (A.16) as follows.

$\displaystyle\lim_{i\to\infty}\int_{X\setminus
X^{\epsilon}_{i}}F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)f_{P}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}\int_{X\setminus
X^{\epsilon}_{i}}\left(A\frac{f_{Q}(x)}{f_{P}(x)}+B\right)f_{P}(x)\text{d}x$
$\displaystyle=\lim_{i\to\infty}\int_{X\setminus
X^{\epsilon}_{i}}Af_{Q}(x)\text{d}x+\int_{X\setminus
X^{\epsilon}_{i}}Bf_{P}(x)\text{d}x$
$\displaystyle=\lim_{i\to\infty}A\mu_{Q}+B\mu_{P}$ $\displaystyle=0.$

We bound (A.17) similarly.

$\displaystyle\lim_{i\to\infty}\int_{X_{i}\setminus
X^{\epsilon}_{i}}F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)f_{P_{i}}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}\int_{X_{i}\setminus
X^{\epsilon}_{i}}\left(A\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}+B\right)f_{P_{i}}(x)\text{d}x$
$\displaystyle=\lim_{i\to\infty}\int_{X_{i}\setminus
X^{\epsilon}_{i}}Af_{Q_{i}}(x)\text{d}x+\int_{X_{i}\setminus
X^{\epsilon}_{i}}Bf_{P_{i}}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}A(\mu_{Q}+\epsilon-\mu_{Q}\epsilon)+B(\mu_{P}+\epsilon-\mu_{Q}\epsilon)$
$\displaystyle=(A+B)\epsilon.$

Regarding (A.18), note that within $X^{\epsilon}_{i}$ we know that
$|f_{P_{i}}(x)-f_{P}(x)|\leq f_{P_{i}}(x)\frac{\epsilon}{1-\epsilon}$. For
this reason, we can bound (A.18) as follows.

$\displaystyle\lim_{i\to\infty}\int_{X^{\epsilon}_{i}}\left|F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)\right|$
$\displaystyle|f_{P_{i}}(x)-f_{P}(x)|\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}\int_{X^{\epsilon}_{i}}\left(A\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}+B\right)\left(\frac{\epsilon}{1-\epsilon}\right)f_{P_{i}}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}\left(\frac{\epsilon}{1-\epsilon}\right)\left(\int_{X_{i}}Af_{Q_{i}}(x)\text{d}x+\int_{X_{i}}Bf_{P_{i}}(x)\text{d}x\right)$
$\displaystyle=\left(\frac{\epsilon}{1-\epsilon}\right)(A+B).$

It now only remains to bound (A.19). This is once again an integral over
$X^{\epsilon}_{i}$, which is a domain where we know that

$\frac{1-\epsilon}{1+\epsilon}\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)\leq\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\leq\frac{1+\epsilon}{1-\epsilon}\left(\frac{f_{Q}(x)}{f_{P}(x)}\right).$

Let us therefore define $\alpha(x)$ as the value in the range

$\frac{1-\epsilon}{1+\epsilon}\leq
1+\alpha(x)\leq\frac{1+\epsilon}{1-\epsilon}$

that maximises

$\left|F\left((1+\alpha(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)-F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)\right|.$

By the Mean Value Theorem, for each $x$ there is a $\gamma(x)$,

$\frac{1-\epsilon}{1+\epsilon}\leq
1+\gamma(x)\leq\frac{1+\epsilon}{1-\epsilon},$

such that

$F\left((1+\alpha(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)-F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)=F^{\prime}\left((1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)\alpha(x)\frac{f_{Q}(x)}{f_{P}(x)}.$

The value of (A.19) can therefore be bounded from above by

$\displaystyle\lim_{i\to\infty}$
$\displaystyle\int_{X^{\epsilon}_{i}}\left|F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)-F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)\right|f_{P}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}\int_{X^{\epsilon}_{i}}\left|F^{\prime}\left((1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)\right||\alpha(x)|\frac{f_{Q}(x)}{f_{P}(x)}f_{P}(x)\text{d}x.$

As these equations should be valid for all $\epsilon$, let us assume that
$\epsilon\leq 1/4$, in which case $\alpha(x)$ and $\gamma(x)$ are both bounded
in the range $[-2/5,2/3]$.

Because of our assumption that $\lim_{y\to 0}F(y)<\infty$, we can now use
Lemma A.11 to know that there is a $y_{0}$ such that if $y\leq y_{0}$, the
value of $F^{\prime}(y)$ must be greater than $-1/y$.

Let us now partition $X^{\epsilon}_{i}$ to those $x$ values for which
$(1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}\leq y_{0}$ and
$F^{\prime}\left((1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)<0$, which we
will refer to as $L^{\epsilon}_{i}$, and the rest, which we will refer to as
$H^{\epsilon}_{i}$.

Within the domain $L^{\epsilon}_{i}$, we have

$\displaystyle\lim_{i\to\infty}\int_{L^{\epsilon}_{i}}\left|F^{\prime}\left((1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)\right|$
$\displaystyle|\alpha(x)|\frac{f_{Q}(x)}{f_{P}(x)}f_{P}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}\int_{L^{\epsilon}_{i}}\frac{|\alpha(x)|}{1+\gamma(x)}\left(\frac{f_{P}(x)}{f_{Q}(x)}\right)\frac{f_{Q}(x)}{f_{P}(x)}f_{P}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}2\epsilon\frac{1+\epsilon}{(1-\epsilon)^{2}}\int_{X}f_{P}(x)\text{d}x$
$\displaystyle\leq\frac{40}{9}\epsilon,$

where the last equality plugs in our condition $\epsilon\leq 1/4$.

Finally, within the domain $H^{\epsilon}_{i}$, we know that $|F^{\prime}(x)|$
is bounded. This is because of the following conditions. First, because $F$ is
convex, $F^{\prime}$ is monotone increasing, so its supremum is
$\lim_{x\to\infty}F^{\prime}(x)$, which we assumed to be bounded. On the other
hand, its values must either satisfy
$F^{\prime}\left((1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)\geq 0$, in
which case they are bounded from below by $0$, or
$(1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}>y_{0}$, in which case they are bounded
from below by $F^{\prime}(y_{0})$.

Let $A^{\prime}\geq 0$ be an upper bound for $|F^{\prime}(x)|$.

We now have

$\displaystyle\lim_{i\to\infty}\int_{H^{\epsilon}_{i}}\left|F^{\prime}\left((1+\gamma(x))\frac{f_{Q}(x)}{f_{P}(x)}\right)\right||\alpha(x)|\frac{f_{Q}(x)}{f_{P}(x)}f_{P}(x)\text{d}x$
$\displaystyle\leq\lim_{i\to\infty}A^{\prime}\frac{2\epsilon}{1-\epsilon}\int_{X}f_{Q}(x)\text{d}x$
$\displaystyle\leq\epsilon\frac{8A^{\prime}}{3},$

where the last equality, as before, stems from our assumption $\epsilon\leq
1/4$.

In total, we’ve established that

$\displaystyle\left|\lim_{i\to\infty}\int_{X_{i}}F\left(\frac{f_{Q_{i}}(x)}{f_{P_{i}}(x)}\right)f_{P_{i}}(x)\text{d}x-\int_{X}F\left(\frac{f_{Q}(x)}{f_{P}(x)}\right)f_{P}(x)\text{d}x\right|$
(A.20)
$\displaystyle\quad\quad\quad\quad\leq\left(\frac{7}{3}(A+B)+\frac{40}{9}+\frac{8A^{\prime}}{3}\right)\epsilon,$
(A.21)

under our assumption $\epsilon\leq 1/4$.

Because $A$, $B$ and $A^{\prime}$ are nonnegative constants, independent of
$\epsilon$, and because this equality must hold for any $\epsilon$ in
$(0,1/4]$, we conclude that the true difference on the left-hand side of
(A.20) (which is independent of $\epsilon$) must be zero, so (A.13) holds and
the lemma is proved. ∎

###### Proof of Lemma A.10.

Any $f$-divergence whose $F$-function has 3 continuous derivatives satisfies
the smoothness condition. This can be verified simply by explicitly computing
the required derivatives from the $f$-divergence’s formula. In particular,
this computation shows that the Hessian equals $F^{\prime\prime}(1)$ times the
Fisher information matrix. Therefore, if $F^{\prime\prime}(1)>0$ the
$f$-divergence also satisfies sensitivity for all problem types. Problem
continuity, if relevant, is then proved by Lemma A.12. ∎

###### Proof of Theorem A.8.

From Lemma A.9 and Lemma A.10, we know that it is enough to show an example of
a loss function $L$ that is an $f$-divergence satisfying the extra conditions
of Lemma A.10.

A concrete example of a commonly-used $L$ function satisfying all criteria is
squared Hellinger distance (Pollard, 2002),

$H^{2}(p,q)=\frac{1}{2}\int_{X}\left(\sqrt{p(x)}-\sqrt{q(x)}\right)^{2}\text{d}x,$

which is the $f$-divergence whose $F$-function is $F(r)=1-\sqrt{r}$. ∎

######  Theorem A.13 (Theorem 4.4 of the main paper).

All axioms used in Theorem 3.1 are necessary.

###### Proof.

Much as the proof of feasibility was simply a proof by example, proving
necessity will be done by counterexample: we will show alternate loss
functions, $L$, leading to alternate estimations, that satisfy all axioms but
one. For each such $L$, we will show that its $|H^{\theta}_{L}|$ is not
proportional to $|\mathcal{I}_{\theta}|$, and hence produces an estimator
different to WF.

A well-known loss function that satisfies IRO and IIA but not IRP is quadratic
loss,

$L(\theta_{1},\theta_{2})=|\theta_{1}-\theta_{2}|^{2}.$ (A.22)

As was demonstrated in the main paper, an error intolerant estimator with this
loss function yields the continuous MAP estimate, different to the WF
estimate.

A loss function satisfying IRP and IIA but not IRO is

$L(P,Q)=\int_{X}f_{Q}(x)(f_{P}(x)-f_{Q}(x))^{2}\text{d}x,$ (A.23)

which is the expected square difference between the probability densities at
$\bm{x}\sim Q$. Calculating $H_{L}^{\theta}$ we get

$H_{L}^{\theta}(i,j)=\mathbf{E}_{\bm{x}\sim
f_{\theta}}\left[2\left(\frac{\partial
f_{\theta}(\bm{x})}{\partial\theta(i)}\right)\left(\frac{\partial
f_{\theta}(\bm{x})}{\partial\theta(j)}\right)\right],$

which is different to the Fisher information matrix, and defines an estimator
that is not WF.

To prove necessity of our third axiom, IIA, we construct a loss function $L$
that satisfies IRP and IRO but not IIA as follows. Let $L_{1}$ and $L_{2}$ be
two smooth and problem continuous loss functions, sensitive on the relevant
problem type, satisfying all axioms (such as, for example, two $f$-divergences
matching the criteria of Lemma A.10) and let $t$ be a threshold value.

Consider the function

$P(\theta)=\mathbf{P}(L_{1}(\bm{\theta},\theta)\leq t).$ (A.24)

By construction, this function is independent of representation.

Define

$L(\theta_{1},\theta_{2})=P(\theta_{2})L_{2}(\theta_{1},\theta_{2}).$ (A.25)

The resulting $|H^{\theta}_{L}|$ equals $P(\theta)^{M}|H_{L_{2}}^{\theta}|$,
where the equality stems from the fact that $P(\theta_{2})$ is independent of
$\theta_{1}$ and therefore acts as a constant multiplier in the calculation of
the Hessian.

This new estimator is different to the Wallace-Freeman estimator in the fact
that it adds a weighing factor $P(\theta)^{M/2}$. ∎

######  Theorem A.14 (Theorem 4.5 of the main paper).

All axioms used in Theorem 3.6 are necessary.

###### Proof.

This proof is a direct continuation of the proof of Theorem A.13, and re-uses
the same techniques and some of the same examples.

To begin with, quadratic loss, given in (A.22) as an example of a loss
function that satisfies IIA and IRO but not IRP also demonstrates that IRP is
necessary in the semi-continuous case because (by not depending on the
estimation problem at all) it also satisfies ISI.

Similarly, the example of (A.25) can be re-used here to demonstrate the
necessity of IIA also in the semi-continuous case. We have already shown that
this loss function satisfies IRP and IRO. To show the remaining condition,
namely that it also satisfies ISI, one merely needs to choose two loss
functions $L_{1}$ and $L_{2}$ that satisfy the conditions of Theorem A.8. By
assumption, these satisfy ISI, and so by construction (A.24) is independent of
superfluous information and (A.25) must be, too, proving the claim.

To demonstrate that ISI is necessary, consider

$L_{1}(P,Q)=\sum_{x\in X}\mathbf{P}_{\mathbf{x}\sim
Q}(\mathbf{x}=x)\left(\mathbf{P}_{\mathbf{x}\sim
P}(\mathbf{x}=x)-\mathbf{P}_{\mathbf{x}\sim Q}(\mathbf{x}=x)\right)^{2},$
(A.26)

which is the discrete analogue of (A.23).

This loss function clearly satisfies IRP and IIA, just as (A.23) does. Neither
satisfies ISI, however: if we, for example, replace $\mathbf{x}$ by
$(\mathbf{x},\mathbf{y})$, where $\mathbf{y}$ is a Bernoulli random variable
with $p=1/2$, losses will not be preserved. (All losses will, in fact, be
uniformly scaled down by a factor of $4$.)

In the proof of Theorem A.13 we use (A.23) as an example of a loss function
that does not satisfy IRO. In the semi-continuous domain, however, (A.26) does
satisfy IRO. This is because the impact of applying piecewise-diffeomorphic
deformations on discrete domains is quite different to applying them on
continuous domains. In the continuous domain, such functions can stretch and
contract the probability space, causing probability densities, such as those
used in (A.23), to change. In the discrete domain, no such effects are
possible: the probabilities used in (A.26) do not change as a result of
deformations of the observation space, and so IRO is met.

However, as before, we can use the Hessian, which in this case is

$H_{L}^{\theta}(i,j)=\mathbf{E}_{\bm{x}\sim
P_{\theta}}\left[2\left(\frac{\partial
P_{\theta}(\bm{x})}{\partial\theta(i)}\right)\left(\frac{\partial
P_{\theta}(\bm{x})}{\partial\theta(j)}\right)\right],$

to show that the resulting estimator is different to WF, proving the necessity
of ISI.

Lastly, we want to prove that IRO is necessary. For this, let $x(1:k)$ be the
value of $x$’s first $k$ dimensions, let $x(k)$ be the value of its $k$’th
dimension alone, and for a distribution $P$, if $x\sim P$, let $P^{y}_{k}$ be
the distribution of $x(k)$ given that $x(1:k-1)=y$. Consider, now, a function
$L_{N}(P,Q)$ calculated in the following way.

$L_{N}(P,Q)=\sum_{k=1}^{N}\mathbf{E}_{\bm{x}\sim
Q}\left[L_{1}\left(P^{\bm{x}(1:k-1)}_{k},Q^{\bm{x}(1:k-1)}_{k}\right)\right],$

where we reuse the function $L_{1}$ from (A.26), but use it only to compare
between distributions over a one-dimensional observation space.

By construction, $L_{N}$ satisfies ISI for any $L_{1}$, because any dimension
that does not add information also does not add to the value of $L_{N}(P,Q)$.
Therefore, it satisfies all of IRP, IIA and ISI.

However, as was already demonstrated regarding $L_{1}$ in the one-dimensional
case, it does not lead to WF, proving the necessity of IRO. ∎

######  Theorem A.15 (Theorem 4.7 of the main text).

MLE satisfies IRP, IRO, IIA and ISI.

###### Proof.

Consider loss functions $L$ of the form

$L(\theta_{1},\theta_{2})=\gamma(\theta_{2})L^{\prime}(\theta_{1},\theta_{2}),$

where $L^{\prime}$ is a smooth and problem-continuous loss function, sensitive
on the relevant problem type.

In the proof to Theorem 2.8 of the main text, we showed that any PMLE
estimator can be described in this way, with $L^{\prime}$ being quadratic
loss. Substituting the identity function for $g$ in (2.7), we get that MLE can
be described by

$\gamma(\theta_{2})=f(\theta_{2})^{2/M}.$

This $L$ satisfies IRO, IIA and ISI, but not IRP. We conclude from it that MLE
satisfies IRO, IIA and ISI.

To prove that MLE also satisfies IRP, let $L^{\prime}$ be squared Hellinger
distance and let

$\gamma(\theta_{2})=\left(\frac{f(\theta_{2})}{\sqrt{|\mathcal{I}_{\theta_{2}}|}}\right)^{2/M}.$

This $L$, too, leads to MLE, because $H_{L^{\prime}}^{\theta_{2}}$ equals
$\mathcal{I}_{\theta_{2}}$.

The new $L$ satisfies IRP because both $L^{\prime}$ and $\gamma$ are invariant
to the representation of $\theta$. ∎

We note that even though the theorem is solely about MLE, our construction is
equally applicable to any PMLE estimator whose penalty function $g$ satisfies
the four expected invariance properties.

##  Appendix B Supplementary discussion

###  B.1 Comparison with MML

In the main text we discuss axioms that give rise to the Wallace-Freeman point
estimator (WF). This estimator was originally developed in the Minimum Message
Length (MML) literature, where it is most commonly referred to as the Wallace-
Freeman approximation.

In this section we discuss in greater detail the relationship between Wallace-
Freeman estimation as it appears in the MML literature, and how it appears
here, in the context of Error Intolerant Estimation.

####  B.1.1 Introduction to MML

MML is a general name for any member of the family of Bayesian statistical
inference methods based on the minimum message length principle. They are
closely related to the family of minimum description length (MDL) estimators
(Grünwald, 2007; Rissanen, 1987, 1999), but predate them.

The minimum message length principle was first introduced in Wallace and
Boulton (1968), and the estimator that follows the principle directly, which
was first described in Wallace and Boulton (1975), is known as Strict MML
(SMML).

SMML can be defined as follows. Given an estimation problem
$(\bm{x},\bm{\theta})$ and given an observation, $x$, we wish to choose
$\hat{\theta}(x)$ so that it optimally trades off two ideals. First, it must
be a good approximation to $\theta$, the true value of $\bm{\theta}$, in that
the distribution of $\bm{x}$ given $\bm{\theta}=\hat{\theta}(x)$ is a good
approximation to its distribution given $\bm{\theta}=\theta$. Second, the
choice of $\hat{\theta}$ must be “simple”, in the sense that it upholds the
ideal of Occam’s razor. Both these ideals can be formulated in information-
theoretic terms.

Namely, suppose we wish to communicate both our estimate $\hat{\theta}(x)$ and
the observation $x$, and suppose, further, that we choose for this the
following protocol known as the MML _two-part message_. First, we communicate
$\hat{\theta}(x)$ via the optimal communication protocol. (The choice of this
optimal protocol is different, depending on what the function $\hat{\theta}$
is.) Second, we communicate $x$ via a protocol that is optimal only under the
assumption that $\bm{\theta}=\hat{\theta}(x)$.

The expected length of the first part of the message is the Shannon entropy
(Shannon, 1948) of $\hat{\theta}(\bm{x})$. Thus, a “simpler” function
$\hat{\theta}:X\to\mathbb{R}^{M}$ will result in a shorter first part message
in expectation. On the other hand, the second part of the message introduces
inefficiencies due to its approximation that $\bm{\theta}=\hat{\theta}(x)$.
Thus, the better the approximation the shorter the expected length of the
second part of the message becomes.

Strict MML chooses the function $\hat{\theta}$ so that the overall length of
the two-part message is minimised in expectation, creating, according to MML
theory, an optimal trade-off between simplicity and accuracy.

Notably, this construction only works if $X$ is countable (because otherwise
the second part of the message becomes infinite) and always results in a
$\hat{\theta}$ function whose range is only a countable subset of $\Theta$,
even if $\Theta$ itself is continuous (because otherwise the first part of the
message becomes infinite).

SMML is, however, used also for continuous $X$. This is done by extending the
construction above in one of several equivalent ways. For example, instead of
optimising the message length, it is possible to optimise the _excess message
length_ , which is the difference between the expected message length and the
information-theoretical optimum to deliver the information without
restrictions on the message format. This optimum is the Shannon entropy of
$\bm{x}$ and is independent of the choice of $\hat{\theta}$. By subtracting
this entropy, the excess message length can be represented as a Kullback-
Leibler divergence. Because this divergence is computable also on problems
with a continuous $X$, it provides a natural way to extend the definition of
SMML to the continuous domain.

Nevertheless, even in those cases, SMML will advocate a $\hat{\theta}$ whose
range is countable.

####  B.1.2 Wallace-Freeman Estimation

Strict MML, as a representation-invariant Bayesian point-estimation method,
has much theoretical appeal, but it cannot be used in practice because it is
computationally and analytically intractable in all but a select few single-
parameter cases (Dowty, 2015). It was proved to be NP-Hard to compute even in
estimating the parameters of a trinomial distribution (Farr and Wallace,
2002).

Even beyond the computational problems, SMML is a difficult estimate to work
with. For example, it provides piecewise-constant estimates and can be
asymmetrical even when working on an estimation problem exhibiting symmetry.

Wallace and Freeman (1987b) addressed these problems by developing the
Wallace-Freeman estimator (WF) as a computationally convenient approximation
to SMML. The approximation itself makes many assumptions regarding the
underlying estimation problem, and, furthermore, makes no guarantees regarding
the inaccuracy incurred by using it. The main guarantees given are that WF,
like SMML, is representation-invariant, and that WF is in general not
identical to SMML, because WF’s estimates are a continuous function of $x$ for
problems with a smooth $(\bm{x},\bm{\theta})$, whereas SMML is piecewise
constant.

####  B.1.3 The evolution of MML justifications

Interestingly, though the MML criterion, as described above, is at heart an
information-theoretic criterion, when Wallace and Boulton (1975) describe
their motivation, it is not an information-theoretic one. The paper sets out
with a motivation remarkably similar to ours here: it aims to extend discrete
MAP into the continuous domain, while preserving certain good properties of
discrete MAP that are not met by continuous MAP. Wallace and Boulton
reference, specifically, the idea of representation invariance. They do not,
however, provide any theoretical rationale regarding why one should start with
the discrete MAP estimate, other than the following:

> We do not here advance any argument in favour of [discrete MAP], save to
> note that it in some sense yields the most plausible, or least improbable
> account of what has been observed.

Wallace and Boulton derive Strict MML (SMML) as a representation invariant
extension of discrete MAP by several approximations, chief of which is the
discretisation of the observation space. This particular approximation is a
choice that makes a significant difference to the meaning of representation
invariance, and the approximate nature of the resulting solution can be
observed directly, e.g. when applying it on a discrete problem, where discrete
MAP can also be used: the SMML estimate and the discrete MAP estimate are not
the same.

At the conclusion of Wallace and Boulton (1975), however, the paper points out
that the SMML estimate can be given a different interpretation, this time from
an information-theoretic perspective rather than a Bayesian perspective, and
under this alternate description SMML becomes an exact solution.

According to Dowe (2008), this dual justification is not a fluke, but a
reflection of the different authors’ perspectives:

> Chris [Wallace] was already a Bayesian from his mid-20s in the 1950s while
> David Boulton was clearly talking here in the spirit of (en)coding. Story
> has it that they had their separate approaches, went away and did their
> mathematics separately, re-convened about 6 weeks later and found out that
> they were doing essentially the same thing.

The paper then goes on to quote Wallace himself, from a talk given 2003,
describing the birth of SMML as a synthesis of these two ideas:

> `[`W`]`e came together again and I looked at his maths and he looked at my
> maths and we discovered that we had ended up at the same place. There really
> wasn’t any difference between doing Bayesian analyses the way I thought one
> ought to do Bayesian analyses and doing data compression the way he thought
> you ought to do data compression. Great light dawned…

By the time of the writing of the other seminal paper of MML, Wallace and
Freeman (1987a); Wallace and Freeman (1987b), where the Wallace-Freeman
estimator was first introduced, the information-theoretic view of MML was
already more deeply entrenched, but the authors nevertheless use
justifications that align remarkably well with the justifications for error
intolerant estimation.

They note, for example, as a weakness of the Bayes estimator approach that it
can yield an estimate outside of the given parameter space, and stress the
usefulness of their method in settling on a single hypothesis rather than on a
weighted mixture of possibilities, even bringing up, as we do here, the need
for this in scientific inference. They write:

> `[`W`]`ould we be happy with a scientist who proposed a Bayesian mixture of
> a countably infinite set of incompatible models for electromagnetic fields?

Thus, the early motivations for MML align remarkably well with our motivations
in deriving error intolerant estimation. However, they rely on a cascade of
approximations, first from Discrete MAP to SMML, and then from SMML to WF.
Regarding the second of these, Wallace and Freeman write:

> Even if the approximations used in this paper cannot be justified, the
> general concepts can still be applied.

In other words, the authors acknowledge the lack of theoretical justification
for their approximations, leave room for the possibility that better
approximations can be found, and stress that the importance is in the
underlying concepts that these results embody.

Throughout the years, other good properties were alleged for SMML and WF,
which would have provided additional justification for them. For example, a
general form of consistency for MML results was argued by Dowe and Wallace
(1997). However, these good properties have since been refuted for both WF and
Strict MML itself (Brand, 2019).

By 2005, when Wallace (2005), the canonical text of MML literature, was
published, the information-theoretic view of MML was the orthodox view. Thus,
Wallace (2005) describes Strict MML as an _exact_ solution, not as a
derivative of Discrete MAP, and WF as an approximation thereof.

In the same year, Comley and Dowe (2005) advocated a simple recipe for the MML
practitioner to follow: if one’s estimation problem has continuous parameters,
one should use WF; if it is discrete—use Discrete MAP. Thus, while originally
(working with a Bayesian motivation) Discrete MAP was the ideal starting
point, and SMML—a derivative thereof, now, from an information-theoretic lens,
SMML was seen as the ideal, and both Discrete MAP and WF were viewed as its
approximations.

Figure 1 summarises this discussion, providing a graphical depiction of the
relationships between Error Intolerant Estimation and MML, in their view of
the Wallace-Freeman estimator: while MML sees the connections between Discrete
MAP and WF as approximative and going through SMML as an intermediary, whether
in the original Bayesian view in which these results were first presented or
in the information-theoretic view that was later adopted, Error Intolerant
Estimation derives exactly both Discrete MAP and the Wallace-Freeman estimator
(which is not an “approximation” in this context), and does so using purely
Bayesian reasoning and the presented axioms.

Discrete MAPStrict MMLError IntolerantEstimationWallace-Freeman(Wallace and
Boulton, 1975)(Comley and Dowe, 2005)(Wallace and Freeman, 1987b)(Wallace,
2005) Figure 1: A graphical comparison of Error Intolerant Estimation’s
derivation of the Wallace-Freeman estimator with MML’s. Squiggly lines
represent approximations and straight lines represent exact derivations. In
light grey: When Strict MML was first introduced in Wallace and Boulton
(1975), Discrete MAP was taken as the ideal estimator, and SMML was derived as
an approximation. Similarly, when WF was first introduced in Wallace and
Freeman (1987b), it was a further, secondary approximation, but still in
keeping with the view that these estimators formalise the ideal of finding a
“most plausible” estimate. In dark grey: More modern MML texts (Comley and
Dowe, 2005; Wallace, 2005) consider Strict MML to be the ideal estimator, and
consider both Discrete MAP and WF its approximations. In black: Error
Intolerant Estimation, on the other hand, derives both Discrete MAP and WF
directly and independently from its axiomatic principles, and requires for
this neither approximations nor intermediaries.

We believe this places the methodology of working with Discrete MAP in the
discrete setting and with Wallace-Freeman estimation in a continuous setting
on much stronger theoretical footing.

It not only explains the many successes of this recipe over the past 30 years,
of which recent examples include Bregu et al. (2021b, a); Hlaváčková-Schindler
and Plant (2020); Bourouis et al. (2021); Sumanaweera et al. (2018); Schmidt
and Makalic (2016); Saikrishna et al. (2016); Jin and Rumantir (2015);
Kasarapu et al. (2014), it also explains the recipe’s failings, because it
provides robust reasoning regarding why the method works, and therefore when
we should expect it to underperform.

For example, in Webb and Petitjean (2016) the WF results are less competitive,
but this is clear when analysed from an Error Intolerant Estimation
perspective: the paper utilises WF in a context where inferences have to be
made piecewise, serially. The success criterion in such a study is “best-fit
for later inferences”, not an error intolerant choice of a most-plausible
“truth”. Because of this, the error intolerance criterion is ill-fitting for
the task, and the WF results, being overly conservative, were outperformed by
other methods. MML reasoning alone does not predict this.

###  B.2 Error intolerance and trade-off based methods

In the main paper, we describe error intolerant estimation as a “bet-the-
ranch” type situation, where every mistake is equally and prohibitively bad,
and mention that trade-off based methods are an ill-fit for such a scenario.

In this supplementary section, we expand on this description in greater
detail, and provide intuition regarding why setting all mistakes as equally
bad avoids such trade-offs.

####  B.2.1 A trade-off example

Consider a physicist measuring extremely low electrical charges as part of an
experiment. The physicist is trying to determine whether only one electron was
discharged as part of the experiment ($\theta=1$), or whether it was two
electrons ($\theta=2$). For simplicity, let us assume these are the only
choices, and that there is no reason to consider either option more likely a
priori.

If the physicist’s equipment measures the electric charge discharged in the
experiment as equivalent to $1.5$ electrons (which we’ll take to mean that the
probability of either event is the same) and if the costs of either possible
type of mistake is the same, most estimation methods will have difficulty
deciding which way to go.

Lamport (2012) named this problem _Buridan’s principle_ , in reference to the
dilemma of Buridan’s ass (named after the fourteenth century French
philosopher Jean Buridan) where an ass placed equidistant between two bales of
hay must starve to death because it has no reason to choose one bale over the
other.

When using Bayes estimators, however, the problem becomes much worse. Consider
_any_ value, $x$, reported by the measuring equipment. Consider _any_ non-
trivial joint probability distribution on $(\bm{x},\bm{\theta})$. For example,
suppose that the posterior distribution given the data is
$\mathbf{P}(\bm{\theta}=1|x)=0.9$ and $\mathbf{P}(\bm{\theta}=2|x)=0.1$.
Consider, further, any choice of a loss function (as long as loss increases
super-linearly with error magnitude). For example, let us assume quadratic
loss:

$L(\theta_{1},\theta_{2})=|\theta_{1}-\theta_{2}|^{2}.$

Recall that a Bayes estimator is defined as a minimiser of expected loss:

$\hat{\theta}^{L}_{\text{Bayes}}(x)\stackrel{{\scriptstyle\text{def}}}{{=}}\operatornamewithlimits{argmin}_{\theta\in\Theta}\mathbf{E}[L(\bm{\theta},\theta)|\bm{x}=x].$

So, for example, the expected loss for the choice $\hat{\theta}(x)=1$ is

$0.9\times L(1,1)+0.1\times L(2,1)=0.9\times 0+0.1\times 1=0.1,$

which is clearly superior to the expected loss for the choice
$\hat{\theta}(x)=2$,

$0.9\times L(1,2)+0.1\times L(2,2)=0.9\times 1+0.1\times 0=0.9.$

Given that our chosen $L$ is defined not only over $\Theta\times\Theta$ but
rather over $\mathbb{R}^{M}\times\mathbb{R}^{M}$, let us, however, consider
what would happen if we extend the definition of our Bayes estimator to

$\operatornamewithlimits{argmin}_{\theta\in\mathbb{R}^{M}}\mathbf{E}[L(\bm{\theta},\theta)|\bm{x}=x]$

(noting that in general Bayes inference, the space of actions, which is here
the possible estimation values, is unrelated to the hypothesis space,
$\Theta$.)

When all $\mathbb{R}^{M}$ is considered, the optimal choice is
$\hat{\theta}(x)=1.1$, for which the expected loss is only

$0.9\times L(1,1.1)+0.1\times L(2,1.1)=0.9\times 0.01+0.1\times 0.81=0.09.$

So, in this case, such extended Bayes estimation would have advocated a value
in the open interval $(1,2)$, a value that clearly cannot be the true value of
$\theta$ because it lies outside $\Theta$.

One can repeat this same example with any choice of observations, any choice
of positive posterior probabilities, and any choice of loss function that is
super-linear with distance, and the estimate will never be within $\Theta$.

This is a general problem with trade-off-based estimation methods, and while
it is particularly jarring in cases of discrete estimation or of continuous
estimation over a non-convex $\Theta$, where the estimation can fall outside
of $\Theta$ altogether, the issue exists in all estimation problems, and does
not require any extension to the definition of Bayes estimation. Consider
exactly the same situation as before, except this time let us define
$\Theta=[1,2]$, and choose our prior so that the probability that $\theta$ is
in the open interval $(1,2)$ is negligible.111This could model, e.g., a
situation in which the scientist, being a scientist, allows some tiny
probability to the possibility that everything we think we know about the
discrete nature of electric charges will ultimately be proven false. The Bayes
estimate (this time, without any extension to its definition) will be the same
as in the extended Bayes one: $1.1$, in our example.

Thus, a Bayes estimator is able to choose a highly unlikely $\theta$, simply
because it is a convenient trade-off between other $\theta$ values, each of
which is far more likely than the one actually chosen. In error intolerant
estimation we specifically aim to avoid such trade-off based decisions.

####  B.2.2 How error intolerance limits trade-offs

In the proofs of both Theorem 2.6 and Theorem A.1, we transform the risk
attitude functions, $T_{\epsilon}$, into new functions, $A_{\epsilon}$, that
map from objective loss to subjective utility.

One way to interpret this rescaling is that we use a loss function, $L$, to
determine how similar or different $\theta$ values are to each other, and then
use an attenuation function, $A_{\epsilon}$, to translate this divergence into
a _similarity measure_ : a $1$ indicates an exact match and a $0$ that the two
$\theta$ are not materially similar.

This rescaling gives a better intuitive understanding of why lowering one’s
error tolerance leads to decisions more appropriate to the error intolerance
scenario: for each $\theta_{1}$, the neighbourhood in which
$A_{\epsilon}(L(\theta_{1},\theta_{2}))$ is positive is the neighbourhood for
which mistaking $\theta_{1}$ for $\theta_{2}$ is at all an acceptable error.
Beyond that, the utility of choosing a given $\theta_{2}$ is zero. Lowering
the error tolerance contracts this neighbourhood of partial similarity; at the
limit, anything that is not “essentially identical” to $\theta_{1}$ according
to the loss function contributes zero utility, the minimal possible value. The
optimal decision rule in such a scenario is therefore one that strives, as
much as possible, to avoid this possibility, and therefore maximises the
probability that its decision is, essentially, “exactly right”.

There is, in particular, no reason for such an estimator to ever produce an
estimate $\hat{\theta}(x)$ outside $\Theta$ even when such values are
considered, because the neighbourhood of $\theta$ values for which
$A_{\epsilon}(L(\theta,\hat{\theta}(x)))$ is positive will ultimately shrink
to exclude all $\theta\notin\Theta$, thus yielding for such a decision a
utility of zero.

It is interesting to note that while the loss function, $L$, retains in error
intolerant estimation its basic meaning from Bayes estimation, it changes how
it impacts the estimator. Unlike in standard Bayesian risk minimisation, in
our scenario the only parts of the loss function that are of interest are
those within an $\epsilon$-neighbourhood of the diagonal. One can think of
this as using the loss function to convey the local topology of the hypothesis
space. This is because error intolerant estimation does not concern itself
with errors larger than any $\epsilon$: these are all considered equally
“wrong”.

###  B.3 Rationale of the axioms on loss

In the main paper we introduce four axioms on the choice of a loss function to
be used in error intolerant estimation. In this section, we provide a deeper
analysis of the rationale and incontrovertibility of each.

####  B.3.1 IRP

Our first axiom is Invariance to Representation of Parameter Space (IRP).
Informally, it states that our loss evaluation (and therefore our choice of
estimate) should not depend on how we name our hypotheses.

While the IRP axiom may seem highly intuitive, it is actually the one axiom
among our four for which it is least obvious why it is “incontrovertible”.
After all, common loss functions in use, such as quadratic loss, do not
satisfy it.

Not only that, but the very fact that common loss functions do not satisfy IRP
is often put to good use. When estimating, say, the size of a cohort it makes
a big difference whether one is interested in getting the answer right in
terms of the minimum error in number of people or in terms of minimum error in
percents. One favours the use of a linear representation, the other—a
logarithmic one.

As another example, consider estimation of a probability. When estimating the
probability that a person is likely to respond to a marketing message, the
error needs to be measured on a linear scale, because the underlying analysis
is about maximising the expected size of the total cohort that will respond to
the message. On the other hand, when estimating the probability that a
dangerous experiment has crossed its safety bounds, the difference between a
probability of $0.001\%$ and $0.0001\%$ is far more substantial than between
$4\%$ and $5\%$. Again, a logarithmic scale becomes more appropriate, in order
to accentuate the difference between near-zero figures. In yet other domains,
both near-zero and near-one probability ranges will need to be magnified, and
a log-odds metric may become appropriate.

In light of this, what is the justification for IRP?

The important element to note is that in all these examples the choice of
estimation method was led by a desire to manage errors in particular ways.
This is not the case in error intolerant estimation. Here, we are in a bet-
the-house situation. Everything rests on the question of whether our estimate
is _correct_ , not how well it approximates the correct answer.

In such a situation, a person faced with, for example, the need to estimate
the side length of a cube and estimating it to be, say, $2$ metres in length
will undoubtedly answer $8$ cubic metres if asked to estimate the volume of
said cube. Both answers must be informed by that person’s view of what the
most likely “truth” is regarding the dimensions of the cube, which is why this
truth must always be the same, regardless of how the person is asked to
express it.

We see, therefore, that while IRP is not universally incontrovertible, and is,
in fact, frequently violated, that is not the case within the world of error
intolerant estimation. Here, there is no reason to doubt its veracity.

####  B.3.2 IRO

Our second axiom, Invariance to Representation of Observation Space (IRO), is
also regarding invariance to representation, but this time regarding the
representation of observation space. Informally, IRO guarantees that the
estimate should remain the same regardless of how the observations are
presented to us.

Is IRO incontrovertible?

Statisticians analysing real-world data spend much of their time wrangling
this data. This wrangling, bringing the raw input data into a usable format,
is a process that only works under the assumption that the raw formatting of
the input does not matter: regardless of how the input is initially presented,
one spends the necessary effort to bring the information into the format most
appropriate for the analysis of the problem.

This is precisely the statement of IRO: the choice of the raw input format
should not influence the insights derived from the data.

####  B.3.3 IIA

The third axiom, Invariance to Irrelevant Alternatives (IIA), takes its name
from Nash (1950), a seminal work in game theory, where it was introduced in
the following formulation: if, when faced with a choice among alternatives in
a set $\Theta$, a certain rational player would choose $\theta\in\Theta$,
then, with all other conditions being equal, the same rational player would
make the same choice if the alternatives were narrowed down to some
$\Theta^{\prime}\subset\Theta$, as long as $\theta\in\Theta^{\prime}$. For
example, if, given any evidence $x$, we estimate that the best place to dig
for gold in Australia is, say, in Melbourne, then IIA states that we should
reach the same conclusion, given the same evidence, also if the question asked
is where to dig for gold in only _mainland_ Australia.

Our IIA axiom merely reformulates this principle so that it addresses the
underlying loss function rather than the estimate itself.

Informally, our formulation of IIA states that the loss experienced when
choosing hypothesis $\theta_{2}$ in cases where the true value is $\theta_{1}$
should only relate to the nature of these two hypotheses, not any other
property of the estimation problem. This follows directly from the semantics
of what loss function values are meant to represent.

The IIA axiom has been a hotly debated axiom ever since its introduction in
1950. In our context, however, it is a direct statement regarding the nature
of the error intolerant scenario: we aim for an estimate that is “best”
according to its own merits, not based on its relationship with other
potential estimates (as is the case with trade-off-based methods). This was
already stipulated by AIA in the context of the estimator itself, as
parameterised by a loss function, and IIA merely extends this requirement also
to the underlying loss function, and thus to the estimator as a whole.

####  B.3.4 ISI

What ISI states is that merely adding data bits to the observation that do not
convey any information about $\bm{\theta}$ should not change our estimate of
it.

Like our other axioms, ISI, too, seems nearly tautological, as one would be
hard put to justify why such extra, information-free data should influence a
point estimate in any way.

## References

  * Ali and Silvey (1966) Ali, S. and Silvey, S. (1966).  “A general class of coefficients of divergence of one distribution from another.”  Journal of the Royal Statistical Society: Series B (Methodological), 28(1): 131–142.
  * Amari and Nagaoka (2000) Amari, S. and Nagaoka, H. (2000).  Methods of information geometry, 56.  Translations of mathematical monographs. American Mathematical Society.
  * Bobkov et al. (2014) Bobkov, S., Chistyakov, G., and Götze, F. (2014).  “Fisher information and the central limit theorem.”  Probability Theory and Related Fields, 159(1-2): 1–59.
  * Bourouis et al. (2021) Bourouis, S., Channoufi, I., Alroobaea, R., Rubaiee, S., Andejany, M., and Bouguila, N. (2021).  “Color object segmentation and tracking using flexible statistical model and level-set.”  Multimedia Tools and Applications, 80(4): 5809–5831.
  * Brand (2019) Brand, M. (2019).  “MML is not consistent for Neyman-Scott.”  IEEE Transactions on Information Theory, 66(4): 2533–2548.
  * Bregu et al. (2021a) Bregu, O., Zamzami, N., and Bouguila, N. (2021a).  “Mixture-based clustering for count data using approximated Fisher Scoring and Minorization–Maximization approaches.”  Computational Intelligence, 37(1): 596–620.
  * Bregu et al. (2021b) — (2021b).  “Online mixture-based clustering for high dimensional count data using Neerchal–Morel distribution.”  Knowledge-Based Systems, 225: 107051.
  * Comley and Dowe (2005) Comley, J. and Dowe, D. (2005).  “Minimum Message Length and generalized Bayesian nets with asymmetric languages.”  In Advances in Minimum Description Length: Theory and applications, 265–294. MIT Press.
  * Dowe (2008) Dowe, D. (2008).  “Foreword re C. S. Wallace.”  The Computer Journal, 51(5): 523–560.
  * Dowe and Wallace (1997) Dowe, D. and Wallace, C. (1997).  “Resolving the Neyman-Scott problem by Minimum Message Length.”  Computing Science and Statistics, 614–618.
  * Dowty (2015) Dowty, J. (2015).  “SMML estimators for 1-dimensional continuous data.”  The Computer Journal, 58(1): 126–133.
  * Farr and Wallace (2002) Farr, G. and Wallace, C. (2002).  “The complexity of Strict Minimum Message Length inference.”  The Computer Journal, 45(3): 285–292.
  * Grünwald (2007) Grünwald, P. D. (2007).  The Minimum Description Length principle.  MIT press.
  * Hlaváčková-Schindler and Plant (2020) Hlaváčková-Schindler, K. and Plant, C. (2020).  “Heterogeneous Graphical Granger Causality by Minimum Message Length.”  Entropy, 22(12): 1400.
  * Jin and Rumantir (2015) Jin, Y. and Rumantir, G. (2015).  “A two tiered finite mixture modelling framework to cluster customers on EFTPOS network.”  In Australasian Joint Conference on Artificial Intelligence, 276–284. Springer.
  * Kasarapu et al. (2014) Kasarapu, P., Garcia de la Banda, M., and Konagurthu, A. (2014).  “On representing protein folding patterns using non-linear parametric curves.”  IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB), 11(6): 1218–1228.
  * Lamport (2012) Lamport, L. (2012).  “Buridan’s principle.”  Foundations of Physics, 42(8): 1056–1066.
  * Nash (1950) Nash, J., Jr (1950).  “The bargaining problem.”  Econometrica: Journal of the Econometric Society, 18(2): 155–162.
  * Pollard (2002) Pollard, D. (2002).  A user’s guide to measure theoretic probability, volume 8.  Cambridge University Press.
  * Rissanen (1987) Rissanen, J. (1987).  “Stochastic complexity and the MDL principle.”  Econometric Reviews, 6(1): 85–102.
  * Rissanen (1999) — (1999).  “Hypothesis selection and testing by the MDL principle.”  The Computer Journal, 42(4): 260–269.
  * Rudin (1964) Rudin, W. (1964).  Principles of mathematical analysis, volume 3.  McGraw-Hill New York.
  * Saikrishna et al. (2016) Saikrishna, V., Dowe, D., and Ray, S. (2016).  “Statistical compression-based models for text classification.”  In Eco-friendly Computing and Communication Systems (ICECCS), 2016 Fifth International Conference on, 1–6. IEEE.
  * Schmidt and Makalic (2016) Schmidt, D. and Makalic, E. (2016).  “Minimum Message Length analysis of multiple short time series.”  Statistics & Probability Letters, 110: 318–328.
  * Shannon (1948) Shannon, C. (1948).  “A mathematical theory of communication.”  The Bell System Technical Journal, 27.
  * Sumanaweera et al. (2018) Sumanaweera, D., Allison, L., and Konagurthu, A. (2018).  “The bits between proteins.”  In 2018 Data Compression Conference, 177–186. IEEE.
  * Wallace (2005) Wallace, C. (2005).  Statistical and inductive inference by Minimum Message Length.  Information Science and Statistics. Springer Verlag.
  * Wallace and Boulton (1968) Wallace, C. and Boulton, D. (1968).  “An information measure for classification.”  The Computer Journal, 11(2): 185–194.
  * Wallace and Boulton (1975) — (1975).  “An invariant Bayes method for point estimation.”  Classification Society Bulletin, 3(3): 11–34.
  * Wallace and Freeman (1987a) Wallace, C. and Freeman, P. (1987a).  “Contribution to the discussion of the papers by Dr. Rissanen and Professors Wallace and Freeman.”  Journal of the Royal Statistical Society: Series B (Methodological), 49(3): 262–265.
  * Wallace and Freeman (1987b) — (1987b).  “Estimation and inference by compact coding.”  Journal of the Royal Statistical Society: Series B (Methodological), 49(3): 240–252.  See also Discussion on pp. 252–265.
  * Webb and Petitjean (2016) Webb, G. and Petitjean, F. (2016).  “A multiple test correction for streams and cascades of statistical hypothesis tests.”  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1255–1264. ACM.