# Asymptotics for power posterior mean estimation

Ruchira Ray  Department of Statistics
Columbia University
New York, United States
<EMAIL_ADDRESS>Marco Avella Medina  Department of Statistics
Columbia University
New York, United States
<EMAIL_ADDRESS>Cynthia Rush  Department of Statistics
Columbia University
New York, United States
<EMAIL_ADDRESS>
###### Abstract

Power posteriors “robustify” standard Bayesian inference by raising the
likelihood to a constant fractional power, effectively downweighting its
influence in the calculation of the posterior. Power posteriors have been
shown to be more robust to model misspecification than standard posteriors in
many settings. Previous work has shown that power posteriors derived from low-
dimensional, parametric locally asymptotically normal models are
asymptotically normal (Bernstein-von Mises) even under model misspecification.
We extend these results to show that the power posterior moments converge to
those of the limiting normal distribution suggested by the Bernstein-von Mises
theorem. We then use this result to show that the mean of the power posterior,
a point estimator, is asymptotically equivalent to the maximum likelihood
estimator.

###### Index Terms:

power posterior, $\alpha$-posterior, Bernstein-von Mises, posterior mean
convergence

##  I Introduction

In this work, we study $\alpha$-posteriors (also known as power posteriors),
which are proportional to the product of the prior and the likelihood raised
to a constant, fractional power [4, Chapter 8.6]. This procedure (also known
as posterior tempering) decreases the influence of the likelihood in the
calculation of the posterior. Decreasing the influence of the likelihood has
been explored for several problems as a way to improve Bayesian procedures [9,
12, 13]. More recent literature about this area – under the name
$\alpha$-posteriors – has focused on the robustness to model misspecification
and data corruption of the $\alpha$-posterior compared to that of the ordinary
posterior both empirically [5, 6, 11] and theoretically [11, 2]. To this end,
both the theoretical and empirical properties of this procedure have been
investigated in recent years [5, 11, 3, 1, 2]. In particular, [2] established
that $\alpha$-posteriors derived from low-dimensional, parametric models are
asymptotically normal. In other words, they obey a Bernstein-von Mises
theorem. For standard posteriors, the Bernstein-von Mises theorem says that
the large sample distribution of the posterior is close to a normal
distribution centered around the maximum likelihood estimator (MLE) with
posterior variance depending on the “curvature” of the likelihood function [8,
7, 10]. The work in [2] shows that raising the likelihood to the power
$\alpha$ while calculating the posterior distribution leaves the mean of the
limiting Gaussian unchanged but divides the variance of the limiting Gaussian
by $\alpha$.

We extend the results of [2] by showing that the $k^{th}$ moment of the
$\alpha$-posterior converges in probability to the $k^{th}$ moment of the
limiting normal distribution suggested by the Bernstein-von Mises theorem in
[2]. This result (Theorem 1) holds under similar conditions to those of [2],
but we additionally assume the finiteness of the $k^{th}$ $\alpha$-posterior
moment to prove $k^{th}$ moment convergence. We use Theorem 1 to establish the
$\sqrt{n}$-asymptotic normality of the $\alpha$-posterior mean (Theorem 2). In
particular, we show that the $\alpha$-posterior mean has the same asymptotic
distribution as that of the MLE. This result shows that, asymptotically,
tempering does not affect point estimation from tempered posteriors.

The proof of our first result is adapted from the Bernstein-von Mises proofs
of [7] and [2] in order to show convergence of the moments of the
$\alpha$-posterior rather than convergence to a limiting normal distribution
in total variation. The second result follows almost immediately from our
first result. The main assumptions required in our proofs are slightly
stronger than those from [2] and [7], as discussed in (II-B). Like [2] and
[7], our results make no assumptions of model correctness or independent and
identically distributed (i.i.d.) data.

##  II Main Results

###  II-A Notation and preliminaries

Let $\phi(\cdot|\mu,\Sigma)$ denote the multivariate normal density with mean
$\mu$ and covariance matrix $\Sigma$. The indicator function of an event $A$
is denoted $\mathbbm{1}\\{A\\}$. For a sequence of distributions $P_{n}$ on
random variables (or matrices) $X_{n}$, if
$\lim_{n}P_{n}(\lvert|X_{n}\rvert|_{2}>\epsilon)=0$ for every $\epsilon>0$, we
say $X_{n}=o_{P_{n}}(1)$. We say $X_{n}$ converges in $L_{1}$ to a random
variable $X$ if $\lim_{n\rightarrow\infty}\mathbb{E}[|X_{n}-X|]=0$ and
$\mathbb{E}[|X_{n}|]$ and $\mathbb{E}[|X|]$ exist. We say $X_{n}$ is ‘bounded
in $P_{n}$-probability’, or $O_{P_{n}}(1)$, if for every $\epsilon>0$ there
exists $M_{\epsilon}>0$ such that
$P_{n}(\lvert|X_{n}\rvert|_{2}<M_{\epsilon})\geq 1-\epsilon$. We use
$||X||_{p}=\left(\sum_{i}|X_{i}|^{p}\right)^{1/p}$ to denote the $p$-norm. Let
$B_{v}(\delta)$ denote a closed ball of radius $\delta$ around vector
$v\in\mathbb{R}^{p}$.

Statistical Model: Let
$\mathcal{F}_{n}=\\{f_{n}(\cdot|\theta):\theta\in\Theta\subseteq\mathbb{R}^{p}\\}$
be a parametric family of densities used as a statistical model for the random
sample $X^{n}=(X_{1},\dots,X_{n})$. This model is allowed to be misspecified
in the sense that $f_{0,n}$, the true density of the random sample $X^{n}$,
may not belong to $\mathcal{F}_{n}$. We will use the notion of the pseudo-true
parameter $\theta^{*}$, which is the true data-generating $\theta$ if the
model is well-specified, meaning $f_{0,n}(X^{n}|\theta)\in\mathcal{F}_{n}$. If
the model is misspecified, $\theta^{*}$ is the value of $\theta\in\Theta$ such
that the model $f_{n}(X^{n}|\theta^{*})$ is closest in KL-divergence to the
true model. Denote the (pseudo) MLE by
$\hat{\theta}^{ML}=\arg\max_{\theta}f_{n}(X^{n}|\theta)$, where the asymptotic
behavior of the MLE is determined with respect to the model $f_{0,n}$. Let
$\mathbb{E}_{f_{0,n}}$ and $\mathbb{P}_{f_{0,n}}$ denote the expectation and
probability taken with respect to $f_{0,n}$.

Power posterior: Given the statistical model $\mathcal{F}_{n}$, a prior
density $\pi(\theta)$ for $\theta$, and a scalar $\alpha>0$, the power
posterior, or $\alpha$-posterior, has the density:

$\pi_{n,\alpha}(\theta|X^{n})=\frac{f_{n}(X^{n}|\theta)^{\alpha}\pi(\theta)}{\int_{\Theta}f_{n}(X^{n}|\theta)^{\alpha}\pi(\theta)d\theta}.$
(1)

###  II-B Assumptions

Our main results hold under the following assumptions:

  * (A0)

The MLE, $\hat{\theta}^{ML}$, is unique and there exists a $\theta^{*}$ in the
interior of $\Theta$ such that
$\hat{\theta}^{ML}\overset{p}{\rightarrow}\theta^{*}$ and
$\sqrt{n}(\hat{\theta}^{ML}-\theta^{*})$ is asymptotically normal.

  * (A1)

The prior density $\pi(\theta)$ is continuous and positive in a neighborhood
of $\theta^{*}$.

  * (A2)

Given $\Delta_{n,\theta^{*}}=\sqrt{n}(\hat{\theta}^{ML}-\theta^{*})$, there
exists a positive definite matrix $V_{\theta^{*}}$ such that

$\begin{split}R_{n}(h)&\equiv\log\left(\frac{f_{n}(X^{n}|\theta^{*}+h/\sqrt{n})}{f_{n}(X^{n}|\theta^{*})}\right)\\\
&\qquad-h^{T}V_{\theta^{*}}\Delta_{n,\theta^{*}}+\frac{1}{2}h^{T}V_{\theta^{*}}h,\end{split}$
(2)

satisfies $\sup_{h\in K}|R_{n}(h)|\rightarrow 0$ in $f_{0,n}$-probability for
all compact sets $K\subseteq\mathbb{R}^{p}$.

  * (A3)

There exists a $\gamma>0$ such that the following is finite

$\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k_{0}(1+\gamma)}n^{-\frac{p}{2}}\pi_{n,\alpha}\left(\theta^{*}+\frac{h}{\sqrt{n}}\right)dh\right],$

for some $k_{0}\in\mathbb{N}$.

We adapt the proof of [2, Theorem 1] – which in turn adapts the arguments of
[7, Theorem 2.1] – to prove Theorem 1. We retain assumptions (A0) \- (A2) from
[7] and [2], but make a slightly stronger posterior concentration assumption
in (A3). Indeed, our assumption implies the posterior concentration assumption
in [7] and [2], which states that $\pi_{n,\alpha_{n}}(\cdot|X^{n})$
concentrates at rate $\sqrt{n}$ around $\theta^{*}$ if for every sequence
$r_{n}\rightarrow\infty$,

$\mathbb{E}_{f_{0,n}}\left[\mathbb{P}_{\pi_{n,\alpha}(\theta|X^{n})}\left(||\sqrt{n}(\theta-\theta^{*})||_{2}>r_{n}\right)\right]\rightarrow
0.$ (3)

Applying Markov’s inequality to the left side of (3) and using the change of
variable $h=\sqrt{n}\left(\theta-\theta^{*}\right)$ gives us the upper bound

$r_{n}^{-k_{0}}\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k_{0}}n^{-\frac{p}{2}}\pi_{n,\alpha}\left(\theta^{*}+\frac{h}{\sqrt{n}}\right)dh\right].$
(4)

Because $r_{n}\rightarrow\infty$, the posterior concentration assumption is
satisfied if

$\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k_{0}}n^{-\frac{p}{2}}\pi_{n,\alpha}\left(\theta^{*}+\frac{h}{\sqrt{n}}\right)dh\right]<\infty,$

which is implied by (A3).

###  II-C Convergence of power posterior moments

We denote by $\phi(\cdot|\mu,\Sigma)$ the density of a normal random vector
with mean $\mu$ and covariance matrix $\Sigma$. Recall that [2] showed the
following result:

$d_{TV}\left(\pi_{n,\alpha}\left(\theta|X^{n}\right),\phi\left(\theta\bigg{|}\hat{\theta}^{ML},\frac{1}{\alpha
n}V_{\theta^{*}}^{-1}\right)\right)\rightarrow 0,$

in $f_{0,n}$-probability, where $V_{\theta^{*}}$ is the matrix from assumption
(A2) and it can be thought of as measuring the curvature of the likelihood.
Theorem 1 essentially shows that the $k^{th}$ moment of the $\alpha$-posterior
approaches the $k^{th}$ moment of the limiting Gaussian from [2] with growing
sample size.

######  Theorem 1.

Assume (A0)–(A3) hold. Then, for any integer $k\in[1,k_{0}]$,

$\begin{split}&\int_{\mathbb{R}^{p}}\left|\left|h^{\otimes
k}\right|\right|_{1}\left|\pi_{n,\alpha}^{LAN}(h)-\phi_{n}(h)\right|dh\rightarrow
0,\end{split}$

in $f_{0,n}$-probability, where

$\left|\left|h^{\otimes
k}\right|\right|_{1}=\sum_{i_{1}=1}^{p}\cdots\sum_{i_{k}=1}^{p}\left|h_{i_{1}}\times\cdots\times
h_{i_{k}}\right|,$ (5)

is the $1$-norm of a $k^{th}$ order tensor and

$\begin{split}&\pi_{n,\alpha}^{LAN}(h)\equiv
n^{-p/2}\pi_{n,\alpha}(\theta^{*}+h/\sqrt{n}|X^{n}),\\\ &\phi_{n}(h)\equiv
n^{-p/2}\phi\left(h\big{|}\sqrt{n}(\hat{\theta}^{ML}-\theta^{*}),V_{\theta^{*}}^{-1}/\alpha\right),\end{split}$
(6)

are the centered and scaled versions of the $\alpha$-posterior and its
limiting normal.

######  Remark 1.

We mention that the objects in 6 are proper densities. Indeed, using a change
of variable $h=\sqrt{n}(\theta-\theta^{*})$, we see

$\displaystyle\int\pi_{n,\alpha}^{LAN}(h)dh$ $\displaystyle=\int
n^{-p/2}\pi_{n,\alpha}(\theta^{*}+h/\sqrt{n}|X^{n})dh$
$\displaystyle=\int\pi_{n,\alpha}(\theta|X^{n})d\theta=1.$

Similarly,

$\begin{split}\int\phi_{n}(h)dh&=\int
n^{-\frac{p}{2}}\phi\left(h\big{|}\sqrt{n}(\hat{\theta}^{ML}-\theta^{*}),\frac{V_{\theta^{*}}^{-1}}{\alpha}\right)dh\\\
&=\int\phi\left(\theta\big{|}\hat{\theta}^{ML},\frac{V_{\theta^{*}}^{-1}}{n\alpha}\right)d\theta=1.\end{split}$

######  Remark 2.

With a change of variable $h=\sqrt{n}(\hat{\theta}^{ML}-\theta^{*})$, we see
that as long as $k_{0}\geq 2$, Theorem 1 shows that both of the following
converge to zero in $f_{0,n}$-probability:

$\begin{split}&\int_{\mathbb{R}^{p}}\sqrt{n}\|\theta-\theta^{*}\|_{1}\left|\pi_{n,\alpha}(\theta|X^{n})-\phi\bigg{(}\theta\bigg{|}\hat{\theta}^{ML},\frac{V_{\theta^{*}}^{-1}}{n\alpha}\bigg{)}\right|d\theta,\\\
&\int_{\mathbb{R}^{p}}n\|[\theta-\theta^{*}][\theta-\theta^{*}]^{T}\|_{1}\left|\pi_{n,\alpha}(\theta|X^{n})-\phi\bigg{(}\theta\bigg{|}\hat{\theta}^{ML},\frac{V_{\theta^{*}}^{-1}}{n\alpha}\bigg{)}\right|d\theta.\end{split}$

The proof of [2] and [7] relates “closeness” of the likelihood ratios of two
densities to closeness in terms of total variation distance. Since the results
in [2] and [7] are Bernstein-von Mises results, we modify their proof to
relate “closeness” of likelihood ratios to “closeness” of moments. To
accomplish this, we modify Lemma 4 from [2] (Lemma 1 in this work) to express
the moments of the $\alpha$-posterior in terms of the moments and likelihood
ratios of these two densities. Lemma 1 reveals that the moments of the
$\sqrt{n}$-scaled $\alpha$-posterior must be (I) finite on a growing
neighborhood surrounding the pseudo-true parameter and Lemma 2 shows that the
moments are (II) vanishing on the complement of this neighborhood. Lemmas 1
and 2 require (A3) instead of the weaker posterior concentration assumption
from [2].

###  II-D Asymptotic normality of the Bayes estimator

The result in Section (II-C) is useful for determining the large sample
behavior of point estimators derived from the $\alpha$-posterior. In
particular, because $\alpha$ does not affect the centering of the
$\alpha$-posterior, as the the mean of the posterior approaches
$\hat{\theta}^{ML}$, it stands to reason that the limiting distribution of the
mean of the $\alpha$-posterior is unaffected by $\alpha$. Our second result
verifies this.

######  Theorem 2.

Assume (A0)–(A3) hold. Then the $\alpha$-posterior mean estimator
$\hat{\theta}^{B}=\int_{\mathbb{R}^{p}}\theta\pi_{n,\alpha}(\theta|X^{n})d\theta$
has the following limiting distribution
$\sqrt{n}(\hat{\theta}^{B}-\theta^{*})\overset{d}{\rightarrow}N(0,V_{\theta^{*}}^{-1}).$

The high-level argument in our proof of Theorem 2 involves showing that the
Bayes estimator is asymptotically equivalent to the (pseudo) MLE, which
implies that they have the same limiting distribution. In other words, we need
to show that $\sqrt{n}(\hat{\theta}^{B}-\hat{\theta}^{ML})=o_{p}(1)$. Since
$\hat{\theta}^{B}$ is defined as the first moment of the $\alpha$-posterior,
we can obtain the desired result by examining the moments of the
$\alpha$-posterior centered around the pseudo-true parameter, which we do in
Theorem 1.

##  III Discussion

In this work, we have studied the moments of $\alpha$-posteriors in
parametric, low-dimensional models to better understand point estimators
derived from posterior tempering. We found that because posterior tempering
does not affect the “first-order” properties of the posterior, the limiting
distribution of the posterior mean is also unaffected by tempering. This
result demonstrates that while tempering may induce robustness in the
posterior distribution, it does not do so in the mean of the tempered
posterior. We believe this phenomenon should hold for other point estimators
derived from posteriors (e.g., the posterior median and mode), which is left
for future work.

We also mention some other directions for future work. First, as noted in
Section II-B, our (A3) is stronger than the posterior concentration assumption
used to prove the Bernstein-von Mises results in [2, 7]. However, we believe
that posterior concentration is a sufficient condition for our Theorem 1 to
hold for at least $k=1$ and $k=2$. Future work could involve modifying our
proof to reflect this. Second, while we have advanced our understanding of
$\alpha$-posteriors for fixed, constant, $\alpha$, we would like to establish
similar guarantees to $\alpha$-posteriors where $\alpha$ is selected in a
data-driven way (i.e., in settings where $\alpha$ might change with the sample
size, $n$).

##  IV Proofs of Main Results

###  IV-A Proof of Theorem 1

The proof is adapted from the proofs of [2, Theorem 1] and [7, Theorem 2.1] to
show convergence of the posterior moments rather than the total variation
distance.

Let $Z_{0}$ denote the integral we are trying to control, namely

$Z_{0}\coloneqq\int_{\mathbb{R}^{p}}\left|\left|h^{\otimes
k}\right|\right|_{1}\left|\pi^{LAN}_{n,\alpha}(h)-\phi_{n}(h)\right|dh.$ (7)

Recalling the definition of $\left|\left|h^{\otimes k}\right|\right|_{1}$ in 5
and using the fact that
$\left|\left|h\right|\right|_{1}\leq\sqrt{p}||h||_{2}$, we note that

$\begin{split}\left|\left|h^{\otimes
k}\right|\right|_{1}&=\sum_{i_{1}=1}^{p}\cdots\sum_{i_{k}=1}^{p}\left|h_{i_{1}}\times\cdots\times
h_{i_{k}}\right|\\\
&=\sum_{i_{1}=1}^{p}\left|h_{i_{1}}\right|\cdots\sum_{i_{k}=1}^{p}\left|h_{i_{k}}\right|=\left|\left|h\right|\right|_{1}^{k}\leq
p^{\frac{k}{2}}\left|\left|h\right|\right|_{2}^{k},\end{split}$ (8)

with $p$ fixed. By the bound in 8 and monotonicity of the integral, we have
that

$Z_{0}\leq
p^{\frac{k}{2}}\int_{\mathbb{R}^{p}}\left|\left|h\right|\right|_{2}^{k}\left|\pi^{LAN}_{n,\alpha}(h)-\phi_{n}(h)\right|dh\coloneqq
Z.$ (9)

Thus, it suffices to show that $\mathbb{E}_{f_{0,n}}[Z]\rightarrow 0$, which
implies $\mathbb{E}_{f_{0,n}}[Z_{0}]\rightarrow 0$. This, in turn, implies
that $Z_{0}\overset{p}{\rightarrow}0$ by Markov’s inequality. We show
convergence in $L_{1}$ by (I) showing convergence in $L_{1}$ conditioned on an
intersection of events and (II) showing the complement of these events occur
with vanishing probability.

We now show convergence in $L_{1}$. We will use the following strategy to do
this. For vectors $g,h\in K_{0}$, define the random variables

$\displaystyle
f_{n}^{+}(g,h)=\left\\{1-\frac{\phi_{n}(h)\pi_{n,\alpha}^{LAN}(g|X^{n})}{\pi_{n,\alpha}^{LAN}(h|X^{n})\phi_{n}(g)}\right\\}^{+},$
(10) $\displaystyle
f_{n}^{-}(g,h)=\left\\{\frac{\pi_{n,\alpha}^{LAN}(h|X^{n})\phi_{n}(g)}{\phi_{n}(h)\pi_{n,\alpha}^{LAN}(g|X^{n})}-1\right\\}^{-},$
(11)

where $\\{x\\}^{+}=\max\\{0,x\\}$ and $\\{x\\}^{-}=\max\\{0,-x\\}$. We appeal
to [2, Lemma 5] to show that $f_{n}^{+}(g,h)$ and $f_{n}^{-}(g,h)$ are well-
defined with high-probability. Consider $\eta>0$ and define the following
events,

$\begin{split}\mathcal{A}&=\left\\{\sup_{g,h\in
B_{0}(r_{n})}f_{n}^{+}(g,h)\leq\eta\right\\},\\\
\mathcal{B}&=\left\\{\sup_{g,h\in
B_{0}(r_{n})}f_{n}^{-}(g,h)\leq\eta\right\\}.\end{split}$ (12)

The events $\mathcal{A}$ and $\mathcal{B}$ hold with high probability for
large enough $n$ by [2, Lemma 5]. To apply [2, Lemma 5], we mention that (I)
because $\theta^{*}$ is in the interior of $\Theta\subseteq\mathbb{R}^{p}$ by
(A1), there exists a $\delta>0$ such that ball
$B_{\theta^{*}}(\delta)\subset\Theta$ and $\pi$ is continuous and positive on
the ball and (II) $\sqrt{n}$-stochastic local asymptotic normality holds by
(A2).

Next, we upper bound $\mathbb{E}_{f_{0,n}}[Z]$ as follows using Hölder’s
Inequlity for some $\gamma>0$:

$\begin{split}&\mathbb{E}_{f_{0,n}}[Z]=\mathbb{E}_{f_{0,n}}[Z\mathbbm{1}\\{\mathcal{A}\cap\mathcal{B}\\}]+\mathbb{E}_{f_{0,n}}[Z\mathbbm{1}\\{(\mathcal{A}\cap\mathcal{B})^{c}\\}]\\\
&\leq\mathbb{E}_{f_{0,n}}[Z\mathbbm{1}\\{\mathcal{A}\cap\mathcal{B}\\}]\\\
&\quad+\mathbb{E}_{f_{0,n}}[Z^{1+\gamma}]^{\frac{1}{1+\gamma}}\left(\mathbb{P}_{f_{0,n}}(\mathcal{A}^{c})+\mathbb{P}_{f_{0,n}}(\mathcal{B}^{c})\right)^{\frac{\gamma}{1+\gamma}}.\end{split}$
(13)

We aim to show that each term in (13) converges to zero.

First term in (13). We upper bound the first term of (13), which is
$\mathbb{E}_{f_{0,n}}[Z\mathbbm{1}\\{\mathcal{A}\cap\mathcal{B}\\}]$.

Using (9) and appealing to Lemma 1 with $s(h)=p^{\frac{k}{2}}||h||_{2}^{k}$,
we have that

$\begin{split}&Z=p^{\frac{k}{2}}\int_{\mathbb{R}^{p}}||h||_{2}^{k}\left|\pi_{n,\alpha}^{LAN}(h|X^{n})-\phi_{n}(h)\right|dh\\\
&\leq p^{\frac{k}{2}}\left[\sup_{g,h\in
B_{0}(r_{n})}f^{+}_{n}(g,h)\right]\int_{\mathbb{R}^{p}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh\\\
&\quad+p^{\frac{k}{2}}\left[\sup_{g,h\in
B_{0}(r_{n})}f_{n}^{-}(g,h)\right]\int_{\mathbb{R}^{p}}||h||_{2}^{k}\phi_{n}(h)dh\\\
&\quad+p^{\frac{k}{2}}\int_{||h||_{2}>r_{n}}||h||_{2}^{k}\left(\pi_{n,\alpha}^{LAN}(h|X^{n})+\phi_{n}(h)\right)dh.\end{split}$
(14)

Therefore, to study
$\mathbb{E}_{f_{0,n}}[Z\mathbbm{1}\\{\mathcal{A}\cap\mathcal{B}\\}]$, we
notice that the suprema in (14) are bounded. This gives the upper bound

$\begin{split}&\mathbb{E}_{f_{0,n}}[Z\mathbbm{1}\\{\mathcal{A}\cap\mathcal{B}\\}]\\\
&\leq
p^{\frac{k}{2}}\eta\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh\right]\\\
&\quad+p^{\frac{k}{2}}\eta\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k}\phi_{n}(h)dh\right]\\\
&\quad+p^{\frac{k}{2}}\mathbb{E}_{f_{0,n}}\left[\int_{||h||_{2}>r_{n}}||h||_{2}^{k}\phi_{n}(h)dh\right]\\\
&\quad+p^{\frac{k}{2}}\mathbb{E}_{f_{0,n}}\left[\int_{||h||_{2}>r_{n}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh\right].\end{split}$
(15)

Label the four terms in (15) as $T_{1}$, $T_{2}$, $T_{3}$, and $T_{4}$. We
argue that, for arbitrary $\epsilon>0$, each can be upper bounded by
$\epsilon/4$ (possibly for large enough $n$).

Consider term $T_{1}$ of (15). By (A3), there exists $M_{1}<\infty$ such that

$\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k}\,\pi_{n,\alpha}^{LAN}(h|X^{n})dh\right]<M_{1}.$

Thus, we can pick $\eta<\frac{\epsilon}{4M_{1}p^{k/2}}$ such that

$T_{1}=p^{k/2}\eta\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k}\,\pi_{n,\alpha}^{LAN}(h|X^{n})dh\right]<\frac{\epsilon}{4}.$

For term $T_{2}$, we notice that by a change of variables,

$\begin{split}&\int_{\mathbb{R}^{p}}\|h\|_{2}^{k}\phi_{n}(h)dh\\\
&=\int_{\mathbb{R}^{p}}n^{-\frac{p}{2}}\|h\|_{2}^{k}\phi\left(h\bigg{|}\sqrt{n}(\hat{\theta}^{ML}-\theta^{*}),\frac{V_{\theta^{*}}^{-1}}{\alpha}\right)dh\\\
&=\int_{\mathbb{R}^{p}}n^{\frac{k}{2}}\|\theta-\hat{\theta}^{ML}+\theta^{*}\|_{2}^{k}\phi\left(\theta\bigg{|}0,\frac{V_{\theta^{*}}^{-1}}{n\alpha}\right)d\theta\\\
&=n^{\frac{k}{2}}\mathbb{E}_{\mathcal{N}(0,V_{\theta^{*}}^{-1}/(n\alpha))}\left[\|\theta-\hat{\theta}^{ML}+\theta^{*}\|_{2}^{k}\right].\end{split}$
(16)

Next we upper bound the above using Hölder’s Inequality111Note that for $m>0$
and $v=(v_{1},\dots,v_{p})\in\mathbb{R}^{p}$, Hölder’s inequality implies
$(\sum_{j=1}^{p}|v_{j}|)^{m}\leq p^{m-1}\sum_{j=1}^{p}|v_{j}|^{m}$ and
centered absolute Gaussian moments:

$\begin{split}&n^{\frac{k}{2}}\mathbb{E}\left[\|\theta-\hat{\theta}^{ML}+\theta^{*}\|_{2}^{k}\right]\\\
&=n^{\frac{k}{2}}\mathbb{E}\left[\left(\sum_{i=1}^{p}(\theta_{i}-\hat{\theta}_{i}^{ML}+\theta_{i}^{*})^{2}\right)^{\frac{k}{2}}\right]\\\
&\leq
n^{\frac{k}{2}}p^{\frac{k}{2}-1}\sum_{i=1}^{p}\mathbb{E}\left[(\theta_{i}-\hat{\theta}_{i}^{ML}+\theta_{i}^{*})^{k}\right]\\\
&\leq
n^{\frac{k}{2}}p^{\frac{k}{2}-1}\sum_{i=1}^{p}\mathbb{E}\left[\lvert\theta_{i}-\hat{\theta}_{i}^{ML}+\theta_{i}^{*}\lvert^{k}\right]\\\
&\leq
n^{\frac{k}{2}}2^{k-1}p^{\frac{k}{2}-1}\sum_{i=1}^{p}\left(\mathbb{E}\left[\lvert\theta_{i}\lvert^{k}\right]+\lvert\hat{\theta}_{i}^{ML}-\theta_{i}^{*}\lvert^{k}\right).\end{split}$
(17)

The first term on the right side of (LABEL:eq:t2_eq2a) equals

$\begin{split}&n^{\frac{k}{2}}2^{k-1}p^{\frac{k}{2}-1}\sum_{i=1}^{p}\mathbb{E}[\lvert\theta_{i}\lvert^{k}]\\\
&\quad=\frac{2^{\frac{3k}{2}-1}p^{\frac{k}{2}-1}}{\sqrt{\pi}}\Gamma\left(\frac{k+1}{2}\right)\sum_{i=1}^{p}\left[\frac{[V_{\theta^{*}}^{-1}]_{ii}}{\alpha}\right]^{\frac{k}{2}},\end{split}$
(18)

and the second term on the right side of (LABEL:eq:t2_eq2a) equals
$2^{k-1}p^{\frac{k}{2}-1}\sum_{i=1}^{p}(\sqrt{n}\lvert\hat{\theta}_{i}^{ML}-\theta_{i}^{*}\lvert)^{k}$.
Because $\sqrt{n}(\hat{\theta}_{i}^{ML}-\theta_{i}^{*})$ is asymptotically
normal by Assumption (A0), there exists an $N_{1}\coloneqq N_{1}(p,k,M)$ such
that for all $n>N_{1}$, we have that the second term on the right side of
(LABEL:eq:t2_eq2a) is bounded by some $M<\infty$ with high probability. From
(LABEL:eq:t2_eq1) - (LABEL:eq:t2_eq2b) we have shown for all $n>N_{1}$,

$\begin{split}&\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}\|h\|_{2}^{k}\phi_{n}(h)dh\right]\leq
M_{2},\end{split}$ (19)

where

$\begin{split}&M_{2}\coloneqq\frac{2^{\frac{3k}{2}-1}p^{\frac{k}{2}-1}}{\sqrt{\pi}}\Gamma\left(\frac{k+1}{2}\right)\sum_{i=1}^{p}\left[\frac{[V_{\theta^{*}}^{-1}]_{ii}}{\alpha}\right]^{\frac{k}{2}}+M.\end{split}$

Putting this together, we have

$\begin{split}&T_{2}=p^{\frac{k}{2}}\eta\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}\|h\|_{2}^{k}\phi_{n}(h)dh\right]\leq
p^{\frac{k}{2}}\eta M_{2}.\end{split}$

Hence, by choosing $\eta<{\epsilon}/{(4M_{2}p^{\frac{k}{2}})}$ we have
$T_{2}\leq\epsilon/4$.

We control $T_{3}$ by leveraging (LABEL:eq:t2_eq3). Indeed, by Cauchy-Schwarz,

$\begin{split}&T_{3}=p^{\frac{k}{2}}\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}\|h\|_{2}^{k}(1-\mathbbm{1}\\{||h||_{2}\leq
r_{n}\\})\phi_{n}(h)dh\right]\\\ &\leq
p^{\frac{k}{2}}\mathbb{E}_{f_{0,n}}\Bigg{[}\sqrt{\int_{\mathbb{R}^{p}}\|h\|_{2}^{2k}\phi_{n}(h)dh}\\\
&\qquad\quad\times\sqrt{\int_{\mathbb{R}^{p}}(1-\mathbbm{1}\\{||h||_{2}\leq
r_{n}\\})\phi_{n}(h)dh}\,\Bigg{]}\\\ &\leq
p^{\frac{k}{2}}\sqrt{\mathbb{E}_{f_{0,n}}\Bigg{[}\int_{\mathbb{R}^{p}}\|h\|_{2}^{2k}\phi_{n}(h)dh\Bigg{]}}\\\
&\qquad\quad\times\sqrt{\mathbb{E}_{f_{0,n}}\Bigg{[}\int_{\mathbb{R}^{p}}(1-\mathbbm{1}\\{||h||_{2}\leq
r_{n}\\})\phi_{n}(h)dh\Bigg{]}}.\end{split}$ (20)

By appealing to (LABEL:eq:t2_eq3) with the power $2k$ instead of $k$, we know
that
$\mathbb{E}_{f_{0,n}}[\int_{\mathbb{R}^{p}}\|h\|_{2}^{2k}\phi_{n}(h)dh]\leq\tilde{M}_{2}$
for $n>N_{1}$.

Because $r_{n}\rightarrow\infty$ and
$\phi_{n}(h)\leq(2\pi)^{-p/2}|V_{\theta^{*}}^{-1}/\alpha|^{-1/2}$ for all
$h\in\mathbb{R}^{p}$, we see that $(1-\mathbbm{1}\\{||h||_{2}\leq
r_{n}\\})\phi_{n}(h)\rightarrow 0$. Thus, by the bounded convergence theorem,
there exists $N_{2}\coloneqq N_{2}(\epsilon,\tilde{M}_{2},p,k)$ such that for
all $n>N_{2}$,

$\mathbb{E}_{f_{0,n}}\Bigg{[}\int_{\mathbb{R}^{p}}(1-\mathbbm{1}\\{||h||_{2}\leq
r_{n}\\})\phi_{n}(h)dh\Bigg{]}<\frac{\epsilon^{2}}{16\tilde{M}_{2}p^{k}}.$

Thus, for $n>\max(N_{1},N_{2})$, we have that $T_{3}<\frac{\epsilon}{4}$.

By appealing to Lemma 2 (assumptions are met by appealing to (A3)) with
$f_{Z}(z)$ being $\pi_{n,\alpha}^{LAN}(z|X^{n})$, we pick $N_{3}\coloneqq
N_{3}(p,k,\epsilon,\gamma)$ such that for all $n>N_{3}$,

$\mathbb{E}_{f_{0,n}}\left[\int_{||h||_{2}>r_{n}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh\right]<\frac{\epsilon}{4p^{k/2}}.$

Thus,

$T_{4}=p^{\frac{k}{2}}\mathbb{E}_{f_{0,n}}\left[\int_{||h||_{2}>r_{n}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh\right]<\frac{\epsilon}{4}$

for $n>N_{3}$. Therefore, we have shown that the first term of (13) is
upperbounded by $\epsilon$ whenever
$\eta<\frac{\epsilon}{4\min\\{M_{1},M_{2}\\}p^{\frac{k}{2}}}$ and
$n\geq\max\\{N_{1},N_{2},N_{3}\\}$.

Second term in (13). First, we consider upper bounding the term
$\mathbb{E}_{f_{0,n}}[Z^{1+\gamma}]$. Notice that by the fact that
$|\pi_{n,\alpha}^{LAN}(h|X^{n})-\phi_{n}(h)|\leq\pi_{n,\alpha}^{LAN}(h|X^{n})+\phi_{n}(h)$,

$\begin{split}&Z^{1+\gamma}=\left[p^{\frac{k}{2}}\int_{\mathbb{R}^{p}}||h||_{2}^{k}\left|\pi_{n,\alpha}^{LAN}(h|X^{n})-\phi_{n}(h)\right|dh\right]^{1+\gamma}\\\
&\leq\Bigg{[}2p^{\frac{k}{2}}\max\bigg{\\{}\int_{\mathbb{R}^{p}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh,\\\
&\hskip
100.0pt\int_{\mathbb{R}^{p}}||h||_{2}^{k}\phi_{n}(h)dh\bigg{\\}}\Bigg{]}^{1+\gamma},\end{split}$

Next, applying Jensen’s Inequality, we have that

$\begin{split}&\Bigg{[}\max\bigg{\\{}\int_{\mathbb{R}^{p}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh,\int_{\mathbb{R}^{p}}||h||_{2}^{k}\phi_{n}(h)dh\bigg{\\}}\Bigg{]}^{1+\gamma}\\\
&=\max\Bigg{\\{}\bigg{(}\int_{\mathbb{R}^{p}}||h||_{2}^{k}\pi_{n,\alpha}^{LAN}(h|X^{n})dh\bigg{)}^{1+\gamma},\\\
&\hskip
100.0pt\bigg{(}\int_{\mathbb{R}^{p}}||h||_{2}^{k}\phi_{n}(h)dh\bigg{)}^{1+\gamma}\Bigg{\\}}\\\
&\leq\max\Bigg{\\{}\int_{\mathbb{R}^{p}}||h||_{2}^{k(1+\gamma)}\pi_{n,\alpha}^{LAN}(h|X^{n})dh,\\\
&\hskip
100.0pt\int_{\mathbb{R}^{p}}||h||_{2}^{k(1+\gamma)}\phi_{n}(h)dh\Bigg{\\}}\\\
&\leq\int_{\mathbb{R}^{p}}||h||_{2}^{k(1+\gamma)}(\pi_{n,\alpha}^{LAN}(h|X^{n})+\phi_{n}(h))dh.\end{split}$

Therefore, we have the bound

$\begin{split}&\mathbb{E}_{f_{0,n}}[Z^{1+\gamma}]\leq(2p^{\frac{k}{2}})^{1+\gamma}\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k(1+\gamma)}\phi_{n}(h)dh\right]\\\
&\hskip
20.0pt+(2p^{\frac{k}{2}})^{1+\gamma}\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k(1+\gamma)}\pi_{n,\alpha}^{LAN}(h|X^{n})dh\right].\end{split}$

We argue that the two terms on the right side of the above are upper bounded
by constants. By Assumption (A3), there exists an $M_{3}<\infty$ such that
$\mathbb{E}_{f_{0,n}}[\int_{\mathbb{R}^{p}}||h||_{2}^{k(1+\gamma)}\pi_{n,\alpha}^{LAN}(h|X^{n})dh]<M_{3}$.
By appealing to (LABEL:eq:t2_eq3) with $k(1+\gamma)$ instead of $k$, we know
that for $n>N_{1}$,

$\mathbb{E}_{f_{0,n}}\left[\int_{\mathbb{R}^{p}}||h||_{2}^{k(1+\gamma)}\phi_{n}(h)dh\right]<M_{4}.$

Thus, we have shown that

$\begin{split}\mathbb{E}_{f_{0,n}}[Z^{1+\gamma}]&\leq(2p^{k/2})^{1+\gamma}(M_{3}+M_{4})\\\
&\leq 2(2p^{k/2})^{1+\gamma}\max\\{M_{3},M_{4}\\}.\end{split}$

We finally upper bound $\mathbb{P}(\mathcal{A}^{c})$ and
$\mathbb{P}(\mathcal{B}^{c})$. First, note that $f_{n}^{+}(g,h)$ is the same
as $f_{n}(g,h)$ from [2]. Next, we use the fact that
$\left\\{1-x\right\\}^{+}=\left\\{x-1\right\\}^{-}$ to see

$\begin{split}f_{n}^{-}(g,h)&=\left\\{\frac{\pi_{n,\alpha}^{LAN}(h|X^{n})\phi_{n}(g)}{\phi_{n}(h)\pi_{n,\alpha}^{LAN}(g|X^{n})}-1\right\\}^{-}\\\
&=\left\\{1-\frac{\pi_{n,\alpha}^{LAN}(h|X^{n})\phi_{n}(g)}{\phi_{n}(h)\pi_{n,\alpha}^{LAN}(g|X^{n})}\right\\}^{+}=f_{n}^{+}(h,g).\end{split}$
(21)

Taking the supremum in $g,h$ over $B_{0}(r_{n})$ of both sides of 21 gives us
that

$\sup_{g,h\in B_{0}(r_{n})}f_{n}^{-}(g,h)=\sup_{g,h\in
B_{0}(r_{n})}f_{n}^{+}(h,g).$ (22)

Thus, there exists $N_{4}\coloneqq
N_{4}(\eta,\epsilon,M_{3},M_{4},\gamma,p,k)$ such that
$\mathbb{P}(\mathcal{A}^{c})=\mathbb{P}(\mathcal{B}^{c})$ is sufficiently
small for all $n>N_{4}$ by [2, Lemma 5]:

$\begin{split}\mathbb{P}(\mathcal{A}^{c})=\mathbb{P}(\mathcal{B}^{c})&<\frac{\epsilon^{\frac{1+\gamma}{\gamma}}}{2^{1+\frac{2+\gamma}{\gamma}}p^{\left(\frac{k}{2}\right)\left(\frac{1+\gamma}{\gamma}\right)}\max\\{M_{3},M_{4}\\}^{\frac{1}{\gamma}}}.\end{split}$

Putting this together, we find that for $n>N_{4}$,

$\displaystyle\mathbb{E}_{f_{0,n}}[Z^{1+\gamma}]^{\frac{1}{1+\gamma}}\left(\mathbb{P}_{f_{0,n}}(\mathcal{A}^{c})+\mathbb{P}_{f_{0,n}}(\mathcal{B}^{c})\right)^{\frac{\gamma}{1+\gamma}}$
$\displaystyle\leq[2(2p^{k/2})^{1+\gamma}\max\\{M_{3},M_{4}\\}]^{\frac{1}{1+\gamma}}$
$\displaystyle\quad\times\left[\frac{\epsilon^{\frac{1+\gamma}{\gamma}}}{2^{\frac{2+\gamma}{\gamma}}p^{\left(\frac{k}{2}\right)\left(\frac{1+\gamma}{\gamma}\right)}\max\\{M_{3},M_{4}\\}^{\frac{1}{\gamma}}}\right]^{\frac{\gamma}{1+\gamma}}=\epsilon.$

Final bound. We have shown that whenever $\eta$ is chosen small enough, for
all $n>\max(N_{1},N_{2},N_{3},N_{4})$, for arbitrary $\epsilon>0$,

$\begin{split}\mathbb{E}[Z]&=p^{k/2}\mathbb{E}\Bigg{[}\int_{\mathbb{R}^{p}}||h||_{2}^{k}\left|\pi_{n,\alpha}^{LAN}(h|X^{n})-\phi_{n}(h)\right|dh\Bigg{]}\\\
&<2\epsilon.\end{split}$

This gives us the desired result.

###  IV-B Proof of Theorem 2

Recall,
$\hat{\theta}^{B}=\int_{\mathbb{R}^{p}}\theta\pi_{n,\alpha}(\theta|X^{n})d\theta$.
Using the change of variable $h=\sqrt{n}(\theta-\theta^{*})$,

$\begin{split}&\hat{\theta}^{B}=\int\theta\pi_{n,\alpha}(\theta|X^{n})d\theta=\frac{1}{\sqrt{n}}\int
h\pi_{n,\alpha}^{LAN}(h)dh+\theta^{*}.\end{split}$

In the above, we used the definition
$\pi_{n,\alpha}^{LAN}(h)=n^{-\frac{p}{2}}\pi_{n,\alpha}(\theta^{*}+\frac{h}{\sqrt{n}}|X^{n})$
in (6). Recalling the definition of $\phi_{n}(h)$ in (6), we can write,

$\begin{split}&\sqrt{n}(\hat{\theta}^{B}-\theta^{*})=\int
h\pi_{n,\alpha}^{LAN}(h)dh\\\ &=\int
h(\pi_{n,\alpha}^{LAN}(h)-\phi_{n}(h))dh+\int h\phi_{n}(h)dh\\\ &=\int
h(\pi_{n,\alpha}^{LAN}(h)-\phi_{n}(h))dh+\sqrt{n}(\hat{\theta}^{ML}-\theta^{*}).\end{split}$
(23)

The second term of 23 converges weakly to $\mathcal{N}(0,V_{\theta^{*}}^{-1})$
by Assumption (A0). It remains to show that the first term of the last line of
23 is $o_{p}(1)$. This follows from

$\begin{split}&\left|\left|\int
h(\pi_{n,\alpha}^{LAN}(h)-\phi_{n}(h))dh\right|\right|_{1}\\\
&\leq\int\left|\left|h\right|\right|_{1}\left|\pi_{n,\alpha}^{LAN}(h)-\phi_{n}(h)\right|dh=o_{p}(1)\end{split}$

where the equality follows from Theorem 1 for $k=1$.

##  V Technical Lemmas

######  Lemma 1.

Consider sequences of densities $\varphi_{n}$ and $\psi_{n}$. For a given
compact set $K\subset\mathbb{R}^{p}$, suppose that the densities $\varphi_{n}$
and $\psi_{n}$ are positive on $K$. Then for any function, $s(h)$, that is
nonnegative on all of $\mathbb{R}^{p}$,

$\displaystyle\int_{\mathbb{R}^{p}}s(h)\left|\varphi_{n}(h)-\psi_{n}(h)\right|dh$
$\displaystyle\leq\left[\sup_{g,h\in
K}\tilde{f}^{+}_{n}(g,h)\right]\int_{\mathbb{R}^{p}}s(h)\psi_{n}(h)dh$
$\displaystyle\quad+\left[\sup_{g,h\in
K}\tilde{f}_{n}^{-}(g,h)\right]\int_{\mathbb{R}^{p}}s(h)\varphi_{n}(h)dh$
$\displaystyle\quad+\int_{\mathbb{R}^{p}\setminus
K}s(h)\left(\psi_{n}(h)+\varphi_{n}(h)\right)dh,$

where

$\begin{split}\tilde{f}_{n}^{+}(g,h)\coloneqq\left\\{1-\frac{\varphi_{n}(h)\psi_{n}(g)}{\psi_{n}(h)\varphi_{n}(g)}\right\\}^{+},\\\
\tilde{f}_{n}^{-}(g,h)\coloneqq\left\\{\frac{\psi_{n}(h)\varphi_{n}(g)}{\varphi_{n}(h)\psi_{n}(g)}-1\right\\}^{-}.\end{split}$

######  Proof.

Since $s(h)$, $\psi_{n}(h)$, and $\varphi_{n}(h)$ are nonnegative on all of
$\mathbb{R}^{p}$, we have that $s(h)|\psi_{n}(h)-\varphi_{n}(h)|\leq
s(h)(\psi_{n}(h)+\varphi_{n}(h))$. Using this fact and that
$|\psi_{n}(h)-\varphi_{n}(h)|=\\{\psi_{n}(h)-\varphi_{n}(h)\\}^{-}+\\{\psi_{n}(h)-\varphi_{n}(h)\\}^{+}$,

$\begin{split}&\int_{\mathbb{R}^{p}}s(h)|\psi_{n}(h)-\varphi_{n}(h)|dh\\\
&=\int_{K}s(h)|\psi_{n}(h)-\varphi_{n}(h)|dh\\\
&\quad+\int_{\mathbb{R}^{p}\setminus K}s(h)|\psi_{n}(h)-\varphi_{n}(h)|dh\\\
&\leq\int_{K}s(h)\\{\psi_{n}(h)-\varphi_{n}(h)\\}^{+}dh\\\
&\quad+\int_{K}s(h)\\{\psi_{n}(h)-\varphi_{n}(h)\\}^{-}dh\\\
&\quad+\int_{\mathbb{R}^{p}\setminus
K}s(h)(\psi_{n}(h)+\varphi_{n}(h))dh.\end{split}$ (24)

We bound the first two terms on the right side of (24) separately. Define
$a_{n}=(\int_{K}\psi_{n}(g)dg)^{-1}$ and
$b_{n}=(\int_{K}\varphi_{n}(g)dg)^{-1}$ and assume $a_{n}\geq b_{n}$ without
loss of generality. For the first term in (24), for all $h\in K$,

$\frac{\varphi_{n}(h)}{\psi_{n}(h)}=\frac{a_{n}}{b_{n}}\int_{K}\frac{\varphi_{n}(h)}{\psi_{n}(h)}\frac{\psi_{n}(g)}{\varphi_{n}(g)}b_{n}\varphi_{n}(g)dg.$

Thus,

$\begin{split}&\left\\{\psi_{n}(h)-\varphi_{n}(h)\right\\}^{+}=\left\\{1-\frac{\varphi_{n}(h)}{\psi_{n}(h)}\right\\}^{+}\psi_{n}(h)\\\
&=\left\\{1-\frac{a_{n}}{b_{n}}\int_{K}\frac{\varphi_{n}(h)}{\psi_{n}(h)}\frac{\psi_{n}(g)}{\varphi_{n}(g)}b_{n}\varphi_{n}(g)dg\right\\}^{+}\psi_{n}(h).\end{split}$
(25)

Applying Jensen’s inequality on the convex function $f(x)=\\{1-x\\}^{+}$, we
have $\\{1-\mathbb{E}[X]\\}^{+}\leq\mathbb{E}[\\{1-X\\}^{+}]$. Applying this
to (25),

$\begin{split}&\left\\{1-\frac{a_{n}}{b_{n}}\int_{K}\frac{\varphi_{n}(h)}{\psi_{n}(h)}\frac{\psi_{n}(g)}{\varphi_{n}(g)}b_{n}\varphi_{n}(g)dg\right\\}^{+}\\\
&\leq\int_{K}\left\\{1-\frac{a_{n}}{b_{n}}\frac{\varphi_{n}(h)}{\psi_{n}(h)}\frac{\psi_{n}(g)}{\varphi_{n}(g)}\right\\}^{+}b_{n}\varphi_{n}(g)dg\\\
&=\int_{K}\tilde{f}_{n}^{+}(g,h)b_{n}\varphi_{n}(g)dg.\end{split}$ (26)

The final step above uses that when $a_{n}/b_{n}\geq 1$, we have
$\\{1-(a_{n}/b_{n})x\\}^{+}\leq\\{1-x\\}^{+}$ for $x\geq 0$. Using (25)-(26)
we can rewrite the first term of (24) as

$\displaystyle\int_{K}s(h)\\{\psi_{n}(h)-\varphi_{n}(h)\\}^{+}dh$
$\displaystyle\leq\int_{K}\int_{K}s(h)\tilde{f}^{+}_{n}(g,h)b_{n}\varphi_{n}(g)\psi_{n}(h)dgdh$
$\displaystyle\leq\left[\sup_{g,h\in
K}\tilde{f}^{+}_{n}(g,h)\right]\int_{K}s(h)\psi_{n}(h)\left[\int_{K}b_{n}\varphi_{n}(g)dg\right]dh$
$\displaystyle\leq\left[\sup_{g,h\in
K}\tilde{f}^{+}_{n}(g,h)\right]\int_{K}s(h)\psi_{n}(h)dh,$
$\displaystyle\leq\left[\sup_{g,h\in
K}\tilde{f}^{+}_{n}(g,h)\right]\int_{\mathbb{R}^{p}}s(h)\psi_{n}(h)dh.$

We turn to the second term of (24) and write

$\begin{split}&\big{\\{}\psi_{n}(h)-\varphi_{n}(h)\big{\\}}^{-}=\left\\{\frac{\psi_{n}(h)}{\varphi_{n}(h)}-1\right\\}^{-}\varphi_{n}(h)\\\
&=\left\\{\frac{a_{n}}{b_{n}}\int_{K}\frac{\psi_{n}(h)\varphi_{n}(g)}{\varphi_{n}(h)\psi_{n}(g)}b_{n}\psi_{n}(g)dg-1\right\\}^{-}\varphi_{n}(h).\end{split}$

Applying Jensen’s inequality to the convex function $f(x)=\\{x-1\\}^{-}$, we
have that $\\{\mathbb{E}[X]-1\\}^{-}\leq\mathbb{E}[\\{X-1\\}^{-}]$ and

$\begin{split}&\left\\{\frac{a_{n}}{b_{n}}\int_{K}\frac{\psi_{n}(h)\varphi_{n}(g)}{\varphi_{n}(h)\psi_{n}(g)}b_{n}\psi_{n}(g)dg-1\right\\}^{-}\\\
&\leq\int_{K}\left\\{\frac{a_{n}\psi_{n}(h)\varphi_{n}(g)}{b_{n}\varphi_{n}(h)\psi_{n}(g)}-1\right\\}^{-}b_{n}\psi_{n}(g)dg\\\
&\overset{(a)}{\leq}\int_{K}\left\\{\frac{\psi_{n}(h)\varphi_{n}(g)}{\varphi_{n}(h)\psi_{n}(g)}-1\right\\}^{-}b_{n}\psi_{n}(g)dg\\\
&=\int_{K}\tilde{f}_{n}^{-}(g,h)b_{n}\psi_{n}(g)dg\overset{(b)}{\leq}\int_{K}\tilde{f}_{n}^{-}(g,h)a_{n}\psi_{n}(g)dg.\end{split}$

Step $(a)$ uses that $\\{(a_{n}/b_{n})x-1\\}^{-}\leq\\{x-1\\}^{-}$ for $x\geq
0$ when $a_{n}/b_{n}\geq 1$ and step $(b)$ that $b_{n}\leq a_{n}$. Hence, we
can rewrite the second term of (24) as

$\begin{split}&\int_{K}s(h)\\{\psi_{n}(h)-\varphi_{n}(h)\\}^{-}dh\\\
&\leq\int_{K}\int_{K}s(h)\tilde{f}_{n}^{-}(g,h)a_{n}\psi_{n}(g)\varphi_{n}(h)dgdh\\\
&\leq\left[\sup_{g,h\in
K}\tilde{f}_{n}^{-}(g,h)\right]\int_{K}s(h)\varphi_{n}(h)\left[\int_{K}a_{n}\psi_{n}(g)dg\right]dh.\end{split}$

This establishes the the final bound since

$\begin{split}&\int_{K}s(h)\varphi_{n}(h)\left[\int_{K}a_{n}\psi_{n}(g)dg\right]dh\\\
&=\int_{K}s(h)\varphi_{n}(h)dh\leq\int_{\mathbb{R}^{p}}s(h)\varphi_{n}(h)dh.\end{split}$

######  Lemma 2.

Suppose a random variable $Z\overset{d}{\coloneqq}Y|X^{n}$ has a density
$f_{Z}(\cdot)$ on $\mathbb{R}^{p}$. Assume there exists a $\gamma>0$ such that
$\mathbb{E}_{f_{0,n}}[\mathbb{E}[||Z||_{2}^{k(1+\gamma)}]]<\infty$. Then, for
any $\epsilon>0$ and $r_{n}\rightarrow\infty$, there exists an integer
$N(\epsilon,\gamma,k)>0$ such that for all $n>N(\epsilon,\gamma,k)$,

$\displaystyle\mathbb{E}_{f_{0,n}}\left[\int_{||z||_{2}>r_{n}}||z||_{2}^{k}\,f_{Z}(z)dz\right]<\epsilon,$
(27)

######  Proof.

For an arbitrary sequence $r_{n}\rightarrow\infty$, we choose $N_{0}$ large
enough that $r_{n}>0$ for all $n>N_{0}$. Let
$z=(z_{1},\ldots,z_{p})\in\mathbb{R}^{p}$ and let $f_{||z||_{2}^{k}}(\cdot)$
denote the density function of $||z||_{2}^{k}$. Define the random variable

$\begin{split}V&\coloneqq||Z||_{2}^{k}\mathbbm{1}\left\\{||Z||_{2}>r_{n}\right\\}=\begin{cases}0&\textrm{if
}||Z||_{2}\leq r_{n},\\\ ||Z||_{2}^{k}&\textrm{if
}||Z||_{2}>r_{n}.\end{cases}\end{split}$

Thus, the density of $V$ is given by

$f_{V}(v)=\begin{cases}\mathbb{P}(||Z||_{2}\leq r_{n})&\textrm{ if }v=0,\\\
f_{||z||_{2}^{k}}(v)&\textrm{ if }v>r_{n}.\end{cases}$

Next, notice that

$\begin{split}\mathbb{E}[V]&=\int_{0}^{r_{n}}\mathbb{P}(V>t)dt+\int_{r_{n}}^{\infty}\mathbb{P}(V>t)dt.\end{split}$
(28)

Using the definition of $V$ and its density function, we study (28). Consider
the first term on the right side,

$\begin{split}\int_{0}^{r_{n}}\mathbb{P}(V>t)dt&=\int_{0}^{r_{n}}\int_{t}^{\infty}f_{V}(v)dvdt\\\
&\overset{(a)}{=}\int_{0}^{r_{n}}\mathbb{P}(||Z||_{2}^{k}>r_{n})dt\\\
&=r_{n}\mathbb{P}(||Z||_{2}^{k}>r_{n})\overset{(b)}{\leq}\frac{\mathbb{E}[||Z||_{2}^{k(1+\gamma)}]}{r_{n}^{\gamma}}.\end{split}$
(29)

Step $(a)$ in (29) follows because, for $0<t<r_{n}$,

$\int_{t}^{\infty}f_{V}(v)dv=\int_{r_{n}}^{\infty}f_{||z||_{2}^{k}}(v)dv=\mathbb{P}(||Z||_{2}^{k}>r_{n}),$

and step $(b)$ by Markov’s inequality.

For the second term on the right side of (28), we again use Markov’s
Inequality as

$\begin{split}\int_{r_{n}}^{\infty}\mathbb{P}(V>t)dt&\leq\int_{r_{n}}^{\infty}\frac{\mathbb{E}[V^{1+\gamma}]}{t^{1+\gamma}}dt\\\
&\overset{(c)}{\leq}\int_{r_{n}}^{\infty}\frac{\mathbb{E}[||Z||_{2}^{k(1+\gamma)}]}{t^{1+\gamma}}dt\\\
&=\frac{\mathbb{E}[||Z||_{2}^{k(1+\gamma)}]}{\gamma
r_{n}^{\gamma}}.\end{split}$ (30)

Step $(c)$ in (30) follows because

$\begin{split}&V^{1+\gamma}=\left(||Z||_{2}^{k}\mathbbm{1}\left\\{||Z||_{2}>r_{n}\right\\}\right)^{1+\gamma}\leq||Z||_{2}^{k(1+\gamma)}.\end{split}$

Plugging (29) and (30) into (28), we find

$\begin{split}\mathbb{E}[V]&\leq\frac{\gamma+1}{\gamma
r_{n}^{\gamma}}\mathbb{E}\left[||Z||_{2}^{k(1+\gamma)}\right].\end{split}$
(31)

Taking the expectation with respect to $X^{n}$ in (31) gives,

$\begin{split}\mathbb{E}_{f_{0,n}}\left[\mathbb{E}[V]\right]&=\mathbb{E}_{f_{0,n}}\left[\mathbb{E}[||Z||_{2}^{k}\mathbbm{1}\left\\{||Z||_{2}>r_{n}\right\\}]\right]\\\
&\leq\frac{\gamma+1}{\gamma
r_{n}^{\gamma}}\mathbb{E}_{f_{0,n}}\left[\mathbb{E}[||Z||_{2}^{k(1+\gamma)}]\right]<\epsilon,\end{split}$
(32)

where the final bound in the equations above follows since
$\mathbb{E}_{f_{0,n}}\left[\mathbb{E}[||Z||_{2}^{k(1+\gamma)}]\right]<\infty$
by assumption and $r_{n}\rightarrow\infty$. Thus, there exists
$N(\epsilon,\gamma,k)\geq N_{0}$ such that the bound is true for all
$n>N(\epsilon,\gamma,k)$, which proves (27).

## Acknowledgment

This research was partially supported by the NSF grant DMS-231097 (M. Avella
Medina).

## References

  * [1] Pierre Alquier and James Ridgway  “Concentration of tempered posteriors and of their variational approximations”  In _Annals of Statistics_ 48.3, 2020, pp. 1475–1497
  * [2] Marco Avella Medina, José Luis Montiel Olea, Cynthia Rush and Amilcar Velez  “On the robustness to misspecification of $\alpha$-posteriors and their variational approximations”  In _Journal of Machine Learning Research_ 23.127, 2022, pp. 1–51
  * [3] Anirban Bhattacharya, Debdeep Pati and Yun Yang  “Bayesian fractional posteriors”  In _Annals of Statistics_ 47.1, 2019, pp. 39–66
  * [4] Subhashis Ghosal and Aad Van der Vaart  “Fundamentals of Nonparametric Bayesian Inference”  Cambridge University Press, 2017
  * [5] Peter Grünwald  “The safe Bayesian: learning the learning rate via the mixability gap”  In _International Conference on Algorithmic Learning Theory_ 22, 2012
  * [6] Peter Grünwald and Thijs Van Ommen  “Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it”  In _Bayesian Analysis_ 12.4, 2017, pp. 1069–1103
  * [7] B.J.K. Kleijn and A.W. Vaart  “The Bernstein-Von-Mises theorem under misspecification”  In _Electronic Journal of Statistics_ 6, 2012, pp. 354–381
  * [8] Erich .L. Lehmann and George Casella  “Theory of Point Estimation”  New York, NY: Springer, 2011
  * [9] David A. McAllester  “PAC-Bayesian stochastic model selection”  In _Machine Learning_ 51.1, 2003, pp. 5–21
  * [10] Jeffrey W. Miller  “Asymptotic normality, concentration, and coverage of Generalized Posteriors”  In _Journal of Machine Learning Research_ 22.168, 2021, pp. 1–53
  * [11] Jeffrey W. Miller and David B. Dunson  “Robust Bayesian inference via coarsening”  In _Journal of the American Statistical Association_ 114.527, 2019, pp. 1113–1125
  * [12] Stephen Walker and Nils Lid Hjort  “On Bayesian consistency”  In _Journal of the Royal Statistical Society: Series B_ 63.4, 2001, pp. 811–821
  * [13] Tong Zhang  “From $\epsilon$-entropy to KL-entropy: analysis of minimum information complexity density estimation”  In _Annals of Statistics_ 34.5, 2006, pp. 2180–2210