# Optimal Eigenvalue Shrinkage in the Semicircle Limit

David L. Donoho and Michael J. Feldman
Department of Statistics, Stanford University

###### Abstract

Recent studies of high-dimensional covariance estimation often assume the
proportional growth asymptotic, where the sample size $n$ and dimension $p$
are comparable, with $n,p\rightarrow\infty$ and $\gamma_{n}\equiv
p/n\rightarrow\gamma>0$. Yet, many datasets—perhaps most—have very different
numbers of rows and columns. Consider instead disproportional growth, where
$n,p\rightarrow\infty$ and $\gamma_{n}\rightarrow 0$ or
$\gamma_{n}\rightarrow\infty$. With far fewer dimensions than observations,
the disproportional limit $\gamma_{n}\rightarrow 0$ may seem similar to
classical fixed-$p$ asymptotics. In fact, either disproportional limit induces
novel phenomena distinct from the proportional and fixed-$p$ limits.

We study the spiked covariance model, with population covariance a low-rank
perturbation of the identity. For each of 15 different loss functions, and
each disproportional limit, we exhibit in closed form new optimal shrinkage
and thresholding rules; optimality takes the particularly strong form of
unique asymptotic admissibility. Readers who initially view the
disproportionate limit $\gamma_{n}\rightarrow 0$ as similar to classical
fixed-$p$ asymptotics may expect, given the dominance in that setting of the
sample covariance estimator, that there is no alternative here. On the
contrary, although the sample covariance is consistent as
$\gamma_{n}\rightarrow 0$, our optimal procedures demand extensive eigenvalue
shrinkage and offer substantial performance benefits. The sample covariance is
similarly improvable in the disproportionate limit
$\gamma_{n}\rightarrow\infty$.

Practitioners may worry how to choose between proportional and disproportional
growth frameworks in practice. Conveniently, under the spiked covariance model
there is no conflict between the two and no choice is needed; one unified set
of closed forms (used with the aspect ratio $\gamma_{n}$ of the practitioner’s
data) offers full asymptotic optimality in both regimes.

At the heart of these phenomena is the spiked Wigner model, in which we seek
to recover a low-rank matrix perturbed by symmetric noise. The eigenvalue
distributions of the spiked covariance under disproportionate growth
(appropriately scaled) and the spiked Wigner converge to a common limit—the
semicircle law. Exploiting this connection, we derive optimal performance
levels and eigenvalue shrinkage formulas for the spiked Wigner setting, of
independent and fundamental interest. These formulas visibly correspond to our
formulas for optimal shrinkage in covariance estimation.

##  1 Introduction

Suppose we observe $p$-dimensional Gaussian vectors
$x_{1},\ldots,x_{n}\stackrel{{\scriptstyle
i.i.d.}}{{\sim}}\mathcal{N}(0,\Sigma)$, with $\Sigma=\Sigma_{p}$ the
underlying $p$-by-$p$ population covariance matrix. Traditionally, to estimate
$\Sigma$, we form the empirical (sample) covariance matrix
$S=S_{n}=\frac{1}{n}\sum_{i=1}^{n}x_{i}x_{i}^{\prime}$; this is the maximum
likelihood estimator. Under the classical asymptotic framework that
statisticians have used for centuries, where $p$ is fixed and
$n\rightarrow\infty$, $S$ is a consistent estimator of $\Sigma$ (under any
matrix norm).

In recent decades, many impressive random matrix-theoretic studies consider
$p=p_{n}$ tending to infinity with $n$. Generally, these studies focus on
proportional growth, where the sample size and dimension are comparable:

$\displaystyle
n,p\rightarrow\infty\,,\qquad\gamma_{n}=\frac{p}{n}\rightarrow\gamma>0\,.$
(1.1)

Under this framework, certain beautiful and striking mathematical phenomena
are elegantly brought to light. A striking deliverable for statisticians,
particularly, is the discovery that in such a high-dimensional setting, the
maximum likelihood estimator $S$ is an inconsistent estimator of $\Sigma$
(under various matrix norms).

###  1.1 Standard Covariance Estimation in the Proportional Regime

Inconsistency of $S$ under proportional growth stems from the following
phenomena, not present in classical fixed-$p$ statistics. These results are
due to Marchenko and Pastur [25], Baik, Ben Arous, and Péché [5], Baik and
Silverstein [4], and Paul [28].

  1. 1.

Eigenvalue spreading. Assume proportional growth with $\gamma\in(0,1]$. In the
standard normal case $\Sigma=I_{p}$, the $p$-dimensional identity matrix, the
spectral measure of $S$ converges weakly almost surely to the Marchenko-Pastur
distribution with parameter $\gamma$. This distribution, or bulk, is non-
degenerate, absolutely continuous, and has support
$[(1-\sqrt{\gamma})^{2},(1+\sqrt{\gamma})^{2}]=[\lambda_{-}(\gamma),\lambda_{+}(\gamma)]$.

Intuitively, sample eigenvalues, rather than concentrating near population
eigenvalues (which, in the identity case, are all simply $1$), spread out
across a fixed-size interval, preventing consistency of $S$ for $\Sigma$.

  2. 2.

Eigenvalue inflation. We can quantify asymptotic bias in extreme eigenvalues
more precisely. Consider Johnstone’s spiked covariance model, where all but
finitely many eigenvalues $\ell_{1},\ldots,\ell_{p}$ of $\Sigma$ are identity:

$\displaystyle[\textbf{I}]\hskip
34.14322pt\ell_{1}>\cdots>\ell_{r}>1\,,\qquad\ell_{r+1}=\cdots=\ell_{p}=1\,.$

Under this model, the number of so-called “spiked” eigenvalues $r$ is fixed
and independent of $n$. Spiked eigenvalues are constant.

Let $\lambda_{i}=\lambda_{i,n}$ denote the eigenvalues of $S$, ordered
decreasingly $\lambda_{1}\geq\cdots\geq\lambda_{p}$. As it turns out,
$\lambda_{1},\ldots,\lambda_{r}$—the “leading” eigenvalues—do not converge to
their population counterparts $\ell_{1},\ldots,\ell_{r}$; rather, they are
shifted upwards. Under (1.1), for each fixed $i\geq 1$,

$\displaystyle\phantom{\,.}\lambda_{i}\xrightarrow{a.s.}\lambda(\ell_{i})\,,$
(1.2)

where $\lambda(\ell)\equiv\lambda(\ell,\gamma)$ is the “eigenvalue mapping”
function, given piecewise by

$\displaystyle\lambda(\ell)=\begin{dcases}\ell+\frac{\gamma\ell}{\ell-1}&\ell>1+\sqrt{\gamma}\\\
(1+\sqrt{\gamma})^{2}&\ell\leq 1+\sqrt{\gamma}\end{dcases}\,.$ (1.3)

The transition point $\ell_{+}(\gamma)\equiv 1+\sqrt{\gamma}$ between the two
behaviors is known as the Baik-Ben Arous-Péché (BBP) transition. Below the
transition, $1<\ell\leq\ell_{+}(\gamma)$, “weak signal” leads to a limiting
eigenvalue independent of $\ell$. For fixed $i$ such that
$\ell_{i}\leq\ell_{+}(\gamma)$, $\lambda_{i}$ tends to $\lambda_{+}(\gamma)$,
the upper-bulk-edge of the Marchenko-Pastur distribution with parameter
$\gamma$.

Above the transition, $\ell_{+}(\gamma)<\ell$, “strong signal” produces an
empirical eigenvalue dependant on $\ell$, though with upwards bias:
$\lambda(\ell)>\ell$. This asymptotic bias in extreme eigenvalues is a further
cause of inconsistency of $S$ in several loss measures, obviously including
operator norm.

  3. 3.

Eigenvector rotation. The eigenvectors $v_{1},\ldots,v_{p}$ of $S_{n}$ do not
align asymptotically with the corresponding eigenvectors $u_{1},\ldots,u_{p}$
of $\Sigma$. Under proportional growth, the limiting angles are deterministic,
and obey:

$\displaystyle|\langle u_{i},v_{j}\rangle|\xrightarrow{a.s.}\delta_{ij}\cdot
c(\ell_{i})\,,\hskip 56.9055pt1\leq i,j\leq r\,;$ (1.4)

here the “cosine” function $c(\ell)=c(\ell,\gamma)$ is given piecewise by

$\displaystyle\phantom{\,.}c^{2}(\ell)=\begin{dcases}\frac{1-\gamma/(\ell-1)^{2}}{1+\gamma/(\ell-1)}&\ell>1+\sqrt{\gamma}\\\
0&\ell\leq 1+\sqrt{\gamma}\end{dcases}\,.$ (1.5)

Again, a phase transition occurs at $\ell_{+}(\gamma)$. The misalignment of
empirical and theoretical eigenvectors further contributes to inconsistency in
the spiked case where not all population eigenvalues are equal; this is
easiest to see for Frobenius loss.

###  1.2 Shrinkage Estimation

Charles Stein proposed eigenvalue shrinkage as an alternative to traditional
covariance estimation [31, 32]. Let $S=V\Lambda V^{\prime}$ denote an
eigendecomposition, where $V$ is orthogonal and
$\Lambda=\text{diag}(\lambda_{1},\ldots,\lambda_{p})$. Let
$\eta:[0,\infty)\rightarrow[0,\infty)$ denote a “shrinkage” function or “rule”
and $\eta(\Lambda)=\text{diag}(\eta(\lambda_{1}),\ldots,\eta(\lambda_{p}))$.
Estimators of the form $\widehat{\Sigma}_{\eta}=V\eta(\Lambda)V^{\prime}$ are
studied in hundreds of papers; see for example work of Donoho, Gavish, and
Johnstone [14] and Ledoit and Wolf [21, 22]. Note that while there is
indeterminism in the choice of eigenvectors $V$, $\widehat{\Sigma}_{\eta}$ is
well defined.

The standard sample covariance estimator $S$ is obtained by the identity
“shrinker” $\eta(\lambda)=\lambda$, no shrinkage at all, while “effective”
shrinkers generally act as contractions, obeying
$|\eta(\lambda)-1|<|\lambda-1|$. In the spiked model, a well-chosen shrinker
minimizes the errors induced by eigenvalue spreading and eigenvector rotation.
Working under the spiked model and proportional growth, [14] considers
numerous loss functions $L$ and derives asymptotically unique admissible
shrinkers $\eta^{*}(\cdot|L)$, in many cases far outperforming $S$.

###  1.3 Which Choice of Asymptotic Framework?

The modern “big data” explosion exhibits all manner of ratios of dimension to
sample size. Indeed, there are internet traffic datasets with billions of
samples and thousands of dimensions, and computational biology datasets with
thousands of samples and millions of dimensions. To consider only asymptotic
frameworks where row and column counts are roughly balanced, as they are under
proportional growth, is a restriction, and perhaps, even an obstacle.

Although proportional growth analysis has yielded many valuable insights, it
also raises pressing doubts in applications. Consider the practitioner’s
“scaling conundrum”: in a given application, with one dataset of size
$(n_{\text{data}},p_{\text{data}})$, how can a practitioner know whether the
proportional growth model is applicable?

Implicit in the choice of asymptotic framework is an assumption on how the
data arises in a sequence of growing datasets; this choice has consequences.
The practitioner may view the data as arising from the fixed-$p$ sequence
$(n,p_{\text{data}})$ with only $n$ varying. If so, the practitioner could
appeal to long tradition and estimate $\Sigma$ by $S$. On the other hand,
viewing the dataset size as part of a sequence
$\big{(}n,\frac{p_{\text{data}}}{n_{\text{data}}}\cdot n\big{)}$, with
constant aspect ratio $\gamma=p_{\text{data}}/n_{\text{data}}$, the
practitioner might follow recent trends in the theoretical literature and
apply eigenvalue shrinkage.

Current theory offers little guidance on choice which perspective is more
appropriate, particularly if $p_{\text{data}}$ is large yet relatively small
compared to $n_{\text{data}}$. And—importantly—these two are not the only
possible scaling relations that could produce the given
$(n_{\text{data}},p_{\text{data}})$.

###  1.4 Disproportional Growth

Within the full spectrum of power law scalings $p\asymp n^{\alpha}$,
$\alpha\geq 0$, the much-studied proportional growth limit corresponds to the
single case $\alpha=1$. The classical $p$-fixed, $n$ growing relation again
corresponds to a single case, $\alpha=0$. This paper considers disproportional
growth, encompassing everything else:

$n,p\rightarrow\infty\,,\qquad\gamma_{n}=p/n\rightarrow 0\text{ or }\infty\,.$

Note that all power law scalings $0<\alpha<\infty$, $\alpha\neq 1$ are
included, as well as non-power law scalings, such as $p=\log n$ or $p=e^{n}$.
The disproportional growth framework splits naturally into instances; to
describe them, we use terminology that assumes the underlying data matrices
$X=X_{n}$ are $p\times n$.

  1. 1.

The “wide matrix” disproportional limit obeys:

$n,p\rightarrow\infty\,,\qquad\gamma_{n}=p/n\rightarrow 0.$ (1.6)

In this limit, which includes power laws with $\alpha\in(0,1)$, $n$ is much
larger than $p$, and yet we are outside the classical, fixed-$p$ large-$n$
setting.

  2. 2.

The “tall matrix” disproportional limit involves arrays with many more columns
than rows; formally:

$n,p\rightarrow\infty\,,\qquad\gamma_{n}=p/n\rightarrow\infty\,.$ (1.7)

This limit, including power laws with $\alpha\in(1,\infty)$, admits many
additional scalings of numbers of rows to columns.

Properties of covariance matrices in the two disproportionate limits are
closely linked. Indeed, the non-zero eigenvalues of $XX^{\prime}$ and
$X^{\prime}X$ are equal. In the standard normal case $\Sigma=I$, $XX^{\prime}$
and $X^{\prime}X$ are the unnormalized sample covariance matrices of wide and
tall datasets, respectively. Elaborating this, for any sequence of tall
matrices with $\gamma_{n}\rightarrow\infty$, there is an accompanying sequence
of wide matrices with $\gamma_{n}\rightarrow 0$ and related spectral
properties.

###  1.5 Estimation as $\gamma_{n}\rightarrow 0$

The $\gamma_{n}\rightarrow 0$ regime seems, at first glance, very different
from the proportional case, $\gamma_{n}\rightarrow\gamma>0$. Neither
eigenvalue spreading nor eigenvalue inflation are apparent: under spikes I,
empirical eigenvalues converge to their population counterparts:
$\lambda_{i}\rightarrow\ell_{i}$, almost surely. Eigenvalue shrinkage may
therefore seem irrelevant or unhelpful: $S$ itself is a consistent estimator
of $\Sigma$ in Frobenius and operator norms. To the contrary, we will show
that well-designed eigenvalue shrinkage confers substantial relative gains
over standard covariance estimation, paralleling gains seen earlier under
proportional growth.

Eigenvalue spreading does occur as $\gamma_{n}\rightarrow 0$, though only by
$O(\sqrt{\gamma_{n}})$. Accordingly, introduce the quantities

$\accentset{\leftharpoonup}{\lambda}=\frac{\lambda-1}{\sqrt{\gamma_{n}}}\,,\qquad\,\,\,\accentset{\leftharpoonup}{\ell}=\frac{\ell-1}{\sqrt{\gamma_{n}}}\,,$

measuring leading empirical and population eigenvalues on a finer scale. We
adapt the spiked covariance model to the $\gamma_{n}\rightarrow 0$ setting,
with spiked eigenvalues
$\ell_{i}=\ell_{i;n}=1+\accentset{\leftharpoonup}{\ell}_{i}\sqrt{\gamma_{n}}$
varying with $n$ while maintaining $\accentset{\leftharpoonup}{\ell}_{i}$
fixed. On this scale, as we shall see, in addition to eigenvalue spreading,
eigenvalue inflation and eigenvector rotation occur as well. The consequences
of such high-dimensional phenomena are similar to those uncovered in the
proportional setting. For many choices of loss function, $S$ is outperformed
substantially by well-designed shrinkage functions, particularly near the
phase transition at $\ell_{+}(\gamma_{n})$. We will consider a range of loss
functions $L$, deriving for each a shrinker $\eta^{*}(\cdot|L)$ which is
optimal as $\gamma_{n}\rightarrow 0$. Analogous results hold as
$\gamma_{n}\rightarrow\infty$.

###  1.6 Estimation in the Spiked Wigner Model

At the heart of our analysis is a connection to the spiked Wigner model. Let
$W=W_{n}$ denote a Wigner matrix, a real symmetric matrix of size $n\times n$
with independent entries on the upper triangle distributed as
$\mathcal{N}(0,1)$. Let $\Theta=\Theta_{n}$ denote a symmetric $n\times n$
“signal” matrix of fixed rank $r$; under the spiked Wigner model observed data
$Y=Y_{n}$ obeys

$\displaystyle\phantom{\,.}Y=\Theta+\frac{1}{\sqrt{n}}W\,.$ (1.8)

Let
$\theta_{1}\geq\cdots\geq\theta_{r_{+}}>0>\theta_{r_{+}+1}\geq\dots\geq\theta_{r}$
denote the non-zero eigenvalues of $\Theta$, so there are $r_{+}$ positive
values and $r_{-}=r-r_{+}$ negative.

A standard approach to recovering $\Theta$ from noisy data $Y$ uses the
eigenvalues of $Y$, $\lambda_{1}(Y)\geq\cdots\geq\lambda_{n}(Y)$, and the
associated eigenvectors $v_{1},\ldots,v_{n}$:

$\widehat{\Theta}^{r}=\sum_{i=1}^{r_{+}}\lambda_{i}(Y)v_{i}v_{i}^{\prime}+\sum_{i=n-r_{-}+1}^{n}\lambda_{i}(Y)v_{i}v_{i}^{\prime}\,.$

As it turns out, $\widehat{\Theta}^{r}$ can be improved upon substantially by
estimators of the form

$\displaystyle\phantom{\,.}\widehat{\Theta}_{\eta}=\sum_{i=1}^{n}\eta(\lambda_{i}(Y))v_{i}v_{i}^{\prime}\,,$
(1.9)

with $\eta:\mathbb{R}^{+}\rightarrow\mathbb{R}^{+}$ a well-chosen shrinkage
function.

Optimal shrinkage formulas for the spiked Wigner model are equivalent to
optimal formulas for covariance estimation in the disproportionate,
$\gamma_{n}\rightarrow 0$ limit. This is not mere coincidence. As
$\gamma_{n}\rightarrow 0$, the spectral measure of
$\gamma_{n}^{-1/2}(S-I_{p})$ converges (weakly almost surely) to the
semicircle law (Bai and Yin [2]); this is precisely the limiting spectral
measure of spiked Wigners $Y$. Moreover, we show below that the driving
theoretical quantities in each setting—leading eigenvalue inflation,
eigenvector rotation, optimal shrinkers, and losses—are all “isomorphic.”

###  1.7 Our Contributions

Given this background, we now state our contributions:

  1. 1.

We study the disproportionate $\gamma_{n}\rightarrow 0$ limit with an eye
towards establishing and using analogues of (1.2)-(1.5). In the critical
scaling of this regime, spiked eigenvalues decay towards one as
$1+\accentset{\leftharpoonup}{\ell}\sqrt{\gamma_{n}}$, where
$\accentset{\leftharpoonup}{\ell}$ is a new formal parameter. Analogs of
(1.2)-(1.5) as a function of $\accentset{\leftharpoonup}{\ell}$ are presented
in Lemma 3.1 below. On this scale, the analog of the BBP phase transition—–the
critical spike strength above which leading sample eigenvectors correlate with
population eigenvectors—–now occurs at $\accentset{\leftharpoonup}{\ell}=1$.
These formulas and concepts are our basis for our development of new optimal
eigenvalue shrinkers. Results equivalent to Lemma 3.1 were previously
established under slightly different assumptions by Bloemendal et al. [9]. We
provide a simple, direct argument that permits completely general rates at
which $n,p\rightarrow\infty$ while $\gamma_{n}\rightarrow 0$. Analogous
results hold as $\gamma_{n}\rightarrow\infty$, explored in later sections.

  2. 2.

New sets of asymptotically optimal formulas for shrinkage of eigenvalues are
found under fifteen canonical loss functions. While all losses vanish as
$\gamma_{n}\rightarrow 0$ and diverge as $\gamma_{n}\rightarrow\infty$,
optimal shrinkage provides improvement by multiplicative factors. Under
proportional growth, there is a distinct optimal shrinker for each of the
fifteen losses considered. In the disproportionate regime, there are three
distinct shrinkers, a simplification. These formulas are the limits of
proportional shrinkers as the aspect ratio $\gamma_{n}$ vanishes or diverges
and population spikes decay towards one or diverge. We derive closed forms for
the relative gain of optimal shrinkage versus classical no-shrinkage. In
addition, we find optimal hard thresholding levels under each loss.

  3. 3.

The $n,p\rightarrow\infty$, $\gamma_{n}\rightarrow 0$ limit is dissimilar to
classical fixed-$p$ statistics: for any rate $\gamma_{n}\rightarrow 0$, non-
trivial eigenvalue shrinkage is optimal.

  4. 4.

A consequence of our results is that the optimal shrinkage formulas inherited
from the proportional regime, with $\gamma$ replaced by $\gamma_{n}$, achieve
optimal performance in the disproportionate limits. These shrinkers are
harmless in the classical fixed-$p$ limit. A practitioner with a dataset of
size $(n_{\text{data}},p_{\text{data}})$ implementing the proportionate
shrinkers of Donoho, Gavish, and Johnstone [14] would substitute
$p_{\text{data}}/n_{\text{data}}$ for $\gamma$. We prove that this heuristic
is asymptotically optimal in the disproportionate limits. Thus, for a given
loss function, we provide a single shrinkage function to be used with the
current aspect ratio that achieves optimal performance in any asymptotic
embedding of $(n_{\text{data}},p_{\text{data}})$. A practitioner need not have
the asymptotic embedding (unknown, of course, in practice) to perform optimal
shrinkage.

  5. 5.

We obtain asymptotically optimal shrinkage formulas for the spiked Wigner
model. As the empirical spectral distributions of the spiked covariance as
$\gamma_{n}\rightarrow 0$ and the spiked Wigner converge to a common limit—the
semicircle law—optimal shrinkage formulas are equivalent after a change of
variables.

Our assumption that non-spiked population eigenvalues are identity is for
convenience. In the case of an arbitrary noise level, where $\Sigma$ is a low-
rank perturbation of $\sigma^{2}I$, procedures herein may be appropriately
scaled. If the noise level $\sigma^{2}$ is unknown, it is consistently
estimated by the median eigenvalue of $S$ as $\gamma_{n}\rightarrow 0$. As
$\gamma_{n}\rightarrow\infty$, the median of non-zero eigenvalues suffices.
Asymptotic optimality extends to this setting by the continuity of the
provided shrinkers. Knowledge of the number of spikes $r$ is similarly
unnecessary. In practice, optimal shrinkers may be applied to each empirical
eigenvalue. Rigorous proof of this claim is given in Section 7 of [14].
Similarly, the rank and variance assumptions (1.8) may be relaxed.

##  2 The Fixed-Spike, $\gamma_{n}\rightarrow\gamma$ Limit

We briefly review certain tools and background concepts.

######  Definition 2.1.

As discussed in Section 1.7, the model rank $r$ is assumed known. It therefore
makes sense to employ rank-aware covariance shrinkage estimators: for a
shrinkage function $\eta:[0,\infty)\rightarrow[0,\infty)$,

$\displaystyle\widehat{\Sigma}_{\eta}=\widehat{\Sigma}_{\eta,n,r}$
$\displaystyle=\sum_{i=1}^{r}\eta(\lambda_{i})v_{i}v_{i}^{\prime}+\sum_{i=r+1}^{n}v_{i}v_{i}^{\prime}$
$\displaystyle=\sum_{i=1}^{r}(\eta(\lambda_{i})-1)v_{i}v_{i}^{\prime}+I\,.$
(2.1)

In the particular case $\eta(\lambda)\equiv\lambda$ —no shrinkage— we may
instead write $S^{r}$ rather than $\widehat{\Sigma}_{\lambda,n,r}$.

######  Definition 2.2.

Let $\|\cdot\|_{F}$, $\|\cdot\|_{op}$, and $\|\cdot\|_{*}$ respectively denote
the Frobenius, operator, and nuclear matrix norms. We consider estimation
under 15 loss functions, each formed by applying one of the 3 matrix norms to
one of 5 pivots. By pivot, we mean a matrix-valued function $\Delta(A,B)$ of
two real positive definite matrices $A,B$; we consider specifically:

$\displaystyle\Delta_{1}$ $\displaystyle=A-B\,,$ $\displaystyle\Delta_{2}$
$\displaystyle=A^{-1}-B^{-1}\,,$ $\displaystyle\Delta_{3}$
$\displaystyle=A^{-1}B-I\,,$ $\displaystyle\Delta_{4}$
$\displaystyle=B^{-1}A-I\,,$ $\displaystyle\Delta_{5}$
$\displaystyle=A^{-1/2}BA^{-1/2}-I\,.$

We apply each norm to each of the pivots, obtaining for $k=1,\ldots,5$, the
loss functions:

$\displaystyle
L_{F,k}(\Sigma,\widehat{\Sigma})=\|\Delta_{k}(\Sigma,\widehat{\Sigma})\|_{F}\,,$
$\displaystyle
L_{O,k}(\Sigma,\widehat{\Sigma})=\|\Delta_{k}(\Sigma,\widehat{\Sigma})\|_{op}\,,$
$\displaystyle
L_{N,k}(\Sigma,\widehat{\Sigma})=\|\Delta_{k}(\Sigma,\widehat{\Sigma})\|_{*}\,.$

######  Lemma 2.1.

(Lemma 7 of [14]) In the proportional limit (1.1) of the spiked model I,
suppose $\eta(\lambda_{i})$ has almost sure limit $\eta_{i}$, $1\leq i\leq r$.
Each loss $L_{\star,k}$ converges almost surely to a deterministic limit:

$L_{\star,k}(\Sigma,\widehat{\Sigma}_{\eta})\xrightarrow{a.s.}{\cal
L}_{\star,k}((\ell_{i})_{i=1}^{r},(\eta_{i})_{i=1}^{r}),\qquad\star\in\\{F,O,N\\},\quad
1\leq k\leq 5.$

The asymptotic loss is sum- or max- decomposable into $r$ terms deriving from
non-unit spiked eigenvalues. The terms involve matrix norms applied to pivots
of $2\times 2$ matrix expressions:

$\displaystyle A(\ell)=\begin{bmatrix}\ell&0\\\ 0&1\end{bmatrix}\,,$
$\displaystyle B(\eta,c)=I_{2}+(\eta-1)\begin{bmatrix}c^{2}&cs\\\
cs&s^{2}\end{bmatrix}\,,$

where $s^{2}=1-c^{2}$. With $\ell_{i}$ denoting a spiked eigenvalue and
$c(\ell_{i})$ the limiting cosine in (1.4), the decompositions are

$\displaystyle\phantom{\,,}{\cal
L}_{F,k}((\ell_{i})_{i=1}^{r},(\eta_{i})_{i=1}^{r})$
$\displaystyle=\bigg{(}\sum_{i=1}^{r}\big{[}L_{F,k}\big{(}A(\ell_{i}),B(\eta_{i},c(\ell_{i}))\big{)}\big{]}^{2}\bigg{)}^{1/2}\,,$
$\displaystyle{\cal L}_{O,k}((\ell_{i})_{i=1}^{r},(\eta_{i})_{i=1}^{r})$
$\displaystyle=\max_{1\leq i\leq
r}L_{O,k}\big{(}A(\ell_{i}),B(\eta_{i},c(\ell_{i}))\big{)}\,,$
$\displaystyle{\cal L}_{N,k}((\ell_{i})_{i=1}^{r},(\eta_{i})_{i=1}^{r})$
$\displaystyle=\sum_{i=1}^{r}L_{N,k}\big{(}A(\ell_{i}),B(\eta_{i},c(\ell_{i}))\big{)}\,.$

For each of the 15 losses studied here, and several others, [14] derives—under
proportional growth $\gamma_{n}\rightarrow\gamma>0$—a shrinker
$\eta^{*}(\lambda|L)=\eta^{*}(\lambda|L,\gamma)$ minimizing the asymptotic
loss ${\cal L}$. Optimal shrinkers depend on the “inverse” of the eigenvalue
mapping (1.3):

$\displaystyle\ell(\lambda)=\ell(\lambda,\gamma)=\begin{dcases}\frac{\lambda+1-\gamma+\sqrt{(\lambda-1-\gamma)^{2}-4\gamma}}{2}&\lambda>\lambda_{+}(\gamma)\\\
\ell_{+}(\gamma)&\lambda\leq\lambda_{+}(\gamma)\end{dcases}\,.$

Although $\ell_{i}$ is unobserved, this inverse provides a consistent
estimator of $\ell_{i}$ above the BBP transition:

$\phantom{\,.}\ell(\lambda_{i})\xrightarrow[]{a.s.}\ell_{i}\,,\hskip
56.9055pt\ell_{i}>\ell_{+}(\gamma)\,.$

In most cases, shrinkers are explicit in terms of $\ell$, $c$, and $s$. For
example, for $L_{F,1}$, the optimal shrinker is
$\eta^{*}(\lambda|L_{F,1})=\ell(\lambda)\cdot
c^{2}(\ell(\lambda))+s^{2}(\ell(\lambda))$, while for $L_{O,1}$, it is simply
$\eta^{*}(\lambda|L_{O,1})=\ell(\lambda)$; a list of 18 such closed forms can
be found in [14]. For notational lightness, we may write
$\eta^{*}(\lambda|L_{F,1})=\ell\cdot c^{2}+s^{2}$, or
$\eta^{*}(\lambda|L_{O,1})=\ell$.

##  3 Covariance Estimation as $\gamma_{n}\rightarrow 0$

###  3.1 The Variable-Spike, $\gamma_{n}\rightarrow 0$ Limit

We now formalize our earlier informal discussion of the asymptotic limit
$\gamma_{n}\rightarrow 0$. Consider the normalized empirical eigenvalues
defined by

$\accentset{\leftharpoonup}{\lambda}_{i}=\accentset{\leftharpoonup}{\lambda}_{i,n}=\frac{\lambda_{i}-1}{\sqrt{\gamma_{n}}}\,,\hskip
56.9055pt1\leq i\leq p\,.$ (3.1)

This transformation “spreads” eigenvalues tightly grouped near $1$. In the
standard normal case $\Sigma=I$, the empirical distribution of
$(\accentset{\leftharpoonup}{\lambda}_{i})_{i=1}^{p}$ converges weakly almost
surely to the semicircle law with support $[-2,2]$ (Bai and Yin [2]).

We generalize the spiked model to allow spiked eigenvalues
$(\ell_{i})_{i=1}^{r}$ to vary with $n$. Matching (3.1), we consider
normalized spiked eigenvalues $\gamma_{n}^{-1/2}(\ell_{i}-1)$ and assume their
convergence to limits
$(\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r}\in(0,\infty)$. That is, we
study spiked eigenvalues of the form

$\displaystyle[\textbf{II}]\hskip
42.67912pt\ell_{i}=\ell_{i,n}=1+\accentset{\leftharpoonup}{\ell}_{i}(1+o(1))\sqrt{\gamma_{n}}\,,\hskip
56.9055pt1\leq i\leq r\,,$

where $(\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r}$ are constant, non-
negative, parameters. Spiked eigenvalues are no longer fixed as under model I,
rather, they decay towards one at the $\sqrt{\gamma_{n}}$ rate. We assume
supercritical eigenvalues—those with
$\accentset{\leftharpoonup}{\ell}_{i}>1$—are simple.

This new disproportional limit and varying-spike model yields formulas
analogous to, yet distinct from, those in effect under proportional growth and
fixed spikes. New formulas seem more elegant: a phase transition occurs simply
at $\accentset{\leftharpoonup}{\ell}=1$. Define the eigenvalue mapping
function

$\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})=\begin{dcases}\accentset{\leftharpoonup}{\ell}+\frac{1}{\accentset{\leftharpoonup}{\ell}}&\accentset{\leftharpoonup}{\ell}>1\\\
2&0<\accentset{\leftharpoonup}{\ell}\leq 1\\\ \end{dcases}$ (3.2)

and the eigenvector cosine function

$\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})=\
\begin{dcases}1-\frac{1}{\accentset{\leftharpoonup}{\ell}^{2}},&\accentset{\leftharpoonup}{\ell}>1\\\
0&0<\accentset{\leftharpoonup}{\ell}\leq 1,\\\ \end{dcases}.$ (3.3)

For future use, we also define
$\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})=1-\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})$.

######  Lemma 3.1.

Under $\gamma_{n}\rightarrow 0$ and varying spikes II,

$\phantom{\,.}\accentset{\leftharpoonup}{\lambda}_{i}\xrightarrow{a.s.}\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}_{i})\,,\hskip
56.9055pt1\leq i\leq r\,.$ (3.4)

With $v_{1},\ldots,v_{p}$ and denoting the eigenvectors of $S$ in decreasing
eigenvalue ordering, and $u_{1},\ldots,u_{p}$ the corresponding eigenvectors
of $\Sigma$, the angles between pairs of eigenvectors have limits

$|\langle
u_{i},v_{j}\rangle|\xrightarrow{a.s.}\delta_{ij}\cdot\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\leftharpoonup}{\ell}_{i}),\hskip
29.87547pt1\leq i,j\leq r\,.$ (3.5)

The reader will no doubt see that Lemma 3.1 exhibits a formal similarity to
proportional regime results (1.2) and (1.4); as in the proportional case,
spiked eigenvalues of a critical scale produce eigenvalue inflation and
eigenvector rotation, now written in terms of the new parameter
$\accentset{\leftharpoonup}{\ell}$. The arrow decorators allow us to preserve
a formal resemblance between (1.3) and (1.5) and previous results, yet
indicate there are important differences.

A stronger, non-asymptotic form of this lemma is established by Bloemendal et
al. [9]. Yet, [9] requires the existence of some $k>0$ such that $n\leq p^{k}$
eventually, while here $\gamma_{n}$ may tend to zero arbitrarily slowly.
Feldman [17] considers a closely-related signal-plus-noise model. For the
reader’s convenience, a simple, direct proof of Lemma 3.1 is provided in
Appendix A.

###  3.2 Asymptotic Loss in the Variable-Spike, $\gamma_{n}\rightarrow 0$
Limit

Recall the families of rank-aware estimates $\widehat{\Sigma}_{\eta}$ and
losses $L_{\star,k}$ defined in Section 2. Under variable-spikes II, the
sequence of estimands

$\Sigma=\sum_{i=1}^{r}(\ell_{i}-1)u_{i}u_{i}^{\prime}+I$

now approaches identity. $L_{\star,k}(\Sigma,\widehat{\Sigma}_{\eta})$
vanishes for any shrinker $\eta$ that is continuous at one and satisfies
$\eta(1)=1$, in particular, $L_{\star,k}(\Sigma,S)\rightarrow 0$. We therefore
consider rescaled losses:

$\accentset{\leftharpoonup}{L}_{\star,k}(\Sigma,\widehat{\Sigma})=\frac{L_{\star,k}(\Sigma,\widehat{\Sigma})}{\sqrt{\gamma_{n}}}\,.$

Observe that
$\accentset{\leftharpoonup}{L}_{\star,1}(\Sigma,\widehat{\Sigma})=\|(\Sigma-I)-(\widehat{\Sigma}-I)\|_{\star}/\sqrt{\gamma_{n}}$,
which we view as transforming to a new coordinate system with origin at the
identity matrix. Let
$\accentset{\leftharpoonup}{\phi}_{n}(x)=\gamma_{n}^{-1/2}(x-1)$ denote the
mapping to these coordinates. Thus, (3.1) and spikes II may be written as
$\accentset{\leftharpoonup}{\lambda}_{i}=\accentset{\leftharpoonup}{\phi}_{n}(\lambda_{i})$
and
$\accentset{\leftharpoonup}{\phi}_{n}(\ell_{i})\rightarrow\accentset{\leftharpoonup}{\ell}_{i}$.

######  Definition 3.1.

Let $\eta=\eta_{n}$ denote a sequence of shrinkers, possibly varying with $n$.
Suppose that under the disproportional $\gamma_{n}\rightarrow 0$ limit and
varying-spikes II, the sequences of normalized shrinker outputs induced by
rescaling converge:

$\phantom{\,.}\accentset{\leftharpoonup}{\phi}_{n}(\eta(\lambda_{i}))\xrightarrow{a.s.}\accentset{\leftharpoonup}{\eta}_{i}\,,\hskip
56.9055pt1\leq i\leq r\,.$

We call the limits $(\accentset{\leftharpoonup}{\eta}_{i})_{i=1}^{r}$ the
asymptotic shrinkage descriptors.

######  Lemma 3.2.

Let $\eta$ denote a sequence of shrinkers with asymptotic shrinkage
descriptors $(\accentset{\leftharpoonup}{\eta}_{i})_{i=1}^{r}$ under the
disproportional $\gamma_{n}\rightarrow 0$ limit and varying-spikes II. Each
loss $\accentset{\leftharpoonup}{L}_{\star,k}$ then converges almost surely to
a deterministic limit:

$\accentset{\leftharpoonup}{L}_{\star,k}(\Sigma,\widehat{\Sigma}_{\eta})\xrightarrow{a.s.}\accentset{\leftharpoonup}{\cal
L}_{\star}((\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\leftharpoonup}{\eta}_{i})_{i=1}^{r})\,,\qquad\star\in\\{F,O,N\\}\,,\quad
1\leq k\leq 5\,.$

The asymptotic loss does not involve $k$. It is sum- or max- decomposable into
$r$ terms deriving from non-unit spike eigenvalues. The terms involve matrix
norms applied to pivots of $2\times 2$ matrices $\widetilde{A}$ and
$\widetilde{B}$:

$\displaystyle\widetilde{A}(\accentset{\leftharpoonup}{\ell})=\begin{bmatrix}\accentset{\leftharpoonup}{\ell}&0\\\
0&0\end{bmatrix}\,,$
$\displaystyle\widetilde{B}(\accentset{\leftharpoonup}{\eta},\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt})=\accentset{\leftharpoonup}{\eta}\cdot\begin{bmatrix}\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}&\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}\\\
\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}&\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}\end{bmatrix}\,,$

where
$\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}=1-\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}$.
With $\accentset{\leftharpoonup}{\ell}_{i}$ denoting a spiked eigenvalue and
$\accentset{\leftharpoonup}{c}(\accentset{\leftharpoonup}{\ell}_{i})$ the
limiting cosine in (3.5), the decompositions are

$\displaystyle\phantom{\,,}\accentset{\leftharpoonup}{\cal
L}_{F}((\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\leftharpoonup}{\eta}_{i})_{i=1}^{r})$
$\displaystyle=\bigg{(}\sum_{i=1}^{r}\big{[}L_{F,1}\big{(}\widetilde{A}(\accentset{\leftharpoonup}{\ell}_{i}),\widetilde{B}(\accentset{\leftharpoonup}{\eta}_{i},\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\leftharpoonup}{\ell}_{i}))\big{)}\big{]}^{2}\bigg{)}^{1/2}\,,$
$\displaystyle\accentset{\leftharpoonup}{\cal
L}_{O}((\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\leftharpoonup}{\eta}_{i})_{i=1}^{r})$
$\displaystyle=\max_{1\leq i\leq
r}L_{O,1}\big{(}\widetilde{A}(\accentset{\leftharpoonup}{\ell}_{i}),\widetilde{B}(\accentset{\leftharpoonup}{\eta}_{i},\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\leftharpoonup}{\ell}_{i}))\big{)}\,,$
$\displaystyle\accentset{\leftharpoonup}{\cal
L}_{N}((\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\leftharpoonup}{\eta}_{i})_{i=1}^{r})$
$\displaystyle=\sum_{i=1}^{r}L_{N,1}\big{(}\widetilde{A}(\accentset{\leftharpoonup}{\ell}_{i}),\widetilde{B}(\accentset{\leftharpoonup}{\eta}_{i},\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\leftharpoonup}{\ell}_{i}))\big{)}\,.$

###### Proof.

Under loss $L_{\star,1}$, the argument is identical to the proofs of Lemma 2
and 7 of [14], only using Lemma 3.1 for eigenvalue inflation and eigenvector
rotation as $\gamma_{n}\rightarrow 0$. The pivots we consider are
asymptotically equivalent: using the simultaneous block decomposition in Lemma
5 of [14] and a Neumann series expansion,

$\phantom{\,.}|\accentset{\leftharpoonup}{L}_{\star,1}(\Sigma,\widehat{\Sigma}_{\eta})-\accentset{\leftharpoonup}{L}_{\star,k}(\Sigma,\widehat{\Sigma}_{\eta})|\xrightarrow[]{a.s.}0\,,\hskip
56.9055pt2\leq k\leq 5\,.$

∎

For example, the asymptotic shrinkage descriptors of the rank-aware sample
covariance estimator
$S^{r}=S_{n}^{r}=\sum_{i=1}^{r}(\lambda_{i}-1)v_{i}v_{i}^{\prime}+I$ are
$\accentset{\leftharpoonup}{\eta}_{i}=\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}_{i})$.
For $r=1$, suppressing the subscript of
$\accentset{\leftharpoonup}{\ell}_{1}$, squared asymptotic loss evaluates to

$\phantom{\,.}\big{[}\accentset{\leftharpoonup}{\cal
L}_{F}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}))\big{]}^{2}=(\accentset{\leftharpoonup}{\ell}-\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell}))^{2}+\accentset{\leftharpoonup}{\lambda}^{2}(\accentset{\leftharpoonup}{\ell})(1-\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{4}(\accentset{\leftharpoonup}{\ell}))\,.$
(3.6)

By Lemma 3.1, this simplifies to $2+3/\accentset{\leftharpoonup}{\ell}^{2}$
for $\accentset{\leftharpoonup}{\ell}>1$ and to
$\accentset{\leftharpoonup}{\ell}^{2}+4$ for
$\accentset{\leftharpoonup}{\ell}\leq 1$. Hence, the (unsquared) asymptotic
loss attains a global maximum of $\sqrt{5}$ precisely at the phase transition
$\accentset{\leftharpoonup}{\ell}=1$. Asymptotic losses of $S^{1}$ under each
norm are collected below in Table 1, to later facilitate comparison with
optimal shrinkage.

Norm | $\accentset{\leftharpoonup}{\ell}<1$ | $\accentset{\leftharpoonup}{\ell}>1$
---|---|---
Frobenius | $\sqrt{\accentset{\leftharpoonup}{\ell}^{2}+4}$ | $\sqrt{2+3/\accentset{\leftharpoonup}{\ell}^{2}}$
Operator | $2$ | $\big{(}1+\sqrt{5+4\accentset{\leftharpoonup}{\ell}^{2}}\big{)}/(2\accentset{\leftharpoonup}{\ell})$
Nuclear | $\accentset{\leftharpoonup}{\ell}+2$ | $\sqrt{4+5/\accentset{\leftharpoonup}{\ell}^{2}}$
Table 1: Asymptotic Loss $\accentset{\leftharpoonup}{\cal L}_{\star}$ of the
rank-aware sample covariance $S^{1}$ (the subscript of
$\accentset{\leftharpoonup}{\ell}_{1}$ is suppressed).

###  3.3 Optimal Asymptotic Loss

This subsection assumes $r=1$; the subscript of
$\accentset{\leftharpoonup}{\ell}_{1}$ is suppressed. Recalling the relations
between $\accentset{\leftharpoonup}{\ell}$,
$\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})$, and
$\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\leftharpoonup}{\ell})$,
one sees in Lemma 3.2 and (3.6) that
$\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})$ is not
the minimizer of the function
$\accentset{\leftharpoonup}{\eta}\mapsto\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})$.
A sequence estimators
$\widehat{\Sigma}_{\eta}=(\eta(\lambda_{1})-1)v_{1}v^{\prime}_{1}+I$ can
outperform $S^{1}$ substantially, provided the asymptotic shrinkage descriptor
$\accentset{\leftharpoonup}{\eta}_{1}=\lim_{n\rightarrow\infty}\accentset{\leftharpoonup}{\phi}_{n}(\eta(\lambda_{1}))$
exists and $\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta}_{1})<\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}))$.

In this subsection, we calculate the asymptotic shrinkage descriptors that
minimize
$\accentset{\leftharpoonup}{\eta}\mapsto\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})$.
The following subsection shows the existence of shrinkers with such asymptotic
shrinkage descriptors.

######  Definition 3.2.

The formally optimal asymptotic loss in the rank-1 setting is

$\accentset{\leftharpoonup}{\cal
L}_{\star}^{1}(\accentset{\leftharpoonup}{\ell})=\min_{\vartheta}\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\vartheta),\qquad\star\in\\{F,O,N\\}.$

A formally optimal shrinker is a function
$\accentset{\leftharpoonup}{\eta}(\cdot|\star):\mathbb{R}\mapsto\mathbb{R}$
achieving $\accentset{\leftharpoonup}{\cal
L}_{\star}^{1}(\accentset{\leftharpoonup}{\ell})$:

$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\ell}|\star)=\underset{\vartheta}{\mbox{argmin}}\,\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\vartheta),\qquad\accentset{\leftharpoonup}{\ell}>0\,,\qquad\star\in\\{F,O,N\\}\,.$

We write
$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\ell}|\star)$
rather than
$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\ell}|L_{\star,k})$
as by Lemma 3.2, optimal asymptotic losses are independent of the pivot $k$.

######  Lemma 3.3.

Formally optimal shrinkers and corresponding losses are given by

$\displaystyle\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}|F)=(\accentset{\leftharpoonup}{\ell}-1/\accentset{\leftharpoonup}{\ell})_{+}\,,$
$\displaystyle\accentset{\leftharpoonup}{\cal
L}_{F}^{1}(\accentset{\leftharpoonup}{\ell})=\begin{dcases}\sqrt{2-1/\accentset{\leftharpoonup}{\ell}^{2}}&\accentset{\leftharpoonup}{\ell}>1\\\
\accentset{\leftharpoonup}{\ell}&0<\accentset{\leftharpoonup}{\ell}\leq
1\end{dcases}\,,$
$\displaystyle\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}|O)=\accentset{\leftharpoonup}{\ell}\cdot
1_{\\{\accentset{\leftharpoonup}{\ell}>1\\}}\,,$
$\displaystyle\accentset{\leftharpoonup}{\cal
L}_{O}^{1}(\accentset{\leftharpoonup}{\ell})=\begin{dcases}1&\accentset{\leftharpoonup}{\ell}>1\\\
\accentset{\leftharpoonup}{\ell}&0<\accentset{\leftharpoonup}{\ell}\leq
1\end{dcases}\,,$ (3.7)
$\displaystyle\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}|N)=\big{(}\accentset{\leftharpoonup}{\ell}-2/\accentset{\leftharpoonup}{\ell}\big{)}_{+}\,,$
$\displaystyle\accentset{\leftharpoonup}{\cal
L}_{N}^{1}(\accentset{\leftharpoonup}{\ell})=\begin{dcases}2\sqrt{1-1/\accentset{\leftharpoonup}{\ell}^{2}}&\accentset{\leftharpoonup}{\ell}>\sqrt{2}\\\
\accentset{\leftharpoonup}{\ell}&0<\accentset{\leftharpoonup}{\ell}\leq\sqrt{2}\end{dcases}\,.$

###### Proof.

By Lemma 3.2,

$\displaystyle\phantom{\,.}\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\vartheta)$
$\displaystyle=L_{\star,1}\big{(}\widetilde{A}(\accentset{\leftharpoonup}{\ell}),\widetilde{B}(\vartheta,\accentset{\leftharpoonup}{c}(\accentset{\leftharpoonup}{\ell}))\big{)}=\bigg{\|}\begin{bmatrix}\accentset{\leftharpoonup}{\ell}-\vartheta\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})&-\vartheta\accentset{\leftharpoonup}{c}(\accentset{\leftharpoonup}{\ell})\accentset{\leftharpoonup}{s}(\accentset{\leftharpoonup}{\ell})\\\
-\vartheta\accentset{\leftharpoonup}{c}(\accentset{\leftharpoonup}{\ell})\accentset{\leftharpoonup}{s}(\accentset{\leftharpoonup}{\ell})&-\vartheta\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})\end{bmatrix}\bigg{\|}_{\star}$
$\displaystyle=\begin{cases}\sqrt{(\accentset{\leftharpoonup}{\ell}-\vartheta)^{2}+2\accentset{\leftharpoonup}{\ell}\vartheta\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})}&\star=F\\\
\max(|\lambda_{+}|,|\lambda_{-}|)&\star=O\\\
|\lambda_{+}|+|\lambda_{-}|&\star=N\end{cases}\,,$ (3.8)

where
$\lambda_{\pm}=\big{(}\vartheta-\accentset{\leftharpoonup}{\ell}\pm\sqrt{(\vartheta-\accentset{\leftharpoonup}{\ell})^{2}+4\vartheta\accentset{\leftharpoonup}{\ell}\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})}\big{)}/2$
($\lambda_{\pm}$ are the eigenvalues of the above $2\times 2$ matrix,
according to Lemma 14 of [15]). Differentiating, Frobenius loss is minimized
by
$\vartheta=\accentset{\leftharpoonup}{\ell}\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})=(\accentset{\leftharpoonup}{\ell}-1/\accentset{\leftharpoonup}{\ell})_{+}$.
For $\accentset{\leftharpoonup}{\ell}>1$, operator norm loss is minimized by
$\vartheta=\accentset{\leftharpoonup}{\ell}$, for which
$\lambda_{+}=-\lambda_{-}$. For $\accentset{\leftharpoonup}{\ell}\leq 1$,
$\lambda_{+}=\vartheta$, while
$-\lambda_{-}=\accentset{\leftharpoonup}{\ell}$. In this case, we take
$\vartheta=0$. For $\vartheta\geq 0$, nuclear norm loss may be rewritten as

$\ \accentset{\leftharpoonup}{\cal
L}_{N}(\accentset{\leftharpoonup}{\ell},\vartheta)=\sqrt{(\vartheta-\accentset{\leftharpoonup}{\ell})^{2}+4\vartheta\accentset{\leftharpoonup}{\ell}\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell})}\,;$

this is minimized by
$\vartheta=\accentset{\leftharpoonup}{\ell}(1-2\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell}))_{+}$.
If $\theta\leq 0$, $\accentset{\leftharpoonup}{\cal
L}_{N}(\accentset{\leftharpoonup}{\ell},\vartheta)=-\vartheta+\accentset{\leftharpoonup}{\ell}$
is minimized by $\vartheta=0$. We collect below formally optimal shrinkers:

$\displaystyle\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}|F)=\accentset{\leftharpoonup}{\ell}\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\leftharpoonup}{\ell})\,,$
$\displaystyle\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}|O)=\accentset{\leftharpoonup}{\ell}\cdot
1_{\\{\accentset{\leftharpoonup}{\ell}>1\\}}\,,$
$\displaystyle\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}|N)=\accentset{\leftharpoonup}{\ell}(1-2\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\leftharpoonup}{\ell}))_{+}\,.$
(3.9)

Substitution of (3.3) completes the proof. ∎

###  3.4 Unique Admissibility

Formally optimal shrinkers derived in the previous subsection depend on
$\accentset{\leftharpoonup}{\ell}$, which is not observable. We define the
partial inverse of the eigenvalue mapping
$\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})$ (3.2):

$\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\lambda})=\left\\{\begin{array}[]{ll}(\accentset{\leftharpoonup}{\lambda}+\sqrt{\accentset{\leftharpoonup}{\lambda}^{2}-4})/2&\accentset{\leftharpoonup}{\lambda}>2\\\
1&\accentset{\leftharpoonup}{\lambda}\leq 2\\\ \end{array}\right..$ (3.10)

Recall the rescaling mapping $\accentset{\leftharpoonup}{\phi}_{n}$, with
inverse
$\accentset{\leftharpoonup}{\phi}_{n}^{-1}(\accentset{\leftharpoonup}{\eta})=1+\sqrt{\gamma_{n}}\accentset{\leftharpoonup}{\eta}$.
Through these functions, we may “change coordinates” in shrinkers defined in
terms of $\accentset{\leftharpoonup}{\ell}$ to obtain shrinkers defined on
observables. Thanks to the sum-/max- decomposibility of asymptotic losses,
these shrinkers generate covariance estimates which are asymptotically optimal
in the rank-$r$ case.

######  Definition 3.3.

A sequence of shrinkers $\eta^{*}(\lambda|\star)=\eta_{n}^{*}(\lambda|\star)$
is asymptotically optimal under the disproportional limit
$\gamma_{n}\rightarrow 0$, variable spikes II, and loss
$\accentset{\leftharpoonup}{L}_{\star,k}$ if the formally optimal asymptotic
loss is achieved:

$\displaystyle\phantom{\,.}\accentset{\leftharpoonup}{L}_{F,k}(\Sigma,\widehat{\Sigma}_{\eta^{*}(\lambda|F)})$
$\displaystyle\xrightarrow{a.s.}\bigg{(}\sum_{i=1}^{r}\big{[}\accentset{\leftharpoonup}{\cal
L}_{F}^{1}(\accentset{\leftharpoonup}{\ell}_{i})\big{]}^{2}\bigg{)}^{1/2}\,,$
$\displaystyle\phantom{\,.}\accentset{\leftharpoonup}{L}_{O,k}(\Sigma,\widehat{\Sigma}_{\eta^{*}(\lambda|O)})$
$\displaystyle\xrightarrow{a.s.}\max_{1\leq i\leq
r}\accentset{\leftharpoonup}{\cal
L}_{O}^{1}(\accentset{\leftharpoonup}{\ell}_{i})\,,$
$\displaystyle\phantom{\,.}\accentset{\leftharpoonup}{L}_{N,k}(\Sigma,\widehat{\Sigma}_{\eta^{*}(\lambda|N)})$
$\displaystyle\xrightarrow{a.s.}\sum_{i=1}^{r}\accentset{\leftharpoonup}{\cal
L}_{N}^{1}(\accentset{\leftharpoonup}{\ell}_{i})\,.$

######  Theorem 3.4.

For $\star\in\\{F,N\\}$, define the following shrinkers through the formally
optimal shrinkers
$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\ell}|\star)$ of
Lemma 3.3:

$\displaystyle\eta^{*}(\lambda|\star)$
$\displaystyle=\accentset{\leftharpoonup}{\phi}_{n}^{-1}\big{(}\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\phi}_{n}(\lambda))|\star)\big{)}$
$\displaystyle=1+\sqrt{\gamma_{n}}\cdot\accentset{\leftharpoonup}{\eta}^{*}\Big{(}\accentset{\leftharpoonup}{\ell}\Big{(}\frac{\lambda-1}{\sqrt{\gamma_{n}}}\Big{)}\Big{|}\star\Big{)}\,.$
(3.11)

For the operator norm, define

$\displaystyle\eta^{*}(\lambda|O)$
$\displaystyle=\accentset{\leftharpoonup}{\phi}_{n}^{-1}\Big{(}\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\phi}_{n}(\lambda))\cdot
1_{\\{\accentset{\leftharpoonup}{\phi}_{n}(\lambda)>\tau_{n}\\}}\Big{)}$
$\displaystyle=1+\sqrt{\gamma_{n}}\cdot\accentset{\leftharpoonup}{\ell}\Big{(}\frac{\lambda-1}{\sqrt{\gamma_{n}}}\Big{)}\cdot
1_{\\{\lambda>1+\tau_{n}\sqrt{\gamma_{n}}\\}}\,,$ (3.12)

where $\tau_{n}=2+o(1)$ is a sufficiently slowly decaying sequence.

The shrinkers $\eta^{*}(\cdot|\star)$ are asymptotically optimal. Consider any
other sequence of shrinkers $\eta^{\circ}=\eta_{n}^{\circ}$; unless
$\eta^{\circ}$ has asymptotic shrinkage descriptors equal to those of
$\eta^{*}$, it has strictly larger asymptotic normalized loss. Thus, up to
asymptotically negligible perturbations, $\eta^{*}(\cdot|\star)$ is the unique
asymptotically admissible shrinker.

Empirically, for the operator norm, bulk edge thresholding performs well:

$\phantom{\,.}\eta^{*}(\lambda|O)=\accentset{\leftharpoonup}{\phi}_{n}^{-1}\big{(}\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\phi}_{n}(\lambda))\cdot
1_{\\{\accentset{\leftharpoonup}{\phi}_{n}(\lambda)>2\\}}\big{)}\,.$

This shrinker, which thresholds normalized eigenvalues exactly at two, is used
in the simulations visualized in Figure 2. Achieved loss is quite close to
$\accentset{\leftharpoonup}{{\cal
L}}_{O}^{1}(\accentset{\leftharpoonup}{\ell})$ on $(0,1]$. The slightly
elevated threshold in (3.12) is an artifact of the proof.

###### Proof.

By Lemma 3.1 and continuity of the partial inverse (3.10),

$\phantom{\,.}\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\lambda}_{i})\xrightarrow{a.s.}\max(\accentset{\leftharpoonup}{\ell}_{i},1)\,,\hskip
56.9055pt1\leq i\leq r\,.$ (3.13)

As $\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}|F)$
and $\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\ell}|N)$ are
continuous and on $(0,1]$ constant, (3.13) implies the asymptotic shrinkage
descriptors of $\eta^{*}(\cdot|F)$ and $\eta^{*}(\cdot|N)$ almost surely exist
and equal the formally optimal evaluates
$(\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}_{i}|F))_{i=1}^{r}$
and
$(\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}_{i}|N))_{i=1}^{r}$,
respectively. The formally optimal shrinker under operator norm loss
$\accentset{\leftharpoonup}{\eta}^{*}(\cdot|O)$ is discontinuous at the phase
transition $\accentset{\leftharpoonup}{\ell}=1$. For
$\accentset{\leftharpoonup}{\ell}_{i}>1$, existence and matching of the $i$-th
asymptotic shrinkage descriptor to
$\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\ell}_{i}|O)$
is immediate. Subcritical spiked eigenvalues converge to the bulk upper edge,
at a rate bounded by equation (1.6) of [17]: for
$\accentset{\leftharpoonup}{\ell}_{i}\leq 1$, almost surely eventually,
$\accentset{\leftharpoonup}{\lambda}_{i}\leq p^{-1/11}$. The $i$-th asymptotic
shrinkage descriptor is therefore zero.

Consider any other sequence of shrinkers $\eta^{\circ}=\eta^{\circ}_{n}$ and
rank-aware shrinkage estimators $\widehat{\Sigma}_{\eta^{\circ}}$. Unless this
sequence has asymptotic shrinkage descriptors
$(\accentset{\leftharpoonup}{\eta}_{i}^{\circ})_{i=1}^{r}$ identical to those
of $\eta^{*}$, its asymptotic loss is strictly larger. Namely, if there is a
subsequence with normalized limits
$\accentset{\leftharpoonup}{\eta}_{i}^{\circ}=\lim_{k\rightarrow\infty}\accentset{\leftharpoonup}{\phi}_{n_{k}}(\eta^{\circ}(\lambda_{i,n_{k}}))$
and
$\|(\accentset{\leftharpoonup}{\eta}_{i}^{\circ})_{i=1}^{r}-(\accentset{\leftharpoonup}{\eta}_{i}^{*})_{i=1}^{r}\|_{\ell_{2}^{r}}=\varepsilon>0$,
then by Lemmas 3.2 and 3.3, there is some $\delta=\delta(\varepsilon)>0$ such
that along this subsequence, the asymptotic loss under shrinker $\eta^{\circ}$
exceeds the asymptotic loss of $\eta^{*}$ by at least $\delta$.

Note that $\eta^{*}$ does not depend on the model parameters
$(\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r}$. Hence,
$\widehat{\Sigma}_{\eta^{*}}$ achieves optimal asymptotic performance at each
possible choice of $(\accentset{\leftharpoonup}{\ell}_{i})_{i=1}^{r}$. Except
for asymptotically negligible perturbations, $\eta^{*}(\cdot|\star)$ is the
unique shrinker asymptotically admissible under $L_{\star,k}$. ∎

######  Corollary 3.4.1.

Under the disproportional limit $\gamma_{n}\rightarrow 0$ and variable spikes
II, both the sample covariance $S$ and the rank-aware sample covariance

$\displaystyle S^{r}=\sum_{i=1}^{r}(\lambda_{i}-1)v_{i}v_{i}^{\prime}+I$
(3.14)

are asymptotically inadmissible for $L_{\star,k}$.

###### Proof.

This is an immediate consequence of Theorem 3.4. Still, we sketch a direct
argument for the Frobenius-norm case. Let $W$ denote projection onto the
combined span of $(u_{i})_{i=1}^{r}$ and $(v_{i})_{i=1}^{r}$. Then, for the
sample covariance matrix,

$\displaystyle\phantom{\,.}\|\Sigma-S\|_{F}^{2}$ $\displaystyle=$
$\displaystyle\|W^{\prime}(\Sigma-S)W\|_{F}^{2}+\|(I-W)(I-S)(I-W)\|_{F}^{2}\,,$

while for the rank-aware estimate,

$\displaystyle\phantom{\,.}\|\Sigma-S^{r}\|_{F}^{2}$ $\displaystyle=$
$\displaystyle\|W(\Sigma-S^{r})W\|_{F}^{2}+\|(I-W)(I-S^{r})(I-W)\|_{F}^{2}$
$\displaystyle=$ $\displaystyle\|W(\Sigma-S^{r})W\|_{F}^{2}\,.$

The terms $\|W^{\prime}(\Sigma-S)W\|_{F}^{2}$ and
$\|W(\Sigma-S^{r})W\|_{F}^{2}$ tend to a common limit, so it suffices to study
the rank-aware case. By Lemma 3.2,
$\|\Sigma-S^{r}\|_{F}^{2}\xrightarrow{a.s.}\sum_{i=1}^{r}[\accentset{\leftharpoonup}{{\cal
L}}(\accentset{\leftharpoonup}{\ell}_{i},\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}_{i}))]^{2}$,
and using equation (3.6) one may verify that

$\phantom{\,.}[\accentset{\leftharpoonup}{\cal
L}_{F}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}))]^{2}-[\accentset{\leftharpoonup}{\cal
L}_{F}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta}^{*})]^{2}=(\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})-\accentset{\leftharpoonup}{\eta}^{*})^{2}\geq
0\,.$

Over the range $\accentset{\leftharpoonup}{\ell}>1$,
$\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})-\accentset{\leftharpoonup}{\eta}^{*}=2/\accentset{\leftharpoonup}{\ell}$,
while over $\accentset{\leftharpoonup}{\ell}\leq 1$,
$\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})-\accentset{\leftharpoonup}{\eta}^{*}=2$;
thus the inequality is strict. ∎

###  3.5 Performance in the $\gamma_{n}\rightarrow 0$ Limit

Figure 1 depicts optimal shrinkers (left) and corresponding asymptotic losses
(right, in the rank-one case $r=1$). In the left-hand panel, the red curve
marks the diagonal
$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\lambda})=\accentset{\leftharpoonup}{\lambda}$,
corresponding to no shrinkage. For each loss function we consider, the optimal
shrinker $\accentset{\leftharpoonup}{\eta}^{*}(\cdot|\star)$ lies below the
diagonal. All optimal shrinkers vanish below the bulk edge,
$\accentset{\leftharpoonup}{\lambda}\leq 2$.

Below the phase transition occuring at $\accentset{\leftharpoonup}{\ell}=1$,
sample and population eigenvectors are asymptotically orthogonal. In that
region, it is futile to use empirical eigenvectors to model low-rank
structure—they are pure noise. Therefore, to achieve optimal loss, we simply
take $\eta=0$. According to (3.2), $\accentset{\leftharpoonup}{\ell}\leq 1$ if
and only if $\accentset{\leftharpoonup}{\lambda}\leq 2$, hence all optimal
rules vanish for $\accentset{\leftharpoonup}{\lambda}<2$. Over the restricted
range $0<\accentset{\leftharpoonup}{\ell}<1$, optimal rules are of course not
unique; we also obtain optimality over that range by simple bulk-edge hard
thresholding of empirical eigenvalues,
$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\lambda})=\accentset{\leftharpoonup}{\lambda}\cdot
1_{\\{\accentset{\leftharpoonup}{\lambda}>2\\}}$.)

The right-hand panel compares performances under various loss functions of the
standard estimator $S^{r}$ (dotted lines) and optimal estimators (solid
lines). Asymptotic losses of the standard estimator are strictly larger than
that of optimal estimators for all $\accentset{\leftharpoonup}{\ell}$—near
$\accentset{\leftharpoonup}{\ell}=1$, standard loss is far larger. As
$\accentset{\leftharpoonup}{\ell}\rightarrow 0^{+}$, optimal losses tend to
zero, while standard losses tend to $2$.

######  Definition 3.4.

The (absolute) regret of a decision rule $\accentset{\leftharpoonup}{\eta}$ is
defined as

$\phantom{\,.}\accentset{\leftharpoonup}{{\cal
R}}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})=\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})-\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta}^{*})\,.$

The possible improvement of a decision rule $\accentset{\leftharpoonup}{\eta}$
is $\accentset{\leftharpoonup}{{\cal
I}}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})=\accentset{\leftharpoonup}{{\cal
R}}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})/\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})$,
i.e., the fractional amount by which performance improves by switching to the
optimal rule.

Losses of $S^{r}$ in the right-hand panel of Figure 1 are well above losses of
optimal estimators below the phase transition
$\accentset{\leftharpoonup}{\ell}\leq 1$; the limit
$\accentset{\leftharpoonup}{\ell}\rightarrow 0^{+}$ produces maximal absolute
regret, $2$, for each of these losses. For example, with operator norm loss,
$\accentset{\leftharpoonup}{\cal
L}_{O}(0^{+},\accentset{\leftharpoonup}{\lambda})=2$, while
$\accentset{\leftharpoonup}{\cal
L}_{O}(0^{+},\accentset{\leftharpoonup}{\eta}^{*})=0$, giving absolute regret
$\accentset{\leftharpoonup}{{\cal
R}}_{O}(0^{+},\accentset{\leftharpoonup}{\lambda})=2$ and possible improvement
$\accentset{\leftharpoonup}{{\cal
I}}_{O}(0^{+},\accentset{\leftharpoonup}{\lambda})=1$ (100% of the standard
loss is avoidable). Similarly, with nuclear norm loss, we have
$\accentset{\leftharpoonup}{{\cal
R}}_{N}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda})=2$
for $\accentset{\leftharpoonup}{\ell}\leq 1$, but
$\accentset{\leftharpoonup}{{\cal
I}}_{N}(0,\accentset{\leftharpoonup}{\lambda})=1$ (100% of the standard loss
is avoidable).

Norm | $\accentset{\leftharpoonup}{{\cal R}}_{\star}(0^{+},\accentset{\leftharpoonup}{\lambda})$ | $\accentset{\leftharpoonup}{{\cal R}}_{\star}(1,\accentset{\leftharpoonup}{\lambda})$ | $\accentset{\leftharpoonup}{{\cal I}}_{\star}(0^{+},\accentset{\leftharpoonup}{\lambda})$ | $\accentset{\leftharpoonup}{{\cal I}}_{\star}(1,\accentset{\leftharpoonup}{\lambda})$
---|---|---|---|---
Frobenius | $2$ | $\sqrt{5}-1$ |  $100$% | 55%
Operator | 2 | 1 |  $100$% | 50%
Nuclear | 2 | 2 |  $100$% | 66%
Table 2: Regret and Improvement, $\gamma_{n}\rightarrow 0$. Absolute Regret
$\accentset{\leftharpoonup}{{\cal R}}$ and Possible Improvement
$\accentset{\leftharpoonup}{{\cal I}}$ of the standard rank-aware estimator
$S^{r}$ (equivalently,
$\accentset{\leftharpoonup}{\eta}=\accentset{\leftharpoonup}{\lambda}$) near
zero and exactly at the phase transition $\accentset{\leftharpoonup}{\ell}=1$.

Figure 1: Optimal shrinkers and losses, $\gamma_{n}\rightarrow 0$. Left:
optimal shrinkage functions. Right: losses of optimal shrinkers (solid) and of
the standard estimator $S^{r}$ under Frobenius (blue), operator (orange),
nuclear (green) norms. Figure 2: Monte-Carlo simulations, small $\gamma_{n}$.
Average over 50 realizations of losses under three norms, both for the
standard and asymptotically optimal estimators. Here, $p=1{,}000$ and
$n=100{,}000$, so $\gamma_{n}=.01$.

##  4 Covariance Estimation as $\gamma_{n}\rightarrow\infty$

###  4.1 The Variable-Spike, $\gamma_{n}\rightarrow\infty$ Limit

We now turn to the dual situation, $\gamma_{n}\rightarrow\infty$. To expose
phase transition phenomena, we consider variable spiked eigenvalues of the
form

$\displaystyle[\textbf{III}]\hskip
42.67912pt\ell_{i}=\ell_{i;n}=1+(\accentset{\rightharpoonup}{\ell}_{i}+o(1))\gamma_{n}\,,\hskip
56.9055pt1\leq i\leq r\,,$

where $(\accentset{\rightharpoonup}{\ell}_{i})_{i=1}^{r}$ are fixed, positive,
and distinct parameters. Correspondingly, we study the normalized empirical
eigenvalues

$\accentset{\rightharpoonup}{\lambda}_{i}=\accentset{\rightharpoonup}{\lambda}_{i,n}=\frac{\lambda_{i}}{\gamma_{n}}\,.$
(4.1)

The next theorem provides the $\gamma_{n}\rightarrow\infty$ analogues of
eigenvalue inflation (1.2) and eigenvalue rotation (1.4).

######  Lemma 4.1.

(Benaych-Georges and Rao Nadakuditi [7], Shen et al. [30]) Under the
disproportional limit $\gamma_{n}\rightarrow\infty$ and variable spikes III,
the leading empirical eigenvalues of $S$ satisfy

$\phantom{\,.}\accentset{\rightharpoonup}{\lambda}_{i}\xrightarrow{a.s.}\accentset{\rightharpoonup}{1}+\accentset{\rightharpoonup}{\ell}_{i}\,.$
(4.2)

The angles between empirical eigenvectors and the corresponding population
eigenvectors have limits

$\begin{aligned} \phantom{\,.}\hskip 28.45274pt&|\langle
u_{i},v_{j}\rangle|\xrightarrow{a.s.}\delta_{ij}\cdot\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\rightharpoonup}{\ell}_{i})\,,\end{aligned}\qquad\hskip
28.45274pt1\leq i,j\leq r\,,$ (4.3)

where the cosine function is given by

$\displaystyle\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\rightharpoonup}{\ell})=\frac{\accentset{\rightharpoonup}{\ell}}{1+\accentset{\rightharpoonup}{\ell}}\,.$
(4.4)

No phase transition appears in this framing of the
$\gamma_{n}\rightarrow\infty$ setting; for example,
$\partial\accentset{\rightharpoonup}{\lambda}/\partial\accentset{\rightharpoonup}{\ell}=1$
and
$\partial\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}/\partial\accentset{\rightharpoonup}{\ell}>0$
for all $\accentset{\rightharpoonup}{\ell}>0$, while as $\gamma_{n}\rightarrow
0$, we had
$\partial\accentset{\leftharpoonup}{\lambda}/\partial\accentset{\leftharpoonup}{\ell}=0$
and
$\partial\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}/\partial\accentset{\leftharpoonup}{\ell}=0$
for $0<\accentset{\leftharpoonup}{\ell}<1$.

Recall that for $i\leq\min(n,p)$,
$\lambda_{i}(X^{\prime}X)=\lambda_{i}(XX^{\prime})$; one might therefore
expect that the phase transition as $\gamma_{n}\rightarrow 0$ would manifest
here as well as a clear phase transition. Such a transition for the eigenvalue
does occur under alternative scalings and coordinates to
$\accentset{\rightharpoonup}{\ell}$, $\accentset{\rightharpoonup}{\lambda}$.
Indeed, remaining in the $\gamma_{n}\rightarrow\infty$ limit, consider
$\tilde{\ell}_{i}=\tilde{\ell}_{i,n}=\gamma_{n}(1+\accentset{\leftharpoonup}{\ell}_{i}(1+o(1))\gamma_{n}^{-1/2})$.
Leveraging $\lambda_{i}(X^{\prime}X)=\lambda_{i}(XX^{\prime})$ and earlier
$\gamma_{n}\rightarrow 0$ results, a phase transition occurs at
$\accentset{\leftharpoonup}{\ell}_{i}=1$. This transition, however, tells us
nothing of the eigenvectors: the properties of eigenvectors of $X^{\prime}X$
and $XX^{\prime}$ are quite different, and on this scale, leading empirical
eigenvectors are asymptotically decorrelated from their population
counterparts. By adopting spikes III, we work on a far coarser scale, one
where eigenvectors correlate but with no visible phase transition.

###  4.2 Asymptotic Loss and Unique Admissibility in the
$\gamma_{n}\rightarrow\infty$ Limit

Under variable spikes III, the population covariance now explodes. As losses
similarly explode, we consider rescaled losses:

$\accentset{\rightharpoonup}{L}_{\star,k}(\Sigma,\widehat{\Sigma})=\frac{L_{\star,k}(\Sigma,\widehat{\Sigma})}{\gamma_{n}}\,.$

Let $\accentset{\rightharpoonup}{\phi}_{n}(\lambda)=\lambda/\gamma_{n}$ denote
the mapping to this new coordinate system. Thus, spikes III and (4.1) may be
written as
$\accentset{\rightharpoonup}{\lambda}_{i}=\accentset{\rightharpoonup}{\phi}_{n}(\lambda_{i})$
and
$\accentset{\rightharpoonup}{\phi}_{n}(\ell_{i})\rightarrow\accentset{\rightharpoonup}{\ell}_{i}$.

######  Definition 4.1.

Let $\eta=\eta_{n}$ denote a sequence of shrinkers, possibly varying with $n$.
Suppose that under the disproportional limit $\gamma_{n}\rightarrow\infty$ and
varying-spikes III, the sequences of normalized shrinker outputs induced by
rescaling converge:

$\phantom{\,.}\accentset{\rightharpoonup}{\phi}_{n}(\eta(\lambda_{i}))\xrightarrow{a.s.}\accentset{\rightharpoonup}{\eta}_{i}\,,\hskip
56.9055pt1\leq i\leq r\,.$

We call the limits $(\accentset{\rightharpoonup}{\eta}_{i})_{i=1}^{r}$ the
asymptotic shrinkage descriptors.

######  Lemma 4.2.

Let $\eta$ denote a sequence of shrinkers with asymptotic shrinkage
descriptors $(\accentset{\rightharpoonup}{\eta}_{i})_{i=1}^{r}$ under the
disproportional limit $\gamma_{n}\rightarrow\infty$ limit and varying-spikes
III. Each loss $\accentset{\rightharpoonup}{L}_{\star,1}$ then converges
almost surely to a deterministic limit:

$\accentset{\rightharpoonup}{L}_{\star,1}(\Sigma,\widehat{\Sigma}_{\eta})\xrightarrow{a.s.}\accentset{\rightharpoonup}{\cal
L}_{\star}((\accentset{\rightharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\rightharpoonup}{\eta}_{i})_{i=1}^{r}),\qquad\star\in\\{F,O,N\\}\,.$

The asymptotic loss is sum- or max- decomposable into $r$ terms involving
matrix norms applied to the $2\times 2$ matrices $\widetilde{A}$ and
$\widetilde{B}$ introduced in Lemma 3.2. With
$\accentset{\rightharpoonup}{\ell}_{i}$ denoting a spiked eigenvalue and
$\accentset{\rightharpoonup}{c}(\accentset{\rightharpoonup}{\ell})$ the
limiting cosine in (4.4), the decompositions are

$\displaystyle\phantom{\,,}\accentset{\rightharpoonup}{\cal
L}_{F}((\accentset{\rightharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\rightharpoonup}{\eta})_{i=1}^{r})$
$\displaystyle=\bigg{(}\sum_{i=1}^{r}\big{[}L_{F,1}\big{(}\widetilde{A}(\accentset{\rightharpoonup}{\ell}_{i}),\widetilde{B}(\accentset{\rightharpoonup}{\eta}_{i},\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}(\accentset{\rightharpoonup}{\ell}_{i}))\big{)}\big{]}^{2}\bigg{)}^{1/2}\,,$
$\displaystyle\accentset{\rightharpoonup}{\cal
L}_{O}((\accentset{\rightharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\rightharpoonup}{\eta})_{i=1}^{r})$
$\displaystyle=\max_{1\leq i\leq
r}L_{O,1}\big{(}\widetilde{A}(\accentset{\rightharpoonup}{\ell}_{i}),\widetilde{B}(\accentset{\rightharpoonup}{\eta}_{i},\accentset{\rightharpoonup}{c}(\accentset{\rightharpoonup}{\ell}_{i}))\big{)}\,,$
$\displaystyle\accentset{\rightharpoonup}{\cal
L}_{N}((\accentset{\rightharpoonup}{\ell}_{i})_{i=1}^{r},(\accentset{\rightharpoonup}{\eta})_{i=1}^{r})$
$\displaystyle=\sum_{i=1}^{r}L_{N,1}\big{(}\widetilde{A}(\accentset{\rightharpoonup}{\ell}_{i}),\widetilde{B}(\accentset{\rightharpoonup}{\eta}_{i},\accentset{\rightharpoonup}{c}(\accentset{\rightharpoonup}{\ell}_{i}))\big{)}\,.$

Proof of this lemma is similar to that of Lemma 3.2 and omitted. Note that
only pivot $\Delta_{1}$ is considered as $S$ and $\widehat{\Sigma}_{\eta}$
have $p-n$ eigenvalues equal to zero. As a simple example, the asymptotic
shrinkage descriptors of the rank-aware sample covariance estimator
$S^{r}=\sum_{i=1}^{r}(\lambda_{i}-1)v_{i}v_{i}^{\prime}+I$ are
$\accentset{\rightharpoonup}{\eta}_{i}=\accentset{\rightharpoonup}{\lambda}(\accentset{\rightharpoonup}{\ell}_{i})$.
Squared asymptotic loss evaluates to (suppressing the subscript of
$\accentset{\rightharpoonup}{\ell}_{1}$)

$[\accentset{\rightharpoonup}{\cal
L}_{F}(\accentset{\rightharpoonup}{\ell},\accentset{\rightharpoonup}{\lambda})]^{2}=(\accentset{\rightharpoonup}{\ell}-\accentset{\rightharpoonup}{\lambda}\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\rightharpoonup}{\ell}))^{2}+\accentset{\rightharpoonup}{\lambda}^{2}(1-\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}^{4}(\accentset{\rightharpoonup}{\ell}))\,.$
(4.5)

By Theorem 3.4,
$\accentset{\rightharpoonup}{\lambda}\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\rightharpoonup}{\ell})=\accentset{\rightharpoonup}{\ell}$,
while
$\accentset{\rightharpoonup}{\lambda}^{2}(1-\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}^{4}(\accentset{\rightharpoonup}{\ell}))=(1+2\accentset{\rightharpoonup}{\ell})$,
so $[\accentset{\rightharpoonup}{\cal
L}_{F}(\accentset{\rightharpoonup}{\ell},\accentset{\rightharpoonup}{\lambda})]^{2}=(1+2\accentset{\rightharpoonup}{\ell})$.
Asymptotic losses of $S^{1}$ under each norm are collected below in Table 3,
to later facilitate comparison with optimal shrinkage.

Norm | $\accentset{\rightharpoonup}{{\cal L}}_{\star}(\accentset{\rightharpoonup}{\ell},\accentset{\rightharpoonup}{\lambda})$
---|---
Frobenius | $\sqrt{1+2\accentset{\rightharpoonup}{\ell}}$
Operator | $\left(1+\sqrt{1+4\accentset{\rightharpoonup}{\ell}}\right)/2$
Nuclear | $\sqrt{1+4\accentset{\rightharpoonup}{\ell}}$
Table 3: Asymptotic Loss $\accentset{\rightharpoonup}{\cal L}_{\star}$ of the
standard rank-aware estimator $S^{1}$.

The intermediate form (4.5) is symbolically isomorphic to the intermediate
form (3.6) seen earlier in the $\gamma_{n}\rightarrow 0$ case (under
replacement of $\overset{\leftharpoonup}{\;}$’s by
$\overset{\rightharpoonup}{\;}$’s), suggesting that the path to optimality
will again lead to eigenvalue shrinkage.

######  Definition 4.2.

In the $\gamma_{n}\rightarrow\infty$ limit, the formally optimal asymptotic
loss in the rank-1 setting is

$\accentset{\rightharpoonup}{\cal
L}_{\star}^{1}(\accentset{\rightharpoonup}{\ell})\equiv\min_{\vartheta}\accentset{\rightharpoonup}{\cal
L}_{\star}(\accentset{\rightharpoonup}{\ell},\vartheta),\qquad\star\in\\{F,O,N\\}.$

A formally optimal shrinker is a function
$\accentset{\rightharpoonup}{\eta}(\cdot|\star):\mathbb{R}\mapsto\mathbb{R}$
achieving $\accentset{\rightharpoonup}{\cal
L}_{\star}^{1}(\accentset{\rightharpoonup}{\ell})$:

$\accentset{\rightharpoonup}{\eta}(\accentset{\rightharpoonup}{\ell}|\star)=\underset{\vartheta}{\mbox{argmin}}\,\accentset{\rightharpoonup}{\cal
L}_{\star}(\accentset{\rightharpoonup}{\ell},\vartheta),\qquad\accentset{\rightharpoonup}{\ell}>0,\qquad\star\in\\{F,O,N\\}.$

In complete analogy with Lemma 3.3, we have explicit forms of formally optimal
shrinkers.

######  Lemma 4.3.

Formally optimal shrinkers (defined analogously to Definition 3.3) and
corresponding losses are given by

$\displaystyle\accentset{\rightharpoonup}{\eta}^{*}(\accentset{\rightharpoonup}{\ell}|F)=\frac{\accentset{\rightharpoonup}{\ell}^{2}}{1+\accentset{\rightharpoonup}{\ell}}\,,$
$\displaystyle[\accentset{\rightharpoonup}{\cal
L}_{F}^{1}(\accentset{\rightharpoonup}{\ell})]^{2}=\frac{\accentset{\rightharpoonup}{\ell}^{2}(2\accentset{\rightharpoonup}{\ell}+1)}{(\accentset{\rightharpoonup}{\ell}+1)^{2}}\,,$
$\displaystyle\accentset{\rightharpoonup}{\eta}^{*}(\accentset{\rightharpoonup}{\ell}|O)=\accentset{\rightharpoonup}{\ell}\,,$
$\displaystyle\accentset{\rightharpoonup}{\cal
L}_{O}^{1}(\accentset{\rightharpoonup}{\ell})=\frac{\accentset{\rightharpoonup}{\ell}}{(1+\accentset{\rightharpoonup}{\ell})^{1/2}}\,,$
(4.6)
$\displaystyle\accentset{\rightharpoonup}{\eta}^{*}(\accentset{\rightharpoonup}{\ell}|N)=\accentset{\rightharpoonup}{\ell}\bigg{(}\frac{\accentset{\rightharpoonup}{\ell}-1}{\accentset{\rightharpoonup}{\ell}+1}\bigg{)}_{+}$
$\displaystyle\accentset{\rightharpoonup}{\cal
L}_{N}^{1}(\accentset{\rightharpoonup}{\ell})=\accentset{\rightharpoonup}{\ell}\cdot\bigg{[}1_{\\{\accentset{\rightharpoonup}{\ell}<1\\}}+1_{\\{\accentset{\rightharpoonup}{\ell}>1\\}}\cdot\frac{2\cdot\sqrt{\accentset{\rightharpoonup}{\ell}}}{\accentset{\rightharpoonup}{\ell}+1}\bigg{]}\,.$

###### Proof.

Asymptotic losses are functions of the limiting formulas for eigenvalue
inflation and eigenvector rotation. Thus, by the proof Lemma 3.3, in
particular (3.9),

$\displaystyle\accentset{\rightharpoonup}{\eta}^{*}(\accentset{\rightharpoonup}{\ell}|F)=\accentset{\rightharpoonup}{\ell}\accentset{\rightharpoonup}{c}\rule{0.0pt}{7.0pt}^{2}(\accentset{\rightharpoonup}{\ell})\,,$
$\displaystyle\accentset{\rightharpoonup}{\eta}^{*}(\accentset{\rightharpoonup}{\ell}|O)=\accentset{\rightharpoonup}{\ell}\,,$
$\displaystyle\accentset{\rightharpoonup}{\eta}^{*}(\accentset{\rightharpoonup}{\ell}|N)=\accentset{\rightharpoonup}{\ell}(1-2\accentset{\rightharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(\accentset{\rightharpoonup}{\ell}))_{+}\,.$
(4.7)

Substitution of (4.4) yields the left-hand column of (4.3). In parallel
fashion, asymptotic losses are isomorphic:

$\accentset{\leftharpoonup}{\cal
L}_{F}^{*}(\accentset{\leftharpoonup}{\ell})=\accentset{\leftharpoonup}{\ell}^{2}\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(2-\accentset{\leftharpoonup}{s}\rule{0.0pt}{7.0pt}^{2})\,,\qquad\accentset{\rightharpoonup}{\cal
L}_{F}^{*}(\accentset{\rightharpoonup}{\ell})=\accentset{\rightharpoonup}{\ell}^{2}\accentset{\rightharpoonup}{s}\rule{0.0pt}{7.0pt}^{2}(2-\accentset{\rightharpoonup}{s}\rule{0.0pt}{7.0pt}^{2})\,.$

∎

######  Theorem 4.4.

Define the following shrinkers through the formally optimal shrinkers
$\accentset{\rightharpoonup}{\eta}(\accentset{\rightharpoonup}{\ell}|\star)$
of Lemma 4.3:

$\displaystyle\phantom{\,.}\eta^{*}(\lambda|\star)=\eta_{n}^{*}(\lambda|\star)$
$\displaystyle=\accentset{\rightharpoonup}{\phi}_{n}^{-1}\big{(}\accentset{\rightharpoonup}{\eta}(\accentset{\rightharpoonup}{\phi}_{n}(\lambda)-1|\star)\big{)}$
$\displaystyle=\gamma_{n}\accentset{\rightharpoonup}{\eta}(\lambda/\gamma_{n}-1|\star)\,.$

Under $\gamma_{n}\rightarrow\infty$ and variable spikes III,
$\eta^{*}(\cdot|\star)$ achieves the optimal asymptotic normalized loss. Up to
asymptotically negligible perturbations, $\eta^{*}(\cdot|\star)$ is the unique
asymptotically admissible shrinker for
$\accentset{\rightharpoonup}{L}_{\star,1}$.

All formally optimal shrinkers are continuous. The proof of Theorem 4.4 is
analogous to that of Theorem 3.4 and is omitted.

######  Corollary 4.4.1.

Under $\gamma_{n}\rightarrow\infty$ and variable spikes III, both the sample
covariance $S$ and the rank-aware sample covariance $S^{r}$ are asymptotically
inadmissible for $L_{\star,1}$.

###  4.3 Performance in the $\gamma_{n}\rightarrow\infty$ Limit

Figure 3 depicts optimal shrinkers (left) and corresponding asymptotic losses
(right, in the rank-one case $r=1$), similarly to Figure 1. In the left-hand
panel, the red curve again marks no shrinkage,
$\accentset{\rightharpoonup}{\eta}(\accentset{\rightharpoonup}{\lambda})=\accentset{\rightharpoonup}{\lambda}$.
Each optimal shrinker $\accentset{\rightharpoonup}{\eta}^{*}(\cdot|\star)$
lies below the diagonal—especially for small
$\accentset{\rightharpoonup}{\lambda}$. Normalized (non-zero) eigenvalues
converge to $\accentset{\rightharpoonup}{\lambda}=1$; optimal shrinkers all
vanish below $\accentset{\rightharpoonup}{\lambda}\leq 1$.

The right-hand panel compares performances under various loss functions of the
standard estimator $S^{r}$ (dotted lines) and the optimal estimators (solid
lines). Asymptotic losses of the standard estimator are strictly larger than
that of optimal estimators for all $\accentset{\rightharpoonup}{\ell}$. Under
Frobenius and nuclear norms, optimal shrinkage also outperforms the fixed
shrinker $\eta(\lambda)=\lambda-1$. As
$\accentset{\rightharpoonup}{\ell}\rightarrow 0^{+}$, optimal losses
$\accentset{\rightharpoonup}{\cal
L}_{\star}^{*}(\accentset{\rightharpoonup}{\ell})$ tend to zero, while
standard losses tend to 1. The maximal relative regret for $S^{r}$ is
unbounded.

For example, with operator norm loss, $\accentset{\rightharpoonup}{\cal
L}_{O}(1,\accentset{\rightharpoonup}{\lambda})=(1+\sqrt{5})/2$, while
$\accentset{\rightharpoonup}{\cal
L}_{O}(1,\accentset{\rightharpoonup}{\eta}^{*})=1/\sqrt{2}$. The absolute
regret is
$\accentset{\rightharpoonup}{\mathcal{R}}_{O}(1,\accentset{\rightharpoonup}{\lambda})=.91$,
and 57% improvement in loss is possible at
$\accentset{\rightharpoonup}{\ell}=1$. The maximal possible relative
improvement is 100%: at $\accentset{\rightharpoonup}{\ell}=0$, all the loss
incurred by the standard estimator is avoidable. Under Frobenius norm,
$\accentset{\rightharpoonup}{\cal
L}_{F}(1,\accentset{\rightharpoonup}{\lambda})=\sqrt{3}$,
$\accentset{\rightharpoonup}{\cal
L}_{F}(1,\accentset{\rightharpoonup}{\eta}^{*})=\sqrt{3}/2$, and
$\accentset{\rightharpoonup}{\mathcal{R}}_{F}(1,\accentset{\rightharpoonup}{\lambda})=\sqrt{3}/2$.
There is 50% possible improvement over the standard estimator at
$\accentset{\rightharpoonup}{\ell}=1$, and fully $100$% of the standard loss
is avoidable using shrinkage at $\accentset{\rightharpoonup}{\ell}=0$.

Norm | $\accentset{\rightharpoonup}{\mathcal{R}}_{\star}(0^{+},\accentset{\rightharpoonup}{\lambda})$ | $\accentset{\rightharpoonup}{\mathcal{R}}_{\star}(1,\accentset{\rightharpoonup}{\lambda})$ | $\accentset{\rightharpoonup}{{\cal I}}_{\star}(0^{+},\accentset{\rightharpoonup}{\lambda})$ | $\accentset{\rightharpoonup}{{\cal I}}_{\star}(1,\accentset{\rightharpoonup}{\lambda})$
---|---|---|---|---
Frobenius | 1 | $\sqrt{3}/2$ | 100% | 50%
Operator | 1 | 2.52 | 100% | 57%
Nuclear | 1 | $\sqrt{5}-1$ | 100% | 56%
Table 4: Regret and Improvement, $\gamma_{n}\rightarrow\infty$. Absolute
Regret $\accentset{\rightharpoonup}{\mathcal{R}}$ and possible relative
improvement $\accentset{\rightharpoonup}{{\cal I}}$ of the standard rank-aware
estimator $S^{r}$ (equivalently,
$\accentset{\rightharpoonup}{\eta}=\accentset{\rightharpoonup}{\lambda}$) near
zero and exactly at $\accentset{\rightharpoonup}{\ell}=1$.

Figure 3: Optimal shrinkers and losses, $\gamma_{n}\rightarrow\infty$. Left:
optimal shrinkage functions. Right: losses of optimal shrinkers (solid) and of
the standard estimator $S^{r}$, under Frobenius (blue), operator (orange), and
nuclear (green) norms. Figure 4: Monte-Carlo simulations, large $\gamma_{n}$.
Averages over 50 realizations of losses under three norms for the standard and
asymptotically optimal estimators. Here, $p=10{,}000$ and $n=100$, so
$\gamma_{n}=100$.

##  5 Optimal Hard Thresholding

A natural alternative to optimal shrinkage is simple hard thresholding,
applying the zero-based shrinker $H_{\tau}(\theta)=\theta\cdot
1_{\\{\theta\geq\tau\\}}+1_{\\{\theta<\tau\\}}$. In original coordinates,
$\widehat{\Sigma}_{H_{\tau}}=\sum_{i=1}^{r}H_{\tau_{n}}(\lambda_{i})v_{i}v_{i}^{\prime}+I$;
in rescaled coordinates, this corresponds to hard thresholding the normalized
eigenvalues at the below levels:

  * •

$\accentset{\leftharpoonup}{\tau}_{n}=\accentset{\leftharpoonup}{\phi}_{n}(\tau_{n})$,
i.e.
$H_{\accentset{\leftharpoonup}{\tau}_{n}}(\accentset{\leftharpoonup}{\lambda}_{j})$
($\gamma_{n}\rightarrow 0$),

  * •

$\accentset{\rightharpoonup}{\tau}_{n}=\accentset{\rightharpoonup}{\phi}_{n}(\tau_{n})$,
i.e.
$H_{\accentset{\rightharpoonup}{\tau}_{n}}(\accentset{\rightharpoonup}{\lambda}_{j})$
($\gamma_{n}\rightarrow\infty$).

Under the appropriate varying-spikes model II or III, it makes sense to choose
threshold sequences $\tau_{n}$ that, after normalization, are essentially
constant:

  * •

$\accentset{\leftharpoonup}{\tau}_{n}=\accentset{\leftharpoonup}{\tau}\cdot(1+o(1))$
($\gamma_{n}\rightarrow 0$),

  * •

$\accentset{\rightharpoonup}{\tau}_{n}=\accentset{\rightharpoonup}{\tau}\cdot(1+o(1))$
($\gamma_{n}\rightarrow\infty$).

It may seem natural to place the threshold exactly at the bulk edge. We
propose, however, a performance-dominating alternative, with thresholds
notably beyond the bulk edge. See Table 5.

Norm | $\accentset{\leftharpoonup}{\tau}(\star)$ | $\accentset{\rightharpoonup}{\tau}(\star)$
---|---|---
Frobenius | $4/\sqrt{3}$ | $1+\sqrt{2}$
Operator | $\sqrt{2(1+\sqrt{2})}$ | $3$
Nuclear | $6/\sqrt{5}$ | $3+\sqrt{5}$
Bulk Edge | 2 | 1
Table 5: Optimal thresholding parameters. Thresholds in rows 2 through 4 are
considerably beyond the bulk edge in row 5. To use these (normalized)
thresholds with unnormalized eigenvalues, back-translate: use
$\tau_{n}=\accentset{\leftharpoonup}{\phi}_{n}^{-1}(\accentset{\leftharpoonup}{\tau})$
as $\gamma_{n}\rightarrow 0$ and
$\tau_{n}=\accentset{\rightharpoonup}{\phi}_{n}^{-1}(\accentset{\rightharpoonup}{\tau})$
as $\gamma_{n}\rightarrow\infty$.

######  Definition 5.1.

We say that $\accentset{\leftharpoonup}{\tau}$ is the unique admissible
normalized threshold for asymptotic loss $\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\cdot)$ as $\gamma_{n}\rightarrow
0$ if, for any other deterministic normalized threshold
$\accentset{\leftharpoonup}{\nu}$, we have

$\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},H_{\accentset{\leftharpoonup}{\tau}})\leq\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},H_{\accentset{\leftharpoonup}{\nu}})\,,\hskip
56.9055pt\forall\,\accentset{\leftharpoonup}{\ell}\geq 0\,,$

with strict inequality at some $\accentset{\leftharpoonup}{\ell}^{\prime}\geq
0$. We similarly define the unique admissible normalized threshold
$\accentset{\rightharpoonup}{\tau}$ for $\accentset{\rightharpoonup}{{\cal
L}}_{\star}(\accentset{\rightharpoonup}{\ell},\cdot)$ as
$\gamma_{n}\rightarrow\infty$.

######  Theorem 5.1.

For $\star\in\\{F,O,N\\}$, there are unique admissible thresholds
$\accentset{\leftharpoonup}{\tau}(\star)$ and
$\accentset{\rightharpoonup}{\tau}(\star)$ for asymptotic losses
$\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\eta})$
and $\accentset{\rightharpoonup}{\cal
L}_{\star}(\accentset{\rightharpoonup}{\ell},\accentset{\rightharpoonup}{\eta})$,
respectively, given in Table 5.

###### Proof.

Consider $\gamma_{n}\rightarrow 0$.
$\accentset{\leftharpoonup}{\ell}\mapsto\accentset{\leftharpoonup}{\cal
L}(\accentset{\leftharpoonup}{\ell},0)$ and
$\accentset{\leftharpoonup}{\ell}\mapsto\accentset{\leftharpoonup}{\cal
L}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda})$
denote the asymptotic losses of the null
$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\lambda})=0$ and
identity
$\accentset{\leftharpoonup}{\eta}(\accentset{\leftharpoonup}{\lambda})=\accentset{\leftharpoonup}{\lambda}$
rules, respectively. In each case of Table 5, there is an unique crossing
point $\accentset{\leftharpoonup}{\theta}(\star)$ exceeding the bulk edge such
that

$\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},0)<\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda})\,,\quad\accentset{\leftharpoonup}{\ell}<\accentset{\leftharpoonup}{\theta}(\star)\,;\qquad\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},0)<\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda})\,,\quad\accentset{\leftharpoonup}{\ell}>\accentset{\leftharpoonup}{\theta}(\star)\,.$

Equality occurs only for
$\accentset{\leftharpoonup}{\ell}=\accentset{\leftharpoonup}{\theta}(\star)$.
Calculations of $\accentset{\leftharpoonup}{\theta}(\star)$ are omitted.
Define
$\accentset{\leftharpoonup}{\tau}(\star)=\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\theta}(\star))$.
Note that

$\displaystyle
H_{\accentset{\leftharpoonup}{\tau}(\star)}(\accentset{\leftharpoonup}{\lambda})\xrightarrow[]{a.s.}0\,,$
$\displaystyle\accentset{\leftharpoonup}{\ell}<\accentset{\leftharpoonup}{\theta}\,,$
$\displaystyle
H_{\accentset{\leftharpoonup}{\tau}(\star)}(\accentset{\leftharpoonup}{\lambda})\xrightarrow{a.s.}\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell})\,,$
$\displaystyle\accentset{\leftharpoonup}{\ell}>\accentset{\leftharpoonup}{\theta}\,.$

Consequently,

$\accentset{\leftharpoonup}{L}_{\star,k}(\accentset{\leftharpoonup}{\ell},H_{\tau(\star)})\xrightarrow{a.s.}\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},H_{\tau(\star)})=\min\big{(}\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},0),\,\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda})\big{)}\,.$

Let $\accentset{\leftharpoonup}{\nu}$ denote another choice of threshold. Now,
for every $\accentset{\leftharpoonup}{\ell}$,

$\accentset{\leftharpoonup}{\cal
L}_{\star}(\ell,{H}_{\accentset{\leftharpoonup}{\nu}})\in\big{\\{}\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},0),\,\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda})\big{\\}}.$

The loss $\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},H_{\accentset{\leftharpoonup}{\tau}(\star)})$
is the minimum of these two. Hence, for every
$\accentset{\leftharpoonup}{\ell}$,

$\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},H_{\accentset{\leftharpoonup}{\tau}(\star)})\leq\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\ell},H_{\accentset{\leftharpoonup}{\nu}})\,.$
(5.1)

Since
$\accentset{\leftharpoonup}{\nu}\neq\accentset{\leftharpoonup}{\tau}(\star)$,
there is an intermediate value $\accentset{\leftharpoonup}{\theta}^{\prime}$
between $\accentset{\leftharpoonup}{\theta}$ and
$\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\nu})$ such that
$\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\theta}^{\prime})$
is intermediate between $\accentset{\leftharpoonup}{\tau}(\star)$ and
$\accentset{\leftharpoonup}{\nu}$. At
$\accentset{\leftharpoonup}{\theta}^{\prime}$, of the two procedures behaves
as the null rule while the other behaves as the identity. The two asymptotic
loss functions cross only at a single point
$\accentset{\leftharpoonup}{\theta}(\star)$. Hence, at
$\accentset{\leftharpoonup}{\theta}^{\prime}$ the asymptotic loss functions
are unequal. By (5.1),

$\phantom{\,.}\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\theta}^{\prime},H_{\accentset{\leftharpoonup}{\tau}(\star)})<\accentset{\leftharpoonup}{\cal
L}_{\star}(\accentset{\leftharpoonup}{\theta}^{\prime},H_{\accentset{\leftharpoonup}{\nu}})\,.$
(5.2)

Together, (5.1) and (5.2) establish unique asymptotic admissibility. The
argument as $\gamma_{n}\rightarrow\infty$ is similar. ∎

Figure 5 depicts two of the six cases: Frobenius norm as
$\gamma_{n}\rightarrow 0$, and nuclear norm as $\gamma_{n}\rightarrow\infty$.

Figure 5: Determining the optimal threshold. Left: Frobenius norm,
$\gamma_{n}\rightarrow 0$. The two loss functions
$\accentset{\leftharpoonup}{\ell}\mapsto\accentset{\leftharpoonup}{\cal
L}_{F}(\accentset{\leftharpoonup}{\ell},0)$,
$\accentset{\leftharpoonup}{\ell}\mapsto\accentset{\leftharpoonup}{\cal
L}_{F}(\accentset{\leftharpoonup}{\ell},\accentset{\leftharpoonup}{\lambda})$
cross in a single point
$\accentset{\leftharpoonup}{\ell}=\accentset{\leftharpoonup}{\theta}(F)$. The
optimal threshold is
$\accentset{\leftharpoonup}{\tau}=\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\theta}(F))$.
Right: nuclear norm, $\gamma_{n}\rightarrow\infty$.

##  6 Universal Closed Forms

The spiked covariance model poses a “scaling dilemma” for practitioners:

> I only have my one dataset, with its own specific $p$ and $n$. I don’t know
> what asymptotic scaling $(n,p)$ my dataset “obeys.” Yet, I have two theories
> seemingly competing for my favor: proportional and disproportional growth.
> Each theory has its own optimal formulas. Which should I apply? Fortunately,
> this dilemma can be avoided.

######  Definition 6.1.

Let $\eta^{+}(\lambda|L,\gamma)$ denote a closed-form shrinker for the
proportional growth regime, mentioned following Lemma 2.1. Given a dataset of
size $n\times p$, define the universal shrinker

$\phantom{\,.}\eta^{u}(\lambda|L)=\eta_{n}^{u}(\lambda|L)=\eta^{+}(\lambda|L,p/n)\,.$

That is, we evaluate $\eta^{+}$ using the aspect ratio $\gamma_{n}=p/n$ of the
given dataset. This requires no hypothesis on scaling of $p$ with $n$. We also
denote by $\widehat{\Sigma}^{u}$ the shrinkage estimator
$\widehat{\Sigma}_{\eta^{u}}$.

######  Observation 6.1.

Adopt loss $L=L_{\star,k}$ for $\star\in\\{F,O,N\\}$. The asymptotic shrinkage
descriptors of the universal shrinker $\eta^{u}(\lambda|L)$ are optimal in the
proportional and disproportional limits.

  1. 1.

Assume the proportional limit $\gamma_{n}\rightarrow\gamma$. The asymptotic
shrinkage descriptors, or shrinkage limits, of the optimal proportional-regime
rule $\eta^{+}(\lambda|L,\gamma)$ are

$\phantom{\,.}\eta_{i}^{+}=\lim_{n\rightarrow\infty}\eta^{+}(\lambda_{i}|L,\gamma)\,.$

The corresponding shrinkage limits

$\eta_{i}^{u}=\lim_{n\rightarrow\infty}\eta^{u}(\lambda_{i}|L,\gamma_{n})=\lim_{n\rightarrow\infty}\eta^{+}(\lambda_{i}|L,\gamma_{n})$

almost surely exist and are identical:

$\eta_{i}^{u}\stackrel{{\scriptstyle a.s.}}{{=}}\eta_{i}^{+}\,,\qquad
i=1,\dots,r\,.$

The asymptotic losses of the two shrinkers as calculated by Lemma 2.1 are
almost surely identical.

  2. 2.

Assume the disproportional limit $\gamma_{n}\rightarrow 0$ and varying spikes
II. The shrinkage limits of $\eta^{u}$ and $\eta^{*}$ are

$\accentset{\leftharpoonup}{\eta}_{i}^{u}=\lim_{n\rightarrow\infty}\accentset{\leftharpoonup}{\phi}_{n}(\eta^{u}(\lambda_{i}|L))\,,\hskip
56.9055pt\accentset{\leftharpoonup}{\eta}_{i}^{*}=\lim_{n\rightarrow\infty}\accentset{\leftharpoonup}{\eta}^{*}(\accentset{\leftharpoonup}{\phi}_{n}(\lambda_{i})|L)\,.$

These limits almost surely exist and are identical:

$\phantom{\,.}\accentset{\leftharpoonup}{\eta}_{i}^{u}\stackrel{{\scriptstyle
a.s.}}{{=}}\accentset{\leftharpoonup}{\eta}_{i}^{*}\,,\qquad i=1,\dots,r\,.$

The asymptotic losses of the two shrinkers as calculated by Lemma 3.2 are
almost surely identical.

  3. 3.

Assume the disproportional limit $\gamma_{n}\rightarrow\infty$ and varying
spikes III. The shrinkage limits of $\eta^{u}$ and $\eta^{*}$ are

$\accentset{\rightharpoonup}{\eta}_{i}^{u}=\lim_{n\rightarrow\infty}\accentset{\rightharpoonup}{\phi}_{n}(\eta^{u}(\lambda_{i}|L))\,,\hskip
56.9055pt\accentset{\rightharpoonup}{\eta}_{i}^{*}=\lim_{n\rightarrow\infty}\accentset{\rightharpoonup}{\eta}^{*}(\accentset{\rightharpoonup}{\phi}_{n}(\lambda_{i})|L)\,.$

These limits almost surely exist and are identical:

$\accentset{\rightharpoonup}{\eta}_{i}^{u}\stackrel{{\scriptstyle
a.s.}}{{=}}\accentset{\rightharpoonup}{\eta}_{i}^{*}\,,\qquad i=1,\dots,r\,.$

The asymptotic losses of the two shrinkers as calculated by Lemma 4.2 are
almost surely identical.

For example, recall the proportional-regime shrinker for $L_{F,1}$:

$\displaystyle\phantom{\,.}\eta^{+}(\lambda|L_{F,1},\gamma)=1+(\ell(\lambda,\gamma)-1)c^{2}(\ell(\lambda,\gamma),\gamma)\,.$
(6.1)

Note that

$\phantom{\,.}\accentset{\leftharpoonup}{\phi}_{n}(\ell(\lambda,\gamma_{n}))=\frac{\accentset{\leftharpoonup}{\phi}_{n}(\lambda)-\sqrt{\gamma_{n}}+\sqrt{(\accentset{\leftharpoonup}{\phi}_{n}(\lambda)-\sqrt{\gamma_{n}})^{2}-4}}{2}=\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\phi}_{n}(\lambda)-\sqrt{\gamma_{n}})\,,$

so
$\accentset{\leftharpoonup}{\phi}_{n}(\ell(\lambda_{i},\gamma_{n}))\xrightarrow{a.s.}\accentset{\leftharpoonup}{\ell}(\accentset{\leftharpoonup}{\lambda}_{i})$.
Thus,

$\displaystyle\phantom{\,.}\accentset{\leftharpoonup}{\phi}_{n}(\eta^{u}(\lambda_{i}|L_{F,1}))$
$\displaystyle=\frac{\ell(\lambda_{i},\gamma_{n})-1}{\sqrt{\gamma_{n}}}\cdot
c^{2}(\ell(\lambda_{i},\gamma_{n}),\gamma_{n})$
$\displaystyle=\accentset{\leftharpoonup}{\phi}_{n}(\ell(\lambda_{i},\gamma_{n}))\cdot\frac{1-1/\phi_{n}^{2}(\ell(\lambda_{i},\gamma_{n}))}{1+\sqrt{\gamma_{n}}/\phi_{n}(\ell(\lambda_{i},\gamma_{n}))}\cdot
1_{\\{\accentset{\leftharpoonup}{\lambda}_{i}>1\\}}$
$\displaystyle\xrightarrow{a.s.}(\accentset{\leftharpoonup}{\ell}_{i}-1/\accentset{\leftharpoonup}{\ell}_{i})_{+}\,,$

agreeing with Lemma 3.3.

######  Corollary 6.1.1.

The estimator sequence $\widehat{\Sigma}_{\eta^{u}}$ is uniquely
asymptotically admissible under either the proportional or disproportional
growth regimes.

This principle applies more broadly; consider thresholding. Constructed in the
previous section as $\gamma_{n}\rightarrow 0$ and
$\gamma_{n}\rightarrow\infty$, optimal thresholds also exist in the
proportional limit $\gamma_{n}\rightarrow\gamma\in(0,\infty)$. These three
choices of threshold, depending on the limit regime, again present a scaling
conundrum to practitioners.

One can easily check, however, that under each loss there exists a simple
close-form threshold which performs optimally in all three limits. Under
proportional growth and $L_{F,1}$, the losses of the null and identity rules
are

${\cal L}_{F,1}(\ell,0)=(\ell-1)^{2}\,,\qquad{\cal
L}_{F,1}(\ell,\lambda)=(\ell-\lambda(\ell,\gamma))^{2}+(\lambda(\ell,\gamma)-1)^{2}(1-c^{4}(\ell,\gamma))\,.$

$\ell\mapsto{\cal L}_{F,1}(\ell,0)$ is increasing and, for
$\ell>\ell_{+}(\gamma)$, $\ell\mapsto{\cal L}_{F,1}(\ell,\lambda)$ is
decreasing. The crossing point $\theta_{\gamma}(F,1)$ of the two losses ${\cal
L}_{F,1}(\theta_{\gamma},0)={\cal L}_{F,1}(\theta_{\gamma},\lambda)$ can be
shown to be the largest real root of

$\gamma^{3}\theta^{2}+3\gamma^{2}\theta^{2}(\theta-1)+\gamma(\theta+1)(\theta-1)^{3}=(\theta-1)^{5}.$

The corresponding threshold is
$\lambda_{\gamma}(F,1)=\lambda(\theta_{\gamma}(F,1),\gamma)$. One may verify
that

$\lambda_{\gamma}(F,1)\sim
1+\sqrt{\gamma}\cdot\accentset{\leftharpoonup}{\tau}(F)\,,\quad\gamma\rightarrow
0\,;\qquad\lambda_{\gamma}(F,1)\sim
1+\gamma\cdot\accentset{\rightharpoonup}{\tau}(F)\,,\quad\gamma\rightarrow\infty\,.$

Define the universal threshold rule $\lambda^{u}(F,1)=\lambda^{u}_{n}(F,1)$ by
evaluating the proportional rule with the aspect ratio $\gamma_{n}=p/n$ of the
given dataset: $\lambda^{u}(F,1)=\lambda_{\gamma_{n}}(F,1)$. This threshold
can be applied as is—it requires no scaling hypothesis. Nevertheless, it is an
optimal threshold in both the proportional limit and either disproportional
limit.

##  7 Estimation in the Spiked Wigner model

We now develop a connection to the spiked Wigner model. Let $W=W_{n}$ denote a
Wigner matrix, a real symmetric matrix of size $n\times n$ with independent
entries on the upper triangle distributed as $\mathcal{N}(0,1)$. The empirical
distribution of eigenvalues of $W$ converges (weakly almost surely) to
$\omega(x)=(2\pi)^{-1}\sqrt{4-x^{2}}$, the standard semicircle density with
support $\overline{\lambda}_{\pm}=\pm 2$.

Let $\Theta=\Theta_{n}$ denote a symmetric $n\times n$ “signal” matrix of
fixed rank $r$; under the spiked Wigner model observed data $Y=Y_{n}$ obeys

$\displaystyle\phantom{\,.}Y=\Theta+\frac{1}{\sqrt{n}}W\,.$ (7.1)

Let
$\theta_{1}\geq\cdots\geq\theta_{r_{+}}>0>\theta_{r_{+}+1}\geq\dots\geq\theta_{r}$
denote the non-zero eigenvalues of $\Theta$, so there are $r_{+}$ positive
values and $r_{-}=r-r_{+}$ negative, and $u_{1},\ldots,u_{n}$ the
corresponding eigenvectors. The standard (rank-aware) reconstruction is

$\phantom{\,.}\widehat{\Theta}^{r}=\sum_{i=1}^{r_{+}}\lambda_{i}(Y)v_{i}v_{i}^{\prime}+\sum_{i=n-r_{-}+1}^{n}\lambda_{i}(Y)v_{i}v_{i}^{\prime}\,,$

where $\lambda_{1}(Y)\geq\cdots\geq\lambda_{n}(Y)$ are the eigenvalues of $Y$
and $v_{1},\ldots,v_{n}$ the associated eigenvectors.

Maïda [24], Capitaine, Donati-Martin and Feral [11], and Benaych-Georges and
Rao Nadakuditi [7], among others, derive an eigenvalue mapping
$\overline{\lambda}(\theta)$ describing the empirical eigenvalues induced by
signal eigenvalues $\theta_{i}$. Their results imply that the top $r_{+}$
empirical eigenvalues of $Y$ obey
$\lambda_{i}(Y)\xrightarrow{a.s.}\overline{\lambda}(\theta_{i})$,
$i=1,\dots,r_{+}$, while the lowest $r_{-}$ obey
$\lambda_{n-i}\xrightarrow{a.s.}\overline{\lambda}(\theta_{r-i})$, $0\leq
i<r_{-}$. Here the eigenvalue mapping function is defined by

$\displaystyle\phantom{\,.}\overline{\lambda}(\theta)=\begin{dcases}\theta+\frac{1}{\theta}&|\theta|>1\\\
2\,\text{sign}(\theta)&0<|\theta|\leq 1\end{dcases},$ (7.2)

with phase transitions at $\pm 1$ mapping to bulk edges
$\overline{\lambda}_{\pm}=\pm 2$. There is a partial inverse to
$\theta\mapsto\overline{\lambda}(\theta)$:

$\theta(\lambda)=\begin{dcases}(\lambda+\text{sign}(\lambda)\sqrt{\lambda^{2}-2\sigma^{2}})/2&|\lambda|>2\\\
0&|\lambda|\leq 2\end{dcases}\,.$ (7.3)

Empirical eigenvectors are not perfectly aligned with the corresponding signal
eigenvectors:

$|\langle
u_{i},v_{i}\rangle|^{2}\xrightarrow{a.s.}\overline{c}^{2}(\theta_{i})\,,\qquad
i\in\\{1,\ldots,r_{+},n-r_{-}+1,\ldots,n\\}\,,$

where the cosine function is given by

$\displaystyle\phantom{\,.}\overline{c}^{2}(\theta)=\begin{dcases}1-\frac{1}{\theta^{2}}&|\theta|>1\\\
0&\theta\leq 1\end{dcases}.$ (7.4)

The phenomena of spreading, inflation, and rotation imply that
$\widehat{\Theta}^{r}$ can be improved upon, substantially, by well-chosen
shrinkage estimators:

$\displaystyle\widehat{\Theta}_{\eta}=\sum_{i=1}^{n}\eta(\lambda_{i}(Y))v_{i}v_{i}^{\prime}\,,$
(7.5)

with $\eta:\mathbb{R}\rightarrow\mathbb{R}$ a shrinkage function. For numerous
loss functions $L$, specific shrinkers $\eta^{*}(\cdot|L)$ outperform the
standard rank-aware estimator $\widehat{\Theta}^{r}$.

We evaluate performance under a fixed-spike model, in which the signal
eigenvalues $(\theta_{i})_{i=1}^{r}$ do not vary with $n$. We measure loss
using matrix norms $L_{\star,1}(\Theta,\widehat{\Theta})$,
$\star\in\\{F,O,N\\}$, as earlier, and evaluate asymptotic loss following the
“shrinkage descriptor” approach.

######  Lemma 7.1.

Let $\eta_{n}$ denote a sequence of shrinkers, possibly varying with $n$.
Under the fixed-spike model, suppose that the sequences of shrinker outputs
converge:

$\displaystyle\eta_{n}(\lambda_{i})$
$\displaystyle\xrightarrow{a.s.}\overline{\eta}_{i}\,,$ $\displaystyle 1\leq
i\leq r_{+}\,,$ $\displaystyle\eta_{n}(\lambda_{n-(r-i)})$
$\displaystyle\xrightarrow{a.s.}\overline{\eta}_{i}\,,$ $\displaystyle
r_{+}<i\leq r\,.$

As before, we call the limits $(\overline{\eta}_{i})_{i=1}^{r}$ the asymptotic
shrinkage descriptors. Each loss $L_{\star,1}$ converges almost surely to a
deterministic limit:

$\phantom{\,.}L_{\star,1}(\Theta,\widehat{\Theta}_{\eta_{n}})\xrightarrow{a.s.}\overline{\cal
L}_{\star}((\theta_{i})_{i=1}^{r},(\overline{\eta}_{i})_{i=1}^{r})\,.$

The asymptotic loss is sum- or max- decomposable into $r$ terms involving
matrix norms applied to pivots of the $2\times 2$ matrices $\widetilde{A}$ and
$\widetilde{B}$ introduced earlier. With $\theta_{i}$ denoting a spike
parameter, $\overline{c}(\theta_{i})$ the limiting cosine in (7.4), and
$\overline{s}^{2}(\theta_{i})=1-\overline{c}^{2}(\theta_{i})$, the
decompositions are

$\displaystyle\phantom{\,,}\overline{\cal
L}_{F}((\theta_{i})_{i=1}^{r},(\overline{\eta}_{i})_{i=1}^{r})$
$\displaystyle=\bigg{(}\sum_{i=1}^{r}\big{[}L_{F,1}\big{(}\widetilde{A}(\theta_{i}),\widetilde{B}(\overline{\eta}_{i},\overline{c}(\theta_{i}))\big{)}\big{]}^{2}\bigg{)}^{1/2}\,,$
$\displaystyle\overline{\cal
L}_{O}((\theta_{i})_{i=1}^{r},(\overline{\eta}_{i})_{i=1}^{r})$
$\displaystyle=\max_{1\leq i\leq
r}L_{O,1}\big{(}\widetilde{A}(\theta_{i}),\widetilde{B}(\overline{\eta}_{i},\overline{c}(\theta_{i}))\big{)}\,,$
$\displaystyle\overline{\cal
L}_{N}((\theta_{i})_{i=1}^{r},(\overline{\eta}_{i})_{i=1}^{r})$
$\displaystyle=\sum_{i=1}^{r}L_{N,1}\big{(}\widetilde{A}(\theta_{i}),\widetilde{B}(\overline{\eta}_{i},\overline{c}(\theta_{i}))\big{)}\,.$

Proceeding as before, we obtain closed forms of formally optimal shrinkers and
losses, explicit in terms of $\theta$. As in previous sections, asymptotically
optimal shrinkers on observables are constructed using the partial inverse
$\theta(\lambda)$ (7.3).

######  Lemma 7.2.

Formally optimal shrinkers and corresponding losses are given by

$\displaystyle\overline{\eta}^{*}(\theta|F)=\textup{sign}(\theta)(|\theta|-1/|\theta|)_{+}\,,$
$\displaystyle[\overline{\cal
L}_{F}^{1}(\theta)]^{2}=\begin{dcases}\theta^{2}(1-1/\theta^{4})&|\theta|>1\\\
\theta^{2}&0\leq|\theta|\leq 1\end{dcases}\,,$
$\displaystyle\overline{\eta}^{*}(\theta|O)=\theta\cdot
1_{\\{|\theta|>1\\}}\,,$ $\displaystyle\overline{\cal
L}_{O}^{1}(\theta)=\begin{dcases}1&|\theta|>1\\\ |\theta|&0<|\theta|\leq
1\end{dcases}\,,$ (7.6)
$\displaystyle\overline{\eta}^{*}(\theta|N)=\textup{sign}(\theta)\big{(}|\theta|-2/|\theta|\big{)}_{+}\,,$
$\displaystyle\overline{\cal
L}_{N}^{1}(\theta)=\begin{dcases}2\sqrt{1-1/|\theta|^{2}}&|\theta|>\sqrt{2}\\\
|\theta|&0<|\theta|\leq\sqrt{2}\end{dcases}\,.$

Evidently, these expressions bear a strong formal resemblance to those we
found earlier for covariance shrinkage as $\gamma_{n}\rightarrow 0$: for
$x>0$,

$\displaystyle\overline{\lambda}(x)=\accentset{\leftharpoonup}{\lambda}(x)\,,$
$\displaystyle\overline{c}(x)=\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}(x)\,,$
$\displaystyle\overline{\eta}^{*}(x|\star)=\accentset{\leftharpoonup}{\eta}^{*}(x|\star)\,,$
$\displaystyle\overline{\cal
L}_{\star}^{1}(x)=\accentset{\leftharpoonup}{{\cal L}}_{\star}^{1}(x)\,.$

Such similarities extend to hard thresholding; namely, the
$L_{\star,1}$-optimal thresholds $\overline{\tau}(\star)$ for the spiked
Wigner model (to which eigenvalue magnitudes are compared to) are equal to
their counterparts in the $\gamma_{n}\rightarrow 0$ setting:

$\phantom{\,.}\overline{\tau}(\star)=\accentset{\leftharpoonup}{\tau}(\star)\,,\qquad\star\in\\{F,O,N\\}\,.$

These are not chance similarities. The empirical spectral distribution of
$\gamma_{n}^{-1/2}(S-I)$ converges as $\gamma_{n}\rightarrow 0$ to the
semicircle law (Bai and Yin [2]). Spiked covariance formulas as
$\gamma_{n}\rightarrow 0$ for eigenvalue inflation and eigenvector
rotation—functions of the limiting spectral distribution—are therefore
equivalent to those under the spiked Wigner model. By Lemmas 3.5 and 7.2, this
mandates identical shrinkage. In all essential quantitative aspects—eigenvalue
inflation, eigenvector rotation, and optimal shrinkers and losses—the
$\gamma_{n}\rightarrow 0$ covariance estimation and spiked Wigner settings are
“isomorphic.”

##  8 Bidirectional Spiked Covariance Model

Thus far we have discussed the spike covariance model assuming the spiked
eigenvalues deviate are elevated, $\ell_{i}>1$. We now consider the
possibility of depressed values, $\ell_{i}<1$. Earlier results as
$\gamma_{n}\rightarrow 0$ quickly adapt to this setting. We adopt the more
general spike covariance model
$\ell_{i}=\ell_{i,n}=1+\accentset{\leftharpoonup}{\ell}_{i}(1+o(1))\sqrt{\gamma_{n}}$
where $\accentset{\leftharpoonup}{\ell}_{i}$ may now be either positive or
negative. In the interest of brevity, results in this section are stated
informally without proof.

Normalized bulk edges lie at $\pm 2$, and phase transitions of normalized
eigenvalues occur at $\pm 1$. The “bidirectional” eigenvalue mapping function,
$\accentset{\leftharpoonup}{\lambda}^{\pm}(\accentset{\leftharpoonup}{\ell})$,
is the odd extension of the “unidirectional” mapping (previously denoted by
$\accentset{\leftharpoonup}{\lambda}$, now by
$\accentset{\leftharpoonup}{\lambda}^{+}$ for clarity):

$\phantom{\,.}\accentset{\leftharpoonup}{\lambda}^{\pm}(\accentset{\leftharpoonup}{\ell})=\mbox{sign}(\accentset{\leftharpoonup}{\ell})\cdot\accentset{\leftharpoonup}{\lambda}^{+}(|\accentset{\leftharpoonup}{\ell}|)\,,$

while the cosine function
$\accentset{\leftharpoonup}{c}^{\pm}(\accentset{\leftharpoonup}{\ell})=(1-|\accentset{\leftharpoonup}{\ell}|^{-2})_{+}$
is the even extension of $\accentset{\leftharpoonup}{c}^{+}$ (previously
denoted by $\accentset{\leftharpoonup}{c}$).

The connection to the spiked Wigner model is now even more apparent,
eigenvalue mappings and cosine functions are identical:

$\accentset{\leftharpoonup}{\lambda}^{\pm}(\accentset{\leftharpoonup}{\ell})=\overline{\lambda}(\accentset{\leftharpoonup}{\ell})\,,\hskip
56.9055pt\accentset{\leftharpoonup}{c}\rule{0.0pt}{7.0pt}^{\pm}(\accentset{\leftharpoonup}{\ell})=\overline{c}(\accentset{\leftharpoonup}{\ell})\,.$

For Frobenius norm loss, we have the “bidirectionally optimal” shrinker

$\phantom{\,.}\accentset{\leftharpoonup}{\eta}^{\pm}(\accentset{\leftharpoonup}{\ell}|F)=\overline{\eta}(\accentset{\leftharpoonup}{\ell}|F)=\mbox{sign}(\accentset{\leftharpoonup}{\ell})\cdot(|\accentset{\leftharpoonup}{\ell}|-1/|\accentset{\leftharpoonup}{\ell}|)_{+}\,.$

the odd extension of the “unidirectionally optimal” shrinker, while the
optimal (rank-one) loss is $\sqrt{2-1/|\accentset{\leftharpoonup}{\ell}|^{2}}$
for $|\accentset{\leftharpoonup}{\ell}|>1$ and
$|\accentset{\leftharpoonup}{\ell}|$ otherwise, the even extension of
$\accentset{\leftharpoonup}{{\cal L}}_{F}^{1}$. Similarly, bidirectionally
optimal shrinkers and corresponding losses under operator and nuclear norm
losses are respectively the odd and even extensions of functions in Lemma 3.3.

##  9 Conclusion

Although proportional-limit analysis has become popular in recent years, many
datasets—perhaps most—have many more rows than columns or many more columns
than rows. We have studied eigenvalue shrinkage in each of these
disproportional regimes and identified optimal procedures under each,
exhibiting closed-form expressions for asymptotically optimal shrinkage
functions and corresponding losses. We further identified a single closed-form
nonlinearity for each loss function considered which can be “universally”
applied across the proportional fixed-spiked or either disproportional
varying-spike limit. Equivalent optimal shrinkage rules independently arise
for matrix recovery under the spiked Wigner model.

## Acknowledgements

This work was supported by NSF DMS grant 1811614.

##  Appendix A Appendix

###### Proof of Lemma 3.1.

The proof is a modification of standard arguments, see Section 4 of [19] or
Chapter 11 of [35]. Without loss of generality, we may assume that
$\Sigma=\text{diag}(\accentset{\leftharpoonup}{\ell}_{1},\ldots,\accentset{\leftharpoonup}{\ell}_{r},1,\ldots,1)$.
An eigenvalue $\accentset{\leftharpoonup}{\ell}_{i}$ that is nosolves the
equation Partition the data matrix $X$ and covariance $\Sigma$ into blocks of
$r$ and $p-r$ rows:

$\displaystyle X=\begin{bmatrix}X_{1}\\\ X_{2}\end{bmatrix}\,,$
$\displaystyle\Sigma=\begin{bmatrix}\Sigma_{1}&0\\\ 0&I_{p-r}\end{bmatrix}\,.$
(A.1)

We define the transformed sample covariance
$\accentset{\leftharpoonup}{S}=\accentset{\leftharpoonup}{S}_{n}=(np)^{-1/2}(XX^{\prime}-nI_{p})$
with eigenvalues
$\accentset{\leftharpoonup}{\lambda}_{1}\geq\accentset{\leftharpoonup}{\lambda}_{2}\geq\cdots\geq\accentset{\leftharpoonup}{\lambda}_{p}$
and partition analogously,

$\displaystyle\phantom{\,.}\accentset{\leftharpoonup}{S}=\begin{bmatrix}\accentset{\leftharpoonup}{S}_{11}&\accentset{\leftharpoonup}{S}_{12}\\\
\accentset{\leftharpoonup}{S}_{21}&\accentset{\leftharpoonup}{S}_{22}\end{bmatrix}=\frac{1}{\sqrt{np}}\begin{bmatrix}X_{1}X_{1}^{\prime}-nI_{r}&X_{1}X_{2}^{\prime}\\\
X_{2}X_{1}^{\prime}&X_{2}X_{2}^{\prime}-nI_{p-r}\end{bmatrix}\,.$ (A.2)

Let
$\underline{\accentset{\leftharpoonup}{S}}_{22}=(np)^{-1/2}(X_{2}^{\prime}X_{2}-nI_{n})$
denote the companion matrix to $\accentset{\leftharpoonup}{S}_{22}$. An
eigenvalue $\accentset{\leftharpoonup}{\lambda}$ of
$\accentset{\leftharpoonup}{S}$ that is not an eigenvalue of
$\accentset{\leftharpoonup}{S}_{22}$ satisfies

$\displaystyle\phantom{\,.}|\accentset{\leftharpoonup}{S}-\accentset{\leftharpoonup}{\lambda}I_{p}|=|\accentset{\leftharpoonup}{S}_{22}-\accentset{\leftharpoonup}{\ell}I_{p-r}||K_{n}(\accentset{\leftharpoonup}{\lambda})-\accentset{\leftharpoonup}{\lambda}I_{r}|=0\,,$
(A.3)

where the $r\times r$ matrix $K_{n}(z)$ is given by

$\displaystyle K_{n}(z)$
$\displaystyle=\accentset{\leftharpoonup}{S}_{11}+\accentset{\leftharpoonup}{S}_{12}(zI_{p-r}-\accentset{\leftharpoonup}{S}_{22})^{-1}\accentset{\leftharpoonup}{S}_{21}$
(A.4)
$\displaystyle=\Big{(}\frac{z}{\sqrt{np}}+\frac{1}{p}\Big{)}X_{1}(zI_{n}-\underline{\accentset{\leftharpoonup}{S}}_{22})^{-1}X_{1}^{\prime}-\sqrt{\frac{n}{p}}I_{r}\,.$

Now, fix $z$ with $\Re(z)>2$ and $\delta\in(0,\Re(z)-2)$. Define the event
$E_{n,\delta}=\\{\lambda_{1}(\accentset{\leftharpoonup}{S}_{22})\leq
2+\delta\\}$, which occurs almost surely eventually (Theorem 1 of [12]).
Recalling that the non-zero eigenvalues of $X_{2}X_{2}^{\prime}$ and
$X_{2}^{\prime}X_{2}$ are equal,
$\underline{\accentset{\leftharpoonup}{S}}_{22}$ has an eigendecomposition

$\phantom{\,.}\underline{\accentset{\leftharpoonup}{S}}_{22}=\begin{bmatrix}U_{1}&U_{2}\end{bmatrix}\begin{bmatrix}\Lambda_{1}&0\\\
0&-\sqrt{n/p}I_{n-p+r}\end{bmatrix}\begin{bmatrix}U_{1}&U_{2}\end{bmatrix}^{\prime}\,.$

Here, $\Lambda_{1}$ contains the $p-r$ eigenvalues of
$n^{-1}X_{2}X_{2}^{\prime}$ and the blocks $U_{1}$ and $U_{2}$ contain as
columns $p-r$ and $n-p+r$ eigenvectors, respectively. Thus,

$\displaystyle\phantom{\,.}X_{1}(zI_{n}-\underline{\accentset{\leftharpoonup}{S}}_{22})^{-1}X_{1}^{\prime}$
$\displaystyle=X_{1}U_{1}(zI_{p-r}-\Lambda_{1})^{-1}U_{1}^{\prime}X_{1}^{\prime}+(z+\sqrt{n/p})^{-1}X_{1}U_{2}U_{2}^{\prime}X_{1}^{\prime}\,.$
(A.5)

Let $y_{i}$ denote the $i$-th row of $X_{1}U_{1}$, distributed as
$\mathcal{N}(0,\Sigma_{ii}I_{p-r})$ and independent of $X_{2}$. By the
boundedness of the spectral norm of $(zI_{p-r}-\Lambda_{1})^{-1}$ on
$E_{n,\delta}$ and concentration of quadratic forms (Lemma B.26 of [3]), we
obtain

$\displaystyle\phantom{\,.}p^{-k}\mathbf{E}\big{|}y_{i}^{\prime}(zI_{p-r}-\Lambda_{1})^{-1}y_{i}^{\prime}-\mathrm{tr}(zI_{p-r}-\Lambda_{1})^{-1}\Sigma_{ii}\big{|}^{k}I(E_{n,\delta})$
$\displaystyle\leq
C_{k}p^{-k/2}\mathbf{E}\|zI_{p-r}-\Lambda_{1}\|^{-k}I(E_{n,\delta})$ (A.6)
$\displaystyle\leq C_{k}p^{-k/2}\,.$

Taking $k>2$ and using the Borel-Cantelli lemma,

$\displaystyle\phantom{\,.}\frac{1}{p}X_{1}U_{1}(zI_{p-r}-\Lambda_{1})^{-1}U_{1}^{\prime}X_{1}^{\prime}-\mathrm{tr}(zI_{p-r}-\Lambda_{1})^{-1}\Sigma_{1}\xrightarrow{a.s.}0\,.$
(A.7)

Furthermore, by a basic calculation,

$\displaystyle\phantom{\,.}\frac{1}{p(z+\sqrt{n/p})}X_{1}U_{2}U_{2}^{\prime}X_{1}^{\prime}-\sqrt{\frac{n}{p}}I_{r}\xrightarrow{a.s.}\accentset{\leftharpoonup}{\Sigma}_{1}\,,$
(A.8)

where we define
$\accentset{\leftharpoonup}{\Sigma}_{1}=\text{diag}(\accentset{\leftharpoonup}{\ell}_{1},\accentset{\leftharpoonup}{\ell}_{2},\ldots,\accentset{\leftharpoonup}{\ell}_{r})=\lim_{n\rightarrow\infty}\gamma_{n}^{-1/2}(\Sigma_{1}-I_{r})$.
From (A.4) - (A.8) and the fact that

$\phantom{\,.}-\frac{1}{p}\mathrm{tr}(zI_{p-r}-\Lambda_{1})\xrightarrow{a.s.}s(z)=\frac{-z+\sqrt{z^{2}-4}}{2}\,,$

the Stieltjes transform of the semicircle law [2], we conclude that $K_{n}(z)$
tends almost surely to a deterministic limit $K(z)$:

$\displaystyle\phantom{\,.}K_{n}(z)\xrightarrow[]{a.s.}K(z)=-s(z)I_{r}+\accentset{\leftharpoonup}{\Sigma}_{1}\,.$
(A.9)

Notice that the zeros of $|K(z)-zI_{r}|$ are precisely
$\\{\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}_{i}):\accentset{\leftharpoonup}{\ell}_{i}\geq
1\\}$. We establish convergence of the zeros of $|K_{n}(z)-zI_{r}|$ to those
of $|K_{n}(z)-zI_{r}|$ via the method of Lemma 6.1 of [7]. Let
$\text{Card}_{n}(a,b)$ and $\text{Card}(a,b)$ denote the number of zeros in
$(a,b)$ of $|K_{n}(z)-zI_{r}|$ and $|K(z)-zI_{r}|$, respectively.

Consider an interval $(a,b)$ with $a>\lambda_{+}=2$ and
$a,b\not\in\\{\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}_{i}):\accentset{\leftharpoonup}{\ell}_{i}\geq
1\\}$. Let $\mathcal{C}$ denote a circular contour with diameter $[a,b]$ and
$\delta\in(0,a-2)$. On $\mathcal{C}$, $E_{n,\delta}$ occurring almost surely
eventually, (A.9) converges uniformly by the Arzela-Ascoli theorem. Thus,

$\displaystyle\phantom{\,.}\text{Card}_{n}(a,b)=\frac{1}{2\pi
i}\int_{\mathcal{C}}\frac{\partial_{z}|K_{n}(z)-zI_{r}|}{|K_{n}(z)-zI_{r}|}dz\xrightarrow{a.s.}\frac{1}{2\pi
i}\int_{\mathcal{C}}\frac{\partial_{z}|K(z)-zI_{r}|}{|K(z)-zI_{r}|}dz=\text{Card}(a,b)\,,$
(A.10)

where we have used that uniform convergence of an analytic sequence implies
uniform convergence of the derivative. Since the right-hand side is integral,
$\text{Card}_{n}(a,b)=\text{Card}(a,b)$ almost surely eventually.

As the spectral norm of $\accentset{\leftharpoonup}{S}$ is almost surely
eventually bounded (one may verify the norm of each block of (A.2) is
bounded), (A.10) implies (3.4):

$\displaystyle\phantom{\,.}\accentset{\leftharpoonup}{\lambda}_{i}\xrightarrow{a.s.}\accentset{\leftharpoonup}{\lambda}(\accentset{\leftharpoonup}{\ell}_{i})\,,\hskip
56.9055pt1\leq i\leq r\,.$

The proof of (3.5) is similar to Section 4.2 of [19] and is omitted.

∎

## References

  * [1] Z. Bai and J. Yao. On sample eigenvalues in a generalized spiked population model. Journal of Multivariate Analysis, 106:167-177, 2012.
  * [2] Z. Bai and Y. Yin. Convergence to the Semicircle Law. The Annals of Probability, 16(2):863-875, 1988.
  * [3] Z. Bai and J. Silverstein. Spectral Analysis of Large Dimensional Random Matrices. Springer, 2010.
  * [4] J. Baik and J. Silverstein. Eigenvalues of large sample covariance matrices of spiked population models. Journal of Multivariate Analysis, 97(6):1382–1408, 2006.
  * [5] J. Baik, G. Ben Arous, and S. Péché. Phase transition of the largest eigenvalue for non-null complex sample covariance matrices. The Annals of Probability, 33:1643-1697, 2005.
  * [6] F. Benaych-Georges, A. Guionnet, and M. Maïda. Fluctuations of the extreme eigenvalues of finite rank deformations of random matrices. Electronic Journal of Probability, 16:1621-1662, 2011.
  * [7] F. Benaych-Georges and R. Rao Nadakuditi. The eigenvalues and eigenvectors of low rank perturbations of large random matrices. Advances in Mathematics, 227(1):494–521, 2011.
  * [8] F. Benaych-Georges and R. Rao Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis, 111:120–135, 2012.
  * [9] A. Bloemendal, A. Knowles, H. Yau, J. Yin. On the principal components of sample covariance matrices. Probab. Theory Related Fields 164 (2016) 459–552.
  * [10] T. Cai, X. Han, and G. Pan. Limiting Laws for Divergent Spiked Eigenvalues and Largest Non-spiked Eigenvalue of Sample Covariance Matrices. The Annals of Statistics, 48(3):1255-1280, 2020.
  * [11] C. Mireille, C. Donati-Martin, and D. Féral. The Largest Eigenvalues of Finite Rank Deformation of Large Wigner Matrices: Convergence and Nonuniversality of the Fluctuations. The Annals of Probability, 37(1):1–47, 2009.
  * [12] B. Chen and G. Pan. Convergence of the largest eigenvalue of normalized sample covariance matrices when p and n both tend to infinity with their ratio converging to zero. Bernoulli, 18(4):1405-1420, 2012.
  * [13] B. Chen and G. Pan. CLT for linear spectral statistics of normalized sample covariance matrices with the dimension much larger than the sample size. Bernoulli, 21(2):1089-1133, 2015.
  * [14] D. Donoho, M. Gavish, and I. Johnstone. Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model. The Annals of Statistics, 46(4):1742-1778, 2018.
  * [15] D. Donoho, M. Gavish, and I. Johnstone. Supplementary Material for “Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model.” http://purl.stanford.edu/xy031gt1574. 2016.
  * [16] N. El Karoui. Spectrum estimation for large dimensional covariance matrices using random matrix theory. The Annals of Statistics, 36(6):2757-2790, 2008.
  * [17] M. Feldman. Spiked Singular Values and Vectors under Extreme Aspect Ratios. arXiv preprint arXiv:2104.15127, 2021.
  * [18] M. Gavish and D. Donoho. The Optimal Hard Threshold for Singular Values is $4/\sqrt{3}$. IEEE Transactions on Information Theory, 60:5040-5053, 2014.
  * [19] I. Johnstone and J. Yang. Notes on asymptotics of sample eigenstructure for spiked models with non-Gaussian data. arXiv preprint arXiv:1810.10427, 2018.
  * [20] O. Ledoit and S. Péché. Eigenvectors of some large sample covariance matrix ensembles. Probability Theory and Related Fields, 151:233-264, 2011.
  * [21] O. Ledoit and M. Wolf. Analytical Nonlinear Shrinkage of Large-Dimensional Covariance Matrices. The Annals of Statistics, 48(5):3043-3065, 2020.
  * [22] O. Ledoit and M. Wolf. The power of (Non-)Linear Shrinking: A Review and Guide to Covariance Matrix Estimation. Journal of Financial Econometrics, 1-32, 2020.
  * [23] Y. Long, X. Jiahui, and Z. Wang. Central Limit Theory for Linear Spectral Statistics of Normalized Separable Sample Covariance Matrix. arXiv preprint arXiv:2105.12975, 2021.
  * [24] M. Maida, Large deviations for the largest eigenvalue of rank one deformations of Gaussian ensembles. Electronic Journal of Probability, 12:1131-1150, 2007.
  * [25] V. Marchenko and L. Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1:457-483, 1967.
  * [26] D. Morales-Jimenez, I. Johnstone, M. McKay, J. Yang. Asymptotics of eigenstructure of sample correlation matrices for high-dimensional spiked models. Statistica Sinica, 31:571-601, 2021.
  * [27] J. Novembre et. al. Genes mirror geography within Europe. Nature, 456:98-101, 2008.
  * [28] D. Paul. Asymptotics of Sample Eigenstructure for a Large Dimensional Spiked Covariance Model. Statistica Sinica, 17:1617-1642, 2007.
  * [29] M. Rudelson and R. Vershynin. Hanson-Wright inequality and sub-gaussian concentration. Electronic Communications in Probability, 18:1-9, 2013. a
  * [30] D. Shen, H. Shen, H. Zhu, and J. Marron. The statistics and mathematics of high dimension low sample size asymptotics. Statistica Sinica, 26(4):1747–1770, 2016.
  * [31] C. Stein. Lectures on the theory of estimation of many parameters. Journal of Mathematical Sciences, 34(1):1373–1403, 1986.
  * [32] C. Stein. Some problems in multivariate analysis. Technical report, Department of Statistics, Stanford University, 1956.
  * [33] M. Wainwright. High-Dimensional Statistics. Cambridge, 2019.
  * [34] L. Wang and D. Paul. Limiting spectral distribution of renormalized separable sample covariance matrices when $p/n\rightarrow 0$. Journal of Multivariate Analysis, 126:25-52, 2014.
  * [35] J. Yao, S. Zheng, and Z. Bai. Large Sample Covariance Matrices and High-Dimensional Data Analysis. Cambridge, 2015.