notice that

$\displaystyle~{}\left\|{\Psi}_{n}^{-1/2}{\widehat{\mathcal{T}}_{n,a}}{\Psi}_{n}^{-1/2}\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}\left\|{\Psi}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\widehat{\bm{K}}_{n,a}^{-1}\widehat{\bm{V}}_{n,a}\widehat{\bm{K}}_{n,a}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{\Psi}_{n}^{-1/2}\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}\left\|{\Psi}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1/2}\cdot\bm{K}_{n}^{1/2}\widehat{\bm{K}}_{n,a}^{-1}\widehat{\bm{V}}_{n,a}\widehat{\bm{K}}_{n,a}^{-1}\bm{K}_{n}^{1/2}\cdot\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{\Psi}_{n}^{-1/2}\right\|_{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|\bm{K}_{n}^{1/2}\widehat{\bm{K}}_{n,a}^{-1}\widehat{\bm{V}}_{n,a}\widehat{\bm{K}}_{n,a}^{-1}\bm{K}_{n}^{1/2}\right\|_{2}\cdot\left\|\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{\Psi}_{n}^{-1/2}\right\|_{2}^{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|\bm{K}_{n}\right\|_{2}\left\|\widehat{\bm{K}}_{n,a}^{-1}\widehat{\bm{V}}_{n,a}\widehat{\bm{K}}_{n,a}^{-1}\right\|_{2}\cdot\left\|{\Psi}_{n}^{-1/2}{\Psi}_{n}{\Psi}_{n}^{-1/2}\right\|_{2}$
$\displaystyle=$ $\displaystyle~{}O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right),$

where the last equation is from the definition of $\Psi_{n}$ as well as the
fact that $\lambda_{\max}(\bm{K}_{n})=O(1)$ and result (42) proved before.
Therefore, we proved that
$\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}\left({\widehat{\Psi}_{n,a}}^{-1}-{\Psi}_{n}^{-1}\right){\widehat{\mathcal{T}}_{n,a}}^{1/2}\right\|_{2}=O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right)$.

For the second term in equation (61), assume for now that we have the
following result

$\left\|{\widehat{\mathcal{T}}_{n,a}}^{-1/2}{\Psi}_{n}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}-{{\mathcal{T}}_{n}}^{-1/2}{\Psi}_{n}{{\mathcal{T}}_{n}}^{-1/2}\right\|_{2}=O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right).$
(62)

Next, we prove that
$\left\|{{\mathcal{T}}_{n}}^{1/2}{\Psi}_{n}^{-1}{{\mathcal{T}}_{n}}^{1/2}\right\|_{2}=O(1)$
and
$\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\Psi}_{n}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\right\|_{2}=O_{p}(1)$.
Notice that

$\displaystyle~{}\lim\inf_{n}\lambda_{\min}\left({{\mathcal{T}}_{n}}^{-1/2}{\Psi}_{n}{{\mathcal{T}}_{n}}^{-1/2}\right)$
$\displaystyle=$
$\displaystyle~{}\lim\inf_{n}\min_{\bm{a}:\|\bm{a}\|_{2}=1}\bm{a}^{T}{{\mathcal{T}}_{n}}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{\bm{K}}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{{\mathcal{T}}_{n}}^{-1/2}\bm{a}$
$\displaystyle=$
$\displaystyle~{}\lim\inf_{n}\min_{\bm{a}:\|\bm{a}\|_{2}=1}\bm{a}^{T}{{\mathcal{T}}_{n}}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{\bm{K}}_{n}^{-1}\bm{V}_{n}^{1/2}\cdot\bm{V}_{n}^{-1/2}{\bm{K}}_{n}\bm{V}_{n}^{-1/2}\cdot\bm{V}_{n}^{1/2}{\bm{K}}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{{\mathcal{T}}_{n}}^{-1/2}\bm{a}$
$\displaystyle\geq$
$\displaystyle~{}\lim\inf_{n}\lambda_{\min}\left(\bm{V}_{n}^{-1/2}{\bm{K}}_{n}\bm{V}_{n}^{-1/2}\right)\lambda_{\min}\left({{\mathcal{T}}_{n}}^{-1/2}{{\mathcal{T}}_{n}}{{\mathcal{T}}_{n}}^{-1/2}\right)>0,$
(63)

where the last inequality is from the definition of $\Psi_{n}$ as well as the
fact that $\lambda_{\max}(\bm{K}_{n}^{-1})=O(1)$ and
$\lambda_{\max}(\bm{V}_{n})=O(1)$. Similarly, we can prove result (48). By the
definition of $\Psi_{n}$ and ${\widehat{\mathcal{T}}_{n,a}}$, we have

$\displaystyle~{}\lim\inf_{n}\lambda_{\min}\left({\widehat{\mathcal{T}}_{n,a}}^{-1/2}{\Psi}_{n}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}\right)$
$\displaystyle=$
$\displaystyle~{}\lim\inf_{n}\min_{\bm{a}:\|\bm{a}\|_{2}=1}\bm{a}^{T}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\widehat{\bm{K}}_{n,a}^{-1}\widehat{\bm{V}}_{n,a}^{1/2}\cdot\widehat{\bm{V}}_{n,a}^{-1/2}\widehat{\bm{K}}_{n,a}{\bm{K}}_{n}^{-1}\widehat{\bm{K}}_{n,a}\widehat{\bm{V}}_{n,a}^{-1/2}\cdot\widehat{\bm{V}}_{n,a}^{1/2}\widehat{\bm{K}}_{n,a}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}\bm{a}$
$\displaystyle\geq$
$\displaystyle~{}\lim\inf_{n}\lambda_{\min}\left(\widehat{\bm{V}}_{n,a}^{-1/2}\widehat{\bm{K}}_{n,a}{\bm{K}}_{n}^{-1}\widehat{\bm{K}}_{n,a}\widehat{\bm{V}}_{n,a}^{-1/2}\right)>0,$
(64)

where the last inequality is because

$\displaystyle\left\|\widehat{\bm{V}}_{n,a}^{1/2}\widehat{\bm{K}}_{n,a}^{-1}{\bm{K}}_{n}\widehat{\bm{K}}_{n,a}^{-1}\widehat{\bm{V}}_{n,a}^{1/2}\right\|_{2}\leq\left\|{\bm{K}}_{n}\right\|_{2}\left\|\widehat{\bm{K}}_{n,a}^{-1}\widehat{\bm{V}}_{n,a}\widehat{\bm{K}}_{n,a}^{-1}\right\|_{2}=O_{p}(1)$

along with the fact that $\lambda_{\max}(\bm{K}_{n}^{-1})=O(1)$ and result
(42) proved before. Then, we have

$\displaystyle~{}\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\Psi}_{n}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{{\mathcal{T}}_{n}}^{1/2}{\Psi}_{n}^{-1}{{\mathcal{T}}_{n}}^{1/2}\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\Psi}_{n}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}({\widehat{\mathcal{T}}_{n,a}}^{-1/2}{\Psi}_{n}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}-{{\mathcal{T}}_{n}}^{-1/2}{\Psi}_{n}{{\mathcal{T}}_{n}}^{-1/2}){\widehat{\mathcal{T}}_{n,a}}^{1/2}{\Psi}_{n}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\right\|_{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\Psi}_{n}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\right\|_{2}\left\|{\widehat{\mathcal{T}}_{n,a}}^{-1/2}{\Psi}_{n}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}-{{\mathcal{T}}_{n}}^{-1/2}{\Psi}_{n}{{\mathcal{T}}_{n}}^{-1/2}\right\|_{2}\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\Psi}_{n}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\right\|_{2}$
$\displaystyle=$ $\displaystyle~{}O_{p}(1)\cdot
O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right)\cdot
O(1)=O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right).$

So it remains to prove equation (62). We start from proving that

$\left\|{\widehat{\mathcal{T}}_{n,a}}^{-1}{\Psi}_{n}^{2}{\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}^{2}{{\mathcal{T}}_{n}}^{-1}\right\|_{2}=O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right).$
(65)

Notice that

$\displaystyle~{}\left\|{\widehat{\mathcal{T}}_{n,a}}^{-1}{\Psi}_{n}^{2}{\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}^{2}{{\mathcal{T}}_{n}}^{-1}\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}\left\|({\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1}){\Psi}_{n}^{2}({\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1})+({\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1}){\Psi}_{n}^{2}{{\mathcal{T}}_{n}}^{-1}+{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}^{2}({\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1})\right\|_{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|{\Psi}_{n}({\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1})\right\|_{2}^{2}+2\left\|{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}\right\|_{2}\left\|{\Psi}_{n}({\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1})\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}O_{p}\left(\frac{(s+m+K)^{2}}{n}\right)+\left\|{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}\right\|_{2}O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right),$

where the last equation is from the result (45) proved before. And
$\left\|{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}\right\|_{2}=O(1)$ because

$\displaystyle~{}\left\|{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}\right\|_{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|{{\mathcal{T}}_{n}}^{-1/2}\right\|_{2}\left\|{{\mathcal{T}}_{n}}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{\bm{K}}_{n}^{-1}{\bm{V}}_{n}^{1/2}\cdot{\bm{V}}_{n}^{-1/2}{\bm{K}}_{n}{\bm{V}}_{n}^{-1/2}\cdot{\bm{V}}_{n}^{1/2}{\bm{K}}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\right\|_{2}$ $\displaystyle\leq$
$\displaystyle~{}\left\|{{\mathcal{T}}_{n}}^{-1/2}\right\|_{2}\left\|{{\mathcal{T}}_{n}}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{\bm{K}}_{n}^{-1}{\bm{V}}_{n}^{1/2}\right\|_{2}\left\|{\bm{V}}_{n}^{-1/2}{\bm{K}}_{n}{\bm{V}}_{n}^{-1/2}\right\|_{2}\left\|{\bm{V}}_{n}^{1/2}{\bm{K}}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\right\|_{2}$ $\displaystyle=$
$\displaystyle~{}\left\|{{\mathcal{T}}_{n}}^{-1/2}\right\|_{2}\cdot 1\cdot
O(1)\cdot\left\|{{\mathcal{T}}_{n}}^{1/2}\right\|_{2}=O(1).$

Along with the fact that $s+m+K=o(\sqrt{n})$, we proved the desired equation
(65). By applying Lemma 1, we have

$\displaystyle~{}\left\|{\widehat{\mathcal{T}}_{n,a}}^{-1/2}{\Psi}_{n}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}-{{\mathcal{T}}_{n}}^{-1/2}{\Psi}_{n}{{\mathcal{T}}_{n}}^{-1/2}\right\|_{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|{\widehat{\mathcal{T}}_{n,a}}^{-1}{\Psi}_{n}^{2}{\widehat{\mathcal{T}}_{n,a}}^{-1}-{{\mathcal{T}}_{n}}^{-1}{\Psi}_{n}^{2}{{\mathcal{T}}_{n}}^{-1}\right\|_{2}\frac{1}{\lambda_{\min}({\widehat{\mathcal{T}}_{n,a}}^{-1/2}{\Psi}_{n}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}+{{\mathcal{T}}_{n}}^{-1/2}{\Psi}_{n}{{\mathcal{T}}_{n}}^{-1/2})}.$

From results (63) and (64), we have
$1/{\lambda_{\min}({\widehat{\mathcal{T}}_{n,a}}^{-1/2}{\Psi}_{n}{\widehat{\mathcal{T}}_{n,a}}^{-1/2}+{{\mathcal{T}}_{n}}^{-1/2}{\Psi}_{n}{{\mathcal{T}}_{n}}^{-1/2})}=O_{p}(1)$.
Along with result (65), we proved the equation (62) as desired. This ends the
proof for equation (46).

∎

##### Proof of equation (49)

###### Proof.

First proof that $\lambda_{\max}(\mathcal{T}_{n}^{-1})=O(1)$. By definition of
$\mathcal{T}$, we have

$\displaystyle\lim\inf_{n}\lambda_{\min}(\mathcal{T}_{n})$
$\displaystyle=\lim\inf_{n}\min_{\bm{a}:\|\bm{a}\|_{2}=1}\bm{a}^{T}\mathcal{T}_{n}\bm{a}$
$\displaystyle=\lim\inf_{n}\min_{\bm{a}:\|\bm{a}\|_{2}=1}\bm{a}^{T}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{\bm{K}}_{n}^{-1}{\bm{V}}_{n}{\bm{K}}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\bm{a}$
$\displaystyle\geq\lim\inf_{n}\lambda_{\min}\left({\bm{K}}_{n}^{-1}{\bm{V}}_{n}{\bm{K}}_{n}^{-1}\right)\cdot\lim\inf_{n}\lambda_{\min}\left(\bm{C}\bm{C}^{T}\right)>0,$

where the last inequality is from the fact that
$\lambda_{\max}(\bm{K}_{n})=O(1)$, $\lambda_{\max}(\bm{V}_{n}^{-1})=O(1)$, as
well as $\lambda_{\max}((\bm{C}\bm{C}^{T})^{-1})=O(1)$ given in assumption
(A3). Along with the fact that
$\|\bm{h}_{n}\|_{2}=O(\sqrt{\min(m+s+K-r,r)/n})\leq O(\sqrt{r/n})$ given in
assumption (A3), we proved the desired result (49). ∎

###  A.2 Proof of Theorem 1

###### Proof.

Since the true coefficients satisfy $\mathcal{B}^{*}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0}$, we define an oracle estimator
$\hat{\mathcal{B}}$ as solution to

$\max_{\mathcal{B}}Q_{n}(\mathcal{B})~{}~{}\text{s.t.}~{}\bm{C}\mathcal{B}_{\mathcal{M}}=\bm{t},~{}\mathcal{B}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0}.$ (66)

##### Step 1: Show that
$\|\hat{\mathcal{B}}-\mathcal{B}^{*}\|_{2}=O_{p}(\sqrt{(m+s+K-r)/n})$.

Define a $(p+K)$-dimensional vector $\mathcal{B}_{0}$ as

$\begin{cases}\mathcal{B}_{0,\mathcal{M}}&=\mathcal{B}_{\mathcal{M}}^{*}-\bm{C}^{T}(\bm{C}\bm{C}^{T})^{-1}\bm{h}_{n},\\\
\mathcal{B}_{0,\mathcal{M}^{c}}&=\mathcal{B}_{\mathcal{M}^{c}}^{*}.\end{cases}$

It is immediate to see that

$\bm{C}\mathcal{B}_{0,\mathcal{M}}-\bm{t}=\bm{0}.$

And it follows from (A3) that

$\|\mathcal{B}_{0}-\mathcal{B}^{*}\|_{2}^{2}=\|\bm{C}^{T}(\bm{C}\bm{C}^{T})^{-1}\bm{h}_{n}\|_{2}^{2}\leq\lambda_{\max}((\bm{C}\bm{C}^{T})^{-1})\|\bm{h}_{n}\|_{2}^{2}\leq
O((m+s+K-r)/n).$ (67)

So it suffices to show that
$\|\hat{\mathcal{B}}-\mathcal{B}_{0}\|_{2}^{2}=O_{p}((m+s+K-r)/n)$.
Observe that $\bm{C}\mathcal{B}_{0,\mathcal{M}}=\bm{t}$, then for any
$\mathcal{B}$ with $\bm{C}\mathcal{B}_{\mathcal{M}}=\bm{t}$, we have
$\bm{C}(\mathcal{B}_{\mathcal{M}}-\mathcal{B}_{0,\mathcal{M}})=\bm{0}$, i.e.,
$\mathcal{B}_{\mathcal{M}}-\mathcal{B}_{0,\mathcal{M}}\in Null(\bm{C})$. Take
basis matrix $\bm{Z}\in\mathbb{R}^{m\times(m-r)}$ of $Null(\bm{C})$, i.e.,
$\bm{C}\bm{Z}=\bm{0}$, and $\bm{Z}^{T}\bm{Z}=\bm{I}_{m-r}$. Then for any
$\mathcal{B}_{\mathcal{M}}$ s.t. $\bm{C}\mathcal{B}_{\mathcal{M}}=\bm{t}$,
$\exists\bm{v}\in\mathbb{R}^{m-r}$, s.t.
$\mathcal{B}_{\mathcal{M}}=\mathcal{B}_{0,\mathcal{M}}+\bm{Z}\bm{v}$. Then
define $\overline{Q}_{n}(\bm{\delta})=Q_{n}(\mathcal{B}(\bm{\delta}))$ for any
$\bm{\delta}\in\mathbb{R}^{(m-r)+s+K}$ where $\mathcal{B}(\bm{\delta})$ is
defined as

$\begin{cases}\mathcal{B}(\bm{\delta})_{\mathcal{M}}&=\mathcal{B}_{0,\mathcal{M}}+\bm{Z}\bm{\delta}_{J_{0}},\\\
\mathcal{B}(\bm{\delta})_{S\cup\mathcal{K}}&=\mathcal{B}_{0,S\cup\mathcal{K}}+\bm{\delta}_{J_{0}^{c}},\\\
\mathcal{B}(\bm{\delta})_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}&=\mathcal{B}_{0,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}},\end{cases}$

where $J_{0}=[1,2,\dots,m-r]$. Then solving the constrained optimization
problem (66) is equivalent to solving
$\max_{\bm{\delta}\in\mathbb{R}^{(m-r)+s+K}}\overline{Q}_{n}(\bm{\delta})$.
Since $\|\bm{Z}\bm{\delta}_{J_{0}}\|_{2}^{2}=\|\bm{\delta}_{J_{0}}\|_{2}^{2}$,
it suffices to show that there exists a local maximizer $\bm{\delta}_{0}$ of
$\overline{Q}_{n}(\bm{\delta})$ such that
$\|\bm{\delta}_{0}\|_{2}=O_{p}(\sqrt{(m+s+K-r)/n})$.

Define a neighborhood
$\mathcal{N}_{\tau}=\\{\bm{\delta}\in\mathbb{R}^{(m-r)+s+K}:\|\bm{\delta}\|_{2}\leq\tau\sqrt{(m+s+K-r)/n}\\}$,
and an event
$H_{n}:=\left\\{\overline{Q}_{n}(\bm{0})>\max_{\bm{\delta}\in{\partial\mathcal{N}_{\tau}}}\overline{Q}_{n}(\bm{\delta})\right\\}$,
where ${\partial\mathcal{N}_{\tau}}$ denotes the boundary of
${\mathcal{N}_{\tau}}$. On the event $H_{n}$, there must exist a local
maximizer in $\mathcal{N}_{\tau}$. Hence, it suffices to show
$\mathbb{P}(H_{n})\rightarrow 1$ as $n\rightarrow\infty$ and some sufficiently
large $\tau$.

For any $\bm{\delta}\in\partial\mathcal{N}_{\tau}$, by the second-order
Taylor’s expansion

$\overline{Q}_{n}(\bm{\delta})-\overline{Q}_{n}(\bm{0})=\bm{\delta}^{T}\bm{\nu}-\frac{1}{2}\bm{\delta}^{T}\bm{D}\bm{\delta},$
(68)

where

$\displaystyle\bm{\nu}$
$\displaystyle=\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
\bm{X}_{S}^{T}\\\
-\bm{E}_{k}^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}\mathcal{B}_{0})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\mathcal{B}_{0})\right\\}-\begin{pmatrix}\bm{0}_{m-r}\\\
\lambda_{n,0}\bar{\rho}(\mathcal{B}_{0,S},\lambda_{n,0})\\\
\bm{0}_{K}\end{pmatrix},$ $\displaystyle\bm{D}$
$\displaystyle=\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
\bm{X}_{S}^{T}\\\
-\bm{E}_{k}^{T}\end{pmatrix}\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
\bm{X}_{S}^{T}\\\
-\bm{E}_{k}^{T}\end{pmatrix}^{T}-\begin{pmatrix}\bm{0}_{(m-r)\times(m-r)}&\bm{0}_{(m-r)\times
s}&\bm{0}_{(m-r)\times K}\\\ \bm{0}_{s\times(m-r)}&\Lambda_{0}&\bm{0}_{s\times
K}\\\ \bm{0}_{K\times(m-r)}&\bm{0}_{K\times s}&\bm{0}_{K\times
K}\end{pmatrix},$

where $\mathcal{B}^{**}$ lies in the line segment joining
$\mathcal{B}(\bm{\delta})$ and $\mathcal{B}_{0}$, and $\Lambda_{0}$ is a
$r\times r$ diagonal matrix with non-negative diagonal elements and

$\displaystyle\bar{\rho}(\mathcal{B}_{S},\lambda_{n,0})$
$\displaystyle=\left(\text{sign}(\mathcal{B}_{j})\rho^{\prime}(|\mathcal{B}_{j}|,\lambda_{n,0}),j\in
S\right)^{T}$ $\displaystyle\bm{H}(\bm{X}^{k}\mathcal{B})$
$\displaystyle=\text{diag}\left\\{h^{\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}),~{}i=1,\dots,n\right\\},$
$\displaystyle\bm{\mu}(\bm{X}^{k}\mathcal{B})$
$\displaystyle=\begin{pmatrix}\Phi((\bm{x}_{1}^{k})^{T}\mathcal{B}),\cdots,\Phi((\bm{x}_{n}^{k})^{T}\mathcal{B})\end{pmatrix}^{T},$
$\displaystyle\hat{\Sigma}(\bm{X}^{k}\mathcal{B})$
$\displaystyle=\text{diag}\left\\{\tilde{y}_{ki}\ell^{\prime\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B})+(1-\tilde{y}_{ki})\ell^{\prime\prime}(-(\bm{x}_{i}^{k})^{T}\mathcal{B}),~{}i=1,\dots,n\right\\},$

where $\ell(t)=-\log(\Phi(t))$ is the probit loss function introduced and
studied in Zhou and Zou 2023 with $\ell(t)^{\prime\prime}$ bounded within
$[0,1]$. By the definition of $\mathcal{B}(\bm{\delta})$ and
$\mathcal{B}_{0}$, we have $\mathcal{B}^{**}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0}$. Moreover, it follows from equation (67) and
$\bm{\delta}\in\partial\mathcal{N}_{\tau}$ that

$\displaystyle\|\mathcal{B}^{**}-\mathcal{B}^{*}\|_{2}$
$\displaystyle\leq\tau\sqrt{(s-r+m+K)/n}+O(\sqrt{(s-r+m+K)/n})$
$\displaystyle\ll\sqrt{(s+m+K)\log(n)/n},$ (69)

for $n$ sufficiently large and for $\tau\ll\sqrt{\log{n}}$. Therefore, we have
$\mathcal{B}^{**}\in\mathcal{N}^{*}$. By definition of $\kappa_{0,0}$, we have
$\lambda_{\max}(\Lambda_{0})\leq\lambda_{n,0}\kappa_{0,0}$. Let

$\bm{L}=\begin{pmatrix}\bm{Z}&\bm{0}_{m\times(s+K)}\\\
\bm{0}_{(s+K)\times(m-r)}&\bm{I}_{s+K}\end{pmatrix},$

we have

$\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}=\sum_{k=1}^{K}w_{k}\frac{1}{n}\bm{L}^{T}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\bm{L}.$

Then on the event defined below:

$E_{n}:=\left\\{\left\|\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}<\frac{c}{8}\right\\},$
(70)

where
${\Sigma}(\bm{X}^{k}\mathcal{B})=\text{diag}\left\\{\frac{\varphi^{2}}{\Phi(1-\Phi)}((\bm{x}_{i}^{k})^{T}\mathcal{B}),~{}i=1,\dots,n\right\\}$,
and $c$ is the constant in the 6th condition of (A1), we must have

$\lambda_{\min}\left\\{\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\\}\geq\frac{3}{4}c,$
(71)

for sufficiently large $n$. This is because for any
$\bm{\delta}\in\mathbb{R}^{m-r+s+K}$, notice that
$\|\bm{L}\bm{\delta}\|_{2}=\|\bm{\delta}\|_{2}$, then

$\displaystyle~{}\bm{\delta}^{T}\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\bm{\delta}$
$\displaystyle=$
$\displaystyle~{}(\bm{L}\bm{\delta})^{T}\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}\mathcal{B}^{**})+\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}(\bm{L}\bm{\delta})$
$\displaystyle\geq$
$\displaystyle~{}\lambda_{\min}\left(\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)\|\bm{L}\bm{\delta}\|_{2}^{2}$
$\displaystyle-$
$\displaystyle~{}\lambda_{\max}\left(\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)\|\bm{L}\bm{\delta}\|_{2}^{2}$
$\displaystyle\geq$
$\displaystyle~{}c\|\bm{\delta}\|_{2}^{2}-\lambda_{\max}(\bm{U_{n}})\|\bm{\delta}\|_{2}^{2},$
(72)

where the last inequality is from the fact that
$\mathcal{B}^{**}\in\mathcal{N}^{*}$ and the 6th condition of (A1), and

$\bm{U}_{n}:=\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}.$

Since

$\displaystyle\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})$
$\displaystyle=\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})-\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})\right)+\left({\Sigma}(\bm{X}^{k}\mathcal{B}^{*})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\right)$
$\displaystyle+\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})\right),$

we have $\bm{U}_{n}=\bm{U}_{n}^{1}+\bm{U}_{n}^{2}+\bm{U}_{n}^{3}$, where

$\displaystyle\bm{U}_{n}^{1}$
$\displaystyle=\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})-\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T},$
$\displaystyle\bm{U}_{n}^{2}$
$\displaystyle=\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}\mathcal{B}^{*})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T},$
$\displaystyle\bm{U}_{n}^{3}$
$\displaystyle=\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}.$

By the 1st condition of (A1) and the fact that
$\|\mathcal{B}^{**}-\mathcal{B}^{*}\|_{2}\ll\sqrt{(s+m+K)\log(n)/n}$ as long
as $\tau\ll\sqrt{\log(n)}$ proved in (69), we have $\max_{1\leq i\leq
n}\max_{1\leq k\leq
K}|(\bm{x}_{i}^{k})^{T}(\mathcal{B}^{**}-\mathcal{B}^{*})|=o(1)$. This,
combined with the 4th condition of (A1), implies that
$\lambda_{\max}(\bm{U}_{n}^{1})<c/16$ and
$\lambda_{\max}(\bm{U}_{n}^{2})<c/16$ when $n$ is sufficiently large.
Therefore, on event $E_{n}$, we must have

$\lambda_{\max}(\bm{U}_{n})\leq\lambda_{\max}(\bm{U}_{n}^{1})+\lambda_{\max}(\bm{U}_{n}^{2})+\lambda_{\max}(\bm{U}_{n}^{3})\leq\frac{c}{4},$

for sufficiently large $n$. This, combined with inequality (72), implies that

$\displaystyle\bm{\delta}^{T}\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{**})\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\bm{\delta}\geq
c\|\bm{\delta}\|_{2}^{2}-\frac{c}{4}\|\bm{\delta}\|_{2}^{2}=\frac{3c}{4}\|\bm{\delta}\|_{2}^{2},$

i.e., inequality (71) holds on event $E_{n}$.

By condition $\lambda_{n,0}\kappa_{0,0}=o(1)$ in (A2), we have
$\lambda_{\max}(\Lambda_{0})=o(1)$. Thus, combining with inequality (71), we
have $\lambda_{\min}(\bm{D})\geq\frac{1}{2}{c}$ for some sufficiently large
$n$. Therefore, on the event $E_{n}$ defined in (70), it follows from equation
(68) that

$\displaystyle\sup_{\bm{\delta}\in\partial\mathcal{N}_{\tau}}\overline{Q}_{n}(\bm{\delta})-\overline{Q}_{n}(\bm{0})$
$\displaystyle\leq\sup_{\bm{\delta}\in\partial\mathcal{N}_{\tau}}\left(\|\bm{\delta}\|_{2}\|\bm{\nu}\|_{2}-\frac{1}{2}\lambda_{\min}(D)\|\bm{\delta}\|_{2}^{2}\right)$
$\displaystyle\leq\sup_{\bm{\delta}\in\partial\mathcal{N}_{\tau}}\left(\|\bm{\delta}\|_{2}\|\bm{\nu}\|_{2}-\frac{c}{4}\|\bm{\delta}\|_{2}^{2}\right)$
$\displaystyle=\|\bm{\nu}\|_{2}\tau\sqrt{\frac{m-r+s+K}{n}}-\frac{c\tau^{2}}{4}\frac{(m-r+s+K)}{n}.$

By Markov’s inequality,

$\displaystyle\mathbb{P}(H_{n})$ $\displaystyle\geq\mathbb{P}(H_{n}\cap
E_{n})$
$\displaystyle\geq\mathbb{P}\left(\left\\{\|\bm{\nu}\|_{2}<\bar{c}\tau\sqrt{\frac{m-r+s+K}{n}}\right\\}\cap
E_{n}\right)$
$\displaystyle\geq\mathbb{P}\left(\|\bm{\nu}\|_{2}<\bar{c}\tau\sqrt{\frac{m-r+s+K}{n}}\right)-\mathbb{P}(E_{n}^{c})$
$\displaystyle\geq
1-\frac{n\mathbb{E}\|\bm{\nu}\|_{2}^{2}}{\bar{c}^{2}\tau^{2}(m-r+s+K)}-\mathbb{P}(E_{n}^{c}),$

for any fixed $\tau$ satisfying $\tau\leq\frac{1}{2}\sqrt{\log n}$ and for $n$
sufficiently large.

Next show that $\mathbb{E}\|\bm{\nu}\|_{2}^{2}=O((m-r+s+K)/n)$. By Cauchy-
Schwartz inequality,

$\displaystyle\mathbb{E}\|\bm{\nu}\|_{2}^{2}$ $\displaystyle\leq
2\|\lambda_{n,0}\bar{\rho}(\mathcal{B}_{0,S};\lambda_{n,0})\|_{2}^{2}+2\mathbb{E}\left\|\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}\mathcal{B}_{0})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\mathcal{B}_{0})\right\\}\right\|_{2}^{2}$
$\displaystyle\overset{\Delta}{=}2I_{0}+2I_{1}$

By the concavity of $\rho$, its derivative $\rho^{\prime}$ is monotone
decreasing. Along with the definition of $d_{n}$, and the 2nd condition of
(A2), we have

$I_{0}\leq s\left\\{\lambda_{n,0}\rho^{\prime}(d_{n})\right\\}^{2}=o(1/n).$

Next show that $I_{1}=O(\frac{m-r+s+K}{n})$:

$\displaystyle I_{1}$
$\displaystyle=\frac{1}{n^{2}}\mathbb{E}\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}\mathcal{B}_{0})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\mathcal{B}_{0})\right\\}\right\|_{2}^{2}$
$\displaystyle\leq\frac{1}{n^{2}}\mathbb{E}\left[\sum_{k=1}^{K}w_{k}\left\|\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}\mathcal{B}_{0})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\mathcal{B}_{0})\right\\}\right\|_{2}^{2}\right]$
$\displaystyle=\frac{1}{n}\text{tr}\left[\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\text{diag}\left\\{(h^{\prime})^{2}((\bm{x}_{i}^{k})^{T}\mathcal{B}_{0})\cdot\Phi(1-\Phi)((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*}),i=1,\dots,n\right\\}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right]$
$\displaystyle+\frac{1}{n}\cdot\frac{1}{n}\sum_{k=1}^{K}w_{k}\left\|\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}\mathcal{B}_{0})\left\\{\bm{\mu}(\bm{X}^{k}\mathcal{B}_{0})-\bm{\mu}(\bm{X}^{k}\mathcal{B}^{*})\right\\}\right\|_{2}^{2}$
$\displaystyle\overset{\Delta}{=}\frac{1}{n}I_{2}+\frac{1}{n}I_{3},$

where the first inequality is by the convexity of $\|\cdot\|_{2}^{2}$.

To bound $I_{2}$, consider for any $i=1,\dots,n$, $k=1,\dots,K$,

$\displaystyle~{}(h^{\prime})^{2}((\bm{x}_{i}^{k})^{T}\mathcal{B}_{0})\cdot\Phi(1-\Phi)((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})$
$\displaystyle=$
$\displaystyle~{}\left[h^{\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})+h^{\prime\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}^{**})\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})\right]^{2}\cdot\Phi(1-\Phi)((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})$
$\displaystyle=$
$\displaystyle~{}\frac{\varphi^{2}}{\Phi(1-\Phi)}((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})+2\varphi((\bm{x}_{i}^{k})^{T}\mathcal{B})h^{\prime\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}^{**})\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})$
$\displaystyle~{}+\left[h^{\prime\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}^{**})\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})\right]^{2}\cdot\Phi(1-\Phi)((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})$
$\displaystyle\leq$
$\displaystyle~{}4{\varphi(0)^{2}}+O(\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}))+O(\left[\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})\right]^{2})$
$\displaystyle=$ $\displaystyle~{}4{\varphi(0)^{2}}+o(1)$

where the first equality is from the Taylor’s expansion and
$\mathcal{B}_{\mathcal{M}^{c}}^{*}=\mathcal{B}_{0,\mathcal{M}^{c}}$; the
inequality is from the fact that
$\frac{\varphi^{2}}{\Phi(1-\Phi)}(t)\leq\frac{\varphi^{2}}{\Phi(1-\Phi)}(0)$,
$\varphi(t)\leq\varphi(0)$, $|h^{\prime\prime}(t)|\leq 1$ and
$\Phi(1-\Phi)(t)\leq 1/4$; the last equality is because
$\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})=o(1)$
derived from the 1st condition of (A1) and the fact that
$\|\mathcal{B}_{\mathcal{M}}^{*}-\mathcal{B}_{0,\mathcal{M}}\|_{2}=O(\sqrt{(m-r+s+K)/n})$.
Therefore, we have

$\displaystyle I_{2}\leq$
$\displaystyle~{}\text{tr}\left[\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\begin{pmatrix}\bm{Z}^{T}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right]\cdot O(1)$
$\displaystyle=$
$\displaystyle~{}\text{tr}\left[\bm{L}^{T}\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\bm{L}\right]\cdot
O\left(1\right)$ $\displaystyle\leq$
$\displaystyle~{}(m-r+s+K)\cdot\lambda_{\max}\left(\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)\cdot
O\left(1\right).$

Along with the 4th condition of (A1), we have $I_{2}/n=O((m-r+s+K)/n)$.

To bound $I_{3}$, consider the $i$-th coordinate of vector
$\bm{H}(\bm{X}^{k}\mathcal{B}_{0})\left(\bm{\mu}(\bm{X}^{k}\mathcal{B}_{0})-\bm{\mu}(\bm{X}^{k}\mathcal{B}^{*})\right)$
for any $i=1,\dots,n$, $k=1,\dots,K$,

$\displaystyle~{}h^{\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}_{0})\cdot\left(\Phi((\bm{x}_{i}^{k})^{T}\mathcal{B}_{0})-\Phi((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})\right)$
$\displaystyle=$
$\displaystyle~{}\left.\left[h^{\prime\prime}(\eta)\left(\Phi(\eta)-\Phi((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})\right)+h^{\prime}(\eta)\varphi(\eta)\right]\right|_{\eta=(\bm{x}_{i}^{k})^{T}\mathcal{B}^{**}}\cdot\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})$
$\displaystyle=$
$\displaystyle~{}O(\left[\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})\right]^{2})+O(\bm{x}_{i,\mathcal{M}}^{T}(\mathcal{B}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}))$

where the last equality is from the fact that $|h^{\prime\prime}(t)|\leq 1$,
$|\Phi(t_{1})-\Phi(t_{2})|\leq\varphi(0)|t_{1}-t_{2}|$, and
$h^{\prime}(t)\varphi(t)=\frac{\varphi^{2}}{\Phi(1-\Phi)}(t)\leq\frac{\varphi^{2}}{\Phi(1-\Phi)}(0)$.
Along with the 4th condition of (A1), we have $I_{3}/n=o(1/n)$. Therefore,
$I_{1}=O((m-r+s+K)/n)$, as desired.

It remains to bound

$\mathbb{P}(E_{n}^{c})=\mathbb{P}\left(\left\|\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}\geq\frac{c}{8}\right).$

Denote

$\displaystyle{\mathcal{R}_{n}}=1+\max_{1\leq i\leq
n}\|\bm{x}_{i,\mathcal{M}\cup S}\|_{2}^{2},~{}T_{n}=\max_{1\leq k\leq
K}\lambda_{\max}\left(\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)$

By the matrix Chernoff bound, we have

$\displaystyle~{}\mathbb{P}\left(\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}>t\right)$
$\displaystyle\leq$
$\displaystyle~{}(s+m+K)\exp\left(-\frac{t^{2}}{{\mathcal{R}_{n}}T_{n}+{\mathcal{R}_{n}}t/3}\right)$
$\displaystyle=$
$\displaystyle~{}\exp\left(-\frac{t^{2}}{{\mathcal{R}_{n}}(T_{n}+t/3)}+\log(s+m+K)\right)$
(73)

From the 1st and 4th conditions of (A1), we have
${\mathcal{R}_{n}}=O\left({\frac{n}{(s+m+K)\log n}}\right)$, $T_{n}=O(n)$.
Then there exists constants $c_{4}$, $c_{5}$ such that
${\mathcal{R}_{n}}\leq{\frac{c_{4}n}{(s+m+K)\log n}}$, $T_{n}\leq c_{5}n$ for
sufficiently large $n$. Thus

$\displaystyle~{}\mathbb{P}\left(\left\|\sum_{k=1}^{K}w_{k}\frac{1}{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left(\hat{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})-{\Sigma}(\bm{X}^{k}\mathcal{B}^{*})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}\geq\frac{c}{8}\right)$
$\displaystyle\leq$
$\displaystyle~{}\exp\left(-\tilde{c}{(s+m+K)\log(n)}+\log(s+m+K)\right)$
$\displaystyle\leq$
$\displaystyle~{}\exp\left(-\frac{\tilde{c}}{2}{(s+m+K)\log(n)}\right),$

for some constant $\tilde{c}$ and sufficiently large $n$.

##### Step 2: Show that with probability tending to 1, the oracle local
maximizer $\hat{\mathcal{B}}$ is indeed a maximizer of $Q_{n}(\mathcal{B})$
with the linear constraint $\bm{C}\mathcal{B}_{\mathcal{M}}=\bm{t}$.

From step 1, we have for some constant $c$, and any $\tau\ll{\sqrt{\log n}}$,

$\mathbb{P}\left(\|\hat{\mathcal{B}}-\mathcal{B}^{*}\|_{2}\leq\tau\sqrt{\frac{m-r+s+K}{n}}\right)\geq
1-c\frac{1}{\tau^{2}}-\exp\left(-\frac{\tilde{c}}{2}{(s+m+K)\log(n)}\right).$

Similar to Theorem 1 in Fan and Lv 2011, it suffices to show that with
probability tending to 1, $\hat{\mathcal{B}}$ satisfies the following
inequality:

$\left\|\sum_{k=1}^{K}w_{k}(\bm{X}^{k}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}})^{T}\bm{H}(\bm{X}^{k}\hat{\mathcal{B}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\hat{\mathcal{B}})\right\\}\right\|_{\infty}<n\lambda_{n,0}{\rho}^{\prime}(0+).$

By the Taylor’s expansion, $\forall j\in(\mathcal{M}\cup
S\cup\mathcal{K})^{c}$,

$\displaystyle\sum_{k=1}^{K}w_{k}\bm{x}_{(j)}^{T}\bm{H}(\bm{X}^{k}\hat{\mathcal{B}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\hat{\mathcal{B}})\right\\}$
$\displaystyle=$
$\displaystyle\sum_{k=1}^{K}w_{k}\bm{x}_{(j)}^{T}\bm{H}(\bm{X}^{k}{\mathcal{B}^{*}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\mathcal{B}^{*}})\right\\}-\sum_{k=1}^{K}w_{k}\bm{x}_{(j)}^{T}\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\bm{X}^{k}(\hat{\mathcal{B}}-\mathcal{B}^{*})+R_{j},$
(74)

where

$R_{j}=\sum_{k=1}^{K}w_{k}\hat{\bm{\Delta}}_{{\mathcal{M}\cup
S\cup\mathcal{K}}}^{T}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\text{diag}\left\\{\bm{x}_{(j)}\circ\hat{\Sigma}^{\prime}(\bm{X}^{K}\bar{\mathcal{B}}_{j})\right\\}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\bm{\Delta}}_{{\mathcal{M}\cup
S\cup\mathcal{K}}},$

for some $\bar{\mathcal{B}}_{j}$ lying on the line segment joining
$\mathcal{B}^{*}$ and $\hat{\mathcal{B}}$, where
$\hat{\bm{\Delta}}_{{\mathcal{M}\cup
S\cup\mathcal{K}}}=\hat{\mathcal{B}}_{\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup S\cup\mathcal{K}}$. Under
the event

$\left\\{\|\hat{\mathcal{B}}-\mathcal{B}^{*}\|_{2}\leq\tau\sqrt{{(m-r+s+K)}/{n}}\right\\},$
(75)

we have $\hat{\mathcal{B}}\in\mathcal{N}^{*}$. Along with
$\mathcal{B}^{*}\in\mathcal{N}^{*}$, we have
$\bar{\mathcal{B}}_{j}\in\mathcal{N}^{*}$. Then under the same event (75), it
follows from the 5th condition of (A1) that

$\displaystyle|R_{j}|$
$\displaystyle\leq\sum_{k=1}^{K}w_{k}\hat{\bm{\Delta}}_{{\mathcal{M}\cup
S\cup\mathcal{K}}}^{T}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\text{diag}\left\\{|\bm{x}_{(j)}|\circ|\hat{\Sigma}^{\prime}(\bm{X}^{K}\bar{\mathcal{B}}_{j})|\right\\}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\bm{\Delta}}_{{\mathcal{M}\cup
S\cup\mathcal{K}}},$ $\displaystyle\leq
n\lambda_{\max}\left(\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\text{diag}\left\\{|\bm{x}_{(j)}|\right\\}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)\cdot\|\hat{\bm{\Delta}}_{{\mathcal{M}\cup
S\cup\mathcal{K}}}\|_{2}^{2}$ $\displaystyle\leq
c_{1}n\cdot\tau^{2}{\frac{m-r+s+K}{n}}$
$\displaystyle=c_{1}\tau^{2}{(m-r+s+K)}.$

Therefore, we have

$\sup_{j\in(\mathcal{M}\cup S\cup\mathcal{K})^{c}}|R_{j}|\leq
c_{1}\tau^{2}{(m-r+s+K)}.$ (76)

For the second term of the RHS of inequality (74), we can prove using 2nd and
4th conditions of (A1) that

$\left\|\frac{1}{n}\sum_{k=1}^{K}w_{k}\bm{X}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}^{T}\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\bm{X}_{\mathcal{M}\cup
S\cup\mathcal{K}}^{k}\right\|_{2,\infty}=O(1).$

Therefore, under the same event (75), we have

$\displaystyle~{}\left\|\sum_{k=1}^{K}w_{k}\bm{X}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}^{T}\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\bm{X}^{k}(\hat{\mathcal{B}}-\mathcal{B}^{*})\right\|_{\infty}$
$\displaystyle\leq$
$\displaystyle~{}\left\|\sum_{k=1}^{K}w_{k}\bm{X}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}^{T}\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\bm{X}_{\mathcal{M}\cup
S\cup\mathcal{K}}^{k}\right\|_{2,\infty}\cdot\left\|\hat{\mathcal{B}}_{\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\right\|_{2}$ $\displaystyle\leq$
$\displaystyle~{}c_{2}n\cdot\tau\sqrt{\frac{m-r+s+K}{n}}$ $\displaystyle=$
$\displaystyle~{}c_{2}\tau\sqrt{(m-r+s+K)n}.$ (77)

For the first term in the RHS of inequality (74), by Hoeffding’s inequality,
given some constant $\gamma>0$, $\forall j\in(\mathcal{M}\cup
S\cup\mathcal{K})^{c}$,

$\displaystyle\mathbb{P}\left(\left|\sum_{k=1}^{K}w_{k}\bm{x}_{(j)}^{T}\bm{H}(\bm{X}^{k}{\mathcal{B}^{*}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\mathcal{B}^{*}})\right\\}\right|>\gamma\sqrt{n\log
p}\right)$ $\displaystyle\leq 2\exp\left(-\frac{2\gamma^{2}n\log
p}{\sum_{i=1}^{n}x_{ij}^{2}[\sum_{k=1}^{K}w_{k}h^{\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})]^{2}}\right).$

By the 2nd and 3rd conditions of (A1) and $|h^{\prime\prime}(\cdot)|\leq 1$,
we have

${\sum_{i=1}^{n}\left[\sum_{k=1}^{K}w_{k}|h^{\prime}((\bm{x}_{i}^{k})^{T}\mathcal{B}^{*})\cdot
x_{ij}|\right]^{2}}=O(n).$

Therefore, combined with the previous inequality, we have

$\displaystyle\mathbb{P}\left(\left|\sum_{k=1}^{K}w_{k}\bm{x}_{(j)}^{T}\bm{H}(\bm{X}^{k}{\mathcal{B}^{*}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\mathcal{B}^{*}})\right\\}\right|>\gamma\sqrt{n\log
p}\right)\leq 2\exp\left(-\frac{2\gamma^{2}\log p}{M}\right)$

for some constant $M$ and for $n$ sufficiently large. Thus, by Bonferroni’s
inequality, we have

$\displaystyle~{}\mathbb{P}\left(\left\|\sum_{k=1}^{K}w_{k}\bm{X}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}^{T}\bm{H}(\bm{X}^{k}{\mathcal{B}^{*}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\mathcal{B}^{8}})\right\\}\right\|_{\infty}>\gamma\sqrt{n\log
p}\right)$ $\displaystyle\leq$
$\displaystyle~{}2\exp\left(-\frac{2\gamma^{2}\log p}{M}+\log(p-m-s)\right)$
$\displaystyle\leq$ $\displaystyle~{}2\exp\left(-\frac{\gamma^{2}\log
p}{M}\right),$

as long as $\gamma^{2}\geq M$.

Combining with inequalities (76), and (77), we have under event (75) and

$\left\\{\left\|\sum_{k=1}^{K}w_{k}\bm{X}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}^{T}\bm{H}(\bm{X}^{k}{\mathcal{B}^{*}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\mathcal{B}^{*}})\right\\}\right\|_{\infty}\leq\gamma\sqrt{n\log
p}\right\\},$

we have

$\displaystyle~{}\left\|\sum_{k=1}^{K}w_{k}(\bm{X}^{k}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}})^{T}\bm{H}(\bm{X}^{k}\hat{\mathcal{B}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\hat{\mathcal{B}})\right\\}\right\|_{\infty}$
$\displaystyle\leq$ $\displaystyle~{}\gamma\sqrt{n\log
p}+c_{2}\tau\sqrt{(m-r+s+K)n}+c_{1}\tau^{2}{(m-r+s+K)}$ $\displaystyle<$
$\displaystyle~{}n\lambda_{n,0}{\rho}^{\prime}(0+),$

as long as we have
$\gamma<\frac{{\rho}^{\prime}(0+)}{3}\lambda_{n,0}\sqrt{{n}/{\log p}}$,
$\tau<\frac{{\rho}^{\prime}(0+)}{3c_{2}}\lambda_{n,0}\sqrt{{n}/{(m-r+s+K)}}$,
and

$\tau^{2}<\frac{{\rho}^{\prime}(0+)}{3c_{1}}{\frac{\lambda_{n,0}n}{(m-r+s+K)}}.$

By the 3rd condition of (A2) and $s+m+K=o(\sqrt{n})$, we know that
$\lambda_{n,0}\sqrt{{n}/{\log p}}$, $\lambda_{n,0}\sqrt{{n}/{(m-r+s+K)}}$, and
$\lambda_{n,0}{{n}/{(m-r+s+K)}}$ could be arbitrarily large if $n$ is
sufficiently large. Therefore, by choosing a diverging sequence of
$\\{\tau_{n}\\}$ and a sufficiently large positive constant $\gamma$ such that
all the previous inequality constraints on $\gamma$ and $\tau$ hold, we could
have

$\displaystyle~{}\mathbb{P}\left(\left\|\sum_{k=1}^{K}w_{k}(\bm{X}^{k}_{(\mathcal{M}\cup
S\cup\mathcal{K})^{c}})^{T}\bm{H}(\bm{X}^{k}\hat{\mathcal{B}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\hat{\mathcal{B}})\right\\}\right\|_{\infty}<n\lambda_{n,0}{\rho}^{\prime}(0+).\right)$
$\displaystyle\geq$
$\displaystyle~{}1-c\frac{1}{\tau^{2}}\exp\left(-\frac{\tilde{c}}{2}{(s+m+K)\log(n)}\right)-2\exp\left(-\frac{\gamma^{2}\log
p}{M}\right)\rightarrow 1,$

as desired.

##### Step 3: Derive the asymptotic distribution of $\hat{\mathcal{B}}_{0}$

We have shown in Step 1 and Step 2 that for any $\tau\ll\sqrt{\log n}$ and
sufficiently large $n$,

$\displaystyle\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{2}=O_{p}\left(\sqrt{\frac{m-r+s+K}{n}}\right),$ (78)
$\displaystyle\mathbb{P}\left(\hat{\mathcal{B}}_{0,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0}\right)\rightarrow 1,$ (79)
$\displaystyle\bm{C}\hat{\mathcal{B}}_{0,\mathcal{M}}=\bm{t}.$ (80)

Finally, we show that

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}(\bm{I}-\bm{P}_{n})\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}\mathcal{B}^{*})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}\mathcal{B}^{*})\right\\}$
$\displaystyle-\sqrt{n}\bm{K}_{n}^{-1/2}\bm{P}_{n}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}\bm{h}_{n}\\\
\bm{0}_{s+K}\end{pmatrix}+o_{p}(1).$

From the results of Step 1 and Step 2, we know that with probability tending
to 1, $\hat{\mathcal{B}}_{0}$ is the local maximizer of $Q_{n}(\mathcal{B})$
with the constraints $\bm{C}\mathcal{B}_{\mathcal{M}}=\bm{t}$, and
$\mathcal{B}_{(\mathcal{M}\cup S\cup\mathcal{K})^{c}}=\bm{0}$. This implies
that there exists some vector $\bm{v}\in\mathbb{R}^{r}$ such that

$\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
\bm{X}_{S}^{T}\\\
(\bm{X}^{k}_{\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}=\begin{pmatrix}\sqrt{n}\bm{C}^{T}\bm{v}\\\
n\lambda_{n,0}\bar{\rho}({\hat{\mathcal{B}}_{0,S}})\\\
\bm{0}_{K}\end{pmatrix}$ (81)

By Taylor’s expansion, the LHS of equation (81) is equal to

$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle+$
$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}+\bm{R},$
(82)

where

$R_{j}=\sum_{k=1}^{K}w_{k}\hat{\Delta}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}^{T}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\text{diag}\left\\{\bm{x}^{k}_{(j)}\circ\hat{\Sigma}^{\prime}(\bm{X}^{K}\bar{\mathcal{B}}_{j})\right\\}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{0,\mathcal{M}\cup
S\cup\mathcal{K}},$

for some $\bar{\mathcal{B}}_{j}$ lying on the line segment joining
$\mathcal{B}^{*}$ and $\hat{\mathcal{B}}_{0}$, and
$\hat{\Delta}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}=\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup S\cup\mathcal{K}}$. For
$j\in\mathcal{M}\cup S$,

$\displaystyle|R_{j}|$
$\displaystyle\leq\sum_{k=1}^{K}w_{k}\hat{\Delta}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}^{T}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\text{diag}\left\\{|\bm{x}_{(j)}|\circ|\hat{\Sigma}^{\prime}(\bm{X}^{K}\bar{\mathcal{B}}_{j})|\right\\}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{0,\mathcal{M}\cup
S\cup\mathcal{K}},$ $\displaystyle\leq
n\lambda_{\max}\left(\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\text{diag}\left\\{|\bm{x}_{(j)}|\right\\}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)\cdot\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{2}^{2}$ $\displaystyle=O(n)\cdot
O_{p}\left(\frac{m-r+s+K}{n}\right),$

where the last equality is by the 5th condition of (A1) and result (78) proved
in step 1 and 2. Similarly, for $j\in\mathcal{K}$, we have
$\bm{x}^{k}_{(j)}=-\mathbf{1}_{n}\mathbf{I}\\{k=j\\}$. By the 4th condition of
(A1), we have

$\displaystyle|R_{j}|$ $\displaystyle\leq
w_{j}\lambda_{\max}\left(\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{j}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{I}_{n}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{j}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)\cdot\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{2}^{2}$ $\displaystyle=O(n)\cdot
O_{p}\left(\frac{m-r+s+K}{n}\right).$

Therefore, we have

$\|\bm{R}\|_{2}\leq\sqrt{s+m+K}\|\bm{R}\|_{\infty}=O_{p}\left(\sqrt{(s+m+K)^{3}}\right)=o_{p}(\sqrt{n}),$
(83)

where the last equality is because we assume that $s+m+K=o(n^{1/3})$.

Next we show that

$\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}\right\|_{2}=o_{p}(\sqrt{n}).$
(84)

Since $\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{2}=O_{p}(\sqrt{(m-r+s+K)/n})$, it suffices to show that

$\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}=o_{p}\left(\frac{n}{\sqrt{s+m+K}}\right).$

By the matrix Chernoff bound (73) in step 1, we have

$\displaystyle~{}\mathbb{P}\left(\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}>\tau\frac{n}{\sqrt{s+m+K}}\cdot\sqrt{\frac{{\log(s+m+K)}}{{\log
n}}}\right)$ $\displaystyle\leq$
$\displaystyle~{}\exp\left(-\frac{\tau^{2}\frac{n^{2}}{s+m+K}\frac{\log(s+m+K)}{{\log
n}}}{{\frac{c_{4}n}{(s+m+K)\log
n}}\left(c_{5}n+\frac{\tau}{3}\frac{n}{\sqrt{s+m+K}}\sqrt{\frac{{\log(s+m+K)}}{{\log
n}}}\right)}+\log(s+m+K)\right)$ $\displaystyle=$
$\displaystyle~{}\exp\left(-\frac{\tau^{2}\log(s+m+K)}{c_{4}\left(c_{5}+\frac{\tau}{3}\sqrt{\frac{{\log(s+m+K)}}{{(s+m+K)\log
n}}}\right)}+\log(s+m+K)\right)$ $\displaystyle\leq$
$\displaystyle~{}\exp\left(-\frac{\tau^{2}\log(s+m+K)}{c_{4}\left(c_{5}+\frac{\tau}{3}c_{6}\right)}+\log(s+m+K)\right),$

where the last inequality is because $\sqrt{\frac{{\log(s+m+K)}}{{(s+m+K)\log
n}}}=o(1)$ and $c_{6}>0$ is some constant. The RHS of the above inequality
could be arbitrarily small as long as $\tau$ is sufficiently large and $n$ is
sufficiently large. Therefore, we proved that

$\displaystyle\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}$
$\displaystyle=O_{p}\left(\frac{n}{\sqrt{s+m+K}}\cdot\sqrt{\frac{{\log(s+m+K)}}{{\log
n}}}\right)$ $\displaystyle=o_{p}\left(\frac{n}{\sqrt{s+m+K}}\right),$ (85)

as desired.

Combining equations (82), (83), and (84), we have

$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}$
$\displaystyle=$
$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}+o_{p}(\sqrt{n}).$
(86)

Combining equations (81) and (86), we have

$\displaystyle\begin{pmatrix}\sqrt{n}\bm{C}^{T}\bm{v}\\\
n\lambda_{n,0}\bar{\rho}({\hat{\mathcal{B}}_{0,S}})\\\
\bm{0}_{K}\end{pmatrix}=$
$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}+o_{p}(\sqrt{n}).$
(87)

Define that

$\bm{K}_{n}=\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}$

Under the 6th condition of (A1), we have
$\lim\inf_{n}\lambda_{\min}(\bm{K}_{n})>0$. This implies
$\|\bm{K}_{n}^{-1}\bm{R}\|_{2}=o_{p}(\sqrt{n})$. Combining this together with
equation (87), we have

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\bm{v}\\\
\sqrt{n}\lambda_{n,0}\bar{\rho}({\hat{\mathcal{B}}_{0,S}})\\\
\bm{0}_{K}\end{pmatrix}+o_{p}(1)$ (88)

Since $\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{2}=O_{p}\left(\sqrt{\frac{m-r+s+K}{n}}\right)$ and
$d_{n}\gg\sqrt{(s+m+K)/n}$ by the 3rd condition of (A2), we have
$\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{\infty}\leq\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{2}=o_{p}(d_{n})$. Thus, with probability tending to $1$,

$\min_{j\in{S}}|\hat{\mathcal{B}}_{0,j}|\geq\min_{j\in{S}}|{\mathcal{B}^{*}_{j}}|-\|\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}-\mathcal{B}^{*}_{\mathcal{M}\cup
S\cup\mathcal{K}}\|_{\infty}>d_{n}.$

By the monotonicity of $\rho^{\prime}$ and the 2nd condition of (A2), we
obtain

$\|\sqrt{n}\lambda_{n,0}\bar{\rho}({\hat{\mathcal{B}}_{0,S}})\|_{2}\leq\sqrt{s}\sqrt{n}\lambda_{n,0}|\rho^{\prime}(d_{n})|=o(1),$

with probability tending to 1. This together with equation (88) suggests that

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\bm{v}+o_{p}(1).$ (89)

Since $\bm{C}\mathcal{B}^{*}_{\mathcal{M}}-\bm{t}=\bm{h}_{n}$, we have
$\bm{C}(\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})=-\bm{h}_{n}$.
Therefore, multiplying both sides of equation (89) by
$\begin{pmatrix}\bm{C}&\bm{0}_{r\times(s+K)}\end{pmatrix}$, we have

$\displaystyle~{}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\bm{v}-\sqrt{n}\bm{h}_{n}$
$\displaystyle=$
$\displaystyle~{}\frac{1}{\sqrt{n}}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}+\bm{C}\bm{R}^{*}$
(90)

for some $m$-dimensional vector $\bm{R}^{*}$ such that
$\|\bm{R}^{*}\|_{2}=o_{p}(1)$. Define $\Omega_{n}=\bm{K}_{n}^{-1}$ and
$\Omega_{mm}$ is the $m\times m$ top-left block of $\Omega_{n}$. By
multiplying ${(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}$ on both sides of equation
(90), we have

$\displaystyle\bm{v}$
$\displaystyle=\frac{1}{\sqrt{n}}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle+{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\bm{R}^{*}+\sqrt{n}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}.$

This together with equation (89) yields

$\displaystyle~{}\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\sqrt{n}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}-\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\bm{R}^{*}+o_{p}(1)$
$\displaystyle=$
$\displaystyle~{}\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}(\bm{I}-\bm{P}_{n})\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\sqrt{n}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}-\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\bm{R}^{*}+o_{p}(1),$
(91)

where $\bm{P}_{n}$ is a $(m+s+K)\times(m+s+K)$ projection matrix of rank $r$.

$\bm{P}_{n}=\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\left(\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\right)^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1/2}.$

Next, we show that

$\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\bm{R}^{*}=o_{p}(1)$
(92)

Under the 4th condition in (A1), we have $\lambda_{\max}(\bm{K}_{n})=O(1)$.
This implies that $\lim\inf_{n}\lambda_{\min}(\bm{K}_{n}^{-1})>0$. Thus
$\lim\inf_{n}\lambda_{\min}(\Omega_{mm})>0$. Similarly, by the 6th condition
in (A1), we have $\lim\inf_{n}\lambda_{\min}(\bm{K}_{n})>0$, which implies
that $\lambda_{\max}(\Omega_{mm})=O(1)$. Therefore, using Cauchy-Schwartz
inequality, we have

$\displaystyle~{}\left\|\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\bm{R}^{*}\right\|_{2}^{2}$
$\displaystyle\leq$
$\displaystyle~{}\|\bm{K}_{n}^{-1/2}\|_{2}^{2}\cdot\left\|\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\right\|_{2}^{2}\cdot\|\bm{R}^{*}\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\lambda_{\max}(\bm{K}_{n}^{-1})\cdot\left\|\bm{C}^{T}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\right\|_{2}\cdot\|\bm{R}^{*}\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\lambda_{\max}(\bm{K}_{n}^{-1})\cdot\left\|\bm{C}^{T}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{C}\right\|_{2}\cdot\|\bm{R}^{*}\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\lambda_{\max}(\bm{K}_{n}^{-1})\cdot\left\|{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1/2}}\bm{C}\Omega_{mm}^{1/2}\cdot\Omega_{mm}^{-1/2}\right\|_{2}^{2}\cdot\|\bm{R}^{*}\|_{2}^{2}$
$\displaystyle\leq$
$\displaystyle~{}\lambda_{\max}(\bm{K}_{n}^{-1})\cdot\left\|{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1/2}}\bm{C}\Omega_{mm}^{1/2}\right\|_{2}^{2}\cdot\|\Omega_{mm}^{-1/2}\|_{2}^{2}\cdot\|\bm{R}^{*}\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\lambda_{\max}(\bm{K}_{n}^{-1})\cdot\|\bm{I}_{r}\|_{2}\cdot\lambda_{\max}(\Omega_{mm}^{-1})\cdot\|\bm{R}^{*}\|_{2}^{2}=o_{p}(1).$

This proves equation (92). Hence, it follows from equation (91) that

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}(\bm{I}-\bm{P}_{n})\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-\sqrt{n}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}+o_{p}(1).$
(93)

Moreover, observe the following equality

$\displaystyle\bm{h}_{n}$
$\displaystyle={\bm{C}\Omega_{mm}\bm{C}^{T}}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}$
$\displaystyle=\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}$
$\displaystyle=\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}\\\
\bm{0}_{s+K}\end{pmatrix}.$

Hence, we have

$\displaystyle~{}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}$
$\displaystyle=\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}\\\
\bm{0}_{s+K}\end{pmatrix}$
$\displaystyle=\bm{K}_{n}^{-1/2}\bm{P}_{n}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}\\\
\bm{0}_{s+K}\end{pmatrix}$ (94)

This together with equation (94) implies that

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}(\bm{I}-\bm{P}_{n})\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-\sqrt{n}\bm{K}_{n}^{-1/2}\bm{P}_{n}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}\\\
\bm{0}_{s+K}\end{pmatrix}+o_{p}(1).$ (95)

Similarly, one can prove that

$\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{8}}})\right\\}+o_{p}(1).$
(96)

∎

###  A.3 Proof of Theorem 2

###### Proof.

Define

$T_{0}=({\bm{\omega}_{n}+\sqrt{n}\bm{h}_{n}})^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}({\bm{\omega}_{n}+\sqrt{n}\bm{h}_{n}})$

and

$\bm{\omega}_{n}=\frac{1}{\sqrt{n}}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}.$

##### Step 1: Show that $T_{W}/r$ is equivalent to $T_{0}/r$.

By Theorem 2.1,

$\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}+\bm{R}_{a},$

where $\bm{R}_{a}$ is a $(m+s+K)$-dimensional vector satisfying
$\|\bm{R}_{a}\|_{2}=o_{p}(1)$. Therefore, by multiplying both sides of the
above equation by $\begin{pmatrix}\bm{C}&\bm{0}_{r\times(s+K)}\end{pmatrix}$,
we have

$\sqrt{n}\bm{C}(\hat{\mathcal{B}}_{a,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}})=\bm{\omega}_{n}+\bm{C}\bm{R}_{a,J_{0}},$
(97)

where $J_{0}=[1,\dots,m]$. Since
$\bm{C}\mathcal{B}^{*}_{\mathcal{M}}=\bm{t}+\bm{h}_{n}$, it follow from
equation (97) that

$\sqrt{n}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})=\bm{\omega}_{n}+\bm{C}\bm{R}_{a,J_{0}}+\sqrt{n}\bm{h}_{n}$

and hence

$\displaystyle\sqrt{n}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})$
$\displaystyle=({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{\omega}_{n}+\bm{C}\bm{R}_{a,J_{0}}+\sqrt{n}\bm{h}_{n})$
$\displaystyle=({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{\omega}_{n}+\sqrt{n}\bm{h}_{n})+({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{C}\bm{R}_{a,J_{0}}$
(98)

by the 6th condition in (A1), we have
$\lim\inf_{n}\lambda_{\min}(\bm{K}_{n})>0$, which implies that
$\lambda_{\max}(\Omega_{mm})=O(1)$. Then we can prove that
$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{C}\|_{2}=O(1)$,

$\displaystyle\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{C}\|_{2}^{2}$
$\displaystyle=\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{C}\Omega_{mm}^{1/2}\cdot\Omega_{mm}^{1/2}\|_{2}^{2}$
$\displaystyle\leq\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{C}\Omega_{mm}^{1/2}\|_{2}^{2}\|\Omega_{mm}^{1/2}\|_{2}^{2}$
$\displaystyle=\lambda_{\max}\left(({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}{\bm{C}\Omega_{mm}\bm{C}^{T}}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\right)\cdot\|\Omega_{mm}\|_{2}$
$\displaystyle=\lambda_{\max}(\bm{I}_{r})\|\Omega_{mm}\|_{2}=O(1).$

This implies that

$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{C}\bm{R}_{a,J_{0}}\|_{2}\leq\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{C}\|_{2}\|\bm{R}_{a,J_{0}}\|_{2}=o_{p}(1).$

This together with equation (98) gives

$\sqrt{n}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})=({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{\omega}_{n}+\sqrt{n}\bm{h}_{n})+o_{p}(1).$
(99)

Next, show that

$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{\omega}_{n}+\sqrt{n}\bm{h}_{n})\|_{2}=O_{p}(\sqrt{r}).$
(100)

To prove
$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{\omega}_{n}\|_{2}=O_{p}(\sqrt{r})$,
we notice that

$\displaystyle~{}\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{\omega}_{n}\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\frac{1}{\sqrt{n}}\left(\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}\right)^{T}\bm{K}_{n}^{-1/2}\cdot$
$\displaystyle~{}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1/2}\cdot$
$\displaystyle~{}\bm{K}_{n}^{-1/2}\frac{1}{\sqrt{n}}\left(\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}\right)$
$\displaystyle=$
$\displaystyle~{}\left\|\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\frac{1}{\sqrt{n}}\left(\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}\right)\right\|_{2}^{2}.$

Since

$\displaystyle~{}\mathbb{E}\left[\left\|\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\frac{1}{\sqrt{n}}\left(\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}\right)\right\|_{2}^{2}\right]$
$\displaystyle=$
$\displaystyle~{}\frac{1}{n}\mathbb{E}\left[\left\|\left.\sum_{k=1}^{K}w_{k}\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}\right.\right\|_{2}^{2}\right]$
$\displaystyle\leq$
$\displaystyle~{}\frac{1}{n}\sum_{k=1}^{K}w_{k}\mathbb{E}\left[\left\|\left.\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}\right.\right\|_{2}^{2}\right]$
$\displaystyle=$
$\displaystyle~{}\frac{1}{n}\sum_{k=1}^{K}w_{k}\text{tr}\left[\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1/2}\bm{P}_{n}^{1/2}\right]$
$\displaystyle=$
$\displaystyle~{}\text{tr}\left[\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\left(\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right)\bm{K}_{n}^{-1/2}\bm{P}_{n}^{1/2}\right]$
$\displaystyle=$
$\displaystyle~{}\text{tr}\left[\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\bm{K}_{n}\bm{K}_{n}^{-1/2}\bm{P}_{n}^{1/2}\right]$
$\displaystyle=$ $\displaystyle~{}\text{tr}\left[\bm{P}_{n}\right]=r,$

where the first inequality is by the convexity of $\|\cdot\|_{2}^{2}$, and
$\text{tr}\left[\bm{P}_{n}\right]=r$ is because $\bm{P}_{n}$ is the projection
matrix of rank $r$. By Markov’s inequality, we have

$\left\|\bm{P}_{n}^{1/2}\bm{K}_{n}^{-1/2}\frac{1}{\sqrt{n}}\left(\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}\right)\right\|_{2}^{2}=O_{p}(r).$

Therefore, we proved that
$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{\omega}_{n}\|_{2}=O_{p}(\sqrt{r})$.
Besides, we can show that
$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\|_{2}=O(1)$:

$\displaystyle\lambda_{\min}({\bm{C}\Omega_{mm}\bm{C}^{T}})$
$\displaystyle=\inf_{\bm{a}\in\mathbb{R}^{r},\|\bm{a}\|_{2}=1}\bm{a}^{T}{\bm{C}\Omega_{mm}\bm{C}^{T}}\bm{a}$
$\displaystyle\geq\inf_{\bm{a}\in\mathbb{R}^{r},\|\bm{a}\|_{2}=1}\lambda_{\min}({\Omega_{mm}})\cdot\bm{a}^{T}{\bm{C}\bm{C}^{T}}\bm{a}$
$\displaystyle\geq\lambda_{\min}({\Omega_{n}})\cdot\lambda_{\min}({\bm{C}\bm{C}^{T}}).$

Under the 4th condition in (A1), we have $\lambda_{\max}(\bm{K}_{n})=O(1)$.
This implies that $\liminf_{n}\lambda_{\min}(\bm{K}_{n}^{-1})>0$. And
condition (A3): $\lambda_{\max}(({\bm{C}\bm{C}^{T}})^{-1})=O(1)$ implies that
$\liminf_{n}\lambda_{\min}({\bm{C}\bm{C}^{T}})>0$. Thus,
$\lim\inf_{n}\lambda_{\min}({\bm{C}\Omega_{mm}\bm{C}^{T}})>0$, or
equivalently, $\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\|_{2}=O(1)$. This
together with condition (A3) $\|\bm{h}_{n}\|_{2}=O(\sqrt{r/n})$ implies that

$\|\sqrt{n}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}\bm{h}_{n}\|_{2}=O(\sqrt{r}).$

Thus equation (100) is proved as desired. Along with (99) implies that

$\|\sqrt{n}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})\|_{2}=O_{p}(\sqrt{r}).$
(101)

By Lemma 2, we have

$\left\|\bm{I}-\Psi_{n}^{1/2}({\bm{C}\widehat{\Omega}_{a,mm}\bm{C}^{T}})^{-1}\Psi_{n}^{1/2}\right\|_{2}=O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right),$
(102)

where $\Psi_{n}={\bm{C}\Omega_{mm}\bm{C}^{T}}$, then we will have

$\displaystyle~{}\left|n(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})-n(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})^{T}({\bm{C}\widehat{\Omega}_{a,mm}\bm{C}^{T}})^{-1}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})\right|$
$\displaystyle=$
$\displaystyle~{}\left|n(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})^{T}\Psi_{n}^{-1/2}\left(\bm{I}-\Psi_{n}^{1/2}({\bm{C}\widehat{\Omega}_{a,mm}\bm{C}^{T}})^{-1}\Psi_{n}^{1/2}\right)\Psi_{n}^{-1/2}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})\right|$
$\displaystyle\leq$
$\displaystyle~{}\left\|\bm{I}-\Psi_{n}^{1/2}({\bm{C}\widehat{\Omega}_{a,mm}\bm{C}^{T}})^{-1}\Psi_{n}^{1/2}\right\|_{2}\left\|\sqrt{n}\Psi_{n}^{-1/2}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})\right\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right)\cdot
O_{p}(\sqrt{r})=o_{p}(r),$

where the last equality is because the assumption that $s+m+K=o(n^{1/3})$.
Therefore, by the definition of $T_{W}$, we have

$\displaystyle T_{W}$
$\displaystyle=n(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}(\bm{C}\hat{\mathcal{B}}_{a,\mathcal{M}}-\bm{t})+o_{p}(r)$
$\displaystyle=T_{0}+o_{p}({r}).$

##### Step 2: Show that $T_{S}/r$ is equivalent to $T_{0}/r$.

Based on the proof of Theorem 2.1, we have:

$\displaystyle~{}\frac{1}{\sqrt{n}}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}$
$\displaystyle=$
$\displaystyle~{}\frac{1}{\sqrt{n}}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\frac{1}{\sqrt{n}}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}+o_{p}(1).$
(103)

and

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}(\bm{I}-\bm{P}_{n})\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-\sqrt{n}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}+o_{p}(1).$
(104)

Since $\lambda_{\max}(\bm{K}_{n})=O(1)$, we have

$\displaystyle\sqrt{n}\bm{K}_{n}\begin{pmatrix}\hat{\mathcal{B}}_{0,\mathcal{M}}-\mathcal{B}^{*}_{\mathcal{M}}\\\
\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}-\mathcal{B}^{*}_{S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{1/2}(\bm{I}-\bm{P}_{n})\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-\sqrt{n}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}+o_{p}(1),$
(105)

The LHS of above equation is the same as the second term in the RHS of
equation (103). Thus, this together with (103) implies that

$\displaystyle~{}\frac{1}{\sqrt{n}}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}$
$\displaystyle=$
$\displaystyle~{}\frac{1}{\sqrt{n}}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle-$
$\displaystyle~{}\frac{1}{\sqrt{n}}\bm{K}_{n}^{1/2}(\bm{I}-\bm{P}_{n})\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle+$ $\displaystyle~{}\sqrt{n}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}+o_{p}(1)$
$\displaystyle=$
$\displaystyle~{}\frac{1}{\sqrt{n}}\bm{K}_{n}^{1/2}\bm{P}_{n}\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle+$ $\displaystyle~{}\sqrt{n}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\bm{h}_{n}+o_{p}(1)$
$\displaystyle=$ $\displaystyle~{}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})+o_{p}(1),$
(106)

where the last equality is by the definition of $\bm{P}_{n}$ and
$\bm{\omega}_{n}$. Multiplying both sides by $\bm{K}_{n}^{-1/2}$ and from the
fact that $\lim\inf_{n}\lambda_{\min}(\bm{K}_{n})>0$, we have

$\displaystyle~{}\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}$
$\displaystyle=$
$\displaystyle~{}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})+o_{p}(1),$
(107)

We proved in Step 1 that
$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\|_{2}=O_{p}(\sqrt{r})$,
thus

$\left\|\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}=\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\|_{2}^{2}=O_{p}({r}).$

This implies that

$\left\|\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}\right\|_{2}=O_{p}({\sqrt{r}}).$

By Lemma 2, we have

$\left\|\bm{I}-\Omega_{n}^{-1/2}\widehat{\Omega}_{0}\Omega_{n}^{-1/2}\right\|_{2}=O_{p}\left(\frac{s+m+K}{\sqrt{n}}\right).$

Then under event $\\{\widehat{S}_{0}=S\\}$, we have

$\displaystyle~{}\left|\frac{1}{n}\mathcal{S}_{n}(\hat{\mathcal{B}}_{0})^{T}(\Omega_{n}-\widehat{\Omega}_{0})\mathcal{S}_{n}(\hat{\mathcal{B}}_{0})\right|$
$\displaystyle\leq$
$\displaystyle~{}\left\|\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{\widehat{S}_{0}\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}\right\|_{2}^{2}\left\|\bm{I}-\Omega_{n}^{-1/2}\widehat{\Omega}_{0}\Omega_{n}^{-1/2}\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}\left\|\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{0}})\right\\}\right\|_{2}^{2}\left\|\bm{I}-\Omega_{n}^{-1/2}\widehat{\Omega}_{0}\Omega_{n}^{-1/2}\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}O_{p}\left(\frac{r(s+m+K)}{\sqrt{n}}\right)=o_{p}(r).$

Since $\mathbb{P}(\widehat{S}_{0}=S)\rightarrow 1$ and $s+m+K=o(\sqrt{n})$, we
have

$T_{S}=T_{0}+o_{p}(r).$

##### Step 3: Show that $T_{L}/r$ is equivalent to $T_{0}/r$.

By Theorem 2.1, we have

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}{\bm{P}_{n}}\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle+\sqrt{n}\bm{K}_{n}^{-1/2}\bm{P}_{n}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}\\\
\bm{0}_{s+K}\end{pmatrix}+o_{p}(1).$

Notice that

$\displaystyle~{}\bm{K}_{n}^{-1/2}\bm{P}_{n}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}\\\
\bm{0}_{s+K}\end{pmatrix}$ $\displaystyle=$
$\displaystyle~{}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}$
$\displaystyle=$ $\displaystyle~{}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}.$

It follows that

$\displaystyle\sqrt{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=\frac{1}{\sqrt{n}}\bm{K}_{n}^{-1/2}{\bm{P}_{n}}\bm{K}_{n}^{-1/2}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle+\sqrt{n}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}\bm{h}_{n}+o_{p}(1)$
$\displaystyle=\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1}(\bm{\omega}_{n}+\sqrt{n}\bm{h}_{n})+o_{p}(1).$
(108)

We proved in Step 1 that
$\|({\bm{C}\Omega_{mm}\bm{C}^{T}})^{-1/2}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\|_{2}=O_{p}(\sqrt{r})$,
thus

$\displaystyle~{}\left\|\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|\bm{K}_{n}^{-1/2}\right\|_{2}^{2}\left\|\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\lambda_{\max}(\bm{K}_{n}^{-1})\left\|{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1/2}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}$
$\displaystyle=$ $\displaystyle~{}O_{p}({r}).$ (109)

Thus, we proved that

$\left\|\hat{\mathcal{B}}_{a,\mathcal{M}\cup
S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,\mathcal{M}\cup
S\cup\mathcal{K}}\right\|_{2}=O_{p}({\sqrt{r/n}})$ (110)

Under the event $\left\\{\hat{\mathcal{B}}_{a,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\hat{\mathcal{B}}_{0,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0}\right\\}$, denote
$\hat{\Delta}_{a,0}=\hat{\mathcal{B}}_{a,\mathcal{M}\cup
S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,\mathcal{M}\cup S\cup\mathcal{K}}$. By
the second-order Taylor expansion, we obtain that

$\displaystyle M_{n}(\hat{\mathcal{B}}_{a})-M_{n}(\hat{\mathcal{B}}_{0})$
$\displaystyle=\frac{1}{n}\hat{\Delta}_{a,0}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\right\\}$
$\displaystyle+\frac{1}{2n}\hat{\Delta}_{a,0}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\hat{\Sigma}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{a,0}+\frac{1}{n}\hat{\Delta}_{a,0}^{T}\bm{R},$
(111)

where $\bm{R}$ satisfies $\|\bm{R}\|_{2}=O_{p}(r\sqrt{s+m+K})$. Therefore, we
have

$\displaystyle\left\|\frac{1}{n}\hat{\Delta}_{a,0}^{T}\bm{R}\right\|_{2}$
$\displaystyle\leq\frac{1}{n}\left\|\hat{\Delta}_{a,0}\right\|_{2}\left\|\bm{R}\right\|_{2}=O_{p}\left(\frac{\sqrt{r}}{n}\cdot\frac{r\sqrt{s+m+K}}{\sqrt{n}}\right)=o_{p}\left(\frac{{\sqrt{r}}}{n}\right),$
(112)

where the last equality is because $(s+m+K)=o(n^{1/3})$. Notice that

$\hat{\Sigma}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})=(\hat{\Sigma}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})-\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}}))+(\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})-{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}}))+{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}}).$

By Cauchy Schwartz inequality,

$\displaystyle~{}\left|\frac{1}{n}\hat{\Delta}_{a,0}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}(\hat{\Sigma}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})-\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{a,0}\right|$
$\displaystyle\leq$
$\displaystyle~{}\left\|\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}(\hat{\Sigma}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})-\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}\left\|\hat{\Delta}_{a,0}\right\|_{2}^{2}.$

By the the 4th condition of (A1) and
$\sup_{t}|\hat{\Sigma}^{\prime}(t)|=O_{p}(1)$, we have

$\displaystyle~{}\left\|\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}(\hat{\Sigma}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})-\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}$
$\displaystyle=$
$\displaystyle~{}\left\|\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\hat{\Sigma}^{\prime}(\bm{X}^{k}{\bar{\mathcal{B}}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{a,0}\right\|_{2}$
$\displaystyle\leq$
$\displaystyle~{}\left\|\frac{1}{n}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\sup_{t}|\hat{\Sigma}^{\prime}(t)|\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}\left\|\hat{\Delta}_{a,0}\right\|_{2}$
$\displaystyle=$ $\displaystyle~{}O_{p}(1)\cdot
O_{p}\left(\sqrt{{r}/{n}}\right).$

Therefore, we have

$\displaystyle~{}\left|\frac{1}{n}\hat{\Delta}_{a,0}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}(\hat{\Sigma}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})-\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{a,0}\right|$
$\displaystyle\leq$ $\displaystyle~{}O_{p}(\sqrt{r/n})\cdot
O_{p}(r/n)=o_{p}(\sqrt{r}/n).$ (113)

Next, show that

$\displaystyle\left|\frac{1}{n}\hat{\Delta}_{a,0}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}(\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})-{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}}))\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{a,0}\right|=o_{p}\left(\frac{{\sqrt{r}}}{n}\right).$
(114)

When proving the asymptotic distribution in Theorem 2.1 (85), we proved that

$\displaystyle\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}$
$\displaystyle=O_{p}\left(\frac{n}{\sqrt{s+m+K}}\cdot\sqrt{\frac{{\log(s+m+K)}}{{\log
n}}}\right).$

This implies that

$\displaystyle~{}\left|\frac{1}{n}\hat{\Delta}_{a,0}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}(\hat{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}})-{\Sigma}(\bm{X}^{k}{{\mathcal{B}^{*}}}))\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\hat{\Delta}_{a,0}\right|$
$\displaystyle\leq$
$\displaystyle~{}\frac{1}{n}\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\left({\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})-\hat{\Sigma}(\bm{X}^{k}{\mathcal{B}^{*}})\right)\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}^{T}\right\|_{2}\cdot\left\|\hat{\Delta}_{a,0}\right\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}O_{p}\left(\frac{1}{\sqrt{s+m+K}}\cdot\sqrt{\frac{{\log(s+m+K)}}{{\log
n}}}\right)\cdot O_{p}\left({\frac{r}{n}}\right)$ $\displaystyle=$
$\displaystyle~{}O_{p}\left(\frac{\sqrt{r}}{n}\cdot\sqrt{\frac{r}{s+m+K}}\sqrt{\frac{{\log(s+m+K)}}{{\log
n}}}\right)=o_{p}\left(\frac{\sqrt{r}}{n}\right).$

where the last equality is because $r\leq s+m+K$ and $s+m+K=o(\sqrt{n})$.
Combining (111), (112), (113), and (114), we have under the event
$\left\\{\hat{\mathcal{B}}_{a,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\hat{\mathcal{B}}_{0,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0}\right\\}$,

$\displaystyle
n\left[M_{n}(\hat{\mathcal{B}}_{a})-M_{n}(\hat{\mathcal{B}}_{0})\right]$
$\displaystyle=\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\right\\}$
$\displaystyle+\frac{n}{2}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}^{T}\bm{K}_{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle+o_{p}\left(\sqrt{r}\right).$ (115)

Next show that

$\left\|\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\right\\}\right\|_{2}=o_{p}(\sqrt{n})$
(116)

By Theorem 2.1, we have with probability tending to 1 that $\min_{j\in
S}|\hat{\mathcal{B}}_{a,j}|\geq d_{n}$. By the first-order optimality
condition of $\hat{\mathcal{B}}_{a}$, we have

$\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\right\\}=\begin{pmatrix}\bm{0}_{m}\\\
n\lambda_{n,a}\bar{\rho}(\hat{\mathcal{B}}_{a,S)},\lambda_{n,a})\\\
\bm{0}_{K}\end{pmatrix}.$

Under the 2nd condition of (A2) that
$p^{\prime}_{\lambda_{n,a}}(d_{n})=o(1/\sqrt{(s+m)n})$, we have

$\|n\lambda_{n,a}\bar{\rho}(\hat{\mathcal{B}}_{a,S\cup\mathcal{K})},\lambda_{n,a})\|_{2}\leq
n\sqrt{s}|p^{\prime}_{\lambda_{n,a}}(d_{n})|=o(\sqrt{n}).$

This proves equation (116) as desired. Along with (110), we have

$\left|\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}^{T}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{\hat{\mathcal{B}}_{a}})\right\\}\right|=o_{p}(\sqrt{r}).$
(117)

As a result, we have under the event
$\left\\{\hat{\mathcal{B}}_{a,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\hat{\mathcal{B}}_{0,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0}\right\\}$,

$\displaystyle
2n\left[M_{n}(\hat{\mathcal{B}}_{a})-M_{n}(\hat{\mathcal{B}}_{0})\right]$
$\displaystyle={n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}^{T}\bm{K}_{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}+o_{p}\left(\sqrt{r}\right).$
(118)

From the fact that
$\left\|{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1/2}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}=O_{p}(\sqrt{r})$
and $\lambda_{\max}(\bm{K}_{n})=O(1)$, we have

$\displaystyle~{}\left\|\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\left\|\bm{K}_{n}^{1/2}\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}$
$\displaystyle\leq$
$\displaystyle~{}\|\bm{K}_{n}^{1/2}\|_{2}^{2}\left\|\bm{K}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}$
$\displaystyle=$
$\displaystyle~{}\|\bm{K}_{n}\|_{2}\left\|{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1/2}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right\|_{2}^{2}=O_{p}(r)$

In view of equation (108), we have

$\displaystyle~{}\left|{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}^{T}\bm{K}_{n}\begin{pmatrix}\hat{\mathcal{B}}_{a,\mathcal{M}}-\hat{\mathcal{B}}_{0,\mathcal{M}}\\\
\hat{\mathcal{B}}_{a,S\cup\mathcal{K}}-\hat{\mathcal{B}}_{0,S\cup\mathcal{K}}\end{pmatrix}-(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})^{T}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}(\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})\right|$
$\displaystyle=$
$\displaystyle~{}o_{p}\left((\bm{\omega_{n}}+\sqrt{n}\bm{h}_{n})^{T}{(\bm{C}\Omega_{mm}\bm{C}^{T})^{-1}}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\right)+o_{p}(\bm{K}_{n})$
$\displaystyle=$ $\displaystyle~{}o_{p}(\sqrt{r}).$

Along with equation (118) and the fact that
$\mathbb{P}(\hat{\mathcal{B}}_{a,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\hat{\mathcal{B}}_{0,(\mathcal{M}\cup
S\cup\mathcal{K})^{c}}=\bm{0})\rightarrow 1$ proved in Theorem 1, we can prove
that

$T_{L}=T_{0}+o_{p}(\sqrt{r})=T_{0}+o_{p}(r).$

##### Step 4: Find the asymptotic distribution of $T_{0}$.

Define $\mathcal{T}_{n}=\text{Cov}(\bm{\omega}_{n})$, we then prove that

$\sup_{\mathcal{A}}\left|\mathbb{P}\left(\mathcal{T}_{n}^{-1/2}\bm{\omega}_{n}\in\mathcal{A}\right)-\mathbb{P}\left(\bm{Z}\in\mathcal{A}\right)\right|\rightarrow
0,$ (119)

where $\bm{Z}\in\mathbb{R}^{r}$ stands for a mean zero Gaussian random vector
with identity covariance matrix, and the supremum is taken over all convex
sets $\mathcal{A}\subseteq\mathbb{R}^{r}$. By the definition of
$\bm{\omega}_{n}$, we have

$\displaystyle\mathcal{T}_{n}^{-1/2}\bm{\omega}_{n}$
$\displaystyle=\frac{1}{\sqrt{n}}\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}\begin{pmatrix}\bm{X}_{\mathcal{M}}^{T}\\\
(\bm{X}^{k}_{S\cup\mathcal{K}})^{T}\end{pmatrix}\bm{H}(\bm{X}^{k}{{\mathcal{B}^{*}}})\left\\{\bm{Y}^{k}-\bm{\mu}(\bm{X}^{k}{{\mathcal{B}^{*}}})\right\\}$
$\displaystyle=\sum_{i=1}^{n}\frac{1}{\sqrt{n}}\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\sum_{k=1}^{K}w_{k}h^{\prime}((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\left\\{\tilde{y}_{ik}-\mu((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\right\\}\begin{pmatrix}\bm{x}_{i,\mathcal{M}}\\\
\bm{x}^{k}_{i,S\cup\mathcal{K}}\end{pmatrix}$
$\displaystyle=:\sum_{i=1}^{n}\xi_{i},$

where $\xi_{i}$’s are independent. Since
$\mathcal{T}_{n}=\text{Cov}(\bm{\omega}_{n})$, we have
$\sum_{i=1}^{n}\text{Cov}(\xi_{i})=I_{r}$. By Theorem 1 in Bentkus 2005, we
have

$\sup_{\mathcal{A}}\left|\mathbb{P}\left(\mathcal{T}_{n}^{-1/2}\bm{\omega}_{n}\in\mathcal{A}\right)-\mathbb{P}\left(\bm{Z}\in\mathcal{A}\right)\right|\leq
c_{0}r^{1/4}\sum_{i=1}^{n}\mathbb{E}\|\xi_{i}\|_{2}^{3},$

for some constant $c_{0}$, where the supremum is taken for all convex subsets
in $\mathbb{R}^{r}$. To bound $\sum_{i=1}^{n}\mathbb{E}\|\xi_{i}\|_{2}^{3}$,
notice that by convexity of $\|\cdot\|_{2}^{3}$,

$\displaystyle\sum_{i=1}^{n}\mathbb{E}\|\xi_{i}\|_{2}^{3}$
$\displaystyle=\frac{1}{n^{3/2}}\sum_{i=1}^{n}\mathbb{E}\left\|\sum_{k=1}^{K}w_{k}\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}h^{\prime}((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\left\\{\tilde{y}_{ik}-\mu((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\right\\}\begin{pmatrix}\bm{x}_{i,\mathcal{M}}\\\
\bm{x}^{k}_{i,S\cup\mathcal{K}}\end{pmatrix}\right\|_{2}^{3}$
$\displaystyle\leq\frac{1}{n^{3/2}}\sum_{i=1}^{n}\sum_{k=1}^{K}w_{k}\mathbb{E}\left\|\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}h^{\prime}((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\left\\{\tilde{y}_{ik}-\mu((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\right\\}\begin{pmatrix}\bm{x}_{i,\mathcal{M}}\\\
\bm{x}^{k}_{i,S\cup\mathcal{K}}\end{pmatrix}\right\|_{2}^{3}$
$\displaystyle=\frac{1}{n^{3/2}}\sum_{i=1}^{n}\sum_{k=1}^{K}w_{k}\left\|\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{x}_{i,\mathcal{M}}\\\
\bm{x}^{k}_{i,S\cup\mathcal{K}}\end{pmatrix}\right\|_{2}^{3}\left|h^{\prime}((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\right|^{3}\mathbb{E}\left|\tilde{y}_{ik}-\mu((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\right|^{3}.$

Since
$\left|\tilde{y}_{ik}-\mu((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\right|\leq
1$, we have
$\mathbb{E}\left|\tilde{y}_{ik}-\mu((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})\right|^{3}=O(1)$.
By the 3rd condition of (A1) and $|h^{\prime\prime}|\leq 1$, we have
$|h^{\prime}((\bm{x}_{i}^{k})^{T}{\mathcal{B}^{*}})|=O(1)$. Therefore,

$\displaystyle\sum_{i=1}^{n}\mathbb{E}\|\xi_{i}\|_{2}^{3}$
$\displaystyle\leq\frac{1}{n^{3/2}}\sum_{i=1}^{n}\sum_{k=1}^{K}w_{k}\left\|\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{x}_{i,\mathcal{M}}\\\
\bm{x}^{k}_{i,S\cup\mathcal{K}}\end{pmatrix}\right\|_{2}^{3}O(1)$
$\displaystyle\leq\frac{1}{n^{3/2}}\sum_{i=1}^{n}\sum_{k=1}^{K}w_{k}\left\|\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\bm{V}_{n}^{1/2}\right\|_{2}^{3}\left\|\bm{V}_{n}^{-1/2}\begin{pmatrix}\bm{x}_{i,\mathcal{M}}\\\
\bm{x}^{k}_{i,S\cup\mathcal{K}}\end{pmatrix}\right\|_{2}^{3}O(1).$

Notice that

$\displaystyle\left\|\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\bm{V}_{n}^{1/2}\right\|_{2}$
$\displaystyle=\lambda_{\max}\left(\mathcal{T}_{n}^{-1/2}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}^{T}\bm{K}_{n}^{-1}\bm{V}_{n}\bm{K}_{n}^{-1}\begin{pmatrix}\bm{C}^{T}\\\
\bm{0}_{r\times(s+K)}^{T}\end{pmatrix}\mathcal{T}_{n}^{-1/2}\right)^{1/2}=1,$

this along with the additional condition (18) in Theorem 2:

$\frac{r^{1/4}}{n^{3/2}}\sum_{i=1}^{n}\sum_{k=1}^{K}w_{k}\left\\{(\bm{x}_{i,\mathcal{M}\cup{S}\cup\mathcal{K}}^{k})^{T}\bm{V}_{n}^{-1}(\bm{x}_{i,\mathcal{M}\cup{S}\cup\mathcal{K}}^{k})\right\\}^{3/2}\rightarrow
0,$

implies that

$\sup_{\mathcal{A}}\left|\mathbb{P}\left(\mathcal{T}_{n}^{-1/2}\bm{\omega}_{n}\in\mathcal{A}\right)-\mathbb{P}\left(\bm{Z}\in\mathcal{A}\right)\right|\rightarrow
0.$

Since for any $x\in\mathbb{R}^{+}$, the event $\\{T_{0}\leq x\\}$ is
equivalent to the event
$\\{\mathcal{T}_{n}^{-1/2}\bm{\omega}_{n}\in\mathcal{A}_{x}\\}$, where

$\mathcal{A}_{x}=\left\\{\bm{z}\in\mathbb{R}^{r}:\left\|\Psi_{n}^{-1/2}\left({\mathcal{T}_{n}}^{1/2}\cdot\bm{z}+\sqrt{n}\bm{h}_{n}\right)\right\|_{2}^{2}\leq
x\right\\}.$

It follows that

$\sup_{x}\left|\mathbb{P}\left(T_{0}\leq
x\right)-\mathbb{P}\left(\bm{Z}\in\mathcal{A}_{x}\right)\right|\rightarrow 0,$
(120)

where
$\left\|\Psi_{n}^{-1/2}\left({\mathcal{T}_{n}}^{1/2}\cdot\bm{Z}+\sqrt{n}\bm{h}_{n}\right)\right\|_{2}^{2}$
follows a generalized Chi-squared distribution.

Consider any statistic $T=T_{0}+o_{p}(r)$. For any $x$ and any sufficiently
small $\varepsilon>0$, we have $\mathbb{P}(|T-T_{0}|>\varepsilon r)=o(1)$. It
follows from (120) that

$\displaystyle\mathbb{P}(T\leq x)$ $\displaystyle\leq\mathbb{P}(T_{0}\leq
x+\varepsilon r)+\mathbb{P}(|T-T_{0}|>\varepsilon
r)\leq\mathbb{P}\left(\bm{Z}\in\mathcal{A}_{x+\varepsilon r}\right)+o(1)$
(121) $\displaystyle\mathbb{P}(T\leq x)$
$\displaystyle\geq\mathbb{P}(T_{0}\leq x-\varepsilon
r)-\mathbb{P}(|T-T_{0}|>\varepsilon
r)\geq\mathbb{P}\left(\bm{Z}\in\mathcal{A}_{x-\varepsilon r}\right)-o(1).$

It remains to prove that

$\lim_{\varepsilon\rightarrow
0^{+}}{\lim\sup}_{n}\left|\mathbb{P}\left(\bm{Z}\in\mathcal{A}_{x+\varepsilon
r}\right)-\mathbb{P}\left(\bm{Z}\in\mathcal{A}_{x-\varepsilon
r}\right)\right|=0$ (122)

or equivalently,

$\lim_{\varepsilon\rightarrow
0^{+}}{\lim\sup}_{n}\mathbb{P}\left(x-\varepsilon
r\leq\left(\bm{Z}+\sqrt{n}{\mathcal{T}_{n}}^{-1/2}\bm{h}_{n}\right)^{T}{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2}\left(\bm{Z}+\sqrt{n}{\mathcal{T}_{n}}^{-1/2}\bm{h}_{n}\right)\leq
x+\varepsilon r\right)=0$

This result can be proved using the fact that
$\|\sqrt{n}{\mathcal{T}}_{n}^{-1/2}\bm{h}_{n}\|_{2}=O(\sqrt{r})$ ,
$\lambda_{\max}({\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})=O(1)$
as well as
$\lim\sup_{n}\lambda_{\min}({\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})=O(1)$
given in results (47), (48) and (49) of Lemma 2. Combining (121) with (122),
we obtain that

$\sup_{x}\left|\mathbb{P}\left(T\leq
x\right)-\mathbb{P}\left(\bm{Z}\in\mathcal{A}_{x}\right)\right|\rightarrow 0.$
(123)

In the first three steps, we have shown
$T_{0}=T_{W}+o_{p}(1)=T_{S}+o_{p}(1)=T_{L}+o_{p}(1)$. This together with (123)
implies that the generalized $\chi^{2}$ approximation holds for our partial
penalized statistics. Hence, the proof is completed.

∎

###  A.4 Proof of Theorem 3

From the results (122) proved in Theorem 2, it suffices to prove that

$\mathbb{P}\left(\bm{Z}^{T}{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2}\bm{Z}\leq\hat{\chi}_{n,(1-\alpha)}|\bm{X},\bm{Y}\right)=1-\alpha+o_{p}(1),$
(124)

For any fixed $\varepsilon>0$, we have the following inequality

$\displaystyle~{}\mathbb{P}\left(\bm{Z}^{T}{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\bm{Z}\leq\hat{\chi}_{n,(1-\alpha)}-\varepsilon|\bm{X},\bm{Y}\right)-\mathbb{P}\left(\left|\bm{Z}^{T}({\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})\bm{Z}\right|>\varepsilon|\bm{X},\bm{Y}\right)$
$\displaystyle\leq$
$\displaystyle~{}\mathbb{P}\left(\bm{Z}^{T}{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2}\bm{Z}\leq\hat{\chi}_{n,(1-\alpha)}|\bm{X},\bm{Y}\right)$
$\displaystyle\leq$
$\displaystyle~{}\mathbb{P}\left(\bm{Z}^{T}{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\bm{Z}\leq\hat{\chi}_{n,(1-\alpha)}+\varepsilon|\bm{X},\bm{Y}\right)+\mathbb{P}\left(\left|\bm{Z}^{T}({\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})\bm{Z}\right|>\varepsilon|\bm{X},\bm{Y}\right).$

On the event

$\mathcal{E}_{n}:=\left\\{\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{{\mathcal{T}}_{n}}^{1/2}{\Psi}_{n}^{-1}{{\mathcal{T}}_{n}}^{1/2}\right\|_{2}\leq\frac{\varepsilon}{\frac{\varepsilon}{2}\frac{\sqrt{n}}{s+m+K}+r}\right\\},$
(125)

by the sub-exponential tail bound for $\chi^{2}(0,r)$, we have

$\displaystyle~{}\mathbb{P}\left(\left|\bm{Z}^{T}({\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})\bm{Z}\right|>\varepsilon|\bm{X},\bm{Y}\right)$
$\displaystyle\leq$
$\displaystyle~{}\mathbb{P}\left(\left\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{{\mathcal{T}}_{n}}^{1/2}{\Psi}_{n}^{-1}{{\mathcal{T}}_{n}}^{1/2}\right\|_{2}\bm{Z}^{T}\bm{Z}>\varepsilon|\bm{X},\bm{Y}\right)$
$\displaystyle\leq$
$\displaystyle~{}\exp\left(-\frac{1}{8}\left(\frac{\varepsilon}{\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{{\mathcal{T}}_{n}}^{1/2}{\Psi}_{n}^{-1}{{\mathcal{T}}_{n}}^{1/2}\|_{2}}-r\right)\right)$
$\displaystyle\leq$
$\displaystyle~{}\exp\left(-\frac{\varepsilon}{16}\frac{\sqrt{n}}{s+m+K}\right),$

which could be arbitrarily small as long as $n$ is large enough. Furthermore,
combining the fact that
$\|{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{{\mathcal{T}}_{n}}^{1/2}{\Psi}_{n}^{-1}{{\mathcal{T}}_{n}}^{1/2}\|_{2}=O_{p}\left({s+m+K}/{\sqrt{n}}\right)$
proved in Lemma 2 as well as the additional condition $(s+m+K)r=o(\sqrt{n})$,
we know that with probability tending to $1$, the event $\mathcal{E}_{n}$
(125) holds. Therefore, the above ineuqality holds with probability tending to
$1$. Then we have

$\displaystyle~{}\mathbb{P}\left(\left|\bm{Z}^{T}({\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})\bm{Z}\right|>\varepsilon\right)$
$\displaystyle\leq$
$\displaystyle~{}\mathbb{E}\left[\bm{1}_{\mathcal{E}_{n}}\cdot\mathbb{P}\left(\left|\bm{Z}^{T}({\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})\bm{Z}\right|>\varepsilon|\bm{X},\bm{Y}\right)\right]+\mathbb{P}(\mathcal{E}_{n}^{c})$
$\displaystyle\leq$
$\displaystyle~{}\exp\left(-\frac{\varepsilon}{16}\frac{\sqrt{n}}{s+m+K}\right)+\mathbb{P}(\mathcal{E}_{n}^{c}),$

which implies that

${\lim\sup}_{n}\mathbb{P}\left(\left|\bm{Z}^{T}({\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}-{\mathcal{T}_{n}}^{1/2}\Psi_{n}^{-1}{\mathcal{T}_{n}}^{1/2})\bm{Z}\right|>\varepsilon\right)=0.$
(126)

Similar to the proof of (122) with $\bm{h}_{n}=\bm{0}$ under the null
hypothesis, we can prove that

$\lim_{\varepsilon\rightarrow
0^{+}}{\lim\sup}_{n}\mathbb{P}\left(x-\varepsilon
r\leq\bm{Z}^{T}{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\bm{Z}\leq
x+\varepsilon r|\bm{X},\bm{Y}\right)\overset{a.s.}{=}0.$

Combining the previous results with the fact that

$\mathbb{P}\left(\left.\bm{Z}^{T}{\widehat{\mathcal{T}}_{n,a}}^{1/2}{\widehat{\Psi}_{n,a}}^{-1}{\widehat{\mathcal{T}}_{n,a}}^{1/2}\bm{Z}\leq\hat{\chi}_{n,(1-\alpha)}\right|\bm{X},\bm{Y}\right)=1-\alpha,$

given by the definition of $\hat{\chi}_{n,(1-\alpha)}$, we proved the desired
result (124).

###  A.5 Analysis of the Supermarket Data

In our empirical study, we use the supermarket data in Lan et al. 2016. This
data set contains a total of $n=464$ daily records. For each record, the
response is the number of customers and the predictors are the sales volumes
of $p=6398$ products. Both the response and predictors are standardized so
that they have zero mean and unit variance. This study identifies the products
that attract the most customers and further performs hypothesis testings on
those products to evaluate the impact of those products on the customers.

We apply the testing procedures of Shi et al. 2019 under the standard normal
linear model assumptions, and our testing procedures under non-parametric Box-
Cox model to this dataset. To formulate the testing hypotheses, we randomly
split the data set into two subsets: one for preliminary analysis ($232$
observations) and one for hypothesis testings ($232$ observations).

###  A.6 Computing the partial penalized CPR estimators $\hat{\bm{\beta}}_{a}$

From (7), $(\hat{\bm{\beta}}_{a},\hat{\bm{b}}_{a})$ is the minimizer of the
following objective function

${\sum_{k=1}^{K}w_{k}\left\\{\frac{1}{n}\sum_{i=1}^{n}L(\check{y}_{ki}(\bm{x}_{i}^{T}\bm{\beta}-b_{k}))\right\\}}+\sum_{j\notin\mathcal{M}}p_{\lambda}(|\beta_{j}|).$
(127)

It can be solved by combining the local linear approximation (LLA) algorithm
and the coordinate-majorization-descent (CMD) algorithm. Details of the LLA-
CMD algorithm are discussed below.

##### Outer loop: local linear approximation

The local linear approximation (LLA) algorithm Zou and Li, 2008 takes
advantage of the special folded concave structure and utilizes the
majorization-minimization (MM) principle to turn a concave regularization
problem into a sequence of weighted $\ell_{1}$ penalized problems. Let
$(\hat{\bm{\beta}}^{\textit{curr}},\hat{\bm{b}}^{\textit{curr}})$ be the
current estimate. The folded concave penalty could be majorized by a local
linear approximation function:

$\sum_{j\notin\mathcal{M}}p_{\lambda}(|\beta_{j}|)\leq\sum_{j\notin\mathcal{M}}p_{\lambda}(|\hat{\beta}_{j}^{\textit{curr}}|)+p_{\lambda}^{\prime}(|{\hat{\beta}_{j}^{\textit{curr}}}|)(|\beta_{j}|-|{\hat{\beta}_{j}^{\textit{curr}}}|),$
(128)

which is the best convex majorization of the concave penalty function (Theorem
2 of Zou and Li 2008). Then the objective function of (127) could be majorized
by a weighted $\ell_{1}$ penalized problem:

$M_{n}(\bm{\beta},\bm{b})+\sum_{j\notin\mathcal{M}}p_{\lambda}(|\hat{\beta}_{j}^{\textit{curr}}|)+p_{\lambda}^{\prime}(|{\hat{\beta}_{j}^{\textit{curr}}}|)(|\beta_{j}|-|{\hat{\beta}_{j}^{\textit{curr}}}|).$
(129)

The details of the LLA algorithm are summarized in Algorithm 3.

##### Inner loop: coordinate-majorization-descent

For our weighted $\ell_{1}$ penalized composite probit regression problem
(136) within each LLA iteration in Algorithm 3, we may also apply the
coordinate descent algorithm Friedman et al., 2010 which has been successfully
used in solving some high-dimensional models. In the case of probit
regression, we need to pay attention to computer overflow errors that may
occur during the computation of the CDF $\Phi(\cdot)$. Standard algorithms
like Newton-Raphson are very sensitive to large values of the linear predictor
Demidenko, 2001.

We prefer to use a numerically stable and efficient algorithm to solve (136).
Due to the good property of probit regression loss given in Lemma 2 (refer to
the first paper), that is, the second derivative of the probit regression loss
function can be bounded by 1, we can fix the computer overflow error issue by
using the coordinate-majorization-descent algorithm Yang and Zou, 2013 which
only uses the gradient information of the composite probit loss function.

Let $(\hat{\bm{\beta}}^{\textit{curr}},\hat{\bm{b}}^{\textit{curr}})$ be the
current estimate. Define the current margin
$\hat{r}_{ki}^{\textit{curr}}=\check{y}_{ki}(\bm{x}_{i}^{T}{\hat{\bm{\beta}}^{\textit{curr}}}-\hat{{b}}_{k}^{\textit{curr}})$
for $k=1,\cdots,K$, $i=1,\cdots,n$ and current $\ell_{1}$ weights
$\hat{\omega}_{j}^{\textit{curr}}=p^{\prime}_{\lambda}(|\hat{\beta}^{\textit{curr}}_{j}|)$
for $j\in\\{1,\cdots,p\\}\backslash\mathcal{M}$.

To update the $\zeta$-th coordinate of $\bm{\beta}$, where
$\zeta\in\\{1,\cdots,p\\}\backslash\mathcal{M}$, define the $F$ function:

$F(\beta_{\zeta}|\hat{\bm{\beta}}^{\textit{curr}},\hat{\bm{b}}^{\textit{curr}}):=\sum_{k=1}^{K}w_{k}\left\\{\frac{1}{n}\sum_{i=1}^{n}L(\check{y}_{ki}x_{i\zeta}(\beta_{\zeta}-{\hat{\beta}^{\textit{curr}}}_{\zeta})+\hat{r}_{ki}^{\textit{curr}})\right\\}+\hat{\omega}^{\textit{curr}}_{\zeta}|\beta_{\zeta}|.$
(130)

By Lemma LABEL:lemma:2deriv_bounded and $\check{y}_{ki}^{2}=1$, this $F$
function can be majorized by a penalized quadratic function defined as

$\mathcal{F}(\beta_{\zeta}|\hat{\bm{\beta}}^{\textit{curr}},\hat{\bm{b}}^{\textit{curr}}):=\sum_{k=1}^{K}w_{k}\cdot\frac{1}{n}\sum_{i=1}^{n}\left[L(\hat{r}_{ki}^{\textit{curr}})+L^{\prime}(\hat{r}_{ki}^{\textit{curr}})\check{y}_{ki}x_{i\zeta}(\beta_{\zeta}-{\hat{\beta}^{\textit{curr}}}_{\zeta})+\frac{1}{2}x_{is}^{2}(\beta_{\zeta}-{\hat{\beta}^{\textit{curr}}}_{\zeta})^{2}\right]+\hat{\omega}^{\textit{curr}}_{\zeta}|\beta_{\zeta}|.$

We can easily solve the minimizer of the majorization function by a simple
soft thresholding rule:

${\hat{\beta}^{\textit{new}}}_{\zeta}=\mathcal{S}\left({\hat{\beta}^{\textit{curr}}}_{\zeta}+\frac{-\sum_{k=1}^{K}w_{k}\sum_{i=1}^{n}L^{\prime}(\hat{r}_{ki}^{\textit{curr}})\check{y}_{ki}x_{i\zeta}}{\sum_{i=1}^{n}x_{i\zeta}^{2}},\frac{\hat{\omega}^{\textit{curr}}_{\zeta}}{\frac{1}{n}\sum_{i=1}^{n}x_{i\zeta}^{2}}\right),$
(131)

where $\mathcal{S}(z,t)=(|z|-t)_{+}\cdot\text{sgn}(z)$. The updating of the
margins is given by

$\hat{r}_{ki}^{\textit{new}}=\hat{r}_{ki}^{\textit{curr}}+\check{y}_{ki}x_{i\zeta}\cdot({\hat{\beta}^{\textit{new}}}_{\zeta}-{\hat{\beta}^{\textit{curr}}}_{\zeta}),~{}\text{for}~{}k=1,\dots,K,~{}i=1,\dots,n.$

To update $\zeta$-th coordinate of $\bm{\beta}$, where $\zeta\in\mathcal{M}$,
we follow the same trick and the updating rule is as follows:

$\hat{\beta}_{\zeta}^{\textit{new}}={\hat{\beta}^{\textit{curr}}}_{\zeta}+\frac{-\sum_{k=1}^{K}w_{k}\sum_{i=1}^{n}L^{\prime}(\hat{r}_{ki}^{\textit{curr}})\check{y}_{ki}x_{i\zeta}}{\sum_{i=1}^{n}x_{i\zeta}^{2}}.$
(132)

The updating of margins is also given by equation (31).

Simiarly, for the $\zeta$-th intercept $b_{\zeta}$, we consider minimizing the
quadratic majorization:

$\mathcal{F}(b_{\zeta}|\hat{\bm{\beta}}^{\textit{curr}},\hat{\bm{b}}^{\textit{curr}}):=\frac{1}{n}\sum_{i=1}^{n}\left\\{L(\hat{r}_{{\zeta}i}^{\textit{curr}})+L^{\prime}(\hat{r}_{{\zeta}i}^{\textit{curr}})(-\check{y}_{{\zeta}i})(b_{\zeta}-\hat{b}_{\zeta}^{\textit{curr}})+\frac{1}{2}(b_{\zeta}-\hat{b}_{\zeta}^{\textit{curr}})^{2}\right\\},$
(133)

which has a minimizer

$\hat{b}_{\zeta}^{\textit{new}}=\hat{b}_{\zeta}^{\textit{curr}}+\frac{1}{n}\sum_{i=1}^{n}L^{\prime}(r_{{\zeta}i})\check{y}_{{\zeta}i}.$
(134)

The updating of margins is given by

$\hat{r}_{ki}^{\textit{new}}=\begin{cases}\hat{r}_{\zeta
i}^{\textit{curr}}-\check{y}_{{\zeta}i}(\hat{b}_{\zeta}^{\textit{new}}-\hat{b}_{\zeta}^{\textit{curr}}),&\text{if
}k=\zeta;\\\ \hat{r}_{ki}^{\textit{curr}},&\text{if }k\neq\zeta.\end{cases}$
(135)

To sum up, the CMD algorithm for solving the weighted $\ell_{1}$-penalized CPR
is given in Algorithm 4.

1:Initialize $\hat{\bm{\beta}}^{(0)}=\hat{\bm{\beta}}^{\text{initial}}$ and
compute the adaptive weight
$\hat{\bm{\omega}}^{(0)}=\left(\hat{\omega}_{j}^{(0)},~{}j\in\\{1,\dots,p\\}\backslash\mathcal{M}\right)=\left(p_{\lambda}^{\prime}(|\hat{\beta}_{j}^{(0)}|),~{}j\in\\{1,\cdots,p\\}\backslash\mathcal{M}\right)$

2:For $m=1,2,\cdots,$ repeat the LLA iteration till convergence

  * (2.a)

Obtain $(\hat{\beta}^{(m)},\hat{\bm{b}}^{(m)})$ by solving the following
optimization problem

$(\hat{\bm{\beta}}^{(m)},\hat{\bm{b}}^{(m)})=\arg\min_{\bm{\beta},\bm{b}}M_{n}(\bm{\beta},\bm{b})+\sum_{j\notin\mathcal{M}}\hat{\omega}_{j}^{(m-1)}\cdot|\beta_{j}|,$
(136)

  * (2.b)

Update the adaptive weight vector $\hat{\bm{\omega}}^{(m)}$ with
$\hat{\omega}_{j}^{(m)}=p_{\lambda}^{\prime}(|\hat{\beta}_{j}^{(m)}|)$.

Algorithm 3 The LLA Algorithm for solving unconstrained estimator

1:Input the weight vector $\hat{\bm{\omega}}$.

2:Initialize $(\hat{\bm{\beta}},\hat{\bm{b}})$ and its corresponding margin
$\hat{r}_{ki}=\check{y}_{ki}(\bm{x}_{i}^{T}\hat{\bm{\beta}}-\hat{b}_{k})$, for
$k=1,\dots,K$, and $i=1,\dots,n$.

3:Iterate (3.a)-(3.b) until convergence:

  * (3.a)

Cyclic coordinate descent for coefficients: for $j=1,2,\cdots,p$,

    * (3.a.1)

Compute

$\hat{\beta}_{j}^{\textit{new}}=\begin{cases}\mathcal{S}\left(\hat{\beta}_{j}+\frac{-\sum_{k=1}^{K}w_{k}\sum_{i=1}^{n}L^{\prime}(\hat{r}_{ki})\check{y}_{ki}x_{ij}}{\sum_{i=1}^{n}x_{ij}^{2}},\frac{\tilde{\omega}_{j}}{\frac{1}{n}\sum_{i=1}^{n}x_{ij}^{2}}\right),&\text{if
}j\in\\{1,\dots,p\\}\backslash\mathcal{M};\\\
\hat{\beta}_{j}+\frac{-\sum_{k=1}^{K}w_{k}\sum_{i=1}^{n}L^{\prime}(\hat{r}_{ki})\check{y}_{ki}x_{ij}}{\sum_{i=1}^{n}x_{ij}^{2}},&\text{if
}j\in\mathcal{M}.\end{cases}$

    * (3.a.2)

Compute the new margins:
$\hat{r}_{ki}^{\textit{new}}=\hat{r}_{ki}+\check{y}_{ki}x_{ij}\cdot({\hat{\beta}^{\textit{new}}}_{j}-\hat{\beta}_{j})$,
for $k=1,\dots,K$, $i=1,\dots,n$.

    * (3.a.3)

Update $\hat{\beta}_{j}={\hat{\beta}_{j}}^{\textit{new}}$, and
$\hat{r}_{ki}=\hat{r}_{ki}^{\textit{new}}$, for $k=1,\dots,K$, $i=1,\dots,n$.

  * (3.b)

Cyclic coordinate descent for intercepts: for $k=1,2,\cdots,K$,

    * (3.b.1)

Compute

$\hat{b}_{k}^{\textit{new}}=\hat{b}_{k}+\frac{1}{n}\sum_{i=1}^{n}L^{\prime}(r_{ki})\check{y}_{ki}.$

    * (3.b.2)

Compute the new margins:
$\hat{r}_{ki}^{\textit{new}}=\hat{r}_{ki}-\check{y}_{ki}(\hat{b}_{k}^{\textit{new}}-\hat{b}_{k}),$
for $i=1,\dots,n$.

    * (3.b.3)

Update $\hat{b}_{k}=\hat{b}_{k}^{\textit{new}}$, and
$\hat{r}_{ki}=\hat{r}_{ki}^{\textit{new}}$, for $i=1,\dots,n$.

Algorithm 4 The CMD algorithm for solving (136).