XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX
OJCOMS.2022.1234567

# Protecting Massive MIMO-Radar Coexistence: Precoding Design and Power
Control

Mohamed Elfiatoure1  Mohammadali Mohammadi1  Senior Member, IEEE,
Hien Quoc Ngo1  Senior Member, IEEE, Peter J. Smith2  Fellow, IEEE,
Michail Matthaiou1  Fellow, IEEE CORRESPONDING AUTHOR: Mohamed Elfiatoure
(e-mail: melfiatoure01@qub.ac.uk). This work is a contribution by Project
REASON, a UK Government funded project under the Future Open Networks Research
Challenge (FONRC) sponsored by the Department of Science Innovation and
Technology (DSIT). The work of M. Mohamamdi and M. Matthaiou was supported by
the European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme (grant agreement No. 101001331).The work of
H. Q. Ngo was supported by the U.K. Research and Innovation Future Leaders
Fellowships under Grant MR/X010635/1  Centre for Wireless Innovation (CWI),
Queen’s University Belfast, U.K.  School of Mathematics and Statistics,
Victoria University of Wellington, Wellington 6012, New Zealand

###### Abstract

This paper studies the coexistence between a downlink multiuser massive multi-
input-multi-output (MIMO) communication system and MIMO radar. The performance
of the massive MIMO system with maximum ratio ($\mathrm{MR}$), zero-forcing
($\mathrm{ZF}$), and protective $\mathrm{ZF}$ ($\mathrm{PZF}$) precoding
designs is characterized in terms of spectral efficiency (SE) and by taking
the channel estimation errors and power control into account. The idea of
$\mathrm{PZF}$ precoding relies on the projection of the information-bearing
signal onto the null space of the radar channel to protect the radar against
communication signals. We further derive closed-form expressions for the
detection probability of the radar system for the considered precoding
designs. By leveraging the closed-form expressions for the SE and detection
probability, we formulate a power control problem at the radar and base
station (BS) to maximize the detection probability while satisfying the per-
user SE requirements. This optimization problem can be efficiently tackled via
the bisection method by solving a linear feasibility problem. Our analysis and
simulations show that the $\mathrm{PZF}$ design has the highest detection
probability performance among all designs, with intermediate SE performance
compared to the other two designs. Moreover, by optimally selecting the power
control coefficients at the BS and radar, the detection probability improves
significantly.

Beamforming, detection probability, massive multiple-input multiple-output
(MIMO), power allocation, spectral efficiency.

##  1 INTRODUCTION

Spectrum sharing between the radar and cellular communication systems, termed
as _communication/radar co-existing systems_ , has been envisioned as an
enabling solution to address the explosive growth of wireless traffic demands
and shortage of licensed spectra [1, 2, 3, 4]. Nevertheless, the inherent
challenge of spectrum sharing, i.e., inter-system interference that
compromises the performance of both systems, calls for an efficient cross
interference management and spectrum assignment. To facilitate co-existence
with overlaid communication systems, a variety of techniques, such as
opportunistic spectrum sharing between cellular and rotating radar [5],
interference mitigation [1, 6], precoding or spatial separation [1, 7], and
waveform design for radar [1, 8, 9] have been proposed in the literature.

The widespread deployment of the massive multiple-input, multiple-output
(MIMO) technology in cellular networks on one hand [10, 11], and the potential
of MIMO radars on the other hand [12], has paved the way to the co-existence
of MIMO structures. MIMO technology offers waveform diversity and higher
detection capability for the radars, while at the same time, massive MIMO
technology delivers the significant user coverage and spectral efficiency (SE)
enhancement for the cellular systems. While the literature has focused more on
conventional MIMO for both systems, the potential of massive MIMO technology
to further boost the system’s performance has not been thoroughly studied yet.
Massive MIMO is a key enabling technology for 5G and beyond networks, which
relies on a large number of antennas at the BS to provide high spectral and
energy efficiency using relatively simple processing [13]. More importantly, a
BS with a large-antenna array can easily form a null to minimize interference
to a coexistent radar. This motivates us to develop a massive MIMO
communication system overlaid with a radar system.

###  1.1 Related Works

To manage the interference between the MIMO radar system and MIMO cellular
networks, the null space projection method [14, 15] and optimum beamforming
design [16, 17, 18, 19] have been widely discussed in the literature. More
specifically, Mahal  _et al._ [14] proposed a radar precoder design using
subspace projection methods and based on zero-forcing ($\mathrm{ZF}$) and
minimum mean-square-error (MMSE) criteria. Biswas  _et al._ [15] applied null-
space based waveform projection to mitigate the interference from the radar
system toward a full-duplex cellular system and proposed a joint transceiver
design at the base station (BS) and users to maximize the detection
probability of the MIMO radar system. The authors in [16] considered the
transmit beamforming design for spectrum sharing between downlink multiuser
MIMO communication and colocated MIMO radar to maximize the detection
probability of the radar, while guaranteeing the transmit power budget of the
BS and the received signal-to-interference-plus-noise-ratio (SINR) of each
downlink user. The authors in [16] focused on the transmit beamforming design
for spectrum sharing between downlink multiuser MIMO communication and
colocated MIMO radar. Their aim was to maximize the radar’s detection
probability, while ensuring that the base station’s transmit power budget and
each downlink user’s received signal-to-interference-plus-noise ratio (SINR)
are maintained. Qian  _et al._ [17] addressed the problem of joint design of
the radar transmit code, radar receive filter, and the communication system
codebook for the co-existence of MIMO radar and MIMO communication. Liu  _et
al._ [18] investigated power efficient transmission in the overlaid systems,
where the BS beamforming is designed to minimize the transmit power at the BS,
while guaranteeing the receive SINR at the users and the interference level
from BS to radar. Pu  _et al._ [19] extended the proposed design in [18] by
taking the radar transmit waveform design into consideration and maximized the
radar SINR, under the constraints of communication constructive interference,
radar waveform similarity and constant modulus.

###  1.2 Research Gap and Main Contributions

The integration of the massive MIMO into communication/radar co-existing
systems has been recently studied in [20, 21]. In [20], the rate region of a
coexistence based joint radar and communications system, comprising a single
cell massive MIMO communication system and a static MIMO radar has been
characterized. In the presence of radar interference, the uplink and downlink
achievable rates of the cellular system have been derived by applying MMSE
combining and regularized $\mathrm{ZF}$ beamformer at the BS, respectively. In
[21], closed-form expressions for the detection probability of the radar
system and the downlink SE of the massive MIMO system with maximum ratio
($\mathrm{MR}$) precoding were derived. Nonetheless, research on the
coexistence of massive MIMO communications and radar systems is still in its
infancy and how joint precoding design and power allocation affects its
performance remains unclear.

Thanks to the promising features of the massive MIMO technology, in this
paper, we investigate the potential benefits of massive MIMO coexisting with
MIMO radar to alleviate the inter-system interference. We consider a joint
radar communication system comprising a single-cell massive MIMO cellular
communication system and a MIMO radar operating over the same frequency band.
The specific contributions of our paper can be summarized as follows:

  * •

We characterize the performance of the cellular and radar system in terms of
downlink SE and detection probability, respectively, in the presence of
imperfect channel state information (CSI). Analytical results for
$\mathrm{MR}$ and $\mathrm{ZF}$ beamforming at the BS are derived. In order to
manage the inter-system interference, we design protective $\mathrm{ZF}$
($\mathrm{PZF}$) precoding at the BS to ensure the radar’s functionality is
not significantly impaired by the interference caused by downlink transmission
towards users. Accordingly, we characterize the performance of the cellular
and radar system with the $\mathrm{PZF}$ scheme.

  * •

We formulate a power allocation problem with the objective of maximizing the
detection probability for MIMO radar, subject to a power budget constraint at
the radar and minimum SE requirements at the cellular users. This problem is
efficiently solved via the bisection method. Our proposed power allocation
strategy provides a significant detection probability gain for all precoding
designs compared to the case without power allocation.

  * •

Our numerical results show that by increasing the number of BS antennas, the
gap between the PZF and MR design reduces, while ZF constantly outperforms the
MR. Nevertheless, while increasing the number of antennas at the radar results
in SE degradation, it can also significantly enhance the detection probability
at the radar site. By implementing power control at both the BS and radar, the
$\mathrm{MR}$, $\mathrm{ZF}$, and $\mathrm{PZF}$ schemes achieve detection
probability improvements of up to $55\%$, $49\%$, and $38\%$, respectively,
compared to their baseline values, while the detection probability approaches
to $1$. Finally, our findings demonstrate that the $\mathrm{PZF}$ scheme, when
combined with optimal power control, can consistently achieve a probability of
detection exceeding $0.8$ with a high probability (greater than $0.6$).

###  1.3 Paper Organization and Notation

The rest of this paper is organized as follows. Section 2 describes the system
model. Our performance analysis is pursued in Section 3. In Section 4, we
develop the proposed power allocation problem. Simulation results are
presented in Section 6. Finally, Section 7 contains concluding remarks.

Notations: We denote vectors and matrices by lower-case boldface symbols and
upper-case boldface symbols, respectively; $\|\cdot\|$ denotes the $l_{2}$
norm, $\mathrm{tr}(\cdot)$ denotes the matrix trace, $(\cdot)^{*}$,
$(\cdot)^{H}$ and $(\cdot)^{T}$ denote the conjugate, conjugate transpose, and
transpose, respectively; ${\mathbf{I}}_{M}$ denotes an $M\times M$ identity
matrix; $[{\bf A}]_{(:,k)}$ denotes the $k$-th column of ${\bf A}$;
$\mathcal{CN}\left(\boldsymbol{0},\mathbf{C}_{x}\right)$ denotes a circularly
symmetric complex Gaussian vector with zero-mean and covariance matrix
$\mathbf{C}_{x}$; $X\sim\mathcal{CN}(0,\sigma^{2})$ denotes a circularly
symmetric complex Gaussian random variable (RV) $X$ zero mean and variance
$\sigma^{2}$; $X\sim\mathcal{N}(0,1)$ denotes a real valued Gaussian RV;
$\mathbb{E}\\{\cdot\\}$ denotes the statistical expectation;
$Q_{n}(\cdot,\cdot)$ denotes the Marcum $Q$-function of order $n$ defined in
[22, Eq.(4.60)].

##  2 System Model

We consider a time division duplex (TDD) downlink massive MIMO communication
system coexisting with a MIMO radar system on the same time-frequency
resource. Radar is equipped with $N$ transmit and $N$ receive antennas, while
the massive MIMO communication system includes an $M$-antenna BS serving $K$
single-antenna users $(M>K)$. The basic structure of the model is illustrated
in Fig. 1, where

  * •

${\bf g}_{k}\in\mathbb{C}^{M\times 1}$ is the channel vector response between
the BS and the $k$-th user. The channel ${\bf g}_{k}$ is modeled as follows:

${\bf g}_{k}=\sqrt{\beta_{k}}\mathbf{z}_{k},$ (1)

where $\mathbf{z}_{k}$ represents the small-scale fading, assuming to include
independent and identically distributed (i.i.d.) RVs, i.e., $\mathbf{z}_{k}$
$\sim\mathcal{CN}\left(0,{\bf I}_{M}\right)$, while $\beta_{k}$ represents the
large-scale fading. Denote by ${\bf G}=[{\bf g}_{1}\ldots{\bf
g}_{K}]\in\mathbb{C}^{M\times K}$ the corresponding channel matrix from the BS
to all $K$ users.

  * •

${\bf R}\in\mathbb{C}^{N\times M}$ represents the channel matrix from the BS
to the radar receiver, whose elements are i.i.d.
$\sim\mathcal{CN}\left(0,\beta_{br}\right)$ RVs, where $\beta_{br}$ is the
corresponding large-scale fading coefficient.

  * •

$\mathbf{{f}}_{k}\in\mathbb{C}^{N\times 1}$ is the channel from the radar
transmitter to the $k$-th user. Denote by
$\mathbf{{F}}=[\mathbf{{f}}_{1}\ldots\mathbf{{f}}_{K}]\in\mathbb{C}^{N\times
K}$ the corresponding channel matrix from the radar transmitter to all $K$
users. We assume that
$\mathbf{{f}}_{k}\sim\mathcal{CN}\left(0,\bar{\beta}_{k}{\bf I}_{N}\right)$,
where $\bar{\beta}_{k}$ represents the large-scale fading.

In the following subsections, we provide details on the architecture of the
two systems.

###  2.1 Massive MIMO Communication System

Our focus here is on data transmission over the downlink with TDD operation.
Each transmission frame is divided to two phases: 1) uplink training phase and
2) downlink data transmission. Relying on the channel estimates obtained in
the uplink training phase, different linear processing schemes are applied at
the BS to transmit information towards all users.

Figure 1: Coexistence between massive MIMO cellular systems and MIMO radar.

####  2.1.1 Uplink Training

In this phase, the users and radar will first send the pilot sequences to the
BS. Then, based on the received pilot signals, the BS will estimate the
channels to users and radar. The CSI of the radar at the BS is required to
design the $\mathrm{PZF}$ scheme and protect the radar against the
interference from the BS. Let $\tau_{\mathrm{p}}$ be the number of symbols per
coherence interval used for uplink training phase. All users and radar
simultaneously transmit pilot sequences of length $\tau_{\mathrm{p}}$ symbols.

Let
$\sqrt{\tau_{\mathrm{p}}}\boldsymbol{\Phi}_{r}\in\mathbb{C}^{\tau_{\mathrm{p}}\times
N}$, and
$\sqrt{\tau_{\mathrm{p}}}\boldsymbol{\Phi}_{p}\in\mathbb{C}^{\tau_{\mathrm{p}}\times
K}$ be the pilot sequences sent by the radar and the $K$ users, respectively.
It is assumed that pilot sequences transmitted from all users and radar are
pairwisely orthogonal, i.e.,
$\boldsymbol{\Phi}_{r}^{H}\boldsymbol{\Phi}_{r}=\boldsymbol{I}_{N}$,
$\boldsymbol{\Phi}_{p}^{H}\boldsymbol{\Phi}_{p}=\boldsymbol{I}_{K}$,
$\boldsymbol{\Phi}_{r}^{H}\boldsymbol{\Phi}_{p}=\boldsymbol{0}_{N\times K}$,
and $\boldsymbol{\Phi}_{p}^{H}\boldsymbol{\Phi}_{r}=\boldsymbol{0}_{K\times
N}$. This requires $\tau_{\mathrm{p}}\geq N+K$. Then, the received pilot
signal at the BS can be expressed as

$\displaystyle~{}{\bf Y}_{\mathrm{\text{p}}}=\sqrt{\tau_{\mathrm{p}}P_{R}}{\bf
R}^{H}\boldsymbol{\Phi}_{r}^{H}+\sqrt{\tau_{\mathrm{p}}\rho_{u}}\mathbf{G}\boldsymbol{\Phi}_{p}^{H}+{\mathbf{N}},$
(2)

where $\mathbf{N}\in\mathbb{C}^{M\times\tau_{\mathrm{p}}}$ is the AWGN, whose
elements have zero mean and variance $\sigma^{2}_{C}$.

Denote by $\hat{{\bf R}}$ and $\hat{{\bf g}}_{k}$ the MMSE estimates of ${\bf
R}$ and ${{\bf g}}_{k}$, respectively. Then, they can be expressed as

$\displaystyle\hat{{\bf R}}$ $\displaystyle={\bf R}+\tilde{{\bf R}},$ (3)
$\displaystyle\hat{{\bf g}}_{k}$ $\displaystyle={\bf g}_{k}+\tilde{{\bf
g}}_{k},$ (4)

where $\tilde{{\bf R}}$ and $\tilde{{\bf g}}_{k}$ represent the corresponding
channel estimation errors. Following [23], the elements of $\hat{{\bf R}}$ and
the elements of $\hat{{\bf g}}_{k}$ are independent and distributed as
$\mathcal{CN}(0,\gamma_{r})$ and $\mathcal{CN}(0,\gamma_{k})$, respectively,
with

$\displaystyle\gamma_{r}$
$\displaystyle=\frac{\tau_{\mathrm{p}}P_{R}\beta_{br}^{2}}{\tau_{\mathrm{p}}P_{R}\beta_{br}+\sigma^{2}_{C}},$
(5) $\displaystyle\gamma_{k}$
$\displaystyle=\frac{{\tau_{\mathrm{p}}\rho_{u}}\beta_{k}^{2}}{\tau_{\mathrm{p}}\rho_{u}\beta_{k}+\sigma^{2}_{C}},$
(6)

where $P_{R}$ and $\rho_{u}$ are the transmit power of each pilot symbol at
the radar and the user, respectively, and $\sigma^{2}_{C}$ is the noise power
at the BS. From the MMSE estimation property, $\tilde{{\bf R}}$ is independent
of $\hat{{\bf R}}$ and $\tilde{{\bf g}}_{k}$ is independent of $\hat{{\bf
g}}_{k}$, and hence, $[\tilde{{\bf
R}}]_{n,m}\sim\mathcal{CN}\big{(}{0},(\beta_{br}-\gamma_{r})\big{)}$ and
$\tilde{{\bf
g}}_{k}\sim\mathcal{CN}\big{(}\boldsymbol{0},(\beta_{k}-\gamma_{k}){\bf
I}_{M}\big{)}$. We can represent the overall channel estimation matrix between
the BS and users as

$\displaystyle\hat{\mathbf{G}}$ $\displaystyle={\bf
H}\mathbf{D}^{\frac{1}{2}}_{\gamma},$ (7)

where $\hat{\mathbf{G}}=[\hat{{\bf g}}_{1},\ldots,\hat{{\bf
g}}_{K}]\in\mathbb{C}^{M\times K}$, ${\bf H}=[{\bf h}_{1},\ldots,{\bf
h}_{K}]\in\mathbb{C}^{M\times K}$ with ${\bf
h}_{k}\sim\mathcal{CN}(\boldsymbol{0},{\bf I}_{M})$, while ${\bf
D}_{\gamma}=\mathrm{diag}\\{\gamma_{1},\ldots,\gamma_{K}\\}$ is a diagonal
matrix. We can further represent $\hat{{\bf g}}_{k}$ as

$\displaystyle~{}\hat{{\bf g}}_{k}$ $\displaystyle=\sqrt{\gamma_{k}}{{\bf
h}}_{k}.$ (8)

####  2.1.2 Downlink Signal Transmission

By using the channel estimates obtained during the training phase, precoding
matrix is designed at the BS for downlink data transmission phase. Suppose
that the information symbols ${\bf
d}=[d_{1},\ldots,d_{K}]^{T}\in\mathbb{C}^{K\times 1}$ are independent with
$\mathbb{E}\big{\\{}{\bf d}{\bf d}^{H}\big{\\}}={\bf I}_{K}$, where $d_{k}$
denotes the information symbol intended for the $k$-th user. Then, the signal
transmitted from the BS can be expressed as

$\displaystyle\boldsymbol{x}={\bf T}{\bf D}^{\frac{1}{2}}_{\eta}{\bf
d}=\sum_{k=1}^{K}\sqrt{{\eta}_{k}}{\bf t}_{k}{d}_{k},$ (9)

where ${\bf T}=[{\bf t}_{1}\ldots{\bf t}_{K}]\in\mathbb{C}^{M\times K}$ is the
precoding matrix, which is a function of the channel estimate
$\hat{\mathbf{G}}$ and ${\bf t}_{k}\in\mathbb{C}^{N\times 1}$, where
$\mathbb{E}\big{\\{}{\|{\bf t}_{k}\|}^{2}\big{\\}}=1$, is the precoding vector
for user $k$, ${\bf D}_{\eta}=\mathrm{diag}\\{\eta_{1},\ldots,\eta_{K}\\}$ is
a diagonal matrix whose $k$-th diagonal element, $\eta_{k}$, represents the
power control coefficient for user $k$, chosen to satisfy the power constraint
at the BS $\mathbb{E}\\{\|\mathbf{x}\|^{2}\\}\leq 1$, which implies
$\sum_{k=1}^{K}{\eta_{k}}\leq 1$. The received signal vector at the users can
be expressed as

$\displaystyle\boldsymbol{y}=\underbrace{\sqrt{\rho}\boldsymbol{G}^{\mathrm{T}}\boldsymbol{x}}_{\text{desired
signal}}+\underbrace{\sqrt{P_{R}}\mathbf{F}^{T}\mathbf{s},}_{\text{interference
from radar}}+\boldsymbol{n},$ (10)

where $\rho$ denotes the BS transmit power, ${\bf s}\in\mathbb{C}^{N\times 1}$
is the transmitted probing signal from the radar, with
$\mathbb{E}\left\\{\mathbf{s}\mathbf{s}^{H}\right\\}={\bf I}_{N}$, to the
target, while $\boldsymbol{n}\sim C\mathcal{N}\left(0,\sigma_{C}^{2}{\bf
I}_{N}\right)$ is the AWGN at the users.

###  2.2  MIMO Radar

We consider a MIMO radar system that detects targets located in the far field.
Assuming the MIMO radar-to-target channel is line-of-sight (LoS), the
reflected signal (echo from the target) from one point-like target to the
radar receiver is interfered by the signal transmitted from the BS. Assuming
that a uniform linear array (ULA) is used at the radar, at the $l$-th
snapshot, the discrete signal vector ${\bf y}_{R}[l]$ received by the radar is
given by [16, 24]

$\displaystyle{\bf y}_{R}[l]=$ $\displaystyle\alpha\sqrt{P_{R}}{\bf
A}(\theta){\bf s}[l]+{\bf R}\sum_{k=1}^{K}\sqrt{{\eta}_{k}}{\bf
t}_{k}{d}_{k}[l],+{\bf w}[l],$ (11)

where $\alpha$ denotes the complex path loss of the radar-target-radar path;
$P_{R}$ is the transmitted power from the MIMO radar; $\theta$ is the azimuth
angle of the target; ${\bf w}[l]=\left[w_{i}[l],\ldots
w_{N}[l]\right]^{T}\in\mathbb{C}^{N\times 1}$ is the received additive white
Gaussian noise (AWGN) vector at the $l$-th snapshot with
$w_{m}[l]\sim\mathcal{CN}\left(0,\sigma_{R}^{2}\right)$, $\forall m$; ${\bf
A}(\theta)={\bf a}_{R}(\theta){\bf a}_{T}^{T}(\theta)$, in which ${\bf
a}_{T}(\theta)\in\mathbb{C}^{N\times 1}$ and ${\bf
a}_{R}(\theta)\in\mathbb{C}^{N\times 1}$ are the transmit and receive steering
vectors of the radar antenna array. Similar to [16, 24], and without
significant loss of generality, we assume that ${\bf a}_{R}(\theta)={\bf
a}_{T}(\theta)={\bf a}(\theta)$, where

$\displaystyle{\bf a}(\theta)=\left[1,e^{-j{2\pi
d}\sin\left(\theta\right)},\ldots,e^{-j{2\pi
d}\left(N-1\right)\sin\left(\theta\right)}\right]^{T},$ (12)

where $d$ represents the inter-antenna spacing normalized by the carrier
wavelength.

##  3 Performance Analysis

We evaluate the performance, in terms of downlink SE of the cellular
communication systems and detection probability for the radar system, for
different precoding schemes. Theoretically, the precoding matrix, ${\bf T}$,
can be optimized to achieve the optimal performance of the system. However,
the complexity of the optimum precoding grows dramatically with $M$ and $K$.
For the massive antenna regime with $M\gg K$, it is known that linear
precoders, i.e., $\mathrm{MR}$ and $\mathrm{ZF}$ perform fairly well [25, 26,
27]. Therefore, we focus on the performance of those precoders in the
following subsections. While these two precoders provide cellular users with
satisfactory performance, the BS may create interference at the radar
receiver. To address this issue, we propose to use $\mathrm{PZF}$ precoding at
the BS, to guarantee full protection for the radar against signals intended
for cellular users.

###  3.1 Spectral Efficiency

We assume that estimated CSI is not available at the user side and each user
uses only the statistical CSI for signal detection. Therefore, users treat the
mean effective channel gain as the channel knowledge for data detection. By
invoking (10) and using the use-and-then-forget bounding technique in [13, Eq.
(2.44)][28], known as hardening bound, we derive a lower bound on the downlink
SE of user $k$. To this end, using (10) we first rewrite the received signal
at user $k$ as

$\displaystyle~{}y_{k}$
$\displaystyle=\mathrm{DS}_{k}d_{k}+\mathrm{BU}_{k}d_{k}+\sum_{k^{\prime}\neq
k}\mathrm{IUI}_{kk^{\prime}}d_{k^{\prime}}+\sqrt{P_{R}}{\bf f}^{T}_{k}{\bf
s}+n_{k},$ (13)

where

$\displaystyle\mathrm{DS}_{k}$
$\displaystyle=\sqrt{{\rho\eta_{k}}}\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf
t}_{k}\big{\\}},~{}$ (14) $\displaystyle\mathrm{BU}_{k}$
$\displaystyle=\sqrt{{\rho\eta_{k}}}\Big{(}{\bf g}_{k}^{T}{\bf
t}_{k}-\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf t}_{k}\big{\\}}\Big{)},~{}$ (15)
$\displaystyle\mathrm{IUI}_{kk^{\prime}}$
$\displaystyle=\sqrt{{\rho\eta_{k^{\prime}}}}{\bf g}_{k}^{T}{\bf
t}_{k^{\prime}},~{}$ (16)

represent the strength of the desired signal ($\mathrm{DS}_{k}$), the
beamforming gain uncertainty ($\mathrm{BU}_{k}$), and the interference caused
by the $k^{\prime}$-th user, respectively.

Accordingly, an achievable downlink SE at the $k$-th user can be expressed as

$\displaystyle~{}\mathrm{SE}_{k}=\Big{(}1-\frac{\tau_{\mathrm{p}}}{\tau}\Big{)}\log_{2}(1+\mathrm{SINR}_{k}),$
(17)

where the effective SINR is given by

$\displaystyle\mathrm{SINR}_{k}=$
$\displaystyle\\!\frac{\big{|}\mathrm{DS}_{k}\big{|}^{2}}{\mathbb{E}\Big{\\{}\big{|}\mathrm{BU}_{k}\big{|}^{2}\Big{\\}}+\sum_{k^{\prime}\neq
k}\mathbb{E}\Big{\\{}\big{|}\mathrm{IUI}_{kk^{\prime}}\big{|}^{2}\Big{\\}}\\!+P_{R}\bar{\beta}_{k}N+\sigma_{C}^{2}}.$
(18)

The achievable downlink SE in (17) is general and valid regardless of the
precoding scheme used at the BS. We derive closed-form expressions for the
$\mathrm{MR}$, $\mathrm{ZF}$, and $\mathrm{PZF}$ precoding schemes in the
following subsection.

####  3.1.1 Maximum-Ratio Precoder

The $\mathrm{MR}$ precoder is employed at the BS, particularly due to its
advantages including low computational complexity, ease of analysis, and
reasonable performance, as shown in [25, 26, 27, 29]. With $\mathrm{MR}$
precoding design, the linear precoding vectors ${\bf t}_{k}={\bf
t}^{\mathrm{MR}}_{k}$ given by [26]

$\displaystyle{\bf t}^{\mathrm{MR}}_{k}=\alpha_{\mathrm{MR}}{{\bf
h}}^{*}_{k},$ (19)

where $\alpha_{\mathrm{MR}}=\frac{1}{\sqrt{M}}$ is the normalization factor.

######  Proposition 1.

The SE of the $k$-th user achieved by the $\mathrm{MR}$ precoding can be
expressed by (20), at the top of the next page.

$\displaystyle{\mathrm{SE}}^{\mathrm{MR}}_{k}=\Big{(}1-\frac{\tau_{\mathrm{p}}}{\tau}\Big{)}\log_{2}\left(1+\frac{M\rho\gamma_{k}\eta_{k}}{\rho\gamma_{k}\sum_{k^{\prime}=1}^{K}{\eta_{k^{\prime}}}+P_{R}\bar{\beta}_{k}N+\sigma_{C}^{2}}\right),\forall
k.$ (20)

###### Proof.

See Appendix 8.2. ∎

####  3.1.2 Zero-Forcing Precoder

In order to mitigate the inter-user interference, we exploit the $\mathrm{ZF}$
principle for precoding design. With ZF, the precoding vector ${\bf t}_{k}$
can be expressed as

$\displaystyle{\bf t}^{\mathrm{ZF}}_{k}=\alpha_{\mathrm{ZF}}\left[{{\bf
H}}^{*}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}\right]_{(:,k)},$ (21)

where $\alpha_{\mathrm{ZF}}=\sqrt{M-K}$ is the normalization factor [30].

######  Proposition 2.

The SE of the $k$-th user achieved by the $\mathrm{ZF}$ precoding can be
expressed by (22), at the top of the next page.

$\displaystyle{\mathrm{SE}}^{\mathrm{ZF}}_{k}=\Big{(}1-\frac{\tau_{\mathrm{p}}}{\tau}\Big{)}\log_{2}\left(1+\frac{(M-K)\rho\gamma_{k}\eta_{k}}{\rho(\beta_{k}-\gamma_{k})\sum_{k^{\prime}=1}^{K}{\eta_{k^{\prime}}}+P_{R}\bar{\beta}_{k}N+\sigma_{C}^{2}}\right),\forall
k.$ (22)

###### Proof.

See Appendix 8.3. ∎

####  3.1.3 Protective Zero-Forcing Precoder

The $\mathrm{MR}$ and $\mathrm{ZF}$ precoders do not take the interference
from the BS to radar into account. To protect the radar against the
interference caused by the downlink transmission towards cellular users, we
now elaborate on the $\mathrm{PZF}$ precoding scheme. The $\mathrm{PZF}$
scheme ensures that the radar is fully protected from BS interference if the
BS has perfect CSI from the radar. To achieve this objective, some degrees-of-
freedom at the BS are used to steer the BS beams into the radar’s null-space.
Therefore, from the perspective of the cellular communication network, we can
see that $\mathrm{PZF}$ is inferior to $\mathrm{ZF}$, while it performs better
than the $\mathrm{MR}$ scheme. Importantly, it outperforms both $\mathrm{ZF}$
and $\mathrm{MR}$ in terms of the detection probability.

Let now ${\bf B}$ denote the projection matrix onto the orthogonal complement
of $\hat{{\bf R}}$, which can be expressed as

$\displaystyle\mathbf{B}=\mathbf{I}_{M}-{\hat{{\bf R}}^{H}}\left({\hat{{\bf
R}}}{\hat{{\bf R}}^{H}}\right)^{-1}{\hat{{\bf R}}}.$ (23)

Then, the $\mathrm{PZF}$ precoder is designed as

$\displaystyle~{}{\mathbf{t}^{\mathrm{PZF}}_{k}}$
$\displaystyle=\alpha_{\mathrm{PZF}}\mathbf{B}\mathbf{w}_{k}^{\mathrm{ZF}},$
(24)

where ${\bf w}_{k}^{\mathrm{ZF}}$ and ${\bf t}^{\mathrm{PZF}}_{k}$ are $k$-th
column of ${\bf W}^{\mathrm{ZF}}$ and ${\bf T}^{\mathrm{PZF}}$ matrices,
respectively, where ${\bf W}^{\mathrm{ZF}}={{\bf H}}^{*}({\bf H}^{T}{\bf
H}^{*})^{-1}$ and ${\bf T}^{\mathrm{PZF}}=\alpha_{\mathrm{PZF}}{\bf B}{\bf
W}^{\mathrm{ZF}}$. Moreover, the normalization factor is expressed as:

$\displaystyle\alpha_{\mathrm{PZF}}=\frac{1}{\sqrt{\mathbb{E}\\{\|\mathbf{B}\mathbf{w}_{k}^{\mathrm{ZF}}\|^{2}\\}}}=\frac{1}{\sqrt{\mathbb{E}\\{\mathrm{tr}(\mathbf{w}_{k}^{\mathrm{ZF}}(\mathbf{w}_{k}^{\mathrm{ZF}})^{H}\mathbf{B})\\}}},$
(25)

where we have used $\mathbf{B}^{H}\mathbf{B}=\mathbf{B}$. By noticing that
${\bf B}$ is independent of ${\bf w}_{k}^{\mathrm{ZF}}$, we have

$\displaystyle\alpha_{\mathrm{PZF}}$
$\displaystyle=\frac{1}{\sqrt{\mathrm{tr}\Big{(}\mathbb{E}\big{\\{}\mathbf{w}_{k}^{\mathrm{ZF}}(\mathbf{w}_{k}^{\mathrm{ZF}})^{H}\big{\\}}\mathbb{E}\\{\mathbf{B}\\}\Big{)}}}$
$\displaystyle=\sqrt{\frac{M(M-K)}{(M-N)}},$ (26)

where we have applied Lemma 1 from Appendix 8.1 and [31, Lemma 2.10] to derive
the final result. Accordingly, the achievable SE by the $k$-th user using
$\mathrm{PZF}$ precoding is given in the following proposition.

The idea of PZF is the same as interference nulling which has been introduced
in the space of multi-cell cellular systems [32] to mitigate the inter-cell
interference and next applied to the cell-free massive MIMO systems in [33] to
suppress the interference caused by each access point to part of users.

######  Proposition 3.

The SE of the $k$-th user achieved by the $\mathrm{PZF}$ precoding can be
expressed by (27) at the top of the next page.

$\displaystyle{\mathrm{SE}}^{\mathrm{PZF}}_{k}=\Big{(}1-\frac{\tau_{\mathrm{p}}}{\tau}\Big{)}\log_{2}\left(1+\frac{\frac{\rho}{M}(M-K)(M-N)\gamma_{k}\eta_{k}}{\frac{\rho}{M}{(M-N)(M-K)}(\beta_{k}-\gamma_{k})\sum_{k^{\prime}=1}^{K}\eta_{k^{\prime}}+P_{R}\bar{\beta}_{k}N+\sigma_{C}^{2}}\right),\forall
k.$ (27)

###### Proof.

See Appendix 8.4. ∎

###  3.2  Detection Probability

In this subsection, we derive the detection probability for radar system,
under the Neyman-Pearson criterion. By using the Generalized Likelihood Ratio
Test, the asymptotic detection probability for radar $P_{d}$ is given as [21]

$\displaystyle~{}P_{d}=1-\mathfrak{F}_{X_{2}^{2}(\mu)}\left(\mathfrak{F}_{X_{2}^{2}}^{-1}\left(1-P_{FA}\right)\right),$
(28)

where $P_{FA}$ is the radar’s probability of false alarm,
$\mathfrak{F}_{X_{2}^{2}(\mu)}$ is the non-central chi-square cumulative
distribution function (CDF) with two degrees-of-freedom (DoF),
$\mathfrak{F}_{X_{2}^{2}}^{-1}$ is the inverse function of chi-square CDF,
while the non-centrality parameter, $\mu$, for $X_{2}^{2}$ is given by [21]

$\mu=|\alpha|^{2}LP_{R}\mathrm{tr}\left({\bf A}{\bf A}^{H}\left({\bf
R}\tilde{{\bf T}}{\bf R}^{H}+\sigma_{R}^{2}{\bf I}_{N}\right)^{-1}\right),$
(29)

where $\tilde{{\bf T}}={\bf T}{\bf D}_{\eta}{\bf T}^{H}$.

By invoking the generalized Marcum Q-function, $P_{d}$ can be written as

$P_{d}=Q_{1}\big{(}\sqrt{\mu},\sqrt{C_{FA}}\big{)},$ (30)

where $C_{FA}=\mathfrak{F}_{x_{2}^{2}}^{-1}(1-P_{FA})$ [21]. Hence, in order
to derive $P_{d}$ for different precoding schemes, we need to find $\mu$.

By leveraging the massive MIMO concept, we are able to obtain a tight
approximation to $\mu$. Specifically, by using the trace lemma [34], as $M$
$\rightarrow\infty$, for the considered linear precoding schemes, we get

$\displaystyle\begin{cases}\frac{1}{M}{\bf R}\tilde{{\bf T}}^{i}{\bf
R}^{H}-\frac{\beta_{br}}{M}\mathrm{tr}(\tilde{{\bf T}}^{i}){\bf
I}_{N}\buildrel a.s.\over{\rightarrow}0&{i}\in\\{\mathrm{MR},\mathrm{ZF}\\}\\\
\frac{1}{M}{\bf R}\tilde{{\bf T}}^{\mathrm{PZF}}{\bf
R}^{H}-\frac{(\beta_{br}-\gamma_{r})}{M}\mathrm{tr}(\tilde{{\bf
T}}^{\mathrm{PZF}}){\bf I}_{N}\buildrel a.s.\over{\rightarrow}0,&\end{cases}$
(31)

where $\buildrel a.s.\over{\rightarrow}$ denotes the almost sure convergence.
Moreover, in the case of $\mathrm{PZF}$ scheme, we have used the fact that

$\displaystyle~{}{\bf R}\tilde{{\bf T}}^{\mathrm{PZF}}{\bf R}^{H}$
$\displaystyle=(\hat{{\bf R}}+\tilde{{\bf R}})\tilde{{\bf
T}}^{\mathrm{PZF}}(\hat{{\bf R}}+\tilde{{\bf R}})^{H}$
$\displaystyle=\tilde{{\bf R}}\tilde{{\bf T}}^{\mathrm{PZF}}\tilde{{\bf
R}}^{H}.$ (32)

As a result, (29) can be tightly approximated as

$\displaystyle\mu^{i}$
$\displaystyle\approx|\alpha|^{2}LP_{R}\mathrm{tr}\left({\bf A}{\bf
A}^{H}\left(\zeta_{i}\mathrm{tr}(\tilde{{\bf T}}^{i}){\bf
I}_{N}+\sigma_{R}^{2}{\bf I}_{N}\right)^{-1}\right)$
$\displaystyle=\frac{|\alpha|^{2}LP_{R}\mathrm{tr}({\bf A}{\bf
A}^{H})}{\zeta_{i}\mathrm{tr}\big{(}\tilde{{\bf
T}}^{i}\big{)}+\sigma_{R}^{2}},$ (33)

where $\zeta_{\mathrm{MR}}=\zeta_{\mathrm{ZF}}=\beta_{br}$ and
$\zeta_{\mathrm{PZF}}=(\beta_{br}-\gamma_{r})$. We now derive closed-form
expressions of $\mu$ for the $\mathrm{MR}$, $\mathrm{ZF}$, and $\mathrm{PZF}$
precoding schemes in the following subsections.

####  3.2.1 Maximum-Ratio Precoder:

From (19), $\tilde{{\bf T}}^{\mathrm{MR}}$ can be defined as

$\displaystyle\tilde{{\bf T}}^{\mathrm{MR}}$ $\displaystyle=\sum_{k=1}^{K}{\bf
t}_{k}^{\mathrm{MR}}\eta_{k}({\bf t}^{\mathrm{MR}}_{k})^{H}={\bf
T}^{\mathrm{MR}}{\bf D}_{\eta}({\bf T}^{\mathrm{MR}})^{H}$
$\displaystyle=\frac{1}{M}({{\bf H}^{*}}{\bf D}_{\eta}{{\bf H}^{T}}).$ (34)

Since $\frac{1}{M}{{\bf H}^{T}}{{\bf H}^{*}}\to{\bf I}_{K}$ for sufficiently
large $M$ [26], we have $\mathrm{tr}\big{(}\tilde{{\bf
T}}^{\mathrm{MR}}\big{)}=\mathrm{tr}\left({\bf D}_{\eta}\frac{{{\bf
H}^{T}}{{\bf H}^{*}}}{M}\right)\approx\sum_{k=1}^{K}\eta_{k}$. As a result,

$\mu^{\mathrm{MR}}\approx\frac{|\alpha|^{2}LP_{R}\mathbf{\mathrm{tr}}\big{(}{\bf
A}{\bf A}^{H}\big{)}}{\beta_{br}\sum_{k=1}^{K}\eta_{k}+\sigma_{R}^{2}}.$ (35)

####  3.2.2 Zero-Forcing Precoder:

Form (21), $\tilde{{\bf T}}^{\mathrm{ZF}}$ can be derived as

$\displaystyle~{}\tilde{{\bf T}}^{\mathrm{ZF}}$ $\displaystyle={\bf
T}^{\mathrm{ZF}}{\bf D}_{\eta}({\bf T}^{\mathrm{ZF}})^{H}$
$\displaystyle=(M-K){\bf H}^{*}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}{\bf
D}_{\eta}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}{\bf H}^{T}.$ (36)

To this end, by using (3.2.2), we get

$\displaystyle\mathrm{tr}\big{(}\tilde{{\bf T}}^{ZF}\big{)}$
$\displaystyle=(M-K)\mathrm{tr}\big{(}{\bf D}_{\eta}({{\bf H}}^{T}{{\bf
H}}^{*})^{-1}\big{)}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\approx}}\frac{M-K}{M}\mathrm{tr}\left({\bf
D}_{\eta}\right)$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}\frac{M-K}{M}\sum_{k=1}^{K}\eta_{k},$
(37)

where (a) follows from the fact that $({{\bf H}}^{T}{{\bf
H}}^{*})^{-1}\approx\frac{1}{M}{\bf I}_{K}$ for sufficiently large values of
$M$, and (b) holds since $\mathrm{tr}\left({\bf
D}_{\eta}\right)=\sum_{k=1}^{K}\eta_{k}$. Then, the non-centrality parameter
$\mu^{\mathrm{ZF}}$ can be approximated by

$\displaystyle\mu^{\mathrm{ZF}}\approx\frac{|\alpha|^{2}LP_{R}\mathrm{tr}\big{(}{\bf
A}{\bf
A}^{H}\big{)}}{\frac{M-K}{M}\beta_{br}\sum_{k=1}^{K}\eta_{k}+\sigma_{R}^{2}}.$
(38)

####  3.2.3 Protective Zero-Forcing Precoder:

######  Proposition 4.

With PZF precoding, the non-centrality parameter, $\mu$, can be derived as

$\displaystyle\mu^{\mathrm{PZF}}\approx\frac{|\alpha|^{2}LP_{R}\mathbf{\mathrm{tr}}\big{(}{\bf
A}{\bf
A}^{H}\big{)}}{\frac{M-K}{M}(\beta_{br}-\gamma_{r})\sum_{k=1}^{K}\eta_{k}+\sigma_{R}^{2}}.$
(39)

###### Proof.

See Appendix 8.5. ∎

##  4 Power Optimization

To protect the radar system against the BS transmissions and at the same time
to guarantee the performance requirements of the cellular users, power control
at the BS and radar can be applied. In this regard, we aim at selecting the
radar transmit power $P_{R}$ and BS power control coefficients $\eta_{k}$ to
maximize the detection probability for MIMO radar, under the constraints on
per-user SE and sum of power control coefficients $\eta_{k}$. More precisely,
the optimization problem is mathematically described as

$\displaystyle(\mathbb{P}1):\hskip
10.00002pt\underset{{P_{R},\eta_{k}}}{\max}\,\,\hskip 10.00002pt$
$\displaystyle P_{d}$ (40a) $\displaystyle\mathrm{s.t.}\,\,\hskip 10.00002pt$
$\displaystyle{\mathrm{SE}}_{k}\geq\mathrm{SE}_{k,\mathrm{th}},~{}\forall k,$
(40b) $\displaystyle 0\leq{P}_{R}\leq P_{R,\mathrm{max}},$ (40c)
$\displaystyle\sum_{k=1}^{K}\eta_{k}\leq 1,$ (40d)

where $P_{R,\mathrm{max}}$ is the maximum transmit power at the MIMO radar and
$\mathrm{SE}_{k,\mathrm{th}}$ is the minimum SE requirement by the $k$-th
user. The constraint (40d) represents the maximum transmit power constraint at
the BS. By invoking (30), and since the Marcum $Q$-function is an increasing
function of $\mu$, we can further rewrite $(\mathbb{P}1)$ as the following
optimization problem

$\displaystyle(\mathbb{P}2):\hskip
10.00002pt\underset{{P_{R},\eta_{k}}}{\max}\,\,\hskip 10.00002pt$
$\displaystyle\mu$ (41a) $\displaystyle\mathrm{s.t.}\,\,\hskip 10.00002pt$
$\displaystyle{\mathrm{SE}}_{k}\geq\mathrm{SE}_{k,\mathrm{th}},~{}\forall k,$
(41b) $\displaystyle 0\leq{P}_{R}\leq P_{R,\mathrm{max}},$ (41c)
$\displaystyle\sum_{k=1}^{K}\eta_{k}\leq 1.$ (41d)

By invoking (35), (38), and (39), we can rewrite the optimization problem
$(\mathbb{P}2)$ in the general form

$\displaystyle(\mathbb{P}3):\hskip
10.00002pt\underset{P_{R},\eta_{k}}{\max}\,\,\hskip 10.00002pt$
$\displaystyle\frac{|\alpha|^{2}LP_{R}\mathrm{tr}\big{(}{\bf A}{\bf
A}^{H}\big{)}}{a_{i}\sum_{k=1}^{K}\eta_{k}+\sigma_{R}^{2}}$ (42a)
$\displaystyle\mathrm{s.t.}\,\,\hskip 10.00002pt$
$\displaystyle\frac{b_{i}\rho\eta_{k}\gamma_{k}}{\rho
c_{i,k}\sum_{k^{\prime}=1}^{K}\eta_{k^{\prime}}\\!+\\!\bar{\beta}_{k}P_{R}N\\!+\\!1}\geq\gamma_{k,\mathrm{th}},~{}\forall
k,$ (42b) $\displaystyle 0\leq{P}_{R}\leq P_{R,\mathrm{max}},$ (42c)
$\displaystyle\sum_{k=1}^{K}\eta_{k}\leq 1,$ (42d)

where $\gamma_{k,\mathrm{th}}=2^{\mathrm{SE}_{k,\mathrm{th}}}-1$,
$i\in\\{\mathrm{MR},\mathrm{PZF},\mathrm{ZF}\\}$ and
$a_{\mathrm{MR}}=\beta_{br}$, $a_{\mathrm{ZF}}=\frac{(M-K)}{M}\beta_{br}$,
$a_{\mathrm{PZF}}=\frac{M-K}{M}(\beta_{br}-\gamma_{r})$, $b_{\mathrm{MR}}=M$;
$b_{\mathrm{PZF}}=\frac{1}{M}(M-K)(M-N)$, $b_{\mathrm{ZF}}=(M-K)$;
$c_{\mathrm{MR},k}=\gamma_{k}$,
$c_{\mathrm{PZF},k}=b_{\mathrm{PZF}}(\beta_{k}-\gamma_{k})$, and
$c_{\mathrm{ZF},k}=\beta_{k}-\gamma_{k}$.

###  4.1 Bisection Search-Based Solution

Since expression (42a) is quasi-concave and constraints of the problem are
linear functions of the optimization variables, the optimization problem
($\mathbb{P}3$) is quasi-concave. Thus, it can be equivalently reformulated as

$\displaystyle(\mathbb{P}4):\hskip
5.0pt\underset{{P_{R},\eta_{k}}}{\max}\,\,\hskip 10.00002pt$ $\displaystyle t$
(43a) $\displaystyle\mathrm{s.t.}\,\,\hskip 5.0pt$
$\displaystyle{b_{i}\rho\eta_{k}\gamma_{k}}\geq$
$\displaystyle\gamma_{k,\mathrm{th}}\Big{(}{\rho
c_{i,k}\sum_{k^{\prime}=1}^{K}\eta_{k^{\prime}}\\!+\\!\bar{\beta}_{k}P_{R}N\\!+\\!1}\Big{)},\forall
k,$ (43b) $\displaystyle{\left|\alpha|^{2}LP_{R}\mathrm{tr}\right({\bf A}{\bf
A}^{H})}\\!\geq\\!t\Big{(}{a_{i}\sum_{k=1}^{K}\eta_{k}+\sigma_{R}^{2}}\Big{)}$
(43c) $\displaystyle 0\leq{P}_{R}\leq P_{R,\mathrm{max}},$ (43d)
$\displaystyle\sum_{k=1}^{K}\eta_{k}\leq 1,$ (43e)

where

$\displaystyle t=\frac{\left|\alpha|^{2}LP_{R}\mathbf{\mathrm{tr}}\right({\bf
A}{\bf A}^{H})}{a_{i}\sum_{k=1}^{K}\eta_{k}+\sigma_{R}^{2}},$ (44)

is an auxiliary variable.

Problems (43) can be solved efficiently by a bisection search[35], where in
each step we solve a sequence of convex feasibility problems as detailed in
Algorithm 1.

Algorithm 1 Bisection Algorithm for Solving $(\mathbb{P}4)$

1: Initialization: choose the initial values of $t_{\min}$ and $t_{\max}$,
where $t_{\min}$ and $t_{\max}$ define a range of relevant values of the
objective function, choose a tolerance $\varsigma\geq 0$.

2: While ${t_{\min}-t_{\max}}\leq\varsigma$.

3: Set $t=\frac{t_{\min}+t_{\max}}{2}$ and solve the feasibility problem.

4: If problem $(\mathbb{P}4)$ is feasible, then set $t_{\min}=t$, else set
$t_{max}=t$.

5: Stop if $t_{\max}-t_{\min}\leq\epsilon$. Otherwise, go to Step 2.

####  4.1.1 Convergence and Complexity Analysis

Let $p^{\star}$ denote the optimal value of the quasiconvex optimization
problem ($\mathbb{P}3$). If the feasibility problem ($\mathbb{P}4$) is
feasible, then we have $p^{\star}\leq t$. Conversely, if the problem
($\mathbb{P}4$) is infeasible, then we can conclude that $p^{\star}\geq t$. We
can assess whether the optimal value, denoted as $p^{\star}$, for a
quasiconvex optimization problem is less than or greater than a predefined
value $t$ by solving the convex feasibility problem ($\mathbb{P}4$). If the
convex feasibility problem yields a feasible solution, we can conclude that
$p^{\star}\leq t$. Conversely, if the convex feasibility problem is
infeasible, it implies that $p^{\star}\geq t$. To implement a bisection
algorithm, the interval $[t_{\min},t_{\max}]$ is guaranteed to contain
$p^{\star}$, i.e., we have $t_{\min}\leq p^{\star}\leq t_{\min}$ at each step.
In each iteration, the interval is divided in two, i.e., bisected, so the
length of the interval after $k$ iterations is
$\frac{(t_{\max}-t_{\min})}{2^{k}}$, where $(t_{\max}-t_{\min})$ is the length
of the initial interval. It follows that exactly $\lceil\log
2((t_{\max}-t_{\min})/\epsilon)\rceil$ iterations are required before the
algorithm terminates. Each step involves solving the convex feasibility
problem ($\mathbb{P}4$).

According to [36], the per-iteration cost to solve the feasibility problem
($\mathbb{P}4$) is
$\mathcal{O}\big{(}(n_{l}+n_{v})n_{v}^{2}n_{l}^{0.5}\big{)}$, where
$n_{l}=K+3$ denotes the number of linear constraints and $n_{v}=K+1$ is the
number of real valued scalar decision variables. Therefore, the overall
complexity of the bisection algorithm is $\lceil\log
2((t_{\max}-t_{\min})/\epsilon)\rceil\mathcal{O}\big{(}(n_{l}+n_{v})n_{v}^{2}n_{l}^{0.5}\big{)}$.

###  4.2  Linear Programming Solution

The computational complexity of the bisection-based search algorithm can be
reduced by noticing that ($\mathbb{P}$1) can be presented as a linear
programming (LP) problem. The optimization problem in (42) is a non-concave
problem due to non-concavity of the objective function. Before proceeding, we
notice that both the objective function and first constraint (42a) are coupled
together via the optimization variables $P_{R}$ and $\eta_{k}$, for
$k=1,\ldots,K$. We now prove that the optimal value for $P_{R}$ is
$P_{R}^{\star}=P_{R,\mathrm{max}}$. To this end, assume that
$P_{R}^{\star}\geq P_{R,\mathrm{max}}$. Now, we define the new variables
$P_{R,{\mathsf{new}}}=cP_{R}^{\star}$ and
$\eta_{k,{\mathsf{new}}}=c\eta^{\star}_{k}$, where $c\geq 1$. Then, the
objective function (42a) can be obtained as

$\displaystyle f(P_{R,{\mathsf{new}}},\\{\eta_{k,{\mathsf{new}}}\\})$
$\displaystyle=\frac{|\alpha|^{2}LcP^{*}_{R}\mathrm{tr}\big{(}{\bf A}{\bf
A}^{H}\big{)}}{a_{i}c\sum_{k=1}^{K}\eta^{*}_{k}+\sigma_{R}^{2}}$
$\displaystyle\geq\frac{|\alpha|^{2}LcP^{\star}_{R}\mathrm{tr}\big{(}{\bf
A}{\bf
A}^{H}\big{)}}{a_{i}c\sum_{k=1}^{K}\eta^{*}_{k}+\sigma_{R}^{2}}=f(P_{R}^{\star},\\{\eta_{k}^{\star}\\}).$
(45)

Moreover, at given $P_{R,{\mathsf{new}}}$ and $\eta_{k,{\mathsf{new}}}$, the
first constraint is upper bounded as

$\displaystyle\frac{b_{i}\rho c\eta^{\star}_{k}\gamma_{k}}{\rho
c_{i,k}c\sum_{k^{\prime}=1}^{K}\\!\\!\eta^{\star}_{k^{\prime}}\\!+\\!\bar{\beta}_{k}cP^{*}_{R}N\\!+\\!1}\geq$
$\displaystyle\hskip 40.00006pt\frac{b_{i}\rho\eta^{\star}_{k}\gamma_{k}}{\rho
c_{i,k}\sum_{k^{\prime}=1}^{K}\eta^{\star}_{k^{\prime}}\\!+\\!\bar{\beta}_{k}P^{*}_{R}N+1},~{}\forall
k.$ (46)

Therefore, by increasing $P_{R}$ and $\eta_{k}$, respectively, beyond
$P_{R,\mathrm{max}}$ and $\eta_{k}^{\star}$, the objective function is
increased, and the first constraint is still satisfied. This implies that at
the optimal point we have $P_{R}^{\star}=P_{R,\mathrm{max}}$.

Now, for the sake of notational simplicity, let
$\boldsymbol{1}_{K}=[1,\ldots,1]^{T}$ be a vector of size $K\times 1$.
Moreover, define $\boldsymbol{\eta}=[\eta_{1},\ldots,\eta_{K}]^{T}$. Then, the
optimization problem ($\mathbb{P}$3) can be reduced to

$\displaystyle(\mathbb{P}5):\underset{\boldsymbol{\eta}}{\max}\,\,\hskip
1.99997pt$
$\displaystyle\boldsymbol{1}_{K}^{T}\boldsymbol{\eta}+\frac{\sigma_{R}^{2}}{a_{i}}$
(47a) $\displaystyle\mathrm{s.t.}\,\,\hskip 1.99997pt$ $\displaystyle{\bf
C}\boldsymbol{\eta}\leq{\bf b},$ (47b)

where ${\bf b}=[\Lambda_{1},\ldots,\Lambda_{K},1]$, with
$\Lambda_{k}=-\gamma_{k,\mathrm{th}}(\bar{\beta}_{k}P_{R,\mathrm{max}}N\\!+\\!1)$
and

$\displaystyle{\bf
C}=\begin{bmatrix}\Psi_{1}&\Theta_{1}&\Theta_{1}&\ldots&\Theta_{1}&\Theta_{1}\\\
\Theta_{2}&\Psi_{2}&\Theta_{2}&\ldots&\Theta_{2}&\Theta_{2}\\\
\vdots&\vdots&\vdots&\ddots&\vdots&\vdots\\\
\Theta_{K}&\Theta_{K}&\Theta_{K}&\ldots&\Theta_{K}&\Psi_{K}\\\
1&1&1&\ldots&1&1\end{bmatrix},$ (48)

with $\Theta_{k}=\rho c_{i,k}\gamma_{k,\mathrm{th}}$ and $\Psi_{k}=\rho
c_{i,k}\gamma_{k,\mathrm{th}}-\rho b_{i}\gamma_{k}$. The problem
($\mathbb{P}$5) is an LP problem and can be efficiently solved via interior-
point algorithms [35]. We use the linprog function in Matlab platform to solve
Problem ($\mathbb{P}5$).

According to [36], the complexity of the optimization problem ($\mathbb{P}5$)
is $\mathcal{O}\big{(}(n_{l}+n_{v})n_{v}^{2}n_{l}^{0.5}\big{)}$, where
$n_{l}=K+1$ denotes the number of linear constraints while $n_{v}=K$ is the
number of real valued scalar decision variables.

##  5 Extensions

###  5.1 Multi-target Scenarios

The considered spectrum sharing MIMO radar and communication system has
interesting application scenarios. For instance, identifying abnormal or
suspicious activities in a specific area and sending an alert is one potential
scenario. Therefore, the coexistence of co-located MIMO radar for single-
target detection and communication has been the subject of several research
studies [16, 37]. On the other hand, multi-target detection can be achieved
through the multibeam/beampattern design at the MIMO radar [38]. Accordingly,
new performance metrics such as beampattern gain are used in the literature to
study the performance of such designs.

The primary concept of beampattern design is to determine the covariance
matrix of the radar probing signals through convex optimization problems. Let
${\bf R}_{w}=\frac{1}{L}\sum_{l=1}^{L}{\bf s}_{l}{\bf s}_{l}^{H}$ denote the
covariance matrix of the probing signals, where ${\bf s}_{l}$ is the $l$-th
snapshot across the radar antennas. Three different optimization problems have
been proposed in the literature to determine ${\bf R}_{w}$. More specifically,
the authors in [39] formulated a constrained least-squares problem to approach
an ideal beampattern as

$\displaystyle\min\limits_{\alpha,{{\bf
R}_{w}}}~{}\sum\limits_{m=1}^{M}{{{\left|{{\alpha{\tilde{P}_{d}}\left({{\theta_{m}}}\right)-{{\mathbf{a}}^{H}}\left({{\theta_{m}}}\right){{\bf
R}_{w}{\bf a}}\left({{\theta_{m}}}\right)}}\right|}^{2}}}$ (49a)
$\displaystyle~{}s.t.~{}\mathrm{diag}\left({{{\bf
R}_{w}}}\right)=\frac{{P_{0}{\mathbf{1}_{N}}}}{N},$ (49b)
$\displaystyle\qquad~{}{{\bf R}_{w}}\succeq\mathbf{0},{{\bf R}_{w}}={{{\bf
R}_{w}}^{H}},$ (49c) $\displaystyle\qquad~{}\alpha\geq 0$ (49d)
$\displaystyle\qquad\mathrm{tr}({\bf f}_{i}^{*}{\bf f}_{i}^{T}{\bf
R}_{w})=0,~{}\forall i,~{}$ (49e)

where $\\{\theta_{m}\\}_{m=1}^{M}$ is defined as a fine angular grid that
covers the detection angle range of $[-\frac{\pi}{2},\frac{\pi}{2}]$, ${\bf
a}(\theta_{m})=[1,e^{-j2\pi d\sin(\theta_{m})},\ldots,e^{-j2\pi
d(N-1)\sin(\theta_{m})}]^{T}\in\mathbb{C}^{N\times 1}$ is the steering vector
of the transmit antenna array, ${\tilde{P}_{d}}\left({{\theta_{m}}}\right)$ is
the desired ideal beampattern gain at $\theta_{m}$, $P_{0}$ is the power
budget, $\alpha$ is a scaling factor, and
$\boldsymbol{1}_{N}\in\mathbb{R}^{N\times 1}$. We notice that via constraint
(49e), we force the radar signals to fall into the nullspace of the channel
between the radar antennas and downlink users.

Nevertheless, finding the detection probability of the radar is challenging.
We leave the performance evaluation of the multi-target scenario for future
investigation.

###  5.2 Correlated Fading

In this subsection, we extend our analysis to consider the spatially
correlated Raleigh fading scenario. The correlated Rayleigh channel vector
between the $k$-th user and the BS can be modeled as

${{{\mathbf{g}}}_{k}}\sim{\mathcal{CN}}\left(\boldsymbol{0},\boldsymbol{\Omega}_{k}\right),$
(50)

where $\boldsymbol{\Omega}_{k}\in\mathbb{C}^{M\times M}$ is the transmit
correlation matrix specific to terminal $k$.

Training phase: Based on the observable pilot matrix in (2), the BS correlates
${\mathbf{Y}}_{\mathrm{P}}$ with the pilot sequence
$\boldsymbol{\varphi}_{p,k}$ of user $k$, leading to the processed pilot
sequence

$\displaystyle~{}\breve{{\bf y}}_{k,\mathrm{p}}={\bf
Y}_{\mathrm{\text{p}}}{{\boldsymbol{\varphi}}_{p,k}}=\sqrt{\tau_{\mathrm{p}}{\rho_{u}}}{{{{\mathbf{g}}}_{k}}}+{{{\mathbf{N}}}}{{\boldsymbol{\varphi}}_{p,k}},$
(51)

where $\boldsymbol{\varphi}_{p,k}\in\mathbb{C}^{\tau_{\mathrm{p}}\times 1}$
indicates the pilot signal assigned to the $k$-th user. Based on the
observation $\breve{{\bf y}}_{k,\mathrm{p}}$, the MMSE estimate of the channel
${\bf g}_{k}$ is given by

${{{\hat{{\bf
g}}}}_{k}}=\sqrt{\tau_{\mathrm{p}}{\rho_{u}}}\boldsymbol{\Omega}_{k}{{\boldsymbol{\Xi}}_{k}}\breve{{\bf
y}}_{k,\mathrm{p}},$

where

$\displaystyle{{\boldsymbol{\Xi}}_{k}}={\bigg{(}{{\tau_{\mathrm{p}}\rho_{u}\boldsymbol{\Omega}_{k}+{{{\mathbf{I}}}_{M}}}}\bigg{)}^{-1}}.$
(52)

Therefore, the MMSE estimated channel ${{{\hat{\mathbf{g}}}}_{k}}$ and the
corresponding estimation error ${{{\tilde{\mathbf{g}}}}_{k}}$ are complex
Gaussian random vectors, distributed as follows

$\displaystyle{{{\hat{\mathbf{g}}}}_{k}}\sim$
$\displaystyle{\mathcal{N}_{\mathbb{C}}}\left({{{\boldsymbol{0}},\;{{{\mathbf{Q}}}_{k}}}}\right),$
$\displaystyle{{{\tilde{\mathbf{g}}}}_{k}}\sim$
$\displaystyle{\mathcal{N}_{\mathbb{C}}}\left({{{\boldsymbol{0}},{{{\mathbf{C}}}_{k}}}}\right),$

where ${\bf Q}_{k}=\boldsymbol{\Omega}_{k}-{\bf C}_{k}$ and ${\bf C}_{k}$ is
expressed as

$\displaystyle~{}{{{\mathbf{C}}}_{k}}=\boldsymbol{\Omega}_{k}-\tau_{\mathrm{p}}\rho_{u}\boldsymbol{\Omega}_{k}{{\boldsymbol{\Xi}}_{k}}\boldsymbol{\Omega}_{k}.$
(53)

The following Proposition provides the SE at the $k$th cellular user and
detection probability at the radar over the spatially correlated fading
environments and with MR precoding design at the BS.

We proceed to find closed-form expression for $\mathrm{SINR}_{k}$ with
$\mathrm{MR}$ precoding.

######  Proposition 5.

For the considered spatially correlated Rayleigh channels, the SE of the
$k$-th user achieved by the $\mathrm{MR}$ precoding is given by (54) at the
top of the next page.

$\displaystyle~{}\mathrm{SE}_{k}^{\mathrm{MR},\mathrm{Cor}}=\Big{(}1-\frac{\tau_{\mathrm{p}}}{\tau}\Big{)}\log_{2}\left(1+\frac{{{\eta_{k}}\rho}\tau_{\mathrm{p}}\rho_{u}\mathrm{tr}(\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}\boldsymbol{\Omega}_{k})}{\sum_{k^{\prime}=1}^{K}\eta_{k^{\prime}}\rho\frac{\mathrm{tr}(\boldsymbol{\Omega}_{k^{\prime}}{{\boldsymbol{\Xi}}_{k^{\prime}}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k})}{\mathrm{tr}(\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Xi}_{k^{\prime}}\boldsymbol{\Omega}_{k^{\prime}})}+P_{R}\bar{\beta}_{k}N+\sigma_{C}^{2}}\right),$
(54)

Moreover, the detection probability at the radar is given by (28), where

$\displaystyle\mu^{\mathrm{MR},\mathrm{Cor}}$
$\displaystyle\approx\frac{|\alpha|^{2}LP_{R}\mathrm{tr}({\bf A}{\bf
A}^{H})}{\zeta_{\mathrm{MR}}\sum_{k=1}^{K}\eta_{k}+\sigma_{R}^{2}}.$ (55)

###### Proof.

See Appendix 8.6. ∎

We notice that, in the case of uncorrelated fading channels
$\mathrm{tr}(\boldsymbol{\Omega}_{k}{{\boldsymbol{\Xi}}_{k}}\boldsymbol{\Omega}_{k})=\frac{M\gamma_{k}}{\tau_{\mathrm{p}}\rho_{u}}$
and
$\mathrm{tr}(\boldsymbol{\Omega}_{k^{\prime}}{{\boldsymbol{\Xi}}_{k^{\prime}}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k})=\frac{M\gamma_{k}\gamma_{k^{\prime}}}{\tau_{\mathrm{p}}\rho_{u}}$.
Therefore, $\mathrm{SE}_{k}^{\mathrm{MR},\mathrm{Cor}}$ reduces to
$\mathrm{SE}_{k}^{\mathrm{MR}}$ in (20). Moreover, by comparing (55) and (35),
we observe that when using MR precoding at the BS, the same detection
probability is achieved over both correlated and uncorrelated Rayleigh fading
channels.

######  Remark 1.

Finding closed-form expressions for the achievable SE and detection
probability for $\mathrm{ZF}$ and $\mathrm{PZF}$ precoding designs seems
intractable. However, the corresponding results can be obtained through
numerical simulations.

##  6 Numerical results

In this section, we provide numerical results to verify our analysis and
evaluate the performance of the proposed power control design. We assume that
the MIMO radar system deploys a uniform linear array with inter-antenna
spacing $d=\lambda/2$. The radar SNR is defined as
$\mathtt{SNR}_{R}=\frac{L|\alpha|^{2}P_{R}}{\sigma_{R}^{2}}$ [16]. The target
is set to be located at the direction of $\theta=10^{\circ}$. In the
communication system, we consider large-scale fading with path loss, shadow
fading, and random user locations. To generate the large-scale fading
coefficients, we adopt the model proposed in [30]. For the sake of
understanding, let us visualize a circular cell with a radius of approximately
$15$ kilometres. At the center of the cell, the BS equipped with $M$ antennas
serves $K$ users randomly distributed within the cell. Furthermore, a MIMO
radar is positioned $10$ kilometres away from the BS. We model the large-scale
fading coefficient $\beta$ as [30]

$\beta=\text{PL}_{0}\left({\frac{d}{R_{\text{min}}}}\right)^{\upsilon}\times
10^{\frac{\sigma_{\text{sh}}z}{10}},$ (56)

where $\beta\in\\{\beta_{k},\bar{\beta}_{k},\beta_{br}\\}$,
$d\in\\{d_{k},\bar{d}_{k},d_{br}\\}$ with $d_{k}$ ($\bar{d}_{k}$) being the
distance between the BS (radar) and user $k$, while $d_{br}$ is the distance
between the BS and radar; ${PL}_{0}$ denotes a reference path loss constant
which is chosen to satisfy a given downlink cell-edge SNR; $\upsilon$ and
${\sigma_{\text{sh}}}$ represent the path loss exponent and standard deviation
of the shadow fading, respectively; and $z\sim\mathcal{N}(0,1)$. In our
examples, we chose $\upsilon$ = 3.8, ${\sigma_{\text{sh}}}=8$ dB.

The $(i,j)$ entry in the correlation matrix of terminal $k$ is given by [40]

$\displaystyle\left[{\boldsymbol{\Omega}_{k}}\right]_{i,j}=\frac{1}{2\Delta}\int\nolimits_{-\Delta+\phi_{k}}^{\Delta+\phi_{k}}e^{-j2\pi{}d\left({i-j}\right)\sin\left({\theta_{k}}\right)}d\theta_{k},$
(57)

where $\Delta$ denotes the azimuth angular spread, $\phi_{k}$ is the central
azimuth angle from BS array to the terminal $k$, $\theta_{k}$ is the actual
angle-of-departure and $d\left({i-j}\right)$ captures the inter-element
spacing normalized by the carrier wavelength between $i$-th and $j$-th antenna
elements. Unless explicitly stated, we set $d(1)=0.5$ and assume that
$\phi_{k}$ drawn from a uniform distribution on $0$, $2\pi$, i.e.,
$\phi_{k}\sim\mathcal{U}[0,2\pi]$. The instantaneous value of $\theta_{k}$ is
also drawn from a uniform distribution on $-\frac{\Delta}{2}$,
$\frac{\Delta}{2}$ , i.e.,
$\theta_{k}\sim\mathcal{U}[-\frac{\Delta}{2},\frac{\Delta}{2}]$. As such,
$\Delta$ represents the tota angular spread, naturally bounded from $0$ to
$2\pi$ radians.

Figure 2: Per-user SE versus the number of BS antennas, $M$, ($N=20$,
$\rho=20$ dB, $K=20$, $\Delta=20^{o}$). Figure 3: Sum-SE versus the number of
users, $K$, ($M=200$, $\rho=20$ dB, $N=20$). Figure 4: Per-user SE versus the
number of radar antennas, $N$, ($M=200$, $\rho=20$ dB, $K=20$).

Figure 2 shows the downlink SE of different massive MIMO systems versus the
number of BS antennas $M$, for different precoding schemes
$\mathrm{MR},\mathrm{ZF}$ and $\mathrm{PZF}$. Our analysis has been confirmed
by the simulation results, demonstrating a precise alignment between the
analytical and simulated outcomes. As anticipated, an increase in the number
of BS antennas directly corresponds to SE enhancement. We observe that the
performance gaps between the considered precoding schemes are decreased, when
the number of BS antennas increases. For example, by increasing the number of
BS antennas from $M=200$ to $M=400$, the gap between the $\mathrm{ZF}$
($\mathrm{PZF}$) scheme and $\mathrm{MR}$ is reduced from $85\%$ ($50\%$) to
$72\%$ ($27\%$). This behavior can be explained as follows: By comparing (22)
and (27), we observe that by increasing the number of BS antennas $M$, for a
fixed number of receive antennas at the radar, $N$, we have
$\frac{M-N}{N}\xrightarrow{M\rightarrow\infty}1$. Therefore, the numerators of
the SINRs inside the logarithmic functions are approximately the same for the
$\mathrm{ZF}$ and $\mathrm{PZF}$ schemes. However, the first term in the
denominator of $\mathrm{SINR}^{\mathrm{PZF}}_{k}$ scales as $(M-K)$, whereas
the corresponding term in $\mathrm{SINR}^{\mathrm{ZF}}_{k}$ scales as $1$.
Therefore, by increasing $M$, the reduction in
$\mathrm{SINR}^{\mathrm{PZF}}_{k}$ becomes greater compared to
$\mathrm{SINR}^{\mathrm{ZF}}_{k}$. As a result, the gap between the SE
achieved by $\mathrm{ZF}$ and $\mathrm{PZF}$ increases with higher values of
$M$. Finally, when comparing the results between the correlated and
uncorrelated channels, we find that the same insights apply to correlated
environments.

Figure 5: CDF of the per-user SE for different number of users ($N=20$,
$M=200$, $\rho=20$ dB). Figure 6: Average detection probability versus the
radar SNR ($M=200$, $K=20$, $N=20$, $\mathtt{SNR}_{R}=0$ dB).

Figure 3 shows the downlink sum-SE of massive MIMO system versus the number of
users $K$, for different precoding schemes, i.,e $\mathrm{MR},\mathrm{ZF}$ and
$\mathrm{PZF}$. By increasing the number of users, sum-SE increases and tends
to the maximum value and then starts to reduce due to increasing pilot
overhead. Moreover, we observe that by increasing $K$, the gap between the
$\mathrm{ZF}$ and $\mathrm{MR}$ increases, while the gap between the
$\mathrm{ZF}$ and $\mathrm{PZF}$ remains constant, which is due to the severe
increase of the inter-user interference in the case of $\mathrm{MR}$
processing. These results reveal the necessity of developing pilot reuse among
different users and designing pilot assignment algorithms to manage the pilot
contamination effects.

To demonstrate the impact of radar interference on massive MIMO systems, we
examine the SE as a function of the number of transmit antennas at the radar
in Figure 4. As the number of antennas at the radar increases, the
interference towards the massive MIMO system increases, leading to SE
reduction. For different number of antennas at the radar, $\mathrm{MR}$ yields
the worst performance, while $\mathrm{ZF}$ outperforms all schemes. Notably,
$\mathrm{ZF}$ exhibits a remarkable improvement of 78$\%$ over $\mathrm{MR}$,
while $\mathrm{PZF}$ demonstrates a substantial enhancement of 50$\%$ compared
to $\mathrm{MR}$, when the number of radar antennas is $N=20$. When the number
of radar antennas is increased to $N=80$, $\mathrm{ZF}$ achieves a significant
improvement of up to $50\%$ over $\mathrm{MR}$, while $\mathrm{PZF}$ exhibits
a substantial enhancement of $24\%$ over $\mathrm{MR}$. This further
solidifies the superiority of $\mathrm{ZF}$ and $\mathrm{PZF}$ techniques over
the $\mathrm{MR}$ scheme in the given context.

Figure 5 shows the cumulative distribution of the per-user SE for different
values of $K=5$ and $K=20$, with $M=200$ and $\rho=20$ dB. We can observe that
the per-user SE in the case of $K=5$ is higher than that in the case of
$K=20$. This difference can be attributed to the fact that when there are more
users, inter-user interference becomes more prevalent. Additionally, it is
evident from the figure that $\mathrm{MR}$ suffers more from inter-user
interference compared to $\mathrm{ZF}$ and $\mathrm{PZF}$.

Figure 7: CDF of detection probability ($M=200$, $K=20$, $N=20$). Figure 8:
Detection probability versus the SE threshold ($M=200$, $K=20$,
$\mathtt{SNR}_{R}=0$ dB). Figure 9:  Convergence behavior of the bisection
algorithm to solve the problem $(\mathbb{P}4)$ for different number of BS
antennas ($K=20$, $N=20$). Figure 10: Average detection probability versus the
radar SNR for single and multiple target scenarios ($M=200$, $K=20$, $N=20$,
$\mathtt{SNR}_{R}=0$ dB)

Figure 6 provides a comparative analysis of detection probabilities under two
power allocation schemes: our proposed optimal power allocation (OPA)
solutions and equal power allocation (EPA). It can be observed that the
results obtained using the OPA via the bisection algorithm closely match those
obtained using LP. In the EPA scenario, the radar transmits with maximum power
level, i.e, $P_{R}=P_{R,\mathrm{max}}$ and power control coefficients at the
BS are set as $\eta_{k}=\frac{1}{K}$, $\forall k=1,\ldots,K$. To ensure a fair
comparison, the selection of the SE threshold ($\mathrm{SE_{th}}$) in the
power allocation scheme is based on the EPA scenario. Among the considered
schemes, our proposed $\mathrm{PZF}$ scheme yields the highest performance,
followed by $\mathrm{ZF}$ and $\mathrm{MR}$, respectively. Moreover, our
proposed power allocation scheme yields a remarkable enhancement in the
detection probability. For example, when the radar SNR is $2$ dB and OPA is
applied, the detection probability of $\mathrm{PZF}$, $\mathrm{ZF}$ and
$\mathrm{MR}$ precoding schemes improved from $0.5$, $0.35$, and $0.3$ to
$0.97$, $0.72$, and $0.68$, respectively.

Figure 7 shows the CDF of the detection probability for $\mathrm{PZF}$,
$\mathrm{ZF}$, and $\mathrm{MR}$ precoding designs and for OPA and EPA. It is
evident that our proposed OPA strategy yields a substantial improvement in the
detection probability performance and $\mathrm{PZF}$ design provides the best
performance.

Figure 8 examines the trade-off between the detection probability and SE for
$\mathrm{PZF}$, $\mathrm{ZF}$, and $\mathrm{MR}$ precoding designs and for two
different radar antenna number. For a specific network realization, we first
derive the maximum value for $\mathrm{SE}_{k,\mathrm{th}}$, termed as
$\mathrm{SE}_{k,\mathrm{th}}^{\max}$, according to the EPA. Then, the
optimization problem ($\mathbb{P}4$) is solved for different values of
$\mathrm{SE}_{k,\mathrm{th}}=[0,\mathrm{SE}_{k,\mathrm{th}}^{\max}]$, and
corresponding detection probability is derived. From Fig. 8, we can observe
that for small values of $\mathrm{SE}_{k,\mathrm{th}}$, the radar can transmit
with more power, yielding better detection probability performance. Moreover,
by increasing the number of radar receive antennas the detection probability
is improved, while it degrades the SE of the users in the communication system
(cf Fig. 4).

Figure 9 shows value of $(t_{\max}-t_{\min})$ versus the number of iterations
for one randomly generated set of channel realizations for different number of
BS antennas. It can be observed that the algorithm with different number of BS
antennas has a similar convergence speed and, as a matter of fact, converges
to the optimal value after only a small number of iterations.

Figure 10 presents a comparison of average detection probabilities versus the
radar SNR for both single and multiple target scenarios. In the multiple
target scenario, three targets are located at specific angular directions. The
comparison is conducted for three precoding techniques: $\mathrm{MR}$,
$\mathrm{ZF}$, and $\mathrm{PZF}$. A key observation from the figure is the
enhanced detection probability in single target scenarios compared to multiple
targets, which is likely due to the increased interference challenges in
multi-target environments. Notably, the $\mathrm{PZF}$ scheme emerges as the
most effective technique, achieving the highest detection probability across
various SNR levels. The superiority of $\mathrm{PZF}$ is particularly marked
in multiple target scenarios, underlining its efficiency in interference
mitigation.

######  Remark 2.

To improve the detection probability at the radar, the multiple-target
detection problem can be modeled as a multi-hypothesis testing, followed by
beampattern design at the radar [41, 42]. Exploring the development of a
closed-form expression for the detection probability and subsequently applying
joint power control and beamforming design at the BS represent interesting
future research directions.

##  7 Conclusion

We have studied the coexistence of massive MIMO communication and MIMO radar.
$\mathrm{PZF}$ precoding design has been proposed to achieve favourable SE
performance in the cellular system, while delivering substantial improvements
to the radar system. Closed-form expressions for the SE of users and the
detection probability at the radar have been derived for $\mathrm{MR}$,
$\mathrm{ZF}$, and $\mathrm{PZF}$ precoding designs. Accordingly, power
control at the BS and radar has been developed to manage the inter-system
interference, subject to the SE requirements of the users. The optimization
problem has been efficiently solved, showing substantial improvement in
detection probability, when compared with the equal power allocation scenario.

##  8 Appendix

###  8.1 Useful Result

######  Lemma 1.

For the projection matrix
$\mathbf{B}=\mathbf{I}_{M}-\mathbf{\hat{R}^{H}}\left(\mathbf{\hat{R}}\mathbf{\hat{R}}^{H}\right)^{-1}\mathbf{\hat{R}}$,
we have

$\displaystyle\mathbb{E}\\{{\bf B}\\}=\frac{M-N}{M}{\bf I}_{M},\quad M>N.$
(58)

###### Proof.

We denote ${\bf S}=\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}$ with $\hat{{\bf R}}\in\mathbb{C}^{N\times M}$.
Assuming that $N<M$, by using the SVD technique, ${\bf R}$ can be expressed as

$\displaystyle~{}\hat{{\bf R}}={\bf U}{\bf D}{\bf V}^{H},$ (59)

where ${\bf U}\in\mathbb{C}^{N\times N}$ and ${\bf V}\in\mathbb{C}^{M\times
M}$ are unitary matrices and ${\bf D}=[{\bf
D}_{1},\boldsymbol{0}]\in\mathbb{C}^{N\times M}$ with ${\bf D}_{1}$ is
$N\times N$ nonnegative diagonal matrix. Therefore, by using (59), we can
express ${\bf S}$ as

$\displaystyle~{}{\bf S}$ $\displaystyle={\bf V}{\bf D}^{H}{\bf U}^{H}({\bf
U}{\bf D}{\bf V}^{H}{\bf V}{\bf D}^{H}{\bf U}^{H})^{-1}{\bf U}{\bf D}{\bf
V}^{H}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}{\bf V}{\bf D}^{H}{\bf
U}^{H}({\bf U}{\bf D}^{2}_{1}{\bf U}^{H})^{-1}{\bf U}{\bf D}{\bf V}^{H}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}{\bf V}{\bf D}^{H}{\bf
D}^{-2}_{1}{\bf D}{\bf V}^{H}$ $\displaystyle={\bf V}\begin{bmatrix}{\bf
I}_{N}&\boldsymbol{0}\\\ \boldsymbol{0}&\boldsymbol{0}\end{bmatrix}{\bf
V}^{H},$ (60)

where (a) follows from the fact that ${\bf V}^{H}{\bf V}={\bf I}_{M}$ and (b)
holds since ${\bf U}^{H}{\bf U}={\bf I}_{N}$. Now assuming that ${\bf
V}\triangleq[{\bf V}_{1},{\bf V}_{2}]$ with ${\bf V}_{1}=[{\bf
v}_{1,1},\ldots,{\bf v}_{1,N}]\in\mathbb{C}^{M\times N}$ and ${\bf
V}_{2}\in\mathbb{C}^{M\times(M-N)}$, we can further simplify (8.1) as

$\displaystyle~{}{\bf S}=\sum_{r=1}^{N}{\bf v}_{1,r}{\bf v}_{1,r}^{H}.$ (61)

By using (61), the expectation of ${\bf S}$ can be expressed as

$\displaystyle~{}\mathbb{E}\big{\\{}{\bf
S}\big{\\}}=\sum^{K}_{r=1}\mathbb{E}\big{\\{}{\bf v}_{1,r}.{\bf
v}_{1,r}^{H}\big{\\}}.$ (62)

Noticing that the distribution of ${\bf v}_{1,r}$ is the same as $\frac{{\bf
z}}{\|{\bf z}\|}$, with ${\bf z}\sim\mathcal{CN}\left(\boldsymbol{0},{\bf
I}_{M}\right)$, (62) can be further simplified as

$\displaystyle~{}\mathbb{E}\big{\\{}{\bf
S}\big{\\}}=N\mathbb{E}\bigg{\\{}\frac{{\bf z}{\bf z}^{H}}{{\bf z}^{H}{\bf
z}}\bigg{\\}}.$ (63)

The diagonal elements of the expectation matrix in (63) can be obtained as

$\displaystyle\left[\mathbb{E}\big{\\{}{\bf S}\big{\\}}\right]_{(r,r)}$
$\displaystyle=N\mathbb{E}\\{t\\}$ $\displaystyle=\frac{N}{M},$ (64)

where $t=\frac{|z_{r}|^{2}}{\sum^{M}_{s=1}|z_{s}|^{2}}$ is Beta distributed
random variables with parameter $(1,M-1)$ and $\mathbb{E}\\{t\\}=\frac{1}{M}$
[43].

To find the off-diagonal elements of $\mathbb{E}\big{\\{}{\bf S}\big{\\}}$, we
have [43]

$\displaystyle\left[\mathbb{E}\big{\\{}{\bf S}\big{\\}}\right]_{(r,s)}$
$\displaystyle=K\mathbb{E}\bigg{\\{}\frac{z_{r}z_{s}^{*}}{\sum^{M}_{t=1}|z_{t}|^{2}}\bigg{\\}}$
$\displaystyle=0,\forall r\neq s.$ (65)

To this end, we can conclude that

$\displaystyle\mathbb{E}\big{\\{}{\bf B}\big{\\}}$ $\displaystyle={\bf
I}_{M}-\mathbb{E}\big{\\{}{\bf S}\big{\\}}=\frac{M-N}{M}{\bf I}_{M}.$ (66)

∎

###  8.2 Proof of Proposition 1

To derive the closed-form expression for the
$\mathrm{SINR}^{\mathrm{MR}}_{k}$, we need to compute $\mathrm{DS}_{k}$,
$|\mathrm{BU}_{k}|^{2}$ and $|\mathrm{IUI}_{kk^{\prime}}|^{2}$. By
substituting (19) into the desired signal in (14), we obtain firstly
$\mathrm{DS}_{k}$ as

$\displaystyle~{}\mathrm{DS}_{k}$
$\displaystyle=\sqrt{\rho\eta_{k}}\mathbb{E}\Big{\\{}(\hat{{\bf
g}}_{k}^{T}-\tilde{{\bf g}}_{k}^{T}){{\bf t}}^{\mathrm{MR}}_{k}\Big{\\}}$
$\displaystyle=\sqrt{\frac{\rho\eta_{k}}{M}}\mathbb{E}\left\\{\hat{{\bf
g}}_{k}^{T}{\bf h}_{k}^{*}\right\\}$
$\displaystyle=\sqrt{\frac{\rho\eta_{k}\gamma_{k}}{M}}\mathbb{E}\\{\left\|{{\bf
h}}_{k}\|^{2}\right\\}=\sqrt{M\rho\gamma_{k}\eta_{k}}.$ (67)

In order to compute $|\mathrm{BU}_{k}|^{2}$, we have

$\displaystyle~{}\mathbb{E}\Big{\\{}|\mathrm{BU}_{k}|^{2}\Big{\\}}$
$\displaystyle=\rho\eta_{k}\mathbb{E}\Big{\\{}|{{\bf g}}_{k}^{T}{{\bf
t}}^{\mathrm{MR}}_{k}|^{2}\Big{\\}}-|\mathrm{DS}_{k}|^{2}$
$\displaystyle=\rho\gamma_{k}\eta_{k}(M+1)-M\rho\gamma_{k}\eta_{k}$
$\displaystyle=\rho\gamma_{k}\eta_{k}.$ (68)

We now turn our attention to derive $|\mathrm{IUI}_{kk^{\prime}}|^{2}$, which
is given by

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}\mathrm{IUI}_{kk^{\prime}}\big{|}^{2}\Big{\\}}$
$\displaystyle=\rho\eta_{k^{\prime}}\mathbb{E}\Big{\\{}\big{|}{\bf
g}_{k}^{T}{\bf t}_{k^{\prime}}^{\mathrm{MR}}\big{|}^{2}\Big{\\}}$
$\displaystyle={\rho\eta_{k^{\prime}}}\mathbb{E}\Big{\\{}{\bf
g}_{k}^{T}\mathbb{E}\big{\\{}{\bf t}_{k^{\prime}}^{\mathrm{MR}}({\bf
t}_{k^{\prime}}^{\mathrm{MR}})^{H}\big{\\}}{\bf g}_{k}^{*}\Big{\\}}$
$\displaystyle=\rho\eta_{k^{\prime}}\gamma_{k}.$ (69)

Plugging (8.2), (8.2), and (8.2) into (3.1), we obtain the desired result in
(20).

###  8.3 Proof of Proposition 2

By using ${\bf t}_{k}^{\mathrm{ZF}}$, i.e., the $k$-th column of ${\bf
T}^{\mathrm{ZF}}$ in (21), the desired signal in (14), can be obtained as

$\displaystyle~{}\mathrm{DS}_{k}$
$\displaystyle=\sqrt{\rho\eta_{k}}\mathbb{E}\Big{\\{}(\hat{{\bf
g}}_{k}^{T}-\tilde{{\bf g}}_{k}^{T}){{\bf t}}^{\mathrm{ZF}}_{k}\Big{\\}}$
$\displaystyle=\sqrt{\rho\eta_{k}}\mathbb{E}\left\\{\hat{{\bf g}}_{k}^{T}{\bf
t}_{k}^{\mathrm{ZF}}\right\\}$
$\displaystyle=\sqrt{\rho\eta_{k}\gamma_{k}}\mathbb{E}\left\\{{\bf
h}_{k}^{T}{\bf t}_{k}^{\mathrm{ZF}}\right\\}$
$\displaystyle=\sqrt{(M-K)\rho\eta_{k}\gamma_{k}}.$ (70)

In order to derive $\mathbb{E}\\{|\mathrm{BU}_{k}|^{2}\\}$, which can be
further expressed by

$\displaystyle~{}\mathbb{E}\Big{\\{}|\mathrm{BU}_{k}|^{2}\Big{\\}}$
$\displaystyle={\rho\eta_{k}}\mathbb{E}\Big{\\{}|{{\bf g}}_{k}^{T}{{\bf
t}}^{\mathrm{PZF}}_{k}|^{2}\Big{\\}}-|\mathrm{DS}_{k}|^{2},$ (71)

we need to obtain the first term in (8.3), which can be written as

$\displaystyle~{}\mathbb{E}\big{\\{}|{\bf g}_{k}^{T}{{\bf
t}}^{\mathrm{PZF}}_{k}|^{2}\big{\\}}$ $\displaystyle=\mathbb{E}\Big{\\{}{\bf
g}_{k}^{T}{\bf t}_{k}^{\mathrm{ZF}}({\bf t}^{{\mathrm{ZF}}}_{k})^{H}{\bf
g}_{k}^{*}\Big{\\}}.$ (72)

Denote ${\bf V}^{\mathrm{ZF}}_{k}={\bf t}_{k}^{\mathrm{ZF}}({\bf
t}^{\mathrm{ZF}}_{k})^{H}$, we can express (72) as

$\displaystyle~{}\mathbb{E}\big{\\{}|{\bf g}_{k}^{T}{{\bf
t}}^{\mathrm{ZF}}_{k}|^{2}\big{\\}}$
$\displaystyle=\mathbb{E}\big{\\{}(\hat{{\bf g}}_{k}^{T}-\tilde{{\bf
g}}_{k}^{T}){\bf V}^{\mathrm{ZF}}_{k}(\hat{{\bf g}}_{k}^{*}-\tilde{{\bf
g}}_{k}^{*})\big{\\}}$ $\displaystyle=\mathbb{E}\big{\\{}(\hat{{\bf
g}}_{k}^{T}{\bf V}^{\mathrm{ZF}}_{k}\hat{{\bf g}}_{k}^{*}+\tilde{{\bf
g}}_{k}^{T}{\bf V}^{\mathrm{ZF}}_{k}\tilde{{\bf g}}_{k}^{*})\big{\\}}$
$\displaystyle=\mathbb{E}\big{\\{}\hat{{\bf g}}_{k}^{T}{\bf
V}^{\mathrm{ZF}}_{k}\hat{{\bf
g}}_{k}^{*}\big{\\}}+\mathbb{E}\big{\\{}\tilde{{\bf g}}_{k}^{T}{\bf
V}^{\mathrm{ZF}}_{k}\tilde{{\bf g}}_{k}^{*}\big{\\}}$
$\displaystyle=(M-K)\rho\eta_{k}\gamma_{k}+(\beta_{k}-\gamma_{k}).$ (73)

To this end, by substituting (8.3) into (8.3), we have

$\displaystyle~{}\mathbb{E}\Big{\\{}|\mathrm{BU}_{k}|^{2}\Big{\\}}$
$\displaystyle=\mathbb{E}\Big{\\{}|{{\bf g}}_{k}^{T}{{\bf
t}}^{\mathrm{ZF}}_{k}|^{2}\Big{\\}}-|\mathrm{DS}_{k}|^{2}$
$\displaystyle=\rho\eta_{k}(\beta_{k}-\gamma_{k}).$ (74)

Then, we obtain the inter-user interference term as

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}\mathrm{IUI}_{kk^{\prime}}\big{|}^{2}\Big{\\}}$
$\displaystyle=\rho\eta_{k^{\prime}}\mathbb{E}\Big{\\{}\big{|}{\bf
g}_{k}^{T}{\bf t}_{k^{\prime}}^{\mathrm{ZF}}\big{|}^{2}\Big{\\}}$
$\displaystyle={\rho\eta_{k^{\prime}}}\mathbb{E}\Big{\\{}\tilde{{\bf
g}}_{k}^{T}\mathbb{E}\big{\\{}{\bf t}_{k^{\prime}}^{\mathrm{ZF}}({\bf
t}_{k^{\prime}}^{\mathrm{ZF}})^{H}\big{\\}}\tilde{{\bf g}}_{k}^{*}\Big{\\}}$
$\displaystyle=\rho\eta_{k^{\prime}}(\beta_{k}-\gamma_{k}).$ (75)

Finally, (22) is obtained by plugging (8.3), (72), and (8.3) into (3.1).

###  8.4 Proof of Proposition 3

By using ${\bf t}^{\mathrm{PZF}}$ in (3.1.3), the desired signal in (14), can
be obtained as

$\displaystyle~{}\mathrm{DS}_{k}$
$\displaystyle=\sqrt{\rho\eta_{k}}\mathbb{E}\Big{\\{}(\hat{{\bf
g}}_{k}^{T}-\tilde{{\bf g}}_{k}^{T}){{\bf t}}^{\mathrm{PZF}}_{k}\Big{\\}}$
$\displaystyle=\alpha_{\mathrm{PZF}}\sqrt{\rho\eta_{k}\gamma_{k}}\mathbb{E}\big{\\{}{\bf
h}_{k}^{T}\mathbb{E}\\{{\bf B}\\}{\bf w}^{\mathrm{ZF}}_{k}\big{\\}}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\alpha_{\mathrm{PZF}}\sqrt{\rho\eta_{k}\gamma_{k}}\bigg{(}\frac{M-N}{M}\bigg{)}\mathbb{E}\left\\{{\bf
h}_{k}^{T}{{\bf w}}^{\mathrm{ZF}}_{k}\right\\}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}\sqrt{\frac{\rho\eta_{k}\gamma_{k}(M-K)(M-N)}{M}},$
(76)

where we exploited, in (a) Lemma 1 and in (b) $\mathbb{E}\left\\{{\bf
h}_{k}^{T}{{\bf w}}^{\mathrm{ZF}}_{k}\right\\}=1$.

In order to derive $\mathbb{E}\\{|\mathrm{BU}_{k}|^{2}\\}$, which can be
expressed as

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}\mathrm{BU}_{k}\big{|}^{2}\Big{\\}}$
$\displaystyle={\rho\eta_{k}}\mathbb{E}\Big{\\{}\big{|}{{\bf g}}_{k}^{T}{{\bf
t}}^{\mathrm{PZF}}_{k}\big{|}^{2}\Big{\\}}-|\mathrm{DS}_{k}|^{2},$ (77)

we need to obtain the first term in (77), given by

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{{\bf
t}}^{\mathrm{PZF}}_{k}\big{|}^{2}\Big{\\}}$
$\displaystyle=\mathbb{E}\Big{\\{}{\bf g}_{k}^{T}{\bf B}{\bf
w}_{k}^{\mathrm{ZF}}({\bf w}^{{\mathrm{ZF}}}_{k})^{H}{\bf B}^{H}{\bf
g}_{k}^{*}\Big{\\}}.$ (78)

Denoting ${\bf V}_{k}={\bf w}_{k}^{\mathrm{ZF}}({\bf
w}^{\mathrm{ZF}}_{k})^{H}$, we can express (78) as

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{{\bf
t}}^{\mathrm{PZF}}_{k}\big{|}^{2}\Big{\\}}$
$\displaystyle=\mathbb{E}\left\\{{\bf g}_{k}^{T}{\bf B}{\bf V}_{k}{\bf
B}^{H}{\bf g}_{k}^{*}\right\\}$ $\displaystyle=\mathbb{E}\Big{\\{}{\bf
g}_{k}^{T}\big{(}{\bf I}_{M}-\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}\big{)}{\bf V}_{k}$ $\displaystyle\hskip
10.00002pt\times\big{(}{\bf I}_{M}-\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}\big{)}{\bf g}_{k}^{*}\Big{\\}}$
$\displaystyle=\mathbb{E}\Big{\\{}{\bf g}_{k}^{T}{\bf V}_{k}{\bf
g}_{k}^{*}-2{\bf g}_{k}^{T}{\bf V}_{k}\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}$ $\displaystyle\hskip 10.00002pt\times\hat{{\bf R}}{\bf
g}_{k}^{*}+{\bf g}_{k}^{T}\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}$ $\displaystyle\hskip 10.00002pt\times{\bf
V}_{k}(\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf R}}^{H})^{-1}\hat{{\bf R}}{\bf
g}_{k}^{*}\Big{\\}}.$ (79)

Now, by using the fact that $\hat{{\bf R}}$ is independent of ${\bf g}_{k}$
and ${\bf V}_{k}$, we can derive (8.4) as

$\displaystyle\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{{\bf
t}}^{\mathrm{PZF}}_{k}\big{|}^{2}\Big{\\}}=\mathbb{E}\Big{\\{}{\bf
g}_{k}^{T}{\bf V}_{k}{\bf g}_{k}^{*}\Big{\\}}$ $\displaystyle\hskip
5.0pt-2\mathbb{E}\Big{\\{}{\bf g}_{k}^{T}{\bf
V}_{k}\mathbb{E}\Big{\\{}\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}\Big{\\}}{\bf g}_{k}^{*}\Big{\\}}$
$\displaystyle\hskip 5.0pt+\mathbb{E}\Big{\\{}{\bf
g}_{k}^{T}\mathbb{E}\Big{\\{}\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}{\bf V}_{k}(\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}\big{\\}}{\bf g}_{k}^{*}\Big{\\}}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\\!\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf V}_{k}{\bf
g}_{k}^{*}\big{\\}}\\!-\\!2\frac{N}{M}\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf
V}_{k}{\bf
g}_{k}^{*}\big{\\}}\\!+\\!\bigg{(}\frac{N}{M}\bigg{)}^{2}\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf V}_{k}{\bf g}_{k}^{*}\Big{\\}}$
$\displaystyle=\Big{(}1-\frac{N}{M}\Big{)}^{2}\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf V}_{k}{\bf g}_{k}^{*}\big{\\}},$ (80)

where (a) holds since according to Lemma 1 we have
$\mathbb{E}\left\\{\hat{{\bf R}}^{H}(\hat{{\bf R}}\hat{{\bf
R}}^{H})^{-1}\hat{{\bf R}}\right\\}=\frac{N}{M}{\bf I}_{M}$, while the final
result follows from the fact that $\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf
V}_{k}{\bf g}_{k}^{*}\big{\\}}=1$.

Then, we need to derive $\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf V}_{k}{\bf
g}_{k}^{*}\big{\\}}$, which after vanishing the cross-expectations can be
obtained as

$\displaystyle~{}\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf V}_{k}{\bf
g}_{k}^{*}\big{\\}}$ $\displaystyle=\mathbb{E}\big{\\{}(\hat{{\bf
g}}_{k}^{T}-\tilde{{\bf g}}_{k}^{T}){\bf V}_{k}(\hat{{\bf
g}}_{k}^{*}-\tilde{{\bf g}}_{k}^{*})\big{\\}}$
$\displaystyle=\mathbb{E}\big{\\{}\hat{{\bf g}}_{k}^{T}{\bf V}_{k}\hat{{\bf
g}}_{k}^{*}\big{\\}}+\mathbb{E}\big{\\{}\tilde{{\bf g}}_{k}^{T}{\bf
V}_{k}\tilde{{\bf g}}_{k}^{*}\big{\\}}$
$\displaystyle=1+(\beta_{k}-\gamma_{k}).$ (81)

To this end, by substituting (8.4) into (8.4) and then plugging the result
into (77), we get

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}\mathrm{BU}_{k}\big{|}^{2}\Big{\\}}$
$\displaystyle=\mathbb{E}\Big{\\{}\big{|}{{\bf g}}_{k}^{T}{{\bf
t}}^{\mathrm{PZF}}_{k}\big{|}^{2}\Big{\\}}-|\mathrm{DS}_{k}|^{2}$
$\displaystyle={\rho\eta_{k}}\alpha_{\mathrm{PZF}}^{2}\Big{(}1-\frac{N}{M}\Big{)}^{2}(\beta_{k}-\gamma_{k}).$
(82)

In order to derive IUI in (16), by using similar steps as in (8.4), we have

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}\mathrm{IUI}_{kk^{\prime}}\big{|}^{2}\Big{\\}}$
$\displaystyle={\rho\eta_{k^{\prime}}}\mathbb{E}\Big{\\{}\big{|}{\bf
g}_{k}^{T}{\bf B}{\bf w}_{k^{\prime}}^{\mathrm{ZF}}\big{|}^{2}\Big{\\}}$
$\displaystyle={\rho\eta_{k^{\prime}}}\Big{(}\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf V}_{k^{\prime}}{\bf
g}_{k}^{*}\big{\\}}-2\frac{N}{M}\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf
V}_{k^{\prime}}{\bf g}_{k}^{*}\big{\\}}$ $\displaystyle\hskip
10.00002pt+\bigg{(}\frac{N}{M}\bigg{)}^{2}\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf V}_{k^{\prime}}{\bf g}_{k}^{*}\big{\\}}\Big{)}$
$\displaystyle={\rho\eta_{k^{\prime}}}\Big{(}1-\frac{N}{M}\Big{)}^{2}\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf V}_{k^{\prime}}{\bf g}_{k}^{*}\big{\\}}.$ (83)

Therefore, we need to derive $\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf
V}_{k^{\prime}}{\bf g}_{k}^{*}\big{\\}}$, which, after averaging out the
cross-expectations, it becomes

$\displaystyle\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf V}_{k^{\prime}}{\bf
g}_{k}^{*}\big{\\}}$ $\displaystyle=\mathbb{E}\big{\\{}(\hat{{\bf
g}}_{k}^{T}-\tilde{{\bf g}}_{k}^{T}){\bf V}_{k^{\prime}}(\hat{{\bf
g}}_{k}^{*}-\tilde{{\bf g}}_{k}^{*})\big{\\}}$
$\displaystyle=\mathbb{E}\big{\\{}(\hat{{\bf g}}_{k}^{T}{\bf
V}_{k^{\prime}}\hat{{\bf g}}_{k}^{*}+\tilde{{\bf g}}_{k}^{T}{\bf
V}_{k^{\prime}}\tilde{{\bf g}}_{k}^{*})\big{\\}}$
$\displaystyle=\mathbb{E}\big{\\{}\hat{{\bf g}}_{k}^{T}{\bf
V}_{k^{\prime}}\hat{{\bf g}}_{k}^{*}\big{\\}}+\mathbb{E}\big{\\{}\tilde{{\bf
g}}_{k}^{T}{\bf V}_{k^{\prime}}\tilde{{\bf g}}_{k}^{*}\big{\\}}$
$\displaystyle=(\beta_{k}-\gamma_{k}).$ (84)

Therefore, the IUI in (8.4), can be derived as

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}\mathrm{IUI}_{kk^{\prime}}\big{|}^{2}\Big{\\}}={\rho\eta_{k^{\prime}}}\Big{(}1-\frac{N}{M}\Big{)}^{2}(\beta_{k}-\gamma_{k}).$
(85)

Finally, by substituting (8.4), (8.4), and (85), into (3.1), the desired
result in (27) is obtained.

###  8.5 Proof of Proposition 4

From (24), $\tilde{{\bf T}}^{\mathrm{PZF}}$ can be expressed as

$\displaystyle~{}\tilde{{\bf T}}^{\mathrm{PZF}}$ $\displaystyle={\bf
T}^{\mathrm{PZF}}{\bf D}_{\eta}({\bf T}^{\mathrm{PZF}})^{H}$
$\displaystyle=\frac{M(M-K)}{(M-N)}\mathbf{B}\mathbf{W}^{\mathrm{ZF}}{\bf
D}_{\eta}(\mathbf{W}^{\mathrm{ZF}})^{H}\mathbf{B}^{H}.$ (86)

Therefore, $\mathrm{tr}(\tilde{{\bf T}}^{\mathrm{PZF}})$ can be obtained as

$\displaystyle~{}\mathrm{tr}(\tilde{{\bf T}}^{\mathrm{PZF}})$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\\!\frac{M(M\\!-\\!K)}{(M\\!-\\!N)}\mathrm{tr}(\mathbf{W}^{\mathrm{ZF}}{\bf
D}_{\eta}({\bf W}^{\mathrm{ZF}})^{H}{\bf B}^{H}{\bf B})$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{=}}\frac{M(M-K)}{(M-N)}$
$\displaystyle\times\mathrm{tr}({\bf H}^{*}({{\bf H}}^{T}{{\bf
H}}^{*})^{-1}{\bf D}_{\eta}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}{\bf H}^{T}{\bf
B}),$ (87)

where we have exploited: in (a) $\mathrm{tr}({\bf X}{\bf Y})=\mathrm{tr}({\bf
Y}{\bf X})$, in (b) ${\bf B}^{H}{\bf B}={\bf B}$. To derive
$\mathrm{tr}(\tilde{{\bf T}}^{\mathrm{PZF}})$, let us denote ${\bf C}$ as

$\displaystyle{\bf C}={\bf H}^{*}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}{\bf
D}_{\eta}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}{\bf H}^{T}.$ (88)

Then, by substituting (23) into (8.5), we get

$\displaystyle\mathrm{tr}(\tilde{{\bf T}}^{\mathrm{PZF}})$
$\displaystyle=\frac{M(M-K)}{(M-N)}$
$\displaystyle\times\Big{(}\underbrace{\mathrm{tr}({\bf
C})}_{\text{T}_{1}}-\underbrace{\mathrm{tr}({\bf
C}\mathbf{\hat{R}^{H}}\left(\mathbf{\hat{R}}\mathbf{\hat{R}}^{H}\right)^{-1}\mathbf{\hat{R}})}_{\text{T}_{2}}\Big{)}.$
(89)

The first term $\text{T}_{1}$ can be calculated as

$\displaystyle~{}T_{1}$ $\displaystyle=\mathrm{tr}({\bf H}^{*}({{\bf
H}}^{T}{{\bf H}}^{*})^{-1}{\bf D}_{\eta}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}{\bf
H}^{T})$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\mathrm{tr}({\bf
D}_{\eta}({{\bf H}}^{T}{{\bf H}}^{*})^{-1})$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\approx}}\frac{1}{M}\mathrm{tr}({\bf
D}_{\eta}{\bf I}_{K})$ $\displaystyle=\frac{1}{M}\sum_{k=1}^{K}\eta_{k},$ (90)

where we have exploited: in (a) $\mathrm{tr}({\bf X}{\bf Y})=\mathrm{tr}({\bf
Y}{\bf X})$, in (b) $({{\bf H}}^{T}{{\bf H}}^{*})^{-1}\approx\frac{1}{M}{\bf
I}_{K}$.

Before proceeding to derive $T_{2}$, we notice that

$\displaystyle{\bf C}$ $\displaystyle={\bf H}^{*}({{\bf H}}^{T}{{\bf
H}}^{*})^{-1}{\bf D}_{\eta}({{\bf H}}^{T}{{\bf H}}^{*})^{-1}{\bf H}^{T}$
$\displaystyle\approx{\bf H}^{*}(M{\bf I}_{K})^{-1}{\bf D}_{\eta}(M{\bf
I}_{K})^{-1}{\bf H}^{T}$ $\displaystyle=\frac{1}{M^{2}}{\bf H}^{*}{\bf
D}_{\eta}{\bf H}^{T}.$ (91)

Therefore, we have

$\displaystyle T_{2}$ $\displaystyle=\mathrm{tr}\big{(}{\bf C}{\hat{{\bf
R}}}^{H}\left(\mathbf{\hat{R}}\mathbf{\hat{R}^{H}}\right)^{-1}\mathbf{\hat{R}}\big{)}$
$\displaystyle\approx\frac{1}{M^{2}}\mathrm{tr}({\bf D}_{\eta}{\bf
H}^{T}{\hat{{\bf
R}}}^{H}\left(\mathbf{\hat{R}}\mathbf{\hat{R}^{H}}\right)^{-1}\mathbf{\hat{R}}{\bf
H}^{*}).$ (92)

Accordingly, by using the law of large number when $M\to\infty$, we get

$\displaystyle\frac{{\bf H}^{T}{\hat{{\bf
R}}}^{H}\left(\mathbf{\hat{R}}\mathbf{\hat{R}^{H}}\right)^{-1}\mathbf{\hat{R}}{\bf
H}^{*}}{M}$ $\displaystyle-\underbrace{\frac{\mathrm{tr}({\hat{{\bf
R}}}^{H}\left(\mathbf{\hat{R}}\mathbf{\hat{R}^{H}}\right)^{-1}\mathbf{\hat{R}})}{M}}_{=N/M}{\bf
I}_{K}$ $\displaystyle\stackrel{{\scriptstyle M\to\infty}}{{\rightarrow}}0.$
(93)

Finally, by substituting (8.5) into (8.5), $T_{2}$ can be approximated as

$\displaystyle~{}T_{2}$ $\displaystyle\approx\frac{N}{M^{2}}\mathrm{tr}({\bf
D}_{\eta}\mathbf{)}$ $\displaystyle=\frac{N}{M^{2}}\sum_{k=1}^{K}\eta_{k}.$
(94)

To this end, by substituting (8.5) and (8.5) into (8.5) and then plugging the
results into (3.2), the desired result in (39) is obtained.

$\displaystyle~{}\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{\bf
t}^{\mathrm{MR}}_{k^{\prime}}\big{|}^{2}\Big{\\}}$
$\displaystyle=\begin{cases}\frac{\tau_{\mathrm{p}}\rho_{u}}{\mathrm{tr}({\bf
Q}_{k^{\prime}})}\Big{(}{\tau_{\mathrm{p}}{\rho_{\mathrm{\text{p}}}}}\Big{|}\mathrm{tr}\big{(}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k}{\boldsymbol{\Xi}_{k}}\big{)}\Big{|}^{2}+\mathrm{tr}\big{(}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}\boldsymbol{\Omega}_{k}\big{)}\Big{)}&k^{\prime}=k\\\
\frac{\tau_{\mathrm{p}}\rho_{u}}{\mathrm{tr}({\bf
Q}_{k^{\prime}})}\mathrm{tr}(\boldsymbol{\Omega}_{k^{\prime}}{{\boldsymbol{\Xi}}_{k^{\prime}}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k})&k^{\prime}\neq
k\end{cases}$ (103)

###  8.6 Proof of Proposition 5

An achievable downlink SE at the $k$-th user can be obtained using (17) where
the effective SINR is given by

$\displaystyle\mathrm{SINR}_{k}=$
$\displaystyle\\!\frac{{{\rho\eta_{k}}}\big{|}\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf
t}_{k}\big{\\}}\big{|}^{2}}{\sum_{k^{\prime}=1}^{K}\\!\\!{{\rho\eta_{k^{\prime}}}}\mathbb{E}\Big{\\{}\big{|}{\bf
g}_{k}^{T}{\bf
t}_{k^{\prime}}\big{|}^{2}\Big{\\}}\\!-\\!{\rho\eta_{k}}\Big{|}\mathbb{E}\big{\\{}{\bf
g}_{k}^{T}{\bf
t}_{k}\big{\\}}\Big{|}^{2}\\!+\\!P_{R}\bar{\beta}_{k}N+\sigma_{C}^{2}}.$ (95)

Noticing that the MR precoding is given ${\bf
t}^{\mathrm{MR}}_{k}=\frac{\hat{{\bf
g}}_{k}^{*}}{\sqrt{\mathbb{E}\\{\|\hat{{\bf g}}_{k}\|^{2}}\\}}$, where
$\mathbb{E}\\{\|\hat{{\bf g}}_{k}\|^{2}\\}=\mathrm{tr}({\bf Q}_{k})$, we have

$\displaystyle~{}\mathbb{E}\big{\\{}{\bf g}_{k}^{T}{\bf
t}^{\mathrm{MR}}_{k}\big{\\}}$ $\displaystyle=\mathbb{E}\Big{\\{}(\hat{{\bf
g}}_{k}^{T}+\tilde{{\bf g}}_{k}^{T}){{\bf t}}^{\mathrm{MR}}_{k}\Big{\\}}$
$\displaystyle=\frac{1}{\sqrt{\mathrm{tr}({\bf
Q}_{k})}}\mathbb{E}\Big{\\{}\|\hat{{\bf g}}_{k}\|^{2}\Big{\\}}$
$\displaystyle=\sqrt{\mathrm{tr}({\bf
Q}_{k})}=\sqrt{\tau_{\mathrm{p}}\rho_{u}\mathrm{tr}(\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}\boldsymbol{\Omega}_{k})}.$
(96)

To derive $\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{\bf
t}^{\mathrm{MR}}_{k^{\prime}}\big{|}^{2}\Big{\\}}$, we consider two different
cases as follows

  * •

When $k^{\prime}\neq k$:

$\displaystyle\mathbb{E}\big{\\{}|{{\bf g}}_{k}^{T}\hat{{\bf
g}}_{k^{\prime}}^{*}|^{2}\big{\\}}$ $\displaystyle=\mathbb{E}\\{{{\bf
g}}_{k}^{T}\hat{{\bf g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}{{\bf
g}}_{k}^{*}\\}$ $\displaystyle=\mathrm{tr}\big{(}\mathbb{E}\\{\hat{{\bf
g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}{{\bf g}}_{k}^{*}{{\bf
g}}_{k}^{T}\\}\big{)}$ $\displaystyle=\mathrm{tr}\big{(}\mathbb{E}\\{\hat{{\bf
g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}\\}\mathbb{E}\\{{{\bf
g}}_{k}^{*}{{\bf g}}_{k}^{T}\\}\big{)}$
$\displaystyle=\tau_{\mathrm{p}}\rho_{u}\mathrm{tr}(\boldsymbol{\Omega}_{k^{\prime}}{{\boldsymbol{\Xi}}_{k^{\prime}}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k}).$
(97)

  * •

When $k^{\prime}=k$: We notice that according to (51) and (52), we have
$\breve{{\bf y}}_{k,\mathrm{p}}=\breve{{\bf y}}_{k^{\prime},\mathrm{p}}$ and
$\boldsymbol{\Xi}_{k}=\boldsymbol{\Xi}_{k^{\prime}}$, respectively. We first
rewrite $\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{\bf
t}^{\mathrm{MR}}_{k^{\prime}}\big{|}^{2}\Big{\\}}$ as

$\displaystyle\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{\bf
t}^{\mathrm{MR}}_{k^{\prime}}\big{|}^{2}\Big{\\}}=\frac{1}{\mathrm{tr}({\bf
Q}_{k^{\prime}})}\mathbb{E}\big{\\{}|{{\bf g}}_{k}^{T}\hat{{\bf
g}}_{k^{\prime}}^{*}|^{2}\big{\\}}$ $\displaystyle=\frac{1}{\mathrm{tr}({\bf
Q}_{k^{\prime}})}\mathbb{E}\big{\\{}(\hat{{\bf g}}_{k}^{T}\hat{{\bf
g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}\hat{{\bf
g}}_{k}^{*}+\tilde{{\bf g}}_{k}^{T}\hat{{\bf g}}_{k^{\prime}}^{*}\hat{{\bf
g}}_{k^{\prime}}^{T}\tilde{{\bf g}}_{k}^{*})\big{\\}},$ (98)

where the cross-expectations are vanished as the channel estimation error is
zero-mean vector. Now, we derive the first term in (• ‣ 8.6), which can be
expressed as

$\displaystyle~{}\mathbb{E}\big{\\{}\hat{{\bf g}}_{k}^{T}\hat{{\bf
g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}\hat{{\bf
g}}_{k}^{*}\big{\\}}$
$\displaystyle={\tau_{\mathrm{p}}^{2}{\rho^{2}_{\mathrm{\text{p}}}}}\mathbb{E}\Big{\\{}\Big{|}\breve{{\bf
y}}_{k,\mathrm{p}}^{T}{\boldsymbol{\Xi}_{k}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}\breve{{\bf
y}}_{k,\mathrm{p}}^{*}\Big{|}^{2}\Big{\\}}.$ (99)

Before proceeding, we present the following results [44]

$\displaystyle~{}\mathbb{E}\left\\{{{{{\left|{\bf u}^{H}{\bf B}{\bf
u}\right|}^{2}}}}\right\\}=|{\mathop{\text{tr}}\nolimits}({{{\bf
B}\boldsymbol{\Psi}}}){|^{2}}+{\mathop{\text{tr}}\nolimits}\left({{{{\mathbf{B}}\boldsymbol{\Psi}}{{{\mathbf{B}}}^{H}}{{\boldsymbol{\Psi}}}}}\right),$
(100)

where ${\bf u}\sim\mathcal{CN}(\boldsymbol{0},\boldsymbol{\Psi})$ with
covariance matrix $\boldsymbol{\Psi}\in\mathbb{C}^{M\times M}$ and any
diagonalizable matrix ${\bf B}\in\mathbb{C}^{M\times M}$.

By exploiting (100) and noticing that $\breve{{\bf
y}}_{k,\mathrm{p}}\sim\mathcal{CN}(\boldsymbol{0},({\boldsymbol{\Xi}_{k}})^{-1})$,
we can derive (99) as

$\displaystyle~{}\mathbb{E}\big{\\{}\hat{{\bf g}}_{k}^{T}\hat{{\bf
g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}\hat{{\bf
g}}_{k}^{*}\big{\\}}$
$\displaystyle={\tau_{\mathrm{p}}^{2}{\rho^{2}_{\mathrm{\text{p}}}}}\Big{(}\Big{|}\mathrm{tr}\big{(}{\boldsymbol{\Xi}_{k}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}({\boldsymbol{\Xi}_{k}})^{-1}\big{)}\Big{|}^{2}$
$\displaystyle+\mathrm{tr}\big{(}{\boldsymbol{\Xi}_{k}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}({\boldsymbol{\Xi}_{k}})^{-1}{\boldsymbol{\Xi}_{k}}\boldsymbol{\Omega}_{k}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Xi}_{k}({\boldsymbol{\Xi}_{k}})^{-1}\big{)}\Big{)}$
$\displaystyle={\tau_{\mathrm{p}}^{2}{\rho^{2}_{\mathrm{\text{p}}}}}\Big{|}\mathrm{tr}\big{(}{\boldsymbol{\Xi}_{k}}\boldsymbol{\Omega}_{k^{\prime}}\boldsymbol{\Omega}_{k}\big{)}\Big{|}^{2}$
$\displaystyle+{\tau_{\mathrm{p}}{\rho_{\mathrm{\text{p}}}}}\mathrm{tr}\big{(}(\boldsymbol{\Omega}_{k^{\prime}}-{\bf
C}_{k^{\prime}})\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}\boldsymbol{\Omega}_{k}\big{)},$
(101)

where in the last step we used the fact that $\mathrm{tr}({\bf A}{\bf
B})=\mathrm{tr}({\bf B}{\bf A})$ and applied (53). We now turn our attention
to the second term in (• ‣ 8.6), which can be expressed as

$\displaystyle~{}\mathbb{E}\big{\\{}\tilde{{\bf g}}_{k}^{T}\hat{{\bf
g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}\tilde{{\bf
g}}_{k}^{*}\big{\\}}$
$\displaystyle=\mathrm{tr}\big{(}\mathbb{E}\big{\\{}\tilde{{\bf
g}}_{k}^{*}\tilde{{\bf g}}_{k}^{T}\\}\mathbb{E}\\{\hat{{\bf
g}}_{k^{\prime}}^{*}\hat{{\bf g}}_{k^{\prime}}^{T}\big{\\}}\big{)}$
$\displaystyle={\tau_{\mathrm{p}}{\rho_{\mathrm{\text{p}}}}}\mathrm{tr}({\bf
C}_{k^{\prime}}\boldsymbol{\Omega}_{k}\boldsymbol{\Xi}_{k}\boldsymbol{\Omega}_{k}).$
(102)

To this end, by substituting (• ‣ 8.6) and (• ‣ 8.6) into (• ‣ 8.6), we obtain
$\mathbb{E}\Big{\\{}\big{|}{\bf g}_{k}^{T}{\bf
t}^{\mathrm{MR}}_{k^{\prime}}\big{|}^{2}\Big{\\}}$ as (103) at the top of the
page. Finally, by substituting (8.6) and (103) into (8.6) and then plugging
the result into (17), the desired result in (54) is obtained.

To derive the detection probability over spatially correlated fading channels,
according to (3.2), we need to obtain $\mathrm{tr}(\tilde{{\bf
T}}^{\mathrm{MR}})$, which can be expressed as

$\displaystyle~{}\mathrm{tr}(\tilde{{\bf T}}^{\mathrm{MR}})$
$\displaystyle=\sum_{k=1}^{K}\frac{\eta_{k}}{\mathrm{tr}({\bf
Q}_{k})}\mathrm{tr}({\hat{{\bf g}}_{k}^{*}}{\hat{{\bf g}}_{k}^{T}})$
$\displaystyle=\sum_{k=1}^{K}\frac{\eta_{k}}{\mathrm{tr}({\bf
Q}_{k})}\|\hat{{\bf g}}_{k}\|^{2}.$ (104)

We notice that $\hat{{\bf g}}_{k}\sim\mathcal{CN}\left(\boldsymbol{0},\;{\bf
Q}_{k}\right)$ can be represented as

$\displaystyle\hat{{\bf g}}_{k}={\bf Q}_{k}^{\frac{1}{2}}{\bf z}_{k},$ (105)

where ${{\bf z}}_{k}\sim\mathcal{CN}\left(\boldsymbol{0},\;{\bf
I}_{M}\right)$. Therefore, $\|\hat{{\bf g}}_{k}\|^{2}$ in (8.6) can be
expressed as

$\displaystyle~{}\|\hat{{\bf g}}_{k}\|^{2}={\bf z}_{k}^{H}{\bf Q}_{k}{\bf
z}_{k}.$ (106)

By substituting (106) into (8.6) and then applying the trace lemma, i.e.,
${\bf z}_{k}^{H}{\bf Q}_{k}{\bf z}_{k}\approx\mathrm{tr}({\bf Q}_{k})$ as $M$
is large, we get

$\displaystyle~{}\mathrm{tr}(\tilde{{\bf T}}^{\mathrm{MR}})$
$\displaystyle\approx\sum_{k=1}^{K}\eta_{k}.$ (107)

To this end, by substituting (107) to (3.2), the non-centrality parameter for
MR processing and over spatially correlated fading channels is obtained as
(55).

## References

  * [1] L. Zheng, M. Lops, Y. C. Eldar, and X. Wang, “Radar and communication coexistence: An overview: A review of recent methods,” _IEEE Signal Process. Mag._ , vol. 36, no. 5, pp. 85–99, Sept. 2019.
  * [2] A. R. Chiriyath, B. Paul, G. M. Jacyna, and D. W. Bliss, “Inner bounds on performance of radar and communications co-existence,” _IEEE Trans. Signal Process._ , vol. 64, no. 2, pp. 464–474, Jan. 2016.
  * [3] F. Liu, C. Masouros, A. Petropulu, H. Griffiths, and L. Hanzo, “Joint radar and communication design: Applications, state-of-the-art, and the road ahead,” _IEEE Trans. Commun._ , vol. 68, no. 6, pp. 3834–3862, Jun. 2020\.
  * [4] A. R. Chiriyath, B. Paul, and D. W. Bliss, “Radar-communications convergence: Coexistence, cooperation, and co-design,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 3, no. 1, pp. 1–12, Mar. 2017.
  * [5] R. Saruthirathanaworakun, J. M. Peha, and L. M. Correia, “Opportunistic sharing between rotating radar and cellular,” _IEEE J. Sel. Areas Commun._ , vol. 30, no. 10, pp. 1900–1910, Nov. 2012.
  * [6] H. Deng and B. Himed, “Interference mitigation processing for spectrum-sharing between radar and wireless communications systems,” _IEEE Trans. Aerosp. Electron. Syst._ , vol. 49, no. 3, pp. 1911–1919, Jul. 2013.
  * [7] A. Khawar, A. Abdel-Hadi, and T. C. Clancy, “Spectrum sharing between S-band radar and LTE cellular system: A spatial approach,” in _Proc. IEEE DYSPAN_ , Apr. 2014, pp. 7–14.
  * [8] A. Aubry, A. De Maio, M. Piezzo, and A. Farina, “Radar waveform design in a spectrally crowded environment via nonconvex quadratic optimization,” _IEEE Trans. Aerosp. Electron. Syst._ , vol. 50, no. 2, pp. 1138–1152, Apr. 2014.
  * [9] L. Zheng, M. Lops, X. Wang, and E. Grossi, “Joint design of overlaid communication systems and pulsed radars,” _IEEE Trans. Signal Process._ , vol. 66, no. 1, pp. 139–154, Jan. 2017.
  * [10] M. Matthaiou, O. Yurduseven, H. Q. Ngo, D. Morales-Jimenez, S. L. Cotton, and V. F. Fusco, “The road to 6G: Ten physical layer challenges for communications engineers,” _IEEE Commun. Mag._ , vol. 59, no. 1, pp. 64–69, Jan. 2021.
  * [11] J. Zhang, E. Björnson, M. Matthaiou, D. W. K. Ng, H. Yang, and D. J. Love, “Prospective multiple antenna technologies for beyond 5G,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1637–1660, Aug. 2020.
  * [12] J. Li and P. Stoica, “MIMO radar with colocated antennas,” _IEEE Signal Process. Mag._ , vol. 24, no. 5, pp. 106–114, Sept. 2007.
  * [13] T. L. Marzetta, _Fundamentals of Massive MIMO_. Cambridge University Press, 2016.
  * [14] J. A. Mahal, A. Khawar, A. Abdelhadi, and T. C. Clancy, “Spectral coexistence of MIMO radar and MIMO cellular system,” _IEEE Trans. Aerosp. Electron. Syst._ , vol. 53, no. 2, pp. 655–668, Apr. 2017.
  * [15] S. Biswas, K. Singh, O. Taghizadeh, and T. Ratnarajah, “Coexistence of MIMO radar and FD MIMO cellular systems with QoS considerations,” _IEEE Trans. Wireless Commun._ , vol. 17, no. 11, pp. 7281–7294, Nov. 2018\.
  * [16] F. Liu, C. Masouros, A. Li, and T. Ratnarajah, “Robust MIMO beamforming for cellular and radar coexistence,” _IEEE Wireless Commun. Lett._ , vol. 6, no. 3, pp. 374–377, Apr. 2017.
  * [17] J. Qian, M. Lops, L. Zheng, X. Wang, and Z. He, “Joint system design for coexistence of MIMO radar and MIMO communication,” _IEEE Trans. Signal Process._ , vol. 66, no. 13, pp. 3504–3519, Jul. 2018.
  * [18] F. Liu, C. Masouros, A. Li, T. Ratnarajah, and J. Zhou, “MIMO radar and cellular coexistence: A power-efficient approach enabled by interference exploitation,” _IEEE Trans. Signal Process._ , vol. 66, no. 14, pp. 3681–3695, Jul. 2018.
  * [19] Z. Pu, W. Wang, and F. Dong, “Beamforming and waveform designing for spectrum coexistence system based on constructive interference,” _IEEE Commun. Lett._ , vol. 26, no. 11, pp. 2695–2699, Nov. 2022.
  * [20] A. Mishra and R. Chopra, “MIMO radars and massive MIMO communication systems can coexist,” _arXiv preprint arXiv:2304.00890_ , 2023.
  * [21] M. Elfiatoure, H. Q. Ngo, and M. Matthaiou, “Coexistence between massive MIMO and radar communications: Performance analysis,” _J. Commun. Inf. Networks_ , vol. 8, no. 1, pp. 37–47, Mar. 2023.
  * [22] M. K. Simon and M.-S. Alouini, _Digital Communication over Fading Channels_ , 2nd ed. New York: Wiley, 2001\.
  * [23] S. M. Kay, _Fundamentals of Statistical Signal Processing: Estimation Theory_. Upper Saddle River, NJ, USA: Prentice-Hall, Inc., 1993.
  * [24] A. M. Ahmed, A. A. Ahmad, S. Fortunati, A. Sezgin, M. S. Greco, and F. Gini, “A reinforcement learning based approach for multitarget detection in massive MIMO radar,” _IEEE Trans. Aerosp. Electron. Syst_ , vol. 57, no. 5, pp. 2622–2636, Oct. 2021.
  * [25] T. L. Marzetta, “Noncooperative cellular wireless with unlimited numbers of base station antennas,” _IEEE Trans. Wireless Commun._ , vol. 9, no. 11, pp. 3590–3600, Nov. 2010.
  * [26] H. Q. Ngo, E. G. Larsson, and T. L. Marzetta, “Energy and spectral efficiency of very large multiuser MIMO systems,” _IEEE Trans. Commun._ , vol. 61, no. 4, pp. 1436–1449, Apr. 2013.
  * [27] A. Khansefid and H. Minn, “Achievable downlink rates of MRC and ZF precoders in massive MIMO with uplink and downlink pilot contamination,” _IEEE Trans. Commun._ , vol. 63, no. 12, pp. 4849–4864, Dec. 2015.
  * [28] H. Q. Ngo, A. Ashikhmin, H. Yang, E. G. Larsson, and T. L. Marzetta, “Cell-free massive MIMO versus small cells,” _IEEE Trans. Wireless Commun._ , vol. 16, no. 3, pp. 1834–1850, Mar. 2017.
  * [29] J. A. Sutton, H. Q. Ngo, and M. Matthaiou, “Hardening the channels by precoder design in massive MIMO with multiple-antenna users,” _IEEE Trans. Veh. Technol._ , vol. 70, no. 5, pp. 4541–4556, May 2021.
  * [30] H. Q. Ngo and E. G. Larsson, “No downlink pilots are needed in TDD massive MIMO,” _IEEE Trans. Wireless Commun._ , vol. 16, no. 5, pp. 2921–2935, May 2017.
  * [31] A. M. Tulino and Verdú, “ _Random matrix theory and wireless communications_ ,” _Foundations and Trends in Communications and Information Theory_ , vol. 1, no. 1, pp. 1–182, 2004.
  * [32] J. Tang and S. Lambotharan, “Interference alignment techniques for MIMO multi-cell interfering broadcast channels,” _IEEE Trans. Commun._ , vol. 61, no. 1, pp. 164–175, Jan. 2013.
  * [33] G. Interdonato, M. Karlsson, E. Björnson, and E. G. Larsson, “Local partial zero-forcing precoding for cell-free massive MIMO,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 7, pp. 4758–4774, Jul. 2020.
  * [34] S. Wagner, R. Couillet, M. Debbah, and D. T. M. Slock, “Large system analysis of linear precoding in correlated MISO broadcast channels under limited feedback,” _IEEE Trans. Inf. Theory_ , vol. 58, no. 7, pp. 4509–4537, Jul. 2012.
  * [35] S. Boyd, S. P. Boyd, and L. Vandenberghe, _Convex Optimization_. Cambridge Univ. Press, 2004.
  * [36] H. H. M. Tam, H. D. Tuan, D. T. Ngo, T. Q. Duong, and H. V. Poor, “Joint load balancing and interference management for small-cell heterogeneous networks with limited backhaul capacity,” _IEEE Trans. Wireless Commun._ , vol. 16, no. 2, pp. 872–884, Feb. 2017.
  * [37] S. Fortunati, L. Sanguinetti, F. Gini, M. S. Greco, and B. Himed, “Massive MIMO radar for target detection,” _IEEE Trans. Signal Process._ , vol. 68, pp. 859–871, Jan. 2020.
  * [38] A. Liu, Z. Huang, M. Li, Y. Wan, W. Li, T. X. Han, C. Liu, R. Du, D. K. P. Tan, J. Lu, Y. Shen, F. Colone, and K. Chetty, “A survey on fundamental limits of integrated sensing and communication,” _IEEE Commun. Surveys Tuts._ , vol. 24, no. 2, pp. 994–1034, Secondquarter 2022.
  * [39] F. Liu, C. Masouros, A. Li, H. Sun, and L. Hanzo, “MU-MIMO communications with MIMO radar: From co-existence to joint transmission,” _IEEE Trans. Wireless Commun._ , vol. 17, no. 4, pp. 2755–2770, 2018.
  * [40] H. Tataria, P. J. Smith, L. J. Greenstein, P. A. Dmochowski, and M. Matthaiou, “Impact of line-of-sight and unequal spatial correlation on uplink MU-MIMO systems,” _IEEE Wireless Commun. Lett._ , vol. 6, no. 5, pp. 634–637, Oct. 2017.
  * [41] L. Wang, W. Zhu, Y. Zhang, Q. Liao, and J. Tang, “Multi-target detection and adaptive waveform design for cognitive MIMO radar,” _IEEE Sensors J._ , vol. 18, no. 24, pp. 9962–9970, Dec. 2018.
  * [42] W. Yi, T. Zhou, Y. Ai, and R. S. Blum, “Suboptimal low complexity joint multi-target detection and localization for non-coherent MIMO radar with widely separated antennas,” _IEEE Trans. Signal Process._ , vol. 68, pp. 901–916, 2020.
  * [43] P. Z. Peebles Jr, _Probability, Random Variables, and Random Signal Principles_. McGraw-Hill, 2001\.
  * [44] E. Björnson, M. Matthaiou, and M. Debbah, “Massive MIMO with non-ideal arbitrary arrays: Hardware scaling laws and circuit-aware design,” _IEEE Trans. Wireless Commun._ , vol. 14, no. 8, pp. 4353–4368, Aug. 2015\.

[] Mohamed Elfiatoure received his B.S. Degree in Electrical and Electronic
Engineering from the High Institute Comprehensive Al-Shati, Libya, in 2008,
and his M.S. Degree in Electrical and Electronic Engineering from Sheffield
Hallam University, U.K., in 2017. He is currently pursuing his Ph.D. at the
Centre for Wireless Innovation, Queen’s University Belfast, U.K. His research
focuses on the coexistence of massive MIMO and MIMO radar, with particular
emphasis on multiple-antenna users and precoding techniques.

[] Mohammadali Mohammadi (M’15, SM’23) received the B.Sc. degree in electrical
engineering from the Isfahan University of Technology, Isfahan, Iran, the
M.Sc. and Ph.D. degree in wireless communication engineering from K. N. Toosi
University of Technology, Tehran, Iran in 2007 and 2012, respectively. From
November 2010 to November 2011, he was a visiting researcher in the Research
School of Engineering, the Australian National University, Australia. In 2013,
he joined the Shahrekord University, Iran, as an Assistant Professor, and was
promoted to Associate Professor before joining Queen’s University Belfast in
2021 as a Research Fellow. His research interests span diverse areas, such as
full-duplex communications, wireless power transfer, OTFS modulation,
reconfigurable intelligent surface, and cell-free massive MIMO. He has
published more than 70 research papers in accredited international peer
reviewed journals and conferences in the area of wireless communication. He
has co-authored two book chapters, “Full-Duplex Non-orthogonal Multiple Access
Systems”, invited chapter in Full-Duplex Communication for Future Networks,
Springer-Verlag, 2020 and “Full-Duplex wireless-powered communications”,
invited chapter in Wireless Information and Power Transfer: A New Green
Communications Paradigm, Springer-Verlag, 2017. He was a recipient of the
Exemplary Reviewer Award for IEEE Transactions on Communications, in 2020 and
2022 and IEEE Communications Letters in 2023. He has actively served as the
Technical Program Committee member of a variety of conferences, such as ICC,
VTC, and GLOBECOM.

[] Hien Quoc Ngo (Senior Member, IEEE) received the B.S. degree in electrical
engineering from the Ho Chi Minh City University of Technology, Vietnam, in
2007, the M.S. degree in electronics and radio engineering from Kyung Hee
University, South Korea, in 2010, and the Ph.D. degree in communication
systems from Linköping University (LiU), Sweden, in 2015. In 2014, he visited
the Nokia Bell Labs, Murray Hill, New Jersey, USA. From January 2016 to April
2017, Hien Quoc Ngo was a VR researcher at the Department of Electrical
Engineering (ISY), LiU. He was also a Visiting Research Fellow at the School
of Electronics, Electrical Engineering and Computer Science, Queen’s
University Belfast, UK, funded by the Swedish Research Council.

Hien Quoc Ngo is currently a Reader (Associate Professor) at Queen’s
University Belfast, UK. His main research interests include massive MIMO
systems, cell-free massive MIMO, physical layer security, and cooperative
communications. He has co-authored many research papers in wireless
communications and co-authored the Cambridge University Press textbook
_Fundamentals of Massive MIMO_ (2016).

Dr. Hien Quoc Ngo received the IEEE ComSoc Stephen O. Rice Prize in
Communications Theory in 2015, the IEEE ComSoc Leonard G. Abraham Prize in
2017, and the Best PhD Award from EURASIP in 2018. He also received the IEEE
Sweden VT-COM-IT Joint Chapter Best Student Journal Paper Award in 2015. He
was awarded the UKRI Future Leaders Fellowship in 2019. Dr. Hien Quoc Ngo
currently serves as an Editor for the IEEE Transactions on Communications, the
IEEE Transactions on Wireless Communications, the IEEE Wireless Communications
Letters, the Digital Signal Processing, and the Elsevier Physical
Communication. He was a Guest Editor of IET Communications, special issue on
“Recent Advances on 5G Communications” and a Guest Editor of IEEE Access,
special issue on “Modelling, Analysis, and Design of 5G Ultra-Dense Networks”,
in 2017. He has been a member of Technical Program Committees for many IEEE
conferences such as ICC, GLOBECOM, WCNC, and VTC.

[] Peter Smith (M’93–SM’01-F’15) received the B.Sc degree in Mathematics and
the Ph.D degree in Statistics from the University of London, London, U.K., in
1983 and 1988, respectively. From 1983 to 1986 he was with the
Telecommunications Laboratories at GEC Hirst Research Centre. From 1988 to
2001 he was a lecturer in statistics at Victoria University of Wellington, New
Zealand. From 2001-2015 he worked in Electrical and Computer Engineering at
the University of Canterbury. In 2015 he joined Victoria University of
Wellington as Professor of Statistics. He is also an Adjunct Professor in
Electrical and Computer Engineering at the University of Canterbury, New
Zealand and an Honorary Professor in the School of Electronics, Electrical
Engineering and Computer Science, Queen’s University Belfast. He was elected a
Fellow of the IEEE in 2015 and in 2017 was awarded a Distinguished Visiting
Fellowship by the UK based Royal Academy of Engineering at Queen’s University
Belfast. In 2018-2019 he was awarded Visiting Fellowships at the University of
Bologna, the University of Bristol and the University of Melbourne. His
research interests include the statistical aspects of design, modeling and
analysis for communication systems, especially antenna arrays, MIMO, cognitive
radio, massive MIMO, mmWave systems, reconfigurable intelligent surfaces and
the fusion of radar sensing and communications.

[] Michail Matthaiou (Fellow, IEEE) was born in Thessaloniki, Greece in 1981.
He obtained the Diploma degree (5 years) in Electrical and Computer
Engineering from the Aristotle University of Thessaloniki, Greece in 2004. He
then received the M.Sc. (with distinction) in Communication Systems and Signal
Processing from the University of Bristol, U.K. and Ph.D. degrees from the
University of Edinburgh, U.K. in 2005 and 2008, respectively. From September
2008 through May 2010, he was with the Institute for Circuit Theory and Signal
Processing, Munich University of Technology (TUM), Germany working as a
Postdoctoral Research Associate. He is currently a Professor of Communications
Engineering and Signal Processing and Deputy Director of the Centre for
Wireless Innovation (CWI) at Queen’s University Belfast, U.K. after holding an
Assistant Professor position at Chalmers University of Technology, Sweden. His
research interests span signal processing for wireless communications, beyond
massive MIMO, intelligent reflecting surfaces, mm-wave/THz systems and deep
learning for communications.

Dr. Matthaiou and his coauthors received the IEEE Communications Society
(ComSoc) Leonard G. Abraham Prize in 2017. He currently holds the ERC
Consolidator Grant BEATRICE (2021-2026) focused on the interface between
information and electromagnetic theories. To date, he has received the
prestigious 2023 Argo Network Innovation Award, the 2019 EURASIP Early Career
Award and the 2018/2019 Royal Academy of Engineering/The Leverhulme Trust
Senior Research Fellowship. His team was also the Grand Winner of the 2019
Mobile World Congress Challenge. He was the recipient of the 2011 IEEE ComSoc
Best Young Researcher Award for the Europe, Middle East and Africa Region and
a co-recipient of the 2006 IEEE Communications Chapter Project Prize for the
best M.Sc. dissertation in the area of communications. He has co-authored
papers that received best paper awards at the 2018 IEEE WCSP and 2014 IEEE
ICC. In 2014, he received the Research Fund for International Young Scientists
from the National Natural Science Foundation of China. He is currently the
Editor-in-Chief of Elsevier Physical Communication, a Senior Editor for IEEE
Wireless Communications Letters and IEEE Signal Processing Magazine, and an
Associate Editor for IEEE Transactions on Communications. He is an IEEE
Fellow.