# RF Lens Antenna Array-Based One-Shot Coarse Pointing for
Hybrid RF/FSO Communications

Hyung-Joo Moon, , Hong-Bae Jeon, , and
Chan-Byoung Chae This work was supported in part by IITP grant funded by the
Korea government (MSIT) (No. 2019-0-00685, Free space optical communication
based vertical mobile network), and (No. 2016-0-00208, High Accurate
Positioning Enabled MIMO Transmission and Network Technologies for Next 5G-V2X
Services). (Corresponding author: C.-B. Chae.)H.-J. Moon, H.-B. Jeon and C.-B.
Chae are with the School of Integrated Technology, Yonsei University, Seoul
120-749, Korea (e-mail: {moonhj, hongbae08, cbchae}@yonsei.ac.kr).

###### Abstract

Because of its high directivity, free-space optical (FSO) communication offers
a number of advantages. It can, however, give rise to major system
difficulties concerning alignment between two terminals. During the link-
acquisition step (a.k.a. coarse pointing), a ground station can be prevented
from acquiring optical links due to pointing errors and insufficient
information about unmanned aerial vehicle locations. We propose, in this
letter, a radio-frequency (RF) lens antenna array to increase the performance
of coarse pointing in hybrid RF/FSO communications. The proposed algorithm
using a novel closed-form angle estimator, compared to conventional methods,
reduces the minimum outage probability by over a thousand times.

###### Index Terms:

Free-space optics, lens antenna array, coarse pointing.

##  I Introduction

Researchers are developing a major future communication technology known as
free-space optical (FSO) communications. FSO has two primary advantages–high
capacity and unlicensed frequency bands [1]. Currently, the tradeoff is
twofold–the unpredictability of channels and significant pointing
difficulties–making such systems less stable than conventional radio-frequency
(RF) communications systems. Therefore, researchers have begun to investigate
the use of RF links to compensate for system instability (i.e., hybrid RF/FSO
communication systems). In [2, 3], the authors claimed that single-hop
parallel RF/FSO communication is a promising technology that utilizes both
robustness and high data rate, the complementary advantages of RF and FSO
link.

It is widely known that although RF communications have a low risk of link
failure, beam-directivity lags far behind that of optical communications.
Hence, many studies on hybrid RF/FSO systems assume a link distance of no
greater than a few kilometers to maximize the advantages of additional RF
links [4]. One of the approaches in this field is to distribute the high
demand for capacitance by deploying unmanned aerial vehicles (UAVs) in an FSO
backhaul network; this supplies a sufficient data rate with low latency [5].
The UAVs in Fig. 1, for example, can be deployed efficiently based on the
demands of local users. In this letter, we propose a performance-enhancement
strategy for ground-to-air hybrid RF/FSO communications by focusing on link-
acquisition capability.

Link-acquisition capability is determined by the performance of the pointing,
acquisition, and tracking (PAT) system and is generally broken down into two
steps–coarse pointing and fine tracking. The coarse-pointing process begins
with the transmission of an optical beacon signal from a ground station to the
location that is most likely to be occupied by a UAV. In most hybrid RF/FSO
communication systems for ground-to-air applications, this location is
determined by the ground station using global positioning system (GPS)
information sent from a UAV through RF signals [6]. We propose utilizing RF
signals not only for backup data links and passing GPS information, but also
for angle of arrival (AoA) estimation at ground stations to reduce UAV
location uncertainty and provide a reliable link-formation process. When a UAV
successfully detects a beacon sent from a ground station, it transmits an
optical beam back in the direction of arrival using a transmitter that is
tightly aligned with its receiver. Once the ground station receives a beam
sent back from a UAV, both the ground station and UAV begin transmitting to
each other and controlling the beam direction using fine-tracking modules [1].

From the perspective of a PAT system, an RF lens antenna array is an
appropriate choice for hybrid RF/FSO systems for several reasons. First, prior
work has demonstrated that the Cramér-Rao lower bound of AoA estimation with a
lens antenna array is considerably lower than that with a uniform linear
antenna array [7]. Moreover, since most of the received power is concentrated
around the focal point, activating only a few antenna elements does not bring
much performance degradation but allows a low-complexity RF beam-tracking on
multiple UAVs due to the reduced RF chains [8, 9, 2020wcnc]. In addition, even
if the UAVs use the same bandwidth, signal interference may be resolved by
antenna selection of the lens antenna if they are in different directions.
This antenna selection property is also considered in this letter to reduce
the complexity of our algorithm. Finally, RF signal-based UAV tracking is
possible [10]. This allows a ground station to select the most appropriate UAV
for communication depending on the channel status and quickly establish an
optical link through the algorithm we will propose. As shown in Fig. 1, an RF
module receives RF signals from multiple UAVs. When link failure occurs, this
system facilitates rapid reacquisition of an FSO link.

The rest of this letter is organized as follows. In Section II, we introduce
the lens antenna array structure and optical channel model used in our
analysis. In Section III, we present a closed-form expression of this
estimator for a given GPS information. In Section IV, a novel one-shot,
coarse-pointing algorithm is proposed and described from the perspective of
outage probability and processing time. Also proposed in Section IV is an
evaluation method for the processing time of one-shot, coarse-pointing.
Numerical simulation results are presented in Section V and in Section VI we
offer our conclusions.

Figure 1: System model for lens antenna array-based RF/FSO hybrid
communications. The key operations of our algorithm are highlighted in red.

##  II System Model

Fig. 1 presents an integrated scenario with an RF lens antenna array and
optical PAT module for a hybrid RF/FSO system. The RF module at the ground
station utilizes a lens antenna array to track multiple UAVs. It leads to a
fast acquisition of an optical link when the high-throughput backhaul or
feeder link is required at a specific aerial node.

###  II-A Signal Model for RF Lens Antenna Array

The received power of incoming RF signals through a lens differs for each
antenna element at the focal distance. The received signal vector $\mathbb{x}$
can be modeled as ${\mathbb{x}=g\mathbb{H}(\phi)\mathbb{r}e^{jb}+\mathbb{z}}$,
where the diagonal amplitude matrix $\mathbf{H}(\phi)$ is given as follows
[7]:

$\displaystyle\mathbb{H}(\phi)_{n+\frac{N+1}{2},n+\frac{N+1}{2}}=\frac{L}{\sqrt{z}}\operatorname{sinc}{\left[\frac{L}{\lambda}\left(\frac{dn}{z}-\sin(\phi)\right)\right]},$
(1)

where the index $n$ of each antenna element is a set from $-(N-1)/2$ to
$(N-1)/2$, and $N$ is the number of antenna elements. The vector $\mathbf{r}$
of length $N$, which varies with the shape of the antenna array, has the form
of

$\mathbb{r}=\begin{cases}\begin{aligned}
s\Bigg{[}\exp\Bigg{\\{}-\frac{j2\pi}{\lambda}\sqrt{d^{2}\left(-\frac{N-1}{2}\right)^{2}+f^{2}}\Bigg{\\}},\cdots\\\
,\exp\Bigg{\\{}-\frac{j2\pi}{\lambda}\sqrt{d^{2}\left(\frac{N-1}{2}\right)^{2}+f^{2}}\Bigg{\\}}\Bigg{]}\end{aligned}&\text{(linear)}\\\
s\left[1,\cdots,1\right]^{T}\exp\left\\{{-2\pi
z/\lambda}\right\\}&\text{(arc)},\end{cases}$ (2)

where $s$ is a unit-power signal. The symbols $g$, $b$, ${\phi}$, and
$\mathbf{\mathbb{z}}$ are the signal gain, signal phase, AoA of the signal,
and zero-mean complex normal vector with a standard deviation of
${\sigma_{\text{n}}}$, respectively. The parameters ${L}$, ${d}$, ${n}$,
${\lambda}$ and ${z}$ denote the lens diameter, antenna spacing, antenna
index, signal wavelength, and distance between the center of the lens and each
antenna, respectively. The value of ${z}$ is determined as ${z=f}$ for an arc
antenna array, and ${z=\sqrt{d^{2}n^{2}+f^{2}}}$ for a linear array.

Compared to a uniform linear antenna array, a lens antenna array has the
advantage of requiring a small number of RF chains. Suppose the receiver only
has $k$ RF chains. At this point, the natural option is to estimate the focal
point by substituting an angle $\phi_{\text{gps}}$ calculated by GPS
information into (1), and select the nearest $k$ antennas. The smallest index
$u$ and the largest index $v$ can be expressed as
${u=[\frac{z}{d}\sin(\phi_{\text{gps}})-\frac{k}{2}+\frac{1+(-1)^{N}}{4}]+\frac{1-(-1)^{N}}{4}}$
and
${v=[\frac{z}{d}\sin(\phi_{\text{gps}})+\frac{k}{2}+\frac{1+(-1)^{N}}{4}]+\frac{1-(-1)^{N}}{4}}$.
Therefore, the received signal vector with the limited number of RF chains
becomes ${\mathbb{y}=g\mathbb{A}(\phi)\mathbb{s}e^{jb}+\mathbb{n}}$, where
$\mathbb{y}=\mathbb{x}[u:v]$, $\mathbb{A}(\phi)=\mathbb{H}(\phi)[u:v,u:v]$,
$\mathbb{s}=\mathbb{r}[u:v]$, and $\mathbb{n}=\mathbb{z}[u:v]$.

###  II-B Optical Channels for Performance Evaluation

A received power ${P_{R}}$ can be modeled as
${P_{R}={h_{\ell}}{h_{a}}{h_{p}}RP_{T}}$, where each of the parameters
indicates attenuation, fading, pointing loss, responsivity, and transmitted
power, respectively [11].

####  II-B1 Attenuation Loss

The term ${h_{\ell}}$ representing attenuation loss follows the Beer-Lambert
law as

$\displaystyle h_{\ell}(z_{\ell})=\frac{P(z_{\ell})}{P(0)}=\text{exp}(-\sigma
z_{\ell}),$ (3)

where ${z_{\ell}}$ is a propagation distance, ${P(z_{\ell})}$ is a power level
at ${z_{\ell}}$, and ${\sigma}$ is an attenuation coefficient determined by
the visibility range.

####  II-B2 Atmospheric Fading

One of the conventional fading models of optical channels is a log-normal
distribution, which usually stands for a stable atmospheric condition. The
probability density function (PDF) of ${h_{a}}$ is given by

$\displaystyle
f_{\text{LN}}(h_{a})=\frac{1}{2h_{a}\sqrt{2\pi\sigma_{\text{X}}^{2}}}\text{exp}\left(\frac{(\text{ln}h_{a}+2\sigma_{\text{X}}^{2})^{2}}{8\sigma_{\text{X}}^{2}}\right),$
(4)

and ${\sigma_{\text{X}}}$ here, is approximated as ${\sigma_{\text{R}}/2}$,
where ${\sigma_{\text{R}}^{2}}$ is the Rytov variance [12]. On the other hand,
$h_{a}$ follows gamma-gamma distribution considering scintillation:

$\displaystyle
f_{\text{GG}}(h_{a})=\frac{2(\alpha\beta)^{(\alpha+\beta)/2}}{\Gamma(\alpha)\Gamma(\beta)}h_{a}^{\frac{\alpha+\beta}{2}-1}K_{\alpha-\beta}\left(2\sqrt{\alpha\beta
h_{a}}\right),$ (5)

where ${1/\alpha}$ and ${1/\beta}$ are the variances of the large-and small-
scale eddies, and ${K_{\alpha-\beta}}$ is the modified Bessel function of the
second kind [12].

####  II-B3 Pointing Error

The two main factors determining the pointing error are beamwidth ${w_{z}}$ at
a distance ${z}$ and beam displacement ${r}$ [13]. Coordinates of the UAV
location are a superposition of the estimated point and Gaussian error. Then,
the distribution of the beam displacement can be modeled as

$\displaystyle
f_{\text{dis}}(r)=\frac{r}{\sigma_{d}^{2}}\text{exp}\left(-\frac{r^{2}}{2\sigma_{d}^{2}}\right),$
(6)

which is a Rayleigh distribution. Here, ${\sigma_{d}}$ is the one-dimensional
standard deviation of the Gaussian error, which satisfies
${\sigma_{d}=\sqrt{\sigma_{\text{est}}^{2}+\sigma_{\text{jit}}^{2}}}$ where
${\sigma_{\text{est}}}$ and ${\sigma_{\text{jit}}}$ are the standard deviation
of the angle estimation error and the error induced by jitter, respectively.
Therefore, ${h_{\text{p}}}$ is given by ${r}$ as

$\displaystyle
h_{\text{p}}(r,z)=A_{\text{0}}\text{exp}\left(-\frac{2r^{2}}{w_{z}^{2}}\right),$
(7)

as the beamwidth ${w_{z}}$ is approximately proportional to the distance and
${A_{\text{0}}}$ is the power gain at the center of the beam. In this case,
the beamwidth can be described as ${w_{z}=z\cdot\theta_{\text{div}}}$ where
${\theta_{\text{div}}}$ is the beam divergence angle.

##  III Closed-form Angle Estimator

If during the coarse-pointing stage, the uncertainty area of a UAV is wider
than the typical beam-divergence angle, then a ground station scans the area
so that the UAV can receive the beacon beam [14]. In practice, GPS information
provides position information within an uncertainty range of a few meters.
Therefore, we assume that the ground station selects a beam divergence angle
to cover the entire uncertainty area when transmitting a beacon beam [6].
Given there is no scanning process using a beacon-beam footprint, this method
is referred to as one-shot coarse pointing.

For successful link acquisition via one-shot coarse pointing, a UAV must
detect a beacon beam transmitted from a ground station and transmit the beam
back to the ground station. During this process, a beam-divergence angle must
be sufficiently wide to completely cover the uncertainty area. Also, minimal
beam divergence is required to intensify the beam and maximize the probability
of detection. This trade-off demonstrates that the robustness of the link-
acquisition step is directly affected by the uncertainty of the UAV location
information. Here, the proposed advanced estimator reduces the outage
probability of the coarse pointing process by improving the initial accuracy
of UAV location information.

Figure 2: Performance enhancement of the proposed estimator compared to the
conventional GPS-only scenario.

We designed a highly accurate AoA estimation method with low complexity by
fully utilizing both received RF signals and GPS information from a UAV. A
closed-form suboptimal solution for the UAV location is first derived using
the maximum a posteriori (MAP) criterion. Assume that the antenna array
receives a superposition of the signal vector ${\mathbb{y}}$ and a Gaussian
noise vector ${\mathbb{n}}$. Then the conditional distribution of
${\mathbb{y}}$ given an AoA of ${\phi}$ can be written as

$\displaystyle
f_{\text{lens}}(\mathbb{y}|\phi)=\frac{1}{(\pi\sigma_{n})^{N}}\exp\left[-\frac{\left\lVert\mathbb{y}-g\mathbb{A}(\phi)\mathbb{s}e^{jb}\right\rVert^{2}}{\sigma_{n}^{2}}\right]$
(8)

and the given GPS coordinate information can be equivalently converted into an
angle ${\phi_{\text{gps}}}$. Thus the PDF of ${\phi}$ is given by

$\displaystyle
f_{\text{gps}}(\phi)=\frac{1}{\sqrt{2\pi}\sigma_{\text{gps}}}\exp\left[-\frac{(\phi-\phi_{\text{gps}})^{2}}{2\sigma_{\text{gps}}^{2}}\right],$
(9)

where ${\sigma_{\text{gps}}}$ is the standard deviation of the GPS information
and has a value of approximately 5 m divided by the distance to the UAV [6].

Algorithm 1 Proposed coarse-pointing algorithm

1:while FSO link failure do

2:
${u=[\frac{z}{d}\sin(\phi_{\text{gps}})-\frac{k}{2}+\frac{1+(-1)^{N}}{4}]+\frac{1-(-1)^{N}}{4}}$.

3:
${v=[\frac{z}{d}\sin(\phi_{\text{gps}})+\frac{k}{2}+\frac{1+(-1)^{N}}{4}]+\frac{1-(-1)^{N}}{4}}$.

4: Choose antenna indices $\forall i\in\mathbb{Z}:i\in[u,v]$ and connect to
each RF chain.

5: Obtain ${\phi_{\text{gps}},g,b,}$ and ${\sigma_{\text{n}}}$ during RF
communication.

6: if FSO channel is known then

7: Choose beam divergence angle.

8: Compute (10) and transmit beacon beam to UAV.

9: if UAV detects beacon then

10: Begin communication.

11: else

12: Compare ${t_{0}}$ and ${t_{\text{rot}}}$ to determine the appropriate
policy.

13:end

###### Theorem 1

Given $\mathbb{y}$, ${\phi_{\text{gps}}}$, ${g}$, ${b}$, and
${\sigma_{\text{n}}}$, a closed-form angle estimation can be expressed as
follows 111In practice, two-dimensional angle parameters are required for the
complete location estimation of a UAV. Note that the result in (10) is
sufficient since the dimension extension of the antenna array can equivalently
extend the dimension of estimation.:

$\phi_{\text{est}}=\phi_{\text{gps}}+\\\
\frac{e^{jb}\frac{\mathbb{y}}{g}^{*}\mathbb{A}^{\prime}(\phi_{\text{gps}})\mathbb{s}+e^{-jb}\mathbb{s}^{*}\mathbb{A}^{\prime}(\phi_{\text{gps}})\frac{\mathbb{y}}{g}-2\mathbb{s}^{*}\mathbb{A}(\phi_{\text{gps}})\mathbb{A}^{\prime}(\phi_{\text{gps}})\mathbb{s}}{2\mathbb{s}^{*}\mathbb{A}^{\prime}(\phi_{\text{gps}})^{2}\mathbb{s}+2\sigma_{\text{n}}^{2}/g^{2}\sigma_{\text{gps}}^{2}}.$
(10)

_Proof:_ See the Appendix. $\blacksquare$

By Theorem 10, the ground station can accurately estimate the UAV location
based on the received RF signal. Fig. 2 demonstrates that, within a range of
10 km, the proposed estimator effectively reduces the standard deviation of
UAV location inference. This results in a narrower uncertainty area because
the uncertainty area is proportional to the square of the UAV location error.
If the receiver connects four selected antennas to each RF chain, there is a
slight performance degradation but a significant reduction in system
complexity.

##  IV Proposed Coarse Pointing Algorithm

Algorithm 1 is proposed; it allows flexible feeder-link acquisition for mobile
UAVs supporting dynamic demands of data rate. Parameters for angle estimation
are given through continuous RF tracking. To quickly connect the link, we now
discuss optimizing the coarse-pointing factors: Beam divergence angle and
coarse pointing policy. To this end, we assume that a ground station knows
optical channel information.

###  IV-A Beam Divergence Selection

The coarse-pointing process can be successfully done when the received power
${P_{R}}$ is greater than the threshold power ${P_{\text{th}}}$. The outage
probability ${P_{\text{out}}=\mathbb{Prob}[P_{R}<P_{\text{th}}]}$ is given by

$\displaystyle
P_{\text{out}}=\int_{0}^{\infty}f_{\text{dis}}(r)\bigg{\\{}\int_{0}^{P_{\text{th}}}f\left(P_{R}|r\right)d{P_{R}}\bigg{\\}}dr,$
(11)

where ${f(P_{R}|r)}$ is the PDF of ${P_{R}}$ for a given beam displacement
${r}$. The distribution is determined by using the channel model introduced in
Section II-B. The numerical results in Section V allow the appropriate beam-
divergence selection that minimizes the outage probability.

###  IV-B Coarse Pointing Policy

The processing time for coarse pointing strongly depends on the outage
probability and channel-coherence time since the proposed algorithm must be
executed until the UAV successfully receives a signal power greater than the
threshold power. The channel-coherence time ${t_{0}}$ can be expressed as

$\displaystyle t_{0}=\bar{\rho}_{0}/v_{\perp},$ (12)

where ${\bar{\rho}=\left(\frac{1}{2.91k^{2}C_{n}^{2}L}\right)^{3/5}}$ is the
atmospheric correlation length, and ${v_{\perp}}$ is the wind speed
perpendicular to the link [16].

TABLE I: Simulation parameters Parameter | Value
---|---
Optical transmitted power ($P_{\text{t}}$) | $1\,\,$\mathrm{W}$$
Optical threshold power ($P_{\text{th}}$) | $1\,\,$\mathrm{\SIUnitSymbolMicro W}$$
Responsivity ($R$) | $0.5$
Standard deviation of mechanical error ($\sigma_{\text{jit}}$) | $3\,\,$\mathrm{mrad}$$
Visibility range ($V$) | $3,\,10\,\,$\mathrm{km}$$
Log-amplitude standard deviation ($\sigma_{\text{X}}$) | $0.3$
Gamma-gamma fading parameters ($\alpha,\beta$) | $8.05,1.03$
Standard deviation of GPS error ($\sigma_{\text{gps}}$) | $5\,\,$\mathrm{m}$$

We propose two different policies for attempting link acquisition. One re-
estimates the AoA for every acquisition failure and transmits the beacon beam
in a revised direction. The other relies on the initial AoA estimation result
and waits until the UAV receives sufficient signal power as the channel
fluctuates. Assuming that ${t_{\text{rot}}}$ is the time required to rotate a
beam to the desired point, the average coarse pointing processing time
${\overline{T}_{\text{re}}}$ for the re-estimation policy is given by

$\displaystyle\overline{T}_{\text{re}}=\frac{t_{0}+t_{\text{rot}}}{1-P_{\text{out}}}.$
(13)

In the single-estimation policy, the value of $r$ is fixed when the estimation
is performed. The average processing time ${\overline{T}_{\text{sing}}}$ for
the latter policy is

$\displaystyle\overline{T}_{\text{sing}}=\underset{r}{\mathbb{E}}\left[\frac{t_{0}}{1-F(r)}\right]+t_{\text{rot}},$
(14)

where ${F(r)=\int_{0}^{P_{\text{th}}}f\left(P_{R}|r\right)d{P_{R}}}$ is the
outage probability for a given $r$. If ${t_{0}{\ll}t_{\text{rot}}}$, then it
is clear that the single-estimation strategy takes less time on average. On
the other hand, ${t_{0}{\gg}t_{\text{rot}}}$ guarantees a shorter processing
time for the re-estimation method. This can be proven as follows:

$\displaystyle\overline{T}_{\text{re}}=\frac{t_{0}}{1-P_{\text{out}}}=\frac{t_{0}}{1-\mathbb{E}\left[F(r)\right]}\leq\underset{r}{\mathbb{E}}\left[\frac{t_{0}}{1-F(r)}\right]=\overline{T}_{\text{sing}}.$
(15)

Since ${g(x)=\frac{1}{1-x}}$ is convex in $x\in[0,1)$, Jensen’s inequality
supports (15). The ground station can select the best policy based on a
comparison between $t_{0}$ and $t_{\text{rot}}$, which we demonstrate
numerically in Section V.

##  V Numerical Results

Figure 3: Outage probability for a ${1}$ km link distance.

(a)  (b)

Figure 4: (a) Outage probability under harsh weather for different distances
and (b) the amount of reduction compared to the conventional method.

(a)

(b)

Figure 5: Probability that a link is yet not connected at a time $t$ for (a)
small and (b) large ${t_{0}}/{t_{\text{rot}}}$ values.

The detailed parameters used for our simulations are listed in Table I. In
Fig. 3, the purple lines represent the outage probability when only GPS
information is utilized for coarse pointing; the orange lines represent the
proposed method. The lowest point on each line represents the lowest
achievable outage probability and beam-divergence angle for the achievement.
The result shows that the minimum outage probability is significantly reduced
and can be achieved with a narrower beam divergence when using the proposed
method. When the atmospheric channel is known to the ground station, lookup
table-based, beam-divergence selection is possible. Fig. 4 presents outage
probability measurements with varying distances between the ground station and
UAV. As the distance increases, the gap narrows between the solid and dashed
lines. This shows that when the link distance is short, the proposed method is
more effective. This is because the intensity of an RF signal attenuates
significantly with propagation distance.

Fig. 5 shows the numerical results for the processing time of one-shot, coarse
pointing using (13) and (14). The values indicate the probability that the
link-acquisition time ${t_{\text{acq}}}$ is greater than $t$. The value of
${t_{\text{rot}}}$ is fixed at $20$ ms and the link distance is assumed to be
$2$ km. As shown in Fig. 5, when ${t_{0}}$ is much smaller than
${t_{\text{rot}}}$, the processing time of the single-estimation method is
almost certainly faster than that of the re-estimation method. Conversely, the
re-estimation method is always faster when ${t_{0}}$ is sufficiently large.
When mobility exists, the performance degrades over time only for a single-
estimation policy since the estimated AoA is not updated while the UAV is
moving.

##  VI Conclusion

In this letter, we derived, as a component of a PAT system, a closed-form
solution for angle estimation and proposed a fast one-shot coarse-pointing
algorithm for hybrid RF/FSO communication systems. We showed that the proposed
algorithm effectively reduces the outage probability of link acquisition.
Additionally, when an accurate prediction of the optical channel conditions is
available, it is possible to find the beam-divergence angle that minimizes the
outage probability. Further, we formulated and analyzed the processing time of
the one-shot, coarse-pointing process. Finally, we numerically verified that,
in terms of outage probability and processing time, it is superior to the
conventional method. [Proof of Theorem 10] Following MAP criterion, the
proposed estimator can be formulated as

$\displaystyle\phi_{\text{est}}=\underset{\phi}{\operatorname{argmax}}f(\phi|\mathbb{y})=\underset{\phi}{\operatorname{argmax}}\frac{f_{\text{lens}}(\mathbb{y}|\phi)f_{\text{gps}}(\phi)}{f(\mathbb{y})},$
(16)

where ${f(\mathbb{y})=\int
f_{\text{lens}}(\mathbb{y}|\phi)f_{\text{gps}}(\phi)\,d\phi}$. Given
$\mathbb{y}$, since ${f(\mathbb{y})}$ is a constant, the solution of (16) can
be obtained by searching for the local maximum of
${f_{\text{lens}}(\mathbb{y}|\phi)f_{\text{gps}}(\phi)}$ with respect to
$\phi$ if it is a concave function. To this end, we apply the first-order
Taylor approximation
${\mathbb{A}(\phi)=\mathbb{A}(\phi_{\text{gps}})+\mathbb{A}^{\prime}(\phi_{\text{gps}})(\phi-\phi_{\text{gps}})}$
to the following function:

$\displaystyle f^{\prime}_{\text{lens}}(\mathbb{y}|\phi)=$
$\displaystyle\left\\{\frac{\partial}{\partial\phi}{\left\lVert\mathbb{y}-g\mathbb{A}(\phi){\mathbb{s}}e^{jb}\right\rVert^{2}}\right\\}$
(17)
$\displaystyle\cdot\frac{1}{(\pi\sigma_{n})^{N}}\exp\left[-\frac{\left\lVert\mathbb{y}-g\mathbb{A}(\phi){\mathbb{s}}e^{jb}\right\rVert^{2}}{\sigma_{n}^{2}}\right].$

Note that a diagonal matrix
$\mathbb{A}^{\prime}(\phi)=\mathbb{H}^{\prime}(\phi)[u:v,u:v]$ and
$\mathbb{H}^{\prime}(\phi)=\frac{d}{d\phi}\mathbb{H}(\phi)$. Each component of
$\mathbb{H}^{\prime}(\phi)$ then satisfies
$\mathbb{H}^{\prime}(\phi)_{n+\frac{N+1}{2},n+\frac{N+1}{2}}=\frac{L^{2}\cos(\phi)}{\lambda\sqrt{z}{{\phi_{1}}^{2}}}\\{\sin(\phi_{1})-\phi_{1}\cos({\phi_{1}})\\}$,
where $\phi_{1}=\frac{L}{\lambda}(\frac{dn}{z}-\sin({\phi}))$ and $n$ is a set
from $-(N-1)/2$ to $(N-1)/2$. Hence, the derivatives of
${f_{\text{gps}}(\phi)}$ and ${f_{\text{lens}}({\mathbb{y}}|\phi)}$ are given
by

$\displaystyle
f^{\prime}_{\text{gps}}(\phi)=\frac{\Delta\phi}{\sigma_{\text{gps}}^{2}}f_{\text{gps}}(\phi),$
(18) $\displaystyle f^{\prime}_{\text{lens}}({\mathbb{y}}|\phi)\approx$
$\displaystyle-\frac{g}{\sigma_{\text{n}}^{2}}\big{\\{}2g\mathbb{s}^{*}(\mathbb{A}(\phi_{\text{gps}})\mathbb{A}^{\prime}(\phi_{\text{gps}})+\mathbb{A}^{\prime}(\phi_{\text{gps}})^{2}\Delta\phi)\mathbb{s}$
(19)
$\displaystyle-e^{jb}\mathbb{y}^{*}\mathbb{A}^{\prime}(\phi_{\text{gps}})\mathbb{s}-e^{-jb}\mathbb{s}^{*}\mathbb{A}^{\prime}(\phi_{\text{gps}})\mathbb{y}\big{\\}}f_{\text{lens}}(\mathbb{y}|\phi),$

respectively, where ${\Delta\phi=\phi-\phi_{\text{gps}}}$. Therefore, by
solving
${\frac{\partial}{\partial\phi}f_{\text{lens}}(\mathbb{y}|\phi)f_{\text{gps}}(\phi)=0}$,
the local maximum $\phi_{\text{est}}$ satisfies

$\displaystyle\Big{[}-\frac{g}{\sigma_{\text{n}}^{2}}\big{\\{}2g\mathbb{s}^{*}(\mathbb{A}(\phi_{\text{gps}})\mathbb{A}^{\prime}(\phi_{\text{gps}})+\mathbb{A}^{\prime}(\phi_{\text{gps}})^{2}\Delta\phi)\mathbb{s}$
(20)
$\displaystyle-e^{jb}\mathbb{y}^{*}\mathbb{A}^{\prime}(\phi_{\text{gps}})\mathbb{s}-e^{-jb}\mathbb{s}^{*}\mathbb{A}^{\prime}(\phi_{\text{gps}})\mathbb{y}\big{\\}}+\frac{\Delta\phi}{\sigma_{\text{gps}}^{2}}\Big{]}$
$\displaystyle{\cdot}f_{\text{lens}}(\mathbb{y}|\phi)f_{\text{gps}}(\phi)$
$\displaystyle=0,$

which has the unique solution of (10).

## References

  * [1] H. Kaushal and G. Kaddoum, “Optical communication in space: Challenges and mitigation techniques,” _IEEE Commun. Surveys Tuts._ , vol. 19, no. 1, pp. 57–96, Feb. 2017.
  * [2] H. Dahrouj _et al._ , “Cost-effective hybrid RF/FSO backhaul solution for next generation wireless systems,” _IEEE Wireless Commun._ , vol. 22, no. 5, pp. 98–104, Oct. 2015.
  * [3] A. AbdulHussein _et al._ , “Rateless coding for hybrid free-space optical and radio-frequency communication,” _IEEE Trans. Wireless Commun._ , vol. 9, no. 3, pp. 907–913, Mar. 2010.
  * [4] L. Chen _et al._ , “Multiuser diversity over parallel and hybrid FSO/RF links and its performance analysis,” _IEEE Photon. J._ , vol. 8, no. 3, pp. 1–10, Jun. 2016.
  * [5] K. Zhou _et al._ , “Distributed channel allocation and rate control for hybrid FSO/RF vehicular ad hoc networks,” _J. Opt. Commun. Netw._ , vol. 9, no. 8, pp. 669–681, Aug. 2017.
  * [6] H. Henniger and B. Epple, “Free-space optical transmission improves land-mobile communications,” _SPIE Newsroom_ , Jan. 2007.
  * [7] J.-N. Shim _et al._ , “Cramér-Rao lower bound on AoA estimation using an RF lens-embedded antenna array,” _IEEE Antennas Wireless Propag. Lett._ , vol. 17, no. 12, pp. 2359–2363, Dec. 2018.
  * [8] T. Kwon _et al._ , “RF lens-embedded massive MIMO systems: Fabrication issues and codebook design,” _IEEE Trans. Microw. Theory Techn._ , vol. 64, no. 7, pp. 2256–2271, Jul. 2016.
  * [9] Y. J. Cho _et al._ , “RF lens-embedded antenna array for mmWave MIMO: Design and performance,” _IEEE Commun. Mag._ , vol. 56, no. 7, pp. 42–48, Jul. 2018.
  * [10] R. Kingsbury _et al._ , “Design of a free-space optical communication module for small satellites,” in _Proc. Annu. AIAA/USU Conf. Small Satellites_ , 2014, pp. 1–10.
  * [11] J. Park _et al._ , “Impact of pointing errors on the performance of coherent free-space optical systems,” _IEEE Photon. Technol. Lett._ , vol. 28, no. 2, pp. 181–184, Jan. 2016.
  * [12] X. Zhu and J. Kahn, “Free-space optical communication through atmospheric turbulence channels,” _IEEE Trans. Commun._ , vol. 50, no. 8, pp. 1293–1300, Aug. 2002.
  * [13] A. A. Farid and S. Hranilovic, “Outage capacity optimization for free-space optical links with pointing errors,” _J. Lightw. Technol._ , vol. 25, no. 7, pp. 1702–1710, Jul. 2007.
  * [14] A. Harris and T. A. Giuma, “Minimization of acquisition time in a wavelength diversified FSO link between mobile platforms,” in _Proc. SPIE Atmospheric Propag. IV_ , vol. 6551, 2007, p. 655108.
  * [15] A. Carrasco-Casado _et al._ , “Low-impact air-to-ground free-space optical communication system design and first results,” in _Proc. IEEE Int. Conf. on Space Opt. Syst. and Appl. (ICSOS)_ , 2011, pp. 109–112.
  * [16] A. Puryear and V. W. S. Chan, “On the time dynamics of optical communication through atmospheric turbulence with feedback,” _J. Opt. Commun. Netw._ , vol. 3, no. 8, pp. 594–609, Aug. 2011.


# FAR Tracker: A Fast, Accurate and Robust Multiple Face Tracker

First Author
Institution1
Institution1 address
<EMAIL_ADDRESS>Second Author
Institution2
First line of institution2 address
<EMAIL_ADDRESS>
###### Abstract

Multiple face tracking in videos is a challenging task due to face motion,
occlusion and shot change. Current tracking by detection approaches
[Wojke2018deep, zhang2016tracking, lin2018prior] rely on detectors and thus
suffer from errors made by the detectors. In this paper, we apply bounding box
re-detection to a single-stage detection framework and use facial embedding
feature to effectively associate the tracklets. By exploring the spatial and
temporal information our solution is able to generate tracklets directly from
a detector backbone which allows us to use a light-weight linkage algorithm
for online identity link. Thanks to the single-stage structure and the novel
anchor reassignment algorithm, the runtime of our method is agnostic to the
number of faces. The proposed FAR tracker outperforms the state-of-the-art
multi object tracking algorithms significantly. Our online tracking method
achieves 0.668 of MOTA on the MusicVideo dataset and 5.3% improvement in
recall against the baseline approach on the IQIYI dataset.

##  1 Introduction

Similar to the role of face detection in image-based face analysis, face
tracking is an essential component for face understanding in videos
[tapaswi2019video]. Its outputs are subsequently used in facial expression
analysis [kanade2000comprehensive], face attributes detection
[shen2017learning], and face recognition [deng2019arcface]. One important
tenet of a multi-face tracker is to distinguish faces of different persons
into different tracks. Despite the rapid advance of tracking algorithms with
the help of deep learning [bergmann2019tracking, feichtenhofer2017detect,
zhang2016tracking], designing a face tracker to achieve this goal still
remains a difficult problem. Besides, due to the high volume size of the video
data, the face tracker is also expected to have: 1) fast runtime speed; 2)
accurate and robust face localization and 3) the capacity for online
processing.

Figure 1: The proposed FAR tracker consists of two stages. The first stage
generates temporally consecutive tracklets by exploring the temporal and
spatial correlations from previous frame. The second stage performs online
linking of the tracklets to generate a face track for each person (view in
color is suggested).

Most existing multi-face tracking algorithms [lin2018prior, zhang2016tracking,
wu2013simultaneous] rely on a three-stage scheme, known as _tracking by
detection_. The first stage exercises a _per-frame face detector_ to localize
face bounding boxes in each frame. The second stage, _tracklet generation_ ,
merges detection results to create a set of tracklets, _i.e._ , short tracks,
based on short-term cues. These short-term cues include motion
[wojke2017simple], appearance features [Wojke2018deep], and geometrical
affinity [sun2019deep]. Then the third stage, _long-term tracking_ , links the
tracklets in the whole video to form long-term tracking outputs. The three-
stage tracking-by-detection scheme is known to provide reasonable tracking
quality. However, there are several limitations in this scheme. Firstly, the
division of face detection and tracklet makes the tracking quality highly
reliant on the quality of face detectors. These detectors, mostly designed for
still-images and operated in a per-frame manner, are not able to utilize
temporal context and thus are not robust against video-specific nuance
factors, such as motion blurring, large pose variations, and video compression
artifacts. Secondly, it is impossible to recover a face bounding box through
temporal context once it is missed by the detector. Thirdly, the low-cost
matching used in the tracklet creation process is prone to cause false
matches, thus creating tracks with multiple persons’ faces in it.

In this work, we propose a new tracker architecture to address the issues
above based on the idea of perform detection and tracking simultaneously. This
tracker is composed of two stages, first for generating tracklets and second
for linking them in long-term tracking.

For the first stage we design a tracklet generation model that creates
tracklet directly from a sequence of input frames, without the need for
grouping. This tracklet generator can be transformed from an existing face
detector with a single-shot multibox detector (SSD) [liu2016ssd] design. It
runs at an almost constant frame per second (FPS) speed regardless of the
number of faces in each frame. As we will later show in the experimental
analysis, it is also less prone to cause false matches in tracklets. Our
tracklet generation model also gives much higher recall of faces comparing
with other divided face detection and grouping approaches, due to the ability
to utilize temporal context for detecting faces. The second stage is a simple
tracklets linking module based on averaging face identity embedding features
of the tracklets, which can be implemented in an online manner. The proposed
multi face tracker outperforms other online state–the-art multi-face trackers
[Wojke2018deep] and can achieve comparable results with the global
optimization methods [lin2018prior, zhang2016tracking].

To build the tracklet generation module that detects and tracks faces
simultaneously, we took inspiration from the work in [bergmann2019tracking].
In [bergmann2019tracking], a faster-RCNN [ren2015faster] model for human
detection is converted to multi-person tracker by passing the detection
results in the previous frame as additional object proposals for the current
frame. This operation is based on the observation that the bounding box
regression modules in fast RCNN [ren2015faster] has a good capacity in
correcting slightly shifted object bounding boxes. It is then used to re-
detect the one object tracked in the previous frame from its estimated
location in the current frame, so the track identity can also be carried over
to this re-detection output. We adopt this simple but effective strategy and
apply it to the case of SSD-based face detectors. We choose SSD-based detector
since the face detection has high runtime efficiency requirement which can be
verified in recent face detection algorithms [zhang2019refineface,
zhang2019accurate, deng2019retinaface, li2019pyramidbox++]. But one question
arises for this adoption, that is the SSD-based face detector does not have a
explicit proposal stage. Thus it is not straightforward how to convey the
detection results between frames. We solve this problem by reckoning the fact
that SSD detector just uses sliding window proposals and designing a process
that locate the proposal for re-detection. Using the SSD-based architecture
enables us to run the model at a constant speed w.r.t. number of faces in each
frame.

Overall, we propose a new face tracker that is composed of a novel tracklet
generation module and a simple online linker. We call this tracker _FAR_
tracker in the sense that it provides fast, accurate and robust long-term
tracking of faces in videos. The contributions of this work are: 1) we propose
a compact design of multi-face tracker that unites face detection and tracklet
creation into a single model followed by a light-weight linkage post-
processing algorithm; 2) we implement the tracklet generation by carefully
adapting an SSD-based face detector to perform detection and tracking
simultaneously; 3) In experiments, the simple yet effective FAR tracker lead
to superior results than other state-of-the-art methods while operating in an
online manner.

Figure 2: Given a frame $F_{t}$, we first use motion estimator to align
existing anchor box $A_{i}^{t-1}$ associating with the identity $i$ to a new
location in the corresponding single shot detector feature map. The regressor
and classifier is then used to generate bounding box prediction $B_{i}^{t}$
and confidence score $C_{i}^{t}$. The $C_{i}^{t}$ is used to kill potentially
occluded tracks and the $B_{i}^{t}$ is used to update the anchor box
$A_{i}^{t}$. At the same time, the face detector provides a set of detections.
We initialize a new track, if a detection has no substantial intersection over
union with any bounding box in the set of active tracks. (view in color is
suggested)

##  2 Related Works

Deep learning-based object detection Object detection is a basic computer
vision problem. Based on whether using explicit detection proposals, the deep
learning detectors can be divided into two main groups: two-stage detectors
[lin2017feature, ren2015faster, girshick2015fast, he2017mask] and one-shot
detectors [liu2016ssd, redmon2016you, redmon2018yolov3]. The two-stage
detectors first generate detection proposals and then conduct classification
as well as position regression. One known issue for the two stage detectors is
that the runtime increase dramatically when there is multiple instances to
detect, affecting the practical usage in real scenario. While single-shot
detectors like SSD [liu2016ssd] conduct the proposal generation in an implicit
manner. Anchors are generated directly from feature map and then regressed to
the ultimate detection results. The anchor-based regression is agnostic to
instance numbers so it could be well applied in scenes multi-target detection
is required like face detection [deng2019retinaface].

Multi-target tracking Benefiting from the advance of object detection, we saw
the emergence of many multi-object tracking approaches [voigtlaender2019mots,
xu2019deepmot, bochinski2017high, insafutdinov2017arttrack, ma2018trajectory].
[wang2019exploit] proposed a multi-scale TrackletNet to merge tracklets
generated through CNN feature, IoU score and epipolar geometry.
[feichtenhofer2017detect] combined the tracking and detection, providing a
simultaneous approach as detection shows the power in regressing accurate
position of targets. The author developed the D2T framework by adding
correlation layer to a Siamese R-FCN [dai2016r] detector. Global tracklets
fusion are employed later to generate final tracking results. Comparing to
these batch based tracking methods, online multi-target tracking draw more and
more attention because of the potential usage in practical. [Wojke2018deep]
provide a simple but effective online multi-target solution. To realize online
multi-target tracking, the author combined the deep learning based detection,
feature embedding with tradition MOT framework and proposed deep feature & IoU
merging based two-gate tracking methods. Different from these tracking by
detection methods where complete detection are employed at the first step,
[bergmann2019tracking] build the connection between tracking and detection in
a more entangled way. The author save detection results and transform as new
detector proposal to regress target positions in new frame. The most
interesting thing of this approach is that, given the assumption that the
target does not have large motion, the new regressed position and the proposal
corresponding to the same object. This means the pair of bounding box
automatically form a tracklet without any additional association efforts.

Multi-face tracking with deep learning As a special case of multi-target
tracking, multi-face tracking also benefits from the overall advance in object
detection. In [zhang2016tracking], the author proposed a multi-stage approach
to realize multi-face tracking in the wild. The approach first employed a shot
detector and face detector to extract face bounding boxes within small shots.
Tracklets are then formed through the similirary comparison of face appearance
and positions. Improved Triplet loss [schroff2015facenet] are conducted to
generate better face embedding feature for tracklets linking. [lin2018prior]
provided a multi-model and multi-stage based multi-face tracking solution. In
order to generate better face tracklets, the author combined multiple body
parts detector to sustain face detection in the scene of large pose, occlusion
and low resolution. Then in the tracklets linking stage, recursive association
are conducted to finetune the tracking results and Gaussian Process model is
used to detect and re-assign outlier tracklets. The additional body parts
detection model and recursive association process enrich the tracklets
information and thus achieve the state-of-the-art performance.

##  3 Pipeline Overview

The proposed FAR tracker is based on the idea of two stage tracking. Before
diving into details of each stage, we first review the overall tracking
pipeline. The input to the tracker is a video, represented by a consecutive
sequence of $T$ frames, $\\{F_{i}\\}_{t=1}^{T}$. The tracker is expected to
output a series of face tracks $\\{S_{i}\\}$ that capture the faces of
different persons in multiple frames. Here one face tracks is represented by a
series of rectangular face bounding boxes $B_{i}^{t}$ in different frames with
the same track ID $i$. Each bounding box is represented by the image space
coordinates of its top-left corner $(x_{i}^{t},y_{i}^{t})$, heigth
$h_{i}^{t}$, and width $w_{i}^{t}$.

Our tracker operates with a cascade of two stages, as illustrated in Fig. 1.
The first stage is called _tracklet generator_ , in the sense that it focuses
on generating tracklets from the video frames. Tracklets $L_{j}$ are short-
term tracks that are consecutive in time. In particular, it encapsules a
series of face bounding boxes of the same person in several consecutive
frames. The tracklet genreator simultaneously perform bounding box detection
and tracking. At a timestep $t$, it observes the current frame $F_{t}$. It
also carries the tracklets that are present in the previous frame. To extend
the tracklets, it tries to detect the current frame but with tracking
information absorbed from previous frame, we call this operation re-detection,
means detecting new frames with remembering old information. Re-detected faces
are used to extend the existing tracklets by inheriting their identity.
Existing tracklets failed to be re-detected will be released from the tracklet
generator. The newly detected bounding boxes will be considered as the start
of a new tracklet. The second stage is called _tracklet linker_. It also works
in an online manner to link tracklets into long-term tracks. The tracklet
linker extract face identity embedding for a newly formed tracklet and matches
it against the tracks in the linker’s memory. The matched tracklets are then
linked to the corresponding tracks and obtain the track ID. Non-match
tracklets will start as new tracks with their own new track ID. In the
following sections, we go through the details of the two stages.

##  4 Tracklet Generator

The trackelet generator is based on the idea of performing detection and
tracking simultaneously, by trying to re-detect the existing faces within
current frame. This idea is not new, a similar approach has been introduced in
[bergmann2019tracking] for the task of person bounding box tracking. The basic
motivation of this approach is that object detectors bounding box regression
has good capability to rectify the locations and shapes if giving a not
accurate initial proposal. Thus by reusing the previous frame’s detection
results as prior information, we can re-detect these objects and can propagate
the same track ID to the re-detected bounding boxes. In
[bergmann2019tracking], this idea is realized on a fasterRCNN [ren2015faster]
person detector trained on the COCO dataset [lin2014microsoft]. FasterRCNN is
by itself a two-stage detector architecture, which first generates proposals
then classifies each proposal. Thus its runtime will grow linearly with the
number of objects it is tracking, which is not desired. In this work, we
starting the design of tracklet generator from a single-stage face detector,
namely, single-short multibox detector (SSD) [liu2016ssd]. This way, the
runtime will not grow no matter how many faces the tracklet generator is
tracking. This is an ideal property especially in crowded scenes. But re-
purposing SSD for the task of tracklet generation with re-detection is not
straightforward. Below we describe how this idea can be implemented.

###  4.1 Tracking by Re-Detection with SSD

Unlike faster-RCNN, single-stage detectors do not have an explicit concept of
proposals. They operate by classifiying and rectifying the locations of a set
of predefined location hypothesis, _i.e._ , anchors. Another way to view
single-stage detectors, is that, they actually have a huge amount of sliding
window proposals. These proposals have their prior locations and shapes. The
network then, based on the image features it extracts, makes prediction for
each proposal. This provides a way to reinstate detected bounding boxes in the
detection process of the current frame.

Given the frame $F_{t}$ and we have detected several face bounding boxes
assigned to tracklets $\\{L_{j}^{t}\\}$. For frame $F_{t+1}$, we adopt the
following steps to re-detect the faces in frame $F_{t+1}$. For each bounding
box $B_{j}^{t}$ of the tracklet $\\{L_{j}^{t}\\}$, we predict its location and
shape at $t+1$ by function $\hat{B}_{j}^{t+1}=H(B_{j}^{t})$. The simplest form
of $H$ can be identity function, which means $\hat{B}_{j}^{t+1}=B_{j}^{t}$. Of
all the detector anchor candidates, the closest one to $\hat{B}_{j}^{t+1}$
will be the one have largest IoU with $\hat{B}_{t+1}^{j}$. In this way, we
obtained the seed anchor for re-detection. We directly look at the network
predictions the anchors matched to the $\\{B_{j}^{t}\\}$. There are two
possible outcomes for these anchors: a) the anchor is classified as a valid
detection. Then the re-detection is successful. The detected bound box is
considered the new appearance of the corresponding tracklet; b) the anchor is
classified as background. Then the re-detection has failed. The tracklet is
considered halted, and released from the tracklet generator. This process is
illustrated in Fig. 3 Though this process looks reasonable, one fundamental
question needs to be answered for it to work.

Figure 3: We show the re-detection results of a set of anchor boxes $A^{t}$
which are spatially closed. The corresponding bounding box $B^{t}$ is
indicated using the same color with the anchor box on the feature map. The
ground truth bounding box is represented using green color (view in color is
suggested).

Is the prediction on the matched anchor always reliable? This question is
critical to the success of the approach described above. To verify it, we
conduct experiments on a small set of images. We look at the predictions of
SSD on the anchors that are similar in terms of IoU to the ground truth
bounding boxes. We find these nearby anchors’ final prediction position all
pointing to the final target as shown in Fig. 3, the color pixels represents
neiboring anchors, the dot bounding box represents the re-detected target
position generated by the neighboring anchors. Quantitative, we demonstrate in
Fig. 4 how the initial shift of bounding box affect the ability correctly re-
detect a face target. Using the $w,h$ of the bounding box in $F_{i}$ as a
scale, we added multiple ratios of shift to the Image for re-detection. Fig. 4
shows the re-detection results. We found that the detector can correctly
localize the face position even with a 0.33 ratio of relative shift. This
means if two consecutive frames only has a small motion, we can accurately
track the faces in new frames. We can observe that, similar to their
counterpart in faster RCNN, the bounding box regressors in SSD also have
strong power to rectify anchors to produce tight bounding boxes of the faces.
This means if we start from the anchor that are close the predict shape and
location of a face, there is high chance of re-detecting this face and produce
precise localization.

With the above question answered, we can formally describe the workflow of our
tracklet generator as follow: With the above question answered, we can
formally describe the workflow of our tracklet generator as follow:

With the above question answered, we can formally describe the workflow of our
tracklet generator as follow:

(1) For the first frame $F_{0}$: run the SSD detector and get all face
detection results $\\{B_{i}^{0}\\}$. Create one tracklet ID for each bounding
box.

(2) Predict the locations and shape of the bounding boxes in frame $F_{1}$.
Find the closest anchors to $\\{B_{i}^{0}\\}$ by IoU-based bi-party matching.
Mark these anchors as seed anchors for re-detection.

(3) Run the SSD detector on frame $F_{1}$. Based on the prediction on the
marked anchors, decide whether to extend or halt the tracklet.

(4) Suppress all detection results that have high IoU to the tracked bounding
boxes.

(5) For remaining detection results without tracklet, create a new tracklet
for each of them.

(6) Repeat the steps 2-5 for the remaining frames one by one.

Figure 4: Detector anchor regression robustness test (a-d corresponds to shift
ratio of 0.1, 0.25, 0.33, 0.45). Red is ground truth, yellow is the shifted
box and green is the re-detected target position (view in color is suggested).

###  4.2 Motion Assisted Re-Detection

The simplest form of our tracklet generator predicts the location of boxes in
the next frame by identity mapping. This effectively relies on a static motion
assumption. When the video contains large motion, this assumption breaks and
it becomes easy for the tracklet generator to loose the target. Thus, it is
important to consider motion in bounding box prediction.

Luckily, motion prediction from two consecutive frames is a widely
investigated problem [ilg2017flownet, ilg2018occlusions]. Dense optical flow
algorithms provide pixel-wise displacement prediction between two frames,
which can be used to predict the bounding box movement. In this work we adopt
the state of the art optical flow network, PWCNet [sun2018pwc] to prediction
optical flow between two frames. It uses deep network with correlation layers
and warping layers to generate dense optical flow fields. With the flow field,
we predict a bounding box’s location and shape using the following steps:

Run dense flow estimation for a pair of input images $F_{t-1}$ and $F_{t}$ and
obtain the dense field, note as $P_{t-1,x},P_{t-1,y}$.

Obtain the ROI dense flow by cropping the dense flow field with current
tracked face bounding box $B_{j}^{t}$, $b_{x},b_{y}$ as the center of
$B_{j}^{t}$.

Average the cropped dense flow, obtain position change
$(\Delta{x},\Delta{y})$, this will be the difference to new bounding box
center.

Compute the average vertical change of left/right half, horizontal change of
upper/bottom half of the cropped dense flow, namely $P_{t-1,x|x<c_{x}}$,
$P_{t-1,x|x>c_{x}}$, $P_{t-1,y|y<c_{y}}$ and $P_{t-1,y|y>c_{y}}$, this will be
the left, right, top and bottom boundary change, noted as $\Delta b_{t}^{j}$.

Apply the boundary change to original bounding box, obtain the motion assisted
bounding box estimation $B_{j}^{t}+\Delta b_{t}^{j}$. And use the new
estimated bounding box to find seed anchor. We call this approach _motion
assisted tracklet generation_. In experiments, we empirically compare this
approach with the static motion based tracklet generator and find it
significantly improve the tracking performance.

Figure 5: Online feature bank updating for tracklet-track linking.
$\phi_{i}^{t}$ denotes identity embedding of track $i$ at time $t$. We store a
bank of the identity embeddings where each track has one feature item. When we
have a new tracklet, we either match it to an existing identity in the bank or
add a new item to the bank (view in color is suggested).

##  5 Online Linking of the Tracklets

Output of tracklet generator are short tracklets, which are sensitive to
occlusion and temporary disappearance of faces in videos. We design an online
tracklet linker module to group the tracklets into the output of the FAR
tracker, face tracks.

The linker carries a set of existing tracks $\\{S_{i}\\}$. Each track is
represented by an identity embedding $\phi(S_{i})$ produced by averaging the
output of a face embedding model $\phi$ on faces in this track. At a certrain
time step $t$, there are two possible states of a track: 1) this track is
linked to an active tracklet in the current frame $F_{t}$; 2) this track has
no active tracklet in the current frame. Fig. 5 illustrates the identity
linking process. For each new tracklet $L_{j}^{t}$ appearing in $F_{t}$, we
try to match it with tracks in state 2). To perform the matching, we first
extract the face identity embedding from the face in $L_{j}^{t}$, then we
calculate the pairwise match cost between $L_{j}^{t}$ and the tracks. Once a
track has a new tracklet linked to it, it goes to state 1) in the next frame.
By solving the bi-party matching problem we can link the matching tracklet to
the tracks. For those new tracklets that are not matched to any existing
track, generate a new track for each of them.

This tracklet linker handles the cases of occlusion and temporal disappearance
by linking tracklet across time. On the other hand it requires extracting face
embedding using an embedding model, usually a deep neural network like
[deng2019arcface], which adds to the runtime cost. However, this additional
cost can be minimal due to two reasons: 1) the linking is only performed for
newly created tracklets, once a tracklet is linked we do not need to extract
it embedding. 2) the extraction of face embedding for the track do not need to
be run on each frame, as in videos the appereance of faces usually change
slowly.

##  6 Experiments

We evaluate the proposed FAR tracker on two benchmark datasets: the Hannah
dataset [ozerov2013evaluating] and the Music Videos dataset
[zhang2016tracking]. The tracker is assessed in term of standard multiple
object tracking metrics, runtime speed, and metrics concerned by downstream
tasks such as video face recognition. We provide qualitative analysis of the
tracker’s ability in handling common nuance factors to face trackers.

###  6.1 Implementation details

In our implementation of FAR Tracker, we first employ a face detector which is
based on Resnet34 as backbone. Feature pyramid network [lin2017feature] is
adopted to enhance the detection performance. We customize the structure of
the detector to allow marking anchors for redetection. So input of the
detector will be an input image and a list of anchor indices. We choose
detection confidence score $0.3$ as the confidence threshold for re-detection
and $0.6$ as the confidence threshold for new detection. In motion assisted
re-detection, we use PWCNet [sun2018pwc] to compute dense optical flow field.
The resolution of the image for flow computing is $256$x$320$ (height vs.
width). We choose this relatively small scale based on trade off between speed
and prediction accuracy. In the identity linking, we employ a face embedding
network [deng2019arcface] which generates 128-dimensional feature embedding.
We threshold the euclidean distance between embedding on $0.97$ in the
tracklet linker.

###  6.2 Dataset

Hannah dataset [ozerov2013evaluating] is a public dataset originally from the
Hollywood movie Hannah and her sisters. There are in total 153825 frames and
245 shots. As a muti-face tracking dataset, it contains 254 people and 53 of
which are named characters. There are 2002 tracks annotated, ranging from 1 to
500 frames in length. In the Hannah dataset, face bounding box contains ears
in the horizontal direction, and the bounding box starts from center of
forehead and ends at center of the jaw in the vertical direction.

Music video dataset [zhang2016tracking] is a widely used face tracking
benchmark with $8$ high resolution music videos from YouTube. The videos are
of high quality, the scenes are mainly music video or live concert recordings.
This dataset is challenging for it contains a lot of fast moving humans,
facial appearance changes, crowd occlusions.

IQIYI single face tracking dataset is a face tracking benchmark we create on
top of the IQIYI Celebrity Video dataset [liu2018iqiyi]. IQIYI Celebrity Video
dataset consists of 600K video clips. The dataset contains 5,000 labeled
celebrities and 84,759 unknown identities (distractors for face recognition
task) and the each video clip is labeled with one identity from the 5,000
celebrities. The dataset only clip-level label and no bounding box annotation.
We manually annoate the bounding boxes in 2595 videos from the evaluation set
of IQIYI datasets using Amazon Mechanical Turk. In the annotation process, we
provide a pre-computed bounding box in WiderFace [yang2016wider] format.
Annotators were asked to to adjust bounding box location, add missing bounding
box and remove false detection. Note that in this dataset each video contains
exactly one person.

###  6.3 Evaluation Metric

We treat multi-face tracking as a special case of multiple object tarcking
(MOT), so the standard MOT metrics including Recall, Precision, F1, MOTA,
MOTP, MT, Fragments are used for evaluation. Because one of the major
downstream tasks for face tracking is face recognition, we design an
additional evaluation metrics, called identity miss-match (ID-missmatch). It
counts how many times in the tracking results faces from different persons are
falsely tracked in one track. This metric is important for face recognition in
the sense that the video recognition usually assumes faces in a track are from
the same person. For IQIYI dataset, since each video only contains one
person’s faces, we use the Recall, Precision, MT/PT/ML/GT, and MOTA, MOTP as
the evaluation metrics.

###  6.4 Evaluation and analysis

Baseline approach We implemented the widely used DeepSORT tracker
[Wojke2018deep] as our baseline approach. It is based on the standard tracking
by detection scheme. As shown in [wojke2017simple], the performance of the
DeepSORT tracker is high dependent on the quality of its object detector and
re-ID features. We equip this tracker with state of the art image based face
detector trained on the WIDERFace [yang2016wider] dataset and the state of the
art face embedding [deng2019arcface] for re-identification features. This
makes it a strong baseline to compete against. This tracker is called deep
face SORT (DFS) in experimental results.

Figure 6: Fig. $(a)$ shows the different bounding box style of Hannah dataset and WiderFace, WiderFace have tight bounding box covering the face region while Hannah style have loosen bounding box in horizontal including ears but shorter in vertical which starts from forehead center to haw center. Fig. $(b)$ illustrates the rare case where shot changes but new face appear at the same region, this leads to identity miss match in tracking Table 1: Multi-face tracking evaluation results | Method | Recall$\uparrow$ | Precision$\uparrow$ | F1$\uparrow$ | MT/GT$\uparrow$ | ID missmatch$\downarrow$ | Frag$\downarrow$ | MOTA$\uparrow$ | MOTP$\uparrow$ | Online
---|---|---|---|---|---|---|---|---|---|---
Hannah | DFS [Wojke2018deep] | 0.666 | 0.822 | 0.732 | 499/864 | 94 | 807 | 0.27 | 0.74 | Y
FAR | 0.894 | 0.687 | 0.776 | 604/864 | 15 | 1349 | 0.49 | 0.75 | Y
FAR-flow | 0.894 | 0.686 | 0.775 | 607/864 | 15 | 1324 | 0.50 | 0.75 | Y
FAR-link | 0.842 | 0.820 | 0.830 | 481/864 | 35 | 602 | 0.66 | 0.78 | Y
Musicvideo | Siamese [zhang2016tracking] | 0.715 | 0.894 | 0.795 | 18/44 | 986 | 2512 | 0.623 | 0.640 | N
SymTriplets [zhang2016tracking] | 0.718 | 0.897 | 0.798 | 19/44 | 699 | 2563 | 0.628 | 0.643 | N
Prior-Less [lin2018prior] | 0.817 | 0.902 | 0.853 | 32/44 | 624 | 1645 | 0.692 | 0.860 | N
DFS [Wojke2018deep] | 0.816 | 0.873 | 0.843 | 38/44 | 362 | 6548 | 0.571 | 0.617 | Y
FAR | 0.788 | 0.810 | 0.798 | 36/44 | 54 | 7081 | 0.580 | 0.683 | Y
FAR-flow | 0.774 | 0.802 | 0.797 | 41/44 | 47 | 6985 | 0.601 | 0.699 | Y
FAR-link | 0.792 | 0.835 | 0.812 | 35/44 | 208 | 4038 | 0.668 | 0.727 | Y
IQIYI | SiamMask [wang2019fast] | 0.354 | 0.368 | - | 473/3001 | - | - | -0.253 | 0.739 | N
DFS [wojke2017simple] | 0.927 | 0.898 | - | 1913/3001 | - | - | 0.825 | 0.873 | Y
FAR | 0.980 | 0.981 | - | 2646/3001 | - | - | 0.964 | 0.91 | Y

Bounding box transformation Both Musicvideo and Hannah dataset have their own
annotation styles for face bounding boxes. In face detection scenario, many of
current face detector follow the annotation styles induced by the WIDERFace
[yang2016wider] data. In this format, the bounding box tightly contains
forehead, cheek, chin. While in Hannah dataset, the bounding box contain
looser bounding box including ears, shorter bounding box in the middle of
forehead and middle of jaw. In Musicvideo dataset, the bounding box is tighter
than WiderFace format in forehead boundaries. The difference between
annotation styles is illustrated in figure 6 In IoU based evaluation, which is
standard for MOT benchmarks, this difference leads to undesired performance
drop. To make fair comparison between algorithms, in evaluaion we obtain
landmark localization results of each output face bounding box and rebuild the
bounding boxes according to the dataset specification.

Results on the Hannah dataset Table 1 shows the Hannah dataset evaluation
results based on baseline DFS and our proposed FAR tracklets generation,
optical flow assisted tracklets and tracklets& identity linking results. We
found that even the raw tracklets generation results have better performance
compared to the DFS methods. We can see that with a powerful face detector and
face embedding extractor, the DFS can have a relative good face detection
results as shown by the precision and recall number. While the approach have
much larger miss matching number compared with the FAR series methods. We also
see that applying optical flow did not change the original detection too much,
but adding flow based motion compensation, the mostly tracked number and MOTP
both increased, This means the optical flow do help the FAR tracker find more
accurate target positions. We find that adding the identity link increase the
miss match number, which is acceptable as merging tracklets will introduce
miss matches due to false positive verification. But the MOTA and fragments
metric have obvious improvements. We saw the significant improvement in
identity miss match compared with DFS [Wojke2018deep], this means within a
given tracklet, the FAR tracker can provide a set of face images with less
incorrect id. This is very important in face recognition as outlier identity
will generate different feature and harm the recognition. It is mainly
contributed by the Re-detection mechanism of FAR tracker, the given dataset
have a relative high frame rate so the assumption that most consecutive frames
have small motion stands. This also explained why the flow assisted motion do
not leads to much boost in detection quality. During the evaluation we also
found that the FAR tracker is robust to shot change most of the time, it is
rare to have a consecutive face across shots. But we do find some error cases,
as shown in Fig. 6$(b)$. This might be even problematic when there are crowds
of face where it is easy to have different faces at same location across
frames. Finding the correct corresponding faces in the challenging case will
be the focus of our future work.

Results on the Musicvideo dataset We conduct the evaluation on MusicVideo
dataset. Musicvideo datasets are used for evaluation in [zhang2016tracking,
lin2018prior]’s work. We tested DFS [Wojke2018deep], and FAR series methods on
the MusicVideo datasets and show the result in table 1.

As shown in table 1, compared to the state of the art multi-face tracker which
multi-stage detection and tracklets association are conducted, our online
methods provide comparable tracking results. In [lin2018prior]’s work,
multiple body parts detection leads to better detection quality as especially
in recall metrics. Our proposed FAR tracker shows much less miss match while
more fragments. This is mainly due to the global multi-step tracklets
association mentioned in [lin2018prior, zhang2016tracking] which tends to find
most tracklets and merge to the same tracklets. In our proposed approaches, we
use a very strict tracklets merging threshold. This results in a large
fragment number in tracking, because we are trying to make sure multiple
identity will not merge into one tracklet as we believe this is a very
important requirement in video based face recognition. It is also important to
notice that additional information is useful in improving the recall of video
face detection.

IQIYI single face tracking evaluation To evaluate our approach, we compared
the proposed FAR tracker with DFS [Wojke2018deep] and the state of the art
single object tracker SiamMask [wang2019fast]. Table 1 shows the evaluation
results on IQIYI dataset. The SiamMask [wang2019fast] single object tracker
fails in most metrics. The main reason is lack of domain knowledge in face
detection. Due to the task type, there is no multi-face linking problems need
to be considered by DFS. This show the detector’s basic performance in video
detection. FAR tracker is running without flow assistance and identity
linking. To our surprise, the FAR tracker can pick up most of the detection
missed by the detector, improving recall from 92.7% to 98.0%. This also help
the tracker generate more Mostly tracked tracklets. We also see that
benefiting from the prior knowledge of face bounding box in previous frame,
precision improves from 89.8% to 98.1% and mota/motp is boosted from
0.825/0.873 to 0.964/0.91 respectively.

####  6.4.1 Runtime analysis

One of the main advantage of FAR is that it is an online multiple face
tracker. To see if it is possible to make the FAR tracker running in realtime,
we conduct a breakdown of the components runtime. All the benchmark is based
on Nvidia V100 graphic card. With a input image size 1024 by 1024, the main
components detection, tracker update, flow computing, mbedding computing take
126 $ms$, 9$ms$, 15$ms$, 14 $ms$ respectively. We can see that the detection
take up the major part of the runtime. The current pipeline can achieve around
6 FPS in tracking. Note that tracklets generation module is independent to the
based detector structures. It can be applied to any single-stage detector. All
the current components can be replaced with a faster module in order to
achieve realtime performance.

##  7 Conclusion

In this paper, we present a two-stage face tracker that is able to generate
fast, accurate and robust tracking results from videos. By applying the
tracking by detection idea to single shot face detectors, its tracklet
generator runs at constant speed w.r.t. the number of faces it is tracking.
With motion assisted redetection, the tracker is robust to large motion in
videos. A simple but effective tracklet linkers links the tracklets into
tracks in an online manner. In experiments, our FAR tracker achieves state-of-
the-art accuracy and is robust to multiple nuance factors. Particularly, it is
able detect and track the faces where the image based detector in three-stage
trackers will fail. Its high accuracy and robustness validates the
effectiveness of our designing principle: simultaneous performing detection
and tracking leads to better detection and better tracking.


# NerfAcc: A General NeRF Acceleration Toolbox.

Ruilong Li
University of California, Berkeley
<EMAIL_ADDRESS>
&Matthew Tancik
University of California, Berkeley
<EMAIL_ADDRESS>
&Angjoo Kanazawa
University of California, Berkeley
<EMAIL_ADDRESS>
###### Abstract

We propose NerfAcc, a toolbox for efficient volumetric rendering of radiance
fields. We build on the techniques proposed in Instant-NGP [1], and extend
these techniques to not only support bounded static scenes, but also for
dynamic scenes and unbounded scenes. NerfAcc comes with a user-friendly Python
API, and is ready for plug-and-play acceleration of most NeRFs. Various
examples are provided to show how to use this toolbox. Code can be found here:
https://github.com/KAIR-BAIR/nerfacc.

##  1 Introduction

Neural Radiance Fields (NeRFs) [2] are a groundbreaking technique for 3D
representation, which model the geometry and view-dependent appearance of the
scene by using a multi-layer perceptron (MLP). In the past two years, they
have been proven to be quite powerful in many downstream applications in 3D
such as static/dynamic scene reconstruction [3, 4, 5, 6], relighting [7, 8, 9,
10, 11] and content generation [12, 13, 14]. Using differentiable volumetric
rendering [15], a radiance field can be optimized / trained from calibrated
RGB images alone. However, with the vanilla MLP NeRF, such an optimization
process usually takes days to converge on a single scene, and tens of seconds
to render a single image.

Recently, many works have demonstrated that the NeRF optimization and
inference process can be drastically accelerated by incorporating explicit 3D
voxels into the radiance field representation [16, 17, 1, 18]. However, voxel-
based radiance fields are less flexible than the MLP-based representation as
they are only applicable to a _single_ , _static_ scene. Applications to
dynamic scene reconstruction [5, 19, 20] and generalization across multiple
scenes [21, 22, 7] have not benefited yet from those methods.

Despite all variations of the radiance field representations and applications,
most of them share the same volumetric rendering process – cast a ray into the
space and accumulate the colors along the ray. However, not too much attention
has been spent on exploring an efficient volumetric rendering process that is
easily applicable to any downstream applications. A few works such as
Plenoctrees [16] and Instant-NGP [1] incorporate some techniques for efficient
volumetric rendering such as skipping empty and occluded areas, but are highly
customized for _bounded static_ scenes and use specialized CUDA
implementations making it difficult to extend to other applications.

In this work, we propose a toolbox, NerfAcc for “NeRF Accelration”, which
focuses on efficient volumetric rendering of radiance fields. We build on the
techniques proposed in Instant-NGP [1], and extend those techniques to support
not only bounded static scenes, but also the dynamic scenes and unbounded
scenes. We also provide a user friendly Python API written with PyTorch [23],
which is ready for plug-and-play acceleration of most NeRFs. Various examples
are included to show how to use the NerfAcc toolbox to accelerate existing
NeRF methods.

With the NerfAcc toobox,

  * •

The vanilla NeRF model [2] with an 8-layer MLP can be trained to better
quality (+0.5 PSNR) in 1 hour rather than $1\sim 2$ days as in the paper.

  * •

The Instant-NGP NeRF [1] model can be trained to equal quality with 4.5
minutes in Python, comparing to the official pure CUDA implementation.

  * •

The D-NeRF [19] model for dynamic scenes can be trained on 1 hour rather than
2 days as in the paper, and with better quality (+2.0 PSNR).

##  2 Design

In NerfAcc, the volumetric rendering pipeline is broken down into two steps,
_ray marching_ and _differentiable rendering_. Here we dive into each step to
introduce some techniques that we incorporated for efficient volumetric
rendering.

###  2.1 Ray Marching

Ray marching is the process of casting a ray through the scene and generating
discrete samples along the ray. As those samples will later be evaluated by a
radiance field, which is usually the main bottleneck of the entire pipeline,
efficiency can be achieved by reducing the number of samples as much as
possible during ray marching. As shown in the volumetric rendering Equation 1,
a sample would have little contribution to the final image color
$C(\mathbf{r})$, if it has either low opacity $\alpha_{i}$ or low
transmittance $T_{i}$. In other words, during ray marching we can safely skip
samples that are in the empty or occluded regions as they do not contribute to
the final image.

$\displaystyle
C(\mathbf{r})=\sum_{i=1}^{N}T_{i}\alpha_{i}\mathbf{c}_{i},\quad\text{where
}T_{i}=\text{exp}(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}),~{}\alpha_{i}=(1-\text{exp}(\sigma_{i}\delta_{i})),$
(1)

#### Pruning Empty and Occluded Regions.

However during training, finding the empty and occluded regions is not
straightforward. In NerfAcc, we incorporate the idea of using an _Occupancy
Grid_ from Instant-NGP [1] 111See Instant-NGP Appendix E.2 for details, in
which a binary grid is cached and updated during training to store which areas
in the scene are empty (e.g., $\alpha_{i}<1\mathrm{e}{-2}$). We skip occluded
regions by terminating the marching early based on the transmittance (e.g.,
$T_{i}<1\mathrm{e}{-4}$) along the ray, similar to Instant-NGP [1] and
PlenOctrees [16]. Note, for skipping occluded regions, we need to know the
precise density of the samples along the ray to compute transmittance. So a
user-defined radiance field (e.g., MLP NeRF or Instant-NGP NeRF) is evaluated
to get the density, but _with gradients disabled_ to minimize the computation.

#### GPU Memory Efficiency.

One way to apply the aforementioned pruning during marching, is to first over-
generate samples for each ray, and then selectively keep the samples that
matter. This can be done efficiently in Python with parallelization across all
the samples, but is not memory friendly as it can over allocate the GPU memory
that is needed in the first place. For a scene that has dense surface such as
the _Lego_ scene in the NeRF-Synthetic dataset, the pruning process can get
rid of $98\%$ of the samples when training converges. To minimize the GPU
usage, NerfAcc prunes the empty space while marching, resulting in samples
only in the non-emtpy space. This changes the parallelization from samples to
rays, which is slightly less efficiently but allows one to use significantly
less GPU memory to generate much denser samples.

#### GPU Parallelization.

There are in general two options for GPU parallelization: across rays or
across samples. In most of the cases, the total number of valid samples in the
scene is much larger than the total number of rays. As a result, we would
prefer to parallelize across samples whenever possible. In NerfAcc, we
minimize operations that are parallelized across rays. Operations that are
independent of rays, such as computing sample opacity from sample density, are
done with parallelization across samples.

#### Scene Contraction for Unbounded Scene

As the _Occupancy Grid_ from Instant-NGP [1] only supports bounded
scenes222Instant-NGP proposes multi-scale grids to support a really large
scenes but is still bounded., we incorporate the scene contraction idea333See
Eq. 10 in the Mip-NeRF 360 paper from Mip-NeRF 360 [24] to support unbounded
scenes. When querying into the occupancy grid, a non-linear function is
applied to the coordinates to map the unbounded space into a finite grid. With
this single change, NerfAcc is able to reconstruct bounded scenes as well as
unbounded scenes.

###  2.2 Differentiable Rendering

Differentiable rendering is the process of accumulating the color of the
samples along the rays, into pixel colors. In this step we make use of the
user-defined radiance field in an differentiable way – all the outputs of the
radiance field will receive gradients from pixel supervisions. We follow
Equation 1 to perform the accumulation. Any sample attributes, not just
colors, are supported and can be rendered in a differentiable way by NerfAcc,
such as depth and opacity. We parallelize across all the samples in this step
to maximize the GPU utilization.

##  3 Plug-and-Play

As the idea of NerfAcc is to perform efficient ray marching and volumetric
rendering, it can work with any user-defined radiance field. To plug the
NerfAcc rendering pipeline into the code and enjoy the acceleration, user only
needs to define two functions (_sigma_fn_ , _rgb_sigma_fn_) with the
customized radiance field. Blow is an example of plug-and-play:

⬇

1 import torch

2 from torch import Tensor

3 import nerfacc

4

5 radiance_field = ... # network: a NeRF model

6 optimizer = ... # network optimizer

7 rays_o: Tensor = ... # ray origins. (n_rays, 3)

8 rays_d: Tensor = ... # ray normalized directions. (n_rays, 3)

9

10 def sigma_fn(

11 t_starts: Tensor, t_ends:Tensor, ray_indices: Tensor

12 ) -> Tensor:

13 """ Query density values from a user-defined radiance field.

14 :params t_starts: Start of the sample interval along the ray.

15 :params t_ends: End of the sample interval along the ray.

16 :params ray_indices: Ray indices that each sample belongs to.

17 :returns The post-activation density values.

18 """

19 t_origins = rays_o[ray_indices] # (n_samples, 3)

20 t_dirs = rays_d[ray_indices] # (n_samples, 3)

21 positions = t_origins + t_dirs * (t_starts + t_ends) / 2.0

22 sigmas = radiance_field.query_density(positions)

23 return sigmas # (n_samples, 1)

24

25 def rgb_sigma_fn(

26 t_starts: Tensor, t_ends: Tensor, ray_indices: Tensor

27 ) -> Tuple[Tensor, Tensor]:

28 """ Query rgb and density values from a user-defined radiance field.

29 :params t_starts: Start of the sample interval along the ray.

30 :params t_ends: End of the sample interval along the ray.

31 :params ray_indices: Ray indices that each sample belongs to.

32 :returns The post-activation rgb and density values.

33 """

34 t_origins = rays_o[ray_indices] # (n_samples, 3)

35 t_dirs = rays_d[ray_indices] # (n_samples, 3)

36 positions = t_origins + t_dirs * (t_starts + t_ends) / 2.0

37 rgbs, sigmas = radiance_field(positions, condition=t_dirs)

38 return rgbs, sigmas # (n_samples, 3), (n_samples, 1)

39

40 # Efficient Raymarching: Skip empty and occluded space.

41 packed_info, t_starts, t_ends = nerfacc.ray_marching(

42 rays_o, rays_d, sigma_fn=sigma_fn, near_plane=0.2, far_plane=1.0,

43 early_stop_eps=1e-4, alpha_thre=1e-2,

44 )

45

46 # Differentiable Volumetric Rendering.

47 # colors: (n_rays, 3). opaicity: (n_rays, 1). depth: (n_rays, 1).

48 color, opacity, depth = nerfacc.rendering(

49 rgb_sigma_fn, packed_info, t_starts, t_ends

50 )

51

52 # Optimize the radience field.

53 optimizer.zero_grad()

54 loss = F.mse_loss(color, color_gt)

55 loss.backward()

56 optimizer.step()

##  4 Example Usages

We provide four example usages of NerfAcc on different radiance field
representations, on different type of scenes, including dynamic scenes and
unbounded scenes. The results below indicate that NerfAcc is a universal
acceleration tool for various NeRFs. Our experiments are all conducted on a
single NVIDIA TITAN RTX GPU.

###  4.1 Vanilla MLP NeRF in 1 hour.

In this example we trained an 8-layer-MLP with the same structure as the one
in the original NeRF [2] method. We used the train split for training and test
split for evaluation as in the NeRF paper. The training memory footprint is
about 10GB. The results on the NeRF-Synthetic dataset is shown here:

| Lego | Mic | Materials | Chair | Hotdog | Ficus | Drums | Ship | Mean
---|---|---|---|---|---|---|---|---|---
NeRF [2] (PSNR: $\sim$days) | 32.54 | 32.91 | 29.62 | 33.00 | 36.18 | 30.13 | 25.01 | 28.65 | 31.00
Ours (PSNR: $\sim$1 hr) | 33.69 | 33.76 | 29.73 | 33.32 | 35.80 | 32.52 | 25.39 | 28.18 | 31.55
Ours (Training time) | 58min | 53min | 46min | 62min | 56min | 42min | 52min | 49min | 52min

Table 1: Performance on NeRF-Synthetic dataset.

#### Note.

The vanilla NeRF paper uses two MLPs for course-to-fine sampling. Instead here
we only use a single MLP with more samples (1024). Both methods accomplish the
same goal of dense sampling around the surface. NerfAcc inherently skips
samples away from the surface so we can simply increase the number of samples
with a single MLP, to achieve the same goal with the coarse-to-fine sampling,
without run-time or memory issue.

###  4.2 Instant-NGP in 4.5 minutes.

In this example we trained a Instant-NGP NeRF model on the NeRF-Synthetic
dataset. We adopt the same settings proposed in the Instant-NGP paper, which
uses train split for training and test split for evaluation. The training
memory footprint is about 3GB.

PSNR | Lego | Mic | Materials | Chair | Hotdog | Ficus | Drums | Ship | Mean
---|---|---|---|---|---|---|---|---|---
Instant-NGP [1] ($\sim$ 4.5 mins) | 35.87 | 36.22 | 29.08 | 35.10 | 37.48 | 30.61 | 23.85 | 30.62 | 32.35
Ours ($\sim$ 4.5 mins) | 35.50 | 36.16 | 29.14 | 35.23 | 37.15 | 31.71 | 24.88 | 29.91 | 32.46

Table 2: Performance on NeRF-Synthetic dataset.

#### Note.

The Instant-NGP paper makes use of the alpha channel in the images to apply
random background augmentation during training. For fair comparison, we rerun
their code with a constant white background during both training and testing.
Also it is worth to mention that we didn’t strictly follow the training recipe
in the Instant-NGP paper, such as the learning rate schedule etc, as the
purpose of this benchmark is to showcase instead of reproducing the paper.

###  4.3 D-NeRF in 1 hour.

In this example we trained a 8-layer-MLP for the radiance field and a 4-layer-
MLP for the warping field, (similar to the T-NeRF model in the D-NeRF paper)
on the D-NeRF dataset. We used train split for training and test split for
evaluation. The training memory footprint is about 11GB.

PSNR | Bouncing balls | Hell warrior | Hook | Jumping jacks | Lego | Mutant | Standup | Trex | Mean
---|---|---|---|---|---|---|---|---|---
D-NeRF [19] ($\sim$ days) | 38.93 | 25.02 | 29.25 | 32.80 | 21.64 | 31.29 | 32.79 | 31.75 | 30.43
Ours ($\sim$ 1 hr) | 39.49 | 25.58 | 31.86 | 32.73 | 24.32 | 35.55 | 35.90 | 32.33 | 32.22
Ours (Training time) | 37min | 52min | 69min | 64min | 44min | 79min | 79min | 39min | 58min

Table 3: Performance on D-NeRF Synthetic dataset.

#### Note.

The Occupancy Grid used in this example is shared by all the frames. In other
words, instead of using it to store the opacity of an area at a single
timestamp, Here we use it to store the maximum opacity at this area over all
the timestamps. It is not optimal but still makes the rendering very
efficient.

###  4.4 Unbounded Scene in 20 minutes.

Here we trained an Instant-NGP NeRF on the Mip-NeRF 360 dataset. We used the
train split for training and test split for evaluation. The training memory
footprint is about 8GB.

PSNR | Garden | Bicycle | Bonsai | Counter | Kitchen | Room | Stump | Mean
---|---|---|---|---|---|---|---|---
NeRF++ [25] ($\sim$ days) | 24.32 | 22.64 | 29.15 | 26.38 | 27.80 | 28.87 | 24.34 | 26.21
Mip-NeRF 360 [24] ($\sim$ days) | 26.98 | 24.37 | 33.46 | 29.55 | 32.23 | 31.63 | 28.65 | 29.55
Ours ($\sim$ 20 mins) | 25.41 | 22.97 | 30.71 | 27.34 | 30.32 | 31.00 | 23.43 | 27.31
Ours (Training time) | 25min | 17min | 19min | 23min | 28min | 20min | 17min | 21min

Table 4: Performance on Mip-NeRF 360 dataset.

#### Note.

Even though we are comparing with NeRF++ and Mip-NeRF 360, the model and
losses are different. There are many ideas from those papers that could be
helpful for the performance. We didn’t adopt them in these experiments as they
are tangential to the library.

##  5 Conclusion

We have demonstrated a general acceleration toolbox, NerfAcc, for accelerating
various NeRFs in different applications. It is built on top of the ideas from
Instant-NGP [1], Plenoctrees [16] and Mip-NeRF 360 [24] for efficient
volumetric rendering. With NerfAcc, various NeRF models can be trained in 1
hour with better performance. We provide full Python API for NerfAcc, which
makes it ready for plug-and-play in future research.

## References

  * [1] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  arXiv preprint arXiv:2201.05989, 2022.
  * [2] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [3] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  arXiv preprint arXiv:2106.10689, 2021.
  * [4] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  * [5] Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhofer, Jurgen Gall, Angjoo Kanazawa, and Christoph Lassner.  Tava: Template-free animatable volumetric actors.  arXiv preprint arXiv:2206.08929, 2022.
  * [6] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao.  Animatable neural radiance fields for modeling dynamic human bodies.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14314–14323, 2021.
  * [7] Z Kuang, K Olszewski, M Chai, Z Huang, P Achlioptas, and S Tulyakov.  Neroic: Neural object capture and rendering from online image collections, 2021.
  * [8] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi.  Neural reflectance fields for appearance acquisition.  arXiv preprint arXiv:2008.03824, 2020.
  * [9] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  ACM Transactions on Graphics (TOG), 40(6):1–18, 2021.
  * [10] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7495–7504, 2021.
  * [11] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely.  Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5453–5462, 2021.
  * [12] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 867–876, 2022.
  * [13] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [14] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu.  Gnerf: Gan-based neural radiance field without posed camera.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6351–6361, 2021.
  * [15] Nelson Max.  Optical models for direct volume rendering.  IEEE Transactions on Visualization and Computer Graphics, 1(2):99–108, 1995.
  * [16] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752–5761, 2021.
  * [17] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  arXiv preprint arXiv:2112.05131, 2021.
  * [18] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  arXiv preprint arXiv:2203.09517, 2022.
  * [19] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [20] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  * [21] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [22] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  * [23] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [24] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [25] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv preprint arXiv:2010.07492, 2020.