# Movable Antenna-Enabled Full-Duplex Wireless

Jingze Ding, , Zijian Zhou, , Wenyao Li, Chenbo Wang, , Lifeng Lin, and Bingli
Jiao This work was jointly supported by the Science and Technology Project of
Guangzhou under Grants 202206010118 and 2023B04J0011. _(Corresponding author:
Zijian Zhou.)_ J. Ding, Z. Zhou, W. Li, C. Wang, L. Lin, and B. Jiao are with
the Department of Electronics, Peking University, Beijing 100871, China
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>{wcb15, linlifeng, jiaobl}@pku.edu.cn).

###### Abstract

Movable antenna (MA) provides an innovative way to arrange antennas that can
contribute to improved signal quality and more effective interference
management. This method is especially beneficial for full-duplex (FD)
wireless, which struggles with self-interference (SI) that usually overpowers
the desired incoming signals. By dynamically repositioning transmit/receive
antennas, we can mitigate the SI and enhance the reception of incoming
signals. Thus, this paper proposes a novel MA-enabled point-to-point FD
wireless system and formulates the minimum achievable rate of two FD
terminals. To maximize the minimum achievable rate and determine the near-
optimal positions of the MAs, we introduce a solution based on projected
particle swarm optimization (PPSO), which can circumvent common suboptimal
positioning issues. Moreover, numerical results reveal that the PPSO method
leads to a better performance compared to the conventional alternating
position optimization (APO). The results also demonstrate that an MA-enabled
FD system outperforms the one using fixed-position antennas (FPAs).

###### Index Terms:

Movable antenna (MA), full-duplex (FD), fixed-position antenna (FPA),
projected particle swarm optimization (PPSO).

##  I Introduction

Wireless communications are experiencing a dramatic increase in the demand for
never-enough capacity and thus, the spectrum resources become ever more
precious towards the sixth generation (6G). To rise to this challenge, full-
duplex (FD) wireless [1, 2] stands out as a potential for doubling spectral
efficiency by enabling the simultaneous exchange of information across the
same frequency band. However, the presence of self-interference (SI) that is
leaked from the transmitter’s antenna poses a troublesome impediment to
practical applications of this technology.

The prior literature has identified three basic strategies to mitigate the SI,
i.e., antenna separation, analog cancellation, and digital cancellation. The
study [3] categorizes antenna separation as a form of passive cancellation,
owing to its reliance on the fixed spatial configurations of antennas, such as
separate distance, orientation, and polarization [4, 5]. On the other hand,
the analog and digital cancellations are termed active cancellation and have
been extensively studied. For better SI cancellation, researchers have delved
into the realm of antenna design, exploring how it can play a more dynamic
role. This pursuit leads to the development of the antenna selection (AS)
method [6], a technique designed to exploit the potential of spatial degrees
of freedom (DoFs). In [7], the authors investigate the impact of increasing
the number of AS elements and observe that, although such an increase can
improve the system performance, it also raises the cost of implementation due
to the requisite rise in radio frequency channels. Furthermore, placing AS
elements in close proximity risks antenna coupling, which could lead to
reduced antenna efficiency.

Considering the limitations of AS, the authors in [8, 9, 10, 11] first
describe the concept of fluid antenna system (FAS), which takes advantage of
liquid-based antennas with flexible antenna architectures. Specifically, they
explore the use of a single antenna that can be moved to various preset
locations along a linear axis. This approach has shown that the FAS can reduce
interference and enhance the signal of interest (SoI) far better than
traditional systems that rely on a limited array of antennas to perform
maximum ratio combining. Further advancements are presented by the authors in
[12], where the FAS is expanded into a system aided by movable antennas (MAs).
These antennas can be precisely positioned using stepper motors, allowing for
movement in a three-dimensional space, which could significantly improve
signal reception. The study [7] provides the modeling and performance analysis
of these MA-enabled wireless communication systems, considering various
channel models to give a comprehensive understanding of their capabilities. In
multiple-input multiple-output (MIMO) systems, the study [13] investigates how
the MA technology can be utilized. It shows that by alternatively optimizing
the transmit covariance matrix and the position of each MA while keeping other
variables fixed, the system’s data rate can be significantly increased. The
challenge of multiuser uplink communication is addressed in [14, 15]. These
studies explore strategies for multiple single-antenna users who communicate
with a base station equipped with an MA array, aiming to minimize overall
transmission power while maximizing the minimum achievable rate of all users,
thereby ensuring a robust connection.

The advantages of the MA technology can be summarized as its capability to
dynamically leverage spatial DoFs through the superposition of
transmit/receive signals across various wireless channels [7, 12, 13, 14, 15].
This characteristic is crucial in reducing interference and enhancing the SoI,
making the MA especially well-suited for FD wireless applications. To the best
of our knowledge, this technology has not yet been considered in FD wireless
systems. Motivated by these considerations, we propose a novel point-to-point
FD system assisted by the MAs. In particular, our main contributions to this
work are threefold. First, we formulate the optimization problem by maximizing
the minimum achievable rate of two FD terminals with the aid of the MAs. Next,
to circumvent an undesired sub-optimal positioning, we introduce projected
particle swarm optimization (PPSO) as a superior alternative to the
conventional alternating position optimization (APO) approach. Last, we
provide simulation results that not only illustrate a better performance of
the PPSO method over traditional solutions but also substantiate the
enhancements of an MA-enabled FD system compared to systems with fixed-
position antennas (FPAs).

Notation: $a$, $\mathbf{a}$, and $\mathbf{A}$ denote a scalar, a vector, and a
matrix, respectively. ${\left(\cdot\right)^{\rm{T}}}$ and
${\left(\cdot\right)^{\rm{H}}}$ stand for transpose and conjugate transpose,
respectively. Besides, $\odot$ represents Hadamard product. We use
${\left[\mathbf{a}\right]_{i}}$ to denote the $i^{\mathrm{th}}$ element of the
vector $\mathbf{a}$. The circularly symmetric complex Gaussian (CSCG)
distribution with mean zero and variance $\sigma^{2}$ is represented by
$\mathcal{CN}\left(0,\sigma^{2}\right)$.

Figure 1: An MA-enabled point-to-point FD wireless system

##  II System Model

Figure 1 depicts an MA-enabled point-to-point FD wireless system that
comprises terminals A and B. Each terminal is equipped with a single transmit
antenna and a single receive antenna. These antennas can be facilitated by
stepper motors, allowing them to move within a two-dimensional region. The
transmit and receive positions of the MAs are denoted using Cartesian
coordinates, i.e.,
$\mathbf{t}_{p}=\left[x_{\mathrm{t}}^{p},y_{\mathrm{t}}^{p}\right]^{\mathrm{T}}\in\mathcal{C}_{\mathrm{t}}^{p}$
and
$\mathbf{r}_{q}=\left[x_{\mathrm{r}}^{q},y_{\mathrm{r}}^{q}\right]^{\mathrm{T}}\in\mathcal{C}_{\mathrm{r}}^{q}$,
where $\mathcal{C}_{\mathrm{t}}^{p}$ and $\mathcal{C}_{\mathrm{r}}^{q}$ are
the respective transmit and receive regions at terminals $p$ and $q$. Herein,
$p$ and $q$ belong to the set $\\{\mathrm{A},\mathrm{B}\\}$, signifying the
terminal labels. This notation will be consistently used in the following
derivations.

In this MA-enabled FD wireless system, the channel response relies on the
variable positions of the MAs. We thus characterize the channel coefficient as
a function of the transmit and receive MAs’ coordinates, i.e.,
${h_{pq}}\left({\mathbf{t}_{p}},{\mathbf{r}_{q}}\right)$. Assuming the
transmit power of the terminals is denoted by $P_{\mathrm{t}}$, the received
signals at terminal $q$ can be expressed as

$z_{q}=h_{pq}\left(\mathbf{t}_{p},\mathbf{r}_{q}\right)\sqrt{P_{\mathrm{t}}}s_{p}+h_{qq}\left(\mathbf{t}_{q},\mathbf{r}_{q}\right)\sqrt{P_{\mathrm{t}}}s_{q}+n_{q},~{}p\neq
q.$ (1)

Here, $s_{p}$ and $s_{q}$ stand for transmitted signals with zero mean and
normalized power of one.
$n_{q}\sim{\mathcal{CN}}\left(0,\sigma^{2}_{q}\right)$ is the additive white
Gaussian noise (AWGN) with power of $\sigma^{2}_{q}$. Besides, the first term
in (1) represents the SoI, while the second term is the inherent SI in FD
systems.

It is worth noting that the positions of the MAs don’t directly influence the
performance of the analog and digital cancellations, as they are capable of
actively mitigating the SI by adapting to channel variations. Considering
this, we will focus on exploring the potential of cancellation in the antenna
domain.

###  II-A Channel Model

Referring to the channel modeling in [7], we denote the coordinates of the
FPAs as ${\hat{\mathbf{t}}}_{p}=\left[0,0\right]^{\mathrm{T}}$ and
${\hat{\mathbf{r}}}_{q}=\left[0,0\right]^{\mathrm{T}}$. Then, the channel
matrix from the transmit MA of terminal $p$ to the receive MA of terminal $q$
can be given by $\mathbf{\Sigma}_{pq}$ of dimension $L^{pq}_{\mathrm{r}}\times
L^{pq}_{\mathrm{t}}$, where $L^{pq}_{\mathrm{t}}$ and $L^{pq}_{\mathrm{r}}$
represent the number of transmit and receive paths, respectively. Compared to
the FPAs, the difference of propagation distance for the MAs at the
$l_{\mathrm{t}}^{\mathrm{th}}$ transmit path can be expressed as

$\displaystyle\rho_{\mathrm{t},l_{\mathrm{t}}}^{p}\left(\mathbf{t}_{p}\right)=x_{\mathrm{t}}^{p}\cos\theta_{\mathrm{t},l_{\mathrm{t}}}^{p}\sin\phi_{\mathrm{t},l_{\mathrm{t}}}^{p}+$
$\displaystyle y_{\mathrm{t}}^{p}\sin\theta_{\mathrm{t},l_{\mathrm{t}}}^{p},$
$\displaystyle l_{\mathrm{t}}=1,2,\cdots,L^{pq}_{\mathrm{t}}.$ (2)

Similarly,

$\displaystyle\rho_{\mathrm{r},l_{\mathrm{r}}}^{q}\left(\mathbf{r}_{q}\right)=x_{\mathrm{r}}^{q}\cos\theta_{\mathrm{r},l_{\mathrm{r}}}^{q}\sin\phi_{\mathrm{r},l_{\mathrm{r}}}^{q}+$
$\displaystyle y_{\mathrm{r}}^{q}\sin\theta_{\mathrm{r},l_{\mathrm{r}}}^{q},$
$\displaystyle l_{\mathrm{r}}=1,2,\cdots,L^{pq}_{\mathrm{r}}$ (3)

represents the difference at the $l_{\mathrm{r}}^{\mathrm{th}}$ receive path.

In (II-A) and (II-A), $\theta_{\mathrm{t},l_{\mathrm{t}}}^{p}$ and
$\phi_{\mathrm{t},l_{\mathrm{t}}}^{p}$ respectively denote the elevation and
azimuth angles of departure (AoDs) at the $l_{\mathrm{t}}^{\mathrm{th}}$
transmit path, while $\theta_{\mathrm{r},l_{\mathrm{r}}}^{q}$ and
$\phi_{\mathrm{r},l_{\mathrm{r}}}^{q}$ respectively denote the elevation and
azimuth angles of arrival (AoAs) at the $l_{\mathrm{r}}^{\mathrm{th}}$ receive
path. Here,
$-\frac{\pi}{2}\leq\theta_{\mathrm{t},l_{\mathrm{t}}}^{p},\phi_{\mathrm{t},l_{\mathrm{t}}}^{p},\theta_{\mathrm{r},l_{\mathrm{r}}}^{q},\phi_{\mathrm{r},l_{\mathrm{r}}}^{q}\leq\frac{\pi}{2}$.
The phase difference between the antennas can be written by
$\frac{2\pi\rho}{\lambda}$, where $\rho$ is the difference of propagation
distance and $\lambda$ is the wavelength. Then, the field-response vectors are
respectively given by

$\mathbf{g}\left(\mathbf{t}_{p}\right)=\left[e^{j\frac{2\pi}{\lambda}\rho_{\mathrm{t},1}^{p}\left(\mathbf{t}_{p}\right)},e^{j\frac{2\pi}{\lambda}\rho_{\mathrm{t},2}^{p}\left(\mathbf{t}_{p}\right)},\cdots,e^{j\frac{2\pi}{\lambda}\rho_{\mathrm{t},L_{\mathrm{t}}^{pq}}^{p}\left(\mathbf{t}_{p}\right)}\right]^{\mathrm{T}}$
(4)

and

$\mathbf{f}\left(\mathbf{r}_{q}\right)=\left[e^{j\frac{2\pi}{\lambda}\rho_{\mathrm{r},1}^{q}\left(\mathbf{r}_{q}\right)},e^{j\frac{2\pi}{\lambda}\rho_{\mathrm{r},2}^{q}\left(\mathbf{r}_{q}\right)},\cdots,e^{j\frac{2\pi}{\lambda}\rho_{\mathrm{r},L_{\mathrm{r}}^{pq}}^{q}\left(\mathbf{r}_{q}\right)}\right]^{\mathrm{T}}.$
(5)

As a result, the channel coefficient between the transmit and receive MAs can
be obtained by

$h_{pq}\left({\mathbf{t}_{p}},{\mathbf{r}_{q}}\right)=\mathbf{f}\left({\mathbf{r}_{q}}\right)^{\mathrm{H}}\mathbf{\Sigma}_{pq}\mathbf{g}\left({\mathbf{t}_{p}}\right).$
(6)

###  II-B Problem Formulation

For FD communications, signal-to-interference-plus-noise ratio (SINR) is an
important metric to evaluate the system performance. In this scenario, it can
be represented by

${\gamma^{q}}=\frac{{{{\left|{{h_{pq}}\left({\mathbf{t}_{p}},{\mathbf{r}_{q}}\right)}\right|}^{2}}{P_{\mathrm{t}}}}}{{{{\left|{{h_{qq}}\left({\mathbf{t}_{q}},{\mathbf{r}_{q}}\right)}\right|}^{2}}{P_{\mathrm{t}}}+\sigma_{q}^{2}}},~{}p\neq
q.$ (7)

Therefore, based on (7), the achievable rate at terminal $q$ is calculated by

$R_{q}={\log_{2}}\left(1+{\gamma^{q}}\right).$ (8)

For the purpose of ensuring fairness between the terminals, we aim to maximize
the minimum achievable rate by jointly optimizing the positions of the four
MAs. To this end, the optimization problem is formulated as

$\displaystyle\mathop{\max}\limits_{\mathbf{t}_{p},\mathbf{r}_{q}}\quad$
$\displaystyle\min\\{{R_{\mathrm{A}}},{R_{\mathrm{B}}}\\}$ (9)
$\displaystyle\mathrm{s.t.}\quad$
$\displaystyle\mathbf{t}_{p}\in\mathcal{C}_{\mathrm{t}}^{p},$ (9a)
$\displaystyle\mathbf{r}_{q}\in\mathcal{C}_{\mathrm{r}}^{q},$ (9b)
$\displaystyle p,q\in\left\\{\mathrm{A,B}\right\\}.$ (9c)

Constraints (9a) and (9b) indicate that the MAs are confined to movement
within a determined transmit/receive region. Even though problem (9) appears
simple, it is difficult to solve due to the non-convex objective function.
Moreover, the position of each transmit/receive MA has a significant impact
not only on the reception of the SoI but also on the SI, thereby leading to a
relatively strong interrelationship among the positions of all four MAs.

##  III Proposed Solution

In virtue of the non-convex nature of the objective function in problem (9),
the conventional APO method, which involves alternately fixing three MAs while
moving only one, risks converging to an undesired local optimum. Additionally,
the exhaustive search approach for identifying the optimal solution requires
exploring an extensive solution space, resulting in prohibitively high
computational complexity. Taking these factors into account, this section
implements the PPSO method [14], which supports the simultaneous optimization
of all four MAs’ positions.

###  III-A PPSO Method

To begin with, denoting $N$ as the total number of particles and $K$ as the
number of iterations, the positions of the MAs during the $k^{\mathrm{th}}$
iteration are represented by a particle matrix, i.e.,
${\mathbf{U}^{(k)}}=\left[{\mathbf{u}_{1}^{(k)},\mathbf{u}_{2}^{(k)},\cdots,\mathbf{u}_{n}^{(k)},\cdots,\mathbf{u}_{N}^{(k)}}\right]$
of dimension $8\times N$, where $k=0,1,\cdots,K$ and $n=1,2,\cdots,N$. The
position vector of the $n^{\mathrm{th}}$ particle is defined as

$\mathbf{u}_{n}^{(k)}={\left[{\mathbf{t}_{\mathrm{A},n}^{(k)}}^{\mathrm{T}},{\mathbf{r}_{\mathrm{A},n}^{(k)}}^{\mathrm{T}},{\mathbf{t}_{\mathrm{B},n}^{(k)}}^{\mathrm{T}},{\mathbf{r}_{\mathrm{B},n}^{(k)}}^{\mathrm{T}}\right]^{\mathrm{T}}},$
(10)

where $\mathbf{t}_{p,n}^{(k)}$ and $\mathbf{r}_{q,n}^{(k)}$ represent the
transmit and receive MAs’ coordinates of the terminals, respectively, and
$\mathbf{u}_{n}^{(k)}$ might be a potential solution to problem (9). The
velocity matrix for the particles is introduced as
${\mathbf{V}^{(k)}}=\left[{\mathbf{v}_{1}^{(k)},\mathbf{v}_{2}^{(k)},\cdots,\mathbf{v}_{n}^{(k)},\cdots,\mathbf{v}_{N}^{(k)}}\right]$
of dimension $8\times N$.

Due to constraints (9a) and (9b), the coordinates $\mathbf{t}_{p,n}^{(k)}$ and
$\mathbf{r}_{q,n}^{(k)}$ in (10) are restricted to given regions. We assume
$\mathcal{C}_{\mathrm{t}}^{p}$ and $\mathcal{C}_{\mathrm{r}}^{q}$ are square
regions, each with size $D\times D$. Therefore, the vector
$\mathbf{u}_{n}^{(k)}$ is updated as

$\mathbf{u}_{n}^{(k)}=\mathcal{D}\left\\{\mathbf{u}_{n}^{(k-1)}+\mathbf{v}_{n}^{(k)}\right\\},~{}k=1,2,\cdots,K.$
(11)

$\mathcal{D}\left(\mathbf{a}\right)$ specifies a function that projects each
component of vector $\mathbf{a}$ to its corresponding maximum/minimum value,
detailed as

${\left[{\mathcal{D}\left\\{\mathbf{a}\right\\}}\right]_{i}}=\left\\{\begin{array}[]{rcl}\frac{D}{2},&\mathrm{if}&{{\left[\mathbf{a}\right]_{i}}>\frac{D}{2}}\\\
{\left[\mathbf{a}\right]_{i}},&\mathrm{if}&{-\frac{D}{2}\leq{\left[\mathbf{a}\right]_{i}}\leq\frac{D}{2}}\\\
-\frac{D}{2},&\mathrm{if}&{{\left[\mathbf{a}\right]_{i}}<-\frac{D}{2}}\\\
\end{array}\right..$ (12)

Algorithm 1 PPSO method

0: $\left\\{{{\mathcal{C}}_{\mathrm{t}}^{p}}\right\\}$,
$\left\\{{{\mathcal{C}}_{\mathrm{r}}^{q}}\right\\}$,
$\left\\{{\theta_{\mathrm{t},l_{\mathrm{t}}}^{p}}\right\\}$,
$\left\\{{\phi_{\mathrm{t},l_{\mathrm{t}}}^{p}}\right\\}$,
$\left\\{{\theta_{\mathrm{r},l_{\mathrm{r}}}^{q}}\right\\}$,
$\left\\{{\phi_{\mathrm{r},l_{\mathrm{r}}}^{q}}\right\\}$, $N$, $K$, $c_{1}$,
$c_{2}$, $\mathbf{e}_{1}$, $\mathbf{e}_{2}$, $\omega_{\min}$, $\omega_{\max}$.

0: ${\mathbf{u}^{\star}}$.

1: Initialize the particle matrix $\mathbf{U}^{(0)}$ and the velocity matrix
$\mathbf{V}^{(0)}$.

2: Initialize the local and global optimum position vectors,
$\mathbf{u}^{\star}_{n}$ and $\mathbf{u}^{\star}$, using (16) and (17),
respectively.

3: for $k=1$ to $K$ do

4: Calculate the inertia weight $\omega$ using (14).

5: for $n=1$ to $N$ do

6: Calculate the position and velocity vectors of the $n^{\mathrm{th}}$
particle, $\mathbf{u}_{n}^{(k)}$ and $\mathbf{v}_{n}^{(k)}$, using (11) and
(III-A), respectively.

7: Update the fitness value of the $n^{\mathrm{th}}$ particle
$F\left({\mathbf{u}_{n}^{(k)}}\right)$ using (15).

8: if
$F\left({\mathbf{u}_{n}^{(k)}}\right)>F\left({\mathbf{u}_{n}^{\star}}\right)$
then

9: $\mathbf{u}_{n}^{\star}=\mathbf{u}_{n}^{(k)}$.

10: end if

11: if
$F\left({\mathbf{u}_{n}^{(k)}}\right)>F\left({\mathbf{u}^{\star}}\right)$ then

12: $\mathbf{u}^{\star}={\mathbf{u}_{n}^{(k)}}$.

13: end if

14: end for

15: end for

16: return $\mathbf{u}^{\star}$.

In the PPSO method, incorporating with (11), the velocity vector for the
$n^{\mathrm{th}}$ particle at the $k^{\mathrm{th}}$ iteration is computed by

$\displaystyle\mathbf{v}_{n}^{(k)}=\omega$
$\displaystyle\mathbf{v}_{n}^{(k-1)}+{c_{1}}{\mathbf{e}_{1}}\odot\left(\mathbf{u}_{n}^{\star}-\mathbf{u}_{n}^{(k-1)}\right)$
$\displaystyle+{c_{2}}{\mathbf{e}_{2}}\odot\left(\mathbf{u}^{\star}-\mathbf{u}_{n}^{(k-1)}\right),~{}k=1,2,\cdots,K.$
(13)

Here, the inertia weight, $\omega$, is pivotal in moderating the impact of the
prior velocity, $\mathbf{v}_{n}^{(k-1)}$, on the subsequent velocity,
$\mathbf{v}_{n}^{(k)}$. The learning factors, $c_{1}$ and $c_{2}$, serve as
step sizes that guide each particle towards the local optimum position vector,
$\mathbf{u}_{n}^{\star}$, and the global optimum position vector,
$\mathbf{u}^{\star}$. Furthermore, the random vectors, $\mathbf{e}_{1}$ and
$\mathbf{e}_{2}$, filled with 8 elements that are typically uniform random
numbers within the range $\left[0,1\right]$, are used to increase the
randomness of the search for the sake of reducing the risk of settling on an
unfavorable local optimum.

To balance the trade-off between the speed and precision of the PPSO search,
$\omega$ is described as a linearly decreasing function across the iteration
count in the interval $\left[{{\omega_{\min}},{\omega_{\max}}}\right]$,
calculated as

$\omega={\omega_{\max}}-\frac{\left({{\omega_{\max}}-{\omega_{\min}}}\right)k}{K}.$
(14)

During each iteration, the fitness function determines the local and global
optimum position vectors, assuming the premise that the best position vector
corresponds to the largest fitness value. To maximize the minimum achievable
rate in (9), the fitness function of each particle is established as

$F\left({\mathbf{u}_{n}^{(k)}}\right)=\min\left\\{{{R_{\mathrm{A}}}\left({\mathbf{u}_{n}^{(k)}}\right),{R_{\mathrm{B}}}\left({\mathbf{u}_{n}^{(k)}}\right)}\right\\},$
(15)

where ${R_{\mathrm{A}}}\left({\mathbf{u}_{n}^{(k)}}\right)$ and
${R_{\mathrm{B}}}\left({\mathbf{u}_{n}^{(k)}}\right)$ denote the achievable
rates of the terminals when the positions of the MAs are specified by
${\mathbf{u}_{n}^{(k)}}$.

The PPSO method to address problem (9) is presented in Algorithm 1. Initially,
in line 1, the particles are set up with the starting positions
$\mathbf{U}^{(0)}$ and initial velocities $\mathbf{V}^{(0)}$. Following this,
in line 2, the local optimum position vector for each particle is initialized
by

$\mathbf{u}^{\star}_{n}=\mathbf{u}_{n}^{(0)},$ (16)

while the global optimum position vector is identified through

$\mathbf{u}^{\star}=\arg\mathop{\max}\limits_{\mathbf{u}_{n}^{(0)}}\left[{F\left({\mathbf{u}_{1}^{(0)}}\right),F\left({\mathbf{u}_{2}^{(0)}}\right),\cdots,F\left({\mathbf{u}_{N}^{(0)}}\right)}\right].$
(17)

After these initializations, the algorithm progresses to the iterative phase
(line 3). For each $k^{\mathrm{th}}$ iteration, the first step involves
calculating the inertia weight $\omega$ (line 4), which is followed by the
computation of both position and velocity vectors for each particle (line 6).
The fitness value for every particle is updated according to (15) in line 7.
Particularly noteworthy is the process from lines 8 to 13, where local and
global optimum position vectors are updated in response to larger fitness
values. Upon completing $K$ iterations, the procedure yields the near-optimal
positions of all four MAs, denoted as ${\mathbf{u}^{\star}}$, in line 16.

###  III-B Convergence and Complexity Analysis

The convergence of the PPSO method is ensured for two key reasons. Firstly,
the global optimum position’s fitness value either remains constant or
increases over iterations since a position is only designated as the global
optimum if it has a superior fitness value. Secondly, the minimum achievable
rate for problem (9) is capped by an upper limit, which prevents indefinite
increase. As for the computational complexity, the PPSO method operates with a
complexity of $\mathcal{O}\left(NK\right)$, which is influenced by the total
number of the particles, $N$, and the iterative times, $K$.

##  IV Simulation Results

TABLE I: Simulation Parameters Parameter | Description | Value
---|---|---
$\alpha$ | Path loss exponent | 2.8
$\beta_{\mathrm{AA}}$, $\beta_{\mathrm{BB}}$ | Path loss coefficients of the SI | -90 dB
$\beta_{\mathrm{AB}}$, $\beta_{\mathrm{BA}}$ | Path loss coefficients of the SoI | -30 dB
$d_{\mathrm{AA}}$, $d_{\mathrm{BB}}$ | Propagation distances for the SI channels | 1 m
$d_{\mathrm{AB}}$, $d_{\mathrm{BA}}$ | Propagation distances for the SoI channels | 100 m
$\sigma_{\mathrm{A}}^{2}$, $\sigma_{\mathrm{B}}^{2}$ | AWGN powers | -80 dBm
$P_{\mathrm{t}}$ | Transmit power | 20 dBm
$N$ | Number of particles | 200
$K$ | Number of iterations | 100
$c_{1},c_{2}$ | Learning factors | 1.4
$\omega_{\min}$ | Minimum inertia weight | 0.4
$\omega_{\max}$ | Maximum inertia weight | 0.9

In this section, we successively outline the simulation setup and provide
performance comparisons. Numerical simulation results illustrate the
performance of our proposed MA-enabled FD wireless system and imply the
effectiveness of the PPSO method in maximizing the minimum achievable rate
between the FD terminals.

For the simulation setup, we adopt the geometry channel model as [13, 14], in
which the channel matrix ${\mathbf{\Sigma}_{pq}}$ is composed of $L_{pq}$
diagonal elements that follow a CSCG distribution
$\mathcal{CN}\left(0,\frac{\beta_{pq}d_{pq}^{-\alpha}}{L_{pq}}\right)$. Here,
$L^{pq}_{\mathrm{t}}=L^{pq}_{\mathrm{r}}\overset{\triangle}{=}L_{pq}$,
$\alpha$ is the path loss exponent, $\beta_{pq}$ represents the path loss
coefficient, and $d_{pq}$ denotes the propagation distance. Notably, when
$p=q$, the matrix ${\mathbf{\Sigma}_{pq}}$ can be interpreted as the SI
channel; for $p\neq q$, it indicates the SoI channel. It is important to
emphasize that the path loss of the SI is considered to be much larger than
that of the SoI, owing to the directionality of antennas and the assistance
provided by both analog and digital cancellations. Thus, we assume
$\beta_{pq}=-90~{}\mathrm{dB}$ with $p=q$ and $\beta_{pq}=-30~{}\mathrm{dB}$
with $p\neq q$ [16]. The elevation and azimuth AoDs and AoAs are assumed to be
the random variables that are i.i.d. within the interval
$\left[-\frac{\pi}{2},\frac{\pi}{2}\right]$. Specifically, the detailed
simulation parameters are listed in Table I.

Figure 2: Maximized minimum achievable rates vs. the moving region size $D$.

We compare the performance of various schemes, including MA, FPA, and AS, as
well as FD and half-duplex (HD) modes. We note that the rate in (8) is divided
by two in the HD mode. Also, we evaluate the performance of our proposed
algorithm (see Algorithm 1), against the conventional APO method. In the APO
method, the moving region is partitioned into multiple grids of size
$\frac{\lambda}{100}\times\frac{\lambda}{100}$, allowing for an alternative
search of each MA’s position while others remain fixed. In the AS-enabled
scheme, the antenna arrays are deployed at intervals of $\frac{\lambda}{2}$
and optimal antennas are selected through alternating optimization. The
antennas used in the aforementioned schemes cannot exceed the region of size
$D\times D$. For convenience, our proposed scheme is referred to as MA-FD-
PPSO, signifying the MA-enabled FD scheme that utilizes the PPSO method. Other
schemes are labeled in a similar manner for straightforward comparison.

As shown in Fig. 2, we investigate the maximized minimum achievable rate
concerning the moving region size $D$, considering $L_{pq}=5$ for $p=q$ and
$L_{pq}=10$ for $q\neq q$. This figure reveals that for MA-enabled schemes,
the rates continuously rise as the moving region expands. In contrast, in AS-
enabled schemes, the increases in the rates are only noticeable when the
number of deployed antennas is augmented and such an augmentation occurs if
the size $D$ is an integer multiple of $\frac{\lambda}{2}$. Compared to FPA-
aided schemes, these schemes demonstrate enhanced performance. The PPSO method
surpasses the APO method in performance, especially when the size $D$ is
smaller than $\lambda$ in the FD mode. However, for the HD mode, the
distinction in improvement is less pronounced, indicating that the PPSO method
brings greater enhancements in scenarios of higher complexity. It is
interesting to observe that the AS-enabled HD scheme exhibits superior
performance over the FD one when the size $D<\frac{\lambda}{2}$. This
superiority can be attributed to the deployment of a single antenna at the
origin coordinate of the moving region, aligning its performance with that of
the FPA-aided schemes. Owing to the presence of the SI, the FD mode
underperforms in comparison to the HD mode, similar to the outcomes of the
FPA-aided schemes.

Figure 3: Maximized minimum achievable rates vs. the number of the SI paths.
Figure 4: Maximized minimum achievable rates vs. the number of the SoI paths.

Figure 3 depicts the maximized minimum achievable rates versus the number of
the SI paths, given the number of the SoI paths $L_{pq}=10$ with $p\neq q$ and
$D=\lambda$. Correspondingly, Figure 4 shows the rates regarding the number of
the SoI paths, where the number of the SI paths $L_{pq}=5$ with $p=q$ and
$D=\lambda$. In the FD mode, it can be seen that the rates for both MA-enabled
and AS-enabled schemes increase with an increase in the number of the SI or
SoI paths. However, in the HD mode, the rates hold steady when the number of
the SoI paths is constant, due to the absence of the SI. In Figs. 3 and 4, the
results also imply that the PPSO method consistently outperforms the
conventional APO method across different settings. Furthermore, all the
discussed schemes present superior performance over the FPA-aided schemes.

##  V Conclusion

In this paper, we propose a point-to-point FD system enhanced with the MAs. By
dynamically repositioning antennas to simultaneously mitigate the SI and
bolster the SoI, the MA-enabled FD system is capable of achieving an
outstanding maximized minimum achievable rate. To effectively optimize the
positions of the MAs, the PPSO method is implemented, addressing the non-
convex problem and preventing convergence to an undesired suboptimal
solutions. Simulation results confirm the performance improvements offered by
the PPSO method compared to the conventional APO method. Furthermore, these
results highlight the superiority of the MA-enabled FD scheme over the AS-
enabled schemes and those deploying FPAs regardless of whether they operate in
the FD or HD modes.

## References

  * [1] A. Sabharwal, P. Schniter, D. Guo, D. W. Bliss, S. Rangarajan, and R. Wichman, “In-band full-duplex wireless: challenges and opportunities,” IEEE J. Sel. Areas Commun., vol. 32, no. 9, pp. 1637-1652, Sep. 2014.
  * [2] M. Ma, S. Tian, Y. Chen, L. Wang, Y. Yang, L. Wan, B. Jiao, and H. V. Poor, “A prototype of co-frequency co-time full duplex networking,” IEEE Wireless Commun., vol. 27, no. 1, pp. 132-139, Feb. 2020.
  * [3] M. Duarte, C. Dick, and A. Sabharwal, “Experiment-driven characterization of full-duplex wireless systems,” IEEE Trans. Wireless Commun., vol. 11, no. 12, pp. 4296-4307, Dec. 2012.
  * [4] M. Duarte and A. Sabharwal, “Full-duplex wireless communications using off-the-shelf radios: Feasibility and first results,” in Proc. Conf. Rec. 44th Asilomar Conf. Signals, Syst. Comput., Nov. 2010, pp. 1558–1562.
  * [5] J. C. Dash and D. Sarkar, “A colinearly polarized full-duplex antenna with extremely high Tx–Rx isolation,” IEEE Antennas Wireless Propag. Lett., vol. 21, no. 12, pp. 2387-2391, Dec. 2022.
  * [6] S. Sanayei and A. Nosratinia, “Antenna selection in MIMO systems,” IEEE Commun. Mag., vol. 42, no. 10, pp. 68–73, Oct. 2004.
  * [7] L. Zhu, W. Ma, and R. Zhang, “Modeling and performance analysis for movable antenna enabled wireless communications,” IEEE Trans. Wireless Commun., Nov. 14, 2023, early access, DOI: 10.1109/TWC.2023.3330887.
  * [8] K. -K. Wong, A. Shojaeifard, K. -F. Tong, and Y. Zhang, “Fluid antenna systems,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 1950-1962, Mar. 2021.
  * [9] K. -K. Wong, W. K. New, X. Hao, K. -F. Tong, and C. -B. Chae, “Fluid antenna system—Part I: Preliminaries,” IEEE Commun. Lett., vol. 27, no. 8, pp. 1919-1923, Aug. 2023.
  * [10] K. -K. Wong, K. -F. Tong, and C. -B. Chae, “Fluid antenna system—Part II: Research opportunities,” IEEE Commun. Lett., vol. 27, no. 8, pp. 1924-1928, Aug. 2023.
  * [11] K. -K. Wong, K. -F. Tong, and C. -B. Chae, “Fluid antenna system—Part III: A new paradigm of distributed artificial scattering surfaces for massive connectivity,” IEEE Commun. Lett., vol. 27, no. 8, pp. 1929-1933, Aug. 2023.
  * [12] L. Zhu, W. Ma, and R. Zhang, “Movable antennas for wireless communication: opportunities and challenges,” IEEE Commun. Mag., Oct. 16, 2023, early access, DOI: 10.1109/MCOM.001.2300212.
  * [13] W. Ma, L. Zhu, and R. Zhang, “MIMO capacity characterization for movable antenna systems,” IEEE Trans. Wireless Commun., Sep. 7, 2023, early access, DOI: 10.1109/TWC.2023.3307696.
  * [14] Z. Xiao, X. Pi, L. Zhu, X. Xia, and R. Zhang, “Multiuser communications with movable-antenna base station: joint antenna positioning, receive combining, and power control,” arXiv preprint arXiv:2308.09512, 2023.
  * [15] G. Hu, Q. Wu, K. Xu, J. Ouyang, J. Si, Y. Cai, and N. Al-Dhahir, “Movable-antenna array enabled multiuser uplink: a low-complexity gradient descent for total transmit power minimization,” arXiv preprint arXiv:2312.05763, 2023.
  * [16] P. Guan, Y. Wang, H. Yu, and Y. Zhao, “Joint beamforming optimization for RIS-aided full-duplex communication,” IEEE Wireless Commun. Lett., vol. 11, no. 8, pp. 1629-1633, Aug. 2022.


# MERF: Memory-Efficient Radiance Fields for
Real-time View Synthesis in Unbounded Scenes

Christian Reiser  University of Tübingen, Tübingen AI CenterGermany Google
ResearchUnited Kingdom ,  Richard Szeliski  Google ResearchUnited States of
America ,  Dor Verbin  Google ResearchUnited States of America ,  Pratul P.
Srinivasan  Google ResearchUnited States of America ,  Ben Mildenhall  Google
ResearchUnited States of America ,  Andreas Geiger  University of Tübingen,
Tübingen AI CenterGermany ,  Jonathan T. Barron  Google ResearchUnited States
of America and  Peter Hedman  Google ResearchUnited Kingdom

###### Abstract.

Neural radiance fields enable state-of-the-art photorealistic view synthesis.
However, existing radiance field representations are either too compute-
intensive for real-time rendering or require too much memory to scale to large
scenes. We present a Memory-Efficient Radiance Field (MERF) representation
that achieves real-time rendering of large-scale scenes in a browser. MERF
reduces the memory consumption of prior sparse volumetric radiance fields
using a combination of a sparse feature grid and high-resolution 2D feature
planes. To support large-scale unbounded scenes, we introduce a novel
contraction function that maps scene coordinates into a bounded volume while
still allowing for efficient ray-box intersection. We design a lossless
procedure for baking the parameterization used during training into a model
that achieves real-time rendering while still preserving the photorealistic
view synthesis quality of a volumetric radiance field.

Interactive web demo at https://merf42.github.io

Figure 1.  Our method, _MERF_ , allows for real-time view synthesis in the
browser (framerates shown here are measured on a Macbook Pro M1). We design a
volumetric scene representation that is first optimized to maximize rendering
quality and then losslessly baked into a more efficient format for fast
rendering. As depicted above, this representation uses a combination of a low-
resolution 3D grid and a set of higher-resolution 2D planes (spatial extent of
each visualization signifies memory consumption). Compared to prior real-time
view synthesis methods such as SNeRG++ (our improved version of Hedman et al.
(2021)), MERF achieves better image quality for the same amount of
compression, matching the output of a SNeRG++ model that is over 17$\times$
larger.

##  1\. Introduction

Neural volumetric scene representations such as Neural Radiance Fields (NeRF)
(Mildenhall et al., 2020) enable photorealistic novel view synthesis of scenes
with complex geometry and appearance, but the compute required to query such
neural representations during volumetric raymarching prohibits real-time
rendering. Subsequent works have proposed discretized volumetric
representations that can substantially increase rendering performance (Garbin
et al., 2021; Yu et al., 2021; Hedman et al., 2021), but these approaches do
not yet enable practical real-time rendering for _large-scale_ scenes.

These representations struggle to scale to larger scenes primarily due to
graphics hardware constraints. Volumetric data is necessarily larger than a 2D
surface representation and occupies more space in memory. Similarly, while a
camera ray intersects a hard surface at most once, rendering a ray through a
volume may require many samples. With state of the art neural or hybrid
representations, each of these sample queries is very expensive to evaluate,
either in terms of compute or memory bandwidth. As a result, methods that work
for scenes with limited extent (single objects in space or forward-facing
scenes) typically do not scale up to larger unbounded scenes.

All neural or hybrid volumetric methods must address two fundamental trade-
offs that arise from these constraints:

  * •

_Volume vs. surface?_ Purely volumetric rendering models are most amenable to
gradient-based optimization and produce excellent view synthesis results
(Barron et al., 2022). On the other hand, increasing sparsity and moving
closer (Yariv et al., 2021; Wang et al., 2021) or completely (Chen et al.,
2022a; Munkberg et al., 2022) to a surface-like representation degrades image
quality but results in compact representations that are cheap to render.

  * •

_Memory bound vs. compute bound?_ The most compact representations (such as
the MLP network in Mildenhall et al. (2020) or the low-rank decomposition in
Chen et al. (2022b)) require many FLOPS to query, and the fastest
representations (such as the sparse 3D data structures used in Yu et al.
(2021) and Hedman et al. (2021)) consume large amounts of graphics memory.

One approach to this trade-off is to embrace a slower, more compact volumetric
model for optimization and to subsequently “bake” it into a larger but faster
representation for rendering. However, baking often affects the representation
or rendering model which can lead to a large drop in image quality. Though
this can partially be ameliorated by fine-tuning the baked representation,
fine-tuning does not easily scale to larger scenes, as computing gradients for
optimization requires significantly more memory than rendering.

The goal of our work is to find a representation that is well suited for both
optimization and fast rendering. Our solution is a single unified radiance
field representation with two different underlying _parameterizations_. In
both stages, our memory-efficient radiance field (MERF) is defined by a
combination of a voxel grid (Sun et al., 2022; Yu et al., 2022) and triplane
data structure (Chan et al., 2022). During optimization, we use the NGP hash
grid structure (Müller et al., 2022) to compress our parameterization, which
allows for differentiable sparsification and provides an inductive bias that
aids convergence. After optimization, we query the recovered NGP to explicitly
bake out the MERF and create a binary occupancy grid to accelerate rendering.
Critically, both the NGP-parameterized and baked MERF _represent the same
underlying radiance field function_. This means that the high quality achieved
by the optimized MERF carries over to our real-time browser-based rendering
engine.

##  2\. Related Work

As our goal is real-time view synthesis in large unbounded scenes, this
discussion is focused on approaches that accelerate rendering or reconstruct
large spaces. For a comprehensive overview of recent view synthesis
approaches, please refer to Tewari et al. (2022).

Early methods for real-time large-scale view synthesis either captured a large
number of images and interpolated them with optical flow (Aliaga et al., 2002)
or relied heavily on hand-made geometry proxies (Debevec et al., 1998; Buehler
et al., 2001). Later techniques used inaccurate, but automatically
reconstructed geometry proxies (Chaurasia et al., 2013), and relied on screen-
space neural networks to compensate for this (Hedman et al., 2018; Rückert et
al., 2022). Neural Radiance Fields (NeRF) (Mildenhall et al., 2020)
facilitated higher quality reconstructions by representing the full scene
volume as a multi-layer perceptron (MLP). This volumetric representation can
easily model thin structures and semi-transparent objects and is also well-
suited to gradient-based optimization.

NeRF was quickly extended to reconstruct large scenes by reconstructing
crowdsourced data (Martin-Brualla et al., 2021), tiling the space with NeRF
networks (Tancik et al., 2022; Turki et al., 2022), and reconstructing the
scene in a warped domain where far-away regions are compressed (Zhang et al.,
2020; Barron et al., 2022). Later, fast radiance field reconstruction was
achieved by representing the scene as a grid, stored either densely (Yu et
al., 2022), as latent features to be decoded (Sun et al., 2022; Karnewar et
al., 2022), or as latent hash grids (Müller et al., 2022) implemented with
specialized CUDA kernels (Li et al., 2022d). While this dramatically reduces
reconstruction time, accurate real-time rendering of large scenes has not yet
been demonstrated at high resolutions.

Other methods addressed real-time rendering by precomputing and storing (i.e.
_baking_) NeRF’s view-dependent colors and opacities in volumetric data
structures (Hedman et al., 2021; Yu et al., 2021; Garbin et al., 2021; Zhang
et al., 2022), or by splitting the scene into voxels and representing each
voxel with a small separate MLP (Reiser et al., 2021). However, these
representations consume a lot of graphics memory and are thus limited to
objects, not scenes. Furthermore, these methods incur a quality loss during
baking due to the mismatch between the slower rendering procedure used for
training and the real-time rendering procedure used for inference.

Alternatively, faster rendering can be achieved by extending the network to
work with ray segments rather than points (Lindell et al., 2021; Wu et al.,
2022a) or by training a separate sampling network (Barron et al., 2022; Neff
et al., 2021; Piala and Clark, 2021; Kurz et al., 2022). However, these
approaches have not achieved real-time rates at high resolutions, likely
because they require evaluating an MLP for each sample along a ray. Light
field coordinates circumvent this problem and require just one MLP evaluation
per ray (Sitzmann et al., 2021; Li et al., 2022c; Wang et al., 2022b; Attal et
al., 2022). However, like traditional light field representations (Gortler et
al., 1996; Levoy and Hanrahan, 1996), this approach has only been demonstrated
to work well within small viewing volumes. Similarly, multi-plane image
(Wizadwongsa et al., 2021; Flynn et al., 2019; Zhou et al., 2018; Mildenhall
et al., 2019) or multi-sphere image (Broxton et al., 2020; Attal et al., 2020)
representations map well to graphics hardware and can be rendered in real-
time, but also support only restricted camera motion.

It is also possible to speed-up NeRF rendering by post-processing the output
image with a convolutional neural network. This makes it possible to perform
an expensive volumetric rendering step at a lower resolution and then upsample
that result to the final desired resolution (Li et al., 2022a; Wang et al.,
2022a). Wu et al. (2022b) combined this approach with baking and showed high-
quality real-time rendering of large scenes. However, to achieve this, they
required a 3D scan of the scene as input, and they used a CUDA implementation
designed for workstation-class hardware. In contrast, our method only needs
posed images as input and runs in a browser on commodity hardware such as
laptops.

Recent methods achieve extremely fast rendering by constraining the NeRF
network evaluations to planes or polygons (Chen et al., 2022a; Lin et al.,
2022). While this works well for objects or limited camera motion, we show
that these approaches introduce a loss in quality for large unbounded scenes.

The problem of compressing NeRF reconstructions has also been explored in
prior work. Several methods achieve this by post-processing an existing
reconstruction through incremental pruning (Deng and Tartaglione, 2023) with
vector quantization (Li et al., 2022b). Takikawa et al. (2022) directly
optimize for a compressed codebook-based representation of the scene. While
these methods all report impressive compression ratios, they all rely on
evaluating an MLP for each volume sample and are therefore too slow for real-
time rendering of large scenes.

Our approach works by projecting 3D samples onto three 2D projections that
correspond to the cardinal axes. Similar representations, often referred to as
_tri-planes_ , have been explored for surface reconstruction from point clouds
(Songyou Peng, 2020) and generative modelling of 3D scenes (DeVries et al.,
2021) or faces (Chan et al., 2022). Recently TensoRF (Chen et al., 2022b) use
tri-planes for NeRF reconstruction. TensorRF decomposes the 3D scene volume
into a sum of vector-matrix outer products, which makes it possible to
directly train a compressed and high quality radiance field. However, TensoRF
trades off memory footprint for more expensive queries that involve a large
matrix multiplication. Our representation significantly speeds up the query
time by removing the need for the matrix product while simultaneously halving
the memory bandwidth consumption.

##  3\. Preliminaries

We begin with a short review of relevant prior work on radiance fields for
unbounded scenes. A radiance field maps every 3D position
$\mathbf{x}\in\mathbb{R}^{3}$ and viewing direction
$\mathbf{d}\in\mathbb{S}^{2}$ to the volumetric density
$\tau\in\mathbb{R}_{+}$ at that location and the RGB color emitted from it
along the view direction, $\mathbf{c}\in\mathbb{R}^{3}$. The color of the ray
emitted from point $\mathbf{o}$ in the direction $\mathbf{d}$ can then be
computed using the radiance field by sampling points along the ray,
$\mathbf{x}_{i}=\mathbf{o}+t_{i}\mathbf{d}$, and compositing the corresponding
densities $\\{\tau_{i}\\}$ and colors $\\{\mathbf{c}_{i}\\}$ according to the
numerical quadrature approach of Max (1995):

(1)
$\mathbf{C}=\sum_{i}w_{i}\mathbf{c}_{i}\,,\,\,\,w_{i}=\alpha_{i}T_{i}\,,\,\,\,T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j})\,,\,\,\,\alpha_{i}=1-e^{-\tau_{i}\delta_{i}}\,,$

where $T_{i}$ and $\alpha_{i}$ denote transmittance and alpha values of sample
$i$, and $\delta_{i}=t_{i+1}-t_{i}$ is the distance between adjacent samples.

The original NeRF work parameterized a radiance field using a Multilayer
Perceptron (MLP), which outputs the volume density and view-dependent color
for any continuous 3D location. In order to reduce the number of MLP
evaluations to one per ray, SNeRG uses a deferred shading model in which the
radiance field is decomposed into a 3D field of densities $\tau$, diffuse RGB
colors $\mathbf{c_{d}}$, and feature vectors $\mathbf{f}$ (Hedman et al.,
2021).

SNeRG’s deferred rendering model volumetrically accumulates the diffuse colors
$\\{\mathbf{c}_{d,i}\\}$ and features $\\{\mathbf{f}_{i}\\}$ along the ray,
similar to Equation 1:

(2)
$\mathbf{C}_{d}=\sum_{i}w_{i}\mathbf{c}_{d,i},\quad\mathbf{F}=\sum_{i}w_{i}\mathbf{f}_{i},$

and computes the ray’s color as the sum of the accumulated diffuse color
$\mathbf{C}_{d}$ and the view-dependent color computed using a small MLP $h$
that takes as input $\mathbf{C}_{d}$, $\mathbf{F}$, and the viewing direction
$\mathbf{d}$:

(3) $\mathbf{C}=\mathbf{C}_{d}+h(\mathbf{C}_{d},\mathbf{F},\mathbf{d}).$

SNeRG uses a large MLP during training and bakes it after convergence into a
block-sparse grid for real-time rendering.

In order for radiance fields to render high quality unbounded scenes
containing nearby objects as well as objects far from the camera, mip-NeRF 360
(Barron et al., 2022) uses a contraction function to warp the unbounded scene
domain into a finite sphere:

(4) $\operatorname{contract}(\mathbf{x})=\begin{cases}\mathbf{x}&\text{if
$\left\lVert\mathbf{x}\right\rVert_{2}\leq 1$}\\\
\left(2-\frac{1}{\prescript{}{\phantom{2}}{\left\lVert\mathbf{x}\right\rVert}_{2}}\right)\frac{\mathbf{x}}{\prescript{}{\phantom{2}}{\left\lVert\mathbf{x}\right\rVert}_{2}}&\text{if
$\left\lVert\mathbf{x}\right\rVert_{2}>1$}\end{cases}$

Figure 2. Our scene representation. For a location $\mathbf{x}$ along a ray:
(1) We query its eight neighbors on a low-resolution 3D grid; and we project
it onto each of the three axis-aligned planes, and then query each
projection’s four neighbors on a high-resolution 2D grid. (2) The eight low-
resolution 3D neighbors are evaluated and trilinearly interpolated while the
three sets of four high-resolution 2D neighbors are evaluated and bilinearly
interpolated, and the resulting features are summed into a single feature
vector $\mathbf{t}$. (3) The feature vector is split and nonlinearly mapped
into three components: density $\tau$, RGB color $\mathbf{c}_{d}$, and a
feature vector $\mathbf{f}$ encoding view dependence effects.

##  4\. Scene Representation

In this section, we describe the MERF scene representation, which is designed
to enable real-time volumetric rendering of unbounded scenes while maintaining
a low memory footprint.

###  4.1. Volume Parameterization

MERF represents a scene using a 3D field of volume densities
$\tau\in\mathbb{R}_{+}$, diffuse RGB colors $\mathbf{c_{d}}\in\mathbb{R}^{3}$,
and feature vectors $\mathbf{f}\in\mathbb{R}^{K}$, as shown in Figure 2. These
quantities are rendered using the deferred shading model from SNeRG, described
in Section 3.

We parameterize this field with a low-resolution 3D $L\times L\times L$ voxel
grid $V$ and three high-resolution 2D $R\times R$ grids $P_{x}$, $P_{y}$, and
$P_{z}$, one for each of the cardinal $yz$, $xz$, and $xy$ planes. Each
element of the low-resolution 3D grid and the three high-resolution 2D grids
stores a vector with $C=4+K$ channels In our experiments, we use $C=8$ and
default to $L=512$ and $R=2048$.

We define the continuous field of $C$-vectors as the sum of trilinearly
interpolated vectors from the 3D grid and bilinearly interpolated vectors from
the three 2D grids:

(5)
$\mathbf{t}(x,y,z)=\mathbf{V}(x,y,z)+\mathbf{P}_{x}(y,z)+\mathbf{P}_{y}(x,z)+\mathbf{P}_{z}(x,y),$

where $\mathbf{V}\colon\mathbb{R}^{3}\rightarrow\mathbb{R}^{C}$ is a trilinear
interpolation operator using the 3D grid values, and
$\mathbf{P}_{i}\colon\mathbb{R}^{2}\rightarrow\mathbb{R}^{C}$ is a bilinear
interpolation operator using the grid perpendicular to the $i$th axis, for
$i\in\\{x,y,z\\}$.

We split the $C$-vector at any 3D location into three components corresponding
to density $\tilde{\tau}\in\mathbb{R}$, diffuse color
$\tilde{\mathbf{c}}_{d}\in\mathbb{R}^{3}$, and view-dependence feature
$\tilde{\mathbf{f}}\in\mathbb{R}^{K}$, and then apply nonlinear functions to
obtain the three values:

(6)
$\tau=\exp(\tilde{\tau}),\quad\mathbf{c_{d}}=\sigma(\tilde{\mathbf{c}}_{d}),\quad\mathbf{f}=\sigma(\tilde{\mathbf{f}}),$

where $\sigma$ is the standard logistic sigmoid function, which constrains
colors and features to lie within $(0,1)$. Note that we apply the
nonlinearities after interpolation and summation, which has been shown to
greatly increase the representational power of grid representations (Karnewar
et al., 2022; Sun et al., 2022).

###  4.2. Piecewise-projective Contraction

Mip-NeRF 360 (Barron et al., 2022) demonstrated the importance of applying a
contraction function to input coordinates when representing large-scale scenes
with unbounded extent. The contraction maps large far-away regions of space
into small regions in contracted space, which has the effect of allocating
model capacity towards representing high-resolution content near the input
camera locations.

(a) $\operatorname{contract}(\cdot)$

(b) $\operatorname{contract_{\pi}}(\cdot)$

Figure 3. A 2D visualization of (a) the spherical contraction function used by
Barron et al. (2022); and (b) our piecewise projective contraction, both
applied to the same ray (in red). The spherical contraction maps a straight
line to a curve, which makes empty space skipping more expensive, while our
contraction maps a straight line to a small number of line segments. The gray
lines show the result of applying each contraction function to a regular grid.

The contraction function used in mip-NeRF 360 (Equation 4) nonlinearly maps
any point in space $\mathbf{x}\in\mathbb{R}^{3}$ into a radius-$2$ ball, and
represents the scene within this contracted space. While mip-NeRF 360’s
contraction function is effective and efficient to evaluate, it cannot be
easily used in a real-time rendering pipeline with discretized voxels. This is
because empty space skipping is critical for efficient volume rendering, and
requires a method for analytically computing the intersection between a ray
and an axis-aligned bounding box (AABB) of any “active” (i.e., occupied)
content. As can be seen in Figure 3(a), mip-NeRF 360’s contraction function
does not preserve straight lines, and thereby makes ray-AABB intersections
challenging to compute.

To address this, we propose a novel contraction function for which ray-AABB
intersections can be computed trivially. The $j$th coordinate of a contracted
point is defined as follows:

(7) $\operatorname{contract_{\pi}}(\mathbf{x})_{j}=\begin{cases}x_{j}&\text{if
}\|\mathbf{x}\|_{\infty}\leq 1\\\
\frac{x_{j}}{\prescript{}{\phantom{j}}{\|\mathbf{x}\|_{\infty}}}&\text{if
}x_{j}\neq\|\mathbf{x}\|_{\infty}>1\\\
\left(2-\frac{1}{|x_{j}|}\right)\frac{x_{j}}{|x_{j}|}&\text{if
}x_{j}=\|\mathbf{x}\|_{\infty}>1\end{cases}\,,$

where $\|\cdot\|_{\infty}$ is the $L_{\infty}$ norm
($\|\mathbf{x}\|_{\infty}=\max_{j}|x_{j}|$). This contraction function is
piecewise-projective: it contains seven regions, and within each region, it is
a projective transformation. The unit cube $\|\mathbf{x}\|_{\infty}$ is
preserved (i.e., the contraction function is the identity), and the other six
regions defined by the coordinate maximizing $|x_{j}|$ and its sign each get
mapped by a different projective transformation. Because projective
transformations preserve straight lines, any contracted ray must be piecewise-
linear, and the only discontinuities in its direction are on the boundaries
between the seven regions (see Figure 3(b) for a 2D example with 5 regions).
The origin and direction of a ray can therefore be computed in contracted
space, which allows us to use standard ray-AABB intersection tests. Table 2
(c) shows that our proposed contraction function performs on par with the
original spherical contraction.

##  5\. Training and Baking

In this section, we describe how to efficiently optimize a MERF and bake it
into a representation that can be used for high-quality real-time rendering.
To achieve our goal of not losing quality during this baking process, we take
care to ensure that the baked representation and the scene representation we
use during optimization both describe the same radiance field. This requires a
training procedure that accounts for any baking discretization or
quantization.

###  5.1. Efficient Training

Modeling high-resolution large-scale scenes requires high-capacity
representations that may consume prohibitive amounts of memory during
training. Significantly more memory is consumed during training than during
rendering because training requires that intermediate activations be stored
for the sake of backpropagation and that higher-precision per-parameter
statistics be accumulated for the Adam optimizer. In our case, we found that
training requires more than twelve times as much video memory as rendering. We
thus optimize a compressed representation of MERF’s low-resolution voxel grid
and high-resolution 2D planes by parameterizing them as MLPs with a multi-
resolution hash encoding (Müller et al., 2022).

However, baking the MLPs’ outputs onto discrete grids for rendering introduces
a mismatch between the representations used for training and for rendering.
Prior work accounted for this by fine-tuning the baked representation (Yu et
al., 2022), but fine-tuning requires the entire representation to be stored in
memory and suffers from the aforementioned scalability issues. Instead, we
simulate finite grid resolutions during training by querying the MLPs at
virtual grid corners and interpolating the resulting outputs using bilinear
interpolation for the high-resolution 2D grids and trilinear interpolation for
the low-resolution voxel grid.

In addition to requiring high-capacity representations, high-resolution large-
scale scenes require dense samples along rays during volume rendering, which
also significantly contributes to the training memory footprint. To address
this, mip-NeRF 360 introduced a hierarchical sampling technique that uses
“proposal” MLPs to represent coarse versions of scene geometry. A proposal MLP
maps 3D positions to density values, which are converted into probability
distributions along rays that are supervised to be consistent with the
densities output by the NeRF MLP. These proposal distributions are used in an
iterative resampling procedure that produces a small number of samples that
are concentrated around visible scene content. While this proposal MLP
hierarchical sampling strategy is effective for reducing the number of samples
along each ray during training, the proposal MLP is too expensive to evaluate
for real-time rendering purposes.

Instead, we use traditional empty space skipping during rendering, which also
concentrates representation queries around surfaces. To avoid introducing a
mismatch between training and rendering, we only bake content in regions
considered occupied by the proposal MLP during training, as detailed in
Section 5.3.

###  5.2. Quantization

To reduce our system’s memory consumption at render time, we wish to quantize
each of the $C$ dimensions at every location in the grid to a single byte (see
further discussion in Section 6). However, simply quantizing the optimized
grid values after training creates mismatches between the optimized model and
the one used for rendering, which leads to a drop in rendering quality as
shown in Table 2 (b).

Our solution to this is to quantize the $C$ values at every location during
optimization. That is, we nonlinearly map them to lie in $[0,1]$ using a
sigmoid $\sigma$, then quantize them to a single byte using a quantization
function $q$, and finally affinely map the result to the range $[-m,m]$, as:

(8) $\tilde{\mathbf{t}}^{\prime}=2m\cdot
q\\!\left(\sigma\\!\left(\tilde{\mathbf{t}}\right)\right)-m\,,$

where we choose $m=14$ for densities (which are computed using an exponential
nonlinearity), and $m=7$ for diffuse colors and features. Note that this only
quantizes the values stored in the grid, and the non-linearities in Equation 6
are subsequently applied after linearly interpolating and summing these
values.

We implement the byte quantization function $q$ as:

(9)
$q(x)=x+\cancel{\nabla}\\!\left(\frac{\lfloor(2^{8}-1)x+\nicefrac{{1}}{{2}}\rfloor}{2^{8}-1}-x\right)\,,$

where $\cancel{\nabla}(\cdot)$ is a stop-gradient, which prevents gradients
from backpropagating to its input. This use of a stop-gradient allows us to
obtain gradients for the non-differentiable rounding function by treating $q$
as the identity function during the backward pass, which is referred to as the
straight-through estimator, as used in (Bengio et al., 2013; Yin et al.,
2019).

###  5.3. Baking

After training, we evaluate and store the MLP’s outputs on discrete grids for
real-time rendering. First, we compute a binary 3D grid A indicating voxels
that contributed to any training image (i.e., voxels should _not_ be stored if
they correspond to occluded content, are not sampled by any training ray, or
have low opacity). To populate A, we render all training rays and extract from
them a set of weighted points $\\{(\mathbf{x}_{i},w_{i})\\}$, where
$\mathbf{x}_{i}$ is the point’s position, and $w_{i}$ is the associated volume
rendering weight from Equation 1. Note that these points cluster around
surfaces in the scene as they are sampled with a proposal-MLP (Barron et al.,
2022).

We mark the eight voxels surrounding a given point $\mathbf{x}_{i}$ as
occupied if both the volume rendering weight $w_{i}$ and the opacity,
$\alpha_{i}$, exceed a threshold set to $0.005$. To cull as aggressively as
possible, we compute $\alpha_{i}$ based on the distance between samples
$\delta_{i}$ used by the real-time renderer — recall that it steps through
contracted space with a small uniform step size. As the proposal-MLP often
suggests steps larger than $\delta_{i}$, computing $\alpha_{i}$ this way leads
to better culling. However, we still guarantee voxels which contribute a
significant opacity value ($\alpha_{i}>0.005$) are not culled in the final
sampling scheme. Note that while the opacity $\alpha_{i}$ only depends on the
density at $\mathbf{x}_{i}$, the weight $w_{i}$ also depends on densities
along the entire ray, making usage of $w_{i}$ necessary to account for
visibility. We observe that the opacity check based on the real-time
renderer’s step size significantly decreases the fraction of the volume marked
as occupied. Note that this is in addition to the sparsity already achieved by
only considering voxels in locations that have been sampled by the proposal-
MLP. In contrast, existing baking pipelines often do not consider the
proposal-MLP and perform visibility culling with uniformly-spaced sample
points. This often results in fog-like artifacts and floating blobs because
the underlying 3D field can have arbitrary values in regions not sampled by
the proposal-MLP. Table 2 demonstrates that our Proposal-MLP-aware baking
pipeline is almost lossless.
After computing the binary grid A, we bake the three high-resolution 2D planes
and the low-resolution 3D voxel grid. Following SNeRG, we store this voxel
grid in a block-sparse format, where we only store data blocks that contain
occupied voxels. For empty space skipping, we create multiple lower resolution
versions of the binary occupancy grid A with max-pooling. To reduce storage,
we encode textures as PNGs.

| Outdoor | Indoor
---|---|---
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
NeRF (Mildenhall et al., 2020) | 21.46 | 0.458 | 0.515 | 26.84 | 0.790 | 0.370
NeRF++ (Zhang et al., 2020) | 22.76 | 0.548 | 0.427 | 28.05 | 0.836 | 0.309
SVS (Riegler and Koltun, 2021) | 23.01 | 0.662 | 0.253 | 28.22 | 0.907 | 0.160
Mip-NeRF 360 (Barron et al., 2022) | 24.47 | 0.691 | 0.283 | 31.72 | 0.917 | 0.180
Instant-NGP (Müller et al., 2022) | 22.90 | 0.566 | 0.371 | 29.15 | 0.880 | 0.216
Deep Blending (Hedman et al., 2018) | 21.54 | 0.524 | 0.364 | 26.40 | 0.844 | 0.261
Mobile-NeRF (Chen et al., 2022a) | 21.95 | 0.470 | 0.470 | $-$ | $-$ | $-$
Ours | 23.19 | 0.616 | 0.343 | 27.80 | 0.855 | 0.271
Table 1.  Quantitative results of our model on all scenes from Mip-NeRF 360
(Barron et al., 2022). Models that render in real-time are highlighted.
Mobile-NeRF did not evaluate on the “indoor” scenes, so those metrics are
absent.

##  6\. Real-time Rendering

We implement our real-time viewer as a Javascript 3D (three.js) web
application, based on SNeRG’s implementation, where rendering is orchestrated
by a single GLSL fragment shader.

For efficient ray marching, we employ a multi-resolution hierarchy of
occupancy grids. The set of occupancy grids is created by max-pooling the
full-resolution binary mask A with filter sizes $16,32$ and $128$. For
instance, if the base resolution is $4096$, this results in occupancy grids of
size $256,128$ and $32$, occupying a total of 18 MB of video memory. We
leverage this multi-resolution hierarchy of occupancy grids for faster space
skipping. Given any sample location, we query the occupancy grids in a coarse-
to-fine manner. If any level indicates the voxel as empty, we can skip the
corresponding volume until the ray enters a new voxel at that level and
compute the new sample location using the efficient ray-AABB intersection
discussed in Section 4.2. We only access the MERF scene representation for
samples where all occupancy grid levels are marked as occupied. Finally, we
terminate ray marching along a ray once the transmittance value $T_{i}$
(defined in Equation 1) falls below $2\times 10^{-4}$.

To further decrease the number of memory accesses during rendering, we split
textures into density and appearance (containing diffuse RGB colors and
feature vectors) components. When accessing the MERF representation at any
location, we first query the density component and only read the corresponding
appearance component if the voxel opacity computed from the returned density
is nonzero. Moreover, we obtain an additional $4\times$ speed-up by optimizing
the deferred rendering MLP. More specifically, we conduct loop unrolling,
modify the memory layout to facilitate linear accesses, and exploit fast
mat4-multiplication.

##  7\. Experiments

We experimentally evaluate our model in terms of rendering quality, video
memory consumption, and real-time rendering performance. We compare MERF to a
variety of offline view synthesis methods (NeRF (Mildenhall et al., 2020),
mip-NeRF 360 (Barron et al., 2022), Stable View Synthesis (Riegler and Koltun,
2021), and Instant-NGP (Müller et al., 2022)), and real-time ones (Deep
Blending (Hedman et al., 2018), Mobile-NeRF (Chen et al., 2022a), and SNeRG
(Hedman et al., 2021)). To make this evaluation as rigorous as possible we
evaluate against an improved version of SNeRG, which we call SNeRG++. SNeRG++
uses many components of our approach: multi-level empty space skipping, an
optimized MLP implementation, our improved baking pipeline, and post-
activation interpolation (which increases model expressivity by allowing for
intra-voxel discontinuities (Sun et al., 2022; Karnewar et al., 2022)). Unless
otherwise stated, for MERF we set the triplane resolution $R$ to 2048 and the
sparse grid resolution $L$ to 512, and for SNeRG++ we set the grid resolution
to $2048$.

For evaluation, we use the challenging mip-NeRF 360 dataset (Barron et al.,
2022) which contains five outdoor and four indoor scenes. All scenes are
unbounded and require a high resolution representation ($2048^{3}$) to be
faithfully reproduced. If not indicated otherwise, reported metrics are
averaged over runs from the five outdoor scenes. We evaluate rendering quality
using peak-signal-to-noise-ratio (PSNR), SSIM (Wang et al., 2004), and LPIPS
(Zhang et al., 2018).

###  7.1. Quality Comparison

We first compare our method to a variety of offline and real-time view
synthesis methods in terms of rendering quality on both outdoor and indoor
scenes. As can be seen in Table 1, MERF achieves better scores than the real-
time competitors DeepBlending and Mobile-NeRF on all three metrics. On the
outdoor scenes, MERF even achieves higher scores than the offline methods
Instant-NGP, NeRF and NeRF++ and performs on par with SVS in terms of PSNR and
SSIM, despite offline rendering quality not being the primary focus of this
work. In contrast, on indoor scenes MERF achieves slightly lower scores than
Instant-NGP, SVS, and NeRF++. This might be a result of the indoor scenes
containing more artificial materials with stronger view-dependent effects that
our shallow decoder MLP $h$ designed for real-time rendering cannot model
well. Qualitatively, Figure 6 shows that our reconstructions of the background
are much sharper compared to Mobile-NeRF and Instant-NGP, which we attribute
to using a scene contraction function.

###  7.2. MERF vs. SNeRG++

We begin our analysis with a comparison of MERF with SNeRG++: both use the
same training pipeline, view-dependent appearance model, and rendering engine
implementation, and this enables controlled experiments. To support our claim
that MERF provides a favorable trade-off between memory consumption and
rendering quality, we train models with varying resolutions: For MERF, we vary
the feature plane resolution $R$ from $512$ and $3072$ while setting the grid
resolution $L$ to either $512$ or $1024$. Likewise, for SNeRG++, we vary the
grid resolution $L$ from $512$ to $2048$. To demonstrate the benefit of adding
a low-resolution sparse 3D grid, we also train our models without the 3D grid.
As can be seen in Figure 4, the memory consumption of SNeRG++ quickly rises to
multiple gigabytes, whereas our model scales better to higher resolutions. For
models without 3D grids we observe that quality saturates well below the other
models. In Figure 5 we see that our model ($R=2048$, $L=512$) achieves similar
quality to SNeRG++, while requiring a fraction of the memory. Additionally, in
Figure 7 we see that ablating the 3D grid from MERF leads to a significant
loss in quality.

###  7.3. Real-time Rendering Evaluation

Finally, we evaluate the rendering speed of MERF, MobileNeRF, SNeRG++, and
Instant-NGP in frames per second (FPS). Note that MERF, Mobile-NeRF and
SNeRG++ all run in the browser and use the view-dependence model introduced by
Hedman et al. (Hedman et al., 2021). In contrast, Instant-NGP uses a different
view-dependence model, and is implemented in CUDA and is therefore less
portable across devices. For benchmarking the methods that include web viewers
(MERF, Mobile-NERF, SNeRG++) we use an M1 MacBook Pro and set the rendering
resolution to $1280\times 720$. When evaluating against Instant-NGP, to make
the comparison fair we use an RTX 3090 GPU (which Instant-NGP is optimized
for) and increase the rendering resolution to $1920\times 1080$ to demonstrate
MERF’s scalability on high-powered devices. As can be seen in Table 3, our
method runs faster than SNeRG++ while consuming only one fifth of the memory.
While MobileNeRF achieves higher frame rates on the Macbook than MERF, it
requires twice as much video memory and reduces rendering quality (a $1.24$ dB
reduction in PSNR). This reduced quality is especially evident in background
regions, as shown in Figure 6. From our experiment with the RTX 3090, we see
that Instant-NGP does not achieve real-time performance (4 FPS), while MERF
renders at frame rates well above 100.

###  7.4. Limitations

Since we use the view-dependence model introduced in SNeRG (Hedman et al.,
2021), we also inherit its limitations: By evaluating view-dependent color
once per ray, we are unable to faithfully model view-dependent appearance for
rays that intersect with semi-transparent objects. Furthermore, since the tiny
MLP has limited capacity, it may struggle to scale to much larger scenes or
objects with complex reflections.

Moreover, our method still performs volume rendering, which limits it to
devices equipped with a sufficiently powerful GPU such as laptops, tablets or
workstations. Running our model on smaller, thermally limited devices such as
mobile phones or headsets will require further reductions in memory and
runtime.

##  8\. Conclusion

We have presented MERF, a compressed volume representation for radiance
fields, which admits real-time rendering of large-scale scenes in a browser.
By using novel hybrid volumetric parameterization, a novel contraction
function that preserves straight lines, and a baking procedure that ensures
that our real-time representation describes the same radiance field as was
used during optimization, MERF is able to achieve faster and more accurate
real-time rendering of large and complicated real-world scenes than prior
real-time NeRF-like models. Out of all real-time methods, ours produces the
highest-quality renderings for any given memory budget. Not only does it
achieve $31.6\%$ (MSE) higher quality in the outdoor scenes compared to
MobileNeRF, the previous state-of-the-art, it also requires less than half of
the GPU memory.

###### Acknowledgements.

We thank Marcos Seefelder, Julien Philip and Simon Rodriguez for their
suggestions on shader optimization. This work was supported by the ERC
Starting Grant LEGO3D (850533) and the DFG EXC number 2064/1 - project number
390727645.

Figure 4. PSNR (higher is better) vs VRAM consumption (lower is better) for our model, our improved SNeRG baseline, MobileNeRF, and an ablation of our model without 3D grids. Each line besides MobileNeRF represents the same model with a varying resolution of its underlying spatial grid, indicated by marker size. | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
(a) Pre-baking | 23.20 | 0.620 | 0.336
(b) w/o quant.-aware training | 22.64 | 0.603 | 0.347
(c) Spherical contraction | 23.22 | 0.619 | 0.341
Ours (Post-baking) | 23.19 | 0.616 | 0.343
Table 2. We compare our final model after baking to the model before baking
(a) demonstrating that our Proposal-MLP-aware baking pipeline is almost
lossless. Omitting quantization-aware training (b) leads to a drop in
rendering quality. Our proposed contraction function performs on par with the
original spherical contraction function (c), while enabling efficient ray-AABB
intersection tests. Results are averaged over all outdoor scenes.

|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  VRAM $\downarrow$ |  DISK $\downarrow$ |  FPS $\uparrow$
---|---|---|---|---|---|---
| MacBook M1 Pro, 1280$\times$720
Mobile-NeRF | 21.95 | 0.470 | 0.470 | 1162 | 345 | 65.7
SNeRG++ | 23.64 | 0.672 | 0.285 | 4571 | 3785 | 18.7
Ours | 23.19 | 0.616 | 0.343 | 524 | 188 | 28.3
| NVIDIA RTX 3090, 1920$\times$1080
Instant-NGP | 22.90 | 0.566 | 0.371 | $-$ | 107 | 4
Ours | 23.19 | 0.616 | 0.343 | 524 | 188 | 119

Table 3. Performance comparison on the “outdoor” scenes.

## References

  * (1)
  * Aliaga et al. (2002) D.G. Aliaga, T. Funkhouser, D. Yanovsky, and I. Carlbom. 2002.  Sea of images.  _IEEE Visualization_ (2002).
  * Attal et al. (2022) Benjamin Attal, Jia-Bin Huang, Michael Zollhöfer, Johannes Kopf, and Changil Kim. 2022.  Learning Neural Light Fields with Ray-Space Embedding Networks.  _CVPR_ (2022).
  * Attal et al. (2020) Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. 2020\.  MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images.  _ECCV_ (2020).
  * Barron et al. (2022) Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022\.  Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields.  _CVPR_ (2022).
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013\.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv:1308.3432_ (2013).
  * Broxton et al. (2020) Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. 2020\.  Immersive Light Field Video with a Layered Mesh Representation.  _ACM Transactions on Graphics_ (2020).
  * Buehler et al. (2001) Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001.  Unstructured Lumigraph Rendering.  _SIGGRAPH_ (2001).
  * Chan et al. (2022) Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. 2022\.  Efficient Geometry-aware 3D Generative Adversarial Networks.  _CVPR_ (2022).
  * Chaurasia et al. (2013) Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis. 2013.  Depth Synthesis and Local Warps for Plausible Image-Based Navigation.  _ACM Transactions on Graphics_ (2013).
  * Chen et al. (2022b) Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022b.  TensoRF: Tensorial Radiance Fields.  _ECCV_ (2022).
  * Chen et al. (2022a) Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. 2022a.  MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures.  _arXiv:2208.00277_ (2022).
  * Debevec et al. (1998) Paul Debevec, Yizhou Yu, and George Borshukov. 1998.  Efficient view-dependent image-based rendering with projective texture-mapping.  _EGSR_ (1998).
  * Deng and Tartaglione (2023) Chenxi Lola Deng and Enzo Tartaglione. 2023.  Compressing Explicit Voxel Grid Representations: Fast NeRFs Become Also Small.  _WACV_ (2023).
  * DeVries et al. (2021) Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. 2021.  Unconstrained Scene Generation with Locally Conditioned Radiance Fields.  _ICCV_ (2021).
  * Flynn et al. (2019) John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. 2019\.  DeepView: View synthesis with learned gradient descent.  _CVPR_ (2019).
  * Garbin et al. (2021) Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. 2021\.  FastNeRF: High-Fidelity Neural Rendering at 200FPS.  _ICCV_ (2021).
  * Gortler et al. (1996) Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996.  The lumigraph.  _SIGGRAPH_ (1996).
  * Hedman et al. (2018) Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018\.  Deep blending for free-viewpoint image-based rendering.  _SIGGRAPH Asia_ (2018).
  * Hedman et al. (2021) Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec. 2021\.  Baking Neural Radiance Fields for Real-Time View Synthesis.  _ICCV_ (2021).
  * Karnewar et al. (2022) Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. 2022.  ReLU fields: The little non-linearity that could.  _SIGGRAPH_ (2022).
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR_.
  * Kurz et al. (2022) Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, and Markus Steinberger. 2022\.  AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields.  _ECCV_ (2022).
  * Levoy and Hanrahan (1996) Marc Levoy and Pat Hanrahan. 1996.  Light field rendering.  _SIGGRAPH_ (1996).
  * Li et al. (2022b) Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Liefeng Bo. 2022b.  Compressing Volumetric Radiance Fields to 1 MB.  _arXiv:2211.16386_ (2022).
  * Li et al. (2022d) Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. 2022d.  NerfAcc: A General NeRF Accleration Toolbox.  _arXiv:2210.04847_ (2022).
  * Li et al. (2022a) Sicheng Li, Hao Li, Yue Wang, Yiyi Liao, and Lu Yu. 2022a.  SteerNeRF: Accelerating NeRF Rendering via Smooth Viewpoint Trajectory.  _arXiv:2212.08476_ (2022).
  * Li et al. (2022c) Zhong Li, Liangchen Song, Celong Liu, Junsong Yuan, and Yi Xu. 2022c.  NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field.  _EGSR_ (2022).
  * Lin et al. (2022) Zhi-Hao Lin, Wei-Chiu Ma, Hao-Yu Hsu, Yu-Chiang Frank Wang, and Shenlong Wang. 2022.  NeurMiPs: Neural Mixture of Planar Experts for View Synthesis.  _CVPR_ (2022).
  * Lindell et al. (2021) David B. Lindell, Julien N.P. Martel, and Gordon Wetzstein. 2021.  AutoInt: Automatic Integration for Fast Neural Rendering.  _CVPR_ (2021).
  * Martin-Brualla et al. (2021) Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  _CVPR_ (2021).
  * Max (1995) Nelson Max. 1995\.  Optical models for direct volume rendering.  _IEEE Transactions on Visualization and Computer Graphics_ (1995).
  * Mildenhall et al. (2019) Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019\.  Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines.  _ACM Transactions on Graphics_ (2019).
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  _ECCV_ (2020).
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _SIGGRAPH_ (2022).
  * Munkberg et al. (2022) Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. 2022\.  Extracting Triangular 3D Models, Materials, and Lighting From Images.  _CVPR_ (2022).
  * Neff et al. (2021) Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. 2021.  DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks.  _Computer Graphics Forum_ (2021).
  * Piala and Clark (2021) Martin Piala and Ronald Clark. 2021.  TermiNeRF: Ray Termination Prediction for Efficient Neural Rendering.  _3DV_ (2021).
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021\.  KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs.  _ICCV_ (2021).
  * Riegler and Koltun (2021) Gernot Riegler and Vladlen Koltun. 2021.  Stable view synthesis.  _CVPR_ (2021).
  * Rückert et al. (2022) Darius Rückert, Linus Franke, and Marc Stamminger. 2022\.  ADOP: Approximate differentiable one-pixel point rendering.  _SIGGRAPH_ (2022).
  * Sitzmann et al. (2021) Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. 2021\.  Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering.  _NeurIPS_ (2021).
  * Songyou Peng (2020) Lars Mescheder Marc Pollefeys Andreas Geiger Songyou Peng, Michael Niemeyer. 2020.  Convolutional Occupancy Networks.  _ECCV_ (2020).
  * Sun et al. (2022) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  _CVPR_ (2022).
  * Takikawa et al. (2022) Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. 2022\.  Variable Bitrate Neural Fields.  _ACM Transactions on Graphics_ (2022).
  * Tancik et al. (2022) Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. 2022.  Block-NeRF: Scalable Large Scene Neural View Synthesis.  _CVPR_ (2022).
  * Tewari et al. (2022) Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, W Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. 2022\.  Advances in neural rendering.  _Computer Graphics Forum_ (2022).
  * Turki et al. (2022) Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. 2022\.  Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs.  _CVPR_ (2022).
  * Wang et al. (2022b) Huan Wang, Jian Ren, Zeng Huang, Kyle Olszewski, Menglei Chai, Yun Fu, and Sergey Tulyakov. 2022b.  R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis.  _ECCV_ (2022).
  * Wang et al. (2021) Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021\.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  _NeurIPS_ (2021).
  * Wang et al. (2004) Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004.  Image quality assessment: from error visibility to structural similarity.  _IEEE TIP_ (2004).
  * Wang et al. (2022a) Zhongshu Wang, Lingzhi Li, Zhen Shen, Li Shen, and Liefeng Bo. 2022a.  4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions.  _arXiv:2212.04701_ (2022).
  * Wizadwongsa et al. (2021) Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021\.  NeX: Real-time View Synthesis with Neural Basis Expansion.  _CVPR_ (2021).
  * Wu et al. (2022a) Liwen Wu, Jae Yong Lee, Anand Bhattad, Yuxiong Wang, and David Forsyth. 2022a.  DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering.  _CVPR_ (2022).
  * Wu et al. (2022b) Xiuchao Wu, Jiamin Xu, Zihan Zhu, Hujun Bao, Qixing Huang, James Tompkin, and Weiwei Xu. 2022b.  Scalable Neural Indoor Scene Rendering.  _ACM TOG_ (2022).
  * Yariv et al. (2021) Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021\.  Volume rendering of neural implicit surfaces.  _NeurIPS_ (2021).
  * Yin et al. (2019) Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. 2019\.  Understanding straight-through estimator in training activation quantized neural nets.  _ICLR_ (2019).
  * Yu et al. (2022) Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022.  Plenoxels: Radiance fields without neural networks.  _CVPR_ (2022).
  * Yu et al. (2021) Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021\.  PlenOctrees for real-time rendering of neural radiance fields.  _ICCV_ (2021).
  * Zhang et al. (2022) Jian Zhang, Jinchi Huang, Bowen Cai, Huan Fu, Mingming Gong, Chaohui Wang, Jiaming Wang, Hongchen Luo, Rongfei Jia, Binqiang Zhao, and Xing Tang. 2022.  Digging into Radiance Grid for Real-Time View Synthesis with Detail Preservation.  _ECCV_ (2022).
  * Zhang et al. (2020) Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. 2020.  NeRF++: Analyzing and Improving Neural Radiance Fields.  _arXiv:2010.07492_ (2020).
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.  _CVPR_ (2018).
  * Zhou et al. (2018) Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018.  Stereo Magnification: Learning View Synthesis using Multiplane Images.  _SIGGRAPH_ (2018).

gardenvase |  |  |  |  |  |
---|---|---|---|---
|  |  |  |
Ground truth | Ours (2048/512) | SNeRG++ (512) | SNeRG++ (1024) | SNeRG++ (2048)
| 198 MB | 117 MB | 489 MB | 2372 MB
kitchenlego |  |  |  |  |  |
---|---|---|---|---
|  |  |  |
Ground truth | Ours (2048/512) | SNeRG++ (512) | SNeRG++ (1024) | SNeRG++ (2048)
| 233 MB | 213 MB | 1000 MB | 4570 MB
stump |  |  |  |  |  |
---|---|---|---|---
|  |  |  |
Ground truth | Ours (2048/512) | SNeRG++ (512) | SNeRG++ (1024) | SNeRG++ (2048)
| 228 MB | 210 MB | 810 MB | 3904 MB

Figure 5. Visual comparison between MERF and various resolution SNeRG++
(Hedman et al., 2021) models. Total VRAM (GPU memory) usage during rendering
is listed beneath each method name. Only SNeRG++ (512) has comparable size to
our model, whereas SNeRG++ (1024) and SNeRG++ (2048) are significantly larger.

flowerbed |  |  |  |  |  |
---|---|---|---|---
|  |  |  |
Ground truth | Ours | Mobile-NeRF | Instant NGP | mip-NeRF 360
treehill |  |  |  |  |  |
---|---|---|---|---
|  |  |  |
Ground truth | Ours | Mobile-NeRF | Instant NGP | mip-NeRF 360

Figure 6. Visual comparison between MERF and other view synthesis methods.
Mobile-NeRF (Chen et al., 2022a) is the only other real-time method (30fps or
better). Instant NGP (Müller et al., 2022) runs at interactive rates (around
5fps) and mip-NeRF 360 (Barron et al., 2022) is an extremely heavyweight
offline method (around 30 seconds to render a single frame), representing the
current state-of-the-art view synthesis quality.

|  |  |
---|---|---|---
With low-resolution 3D grid |  | Without low-resolution 3D grid |

Figure 7. Visual comparison between MERF with and without a low-resolution 3D
voxel grid (both models use three high-resolution 2D grids). Omitting the 3D
grid often results in parts of the scene being poorly reconstructed; note the
missing background in this example.

##  Appendix A Training details

All SNeRG++ and MERF models use the same training hyperparameters and
architectures. We train for 25000 iterations with a batch size of $2^{16}$
pixels. A training batch is created by sampling pixels from all training
images. We use the Adam (Kingma and Ba, 2015) optimizer with an exponentially
decaying learning rate. The learning rate is warmed up during the first 100
iterations where it is increased from $110-4$ to $110-2$. Then the learning
rate is decayed to $110-3$. Adam’s hyperparameters $\beta_{1}$, $\beta_{2}$
and $\epsilon$ are set to $0.9$, $0.99$ and $110-15$, respectively. To
regularize the hash grids we use a weight decay of $0.1$ on its grid values.

##  Appendix B Architecture

For parameterizing all grids (i.e. 3D voxel grids and 2D planes) we use an MLP
with a multi-resolution hash encoding (Müller et al., 2022). The hash encoding
uses 20 levels and the individual hash tables have $2^{21}$ entries. Following
Müller et al. (2022), hash collisions are resolved with a 2-layer MLP with 64
hidden units. This MLP outputs an 8-dimensional vector representing density,
diffuse RGB and the 4-dimensional view-dependency feature vector. For our
deferred view-dependency model we closely follow SNeRG (Hedman et al., 2021)
and use a 3-layer MLP with 16 hidden units. As in SNeRG viewing directions are
encoded with 4 frequencies. Following MipNeRF360 (Barron et al., 2022) we use
hierarchical sampling with three levels and therefore require two Proposal-
MLPs. The Proposal-MLPs consist of 2 layers with 64 hidden units and use a
hash encoding. Since a Proposoal-MLP merely needs to model coarse geometry, we
use for the Proposal-MLPs’ hash encodings only 10 levels, a maximum grid
resolution of $512$ and a hash table size of $2^{16}$.

##  Appendix C Benchmarking

For each test scene we define a camera pose that is used for benchmarking.
Camera poses are set programmatically in each viewer. For fair comparison, we
ensure that resolutions and camera intrinsics (i.e. field of view) are
identical across viewers. We compute the average frame rate across 150 frames.

By default, Instant-NGP makes use of progressive upsampling, where the image
is first rendered at a lower resolution. When the camera rests additional low
resolutions images are rendered that are dynamically combined into the final
high resolution image. We also implemented progressive rendering as part of
our webviewer. Independent of the method progressive upsampling speeds up
rendering in proportion to the ratio of target resolution and initial render
resolution. For simplicity, we disable progressive rendering for benchmarking.