# A revised upper limit to energy extraction from a Kerr black hole

Jeremy D. Schnittman  Gravitational Astrophysics Laboratory, NASA Goddard
Space Flight Center, Greenbelt, MD 20771  Joint Space-Science Institute (JSI),
College Park, MD 20742

###### Abstract

We present a new upper limit on the energy that may be extracted from a Kerr
black hole by means of particle collisions in the ergosphere (i.e., the
“collisional Penrose process”). Earlier work on this subject has focused
largely on particles with critical values of angular momentum falling into an
extremal Kerr black hole from infinity and colliding just outside the horizon.
While these collisions are able to reach arbitrarily high center-of-mass
energies, it is very difficult for the reaction products to escape back to
infinity, effectively limiting the peak efficiency of such a process to
roughly $130\%$. When we allow one of the initial particles to have impact
parameter $b>2M$, and thus not get captured by the horizon, it is able to
collide along outgoing trajectories, greatly increasing the chance that the
products can escape. For equal-mass particles annihilating to photons, we find
a greatly increased peak energy of $E_{\rm out}\approx 6\times E_{\rm in}$.
For Compton scattering, the efficiency can go even higher, with $E_{\rm
out}\approx 14\times E_{\rm in}$, and for repeated scattering events, photons
can both be produced and escape to infinity with Planck-scale energies.

###### pacs:

04.70.Bw, 97.60.Lf

When two particles scatter or annihilate close to a spinning black hole, it is
possible that one of the products can have negative energy, necessarily
getting captured by the horizon, thereby reducing the black hole’s total mass.
This energy is transferred to the other particle, which then may escape to
infinity with more energy than that of the initial particles. This process,
first identified by Penrose in the context of decay products, has long been
studied as a curious property of Kerr black holes, but is of questionable
astrophysical relevance Penrose1969 .

Notably, early work by Wald Wald1974  showed that a maximum efficiency of
$\approx 121\%$ can be a obtained for a single particle falling from rest at
infinity, then decaying into two equal-mass particles in the ergosphere, one
of which escapes, and one of which is captured. Piran et al found
qualitatively similar results for a variety of scattering and annihilation
processes, with only modest net energy extraction from the black hole despite
arbitrarily large center-of-mass energies Piran1975 ; Piran1977 .

More recently, the subject has received renewed interest following a paper by
Banados, Silk, and West (BSW) Banados2009  that proposed to use extremal Kerr
black holes to probe Planck-scale energies by colliding particles on critical
trajectories just outside the horizon. Numerous subsequent papers pointed out
that, while very large center-of-mass energies are indeed obtainable, the
subsequent redshift of the escaping particles ensures that the net efficiency
of the Penrose process remains limited to only about $130\%$ Jacobson2010 ;
Banados2011 ; Harada2012 ; Bejger2012 ; McWilliams2013 .

One feature that all these papers have in common is their analytic approach to
the problem. Based on strong symmetry arguments, they focus almost exclusively
on very specific geodesic trajectories in the equatorial plane, usually taking
the limit as the point of collision approaches the horizon, where the center-
of-mass energy is highest. In a companion work Schnittman2014 , we have taken
a different approach, calculating numerically the distribution function of
dark matter particles around a Kerr black hole using a fully three-dimensional
Monte Carlo code to integrate a huge sample of random geodesic orbits for both
particles and annihilation photons Schnittman2013 . In doing so, we
consistently found a small number of photons that managed to escape the black
hole with energies well in excess of the theoretical bounds of Harada2012 ;
Bejger2012 .

In this Letter, we focus on these extreme events, again returning to analytic
methods to understand the properties of the colliding particles, their
products, and the dependence on physical parameters such as the black hole
spin. We arrive at the somewhat startling conclusion that for annihilation
between equal-mass particles falling from rest at infinity, the escaping
photons can attain energies equal to six times the total rest mass energy of
the incoming particles. For Compton scattering between a photon and massive
particle, the efficiency can reach nearly $1400\%$! Finally, we show how
repeated scattering events can increase the energy of a photon—and the net
efficiency of the Penrose process—almost without bound.

We begin with a description of geodesic trajectories in the equatorial plane
around a Kerr black hole. From the normalization constraint
$g_{\mu\nu}p^{\mu}p^{\nu}=-m^{2}$ we can write down an effective potential:

$V_{\rm
eff}(r)=k\frac{M}{r}+\frac{\ell^{2}}{2r^{2}}+\frac{1}{2}(-k-\varepsilon^{2})\left(1+\frac{a^{2}}{r^{2}}\right)-\frac{M}{r^{3}}(\ell-a\varepsilon)^{2}\,,$
(1)

where $r$ is the radius in Boyer-Lindquist coordinates, $\ell$ and
$\varepsilon$ are the particle’s specific angular momentum and energy, $M$ and
$a$ are the black hole mass and spin, and $k=0$ for photons and $k=-1$ for
massive particles. We set $G=M=c=1$ throughout this paper. For a specific
choice of $a$, $\ell$, $\varepsilon$, and $k$, we can solve for the radial
turning points by setting $V_{\rm eff}(r)=0$.

Figure 1:  Radial turning points in the effective potential $V_{\rm eff}(r,b)$
for (top) massive and (bottom) massless particles, for a black hole with
maximal spin $a=1$. Any particle in the yellow region can escape from the
black hole, but in the blue regions, only particles with outgoing radial
velocities can escape. The static limit is located at $r=2$ and the horizon is
at $r=1$.

Figure 1 shows these turning points as a function of the impact parameter
$b\equiv\ell/\varepsilon$ for both massless and massive particles, for maximal
spin $a=1$. For the massive particles we set $\varepsilon=1$, corresponding to
a particle at rest at infinity. One can imagine a massive particle incoming
from the right with $b<-2(1+\sqrt{2})$ or $b>2$, reflecting off the
centrifugal potential barrier and returning back to infinity (yellow regions).
Alternatively, if the impact parameter is small enough (i.e.,
$-2(1+\sqrt{2})<b<2$), the particle will get captured by the black hole. Due
to frame-dragging, the cross section for capture is much greater for incoming
particles with negative angular momentum.

For a given $\ell=p_{\phi}$ and $\varepsilon=-p_{t}$, the radial momentum
$p_{r}$ can be determined from the normalization condition
$p_{\mu}p_{\nu}g^{\mu\nu}=k$:

$p_{r}=\pm[g_{rr}(k-g^{tt}\varepsilon^{2}+2g^{t\phi}\ell\varepsilon-g^{\phi\phi}\ell^{2})]^{1/2}\,,$
(2)

and the sign of the root is chosen depending on the criterion described below.

Let us now consider a simple reaction between two particles with momenta
$\mathbf{p}^{(1)}$ and $\mathbf{p}^{(2)}$ that collide to produce two
particles with $\mathbf{p}^{(3)}$ and $\mathbf{p}^{(4)}$. Constraining all
trajectories to the equatorial plane, we have to solve for six unknown
momentum components. Conservation of total momentum provides three
constraints, and the normalization conditions on the two daughter particles
provide two more. We are left with a single free parameter: the angle $\psi$
between $\mathbf{p}^{(3)}$ and the $\hat{\mathbf{r}}$-direction in the center-
of-mass frame.

After solving for the momenta $\mathbf{p}^{(3)}$ and $\mathbf{p}^{(4)}$ of the
products, we determine if they can escape the horizon based on the effective
potential barriers and the sign of the radial velocity component, as evident
in Figure 1. Scanning over all values of the angle $\psi$, we solve for the
maximum energy of escaping particles as a function of radius. Then the peak
efficiency of the reaction is defined as $\eta_{\rm
max}=\varepsilon^{(3)}_{\rm max}/(\varepsilon^{(1)}+\varepsilon^{(2)})$.
Interestingly, as pointed out by Piran1977 ; Bejger2012 , the highest energy
photons that can escape are initially ingoing with $b_{3}>2$, giving
$\eta_{\rm max}\approx 1.3$, while those photons with initially outward-
directed trajectories typically have $\eta_{\rm max}<0.5$.

BSW and much of the subsequent work on this topic focus predominantly on
reactions between equal-mass particles falling from rest at infinity with
critical values of the impact parameters ($b_{1}=2$, $-2(1+\sqrt{2})\leq
b_{2}\leq 2$), which lead to divergent center-of-mass energies just outside
the horizon when $a=1$. Yet the products of such reactions are either
immediately captured by the black hole, or lose most of their energy due to
gravitational redshift before reaching an observer Banados2011 ; Harada2012 ;
Bejger2012 ; McWilliams2013 .

Figure 2:  Peak efficiency for annihilation of equal-mass particles falling
from rest at infinity, as a function of the radius at which the annihilation
occurs. The impact parameter $b_{1}=2$ is fixed at the critical value, and
$b_{2}$ varies. The black hole spin is maximal: $a=1$. (Compare to Fig. 2 of
Ref. Bejger2012 )

In Figure 2 we plot $\eta_{\rm max}(r)$ for the same parameters as in Figure
2b of Ref. Bejger2012 , and reassuringly get identical results. In all cases,
$m_{1}=m_{2}=1$, $m_{3}=m_{4}=0$, $\varepsilon_{1}=\varepsilon_{2}=1$, $a=1$,
$b_{1}=2$, and $p_{r}^{(1)},p_{r}^{(2)}<0$. One interesting result is that,
while the center-of-mass energy increases with smaller $r$, the peak
efficiency is reached somewhat outside the horizon, due to the greater
probability of escape. Note also that the efficiency can only surpass unity
for $r<2$, because the Penrose process only works inside the ergosphere. The
dashed line, for $b_{2}=b_{1}=2$, is physically equivalent to spontaneous
breakup of a single particle into two photons, as in Wald1974 .

While exploring the distribution function and annihilation products of dark
matter around Kerr black holes Schnittman2014 , we consistently found escaping
photons with energies greater than the limits of Bejger et al. Bejger2012 . An
important clue to understanding these seemingly impossible photons came from
Figure 2: the greatest efficiency actually comes from trajectories somewhat
outside the horizon. When considering initial particles with $b>2$ that never
reach the horizon, it soon became obvious that they can reflect off the
effective potential barrier and escape with $p_{r}>0$. When colliding with
particles on incoming trajectories, these “super-critical” particles
contribute some positive radial momentum to the center-of-mass frame, thereby
increasing the range of angles—as measured in the coordinate frame—accessible
to the product photons. This in turn greatly increasing the chance that high-
energy products can escape to infinity (even if they are initially on ingoing
trajectories).

Figure 3:  Peak efficiency for annihilation of equal-mass particles falling
from rest at infinity, as a function of the radius at which the annihilation
occurs. Unlike in Fig. 2, here we allow $p_{r}^{(1)}>0$, which greatly
increases the fraction and energy of escaping photons. The impact parameter
$b_{1}=2$ is fixed at the critical value, and $b_{2}$ varies.

In Figure 3 we again plot the peak efficiency for annihilation reactions
between two equal-mass particles with $\varepsilon_{1,2}=1$. Yet now we set
$b_{1}$ to be infinitesimally larger than the critical value, allowing
$p_{r}^{(1)}>0$. The difference is quite profound. Not only is the peak
efficiency much greater than previously calculated, but it is also attainable
for a wider range of $b_{2}$ values.

Figure 4:  Center-of-mass energy for the annihilation reactions shown in Figs.
2 and 3. The color scheme is as in Fig. 3, with solid lines corresponding to
$p_{r}^{(1)}>1$ and dashed lines for $p_{r}^{(1)}<1$. The highest COM energies
naturally occur for the prograde-retrograde collisions, which are closest to
“head-on.”

To better understand the cause of this increase, in Figure 4 we plot the
center-of-mass energy $E_{\rm com}/m$ as a function of radius, for both
$p_{r}^{(1)}>0$ (solid curves) and $p_{r}^{(1)}<0$ (dashed curves). The values
for $b_{2}$ and the color scheme are identical to Figure 3. Interestingly, the
peak center-of-mass energy is only a factor of $\sim 2$ higher for outgoing
particles relative to ingoing particles. Thus we conclude that the huge
increase in efficiency in our case is due to a higher escape probability for
the products, rather than a much higher initial photon energy.

We also considered Compton-like scattering between a photon and massive
particle in a similar geometry to the annihilation mechanism, with $b_{1}$
just above the critical value, $p_{r}^{(1)}>0$, $b_{2}=-2(1+\sqrt{2})$, and
$p_{r}^{(2)}<0$. The results are shown in Figure 5, where the different curves
correspond to different photon energies $\varepsilon_{1}$, and the efficiency
is still defined as $\varepsilon_{3}/(\varepsilon_{1}+\varepsilon_{2})$111The
“kinks” in the curves correspond to different roots of the quadratic equation
that arise when solving for the momentum constraints and escape conditions for
$\mathbf{p}^{(3)}$. When $\varepsilon_{1}\gg m_{2}$, we find the escaping
photons can reach energies roughly 14 times that of the incoming photons.

Figure 5:  Peak efficiency for Compton-like scattering events between a photon
with $b_{1}=2$, $p_{r}^{(1)}>1$ and a massive particle falling from rest at
infinity with rest mass $m_{2}$ and impact parameter $b_{2}=-2(1+\sqrt{2})$.

As noted above, the highest-energy photons that are able to escape are
actually emitted with ingoing radial momentum before reflecting off the
potential barrier and then reaching infinity. This allows for the fortunate
possibility of multiple scattering events with the same photon: each time, the
photon first reflects off the potential barrier so that $p_{r}^{(1)}>0$, then
scatters off a new infalling particle with $\varepsilon_{2}=1$,
$b_{2}=-2(1+\sqrt{2})$, and the resulting photon has $p_{r}^{(3)}<0$ and
$b_{3}>2$, allowing the process to repeat.

Thus it is possible to reach photon energies on the order of $m_{2}\,10^{N}$
after $N$ scattering events, for a net efficiency of $\eta\approx 10^{N}/N$.
The only limitation, as can be seen in Figure 5, is that the radius at which
such high-efficiency reactions ($\eta\gtrsim 10$) can occur moves steadily
inward with each subsequent scattering event. We call this radius $r_{{\rm
crit},N}$ and find that $\log_{10}(r_{{\rm crit},N}-1)\approx-1-N$.

At the same time, each scattering event deposits some negative angular
momentum into the black hole, reducing its spin. This in turn changes the
effective potential, and increases both the impact parameter and turning
radius for the critical trajectories required for high-efficiency scattering
events. In the limit of $\epsilon\equiv(1-a)\ll 1$, we find the critical value
is $b\approx 2(1+\epsilon^{1/2})$, which corresponds to a turning radius of
$r_{\rm turn}\approx 1+2\epsilon^{1/2}$. Eventually, the spin is reduced to a
point where the turning radius is outside of $r_{\rm crit}$ and the efficiency
begins to suffer (see Fig. 6).

Figure 6:  Spin dependency for efficiency of the scattering events plotted in
Fig. 5 with $\varepsilon_{1}/m_{2}=10$.

Following Thorne1974 , we can write an expression for the spin evolution due
to capture of a particle with angular momentum $\delta J$ and energy $\delta
E$:

$\delta a=-2a\frac{\delta E}{M}+\frac{\delta J}{M^{2}}\,.$ (3)

Assuming each scattering event has an efficiency of $\eta=10$, and we start
with $a=1$, the change in spin due to the $N^{th}$ event is to leading
order222While each post-scattering particle $m_{4}$ deposits a greater amount
of negative angular momentum, it also deposits more negative energy, so the
net change in spin is relatively constant.

$\delta\epsilon_{N}\approx
2\frac{m_{2}}{M}\left(2+\sqrt{2}+\epsilon_{N}^{1/2}10^{N+1}\right)\,,$ (4)

where $1-\epsilon_{N}$ is the spin just before the $N^{th}$ event. Since the
condition $r_{\rm turn}<r_{\rm crit}$ requires
$\epsilon_{N}^{1/2}10^{N+1}<1/2$, we can write

$\epsilon_{N+1}\approx(4+2\sqrt{2})N\frac{m_{2}}{M}\,,$ (5)

which in turn gives the maximum number of high-efficiency scattering events:
$N_{\rm max}\approx\log_{10}(M/m_{2})^{1/2}$.

Taking $m_{2}$ to be the electron mass and $M=10M_{\odot}$ gives $N_{\rm
max}\approx 30$, for a peak energy of $10^{26}$ GeV. Thus photons undergoing
repeated Compton scattering events could not only far surpass the Planck
energy scale, but these hyper-energetic photons could even escape to an
observer at infinity!

By simply relaxing a single assumption about the inital trajectories of
colliding particles, we have identified a mechanism that allows high-energy
particles to be produced by the collisional Penrose process and also escape to
infinity. It is entirely possible that further exploration of parameter space
will find even more extreme cases that have not yet been considered in this
study.

At the same time, it is eminently clear that the most extreme effects will
have little or no application to astrophysical processes, where random
particle trajectories and realistic black hole spins will likely never reach
the critical levels required for such events. For example, quantum effects and
Hawking radiation will naturally spin down even an isolated extremal black
hole Page1976 , and gravitational radiation from the scattering particles
could further degrade the spin Berti2009  (although a carefully constructed
steady-state stream of incoming particles might avoid this problem by removing
the time-varying quadrupole moment). A more astrophysically realistic
calculation of potentially observable signatures from dark matter
annihilations is given in Schnittman2014 .

One might imagine an advanced civilization that can create extremal black
holes and harness them as energy storage devices or particle accelerators. Of
course, there is the problem of spinning down the black hole in the process,
so that only a single extreme event might ever be achieved Berti2009 , but
this too could be avoided by providing a simultaneous source of positive
angular momentum via accretion. The net efficiency would suffer, but such is
the cost of probing Planck-scale physics in the laboratory.

It is a pleasure to acknowledge helpful discussions with A. Buonanno, T.
Jacobson, and J. Silk. Special gratitude is due to HKB”H for providing us a
world full of elegant wonder and beauty. This work was partially supported by
NASA grants ATP12-0139 and ATP13-0077.

## References

  * (1) R. Penrose, Nuovo Cimento 1, 252 (1969).
  * (2) R. M. Wald, Astroph. J. 191, 231 (1974)
  * (3) T. Piran, J. Shaham, and J. Katz, Astroph. J. Lett., 196, 107 (1975)
  * (4) T. Piran and J. Shaham, Phys. Rev. D, 16, 1615 (1977)
  * (5) M. Banados, J. Silk, and S. M. West, Phys. Rev. Lett. 103, 111102 (2009).
  * (6) T. Jacobson and T. P. Sotiriou, Phys. Rev. Lett. 104, 021101 (2010).
  * (7) M. Banados, B. Hassanain, J. Silk, and S. M. West, Phys. Rev. D 83, 023004 (2011).
  * (8) T. Harada, H. Nemoto, and U. Miyamoto, Phys. Rev. D, 86, 024027 (2012).
  * (9) M. Bejger, T. Piran, M. Abramowicz, and F. Hakanson, Phys. Rev. Lett. 109, 121101 (2012).
  * (10) S. T. McWilliams, Phys. Rev. Lett. 110, 011102 (2013).
  * (11) J. D. Schnittman, Astroph. J. in prep. (2014).
  * (12) J. D. Schnittman and J. H. Krolik, Astroph. J. 777, 11 (2013).
  * (13) K. S. Thorne, Astroph. J. 191, 507 (1974).
  * (14) D. N. Page, Phys. Rev. D 14, 3260 (1976).
  * (15) E. Berti, V. Cardoso, L. Gualtieri, F. Pretorius, and U. Sperhake, Phys. Rev. Lett. 103, 239001 (2009).


Chen & Li Learning on the correct class

# Learning on the correct class for domain inverse problems of gravimetry

Yihang Chen11footnotemark: 1 and Wenbin Li11footnotemark: 1

###### Abstract

We consider end-to-end learning approaches for inverse problems of gravimetry.
Due to ill-posedness of the inverse gravimetry, the reliability of learning
approaches is questionable. To deal with this problem, we propose the strategy
of learning on the correct class. The well-posedness theorems are employed
when designing the neural-network architecture and constructing the training
set. Given the density-contrast function as a priori information, the domain
of mass can be uniquely determined under certain constrains, and the domain
inverse problem is a correct class of the inverse gravimetry. Under this
correct class, we design the neural network for learning by mimicking the
level-set formulation for the inverse gravimetry. Numerical examples
illustrate that the method is able to recover mass models with non-constant
density contrast.

##  1 Introduction

The gravimetry approach is of great value in large-scale geophysical
explorations. The inverse problem of gravimetry aims to determine the earth’s
mass distribution from data of the gravity potential or related gravity
quantities (Li and Oldenburg, 1998). Because of the equivalent source
principle of gravity potential, the inverse problem of gravimetry is severely
ill-posed in the Hadamard sense (Isakov, 1990). To regain well-posedness, some
a priori conditions should be imposed on the anomalous mass distribution,
restricting the solution into a correct class. Based on the uniqueness
theorems (Isakov, 1990), Isakov and his collaborators developed a level-set
framework to solve domain inverse problems of gravimetry (Isakov et al., 2011;
Lu et al., 2015; Li et al., 2016; Li and Qian, 2021, 2022), which is one
correct class of inverse gravimetry problems. Consider the volume mass
distribution $\mu=f\chi_{D}$, where $\chi_{D}$ denotes the characteristic
function of a domain $D$, and $f$ is a density-contrast function. Given some a
priori information of $f$ and imposing certain geometric constraints on $D$,
the domain inverse problem of gravimetry admits a unique solution. In the
level-set approach, a level-set function $\phi$ is employed to describe the
unknown domain $D$, and an iterative algorithm is developed to solve the
level-set function by fitting the measurement data.

In recent years, deep learning approaches are widely used to solve inverse
problems in many fields of science. Different from classical iterative
algorithms, an end-to-end learning method aims to develop a neural-network
architecture which can generate recovered solutions directly by inputting
measurement data. The learning approaches have the potential to handle high-
level noise contamination of data; moreover, the end-to-end learning methods
are much more efficient than classical iterative algorithms. Due to these
advantages, using deep neural networks and learning strategies has arisen as a
new trend in the study of inverse gravimetry (Huang et al., 2021; Yang et al.,
2022; Zhang et al., 2022b, a; Zhou et al., 2023). On the other hand, the
learning approaches have limitations in several aspects, including the
reliability of solutions and the generalization ability of networks.
Especially in the inverse gravimetry, due to the severe ill-posedness of the
inverse problem, the end-to-end neural networks may have poor performance when
moving to a test set deviating from the training set. In this work, we try to
deal with this problem in domain inverse problems of gravimetry. Different
from the popular trend that promotes enlarging the training set and employing
diverse models (Huang et al., 2021; Zhou et al., 2023), we propose the
strategy of learning on the correct class.

##  2 Learning on the correct class

The ill-posed inverse problems can be solved only on a conditionally correct
class. For the inverse gravimetry problem, since the solution is theoretically
nonunique, there may exist infinitely many structures that reproduce a given
set of measurement data within the same level of accuracy. Even though the
learning approaches have the magic to produce reasonable solutions at times,
the reliability of solutions is questionable if not restricted on the correct
class.

Let $U$ denote the gravity potential generated by a mass distribution $\mu$
with $\mathrm{supp}\mu\subset\Omega$,

$U(\mathbf{y};\mu)=\gamma\int_{\Omega}K(\mathbf{y},\mathbf{x})\mathrm{d}\mu(\mathbf{x}),$
(1)

where $K(\mathbf{y},\mathbf{x})=K(|\mathbf{y}-\mathbf{x}|)$ is the fundamental
solution of Laplace’s equation,

$K(\mathbf{y},\mathbf{x})=\left\\{\begin{array}[]{ccc}-\frac{1}{2\pi}\mathrm{ln}|\mathbf{y}-\mathbf{x}|&,&\mathbf{x},\mathbf{y}\in\mathbf{R}^{2}\,,\\\
\frac{1}{4\pi|\mathbf{y}-\mathbf{x}|}&,&\mathbf{x},\mathbf{y}\in\mathbf{R}^{3}\,,\end{array}\right.$
(2)

and $\gamma$ is a constant related to the universal gravitational constant.
The vector $\nabla U(\mathbf{y};\mu)$ represents the gravity force at
$\mathbf{y}$. The inverse problem of gravimetry reads as follows: Given the
gravity force $\nabla U(\mathbf{y};\mu)$ on
$\Sigma_{0}\subset\mathbf{R}^{n}\setminus\Omega$, find the mass distribution
$\mu$ with $\mathrm{supp}\,\mu\subset\Omega$.

In this work, we consider a conditionally correct class of the inverse problem
as the volume mass distribution takes the form of $\mu=f\chi_{D}$, where
$f(\mathbf{x})$ is a density-contrast function, and $\chi_{D}$ denotes the
characteristic function of the domain $D$: $\chi_{D}(\mathbf{x})=1$,
$\mathbf{x}\in D$; $\chi_{D}(\mathbf{x})=0$, $\mathbf{x}\notin D$. In this
formulation, we have the following uniqueness results for the inverse problem
of gravimetry (Isakov, 1990; Li and Qian, 2021).

###### Theorem 2.1

Let $\Omega_{0}$ be a convex domain with analytic (regular) boundary,
$\Sigma_{0}\subset\partial\Omega_{0}$ be a nonempty hyper-surface, and
$\Omega\subset\Omega_{0}$ be a bounded domain with connected
$\mathbf{R}^{n}\setminus\overline{\Omega}$. $D\subset\Omega$ denotes the
domain of mass anomaly having piecewise smooth boundaries. Given the gravity
force $\nabla U$ on $\Sigma_{0}$ and given $f\geq 0$ in $\Omega$, the domain
$D$ can be uniquely determined if one of the following constraints is
satisfied:

(1) $D$ is star-shaped with respect to its center of gravity, and $f$ is
constant;

(2) $D$ is convex in one direction, e.g. in $x_{n}$, where $x_{n}$ denotes a
component of the spatial coordinate
$\mathbf{x}=(x_{1},\cdots,x_{n})\in\mathbf{R}^{n}$, and $f$ is constant;

(3) $D$ is convex in $x_{n}$, $f$ does not depend on $x_{n}$, $f\in
C(\Omega)$, and $\Omega\subset\mathrm{supp}\,f$;

(4) $D$ is convex, $f\in L_{1}(\Omega)$, and $0<f$ on $\Omega$.

Theorem 2.1 indicates that the domain of mass can be uniquely determined if
the density-contrast function is given and certain constraints are imposed.
The domain inverse problem is a correct class of the inverse gravimetry. Under
the correct class, the learning approaches are more likely to achieve reliable
solutions.

In the level-set approach (Isakov et al., 2011; Li and Qian, 2021), a level-
set formulation is proposed for the volume mass distribution $\mu=f\chi_{D}$,

$\mu(x)=f(x)H(\phi(x))\,,$ (3)

where $\phi(x)$ is the level-set function and $H(\cdot)$ is the standard
Heaviside function,

$\displaystyle\phi(x)\left\\{\begin{array}[]{ccc}\geq 0&,&x\in\bar{D}\\\
<0&,&x\in\bar{D}^{C}\end{array}\right.,\quad
H(\cdot)=\left\\{\begin{array}[]{ccc}1&,&x\geq 0\\\
0&,&x<0\end{array}\right.\,.$ (8)

Mimic to this expression, we propose the following formula for $\mu$ in the
learning approach,

$\mu(\cdot)=f\,\sigma\left(\tilde{\Lambda}_{\Theta}(\cdot)\right)\,,$ (9)

where $\tilde{\Lambda}_{\Theta}(\cdot)$ denotes a neural network with the
parameter $\Theta$, and $\sigma(\cdot)$ is the sigmoid activation function,

$\sigma(s)=1/(1+e^{-s})\,,$ (10)

which can be viewed as a smooth version of the Heaviside.

Researchers in geophysics have the intuition to consider the so-called sparse
inversion in learning approaches for inverse gravimetry (Huang et al., 2021),
which is essentially the domain inverse problem. While they suggest diverse
models or data augmentation to improve the network generalization ability, we
consider an opposite direction to achieve reliable solutions, i.e. restricting
the solution on the correct class according to uniqueness theorems.

##  3 Network architecture and training

###  3.1 Building training set on the correct class

Consider a given geometry of exploration: $\Omega=(0,1)\times(0,0.5)$ km, and
$\Sigma_{0}=\\{(x_{1},x_{2})\mid 0\leq x_{1}\leq 1,x_{2}=-0.01\\}$ km; $256$
measurements are uniformly sampled along $\Sigma_{0}$. We build the pairs of
mass models and gravity data for the training of neural networks.

According to Theorem 2.1, we construct the mass model $\mu$ on a correct class
of the domain inverse problem. The constraint of star-shape or convexity seems
too strong for the unknown domain $D$, which avoids the condition of
disconnected regions. But the convexity in one direction is reasonable in many
applications. As a result, we focus on situations (2) and (3) in Theorem 2.1,
where (2) can be viewed as a special case of (3).

The left column of Figure 1 shows 4 examples of 20,000 mass models we have
built. The computational domain $\Omega$ is divided into $256\times 128$
grids. The mass model $\mu$ is composed of ellipses and polygons, which are
both convex, and the overall mass model is convex in $x_{2}$, i.e. the
vertical direction. The density-contrast function $f$ is given as a priori
information; e.g. $f(x_{1},x_{2})=1+x_{1}^{2}$ in this application, which
satisfies the constraint (3) of Theorem 2.1. The gravity data $\nabla U$ is
calculated on $\Sigma_{0}$ according to the forward model (1). $0-5\%$
Gaussian noises are added to $\nabla U$, simulating the situation of practical
measurements. The right column of Figure 1 plots the simulated measurement
data of the 4 mass models in the training set.

###  3.2 Network architecture

The end-to-end neural network, $\Lambda_{\Theta}:\ \nabla U\mapsto\mu$, aims
to recover the mass model $\mu$ by inputting the gravity data $\nabla U$. As
shown in the preceding section, learning on the correct class suggests us to
consider the formulation of (9):

$\Lambda_{\Theta}(\cdot)=f\,\sigma\big{(}\tilde{\Lambda}_{\Theta}(\cdot)\big{)}\,,$

where $f$ is the given density-contrast function, $\sigma(\cdot)$ is the
sigmoid activation function mimicking the Heaviside function, and
$\tilde{\Lambda}_{\Theta}(\cdot)$ is a neural network.

We propose to use a modified U-net architecture (Ronneberger et al., 2015) for
$\tilde{\Lambda}_{\Theta}(\cdot)$, and Figure 2 shows the structure. It is a
U-shaped convolutional neural network with 2 channels of input and 1 channel
of output, where the 2 channels of input corresponds to the two components of
$\nabla U$. In most applications, the measurement surface $\Gamma_{0}$ is one-
dimensional lower than the exploration domain $\Omega$. To map from the data
surface $\Gamma_{0}$ to the model domain $\Omega$, the convolutional neural
network $\tilde{\Lambda}_{\Theta}(\cdot)$ must have the ability to expand one
dimension, e.g. $\tilde{\Lambda}_{\Theta}:\
\mathbf{R}^{n-1}\times\mathbf{R}^{n-1}\to\mathbf{R}^{n}$. We employ the
strategy of joining multiple channels to achieve the dimension extension (Yang
et al., 2022). As shown in Figure 2, the layer before the last has $128$
channels, each of which corresponds to a horizontal slice of the model domain
$\Omega$, and the number of channels expands along the $x_{2}$-direction. The
last layer reshapes the output, where a convolution is added after joining the
channels. For rigorous math modeling of U-net architectures and general neural
networks, we refer readers to Bianchi et al. (2023).

###  3.3 Training and optimization

Let $\\{(\nabla U_{s},\mu_{s})\mid s=1,\cdots M\\}$ denote the training set
built on the correct class. The loss function for training is defined as
follows,

$\mathcal{L}_{t}(\Theta)=\frac{1}{|\mathcal{I}_{t}|}\sum_{s\in\mathcal{I}_{t}}\frac{1}{N}\left(\left\|f\,\sigma\big{(}\tilde{\Lambda}_{\Theta}(\nabla
U_{s})\big{)}-\mu_{s}\right\|_{2}^{2}+\lambda_{1}\left\|\sigma\big{(}\tilde{\Lambda}_{\Theta}(\nabla
U_{s})\right\|_{\mathrm{TV}}\right)+\lambda_{2}\|\Theta_{w}\|_{2}^{2}\,,$ (11)

where $\mathcal{I}_{t}\subset\\{1,\cdots M\\}$ denotes a mini-batch of the
index set, $|\mathcal{I}_{t}|$ denotes the batch size, and $N$ is the pixel of
the mass model, e.g. $N=256\times 128$. In (11), the 1st term measures the
discrepancy between the true model $\mu_{s}$ and the neural-network output.
The 2nd term is a regularization aiming to enhance the piecewise constant
property of the recovered domain; since the end-to-end learning approach
generates solutions directly, the regularization is introduced in the training
process. The TV-norm is implemented as:

$\|\mathbf{p}\|_{TV}=\sum_{i_{1},\cdots,i_{n}}\big{(}\sqrt{(\mathbf{p}_{i_{1}+1,\cdots,i_{n}}-\mathbf{p}_{i_{1},\cdots,i_{n}})^{2}+\epsilon}+\cdots+\sqrt{(\mathbf{p}_{i_{1},\cdots,i_{n}+1}-\mathbf{p}_{i_{1},\cdots,i_{n}})^{2}+\epsilon}\,\big{)},$

where $\epsilon$ is a small constant, e.g. $\epsilon=10^{-10}$. The 3rd term
of (11) is a standard regularization for the neural network, where
$\Theta_{w}\subset\Theta$ denotes the parameter of convolution kernels.
$\lambda_{1}$ and $\lambda_{2}$ are two constants controlling the amount of
regularization. Similar to Bianchi et al. (2023), we use a random shuffling
strategy for the sampling of the mini-batch set $\mathcal{I}_{t}$, and the
training parameter $\Theta$ is updated according to the Adam algorithm.

##  4 Results

As shown in the preceding section, we have built 20,000 pairs of mass models
and gravity data on the correct class. We separate them into 3 sets: the
training set of 16,000 pairs, $\\{(\nabla U_{s},\mu_{s})\mid s=1,\cdots
16000\\}$, the validation set of 2000 pairs, $\\{(\nabla U_{s},\mu_{s})\mid
s=16001,\cdots 18000\\}$, and the test set of the remaining 2000 pairs,
$\\{(\nabla U_{s},\mu_{s})\mid s=18001,\cdots 20000\\}$. The neural network
$\Lambda_{\Theta}$ is trained on the 1st set and validated on the 2nd set,
where we take $\lambda_{1}=10^{-2}$, $\lambda_{2}=10^{-4}$, and the batch size
$|\mathcal{I}_{t}|=40$. The trained neural network $\Lambda_{\Theta}$ is then
implemented on the test set of 2000 samples. Figure 3 shows the recovered
solutions in 4 of the 2000 samples. Note that the test set has no intersection
with the training set. To quantitatively evaluate the performance of
reconstructions, we compute the peak signal-to-noise ratio (PSNR) and the
structural similarity index (SSIM). Table 1 lists the average values of PSNR
and SSIM for the reconstructions on the 2000 tests samples.

To test the generalization ability, we further implement the trained neural
network $\Lambda_{\Theta}$ on a set of 200 salt domes, which has mass models
different from those of the training set. The domain of each salt dome is
convex in $x_{2}$, and the density-contrast function is still given as
$f=1+x_{1}^{2}$, satisfying the uniqueness theorem of inverse gravimetry.
Figure 4 shows the results in 4 examples of the 200 samples. Table 1 lists the
average values of PSNR and SSIM for the reconstructions on the 200 salt
models. Considering the ill-posedness of the inverse problem of gravimetry,
the reconstructions are adequate and the learning method is promising.
Moreover, the end-to-end neural network is super efficient.

| on the test set of 2000 samples | on the 200 salt models
---|---|---
average PSNR | 22.64 | 18.62
average SSIM | 0.91 | 0.79
Table 1: List of PSNR and SSIM for the reconstruction results.

##  5 Conclusion

Using learning approaches has arisen as a new trend in the study of inverse
gravimetry, but the reliability of learning approaches is questionable. In a
mathematical view, the ill-posed inverse problems can be solved only on a
conditionally correct class, which should be the same situation when employing
the learning approaches. We propose the strategy of learning on the correct
class for the domain inverse problems of gravimetry. With the density-contrast
function given as a-priori information, the end-to-end neural network is
developed to recover the mass model. The well-posedness theorems are employed
when designing the neural-network architecture and constructing the training
set. Numerical examples demonstrate the efficacy of the proposed method. In
geophysical explorations using gravity data, the correct class of solutions
should be firstly prescribed, and the proposed method is efficient and
promising.

## Acknowledgments

Wenbin Li is supported by Natural Science Foundation of Shenzhen (grant no.
JCYJ20190806144005645).

## References

  * Bianchi et al. (2023) Bianchi, D., Lai, G., and Li, W., 2023, Uniformly convex neural networks and non-stationary iterated network Tikhonov (iNETT) method: Inverse Problems, 39, no. 5, 055002.
  * Huang et al. (2021) Huang, R., Liu, S., Qi, R., and Zhang, Y., 2021, Deep learning 3D sparse inversion of gravity data: Journal of Geophysical Research: Solid Earth, 126, no. 11, e2021JB022476.
  * Isakov et al. (2011) Isakov, V., Leung, S., and Qian, J., 2011, A fast local level set method for inverse gravimetry: Communications in Computational Physics, 10, 1044–1070.
  * Isakov (1990) Isakov, V., 1990, Inverse source problems: American Mathematical Society, Providence, Rhode Island.
  * Li and Oldenburg (1998) Li, Y., and Oldenburg, D. W., 1998, 3-D inversion of gravity data: Geophysics, 63, no. 1, 109–119.
  * Li and Qian (2021) Li, W., and Qian, J., 2021, Simultaneously recovering both domain and varying density in inverse gravimetry by efficient level-set methods: Inverse Problems and Imaging, 15, no. 3, 387–413.
  * Li and Qian (2022) Li, W., and Qian, J., 2022, Kantorovich-rubinstein metric based level-set methods for inverting modulus of gravity-force data: Inverse Problems and Imaging, 16, no. 6, 1643–1667.
  * Li et al. (2016) Li, W., Lu, W., and Qian, J., 2016, A level set method for imaging salt structures using gravity data: Geophysics, 81, no. 2, G35–G51.
  * Lu et al. (2015) Lu, W., Leung, S., and Qian, J., 2015, An improved fast local level set method for three-dimensional inverse gravimetry: Inverse Problems and Imaging, 9, 479–509.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T., 2015, U-net: Convolutional networks for biomedical image segmentation: U-net: Convolutional networks for biomedical image segmentation:, Proc. Med. Image Comput. Comput. Assist. Interv., 234–241.
  * Yang et al. (2022) Yang, Q., Hu, X., Liu, S., Jie, Q., Wang, H., and Chen, Q., 2022, 3-D gravity inversion based on deep convolution neural networks: IEEE Geoscience and Remote Sensing Letters, 19, 3001305.
  * Zhang et al. (2022a) Zhang, L., Zhang, G., Fan, Z., and Ma, J., 2022a, A multitask deep learning for simultaneous denoising and inversion of 3-D gravity data: IEEE Transactions on Geoscience and Remote Sensing, 60, 5923117.
  * Zhang et al. (2022b) ——– 2022b, Deep learning for 3-D inversion of gravity data: IEEE Transactions on Geoscience and Remote Sensing, 60, 5905918.
  * Zhou et al. (2023) Zhou, X., Chen, Z., Lv, Y., and Wang, S., 2023, 3-D gravity intelligent inversion by U-Net network with data augmentation: IEEE Transactions on Geoscience and Remote Sensing, 61, 5902713.

(a) (b) (c) (d)

Figure 1: Training set on the correct class. (a)-(d) show 4 examples of the
20,000 models we have built. The left column plots the mass distributions
$\mu$, and the right column plots the simulated measurement data $\nabla U$
with $0-5\%$ Gaussian noises. Figure 2: A modified U-net architecture for
$\tilde{\Lambda}_{\Theta}(\cdot)$. The overall neural network for training is
$\Lambda_{\Theta}(\cdot)=f\,\sigma\big{(}\tilde{\Lambda}_{\Theta}(\cdot)\big{)}$.

(a) (b) (c) (d)

Figure 3: Results 1: The trained neural network $\Lambda_{\Theta}$ is
implemented on the test set of 2000 samples. (a)-(d) show 4 examples of the
2000 test samples. The 1st column plots the measurement data with $0-5\%$
Gaussian noises, which are the inputs of $\Lambda_{\Theta}$; the 2nd column
plots the recovered solutions; the 3rd column plots the true models.

(a) (b) (c) (d)

Figure 4: Results 2: To test the generalization ability, the trained neural
network $\Lambda_{\Theta}$ is implemented on the set of 200 salt models.
(a)-(d) show 4 examples of the 200 salt models. The 1st column plots the
measurement data with $0-5\%$ Gaussian noises, which are the inputs of
$\Lambda_{\Theta}$; the 2nd column plots the recovered solutions; the 3rd
column plots the true models.


(a) Off-center perspective projections of a panorama contain barrel
distortions and break realism. (b) With our distortion-reduction measures,
distortions are significantly reduced and the visualization is more like a
genuine perspective projection in which linear lines in 3D remain linear in
2D. In comparison, augmenting the panorama with (c) depth or (d) room layouts
do not lead to visually convincing results. Observe the broken lamp wires and
doorway in (d). We show the panorama and room layout (predicted by LED2-Net
[WYS*21]) and the camera position and looking direction on the left. The
panorama and the ground-truth depth are from the Structure3D dataset [ZZL*20].

# Distortion Reduction for Off-Center Perspective Projection of Panoramas

Chi-Han Peng1 and Jiayao Zhang2
1National Chiao Tung University 2King Abdullah University of Science and
Technology

###### Abstract

A single Panorama can be drawn perspectively without distortions in arbitrary
viewing directions and field-of-views when the camera position is at the
origin. This is a key advantage in VR and virtual tour applications because it
enables the user to freely "look around" in a virtual world with just a single
panorama, albeit at a fixed position. However, when the camera moves away from
the center, barrel distortions appear and realism breaks. We propose
modifications to the equirectangular-to-perspective (E2P) projection that
significantly reduce distortions when the camera position is away from the
origin. This enables users to not only "look around" but also "walk around"
virtually in a single panorama with more convincing renderings. We compare
with other techniques that aim to augment panoramas with 3D information,
including: 1) panoramas with depth information and 2) panoramas augmented with
room layouts, and show that our approach provides more visually convincing
results. <ccs2012> <concept>
<concept_id>10010147.10010371.10010387.10010866</concept_id>
<concept_desc>Computing methodologies Virtual reality</concept_desc>
<concept_significance>500</concept_significance> </concept> </ccs2012>

[500]Computing methodologies Virtual reality

##  1 Introduction

A spherical panorama stores incoming ray intensities toward a fixed camera
point in all possible directions. Commonly, the directions are sampled on a
sphere (centered at the point) in a 2D equirectangular format. Using just a
single panorama, one can render accurate perspective projections from the
camera point in arbitrary viewing directions and field-of-views through
sampling strategies commonly known as the equirectangular-to-perspective (E2P)
projection. In VR and virtual tour applications, this is known as the 3DoF
(i.e., rotations along three axis) of a perspective camera in a virtual
environment, albeit at a fixed position.

It is actually straightforward to render perspective projections from an off-
center position, although the results would contain barrel distortions. One
way is to map the panorama onto a spherical mesh through projective texture
mapping and draw perspective views using standard perspective camera models
such as OpenGL. An analytical form of the off-center E2P projection is
described in Section 3.1. The rendering results (see the accompanying video
for examples) are "intuitive" in general, i.e., scenery becomes bigger when
the camera moves closer and the scenery moves to the right when the camera
moves to the left, and vice versa. Parallax effect and occlusion on-and-off
are lacking. However, by our observations, lacking them usually doesn’t ruin
the plausibility of the rendering, unless there are large variations in the
depth disparity (e.g., having objects very close to the camera).

We opt for solving another major defect of the rendering - the barrel
distortions (e.g., Figure Distortion Reduction for Off-Center Perspective
Projection of Panoramas (a)), which are inevitable when viewing a textured
sphere from an interior position away from the center. The reason why barrel
distortions break realism is that straight lines in 3D no longer remain
straight in the 2D view, breaching a key assumption of perspective
projections.

Such task is often called the novel view synthesis problem in visual
computing. Powerful methods such as [XZX*21] leverage deep learning (DL)
models and comprehensive data to synthesize novel views of the same scene that
extrude parallax effect and even contain scenery that was previously occluded.
These DL-based methods either take panoramic videos as input (e.g., "one-shot
3D photography" [KMA*20]), or need a dataset trained on such ([XZX*21]). In
comparison, our method takes a lightweight approach to the problem (no DL
training/inference nor video-sequence inputs are needed, and is very
computationally light), with a focus on tackling barrel distortions.

We introduce two modifications to the standard E2P projection process. First,
in order to preserve the linearity of vertically straight lines in 3D in the
2D view, we re-project the panorama to a cylinder so that all vertically
straight lines in 3D, which are necessarily mapped to meridians on the sphere,
would be projected to vertically straight lines in 3D again (i.e., the
straight lines on the cylinder), which are guaranteed to be projected to
straight lines in 2D. Second, we introduce computational dolly-zoom effects to
find alternative camera position and field-of-view for the same viewing region
that minimizes barrel distortions. In summary, the first measure effectively
eliminates barrel distortions of vertical lines while the second measure
reduces distortions in both horizontal and vertical directions to a degree.

The endgame for the novel view synthesis problem is to build an accurate 3D
model of the scene. To our best knowledge, [XZX*21] is the only DL-based work
that takes a single panorama as input for such a goal. However, their code is
not yet available. Therefore, we opt for comparing to two common strategies to
build a rough 3D model out of a single panorama: 1) panoramas with per-pixel
depths (ground-truth or predicted by neural networks), and 2) augmenting
panoramas with room layouts. In our experiments (see Section 4), we show that
both strategies are inadequate to produce good quality rendering results.

The paper outline is as follows. In Section 2, we describe related work in the
novel view synthesis problem based on panoramic inputs. In Section 3, we
describe our two distortion reduction measures and provide a thorough
analysis. In Section 4, we compare the renderings of off-center perspective
projections of our method to vanilla E2P projections and common approaches to
build 3D panoramas. We further compare to results of [XZX*21] (by using
panorama images taken from their paper) and discuss the pros and cons. Results
on a large variety of panoramas taken from different datasets are shown.
Finally, we conclude the paper in Section 5.

##  2 Related work

###  2.1 Panoramic 3D modeling and datasets

3D Modeling of indoor scenes based on spherical panoramic (also called "360°"
in commercial settings) image inputs is a popular field in recent years. Key
tasks include depth estimation [PAA*, AZD*21, WYS*20, LRSK19], room layout
estimation [LXM*20, XZX*21, YWP*19], object detection and segmentation [SSC21,
XZH*18], and more generally 3D reconstruction tasks such as registration of
multiple panoramas [YLC*20, CCYC21].

A number of panoramic image datasets have been produced to aid the research.
Matterport3D [CDF*17] and Standford2D3D [ASZS17] are real-world large-scale
RGB-D datasets, which provide panoramic views of diverse indoor scenes. They
include various kinds of 2D and 3D semantics, meshes, and even video
walkthroughs. Structure3D [ZZL*20] and SunCG [ZKZD18] dataset are synthetic
datasets with richly decorated indoor scenes. They provide realistically
rendered indoor RGB-D images and annotations of 3D structures. 3D60 [ZKZ*19]
is a collective dataset with 3 different modalities (color, depth and normal)
and comprises of realistic and synthetic 3D datasets (Matterport3D,
Stanford2D3D, and SunCG). Gibson [XZH*18] is a real-world dataset that
includes high-quality RGB panoramas, global camera poses, and 3D meshes.

###  2.2 Novel view synthesis

Novel view synthesis is one of the core tasks in visual computing. We limit
our scope to methods rely on panoramic image inputs. Layered Depth Images
(LDI) [SGHS98] and Multiplane Images (MPI) [ZTF*18] are used as image-based
representations for novel view synthesis, but for large translations, they
might lack sufficient information to render correctly. Multi Depth Panoramas
(MDPs) [LXM*20] and PerspectiveNet [NGR19] comprise of multi-RGBD$\alpha$
images for high-quality and efficient novel view generations. Hedman et al.
[HK18] input burst of aligned color-and-depth photos to generate 3D panorama,
and their 3D effects could also interact with the scene. Gao et al. [GSKH21]
propose an algorithm to generate novel views from dynamic monocular videos. Xu
et al. [XZX*21] make the first attempt to generate a target-view panorama from
one single source-view panorama with a large camera translation. Jin et al.
[JXZ*20] and Zeng et al. [ZKG20] leverage the geometric structure of a 360°
indoor image for depth estimation.

Attal et al. [ALG*20] simultaneously learn depth and occlusions via a multi-
sphere image representation, which could greatly handle occluded regions in
dynamic scenes. With 46 input light field video, Broxton et al. [BFO*20]
present a system that is able to reproduce view-dependent reflections, semi-
transparent surfaces, and near-field objects. Tobias et al. introduce
OmniPhotos [BYLR20] for quickly and casually capturing 360° VR panoramas, and
improve the visual rendering quality by alleviating distortion using a novel
deformable proxy geometry. Serrano et al. [SKC*19] present a device which
enable head motion parallax in 360°video, thus tackled silhouettes and
occlusions. Other works extend these approach to point clouds, Aliev et al.
[ASK*20] present a point-based approach to generate novel views of the scene.
Voxel grid-based methods such as DeepVoxels [STH*19] encodes the view-
dependent appearance of a 3D scene as 3D voxels. Implicit function-based
methods such as Sitzmann et al [SZW19] propose a continuous, 3D structure-
aware scene representation that encodes both geometry and appearance.

Overall, nearly all existing methods reply on input data that consists of
multiple panoramas (taken at different camera positions), often as panoramic
videos shoot either from a single moving 360° camera or an array of fixed (or
even moving) cameras (capturing rigs). One exception is [XZX*21], in which
they rely on DL-based methods to synthesize novel views from just a single
panorama. Our work is most similar to theirs.

##  3 Method

We first describe the analytical form of the vanilla off-center E2P projection
in Section 3.1. We then describe our two measures to reduce barrel distortions
in Section 3.2 and 3.3. Each approach can be applied independently. We assume
a system would apply both unless otherwise specified.

###  3.1 Off-center E2P projection

Figure 1:  An off-center E2P projection. $P$ is the camera center and $I$ is
the intersection of the looking ray and the unit sphere. We use a right-handed
system with +z direction at azimuth=0° and zenith=0° and +x direction at
azimuth=0° and zenith=90°. The viewing pyramid is drawn in blue. Red lines
show a column in the view plane that depicts a non-straight geometry in the 3D
scene due to barrel distortions.

Without loss of generality, we assume the panorama is mapped to a unit sphere
centered at the origin. We assume the perspective camera’s position is
$P=(p_{x},p_{y},p_{z})$, looking direction is $Dir=(dir_{x},dir_{y},dir_{z})$,
and up direction is $Up=(up_{x},up_{y},up_{z})$. $Up$ is orthogonal to $Dir$.
We assume the image plane is rectangular and its local y-axis is aligned to
the up direction. The dimensionality of the image plane can be described by
two angles $fovx$ and $fovy$, i.e., the horizontal and vertical angles of the
pyramid formed by the camera center and the view plane. Therefore, the width
and height of the image plane are $width=near\cdot tan(fovx/2)\cdot 2$ and
$height=near\cdot tan(fovy/2)\cdot 2$, respectively. $near$ is the distance
between the camera position and the view plane. We denote the "left" direction
as $Left=Up\times Dir$ by right-handed cross product. See Figure 1 for an
example.

To retrieve the ray intensity (e.g., color or depth) at a normalized 2D
position on the image plane, $(X,Y)$, $0\leq X,Y\leq 1$, $X$ is left to right
direction and $Y$ is top to bottom direction, we derive the corresponding 3D
ray direction, $Ray=(ray_{x},ray_{y},ray_{z})$, as:

$|Dir\cdot near+Left\cdot(0.5-X)\cdot width+Up\cdot(0.5-Y)\cdot height|,$

which can be shortened as:

$|Dir+Left\cdot(1-2\dot{X})\cdot tan(fovx/2)+Up\cdot(1-2\dot{Y})\cdot
tan(fovy/2)|.$ (1)

To draw a perspective projection, for each 2D position $(X,Y)$ on the image
plane (e.g., a pixel), our goal is to find the spherical coordinate,
$(\theta,\phi)$, of the point $I$ on the unit sphere that intersects with the
ray from the camera position to the point on the 2D image plane. Recall that
$\theta$ is the zenith (angle from the +z axis), $\phi$ is the azimuth (angle
of counterclockwise rotation along +z-axis from the +x axis), and the
corresponding 2D coordinate in equirectangular projection can be trivially
derived as $(\frac{\phi}{2\pi},\frac{\theta}{\pi})$.

We find the 3D coordinate of $I$ by solving $t$ in $I=P+t\cdot Ray$, $|I|=1$.
We have $a=ray_{x}^{2}+ray_{y}^{2}+ray_{z}^{2}$, $b=2\cdot(pos_{x}\cdot
ray_{x}+pos_{y}\cdot ray_{y}+pos_{z}\cdot ray_{z})$, and
$c=pos_{x}^{2}+pos_{y}^{2}+pos_{z}^{2}-1$. $t$ equals
$(-b+\sqrt{b^{2}-4ac})/2a$ (i.e., we take the positive-signed solution).
Finally, convert $I$’s 3D coordinate to spherical coordinate $(\theta,\phi)$.

Note that the above calculation can be done in the OpenGL rendering pipeline
by setting up a unit spherical model centered at origin with the panorama as
the 2D texture and equirectangular coordinates as UV coordinates, and drawing
the 2D view defined by a view frustum defined by the aforementioned view
pyramid cut through the near plane. A pixel shader is used to calculate the UV
coordinates per pixel.

###  3.2 Cylindrical projection

Figure 2:  Off-center cylindrical projections. (a): A rendering using the same
camera position and looking direction as in Figure 1. All vertical features in
the scene are now drawn as straight lines. (b): After the camera is pitched
downward. The vertical features are still drawn as 2D straight lines, but not
vertical. (c): After the camera is rolled clockwise. Again, vertical features
are drawn as 2D straight lines.

Modern 360° cameras have strong image stabilization features to ensure that
the shoot panoramas are nearly in upright position, i.e., the -z direction in
3D or the +y direction in equirectangular projection are aligned to the
direction of gravity. There also exist algorithms to further transform a
panorama to make it in upright position (e.g., the "camera rotation pose
alignment" in [SHSC19]). This means that vertical features in 3D in the scene,
such as wall corners, are nearly always mapped to vertical lines in the
panorama. However, this doesn’t mean that they are mapped to vertical lines in
3D through the equirectangular projection. Instead, they are mapped to
meridians on the unit sphere, which are curved in 3D and are perspectively
drawn as curved lines in 2D except when the camera position is at the origin.

We propose a simple solution to ensure that vertical features in the scene are
perspectively drawn as straight lines in 2D even when the camera is not at the
origin: projecting the panorama to a cylinder. We choose the cylinder to be of
radius $1$, centered at the origin, in the "upright" position (i.e., axis is
aligned to the z axis), and of an infinite height. We now have off-center
equirectangular-to-cylinder (E2C) projection as follows.

Equation 1 again describes the 3D ray direction, $Ray$, of a 2D position
$(X,Y)$ on the image plane of the camera. Its intersection point with the
cylinder, $I^{c}$, is calculated by solving $t^{\prime}$ in
$I^{c}=P+t^{\prime}\cdot Ray$, $|I^{c}_{x}+I^{c}_{y}|=1$. We now have
$a^{\prime}=ray_{x}^{2}+ray_{y}^{2}$, $b^{\prime}=2\cdot(pos_{x}\cdot
ray_{x}+pos_{y}\cdot ray_{y})$, and $c=pos_{x}^{2}+pos_{y}^{2}-1$.
$t^{\prime}$ equals $(-b^{\prime}+\sqrt{b^{\prime
2}-4a^{\prime}c^{\prime}})/2a^{\prime}$. Next, convert $|I^{c}|$ to spherical
coordinate $(\theta,\phi)$, and use it to sample a ray intensity in the
panorama.

We have the following lemma:

###### Lemma 3.1

A vertical feature in 3D in the scene is perspectively drawn as a 2D straight
line under a cylindrical projection.

######  Proof 3.2.

Recall that a vertical feature in 3D is necessarily drawn as a vertical line
in the 2D panorama under equirectangular projections, which is then mapped to
a part of a meridian on the sphere. Trivially, a cylindrical projection maps
every meridian (and parts of it) on the sphere to a vertical straight line on
the cylinder. Finally, recall that perspective projection preserves the
linearity of any 3D straight lines in 2D perspective views.

Note that Lemma 3.1 applies for cameras with arbitrary rolls, pitches, and
yaws, not just "upright" cameras with a left direction perpendicular to the z
axis. See Figure 2 for an example.

To realize the above calculation in the OpenGL rendering pipeline, replace the
spherical model with a cylindrical model. The model is textured using the
panorama as the 2D texture and "normalized" equirectangular coordinates, which
are based on the spherical coordinates of the normalized 3D point positions,
as UV coordinates.

In summary, Lemma 3.1 states that any vertical features in the 3D scene are
guaranteed to be drawn as straight lines in the 2D perspective views. However,
the inverse is not necessarily true - everything depicted on a column in the
panorama would remain straight in 2D perspective views, no matter they are
genuinely vertically aligned in 3D or not. In practice, we observe that the
renderings largely remain smooth and intuitive.

###  3.3 Computational dolly-zoom effect

In cinematography, a "dolly-zoom" effect refers to the act that the field-of-
view (FOV) angle is continuously narrowed while the camera is moving away from
an object in the scene, or vice versa, in a calibrated manner such that the
object would appear at the same size on the 2D frame during the movement. The
goal is to create so called "perspective distortions" in which the relative
sizes of other objects in the scene change w.r.t. to the size of the
particular object.

Recall that in perspective drawing of panoramas, in general, the amount of
distortions is proportional to the distance between the camera position and
the origin (zero when they coincide). Therefore, our main idea is to leverage
dolly-zoom principles to draw roughly the same subset of the panorama in the
image plane but from a camera position that is as close to the origin as
possible. We call the new camera position, $P^{h}$, the heuristic solution to
the computational dolly-zoom problem.

Figure 3:  Inspired by the dolly-zoom effects in cinematography, we find an
alternative camera position and FOV angles setting that with which the
perspective camera draws roughly the same subset of the panorama but with less
barrel distortions. The original camera is shown in blue and the adjusted
camera is shown in cyan. Observe that the two drawn images have roughly the
same boundary and center. Just that the images are stretched in different
ways.

To elaborate, $P^{h}$ is the solution to the following optimization problem:

$\displaystyle\underset{t}{\text{argmin}}$ $\displaystyle|P^{h}|$ (2) subject
to $\displaystyle P^{h}=P+t\cdot Dir$

where $P$ is the original camera position and $Dir$ is the looking direction.
Given the new camera position $P^{h}$, we solve the new FOV angles as follows.
First, we denote the "left-middle" and "right-middle" viewing rays,
$Ray_{left}$ and $Ray_{right}$, as the rays from $P$ toward the left-middle
and right-middle points of the original image plane. See Figure 3 for an
example. Next, we find the intersections of $Ray_{left}$ and $Ray_{right}$ to
the sphere or cylinder (depending on which projection scheme is used), denoted
as $I_{left}$ and $I_{right}$, respectively. Our goal is that in the new
viewing pyramid, the new left-middle and right-middle viewing rays should
intersect the sphere/cylinder at the same positions. Therefore, the new left-
middle viewing ray, $Ray_{left}^{new}$, is the ray from $P^{h}$ toward
$I_{left}$. The new right-middle viewing ray, $Ray_{right}^{new}$, is the ray
from $P^{h}$ toward $I_{right}$. Finally, we calculate the new "left-
horizontal" and "right-horizontal" FOV angles, $fovx_{left}$ and
$fovx_{right}$, as the angles between $Dir$ and $Ray_{left}^{new}$ and between
$Dir$ and $Ray_{left}^{new}$, respectively. The new vertical FOV angle is
calculated as $atan(\frac{(tan(fovx_{left})+tan(fovx_{right}))}{aspect})$,
$aspect$ is the aspect ratio of the original image plane. Note that the new
viewing pyramid could be skewed as $fovx_{left}$ and $fovx_{right}$ are not
necessarily the same.

####  3.3.1 Performance analysis

Figure 4:  Distortion analysis of perspective views of a original camera (a)
and the dolly-zoom adjusted version (b). The original camera has a distortion
value of $0.00012$ while the adjusted version’s amount is just $4.413e-5$.
Blue grids come from bilinear interpolations of the four projected corners of
the viewing pyramids. Red grids are their projections to the cylinder. Grey
lines connect the grid vertices to the origin while cyan lines connect them to
the camera positions. In short, the cyan grids in perspective views visualize
how regular grids on each camera’s image plane (blue) are first projected to
the cylinder (red) and then projected back to the respective image planes,
causing distortions. We used a coarser grid for visualization with less
clutter.

We propose the following scheme to evaluate how the heuristic solution reduces
distortions. In short, for a given camera position, looking direction, and FOV
angles (a "camera pose" in short), we estimate a subset of the sphere/cylinder
that is projected to roughly cover the whole image plane. We subdivide the
subset to be a regular grid. We then calculate the curvature of the grid’s 2D
projection on the image plane as the way to measure the "distortion value" of
the camera.

How to estimate such a subset of the sphere/cylinder? We first project the
four corners of the view plane onto to the sphere/cylinder. Afterwards, we
build a nearly-planar regular grid through bilinear interpolation of the four
projected corners. We denote the regular grid as a 2D array of vertices
$v[i,j]\in R^{3}$, $0\leq i\leq ROWS,0\leq j\leq COLS$. In practice, the
vertices are projected onto the sphere/cylinder in a panoramic image, so the
3D position of the $(i,j)$-th vertex is actually $v^{\prime}[i,j]=|v[i,j]|$.
Note that this means that the grid becomes a discrete curved surface and the
2D projections of its rows and columns of vertices would not be straight
unless it is viewed exactly from the origin. Finally, to evaluate the
curvature of the grid’s 2D projection on the image plane, we sum up a
linearity measurement (Equation 3.3.2) of the consecutive edges of every rows
and columns in the grid. See Figure 4 for an example.

At a particular camera pose, the distortion improvement is the distortion
value of the original camera pose minus the distortion value of the dolly
zoom-adjusted camera pose. As shown in Figure 6, we measure the distortion
improvements at every possible camera poses up to rotational and reflective
symmetries in a sphere, sampled to avoid clutter. We can see that the
improvements vary greatly at different camera poses. For example, distortions
are reduced to zero when we "dolly" the camera (i.e., moving forward or
backward without changing the looking direction). However, improvements are
non-existent if we "truck" the camera (i.e., moving sideways while fixing the
looking direction). We leave finding an analytical form to explain the
distribution of improvements among different camera poses to future work.

####  3.3.2 Optimization-based solution

Figure 5:  The derivation of the 3D-to-2D perspective projection formula
(Equation 3). We show the derivation of $V_{x}$ of a point on the cylinder
(red dot) and assume $Aspect=1$. The distance to the view plane, $near$, is 1.
Left: an original camera without position offsets (i.e., $t=0$). $h$ equals
$\frac{v^{\prime\prime}_{x}}{-v^{\prime\prime}_{z}}$ ($v^{\prime\prime}_{z}$
is negative). Therefore, $V_{x}$ equals $\frac{h}{\frac{H}{B}}$ (i.e., we need
to normalize $h$ by the half-width of the view plane). Right: the camera is
moved along the z-axis by amount $t$. $h$ now equals
$\frac{v^{\prime\prime}_{x}}{-v^{\prime\prime}_{z}+t}$. The new half-width of
the view plane becomes $\frac{H}{B+t}$. Therefore, we have
$V_{x}=\frac{v^{\prime\prime}_{x}}{\frac{H}{B+t}\cdot(-v^{\prime\prime}_{z}+t)}$.
Figure 6:  Comparing distortions of various camera models. (a) to (c): the
distortion values of the original camera model, heuristic solution, and
optimized solutions with cylindrical projection. We only show results up to
rotational and reflective symmetrices in a sphere. Therefore, we sample the
camera positions on a radially sampled first quartile of the Y-Z plane
intersect with the unit sphere. At each position, we sample all possible
looking directions on the half hemisphere facing the +x direction. The
distortion values are converted to colors from blue (smallest) to yellow to
red (biggest). For clarity, we cap the upper bound (there exist some very
large distortion values so using the full range would result in all blue-ish
colors). We can see that optimized solutions clearly have the lowest
distortion values among the three. While not clearly shown by the
visualizations here, heuristic solutions actually win over the original camera
models (see Table 1 for statistics). In (d) and (e), we show the distortion
improvements of the optimized solutions over the original and the heuristic
solutions, respectively.

The aforementioned evaluation scheme inspired us to formulate the task of
finding $t$ as an optimization problem to minimize barrel distortions. In
short, we have a single variable $t$ (i.e., offset of the camera position
along the looking direction). Given the 3D positions of the grid vertices,
$v^{\prime}[i,j]$, as constants, we first transform them to a new coordinate
space that is the same as the view space in OpenGL. That is, the camera
position is aligned to the origin, the three axis, $Left$, $Up$, and $Dir$,
are aligned to the -x-axis, +y-axis, and -z-axis, respectively, through a
rotation and a translation (e.g., by applying OpenGL’s MODELVIEW transform
matrix). We denote the transformed grid vertex positions as
$v^{\prime\prime}[i,j]\in R^{3}$. In this manner, we can concisely formulate
their 2D projected positions onto the image plane, $V[i,j]\in R^{2}$, as a
function of $t$ as follows:

$\displaystyle
V[i,j]_{x}=\frac{v^{\prime\prime}[i,j]_{x}}{Aspect\cdot\frac{H}{B+t}\cdot(-v^{\prime\prime}[i,j]_{z}+t)},$
(3) $\displaystyle
V[i,j]_{y}=\frac{v^{\prime\prime}[i,j]_{y}}{\frac{H}{B+t}\cdot(-v^{\prime\prime}[i,j]_{z}+t)},$

where $B$ denotes the distance of the grid plane to the origin in the new
coordinate space. $H$ equals $B\cdot tan(fovy/2)$. $B$ and $H$ can be
understood as the base and height of the triangle from the origin to half of
the grid plane (see Figure 5 for an illustration).

The objective function to minimize the sum of the linearity measurement of
consecutive edges of every rows and columns in the grid is formulated as
follows:

$\displaystyle Obj=$
$\displaystyle\forall_{i}\;\Sigma_{j=[1,COLS-1]}angle(V[i,j-1],V[i,j],V[i,j+1])$
$\displaystyle+$ (4)
$\displaystyle\forall_{j}\;\Sigma_{i=[1,ROWS-1]}angle(V[i-1,j],V[i,j],V[i+1,j])$
,

where $linearity(a,b,c)$, $a,b,c\in R^{2}$, denotes the linearity measurement
of edge $a,b$ and edge $b,c$. We formulate it as:

$\displaystyle linearity(a,b,c)=$ $\displaystyle
pow((b_{x}-a_{x})(c_{y}-a_{y})-$ $\displaystyle(c_{x}-a_{x})(b_{y}-a_{y}),2).$

To sum up, the optimization problem takes the form:

$\displaystyle\underset{t}{\text{argmin}}$ $\displaystyle
Equation\leavevmode\nobreak\ \ref{equ:dolly2}$ (5) subject to $\displaystyle
Equation\leavevmode\nobreak\ \ref{equ:dolly1}$
$\displaystyle\forall\;V[i,j]\in grid.$

After $t$ is solved, the new camera position and FOV angles are derived
accordingly.

In Figure 6, we compare the distortion improvements of the original camera,
heuristic solutions, and the optimized solutions at different camera poses. We
find that the optimized solutions improves upon both the original and the
heuristic solutions. In our comparisons, we used 10 for $ROWS$ and $COLS$. We
experimented with other grid resolutions and the results are similar. We
conclude that using the optimization approach led to better results at a small
computational cost.

##  4 Results and comparisons

Figure 7: Our attempts to recreate the same novel view synthesis results in
Figure 5 in [XZX*21]. For each row, left: a perspectives view of a ground
truth "target" panorama. Middle: the same perspective view of the synthesized
panorama generated by their method using a different "source" panorama and the
camera pose of the "target" panorama. Right: our result with our best effort
to find the corresponding camera pose. Our results aren’t as geometrically
correct as theirs but are free of blurs and ghosts. The red markers can be
ignored.

We first compare our method to [XZX*21]. As shown in Figure 7, we attempt to
create the same novel view synthesis results as done in their paper. In
general, we find their results to be quite good in getting the overall 3D
layouts correct. However, two shortcomings are: 1) the occasional blurs and
ghosts, and 2) the resolution of their predicted panoramas is only 512x256,
too low for drawing perspective views. In comparison, our results are not as
geometrically accurate as theirs, but are of higher resolutions (since the
source panoramas tend to have higher resolutions than the predicted
panoramas), and largely free of any visual artifacts.

In Figure 8, we compare off-center perspective projection results of our
method with the vanilla E2P projection and two common approaches to augment
panoramas with 3D information (per-pixel depth and room layouts). We test on
panoramas from the Structure3D dataset [ZZL*20] (synthetic), the Gibson
dataset [XZH*18] (real-world), and several panoramas shoot by ourselves (we
used a RICOH THETA Z1 360∘ camera). Note that the teaser (Figure Distortion
Reduction for Off-Center Perspective Projection of Panoramas) is also based on
the Structure3D dataset. The depth information are either ground truth
(available in the Structure3D dataset only) or predicted by a state-of-the-art
depth prediction model ("MiDaS v3.0" [RBK21]) that we found to be accurate and
robust in general. We also tried another modern depth prediction model
specifically for panoramas ( [WYS*20]) but found the results to be less
accurate. The layouts are either ground truth (available in the Structure3D
dataset only) or predicted by LED2-Net [WYS*21]. In summary, our results
significantly reduced the barrel distortions produced by the vanilla E2P
projection. Augmenting panoramas with depths or room layouts provides more
realistic 3D effects such as parallax and occlusion on-and-offs. However,
glaring artifacts, such as blurs (happen when viewing "extruded" pixels by
depths from sideways) and broken images features (happen when room layouts
mismatch 3D objects, such as furniture, in the scene), may happen. See the
accompanying video for animated versions of the results. We also provide a
computer program for readers to try out on their own panoramas.

###  4.1 Statistics

Method | Zeroth-q | First-q | Second-q | Third-q | Fourth-q
---|---|---|---|---|---
Ori. | 1.832e-05 | 0.00038 | 0.00172 | 0.00689 | 5.837e+11
Heu. | 7.254e-15 | 0.00019 | 0.00098 | 0.00375 | 6.293e+09
Opt. | 9.450e-31 | 0.00015 | 0.00071 | 0.00255 | 1.546e+06
Table 1: The zeroth- (minimum), first-, second- (median), third-, and fourth-
quartiles (maximum) of the distortion values of the original camera model, the
heuristic solutions, and the optimized solutions of the computational dolly-
zoom effects.

We tested on a laptop computer with 6-core 2.6GHZ CPU, 16GB ram, NVidia GTX
1650 Ti graphics card, and Windows system. We use Google Ceres-Solver to solve
the computational dolly-zoom effect optimization problem. We measure the times
to solve the problem in all possible camera poses (the same sampling as in
Figure 6). The average and largest times are 0.53 and 7 milliseconds. This
means that the method is suitable for real-time applications on a reasonable
computer. The computational costs of the heuristic solutions are negligible.
The quartiles of the distortion values of the original cameras, heuristic
solutions, and optimized solutions are shown in Table 1. In summary, both
heuristic and optimized solutions improve upon the original camera model in
terms of distortion values, while the optimized solutions win over the
heuristic ones by a sizeable margin.

###  4.2 Limitations

Our method can be summarized as using a cylinder as the proxy mesh to define
the per-pixel depths, which eliminates distortions of vertical features in 3D,
and then using the computational dolly-zoom effect to find optimal alternative
camera positions and FOV angles to render the same view-able regions but with
minimized distortions. Our method does not create true 3D effects such as
parallax and occlusion on-and-offs. Nevertheless, the illusion of a scene in
3D remains to a degree when the camera movement is small w.r.t. the depth
disparity of the scene. Our hypothesis is that human brains can still deduct
3D depths by the image features (semantics, shading, straight lines, etc).
Realism of the rendering begins to fray when the relative positions of the
camera to the other objects significantly changed. One example is the last row
in Figure 7 (the camera moved to the other side of the table-chairs set).

Figure 8: Off-center perspective projection results. From second left to right
columns: 1) by vanilla E2P projection, 2) our method, panoramas augmented with
3) depths and 4) room layouts. The first two rows are based on real-world
indoor panoramas shoot by ourselves with predicted depths ("MiDaS v3.0"
[RBK21]) and predicted room layouts (LED2-Net [WYS*21]). The third row is
based on a panorama from the Structure3D dataset (synthetic) with ground truth
depths and room layout. The last three rows are based on Gibson dataset (real-
world) with predicted depths and room layouts. We show the panoramas annotated
with room layouts and visualization of the off-center cameras in the left
column. Bigger versions of the images can be found in the Additional
Materials.

##  5 Conclusion and future work

In this paper, we explore how to tackle barrel distortions in E2P projections
when the camera moves away from the center. The experiments show that our two
proposed approaches, namely the cylindrical projection and the computational
dolly-zoom effect, significantly reduced distortions in different indoor scene
datasets. The resulting renderings are smooth and intuitive, albeit lacking
true 3D effects such as parallax and occlusion on-and-offs. Comparing to the
state-of-the-art DL-based method [XZX*21], one key advantage of our method is
that the resolution of perspective views is much higher. We compare our method
with other methods that aim to augment panoramas with 3D information (i.e.,
depth information and room layouts), and show that these methods are actually
more likely to produce results with noticeable artifacts.

In summary, our approach relies on novel strategies to re-sample the existing
pixels in a panorama to synthesize off-center perspective views with minimized
barrel distortions. For future work, we would like to expand our methods to
leverage readily-available 3D information of a panorama, such as depths and
room layouts predicted by neural networks. For example, creating more complex
proxy meshes according to depths or layouts instead of a cylinder. More
ambitious goals include creating datasets and deep-learning methods that
directly synthesize novel full-resolution perspective views out of a single or
a sparse set of panoramas.

## References

  * [ALG*20] Benjamin Attal et al.  “Matryodshka: Real-time 6dof video view synthesis using multi-sphere images”  In _European Conference on Computer Vision_ , 2020, pp. 441–459  Springer
  * [ASK*20] Kara-Ali Aliev et al.  “Neural point-based graphics”  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_ , 2020, pp. 696–712  Springer
  * [ASZS17] Iro Armeni, Sasha Sax, Amir R Zamir and Silvio Savarese  “Joint 2d-3d-semantic data for indoor scene understanding”  In _arXiv preprint arXiv:1702.01105_ , 2017
  * [AZD*21] Georgios Albanis et al.  “Pano3D: A Holistic Benchmark and a Solid Baseline for 360deg Depth Estimation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3727–3737
  * [BFO*20] Michael Broxton et al.  “Immersive light field video with a layered mesh representation”  In _ACM Transactions on Graphics (TOG)_ 39.4 ACM New York, NY, USA, 2020, pp. 86–1
  * [BYLR20] Tobias Bertel, Mingze Yuan, Reuben Lindroos and Christian Richardt  “OmniPhotos: casual 360° VR photography”  In _ACM Transactions on Graphics (TOG)_ 39.6 ACM New York, NY, USA, 2020, pp. 1–12
  * [CCYC21] Kuo-Wei Chen, Felix Chang, Chih-Yuan Yao and Hung-Kuo Chu  “Sio-Keong Si”, 2021
  * [CDF*17] Angel Chang et al.  “Matterport3d: Learning from rgb-d data in indoor environments”  In _arXiv preprint arXiv:1709.06158_ , 2017
  * [GSKH21] Chen Gao, Ayush Saraf, Johannes Kopf and Jia-Bin Huang  “Dynamic View Synthesis from Dynamic Monocular Video”  In _arXiv preprint arXiv:2105.06468_ , 2021
  * [HK18] Peter Hedman and Johannes Kopf  “Instant 3d photography”  In _ACM Transactions on Graphics (TOG)_ 37.4 ACM New York, NY, USA, 2018, pp. 1–12
  * [JXZ*20] Lei Jin et al.  “Geometric structure based and regularized depth estimation from 360 indoor imagery”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 889–898
  * [KMA*20] Johannes Kopf et al.  “One Shot 3D Photography”  In _Transactions on Graphics (Proceedings of SIGGRAPH)_ 39.4, 2020
  * [LRSK19] Katrin Lasinger, René Ranftl, Konrad Schindler and Vladlen Koltun  “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer”  In _arXiv preprint arXiv:1907.01341_ , 2019
  * [LXM*20] Kai-En Lin et al.  “Deep multi depth panoramas for view synthesis”  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16_ , 2020, pp. 328–344  Springer
  * [NGR19] David Novotny, Ben Graham and Jeremy Reizenstein  “PerspectiveNet: A scene-consistent image generator for new view synthesis in real indoor environments”  In _Advances in Neural Information Processing Systems_ 32, 2019, pp. 7601–7612
  * [PAA*] Giovanni Pintore et al.  “SliceNet: deep dense depth estimation from a single indoor panorama using a slice-based representation: Supplementary Material”
  * [RBK21] René Ranftl, Alexey Bochkovskiy and Vladlen Koltun  “Vision Transformers for Dense Prediction”  In _ArXiv preprint_ , 2021
  * [SGHS98] Jonathan Shade, Steven Gortler, Li-wei He and Richard Szeliski  “Layered depth images”  In _Proceedings of the 25th annual conference on Computer graphics and interactive techniques_ , 1998, pp. 231–242
  * [SHSC19] Cheng Sun, Chi-Wei Hsiao, Min Sun and Hwann-Tzong Chen  “HorizonNet: Learning Room Layout With 1D Representation and Pano Stretch Data Augmentation”  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , 2019, pp. 1047–1056
  * [SKC*19] Ana Serrano et al.  “Motion parallax for 360 RGBD video”  In _IEEE Transactions on Visualization and Computer Graphics_ 25.5 IEEE, 2019, pp. 1817–1827
  * [SSC21] Cheng Sun, Min Sun and Hwann-Tzong Chen  “Hohonet: 360 indoor holistic understanding with latent horizontal features”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2573–2582
  * [STH*19] Vincent Sitzmann et al.  “DeepVoxels: Learning Persistent 3D Feature Embeddings”  In _Proc. Computer Vision and Pattern Recognition (CVPR), IEEE_ , 2019
  * [SZW19] Vincent Sitzmann, Michael Zollhöfer and Gordon Wetzstein  “Scene representation networks: Continuous 3d-structure-aware neural scene representations”  In _arXiv preprint arXiv:1906.01618_ , 2019
  * [WYS*20] Fu-En Wang et al.  “Bifuse: Monocular 360 depth estimation via bi-projection fusion”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 462–471
  * [WYS*21] Fu-En Wang et al.  “LED2-Net: Monocular 360deg Layout Estimation via Differentiable Depth Rendering”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12956–12965
  * [XZH*18] Fei Xia et al.  “Gibson env: Real-world perception for embodied agents”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9068–9079
  * [XZX*21] Jiale Xu et al.  “Layout-Guided Novel View Synthesis from a Single Indoor Panorama”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 16438–16447
  * [YLC*20] Sheng Yang et al.  “Noise-resilient reconstruction of panoramas and 3D scenes using robot-mounted unsynchronized commodity RGB-D cameras”  In _ACM Transactions on Graphics (TOG)_ 39.5 ACM New York, NY, USA, 2020, pp. 1–15
  * [YWP*19] Shang-Ta Yang et al.  “Dula-net: A dual-projection network for estimating room layouts from a single rgb panorama”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3363–3372
  * [ZKG20] Wei Zeng, Sezer Karaoglu and Theo Gevers  “Joint 3d layout and depth prediction from a single indoor panorama image”  In _European Conference on Computer Vision_ , 2020, pp. 666–682  Springer
  * [ZKZ*19] Nikolaos Zioulis et al.  “Spherical view synthesis for self-supervised 360 depth estimation”  In _2019 International Conference on 3D Vision (3DV)_ , 2019, pp. 690–699  IEEE
  * [ZKZD18] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas and Petros Daras  “Omnidepth: Dense depth estimation for indoors spherical panoramas”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 448–465
  * [ZTF*18] Tinghui Zhou et al.  “Stereo magnification: Learning view synthesis using multiplane images”  In _arXiv preprint arXiv:1805.09817_ , 2018
  * [ZZL*20] Jia Zheng et al.  “Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling”  In _Proceedings of The European Conference on Computer Vision (ECCV)_ , 2020