# Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction

Feng Wang∗1 Zilong Chen∗1 Guokang Wang1 Yafei Song2 Huaping Liu†1
1Beijing National Research Center for Information Science and
Technology(BNRist),
Department of Computer Science and Technology, Tsinghua University
2Alibaba Group
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose the Masked Space-Time Hash encoding (MSTH), a novel method for
efficiently reconstructing dynamic 3D scenes from multi-view or monocular
videos. Based on the observation that dynamic scenes often contain substantial
static areas that result in redundancy in storage and computations, MSTH
represents a dynamic scene as a weighted combination of a 3D hash encoding and
a 4D hash encoding. The weights for the two components are represented by a
learnable mask which is guided by an uncertainty-based objective to reflect
the spatial and temporal importance of each 3D position. With this design, our
method can reduce the hash collision rate by avoiding redundant queries and
modifications on static areas, making it feasible to represent a large number
of space-time voxels by hash tables with small size. Besides, without the
requirements to fit the large numbers of temporally redundant features
independently, our method is easier to optimize and converge rapidly with only
twenty minutes of training for a 300-frame dynamic scene. As a result, MSTH
obtains consistently better results than previous methods with only 20 minutes
of training time and 130 MB of memory storage. Code is available at
https://masked-spacetime-hashing.github.io/.

##  1 Introduction

0.31236912Training time (hour)30313233PSNR on Plenoptic
VideoOursOurs†MixVoxels Wang et al. (2022a)K-Planes Sara Fridovich-Keil and
Giacomo Meanti et al. (2023)NeRFPlayer Song et al. (2022)HexPlane Cao and
Johnson (2023)HyperReel Attal et al. (2023)HexPlane† Cao and Johnson
(2023)27282930PSNR on Google ImmersiveOursNeRFPlayer Song et al.
(2022)HyperReel Attal et al. (2023)Plenoptic DatasetImmersive Dataset

Figure 1: Performance comparison with state-of-the-art methods. We compare the
PSNR and training time on two public benchmarks. Our method surpasses other
methods by a non-trivial margin with only 20m of training. ${\dagger}$ denotes
the HexPlane Cao and Johnson (2023) setting which removes the coffee-martini
scene.

Neural radiance fields Mildenhall et al. (2021b); Yu et al. (2021a); Sun et
al. (2022); Müller et al. (2022); Chen et al. (2022a) have achieved great
success in reconstructing static 3D scenes. The learned volumetric
representations can produce photo-realistic rendering for novel views.
However, for dynamic scenes, the advancements lag behind that of static scenes
due to the inherent complexities caused by the additional time dimension.
††footnotetext: ∗ Equal contribution. ††footnotetext: † Corresponding Author.
Specifically, reconstructing dynamic scenes requires more time, memory
footprint, and additional temporal consistency compared to static scenes.
These factors have resulted in unsatisfactory rendering qualities. Recently,
many works Li et al. (2022c, a); Song et al. (2022); Wang et al. (2022a);
Attal et al. (2023); Cao and Johnson (2023); Sara Fridovich-Keil and Giacomo
Meanti et al. (2023) have made great progress in dynamic scene reconstruction,
improving the efficiency and effectiveness of reconstructions. However, there
is still considerable room for improvement in many aspects such as rendering
quality, motion coherence, training and rendering speed, and memory usage.

DyNeRFLi et al. (2022c) 1344h

HexPlaneCao and Johnson (2023) 12h

Ours 20min

NeRFPlayerSong et al. (2022) 6h

HyperReelAttal et al. (2023) 2.7h

Ours 20min

DyNeRFLi et al. (2022c) 1344h

HyperReelAttal et al. (2023) 2.7h

Ours 20min

Figure 2: Qualitative comparisons to state-of-the-art methods Li et al.
(2022c); Attal et al. (2023); Song et al. (2022); Cao and Johnson (2023). We
visualize three scenes: flame salmon, horse, and welder from Plenoptic Video
dataset (Li et al., 2022c) and Google Immersive dataset (Broxton et al.,
2020). Some key patches are zoomed in for better inspection. Our method
performs better in reconstructing details, such as the stripes of the salmon,
the facial features, and the splashing sparks.

This paper aims to develop a method for reconstructing dynamic scenes with
high rendering quality in a space- and time-efficient manner. To achieve this
goal, we base our approach on the multi-resolution hash encoding Müller et al.
(2022) due to its efficiency and compactness for representing and
reconstructing static scenes. However, directly applying a 4D multi-resolution
hash table to represent a dynamic scene would require a much larger hash table
size than that for a static scene due to the much more hash collisions caused
by the additional time dimension. Our finding is that, for the ubiquitous
existence of static areas in a scene, storing the static points into a 4D
time-dependent hash table will result in an information redundancy since each
of them will occupy $\mathcal{T}$ hash table items with an identical value for
a $\mathcal{T}$-frame scene. It will also lead to a high hash collision rate
since it narrows the capacity of the hash table, which negatively impacts the
reconstruction quality. Therefore, for the points with low dynamics, we hope
to establish a mechanism to reduce the frequent queries and updates to the 4D
hash table and automatically save their features into a 3D hash table to avoid
temporal redundancy.

From this observation, we propose Masked Space-Time Hash encoding, which
combines a 3D hash encoder and a 4D hash encoder with a 3D learnable weight
mask. In order to make the mask correlate with the dynamics of the
corresponding positions, we adopt the Bayesian framework of Kendall and Gal
(2017) to estimate the uncertainty Kendall and Gal (2017); Martin-Brualla et
al. (2021b) of each point being static. The non-linear correlation Belghazi et
al. (2018) between the uncertainty and the learnable mask is maximized to make
the mask reflect the dynamics. In this way, the static points indicated with a
low uncertainty will have a low weight for the 4D hash table and a high weight
for the 3D hash table, which prevents modifications to the dynamic
representations. With the proposed masked space-time hash encoding, we can set
the size of a 4D hash table the same as a 3D one without much loss of
rendering qualities, making the representation highly compact. Besides,
without the requirements to fit the large numbers of repeated features
independently, our method is easier to optimize and converge rapidly in only
twenty minutes. To validate the effectiveness of our method on scenes in more
realistic settings with large areas of dynamic regions and more complex
movements, we collect a synchronized multi-view video dataset with 6
challenging dynamic scenes, which will be publicly available. As a result, the
proposed masked space-time hash encoding achieves consistently better
reconstruction metrics on two publicly available datasets consisting of 13
scenes, with only 20 minutes of training and 130 MB of storage. Fig. 1 and
Fig. 2 show the quantitative and qualitative comparisons to other state-of-
the-art methods. In summary, our contributions are:

  * •

We propose Masked Space-time Hash Encoding, a novel representation that
decomposes the dynamic radiance fields into a weighted combination of 3D and
4D hash encoders.

  * •

The proposed method is validated on a wide range of dynamic scenes, surpassing
previous state-of-the-art methods by non-trivial margins with only 20 minutes
of training time and 130 MB of memory storage.

  * •

We propose a new synchronized multi-view dataset with more challenging dynamic
scenes, including scenes with many moving objects and scenes with complex and
rapid movements.

##  2 Related Work

Neural Scene Representations for Static Scenes. Representing 3D scenes with
neural networks has achieved remarkable success in recent years. NeRF
(Mildenhall et al., 2021b) first proposes to use neural networks to represent
radiance fields, demonstrating the remarkable potential of this representation
in conjunction with volume rendering techniques. The high-fidelity rendering
has sparked a surge of interest in related areas. Numerous variants have
emerged to address the inherent challenges of the original approach, including
improving rendering quality (Barron et al., 2021, 2023; Kerbl et al., 2023; Ma
et al., 2021; Wang et al., 2022e), handling sparse input (Niemeyer et al.,
2022; Chen et al., 2021a; Xu et al., 2022; Meng et al., 2021; Yen-Chen et al.,
2020; Chibane et al., 2021), lighting (Boss et al., 2020; Srinivasan et al.,
2020; Pan et al., 2021), editing (Kobayashi et al., 2022; Liu et al., 2021,
2022a; Fan et al., 2022; Mildenhall et al., 2021a), dealing with complicated
(Wang et al., 2023; Guo et al., 2022; Yin et al., 2023; Klenk et al., 2023)
and large scenes (Tancik et al., 2022; Zhang et al., 2020; Martin-Brualla et
al., 2021b; Chen et al., 2021b), generalization (Yu et al., 2021c; Schwarz et
al., 2020; Tancik et al., 2020; Chan et al., 2020; Wang et al., 2021a;
Niemeyer and Geiger, 2021; Jang and Agapito, 2021; Wang et al., 2022d),
jointly optimizing with camera poses (Lin et al., 2021; Wang et al., 2021b;
Bian et al., 2023; Jeong et al., 2021; Sucar et al., 2021; Zhu et al., 2021;
Chen et al., 2022b), accelerating training (Yu et al., 2021a; Chen et al.,
2022a; Sun et al., 2022; Yariv et al., 2023; Deng et al., 2021) and speeding
up rendering (Reiser et al., 2023; Hedman et al., 2021; Yu et al., 2021b;
Reiser et al., 2021; Liu et al., 2020; Rebain et al., 2021; Lindell et al.,
2021; Garbin et al., 2021; Hu et al., 2022; Cao et al., 2022; Wang et al.,
2022b; Lombardi et al., 2021). Our work draws inspiration from two recent
contributions in the field, namely Instant-NGP (Müller et al., 2022) and Mip-
NeRF 360 (Barron et al., 2022). In Instant-NGP, Müller et al. (2022) proposes
a novel data structure that leverages multi-resolution hash grids to
efficiently encode the underlying voxel grid while addressing hash collisions
using a decoding MLP. Furthermore, the proposed method allows for fast and
memory-efficient rendering of large-scale scenes. Similarly, Mip-NeRF 360
(Barron et al., 2022) presents a sample-efficient scheme for unbound scenes,
which utilizes a small density field as a sample generator and parameterizes
the unbounded scene with spherical contraction. In this paper, we build upon
these contributions and propose a novel approach that extends successful
techniques and components to incorporate time dimension with minimal overhead.
Our method is capable of representing a dynamic scene with only 2~3$\times$
memory footprint than that of a static NeRF.

Novel View Synthesis for Dynamic Scenes. Extending the neural radiance field
to express dynamic scenes is a natural yet challenging task that has been
proven crucial to many downstream applications (Li et al., 2022b; Peng et al.,
2021; Su et al., 2021; Jiakai et al., 2021). Many researchers focus on
monocular dynamic novel view synthesis, which takes a monocular video as input
and targets to reconstruct the underlying 4D information by modeling
deformation explicitly (Du et al., 2021; Gao et al., 2021; Li et al., 2021a;
Xian et al., 2021; Pumarola et al., 2021; Yuan et al., 2021) or implicitly
(Park et al., 2021b; Gao et al., 2022; Park et al., 2021a; Fang et al., 2022;
Shao et al., 2023; Liu et al., 2023, 2022b; Zhao et al., 2022; Tretschk et
al., 2021; Jiang et al., 2022). As an exemplary work, D-NeRF (Pumarola et al.,
2021) models a time-varying field through a deformation network that maps a 4D
coordinate into a spatial point in canonical space. Despite the great outcomes
achieved by research in this line, the applicability of these methods is
restricted by the inherent nature of the underlying problem (Gao et al.,
2022). A more practical way of reconstructing dynamic scenes is by employing
multi-view synchronized videos (Kanade et al., 1997; Lombardi et al., 2019;
Zitnick et al., 2004; Li et al., 2022a; Attal et al., 2023; Song et al., 2022;
Cao and Johnson, 2023; Wang et al., 2022a, c; Bansal et al., 2020; Peng et
al., 2023). DyNeRF (Li et al., 2022c) models dynamic scenes by exploiting a 6D
plenoptic MLP with time queries and a set of difference-based importance
sampling strategies. The authors also contributed to the field by presenting a
real-world dataset, which validated their proposed methodology and provided a
valuable resource for future research endeavors. K-Planes (Sara Fridovich-Keil
and Giacomo Meanti et al., 2023) and HexPlane (Cao and Johnson, 2023) speed up
training by decomposing the underlying 4D radiance field into several low-
dimensional planes, which substantially reduces the required memory footprint
and computational complexity compared with explicit 4D voxel grid. HyperReel
(Attal et al., 2023) breaks down the videos into keyframe-based segments and
predicts spatial offset towards the nearest keyframe. NeRFPlayer (Song et al.,
2022) and MixVoxel (Wang et al., 2022a) address the problem by decomposing 4D
space according to corresponding temporal properties. The former separates the
original space into three kinds of fields and applies different structures and
training strategies, while the latter decouples static and dynamic voxels with
a variational field which further facilitates high-quality reconstruction and
fast rendering. Our method implicitly decomposes 3D space with a learnable
mask. This mask eliminates the requirement for manual determination of dynamic
pixels and enables the acquisition of uncertainty information, facilitating
high-quality reconstruction.

##  3 Methodology

Our objective is to reconstruct dynamic scenes from a collection of multi-view
or monocular videos with a compact representation while achieving fast
training and rendering speeds. To this end, we propose Masked Space-Time Hash
encoding, which represents a dynamic radiance field by a weighted combination
of a 3D and a 4D hash encoder. We employ an uncertainty-guided mask learning
strategy to enable the mask to learn the dynamics. In the following, we will
first introduce the preliminary, then the masked space-time hash encoding and
uncertainty-guided mask learning, and finally the ray sampling method.

###  3.1 Preliminary

For neural radiance fields, the input is a 3D space coordinate $\bm{x}$ and a
direction $\bm{d}$ to enable the radiance field to represent the non-
Lambertian effects. The output is the color
$c(\bm{x},\bm{d})\in\mathbb{R}^{3}$ and a direction-irrelevant density
$\sigma(\bm{x})\in\mathbb{R}$. Most existing methods encode the input
coordinate with a mapping function $h$ that maps the raw coordinate into
Fourier features Mildenhall et al. (2021b), voxels-based features Sun et al.
(2022); Yu et al. (2021a), hashing-based features Müller et al. (2022) or
factorized low-rank features Chen et al. (2022a). In this work, we mainly
focus on the multi-resolution hash encoding due to its efficiency and
compactness. Specifically, for an input point $\bm{x}$, the corresponding
voxel index is computed through the scale of each level, and a hash table
index is computed by a bit-wise XOR operation (Nießner et al., 2013). There
are $L$ levels of hash tables corresponding to different grid sizes in a
geometric progressive manner. For each level, the feature for a continuous
point is tri-linearly interpolated by the nearest grid points. The
corresponding features in different levels are concatenated to obtain the
final encoding. For convenience, we denote the output of the multi-resolution
hash encoder for $\bm{x}$ as $\bm{\mathrm{enc}}(\bm{x})$.

After the multi-resolution hash encoder, a density-decoding MLP
$\phi_{\theta_{1}}$ is employed to obtain the density and an intermediate
feature $\bm{g}$ such that:
$\sigma(\bm{x}),\bm{g}(\bm{x})=\phi_{\theta_{1}}(\bm{\mathrm{enc}}(\bm{x}))$.
Then, the color is computed through a color-decoding MLP $\psi_{\theta_{2}}$:
$c(\bm{x})=\psi_{\theta_{2}}(\bm{g}(\bm{x}),\bm{d})$ which is direction-
dependent.

For rendering, points along a specific ray $\bm{r}(s)=\bm{o}+s\bm{d}$ are
sampled and the volumetric rendering Mildenhall et al. (2021b) is applied to
get the rendered color $\hat{C}(\bm{r})$:

$\small\hat{C}(\bm{r})=\int_{n}^{f}T(s)\cdot\sigma(\bm{r}(s))\cdot
c(\bm{r}(s),\bm{d})\mathrm{d}s,~{}\textrm{where}~{}T(s)=\mathrm{exp}\left(-\int_{s_{n}}^{s}\sigma(\bm{r}(s))ds\right).$
(1)

A squared error between the rendered color $\hat{C}(\bm{r})$ and the ground
truth color $C(\bm{r})$ is applied for back-propagation.

###  3.2 Masked Space-Time Hashing

For a dynamic neural radiance field, the input is a 4D space-time coordinate
$(\bm{x},t)$. A straightforward method is to replace the 3D hash table with a
4D one. However, this simple replacement will result in a high hash collision
rate due to the enormous volume of hash queries and modifications brought by
the additional time dimension. For instance, in a dynamic scene comprising
$\mathcal{T}$ frames, the hash collision rate is $\mathcal{T}$ times higher
than a static scene, leading to a degradation in the reconstruction
performance. Enlarging the size of the hash table will cause an unbearable
model size and be difficult to scale up with the frame numbers.

To solve this problem, we propose the masked space-time hash encoding, which
incorporates a 3D multi-resolution hash mapping $\mathrm{h}_{\mathtt{3D}}$, a
4D multi-resolution hash mapping $\mathrm{h}_{\mathtt{4D}}$, and a learnable
mask encoding $\tilde{m}$. The final encoding function is formulated as
follows:

$\bm{\mathrm{enc}}(\bm{x},t)=m(\bm{x})\cdot\mathrm{h}_{\mathtt{3D}}(\bm{x})+(1-m(\bm{x}))\cdot\mathrm{h}_{\mathtt{4D}}(\bm{x},t),~{}~{}\textrm{where}~{}~{}m(\bm{x})=\textrm{sigmoid}(\tilde{m}(\bm{x})).$
(2)

The learnable mask $m$ can be represented by a multi-resolution hash table or
a 3D voxel grid. For the 4D hash table, the sizes of time steps also adopt a
geometric growing multi-resolution scheme, which is reasonable due to the
natural hierarchical properties of motions in time scales.

After obtaining the space-time encoding, the density-decoding and color-
decoding MLPs are applied to obtain the final outputs:

$\sigma(\bm{x},t),~{}\bm{g}(\bm{x},t)=\phi_{\theta_{1}}(\mathrm{enc}(\bm{x},t)),~{}~{}~{}c(\bm{x},\bm{d},t)=\psi_{\theta_{2}}(\bm{g}(\bm{x},t),\bm{d}).$
(3)

Then the volumetric rendering is applied to each ray at each time step, and a
squared error is employed as the loss function:

$\hat{C}(\bm{r},t)=\int_{n}^{f}T(s,t)\cdot\sigma(\bm{r}(s),t)\cdot
c(\bm{r}(s),\bm{d},t)\mathrm{d}s,~{}~{}~{}~{}L_{r}=\mathop{\mathbb{E}}\limits_{\bm{r},t}\left[\|C(\bm{r},t)-\hat{C}(\bm{r},t)\|_{2}^{2}\right].$
(4)

Reduce Hash Collisions. The intuition behind the masked modeling lies in the
fact that many parts in the dynamic radiance fields are time-invariant, such
as the static objects, the background, etc. These static parts can be well
reconstructed from the 3D hash table with a large $m$. For these static
points, storing their features in the 4D hash table will occupy a large number
of storage, largely increasing the hash collision rate. For these time-
invariant parts, the 3D hash encoder can be sufficient to reconstruct, and
these static parts will not modify the 4D hash table significantly when
$(1-m(\bm{x}))$ is small. In this way, the 4D hash table stores the properties
of those points which are really dynamic, and those dynamic features in the 4D
hash table will be protected by the mask term to suppress the gradients from
static points.

Accelerate Rendering. Another advantage of using masked space-time hash
encoding is to accelerate the fixed view-port rendering. Instead of rendering
a novel view video frame-by-frame, we adopt an incremental way of using the
mask to avoid redundant computations. Specifically, for a $\mathcal{T}$-frame
scene, we first render the initial frame $V_{0}$ as usual. The obtained mask
is used to filter the static parts, and we only render the dynamic parts of
other frames with the dynamic weight $(1-m(x))>(1-\epsilon)$, where $\epsilon$
is a hyper-parameter. In this way, except for the initial frame, we only
require to render the dynamic part, improving the rendering fps from $1.4$ to
$15$, without loss of rendering quality.

The key to the masked space-time hash encoding is that $m(\bm{x})$ can reflect
the dynamics of the point. To this end, we design an uncertainty-guided mask
learning strategy to connect the relation between the 3D hash uncertainty
$u(\bm{x})$ and the mask $m(\bm{x})$. We will elaborate on this strategy in
the next part.

Figure 3: We compared the learned masks $m$ by visualizing them using
volumetric rendering. As shown in the figures above, we observed that the mask
learned with uncertainty is cleaner and tends to have a binarized value, which
helps avoid the mixture of both tables.

###  3.3 Uncertainty-guided Mask Learning

To make the mask $m(\bm{x})$ well reflect the dynamics of the corresponding
point, we design an uncertainty branch in our model to estimate the
uncertainty of a point being static, which is a good indicator for the
dynamics of the point. To this end, the uncertainty branch is required to
reconstruct the dynamic scenes only using the 3D hash table with the input-
dependent uncertainties, ignoring the time input. In this way, the dynamic
points are regarded as the inherent noise since its supervision are inherently
inconsistent (different time step describes different geometries while the
uncertainty model is time-agnostic). The inherent noise of dynamic points will
lead to a high uncertainty. We adopt the Bayesian learning framework of
Kendall and Gal (2017) to model the heteroscedastic aleatoric uncertainty for
each point.

Specifically, we construct an uncertainty field $u$ with a voxel grid
representation. $u(\bm{x})$ denotes the uncertainty of a space point $\bm{x}$.
We denote the raw output of the uncertainty voxel-grid as $\tilde{u}$, and a
soft-plus is used as the activation:
$u(\bm{x})=u_{m}+\mathrm{log}\left(1+\mathrm{exp}(\tilde{u}(\bm{x}))\right)$,
where $u_{m}$ is a hyper-parameter for shifting the uncertainty values Martin-
Brualla et al. (2021b). For each ray, the ray-level uncertainty
$\mathrm{U}(\bm{r})$ is calculated through volumetric rendering Martin-Brualla
et al. (2021a):

$\small\mathrm{U}(\bm{r})=\int_{n}^{f}T(s)\cdot\sigma(\bm{r}(s))\cdot
u(\bm{r}(s))\mathrm{d}s.$ (5)

Besides, the color and density estimated by the 3D hash table are:

$\sigma_{s}(\bm{x}),~{}\bm{g}_{s}(\bm{x})=\phi_{\theta_{1}}(\mathrm{h}_{\mathtt{3D}}(\bm{x})),~{}~{}~{}c_{s}(\bm{x},\bm{d})=\psi_{\theta_{2}}(\bm{g}_{s}(\bm{x}),\bm{d}).$
(6)

The rendered color of this branch is $\hat{C}_{s}(\bm{r})$ by applying the
volumetric rendering. After that, the uncertainty-based loss for ray $r$ is
defined as:

$\small
L_{u}=\mathop{\mathbb{E}}\limits_{\bm{r},t}\left[\frac{1}{2\mathrm{U}(\bm{r})^{2}}\|C(\bm{r},t)-\hat{C}_{s}(\bm{r})\|_{2}^{2}+\mathrm{log}\mathrm{U}(\bm{r})\right].$
(7)

Note that the uncertainty branch is time-agnostic, i.e., the predicted color
$\hat{C}(\bm{r})$ is not time-dependent, which is important to make the
uncertainty relevant to dynamics. In this way, the uncertainty for each point
$u(\bm{x})$ can be estimated through the above loss function. For the dynamic
points, the uncertainty is inherently large because of the ambiguous and
inconsistent geometries caused by the inconsistent pixel color supervision. In
this perspective, the uncertainty can well reflect the dynamics of each point
and could guide the mask values.

Bridging uncertainty with mask. Though we find the correlation between the
mask $m$ and uncertainty $u$, it is not trivial to connect them. First, $m$
and $u$ are in different value ranges, $m(\bm{x})\in[0,1]$ while
$u(\bm{x})\in[0,+\infty)$. Second, the distributions of $m$ and $u$ are very
different, and the relations between them are non-linear. Imposing a hard
relationship between them will impact the training of their specific branches.
We instead maximize the mutual information between the two random variables
$m$ and $u$ to maximize the nonlinear correlation between them. Although MI
can not measure whether the correlation is negative, Eq. 7 can guarantee this.
The mutual information $I(m,u)$ describes the decrease of uncertainty in $m$
given $u$: $I(m,u):=H(m)-H(m|u)$, where $H$ is the Shannon entropy. To
estimate the mutual information between $m$ and $u$, we adopt the neural
estimator in Belghazi et al. (2018) to approximate the mutual information
$I(m,u)$ as:

$I_{\Theta}(m,u)=\mathop{\mathrm{sup}}\limits_{\theta\in\Theta}\mathbb{E}_{\mathbb{P}_{m,u}}\left[T_{\theta}\right]-\mathrm{log}(\mathbb{E}_{\mathbb{P}_{m}\otimes\mathbb{P}_{u}}\left[e^{T_{\theta}}\right]).$
(8)

$\mathbb{P}_{m,u}$ is the joint probability distribution of $m$ and $u$, and
$\mathbb{P}_{m}\otimes\mathbb{P}_{u}$ is the product of marginals.
$T_{\theta}$ is a neural network with parameters $\theta\in\Theta$. We choose
a small MLP with two hidden layers to represent $T_{\theta}$ and draw samples
from the joint distribution and the product marginals to compute the empirical
estimation of the expectation. By maximizing the estimated mutual information,
we build the correlation between $m$ and $u$. At last, the overall learning
objective of MSTH is the combination of the above losses:

$L=L_{r}+\lambda\cdot L_{u}-\gamma\cdot I_{\Theta}(m,u),$ (9)

where $\lambda$ and $\gamma$ are two hyper-parameters. For rendering a novel
view, the uncertainty branches and the mutual information network are
disabled.

In practice, the model can learn a reasonable mask $m$ even without the
uncertainty guidance. However, this will make the learned mask very noisy and
tend to learn a "middle value" in $[0,1]$, which will make the static points
still have a relatively large dynamic weight to modify the 4D table. Fig. 3
visualize the learned mask with or without uncertainty guidance. With
uncertainty guidance, the mask will tend to be binarized and the static parts
are with very low dynamic weight. For the mutual information constraint, we
find it will make the distribution of $m$ towards a Bernoulli distribution
which is helpful for reducing hash collision and accelerating rendering speed.
Without the constraint, the model tends to learn more from the 3D hash table
only even for the dynamic point. This will make the failing captures of some
dynamic voxels with transient changes. As a result, the uncertainty guidance
will help learn a finer detail, which will be shown in the ablation part.

###  3.4 Ray Sampling

For a natural video, a significant portion of the scene is usually static or
exhibits only minor changes in radiance at a specific time across the entire
video. Uniformly sampling the space-time rays causes an imbalance between the
static and dynamic observations. Therefore, we sample the space-time queries
according to the quantification of dynamics. Specifically, we sample a space-
time ray $(\bm{r},t)$ according to a pre-computed probability $P(\bm{r},t)$.
We decompose the sampling process into two sub-process: spatial sampling and
temporal sampling. Formally, we decompose $P(\bm{r},t)$ as
$P(\bm{r},t)=P(t|\bm{r})P(\bm{r})$, which forms a simple Markov process that
first sample the space ray according to $P(\bm{r})$ then sample the time step
by $P(t|\bm{r})$.

In practice, we build $P(\bm{r})$ as the softmax of temporal standard
deviation $\mathtt{std}(\bm{r})$ of the corresponding pixel intensities
$\\{C(r,t)\\}_{t}$:

$\small
P(\bm{r})\propto\mathrm{exp}\left(\mathtt{std}(\bm{r})/\tau_{1}\right),$ (10)

where $\tau_{1}$ is the temperature parameter. We set a higher temperature for
a softer probability distribution Hinton et al. (2015). For temporal sampling,
we follow Li et al. (2021b) to compute the weight according to the residual
difference of its color to the global median value across time
$\bar{C}(\bm{r})$ with temperature $\tau_{2}$:

$\small
P(t|\bm{r})\propto\mathrm{exp}\left(|C(\bm{r},t)-\bar{C}(\bm{r})|/\tau_{2}\right).$
(11)

###  3.5 Implementation Details

To improve the sampling efficiency, we utilize the proposal sampler (Barron et
al., 2022) that models density at a coarse granularity with a much smaller
field, thereby generating more accurate samples that align with the actual
density distribution with minor overhead. In our implementation, we use one
layer of proposal net to sample 128 points. For the mask, we use a non-hash
voxel grid with 128 spatial resolution. To encourage the separation of the
static and the dynamic, we utilize a mask loss that aims at generating sparse
dynamic voxels by constraining the mask to be close at 1. We also adopt
distortion loss (Barron et al., 2022) with $\lambda_{dist}=2e-2$. For
uncertainty loss, we set $\gamma=3e-4$ and $\lambda=3e-5$. We find a small
coefficient of mutual information estimator $\gamma$ will have a large impact
on the gradients, which is consistent with the observations of Belghazi et al.
(2018). For the hash grid, we implement a CUDA extension based on PyTorch
(Paszke et al., 2019) to support the rectangular voxel grid required by the
proposed method. For the complete experiment setting and model hyper-
parameters, please refer to the Appendix.

Figure 4: Qualitative results on different datasets of our method. We
visualize four novel views in each column, three for RGB and one for depth.
For other scenes, we visualize them in the Appendix, and the full spiral
videos are presented in our supplemental material.

##  4 Experiments

###  4.1 Dataset

For validating the performances of the proposed method, we conduct experiments
on two public datasets and our collected dataset: (1) The Plenoptic Video
Dataset Li et al. (2022c), which consists of 6 publicly accessible scenes:
coffee-martini, flame-salmon, cook-spinach, cut-roasted-beef, flame-steak, and
sear-steak. Each scene contains 19 videos with different camera views. The
dataset contains some challenging scenes, including objects with topology
changes, objects with volumetric effects, various lighting conditions, etc.
(2) Google Immersive Dataset Broxton et al. (2020): The Google Immersive
dataset contains light field videos of different indoor and outdoor
environments captured by a time-synchronized 46-fisheye camera array. We use
the same 7 scenes (Welder, Flames, Truck, Exhibit, Alexa Meade Exihibit, Face
Paint 1, Face Paint 2) as NeRFPlayer (Song et al., 2022) for evaluation. (3)
To validate the robustness of our method on more complex in-the-wild
scenarios, we collect six time-synchronized multi-view videos including more
realistic observations such as pedestrians, moving cars, and grasses with
people playing. We named the collected dataset as Campus Dataset. The Campus
dataset is much more difficult than the above two curated ones in the movement
complexities and dynamic areas. For detail on the dataset, please see our
Appendix. For the above three multi-view datasets, we follow the experiment
setting in Li et al. (2022c) that employs 18 views for training and 1 view for
evaluation. To quantitatively evaluate the rendering quality on novel views,
we measure PSNR, DSSIM, and LPIPS Zhang et al. (2018) on the test views. We
follow the setting of Li et al. (2022c) to evaluate our model frame by frame.
Our method is also applicable to monocular videos. To validate the
reconstruction quality with monocular input, we conduct experiments on D-NeRF
Pumarola et al. (2021) dataset, which contains eight videos of varying
duration, ranging from 50 frames to 200 frames per video. There is only one
single training image for each time step. For evaluation, we follow the common
setting from Pumarola et al. (2021); Sara Fridovich-Keil and Giacomo Meanti et
al. (2023); Cao and Johnson (2023).

Table 1: Quantitative results on Plenoptic Video dataset Li et al. (2022c). We
report the average metrics and compare them with other state-of-the-art
methods. Our method achieves non-trivial performance improvements on all
metrics. * denotes the DyNeRF setting which only reports results on the flame-
salmon scene. ${\dagger}$ denotes the HexPlane setting that removes the
coffee-martini scene to calculate average metrics. We report the per-scene
metrics in the Appendix.

Model | PSNR$\uparrow$ | D-SSIM$\downarrow$ | LPIPS$\downarrow$ | Training Time$\downarrow$ | Rendering FPS $\uparrow$ | Storage $\downarrow$
---|---|---|---|---|---|---
DyNeRF* Li et al. (2022c) | 29.58 | 0.020 | 0.099 | 1344 h | < 0.01 | 28MB
Ours* | 29.92 | 0.020 | 0.063 | 20 min | 15 | 135MB
NeRFPlayer Song et al. (2022) | 30.69 | 0.034 | 0.111 | 6 h | 0.05 | -
HyperReel Attal et al. (2023) | 31.10 | 0.036 | 0.096 | 9 h | 2.0 | 360MB
MixVoxels Wang et al. (2022a) | 30.80 | 0.020 | 0.126 | 80 min | 16.7 | 500MB
K-Planes Sara Fridovich-Keil and Giacomo Meanti et al. (2023) | 31.63 | 0.018 | - | 108 min | - | -
Ours | 32.37 | 0.015 | 0.056 | 20 min | 15 | 135MB
HexPlane Cao and Johnson (2023) † | 31.705 | 0.014 | 0.075 | 12 h | - | 200MB
Ours† | 33.099 | 0.013 | 0.051 | 20 min | 15 | 135MB

###  4.2 Results

Multi-view Dynamic Scenes. We reconstruct the dynamic scenes from multi-view
time-synchronized observations on the two public multi-view video datasets and
our collected Campus dataset. The quantitative results and comparisons are
presented in Tab. 1 and Tab. 3. For the Plenoptic Video dataset, our method
surpasses previous state-of-the-art methods by a non-trivial margin, with a
$0.7$ to $1.4$ PSNR gains, and > $30\%$ improvements on the perceptual metric
LPIPS. For training, we cost at most $1/5$ training time of other methods,
achieving a speedup of $5-36$ compared with other fast reconstruction
algorithms and a $4000$ speedup compared with DyNeRF Li et al. (2022c).
Besides, we also keep a compact model size with only $135$MB of memory
occupancy, showing the advantages of the masking strategy which can avoid a
large number of hash collisions and make the dynamic hash table size smaller
enough. For the Google Immersive dataset, Tab. 3 also shows consistently non-
trivial improvements in both training efficiency and reconstruction quality.

Fig. 2 visualizes the qualitative comparisons of our method to other state-of-
the-art methods, from which we can observe that our method can capture finer
motion details than other methods. For example, MSTH can well reconstruct the
fire gun with distinct boundaries and specular reflection, the mark of the
hat, and the stripe of the salmon. The Splashing sparks can also be accurately
captured. We provide the representative novel-view rendering results of our
method in Fig. 4.

Monocular Dynamic Scenes. We present the quantitative and qualitative results
of our proposed method on the D-NeRF dataset in Tab. 3 and Fig. 5, accompanied
by a comparative analysis with related approaches. MSTH outperforms all other
methods in terms of SSIM and LPIPS, including methods that focus on dynamic
scenes (K-Planes (Sara Fridovich-Keil and Giacomo Meanti et al., 2023) and
HexPlane (Cao and Johnson, 2023)) and TiNeuVox, the current state-of-the-art
method for monocular dynamic novel view synthesis. The proposed MSTH achieves
superior results without any assumption about the underlying field which
demonstrates that our method could be easily extended to accommodate scenarios
with varying complexity and dynamics.

###  4.3 Ablation Study

Figure 5: Qualitative results on D-NeRF dataset (Pumarola et al., 2021). The
visualized images are rendered from test views. Complete per-scene multi-view
images are shown in the Appendix.

Table 2: Quantitive result on Google Immersive dataset (Broxton et al., 2020)
and the proposed Campus dataset. Check out the Appendix for detailed
comparison and visualization.

Model | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Training Time $\downarrow$
---|---|---|---|---
Google Immersive Video Dataset
NeRFPlayer Song et al. (2022) | 26.6 | 0.870 | 0.1931 | 6 hrs
HyperReel Attal et al. (2023) | 28.8 | 0.874 | 0.193 | 2.7 hrs
Ours | 29.6 | 0.950 | 0.0929 | 20 min
Campus Dataset
Ours | 20.9 | 0.722 | 0.241 | 20 min

Table 3: Quantitative comparison with related work on D-NeRF dataset (Pumarola
et al., 2021). Per-scene metrics are shown in the Appendix.

Model | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
T-NeRF (Pumarola et al., 2021) | 29.51 | 0.95 | 0.08
D-NeRF (Pumarola et al., 2021) | 30.5 | 0.95 | 0.07
TiNeuVox-B (Fang et al., 2022) | 32.67 | 0.97 | 0.04
HexPlane (Cao and Johnson, 2023) | 31.04 | 0.97 | 0.04
K-Planes (Sara Fridovich-Keil and Giacomo Meanti et al., 2023) | 30.84 | 0.96 | -
Ours | 31.34 | 0.98 | 0.02

Ablation on the Masked Hash Encoding. To evaluate the effect of decomposing a
4D dynamic radiance field into the proposed masked space-time hash encoding,
we propose two variants as comparisons: (1) A pure 4D hash encoding. (2) A
simple decomposition which is an addition of a 3D hash table and a 4D hash
table. Fig. 6 (a) and (b) visualize the qualitative comparisons. From Fig. 6
(a) we can observe that only using the 4D hash table will generate blurry
rendering with a 3 PSNR drop. With a simple addition method, the
reconstruction quality is improved compared with only a 4D hash table, while
the dynamic regions are not captured well. Fig. 6 (b) and (c) show the
comparisons of MSTH with the addition. MSTH improves the LPIPS by $20\%$
compared with the addition method.

Figure 6: Qualitative comparisons for different ablations. Zoom in for a
better inspection.

Ablation on the uncertainty-based guidance. We visualize the learned mask with
or without uncertainty-based guidance, Fig. 3 shows the comparison. The
learned mask with uncertainty loss will distinguish the static and dynamic
parts more clearly, with less noisy judgment, leading to an assertive
distribution on the mask, which is beneficial to avoid hash collisions. Fig. 6
(d) visualize the qualitative comparisons. MSTH with uncertainty loss performs
better on the motion details.

##  5 Limitations and Conclusion

Limitations. MSTH tends to generate unsatisfying results when detailed dynamic
information is insufficient, especially in monocular dynamic scenes, since it
relies solely on the input images. When the input is blurred, occluded, or
incomplete, the method may struggle to infer the motion information, leading
to artifacts such as ghosting, flickering, or misalignment, which may degrade
the visual quality. Further research is needed to develop advanced methods
that handle complex scenes, motion dynamics and integrate multiple information
sources to enhance synthesis accuracy and robustness.

Conclusion. In this paper, we propose a new method to reconstruct dynamic 3D
scenes in a time- and space-efficient way. We decouple the representations of
the dynamic radiance field into a time-invariant 3D hash encoding and a time-
variant 4D hash encoding. With an uncertainty-guided mask as weight, our
algorithm can avoid a large number of hash collisions brought about by the
additional time dimension. We conduct extensive experiments to validate our
method and achieve state-of-the-art performances with only 20 minutes of
training. Besides, we collect a complex in-the-wild multi-view video dataset
for evaluating the robustness of our approach on more realistic dynamic
scenes, including many daily activities. We hope our method can serve as a
fast and lightweight baseline to reconstruct dynamic 3D scenes, which may
derive many valuable applications such as interactively free-viewpoint control
for movies, cinematic effects, novel view replays for sporting events, and
other VR/AR applications.

##  6 Acknowledgement

This work was supported in part by the National Natural Science Fund for
Distinguished Young Scholars under Grant 62025304. Thank Xueping Shi for
giving helpful discussion.

## References

  * Attal et al. [2023] B. Attal, J.-B. Huang, C. Richardt, M. Zollhoefer, J. Kopf, M. O’Toole, and C. Kim.  Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling.  _arXiv preprint arXiv:2301.02238_ , 2023.
  * Bansal et al. [2020] A. Bansal, M. Vo, Y. Sheikh, D. Ramanan, and S. Narasimhan.  4d visualization of dynamic events from unconstrained multi-view videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5366–5375, 2020.
  * Barron et al. [2021] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5855–5864, 2021.
  * Barron et al. [2022] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5470–5479, 2022.
  * Barron et al. [2023] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  _arXiv_ , 2023.
  * Belghazi et al. [2018] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and R. D. Hjelm.  Mine: mutual information neural estimation.  _arXiv preprint arXiv:1801.04062_ , 2018.
  * Bian et al. [2023] W. Bian, Z. Wang, K. Li, J. Bian, and V. A. Prisacariu.  Nope-nerf: Optimising neural radiance field with no pose prior.  2023\.
  * Boss et al. [2020] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. Lensch.  NeRD: Neural reflectance decomposition from image collections.  _https://arxiv.org/abs/2012.03918_ , 2020.
  * Broxton et al. [2020] M. Broxton, J. Flynn, R. Overbeck, D. Erickson, P. Hedman, M. Duvall, J. Dourgarian, J. Busch, M. Whalen, and P. Debevec.  Immersive light field video with a layered mesh representation.  _ACM Transactions on Graphics (TOG)_ , 39(4):86–1, 2020.
  * Cao and Johnson [2023] A. Cao and J. Johnson.  Hexplane: a fast representation for dynamic scenes.  _arXiv preprint arXiv:2301.09632_ , 2023.
  * Cao et al. [2022] J. Cao, H. Wang, P. Chemerys, V. Shakhrai, J. Hu, Y. Fu, D. Makoviichuk, S. Tulyakov, and J. Ren.  Real-time neural light field on mobile devices.  _arXiv preprint arXiv:2212.08057_ , 2022.
  * Chan et al. [2020] E. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein.  pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis.  _https://arxiv.org/abs/2012.00926_ , 2020.
  * Chen et al. [2021a] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14124–14133, 2021a.
  * Chen et al. [2022a] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su.  Tensorf: Tensorial radiance fields.  _arXiv preprint arXiv:2203.09517_ , 2022a.
  * Chen et al. [2021b] X. Chen, Q. Zhang, X. Li, Y. Chen, F. Ying, X. Wang, and J. Wang.  Hallucinated neural radiance fields in the wild, 2021b.
  * Chen et al. [2022b] Y. Chen, X. Chen, X. Wang, Q. Zhang, Y. Guo, Y. Shan, and F. Wang.  Local-to-global registration for bundle-adjusting neural radiance fields.  _arXiv preprint arXiv:2211.11505_ , 2022b.
  * Chibane et al. [2021] J. Chibane, A. Bansal, V. Lazova, and G. Pons-Moll.  Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, jun 2021.
  * Deng et al. [2021] K. Deng, A. Liu, J.-Y. Zhu, , and D. Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  _arXiv preprint arXiv:2107.02791_ , 2021.
  * Du et al. [2021] Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu.  Neural radiance flow for 4d view synthesis and video processing.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Fan et al. [2022] Z. Fan, Y. Jiang, P. Wang, X. Gong, D. Xu, and Z. Wang.  Unified implicit neural stylization.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV_ , pages 636–654. Springer, 2022.
  * Fang et al. [2022] J. Fang, T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian.  Fast dynamic radiance fields with time-aware neural voxels.  In _SIGGRAPH Asia 2022 Conference Papers_ , 2022.
  * Gao et al. [2021] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang.  Dynamic view synthesis from dynamic monocular video.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5712–5721, 2021.
  * Gao et al. [2022] H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa.  Dynamic novel-view synthesis: A reality check.  In _NeurIPS_ , 2022.
  * Garbin et al. [2021] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14346–14355, 2021.
  * Guo et al. [2022] Y.-C. Guo, D. Kang, L. Bao, Y. He, and S.-H. Zhang.  Nerfren: Neural radiance fields with reflections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18409–18418, June 2022.
  * Hedman et al. [2021] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec.  Baking neural radiance fields for real-time view synthesis.  _ICCV_ , 2021.
  * Hinton et al. [2015] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hu et al. [2022] T. Hu, S. Liu, Y. Chen, T. Shen, and J. Jia.  Efficientnerf efficient neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12902–12911, June 2022.
  * Jang and Agapito [2021] W. Jang and L. Agapito.  Codenerf: Disentangled neural radiance fields for object categories.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 12949–12958, October 2021.
  * Jeong et al. [2021] Y. Jeong, S. Ahn, A. A. Christopehr Choy, M. Cho, and J. Park.  Self-calibrating neural radiance fields.  In _ICCV_ , 2021.
  * Jiakai et al. [2021] Z. Jiakai, L. Xinhang, Y. Xinyi, Z. Fuqiang, Z. Yanshun, W. Minye, Z. Yingliang, X. Lan, and Y. Jingyi.  Editable free-viewpoint video using a layered neural representation.  In _ACM SIGGRAPH_ , 2021.
  * Jiang et al. [2022] Y. Jiang, P. Hedman, B. Mildenhall, D. Xu, J. T. Barron, Z. Wang, and T. Xue.  Alignerf: High-fidelity neural radiance fields via alignment-aware training.  _arXiv preprint arXiv:2211.09682_ , 2022.
  * Kanade et al. [1997] T. Kanade, P. Rander, and P. Narayanan.  Virtualized reality: Constructing virtual worlds from real scenes.  _IEEE multimedia_ , 4(1):34–47, 1997.
  * Kendall and Gal [2017] A. Kendall and Y. Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in neural information processing systems_ , 30, 2017.
  * Kerbl et al. [2023] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics_ , 42(4), July 2023.  URL https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/.
  * Klenk et al. [2023] S. Klenk, L. Koestler, D. Scaramuzza, and D. Cremers.  E-nerf: Neural radiance fields from a moving event camera.  _IEEE Robotics and Automation Letters_ , 2023.
  * Kobayashi et al. [2022] S. Kobayashi, E. Matsumoto, and V. Sitzmann.  Decomposing nerf for editing via feature field distillation.  In _Advances in Neural Information Processing Systems_ , volume 35, 2022.  URL https://arxiv.org/pdf/2205.15585.pdf.
  * Li et al. [2022a] L. Li, Z. Shen, Z. Wang, L. Shen, and P. Tan.  Streaming radiance fields for 3d video synthesis.  _arXiv preprint arXiv:2210.14831_ , 2022a.
  * Li et al. [2022b] R. Li, J. Tanke, M. Vo, M. Zollhofer, J. Gall, A. Kanazawa, and C. Lassner.  Tava: Template-free animatable volumetric actors.  2022b.
  * Li et al. [2022c] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, et al.  Neural 3d video synthesis from multi-view video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5521–5531, 2022c.
  * Li et al. [2021a] Z. Li, S. Niklaus, N. Snavely, and O. Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021a.
  * Li et al. [2021b] Z. Li, S. Niklaus, N. Snavely, and O. Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6498–6508, 2021b.
  * Lin et al. [2021] C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey.  Barf: Bundle-adjusting neural radiance fields.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2021\.
  * Lindell et al. [2021] D. B. Lindell, J. N. Martel, and G. Wetzstein.  Autoint: Automatic integration for fast neural volume rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14556–14565, 2021.
  * Liu et al. [2022a] H.-K. Liu, I.-C. Shen, and B.-Y. Chen.  Nerf-in: Free-form nerf inpainting with rgb-d priors.  _arxiv preprint arXiv:2206.04901_ , 2022a.
  * Liu et al. [2022b] J.-W. Liu, Y.-P. Cao, W. Mao, W. Zhang, D. J. Zhang, J. Keppo, Y. Shan, X. Qie, and M. Z. Shou.  Devrf: Fast deformable voxel radiance fields for dynamic scenes.  _arXiv preprint arXiv:2205.15723_ , 2022b.
  * Liu et al. [2020] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt.  Neural sparse voxel fields.  _Advances in Neural Information Processing Systems_ , 33:15651–15663, 2020.
  * Liu et al. [2021] S. Liu, X. Zhang, Z. Zhang, R. Zhang, J.-Y. Zhu, and B. Russell.  Editing conditional radiance fields, 2021.
  * Liu et al. [2023] Y.-L. Liu, C. Gao, A. Meuleman, H.-Y. Tseng, A. Saraf, C. Kim, Y.-Y. Chuang, J. Kopf, and J.-B. Huang.  Robust dynamic radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Lombardi et al. [2019] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  _arXiv preprint arXiv:1906.07751_ , 2019.
  * Lombardi et al. [2021] S. Lombardi, T. Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and J. Saragih.  Mixture of volumetric primitives for efficient neural rendering, 2021\.
  * Ma et al. [2021] L. Ma, X. Li, J. Liao, Q. Zhang, X. Wang, J. Wang, and P. V. Sander.  Deblur-nerf: Neural radiance fields from blurry images.  _arXiv preprint arXiv:2111.14292_ , 2021.
  * Martin-Brualla et al. [2021a] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7210–7219, 2021a.
  * Martin-Brualla et al. [2021b] R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In _CVPR_ , 2021b.
  * Meng et al. [2021] Q. Meng, A. Chen, H. Luo, M. Wu, H. Su, L. Xu, X. He, and J. Yu.  Gnerf: Gan-based neural radiance field without posed camera.  _arXiv preprint arXiv:2103.15606_ , 2021.
  * Mildenhall et al. [2021a] B. Mildenhall, P. Hedman, R. Martin-Brualla, P. P. Srinivasan, and J. T. Barron.  NeRF in the dark: High dynamic range view synthesis from noisy raw images.  _arXiv_ , 2021a.
  * Mildenhall et al. [2021b] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021b.
  * Müller et al. [2022] T. Müller, A. Evans, C. Schied, and A. Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _arXiv preprint arXiv:2201.05989_ , 2022.
  * Niemeyer and Geiger [2021] M. Niemeyer and A. Geiger.  Campari: Camera-aware decomposed generative neural radiance fields, 2021\.
  * Niemeyer et al. [2022] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. M. Sajjadi, A. Geiger, and N. Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Nießner et al. [2013] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger.  Real-time 3d reconstruction at scale using voxel hashing.  _ACM Trans. Graph._ , 32(6):169:1–169:11, 2013\.  doi: 10.1145/2508363.2508374.  URL https://doi.org/10.1145/2508363.2508374.
  * Pan et al. [2021] X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai.  A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Park et al. [2021a] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5865–5874, 2021a.
  * Park et al. [2021b] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  _arXiv preprint arXiv:2106.13228_ , 2021b.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8024–8035, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.
  * Peng et al. [2021] S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In _CVPR_ , 2021.
  * Peng et al. [2023] S. Peng, Y. Yan, Q. Shuai, H. Bao, and X. Zhou.  Representing volumetric videos as dynamic mlp maps.  In _CVPR_ , 2023.
  * Pumarola et al. [2021] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10318–10327, 2021.
  * Rebain et al. [2021] D. Rebain, W. Jiang, S. Yazdani, K. Li, K. M. Yi, and A. Tagliasacchi.  Derf: Decomposed radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14153–14161, 2021.
  * Reiser et al. [2021] C. Reiser, S. Peng, Y. Liao, and A. Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14335–14345, 2021.
  * Reiser et al. [2023] C. Reiser, R. Szeliski, D. Verbin, P. P. Srinivasan, B. Mildenhall, A. Geiger, J. T. Barron, and P. Hedman.  Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  _arXiv preprint arXiv:2302.12249_ , 2023.
  * Sara Fridovich-Keil and Giacomo Meanti et al. [2023] Sara Fridovich-Keil and Giacomo Meanti, F. R. Warburg, B. Recht, and A. Kanazawa.  K-planes: Explicit radiance fields in space, time, and appearance.  In _CVPR_ , 2023.
  * Schonberger and Frahm [2016] J. L. Schonberger and J.-M. Frahm.  Structure-from-motion revisited.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4104–4113, 2016.
  * Schwarz et al. [2020] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger.  Graf: Generative radiance fields for 3D-aware image synthesis.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 33, 2020.
  * Shao et al. [2023] R. Shao, Z. Zheng, H. Tu, B. Liu, H. Zhang, and Y. Liu.  Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Song et al. [2022] L. Song, A. Chen, Z. Li, Z. Chen, L. Chen, J. Yuan, Y. Xu, and A. Geiger.  Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields.  _arXiv preprint arXiv:2210.15947_ , 2022.
  * Srinivasan et al. [2020] P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron.  NeRV: Neural reflectance and visibility fields for relighting and view synthesis.  _https://arxiv.org/abs/2012.03927_ , 2020.
  * Su et al. [2021] S.-Y. Su, F. Yu, M. Zollhöfer, and H. Rhodin.  A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Sucar et al. [2021] E. Sucar, S. Liu, J. Ortiz, and A. Davison.  iMAP: Implicit mapping and positioning in real-time.  In _arXiv preprint arXiv:2103.12352_ , 2021.
  * Sun et al. [2022] C. Sun, M. Sun, and H.-T. Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5459–5469, 2022.
  * Tancik et al. [2020] M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. Srinivasan, J. T. Barron, and R. Ng.  Learned initializations for optimizing coordinate-based neural representations.  _https://arxiv.org/abs/2012.02189_ , 2020.
  * Tancik et al. [2022] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar.  Block-nerf: Scalable large scene neural view synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8248–8258, 2022.
  * Tretschk et al. [2021] E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12959–12970, 2021.
  * Wang et al. [2022a] F. Wang, S. Tan, X. Li, Z. Tian, and H. Liu.  Mixed neural voxels for fast multi-view video synthesis.  _arXiv preprint arXiv:2212.00190_ , 2022a.
  * Wang et al. [2022b] H. Wang, J. Ren, Z. Huang, K. Olszewski, M. Chai, Y. Fu, and S. Tulyakov.  R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis.  In _ECCV_ , 2022b.
  * Wang et al. [2022c] L. Wang, J. Zhang, X. Liu, F. Zhao, Y. Zhang, Y. Zhang, M. Wu, J. Yu, and L. Xu.  Fourier plenoctrees for dynamic radiance field rendering in real-time.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13524–13534, 2022c.
  * Wang et al. [2023] P. Wang, Y. Liu, Z. Chen, L. Liu, Z. Liu, T. Komura, C. Theobalt, and W. Wang.  F2-nerf: Fast neural radiance field training with free camera trajectories.  _CVPR_ , 2023.
  * Wang et al. [2021a] Q. Wang, Z. Wang, K. Genova, P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  _arXiv preprint arXiv:2102.13090_ , 2021a.
  * Wang et al. [2022d] T. Wang, B. Zhang, T. Zhang, S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  _arXiv preprint arXiv:2212.06135_ , 2022d.
  * Wang et al. [2021b] Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu.  NeRF$--$: Neural radiance fields without known camera parameters.  _arXiv preprint arXiv:2102.07064_ , 2021b.
  * Wang et al. [2022e] Z. Wang, L. Li, Z. Shen, L. Shen, and L. Bo.  4k-nerf: High fidelity neural radiance fields at ultra high resolutions.  _arXiv preprint arXiv:2212.04701_ , 2022e.
  * Xian et al. [2021] W. Xian, J.-B. Huang, J. Kopf, and C. Kim.  Space-time neural irradiance fields for free-viewpoint video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9421–9431, 2021.
  * Xu et al. [2022] D. Xu, Y. Jiang, P. Wang, Z. Fan, H. Shi, and Z. Wang.  Sinnerf: Training neural radiance fields on complex scenes from a single image.  2022\.
  * Yariv et al. [2023] L. Yariv, P. Hedman, C. Reiser, D. Verbin, P. P. Srinivasan, R. Szeliski, J. T. Barron, and B. Mildenhall.  Bakedsdf: Meshing neural sdfs for real-time view synthesis.  _arXiv preprint arXiv:2302.14859_ , 2023.
  * Yen-Chen et al. [2020] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin.  iNeRF: Inverting neural radiance fields for pose estimation.  _https://arxiv.org/abs/2012.05877_ , 2020.
  * Yin et al. [2023] Z.-X. Yin, J. Qiu, M.-M. Cheng, and B. Ren.  Multi-space neural radiance fields.  2023\.
  * Yu et al. [2021a] A. Yu, S. Fridovich-Keil, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa.  Plenoxels: Radiance fields without neural networks.  _arXiv preprint arXiv:2112.05131_ , 2021a.
  * Yu et al. [2021b] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5752–5761, 2021b.
  * Yu et al. [2021c] A. Yu, V. Ye, M. Tancik, and A. Kanazawa.  pixelNeRF: Neural radiance fields from one or few images.  In _CVPR_ , 2021c.
  * Yuan et al. [2021] W. Yuan, Z. Lv, T. Schmidt, and S. Lovegrove.  Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13144–13152, 2021.
  * Zhang et al. [2020] K. Zhang, G. Riegler, N. Snavely, and V. Koltun.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_ , 2020.
  * Zhang et al. [2018] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhao et al. [2022] F. Zhao, W. Yang, J. Zhang, P. Lin, Y. Zhang, J. Yu, and L. Xu.  Humannerf: Efficiently generated human radiance field from sparse inputs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7743–7753, June 2022.
  * Zhu et al. [2021] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys.  Nice-slam: Neural implicit scalable encoding for slam, 2021.
  * Zitnick et al. [2004] C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski.  High-quality video view interpolation using a layered representation.  _ACM transactions on graphics (TOG)_ , 23(3):600–608, 2004.


B_{\bar{r}}(\bar{x}).$

Recall that a function $\psi:\mathbb{R}^{N-2}\to\mathbb{R}$ is said to be $C$
semiconvex if the function $\psi(x^{\prime\prime})+C|x^{\prime\prime}|^{2}/2$
is convex. In particular the sub-differential of
$\partial\psi(x^{\prime\prime})$ for all $x^{\prime\prime}$ is non-empty,
where

$\partial\psi(x^{\prime\prime})=\Bigl{\\{}p\in\mathbb{R}^{N-2}:\psi(y^{\prime\prime})\geq\psi(x^{\prime\prime})+p\cdot(y^{\prime\prime}-x^{\prime\prime})-\frac{C|x^{\prime\prime}-y^{\prime\prime}|^{2}}{2}\text{
for all $y^{\prime\prime}\in\mathbb{R}^{N-2}$}\Bigr{\\}}.$

Note in particular that if $\psi(0)=0$ and
$p\in\partial\psi(x^{\prime\prime})$ then

(4.2) $0\geq\psi(x^{\prime\prime})-p\cdot
x^{\prime\prime}-\frac{C|x^{\prime\prime}|^{2}}{2}.$

By taking $O$ as above we define for
$x=(0,x^{\prime\prime},x_{N})\in\partial_{\partial H}O$ the normal and the
tangent cones at $x$ in $\partial H$ as

$N_{x}O=\\{\lambda(0,p,-1):\,p\in\partial\psi(x^{\prime\prime}),\lambda\in[0,+\infty)\\}$

and

$T_{x}O=\\{v\in\mathbb{R}^{N}:\,v\cdot e_{1}=0,\,v\cdot\nu\leq 0\quad\text{for
all $\nu\in N_{x}O$}\\}.$

It is well known that for $x=(0,x^{\prime\prime},x_{N})\in\partial_{\partial
H}O$, the sets

$O_{x,r}=\frac{O-x}{r}\to T_{x}O$

as $r\to 0$ where the convergence is the Kuratowski sense. We start with the
following technical lemma.

######  Lemma 4.2.

Let $O\subset\partial H$ be such that (4.1) is satisfied. Then there exist a
smooth vector field $X:\mathbb{R}^{N}\to\mathbb{R}^{N}$ such that

  1. (i)

$X(x)\cdot e_{1}=0$ for all $x\in\partial H$

  2. (ii)

$X(x)\cdot\nu\geq 0$ for all $x\in\partial_{H}O\cap B_{\bar{r}}(\bar{x})$ and
all $\nu\in N_{x}O$

  3. (iii)

For all $x\in B_{\bar{r}}(\bar{x})$ it holds:

(4.3) $\displaystyle X(x)$ $\displaystyle=x+O(|x|^{2})\,,$ (4.4)
$\displaystyle\nabla X(x)$ $\displaystyle=\mathrm{Id}+O(|x|)\,.$

###### Proof.

We may assume that $\bar{x}=0$ and we define

$X(x)=\Bigl{(}x_{1},x^{\prime\prime},x_{N}-\frac{C|x^{\prime\prime}|^{2}}{2}\Bigr{)}$

where $C$ is the semiconvexity constant of $\psi$. Clearly (i) and (iii) are
satisfied. To check (ii), note that if $\nu\in N_{x}O$ then

$\nu=\lambda(0,p,-1)$

for $p\in\partial\psi(x^{\prime})$ and $\lambda\geq 0$ so that

$X(x)\cdot\nu=\lambda\Bigl{(}p\cdot
x^{\prime}-\psi(x^{\prime})+\frac{C|x^{\prime}|^{2}}{2}\Bigr{)}\geq 0$

by (4.2). ∎

We can now state the desired monotonicity formula for $(\Lambda,r_{0})$
minimizers.

######  Theorem 4.3.

Let $E\subset H$ be a $(\Lambda,r_{0})$-minimizer of
$\mathcal{F}_{\sigma}^{O}$ with obstacle $O$. Assume that
$\bar{x}\in\partial_{\partial H}O$ and that (4.1) is satisfied with
$0<\bar{r}\leq r_{0}$ and a $C$-semiconvex function $\psi$ with $\psi(0)=0$.
Then there exists a constant $c_{0}=c_{0}(C,\Lambda,N)$ such that for all
$0<s<r$, with $r<\bar{r}$ sufficiently small,

(4.5)
$e^{c_{0}r}\frac{\mathcal{F}_{\sigma}^{O}(E,B_{r}(\bar{x}))}{r^{N-1}}-e^{c_{0}s}\frac{\mathcal{F}_{\sigma}^{0}(E,B_{s}(\bar{x}))}{s^{N-1}}\geq\frac{1}{2}\int_{s}^{r}t^{1-N}e^{c_{0}t}\frac{d}{dt}\bigg{[}\int_{\partial
E\cap H\cap B_{t}(\bar{x})}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}\bigg{]}\,dt\,.$

###### Proof.

In this proof by $O(r)$ we mean any function bounded by $cr$ for $r$ small,
where the constant $c$ depends only on $\Lambda$, $N$ and the semiconvexity
constant $C$.

We assume that $\bar{x}=0$ and we let $\phi\in C_{c}^{\infty}([0,1);[0,1))$ be
a smooth decreasing function with $\phi(0)=1$ and we consider for
$r\leq\bar{r}/2$ the vector field $T:\mathbb{R}^{N}\to\mathbb{R}^{N}$ defined
as

$T(x)=-\phi\Biggl{(}\frac{|x|}{r}\Biggr{)}X(x)\,,$

where $X$ is as in Lemma 4.2. By (i) and (ii) of the lemma, if we let
$\varphi_{t}$ be the flow generated by $T$, then for all $t>0$

$\varphi_{t}(\partial H)=\partial H,\qquad\varphi_{t}(O)\subset
O\qquad\text{and}\qquad\varphi_{t}(x)=x\quad\text{for $x\not\in B_{r}$\,.}$

In particular $\varphi_{t}(E)\Delta E\Subset B_{\bar{r}}$ and thus the set
$\varphi_{t}(E)$ is a competitor for the $(\Lambda,r_{0})$-minimality of $E$.
Hence

(4.6)
$\begin{split}\mathcal{F}_{\sigma}^{O}(E;B_{\bar{r}})&\leq\mathcal{F}_{\sigma}^{O}(\varphi_{t}(E);B_{\bar{r}})+\Lambda|\varphi_{t}(E)\Delta
E|\\\ &\leq P(\varphi_{t}(\partial E\setminus O))+\sigma
P(\varphi_{t}(\partial E\cap O))+\Lambda|\varphi_{t}(E)\Delta E|\end{split}$

for all $t>0$. By using the coarea formula and recalling that $|X(x)|=O(r)$ on
$\mathrm{spt}\,T$, it is easy to check that for $t>0$ small enough

$|\varphi_{t}(E)\Delta E|=t\int_{\partial E\cap
H}|T(x)\cdot\nu_{E}(x)|\,d\mathcal{H}^{N-1}+o(t)\leq tO(r)\int_{\partial E\cap
H}\phi\biggl{(}\frac{|x|}{r}\biggr{)}\,d\mathcal{H}^{N-1}\,.$

Therefore, differentiating the inequality in (4.6) we get

(4.7) $O(r)\int_{\partial E\cap
H}\phi\biggl{(}\frac{|x|}{r}\biggr{)}\,\mathcal{H}^{N-1}\leq\int_{\partial
E\setminus O}\operatorname{div}_{\tau}Td\mathcal{H}^{N-1}+\sigma\int_{\partial
E\cap O}\operatorname{div}_{\tau}Td\mathcal{H}^{N-1}\,,$

where

$\operatorname{div}_{\tau}T=\operatorname{div}T-\nabla T[\nu_{E}]\cdot\nu_{E}$

is the tangential divergence of $T$. Since

$\nabla T=-\phi\biggl{(}\frac{|x|}{r}\biggr{)}\nabla
X(x)-\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\frac{x}{|x|}\otimes\frac{X(x)}{|x|}$

by exploiting (4.3), (4.4) we have

$\begin{split}\operatorname{div}_{\tau}T=&-\phi\biggl{(}\frac{|x|}{r}\biggr{)}(N-1)-\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\\\
&+\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\biggl{(}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}\biggr{)}+O(r)\phi\biggl{(}\frac{|x|}{r}\biggr{)}+O(r)\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}.\end{split}$

which can be written as

$\begin{split}\operatorname{div}_{\tau}T&=(1+O(r))r^{N}\frac{d}{dr}\Biggl{(}r^{1-N}\phi\biggl{(}\frac{|x|}{r}\biggr{)}\Biggr{)}\\\
&+\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\biggl{(}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}\biggr{)}+O(r)\phi\biggl{(}\frac{|x|}{r}\biggr{)}\end{split}$

Combining the above inequality with (4.7) we infer, after easy computations,

$\begin{split}(1+O(r))&\frac{d}{dr}\Biggl{(}r^{1-N}\int_{\partial E\setminus
O}\phi\biggl{(}\frac{|x|}{r}\biggr{)}\mathcal{H}^{N-1}+\sigma
r^{1-N}\int_{\partial E\cap
O}\phi\biggl{(}\frac{|x|}{r}\biggr{)}d\mathcal{H}^{N-1}\Biggr{)}\\\
\geq&-c\Biggl{(}r^{1-N}\int_{\partial E\setminus
O}\phi\biggl{(}\frac{|x|}{r}\biggr{)}\mathcal{H}^{N-1}+\sigma
r^{1-N}\int_{\partial E\cap
O}\phi\biggl{(}\frac{|x|}{r}\biggr{)}d\mathcal{H}^{N-1}\Biggr{)}\\\
&-r^{-N}\int_{\partial E\setminus
O}\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\biggl{(}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}\biggr{)}-\sigma
r^{-N}\int_{\partial E\cap
O}\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\biggl{(}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}\biggr{)}.\end{split}$

In turn, noticing that

$\int_{\partial E\cap
H}\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}=0\,,$

setting

$\displaystyle h(r)$ $\displaystyle=r^{1-N}\int_{\partial E\setminus
O}\phi\biggl{(}\frac{|x|}{r}\biggr{)}\,d\mathcal{H}^{N-1}+\sigma
r^{1-N}\int_{\partial E\cap
O}\phi\biggl{(}\frac{|x|}{r}\biggr{)}d\mathcal{H}^{N-1},$ $\displaystyle k(r)$
$\displaystyle=-r^{-N}\int_{\partial E\setminus
O}\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}-\sigma
r^{-N}\int_{\partial E\cap
O}\phi^{\prime}\biggl{(}\frac{|x|}{r}\biggr{)}\frac{|x|}{r}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}$
$\displaystyle=r^{1-N}\frac{d}{dr}\biggl{[}\int_{\partial E\cap
H}\phi\biggl{(}\frac{|x|}{r}\biggr{)}\frac{(x\cdot\nu_{E})^{2}}{|x|^{2}}\,d\mathcal{H}^{N-1}\bigg{]}\,,$

the inequality above implies that

$h^{\prime}(r)\geq-c_{0}h(r)+\frac{1}{2}k(r)$

provided that $2\geq 1+O(r)\geq 1/2$, where $c_{0}$ is a costant depending
only on $\Lambda,N$ and $C$. Note also that $k(r)\geq 0$. Multiplying both
sides of this inequality by $e^{c_{0}r}$ and integrating in $(s,r)$, we have

$\displaystyle
h(r)e^{c_{0}r}-h(s)e^{c_{0}s}\geq\frac{1}{2}\int_{s}^{r}e^{c_{0}t}k(t)\,dt\,.$

Then we conclude by letting $\phi\to 1$. ∎

By classical argument, the above monotonicity formula allows for the study of
blow-ups of $\Lambda$-minimizers. To this aim we preliminary observe that the
following compactness property for blow-ups in the case of convex polyhedral
domains holds.

######  Theorem 4.4.

Let $E\subset H$ be a $(\Lambda,r_{0})$-minimizer of $\mathcal{F}_{\sigma}$
with obstacle $O$, where $O$ is a convex polyhedron. Let
$\bar{x}\in\partial_{\partial H}O\cap\partial E$ and set

$E_{h}=\frac{E-\bar{x}}{r_{h}}\,,$

where $r_{h}\to 0^{+}$. Then there exist a (not relabelled) subsequence and a
set $E_{\infty}$ of locally finite perimeter, such that $E_{h}\to E_{\infty}$
in $L^{1}_{loc}(\mathbb{R}^{N})$ with the property that $E_{\infty}$ is a
$0$-minimizer of $\mathcal{F}^{\tilde{O}}_{\sigma}$, where
$\tilde{O}=T_{\bar{x}}O$. Moreover,

$\mu_{E_{h}}\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\mu_{E}\,,\quad|\mu_{E_{h}}|\stackrel{{\scriptstyle*}}{{\rightharpoonup}}|\mu_{E}|\,,$

as Radon measures. In addition, the following Kuratowski convergence type
properties hold:

$\begin{split}&\text{(i)\,\,for every $x\in\partial E_{\infty}$ there exists
$x_{h}\in\partial E_{h}$ such that $x_{h}\to x$;}\\\ &\text{(ii)\,\,if
$x_{h}\in\overline{\partial E_{h}\cap H}$ and $x_{h}\to x$, then
$x\in\overline{\partial E_{\infty}\cap H}$\,.}\end{split}$

Finally, either $\partial E_{\infty}\cap(\partial
H\setminus{\tilde{O}})=\partial H\setminus{\tilde{O}}$ or $\partial
E_{\infty}\cap\partial H\subset\overline{\tilde{O}}$.

The proof of this theorem follows as in the proof of Theorem 2.4 observing
that the density estimates proved in Proposition 2.3 still hold when $O$ is a
convex polyhedron.

######  Proposition 4.5.

Let $E\subset H$ be a $(\Lambda,r_{0})$-minimizer of $\mathcal{F}_{\sigma}$
with obstacle $O$, where $O$ is either of class $C^{1,1}$ or a convex
polyhedron. If $\bar{x}\in\partial_{\partial H}O\cap\partial E$ then the sets

$E_{\bar{x},r}=\frac{E-\bar{x}}{r}$

are pre-compact in $L^{1}$ and every limit point $E_{\infty}$ as $r\to 0$ is a
conical minimizer of $\mathcal{F}_{\sigma}^{\tilde{O}}$ with obstacle
$\tilde{O}=T_{\bar{x}}O$. Moreover if $N=3$ either $\partial
E_{\infty}=\partial H$ or, after a rotation of coordinates in $\partial H$,

$\partial E_{\infty}\cap H=\bigl{\\{}x:x_{N}=\alpha x_{1}\bigr{\\}}\cap H$

with

$\alpha\geq\frac{\sigma}{\sqrt{1-\sigma^{2}}}$

and $\tilde{O}$ is a half space.

###### Proof.

Let $E_{\infty}$ be a limit point of of $E_{\bar{x},r}$ as $r\to 0$. By
Theorem 2.4 or Theorem 4.4 $E_{\infty}$ is a $0$-minimizer of
$\mathcal{F}_{\sigma}^{\tilde{O}}$. Observe that by Theorem 4.3 there exists

$\lim_{r\to 0}\frac{\mathcal{F}_{\sigma}(E,B_{r}(\bar{x}))}{r^{N-1}}$

and it is finite. From this a standard argument, see for instance the proof of
Theorem 28.6 in [12], shows that

$\frac{\mathcal{F}_{\sigma}^{\tilde{O}}(E_{\infty},B_{r}(\bar{x}))}{r^{N-1}}$

is constant with respect to $r$. Thus, the right-hand side of (4.5) with
$E_{\infty}$ in place of $E$ is zero. This immediately implies that
$x\cdot\nu_{E_{\infty}}(x)=0$ for $\mathcal{H}^{N-1}$-a.e. $x$, hence, see
[12, Prop. 28.8], $E_{\infty}$ is a cone.

Note that if $N=3$ the only cones with zero mean curvature are planes and this
forces $\partial E_{\infty}$ to be a union of planes. These planes can not
intersect in $H$ by the interior regularity theory. In particular since
$0\in\partial E_{\infty}$ (which holds true since $\bar{x}\in\partial E$),
either $\partial E_{\infty}=\partial H$ or, after a rotation in the hyperplane
$H$,

$\partial E_{\infty}\cap
H=\bigcup_{i=1}^{k}\bigl{\\{}x:x_{1}=\alpha_{i}x_{N}\bigr{\\}}\cap H.$

Minimality forces $k=1$ and $\partial E_{\infty}\cap\partial H=\tilde{O}$ and
thus $\tilde{O}$ is a half space. ∎

We are now in a position to prove Theorem 1.4.

###### Proof of Theorem 1.4.

We start by proving part (i). In view of the result [3, Theorem 1.2] it is
enough to prove the regularity in a neighborhood of a point
$\bar{x}\in\partial E\cap\partial_{\partial H}O$. Given such a point, by
Proposition 4.5 we know that there exists a sequence
$E_{h}=E_{\bar{x},r_{h}}=\frac{E-\bar{x}}{r_{h}}$, $r_{h}\to 0$, of blow-ups
of $E$ converging in $L^{1}$ to a $0$-minimizer $E_{\infty}$ of
$\mathcal{F}_{\sigma}^{\tilde{O}}$, where $\tilde{O}=T_{\bar{x}}O$. Moreover,
either $\partial E_{\infty}=\partial H$ or, after a rotation of coordinates in
$\partial H$, $\partial E_{\infty}\cap H=\bigl{\\{}x:x_{1}=\alpha
x_{N}\bigr{\\}}\cap H$ with $\alpha\geq\frac{\sigma}{\sqrt{1-\sigma^{2}}}$.
Note that if $E_{\infty}=\partial H$ or if
$\alpha>\frac{\sigma}{\sqrt{1-\sigma^{2}}}$ by conclusion (ii) of Theorem 2.4
we get that $\overline{\partial E_{h}\cap H}$ satisfies the assumptions of
Lemma 3.15 in a neighborhood of $\bar{x}$ and thus $\overline{\partial E\cap
H}\cap\partial H$ coincides with $\partial_{\partial H}O$ in such a
neighborhood. In turn, the conclusion follows by [5, Theorem 6.1]. If instead
$\alpha=\frac{\sigma}{\sqrt{1-\sigma^{2}}}$, using again (ii) of Theorem 2.4
we have that for $E_{h}$ satisfies the assumptions of Theorem 3.21 for $h$
sufficiently large. Hence the conclusion follows also in this case.

Now the proof of part (ii) follows combining (i) with the classical Federer’s
dimension reduction argument, see for instance [17, Appendix A] or [12,
Sections 28.4-28.5]. We leave the details to the reader. ∎

We conclude the section by proving Theorems 1.5 and 1.6. In the following we
make use of the notation introduced in Subsection 1.2.

###### Proof of Theorem 1.5.

We start by recalling that by a standard argument, see for instance Example
21.3 in [12], there exists $\Lambda,r_{0}>0$ such that

$J_{\sigma,\omega}(E)\leq J_{\sigma,\omega}(F)+\Lambda||E|-|F||$

for all $F\subset\mathbb{R}^{3}\setminus\mathbf{C}$ with diam$(E\Delta
F)<r_{0}$. It easily follows that $E$ is a $(\Lambda,r_{0})$-minimizer of
$J_{\sigma,\omega}$ with obstacle $\omega$ in the sense introduced in the
previous sections. Thus the conclusion follows from Theorem 1.4. ∎

Before proving Theorem 1.6 we set the following definition.

######  Definition 4.6.

We say that $E\subset\mathbb{R}^{3}\setminus\mathbf{C}$ has a nontangential
contact with $\mathbf{C}_{top}$ at a point $\bar{x}\in\overline{\partial
E\cap\mathscr{H}}\cap\gamma$ if for any subsequence $E_{\bar{x},r_{h}}$ of
blow-ups of $E$ with $r_{h}\to 0$, converging to $E_{\infty}$, we have that
$\partial E_{\infty}$ does not contain the plane $\\{x_{3}=0\\}$.

###### Proof of Theorem 1.6.

We start by observing that, arguing as in the proof of Theorem 1.5 we have
that $E$ is $(\Lambda,r_{0})$-minimizer of $J_{\sigma,\omega}$ with ostacle
$\omega$.

Assume first that $E$ has a nontangential contact at all points of
$\overline{\partial E\cap\mathscr{H}}\cap\gamma$. Fix a point
$\bar{x}\in\overline{\partial E\cap\mathscr{H}}\cap\gamma$ and consider a
sequence $E_{\bar{x},r_{h}}$ of blow-ups of $E$ with $r_{h}\to 0$, converging
to some set $E_{\infty}$ of locally finite perimeter. By Proposition 4.5 it
follows that $E_{\infty}$ is a $0$-minimizer of $J_{\sigma,\omega}$ with
obstacle $T_{\bar{x}}\omega$, that after a rotation in the plane
$\\{x_{3}=0\\}$

$\partial E_{\infty}\cap\\{x_{3}>0\\}=\bigl{\\{}x:x_{1}=\alpha
x_{3}\bigr{\\}}\cap\\{x_{3}>0\\}\,,$

with $\alpha\geq\frac{\sigma}{\sqrt{1-\sigma^{2}}}$ and thus $\partial
E_{\infty}\cap\\{x_{3}=0\\}$ is a half plane. In turn $T_{\bar{x}}\omega$ is a
half plane and thus it cannot be a vertex of the polygon. Hence, thanks to
Theorem 1.4, $\overline{\partial E\cap\mathscr{H}}$ is of class $C^{1,\tau}$
for all $\tau\in(0,\frac{1}{2})$ in a neighborhood of $\bar{x}$.

Finally, we claim that if $\sigma<0$ then $E$ has a nontangential contact at
all points of $\overline{\partial E\cap\mathscr{H}}\cap\gamma$. Indeed assume
by contradiction that $E_{\infty}=\\{x_{3}>0\\}$ for some point
$\bar{x}\in\overline{\partial E\cap\mathscr{H}}\cap\gamma$ and some sequence
of blow-ups $E_{\bar{x},r_{h}}$ with $r_{h}\to 0$. Then, arguing similarly as
for Theorem 4.4, see also [3, Theorem 2.9], we get that $E_{\infty}$ is a
$0$-minimizer of $J_{\sigma,T_{\bar{x}}\omega}$, that is

$J_{\sigma,T_{\bar{x}}\omega}(E_{\infty})\leq J_{\sigma,T_{\bar{x}}\omega}(F)$

for all $F\subset\mathbb{R}^{3}\setminus(T_{\bar{x}}\omega\times(-\infty,0])$
with $E_{\infty}\Delta F$ bounded. From this it easily follows that
$F_{\infty}:=(\mathbb{R}^{3}\setminus(T_{\bar{x}}\omega\times(-\infty,0]))\setminus
E_{\infty}$ is a $0$-minimizer of $J_{-\sigma,T_{\bar{x}}\omega}$. By a
rotation in the plane $\\{x_{3}=0\\}$ we may assume without loss of generality
that the half line $\ell=\\{(0,x_{2},0):\,x_{2}>0\\}$ is contained in
$\partial T_{\bar{x}}\omega\times\\{0\\}$ and that $F_{\infty}$ (locally
around each point of $\ell$) is contained in the half space $H=\\{x_{1}>0\\}$.
Hence, locally around any point $y\in\ell$, $F_{\infty}$ is a $0$ minimizer of
$\mathcal{F}_{-\sigma}$ with obstacle $O=\\{x_{3}<0\\}\cap\partial H$. Thus,
by the third inequality in (3.14) (with $\sigma$ replaced by $-\sigma$) it
follows that $\partial F_{\infty}$ must form with the vertical plane $\partial
H$ an angle which is strictly larger then $\pi/2$, thus leading to a
contradiction. ∎

##  5\. Appendix

Here we give the proof of Proposition 3.5. We shall closely follow the proofs
of Lemmas 2.1 and 2.2 and of Theorem 1.1 in [16] with the necessary changes.

We denote by $\mathbb{M}^{d\times d}_{\rm sym}$ the space of symmetric
$d\times d$ matrices and by ${{\rm Tr}\,(A)}$ the trace of $A$. Let
$F:\mathbb{M}^{d\times d}_{\rm sym}\times\mathbb{R}^{d}\to\mathbb{R}$ be the
function

$F(A,p):=\frac{{\rm
Tr}\,(A)}{(1+|p+\xi|^{2})^{1/2}}-\sum_{i,j=1}^{d}\frac{A_{ij}(p_{i}+\xi_{i})(p_{j}+\xi_{j})}{(1+|p+\xi|^{2})^{3/2}}\,,$

where $\xi\in\mathbb{R}^{d}$ is a vector, with $|\xi|\leq M$. Observe that
there exist two positive constants $\tilde{\lambda},\lambda>0$, depending only
on $M$ and $d$ such that if $|p|\leq 1$, $A\in\mathbb{M}^{d\times d}_{\rm
sym}$ and $A\geq 0$, then

$\tilde{\lambda}\|A\|\geq F(A,p)\geq\lambda\|A\|\,,$

where $\|A\|:=\sqrt{\sum_{i,j}A^{2}_{ij}}$. Since throughout this section $M$
will be fixed, in the following with a slight abuse of language we will say
that a constant is universal if it depends only on $d$ and $M$. The next
result is essentially as [16, Lemma 2.1].

######  Lemma 5.1.

Let $r\in(0,1)$ and $a\in(0,1/2)$. Let $v:B_{1}\to(0,\infty)$ be a viscosity
$(2a)$-supersolution of the equation

(5.1) $\operatorname{div}\biggl{(}\frac{\nabla u+\xi}{\sqrt{1+|\nabla
u+\xi|^{2}}}\biggr{)}=a\,,$

bounded from above. There exists a universal constant $c_{0}$ with the
following property: Let $\overline{B}_{r}(x_{0})\subset B_{1}$ and let
$B\subset\overline{B}_{1}$ be a closed set. For every $y\in B$ consider the
paraboloid

(5.2) $-\frac{a}{2}|x-y|^{2}+c_{y}$

staying below $v$ in $\overline{B}_{r}(x_{0})$ and touching $v$ in
$\overline{B}_{r}(x_{0})$. Assume that the set $A$ of all such touching points
for $y\in B$ is contained in $B_{r}(x_{0})$. Then $|A|\geq c_{0}|B|$.

###### Proof.

Observe that since $v$ is a bounded lower semicontinuous function, then $A$ is
a closed set.

Assume that $v$ is semi-concave in $B_{r}(x_{0})$, that is there exists $b>0$
such that $v-b|x|^{2}$ is concave in $B_{r}(x_{0})$. Note that this implies in
particular that $v$ is differentiable at all touching points $z\in A$.
Moreover it is not difficult to show that $Dv$ is Lipschitz in $A$ with a
Lipschitz constant depending only on $a$ and $b$. By assumption if $z\in A$
there exists a paraboloid of vertex $y\in B$ and opening $a$ as in (5.2)
touching $v$ at $z$ from below. Moreover,

(5.3) $y=z+\frac{1}{a}Dv(z)\,.$

Let us denote by $Z$ the set of points in $B_{r}(x_{0})$ such that $v$ is
twice differentiable at $z$. By Alexandrov theorem $|B_{r}(x_{0})\setminus
Z|=0$. Fix $z\in Z$. For all $x\in B_{r}(x_{0})$ we have

$v(x)=P(x;z)+o(|x-z|^{2})=v(z)+Dv(z)\cdot(x-z)+\frac{1}{2}(x-z)^{T}D^{2}v(z)(x-z)+o(|x-z|)^{2}\,.$

We claim that there exists a universal constant $C>0$ such that

$-aI\leq D^{2}v(z)\leq CaI\qquad\text{for all $z\in A\cap Z\,.$}$

The left inequality follows from the fact that the paraboloid in (5.2) touches
$v$ from below. To prove the second inequality assume that there exists a unit
vector $e$ such that

$D^{2}v(z)\geq Ca\,e\otimes e-aI\,.$

We will prove that if $C$ is sufficiently large this leads to a contradiction.

Consider now the test function
$\varphi(x)=P(x;z)-\displaystyle\frac{\varepsilon}{2}|x-z|^{2}$, with
$\varepsilon>0$. Note that $\varphi$ lies below $v$ in a neighborhood of $z$
and touches $v$ at $z$ and that by (5.3) $|D\varphi(z)|=|Dv(z)|=a|y-z|\leq
2a$. Therefore $\varphi$ is an admissible test function and we have

$\displaystyle a$ $\displaystyle\geq
F(D^{2}\varphi(z),D\varphi(z))=F(D^{2}v(z)-\varepsilon I,D\varphi(z))\geq
F(Ca\,e\otimes e-(a+\varepsilon)I,D\varphi(z))$ $\displaystyle=F(Ca\,e\otimes
e,D\varphi(z))-(a+\varepsilon)F(I,D\varphi(z))\geq
Ca\lambda-(a+\varepsilon)\tilde{\lambda}\sqrt{n}\,$

which is a contradiction if $C>(\tilde{\lambda}\sqrt{n}+1)/\lambda$ and
$\varepsilon$ is sufficiently small.

Now, from the area formula, using (5.3) we obtain

(5.4) $|B|\leq\int_{A}\Bigl{|}{\rm
det}\Bigl{(}I+\frac{1}{a}D^{2}v(x)\Big{)}\Big{|}\,dx\leq C|A|,$

where $C$ is another universal constant.

If $v$ is not semi-concave for $\varepsilon>0$ we define the inf-convolution,
setting for $x\in\overline{B}_{r}(x_{0})$

$v_{\varepsilon}(x)=\inf_{y\in\overline{B}_{r}(x_{0})}\Big{\\{}v(y)+\frac{1}{\varepsilon}|y-x|^{2}\Big{\\}}.$

It is easily checked that $v_{\varepsilon}$ is semi-concave: Since $v$ is
lower semicontinuous and bounded we have also that $v_{\varepsilon}$ converges
pointwise to $v$ in $\overline{B}_{r}(x_{0})$. Moreover each $v_{\varepsilon}$
is a viscosity $(2a)$-supersolution of the equation (5.1) in $B_{r}(x_{0})$.
In fact, if $\varphi\in C^{2}(B_{r}(x_{0}))$ lies below $v_{\varepsilon}$ in a
neighborhood of $\overline{x}\in B_{r}(x_{0})$ and touches $v_{\varepsilon}$
at $\overline{x}$, let $\overline{y}$ the point in $\overline{B}_{r}(x_{0})$
such that

$\inf_{y\in\overline{B}_{r}(x_{0})}\Big{\\{}v(y)+\frac{1}{\varepsilon}|y-\overline{x}|^{2}\Big{\\}}=v(\overline{y})+\frac{1}{\varepsilon}|\overline{y}-\overline{x}|^{2}.$

Then the function

$\varphi(x+\overline{x}-\overline{y})+v(\overline{y})-\varphi(\overline{x})$

touches $v$ from below at $\overline{y}$ and thus

$F(D^{2}\varphi(\overline{x}),D\varphi(\overline{x}))\leq a.$

Observe that for $\varepsilon$ small enough the contact set $A_{\varepsilon}$
of $v_{\varepsilon}$ is contained in $B_{r}(x_{0})$. Indeed if this is not
true there exist a sequence $\varepsilon_{n}$ converging to $0$ and points
$x_{n}\in A_{\varepsilon_{n}}$, the contact set of $v_{\varepsilon_{n}}$, such
that $x_{n}\in\partial B_{r}(x_{0})$. Therefore there exist $y_{n}\in B$ and
$c_{n}\in\mathbb{R}$ such that

$-\frac{a}{2}|x-y_{n}|^{2}+c_{n}\leq v_{\varepsilon_{n}}(x)\,\,\,\forall
x\in\overline{B}_{r}(x_{0}),\quad-\frac{a}{2}|x_{n}-y_{n}|^{2}+c_{n}=v_{\varepsilon_{n}}(x_{n}).$

Since $B$ is closed and the $v_{\varepsilon_{n}}$ are equibounded we may
assume that $y_{n}\to\bar{y}\in B$, $c_{n}\to\bar{c}$ and
$x_{n}\to\bar{x}\in\partial B_{r}(x_{0})$. Thus, from the first inequality
above we have that the paraboloid $-\frac{a}{2}|x-\bar{y}|^{2}+\bar{c}$ stays
below $v$ in $B_{r}(x_{0})$. Moreover, by lower semicontinuity we have

$\begin{split}-\frac{a}{2}|\bar{x}-\bar{y}|^{2}+\bar{c}&=\lim_{n\to\infty}v_{\varepsilon_{n}}(x_{n})\\\
&=\lim_{n\to\infty}\Big{(}v(z_{n})+\frac{1}{\varepsilon_{n}}|z_{n}-x_{n}|^{2}\Big{)}\geq\liminf_{z\to\bar{x}}\geq
v(z)\geq v(\bar{x}).\end{split}$

Hence $\bar{x}\in A\cap\partial B_{r}(x_{0})$ which is impossible by
assumption. In particular we may assume that (5.4) holds with $A$ replaced by
$A_{\varepsilon}$ for $\varepsilon$ sufficiently small.

Using the fact that $B$ is closed and the pointwise convergence of
$v_{\varepsilon}$ to $v$, a similar argument shows

$\bigcap_{h=1}^{\infty}\bigcup_{k=h}^{\infty}A_{1/k}\subset A.$

From the above inclusion and (5.4) we then conclude that $|B|\leq C|A|$. ∎

Let $v:\overline{B}_{1}\to(0,\infty)$ be a lower semicontinuous function. If
$a>0$ we denote by $A_{a}$ the set of points where $v$ can be touched from
below by a paraboloid of opening $a$ and vertex in $\overline{B}_{1}$ and
where the value of $v$ is also smaller than $a$.

$\displaystyle A_{a}:$
$\displaystyle=\Big{\\{}x\in\overline{B}_{1}:\,\,v(x)\leq a\,\,\text{and there
exists}\,\,y\in\overline{B}_{1}\,\,\text{such that}$
$\displaystyle\qquad\qquad\qquad\min_{z\in\overline{B}_{1}}\Big{(}v(z)+\frac{a}{2}|z-y|^{2}\Big{)}=v(x)+\frac{a}{2}|x-y|^{2}\Big{\\}}\,.$

######  Lemma 5.2.

There exist two constants $c_{1}>0$, $C_{1}>1$, with the following properties:
Let $0<a<1/C_{1}$ and let $v:\overline{B}_{1}\to(0,\infty)$ be a viscosity
$(C_{1}a)$-supersolution of the equation (5.1), bounded from above. If

$\overline{B}_{r}(x_{0})\subset B_{1},\qquad
A_{a}\cap\overline{B}_{r}(x_{0})\not=\emptyset\,,$

then

$\big{|}A_{C_{1}a}\cap B_{r/8}(x_{0})|\geq c_{1}|B_{r}|\,.$

###### Proof.

Up to replace $r$ with $r+\varepsilon$ and then letting $\varepsilon\to 0^{+}$
we may assume that there exists

$x_{1}\in A_{a}\cap B_{r}(x_{0})\,.$

Thus there exists $y_{1}\in\overline{B}_{1}$ such that the paraboloid

$P(x;y_{1})=v(x_{1})+\frac{a}{2}|x_{1}-y_{1}|^{2}-\frac{a}{2}|x-y_{1}|^{2}$

touches $v$ in $x_{1}$ from below. We claim that there exists a universal
constant $C$ such that there exists a point $z\in\overline{B}_{r/16}(x_{0})$
such that

(5.5) $v(z)\leq P(z;y_{1})+Car^{2}\,.$

If $x_{1}\in\overline{B}_{r/16}(x_{0})$ then we may take trivially $z=x_{1}$.
Otherwise, we consider the function

$\varphi(x)=\begin{cases}\alpha^{-1}(|x|^{-\alpha}-1)&\quad\text{if}\,\,\,\displaystyle\frac{1}{16}\leq|x|\leq
1,\cr\alpha^{-1}(16^{\alpha}-1)&\quad\text{if}\,\,\,\displaystyle|x|\leq\frac{1}{16}\,,\end{cases}$

with $\alpha>0$, universal, to be chosen. Then, for $x\in B_{r}(x_{0})$, we
set

$\psi(x)=P(x;y_{1})+ar^{2}\varphi\Big{(}\frac{x-x_{0}}{r}\Big{)}\,.$

Note that $|D\psi|\leq a(16^{\alpha+1}+2)\leq 1$ if $a$ is small enough and
that in the annulus $B_{r}(x_{0})\setminus\overline{B}_{r/16}(x_{0})$ we have

(5.6)
$F(D^{2}\psi,D\psi)=aF\Big{(}D^{2}\varphi\Big{(}\frac{x-x_{0}}{r}\Big{)},D\psi\Big{)}-aF(I,D\psi)\geq
a\lambda(\alpha+1)-a\tilde{\lambda}\sqrt{n}>a,$

provided $\alpha>0$ is chosen large enough.

Let us now denote by $z$ a minimum point of $v-\psi$ in
$\overline{B}_{r}(x_{0})$. Since
$v(x_{1})-\psi(x_{1})=-ar^{2}\varphi\big{(}\frac{x_{1}-x_{0}}{r})<0$, the
minimum of $v-\psi$ is strictly negative. Therefore $z\not\in\partial
B_{r}(x_{0})$ since $v-\psi\geq 0$ on $\partial B_{r}(x_{0})$. On the other
hand, if $r/16<|z-x_{0}|<r$, observing that $|D\psi|\leq
a(16^{\alpha+1}+2)\leq C_{1}a$, if we choose $C_{1}$ large enough, we would
have that $\psi$ is an admissible test function for $v$ in a neighborhood of
$z$ satisfying (5.6), which is impossible. Therefore
$z\in\overline{B}_{r/16}(x_{0})$ and we have

$v(z)\leq\psi(z)=P(z;y_{1})+ar^{2}\alpha^{-1}(16^{\alpha}-1)\,,$

thus proving (5.5).

To conclude the proof, consider for every $y\in\overline{B}_{r/64}(z)$ the
paraboloid

$P(x;y_{1})-C^{\prime}\frac{a}{2}|x-y|^{2}+c_{y}\,,$

where $C^{\prime}$ is a universal constant to be chosen and $c_{y}$ is such
that the above paraboloid touches $v$ from below. Note that the above
paraboloid has opening $(C^{\prime}+1)a$ and vertex at

$\frac{y_{1}}{C^{\prime}+1}+\frac{C^{\prime}y}{C^{\prime}+1}\,.$

Observe that

$P(z;y_{1})-C^{\prime}\frac{a}{2}|z-y|^{2}+c_{y}\leq v(z)\leq
P(z;y_{1})+Car^{2}\,,$

hence $c_{y}\leq Car^{2}+\frac{C^{\prime}a}{2}\big{(}\frac{r}{64}\big{)}^{2}$.
On the other hand, if $|x-z|\geq\frac{r}{16}$, we have

$\displaystyle P(x;y_{1})-C^{\prime}\frac{a}{2}|x-y|^{2}+c_{y}$
$\displaystyle\leq
P(x;y_{1})-C^{\prime}\frac{a}{2}\Big{(}\frac{3r}{64}\Big{)}^{2}+Car^{2}+\frac{C^{\prime}a}{2}\Big{(}\frac{r}{64}\Big{)}^{2}$
$\displaystyle<P(x;y_{1})\leq v(x)\,,$

provided that we choose $C^{\prime}$ large enough independently of $a$ and
$r$. Thus the contact point $x_{y}$ belongs to the ball $B_{r/16}(z)\subset
B_{r/8}(x_{0})$. Note that for $y\in\overline{B}_{r/64}(z)$ the vertex
$\frac{y_{1}}{C^{\prime}+1}+\frac{C^{\prime}y}{C^{\prime}+1}$ spans the ball
with center $\frac{y_{1}}{C^{\prime}+1}+\frac{C^{\prime}z}{C^{\prime}+1}$ and
radius $C^{\prime}r/64(C^{\prime}+1)$, which is contained in $B_{1}$, provided
that $C^{\prime}$ is large enough. Moreover, the gradient of the function
$x\mapsto P(x;y_{1})-C^{\prime}\frac{a}{2}|x-y|^{2}+c_{y}$ is smaller than
$2(C^{\prime}+1)a$ and

$v(x_{y})=P(x_{y};y_{1})-C^{\prime}\frac{a}{2}|x_{y}-y|^{2}+c_{y}\leq
a+2a+c_{y}\leq C^{\prime\prime}a$

for a sufficiently large, universal, constant
$C^{\prime\prime}>2(C^{\prime}+1)a$. Therefore, by applying Lemma 5.1 with
$B_{r}(x_{0})$ and $B$ replaced respectively by $B_{r/8}(x_{0})$ and the ball
of radius $C^{\prime}r/64(C^{\prime}+1)$ and center
$\frac{y_{1}}{C^{\prime}+1}+\frac{C^{\prime}z}{C^{\prime}+1}$, with $a$
replaced by $C^{\prime\prime}a$, we have that, if $3C^{\prime\prime}a\leq 1$

$\big{|}A_{C^{\prime\prime}a}\cap B_{r/8}(x_{0})\big{|}\geq
c_{0}\Big{(}\frac{C^{\prime}}{C^{\prime}+1}\Big{)}^{n}\big{|}B_{r/64}|\,,$

from which (5.2) follows with
$C_{1}=3\max\\{C^{\prime\prime},16^{\alpha+1}+2\\}$. ∎

We can now give the

###### Proof of Proposition 3.5.

Let $x_{0}\in B_{1/2}$ be a point where $v(x_{0})\leq\nu$. Then the function
$u(x)=v(x_{0}+x)$ is a positive viscosity $(C_{0}^{k}\nu)$-supersolution of
(3.9) in $\overline{B}_{1}$ with $u(0)\leq\nu$. Consider the paraboloid with
vertex at the origin and opening $20\nu$ touching $u$ from below and observe
that since $u$ is positive this paraboloid touches $u$ in $B_{1/3}$. Therefore
$A_{20\nu}\cap\overline{B}_{1/3}\not=\emptyset$. Moreover, setting
$D_{0}=A_{20\nu}\cap\overline{B}_{1/3}$, from Lemma 5.2 we know that if
$B_{r}(x)\subset B_{1}$, $B_{r/8}(x)\subset B_{1/3}$ and $D_{0}\cap
B_{r}(x)\not=\emptyset$, then

$|(A_{20C_{1}\nu}\cap\overline{B}_{1/3})\cap B_{r/8}(x)|=|A_{20C_{1}\nu}\cap
B_{r/8}(x)|\geq c_{1}|B_{r}|\,,$

provided $20C_{1}\nu\leq 1$ and $v$, hence $u$, is a viscosity
$(20C_{1}\nu)$-supersolution. By applying the same lemma to
$D_{1}=A_{20C_{1}\nu}\cap\overline{B}_{1/3}$ and proceeding by induction, we
have that the sets $D_{0}\subset D_{1}\subset\dots\subset D_{k}\subset
B_{1/3}$, where $D_{i}=A_{20C_{1}^{i}\nu}\cap\overline{B}_{1/3}$ for
$i=1,\dots,k$, have all the property that if $B_{r}(x)\subset B_{1}$,
$B_{r/8}(x)\subset B_{1/3}$ and $D_{i}\cap B_{r}(x)\not=\emptyset$, for $i\leq
k$, then

$|D_{i+1}\cap B_{r/8}(x)|\geq c_{1}|B_{r}|\,,$

provided $20C_{1}^{k}\nu\leq 1$ and $v$, hence $u$, is a viscosity
$(20C_{1}^{k}\nu)$-supersolution. Then, using Lemma 2.3 in [16] and setting
$C_{2}=20C_{1}$, it follow that for a suitable $\mu$ depending only on
$c_{1}$, hence universal,

$\big{|}\\{x\in B_{1/3}(x_{0}):\,v(x)\leq
C_{2}^{k}\nu\\}\big{|}\leq(1-\mu^{k})|B_{1/3}|\,.$

From this inequality the conclusion then follows by a standard rescaling and
covering argument. ∎

## References

  * [1] L. Ambrosio, N. Fusco, and D. Pallara, Functions of bounded variation and free discontinuity problems, Oxford Mathematical Monographs, The Clarendon Press, Oxford University Press, New York, 2000.
  * [2] J. Dalphin, Uniform ball property and existence of optimal shapes for a wide class of geometric functionals, Interfaces Free Bound., 20 (2018), pp. 211–260.
  * [3] G. De Philippis and F. Maggi, Regularity of free boundaries in anisotropic capillarity problems and the validity of Young’s law, Arch. Ration. Mech. Anal., 216 (2015), pp. 473–568.
  * [4] M. G. Delgadino and F. Maggi, Alexandrov’s theorem revisited, Anal. PDE, 12 (2019), pp. 1613–1642.
  * [5] F. Duzaar and K. Steffen, Optimal interior and boundary regularity for almost minimizers to elliptic variational integrals, J. reine angew. Math., 564 (2002), pp. 73–138.
  * [6] X. Fernández-Real and J. Serra, Regularity of minimal surfaces with lower-dimensional obstacles, J. Reine Angew. Math., 767 (2020), pp. 37–75.
  * [7] R. Finn, Equilibrium capillary surfaces, vol. 284 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], Springer-Verlag, New York, 1986.
  * [8] M. Focardi and E. Spadaro, How a minimal surface leaves a thin obstacle, Ann. Inst. H. Poincaré Anal. Non Linéaire, 37 (2020), pp. 1017–1046.
  * [9] I. Fonseca, N. Fusco, G. Leoni, and M. Morini, Global and local energy minimizers for a nanowire growth model, Ann. Inst. H. Poincaré Anal.Non Linéaire, (2022), to appear.
  * [10] N. Guillen, Optimal regularity for the Signorini problem, Calc. Var. Partial Differential Equations, 36 (2009), pp. 533–546.
  * [11] P. Krogstrup, S. Curiotto, E. Johnson, M. Aagesen, J. Nygård, and D. Chatain, Impact of the liquid phase shape on the structure of iii-v nanowires, Physical Review Letters, 106 (2011), 125505.
  * [12] F. Maggi, Sets of finite perimeter and geometric variational problems: an introduction to Geometric Measure Theory, vol. 135 of Cambridge Studies in Advanced Mathematics, Cambridge University Press, 2012\.
  * [13] E. Milakis and L. Silvestre, Regularity for the nonlinear Signorini problem, Adv. Math., 217 (2008), pp. 1301–1312.
  * [14] M. G. Mora and M. Morini, Functionals depending on curvatures with constraints, Rend. Sem. Mat. Univ. Padova, 104 (2000), pp. 173–199.
  * [15] D. J. A. Richardson, Variational problems with thin obstacles, ProQuest LLC, Ann Arbor, MI, 1978.  Thesis (Ph.D.)–The University of British Columbia (Canada).
  * [16] O. Savin, Small perturbation solutions for elliptic equations, Comm. Partial Differential Equations, 32 (2007), pp. 557–578.
  * [17] L. Simon, Lectures on geometric measure theory, vol. 3 of Proceedings of the Centre for Mathematical Analysis, Australian National University, Australian National University Centre for Mathematical Analysis, Canberra, 1983.
  * [18] J. E. Taylor, Boundary regularity for solutions to various capillarity and free boundary problems, Comm. Partial Differential Equations, 2 (1977), pp. 323–357.