# Painting 3D Nature in 2D:
View Synthesis of Natural Scenes from a Single Semantic Mask

Shangzhan Zhang1 Sida Peng1 Tianrun Chen1 Linzhan Mou1 Haotong Lin1
Kaicheng Yu2 Yiyi Liao1 Xiaowei Zhou1
1Zhejiang University 2Alibaba Group

###### Abstract

We introduce a novel approach that takes a single semantic mask as input to
synthesize multi-view consistent color images of natural scenes, trained with
a collection of single images from the Internet. Prior works on 3D-aware image
synthesis either require multi-view supervision or learning category-level
prior for specific classes of objects, which can hardly work for natural
scenes. Our key idea to solve this challenging problem is to use a semantic
field as the intermediate representation, which is easier to reconstruct from
an input semantic mask and then translate to a radiance field with the
assistance of off-the-shelf semantic image synthesis models. Experiments show
that our method outperforms baseline methods and produces photorealistic,
multi-view consistent videos of a variety of natural scenes. Project website:
https://zju3dv.github.io/paintingnature/

Figure 1: Given only a single semantic map as input (first row), our approach
optimizes neural fields for view synthesis of natural scenes. Photorealistic
images can be rendered via neural fields (the last two rows).

##  1 Introduction

Natural scenes are indispensable content in many applications such as film
production and video games. This work focuses on a specific setting of
synthesizing novel views of natural scenes given a single semantic mask, which
enables us to generate 3D contents by editing 2D semantic masks. With the
development of deep generative models, 2D semantic image synthesis methods
[52, 68, 73, 27] have achieved impressive advances. However, they do not
consider the underlying 3D world and cannot generate multi-view consistent
free-viewpoint videos.

To address this problem, a straightforward approach is first utilizing
semantics-driven image generator like SPADE [52] to synthesize an image from
the input semantic mask and then predicting novel views based on the generated
image. Although the existing single-view view synthesis methods [38, 79, 35,
51, 74, 58] achieve impressive rendering results, they typically require
training networks on posed multi-view images. Compared to urban or indoor
scenes, learning to synthesize natural scenes is a challenging task as it is
difficult to collect 3D data or posed videos of natural scenes for training,
as demonstrated in [36], making the aforementioned methods not applicable.
AdaMPI [20] designs a training strategy to learn the view synthesis network on
single-view image collections. It warps images to random novel views and warps
them back to the original view. An inpainting network is trained to fill the
holes in disocclusion regions to match the original images. After training,
the inpainting network is used to generate pseudo multi-view images for
training a view synthesis network. Our experimental results in Section 4.5
show that the inpainting network struggles to output high-quality image
contents in missing regions under large viewpoint changes, thus limiting the
rendering quality.

In this paper, we propose a novel framework for semantics-guided view
synthesis of natural scenes by learning prior from single-view image
collections. Based on the observation that semantic masks have much lower
complexity than images, we exploratively divide this task into two simpler
subproblems: we first generate semantic masks at novel views and then
translate them to RGB images through SPADE. For view synthesis of semantic
masks, the input semantic mask is first translated to a color image by SPADE,
and a depth map is predicted from the color image by a depth estimator [56].
Then the input semantic mask is warped to novel views using the predicted
depth map and refined by an inpainting network, trained by a self-supervised
learning strategy on single-view image collections. Our experiments show that,
in contrast to images, the novel view synthesis of semantic masks is much
easier to learn by the network.

It is observed that semantic masks generated by the inpainting network tend to
be view inconsistent. Although the inconsistency across views seems minor,
SPADE could generate quite different contents in these regions. Fig. 4
presents two examples. To solve this issue, we learn a neural semantic field
to fuse and denoise these semantic masks for better multi-view consistency.
Finally, we translate the multi-view semantic masks to color images by SPADE
and reconstruct a neural scene representation for view-consistent rendering.

Extensive experiments are conducted on the LHQ dataset [65], a widely-used
benchmark dataset for semantic image synthesis. The results demonstrate that
our approach significantly outperforms baseline methods both qualitatively and
quantitatively. We also show that by editing the input semantic mask, our
approach is capable of generating various high-quality rendering results of
natural scenes, as shown in Fig. 1.

##  2 Related Works

#### Semantics-guided view synthesis.

This task takes a single 2D semantic mask as input and outputs free-viewpoint
videos of the 3D scene. Only a few works attempt to tackle this challenging
task, including SVS [25] and GVS [19]. Given posed videos, SVS [25] first
extracts the semantic map for each video frame. During training, it leverages
a SPADE-like generator [52] to produce an RGB image from the input semantic
mask at a particular frame and then regresses from the image to the multi-
plane images (MPI) as the generated 3D scene for view synthesis. The generated
MPI is rendered into several viewpoints, and the SVS model is optimized by
minimizing the difference between rendered images and observed images of the
input video. However, they train their models on datasets of posed videos [86,
4, 12]. A problem with such a training strategy is that a large number of
videos with calibrated camera poses are hard to obtain in real life, which
could be expensive and limit the diversity of training data. Qiao et al. [55]
mainly focus on novel-view scene layout generation and their model is trained
on posed images, while we concentrate on rendering consistent RGB images with
single image collections for training.

#### Single-view view synthesis.

Recently, many works have focused on single-view view synthesis with monocular
RGB image as the input to generate free-viewpoint videos [33, 24, 74, 63, 38,
49, 70, 57, 58]. Among existing works, some use explicit 3D representations,
such as layered depth image (LDI) [22, 62] and multi-plane image (MPI) [86],
which can capture visible contents and infer disocclusion regions. Another
line of research predicts neural radiance fields [42] from a single image. For
example, PixelNeRF [79] extracts image features using 2D CNN and constructs an
aligned feature field to render novel views. Li et al. [35] combine MPI and
NeRF representation and learn a continuous 3D field. These works learn 3D
representations to perform novel-view synthesis, and the learning is mainly
based on multi-view images or posed videos for supervision. However, similar
to the dataset constraint in semantic view synthesis, large-scale multi-view
datasets are also rare, thus bringing challenges for high-quality view
synthesis for natural scenes.

Recently, some methods have utilized single-view image collections to train a
neural network that performs novel view synthesis given a single-view image.
For example, [33] and [63] use a monocular depth estimation network to
construct LDI representation and leverage the inpainting network to synthesize
disocclusion content to perform view synthesis. AdaMPI [20] proposes a warp-
back training strategy to train MPI on the COCO [5] datasets. Li et al. [36]
propose a cycle-rendering strategy that warps the input image to a virtual
view and then warps it back to the original view, finally resulting in an
image containing holes. A refinement network is trained to refine the final
image to match the original image. They tailored a viable path by
demonstrating the capability of single-view image collections collected from
the Internet in monocular view synthesis. Still, no prior works have attempted
to train models solely using single-view image collections while performing
novel view synthesis from a single semantic mask.

A naive solution to leverage existing single-view image collections [65] to
learn models for semantics-guided semantic view synthesis is extending the
above approaches [63, 20] through a two-stage scheme: first converting the
semantic mask to an RGB image and then applying the view synthesis model.
However, this naive solution does not fully utilize semantic information, thus
leading to poor performance.

#### Neural Radiance Fields.

Neural Radiance Fields [42] and its subsequent works significantly advance the
realm of novel view synthesis [82, 2, 1, 46, 76] and 3D reconstruction [50,
83, 13]. While the above works focus on rendering realistic novel view images
or reconstructing accurate 3D geometry, some works currently aim to exploit 2D
pretrained models to learn a priori knowledge for the neural field. Examples
include creating 3D objects driven by text [71, 28, 54], animating NeRF by
audio signals [18, 40], and stylizing scenes [81, 26, 45]. Unlike the previous
works, in this paper, we focus on utilizing semantic image synthesis models.
Semantic-NeRF [85] and later works [15, 34, 69] use NeRF as a powerful 3D
fusion tool to fuse 2D semantic information. However, they do not explore how
to recover a semantic field from only a single semantic mask. Another line of
work [11, 67, 66] extends 3D-aware generative modeling [59, 60, 48, 8, 47] to
edit 3D appearance and geometry via a semantic mask, but they mainly conduct
their experiments on object-centric datasets (e.g., FFHQ [31]) with known
camera distribution and fail to generate complex scenes on the nature scene
datasets. A closely related work GANcraft [21] and its extended work [29] use
image-to-image translation techniques [52, 68, 73, 27] to synthesize pseudo
ground truths and discriminators to make generative free-viewpoint videos more
realistic from 3D semantic labels. Nevertheless, they need 3D semantic labels
to render consistent 2D semantic masks and fail when they only take a single
semantic mask as input.

#### Image-to-image translations

Image-to-image translations [52, 68, 73, 27] have made tremendous development
and can synthesize realistic natural images. Pix2Pix [27] firstly leverages
conditional GAN [43] to improve the performance of semantic image synthesis.
The performance of image synthesis is significantly enhanced by SPADE [52],
which proposes a spatial-varying normalization layer. OASIS [68] proposes a
novel discriminator for semantic image synthesis [30]. However, these works
can only synthesize 2D images and are not able to generate images of different
views.

##  3 Method

Figure 2: Illustration of our pipeline. Left: Our pipeline can be divided into
two steps: we first generate multi-view semantic masks with an inpainting
network and then convert semantic masks to RGB images using SPADE. In order to
denoise and fuse semantic information, a semantic field is learned for
rendering multi-view consistent masks. Finally, a neural scene representation
is optimized to fuse appearance information provided by SPADE, which enables
view-consistent rendering. Right: Our semantic inpainting network and SPADE
are trained on single-view image collections.

Our goal is to perform photorealistic view synthesis of natural scenes, given
a single semantic mask, by learning prior from single-view image collections.
To this end, we divide this task into two simpler subproblems, as described in
this section. We first introduce how to generate multi-view consistent
semantic masks from the given semantic mask (Section 3.1). Then, Section 3.2
discusses how to translate the multi-view semantic masks to color images by
SPADE and recover a neural scene representation for view-consistent rendering.

###  3.1 Generating view-consistent semantic masks

Fig. 2 illustrates the overview of generating multi-view consistent semantic
masks from a single semantic mask. Specifically, we first warp the given
semantic mask to novel views. Then, an inpainting network is utilized to fill
in the disocclusion areas of the warped semantic mask at each novel view.
After obtaining multiple infilled semantic masks at different viewpoints, we
recover a neural semantic field that can fuse and denoise the multi-view
semantic information. Finally, multi-view semantic masks can be obtained by
the semantic field.

#### Warping semantic mask.

Our approach warps the given semantic map to novel views through the depth-
based warping technique. We first convert the input semantic map to the
corresponding RGB image using SPADE [52], and then use a monocular depth
estimation network [56] to predict the depth map from the generated RGB image.
Then, a 3D triangular mesh is constructed based on the predicted depth
following Shih et al. [63]. The semantic mask is lifted to the mesh, whose
vertices’ color is assigned as the corresponding semantic label. The generated
3D triangular mesh in such a manner may contain spurious edges due to the
depth discontinuities in the depth map. To solve this problem, we remove the
edges whose vertices are far from each other. Eventually, we warp the semantic
mask to the novel views using a mesh renderer following [63].

#### Semantic mask inpainting.

Figure 3: Training a semantic inpainting network. Our semantic inpainting
network takes the $\mathbf{S}_{i\rightarrow j\rightarrow i}$ as input and is
trained to recover the $\mathbf{S}_{i}$.

Directly warping the given semantic mask to novel views brings many holes in
disocclusion regions. To inpaint the missing contents, we train a semantic
inpainting network on single-view natural image collections [65] using the
self-supervised technique in [20]. Fig. 3 shows our training strategy for
semantic inpainting networks. Specifically, we first use a pre-trained image
segmentation model [9] and a monocular depth estimation model [56] to produce
semantic masks and depth maps for the natural images. At each training
iteration, an image is randomly sampled from the dataset as the source image
$\mathbf{I}_{i}$. We then use the corresponding depth map
$\mathbf{\hat{D}}_{i}$ to warp the original semantic mask $\mathbf{S}_{i}$ to
a random target view $j$, producing warped semantic mask
$\mathbf{S}_{i\rightarrow j}$ and depth map $\mathbf{\hat{D}}_{i\rightarrow
j}$ at target view $j$. Next, we warp semantic mask $\mathbf{S}_{i\rightarrow
j}$ back to the source view using depth map $\mathbf{\hat{D}}_{i\rightarrow
j}$, which generates a semantic mask $\mathbf{S}_{i\rightarrow j\rightarrow
i}$ with holes. Finally, we input $\mathbf{S}_{i\rightarrow j\rightarrow i}$
into the semantic inpainting network and train the network to infill these
holes, which is supervised with the original semantic mask $\mathbf{S}_{i}$.

At test time, we first randomly sample a set of viewpoints, then warp the
given semantic mask $\mathbf{S}_{0}$ to these viewpoints to generate warped
semantic masks, and finally apply the inpainting network to fill in their
disocclusion regions to generate the infilled multi-view semantic masks.

#### Semantic field fusion.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
w/ semantic field | w/o semantic field
Figure 4: The effectiveness of semantic field. The cropped patch clearly
indicates the minor change in the semantic masks across different viewpoints
(the first and second columns are adjacent viewpoints) brings the unwanted
large region change in RGB images generated by SPADE.

We observed that the infilled semantic masks are not view-consistent. Although
the artifact regions in semantic masks seem trivial, the generated images from
SPADE could be very different on these regions across different viewpoints.
Fig. 4 presents an example. To tackle this problem, a semantic field is
introduced to fuse and denoise infilled semantic masks. We adopt a continuous
neural field to represent the semantics and geometry of a 3D scene, similar to
[17]. For any query point $\mathbf{x}$ in 3D space, an MLP network
$f_{\theta}$ maps it to an SDF value $d$ and an intermediate feature
$\mathbf{z}$, and another MLP network $f_{\phi}$ maps $\mathbf{z}$ to a
semantic logits $\mathbf{s}$. The neural field is defined as:

$\displaystyle
f_{\theta}:\mathbf{x}\in\mathbb{R}^{3}\mapsto\left(d\in\mathbb{R},\mathbf{z}\in\mathbb{R}^{c}\right)$
(1) $\displaystyle
f_{\phi}:\mathbf{z}\in\mathbb{R}^{c}\mapsto\mathrm{\mathbf{s}}\in\mathbb{R}^{M_{s}},$

where $M_{s}$ denotes the number of semantic classes. We render the semantic
field into semantic logits and depth through the SDF-based volume rendering
[78, 72].

Considering that the sky is very distant from the foreground, we handle the
foreground and the sky separately, following the practice in [21]. The sky is
assumed to be a distant 2D plane, and the semantic probability of the sky is
defined as a constant one-hot vector $\mathbf{P}_{\text{sky}}$. The final
semantic probability is formulated as:

$\mathbf{Y}(\mathbf{r})=\mathbf{P}_{\text{fg}}(\mathbf{r})T_{\text{fg}}(\mathbf{r})+(1-T_{\text{fg}}(\mathbf{r}))\mathbf{P}_{\text{sky}},$
(2)

where $T_{\text{fg}}$ is the accumulated transmittance of the foreground along
camera ray $\mathbf{r}$, and $\mathbf{P}_{\text{fg}}$ is the semantic
probability obtained by applying the softmax layer to the rendered semantic
logits.

To learn the semantic field, the cross entropy loss is applied to compare the
rendered semantic probability $\mathbf{P(\mathbf{r})}$ and the semantic
probability $\mathbf{P}^{*}(\mathbf{r})$ provided by the infilled semantic
masks:

$\mathcal{L}_{\mathbf{P}}=-\sum_{\mathbf{r}\in\mathcal{R}}\sum_{k=1}^{M_{s}}\mathbf{P}_{k}^{*}(\mathbf{r})\log\mathbf{P}_{k}(\mathbf{r}).$
(3)

In addition, we use depth maps to learn the geometry of the semantic field. In
detail, the infilled semantic masks are first converted to RGB images via
SPADE, then processed by a monocular depth estimation network [56] to predict
depth maps. A scale- and shift-invariant loss [56, 80] is utilized to
calculate the difference between the rendered depth map $\mathbf{D}$ and the
predicted depth map $\mathbf{\hat{D}}$, which is defined as:

$\mathcal{L}_{\text{depth}}=\sum_{\mathbf{r}\in\mathcal{R}^{\prime}}\|(w\mathbf{D}(\mathbf{r})+q)-\mathbf{\hat{D}}(\mathbf{r})\|^{2},$
(4)

where $\mathcal{R}^{\prime}$ means camera rays of image pixels excluding the
sky region. $w$ and $q$ are used to align the scale and shift of $\mathbf{D}$
and $\mathbf{\hat{D}}$, which can be obtained using a least-squares criterion
[56, 14].

To separate the foreground and the sky region, we apply a loss on the
accumulated transmittance:

$\mathcal{L}_{\text{trans}}=\sum_{\mathbf{r}\in\mathcal{R}}(\log(T_{\text{fg}}(\mathbf{r}))+\log(1-T_{\text{fg}}(\mathbf{r}))).$
(5)

This loss enforces the transmittance to be either 0 or 1. The overall loss is
described in the supplementary material.

###  3.2 Natural scene representations

Directly translating multi-view semantic masks obtained in Section 3.1 to RGB
images through SPADE fails to produce multi-view consistent rendering, as
shown in the supplementary material. To resolve this issue, we learn a natural
scene representation to fuse appearance information provided by SPADE. This
section describes the generation of natural scenes from the learned semantic
field. We first introduce the neural representation of natural scenes. Then,
the rendering and training of the scene representation are described.

The geometry of the scene is directly modeled as the trained MLP network
$f_{\theta}$ (Eq. 1) of the semantic field. To represent the scene’s
appearance, we recover an appearance field $f_{\xi}$. Following EG3D [7], a
tri-plane feature map is adopted to map a point to a feature vector.
Specifically, given a point $\mathbf{x}$, it is orthogonally projected to the
feature planes to retrieve three feature vectors, which are concatenated into
the final feature vector. We use an MLP network to regress the RGB value
$\mathbf{c}$ from the aggregated feature vector. The appearance field is
defined as:

$f_{\xi}:\mathbf{x}\in\mathbb{R}^{3}\mapsto\mathbf{c}\in\mathbb{R}^{3}$ (6)

For the scene representation, we separately model the foreground and the sky
region. The sky is implemented as a 2D image plane generated by a 2D generator
network, and we place it at a distance. For a ray classified as ‘sky’, the sky
image plane maps the intersection point $(u,v)$ between the ray and the sky
plane to the RGB value.

To efficiently render the scene representation, we leverage the pre-learned
scene geometry to guide the sampling of points along camera rays. The mesh is
first extracted from the trained MLP network $f_{\theta}$. Then, for each
camera ray, we only predict the color for the point on the mesh surface,
similar to [37, 44]. With surface-guided rendering, the computational cost of
synthesizing full-resolution images is significantly reduced.

During training, the geometry network $f_{\theta}$ is fixed. The appearance
network is optimized based on perceptual and adversarial losses. We first use
the learned semantic field to render multi-view semantic masks which are then
translated into images using SPADE. The perceptual loss [30] is adopted to
compare the rendered image $\mathbf{C}$ and generated image
$\mathbf{\hat{C}}$:

$\mathcal{L}_{\text{feat
}}(\mathbf{\hat{C}},\mathbf{C})=\left\|\phi(\mathbf{\hat{C}})-\phi(\mathbf{C})\right\|_{2}^{2},$
(7)

where $\phi$ denotes the VGG network [64]. Perceptual loss makes training
procedure more stable and faster.

Additionally, the adversarial loss is applied to make the rendered images more
photorealistic and prevent blurriness caused by the inconsistency of input
views, which is demonstrated in GANcraft [21]. Our rendered images are taken
as “fake” samples, and generated images are taken as “real” samples. We adopt
the OASIS discriminator [68] as our discriminator, and use the same generator
and discriminator loss as [68].

Methods | FID$\downarrow$ | KID$\downarrow$
---|---|---
SPADE [52]+InfiniteNatureZero [36] | 149.80 | 0.080
SPADE [52]+3DPhoto [63] | 127.74 | 0.064
SPADE [52]+AdaMPI [20] | 185.96 | 0.115
$\text{GVS}^{\ast}$ [19] | 141.64 | 0.084
Ours | 109.85 | 0.050
Table 1: Quantitative comparisons on the LHQ dataset. “SPADE + *” means a two-
stage pipeline that first generates an image with SPADE and then performs
single-view view synthesis. “GVS∗” means that we train GVS on the LHD dataset
using the strategy in AdaMPI.

##  4 Experiments

###  4.1 Datasets

We use the LHQ dataset [65] to train our SPADE and semantic refinement
network. The LHQ dataset is a large collection of landscape photos collected
from the Internet. To prepare the semantic label maps for each image, we use
COCO-Stuff [5] to train a DeepLab v2 [9] network. The training data for
semantic inpainting is synthesized by the back-warping strategy as mentioned
in Section 3.1.

###  4.2 Implementation details

To train the semantic field, 1024 rays are sampled per learning iteration. For
the neural appearance field, we train the neural appearance field at
$256\times 256$ resolution. We train our semantic field and appearance field
on 1 NVIDIA RTX 3090 with 24GB of memory. For the semantic field, each model
is trained for 48k iterations with batch size 1, which takes approximately 13
hours. For the appearance field, each model is trained for 12k iterations with
batch size 4, which takes approximately 3 hours. We use a combination of GAN
loss, L2 loss, and perceptual loss for the appearance field. Their weights are
1.0, 10.0, and 10.0, respectively. More training details are described in the
supplementary material.

###  4.3 Metrics

Following Gancraft [21], we use both quantitative and qualitative metrics to
evaluate our synthesis methods.

#### Quantitative metrics.

We adopt the widely used metrics, FID [23, 53] and KID [3, 53], to measure the
distance between the real and generated distributions. Our experiments are
conducted on 6 test scenes. A collection of landscape images from Flickr is
obtained to evaluate the quality of our generated images. We make sure that
test images are not presented in the LHQ dataset for training. The input
semantic mask is obtained by a pretrained semantic segmentation model [9]. For
each test scene, we render 330 images using a randomly sampled style code from
uniformly sampled camera poses. For both FID and KID metrics, lower values
indicate better image quality.

#### Qualitative metrics.

To quantify the aspects that are not addressed by automatic evaluation
metrics, we conduct a user study to compare our method with the baseline
methods. Specifically, we consider two aspects for human observers: 1) view
consistency and 2) photo-realism. 18 visual designers in a 3D content
designing company are asked to assign a score on a continuous scale of 1-5 for
each aspect per video, where 1 is worse, and 5 is best. Two video sequences of
the same scene rendered by two different methods are presented at the same
time in random orders. A total of 9 videos per method is rendered for
evaluation for each user. For more details about the user study, please refer
to the supplementary material.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Semantic Mask | Ours | 3DPhoto [63] | AdaMPI [20] |  $\text{GVS}^{\ast}$ [19] | InfiniteNatureZero [36]
Figure 5: Qualitative comparisons on the LHQ dataset. We produce more
realistic rendering compared to all baselines, which are demonstrated by the
supplementary video.

###  4.4 Comparisons with baseline methods

We compare our approach with four strong baselines. Each method evaluates at
$256\times 256$ resolution.

SPADE [52]+3DPhoto [63]. Using layered depth images, 3DPhoto [63] generates
free-viewpoint videos from a single image, and it can be trained on the in-
the-wild dataset. Following [36, 58, 38] that directly use official pretrained
models for evaluation, we also use the official pretrained models for our
evaluation. We combine 3DPhoto and SPADE by synthesizing a color image from
the semantic mask using SPADE, then applying 3DPhoto to generate a novel view
image.

SPADE [52]+AdaMPI [20]. AdaMPI [20] regresses the MPI representation from an
image with a network trained on the in-the-wild dataset. We also use official
pretrained models for our evaluation. This method can also be combined with
SPADE to apply in this setting. We obtain generated images from the given
semantic mask using SPADE and then use AdaMPI to synthesize the multi-plane
images.

SPADE [52]+InfiniteNatureZero [36]. InfiniteNature-Zero is a model that
generates flythrough videos of natural scenes, beginning with a single image.
It is trained on the LHQ dataset. We obtain generated images from the given
semantic mask using SPADE and then use InfiniteNature-Zero to perform
perceptual view generation according to the camera poses used in our test set.

$\text{GVS}^{\ast}$ [19]. We carefully train GVS [19] on the LHQ dataset,
using the same training strategy as AdaMPI [63], which takes approximately 2
days. To ensure its style is consistent with other methods, it is finetuned on
the test scenes. We find the official inpainting network pretrained by AdaMPI
[63] can not deal with large disocclusion regions. To prevent these regions
from affecting GVS, GVS is not supervised in these areas. GVS is trained on 3
NVIDIA TITAN A100 40GB graphics cards with 16 batch sizes.

Fig. 5 shows the results of generated videos of different methods. Directly
combining SPADE with 3DPhoto or AdaMPI does not fully utilize the semantic
layout information. Therefore, they are prone to erroneously inpainting
disocclusion regions. In addition, our approach can render free-viewpoint
videos with a wide range of viewpoints, because it is rather easy for our
semantic inpainting network to synthesize a large disocclusion region and the
appearance information at these regions can be generated via SPADE
effortlessly. As $\text{GVS}^{\ast}$ only produces multiple planes at fixed
depths, it struggles to represent complex scene layouts for natural images.
InfiniteNatureZero [36] is not designed for novel view synthesis and struggles
to produce realistic results for the input camera trajectory is different from
the training. Besides, InfiniteNatureZero and $\text{GVS}^{\ast}$ uses a 2D
CNN to generate or refine RGB images, which do not guarantee the inter-view
consistency. On the contrary, our method constructs a continuous neural field
to fuse appearance information from SPADE, so that more realistic and
consistent results can be obtained.

Methods | Consistency$\uparrow$ | Realism$\uparrow$
---|---|---
SPADE [52]+InfiniteNatureZero [36] | 1.31 | 2.06
SPADE [52]+AdaMPI [20] | 3.63 | 1.75
SPADE [52]+3DPhoto [63] | 3.94 | 2.19
$\text{GVS}^{\ast}$ [19] | 2.63 | 2.13
Ours | 4.11 | 3.13

Table 2: Human preference scores. Our method achieves the highest photo-
realism and multi-view consistency according to human raters.

As shown in Table 1, our approach outperforms existing baselines, achieving
the smallest FID and KID. Furthermore, Table 2 indicates that users prefer our
method and rate our videos the most view-consistent and realistic compared to
others.

###  4.5 Ablation studies

|  |  |
---|---|---|---
Input semantic mask | RGB inpainting | Ours w/o SF | Ours
Figure 6: Qualitative results of ablation studies. “RGB inpainting” denotes
the neural scene representation learned on multi-view images generated by an
image inpainting network. This model fails to produce plausible image contents
in disocclusion regions. “Ours w/o SF” denotes the scene representation
learned on images generated from the (post-inpainting) semantic masks and has
the same geometry as our full method. The inconsistency of semantic masks
causes large changes in RGB images, resulting in degraded rendering quality.
Please see the supplementary video for more visual comparisons.

We conduct ablation studies to demonstrate the importance of each component of
our method on one test scene. Our key idea is decomposing the task of view
synthesis from a semantic mask into two simpler steps, which first generate
multi-view semantic masks and then produce RGB images with SPADE for learning
a neural scene representation. To demonstrate the effectiveness of this idea,
we design a naive baseline where we use a pre-trained RGB inpainting network
to generate multi-view images for recovering the 3D scene. All methods use the
same depth map for a fair comparison. Specifically, We generate RGB images by
SPADE from the given semantic mask and then use a monocular depth estimation
model to predict the depth maps of the generated image. The generated image is
then warped to novel views. To infill disocclusion regions, we apply a
pretrained RGB inpainting network to images at novel views, which output the
final multi-view RGBD images. The pretrained RGB and depth inpainting networks
are the official models from AdaMPI [20]. We abbreviate this baseline as “RGB
inpainting”. As shown in Fig. 6, this model fails to produce photorealistic
results under big viewpoint changes. This is because it is challenging for an
RGB inpainting network trained on single-view datasets to infill such large
missing areas. In contrast, our approach renders high-quality images, which
indicates that semantic masks are easier to inpaint than RGB images.

The second important design is our semantic field fusion module, which
leverages the semantic field to denoise and fuse semantic masks generated by
the inpainting network. To illustrate the necessity of this module, we design
the baseline where the infilled semantic masks are directly fed to SPADE to
produce RGB images for learning the scene representation. For this baseline,
we use the same geometry as our full method from the semantic field. We
abbreviate this baseline as “ours w/o SF”. The result in Fig. 6 indicates that
although this model can synthesize reasonable contents in disocclusion
regions, the rendered images tend to be blurry. The reason is that the
infilled semantic masks are not view-consistent, especially near semantic
edges, resulting in that images generated by SPADE differing significantly
across different viewpoints, which makes us difficult to reconstruct the 3D
scene. Besides, to demonstrate that our generated multi-view semantic masks
are better than those generated by GVS [19], we compare our method with it on
the scene used in ablation studies. Following [55], we utilize the average
Negative Log-Likelihood (NLL) score to measure the quality of generated
semantic masks and the View Semantic Consistency (VSC) score to evaluate the
consistency of generated semantic masks. Tab. 3 shows that the quality and
consistency of our generated semantic masks outperform those of GVS.

Methods | NLL$\downarrow$ | VSC$\downarrow$
---|---|---
Ours | 1.60 | 0.049
GVS | 3.33 | 0.089
Table 3: Comparison with GVS. Our mulit-view semantic masks are more
consistent and have better quality.

##  5 Conclusion

In this work, we expanded the AI-enabled content creation tool by using a
single semantic mask to produce a 3D scene that can be rendered from arbitrary
viewpoints. Our method requires single-view image collections for training,
without the need for multi-view images. To achieve this, we proposed a novel
pipeline that first learns an inpainting network to generate novel views of
the input semantic mask and then optimizes a 3D semantic field to render view-
consistent semantic masks, which are subsequently fed into a 2D generator
SPADE to produce RGB images for learning a neural scene representation.
Experiments demonstrated that our method can produce satisfactory
photorealistic and view-consistent results that significantly outperform
baseline methods. We believe that, like AI painting has changed the field, our
method has great potential to revolutionize 3D content creation for humans.

#### Limitation.

Our neural scene representation is optimized per scene, which takes a lot of
time. Amortized inference models are much more convenient for users to edit
digital content with quick feedback. How to train amortized inference models
for view synthesis of natural scenes from a single semantic mask on the
single-view datasets remains unexplored. This problem is left for future work.

## References

  * [1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [3] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  arXiv preprint arXiv:1801.01401, 2018.
  * [4] Yohann Cabon, Naila Murray, and Martin Humenberger.  Virtual kitti 2.  arXiv preprint arXiv:2001.10773, 2020.
  * [5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218, 2018.
  * [6] Yujun Cai, Yiwei Wang, Yiheng Zhu, Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Chuanxia Zheng, Sijie Yan, Henghui Ding, et al.  A unified 3d human motion synthesis model via conditional variational auto-encoder.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11645–11655, 2021.
  * [7] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  * [8] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799–5809, 2021.
  * [9] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [10] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng.  Single-image depth perception in the wild.  Advances in neural information processing systems, 29, 2016.
  * [11] Yuedong Chen, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  Sem2nerf: Converting single-view semantic masks to neural radiance fields.  arXiv preprint arXiv:2203.10821, 2022.
  * [12] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  Carla: An open urban driving simulator.  In Conference on robot learning, pages 1–16. PMLR, 2017.
  * [13] Thibaud Ehret, Roger Marí, and Gabriele Facciolo.  Nerf, meet differential geometry!  arXiv preprint arXiv:2206.14938, 2022.
  * [14] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  Advances in neural information processing systems, 27, 2014.
  * [15] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao.  Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation.  arXiv preprint arXiv:2203.15224, 2022.
  * [16] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  arXiv preprint arXiv:2002.10099, 2020.
  * [17] Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou.  Neural 3d scene reconstruction with the manhattan-world assumption.  In CVPR, 2022.
  * [18] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang.  Ad-nerf: Audio driven neural radiance fields for talking head synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5784–5794, 2021.
  * [19] Tewodros Amberbir Habtegebrial, V. Jampani, Orazio Gallo, and Didier Stricker.  Generative view synthesis: From single-view semantics to novel-view images.  ArXiv, abs/2008.09106, 2020.
  * [20] Yuxuan Han, Ruicheng Wang, and Jiaolong Yang.  Single-view view synthesis in the wild with learned adaptive multiplane images.  arXiv preprint arXiv:2205.11733, 2022.
  * [21] Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu.  Gancraft: Unsupervised 3d neural rendering of minecraft worlds.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14072–14082, 2021.
  * [22] Peter Hedman and Johannes Kopf.  Instant 3d photography.  ACM Transactions on Graphics (TOG), 37:1 – 12, 2018.
  * [23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [24] Ronghang Hu, Nikhila Ravi, Alexander C Berg, and Deepak Pathak.  Worldsheet: Wrapping the world in a 3d sheet for view synthesis from a single image.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12528–12537, 2021.
  * [25] Hsin-Ping Huang, Hung-Yu Tseng, Hsin-Ying Lee, and Jia-Bin Huang.  Semantic view synthesis.  ArXiv, abs/2008.10598, 2020.
  * [26] Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, and Lin Gao.  Stylizednerf: consistent 3d scene stylization as stylized nerf via 2d-3d mutual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18342–18352, 2022.
  * [27] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [28] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 867–876, 2022.
  * [29] Jaebong Jeong, Janghun Jo, Sunghyun Cho, and Jaesik Park.  3d scene painting via semantic image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2262–2272, 2022.
  * [30] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [31] Vahid Kazemi and Josephine Sullivan.  One millisecond face alignment with an ensemble of regression trees.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1867–1874, 2014.
  * [32] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [33] Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, et al.  One shot 3d photography.  ACM Transactions on Graphics (TOG), 39(4):76–1, 2020.
  * [34] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser.  Panoptic neural fields: A semantic object-aware neural scene representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12871–12881, 2022.
  * [35] Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lee.  Mine: Towards continuous depth mpi with nerf for novel view synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12578–12588, 2021.
  * [36] Zhengqi Li, Qianqian Wang, Noah Snavely, and Angjoo Kanazawa.  Infinitenature-zero: Learning perpetual view generation of natural scenes from single images.  ArXiv, abs/2207.11148, 2022.
  * [37] Haotong Lin, Sida Peng, Zhen Xu, Hujun Bao, and Xiaowei Zhou.  Efficient neural radiance fields with learned depth-guided sampling.  arXiv preprint arXiv:2112.01517, 2021.
  * [38] Andrew Liu, Richard Tucker, V. Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa.  Infinite nature: Perpetual view generation of natural scenes from a single image.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14438–14447, 2021.
  * [39] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal.  Towards faster and stabilized gan training for high-fidelity few-shot image synthesis.  In International Conference on Learning Representations, 2020.
  * [40] Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou.  Semantic-aware implicit neural audio-driven video portrait generation.  arXiv preprint arXiv:2201.07786, 2022.
  * [41] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13492–13502, 2022.
  * [42] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [43] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [44] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger.  Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks.  In Computer Graphics Forum, volume 40, pages 45–59. Wiley Online Library, 2021.
  * [45] Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao.  Snerf: stylized neural implicit representations for 3d scenes.  arXiv preprint arXiv:2207.02363, 2022.
  * [46] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5480–5490, 2022.
  * [47] Michael Niemeyer and Andreas Geiger.  Campari: Camera-aware decomposed generative neural radiance fields.  In 2021 International Conference on 3D Vision (3DV), pages 951–961. IEEE, 2021.
  * [48] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [49] Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu.  3d ken burns effect from a single image.  ACM Transactions on Graphics (TOG), 38:1 – 15, 2019.
  * [50] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589–5599, 2021.
  * [51] Byeongjun Park, Hyojun Go, and Changick Kim.  Bridging implicit and explicit geometric transformations for single-image view synthesis.  ArXiv, abs/2209.07105, 2022.
  * [52] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2332–2341, 2019.
  * [53] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On aliased resizing and surprising subtleties in gan evaluation.  In CVPR, 2022.
  * [54] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [55] Xiaotian Qiao, Gerhard P Hancke, and Rynson WH Lau.  Learning object context for novel-view scene layout generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16990–16999, 2022.
  * [56] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [57] C. Rockwell, David F. Fouhey, and Justin Johnson.  Pixelsynth: Generating a 3d-consistent experience from a single image.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14084–14093, 2021.
  * [58] Robin Rombach, Patrick Esser, and Björn Ommer.  Geometry-free view synthesis: Transformers and no 3d priors.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14336–14346, 2021.
  * [59] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  Advances in Neural Information Processing Systems, 33:20154–20166, 2020.
  * [60] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger.  Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids.  arXiv preprint arXiv:2206.07695, 2022.
  * [61] Michael Seufert.  Fundamental advantages of considering quality of experience distributions over mean opinion scores.  In 2019 Eleventh international conference on quality of multimedia experience (QoMEX), pages 1–6. IEEE, 2019.
  * [62] Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski.  Layered depth images.  In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 231–242, 1998.
  * [63] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang.  3d photography using context-aware layered depth inpainting.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8025–8035, 2020.
  * [64] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [65] Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny.  Aligning latent and image spaces to connect the unconnectable.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14144–14153, 2021.
  * [66] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu.  Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.  arXiv preprint arXiv:2205.15517, 2022.
  * [67] Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, and Jue Wang.  Fenerf: Face editing in neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7672–7682, 2022.
  * [68] Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva.  You only need adversarial supervision for semantic image synthesis.  arXiv preprint arXiv:2012.04781, 2020.
  * [69] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi.  Neural feature fusion fields: 3d distillation of self-supervised 2d image representations.  arXiv preprint arXiv:2209.03494, 2022.
  * [70] Shubham Tulsiani, Richard Tucker, and Noah Snavely.  Layer-structured 3d scene inference via view synthesis.  In ECCV, 2018.
  * [71] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  Clip-nerf: Text-and-image driven manipulation of neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3835–3844, 2022.
  * [72] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  arXiv preprint arXiv:2106.10689, 2021.
  * [73] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [74] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7465–7475, 2020.
  * [75] Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao.  Structure-guided ranking loss for single image depth prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 611–620, 2020.
  * [76] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang.  Sinnerf: Training neural radiance fields on complex scenes from a single image.  arXiv preprint arXiv:2204.00928, 2022.
  * [77] Shunyu Yao, RuiZhe Zhong, Yichao Yan, Guangtao Zhai, and Xiaokang Yang.  Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering.  arXiv preprint arXiv:2201.00791, 2022.
  * [78] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  * [79] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [80] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  arXiv preprint arXiv:2206.00665, 2022.
  * [81] Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely.  Arf: Artistic radiance fields.  In European Conference on Computer Vision, pages 717–733. Springer, 2022.
  * [82] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv preprint arXiv:2010.07492, 2020.
  * [83] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu.  Nerfusion: Fusing radiance fields for large-scale scene reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449–5458, 2022.
  * [84] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.  Differentiable augmentation for data-efficient gan training.  Advances in Neural Information Processing Systems, 33:7559–7570, 2020.
  * [85] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place scene labelling and understanding with implicit scene representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15838–15847, 2021.
  * [86] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  ArXiv, abs/1805.09817, 2018.

##  Appendix A Implementation Details

###  A.1 Network architecture

#### Semantic field.

Fig. 7 shows our network architecture of the semantic field. Our network takes
the 3D point $\mathbf{x}$ as the input and outputs SDF $d$, semantic logits
$\mathbf{s}$. For the SDF field, we follow the official implementation of
ManhattanSDF [17]. Following NeRF [42], we utilize a positional encoding to
map the $\mathbf{x}$ to a higher dimensional feature space:

$\displaystyle\gamma(\mathbf{x})=\left[\gamma_{0}(\mathbf{x}),\gamma_{1}(\mathbf{x}),\ldots,\gamma_{L-1}(\mathbf{x})\right]$
(8)
$\displaystyle\gamma_{j}(\mathbf{x})=\left[\sin\left(2^{j}\pi\mathbf{x}\right),\cos\left(2^{j}\pi\mathbf{x}\right)\right]$

where $L=6$.

#### Appearance field.

Fig. 10 and Fig. 9 illustrate the network architecture of our appearance
field. The network architecture of the feature planes generator follows the
official implementation of FastGAN [39]. The generator takes a 256-length
noise vector as input and outputs a $512\times 512\times 180$ feature map,
which is then split to form three 60-channel feature planes. The 256-length
noise vector is fixed.

#### Semantic inpainting network.

Following [38, 36], we view our semantic inpainting step as an image-to-image
translation task and adopt the SPADE generator [52] as our semantic inpainting
network.

#### Label-conditional discriminator.

Our discriminator is the same as the official implementation of OASIS [68].

#### 2D sky generator.

We directly use the generator of FastGAN [39] to the produce the sky image
plane at $512\times 512$ resolution. The padding mode of our sky image plane
is “reflection”.

###  A.2 Loss function

#### Semantic field.

To optimize the semantic field, we use the following overall loss function:

$\displaystyle\mathcal{L}=$
$\displaystyle\lambda_{0}\mathcal{L_{\text{depth}}}+\lambda_{1}\mathcal{L}_{\text{trans}}+\lambda_{2}\mathcal{L}_{\mathbf{P}}+\lambda_{3}\mathcal{L}_{\text{eikonal
}}$ (9) $\displaystyle+\lambda_{4}\mathcal{L}_{\text{rank
}}+\lambda_{5}\mathcal{L}_{\text{src}},$

where
$\lambda_{0}=0.1,\lambda_{1}=10,\lambda_{2}=1,\lambda_{3}=0.01,\lambda_{4}=0.1,\lambda_{5}=1$.
The $\mathcal{L_{\text{depth}}}$, $\mathcal{L}_{\text{trans}}$ and
$\mathcal{L}_{\mathbf{P}}$ are defined in the main paper.

Following common practice, an Eikonal loss [16] is added to regularize the SDF
field.

$\mathcal{L}_{\text{eikonal}}=\sum_{\mathbf{x}\in\mathcal{X}}\left(\left\|\nabla
f_{\theta}(\mathbf{x})\right\|_{2}-1\right)^{2},$ (10)

where $\mathcal{X}$ denotes a set of near-surface and uniformly sampled points
[78].

To encourage the ordinal relation of rendered depth maps matches the ground-
truth ordinal relation, we adopt ranking loss [10, 75], as shown in Equation
11.

$\displaystyle\mathcal{L}_{\text{rank}}=\begin{cases}\log\left(1+\exp\left(-\ell\left(p_{0}-p_{1}\right)\right)\right),&\ell\neq
0\\\ \left(p_{0}-p_{1}\right)^{2},&\ell=0,\end{cases}$ (11)

where $p_{0}=\mathbf{D}_{(u_{0},v_{0})}$ and
$p_{1}=\mathbf{D}_{(u_{1},v_{1})}$ are the rendered depth values, where
$(u_{0},v_{0})$ and $(u_{1},v_{1})$ denote the locations of the randomly
sampled pixels. $\ell$ is induced by the predicted monocular depth values and
a tolerance threshold $\tau$. $p_{0}^{*}=\mathbf{\hat{D}}_{(u_{0},v_{0})}$,
$p_{1}^{*}=\mathbf{\hat{D}}_{(u_{1},v_{1})}$ are the the predicted monocular
depth values at location $(u_{0},v_{0})$, $(u_{1},v_{1})$:

$\ell=\begin{cases}+1,&p_{0}^{*}-p_{1}^{*}>=\tau\\\
-1,&p_{0}^{*}-p_{1}^{*}<=\tau\\\ 0,&\text{ otherwise }.\end{cases}$ (12)

In the main paper, we use a depth map $\mathbf{\hat{D}_{\text{src}}}$
generated by a monocular depth estimation network [56] to warp the input
semantic mask to novel views. The depth map $\mathbf{\hat{D}_{\text{src}}}$ is
also warped to novel views, yielding warped depth maps
$\mathbf{\hat{D}_{\text{src\\{1...N\\}}}}$ at novel views $\\{1...N\\}$. To
further utilize the geometry information of the depth map
$\mathbf{\hat{D}_{\text{src}}}$, we encourage the rendered depth map
$\mathbf{D_{\text{src\\{1...N\\}}}}$ to equal the depth map
$\mathbf{\hat{D}_{\text{src\\{1...N\\}}}}$ at each novel view $\\{1...N\\}$,
using Equation 13:

$\mathcal{L}_{src}=\sum^{N}_{i=1}|\mathbf{D_{\text{src\\{i\\}}}}-\mathbf{\hat{D}_{\text{src\\{i\\}}}}|.$
(13)

Figure 7: Semantic field. All layers are linear layers with softplus
activations except for the final layer.

#### Appearance field.

To learn the appearance field, we use GAN loss, L2 loss, and perceptual loss.
Their weights are 1.0, 10.0, and 10.0, respectively.

#### Semantic inpainting network.

Cross-entropy loss is used to supervise the semantic inpainting network:

$\mathcal{L}=\sum_{\mathbf{p}\in\mathcal{P}}CE(\mathbf{P}^{*}(\mathbf{p}),\mathbf{\hat{P}}(\mathbf{p})),$
(14)

where $\mathbf{P}^{*}(\mathbf{p})$ is the output of the semantic inpainting
network, $\mathbf{\hat{P}}(\mathbf{p})$ denotes the ground-truth label, and
$\mathcal{P}$ means the set of image pixels.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Semantic Mask | Novel View 1 | Novel View 2 | Semantic Mask | Novel View 1 | Novel View 2
Figure 8: Qualitative results. Our results are view-consistent and
photorealistic, as demonstrated by the supplementary video.

###  A.3 Other details

#### Semantic field.

We use the sampling algorithm proposed in VolSDF [78] and choose Adam [32] as
our optimizer, with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and a learning rate of
$5\times 10^{-4}$.

#### Appearance field.

We adopt Adam [32] as our optimizer of generator, with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and a learning rate of $5\times 10^{-4}$. For
discriminator, We also adopt Adam [32], with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and a learning rate of $4\times 10^{-4}$. To prevent the
discriminator from overfitting the training data, the differentiable
augmentation technique [84] is utilized for the training of the discriminator
and the generator.

#### SPADE.

We adopt the official implementation and training procedure of SPADE [52] and
train it on the LHQ [65] dataset. Following common practice, the DeepLabv2
model [9] pre-trained on the COCO-Stuff [5] is adopted to obtain a semantic
mask for each image in the LHQ dataset.

#### Semantic inpainting network.

We train the semantic inpainting network on the LHQ [65] dataset. The semantic
inpainting network is trained on 2 NVIDIA RTX 3090 at $128\times 128$
resolution with batch size 32. The network is trained for a total of 60K
iterations, which takes approximately 2 days. For test time, the input
semantic masks are downsampled to the $128\times 128$ resolution and the
output semantic masks are upsampled to $256\times 256$ resolution.

Figure 9: The overview of the appearance field. The feature retrieved from the
tri-plane generator is converted to RGB value by an MLP with ReLU activations
except for the final layer. Figure 10: The tri-plane generator. The orange
boxes denote feature maps with spatial size. The brown arrows denote an up-
sampling module. The detailed architecture of skip-layer excitation and the
channel number of each layer can be found in FastGAN [39]. The green arrows
represent a $3\times 3$ 2D convolutions layer, which converts the intermediate
feature maps to 45-channel feature maps. The tri-plane feature map is formed
by splitting and concatenating multi-resolution these 45-channel feature maps.

##  Appendix B Experiment Details

###  B.1 Evaluation settings

To ensure the diversity of our test scenes, our test set contains diverse
classes: sky, sea, mountain, hill, sand, rock, waterfall, river, tree, snow,
grass.

We preprocess the depth maps used in our experiments via the code provided by
3DPhoto [63]. All methods use the same depth maps and style codes for a fair
comparison. The camera pose for evaluation is uniformly sampled from a unit
cube.

###  B.2 Discussion of user study

We conducted the user study following the settings of [6, 41, 77] and used the
metric of widely-used Mean Option Score (MOS) ranging from 1-5 [61] to
evaluate our algorithm. Users who participated in the study are designers who
are familiar with 3D content.Prior to the study, we give each participant a
brief and one-to-one introduction to the concept of view consistency and
photo-realism (being realistic as captured by real cameras). During the study
session, we found almost 100% users preferred our method’s result, showing our
proposed method’s superior performance. The ratings they provide further
illustrate the degree of superiority. Although the 3DPhoto [63] and AdaMPI
[20] guarantee view consistency theoretically, they tend to generate
unrealistic content at the views far from the center view, leading to the fact
that users are more likely to lower their view-consistency scores. Our method
produces realistic and consistently high-quality content with fewer artifacts.

###  B.3 More details of ablation studies

The quantitative results of our ablation studies are shown in Table 4. We
render 240 images for evaluation. Because the ablation studies are only
conducted on a single test set, the generated distribution is really different
from the real distribution, thus causing our full model to obtain higher FID
and KID than the quantitative results in the main paper.

Methods | FID$\downarrow$ | KID$\downarrow$
---|---|---
RGB inpainting | 260.46 | 0.279
Ours w/o SF | 250.70 | 0.236
Ours | 166.94 | 0.124
Table 4: Quantitative results of our ablations. “RGB inpainting” denotes the
neural scene representation learned on multi-view images generated by an image
inpainting network. “Ours w/o SF” denotes the scene representation learned on
images generated from the post-inpainting semantic masks. Our full model
outperforms all ablated versions significantly, which demonstrates the
effectiveness of our proposed framework.

##  Appendix C Visualization

|  |  |
---|---|---|---
|  |  |
Figure 11: The results of $\text{SPADE}^{\ast}$. $\text{SPADE}^{\ast}$ denotes
the pipeline which directly translates multi-view consistent semantic masks to
RGB images via SPADE. Although the multi-view semantic masks are view
consistent (first row), the generated RGB images via SPADE (second row) are
inconsistent and have flickering artifacts between different views.

Directly translating multi-view semantic masks obtained by our semantic field
to RGB images through SPADE [52] fails to produce multi-view consistent
results, as shown in Fig. 11. We abbreviate this pipeline as
“$\text{SPADE}^{\ast}$”. Fig. 8 shows several cases of test scenes. For more
visualization results, we recommend watching the supplemental video.

##  Appendix D Results on Other Kinds of Scenes

Our method is general and can extend to other kinds of scenes, including
indoor scenes, as shown in Fig. 12. In the main paper, we focus on natural
scenes, as they are well-suited to our objectives. We will add more results on
other types of scenes.

|  |
---|---|---
Input mask | View 1 | View 2
Figure 12: Results on indoor scenes.

##  Appendix E Discussion of Our Method

Despite the seemingly long pipeline, our high-level idea is clear and simple,
which is to divide a difficult task into two simpler steps: first predicting
multi-view semantic masks as intermediate representations and then translating
semantic masks to RGB images. The additional modules of learning neural fields
are just used for ensuring multi-view consistency.