# Objects With Lighting: A Real-World Dataset for Evaluating
Reconstruction and Rendering for Object Relighting

Benjamin Ummenhofer1,*
Sanskar Agrawal2 Rene Sepulveda1 Yixing Lao3 Kai Zhang4 Tianhang Cheng5
Stephan Richter1 Shenlong Wang5 German Ros6,†
1Intel Labs, 2Preimage, 3The University of Hong Kong, 4Adobe,
5University of Illinois Urbana-Champaign, 6NVIDIA

###### Abstract

Reconstructing an object from photos and placing it virtually in a new
environment goes beyond the standard novel view synthesis task as the
appearance of the object has to not only adapt to the novel viewpoint but also
to the new lighting conditions and yet evaluations of inverse rendering
methods rely on novel view synthesis data or simplistic synthetic datasets for
quantitative analysis. This work presents a real-world dataset for measuring
the reconstruction and rendering of objects for relighting. To this end, we
capture the environment lighting and ground truth images of the same objects
in multiple environments allowing to reconstruct the objects from images taken
in one environment and quantify the quality of the rendered views for the
unseen lighting environments. Further, we introduce a simple baseline composed
of off-the-shelf methods and test several state-of-the-art methods on the
relighting task and show that novel view synthesis is not a reliable proxy to
measure performance. Code and dataset are available at
https://github.com/isl-org/objects-with-lighting .

††*Corresponding author email<EMAIL_ADDRESS>done while
working at Intel Labs.

##  1 Introduction

Inverse rendering, the task of recovering an object’s shape and material
appearance from a collection of images, has been a longstanding goal in the
fields of computer vision and graphics [4, 75]. Its potential applications
span across diverse fields, such as photorealistic rendering [79, 11], video
editing [26], simulation [64, 14] and mixed reality [36, 77]. However, inverse
rendering poses significant challenges due to its ill-posed nature. The color
of a pixel depends on a wide range of factors — often more numerous than the
pixels observed — including but not limited to surface material, illumination,
viewing angle, and surface geometry.

Numerous methods have been proposed to tackle this problem, ranging from data-
driven learning-based methods [48, 53, 37, 46, 77] to model-driven,
optimization-based approaches [3, 79, 11, 29, 65, 15]. These have shown
promising results in creating digital replicas of objects and rendering them
from novel viewpoints and under different lighting conditions. Although
qualitative comparisons among these methods are commonly made, the
establishment of a comprehensive, unbiased, quantitative benchmark for
evaluating inverse graphics approaches remains an elusive goal. This is
largely because measuring "ground truth" surface reflective materials and
illumination in the wild is notoriously difficult. Existing methods often
resort to synthetic data [80, 45] or controlled lab settings [50, 31, 66],
which inevitably restrict the applicability of these findings to real-world
inverse graphics due to the domain shift. Merely evaluating novel view
synthesis is inadequate since one can effectively render an object from a
novel viewpoint without comprehending its materials and illumination. Instead,
it becomes crucial to determine how effectively a relightable digital replica
can generate realistic, high-fidelity images under diverse and novel lighting
conditions. Despite this need, to the best of our knowledge, a well-
calibrated, real-world, multi-illumination dataset enabling quantitative
evaluation of object relighting remains absent.

Motivated by these challenges, we present a novel, naturalistic, real-world
dataset and benchmark for object reconstruction, rendering, and relighting.
This dataset includes ground-truth environmental lighting and posed image
observations for each object, captured across various natural illumination
environments and viewpoints. All images have been geometrically and
photometrically calibrated, facilitating the reconstruction of an object’s
shape, material, and illumination from a single lighting environment, the
object’s relighting under unseen lighting conditions, and the measurement of
realism and fidelity against reference image observations. The dataset is
characterized by a broad diversity of objects and illumination environments,
spanning from non-reflective to reflective surfaces and from indoor man-made
environments to outdoor natural settings. To standardize evaluation, we will
open-source both the training and validation data and the evaluation devkit,
facilitating the development and holistic evaluation of more effective inverse
graphics algorithms.

We conduct a thorough evaluation of recent methods using our new dataset and
existing real and synthetic datasets and provide an analysis of common failure
modes. We also explore the effectiveness and limitations of existing datasets
and discuss potential improvements. In addition, we put forth a simple
baseline integrating a readily available NeuS-based reconstruction [68] with
an off-the-shelf Mitsuba-based differentiable inverse renderer [30].
Interestingly, this straightforward approach performs favorably when compared
to other methods, including the state-of-the-art relightable neural radiance
[32, 79] and neural surface approaches [27, 80], across all benchmarks. We
conduct several discussions based on our findings with the hope to inspire
future research.

In summary, our contributions are as follows. i) We introduce a novel real-
world dataset and benchmark for inverse rendering, captured under natural
illumination. Our benchmark enables quantitative evaluation on relighting,
unlike existing datasets that are either synthetic, qualitative, or confined
to lab environments. ii) We propose a straightforward baseline that
outperforms existing state-of-the-art methods in relighting and reconstruction
across all benchmarks. iii) We provide a systematic evaluation of current
state-of-the-art inverse rendering methods and deliver a comprehensive
analysis. Our experiments underline the significance of geometry estimation
and the use of realistic shading.

##  2 Related work

Material appearance modeling. The Bidirectional Reflectance Distribution
Function (BRDF) defines how light is reflected at a surface point, a concept
that has been widely used in computer graphics and computer vision to
represent surface appearance [52]. Many analytical models were proposed, but
they often lacked strict physical underpinnings [8, 55, 69]. This changed with
the advent of physically-based reflectance models [17, 13, 35, 40, 24], which
incorporated physical principles such as energy conservation, leading to their
widespread use in rendering engines [30, 16, 12] and inverse rendering methods
[60, 79, 29]. However, no analytical model can handle all real-world material.

To surpass analytical BRDF limitations, researchers turned to data-driven
representations [48, 53, 37, 46, 77], with various parameterizations. Classic
representations use dimensionality reduction on lab-measured BRDFs [48]. Other
low-dimensional parametric representations include mixtures of spherical
harmonics [70, 22, 74] and spherical Gaussians [77]. More recently, deep
networks have been used for reflectance modeling [79, 11, 29, 65, 15], shown
to be more capable than low-dimensional parametric models.

Inverse rendering. BRDF estimation from images is inherently challenging due
to the entanglement of scene lighting and material appearance. Researchers
have sought to address this through additional constraints on the capturing
setup [54, 48, 2, 9, 41, 59, 26, 5, 63]. Coded light [60] and flashlights have
provided valuable cues for estimating specularity [51, 58, 6, 5, 7, 76, 38]. A
number of recent works have sought to estimate BRDFs outside of lab
environments by restricting the geometry of surfaces to planes and leveraging
the built-in flash of casual cameras [1, 18, 28, 44, 58, 19]. Another line of
work has recorded videos of rotating objects to estimate reflectance under
natural illumination [20, 71].

Relightable neural radiance field methods, provide another approach to BRDF
estimation under natural illumination [79, 11, 77, 10, 80, 32, 39]. These
methods factorize observed view-conditioned radiance into components
corresponding to illumination and materials, offering excellent rendering and
relighting quality and promising inverse graphics capabilities under natural
environments.

Inverse rendering benchmarks. Obtaining large-scale, ground-truth data for
surface reflectance materials and global illumination in natural environments
has thus far proven to be difficult, if not impossible. As a result, many
researchers have focused their efforts on qualitative comparisons on surface
material estimation or relighting [3], which unfortunately hinders fair,
quantitative comparisons. An alternative approach involves the direct
measurement of surface reflectance within a lab environment [48, 21, 25, 62,
42]. While these techniques yield accurate BRDF measurements, they often carry
limitations related to geometry (such as planar objects), illumination (such
as point light), or scene types (usually fixed).

As photorealistic physics-based rendering has become accessible, synthetic
datasets for inverse rendering have surfaced [43, 56, 80, 45, 73]. These
datasets and benchmarks can cover various objects and scenes with ground truth
across all attributes. However, assumptions about materials, illumination, and
shading must be made during rendering. Furthermore, these datasets still face
challenges bridging the simulation-to-reality gap, thus rendering their
efficacy in assessing real-world performance questionable.

Researchers have also explored benchmarks that assess downstream tasks like
novel view synthesis, relighting, and insertion rendering, which indirectly
gauge the quality of inverse rendering. Existing relighting datasets, however,
are typically either scene-level (for instance, time-lapses of street scenes)
[57] or dependent on simple illumination assumptions (such as point light)
[66]. Moreover, some lack proper photometric calibration [10], which hinders
accurate quantitative measurement. In contrast, our dataset provides a novel
benchmark, captured in various real-world environments across multiple
viewpoints and illuminations, and offers precise geometric and photometric
calibration.

##  3 Objects with Lighting Dataset

Dataset Type Multi-view #Scenes Real data Lighting #Envmaps MIIW [50] Room
(Indoor) No >1000 Yes Limited (Flash) - DiLiGenT [62] Objects (Lab) No 10 Yes
Limited (Point light) - DiLiGenT-MV [42] Objects (Lab) Yes 5 Yes Limited
(Point light) - DTU [31] Objects (Lab) Yes 128 Yes Limited (LED w/ 7 patterns)
- Nerf-OSR [57] Buildings (Outdoor) Yes 8 Yes Environment map 44 ReNe [66]
Objects (Lab) Yes 20 Yes Limited (Point light) - Glossy-Blender [45] Glossy
objects (In- & Outdoor) Yes 8 No Environment map 6 Synthetic4Relight [80]
Objects (In- & Outdoor) Yes 4 No Environment map 3 Ours Objects (In- &
Outdoor) Yes 8 Yes Environment map 72

Table 1: Related datasets featuring multiple illuminations. Only our dataset
enables multiview reconstruction of real-world objects, relighting with
measured complex environment maps, and ground truth data for quantitative
evaluation.

Figure 1: Overview of the Objects With Lighting dataset. First row: Our
dataset contains 8 objects which we capture in 3 different lighting conditions
each. Second and third row: We show 2 HDR environment map examples for each
lighting category which are (left to right): indoor with natural light, indoor
with artificial light, and outdoor.

We present the Objects With Lighting dataset 111The Objects With Lighting
dataset is released under the CDLA-permissive 2.0 license., an open dataset
specifically designed around faithful relighting of real-world objects in new
environments. The Objects With Lighting dataset is more suited for this task
than other related datasets shown in Table 1. Current datasets present
limitations: they are either synthetic, provide only simplified lighting
conditions, or just do not suit the problem, as they lack multiple views or
involve specialized gear.

Our dataset features eight objects, each captured in three distinct
environments with intricate lighting. To ensure varied lighting conditions, we
have chosen these environments to include outdoor, indoor, and indoor settings
with artificial light. Fig. 1 shows surface reconstructions of these objects
and a subset of the environments. Images and environment maps have been
geometrically calibrated and aligned to a common color space to enable
quantitative evaluation of rendered relighted objects against the ground truth
images. In each environment, we have 3 test images, each with its unique
environment map. This map is designed to account for the specific camera that
captured the ground truth image. For evaluation, we use expert annotated masks
to select the object pixels in the ground truth images. For geometric
calibration, we combine natural features using SIFT descriptors, fiducial
markers, and expert keypoint annotation to ensure high-quality camera poses.
To capture the environment lighting we create unclipped HDRI using a Theta Z1
360-degree camera. For the input images and ground truth, we capture images at
high resolution using a Canon EOS 90D DSLR camera. We use RAW format for all
images and implement our own image processing pipeline to control all
transformations of the source material. For color calibration, we use a
24-color chart and transform the images from both cameras to a common color
space. For more details on the calibration process, please refer to Section E
in the appendix.

Figure 2: We take 42-67 photos for each object and roughly sample a hemisphere
with the object in the center.

A key motivation behind our dataset is to assist with the material
reconstruction tasks. To this end, we seek to facilitate the surface
reconstruction task through a comprehensive object coverage, by sampling
images on a hemisphere, as depicted in Fig. 2 . We select objects made from
diverse materials like plastic, wood, metal, porcelain, rubber, and fruit
skin. These materials are not uniform but spatially vary, posing a challenge
to reconstruction methods. In addition to the core data files comprised of
posed images, we provide additional resources for convenience. These include
approximate foreground masks for input images, and NeuS [68] generated meshes,
which can be used to simplify or skip shape reconstruction for methods
focusing on material estimation or fine-tuning. We leverage this additional
data to compare multiple approaches in our experiments.

##  4 A simple baseline

Our baseline method serves as a check to see if we can solve the problem with
existing easy to use building blocks. To keep the method simple, we choose a
sequential approach, which uses NeuS [68] to reconstruct the shape of the
objects first. NeuS optimizes a function $f(\mathbf{x})$ that maps a 3D point
$\mathbf{x}$ to a scalar signed distance value and a function
$c(\mathbf{x},\mathbf{v})$ which maps a position $\mathbf{x}$ and direction
$\mathbf{v}$ to a color. Both functions are implemented as MLPs and
differentiable volume rendering is used to optimize the network parameters.
After optimization the surface $S$, can be extracted from $f$ using the
marching cubes algorithm [47]. We refer to [68] for a detailed description of
the algorithm.

In a second step we compute the material parameters via differentiable
rendering with the path tracer Mitsuba [30]. The loss we optimize is

$\displaystyle\mathcal{L}=$
$\displaystyle\sum_{i}^{N}\int\|I_{i}-\hat{I}_{i}\|\mathrm{d}\mathbf{x}+\alpha_{1}\int\|\nabla\theta_{\text{env}}\|\mathrm{d}\mathbf{x}$
(1)
$\displaystyle+\alpha_{2}\left(\int\|\nabla\theta_{\text{albedo}}\|+\|\nabla\theta_{\text{rough}}\|+\|\nabla\theta_{\text{metal}}\|\mathrm{d}\mathbf{x}\right).$

We minimize the difference between the input images $I_{i}(\mathbf{x})$ and
the rendered images $\hat{I}_{i}(\mathbf{x})$ for all pixels $\mathbf{x}$.
$\hat{I}_{i}$ is the rendered image that depends on the mesh $S$ obtained from
NeuS, the camera parameters, the material parameters
$\theta=\\{\theta_{\text{albedo}},\theta_{\text{rough}},\theta_{\text{metal}}\\}$
of the principled BRDF [13], and the environment map $\theta_{\text{env}}$. We
keep all other parameters of the BRDF model constant at their default values.
We store the material parameters and environment map as 2D textures to allow
them to spatially vary over the surface and sphere and regularize them with a
simple total variation regularization weighted with
$\alpha_{1}=\alpha_{2}=0.01$.

The rendering process that we use is governed by

$L_{o}(\bm{\omega}_{o})=\int_{\Omega}L_{i}(\bm{\omega}_{i})\,f(\bm{\omega}_{i},\bm{\omega}_{o},\theta)\,\langle\mathbf{n}\,,\bm{\omega}_{i}\rangle\;\mathrm{d}\bm{\omega}_{i},$
(2)

which is known as the rendering equation [33] and computed using Monte Carlo
integration over the hemisphere $\Omega$ for a point on the surface with
normal $\mathbf{n}$. $L_{i}$ and $L_{o}$ are the ingoing and outgoing radiance
along the directions $\bm{\omega}_{i},\bm{\omega}_{o}$. The parameters we want
to optimize are part of the BRDF $f$ and $L_{i}$ which is directly and
indirectly affected by $\theta_{\text{env}}$. We refer to [30] for details on
how the rendering equation can be implemented and its parameters optimized
efficiently.

##  5 Experiments

We conduct experiments on our and related datasets with our baseline method
and recent state-of-the-art methods. We present quantitative evaluations for
relighting in comparison with a synthetic relighting dataset and discuss
failure modes. Following common practices we measure novel view synthesis
performance on multiple real and synthetic datasets too and check for
correlations with the more challenging relighting task.

###  5.1 Datasets

We use four datasets for our experiments. (i) Our Objects With Lighting
dataset introduced in Section 3. (ii) The Synthetic4Relight dataset from [80],
which features synthetic images and ground truth for relighting experiments.
(iii) A subset of DTU [31] which is a real-world dataset for evaluating Multi-
View Stereo methods. (iv) A subset of the synthetic BlendedMVS (BMVS) dataset
[72].We use DTU and BMVS as additional datasets for evaluating novel view
synthesis performance.

###  5.2 Test protocol

Relighting The test protocol for relighting is visualized in Fig. 3. For each
method, we reconstruct the object generating a method-specific representation
from a set of images taken in the reconstruction environment. After
reconstruction, we render the object under new lighting using an environment
map and new viewpoints that correspond to the ground truth test images.
Directly testing the rendered images allows us to evaluate all methods
uniformly irrespective of the used representations. When testing, one of the
environment maps corresponds to the reconstruction environment but is not
available during the reconstruction process. Results for this environment are
listed as _Same environment_ in Tables 1 and 3.

To account for the scalar ambiguity between illumination and albedo of the
surface we postprocess the rendered linear images and compute the optimal
exposure for minimizing the MSE between the rendered tone mapped image and the
ground truth image for each channel. For tone mapping and conversion to 8-bit
images we use $y=255(2^{\text{EV}}x)^{\gamma}$. Values $x$ are linear values
and $y$ are the tone mapped 8-bit values. We round all values to the nearest
integer and clip values outside the 8-bit range. $\gamma=\frac{1}{2.2}$ for
all experiments. To measure performance we use the standard metrics PSNR,
SSIM, and LPIPS [78] to compare images. We compute all metrics per-pixel and
exclude background pixels from the averages using ground truth masks to avoid
very large or very small values for the respective metrics. We use the same
evaluation code for our dataset and for Synthetic4Relight.

Novel view synthesis For novel view synthesis we use the lighting estimated by
each method instead of the measured environment maps for rendering. Another
difference to relighting is the tone mapping. Since we use the estimated
lighting we do not have an unknown scalar factor that relates the material and
the illumination. Thus, we skip any post-processing and compare tone mapped
images as generated by each method. We compute the metrics with the same code
as for the relighting experiments.

Figure 3: Test protocol on the _Objects With Lighting_ dataset. We reconstruct
a relightable representation of the object from a set of images taken in one
environment. The reconstruction can be a textured mesh, a neural
representation, or any other representation that can be relit with a new
environment map and rendered from a novel view point. Testing is conducted by
rendering the objects using environment maps from the reconstruction
environment and new environments and comparing the rendered images to the
corresponding test images of that environment.

###  5.3 Methods

We evaluate several state-of-the-art methods and our baseline described in
Section 4, which we name _Mitsuba+NeuS_. Common to all methods is the use of
differential rendering to optimize shape and appearance. Methods differ in the
way they represent shape, materials, and lighting.

NVDiffrec [49] and NVDiffrecMC [27] use tetrahedral grids to represent shapes
during reconstruction. The remaining methods use implicit neural
representations to describe the surface. Materials represented with analytical
BRDF models is used by our baseline, NVDiffrec, NVDiffrecMC, PhySG [77], NeRD
[10] and NeROIC [39]. InvRender [76] and NeuralPIL [11] make use of analytical
BRDFs but combine them with neural representations to enforce sparsity and
improve convergence. TensoIR [32] combines neural rendering and physically-
based rendering and uses joint optimization. For novel view synthesis
experiments we use the neural rendering capabilities while we use the
physically-based rendering for the relighting experiments. NeRFactor [79] uses
albedo and a latent BRDF code trained on a material database to represent
materials. Illuminations is represented as equirectangular environment map in
NVDiffrec, NVDiffrecMC, TensoIR, NeRFactor, and our baseline. InvRender,
PhySG, and NeRD use spherical Gaussians. NeROIC makes use of spherical
harmonics and Neural-PIL uses a neural representation to store preintegrated
illumination.

###  5.4 Relighting

| Same environment | New environment | Mean
---|---|---|---
Method |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
Mitsuba+NeuS | $26.20$ | $0.86$ | $0.205$ | $26.24$ | $0.84$ | $0.227$ | $26.22$ | $0.84$ | $0.219$
InvRender11footnotemark: 1 | $24.42$ | $0.78$ | $0.360$ | $23.45$ | $0.77$ | $0.374$ | $23.77$ | $0.78$ | $0.369$
NeRD11footnotemark: 1 | $22.68$ | $0.63$ | $0.554$ | $21.71$ | $0.65$ | $0.540$ | $22.04$ | $0.64$ | $0.544$
NeRFactor11footnotemark: 1 | $22.15$ | $0.75$ | $0.452$ | $20.62$ | $0.72$ | $0.486$ | $21.13$ | $0.73$ | $0.475$
NeRFactor+NeuS | $25.22$ | $0.82$ | $0.393$ | $25.82$ | $0.81$ | $0.411$ | $25.62$ | $0.81$ | $0.405$
NeROIC11footnotemark: 1 | $22.81$ | $0.82$ | $0.305$ | $21.59$ | $0.78$ | $0.323$ | $22.00$ | $0.79$ | $0.317$
Neural-PIL11footnotemark: 1 | $19.88$ | $0.50$ | $0.602$ | $19.56$ | $0.51$ | $0.604$ | $19.67$ | $0.51$ | $0.604$
NVDiffrec11footnotemark: 1 | $24.45$ | $0.79$ | $0.332$ | $22.60$ | $0.72$ | $0.406$ | $23.21$ | $0.74$ | $0.381$
NVDiffrec+NeuS | $24.47$ | $0.82$ | $0.286$ | $22.78$ | $0.74$ | $0.373$ | $23.34$ | $0.77$ | $0.344$
NVDiffrecMC11footnotemark: 1 | $18.99$ | $0.74$ | $0.379$ | $20.24$ | $0.73$ | $0.393$ | $19.82$ | $0.73$ | $0.389$
NVDiffrecMC+NeuS | $19.14$ | $0.76$ | $0.346$ | $20.32$ | $0.74$ | $0.371$ | $19.92$ | $0.75$ | $0.362$
PhySG11footnotemark: 1 | $23.49$ | $0.82$ | $0.360$ | $22.77$ | $0.82$ | $0.375$ | $23.01$ | $0.82$ | $0.370$
TensoIR11footnotemark: 1 | $25.60$ | $0.80$ | $0.348$ | $24.15$ | $0.77$ | $0.378$ | $24.64$ | $0.78$ | $0.368$
Table 2: Relighting results on our _Objects With Lighting_ dataset. 11footnotemark: 1Uses foreground segmentation masks. | Same environment | New environment | Mean
---|---|---|---
Method |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
Mitsuba+NeuS | $29.02$ | $0.88$ | $0.106$ | $30.47$ | $0.88$ | $0.108$ | $29.99$ | $0.88$ | $0.107$
InvRender | $22.08$ | $0.76$ | $0.297$ | $21.15$ | $0.77$ | $0.279$ | $21.46$ | $0.77$ | $0.285$
NeRD | $22.15$ | $0.61$ | $0.424$ | $22.95$ | $0.65$ | $0.356$ | $22.68$ | $0.64$ | $0.378$
NeRFactor | $23.30$ | $0.69$ | $0.367$ | $21.60$ | $0.62$ | $0.341$ | $22.17$ | $0.65$ | $0.350$
NeROIC | $19.07$ | $0.68$ | $0.358$ | $18.35$ | $0.66$ | $0.338$ | $18.59$ | $0.66$ | $0.345$
Neural-PIL | $16.68$ | $0.45$ | $0.480$ | $17.37$ | $0.47$ | $0.437$ | $17.14$ | $0.47$ | $0.452$
NVDiffrec | $22.60$ | $0.83$ | $0.206$ | $21.03$ | $0.75$ | $0.228$ | $21.55$ | $0.78$ | $0.220$
NVDiffrecMC+NeuS | $26.80$ | $0.81$ | $0.185$ | $27.65$ | $0.82$ | $0.177$ | $27.36$ | $0.81$ | $0.180$
PhySG | $21.03$ | $0.72$ | $0.383$ | $21.46$ | $0.72$ | $0.349$ | $21.32$ | $0.72$ | $0.360$
TensoIR | $26.17$ | $0.80$ | $0.246$ | $27.27$ | $0.82$ | $0.202$ | $26.90$ | $0.81$ | $0.217$
Table 3: Relighting results on the Synthetic4Relight dataset.

We evaluate all methods on our _Objects With Lighting_ dataset and report
results in Table 1. We found that it is challenging for many methods to
compute a good shape representation on our and other datasets, which can
result in failures producing very large errors. To alleviate this we allow the
usage of approximate foreground masks or inject the meshes generated with
NeuS, which we denote with the suffix _+NeuS_. On the Synthetic4Relight
dataset we use foreground masks for all methods and show results in Table 3.

Results show that relighting with illumination from new environments is more
difficult than relighting with the environment map from the reconstruction
environment showing the importance of capturing objects in additional
environments.

Despite being a simple composition of existing methods our baseline
Mitsuba+NeuS performs well on our dataset and the Synthetic4Relight dataset.
On the synthetic dataset we observe very high absolute values for PSNR and
SSIM as well as a bigger lead to other methods. It is striking that
NVDiffrecMC+NeuS achieves a very high PSNR on the synthetic data too. Mitsuba
and NVDiffrecMC use both Monte Carlo ray tracing and the same holds for the
Cycles renderer, which was used to generate the synthetic data. Using similar
rendering techniques and BRDF representations may produce unwanted resonance
between the data and the methods. This may lead to overly optimistic results
and is a problem that can only be resolved by using real-world data.

Our dataset helps us to identify failure modes. The most obvious failures are
caused by bad shape reconstruction. Real-world data is more challenging to
reconstruct even if foreground masks are available. We can see this especially
for NeRD, NeRFactor, and NVDiffrec in Fig. 4 which fail for the apple and
porcelain mug objects. While capturing the concave shape of the mug is
challenging for all methods, NeRFactor, NeROIC, Neural-PIL, NVDiffrec, PhySG,
and TensoIR miss to reconstruct the inside completely.

Another deficiency we can observe is the overestimation of the glossyness of
objects. This can be observed well for our baseline, InvRender, and
NVDiffrecMC+NeuS in the highlights on the apple but is also visible for the
antman object, which shows highlights not visible in the ground truth. It can
also be seen for the gamepad object for NVDiffrecMC+NeuS and InvRender (see
Fig. 13 and Fig. 14 in the appendix for qualitative examples of all objects).
We did not observe these failures in the Synthetic4Relight experiments.

For NeRD and Neural-PIL we observe artifacts on real data. For Neural-PIL we
see similar artifacts on the Synthetic4Relight dataset too but find that the
artifacts are more pronounced on real data.

With respect to the rendering we see that methods that don’t account for
visibility with respect to the illumination have a disadvantage due to missing
shadows. This can be seen best for NVDiffrec+NeuS which misses reproducing the
shadow inside the porcelain mug.

###  5.5 Novel view synthesis

DTU BMVS Synthetic4Relight Objects With Lighting Mean Method PSNR $\uparrow$
SSIM $\uparrow$ LPIPS $\downarrow$ PSNR $\uparrow$ SSIM $\uparrow$ LPIPS
$\downarrow$ PSNR $\uparrow$ SSIM $\uparrow$ LPIPS $\downarrow$ PSNR
$\uparrow$ SSIM $\uparrow$ LPIPS $\downarrow$ PSNR $\uparrow$ SSIM $\uparrow$
LPIPS $\downarrow$ Mitsuba+NeuS $20.49$ $0.65$ $0.241$ $22.18$ $0.69$ $0.225$
$29.64$ $0.88$ $0.091$ $25.08$ $0.87$ $0.186$ $24.35$ $0.77$ $0.186$ InvRender
$17.09$ $0.55$ $0.439$ $18.66$ $0.50$ $0.506$ $22.15$ $0.76$ $0.294$ $24.44$
$0.81$ $0.336$ $20.59$ $0.66$ $0.394$ NeRD $18.85$ $0.49$ $0.434$ $20.17$
$0.46$ $0.489$ $21.37$ $0.62$ $0.408$ $20.52$ $0.66$ $0.535$ $20.23$ $0.56$
$0.467$ NeRFactor $18.47$ $0.54$ $0.412$ $19.77$ $0.49$ $0.481$ $20.39$ $0.69$
$0.363$ $22.99$ $0.78$ $0.434$ $20.41$ $0.63$ $0.423$ NeROIC $18.90$ $0.65$
$0.357$ $20.37$ $0.61$ $0.367$ $20.04$ $0.73$ $0.332$ $20.18$ $0.82$ $0.322$
$19.87$ $0.70$ $0.345$ Neural-PIL $18.92$ $0.48$ $0.428$ $17.57$ $0.34$
$0.646$ $21.98$ $0.64$ $0.392$ $23.85$ $0.68$ $0.523$ $20.58$ $0.54$ $0.497$
NVDiffrec $18.21$ $0.52$ $0.381$ $18.85$ $0.45$ $0.429$ $23.38$ $0.87$ $0.170$
$25.29$ $0.84$ $0.283$ $21.43$ $0.67$ $0.316$ NVDiffrecMC+NeuS $15.81$ $0.42$
$0.531$ $19.20$ $0.45$ $0.454$ $27.97$ $0.84$ $0.165$ $24.73$ $0.86$ $0.255$
$21.93$ $0.64$ $0.351$ TensoIR $20.27$ $0.59$ $0.346$ $22.21$ $0.62$ $0.389$
$27.73$ $0.87$ $0.149$ $26.10$ $0.89$ $0.237$ $24.08$ $0.74$ $0.280$

Table 4: Novel-view synthesis results on multiple real and synthetic datasets.

Novel view synthesis is a related but simpler problem since we can assume that
the illumination is fixed. We use all datasets for evaluation. Note that we
can use test images from the reconstruction environment of our dataset to test
novel view synthesis performance.

We report results in Table 4, which shows that our baseline Mitsuba+NeuS
performs well on this task too. We check for correlations between the
relighting experiments on our dataset and the novel-view performance across
all datasets. We find a significant correlation when ranking the methods using
the LPIPS metric with $r_{s}=0.77$ and $p=0.02$, and the SSIM metric with
$r_{s}=0.91$ and $p=0.002$. For the PSNR metrics, we cannot find a significant
correlation between the tasks. Correlation is expected since we assume that a
method that solves the relighting task perfectly will solve the novel view
synthesis task too. However, not seeing a clear correlation for PSNR and the
small sample size ($N=9$) do not allow for a final conclusion. Additionally,
we know that novel view synthesis does not easily expose problems like baked-
in or missing shadows, something we can observe on our dataset, thus we think
results as shown in Table 4 must be interpreted together with quantitative
relighting experiments.

Figure 4: Qualitative examples for relighting on our dataset. The first row
shows the environment map used for relighting. For each object, we show one of
the test images of the 3 lighting categories. The first image for each object
corresponds to the reconstruction environment. Each object has been captured
in highly different lighting conditions: outdoor, indoor, and indoor with
artificial light.

##  6 Limitations

We use a board with fiducial markers in our dataset to define a common
coordinate frame for all images and environment maps. The board is clearly
visible in all environment maps and violates the assumption that the
environment is a sphere with an infinite radius. Researchers should be able to
bypass this limitation by including the board in the reconstruction. Image
acquisition and camera calibration processes are subject to some level of
noise, which cannot be suppressed completely.

The presented baseline, Mitsuba+NeuS, optimizes geometry, and material and
illumination sequentially. Also, it uses an analytic BRDF, which cannot
represent all possible materials.

##  7 Conclusion & Future work

We present the _Objects With Lighting_ dataset, a new resource designed for
evaluating the reconstruction and relighting of 3D objects. This dataset
enables precise assessment of inverse rendering methods in real-world
scenarios by offering accurate, unclipped HDR images for lighting, and ground
truth images for direct comparison with rendered images. Our dataset uniquely
highlights existing issues with state-of-the-art methods, issues that
synthetic datasets or novel-view synthesis tests may overlook.

The presented dataset, the code involved in the evaluation of all approaches,
and the tools used to build the dataset are released to help the research
community. We intend to continuously expand our dataset and refine the
evaluation methods, enhancing their usefulness and value for the community.

## References

  * Aittala et al. [2018] Miika Aittala, Timo Aila, and Jaakko Lehtinen.  Reflectance modeling by neural texture synthesis.  In _ACM Transactions on Graphics (ToG)_ , 2018.
  * Asselin et al. [2020] Louis-Philippe Asselin, Denis Laurendeau, and Jean-Francois Lalonde.  Deep svbrdf estimation on real materials.  In _2020 International Conference on 3D Vision (3DV)_ , pages 1157–1166. IEEE, 2020.
  * Barron and Malik [2015] Jonathan T Barron and Jitendra Malik.  Shape, illumination, and reflectance from shading.  _IEEE transactions on pattern analysis and machine intelligence_ , 37(8):1670–1687, 2015.
  * Barrow et al. [1978] Harry Barrow, J Tenenbaum, A Hanson, and E Riseman.  Recovering intrinsic scene characteristics.  _Comput. Vis. Syst_ , 2(3-26):2, 1978.
  * Bi et al. [2020a] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi.  Neural Reflectance Fields for Appearance Acquisition.  _arXiv:2008.03824 [cs]_ , 2020a.
  * Bi et al. [2020b] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi.  Deep reflectance volumes: Relightable reconstructions from multi-view photometric images.  In _European Conference on Computer Vision_ , pages 294–311. Springer, 2020b.
  * Bi et al. [2020c] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi.  Deep 3D Capture: Geometry and Reflectance From Sparse Multi-View Images.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020c.
  * Blinn [1977] James F Blinn.  Models of light reflection for computer synthesized pictures.  In _Proceedings of the 4th annual conference on Computer graphics and interactive techniques_ , pages 192–198, 1977.
  * Boss et al. [2018] Mark Boss, Fabian Groh, Sebastian Herholz, and Hendrik P. A. Lensch.  Deep Dual Loss BRDF Parameter Estimation.  In _Workshop on Material Appearance Modeling_ , 2018.
  * Boss et al. [2021a] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch.  NeRD: Neural Reflectance Decomposition from Image Collections.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2021a.
  * Boss et al. [2021b] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P.A. Lensch.  Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021b.
  * Brian [2019] Brian.  freepbr, 2019.  https://freepbr.com.
  * Burley and Studios [2012] Brent Burley and Walt Disney Animation Studios.  Physically-based shading at disney.  In _ACM SIGGRAPH_ , pages 1–7. vol. 2012, 2012.
  * Chen et al. [2021] Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun.  Geosim: Realistic video simulation via geometry-aware composition for self-driving.  In _CVPR_ , 2021.
  * Chen et al. [2020] Zhe Chen, Shohei Nobuhara, and Ko Nishino.  Invertible Neural BRDF for Object Inverse Rendering.  In _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V_ , pages 767–783. Springer, 2020.
  * Community [2018] Blender Online Community.  _Blender - a 3D modelling and rendering package_.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  * Cook and Torrance [1982] Robert L. Cook and Kenneth E. Torrance.  A reflectance model for computer graphics.  _ACM Transactions on Graphics (ToG)_ , 1982.
  * Deschaintre et al. [2018] Valentin Deschaintre, Miika Aitalla, Fredo Durand, George Drettakis, and Adrien Bousseau.  Single-image SVBRDF capture with a rendering-aware deep network.  In _ACM Transactions on Graphics (ToG)_ , 2018.
  * Deschaintre et al. [2020] Valentin Deschaintre, George Drettakis, and Adrien Bousseau.  Guided fine-tuning for large-scale material transfer.  In _Eurographics Symposium on Rendering_ , 2020.
  * Dong et al. [2014] Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong.  Appearance-from-motion: Recovering spatially varying surface reflectance under unknown lighting.  _ACM Transactions on Graphics (TOG)_ , 33(6):1–12, 2014.
  * Dupuy and Jakob [2018] Jonathan Dupuy and Wenzel Jakob.  An adaptive parameterization for efficient material acquisition and rendering.  _ACM Transactions on graphics (TOG)_ , 37(6):1–14, 2018.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _CVPR_ , pages 5491–5500. IEEE, 2022.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets, 2021.
  * Gotanda [2012] Yoshiharu Gotanda.  Beyond a simple physically based blinn-phong model in real-time.  In _SIGGRAPH Physically Based Shading in Theory and Practice course_ , 2012.
  * Grosse et al. [2009] Roger B. Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman.  Ground truth dataset and baseline evaluations for intrinsic image algorithms.  In _IEEE 12th International Conference on Computer Vision, ICCV 2009, Kyoto, Japan, September 27 - October 4, 2009_ , pages 2335–2342. IEEE Computer Society, 2009.
  * Guo et al. [2019] Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, et al.  The relightables: Volumetric performance capture of humans with realistic relighting.  _ACM Transactions on Graphics (ToG)_ , 38(6):1–19, 2019.
  * Hasselgren et al. [2022] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg.  Shape, light, and material decomposition from images using monte carlo rendering and denoising.  In _NeurIPS_ , 2022.
  * Henzler et al. [2021] Philipp Henzler, Valentin Deschaintre, Niloy J Mitra, and Tobias Ritschel.  Generative modelling of BRDF textures from flash images.  _ACM Transactions on Graphics (SIGGRAPH ASIA)_ , 2021.
  * Hu et al. [2020] Bingyang Hu, Jie Guo, Yanjun Chen, Mengtian Li, and Yanwen Guo.  DeepBRDF: A Deep Representation for Manipulating Measured BRDF.  _Computer Graphics Forum_ , 39(2), 2020.
  * Jakob et al. [2022] Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang.  Mitsuba 3 renderer, 2022.  https://mitsuba-renderer.org.
  * Jensen et al. [2014] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 406–413. IEEE, 2014.
  * Jin et al. [2023] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su.  Tensoir: Tensorial inverse rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Kajiya [1986] James T. Kajiya.  The rendering equation.  In _SIGGRAPH_ , pages 143–150, 1986.
  * Kannala and Brandt [2006] Juho Kannala and Sami S. Brandt.  A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 28(8):1335–1340, 2006.
  * Karis [2013] Brian Karis.  Real shading in Unreal Engine 4.  In _SIGGRAPH Physically Based Shading in Theory and Practice course_ , 2013.
  * Karsch et al. [2011] Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem.  Rendering synthetic objects into legacy photographs.  _ACM Transactions on Graphics (TOG)_ , 30(6):1–12, 2011.
  * Kautz et al. [2002] Jan Kautz, John Snyder, and Peter-Pike J Sloan.  Fast arbitrary brdf shading for low-frequency lighting using spherical harmonics.  _Rendering Techniques_ , 2(291-296):1, 2002.
  * Kaya et al. [2021] Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari, and Luc Van Gool.  Uncalibrated neural inverse rendering for photometric stereo of general surfaces.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2021\.
  * Kuang et al. [2022] Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov.  Neroic: Neural rendering of objects from online image collections.  _ACM Trans. Graph._ , 41(4), 2022.
  * Lazarov [2013] Dimitar Lazarov.  Getting more physical in call of duty: Back ops ii.  In _SIGGRAPH Physically Based Shading in Theory and Practice course_ , 2013.
  * Lensch et al. [2003] Hendrik P.A. Lensch, Jochen Lang, M. Sa Asla, and Hans-Peter Seidel.  Planned sampling of spatially varying BRDFs.  In _Computer Graphics Forum_ , 2003.
  * Li et al. [2020] Min Li, Zhenglong Zhou, Zhe Wu, Boxin Shi, Changyu Diao, and Ping Tan.  Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials.  _IEEE Trans. Image Process._ , 29:4159–4173, 2020.
  * Li et al. [2018a] Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, and Stefan Leutenegger.  Interiornet: Mega-scale multi-sensor photo-realistic indoor scenes dataset.  In _British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018_ , page 77. BMVA Press, 2018a.
  * Li et al. [2018b] Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker.  Materials for masses: SVBRDF acquisition with a single mobile phone image.  In _European Conference on Computer Vision (ECCV)_ , 2018b.
  * Liu et al. [2023] Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang.  Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images.  In _SIGGRAPH_ , 2023.
  * Lombardi and Nishino [2015] Stephen Lombardi and Ko Nishino.  Reflectance and illumination recovery in the wild.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(1):129–141, 2015.
  * Lorensen and Cline [1987] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  _ACM siggraph computer graphics_ , 21(4):163–169, 1987.
  * Matusik et al. [2003] Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan.  A data-driven reflectance model.  _SIGGRAPH_ , 22(3):759–769, 2003.
  * Munkberg et al. [2022] Jacob Munkberg, Wenzheng Chen, Jon Hasselgren, Alex Evans, Tianchang Shen, Thomas Müller, Jun Gao, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In _CVPR_ , pages 8270–8280. IEEE, 2022.
  * Murmann et al. [2019] Lukas Murmann, Michaël Gharbi, Miika Aittala, and Frédo Durand.  A dataset of multi-illumination images in the wild.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pages 4079–4088. IEEE, 2019.
  * Nam et al. [2018] Giljoo Nam, Diego Gutierrez, and Min H. Kim.  Practical SVBRDF acquisition of 3d objects with unstructured flash photography.  In _ACM Transactions on Graphics (SIGGRAPH ASIA)_ , 2018.
  * Nicodemus et al. [1977] Fred Edwin Nicodemus, Joseph C Richmond, Jack J Hsia, Irving W Ginsberg, and Thomas Limperis.  Geometrical considerations and nomenclature for reflectance.  Technical report, National Bureau of Standards, 1977.
  * Nishino [2009] Ko Nishino.  Directional statistics brdf model.  In _2009 IEEE 12th International Conference on Computer Vision_ , pages 476–483. IEEE, 2009.
  * Nishino et al. [2001] Ko Nishino, Zhengyou Zhang, and Katsushi Ikeuchi.  Determining reflectance parameters and illumination distribution from a sparse set of images for view-dependent image synthesis.  In _Proceedings Eighth IEEE international conference on computer vision. ICCV 2001_ , pages 599–606. IEEE, 2001.
  * Phong [1975] Bui Tuong Phong.  Illumination for computer generated pictures.  _Communications of the ACM_ , 18(6):311–317, 1975\.
  * Roberts et al. [2021] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind.  Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10912–10922, 2021.
  * Rudnev et al. [2022] Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt.  Nerf for outdoor scene relighting.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Sang and Chandraker [2020] Shen Sang and M. Chandraker.  Single-shot neural relighting and SVBRDF estimation.  In _ECCV_ , 2020.
  * Sato et al. [1997] Yoichi Sato, Mark D Wheeler, and Katsushi Ikeuchi.  Object shape and reflectance modeling from observation.  In _Proceedings of the 24th annual conference on Computer graphics and interactive techniques_ , pages 379–387, 1997.
  * Schmitt et al. [2020] Carolin Schmitt, Simon Donne, Gernot Riegler, Vladlen Koltun, and Andreas Geiger.  On joint estimation of pose, geometry and svbrdf from a handheld scanner.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3493–3503, 2020.
  * Schönberger and Frahm [2016] Johannes L. Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _CVPR_ , pages 4104–4113. IEEE Computer Society, 2016.
  * Shi et al. [2019] Boxin Shi, Zhipeng Mo, Zhe Wu, Dinglong Duan, Sai-Kit Yeung, and Ping Tan.  A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 41(2):271–284, 2019.
  * Srinivasan et al. [2021] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In _CVPR_ , 2021.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al.  The replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Sztrajman et al. [2021] Alejandro Sztrajman, Gilles Rainer, Tobias Ritschel, and Tim Weyrich.  Neural BRDF Representation and Importance Sampling.  _Comput. Graph. Forum_ , 40(6):332–346, 2021\.
  * Toschi et al. [2023] Marco Toschi, Riccardo De Matteo, Riccardo Spezialetti, Daniele De Gregorio, Luigi Di Stefano, and Samuele Salti.  Relight my nerf: A dataset for novel view synthesis and relighting of real world objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 20762–20772, 2023.
  * van der Walt et al. [2014] Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu.  scikit-image: Image processing in python.  _CoRR_ , abs/1407.6245, 2014.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  _NeurIPS_ , 2021.
  * Ward [1992] Gregory J Ward.  Measuring and modeling anisotropic reflection.  In _Proceedings of the 19th annual conference on Computer graphics and interactive techniques_ , pages 265–272, 1992.
  * Wizadwongsa et al. [2021] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Xia et al. [2016] Rui Xia, Yue Dong, Pieter Peers, and Xin Tong.  Recovering shape and spatially-varying surface reflectance under unknown illumination.  _ACM Transactions on Graphics_ , 35(6):187, 2016\.
  * Yao et al. [2020] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  _Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Yeh et al. [2022] Yu-Ying Yeh, Zhengqin Li, Yannick Hold-Geoffroy, Rui Zhu, Zexiang Xu, Miloš Hašan, Kalyan Sunkavalli, and Manmohan Chandraker.  Photoscene: Photorealistic material and lighting transfer for indoor scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18562–18571, 2022.
  * Yu et al. [2021] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for Real-Time Rendering of Neural Radiance Fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5752–5761, 2021.
  * Yu et al. [1999] Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins.  Inverse global illumination: Recovering reflectance models of real scenes from photographs.  In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_ , pages 215–224, 1999.
  * Zhang et al. [2020] Jianzhao Zhang, Guojun Chen, Yue Dong, Jian Shi, Bob Zhang, and Enhua Wu.  Deep inverse rendering for practical object appearance scan with uncalibrated illumination.  In _Advances in Computer Graphics_ , 2020.
  * Zhang et al. [2021a] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely.  PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021a.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.
  * Zhang et al. [2021b] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul E. Debevec, William T. Freeman, and Jonathan T. Barron.  NeRFactor: Neural factorization of shape and reflectance under an unknown illumination.  _ACM Trans. Graph._ , 40(6):237:1–237:18, 2021b.
  * Zhang et al. [2022] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou.  Modeling indirect illumination for inverse rendering.  In _CVPR_ , 2022.

Supplementary Material

Contents

A URL and websiteA
B Broader impactB
C LicenseC
D Data formatD
D.1 File formatsD.1
D.2 Directory structureD.2
D.3 Coordinate systemsD.3
E Data collectionE
E.1 CalibrationE.1
E.2 Capture protocolE.2
E.3 Post processingE.3
F Evaluation metricsF
G Relighting experimentsG
G.1 MethodsG.1
H Novel view synthesis experimentsH
I DatasheetI

##  Appendix A URL and website

The dataset and code is available at https://github.com/isl-org/objects-with-
lighting. The code includes the scripts for evaluation and the tools that were
used for building the dataset.

##  Appendix B Broader impact

The dataset is intended to evaluate the reconstruction and rendering of
objects. We think the risk of negative societal impact directly through our
dataset is low. The dataset does not contain personal information or offensive
content. However, the dataset is intended to help to improve photorealistic
rendering of objects and there is a potential risk that this can be used to
create artificial images that are not labelled as such.

##  Appendix C License

The dataset is licensed under the CDLA-permissive 2.0 license. The source code
is licensed under the Apache License 2.0.

##  Appendix D Data format

###  D.1 File formats

Our dataset uses standard file formats for images and parameters to make it
easy to use. For images we use two file formats: the widely used PNG-format to
store 24-bit RGB images for the input and ground truth images, and the RGBE or
Radiance HDR format to store the equirectangular environment maps. Both
formats are lossless to avoid any artifacts when reading or writing the data.
The widely used OpenCV library can be used to read both formats. Parameters
are stored as text files and their format as well as example reading code
using Numpy is provided in the repository. For convenience we provide meshes
generated with NeuS stored in the PLY format, which is supported by many 3D
viewers and libraries.

###  D.2 Directory structure

The dataset is organized in a simple but extendable structure. Each object is
a directory with a unique name in the root of the dataset. For each object we
collect all data in a subfolder test, which stores the ground truth images for
evaluation and a folder inputs with the data allowed to use for the
reconstruction process. The directory tree for an object with its core data
files looks like this .1 dataset. .2 object name. .3 test. .4
gt_image_xxxx.png. .4 gt_camera_xxxx.txt. .4 gt_env_512_rotated_xxxx.hdr. .4
gt_exposure_xxxx.txt. .4 inputs. .5 image_xxxx.png. .5 camera_xxxx.png. .5
mask_xxxx.png. .5 exposure.txt. .5 object_bounding_box.txt..

###  D.3 Coordinate systems

Our dataset contains images, camera parameters, and environment maps, each
with their respective coordinate system. In the following, we describe these
coordinate systems and how they are related.

Cameras Cameras look in positive z-direction. The intrinsic and extrinsic
camera parameters can be used to directly project a 3D point $\mathbf{X}$ to
image space coordinates with

$\mathbf{x}=\mathbf{K}\left(\mathbf{R}\mathbf{X}+\mathbf{t}\right).$ (3)

$\mathbf{x}$ is a homogeneous point describing a position in the image;
$\mathbf{K}$ is the upper triangular matrix with the camera intrinsics; and
$\mathbf{R},\mathbf{t}$ are the extrinsic parameters as rotation matrix and
translation.

Images The x-axis for images points to the right and the y-axis points down
following the memory order. The coordinates $(x,y)$ of the top left corner are
$(0,0)$. The center of the first pixel is at $(0.5,0.5)$. The bottom right
corner for an image with width $w$ and height $h$ is at $(w,h)$.

Environment maps Environment maps are stored as equirectangular images. We use
a normalized coordinate system similar to regular images. The u-axis points to
the right and the v-axis points down following the memory order. The
coordinates $(u,v)$ of the top left corner are $(0,0)$. The bottom right
corner is at $(1,1)$ irrespective of the size of the environment map. This
corresponds to the texture coordinate convention used by DirectX and is
visualized in Fig. 5(a).

Directions map to the equirectangular image as shown in Fig. 5(b). The
direction +Z $(0,0,1)$ maps to the upper border of the environment map and -Z
$(0,0,-1)$ to the lower border. +X $(1,0,0)$ maps to the center and -X
$(-1,0,0)$ maps to the vertically centered point on the right and left border.
+Y $(0,1,0)$ and -Y $(0,-1,0)$ map to the uv coordinates $(0.25,0.5)$ and
$(0.75,0.5)$ respectively. The environment map projected to a sphere with
coordinate axes is visualized in Fig. 5(c).

(a) UV coordinates.

(b) Spatial directions.

(c) Left half of the environment map visualized as sphere.

Figure 5: Visualization of the coordinate systems used with environment maps.

##  Appendix E Data collection

###  E.1 Calibration

Our dataset requires calibration of the linear color spaces of the cameras
used, the intrinsic parameters of the cameras, and the geometric calibration
of the extrinsics or camera poses.

Color Color calibration is important because we use two cameras that capture
images in their individual color spaces. We capture all images of the object
with a Canon EOS90D but use a Ricoh Theta Z1 to capture environment maps. To
get all cameras into a common linear color space we use the color checker
shown in Fig. 7(a). We collect data for calibrating each camera by taking
multiple images with different exposure values using the shutter speed. We
keep the aperture fixed for both cameras (EOS 90D at $f/8.0$, Theta Z1 at
$f/2.1$). For ISO we always use 80 for the Theta Z1 but calibrate the EOS90D
for multiple ISO values: 100, 500, 1000, 2000. To compute the color
transformation matrix we first merge the exposure brackets to HDR images and
compute the least squares $3\times 3$ color transform that maps the HDRIs from
the EOS90D to the linear color space of the Theta Z1.

Camera Intrinsics To estimate the camera intrinsics we use the calibration
pipeline from OpenCV. We use the OpenCV pinhole camera model but limit the
radial distortion to a single parameter. For calibrating the fisheye cameras
of the Theta Z1 we use the camera model presented in [34]. We use a board with
AprilTag markers as shown in for the calibration. One of the images used for
calibration is shown in Fig. 7(c).

Figure 6: Overview of the camera pose calibration.We show the process for
aligning test images and the environment map for an environment _Env1_ along
with the alignment of images of an _Env0_ which are used as input for the
tested methods. Colors indicate the association of intermediate data and
outputs to the input images where disambiguation is needed.

Camera poses Calibration of the camera poses is crucial since we want to
compare images of the same object taken in different environments and also
relight them with the measured illumination in the form of environment maps.
See Fig. 6 for an overview of the calibration process. We use COLMAP [61] with
a mixture of automatically detected SIFT keypoints, keypoints from a board
with AprilTag markers as seen in Fig. 7(b), and expert annotated keypoints.
For each scene, we detect the AprilTags in all images and inject the corners
into the Structure from Motion pipeline as constant points. Knowing the
geometry of the AprilTag board this allows us to define a metric coordinate
system with the center of the board as the origin. To align the environment
maps to this coordinate system we detect the AprilTags in the fisheye images
by temporarily converting them into 8 pinhole projections for the detector.
With the detected corner points of the markers, we then solve a PnP problem to
compute the pose for the omnidirectional camera. In the last step, we have to
align the test images taken in a different environment to the object in the
environment which we use for the reconstruction. Since the only common
structure is the object itself we need to find correspondences on the object.
We found that automatic detections are not reliable, adding markers on the
objects is in many cases not feasible, and gluing the object to the board does
not allow for easy transport. Thus, we resort to manual annotations that
relate the images of the same objects in different environments. Fig. 8(a)
shows our correspondence annotation tool specialized for this task. We
annotate $\geq 10$ points for each object in the test images and the images
used for reconstruction. We distribute the annotated points over the object
and use these points in the Structure from Motion pipeline too. To register
the test images we solve a PnP problem to compute the camera poses of the test
images.

###  E.2 Capture protocol

We describe the capture process for a scene in our dataset.

Step 1 We start with capturing a reference environment map. We place the 360
camera in the center of the marker board and capture exposure brackets with 11
images at different shutter speeds. In outdoor environments, we shoot
additional images with a custom ND filter to get the full unclipped range of
the sunlight.

Step 2 After capturing a reference for the environment, we remove the 360
camera and place the object in the center of the board. We take 42 to 67
photos of each object setting shutter speed and ISO such that we avoid
overexposure. We use the same settings for all photos of the object meant as
input for the reconstruction.

Step 3 We shoot the ground truth images. To be able to change the exposure
values afterward and to enable evaluation of linear images in the future, we
use a tripod and capture exposure brackets with 7 images, which we merge into
an HDR image. We repeat the process three times to create ground truth images
from three different angles. Each time we mark the position of the tripod for
Step 4.

Step 4 We replace the object with the 360 camera and capture three HDR
environment maps as described in Step 1. For each environment map we put the
tripod with the DSLR camera on one of the previously marked positions. This
allows us to capture the environment as in Step 2 in which the tripod and
camera can act as occluders and affect the illumination. An example of the
environment maps captured in this step is shown in Fig. 9.

(a) Color calibration target.

(b) Board with AprilTags.

(c) A calibration image for the Theta Z1.

Figure 7: Calibration targets for color and intrinsic camera parameters.

(a) Tool for annotating object keypoints.

(b) Bounding box annotation.

Figure 8: Annotation tools to define keypoints on the object and the bounding
box.

Figure 9: The camera and tripod are part of the environment map for test
images because they affect the illumination.

###  E.3 Post processing

Besides the calibration, we apply post-processing steps to make the data
easier to use and reduce errors. For environment maps, we stitch the dual
fisheye images produced by the 360 camera and provide directly equirectangular
images that use the coordinate systems described in Section D.3 and are
aligned with the pinhole cameras of the input and ground truth images.

Many methods require the bounds of the object to be reconstructed to work
efficiently. We manually annotate 3D bounding boxes for all objects using the
sparse reconstruction obtained from COLMAP as seen in Fig. 8.

Additionally, we provide approximate foreground masks for the input images. To
this end, we reconstruct the objects with NeuS and manually clean the mesh.
The masks are generated by projecting the mesh to each input image. For the
foreground masks used in the evaluation with the test images, we use expert
annotations instead.

The last post-processing step is the downsampling of the generated images. For
the input and test images, we reduce the resolution from $6960\times 4640$ to
$1746\times 1165$ to facilitate working with the data and to alleviate any
errors. For the environment maps, we downsample from $6720\times 3360$ to
$1024\times 512$ for the same reasons.

##  Appendix F Evaluation metrics

We use the standard metrics peak signal-to-noise ratio (PSNR), structural
similarity index measure (SSIM), and learned perceptual image patch similarity
(LPIPS) [78] to compare rendered images with their corresponding ground truth.
We briefly describe how we use these metrics in our evaluation script.

For PSNR and SSIM we use the implementation in scikit-image [67]. We convert
and scale our 8-bit tone-mapped RGB images to floating point images with range
$[0..1]$ before computing the two metrics. In addition, we set background
pixels to 0 using the ground truth foreground mask. We compute the PSNR pixel-
wise for all channels and aggregate values over the valid pixels defined by
the foreground mask. For SSIM we use a sliding window with a size of $7\times
7$ pixels and compute the similarity for each pixel and channel and aggregate
over all valid pixels. The values we report are the average PSNR and SSIM over
the valid pixels.

For LPIPS we use the implementation provided by the authors. We transform our
images to floating point images with range $[-1..1]$ and set background pixels
to 0. To account only for the valid pixels we compute the differences per
pixel by scaling the feature difference maps to the original image size. This
corresponds to passing the option spatial=True. We chose AlexNet as the
network. The reported value is the average of the per-pixel loss image for the
valid pixels.

##  Appendix G Relighting experiments

We provide additional details for the relighting experiments on our dataset
and Synthetic4Relight. Quantitative results on our dataset are reported in
Table 1 and Table 3. We show qualitative examples for each object of the
Objects With Lighting dataset in Fig. 13 and Fig. 14. For Synthetic4Relight we
show examples in Fig. 15.

We can identify multiple failure modes on our dataset. The most obvious
artifacts stem from the shape reconstruction. These are especially visible for
NVDiffrec which uses an explicit surface representation and produces noisy
surfaces. Another problem is related to the reconstruction of concave shapes,
which can be observed for multiple methods. We show examples of these problems
in Fig. 10.

Another failure mode can be observed with respect to the estimated materials.
Methods like our baseline, InvRender, and NVDiffrecMC tend towards
overestimating the glossiness of objects, which can be observed by comparing
highlights in the ground truth and rendered images. We show examples in Fig.
11.

Besides the reconstruction, the employed rendering technique is crucial to
produce realistic images. This can be seen on the porcelain mug in Fig. 12 for
which NVDiffrec misses to recreate the shadow. Methods like Mitsuba which use
a more sophisticated rendering process account for the visibility of the
illumination and can more faithfully recover generate the shadow.

(a) NVDiffrec

(b) NeRFactor

(c) NeROIC

(d) InvRender

Figure 10: Artifacts related to failures in shape reconstruction. (a) and (b):
noisy shapes lead to problems in the relighted images. (c) and (d): concave
shapes are problematic for most methods.

(a) GT

(b) Mitsuba+NeuS

(c) InvRender

(d) NVDiffrecMC+NeuS

Figure 11: Examples where objects appear too glossy compared to the ground
truth.

(a) GT

(b) Mitsuba+NeuS

(c) NVDiffrec+NeuS

Figure 12: Methods that do not account for the visibility of the illumination
have a disadvantage in reproducing shadows and are also prone to bake shadows
into the material.

###  G.1 Methods

Mitsuba+NeuS We describe our baseline method in Section 4 and provide the
source code in the accompanying GitHub repository of the dataset to reproduce
results. We use for all experiments the same parameters. Reconstruction of one
scene of our dataset takes about 10 hours on an Nvidia A6000. The majority of
the time is spent on the surface reconstruction using NeuS [68]. The material
reconstruction using Mitsuba [30] takes only about 15 minutes.

InvRender The method is described in [80]. We provide instructions and scripts
for reproducing results in our code release. All experiments use the same
parameters. For each scene, the geometry reconstruction stage takes about 12
hours on an Nvidia A100. The illumination and visibility stage takes about 2
hours. The material optimization stage takes about 3 hours.

NeRD The method is described in [10]. We provide instructions and scripts for
reproducing results in our code release. We use the same parameters for all
scenes in the Objects With Lighting dataset. For DTU and BMVS, we scale the
scene to fit within a unit sphere. For each scene, it takes about 20 hours on
an Nvidia RTX 3090. For relighting, we’ve modified the code to use our tone
mapping equation, and to align the environment map to our coordinate system.
We save relighting output as hdr files to apply exposure correction and color
balancing before evaluation.

NeRFactor(+NeuS) The method is described in [79]. We provide instructions and
scripts for reproducing results in our code release. The coarse NeRF
reconstruction stage takes about 16-24 hours on an Nvidia A6000 and depends on
the number of images and resolution of the scene. Geometry extraction takes
about 20 minutes for each image, which can be parallelized. Further joint
optimization takes about 40 minutes. We keep parameters constant for all
experiments but vary the image resolution to improve reconstruction results.
If combined with NeuS we can skip the NeRF reconstruction.

NeROIC The method is described in [39]. We provide instructions and scripts
for reproducing results in our code release. We use the same parameters for
all scenes in the Objects With Lighting dataset. For DTU and BMVS, we scale
the scene to fit within a unit sphere. For each scene, it takes about 10 hours
to run geometry reconstruction, 1 hour to extract normals, and about 5 hours
to optimize the rendering network on an Nvidia RTX 3090.

Neural-PIL The method is described in [11]. We provide instructions and
scripts for reproducing results in our code release. We use the same
parameters for all scenes in the Objects With Lighting dataset. For DTU and
BMVS, we scale the scene to fit within a unit sphere. For each scene, it takes
about 24 hours to train on an Nvidia RTX 3090. For relighting, we’ve modified
the code to use our tonemapping equation, and to align the environment map to
our coordinate system. We save relighting output as hdr files to apply
exposure correction and color balancing before evaluation.

NVDiffrec(+NeuS) The method is described in [49]. We provide instructions and
scripts for reproducing results in our code release. We use the same
parameters for all the scenes. For each scene, it takes about 1 hour for the
geometry reconstruction pass, and another 1 hour to optimize the materials on
an Nvidia RTX 3090. If combined with NeuS, we can skip the geometry
reconstruction pass.

NVDiffrecMC+NeuS The method is described in [27]. We provide instructions and
scripts for reproducing results in our code release. When training from
scratch, each scene takes about 4 hours to train a coarse mesh and another 4
hours to optimize materials on an Nvidia A100. If training with the meshes
generated with NeuS, we first use Blender to do UV-unwrap for each mesh, and
optimize the material with this fixed mesh. Each experiment only needs about 4
hours in this configuration. All experiments use the same parameters.

PhySG The method is described in [77]. We provide instructions and scripts for
reproducing results in our code release. We modify PhySG’s environment map
convention to be consistent with that of our data. We keep the default hyper-
parameters of PhySG and train from scratch. Each scene takes about 12 hours to
train on an Nvidia RTX 2080Ti. For relighting, we first overfit a Spherical
Gaussian (SG) representation (128 SG using the default PhySG setting) for each
environment lighting, and then use the SG-based lighting to relight the
reconstructed 3D model.

TensoIR The method is described in [32]. We provide instructions and scripts
for reproducing results in our code release. We use the default parameters set
by the author. We use masked input images with white backgrounds for training,
similar to TensoIR’s own synthetic dataset. For DTU and BMVS, we scale the
scene to fit within a unit sphere before training. For each scene, the
training takes about 15 hours on a single Nvidia RTX3090. For novel view
synthesis evaluation, we report the results from the volume rendering network,
which has higher PSNRs than the BRDF rendering network.

Figure 13: Qualitative examples for relighting on our dataset. The first row
shows the environment map used for relighting. For each object, we show one of
the test images of the 3 lighting categories. The first image for each object
corresponds to the reconstruction environment. Each object has been captured
in highly different lighting conditions: outdoor, indoor, and indoor with
artificial light.

Figure 14: Qualitative examples for relighting on our dataset. The first row
shows the environment map used for relighting. For each object, we show one of
the test images of the 3 lighting categories. The first image for each object
corresponds to the reconstruction environment. Each object has been captured
in highly different lighting conditions: outdoor, indoor, and indoor with
artificial light.

Figure 15: Qualitative examples for relighting on the Synthetic4Relight
dataset. The first row shows the environment map used for relighting. For each
object, we show one of the test images. The first image for each object
corresponds to the reconstruction environment.

##  Appendix H Novel view synthesis experiments

We show qualitative examples for novel view synthesis in Figures
17,18,19,20,21,22.

We can observe obvious failures with the shape and shading for this task too
but miss deficiencies like missing or baked shadows, or problems with the
material parameters. We show examples of failures in 16.

(a) GT

(b) Neural-PIL

(c) GT

(d) NeRD

Figure 16: Obvious failure cases that can be observed irrespective of the
task. (a),(b) show an obvious problem in the shape. (c),(d) show the lack of
texture detail.

Figure 17: Qualitative examples for novel view synthesis on scenes from the
DTU dataset. For each scene, we show two novel views.

Figure 18: Qualitative examples for novel view synthesis on scenes from the
DTU dataset. For each scene, we show two novel views.

Figure 19: Qualitative examples for novel view synthesis on scenes from the
BlendedMVS dataset. For each scene, we show two novel views.

Figure 20: Qualitative examples for novel view synthesis on our dataset for
the first environment. For each scene, we show three novel views. Note that
NeRD failed to reconstruct the porcelain mug scene.

Figure 21: Qualitative examples for novel view synthesis on our dataset for
the first environment. For each scene, we show three novel views.

Figure 22: Qualitative examples for novel view synthesis on the
Synthetic4Relight dataset for the first environment. For each scene, we show
three novel views.

##  Appendix I Datasheet

Our datasheet follows the template in [23].

### Motivation

For what purpose was the dataset created?
The dataset was created for evaluating the reconstruction of shape and
material of objects for relighting in novel environments.

Who created the dataset and on behalf of which entity?
The authors of this paper.

Who funded the creation of the dataset?
The dataset was created as part of a research project at Intel Labs.

#### Composition

What do the instances that comprise the dataset represent?
The instances represent photos of objects from multiple view points and
equirectangular projections of environments.

How many instances are there in total?
Currently there are 8 objects each recorded in 3 environments.

Does the dataset contain all possible instances or is it a sample of instances
from a larger set?
N/A. The dataset is not sampled from a larger existing set.

What data does each instance consist of?
Each instance contains images of an object for the reconstruction and a
separate set of images and environment maps for quantitative evaluation. For
all images for reconstruction there are approximate foreground masks. For the
test images there are expert annotated foreground masks. All images come with
a pose and intrinsic camera parameters. A coarse bounding box and exposure
values for each image are given as text files.

Is there a label or target associated with each instance?
The test images for each object are the target.

Is any information missing from individual instances?
No.

Are relationships between individual instances made explicit?
No.

Are there recommended data splits (e.g., training, development/validation,
testing)?
No. The dataset is meant for testing only.

Are there any errors, sources of noise, or redundancies in the dataset?
The imaging and calibration process is subject to noise. No errors are known
at this time. Errors will be fixed in future revisions of the dataset.

Is the dataset self-contained, or does it link to or otherwise rely on
external resources?
The dataset is self-contained.

Does the dataset contain data that might be considered confidential?
No.

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety?
No.

Does the dataset relate to People?
No.

#### Collection process

How was the data associated with each instance acquired?
We use a DSLR camera for the input and test images and a 360 camera for the
environment maps.

What mechanisms or procedures were used to collect the data?
We capture the data in a specific order and with specific camera settings to
enable the creation of HDR images for environment maps and test images. We
refer to Section E for more details.

If the dataset is a sample from a larger set, what was the sampling strategy?
N/A

Who was involved in the data collection process and how were they compensated?
Only the authors of this paper were involved in the process.

Over what timeframe was the data collected?
The images were collected September 2022 to March 2023.

Were any ethical review processes conducted?
No.

Does the dataset relate to People?
No.

#### Preprocesssing / cleaning / labelling

Was any preprocessing/cleaning/labeling of the data done?
Yes, the data is preprocessed to allow researchers to focus on the task. All
calibrations are a result of this preprocessing. We refer to Section E for
more details.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data?
Yes.

Is the software that was used to preprocess/clean/label the data available?
Yes. Please see the source code repository.

#### Uses

Has the dataset been used for any tasks already?
Yes. We evaluated existing methods on our dataset in this paper.

Is there a repository that links to any or all papers or systems that use the
dataset?
No. This paper is the first to use this dataset.

What (other) tasks could the dataset be used for?
A subset of the dataset can be used to evaluate novel view synthesis for
objects.

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses?
No.

Are there tasks for which the dataset should not be used?
No.

#### Distribution

Will the dataset be distributed to third parties outside of the entity on
behalf of which the dataset was created?
Yes. The dataset will be made available to the public.

How will the dataset be distributed?
The dataset will be hosted on GitHub along with the source code used for
creating the dataset.

When will the dataset be distributed?
The dataset will be made available on publication of the accompanying paper.

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)?
The dataset will be licensed under the CDLA-permissive 2.0 license.

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances?
No.

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances?
No.

#### Maintenance

Who will be supporting/hosting/maintaining the dataset?
The authors listed on this paper will support and maintain the dataset.

How can the owner/curator/manager of the dataset be contacted?
The authors listed on this paper can be contacted via email or the datasets
GitHub repository.

Is there an erratum?
No.

Will the dataset be updated?
Yes. The dataset will be updated with new objects to adapt its difficulty to
push the state-of-the-art.

Will older versions of the dataset continue to be supported/hosted/maintained?
Yes. The data will be versioned.

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so?
Yes. The code used for building the dataset in available in the datasets
GitHub repo and others can contact the authors for adding new objects.