# Unsupervised Discovery of Object-Centric Neural Fields

Rundong Luo  Hong-Xing Yu  Jiajun Wu

###### Abstract

We study inferring 3D object-centric scene representations from a single
image. While recent methods have shown potential in unsupervised 3D object
discovery from simple synthetic images, they fail to generalize to real-world
scenes with visually rich and diverse objects. This limitation stems from
their object representations, which entangle objects’ intrinsic attributes
like shape and appearance with extrinsic, viewer-centric properties such as
their 3D location. To address this bottleneck, we propose the unsupervised
discovery of Object-Centric neural Fields (uOCF). uOCF focuses on learning the
intrinsics of objects and models the extrinsics separately. Our approach
significantly improves systematic generalization, thus enabling unsupervised
learning of high-fidelity object-centric scene representations from sparse
real-world images. To evaluate our approach, we collect three new datasets,
including two real kitchen environments. Extensive experiments show that uOCF
enables the unsupervised discovery of visually rich objects from a single real
image, allowing applications such as 3D object segmentation and scene
manipulation. In particular, uOCF demonstrates zero-shot generalization to
unseen objects from a single real image. Project page: https://red-
fairy.github.io/uOCF/.

Figure 1: We propose the unsupervised discovery of Object-Centric neural
Fields (uOCF), which infers factorized 3D scene representations from an unseen
real image, thus enabling scene reconstruction and manipulation from novel
views. We compare uOCF with the state-of-the-art method, uORF (Yu et al.,
2022).

##  1 Introduction

Creating factorized, object-centric 3D scene representations is a fundamental
ability in human vision and a long-standing topic of interest in computer
vision and machine learning. Some recent work has explored unsupervised
learning of 3D factorized scene representations from images alone (Stelzner et
al., 2021; Yu et al., 2022; Smith et al., 2023; Jia et al., 2023). These
methods have delivered promising results in 3D object discovery and
reconstruction from a simple synthetic image.

However, difficulties arise when attempting to generalize these approaches to
complex real-world scenes. The bottleneck lies in their object
representations: they represent each object in the viewer’s frame, entangling
intrinsic object attributes such as shape and appearance with extrinsic
properties such as the object’s 3D location. Such a paradigm means that a
slight shift in the object’s location or a subtle camera adjustment can
significantly change its latent representation. Intuitively, an object’s
intrinsic attributes should remain consistent irrespective of location, yet
this invariance is ignored in existing 3D object-centric learning models. As
demonstrated by convolutional networks (Zhang, 2019), accounting for this
invariance is crucial for generalization (Chattopadhyay et al., 2020; Deng et
al., 2022). The entanglement of object intrinsics and extrinsics substantially
hinders sample efficiency and systematic generalization.

We propose the unsupervised discovery of Object-Centric neural Fields (uOCF),
which learns to infer 3D object-centric scene representations from a single
image. Unlike existing methods, uOCF focuses on learning the intrinsics of
objects and models the extrinsics separately. As shown in Figure 1, uOCF
generalizes to unseen real-world scenes. We train uOCF on sparse multi-view
images without object annotations. During inference, uOCF takes in a single
image and generates a set of object-centric neural radiance fields and a
background radiance field.

Our translation-invariant design facilitates learning object priors from
scenes with different compositional configurations (e.g., the number of
objects and spatial arrangements) and allows learning from multiple different
data sources. For example, uOCF can learn object priors from single-object
room scenes and generalize to multi-object kitchen scenes with unknown spatial
layouts. Moreover, such generalization even facilitates _zero-shot_ 3D object
discovery on real images with unseen objects.

To evaluate our approach, we introduce new challenging datasets for 3D object
discovery, including two real kitchen datasets and a synthetic room dataset.
The two real datasets feature varied kitchen backgrounds and objects from
multiple categories. The synthetic room dataset features chairs with diverse,
realistic shapes and textures. Across all these datasets, uOCF offers high-
fidelity discovery of object-centric neural fields, allowing applications such
as unsupervised 3D object segmentation and scene manipulation from a real
image.

In summary, our contributions are threefold:

  * •

First, we highlight the overlooked role of object-centric modeling in
unsupervised 3D object discovery. We instantiate the idea by proposing the
unsupervised discovery of Object-Centric neural Fields (uOCF).

  * •

Second, we introduce the object-centric prior learning technique, which
leverages uOCF’s translation-invariant property to learn object priors from
scenes with different object-compositional configurations.

  * •

Lastly, we collect three challenging datasets, Room-Texture, Kitchen-Matte,
and Kitchen-Shiny, and show that uOCF significantly outperforms existing
methods on these datasets, unlocking zero-shot, single-image object discovery.

##  2 Related Works

Unsupervised Object Discovery. Prior to the rise of deep learning, traditional
methods for object discovery (often referred to as co-segmentation) primarily
aimed at locating visually similar objects across a collection of images
(Sivic et al., 2005; Russell et al., 2006), where objects are defined as
visual words or clusters of patches (Grauman & Darrell, 2006; Joulin et al.,
2010). This clustering concept was later incorporated into deep learning
techniques for improved grouping results (Li et al., 2019; Vo et al., 2020).
The incorporation of deep probabilistic inference propelled the field towards
factorized scene representation learning (Eslami et al., 2016). These methods
decompose a visual scene into several components, where objects are often
modeled as latent codes that can be decoded into image patches (Kosiorek et
al., 2018; Crawford & Pineau, 2019; Jiang et al., 2020; Lin et al., 2020),
scene mixtures (Greff et al., 2016, 2017, 2019; Burgess et al., 2019; Engelcke
et al., 2019; Locatello et al., 2020; Biza et al., 2023; Didolkar et al.,
2023), or layers (Monnier et al., 2021). Despite their efficacy in scene
decomposition, they do not model the objects’ 3D nature.

To model the 3D nature of scenes and objects, some recent works tried to learn
3D-aware representations from multi-view images of a single-scene (Liang et
al., 2022) or of large datasets for generalization (Eslami et al., 2018; Chen
et al., 2020; Sajjadi et al., 2022), while the latest research emphasizes the
single-image inference of object-centric factorized scene representations
(Stelzner et al., 2021; Yu et al., 2022; Smith et al., 2023). Notably, Yu et
al. (2022) propose the unsupervised discovery of object radiance fields (uORF)
from a single image. Later literature improves the efficiency (Smith et al.,
2023) and segmentation (Jia et al., 2023). However, their object
representations suffer from extrinsic entanglement and limited
generalizability to complex scenes. In contrast, our approach explicitly
disentangles object appearances from their extrinsics.

Figure 2: With a single forward pass, uOCF processes a single image input to
infer a set of object-centric radiance fields along with their 3D locations
and background radiance field. uOCF is trained on sparse multi-view images
from a collection of scenes and uses a single image as input during inference.

Object-Centric 3D Reconstruction. Decomposing visual scenes on an object-by-
object basis and estimating their semantic/geometric attributes has been
explored in several recent works (Wu et al., 2017; Yao et al., 2018; Kundu et
al., 2018; Ost et al., 2021). Some approaches, like AutoRF (Müller et al.,
2022), successfully reconstruct specific objects (e.g., cars) from annotated
images. Others decompose visual scenes into the background and individual
objects represented by neural fields (Yang et al., 2021; Wu et al., 2022). Our
work differs as it emphasizes unsupervised learning. Another line of recent
work focuses on lifting 2D segmentation to reconstructed 3D scenes, such as
Fan et al. (2022); Cen et al. (2023a, b). In contrast, our work aims at
single-image inference, whereas these studies concentrate on multi-view
reconstruction.

Generative Neural Fields. Neural fields have revolutionized 3D scene modeling.
Early works have shown promising geometry representations (Sitzmann et al.,
2019; Park et al., 2019). The seminal work by Mildenhall et al. (2020) on
neural radiance fields has opened up a burst of research on neural fields. We
refer readers to recent survey papers (Tewari et al., 2020; Xie et al., 2022)
for a comprehensive overview. Particularly, compositional generative neural
fields such as GIRAFFE (Niemeyer & Geiger, 2021) and others (Nguyen-Phuoc et
al., 2020; Wang et al., 2023b) also allow learning object representations from
image collections. Yet, they target unconditional generation and cannot tackle
inference.

##  3 Approach

Given a single input image, our goal is to infer object-centric radiance
fields (i.e., each discovered object is represented in its local object
coordinate rather than the world or the viewer coordinates) and the objects’
3D locations. The object-centric design not only boosts generalizability due
to representation invariance but also allows learning object priors from
scenes with different spatial layouts and compositional configurations. The
following provides an overview of our approach and then introduces the
technical details.

###  3.1 Model Overview

As shown in Figure 2, uOCF’s architecture consists of an encoder, a latent
inference module, and a decoder.

Encoder. From an input image $\mathbf{I}$, the encoder extracts a feature map
$\mathbf{f}\in\mathbb{R}^{N\cdot C}$, where $N=H\cdot W$ is the spatial size
of the feature map and $C$ represents the number of channels. We set it as a
frozen DINOv2-ViT (Oquab et al., 2023) followed by two convolutional layers.

Latent Inference Module. The latent inference module infers the latent
representation and position of the objects in the underlying 3D scene from the
feature map. We assume that the scene is composed of a background environment
and no more than $K$ foreground objects. Therefore, the output includes a
background latent $\mathbf{z}^{\mathrm{b}}\in\mathbb{R}^{1\times D}$ and a set
of foreground object latent
$\mathbf{z}^{\mathrm{f}}=[\mathbf{z}_{1}^{\mathrm{f}T}~{}\mathbf{z}_{2}^{\mathrm{f}T}~{}\cdots~{}\mathbf{z}_{K}^{\mathrm{f}T}]^{T}\in\mathbb{R}^{K\times
D}$ with their corresponding positions
$\\{\textbf{p}_{i}^{\mathrm{wd}}\\}_{i=1}^{K}$ , where
$\mathbf{p}_{i}^{\mathrm{wd}}\in\mathbb{R}^{3}$ denotes a position in the
world coordinate. Note that some object latent may not represent any object
when the scene has fewer than $K$ objects.

Decoder. Our decoder employs the conditional NeRF formulation
$g(\mathbf{x}|\mathbf{z})$, which takes the 3D location $\mathbf{x}$ and the
latent $\mathbf{z}$ as input and generates the radiance color and density for
rendering. We use two MLPs, $g^{\mathrm{b}}$ and $g^{\mathrm{f}}$, for the
background environment and the foreground objects, respectively.

###  3.2 Object-Centric 3D Scene Modeling

Object-Centric Latent Inference. Our Latent Inference Module (LIM) aims at
binding a set of learnable object queries
($~{}\mathbf{q}^{\mathrm{f}}=[\mathbf{q}_{1}^{\mathrm{f}T}~{}\mathbf{q}_{2}^{\mathrm{f}T}~{}\cdots~{}\mathbf{q}_{K}^{\mathrm{f}T}]^{T}\in\mathbb{R}^{K\times
D}$) to the visual features of each foreground object, and another query to
the background features ($\mathbf{q}^{\mathrm{b}}\in\mathbb{R}^{1\times D}$).
The binding is modeled via the cross-attention mechanism with learnable linear
functions
$\mathcal{K}^{\mathrm{b}},\mathcal{K}^{\mathrm{f}},\mathcal{Q}^{\mathrm{b}},\mathcal{Q}^{\mathrm{f}},\mathcal{V}^{\mathrm{b}},\mathcal{V}^{\mathrm{f}}$:

$\displaystyle\mathbf{A}_{i,j}=$
$\displaystyle\frac{\exp(\mathbf{M}_{i,j})}{\sum_{k}\exp(\mathbf{M}_{i,k})},~{}\text{where}$
$\displaystyle\mathbf{M}=$
$\displaystyle\frac{1}{\sqrt{D^{s}}}\begin{bmatrix}\mathcal{Q}^{\mathrm{b}}(\mathbf{q}^{\mathrm{b}})\cdot\mathcal{K}^{\mathrm{b}}(\mathbf{f})^{T}\\\
\mathcal{Q}^{\mathrm{f}}(\mathbf{q}^{\mathrm{f}})\cdot\mathcal{K}^{\mathrm{f}}(\mathbf{f})^{T}\end{bmatrix}^{T}\in\mathbb{R}^{N\times(K+1)}.$
(1)

We then calculate the update signals for queries via an attention-weighted
mean of the input:

$\displaystyle\mathbf{u}^{\mathrm{b}}$
$\displaystyle=(\mathbf{W}_{(:,1)})^{T}\cdot\mathcal{V}^{\mathrm{b}}(\mathbf{f})\in\mathbb{R}^{1\times
D},$ $\displaystyle\mathbf{u}^{\mathrm{f}}$
$\displaystyle=(\mathbf{W}_{(:,2:)})^{T}\cdot\mathcal{V}^{\mathrm{f}}(\mathbf{f})\in\mathbb{R}^{K\times
D},$ (2)

$\text{where}~{}\mathbf{W}_{i,j}=\frac{\mathbf{A}_{i,j}}{\sum_{l}\mathbf{A}_{l,j}}$
is the attention map normalized over the spatial dimension. Queries are then
updated by:

$\displaystyle\mathbf{q}^{\mathrm{b}}\leftarrow\mathbf{q}^{\mathrm{b}}+\mathbf{u}^{\mathrm{b}},~{}\mathbf{q}^{\mathrm{f}}\leftarrow\mathbf{q}^{\mathrm{f}}+\mathbf{u}^{\mathrm{f}},$
$\displaystyle\mathbf{q}^{\mathrm{b}}\leftarrow\mathbf{q}^{\mathrm{b}}+t^{\mathrm{b}}(\mathbf{q}^{\mathrm{b}}),~{}\mathbf{q}^{\mathrm{f}}\leftarrow\mathbf{q}^{\mathrm{f}}+t^{\mathrm{f}}(\mathbf{q}^{\mathrm{f}}),$
(3)

where $t^{\mathrm{b}}$ and $t^{\mathrm{f}}$ are MLPs. We repeat this procedure
for $T$ iterations, followed by concatenating the updated object queries with
the corresponding attention-weighted mean of the input feature map
$\mathbf{f}$ (global residual), finally delivering the background latent
$\mathbf{z}^{\mathrm{b}}$ and foreground latent
$\\{\mathbf{z}^{\mathrm{f}}_{i}\\}_{i=1}^{K}$.

Our LIM is related to the Slot Attention (SA) (Locatello et al., 2020) while
differs in several critical aspects. We discuss their relationship in Appendix
C.1.

Position Inference in 2D. To infer objects’ position along with their latent
representation, we assign a normalized image position
$\mathbf{p}_{i}^{\mathrm{img}}\in[-1,1]^{2}$ initialized as zero to each
foreground query, then iteratively update them by momentum $m$ with the
attention-weighted mean over the normalized 2D grid
$\mathbf{E}^{\mathrm{abs}}\in[-1,1]^{N\times 2}$:

$\mathbf{p}_{i}^{\mathrm{img}}\leftarrow(\mathbf{W}_{(:,i+1)})^{T}\cdot\mathbf{E}^{\mathrm{abs}}\cdot(1-m)+\mathbf{p}_{i}^{\mathrm{img}}\cdot
m.$ (4)

To incorporate the inferred positions, we adopt the relative positional
encoding (Biza et al., 2023)
$\mathbf{E}^{\mathrm{pos}}_{i}:=\mathrm{concat}([\mathbf{E}^{\mathrm{abs}}-\mathbf{p}_{i}^{\mathrm{img}},\mathbf{p}_{i}^{\mathrm{img}}-\mathbf{E}^{\mathrm{abs}}])\in\mathbb{R}^{N\times
4}$, where $\mathrm{concat}$ is the concatenation along the last dimension.
Then, we re-write $M$ in Eq. (1) as:

$M=\frac{1}{\sqrt{D^{s}}}\begin{bmatrix}\mathcal{Q}^{\mathrm{b}}(\mathbf{q}^{\mathrm{b}})\cdot\mathcal{K}^{\mathrm{b}}(\mathbf{f}+h_{1}(\mathbf{E}^{\mathrm{abs}}))^{T}\\\
\mathcal{Q}^{\mathrm{f}}(\mathbf{q}_{1}^{\mathrm{f}})\cdot\mathcal{K}^{\mathrm{f}}(\mathbf{f}+h_{1}(\mathbf{E}^{\mathrm{pos}}_{1}))^{T}\\\
\cdots\\\
\mathcal{Q}^{\mathrm{f}}(\mathbf{q}_{K}^{\mathrm{f}})\cdot\mathcal{K}^{\mathrm{f}}(\mathbf{f}+h_{1}(\mathbf{E}^{\mathrm{pos}}_{K}))^{T}\\\
\end{bmatrix}^{T},$ (5)

where $h_{1}:\mathbb{R}^{4}\rightarrow\mathbb{R}^{D}$ is a linear function.

Overall, LIM achieves a gradual binding between the queries and the objects in
the scene through an iterative update of the queries and their locations. To
address potential issues of duplicate object identification, we invalidate one
of two similar object queries with high similarity and positional proximity by
the start of the last iteration. Finally, a small bias term is added to the
position to handle potential occlusion, i.e.,
$\mathbf{p}_{i}^{\mathrm{img}}\leftarrow\mathbf{p}_{i}^{\mathrm{img}}+\mathrm{tanh}(h_{2}((W_{(:,i+1)})^{T}))\cdot\alpha$
, where scaling hyperparameter $\alpha=0.2$ and
$h_{2}:\mathbb{R}^{N}\rightarrow\mathbb{R}^{2}$ is a linear function.

Lifting Position to 3D. The 2D positions $\mathbf{p}_{i}^{\mathrm{img}}$ are
then unprojected into the 3D world coordinate to obtain
$\mathbf{p}_{i}^{\mathrm{wd}}$. We consider two scenarios: for objects on a
known shared plane (e.g., using plane detection (Ge et al., 2023)), a direct
ray extension (extend the ray between the camera center and
$\mathbf{p}_{i}^{\mathrm{img}}$ to intersect the ground plane) is used;
alternatively, in the absence of such a plane, we extend the rays by depth
$d\cdot s_{i}$, where $d$ is the distance between the camera and the scene
center and $\\{s_{i}\\}_{i=1}^{K}$ are scaling terms predicted by a linear
layer using the camera parameters and object latent as input. These two
approaches are referred to as uOCF-P and uOCF-N, respectively, with uOCF
denoting uOCF-P by default. Further details and visualizations of these
processes are provided in Appendix C.1.

Compositional Neural Rendering. The object positions allow us to put objects
in their local coordinates rather than the viewer or world coordinates,
thereby obtaining object-centric neural fields. Technically, for each 3D point
$\mathbf{x}$ in the world coordinate, we transform it to the $i^{\text{th}}$
object’s local coordinate by
$\mathbf{x}_{i}=R\cdot(\mathbf{x}-\mathbf{p}_{i}^{\mathrm{wd}})$, where $R$
denotes the input camera rotation matrix. We then retrieve the color and
density of $\mathbf{x}$ in the foreground radiance fields as
$(\mathbf{c}_{i},\sigma_{i})=g^{\mathrm{f}}(\mathbf{x}_{i}|\mathbf{z}^{\mathrm{f}}_{i})$
and in the background radiance field as
$(\mathbf{c}_{0},\sigma_{0})=g^{\mathrm{b}}(\mathbf{x}|\mathbf{z}^{\mathrm{b}})$.
These values are aggregated into the scene’s composite density and color
$(\overline{\mathbf{c}},\overline{\sigma})$ using density-weighted means:

$\overline{\sigma}=\sum_{i\geq
0}\omega_{i}\sigma_{i},~{}\overline{\mathbf{c}}=\sum_{i\geq
0}\omega_{i}\mathbf{c}_{i},~{}\text{where}~{}~{}\omega_{i}=\frac{\sigma_{i}}{\sum_{j\geq
0}\sigma_{j}}.$ (6)

Finally, we compute the pixel color by volume rendering. Our pipeline is
trivially differentiable, allowing backpropagation through all parameters
simultaneously.

Discussion on Extrinsics Disentanglement. An object’s canonical orientation is
ambiguous without assuming its category (Wang et al., 2019). Thus, we choose
not to disentangle objects’ orientation since we target category-agnostic
object discovery. Further, we observe that uOCF has learned meaningful
representations that can smoothly interpolate an object’s scale and
orientation. Please refer to Appendix B for visualization and analysis.

Figure 3: Our object-centric design allows learning object priors from scenes
with different compositional configurations. We first train our model on
simple configurations (e.g., scenes with a single synthetic object) to learn
_category-agnostic_ object priors. Then, we adapt this prior to complex
scenes. Note that no object annotation is needed in either stage.

###  3.3 Object-Centric Prior Learning

Unsupervised 3D object discovery in complex compositional scenes is inherently
difficult due to ambiguities. For example, when an object is partially
occluded with a visually similar object, the inference model needs to find the
object boundary to separate the occluded one from the blocker and generate
plausible completion to the occluded object.

One way to address such a challenge is to learn object priors. However,
existing methods have difficulties learning generalizable object priors, as
their object representation is sensitive to spatial configurations: a minor
shift in camera pose or object location, rather than the object itself, can
lead to drastic changes in the object representation. Thus, such learned
object priors do not generalize when there are unseen spatial configurations.
Our 3D object-centric representation mitigates this issue due to its
translation-invariance.

Therefore, we introduce object-centric prior learning. The main idea is to
learn general object priors (e.g., physical coherence) from simple scenes
(e.g., scenes with a single synthetic object), then leverage the obtained
priors to learn from more complex scenes that could potentially have very
different scene geometry and spatial layout. We show an illustration in Figure
3.

###  3.4 Model Training

Object-Centric Sampling. To improve the reconstruction quality, we leverage an
object’s local coordinates to concentrate the sampled points in proximity to
the object, as illustrated in Figure 10(b). Specifically, we start dropping
distant samples from the predicted object positions after a few training
epochs when the model has learned to distinguish the foreground objects and
predict their positions. This approach enables us to quadruple the number of
samples with the same amount of computation, leading to significantly improved
robustness and visual quality.

In both training stages, we train our model across scenes, each with
calibrated sparse multi-view images. For each training step, the model
receives an image as input, infers the objects’ latent representations and
positions, renders multiple views from the input and reference poses, and
compares them to the ground truth images to calculate the loss. Model
supervision consists of the MSE reconstruction loss $\ell_{\text{recon}}$ and
the perceptual loss $\ell_{\text{perc}}$ (Johnson et al., 2016) between the
reconstructed and ground truth images. In addition, we incorporate the depth
ranking loss (Wang et al., 2023a) with pre-trained monocular depth estimators
and background occlusion regularization (Yang et al., 2023) to minimize common
floating artifacts in few-shot NeRFs.

The overall loss function is thus formulated as follows:

$\displaystyle\mathcal{L}$
$\displaystyle=\ell_{\text{recon}}+\lambda_{\text{perc}}\ell_{\text{perc}}+\lambda_{\text{depth}}\ell_{\text{depth}}+\lambda_{\text{occ}}\ell_{\text{occ}}.$
(7)

##  4 Experiments

Figure 4: Samples from our datasets. Table 1: Scene segmentation and novel view synthesis result on Room-Texture. Baseline methods are uORF (Yu et al., 2022), BO-QSA (Jia et al., 2023), and COLF (Smith et al., 2023). Method | Scene segmentation | Novel view synthesis
---|---|---
ARI$\uparrow$ |  FG-ARI$\uparrow$ |  NV-ARI$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
uORF (Yu et al., 2022) |  0.670 |  0.093 |  0.578 |  24.23 |  0.711 |  0.254
BO-QSA (Jia et al., 2023) |  0.697 |  0.354 |  0.604 |  25.26 |  0.739 |  0.215
COLF (Smith et al., 2023) |  0.235 |  0.532 |  0.011 |  22.98 |  0.670 |  0.504
uOCF-N (ours) |  0.791 |  0.584 |  0.722 |  28.81 |  0.796 |  0.138
uOCF-P (ours) |  0.785 |  0.563 |  0.704 |  28.85 |  0.798 |  0.136
Figure 5: Scene segmentation qualitative results. Novel view images are for reference only. Table 2: Novel view synthesis results on Kitchen-Shiny and Kitchen-Matte. Method | Kitchen-Shiny | Kitchen-Matte
---|---|---
PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
uORF (Yu et al., 2022) |  19.23 |  0.602 |  0.336 |  26.07 |  0.808 |  0.092
BO-QSA (Jia et al., 2023) |  19.78 |  0.639 |  0.318 |  27.36 |  0.832 |  0.067
COLF (Smith et al., 2023) |  18.30 |  0.561 |  0.397 |  20.68 |  0.643 |  0.236
uOCF-N (ours) |  27.87 |  0.842 |  0.055 |  28.25 |  0.841 |  0.055
uOCF-P (ours) |  28.58 |  0.862 |  0.049 |  29.40 |  0.867 |  0.043
Figure 6: Novel view synthesis qualitative results on Kitchen-Matte (top rows)
and Kitchen-Shiny (bottom rows).

We evaluate our method on unsupervised object segmentation in 3D, novel view
synthesis, and scene manipulation in 3D. Please refer to Appendix D for
limitation analysis and Appendix E for more experimental results.

Data. We curate two real-world datasets to evaluate our method and one
synthetic dataset for learning object priors.

Room-Texture. The synthetic dataset includes 324 object models from the
“armchair” category of the ABO (Collins et al., 2022) dataset. Each scene
contains one object set against a background randomly chosen from a collection
of floor textures. We also render a multi-object variant with 2-4 objects in
each scene to promote comparison with existing methods. The single-object
subset contains 1296 scenes, and the multi-object subset contains 5000 scenes
for training and 100 for evaluation. Each scene is rendered from 4 directions
toward the scene center.

Kitchen-Matte. This dataset features scenes with single-color matte dinnerware
and two kinds of background environments: plain tabletop and complex kitchen
backdrop. There are 735 scenes for training and 102 for evaluation. Each scene
contains 3-4 objects at random positions and is captured from 3 poses (for
tabletop scenes) or 2 poses (for kitchen backdrops).

Kitchen-Shiny. This dataset comprises scenes with textured shiny dinnerware.
Similar to Kitchen-Matte, the first half presents a plain tabletop, while the
latter has a complex kitchen background. There are 324 scenes for training and
56 for evaluation.

Figure 4 shows some samples drawn from our datasets, and more details on data
collection are provided in Appendix C.2.

Details on the Object-Centric Prior Learning. In our experiments, we first
train our model on synthetic room scenes with only one chair. This synthetic
data is easy to generate and scale up. We empirically find that despite the
domain gap, uOCF can learn category-agnostic object priors to generalize to
different object categories. The object priors learned from synthetic chairs
effectively apply to dinnerware in real kitchen scenes. Further experimental
evidence can be found in Appendix E.1, where we show that this technique is
effective even when object priors are learned from simple CLEVR shapes
(monochromatic primitives like cubes and spheres) (Johnson et al., 2017).

Qualitative Metrics. We report the PSNR, SSIM, and LPIPS metrics for novel
view synthesis. For scene segmentation, we use three variants of the Adjusted
Rand Index (ARI): the conventional ARI (calculated on all input image pixels),
the Foreground ARI (FG-ARI, calculated on foreground input image pixels), and
the Novel View ARI (NV-ARI, calculated on novel view pixels). All scores are
computed on images of resolution $128\times 128$.

Baselines. We compare our method with uORF (Yu et al., 2022), BO-QSA (Jia et
al., 2023), and COLF (Smith et al., 2023). During testing, the model uses a
single image with known camera intrinsics as input and outputs the
reconstruction/segmentation results from target poses. We increase the latent
dimensions and training iterations for the baselines for fair comparisons. By
default, we set the number of foreground object queries to $K=4$ across all
methods and discuss its effect in Appendix E.1. Further details on
experimental configuration are provided in Appendix C.3.

Table 3: Quantitative scene manipulation results on Room-Texture. Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
Object Translation
uORF |  23.65 |  0.654 |  0.284
BO-QSA |  25.21 |  0.700 |  0.226
uOCF-N (ours) |  27.70 |  0.773 |  0.156
uOCF-P (ours) |  27.66 |  0.774 |  0.156
Object Removal
uORF |  23.81 |  0.664 |  0.282
BO-QSA |  24.58 |  0.698 |  0.247
uOCF-N (ours) |  29.20 |  0.803 |  0.132
uOCF-P (ours) |  28.99 |  0.802 |  0.136
Figure 7: Qualitative results of single-image 3D scene manipulation. The three
bowls’ positions are shifted for object translation.  Figure 8: Zero-shot
generalization results. We load the model trained on one dataset and test it
on an image from another more complex dataset with a fast test-time
optimization. First/second/third row: scene/background/object reconstruction.

###  4.1 Unsupervised Object Segmentation in 3D

We begin our evaluation by object segmentation. We render a density map
$\mathbf{d}^{i}$ for each latent $i$ and assign each pixel $p$ a segmentation
label $s_{p}=\mathrm{arg\,max}_{i=0}^{K}\mathbf{d}_{p}^{i}$.

The results are presented in Table 1 and Figure 5. On synthetic Room-Texture
scenes, our method outperforms existing methods in all metrics. Notably, no
baseline method can produce reasonable segmentation results in real-world
Kitchen-Shiny scenes. Specifically, uORF binds all objects to the background,
resulting in empty object segmentation; BO-QSA fails to distinguish different
object instances; COLF produces meaningless results on novel views as light
fields do not guarantee multi-view consistency. In contrast, thanks to object-
centric modeling, uOCF better distinguishes the foreground objects and
consistently produces satisfactory scene segmentation results, demonstrating
its effectiveness in object-centric representation learning. Moreover, uOCF
can handle scenes in which objects occlude each other. Please see Appendix E.4
for more results.

###  4.2 Novel View Synthesis

Evaluation then proceeds to novel view synthesis. For each test scene, we use
a single image as input and the remaining images as references.

The results are shown in Table 2 and Figures 6, 19, 20. Our method
significantly surpasses baselines across all metrics. Importantly, while
previous methods often fail to distinguish foreground objects and thus produce
messy results, our approach consistently produces high-fidelity scene
reconstruction and novel view synthesis results.

###  4.3 Scene Manipulation in 3D

This section evaluates uOCF’s ability in scene manipulation. Being able to
infer objects’ positions, uOCF readily supports the following scene editing
functions: 1) object translation, achieved by modifying the object’s position
sent to the decoder, and 2) object removal, achieved by excluding certain
objects during compositional rendering. In the following, we first
quantitatively evaluate uOCF on the Room-Texture dataset, followed by
qualitative evaluations on the real-world Kitchen-Shiny dataset.

Quantitative Evaluation. On the Room-Texture dataset, we randomly select an
object within the scene and relatively shift its position (object translation)
or remove it (object removal). We render images from four viewpoints for every
scene, along with the ground truth mask of the altered object. During testing,
we determine the object to manipulate by selecting the object with the highest
IoU score with the ground truth mask. Note that while existing methods are
limited to relative position adjustment, our approach uniquely allows absolute
translations due to the disentanglement of object positions and
representations. As shown in Table 3, uOCF outperforms baselines across all
metrics, justifying its effectiveness in scene manipulation.

Qualitative Evaluation. Figure 7 offers the qualitative scene manipulation
results on the Kitchen-Shiny dataset. As shown, uORF merges all objects into
the background, making scene manipulation unfeasible (hence, the manipulation
results are identical to the original reconstructions); BO-QSA fails to
distinguish foreground objects correctly, resulting in messy manipulation
results (see Figure 17 for a visual breakdown of each object latent). In
contrast, uOCF delivers much more reasonable and visually satisfactory results
than previous methods. Please refer to the supplementary video for additional
visualization results.

###  4.4 Generalization Analysis

Finally, we explore the zero-shot generalizability of uOCF, where we adapt the
model trained on Kitchen-Matte to an unseen scene in Kitchen-Shiny through a
fast test-time optimization. We also try to adapt the model trained on Room-
Texture to an unseen scene from the indoor scan dataset, HM3D (Ramakrishnan et
al., 2021). Note that only one image is used instead of multi-view images. As
shown in Figure 8, whereas previous methods struggle to generalize to unseen
objects or fail to separate foreground objects from the background, uOCF
demonstrates outstanding generalizability, only requiring a fast test-time
optimization to generalize from one dataset to new, more complex objects.

We also explore the few-shot setting, where the model is trained with
10/50/100 scenes instead of the whole dataset. Please refer to Appendix E.3
for results and analysis.

##  5 Conclusion

In this paper, we study the importance of object-centric modeling for
unsupervised 3D object discovery. We propose the unsupervised discovery of
Object-Centric neural Fields (uOCF) to instantiate this concept. To evaluate
our approach, we collect three challenging new datasets, each with scenes
containing objects of multiple categories set against complex backgrounds. Our
results indicate that our object-centric design and the object-centric prior
learning can substantially improve the systematic generalizability of
unsupervised object discovery.

#### Acknowledgments.

This work is in part supported by NSF RI #2211258, ONR MURI N00014-22-1-2740,
ONR YIP N00014-24-1-2117, the Samsung Global Research Outreach (GRO) program,
Amazon, and Google.

## References

  * Barron et al. (2021) Barron, J. T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., and Srinivasan, P. P.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _ICCV_ , 2021.
  * Biza et al. (2023) Biza, O., van Steenkiste, S., Sajjadi, M. S., Elsayed, G. F., Mahendran, A., and Kipf, T.  Invariant slot attention: Object discovery with slot-centric reference frames.  In _ICML_ , 2023.
  * Burgess et al. (2019) Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A.  Monet: Unsupervised scene decomposition and representation.  _arXiv:1901.11390_ , 2019.
  * Cen et al. (2023a) Cen, J., Fang, J., Yang, C., Xie, L., Zhang, X., Shen, W., and Tian, Q.  Segment any 3d gaussians.  _arXiv:2312.00860_ , 2023a.
  * Cen et al. (2023b) Cen, J., Zhou, Z., Fang, J., Shen, W., Xie, L., Zhang, X., and Tian, Q.  Segment anything in 3d with nerfs.  In _NeurIPS_ , 2023b.
  * Chattopadhyay et al. (2020) Chattopadhyay, P., Balaji, Y., and Hoffman, J.  Learning to balance specificity and invariance for in and out of domain generalization.  In _ECCV_ , 2020.
  * Chen et al. (2020) Chen, C., Deng, F., and Ahn, S.  Learning to infer 3d object models from images.  _arXiv:2006.06130_ , 2020.
  * Chen et al. (2022) Chen, H., Venkatesh, R., Friedman, Y., Wu, J., Tenenbaum, J. B., Yamins, D. L., and Bear, D. M.  Unsupervised segmentation in real-world images via spelke object inference.  In _ECCV_ , 2022.
  * Collins et al. (2022) Collins, J., Goel, S., Deng, K., Luthra, A., Xu, L., Gundogdu, E., Zhang, X., Yago Vicente, T. F., Dideriksen, T., Arora, H., Guillaumin, M., and Malik, J.  Abo: Dataset and benchmarks for real-world 3d object understanding.  In _CVPR_ , 2022.
  * Crawford & Pineau (2019) Crawford, E. and Pineau, J.  Spatially invariant unsupervised object detection with convolutional neural networks.  In _AAAI_ , 2019.
  * Deng et al. (2022) Deng, W., Gould, S., and Zheng, L.  On the strong correlation between model invariance and generalization.  In _NeurIPS_ , 2022.
  * Didolkar et al. (2023) Didolkar, A., Goyal, A., and Bengio, Y.  Cycle consistency driven object discovery.  _arXiv:2306.02204_ , 2023.
  * Engelcke et al. (2019) Engelcke, M., Kosiorek, A. R., Jones, O. P., and Posner, I.  Genesis: Generative scene inference and sampling with object-centric latent representations.  _arXiv:1907.13052_ , 2019.
  * Eslami et al. (2016) Eslami, S., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Kavukcuoglu, K., and Hinton, G. E.  Attend, infer, repeat: Fast scene understanding with generative models.  In _NeurIPS_ , 2016.
  * Eslami et al. (2018) Eslami, S. A., Rezende, D. J., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., et al.  Neural scene representation and rendering.  _Science_ , 2018.
  * Fan et al. (2022) Fan, Z., Wang, P., Jiang, Y., Gong, X., Xu, D., and Wang, Z.  Nerf-sos: Any-view self-supervised object segmentation on complex scenes.  _arXiv2209.08776_ , 2022.
  * Ge et al. (2023) Ge, Y., Yu, H.-X., Zhao, C., Guo, Y., Huang, X., Ren, L., Itti, L., and Wu, J.  3d copy-paste: Physically plausible object insertion for monocular 3d detection.  _Neurips_ , 2023.
  * Grauman & Darrell (2006) Grauman, K. and Darrell, T.  Unsupervised learning of categories from sets of partially matching image features.  In _CVPR_ , 2006.
  * Greff et al. (2016) Greff, K., Rasmus, A., Berglund, M., Hao, T. H., Schmidhuber, J., and Valpola, H.  Tagger: Deep unsupervised perceptual grouping.  In _NeurIPS_ , 2016.
  * Greff et al. (2017) Greff, K., Van Steenkiste, S., and Schmidhuber, J.  Neural expectation maximization.  In _NeurIPS_ , 2017.
  * Greff et al. (2019) Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A.  Multi-object representation learning with iterative variational inference.  In _ICML_ , 2019.
  * Jia et al. (2023) Jia, B., Liu, Y., and Huang, S.  Improving object-centric learning with query optimization.  In _ICLR_ , 2023.
  * Jiang et al. (2020) Jiang, J., Janghorbani, S., de Melo, G., and Ahn, S.  Scalor: Generative world models with scalable object representations.  In _ICLR_ , 2020.
  * Johnson et al. (2016) Johnson, J., Alahi, A., and Fei-Fei, L.  Perceptual losses for real-time style transfer and super-resolution.  In _ECCV_ , 2016.
  * Johnson et al. (2017) Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _CVPR_ , 2017.
  * Joulin et al. (2010) Joulin, A., Bach, F., and Ponce, J.  Discriminative clustering for image co-segmentation.  In _CVPR_ , 2010.
  * Kosiorek et al. (2018) Kosiorek, A. R., Kim, H., Posner, I., and Teh, Y. W.  Sequential attend, infer, repeat: Generative modelling of moving objects.  In _NeurIPS_ , 2018.
  * Kundu et al. (2018) Kundu, A., Li, Y., and Rehg, J. M.  3d-rcnn: Instance-level 3d object reconstruction via render-and-compare.  In _CVPR_ , 2018.
  * Li et al. (2019) Li, B., Sun, Z., Li, Q., Wu, Y., and Hu, A.  Group-wise deep object co-segmentation with co-attention recurrent neural network.  In _CVPR_ , 2019.
  * Liang et al. (2022) Liang, S., Liu, Y., Wu, S., Tai, Y.-W., and Tang, C.-K.  Onerf: Unsupervised 3d object segmentation from multiple views.  _arXiv:2211.12038_ , 2022.
  * Lin et al. (2020) Lin, Z., Wu, Y.-F., Peri, S. V., Sun, W., Singh, G., Deng, F., Jiang, J., and Ahn, S.  Space: Unsupervised object-oriented scene representation via spatial attention and decomposition.  In _ICLR_ , 2020.
  * Locatello et al. (2020) Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T.  Object-centric learning with slot attention.  In _NeurIPS_ , 2020.
  * Mildenhall et al. (2020) Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Monnier et al. (2021) Monnier, T., Vincent, E., Ponce, J., and Aubry, M.  Unsupervised layered image decomposition into object prototypes.  In _ICCV_ , 2021.
  * Müller et al. (2022) Müller, N., Simonelli, A., Porzi, L., Bulo, S. R., Nießner, M., and Kontschieder, P.  Autorf: Learning 3d object radiance fields from single view observations.  In _CVPR_ , 2022.
  * Nguyen-Phuoc et al. (2020) Nguyen-Phuoc, T. H., Richardt, C., Mai, L., Yang, Y., and Mitra, N.  Blockgan: Learning 3d object-aware scene representations from unlabelled images.  In _NeurIPS_ , 2020.
  * Niemeyer & Geiger (2021) Niemeyer, M. and Geiger, A.  Giraffe: Representing scenes as compositional generative neural feature fields.  In _CVPR_ , 2021.
  * Oquab et al. (2023) Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.  Dinov2: Learning robust visual features without supervision.  _arXiv:2304.07193_ , 2023.
  * Ost et al. (2021) Ost, J., Mannan, F., Thuerey, N., Knodt, J., and Heide, F.  Neural scene graphs for dynamic scenes.  In _CVPR_ , 2021.
  * Park et al. (2019) Park, J. J., Florence, P., Straub, J., Newcombe, R., and Lovegrove, S.  Deepsdf: Learning continuous signed distance functions for shape representation.  In _CVPR_ , 2019.
  * Ramakrishnan et al. (2021) Ramakrishnan, S. K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A. X., et al.  Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.  _arXiv:2109.08238_ , 2021.
  * Ranftl et al. (2022) Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., and Koltun, V.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _TPAMI_ , 2022.
  * Russell et al. (2006) Russell, B. C., Freeman, W. T., Efros, A. A., Sivic, J., and Zisserman, A.  Using multiple segmentations to discover objects and their extent in image collections.  In _CVPR_ , 2006.
  * Sajjadi et al. (2022) Sajjadi, M. S., Duckworth, D., Mahendran, A., van Steenkiste, S., Pavetic, F., Lucic, M., Guibas, L. J., Greff, K., and Kipf, T.  Object scene representation transformer.  In _NeurIPS_ , 2022.
  * Sitzmann et al. (2019) Sitzmann, V., Zollhöfer, M., and Wetzstein, G.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In _NeurIPS_ , 2019.
  * Sivic et al. (2005) Sivic, J., Russell, B. C., Efros, A. A., Zisserman, A., and Freeman, W. T.  Discovering objects and their location in images.  In _ICCV_ , 2005.
  * Smith et al. (2023) Smith, C., Yu, H.-X., Zakharov, S., Durand, F., Tenenbaum, J. B., Wu, J., and Sitzmann, V.  Unsupervised discovery and composition of object light fields.  _TMLR_ , 2023.
  * Spelke (1990) Spelke, E. S.  Principles of object perception.  _Cognitive science_ , 1990.
  * Stelzner et al. (2021) Stelzner, K., Kersting, K., and Kosiorek, A. R.  Decomposing 3d scenes into objects via unsupervised volume segmentation.  _arXiv:2104.01148_ , 2021.
  * Tewari et al. (2020) Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin-Brualla, R., Simon, T., Saragih, J., Nießner, M., et al.  State of the art on neural rendering.  _CGF_ , 2020.
  * Vo et al. (2020) Vo, H. V., Pérez, P., and Ponce, J.  Toward unsupervised, multi-object discovery in large-scale image collections.  In _ECCV_ , 2020.
  * Wang et al. (2023a) Wang, G., Chen, Z., Loy, C. C., and Liu, Z.  Sparsenerf: Distilling depth ranking for few-shot novel view synthesis.  In _ICCV_ , 2023a.
  * Wang et al. (2019) Wang, H., Sridhar, S., Huang, J., Valentin, J., Song, S., and Guibas, L. J.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In _CVPR_ , 2019.
  * Wang et al. (2023b) Wang, Q., Wang, Y., Birsak, M., and Wonka, P.  Blobgan-3d: A spatially-disentangled 3d-aware generative model for indoor scenes.  _arXiv:2303.14706_ , 2023b.
  * Wu et al. (2017) Wu, J., Tenenbaum, J. B., and Kohli, P.  Neural scene de-rendering.  In _CVPR_ , 2017.
  * Wu et al. (2022) Wu, Q., Liu, X., Chen, Y., Li, K., Zheng, C., Cai, J., and Zheng, J.  Object-compositional neural implicit surfaces.  In _ECCV_ , 2022.
  * Xie et al. (2022) Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., Sitzmann, V., and Sridhar, S.  Neural fields in visual computing and beyond.  _CGF_ , 2022.
  * Yang et al. (2021) Yang, B., Zhang, Y., Xu, Y., Li, Y., Zhou, H., Bao, H., Zhang, G., and Cui, Z.  Learning object-compositional neural radiance field for editable scene rendering.  In _CVPR_ , 2021.
  * Yang et al. (2023) Yang, J., Pavone, M., and Wang, Y.  Freenerf: Improving few-shot neural rendering with free frequency regularization.  In _CVPR_ , 2023.
  * Yao et al. (2018) Yao, S., Hsu, T. M. H., Zhu, J.-Y., Wu, J., Torralba, A., Freeman, W. T., and Tenenbaum, J. B.  3d-aware scene manipulation via inverse graphics.  In _NeurIPS_ , 2018.
  * Yu et al. (2021) Yu, A., Ye, V., Tancik, M., and Kanazawa, A.  pixelnerf: Neural radiance fields from one or few images.  In _CVPR_ , 2021.
  * Yu et al. (2022) Yu, H.-X., Guibas, L. J., and Wu, J.  Unsupervised discovery of object radiance fields.  In _ICLR_ , 2022.
  * Zhang (2019) Zhang, R.  Making convolutional networks shift-invariant again.  In _ICML_ , 2019.

##  Appendix A Supplementary Document Overview

This supplementary document is structured as follows: We begin with the proof
of concept in Appendix B and provide the implementation details in Appendix C.
Then, we discuss the limitations of our approach in Appendix D and present
additional experiments in Appendix E. For more results, please refer to our
project page: https://red-fairy.github.io/uOCF/.

##  Appendix B Proof of Concept

We conduct a toy experiment (Figure 9) to demonstrate that our model has
successfully learned object position, rotation, and scale. In this experiment,
we begin with two images (input 1 and input 2) of a chair placed at the
scene’s center, exhibiting different sizes (on the left) or rotation angles
(on the right), all captured from the same viewing direction.

We extract the object latents from these images, interpolate them, and then
send the interpolated latents to the decoder. As shown between the two input
images, we observe a smooth transition in object size and rotation, indicating
that the latent representation has effectively captured the scale and rotation
of objects.

In the second row, we placed the chairs in different positions. As shown on
the right, we obtained a smooth transition again, proving that our model could
disentangle object positions from the latent representation.

Figure 9: Proof of concept. We demonstrate that uOCF has effectively learned
objects’ scale and orientation along with the translation-invariant object
representation by interpolating the representation of two identical objects
with different orientations and scales to obtain transitional results.

##  Appendix C Implementation

Figure 10: (a) Architecture of our encoder module. (b) Object-centric
sampling: We drop the samples distant from the predicted object position for
efficient sampling.. Figure 11: Lifting 2D position
$\mathbf{p}_{i}^{\mathrm{img}}$ to 3D position $\mathbf{p}_{i}^{\mathrm{wd}}$
with (left) and without (right) ground plane assumption.

###  C.1 Model Architecture

Encoder. Our encoder module consists of a frozen DINO encoder and two
convolutional layers. We illustrate its architecture in Figure 10(a).

Latent Extraction Module. While motivated by the background-aware slot
attention module proposed by (Yu et al., 2022), our latent inference module
exhibits three key differences: (1) The object queries are initialized with
learnable embeddings instead of being sampled from learnable Gaussians, which
enhances training stability; (2) We jointly extract object positions and their
latent representations and add object-specific positional encoding to utilize
the extracted position information; (3) We remove the Gated Recurrent Unit
(GRU) and replace it with the transformer architecture to smooth the gradient
flow.

When lifting the object image position $\mathbf{p}_{i}^{\mathrm{img}}$ to the
world coordinates, we consider two scenarios: (1) for objects on a known
shared plane (e.g., using plane detection (Ge et al., 2023)), a direct ray
extension (extend the ray between the camera center and
$\mathbf{p}_{i}^{\mathrm{img}}$ to intersect the ground plane) is used (Figure
11, left); (2) we extend the rays by depth $d\cdot s_{i}$, where $d$ is the
distance between the camera and the scene center and $\\{s_{i}\\}_{i=1}^{K}$
are scaling terms predicted by a linear layer using the camera parameters and
object latent as input (Figure 11, right). The two variants are denoted as
uOCF-P and uOCF-N, respectively, and uOCF refers to uOCF-P by default.

###  C.2 Data Collection

This section introduces the details of our datasets.

Room-Texture. In Room-Texture, objects are chosen from 324 ABO objects
(Collins et al., 2022) from the “armchair” category. The single-object subset
contains four scenes for each object instance, resulting in 1296 scenes in
total. The multiple-object subset includes 5,000 scenes for training and 100
for evaluation, with each scene containing 2-4 objects set against a
background randomly chosen from a collection of floor textures. Each scene is
rendered from 4 directions toward the center.

Kitchen-Matte. In Kitchen-Matte, objects are diffuse and have no texture. The
dataset comprises 16 objects and 6 tablecloths in total. We captured 3 images
for each tabletop scene and 2 for each kitchen scene. This dataset contains
735 scenes for training and 102 for evaluation, each containing 3-4 objects.
We calibrate the cameras using the OpenCV library.

Kitchen-Shiny. In Kitchen-Shiny, objects are specular, and the lighting is
more complex. The dataset comprises 12 objects and 6 tablecloths, and the
other settings are identical to Kitchen-Matte. This dataset contains 324
scenes for training and 56 for evaluation, each containing 4 objects.

###  C.3 Training Configuration

This section discusses the training configuration of uOCF.

We employ Mip-NeRF (Barron et al., 2021) as our NeRF backbone and estimate the
depth maps by MiDaS (Ranftl et al., 2022). An Adam optimizer with default
hyper-parameters and an exponential decay scheduler is used across all
experiments. The initial learning rate is $0.0003$ for the first stage and
$0.00015$ for the second stage. Loss weights are set to
$\lambda_{\text{perc}}=0.006,\lambda_{\text{depth}}=1.5,~{}\text{and}~{}\lambda_{\text{occ}}=0.1$.
The position update momentum $m$ is set to $0.5$, and the latent inference
module lasts $T=6$ iterations.

Coarse-To-Fine Progressive Training. We employ a coarse-to-fine strategy in
our second training stage to facilitate training at higher resolutions.
Reference images are downsampled to a lower resolution ($64\times 64$) during
the coarse training stage and replaced by image patches with the same size as
the low-resolution images randomly cropped from the high-resolution
($128\times 128$) input images during the fine training stage.

Locality Constraint and Object-Centric Sampling. We employ the locality
constraint (a bounding box for foreground objects in the world coordinate)
proposed by Yu et al. (2022) in both training stages but only adopt it before
starting object-centric sampling. The number of samples along each ray before
and after starting object-centric sampling is set to 64 and 256, respectively.
We provide an illustration of our object-centric sampling strategy in Figure
10(b).

Training Configuration on Room-Texture. During stage 1, we train the model for
100 epochs directly on images of resolution $128\times 128$. We start with the
reconstruction loss only, add the perceptual $10^{\text{th}}$ epoch, and start
the object-centric sampling at the $20^{\text{th}}$ epoch. During stage 2, we
train the model for 60 epochs on the coarse stage and another 60 on the fine
stage. We start with the reconstruction loss only, add the perceptual loss at
the $10^{\text{th}}$ epoch, and start the object-centric sampling from the
$20^{\text{th}}$ epoch.

Training Configuration on Kitchen-Matte and Kitchen-Shiny. Both kitchen
datasets share the same training configuration with Room-Texture in stage 1.
During stage 2, we train the model for 750 epochs, where the fine stage starts
at the $250^{\text{th}}$ epoch. We add the perceptual loss at the
$50^{\text{th}}$ epoch and start the object-centric sampling from the
$150^{\text{th}}$ epoch.

##  Appendix D Limitations Analysis

Limitation on Reconstruction Quality. Scene-level generalizable NeRFs (Yu et
al., 2021; Sajjadi et al., 2022; Yu et al., 2022) commonly face challenges in
accurately reconstructing detailed object textures. To address this
limitation, we adopted the MipNeRF (Barron et al., 2021) backbone and included
the depth and occlusion regularizers. However, despite these enhancements, our
approach still encounters difficulty capturing extremely high-frequency
details. As shown in Figure 12(a), our method fails to replicate the mug’s
detailed texture. Future research may benefit from exploring more advanced
NeRF backbones to further improve texture detail reconstruction.

Failure in Position Prediction. Our two-stage training pipeline, despite its
robustness in many situations, is not immune to errors, particularly in object
position prediction. Due to the occlusion between objects, using the
attention-weighted mean for determining object positions can sometimes lead to
inaccuracies. Although a bias term can rectify this in most instances (Figure
6), discrepancies persist under a few conditions, as depicted in Figure 12(b).

Figure 12: Failure case visualizations. Our method may fail to reconstruct
intricate object texture or predict biased object position.

##  Appendix E Additional Experiments

###  E.1 Ablation Studies

Training Pipeline. We first justify the efficacy of our two-stage object-
centric prior learning pipeline. Heuristically, the first stage involves
learning the fundamentals of object-centric NeRFs, such as the physical
coherence (Spelke, 1990) property–a crucial aspect in unsupervised
segmentation (Chen et al., 2022). The model then utilizes the learned object
prior in the subsequent training stage on complex real-world scenes.

We note that although the objects in the two datasets might share a large
domain gap, the learned object priors are transferable. The main paper already
presents the model’s adaptability from synthetic chairs to actual dinnerware.
To further validate this concept, we consider using the simple CLEVR shapes
(balls, cubes, and cylinders) for object prior learning, which maintains
fairly good results, as shown in Table 4.

As illustrated in Table 4 and Figure 13, omitting the object prior learning
stage leads to the model incorrectly associating some foreground objects with
the background, yielding poorer results. Excluding the depth and occlusion
losses significantly hampers the visual quality, resulting in a performance
decline. We also present results from eliminating the object-centric sampling
strategy, which slightly impacts the overall reconstruction quality. We also
tried learning object prior from multi-object Room-Texture, which provides
even better results, possibly because the two datasets share the maximum
number of objects in the scene.

Table 4: Ablation study for the training pipeline on Kitchen-Shiny. Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
w/o object-centric prior learning | 26.81 | 0.806 | 0.125
Learn object prior from CLEVR | 27.96 | 0.840 | 0.074
w/o object-centric sampling | 27.31 | 0.852 | 0.072
w/o $\ell_{\text{depth}}$ and $\ell_{occ}$ | 26.79 | 0.819 | 0.081
Learn object prior from single-object Room-Texture (uOCF) | 28.58 | 0.862 | 0.049
Learn object prior from multi-object Room-Texture | 29.58 | 0.878 | 0.037

Encoder Design. We ablate the encoder design to prove that simply substituting
the shallow encoder in previous methods is inadequate for unsupervised object
discovery in complex real-world scenes. As shown by the qualitative results in
Table 5 and quantitative results in 17, simply integrating DINO fails to
address the limitations of existing methods in complex real-world scenes.

Table 5: Ablation study for different encoder designs on Kitchen-Shiny. uORF-DINO modifies the standard uORF by replacing its shallow U-Net encoder with the DINO encoder. Method | Novel view synthesis
---|---
PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
uORF |  19.23 |  0.602 |  0.336
uORF-DINO |  21.06 |  0.662 |  0.282
uOCF (ours) |  28.58 |  0.862 |  0.049
Figure 13: Qualitative ablation study for the training pipeline on Kitchen-
Shiny. Removing either object-centric training strategies or removing the
regularizations significantly impairs the visual quality and may even make the
model fail to discover all objects in the scene. Figure 14: Qualitative
results of uOCF on scenes with larger object queries $K$. The order of the
object reconstructions is rearranged for better visualization.

###  E.2 Discussion on the Number of Object Queries

The above experiments are conducted on scenes with 2-4 objects for training
and evaluation, with the number of object queries $K$ (i.e., the maximum
number of objects supported) all set to 4. This section further explores the
effect of varying $K$, particularly exploring scenarios where $K$ surpasses
the upper limit of objects present in the scene.

The quantitative results in Table 6 and qualitative results in Figure 14
indicate the robustness of our framework to variations in the number of object
queries $K$. Notably, when $K$ is set higher than the actual count of objects
in a scene, certain object queries yield empty reconstructions. This
phenomenon highlights uOCF’s adaptability to complex scenes, i.e., adjusting
its output in response to the excess capacity represented by the larger $K$
value.

Table 6: Quantitative novel view synthesis result Kitchen-Shiny with a larger number of object queries $K$. Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
$K=4$ |  28.58 |  0.862 |  0.049
$K=5$ |  28.28 |  0.846 |  0.059
$K=6$ |  28.04 |  0.848 |  0.058
Figure 15: Ablation study on the few-shot settings, i.e., decrease the number
of scenes available during training. The order of the object reconstructions
is rearranged for better visualization, and the LPIPS values are computed
between images of resolution $256\times 256$. Figure 16: Zero-shot inference
on the HM3D (Ramakrishnan et al., 2021) dataset. Our method can discover and
segment the object instances belonging to the categories it has seen during
training from the scene and deliver plausible reconstruction results.

###  E.3 Generalizability: Few-Shot and Zero-shot Object Discovery

Extending the discussions from Sec. 4.4, this section delves into the
generalizability of uOCF. In contrast to previous experiments that utilized
the full Kitchen-Shiny dataset for training, we now consider a more
challenging scenario, where the model is trained with a limited number of
scenes—specifically, 10, 50, or 100—and then tested on unseen scenes.

Figure 15 illustrates a notable trend: the reconstruction quality of uOCF
deteriorates in correlation with the reduction in the number of training
scenes. Particularly, with an extremely limited training set, the model
struggles to differentiate objects from the background. This underscores the
significance of the dataset size for effective unsupervised object discovery.

Sec. 4.4 also provides experiments on adapting from Room-Texture to HM3D (a
scanned indoor scene dataset) with test-time optimization. To further
understand our approach, we provide the direct inference results (i.e.,
without test-time optimization) in Figure 16. As shown, our method can
accurately identify and segment the chair instances in the scene and deliver
plausible reconstruction results.

###  E.4 Additional Qualitative Results

Scene Decomposition Visualizations. We first present the scene decomposition
results for our scene manipulation experiment (Sec. 4.3) in Figure 17.
Notably, uORF (Yu et al., 2022) puts all objects within the background,
whereas BO-QSA (Jia et al., 2023) binds the same object to all queries,
resulting in identical foreground reconstruction. Contrasting these, uOCF
accurately differentiates between the foreground objects and the background.

Scene Segmentation Visualizations. Afterward, we provide scene segmentation
results on the kitchen datasets in Figure 18. Unlike compared methods that
yield cluttered results, uOCF consistently yields high-fidelity segmentation
results.

Novel View Synthesis Visualizations. Finally, we offer more qualitative
results for novel view synthesis in Figures 18, 19, and 20. Our method
produces much better results than compared methods regarding visual quality.

Figure 17: Scene decomposition reconstruction comparison results on Kitchen-
Shiny. We rearranged the order of the object reconstructions for better
visualization. Figure 18: Additional qualitative comparison results on the
Room-Texture dataset.  Figure 19: Additional qualitative results on the
Kitchen-Matte dataset. Figure 20: Additional qualitative comparison results on
the Kitchen-Shiny dataset.


11institutetext: 1Pattern Recognition Lab, Computer Science, Friedrich-
Alexander-Universität Erlangen-Nürnberg
2Department of Oral and Maxillofacial Surgery, University Hospital Erlangen,
Friedrich-Alexander-Universität Erlangen-Nürnberg
3Erlangen Graduate School in Advanced Optical Technologies (SAOT), Friedrich-
Alexander-Universität Erlangen-Nürnberg
4Department of Oral and Maxillofacial Surgery, University Medical Center
Hamburg-Eppendorf
5First Department of Internal Medicine, University Hospital Mainz, Johannes
Gutenberg-Universität Mainz

# Motion Artifact Detection in Confocal Laser Endomicroscopy Images

Maike Stoeve1 Marc Aubreville1 Nicolai Oetter2,3 Christian Knipfer4,3 Helmut
Neumann5,3 Florian Stelzle2,3 Andreas Maier1,3<EMAIL_ADDRESS>
###### ?abstractname?

Confocal Laser Endomicroscopy (CLE), an optical imaging technique allowing
non-invasive examination of the mucosa on a (sub)- cellular level, has proven
to be a valuable diagnostic tool in gastroenterology and shows promising
results in various anatomical regions including the oral cavity. Recently, the
feasibility of automatic carcinoma detection for CLE images of sufficient
quality was shown. However, in real world data sets a high amount of CLE
images is corrupted by artifacts. Amongst the most prevalent artifact types
are motion-induced image deteriorations. In the scope of this work,
algorithmic approaches for the automatic detection of motion artifact-tainted
image regions were developed. Hence, this work provides an important step
towards clinical applicability of automatic carcinoma detection. Both,
conventional machine learning and novel, deep learning-based approaches were
assessed. The deep learning-based approach outperforms the conventional
approaches, attaining an AUC of 0.90.

##  1 Introduction

With over 500,000 diagnosed cases each year, head and neck squamous cell
carcinoma (HNSCC) is considered the sixth most common cancer type worldwide
[1]. For diagnosis of HNSCC, invasive biopsy and subsequent histopathological
examination is applied as gold standard method [2]. As alternative, non-
invasive optical imaging technologies as narrow-band imaging, autofluorescence
imaging and confocal laser endomicroscopy (CLE) are gaining interest in
research [3]. Among those technologies, CLE has already proven to be a
valuable diagnostic tool in gastroenterology and has been successfully applied
to examine lesions in the oral cavity [4]. Both, the interpretation of CLE
images and the histopathological examination of tissue samples require
experience and proficiency and entail a subjective component [5]. Thus,
promising approaches using machine learning techniques for automatic carcinoma
detection based on CLE images were developed [6].

As the accuracy of these algorithms is highly affected by the occurrence of
artifacts, artifact-tainted images were excluded manually prior to training in
a time-consuming manual labeling step in all known approaches. In the scope of
this work, an automatic motion artifact detection pipeline was developed and
evaluated. Hence, this work provides the basis to integrate a fully automatic
motion artifact detection into existing carcinoma detection frameworks, an
important step towards clinical applicability.

###  1.1 Motion artifacts

A frequent cause of image impairment are motion artifacts. They can either be
caused by movements of the investigated anatomical structures or motion
induced by the physician. The proportion of motion artifact images compared to
good quality images is highly dependent on the experience of the physician
[7]. As shown in Fig. 1, two different manifestations can be observed. The
first manifestation is characterized by streaky patterns originating from the
repeated acquisition of the same, shifted image line. This pattern can be
observed, when the sum of the velocities of probe, organ and scanning is
approximately equal to zero. This requires an organ movement in the same
direction as the sampling pattern or a probe motion in the correspondingly
opposite direction. If a significant relative motion results from organ or
probe motion, the cells are stretched or blurred. This second manifestation is
hard to detect, as distinction between elongated cells is difficult without
the observation of adjacent frames. Finally, motion artifacts can impair the
whole image leading to a total loss of information or only influence parts of
the image still allowing a diagnosis based on untainted regions [7]. Due to
the meander-shaped optical sampling pattern, only whole image rows are
affected by motion artifacts.

?figurename? 1: CLE images containing (a) no artifacts or motion artifacts
manifesting in (b) stretched cells or (c) stripe patterns (region marked in
red).

##  2 Materials and methods

For the present work, 116 CLE sequences comprising 11,234 images from 12
patients and 4 sites within the oral cavity, namely the upper alveolar ridge,
the lower inner labium, the palatal region and the lesion site itself were
utilized. The CLE sequences were recorded at the department of Oral and
Maxillofacial Surgery of the University Hospital Erlangen by a standalone
probe-based CLE system (Cellvizio, Mauna Kea Technologies, Paris, France). The
obtained images are approximately of size 576$\times$578 pixels. The overall
image quality was assessed by an expert and artifact regions were annotated
manually within each image. Details on the data can be found in [6].

In the scope of this work, two different methodical approaches were
established for the detection of motion artifact-tainted CLE image regions.
The first approach uses conventional pattern recognition methods extracting
characteristic image features, whereas the second approach applies deep
learning strategies. For both approaches, images with low signal-to-noise
ratio are excluded. The remaining CLE images are converted to 8-bit integer
values after a quantile-based dynamic range compression following Aubreville
et al. [6] was performed.

###  2.1 Feature-based motion artifact detection

The feature-based approach consists of three steps: pre-processing, feature
extraction and classification. The pre-processing step is required due to the
unusual round shape of CLE images complicating the feature extraction process.
Jaremenko et al. circumvented this problem by dividing the image in
overlapping, square patches and concatenating the information of all patches
for the classification of the whole image [8]. As motion artifacts always
cover the whole width of an image, slices with a width of the maximum extent
of the CLE image in x-direction, a fixed height of 128 pixels and an overlap
of 30% (HOG) or 50% (corrAngle) are extracted. To allow the detection of
motion artifact-tainted slices of an image and differentiate them from
untainted slices of the same image, either Histogram of Oriented Gradient
(HOG) or angle of maximum correlation (corrAngle) features are extracted for
each slice. For classification, a random forest classifier (RF) and a support
vector machine (SVM) classifier were used. Undersampling of the majority class
is applied to deal with the class imbalance. Prior to training of the linear
SVM classifier, the feature vector is standardized. For evaluation purposes of
both classifiers, 5-fold cross-validation is used. The different slices of one
image are all assigned to the same fold.

####  2.1.1 Histogram of oriented gradients (HOG)

The Histogram of Oriented Gradient (HOG) feature descriptor of Dalal and
Triggs is frequently used in the field of object recognition [9]. Basically,
the occurrence of gradient directions is computed in a local image region
characterizing the local shape of an object. In contrast to the original
pipeline, no gamma and color normalization is used. A cell size of
32$\times$32 and block size of 64$\times$64 resulted in the best
classification performance. Due to the varying length of CLE image slices, the
obtained HOG feature vectors are of varying length. Hence, mean, standard
deviation, skewness and kurtosis are computed over all 9 bins of the feature
vector individually. The statistical properties for all bins are subsequently
concatenated to form a feature vector of a fixed length of 36.

####  2.1.2 Angle of Maximum Correlation (corrAngle)

The angle of maximum correlation feature (corrAngle) is designed under
consideration of the origin of the streaky patterns visible in most motion
artifact-tainted images. The direction of the relative motion accounts for the
angle $\theta$ characterizing the direction of the stripe pattern. To create a
feature vector describing the presence or absence of motion artifact patterns
in an image slice with length $L$, a centered reference segment of row $i$ of
the CLE image slice is extracted. For a set of angles equally distributed
between $\frac{\pi}{8}$ and $\frac{7}{8}\pi$ and a fixed radius $R$,
comparative segments are extracted with center at
$(\frac{L}{2}+R\sin(\theta),j+R+R\cos(\theta))$ using bilinear interpolation.
Then, the correlation coefficient of the reference segment and each
comparative segment is computed as measurement of similarity. The angle
responsible for the highest correlation coefficient is saved. This procedure
is repeated for each row despite the first and last $R$ rows. Finally, all
approximated motion angles are concatenated to form the final feature vector.
For a motion artifact-tainted image slice, the approximated angle is constant
(stripe pattern). In contrast, CLE image slices without motion artifacts show
high variations of the approximated motion angle over image rows.

###  2.2 Deep learning-based motion artifact detection (artiNet)

Building on the Inception v3 architecture of Szegedy [10] pretrained on
ImageNet, the deep learning-based detector of motion artifact-tainted CLE
image areas (artiNet) is build by inserting a 2d 1$\times$1 convolutional
layer after the eighth inception block followed by a column fusion and a
softmax layer as depicted in Fig. 2. Thus, the input representation for the
new layers still entails spatial information. The convolutional layer is used
to map the 17$\times$17$\times$768 dimensional input to the two output classes
motion artifact and good quality. The result is a 17$\times$17 grid of
predictions. To obtain predictions of motion artifact regions covering the
whole image width comparable to the slices of the conventional approach, a
column fusion layer extracts the maximum over the image width resulting in a
17$\times$2 representation fed to the final softmax layer.

?figurename? 2: Visualization of the artiNet for motion artifact detection in
CLE images based on an Inception v3 network [10] pre-trained on ImageNet.

Prior to training the grayscale images are transformed to RGB color
representation. Then, a centered image patch of 400$\times$400 pixels is
extracted and resized to 299$\times$299 pixels. Due to the slice-wise
detection of motion artifacts and available corresponding labels effectively
representing a fully convolutional approach with a network capacity according
to one with smaller patches, data augmentation seems less important. Within
the TensorFlow framework, the Inception network was fine-tuned in $2000$
training steps using the Adam optimizer with an initial learning rate of
$5\cdot 10^{-6}$. For the training of the new layers, a learning rate of
$5\cdot 10^{-5}$ was used. In each training step, a minibatch consisting of 25
randomly selected instances of both classes is processed to deal with the
class imbalance of the data set. For evaluation, a leave-one-patient-out
cross-validation was performed.

##  3 Results

The top result is obtained by the artiNet, where an AUC of 0.90 is achieved.
The best performance following the conventional pattern recognition pipeline
is achieved by the corrAngle feature and RF classification. For this
combination, an AUC of 0.85 is reached. The corrAngle-based motion detection
with RF performs significantly better than the RF and SVM predictors trained
on HOG, where approximately equal ROC curves with AUC values of 0.73 and 0.74
are achieved. In contrast to the RF classification performance, the results of
the SVM classifier trained on the corrAngle features are poor. For additional
insight into the performance of the artiNet, the comparison of predictions for
single image slices and respective labels is visualized in Fig. 4(b).

?figurename? 3: Results of motion artifact detection in CLE images.

(a) ROC curves visualizing the average cross-validation performance of the
motion artifact detection. (b) Example images showing slicewise predictions of
the artiNet compared to class labels.

##  4 Discussion

Due to the included patch extraction step of the artiNet, the image borders
are discarded. For motion artifact detection, the center of the slice is
representative as a motion induced deterioration covers the whole extent in
x-direction. Still, the information at the top and bottom of the image is
removed. Thus, motion artifacts only deteriorating the rejected areas can not
be detected. An improved performance is to be expected if the whole image is
used. As the manifestation of stretched cells is underrepresented in the data
set, a performance gap between the two possible manifestations of motion
artifacts, stripe patterns and stretched cells might occur. Moreover, the
performance of the proposed methods was only assessed on data of one clinical
team. Hence, additional training instances are required to obtain a robust
artifact detector.

##  ?refname?

  * [1] Blaszczak W, Barczak, W, Wegner A, et al.  Clinical value of monoclonal antibodies and tyrosine kinase inhibitors in the treatment of head and neck squamous cell carcinoma.  Med Oncol. 2017;34(4):60.
  * [2] Oetter N, Knipfer C, Rohde M, et al.  Development and validation of a classification and scoring system for the diagnosis of oral squamous cell carcinomas through confocal laser endomicroscopy.  J Transl Med. 2016;14(1):159.
  * [3] Nathan CAO, Kaskas NM, Ma X, et al.  Confocal laser endomicroscopy in the detection of head and neck precancerous lesions.  J Otolaryngol Head Neck Surg. 2014;151(1):73–80.
  * [4] Thong PS, Tandjung SS, Movania, MM, et al.  Toward real-time virtual biopsy of oral lesions using confocal laser endomicroscopy interfaced with embedded computing.  J Biomed Opt.2012;17(5):0560091–05600910.
  * [5] Neumann Helmut, Vieth M, Atreya R, et al.  Prospective evaluation of the learning curve of confocal laser endomicroscopy in patients with IBD.  Histol Histopathol.2011;26(7):867.
  * [6] Aubreville M, Knipfer C, Oetter N, et al.  Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning.  Sci Rep.2017;20(7):11979.
  * [7] Sharman MJ, Bacci B, Whittem T, et al.  In vivo confocal endomicroscopy of small intestinal mucosal morphology in dogs.  J Vet Intern Med.2013;27(6):1372–1378.
  * [8] Jaremenko C, Maier A, Steidl S, et al.  Classification of confocal laser endomicroscopic images of the oral cavity to distinguish pathological from healthy tissue.  Proc Bvm.2015; p. 479–485.
  * [9] Dalal N, Triggs B, et al.  Histograms of oriented gradients for human detection.  In: Proc IEEE Comput Soc Conf Comput Vis Pattern Rocognit. vol. 1 of CVPR ‘05.IEEE;2005. p. 886–893.
  * [10] Szegedy C, Vanhoucke V, Ioffe S, et al.  Rethinking the Inception Architecture for Computer Vision.  In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit;2016. p. 2818-2826.