# MagicPony: Learning Articulated 3D Animals in the Wild

Shangzhe Wu Ruining Li∗ Tomas Jakab∗ Christian Rupprecht Andrea Vedaldi
Visual Geometry Group, University of Oxford
{szwu, ruining, tomj, chrisr<EMAIL_ADDRESS>
3dmagicpony.github.io Equal contribution.

###### Abstract

We consider the problem of learning a function that can estimate the 3D shape,
articulation, viewpoint, texture, and lighting of an articulated animal like a
horse, given a single test image. We present a new method, dubbed MagicPony,
that learns this function purely from in-the-wild single-view images of the
object category, with minimal assumptions about the topology of deformation.
At its core is an implicit-explicit representation of articulated shape and
appearance, combining the strengths of neural fields and meshes. In order to
help the model understand an object’s shape and pose, we distil the knowledge
captured by an off-the-shelf self-supervised vision transformer and fuse it
into the 3D model. To overcome common local optima in viewpoint estimation, we
further introduce a new viewpoint sampling scheme that comes at no added
training cost. Compared to prior works, we show significant quantitative and
qualitative improvements on this challenging task. The model also demonstrates
excellent generalisation in reconstructing abstract drawings and artefacts,
despite the fact that it is only trained on real images.

Figure 1: Learning Articulated 3D Animals in the Wild. Our method trains on a
collection of single-view images of an animal category and produces a model
that can predict an articulated 3D shape of the instance from a single input
image, which can be animated and relit.  **footnotetext: Equal contribution.

##  1 Introduction

A model that can reconstruct the 3D shape of a deformable object from a single
image must know _a priori_ what the possible shapes and appearances of the
object are. Learning such a prior usually requires ad-hoc data acquisition
setups [2, 35, 36, 19], involving at least multiple cameras, and often laser
scanners, domes and other hardware, not to mention manual labour. This is
viable for specific objects such as humans that are of particular interest in
applications, but it is unlikely to scale to the long tail of objects that can
appear in natural images.

The alternative is to learn the necessary 3D priors from what is available in
abundance, namely 2D images. Unfortunately, for any given type of object, the
space of possible 3D reconstructions is characterised by rich and yet highly-
structured shape, pose and appearance variations across instances. This prior
must then be _learned while using it to reconstruct_ the available 2D data _in
3D_ , which is a major challenge. This is especially true when no other mode
of manual or automated supervision is given.

In this paper, we propose a novel approach to learning 3D models of
articulated object categories such as horses and birds with only _single-view_
input images for training, which we dub MagicPony. We leverage recent progress
in unsupervised representation learning, unsupervised image matching,
efficient implicit-explicit shape representations and neural rendering, to
devise a new auto-encoder architecture that reconstructs the 3D shape,
articulation and texture of each object instance from a single test image. For
training, we only require a 2D segmenter for the objects and a description of
the topology and symmetry of the 3D skeleton (_e.g_., the number of legs). We
_do not_ require a-priori knowledge of 3D shape models, keypoints, viewpoints
or any other 2D or 3D cue often used in prior work [21, 20, 14, 30]. From
this, we learn a function that, at test time, can estimate the shape and
texture of a new object from a single image, in a feed-forward manner. The
function exhibits remarkable generalisation properties, capable of
reconstructing objects in _abstract drawings_ , despite being trained on real
images only.

We identify a small number of key challenges that must be addressed to solve
this problem, and propose new effective solutions to each of them.

The first problem is viewpoint estimation. Given only raw 2D images of a given
object type, until a 3D model is available it is very difficult to reliably
assign the images to different aspects or viewpoints of the object. Prior
works have shown that this task can be simplified significantly if at least 2D
point correspondences between images are available [21, 20, 30]. To avoid
requiring keypoint supervision, we derive reasonable but noisy correspondences
by _fusing_ into the 3D model knowledge distilled from DINO-ViT [5], a self-
supervised visual transformer network (ViT) [27]. We add to that a new
efficient disambiguation scheme that explores multiple viewpoint assignment
hypotheses at essentially no cost, avoiding local optima that are caused by
greedily matching the noisy 2D correspondences.

The second problem is how to represent the 3D shape, appearance and
deformations of the object. Most prior works have used textured meshes [20,
14, 30, 31, 68, 72, 13], but these are difficult to optimise from scratch,
leading to problems that often require ad-hoc heuristics such as re-meshing
[72, 13]. The other and increasingly popular approach is to use a volumetric
representation such as a neural radiance field [41, 53, 3, 64]. These
representations can model complex shapes that evolve in a seamless manner
during training, including changing their topology. However, this modelling
freedom comes at the cost of over-parameterisation, which is particularly
problematic in monocular reconstruction and often leads to meaningless short-
cut solutions [64]. Furthermore, modelling articulation with a volumetric
representation is difficult. Deformations are naturally defined only for the
object’s surface and its interior and from the canonical/pose-free space to
the posed space. However, ray marching requires extending these
transformations around the object (which is not ‘physical’) and often needs to
invert them (which is hard [8]).

We address these issues by using a hybrid volumetric-mesh representation,
based on DMTet [57]. Shape and appearance are defined volumetrically in
canonical space, but a mesh is extracted on the fly for posing and rendering.
This sidesteps the challenges of using neural rendering directly, while
retaining most of its advantages and enabling the use of powerful shape
regularisers. We compare our method to prior work on two challenging
articulated object categories: horses and birds, and show that our method
obtains _significantly better_ quantitative and qualitative results while
using _significantly less_ supervision.

To summarise, we make the following contributions: (1) A new 3D object
learning framework that combines recent advances in unsupervised learning, 3D
representations and neural rendering, achieving better reconstruction results
with less supervision; (2) An effective mechanism for fusing self-supervised
features from DINO-ViT into the 3D model as a form of self-supervision; (3) An
efficient multi-hypothesis viewpoint prediction scheme that avoids local
optima in reconstruction with no additional cost; (4) Extensive quantitative
and qualitative evaluations, including demonstrating generalisation to
abstract drawings.

##  2 Related Work

Table 1: Related Work Overview on Weakly-supervised Learning of 3D Objects. Annotations: template shape, viewpoint, 2D keypoint, object mask, optical flow, video, 1coarse template shape from keypoints, 2camera estimated from keypoints using SfM, 3outputs texture flow, 4shape bases initialised from CMR. †UMR relies on part segmentations from SCOPS [18].  | Supervision | Output
---|---|---
Method |  |  |  |  |  |  | 3D | 2.5D | Motion | View | Texture
Unsup3D [70] |  |  |  |  |  |  |  | ✓ |  | ✓ | ✓
CSM [29] | ✓ |  |  | ✓ |  |  |  |  |  | ✓ |
A-CSM [28] | ✓ |  |  | ✓ |  |  |  |  | ✓ | ✓ |
CMR [20] |  (✓)1 |  (✓)2 | ✓ | ✓ |  |  | ✓ |  |  | ✓ |  (✓)3
U-CMR [14] | ✓ |  |  | ✓ |  |  | ✓ |  |  | ✓ | ✓
UMR† [31] |  |  |  | ✓ |  |  | ✓ |  |  | ✓ |  (✓)3
ACMR [30] |  (✓)4 |  (✓)2 | ✓ | ✓ |  |  | ✓ |  |  | ✓ |  (✓)3
DOVE [68] |  |  |  | ✓ | ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓
Ours |  |  |  | ✓ |  |  | ✓ |  | ✓ | ✓ | ✓

#### Weakly-supervised Learning of 3D Objects.

Most related to our work are weakly-supervised methods for single-image 3D
object reconstruction that learn from a collection of images or videos of an
object category [23, 20, 34, 22, 66, 17, 14, 31, 30, 69, 26, 25, 68]. Due to
the inherent ambiguity of the problem, these methods usually rely on heavy
geometric supervision (see Tab. 1) in addtion to object masks, such as 2D
keypoints [20, 30], template shapes [14, 29, 28]. UMR [31] forgoes the
requirement of keypoints and template shapes by leveraging weakly-supervised
object part segmentations (SCOPS [18]), but produces coarse symmetric shapes
similar to CMR [20]. DOVE [68] learns coarse articulated objects by exploiting
temporal information from videos with optical flow supervision. All these
methods require object masks for supervision, except Unsup3D [70] which
exploits bilateral symmetry for reconstructing roughly frontal objects, like
faces, and UNICORN [42] which uses a progressive conditioning strategy with a
heavily constrained bottleneck, leading to coarse reconstructions. Another
emerging paradigm is to leverage generative models [46, 56, 7, 50] that
encourages images rendered from viewpoints sampled from a prior distribution
to be realistic. These models, however, often rely heavily on an accurate
estimation of viewpoint distribution and/or a powerful 2D image generative
model, both of which are difficult to obtain in small-data scenarios.

#### Optimization from Multi-views and Videos.

3D reconstruction traditionally relies on epipolar geometry of multi-view
images of static scenes [11, 16]. Neural Radiance Fields (NeRF) [41, 53, 3,
64] have recently emerged as a powerful volumetric representation for multi-
view reconstruction given accurate cameras. A recent line of work, LASR [72],
ViSER [73] and BANMo [74], optimises 3D shapes of deformable objects from a
small set of monocular videos, with heavily engineered optimisation strategies
using optical flow and mask supervision, and additionally DensePose [45]
annotations for highly deformable animals. Concurrent work of LASSIE [75] also
leverages DINO-ViT [5] image features for supervision, but optimises a part-
based model on a small collection of images ($\sim$30) of an object category.
It uses a shared shape model for all instances with only per-instance
articulation.

#### Learnable Deformable 3D Representations.

A common 3D representation for weakly-supervised shape learning is triangular
meshes. Deformation on meshes can be modelled by estimating offsets of
individual vertices [20, 30, 68], which typically requires heavy
regularisation, such as As-Rigid-As-Possible (ARAP) [58]. To constrain the
space of deformations, many works often utilise lower-dimensional models, such
as cages [76] or linear blend skinning controlled with skeletal bones [36, 68]
or Gaussian control points [72, 73]. Parametric models, like SMPL [36] and
SMAL [78], allow for realistic control of the shape through learned
parameters, but often require an extensive collection of 3D scans to learn
from.

Dynamic neural fields [51, 49, 12, 60, 52, 47, 59, 74], such as D-NeRF [51],
extend NeRF [41] with time-varying components. A-NeRF [59] proposes a radiance
field conditioned on an articulated skeleton, while BANMo [74] uses learned
Gaussian control points similarly to [72, 73]. One key limitation of these
implicit representations is the requirement of an inverse transformation from
3D world coordinates back to the canonical space, which is often harder to
learn compared to forward deformation [8]. We propose a hybrid SDF-mesh
representation, extending DMTet [57] with an articulation mechanism, which
combines the expressiveness of implicit models with the simplicity of mesh
deformation and articulation.

#### Viewpoint Prediction in the Wild.

Viewpoint prediction in weakly-supervised settings is challenging, as it is
prone to local optima induced by projection and common object symmetries.
Existing solutions are based around learning a distribution over multiple
possible viewpoints. U-CMR [14] proposes a camera-multiplex that optimises
over 40 cameras for each training sample using a given template shape, which
is expensive as evaluating each hypothesis involves a rendering step. We
propose a new viewpoint exploration scheme that requires evaluating _only_ one
hypothesis in each step with essentially no added training cost, and does not
rely on a given template shape but a jointly learned model. Recent work of
Implicit-PDF [44] proposes to learn viewpoint distributions implicitly by
estimating a probability for each image-pose pair of a fixed object instance.
RelPose [77] extends this to learning relative poses of image pairs. Our model
also predicts a probability associated with each viewpoint hypothesis, but
instead relates an instance to a learned category-level prior model.

##  3 Method

Figure 2: Training Pipeline. Given a collection of single-view images of an
animal category, our model learns a category-specific prior shape using an
implicit-explicit representation, together with a feature field that allows us
to fuse self-supervised correspondences through a feature rendering loss. This
prior shape is deformed, articulated and shaded based on features extracted
from a training image. To combat local minima in viewpoint prediction, we
introduce an efficient scheme that explores multiple hypotheses at essentially
no additional cost. The entire pipeline is trained with reconstruction losses
end-to-end, except for the frozen image encoder pre-trained via self-
supervision.

Given only a collection of single-view images of a deformable object category
collected in the wild, our goal is to train a monocular reconstruction network
that, at test time, predicts the 3D shape, articulated pose, albedo and
lighting from a single image of the object.

During training, our method requires no geometric supervision other than the
images, 2D foreground segmentations of the object obtained automatically from
an off-the-shelf method like [24], and a description of the topology of the
object’s skeleton (_e.g_., the number of legs). It is based on several key
ideas: an implicit-explicit shape representation (Sec. 3.1), a carefully
designed hierarchy of shapes from generic category-specific prior to
articulated instances (Sec. 3.2), and an effective viewpoint prediction scheme
that leverages self-supervised correspondences and an efficient multi-
hypothesis exploration scheme to avoid local minima (Sec. 3.3), as illustrated
in Fig. 2.

###  3.1 Implicit-Explicit 3D Shape Representation

The choice of representation for the 3D shape of the object is critical. It
must be (1) sufficiently _expressive_ for modelling fine-grained shape details
and deformations and (2) _effective_ for learning with only weak supervision
through image reconstruction.

A common choice is to adopt an explicit representation like a triangular mesh
[20, 72, 68]. Meshes are conceptually simple and more compact than volumetric
representations; they provide access to surface properties such as smoothness,
which can be used to regularise the shape; they can be easily deformed by
moving their vertices, which simplifies constructing the shape space, _e.g_.
by using blend skinning. While meshes have been found to be an effective tool
to learn shapes and deformations in weakly-supervised conditions, they are
limited by their inherent difficulty in modelling topology changes, which may
be necessary as the model evolves during training. They can also develop
defects such as folds, requiring occasional remeshing [72, 13].

The alternative to meshes are implicit representations like SDFs and radiance
fields [40, 48, 41, 67], which naturally allow for topological changes during
training. However, due to their high capacity, these representations require
substantial supervision, usually in the form of a dense coverage of the
different object viewpoints [41, 67] or 3D ground-truth [40, 48]. Furthermore,
modelling deformations with implicit representations is harder than for
explicit meshes, so they have found more success in modelling single static
scenes rather than a family of different objects.

Here we adopt a representation that combines the advantages of both: the shape
of the object is represented implicitly by a neural field but converted on-
the-fly into an explicit mesh via a fast marching tetrahedra method of [57,
43], a relative of cubes-like [32]. The object’s shape
$S=\\{\mathbf{x}\in\mathbb{R}^{3}|s(\mathbf{x})=0\\}$ is defined as the
isosurface of the signed distance function (SDF)
$s:\mathbb{R}^{3}\mapsto\mathbb{R}$. We implement the SDF $s$ using a Multi-
Layer Perceptron (MLP) taking 3D coordinates as input. To encourage $s$ to be
a signed distance function, we minimise the Eikonal regulariser [15]:

$\mathcal{R}_{\text{Eik}}=\sum_{x\in\mathbb{R}^{3}}(\|\nabla
s(\mathbf{x})\|_{2}-1)^{2}.$ (1)

We use DMTet’s implementation [57] of Differentiable Marching Tetrahedra [9],
which extracts a watertight mesh $M$ from $s$ on the fly.

###  3.2 3D Shape Modelling and Prediction

In order to represent the range of possible shapes of an object category, we
use a _hierarchical_ model, starting from a category-wise template shape,
deforming the latter to model the shape of a specific object instance, and
finally matching the pose of the object to a specific image using blend
skinning (Fig. 2). These deformations are predicted from a single image by
learning networks that share a common image encoder. We discuss these
components below.

#### Image Encoder.

The networks that predict the object shape and pose from a single image solve
a difficult task and their design is critical. To facilitate their job, we
encode the image by means of a self-supervised Vision Transformer (ViT [27])
like DINO-ViT [5]. The ViT extracts keys and output tokens from the image,
which were respectively found to code for semantic correspondences [1] and
image appearance [62]. Hence, we use keys to predict the object shape and pose
and output tokens to infer its appearance.

Formally, the ViT $\Phi$ divides the input image $I\in\mathbb{R}^{3\times
H\times W}$ into a grid of patches and uses them as tokens in a sequence of
attention layers. We take the _keys_ $\Phi_{\text{k}}\in\mathbb{R}^{D\times
H_{p}\times W_{p}}$ and the _output tokens_
$\Phi_{\text{o}}\in\mathbb{R}^{D\times H_{p}\times W_{p}}$ from the last
attention layer and feed them into two separate (convolutional) networks to
obtain a single global key feature
$\phi_{\text{k}}=f_{\text{k}}(\Phi_{\text{k}})\in\mathbb{R}^{D}$ and a single
global output feature
$\phi_{\text{o}}=f_{\text{o}}(\Phi_{\text{o}})\in\mathbb{R}^{D}$ for the whole
image respectively.

#### Category-wise Template Shape.

We capture the similarities across all instances with a category-wise
template, factoring out instance-specific variations. The template shape is
represented by the SDF $s$, initialised with a generic ellipsoid, elongated
along the $z$-axis. The technique of Sec. 3.1 is used to extract a watertight
mesh from it, the vertices of which we denote by the symbol $V_{\text{pr}}$.

#### Instance-specific Shape.

The shape of each object instance $V_{\text{ins}}$ differs slightly from the
template shape $V_{\text{pr}}$. This is accounted for by the deformation:
$V_{\text{ins},i}=V_{\text{pr},i}+\Delta V_{i},$ where the displacement
$\Delta V_{i}=f_{\Delta V}(V_{\text{pr},i},\phi_{\text{k}})$ is predicted by
an MLP $f_{\Delta V}$ from the image features $\phi_{k}$ for each vertex
$V_{\text{pr},i}$. We exploit the fact that many objects have a bilaterally
symmetric structure, and enforce both the prior shape and instance deformation
to be symmetric by mirroring the query locations for the MLPs about the
$yz$-plane. We further restrict the deformation to be modest in magnitude.

#### Posing.

In order to match the 3D shape observed in the image, the instance-specific
mesh $V_{\text{ins},i}$ is further deformed by the _posing function_
$V_{i}=g(V_{\text{ins},i},\xi)$, where $\xi$ are the pose parameters. Posing
accounts for the large, but very controlled deformations caused by the
underlying skeletal structure of the objects. Attempting to model these
directly by the less structured deformation in $V_{\text{ins},i}$, while
conceptually possible by relaxing the symmetry and magnitude constraints,
would lead to severe overfitting.

We model the pose of the object using blend skinning [38, 6], controlled by a
set of bone rotations $\xi_{b}\in SO(3),b=2,\dots,B$, and a rigid pose (or
viewpoint) $\xi_{1}\in SE(3)$. Specifically, we initialise a set of rest-pose
bone locations $\mathbf{J}_{b}$ on the instance mesh using simple
heuristics.111A chain of bones going through the two farthest end points along
$z$-axis for birds, and for quadrupeds, additionally four legs branching out
from the body bone to the lowest point in each quadrant. See Sec. 7.1 . Each
bone $b$ (except the root) has exactly one parent $\pi(b)$ , forming a tree
structure. Each vertex $V_{i}$ is associated to the bones by the skinning
weights $w_{i,b}$, defined based on the relative proximity to each bone (see
Sec. 7.1 ). The vertices are then posed by the linear blend _skinning
equation_ :

$\displaystyle
V_{i}(\xi)=\left(\sum_{b=1}^{B}w_{i,b}G_{b}(\xi)G_{b}(\xi^{*})^{-1}\right)V_{\text{ins},i},$
(2) $\displaystyle G_{b}=G_{\pi(b)}\circ
g_{b},~{}~{}g_{b}(\xi)=\begin{bmatrix}R_{\xi_{b}}&\mathbf{J}_{b}\\\ 0&1\\\
\end{bmatrix},$

where $\xi^{*}$ denotes the bone rotations at rest pose.

#### Pose Prediction.

Similarly to the instance-specific deformation $\Delta V_{i}$, the bone
rotations $\xi_{2:B}$ are predicted by a network $f_{\text{b}}$ from the image
features $\phi_{\text{k}}$. The viewpoint $\xi_{1}\in SE(3)$ is predicted
separately, given in Sec. 3.3.

Pose articulation depends on estimating local body joints and benefits from
using local features. To extract them, we first project the centroid of each
bone based on the rest pose and the predicted viewpoint $\xi_{1}$:

$\mathbf{u}_{b}=\Pi_{\xi_{1}}\left(\frac{\mathbf{J}_{b}+\mathbf{J}_{\pi(b)}}{2}\right),$

and sample the local feature from the patch key feature map
$\Phi_{\text{k}}(\mathbf{u}_{b})$ at the projected pixel location
$\mathbf{u}_{b}$.

In order to estimate bone rotations given these features, since these
parameters are inter-dependent, we use a transformer architecture [63] for
$f_{\text{b}}$ that operates on the bone descriptors as input tokens
$\nu_{b}=(\phi_{\text{k}},\Phi_{\text{k}}(\mathbf{u}_{b}),b,\mathbf{J}_{b},\mathbf{u}_{b})$
and predicts the bone rotations, parametrized by Euler angles, as output
tokens: $(\xi_{2},\ldots,\xi_{B})=f_{\text{b}}(\nu_{2},\ldots,\nu_{B})$.

###  3.3 Viewpoint Prediction

Estimating the viewpoint is essential in order to learn the object’s shape and
articulation. Here we leverage the self-supervised DINO-ViT features, which
capture noisy semantic correspondences, and a new multi-hypothesis viewpoint
prediction scheme that addresses the remaining ambiguity. These steps are
explained next.

#### Fusing Self-Supervised Correspondences.

Prior works have often established noisy but explicit model-image
correspondences in order to estimate viewpoint. Here, we take a different
approach which is simple, efficient and requires a minimal extension to the
model. Like N3FF [61], we _fuse_ information from the self-supervised ViT
features into the 3D model by learning to render them. This is also similar to
BANMo [74], but we use self-supervised features instead of supervised
DensePose [45] embeddings.

This is done by adding a coordinate network
$\psi:\mathbb{R}^{3}\mapsto\mathbb{R}^{D^{\prime}}$ that predicts a field of
$D^{\prime}$ dimensional features in the canonical space, similar to the
analogous network for the RGB colours. These canonical features are then
_rendered_ as an image $\hat{\Phi}_{\text{k}}$ by using the same process used
to render RGB colours. They are trained to minimise a corresponding rendering
loss:
$\mathcal{L}_{\text{feat}}=\|\tilde{M}\odot(\hat{\Phi}_{\text{k}}-\Phi^{\prime}_{\text{k}})\|_{2}^{2}$,
where $\tilde{M}=\hat{M}\odot M$ is the intersection of the rendered mask and
the ground-truth mask and
$\Phi^{\prime}_{\text{k}}=\texttt{PCA}(\Phi_{\text{k}})$ is a PCA-reduced (to
$D^{\prime}=16$ principal components) version of the original DINO-ViT
features $\Phi_{k}$ for memory efficiency.

#### Multi-Hypothesis Viewpoint Prediction.

Prior works have found that a major challenge in learning the object viewpoint
is the existence of multiple local optima in the reconstruction objective.
Some have addressed this issue by sampling a large number of hypotheses for
the viewpoint [14], but this is somewhat cumbersome and slow. Instead, we
propose a scheme that explores multiple viewpoints _statistically_ , but at
each iteration samples a single one, and thus comes “for free”.

We hence task the model to predict four viewpoint rotation hypotheses
$R_{k},k\in\\{1,2,3,4\\}$, each in one of the four quadrants around the
object.222 For bilaterally symmetric animals, there are often four ambiguous
orientations towards each of the four $xz$-quadrants in the canonical space.
The model also predicts a score $\sigma_{k}$ for each hypothesis, used to
evaluate the probability $p_{k}$ that hypothesis $k$ is the best of the four
options as:
$p_{k}=\frac{\exp(-\sigma_{k}^{\prime})}{\sum_{j}\exp(-\sigma_{j}^{\prime})}$,
where $\sigma_{k}^{\prime}=\frac{\sigma_{k}}{\tau}$ sharpens the distribution
using a temperature parameter $\tau$ that is gradually decreased during
training.

The normal approach for learning $\sigma_{k}$ is to sample multiple hypotheses
and compare their reconstruction loss to determine which one is better.
However, computing the loss is expensive as it requires rendering the model.
Instead, we suggest to _sample_ a single hypothesis each training iteration333
We sample viewpoint $k^{*}=\operatorname{argmax_{k}}p_{k}$ 80% of the time,
and uniformly at random 20% of the time.  and simply learn $\sigma_{k}$ to
predict the expected reconstruction loss $\tilde{\mathcal{L}}_{k}$, minimizing
the objective:

$\mathcal{L}_{\text{hyp}}=(\sigma_{k}-\mathrm{sg}[\tilde{\mathcal{L}}_{k}])^{2},$
(3)

where $\mathrm{sg}[\cdot]$ is the stop-gradient operator.

###  3.4 Appearance and Shading

The albedo of the object is modelled in canonical space by using a coordinate
MLP $a_{i}=f_{\text{a}}(\mathbf{x}_{i},\phi_{\text{o}})\in\mathbb{R}^{3}$. A
lighting network $f_{\text{l}}$ also predicts the dominant light direction
$l\in\mathbb{R}^{2}$ and ambient and diffuse intensities
$k_{s},k_{d}\in\mathbb{R}$. We used _differentiable deferred mesh rendering_
to render the image. This means that the mesh is projected onto the image by a
differentiable renderer, obtaining for each pixel the coordinates of its
corresponding 3D point in the canonical space. Then, in a second pass the
albedo network is evaluated once per pixel, obtaining the value of the albedo
$a_{i}$ which is shaded to obtain the final pixel colours according to
equation $I_{i}=\left(k_{s}+k_{d}\cdot\max\\{0,\langle
l,n_{i}\rangle\\}\right)\cdot a_{i}$, where $n_{i}$ is the pixel-wise normal
of the _posed_ mesh.

###  3.5 Training Objective

The final training objective is:

$\displaystyle\tilde{\mathcal{L}}$
$\displaystyle=\lambda_{\text{m}}\mathcal{L}_{\text{m}}+\lambda_{\text{im}}\mathcal{L}_{\text{im}}+\lambda_{\text{f}}\mathcal{L}_{\text{feat}}$
(4) $\displaystyle\mathcal{L}$
$\displaystyle=\mathbb{E}_{p_{k}}\left[\tilde{\mathcal{L}}+\lambda_{\text{E}}\mathcal{R}_{\text{Eik}}+\lambda_{\text{d}}\mathcal{R}_{\text{def}}+\lambda_{\text{a}}\mathcal{R}_{\text{art}}\right]+\lambda_{\text{h}}\mathcal{L}_{\text{hyp}}.$

This loss includes the main components discussed above plus the following
additional objectives, capturing other available cues. Naturally,
$\mathcal{L}_{\text{m}}=\|\tilde{M}\odot(\hat{I}-I)\|_{1}$, where
$\tilde{M}=\hat{M}\odot M$ encourages the rendered image to match the input.
Here $M$ is the object-mask of the input image and $\hat{M}$ is the
(differentiable) mask of the rendering.

The mask loss
$\mathcal{L}_{\text{m}}=\|\hat{M}-M\|_{2}^{2}+\lambda_{\text{dt}}(\hat{M}\odot\texttt{dt}(M))$
encourages rendered shape to align with the image mask using a distance
transform $\texttt{dt}(\cdot)$ for meaningful gradients.

The feature loss
$\mathcal{L}_{\text{feat}}=\|\tilde{M}\odot(\hat{\Phi}^{\prime}_{\text{k}}-\Phi^{\prime}_{\text{k}})\|_{2}^{2}$,
where $\Phi^{\prime}_{\text{k}}=\texttt{PCA}(\Phi_{\text{k}})$ aligns the mesh
rendered with the feature texture to the pre-trained feature encoder.

Finally, we regularise the magnitude of the deformation:
$\mathcal{R}_{\text{def}}=\sum_{i}\|f_{\Delta V}(\mathbf{x}_{i})\|_{2}^{2}$
and similarly the bone rotations:
$\mathcal{R}_{\text{art}}=\sum_{b}\|\xi_{b}\|_{2}^{2}$.

Due to the supervision of the hypothesis scores with the loss itself, we can
avoid evaluating all hypotheses. We sample only one hypothesis $k$ at a time,
and evaluate $\mathcal{L}={p_{k}}\mathcal{L}_{k}$. To improve the sampling
efficiency, we sample the viewpoint based on the learned probability
distribution $p_{k}$ with a gradually decreasing temperature $\tau$.

##  4 Experiments

We conduct extensive experiments on a few animal categories, including horses,
giraffes, zebras, cows and birds, and compare against prior work both
qualitatively and quantitatively on standard benchmarks. We also show that our
model trained on real images generalises to abstract drawings, demonstrating
the power of unsupervised learning.

###  4.1 Data

For horses, we use the horse dataset from DOVE [68] containing $10.1$k images
extracted from YouTube, and supplement it with $541$ additional images from
three datasets for diversity: Weizmann Horse Database [4], PASCAL [10] and
Horse-10 Dataset [39]. For giraffes, zebras and cows, we source images from
Microsoft COCO Dataset [33] and keep the ones with little occlusion. Since
these datasets are relatively small, we only use them to finetune the pre-
trained horse model. For birds, we combine the DOVE dataset [68] and CUB
dataset [65] consisting of $57.9$k and $11.7$k images respectively. We use an
off-the-shelf PointRend [24] detector to obtain segmentation masks, crop
around the objects and resize them to $256\times 256$. We follow the original
train/test splits from CUB and DOVE, and randomly split the rest, resulting in
$11.5$k/$0.8$k horse, $513$/$57$ giraffe, $574$/$64$ zebra, $719$/$80$ cow and
$63.9$k/$10.5$k bird images. We additionally collect roughly $100$ horse
images from the Internet to test generalisation.

###  4.2 Technical Details

The model is implemented using a total of $8$ different neural networks. The
feature field $\psi$, template SDF $s$, albedo field $f_{\text{a}}$ and
deformation field $f_{\Delta V}$ are implemented using MLPs, which take in 3D
coordinates (concatenated with image features for albedo and deformation
networks) as input. The image encoder consists of a ViT-S [27] architecture
from DINO [5] and two convolutional encoders $f_{\text{k}}$ and $f_{\text{o}}$
that fuse the patch features into global feature vectors. We use self-
supervised pre-trained DINO-ViT with frozen weights, and only train the
convolutional encoders. The lighting network $f_{\text{l}}$ is a simple MLP
that maps global output feature to a $4$-channel vector, and the articulation
network $f_{\text{b}}$ is a transformer architecture with $4$ blocks. We use a
separate encoder for the viewpoint network identical to $f_{\text{k}}$, as we
find empirically that sharing the encoder tends to make the viewpoint learning
unstable. Apart from DINO encoder, all components are trained end-to-end from
scratch for $100$ and $10$ epochs on horses and birds respectively, with the
instance deformation and articulation disabled for the first $30$ and $2$
epochs. The DINO features $\Phi^{\prime}_{\text{k}}$ used to compute
$\mathcal{L}_{\text{feat}}$ are pre-computed with patch size $8$ and stride
$4$ and masked, following [1], and reduced from a channel size of $384$ to
$16$ using PCA. All technical details and hyper-parameters are included in
Sec. 7.

Figure 3: Single Image Reconstruction. We show the reconstructed mesh from the
input view and two additional views together with predicted texture and
animated version of the shape obtained by articulating the estimated skeleton.
Figure 4: Reconstructions of Giraffes, Zebras and Cows. We finetune the horse
model on other animal categories, and show that the method generalises well to
other animals.

###  4.3 Qualitative Results

Fig. 3 shows a few reconstructions of horses and birds produced by our model.
Given a single 2D image at test time, our model predicts a textured 3D mesh of
the object, capturing its fine-grained geometric details, such as the legs and
tails of the horses. Our model predicts the articulated pose of the objects,
allowing us to easily transfer the pose of one instance from another and
animate it in 3D. Although our model is trained on real images only, it
demonstrates excellent generalisation to paintings and abstract drawings.

We also show in Fig. 4 that by finetuning the horse model on other animals
without any additional modifications (except disabling articulation for the
initial $5$k iterations), it generalises to various animal categories with
highly different shape and appearance, such as giraffes, zebras and cows.

Note that for the examples of abstract horse drawings (last rows in Figs. 1
and 3), since the texture is out of the training distribution, we finetune
(only) the albedo network for $100$ iterations, which takes less than $5$
seconds. This is also done for giraffes, zebras and cows in Fig. 4, as the
training datasets are relatively small for learning complex appearance.
Additional qualitative results are presented in Sec. 6 and the _supplementary
video_.

Figure 5: Comparison on Birds. Both CMR [20] and UMR [31] often predict
inaccurate poses, such as the bird on the left. Figure 6: Comparison with UMR
[31] on Horses. Our model produces much higher quality reconstructions with
four legs. UMR is only able to predict symmetrical shapes with two legs.

###  4.4 Comparison with Prior Work

We compare with previous weakly-supervised methods on 3D reconstruction of
deformable objects. The most relevant prior work is UMR [31] and DOVE [68], as
they also only require object masks and weak correspondences from either part
segmentations (SCOPS) [18] or video training [68]. Our method leverages _self-
supervised_ DINO-ViT features, which makes our method the least supervised in
this area. We also compare against the well-established baseline of CMR [20],
which requires stronger supervision in the form of 2D keypoint annotations and
pre-computed cameras, as well as U-CMR [14], which uses a category-specific
template shape. To evaluate other methods, we use their code and pre-trained
models in public repositories (CMR, UMR, U-CMR) or provided by the authors
(DOVE).

#### Qualitative Comparisons.

Figs. 6 and 5 compare our reconstructions against the previous methods on bird
and horse images. Since UMR did not release their horse model, we test our
model on the examples presented in their paper for side-by-side comparison.
Our method predicts more accurate poses and higher quality articulated shapes
(as opposed to only symmetric shapes with CMR and UMR). On horses, our method
reconstructs all four legs with details while UMR predicts only two. Since UMR
copies texture from input image, for fair comparison, we finetune (only) the
albedo network for $100$ iterations at test time.

Table 2: Evaluation on Toy Bird Scans [68]. Baseline results reported by [68]. Annotations: template shape, viewpoint, 2D keypoint, mask, optical flow, video. | Supervision |  Chamfer Distance (cm) $\downarrow$
---|---|---
CMR [20] |  |  1.35 $\pm 0.81$
U-CMR [14] |  |  1.82 $\pm 0.93$
UMR [31] |  \+ SCOPS |  1.24 $\pm 0.75$
DOVE [68] |  |  1.51 $\pm 0.89$
Ours |  \+ DINO |  0.79 $\pm 0.50$
Table 3: Keypoint Transfer on CUB [65] Dataset. In addition to the results on the whole CUB test set, we also report results on a subset w/o aquatic birds which none of the methods can sufficiently reconstruct. Our model produces superior results with significantly less supervision.  |  |<EMAIL_ADDRESS>
---|---|---
Method | Supervision | Entire CUB | w/o aquatic
CMR [20] (from [31]) |  | 0.473 | -
CMR [20] |  | 0.546 | 0.591
CMR [20] |  | 0.255 | 0.277
U-CMR [14] |  | 0.359 | 0.412
UMR [31] |  \+ SCOPS | 0.512 | 0.555
DOVE [68] |  | 0.447 | 0.510
Ours |  \+ DINO | 0.554 | 0.635

#### Quantitative Comparisons.

We evaluate shape reconstruction on the 3D Toy Bird Dataset from DOVE [68],
which contains 3D scans of $23$ realistic toy bird models paired with $345$
photographs of them taken in natural real-world environments. Following [68],
we align the predicted mesh with the ground-truth scan using ICP and compute
the bi-directional Chamfer Distance between two sets of sample points from the
aligned meshes. Tab. 2 summarises the results compared against other methods.
Our model produces significantly more accurate reconstructions, resulting in a
much lower error (visualisations in Fig. 11).

We also evaluate Keypoint Transfer on the CUB [65] benchmark, a common
evaluation metric [20, 31]. We follow the protocol in [31] and sample 10k
source and target image pairs. Given a source image, we project all the
visible vertices of the predicted mesh onto the image using the predicted
viewpoint, and assign each annotated 2D keypoint to its nearest vertex. We
then render these vertices using the mesh and viewpoint predicted from a
different target image. We measure the error between the projected vertices
(corresponding to the transferred keypoints) and the annotated target
keypoints using the Percentage of Correct Keypoints (PCK) metric. As shown in
Tab. 3, our model outperforms previous methods, including CMR [20] which
requires strong geometric supervision in the form of keypoints. Since CUB
dataset contains a wide variety of birds, including aquatic birds like
mallards, which are particularly challenging to model due to heavy occlusion
and large shape variation, we also report the numbers on a subset of the
categories (removing $50$ aquatic bird categories), which show even more
evident improvements.

Figure 7: Ablation Studies. We train the model without feature loss
$\mathcal{L}_{\text{feat}}$, without multi-hypothesis (MH) viewpoint
prediction and without both. Without either component, the viewpoint
predictions collapse to frontal views, resulting in unnatural shapes.

###  4.5 Ablation Studies and Limitations

We present ablation studies on the two key components of the model in Fig. 7,
self-supervised feature loss and multi-hypothesis viewpoint prediction, and
visualise the learned viewpoint distributions in Fig. 8. Without either
component, the learned viewpoints collapse to only frontal views, resulting in
unnatural stretched reconstructions.

Although our model reconstructs highly detailed 3D shapes from only a single
image and generalises to in-the-wild images, the predicted texture may not
preserve sufficient details with a single forward pass and may require
additional test-time finetuning. This can potentially be addressed by training
on a larger dataset and incorporating recent advances in image generation [54,
55]. Another limitation is the requirement of a pre-defined topology for
articulation, which may vary between different animal species. Discovering the
articulation structure automatically from raw in-the-wild images will be of
great interest for future work. Failure cases are discussed in Sec. 6.5.

##  5 Conclusions

We have introduced a new model that can learn a 3D model of an articulated
object category from single-view images taken in the wild. This model can, at
test time, reconstruct the shape, articulation, albedo, and lighting of the
object from a single image, and generalises to abstract drawings. Our approach
demonstrates the power of combining several recent improvements in self-
supervised representation learning together with a new viewpoint sampling
scheme. We have shown significantly superior results compared to prior works,
even when they use more supervision.

#### Acknowledgements.

We would like to thank Tengda Han, Shu Ishida, Dylan Campbell, Eldar
Insafutdinov, Luke Melas-Kyriazi, Ragav Sachdeva and Sagar Vaze for insightful
discussions, and Guanqi Zhan and Jaesung Huh for proofreading. Shangzhe Wu is
supported by Meta Research. Tomas Jakab is supported by ERC-CoG UNION
101001212. Christian Rupprecht is supported by VisualAI EP/T028572/1 and ERC-
CoG UNION 101001212. Andrea Vedaldi is supported by ERC-CoG UNION 101001212.

## References

  * [1] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.  Deep ViT features as dense visual descriptors.  arXiv preprint arXiv:2112.05814, 2021.
  * [2] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis.  SCAPE: Shape completion and animation of people.  ACM TOG, 2005.
  * [3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Mip-NeRF 360: Unbounded anti-aliased neural radiance fields.  CVPR, 2022.
  * [4] Eran Borenstein and Shimon Ullman.  Class-specific, top-down segmentation.  In ECCV, 2002.
  * [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [6] John E Chadwick, David R Haumann, and Richard E Parent.  Layered construction for deformable animated characters.  ACM SIGGRAPH Computer Graphics, 1989.
  * [7] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In CVPR, 2021.
  * [8] Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger.  SNARF: Differentiable forward skinning for animating non-rigid neural implicit shapes.  In ICCV, 2021.
  * [9] Akio Doi and Akio Koide.  An efficient method of triangulating equi-valued surfaces by using tetrahedral cells.  IEICE Transactions on Information and Systems, 1991.
  * [10] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  IJCV, 2015.
  * [11] Olivier Faugeras.  Three-Dimensional Computer Vision: A Geometric Viewpoint.  MIT Press, 1993.
  * [12] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In CVPR, 2021.
  * [13] Shubham Goel, Georgia Gkioxari, and Jitendra Malik.  Differentiable stereopsis: Meshes from multiple views using differentiable rendering.  In CVPR, 2022.
  * [14] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik.  Shape and viewpoints without keypoints.  In ECCV, 2020.
  * [15] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  In ICML, 2020.
  * [16] Richard Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * [17] Paul Henderson and Vittorio Ferrari.  Learning single-image 3d reconstruction by generative modelling of shape, pose and shading.  IJCV, 2019.
  * [18] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz.  SCOPS: Self-supervised co-part segmentation.  In CVPR, 2019.
  * [19] Hanbyul Joo, Tomas Simon, and Yaser Sheikh.  Total capture: A 3d deformation model for tracking faces, hands, and bodies.  In CVPR, 2018.
  * [20] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In ECCV, 2018.
  * [21] Abhishek Kar, Shubham Tulsiani, João Carreira, and Jitendra Malik.  Category-specific object reconstruction from a single image.  In CVPR, 2015.
  * [22] Hiroharu Kato and Tatsuya Harada.  Learning view priors for single-view 3d reconstruction.  In CVPR, 2019.
  * [23] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3d mesh renderer.  In CVPR, 2018.
  * [24] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  PointRend: Image segmentation as rendering.  In CVPR, 2020.
  * [25] Filippos Kokkinos and Iasonas Kokkinos.  Learning monocular 3d reconstruction of articulated categories from motion.  In CVPR, 2021.
  * [26] Filippos Kokkinos and Iasonas Kokkinos.  To the point: Correspondence-driven monocular 3d category reconstruction.  In NeurIPS, 2021.
  * [27] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [28] Nilesh Kulkarni, Abhinav Gupta, David F. Fouhey, and Shubham Tulsiani.  Articulation-aware canonical surface mapping.  In CVPR, 2020.
  * [29] Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani.  Canonical surface mapping via geometric cycle consistency.  In ICCV, 2019.
  * [30] Xueting Li, Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, and Jan Kautz.  Online adaptation for consistent mesh reconstruction in the wild.  In NeurIPS, 2020.
  * [31] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz.  Self-supervised single-view 3d reconstruction via semantic consistency.  In ECCV, 2020.
  * [32] Yiyi Liao, Simon Donné, and Andreas Geiger.  Deep marching cubes: Learning explicit surface representations.  In CVPR, 2018.
  * [33] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In ECCV, 2014.
  * [34] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  In ICCV, 2019.
  * [35] Yebin Liu, Juergen Gall, Carsten Stoll, Qionghai Dai, Hans-Peter Seidel, and Christian Theobalt.  Markerless motion capture of multiple characters using multiview image segmentation.  IEEE TPAMI, 2013.
  * [36] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  SMPL: A skinned multi-person linear model.  ACM TOG, 2015.
  * [37] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng.  Rectifier nonlinearities improve neural network acoustic models.  In ICML, 2013.
  * [38] Nadia Magnenat-Thalmann, E Primeau, and Daniel Thalmann.  Abstract muscle action procedures for human face animation.  The Visual Computer, 1988.
  * [39] Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W. Mathis.  Pretraining boosts out-of-domain robustness for pose estimation.  In WACV, 2021.
  * [40] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, 2019.
  * [41] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [42] Tom Monnier, Matthew Fisher, Alexei A. Efros, and Mathieu Aubry.  Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency.  In ECCV, 2022.
  * [43] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Mueller, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In CVPR, 2021.
  * [44] Kieran A Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia.  Implicit-PDF: Non-parametric representation of probability distributions on the rotation manifold.  In ICML, 2021.
  * [45] Natalia Neverova, David Novotný, and Andrea Vedaldi.  Continuous surface embeddings.  In NeurIPS, 2020.
  * [46] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  HoloGAN: Unsupervised learning of 3d representations from natural images.  In ICCV, 2019.
  * [47] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada.  Neural articulated radiance field.  In ICCV, 2021.
  * [48] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning continuous signed distance functions for shape representation.  In CVPR, June 2019.
  * [49] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In ICCV, 2021.
  * [50] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  DreamFusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [51] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-NeRF: Neural radiance fields for dynamic scenes.  In CVPR, 2020.
  * [52] Amit Raj, Michael Zollhoefer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, and Stephen Lombardi.  PVA: Pixel-aligned volumetric avatars.  In CVPR, 2021.
  * [53] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In ICCV, 2021.
  * [54] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [55] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In NeurIPS, 2022.
  * [56] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  GRAF: Generative radiance fields for 3d-aware image synthesis.  In NeurIPS, 2020.
  * [57] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  In NeurIPS, 2021.
  * [58] Olga Sorkine and Marc Alexa.  As-rigid-as-possible surface modeling.  In Symposium on Geometry processing, 2007.
  * [59] Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin.  A-NeRF: Articulated neural radiance fields for learning human shape, appearance, and pose.  In NeurIPS, 2021.
  * [60] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video.  In ICCV, 2021.
  * [61] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi.  Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representation.  In 3DV, 2022.
  * [62] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Splicing vit features for semantic appearance transfer.  In CVPR, 2022.
  * [63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [64] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan.  Ref-NeRF: Structured view-dependent appearance for neural radiance fields.  CVPR, 2022.
  * [65] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [66] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2Mesh: Generating 3d mesh models from single rgb images.  In ECCV, 2018.
  * [67] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In NeurIPS, 2021.
  * [68] Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  DOVE: Learning deformable 3d objects by watching videos.  arXiv preprint arXiv:2107.10844, 2021.
  * [69] Shangzhe Wu, Ameesh Makadia, Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa.  De-rendering the world’s revolutionary artefacts.  In CVPR, 2021.
  * [70] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3D objects from images in the wild.  In CVPR, 2020.
  * [71] Yuxin Wu and Kaiming He.  Group normalization.  In ECCV, 2018.
  * [72] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T. Freeman, and Ce Liu.  LASR: Learning articulated shape reconstruction from a monocular video.  In CVPR, 2021.
  * [73] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan.  ViSER: Video-specific surface embeddings for articulated 3d shape reconstruction.  In NeurIPS, 2021.
  * [74] Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul.  BANMo: Building animatable 3d neural models from many casual videos.  In CVPR, 2022.
  * [75] Chun-Han Yao, Wei-Chih Hung, Michael Rubinstein, Yuanzhen Lee, Varun Jampani, and Ming-Hsuan Yang.  Lassie: Learning articulated shape from sparse image ensemble via 3d part discovery.  In NeurIPS, 2022.
  * [76] Wang Yifan, Noam Aigerman, Vladimir G Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung.  Neural cages for detail-preserving 3d deformations.  In CVPR, 2020.
  * [77] Jason Y. Zhang, Deva Ramanan, and Shubham Tulsiani.  RelPose: Predicting probabilistic relative rotation for single objects in the wild.  In ECCV, 2022.
  * [78] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black.  3d menagerie: Modeling the 3d shape and pose of animals.  In CVPR, 2017.

##  6 Additional Results

###  6.1 Generalisation to Other Animal Categories

In addition to Fig. 4, we show more reconstruction results of giraffes, zebras
and cows in Fig. 10. Our method is able to produce accurate shape
reconstructions from a single image across large variations of animal shapes.

###  6.2 Additional Comparisons with Prior Works

#### Qualitative Comparisons

Fig. 12 and Fig. 13 show qualitative comparisons with prior works on horses
and birds respectively. Our method is able to predict shapes with finer
details and more accurate poses than prior works. We also plot the
distribution of predicted viewpoints demonstrating that other methods with a
comparable level of supervision collapse to only a limited range of
viewpoints, _e.g_. predicting only frontal poses.

#### Visualisations of the Toy Bird Reconstructions.

Supplementary to Tab. 2, we show a qualitative comparison of the predicted
shapes and the scanned ground-truth shapes on the Toy Bird Scan dataset in
Fig. 11.

###  6.3 Ablation Studies

In addition to Fig. 7, we examine the effects of both the feature rendering
loss $\mathcal{L}_{\text{feat}}$ and the multi-hypothesis viewpoint prediction
in Fig. 8, demonstrating that both of the components are essential to prevent
the collapse of viewpoint prediction.

###  6.4 Texture Finetuning

Fig. 14 shows how a quick test-time finetuning (100 iterations) of the
predicted texture improves their quality. This is especially effective for
images that are far from the training set distribution. Note that the textures
of the real horses in the main paper are single-pass predictions.

###  6.5 Failure Cases

Our texture prediction might not generalise well enough beyond the
distribution of textures observed during training. This is particularly
apparent when the trained model is applied on paintings and abstract drawings
of horses, neither of which are part of the training set. We demonstrate that
a quick finetuning step of the albedo network ($100$ iterations which takes
less than $5$ seconds) can remedy this shortcoming. Fig. 14 illustrates the
difference between the single-pass predicted textures and the finetuned
version.

The viewpoint prediction can fail in the case of more extreme and ambiguous
views as shown Fig. 9. This is often caused by DINO-ViT features that are less
distinctive for these particular views.

When the horse is observed from a side-view, the method might not be able to
disambiguate between left and right legs, for instance, in the second to last
row of Fig. 15. Note that our method uses only object masks and self-
supervised DINO-ViT features, neither of which are sufficient to disambiguate
between different legs of an animal.

###  6.6 Additional Qualitative Results

Additional results of single image reconstruction, animation and relighting
can be found in Fig. 15 and the supplementary video. More generalisation
results on abstract horse drawings, sculptures and toys are presented in Fig.
16, showing that the model has learned to estimate shape, pose and
articulation sufficiently robustly to generalise beyond the training
distribution.

##  7 Additional Technical Details

###  7.1 Articulation Model Details

Recall that our model estimates a set of bones and articulates the instance
mesh using a linear blend skinning model with predicted bone rotations. In the
following, we describe in detail how the rest-pose bones are estimated and how
the skinning weights are defined.

#### Bone Topology.

Our method only assumes a description of the topology of the animal’s
skeleton, and automatically estimates a set of bones at rest pose for the
articulation model based on simple heuristics. Specifically, for birds, we
estimate a chain of $8$ bones with equal lengths that lie on two line segments
going from the centre of the rest-pose mesh to the two most extreme vertices
along $z$-axis ($4$ bones on each side), forming a ‘spine’.

For quadrupedal animals, like horses, we further add $4$ sets of bones for
modelling the legs. We first identify the foot joints as the lowest points of
mesh (in $y$-axis) in each of four $xz$-quadrants. We then draw $4$ line
segments from the foot joints to their closest spine joints, and define a
chain of $3$ bones with equal lengths on each of the segments, representing
each leg.

#### Skinning Weights.

Recall eq. 2, where the instance mesh is further posed by a linear blend
skinning equation. Each vertex $V_{\text{ins},i}$ is associated with the bones
by a skinning weight $w_{i,b}$, defined as:

$\displaystyle w_{i,b}$
$\displaystyle=\frac{e^{-d_{i,b}/\tau_{\text{s}}}}{\sum_{k=1}^{B}{e^{-d_{i,k}/\tau_{\text{s}}}}},$
(5) $\displaystyle\text{where}\quad d_{i,b}$
$\displaystyle=\min_{r\in[0,1]}\|V_{\text{ins},i}-r\mathbf{J}_{b}-(1-r)\mathbf{J}_{\pi(b)}\|^{2}_{2}$

is the minimal distance from the vertex $V_{\text{ins},i}$ to each bone $b$
defined by the joint locations $\mathbf{J}_{b},\mathbf{J}_{\pi(b)}$ at rest
pose, and $\tau_{\text{s}}$ is a temperature parameter set to $0.5$.

#### Constraints on the Bone Rotations.

Our model learns complex articulated 3D poses of animals using reconstruction
losses on single-view images, without any explicit 3D geometric supervision,
which is an extremely ill-posed task. In order to prevent unnatural poses, we
enforce minimal constraints on the bone rotations: (1) all bone rotations are
limited to $(-60^{\circ},60^{\circ})$, (2) for quadrupeds, leg rotations
around $y$\- and $z$-axes (‘twist’ and ‘side-bending’) are further limited to
$(-18^{\circ},18^{\circ})$.

###  7.2 Network Architectures

We implement the feature field $\psi$, template SDF $s$ and the light network
$f_{\text{l}}$ using 5-layer MLPs, and the albedo field $f_{\text{a}}$ and
deformation field $f_{\Delta V}$ with 8-layer MLPs. The articulation network
consists of $4$ transformer blocks. All coordinate inputs are encoded using
$\texttt{sin}(\cdot)$ and $\texttt{cos}(\cdot)$ with $8$ frequencies.

The encoders are simple convolutional networks, described in Tab. 5.
Abbreviations of the components are defined as follows:

  * •

$\text{Conv}(c_{in},c_{out},k,s,p)$: 2D convolution with $c_{in}$ input
channels, $c_{out}$ output channels, kernel size $k$, stride $s$ and padding
$p$

  * •

$\text{GN}(n)$: group normalization [71] with $n$ groups

  * •

$\text{LReLU}(p)$: leaky ReLU [37] with a slope $p$

###  7.3 Hyper-parameters and Training Details

All hyper-parameters are listed in Tab. 4. We enable the articulation after
$10$k iterations and the deformation after $40$k iterations, to prevent the
model from overfitting individual images with excessive articulation and
deformation. During the first $5$k iterations, we allow the model to explore
all four viewpoint hypotheses by randomly sampling the four hypotheses
uniformly, and gradually decrease the chance of random sampling to $20\%$
while sampling the best hypothesis for the rest $80\%$ of the time. The
temperature $\tau$ is decreased from $1$ to $0.01$ over the course of $100$k
iterations. It takes roughly $20$ hours to train the full model for $150$k
iterations on one single NVIDIA A40 GPU.

Parameter | Value/Range
---|---
Optimiser | Adam
Learning rate on prior ($\psi$ and $s$) | $1\times 10^{-3}$
Learning rate on others | $1\times 10^{-4}$
Number of iterations |  $150$k
Batch size | $10$
Loss weight $\lambda_{\text{m}}$ | $10$
Loss weight $\lambda_{\text{dt}}$ | $10$
Loss weight $\lambda_{\text{im}}$ | $1$
Loss weight $\lambda_{\text{f}}$ | $10$
Loss weight $\lambda_{\text{E}}$ | $0.01$
Loss weight $\lambda_{\text{d}}$ | $10$
Loss weight $\lambda_{\text{a}}$ | $0.1$
Loss weight $\lambda_{\text{h}}$ | $1$
Image size | $256\times 256$
Field of view (FOV) | $25^{\circ}$
Camera location | $(0,0,10)$
Tetrahedral grid size | $256$
Initial mesh centre | $(0,0,0)$
Translation in $x$\- and $y$-axes | $(-0.4,0.4)$
Translation in $z$-axis | $(-1.0,1.0)$
Number of spine bones | $8$
Number of bones for each leg | $3$
Viewpoint hypothesis temperature $\tau$ | $(0.01,1.0)$
Skinning weight temperature $\tau_{\text{s}}$ | $0.5$
Ambient light intensity $k_{s}$ | $(0.0,1.0)$
Diffuse light intensity $k_{d}$ | $(0.5,1.0)$
Table 4: Training details and hyper-parameter settings. Table 5: Architecture of the encoders $f_{\text{k}}$, $f_{\text{o}}$ and the viewpoint network $f_{\text{v}}$. Encoder | Output size
---|---
Conv(384, 256, 4, 2, 1) + GN(64) + LReLU(0.2) |  16 $\times$ 16
Conv(256, 256, 4, 2, 1) + GN(64) + LReLU(0.2) |  8 $\times$ 8
Conv(256, 256, 4, 2, 1) + GN(64) + LReLU(0.2) |  4 $\times$ 4
Conv(256, 256, 4, 2, 0) $\rightarrow$ output |  1 $\times$ 1
Figure 8: Visualisations of the Viewpoint Prediction Distributions of the
Ablated Models. We demonstrate that both feature reconstruction loss
$\mathcal{L}_{\text{feat}}$ and multi-hypothesis viewpoint prediction are
needed to successfully recover a full range of viewpoints. The viewpoint
prediction collapses to a limited range as demonstrated by its azimuth without
these two components. Note that for Horses, we only predict the azimuth of the
viewpoint, as most of the horse images were taken with little elevation.
Figure 9: Incorrect Viewpoint Predictions. The viewpoint prediction can be
less reliable in the case of more extreme input views.

Figure 10: Reconstruction of Giraffes, Zebras and Cows. After finetuning on
new categories, our method generalises to various animal classes with highly
different underlying shapes. We show the predicted mesh from the input view
and three additional views together with animated versions of the shape
obtained by articulating the estimated skeleton. Finally, we show a relit
version.  Figure 11: Visual Comparison on Toy Bird Scans Evaluations. We
compare the reconstructed shapes with scanned ground-truth shapes from Toy
Bird Scans dataset. We show the reconstructed mesh from the input view and
three additional views. Our model is able to predict finer shape details
including the bird’s legs as opposed to the prior work of DOVE [68].  Figure
12: Comparison with DOVE [68] on Horses. We visualise the distribution of
predicted viewpoints on the test set together with additional qualitative
results. Our method is able to recover the full range viewpoint azimuth, while
DOVE covers only a portion of possible azimuths. This is further illustrated
by the qualitative results, where DOVE often fails to predict the correct
viewpoint as opposed to our method. Moreover, our predicted shape is far more
detailed. Note that for horses, we only predict azimuth rotations, as most of
the horse images were taken with little elevation.  Figure 13: Comparison with
Previous Methods on Horses. As in Fig. 12 we visualise the distribution of
predicted viewpoints on the test set together with additional qualitative
results. The plot of viewpoint prediction distribution on CUB test set shows
that our method is able to recover a wide range of viewpoints while UMR, which
uses a similar level of supervision, is able to predict only frontal poses. We
also present additional qualitative results on CUB test set demonstrating that
our method recovers shapes with greater details than previous works while
using significantly less supervision.  Figure 14: Texture Finetuning at Test
Time. We show a shape and texture prediction from an input view and one
additional view together with a finetuned version of the texture. We
demonstrate that a simple finetuning of the texture on the input image can
produce high-quality textures for images that are too far from the training
set distribution.  Figure 15: Reconstruction of Real Horse Images. We show the
predicted mesh from the input view and three additional views. We also
demonstrate that our shape can be animated by articulating the estimated
skeleton. Finally, as our method decomposes albedo and lightning, our
predictions can be easily relit.  Figure 16: Reconstruction of Abstract Horse
Drawings and Artefacts. As in Fig. 15, here we show the predicted meshes from
the input view and three additional views together with the animated and relit
versions. The results demonstrate excellent generalisation of our method on
images far from the distribution of the training set which consists only of
real horse images.


# Non-classical correlations between phonons and photons in a MHz-frequency
mechanical oscillator coupled to an optical cavity

Ivan Galinskiy  Niels Bohr Institute, University of Copenhagen, Blegdamsvej
17, 2100 Copenhagen, Denmark  Georg Enzian  Niels Bohr Institute, University
of Copenhagen, Blegdamsvej 17, 2100 Copenhagen, Denmark  Michał Parniak
Centre for Quantum Optical Technologies, Centre of New Technologies,
University of Warsaw, S. Banacha 2c, 02-097 Warsaw, Poland  Niels Bohr
Institute, University of Copenhagen, Blegdamsvej 17, 2100 Copenhagen, Denmark
Eugene Polzik  Niels Bohr Institute, University of Copenhagen, Blegdamsvej 17,
2100 Copenhagen, Denmark<EMAIL_ADDRESS>
###### Abstract

We demonstrate non-classical correlations between phonons and photons created
using opto-mechanical spontaneous parametric down-conversion in a system based
on a soft-clamped ultracoherent membrane oscillator inside of a Fabry-Pérot
optical resonator. We show that phonons stored in the mechanical oscillator,
when subsequently read out, display strong signs of quantum coherence, which
we demonstrate by single-photon counting enabled by our state-of-the-art
optical filtering system. We observe a violation of the classical Cauchy-
Schwarz inequality with a confidence of $>\qty{92}{}$. The presented system
demonstrates the potential for studies of low-frequency quantum effects in
sub-millimeter size nanogram-scale mechanical oscillators.

In recent years, there has been substantial progress towards increasingly
macroscopic systems that display quantum properties. From collective spins of
atomic gases and motion [1], both at room temperature and ultracold [2],
through nano- and microscopic mechanical oscillators [3], and recently with
high-mass systems in bulk resonators [4], quantum effects are now being
observed in multi- systems in gravitational wave detectors [5, 6]. Moreover,
combination of mechanical systems with quantum-level nonlinearities, such as
superconducting qubits [7, 8], have allowed to exploit their non-classical
behavior for quantum storage and processing.

While basic quantum effects such as ground state noise can be seen in even
very massive systems [5], it is more challenging to prepare a macroscopic
mechanical system in a demonstrably non-classical state. One of the most
prominent yet achievable quantum correlations to be explored with such systems
are two-mode squeezed states with correlations between the material system and
the optical or microwave probe. Such squeezed states have been explored across
various regimes, ranging from strong squeezing to particle pair generation.
Generation of a correlated pair of a photon and a material system excitation
has emerged as the crucial step of quantum repeaters, starting with the
original DLCZ protocol [9] for creation and subsequent distribution of
entanglement with light and atomic ensembles.

In the context of this work, recent results in [10] and domains have pushed
the boundary of non-Gaussianity in optomechanical systems, with DLCZ-based
single-phonon protocols [11], and even tomography being done on larger and
larger systems [4, 12].

In this letter, we demonstrate non-classical phonon-photon correlations
created by a mechanical oscillator with a size of 0.3, frequency of
${\Omega_{\mathrm{m}}\approx 2\pi\cdot\qty{1.4}{}}$, and an effective mass of
$m_{\text{eff}}\approx\qty{2}{\nano}$, which places us in the low-frequency,
high-mass regime. Observing such behavior requires precise control over most
of our experimental parameters, which we demonstrate in the domains of noise,
stability, and interaction tuning. By showing a significant violation of the
Cauchy-Schwarz inequality in the photon statistics, we demostrate non-Gaussian
behavior in a low-frequency, macroscopic purely optomechanical system.

Figure 1: Overview of the experimental setup. The optomechanical system (OM)
consists of a mechanical membrane oscillator placed inside a high Q optical
cavity. Single photons are filtered from the pump light with a set of filter
cavities and detected by a superconducting nanowire single photon detector
(SNSPD). The inset shows the fundamental mechanical mode of the membrane
oscillator in which single phonons are stored.

Our mechanical oscillator is the central “defect” of a square
$\mathrm{Si_{3}N_{4}}$ membrane with an overall size of 3 x 3 and a thickness
of 13. The majority of the area is occupied by a phononic shield that
surrounds the defect where the high-Q mode of oscillation is mostly localized
[13] (Fig. 1). The phononic shield has a vibrational bandgap engineered to be
centered on the frequency of the main mode. The shield prevents coupling
between the modes of the defect and the surroundings, allowing us to maintain
Q factors in excess of $200\text{\times}{10}^{6}$ at cryostat temperatures of
69

The phononic-crystal membrane is positioned in free space between two high-
reflectivity mirrors that form a Fabry-Pérot resonator. The mechanical motion
of the dielectric shifts the resonance frequency of this resonator which, upon
interaction with the optical intensity $EE^{*}$, is a fundamentally nonlinear
process that we exploit and which forms the basis of our system’s
optomechanical coupling. This arrangement is often denoted as “Membrane in the
Middle” (MIM). The experimental construction of the optomechanical resonator
consists of a copper holder that houses the membrane, mirrors, and
piezoactuators necessary for tuning, as shown in Fig. (2). We prevent
environmental vibrations from affecting the stability of our system by
suspending the whole cavity assembly on four steel springs, with a copper
braid serving to be the thermal link between the OM cavity and the cryostat.
The optical arrangement consists of two super-polished mirrors 111Fabricated
by FiveNine Optics, forming a cavity with an intrinsic finesse of
approximately $11\text{\times}{10}^{3}$. With a resonator length of 18, the
resulting optical bandwidth is 0.71, with the precise value being dependent on
the relative position between the standing optical wave and the membrane.

In order to generate non-classical motional states, we base our approach on
conditional (heralded) mechanical state preparation, where the heralding comes
from single-photon detection, as theoretically discussed by [15] and
experimentally demonstrated in atomic [16, 17], and optomechanical [11, 10]
systems. For this purpose, we exploit the sideband-resolved properties of our
system. Sideband-resolved operation allows us to change the kind of
optomechanical interaction by either red- or blue-detuning the pumping laser
field with respect to the optical resonance.

Our protocol utilizes a basic step of a quantum repeater protocol [18], where
we begin by creating low gain two-mode squeezing between the mechanical and
optical modes using the “write” pulse, as shown schematically in Fig. 3. As a
result, a photon in the propagating optical mode, when detected by a single-
photon counter, serves as the “herald” of creation of a phonon of mechanical
excitation. The mechanical mode, localized to the membrane, is therefore
“stored” for a chosen time delay, after which it can be converted back into
the optical domain by applying a “read” pulse. Effectively, this leads to
entangled photon-phonon modes, which result in two entangled time-separated
optical modes. Correlations between these two modes are the primary indicator
of non-classical behavior in our system.

The basis of our preparation and measurement scheme is the sideband-resolved
operation of our optomechanical system, as its optical decay rate (cavity
linewidth) $\kappa\approx 2\pi\times\qty{0.8}{}$ is significantly smaller than
the mechanical frequency $\Omega\approx 2\pi\times\qty{1.4}{}$. In the scheme
described by [15], this leads to two kinds of interplay between mechanical
motion and optical cavity mode: beamsplitter interaction and two-mode
squeezing interaction, to which we will refer as spontaneous parametric
downconversion, or SPDC. Beamsplitter interaction, or anti-Stokes scattering,
allows us to perform state swapping between light and mechanics. This mode of
operation is achieved in practice when the pump field is detuned to the “red
side” of the optomechanical cavity’s optical resonance ($\Delta=-\Omega$)
[19], favoring swapping of mechanical information onto the optical (displaced)
vacuum state. SPDC interaction, on the other hand, leads to Stokes scattering,
or the creation of phonon-photon pairs. Experimentally, we activate this
interaction by detuning the pump field to the “blue” side of the resonance
($\Delta=+\Omega_{\mathrm{m}}$).

As the main tool of our investigations of mechanical state properties, we use
photon counting statistics. As mentioned previously, correlations between
scattered photons from the “write” and “read” pulses are equivalent to
correlations between the optical and mechanical state after a period of free
evolution . The statistics, discussed in more detail below, can be used as a
proxy to determine the “classicality” of a system [20]. More precisely, in our
case a violation of the Cauchy-Schwarz inequality demonstrates non-classical
correlations between two time-separated optical modes. The inequality, when
applied to classical states of light (for which the Glauber–Sudarshan P
function has the properties of a probability distribution), yields:

$\mathinner{\\!\left\lVert g^{(2)}_{12}\right\rVert}^{2}\leq g^{(2)}_{1}\cdot
g^{(2)}_{2},$ (1)

where we have defined a shorthand notation for the two-time (two-mode)
correlation $g^{(2)}_{12}\equiv g^{(2)}(t_{1},t_{2})$, and autocorrelation
$g^{(2)}_{n}\equiv g^{(2)}(t_{n},t_{n})$. It is convenient to define the
_Cauchy-Schwarz parameter_ , that we will denote as $R$, such that

$R\equiv\frac{\mathinner{\\!\left\lVert
g^{(2)}_{12}\right\rVert}^{2}}{g^{(2)}_{1}\cdot g^{(2)}_{2}}.$ (2)

Therefore, if $R>1$, then the Cauchy-Schwarz inequality of Eq. (1) is not
applicable, indicating non-classical photon statistics, where $P(\alpha)$ can
no longer be interpreted as a probability distribution.

Note that the knowledge of $g^{(2)}_{12}$ per se is not enough to guarantee
non-classicality, as there exist classical sources with strong bunching
($g^{(2)}_{1}\gg 1$). It is only the Cauchy-Schwarz parameter that can
establish non-classicality in this case.

To reiterate, our experiment measures correlations between photons detected
during two pulses, which we call “write” and “read” respectively. During
“writing”, the optomechanical system undergoes a process equivalent to SPDC.
There, pump photons are scattered down in energy, while creating excitations
in the mechanical oscillator, leading to entangled phonon-photon pairs,
corresponding to the interaction Hamiltonian
$\hat{H}_{\text{int}}\propto\hat{a}^{\dagger}\hat{b}^{\dagger}+\text{h.c.}$.
In the low-gain regime relevant for our case, this interaction leads to a
state $\ket{0,0}+\alpha\ket{1,1}$ with the first (second) index corresponding
to the photon (phonon) number, and the probability amplitude being $\alpha\ll
1$.

While the created phonon remains localized in the mechanical oscillator, the
photon can be detected externally. In what is rather similar to the DLCZ
protocol, the detected photon serves as a herald of the scattering event. As
photodetection is a fundamentally nonlinear process, the detection of this
photon under appropriate conditions (discussed further down) is capable of
collapsing the entangled mechanics-light state onto a mechanical single-phonon
Fock state.

During “reading”, the interplay changes to that of a beamsplitter, where the
state of the intracavity optical state is exchanged with the mechanical state,
to a degree controlled by the strength and duration of the reading pulse. This
corresponds to the beamsplitter interaction Hamiltonian:
$\hat{H}_{\text{int}}\propto\hat{a}^{\dagger}\hat{b}+\text{h.c.}$. This
phonon-to-photon conversion is quite efficient in our case, since the duration
and intensity of the “read” pulse can be adjusted to approach arbitrarily
close to 100 swap efficiency. Quantum cooperativity, defined as the ratio
between the rates of reading and decoherence [19], is $C_{\mathrm{q}}\approx
10\gg 1$ for our experiment, leads to $\approx 0.1$ added photons during
“reading”.

A prominent feature of our mechanical oscillator is its macroscopic size.
However, this results in a relatively low mechanical frequency of the
oscillator which, in turn, constitutes a significant challenge for detection
of single photons at the mechanical sideband frequencies. The frequency
difference between the sidebands created by the mechanical motion and the base
optical frequency is only 1.4, corresponding to a fractional difference of
less than $4\text{\times}{10}^{-9}$. To achieve supression of the unscattered
pump light and neighboring mechanical modes, we designed and built a state-of-
the-art filter system. The filtering is performed by an array of four 60-long
Fabry-Pérot resonators, positioned in series along the path of light that has
interacted with the optomechanical system. The individual linewidth of each
filter resonator is approximately 30, chosen to be broad enough to allow the
optically broadened mechanical signal to be transmitted, while sufficiently
blocking the unscattered pump light, and scattering by nearby out-of-bandgap
mechanical modes. The in-series arrangement compounds the individual transfer
functions into an 8-order effective response, providing ample suppression of
nearly all undesired photons. In the current implementation, we observe an
end-to-end signal transmission of approximately 30, dominated by losses due to
roughness of cavity mirror substrates and coupling imperfections.

(a)
---
(b)
---
(c)
---
Figure 2: Optomechanical cavity suspended on springs inside the helium flow
cryostat (a), with a schematic representation (b), and a cross-section (c)
showing the positioning of the membrane chip, mirrors, and piezoactuators.
Radiation shield and vacuum chamber enclosure removed for illustration
purposes.

Classical noise is a large concern to low-frequency systems such as ours, and
is challenging to counteract. Some of the largest sources of classical
fluctuations are: phase noise coming from our laser system, thermal motion of
the mirrors of the optomechanical cavity, and electronic noise transduced into
the light via e.g. electro-optic modulators. All these sources contribute to
an increase of phase noise in the light entering the system. Especially in the
context of sideband-resolved operation, phase noise is transduced into
intensity noise inside of the optomechanical cavity, leading to force noise on
the mechanics. Additionally, optical noise resonant with the optomechanical
cavity contributes spurious photon counts, which worsen the signal-to-noise
ratio during single-photon counting.

To address phase noise coming from the laser system, we implemented a realtime
feedback system based on an unbalanced Mach-Zehnder interferometer, described
in detail in [21]. A highly unbalanced interferometer, where the arms have an
optical path difference of 70, is used to transduce phase noise of light into
a measurable intensity signal at a frequency offset of 1.4. The measured
signal is then used to apply feedback on an electro-optic modulator,
efficiently cancelling the phase fluctuations at that frequency. As a result,
we achieve a reduction of phase noise by more than 8 with respect to our
laser’s intrinsic noise, down to a phase noise spectral density of -164 [21].

Mirror noise, i.e. thermal motion of reflective surfaces of the optomechanical
cavity’s mirrors , has an effect similar to that of external phase noise [22].
Even at liquid-helium temperatures of our cryostat and a cavity temperature of
$\approx\qty{8}{K}$, this motion is substantial enough to introduce practical
limits on optical cooling and reading. In our design, we work around this
limitation by engineering the key dimensions of our mirrors to avoid any
frequency-domain overlap between the mechanical modes of the mirrors and of
the membrane’s main mode. This optimized geometry yields a limit of
optomechanical cooling, i.e. minimum achievable phonon number, of
$\approx$0.2$$ for our system.

The experiment runs in a pulsed regime, with each of the experimental “shots”
consisting of three main stages: ground-state cooling of the mechanical
oscillator, SPDC-like creation of a photon-phonon pair, and readout of the
created phonon onto a photon. We shall refer to these steps as “cool”,
“write”, and “read”, schematically shown in Fig. 3.

$+\Omega_{\mathrm{m}}$
---
$\hat{a}^{\dagger}\hat{b}$
---
$\hat{a}^{\dagger}\hat{b}^{\dagger}$
---
$-\Omega_{\mathrm{m}}$
---
Cool
---
Write
---
Read
---
$0$
---

Detuning
---

$\hat{a}^{\dagger}\hat{b}$
---
Figure 3: Schematic representation of the protocol used in the experiment. The
top plot is a spectrogram illustrating timing and frequencies of optical
drives entering the optomechanical cavity (red and blue traces), and the
generation of scattered photons resonant with the cavity (yellow). The bottom
drawing illustrates the equivalent “optical diagram” of the sequence,
consisting of a beamsplitter - SPDC - beamsplitter chain of interactions. See
main text for detailed description of the interactions.

The cooling pulse is necessary to bring the motion of the main mode of our
membrane as close to the ground state as possible, as residual thermal
occupation has detrimental effects on the rest of the experiment [15]. During
idle time between experimental sequences, the cooling beam is still active,
though at a much lower power, which prevents spurious heating of the
optomechanical cavity mirrors. At this low cooling level the mechanical mode
equilibrates to a relatively high occupation of about $100$ phonons, while the
cooling pulse at the beginning of each experimental shot serves to bring the
phonon number down to less than one phonon.

After the occupation is reduced by the cooling process, the red-detuned beam
is shut off and a blue-detuned beam is turned on to enable the SPDC
interaction. Spontaneous parametric downconversion interaction begins creating
phonon-photon pairs in accordance to its Hamiltonian. While this interaction
is also equivalent to two-mode squeezing, single-photon detection introduces
the nonlinearity necessary for non-Gaussian state preparation. The power and
duration of the write pulse are set such that the probability of scattering is
low enough, in accordance to our heralded probabilistic scheme.

Finally, after the writing pulse, the mechanical state is read out onto light.
The state readout efficiency, dependent on the interaction strength and length
of the reading pulse, is essentially 100, as we are free to prolong the pulse
if necessary. However, the non-unity quantum efficiency of cavity
overcoupling, the “downstream” light propagation, and detection limits our
overall collection efficiency to approximately 5. The read out phononic mode
converted into a light pulse propagates to the single-photon counter, where it
is detected, serving its role as the second mode in “two-mode correlations”.

As shown in the bottom half of Fig. (3), the whole experimental sequence can
be graphically represented as an optical diagram, noting that the black-
colored “phonon beam propagation” is, in fact, propagation in time. The
interaction between the light and mechanics, shown in the top half of Fig.
(3), is represented as an “optomechanical spectrogram”, where the optical
drives (red and blue lines) are scattered by their corresponding interaction
Hamiltonians. The scattered photons, shown in yellow, are resonant with the
optomechanical cavity and are later transmitted by our filtering system.

Each full experimental run consists of many “shots”, usually in the range of
millions, as our overall collection efficiency is a relatively low 5. An
example of resulting photon rates, averaged over many experimental “shots”, is
shown in Fig. 4, where one can see the cool-write-read sequence. The driving
field strength is shown in the middle graph. Cooling is happening between
150650, followed by the “write” pulse between 750850 , and the “read” pulse
starting around 950 . The resulting photon counts are shown in the top plot,
and the estimated phonon occupation in the bottom graph.

${10^{2}}$${10^{3}}$${10^{4}}$Rate ( )Photon
rate${10^{-2}}$${10^{-1}}$${10^{0}}$Power (a.u.)Drive power02505007501000Time
()${10^{0}}$${10^{1}}$PhononsPhonon occupation Figure 4: Experimental photon
rates during the full sequence (top), driving laser pulses (middle) and
estimated phonon occupation (bottom). See main text for details.

In order to study the behavior of our system in different regimes, we present
two experimental runs. The first run is optimized for high excitation
probability with a gain parameter $g_{w}T_{w}\approx 1$, and increased rate of
pair generation, useful for getting insight into the overall behavior of the
system, as we prepared for the lower-probability run that showed non-classical
correlations.

The high-probability run allowed us to gather significantly more events, which
we used to characterize the general statistical behavior of our system,
valuable despite its reduced level of quantum correlations. As shown in Fig.
5, the correlation map shows distinct off-diagonal features, with increased
$g^{(2)}$. The increased number of events in this particular run (about $6000$
coincidences) allowed us to compute statistics with less uncertainty.
Unfortunately, higher levels of excitation lead, by their nature, to decreased
quantum correlations in schemes such as ours. With this dataset, we estimate a
Cauchy-Schwarz parameter of $0.78(6)$, which is still significantly higher
than the value of $0.5$ expected if the system was purely thermal.

0.00.20.40.60.81.01.2$t_{1}$ ()0.00.20.40.60.81.01.2$t_{2}$
()$g^{(2)}(t_{1},t_{2})$0.51.01.52.02.53.0 Figure 5: Correlation map for an
experiment with an increased preparation probability. The dotted lines
indicate the time of the “write” and “read” pulses (cf Fig. 4). The off-
diagonal regions with ${g^{(2)}>0.5}$ in the map corresponding to write-read
correlations are clearly seen. The rectangles to the top and right of the
intersection are the regions of interest corresponding to non-trivial values
of $R$. The color map threshold of ${g^{(2)}>0.5}$ is chosen for aesthetics,
as $g^{(2)}$ per se is insufficient for conclusions on non-classicality.

In contrast, the experimental run with low probability of excitations
($g_{w}T_{w}\approx 0.1$) shows a less visually suggestive correlation map
(Fig. 6). Despite that, after accumulating the equivalent of about $200$
coincidences, we performed data analysis of the events. This included taking
proper time sub-sections of the whole data trace to maximize the overlap with
temporal modes. With this, we obtained an overall Cauchy-Schwarz parameter of
$R=$1.37(26)$$, which violates the classical limit of $R=1$. The corresponding
probability of violation of the Cauchy-Schwarz inequality is $>\qty{92}{}$
(p-value of $0.08$).

0.00.20.40.60.81.01.2$t_{1}$ ()0.00.20.40.60.81.01.2$t_{2}$
()$g^{(2)}(t_{1},t_{2})$0.51.01.52.02.53.0 Figure 6: Correlation map for an
experiment where the preparation probability was set to low values in order to
achieve non-classical values of the Cauchy-Schwarz parameter. Off-diagonal
regions, corresponding to write-read correlations, show an indication of non-
trivial correlations.

Possible avenues for further improvements have to do with an increase in
overall efficiency. Our losses are not dominated by any single source, but are
rather distributed along the many components in the light path. Notably, the
system of filter cavities in its current state has a signal transmission of
approximately 30, with the rest of the losses being mostly spread across fiber
interconnects, mode matching, and internal losses of the optomechanical
cavity. The strength of the optomechanical interaction, proportional to
$g_{0}$, can be increased by a more resilient system of mounting the mirrors
and the membrane with respect to each other, which would increase the
optomechanical mode overlap. Nevertheless, even with the limitations of our
current setup, our results suggest that mechanical non-classicality is
achievable with photon counting even in such a low-frequency and large size
mechanical system with modest cooling. Further improvements, including
heralded preparation of single-phonon states, are within reach for membrane
optomechanics.

This research has been funded by European Research Council (ERC) Advanced
Grant QUANTUM-N under the EU Horizon 2020 (grant agreement no. 787520) and by
Villum Fonden under a Villum Investigator Grant, grant no. 25880. G.E.
acknowledges support from the European Union’s Horizon 2020 research and
innovation programme under the Marie Sklodowska-Curie grant agreement no.
847523.

## References

  * Thomas  _et al._ [2021] R. A. Thomas, M. Parniak, C. Østfeldt, C. B. Møller, C. Bærentsen, Y. Tsaturyan, A. Schliesser, J. Appel, E. Zeuthen, and E. S. Polzik, Entanglement between distant macroscopic mechanical and spin systems, Nature Physics 17, 228 (2021).
  * Karg  _et al._ [2020] T. M. Karg, B. Gouraud, C. T. Ngai, G.-L. Schmid, K. Hammerer, and P. Treutlein, Light-mediated strong coupling between a mechanical oscillator and atomic spins 1 meter apart, Science 369, 174 (2020).
  * Ockeloen-Korppi  _et al._ [2018] C. F. Ockeloen-Korppi, E. Damskägg, J.-M. Pirkkalainen, M. Asjad, A. A. Clerk, F. Massel, M. J. Woolley, and M. A. Sillanpää, Stabilized entanglement of massive mechanical oscillators, Nature 556, 478 (2018).
  * Bild  _et al._ [2023] M. Bild, M. Fadel, Y. Yang, U. von Lüpke, P. Martin, A. Bruno, and Y. Chu, Schrödinger cat states of a 16-microgram mechanical oscillator, Science 380, 274 (2023).
  * Acernese  _et al._ [2020] Acernese _et al._ (The Virgo Collaboration), Quantum backaction on kg-Scale mirrors: Observation of radiation pressure noise in the advanced virgo detector, Physical Review Letters 125, 131101 (2020).
  * Whittle  _et al._ [2021] C. Whittle, E. D. Hall, S. Dwyer, N. Mavalvala, V. Sudhir, R. Abbott, A. Ananyeva, C. Austin, L. Barsotti, J. Betzwieser, C. D. Blair, A. F. Brooks, D. D. Brown, A. Buikema, C. Cahillane, _et al._ , Approaching the motional ground state of a 10-kg object, Science (New York, N.Y.) 372, 1333 (2021), https://www.science.org/doi/pdf/10.1126/science.abh2634 .
  * Chu  _et al._ [2017] Y. Chu, P. Kharel, W. H. Renninger, L. D. Burkhart, L. Frunzio, P. T. Rakich, and R. J. Schoelkopf, Quantum acoustics with superconducting qubits, Science 358, 199 (2017).
  * Gustafsson  _et al._ [2014] M. V. Gustafsson, T. Aref, A. F. Kockum, M. K. Ekström, G. Johansson, and P. Delsing, Propagating phonons coupled to an artificial atom, Science 346, 207 (2014).
  * Duan  _et al._ [2001a] L.-M. Duan, M. D. Lukin, J. I. Cirac, and P. Zoller, Long-distance quantum communication with atomic ensembles and linear optics, Nature 414, 413 (2001a).
  * Lee  _et al._ [2012] K. C. Lee, B. J. Sussman, M. R. Sprague, P. Michelberger, K. F. Reim, J. Nunn, N. K. Langford, P. J. Bustard, D. Jaksch, and I. A. Walmsley, Macroscopic non-classical states and terahertz quantum processing in room-temperature diamond, Nature Photonics 6, 41 (2012).
  * Hong  _et al._ [2017] S. Hong, R. Riedinger, I. Marinković, A. Wallucks, S. G. Hofer, R. A. Norte, M. Aspelmeyer, and S. Gröblacher, Hanbury Brown and Twiss interferometry of single phonons from an optomechanical resonator, Science 358, 203 (2017).
  * Enzian  _et al._ [2021] G. Enzian, L. Freisem, J. J. Price, A. Ø. Svela, J. Clarke, B. Shajilal, J. Janousek, B. C. Buchler, P. K. Lam, and M. R. Vanner, Non-Gaussian Mechanical Motion via Single and Multiphonon Subtraction from a Thermal State, Physical Review Letters 127, 243601 (2021).
  * Tsaturyan  _et al._ [2017] Y. Tsaturyan, A. Barg, E. S. Polzik, and A. Schliesser, Ultracoherent nanomechanical resonators via soft clamping and dissipation dilution, Nature Nanotechnology 12, 776 (2017).
  * Note [1] Fabricated by FiveNine Optics.
  * Galland  _et al._ [2014] C. Galland, N. Sangouard, N. Piro, N. Gisin, and T. J. Kippenberg, Heralded single-phonon preparation, storage, and readout in cavity optomechanics, Physical Review Letters 112, 1 (2014), arxiv:1312.4303 .
  * Dideriksen  _et al._ [2021] K. B. Dideriksen, R. Schmieg, M. Zugenmaier, and E. S. Polzik, Room-temperature single-photon source with near-millisecond built-in memory, Nature Communications 12, 3699 (2021).
  * Corzo  _et al._ [2019] N. V. Corzo, J. Raskop, A. Chandra, A. S. Sheremet, B. Gouraud, and J. Laurat, Waveguide-coupled single collective excitation of atomic arrays, Nature 566, 359 (2019).
  * Duan  _et al._ [2001b] L.-M. Duan, M. D. Lukin, J. I. Cirac, and P. Zoller, Long-distance quantum communication with atomic ensembles and linear optics, Nature 414, 413 (2001b).
  * Aspelmeyer  _et al._ [2014] M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, Cavity optomechanics, Reviews of Modern Physics 86, 1391 (2014).
  * Reid and Walls [1986] M. D. Reid and D. F. Walls, Violations of classical inequalities in quantum optics, Physical Review A 34, 1260 (1986).
  * Parniak  _et al._ [2021] M. Parniak, I. Galinskiy, T. Zwettler, and E. S. Polzik, High-frequency broadband laser phase noise cancellation using a delay line, Optics Express 29, 6935 (2021).
  * Rabl  _et al._ [2009] P. Rabl, C. Genes, K. Hammerer, and M. Aspelmeyer, Phase-noise induced limitations on cooling and coherent evolution in optomechanical systems, Physical Review A 80, 063819 (2009).