# Universality in driven Potts models

Tim Herpich  Electronic Mail<EMAIL_ADDRESS>Massimiliano Esposito
Complex Systems and Statistical Mechanics, Physics and Materials Science
Research Unit, University of Luxembourg, L-1511 Luxembourg, Luxembourg

(August 27, 2024)

###### Abstract

We study the stochastic dynamics of infinitely many globally interacting units
made of $q$ states distributed uniformly along a ring that is externally
driven. While repulsive interactions always lead to uniform occupations,
attractive interactions give rise to much richer phenomena: We analytically
characterize a Hopf bifurcation which separates a high-temperature regime of
uniform occupations from a low-temperature one where all units coalesce into a
single state. For odd $q$, below the critical temperature starts a
synchronization regime which ends via a second phase transition at lower
temperatures, while for even $q$ this intermediate phase disappears. We find
that interactions have no effects except below critical temperature for
attractive interactions. A thermodynamic analysis reveals that the dissipated
work is reduced in this regime, whose temperature range is shown to decrease
as $q$ increases. The $q$-dependence of the power-efficiency trade-off is also
analyzed.

##  I Introduction

While the thermodynamics of equilibrium phase transitions in interacting
systems has a long history and is well-documented Landau and Lifshitz (1994);
Yeomans (1992), it is only as of recent that the thermodynamics of
nonequilibrium phase transitions started to be explored Herpich  _et al._
(2018); Vroylandt  _et al._ (2017); Tomé and de Oliveira (2012); Zhang and
Barato (2016); Crosato  _et al._ (2018, 2018); Vaikuntanathan  _et al._
(2014); Marcuzzi  _et al._ (2014). This delay can be attributed to the lack of
a theory that systematically describes the thermodynamics of out-of-
equilibrium processes. Over the past two decades, it has become evident that
such a theory is embodied by stochastic thermodynamics that characterizes
thermodynamic properties in systems exhibiting Markovian stochastic dynamics
Seifert (2012); den Broeck and Esposito (2015); Zhang  _et al._ (2012);
Sekimoto (2010). Here, the time-scales of the dynamics are specified by
transition rates that incorporate the energetics of the nonequilibrium system
via the so-called local detailed balance condition Esposito  _et al._ (2010,
2009a). Stochastic thermodynamics has been successfully applied to
characterize energy transduction in noninteracting or interacting few-body-
systems Seifert (2012); Benenti  _et al._ (2017); Proesmans  _et al._ (2016).
Recently, also interacting many-body thermodynamics and the relation between
interactions and power-efficiency trade-offs have been investigated Herpich
_et al._ (2018); Vroylandt  _et al._ (2017); Imparato (2015); Golubeva and
Imparato (2012). Yet, the simplicity of these models allows to explicitly
solve the dynamics. In this letter we demonstrate how stochastic
thermodynamics allows to qualitatively capture a rich dynamical phenomenology
of systems that are too complex to be addressed explicitly.

A popular model in statistical mechanics that exhibits an equilibrium phase
transition is the Potts model Ashkin and Teller (1943) which generalizes the
special case of the Ising model Ising (1925); Onsager (1944) ($q=2$) by
considering interacting spins on a lattice that can take $q$ different values
distributed uniformly about a circle. While in equilibrium statistical
mechanics this model has been largely explored Wu (1982), little is known
about its out-of-equilibrium properties. Progress in that direction was made
in Ref. Herpich  _et al._ (2018) that studies an all-to-all interacting and
driven three-state model (Potts model with $q=3$) across different scales.
This work generalizes the previous one by considering all-to-all interacting
and driven Potts models at the mean-field level with variable $q$. The
dynamics is modeled by thermodynamically consistent Arrhenius rates. We find
that for repulsive interactions there is no phase transition. For attractive
interactions we characterize via thermodynamic principles the distinct
stationary states in the low- and high-temperature regime that are universal
for all finite $q$. The key result is that we derive the $q$-dependent
critical Hopf bifurcation temperature. Numerically, we show that for even $q$
this phase transition separates the high-temperature and the low-temperature
phase, whereas for odd $q$ there are stable limit cycles implying the
existence of a second phase transition. Finally, the dissipated work and the
power-efficiency trade-off in the different phases are discussed.

##  II Model

We consider infinitely many units made of $q$ states with energies
$\epsilon_{i}$ ($i=1,2,\ldots q$) distributed uniformly along a ring in
contact with a heat bath at inverse temperature $\beta=(k_{b}T)^{-1}$, where
we set $k_{b}\equiv 1$ in the following. Any unit in a given state $i$
interacts with all other units that occupy the same state $i$ with the global
coupling constant $u$. The system is autonomously driven by a global and non-
conservative force $f$ that creates a bias along the ring with the rotational
orientation $1\to 2\to\ldots\to q\to 1$, as depicted in the figure in appendix
A that depicts a finite number of $q$-state units with $q=8$. The system is
fully characterized by the occupation densities $\mathcal{P}_{i}$ of the
states $i=1,\ldots,q$ that we identify as the probabilities for any unit in
the mean-field to occupy these states.

We assume that the dynamics of the jump process is governed by a Markovian
master equation (ME)

$\displaystyle\dot{\mathcal{P}}_{i}\\!$
$\displaystyle=\sum\limits_{j}\mathcal{V}_{ij}\,\mathcal{P}_{j}\,,$ (1)

where we choose Arrhenius transition rates from state $j$ to $i$,
$\mathcal{V}_{ij}=\Gamma\,\exp[-\frac{\beta}{2}\\!\left(\Delta\mathcal{E}(i,j)-\Theta(i,j)f\right)]$,
with the kinetic prefactor $\Gamma$ and the function $\Theta(i,j)$ that
selects transitions between adjacent states along the ring according to their
alignment with or against the bias $f$ and that is defined as $\Theta(i,j)=1$
for $(i-j)\text{ mod }q=\\!1$ and $\Theta(i,j)\\!=\\!-1$ otherwise. Moreover,
the change in energy due to that transition is given by
$\Delta\mathcal{E}(i,j)=\epsilon_{i}-\epsilon_{j}+u(\mathcal{P}_{i}-\mathcal{P}_{j})$.
Thus, the transition rates satisfy the local detailed balance condition,
_i.e._

$\displaystyle\ln\frac{\mathcal{V}_{ij}}{\mathcal{V}_{ji}}$
$\displaystyle=-\beta\left(\epsilon_{i}-\epsilon_{j}+u(\mathcal{P}_{i}-\mathcal{P}_{j})-\Theta(i,j)\,f\right),$
(2)

ensuring that the system is thermodynamically consistent. We furthermore note
that probability conservation, $\sum_{i}\mathcal{P}_{i}=1$, removes one degree
of freedom such that the system has $q-1$ dimensions. In the appendix A, the
mean-field Eq. (1) is derived from a microscopic stochastic description for
the many-body problem as an asymptotically exact Eq. in the limit of
infinitely many units.

##  III Dynamics

For a flat energy landscape of the units, $\epsilon_{i}=\epsilon\;\forall i$,
the nonlinear Eq. (1) is solved by a uniform probability distribution

$\displaystyle\mathcal{P}_{i}^{s}=1/q,\quad i=1,2,\ldots q.$ (3)

Yet, no statements can be made about the stability of that fixed point without
a stability analysis that for $q>4$ is difficult.

We demonstrate that the thermodynamic consistency of the Potts model encoded
in Eq. (2) constrains the dynamics and even allows to generically specify the
long-time solution in the low- and high-temperature regime. First, we note
that the high-temperature limit $\beta\to 0$ represents a reversible limit for
finite $f$ since detailed balance
$\mathcal{V}_{ij}\,\mathcal{P}_{j}^{eq}=\mathcal{V}_{ji}\,\mathcal{P}_{i}^{eq},\,\forall
i,j$ holds. Equilibrium statistical mechanics prescribes that the system
behaves entropically and thus the uniform probability distribution from Eq.
(3) represents a stable and unique fixed point. Next, the low-temperature
limit, $\beta\to\infty$ represents a totally irreversible limit for finite
$f$, where Eq. (1) reduces to
$\dot{\mathcal{P}}_{i}\\!=\mathcal{V}_{i,i-1}\,\mathcal{P}_{i-1}-\mathcal{V}_{i+1,i}\,\mathcal{P}_{i}$.
Here, the driving $f$ is acting like a renormalization of the kinetic
prefactor. This suggests that occupation is the only relevant factor that
determines where the dynamics goes to in the long-time limit, _i.e._ the
system with irreversible rates behaves energetically like an equilibrium one
would. In this case one has to distinguish between repulsive ($u>0$) and
attractive ($u<0$) interactions. For the former the system has a unique energy
ground state that coincides with the entropic state from Eq. (3). Conversely,
for attractive interactions there are $q$ energy ground states where all units
occupy the same state

$\displaystyle\mathcal{P}_{i}^{s}=1,\;\mathcal{P}_{j\neq i}^{s}=0,\quad
i=1,2,\ldots,q.$ (4)

These states indeed correspond to fixed points as can be readily verified by
inserting Eq. (4) into the ME (1). To which of those fixed points the dynamics
is striving depends on which state $i$ is initially the most populated one. We
emphasize that these results hold for any finite number of states $q$ and
finite autonomous driving $f$.

At all temperatures, the stability of the symmetric fixed point,
$\mathcal{P}_{i}^{*}\equiv 1/q$, is encoded in the spectrum of the linearized
Jacobian, $\bm{A}_{ij}\\!\\!\\!\leavevmode\nobreak\ \equiv\leavevmode\nobreak\
\\!\\!\\!\sum_{k}[\partial(\mathcal{V}_{ik}\mathcal{P}_{k})/\partial\mathcal{P}_{j}]|_{\bm{\mathcal{P}}^{*}}$.
For $q\leq 4$ we find that the critical point $\beta_{c}$ obeys the relation

$\displaystyle(q+\beta_{c}u)=0.$ (5)

For repulsive interactions this relation is never satisfied and the system
remains in the symmetric fixed point at all temperatures.

A key finding is that for attractive interactions the dynamics has a rich
phenomenology: The main result is that we prove that relation (5) also
characterizes the critical temperature for $q>4$. To this end, we evaluate the
linear stability matrix for $q>2$ at the critical temperature (5) and obtain

$\displaystyle\bm{A}_{ij}$
$\displaystyle=\\!c\big{[}(\delta_{i+1,j}\big{|}_{i\neq
q}-\delta_{i-1,j}\big{|}_{i\neq
0})\\!+\delta_{i,q}\,\delta_{0,j}-\delta_{i,0}\,\delta_{q,j}\big{]},$ (6)

with $c=\Gamma\sinh\left(qf/(2u)\right)$. This skew-symmetric circulant matrix
admits the eigenvalues Davis (1970)

$\displaystyle\lambda_{k}$
$\displaystyle=2\mathrm{i}\,c\sin\left(k\frac{2\pi}{q}\right),\qquad
k=0,1,\ldots,q-1,$ (7)

that are thus either identical zero or purely imaginary. For $q=2$ the linear
stability matrix, $A_{ij}=0$, has only zero eigenvalues. Next, we consider
temperatures in the vicinity of the critical temperature,
$\beta_{c}+\delta\beta$, and expand the linear stability matrix in Eq. (6) up
to linear order,
$\bm{A}(\beta_{c}+\delta\beta)=\bm{A}(\beta)\big{|}_{\beta_{c}}+\delta\beta\;\bm{B}+\mathcal{O}\big{(}\delta\beta^{2}\big{)}$,
with the first-order correction matrix

$\displaystyle\bm{B}_{ij}$
$\displaystyle=a\delta_{ij}+b(\delta_{i+1,j}+\delta_{i,q}\delta_{0,j})+c(\delta_{i-1,j}+\delta_{i,0}\,\delta_{q,j})$
$\displaystyle a$ $\displaystyle=\\!-\\!\tfrac{2\Gamma
u}{q}d,\,b=\Gamma\tfrac{2u\\!-\\!qf}{2q}d,\,c=\Gamma\tfrac{qf\\!+\\!2u}{2q}d,\,d\\!=\\!\cosh\big{(}q\tfrac{f}{2u}\big{)},$

which has the eigenvalues

$\displaystyle\tilde{\lambda}_{k}\\!=\\!-\tfrac{2\Gamma
d\;\sin^{2}\\!\big{(}\tfrac{k\pi}{q}\big{)}}{q}\\!\bigg{[}2u\\!+\\!\mathrm{i}fq\cot\big{(}\tfrac{k\pi}{q}\big{)}\\!\bigg{]}\\!,\,k=0,1,\ldots,q\\!-\\!1.$

We note that for all $q$,
$\operatorname{Re}[\tilde{\lambda}_{k}]\geq\\!0\;\forall k$, such that the
real parts of all perturbed eigenvalues,
$\operatorname{Re}[\lambda_{k}]+\delta\beta\,\operatorname{Re}[\tilde{\lambda}_{k}]$,
change their sign at the critical point from negative ($\delta\beta<0$) to
positive values ($\delta\beta>0$) or remain zero.

Thus Eq. (5) indeed characterizes the critical point that destabilizes the
symmetric fixed point. If the system is at equilibrium, $f=0$, the real parts
of the eigenvalues of the linear stability matrix evaluated at the critical
point change their sign or remain zero while the imaginary parts are
identically zero corresponding to a saddle-node bifurcation that degenerates
the single symmetric fixed point into multiple fixed points. In the out-of-
equilibrium scenario, $f\neq 0$, the real parts of the eigenvalues of the
linearized Jacobian evaluated at the critical temperature change their sign or
remain zero while the imaginary parts remain finite implying a Hopf-
bifurcation that degenerates the symmetric fixed point into a limit cycle.
Equation (5) states that the uniform probability distribution can be observed
over a larger range of temperatures as $q$ increases. The uniform
distribution, however, removes the interactions from the dynamics [cf. Eq.
(2)]. Hence for repulsive interactions the mean-field system is noninteracting
as well as for attractive interactions at temperatures above the first
critical temperature that approaches zero as $q$ becomes large. In the
following we consider attractive interactions ($u=-1$) if not explicitly
stated otherwise. To infer the stability of the limit cycles, an analysis of
the normal form of the Hopf bifurcation and the computation of the first
Lyapunov coefficient would be required Kuznetsov (1998), which for $q>3$
renders analytic progress difficult. A numerical analysis in Fig. 1 depicting
in a parametric $\mathcal{P}_{1}-\mathcal{P}_{2}$ plot the dynamics of the
$q$-model ($q=3,4,5,6$) motivates the following conjecture:

If $q$ is even like in panels b) and d), the Hopf bifurcation occurs
subcritical, _i.e._ the limit cycle is unstable and degenerates into $q$
asymmetric stable fixed points, hence there is only one phase transition at
$\beta_{c}$. Conversely, if $q$ is odd, like in panels a) and c), the Hopf-
bifurcation occurs supercritical, _i.e._ the symmetric fixed point degenerates
at $\beta_{c_{1}}$ into a stable limit cycle. Physically, a limit cycle in the
$q$-dimensional probability space means that the units tend to undergo the
same transition together at a given time, i.e. they synchronize. Since the
low-temperature limit in Eq. (4) is also satisfied for odd $q$, there is a
second critical point $\beta_{c_{2}}$ at which the limit cycle degenerates via
an infinite-period bifurcation Keener (1981) into $q$ asymmetric stable fixed
points. In both cases, the multiple fixed points are related to each other by
permutations of their coordinates in the $q$-dimensional probability space.
For decreasing temperatures these fixed points move towards the respective
energy ground states in Eq. (4). Thus, we have demonstrated that there are two
classes of universal phenomenology: For all $q$ the high-temperature (low-
temperature) regime is characterized by a single (multiple) (a)symmetric
stable fixed point(s), while only for odd $q$ there is also an intermediate
phase exhibiting stable oscillations. This universality is robust to slight
changes of the state energies $\epsilon_{i}$. For large changes, the critical
phenomena vanish and there is a single stable fixed point at all temperatures.
We furthermore emphasize that for sufficiently high-dimensional lattices and
large system sizes, finite-range interactions will also generate the above
phenomenology as discussed in Ref. Herpich  _et al._ (2018) and explicitly
demonstrated in Ref. Wood  _et al._ (2006). The choice of all-to-all
interactions however allows to analytically characterize the universal
phenomenology.

Figure 1: Parametric plot of the probabilities $\mathcal{P}_{1,2}$ for $q=3$
[panel a)], $q=4$ [panel b)], $q=5$ [panel c)] and $q=6$ [panel d)] for
different temperatures $\beta$. The orange-shaded spheres indicate the $q$
asymmetric fixed points and the symmetric fixed point is represented by the
red closed circle. In all plots the initial condition $\mathcal{P}_{1}(0)=1$
is used and the chosen values for the dimensionless parameters read
$\Gamma=0.1$ and $f=1$.

It is natural to consider the large-$q$ limit where the Potts model becomes
the XY model. According to Eq. (5), the phase in which the system exhibits a
unique symmetric fixed point and thus is noninteracting exists over an
increasing range of temperatures. Hence for exceedingly large $q$ the
differences between systems with attractive interactions (for both even or odd
$q$) and repulsive interactions at experimentally meaningful temperatures
vanish.

##  IV Thermodynamics

Our thermodynamically consistent formulation of the Potts model [cf. Eq. (2)]
allows us now to address its nonequilibrium thermodynamic properties. In the
appendix B a thermodynamic description is systematically established. The
first law of thermodynamics

$\displaystyle\mathrm{d}_{t}\,\mathcal{E}=\sum\limits_{i,j}\mathcal{E}_{i}\,\mathcal{V}_{ij}\,\mathcal{P}_{j}=\dot{\mathcal{Q}}+\dot{\mathcal{W}},$
(8)

states that the rate of change in internal energy $\mathcal{E}$ is balanced by
heat and work currents, $\dot{\mathcal{Q}}$ and $\dot{\mathcal{W}}$, hence
ensuring energy conservation.

Furthermore, the non-negativity of the entropy production

$\displaystyle\dot{\mathcal{S}}_{i}=-\mathrm{d}_{t}\sum_{j}\mathcal{P}_{j}\ln\mathcal{P}_{j}-\beta\dot{\mathcal{Q}}\geq
0,$ (9)

constitutes the second law of thermodynamics. Hence, in the long-time limit,
the entropy production, up to temperature, is equal to the work,
$\dot{\mathcal{S}}_{i}=\beta\dot{\mathcal{W}}$.

We proceed by demonstrating that the bifurcations translate into
nonequilibrium phase transitions that can be characterized via the work. Here,
the work is dissipative since the mean-field system takes rotational energy,
$\dot{\mathcal{W}}>0$, and dissipates it into the bath in form of heat,
$\dot{\mathcal{Q}}<0$. First, we recall that except for attractive
interactions below the critical temperature in Eq. (5) the system behaves like
a noninteracting one. The stationary average dissipated work current for a
single unit reads

$\displaystyle\langle\dot{W}_{1}^{s}\rangle=2\Gamma f\sinh\left(\frac{\beta
f}{2}\right)\geq 0,$ (10)

and is independent of the number of states $q$. Next, for
$\beta\gg\beta_{c(c_{2})}$, the stationary mean-field work current can be
approximated as

$\displaystyle\dot{\mathcal{W}}^{s}$ $\displaystyle\approx
2\Gamma\,f\,\mathrm{e}^{\frac{\beta u}{2}}\sinh\left(\tfrac{\beta
f}{2}\right)=\mathrm{e}^{\frac{\beta u}{2}}\,\langle\dot{W}_{1}^{s}\rangle.$
(11)

Hence operating an interacting system in the low-temperatures regime is
exponentially less costly in the interaction strength than maintaining a
noninteracting one.

This can be seen in Fig. 2 that depicts the difference between the stationary
work current of a single unit $\langle\dot{W}_{1}^{s}\rangle$ and the
asymptotic mean-field work current $\overline{\mathcal{W}}$ as a function of
$\beta$ for different $q$. In agreement with Eqs. (10) and (11), we find that
for all $q$ the mean-field system is noninteracting at inverse temperatures
below the inverse Hopf-bifurcation temperature $\beta_{c(c_{1})}$ [Eq. (5)],
while the dissipated work is significantly reduced above that inverse critical
temperature. In fact, we conclude from the monotonotic behavior of the curves
that it is always energetically beneficial to consider attractive
interactions. Since the (first) critical point is shifting to larger values of
$\beta$ as $q$ increases, it is overall favorable to employ small-$q$ units.
At the (first) critical point $\beta_{c(c_{1})}$ the mean-field dissipated
work exhibits for all $q$ a kink that is reminiscent of a first-order
equilibrium phase transition. As a consequence of the two bifurcations for odd
$q$ there is a second non-equilibrium phase transition at $\beta_{c_{2}}(f)$
which displays characteristics of both a saddle and a jump that is more
pronounced for larger $q$.

Figure 2: Difference of the dissipated work for a stationary single-unit,
$\langle\dot{W}_{1}^{s}\rangle$, and the time-averaged mean-field work current
$\overline{\mathcal{W}}$ for inverse temperatures $\beta=0,1,\ldots,10$ and
different $q=2,3,\ldots,7$. The purple closed circle represents the analytic
result in the low-temperature limit.

A central result of this work is that small and odd-$q$ units not only
dissipate less when interacting attractively, but also allow to realize the
optimal power-efficiency trade-off in energy transduction. To construct an
energy converter, we decompose the nonconservative force $f$ into a force
aligned with ($f_{1}>0$) and a force acting against ($f_{2}<0$) the bias,
_i.e._ $f=f_{1}+f_{2}$. We denote by $\mathcal{\mathcal{I}}$ the current
aligned with the net-force $f$ and define the input
$\dot{\mathcal{W}}_{1}\equiv f_{1}\mathcal{I}$ and the output work current
$\dot{\mathcal{W}}_{2}\equiv f_{2}\mathcal{I}$.

This work-to-work conversion is a commonly used concept to model energy
transduction in molecular motors such as kinesin and myosin Jülicher  _et al._
(1997); Imparato (2015). For practical purposes the efficiency at maximum
power (EMP) Curzon and Ahlborn (1975) is of particular interest. The EMP is
obtained by first maximizing the asymptotic output power
$\overline{\mathbb{P}}\equiv\dot{\mathcal{W}}_{2}$ with respect to $f_{2}$.
Next, in the long-time limit the efficiency is defined as
$\eta\equiv-\mathcal{W}_{2}/\mathcal{W}_{1}=-f_{2}/f_{1}\leq 1$ and evaluated
at maximum power.

Fig. 3 shows the asymptotic power output $\overline{\mathbb{P}}$ as a function
of $\beta$ and $f_{2}$ for $q=4$ [panel a)] and $q=5$ [panel b)]. We first
recall that in both cases for inverse temperatures below $\beta_{c(c_{1})}$
the systems are noninteracting and their power output is thus determined via
Eq. (10). Next, the power rapidly drops in the low-temperature regime, that is
for $\beta>\beta_{c}$ [panel a)] or $\beta>\beta_{c_{2}}(f_{2})$ [panel b)].
Thus, we find that the maximum power is achieved in the synchronization
regime, that is for odd $q$-systems.

Figure 3: The time-averaged output power $\overline{\mathbb{P}}$ as a function
of $f_{2}$ and $\beta$ for $q=4$ [panel a)] and $q=5$ [panel b)]. The
simulation time is chosen sufficiently large to ensure convergence of the
time-averaged output power to its asymptotic value. In both plots the white
dashed lines indicate the set of critical points, hence the enclosed area
defines the synchronization phase S. In addition, the global maximum of the
output power is indicated by the green closed circle.

Fig. 4 depicts both the asymptotic global maximum power
$|\overline{\mathbb{P}}|$ \- indicated by green closed circles in Fig. 3 \-
and the EMP as a function of $q$.

Figure 4: The modulus of the time-averaged global maximum power
$|\overline{\mathbb{P}}|$ and the associated EMP for different $q$. The black
dotted lines indicate the $q$-values that exhibit a synchronization (S) phase.

Overall, the maximum power output is increasing with $q$, though there are
striking jumps from even to odd $q$-systems, _i.e._ from asynchronous to
synchronized systems. These jumps are followed by plateaus where the maxima
remain roughly the same. Conversely, the associated EMPs are decreasing
monotonically with $q$. In particular, for $q\leq 3$, the EMP is close to the
optimal value $1/2$ that is universal for a system with a single net-current
responding linearly Van den Broeck (2005); Esposito  _et al._ (2009b).
Therefore, we conclude that the optimal power-efficiency trade-off is achieved
for small and odd-$q$ systems that are compatible with synchronization.

##  V Conclusion

Our work makes progress in the (thermodynamic) study of interacting systems
away from equilibrium - a topic that is still in its infancy and for which
general results are completely lacking. The methods developed show that
combining thermodynamic arguments with tools from nonlinear dynamics can help
to characterize the complex dynamical behaviors of interacting systems
described by stochastic thermodynamics without explicitly solving the
dynamics. While our present study used Arrhenius rates, our methods are
generic and applicable to any rates satisfying local detailed balance in Eq.
(2) (e.g. Glauber dynamics Levin  _et al._ (2010); Yang (1992)). Our findings
also show that engineering interactions among collections of Brownian machines
is a promising strategy to improve their performance.

##  VI Acknowledgments

T. H. thanks Artur Wachtel for fruitful interactions. The simulations were
carried out using the HPC facilities of the University of Luxembourg Varrette
_et al._ (2014). We acknowledge support by the National Research Fund,
Luxembourg, in the frame of the AFR PhD Grant 2016, No. 11271777 and by the
European Research Council project NanoThermo (ERC-2015-CoG Agreement No.
681456).

##  Appendix A Stochastic Dynamics on different Scales

###  A.1 Many-Body Model

The following is devoted to establishing a thermodynamically consistent and
microscopic many-body description of the Potts model for a finite number of
units $N$. From this representation of the Potts model we rigorously derive
the mean-field equation (1) if the macroscopic limit $N\to\infty$ is taken.
Fig. 5 illustrates the setup for a many-body eight-state model ($q$ =8). We
want to remark that the following procedure is to a large extent analogous to
the one detailed in Secs. II and VI of Ref. Herpich  _et al._ (2018) that
studies in great detail the Potts model for $q=3$.

Figure 5: Schematics of identical and globally interacting eight-state units
that are connected with a heat bath at inverse temperature $\beta$ and
subjected to a nonconservative rotational force $f$.

###  A.2 Microscopic Dynamics

We consider $N$ $q$-state units with energies $\epsilon_{i}$,
$i=1,2,\ldots,q$. All units that occupy the same state $i$ interact globally
with the coupling constant $u/N$. The microscopic dynamics of the $N$-body
system is fully characterized by a microstate $\alpha$ which we denote by a
multiindex $\alpha\leavevmode\nobreak\ =\leavevmode\nobreak\
(\alpha_{1},\ldots,\alpha_{i},\ldots,\alpha_{N})$ with $\alpha_{i}=1,2,\ldots
q$. As an expample, we consider a transition from $\alpha^{\prime}$ to
$\alpha$ corresponding to a change in single-unit energy
$\epsilon_{j}\to\epsilon_{i}$. Hence the occupation numbers change along that
transition as $N_{j}\to N_{j}-1$ and $N_{i}\to N_{i}+1$. In order to determine
the change in internal energy for this transition the interaction energy
$U(\alpha)$ of the network is required. One has

$\displaystyle
U(\alpha)=\frac{u}{N}\sum\limits_{k=1}^{q}\;\sum\limits_{l=1}^{N_{k}\\!(\alpha)\\!-\\!1}\\!l=\frac{u}{2N}\sum\limits_{k=1}^{q}N_{k}^{2}(\alpha)+U_{0},$
(12)

thus the change in internal energy reads

$\displaystyle\Delta
E(\alpha,\alpha^{\prime})=\epsilon_{i}-\epsilon_{j}+\frac{u}{N}(N_{i}-N_{j}+1),$
(13)

where $N_{k}^{2}(\alpha)$ denotes the number of units occupying the same
single-unit state for the given microstate $\alpha$. We assume that the
dynamics of the jump process is governed by a Markovian master equation (ME)

$\displaystyle\dot{p}_{\alpha}=\sum\limits_{\alpha^{\prime}}w_{\alpha\alpha^{\prime}}\,p_{\alpha^{\prime}}\,,$
(14)

where $p_{\alpha}$ is the probability to be in the microstate $\alpha$ and the
microscopic transition rates read

$\displaystyle w_{\alpha\alpha^{\prime}}$
$\displaystyle=\Gamma\;\mathrm{e}^{-\frac{\beta}{2}\left(\Delta
E(\alpha,\alpha^{\prime})-\Theta(\alpha,\alpha^{\prime})f\right)}\,.$ (15)

The function $\Theta(\alpha,\alpha^{\prime})$ characterizes the transitions
according to their alignment with the bias $f$. It is defined as
$\Theta(\alpha,\alpha^{\prime})\\!=\\!1$ for
$\sum_{i}(\alpha_{i}\\!-\\!\alpha^{\prime}_{i})\\!=\\!1$ mod $q$ and
$\Theta(\alpha,\alpha^{\prime})\\!=\\!-1$ otherwise. The microscopic
transition rates satisfy local detailed balance

$\displaystyle\ln\frac{w_{\alpha\alpha^{\prime}}}{w_{\alpha^{\prime}\alpha}}=-\beta\left(\Delta
E(\alpha,\alpha^{\prime})-\Theta(\alpha,\alpha^{\prime})f\right)\,,$ (16)

and thus ensure that the microscopic system is thermodynamically consistent.

###  A.3 Mesoscopic Dynamics

In the microscopic formulation the system has a state space that grows
exponentially with the number of units in the network as $q^{N}$. However, the
all-to-all interactions allow to determine the energy changes in the system
without knowledge of the network topology, hence the microscopic ME (14) for
the full microstate dynamics can be coarse-grained as follows

$\displaystyle\dot{P}_{\bm{N}}$
$\displaystyle=\sum\limits_{\alpha\in\bm{N}}\sum\limits_{\bm{N}^{\prime}}\sum\limits_{\alpha^{\prime}\in\bm{N}^{\prime}}w_{\alpha\alpha^{\prime}}\,p_{\alpha^{\prime}}$
(17a)
$\displaystyle=\sum\limits_{\bm{N}^{\prime}}w_{\bm{N}\bm{N}^{\prime}}\sum\limits_{\alpha\in\bm{N}}\sum\limits_{\alpha^{\prime}\in\bm{N}^{\prime}}\;p_{\alpha^{\prime}}\,\chi_{\alpha^{\prime},\alpha}$
(17b)
$\displaystyle=\sum\limits_{\bm{N}^{\prime}}W_{\bm{N}\bm{N}^{\prime}}\;P_{\bm{N}^{\prime}}\,,$
(17c)

where $P_{\bm{N}}\equiv\sum_{\alpha\in\bm{N}}\,p_{\alpha}$ refers to the
mesoscopic probability to observe a mesostate
$\bm{N}\equiv(N_{1},\ldots,N_{q})$ that comprises all microstates
$\alpha\in\bm{N}$. The mesoscopic transition rates are given by
$W_{\bm{N}\bm{N}^{\prime}}=\Omega(\bm{N},\bm{N}^{\prime})w_{\bm{N}\bm{N}^{\prime}}$
with the microscopic transition rates in mesoscopic representation
$w_{\bm{N}\bm{N}^{\prime}}$. Moreover, the characteristic function
$\chi_{\alpha^{\prime},\alpha}=1$ if transitions between $\alpha^{\prime}$ and
$\alpha$ are possible and is 0 otherwise, hence one has

$\displaystyle\Omega(\bm{N},\bm{N}^{\prime})=N^{\prime}_{1}\,\delta_{N^{\prime}_{1},N_{1}+1}+\ldots+N^{\prime}_{q}\,\delta_{N^{\prime}_{q},N_{q}+1}\,.$
(18)

We identify the Boltzmann entropy as the mesoscopic equilibrium entropy

$\displaystyle S^{int}(\bm{N})=\ln\Omega(\bm{N})\,,$ (19)

where $\Omega(\bm{N})=N!/\prod_{i}N_{i}!$ gives the number of microstates
$\alpha$ that belong to the mesostate $\bm{N}$. Collecting results, the
mesoscopic local detailed balance relation is given by

$\displaystyle\ln\frac{W_{\bm{N}\bm{N}^{\prime}}}{W_{\bm{N}^{\prime}\bm{N}}}=-\beta\left[\Delta
A(\bm{N},\bm{N}^{\prime})-\Theta(\bm{N},\bm{N}^{\prime})f\right],$ (20)

where $\Delta A(\bm{N},\bm{N}^{\prime})\\!=\\!\Delta
E(\bm{N},\bm{N}^{\prime})-\\!\beta^{-1}\Delta S^{int}(\bm{N},\bm{N}^{\prime})$
denotes the change in equilibrium free energy between the mesostates
$\bm{N}^{\prime}$ and $\bm{N}$. The function $\Theta(\bm{N},\bm{N}^{\prime})$
is defined as in the microscopic case in Eq. (16). Hence,
$\Theta(\bm{N},\bm{N}^{\prime})=1$ if $(i-j)\text{ mod }q=\\!1$ for the
transition $\bm{N}_{i}^{\prime}\to\bm{N}_{i},\bm{N}_{j}^{\prime}\to\bm{N}_{j}$
and $\Theta(\bm{N},\bm{N}^{\prime})\\!=\\!-1$ otherwise. We emphasize that the
exact coarse-graining of the microscopic dynamics significantly reduces the
complexity of the problem since the state space in the mesoscopic
representation is growing like $N^{q-1}/(q-1)!\,$.

###  A.4 Mean-Field Limit

The asymptotic solution in the macroscopic limit ($N\to\infty$) is simply
given by a mean-field approximation performed on the mesoscopic ME (17c)
scaled with $1/N$ . The mean-field approximation amounts to replacing any
$n$-point correlation function with a product of $n$ averages and thus yields

$\displaystyle\mathcal{P}_{i}\equiv\langle\dot{n}_{i}\rangle$
$\displaystyle=\sum\limits_{j}W_{ij}\big{(}\langle n_{i}\rangle,\langle
n_{j}\rangle\big{)}\equiv\sum\limits_{j}\mathcal{V}_{ij}\big{(}\mathcal{P}_{i},\mathcal{P}_{j}\big{)}\mathcal{P}_{j},$
(21)

which indeed is the nonlinear mean-field equation Eq. (1) . Here, we
introduced the mean-field rates
$\mathcal{V}_{ij}=\Gamma\exp(\epsilon_{i}-\epsilon_{j}+u(\mathcal{P}_{i}-\mathcal{P}_{j})-\Theta(i,j)f)$
with the sign function defined as $\Theta(i,j)=1$ for $(i-j)\text{ mod
}q=\\!1$ and $\Theta(i,j)\\!=\\!-1$ otherwise.

##  Appendix B Thermodynamic Laws on different Scales

###  B.1 Microscopic Thermodynamics

We start with the elementary thermodynamic state functions in this model: the
microscopic internal energy and the microscopic system entropy

$\displaystyle\left\langle e\right\rangle$
$\displaystyle=\sum\limits_{\alpha}E(\alpha)\,p_{\alpha}$ (22)
$\displaystyle\left\langle s\right\rangle$
$\displaystyle=-\sum\limits_{\alpha}p_{\alpha}\ln p_{\alpha}\,.$ (23)

The time-derivative of the internal energy

$\displaystyle\mathrm{d}_{t}\langle
e\rangle=\sum\limits_{\alpha,\alpha^{\prime}}E(\alpha)w_{\alpha\alpha^{\prime}}\,p_{\alpha^{\prime}}=\langle\dot{q}\rangle+\langle\dot{w}\rangle,$
(24)

stipulates the microscopic first law of thermodynamics that ensures energy
conservation. Here, we defined the heat and work current as follows

$\displaystyle\langle\dot{q}\rangle$
$\displaystyle=\sum\limits_{\alpha,\alpha^{\prime}}\left[E(\alpha)-f\,\Theta(\alpha,\alpha^{\prime})\right]w_{\alpha\alpha^{\prime}}\,p_{\alpha^{\prime}}$
(25) $\displaystyle\langle\dot{w}\rangle$
$\displaystyle=\sum\limits_{\alpha,\alpha^{\prime}}f\;\Theta(\alpha,\alpha^{\prime})\;w_{\alpha\alpha^{\prime}}\,p_{\alpha^{\prime}}\,,$
(26)

where the sign function $\Theta(\alpha,\alpha^{\prime})$ is equal to the one
introduced below Eq. (16). The rate of change of the system entropy

$\displaystyle\mathrm{d}_{t}\langle
s\rangle=\langle\dot{s}_{e}\rangle+\langle\dot{\sigma}\rangle$ (27)

can be decomposed into the entropy flow from the bath to the system

$\displaystyle\langle\dot{s}_{e}\rangle=-\sum\limits_{\alpha,\alpha^{\prime}}w_{\alpha\alpha^{\prime}}\,p_{\alpha^{\prime}}\ln\frac{w_{\alpha\alpha^{\prime}}}{w_{\alpha^{\prime}\alpha}}=\beta\langle\dot{q}\rangle,$
(28)

and the non-negative entropy production rate

$\displaystyle\langle\dot{\sigma}\rangle=\sum\limits_{\alpha,\alpha^{\prime}}w_{\alpha\alpha^{\prime}}\,p_{\alpha^{\prime}}\,\ln\frac{w_{\alpha\alpha^{\prime}}p_{\alpha^{\prime}}}{w_{\alpha^{\prime}\alpha}p_{\alpha}}\geq
0.$ (29)

Equation (29) constitutes the second law of thermodynamics.

###  B.2 Mesoscopic Thermodynamics

The exact coarse-graining of the microscopic dynamics from above does not
imply that the statistics of the thermodynamic observables are invariant under
this marginalization Esposito (2012). We define $E_{\bm{N}}$ to be the
internal energy of the system in the mesostate $\bm{N}$ and find for the first
law of thermodynamics

$\displaystyle\mathrm{d}_{t}\langle
E\rangle=\sum\limits_{\bm{N},\bm{N}^{\prime}}E(\bm{N})\,W_{\bm{N}\bm{N}^{\prime}}\,P_{\bm{N}^{\prime}}=\langle\dot{Q}\rangle+\langle\dot{W}\rangle,$
(30)

with the mesoscopic heat and work currents

$\displaystyle\langle\dot{Q}\rangle$
$\displaystyle=\sum\limits_{\bm{N},\bm{N}^{\prime}}\big{(}E(\bm{N})-f\,\Theta(\bm{N},\bm{N}^{\prime})\big{)}W_{\bm{N}\bm{N}^{\prime}}P_{\bm{N}^{\prime}}$
(31) $\displaystyle\langle\dot{W}\rangle$
$\displaystyle=\sum\limits_{\bm{N},\bm{N}^{\prime}}f\,\Theta(\bm{N},\bm{N}^{\prime})W_{\bm{N}\bm{N}^{\prime}}\,P_{\bm{N}^{\prime}}.$
(32)

Furthermore, we define the system entropy in the mesospace

$\displaystyle\left\langle S\right\rangle$
$\displaystyle=\sum\limits_{\bm{N}}P_{\bm{N}}\left(\Omega(\bm{N})-\ln
P_{\bm{N}}\right)\,,$ (33)

consisting of the non-equilibrium entropy defined by Eq. (23) and the
equilibrium entropy from Eq. (19) accounting for the internal structure of the
mesostates. Again, we split the time-derivative of the mesoscopic entropy into
the mesocopic entropy flow

$\displaystyle\langle\dot{S}_{e}\rangle=-\sum\limits_{\bm{N},\bm{N}^{\prime}}W_{\bm{N}\bm{N}^{\prime}}\,P_{\bm{N}^{\prime}}\ln\frac{{w}_{\bm{N}\bm{N}^{\prime}}}{w_{\bm{N}^{\prime}\bm{N}}}=\beta\langle\dot{Q}\rangle,$
(34)

and the mesoscopic EP rate

$\displaystyle\langle\dot{\Sigma}\rangle=\sum\limits_{\bm{N},\bm{N}^{\prime}}W_{\bm{N}\bm{N}^{\prime}}\,P_{\bm{N}^{\prime}}\,\ln\frac{W_{\bm{N}\bm{N}^{\prime}}P_{\bm{N}^{\prime}}}{W_{\bm{N}^{\prime}\bm{N}}P_{\bm{N}}}\geq
0.$ (35)

that constitutes the second law of thermodynamics. A closer inspection shows
that while the first-law quantities are preserved under the coarse-graining
procedure

$\displaystyle\mathrm{d}_{t}\langle E\rangle=\mathrm{d}_{t}\langle
e\rangle,\quad\langle\dot{Q}\rangle=\langle\dot{q}\rangle,\quad\langle\dot{W}\rangle=\langle\dot{w}\rangle,$
(36)

the definitions in Eqs. (33),(35) are in general not equivalent to the
microscopic ones, _i.e._ $\langle S\rangle\neq\langle
s\rangle,\langle\Sigma\rangle\neq\langle\sigma\rangle$. Yet, in the stationary
state it holds that $P_{\bm{N}}=p_{\alpha}\cdot\Omega(\bm{N})$, which in turn
implies that the entropies in mesoscopic representation are identical to those
in microscopic representation, _i.e._ $\langle S^{s}\rangle=\langle
s^{s}\rangle,\langle\Sigma^{s}\rangle=\langle\sigma^{s}\rangle$. For this
particular case, the second law

$\displaystyle\\!\\!\\!\\!\langle\dot{\Sigma^{s}}\rangle\\!=\\!\sum\limits_{\bm{N},\bm{N}^{\prime}}\\!\\!W_{\bm{N}\bm{N}^{\prime}}\,P^{s}_{\bm{N}^{\prime}}\,\ln\frac{\tilde{W}_{\bm{N}\bm{N}^{\prime}}}{\tilde{W}_{\bm{N}^{\prime}\bm{N}}}=-\langle\dot{S}_{e}^{s}\rangle\geq
0,$ (37)

states that the steady mesoscopic entropy flow is equal to minus the
stationary mesoscopic entropy production rate.
As shown above, the mean-field Eq. (21) results from a mean-field
approximation applied on the mesoscopic ME. Thus, only the definitions of the
thermodynamic observables in the mean-field limit stated in the main body are
representing the true physical observables, if the corresponding microscopic
definition coincides with the mesoscopic one. It therefore holds for the mean-
field observables $\mathcal{X}$ that
$\lim_{N\to\infty}\;\langle\dot{X}\rangle/N=\dot{\mathcal{X}},$ with the
mesoscopic observables $X\\!\\!\leavevmode\nobreak\ =\leavevmode\nobreak\
\\!\\!E,Q,W,S_{e},N_{i}$ and
$\lim_{N\to\infty}\;\langle\dot{\Sigma}^{s}\rangle/N=\dot{\mathcal{S}_{i}^{s}}$,
where the superscript s refers to a stationary state.

## References

  * Landau and Lifshitz (1994) L. Landau and E. Lifshitz,  _Course of Theoretical Physics Volum 5: Statistical Physics Part 1_ (Pergamon Press, 1994).
  * Yeomans (1992) J. Yeomans,  _Statistical Mechanics of Phase Transitions_ (Oxford University Press, 1992).
  * Herpich  _et al._ (2018) Tim Herpich, Juzar Thingna, and Massimiliano Esposito, “Collective power: Minimal model for thermodynamics of nonequilibrium phase transitions,” Phys. Rev. X 8, 031056 (2018).
  * Vroylandt  _et al._ (2017) Hadrien Vroylandt, Massimiliano Esposito, and Gatien Verley, “Collective effects enhancing power and efficiency,” EPL (Europhysics Letters) 120, 30009 (2017).
  * Tomé and de Oliveira (2012) Tânia Tomé and Mário J. de Oliveira, “Entropy production in nonequilibrium systems at stationary states,” Phys. Rev. Lett. 108, 020601 (2012).
  * Zhang and Barato (2016) Yirui Zhang and Andre C Barato, “Critical behavior of entropy production and learning rate: Ising model with an oscillating field,” Journal of Statistical Mechanics: Theory and Experiment 2016, 113207 (2016).
  * Crosato  _et al._ (2018) Emanuele Crosato, Richard E. Spinney, Ramil Nigmatullin, Joseph T. Lizier, and Mikhail Prokopenko, “Thermodynamics and computation during collective motion near criticality,” Phys. Rev. E 97, 012120 (2018).
  * Vaikuntanathan  _et al._ (2014) Suriyanarayanan Vaikuntanathan, Todd R. Gingrich, and Phillip L. Geissler, “Dynamic phase transitions in simple driven kinetic networks,” Phys. Rev. E 89, 062108 (2014).
  * Marcuzzi  _et al._ (2014) Matteo Marcuzzi, Emanuele Levi, Sebastian Diehl, Juan P. Garrahan, and Igor Lesanovsky, “Universal nonequilibrium properties of dissipative rydberg gases,” Phys. Rev. Lett. 113, 210401 (2014).
  * Seifert (2012) Udo Seifert, “Stochastic thermodynamics, fluctuation theorems and molecular machines,” Reports on Progress in Physics 75, 126001 (2012).
  * den Broeck and Esposito (2015) C. Van den Broeck and M. Esposito, “Ensemble and trajectory thermodynamics: A brief introduction,” Physica A: Statistical Mechanics and its Applications 418, 6 – 16 (2015).
  * Zhang  _et al._ (2012) Xue-Juan Zhang, Hong Qian, and Min Qian, “Stochastic theory of nonequilibrium steady states and its applications. part i,” Physics Reports 510, 1 – 86 (2012).
  * Sekimoto (2010) K. Sekimoto,  _Stochastic Energetics_ , 1st ed. (Springer-Verlag, 2010).
  * Esposito  _et al._ (2010) Massimiliano Esposito, Katja Lindenberg, and Christian Van den Broeck, “Entropy production as correlation between system and reservoir,” New Journal of Physics 12, 013013 (2010).
  * Esposito  _et al._ (2009a) Massimiliano Esposito, Upendra Harbola, and Shaul Mukamel, “Nonequilibrium fluctuations, fluctuation theorems, and counting statistics in quantum systems,” Rev. Mod. Phys. 81, 1665–1702 (2009a).
  * Benenti  _et al._ (2017) Giuliano Benenti, Giulio Casati, Keiji Saito, and Robert S. Whitney, “Fundamental aspects of steady-state conversion of heat to work at the nanoscale,” Physics Reports 694, 1 – 124 (2017).
  * Proesmans  _et al._ (2016) Karel Proesmans, Yannik Dreher, Momilo Gavrilov, John Bechhoefer, and Christian Van den Broeck, “Brownian duet: A novel tale of thermodynamic efficiency,” Phys. Rev. X 6, 041010 (2016).
  * Imparato (2015) Alberto Imparato, “Stochastic thermodynamics in many-particle systems,” New Journal of Physics 17, 125004 (2015).
  * Golubeva and Imparato (2012) N. Golubeva and A. Imparato, “Efficiency at maximum power of interacting molecular machines,” Phys. Rev. Lett. 109, 190602 (2012).
  * Ashkin and Teller (1943) J. Ashkin and E. Teller, “Statistics of two-dimensional lattices with four components,” Phys. Rev. 64, 178–184 (1943).
  * Ising (1925) E. Ising, “Beitrag zur Theorie des Ferromagnetismus,” Zeitschrift fur Physik 31, 253–258 (1925).
  * Onsager (1944) L. Onsager, “Crystal Statistics. I. A Two-Dimensional Model with an Order-Disorder Transition,” Physical Review 65, 117–149 (1944).
  * Wu (1982) F. Y. Wu, “The potts model,” Rev. Mod. Phys. 54, 235–268 (1982).
  * Davis (1970) Philipp Davis,  _Circulant Matrices_ (Wiley, New York, 1970).
  * Kuznetsov (1998) Y. A. Kuznetsov,  _Elements of applied bifurcation theory_, 2nd ed. (Springer-Verlag, 1998).
  * Keener (1981) Jamse P Keener, “Infinite period bifurcation and global bifurcation branches,” SIAM Journal on Applied Mathematics 41, 127–144 (1981).
  * Wood  _et al._ (2006) Kevin Wood, C. Van den Broeck, R. Kawai, and Katja Lindenberg, “Universality of synchrony: Critical behavior in a discrete model of stochastic phase-coupled oscillators,” Phys. Rev. Lett. 96, 145701 (2006).
  * Jülicher  _et al._ (1997) Frank Jülicher, Armand Ajdari, and Jacques Prost, “Modeling molecular motors,” Rev. Mod. Phys. 69, 1269–1282 (1997).
  * Curzon and Ahlborn (1975) F. L. Curzon and B. Ahlborn, “Efficiency of a Carnot engine at maximum power output,” American Journal of Physics 43, 22–24 (1975).
  * Van den Broeck (2005) C. Van den Broeck, “Thermodynamic efficiency at maximum power,” Phys. Rev. Lett. 95, 190602 (2005).
  * Esposito  _et al._ (2009b) Massimiliano Esposito, Katja Lindenberg, and Christian Van den Broeck, “Universality of efficiency at maximum power,” Phys. Rev. Lett. 102, 130602 (2009b).
  * Levin  _et al._ (2010) D. Levin, M.J. Luczak, and Y. Peres,  _Probability Theory and Related Fields_ (Springer-Verlag, 2010).
  * Yang (1992) Z. R. Yang, “Glauber dynamics of the kinetic ising model,” Phys. Rev. B 46, 11578–11584 (1992).
  * Varrette  _et al._ (2014) S. Varrette, P. Bouvry, H. Cartiaux, and F. Georgatos, “Management of an academic hpc cluster: The ul experience,” in  _Proc. of the 2014 Intl. Conf. on High Performance Computing & Simulation (HPCS 2014)_ (IEEE, Bologna, Italy, 2014) pp. 959–967.
  * Esposito (2012) Massimiliano Esposito, “Stochastic thermodynamics under coarse graining,” Phys. Rev. E 85, 041125 (2012).


# Discovering 3D Parts from Image Collections

Chun-Han Yao
UC Merced  Wei-Chih Hung
Waymo  Varun Jampani
Google  Ming-Hsuan Yang
UC Merced

###### Abstract

Reasoning 3D shapes from 2D images is an essential yet challenging task,
especially when only single-view images are at our disposal. While an object
can have a complicated shape, individual parts are usually close to geometric
primitives and thus are easier to model. Furthermore, parts provide a mid-
level representation that is robust to appearance variations across objects in
a particular category. In this work, we tackle the problem of 3D part
discovery from only 2D image collections. Instead of relying on manually
annotated parts for supervision, we propose a self-supervised approach, latent
part discovery (LPD). Our key insight is to learn a novel part shape prior
that allows each part to fit an object shape faithfully while constrained to
have simple geometry. Extensive experiments on the synthetic ShapeNet,
PartNet, and real-world Pascal 3D+ datasets show that our method discovers
consistent object parts and achieves favorable reconstruction accuracy
compared to the existing methods with the same level of supervision. Our
project page with code is at https://chhankyao.github.io/lpd/.

Figure 1: Discovering 3D parts form single-view image collections. Our method
(LPD) enables self-supervised 3D part discovery while learning to reconstruct
object shapes from single-view images. Compared to other methods using
different part constraints, LPD discovers more faithful and consistent parts,
which improve the reconstruction quality and allow part
reasoning/manipulation.

##  1 Introduction

Recognizing and reasoning about objects surrounding us is essential for many
computer vision systems. While deep learning models have been shown to perform
well at recognizing [27, 46, 17] and localizing [13, 44, 35] objects in a 2D
image, reasoning 3D attributes of objects from a single image remains a
challenging task. Single-view 3D reasoning is fundamentally ill-posed due to
several factors that cause ambiguous object appearance in 2D images, _e.g_.,
camera pose, self-occlusions, lighting, and material properties. Although
objects in general have complicated shapes, they can usually be decomposed
into parts that have simpler geometry and are relatively easy to model.
Furthermore, most object instances of a particular category share similar part
configurations, _e.g_., the wings, body, and tail of airplanes. In this work,
we propose to tackle the problem by discovering faithful and consistent 3D
parts from 2D image collections. Compared to existing single-view 3D
reconstruction approaches that directly predict an object shape, we aim to
learn rich and dense part configurations which form an entire object when
combined.

Although several recent methods [48, 37, 3, 10, 33, 36, 41, 24] leverage part-
based representations for 3D object reasoning, they rely on either 3D object
shapes or explicit part annotations as supervision. Moreover, the learned
parts only serve as additional information and are not exploited to improve 3D
reconstruction. Considering that collecting ground-truth 3D shapes and the
corresponding part labels is labor intensive, we follow a practical scenario
where only single-view images, 2D object silhouettes, and camera viewpoints
are available for model training. In contrast to existing techniques, our
method automatically discovers 3D parts from image collections in a self-
supervised manner.

A common practice to represent 3D parts is to use geometric primitives such as
ellipsoids or cuboids [48]. They provide a strong regularization of part
shapes but are usually too coarse to faithfully represent object parts. As an
alternative, several approaches adopt meshes [23, 50, 14, 22, 34, 18, 15, 32]
or point-cloud [8, 20, 28, 40, 6] representations. Although these
representations are more expressive and can faithfully describe part shapes,
they lack part shape regularization that is particularly needed in a weakly-
supervised or unsupervised setting. The key insight of this work is to
represent 3D parts with deep latent embeddings. Specifically, we propose to
learn a prior distribution of part shapes with a variational auto-encoder
(VAE) [26] that encodes part shapes as latent embeddings. We call this network
Part-VAE and pre-train it with a set of geometric primitives like cones,
cylinders, cuboids, and ellipsoids. We then learn a reconstruction network
that takes an input image and predicts part embeddings to obtain a 3D mesh by
passing through the decoder of Part-VAE. To further improve the quality of
part discovery and reconstruction, we propose a novel part adversarial loss
which involves re-assembling parts from different objects in the same
category. We name the proposed method latent part discovery (LPD). Figure 1
shows several reconstruction results with and without part prior, which
demonstrate that LPD can discover consistent parts and produce faithful
reconstruction to the input image.

We evaluate LPD on the synthetic ShapeNet [1], PartNet [39] and the real-world
Pascal 3D+ [51] datasets. Both quantitative and qualitative results
demonstrate that our approach achieves favorable performance against the
state-of-the-art methods using the same level of supervision. In addition to
part discovery, our part representation enables object manipulation like
selective part swapping, interpolation, and random shape generation from the
latent space. In this work, we make the following contributions:

  * •

We propose a part-based single-view 3D reasoning network which can
automatically discover object parts. To the best of our knowledge, this is the
first work that discovers 3D parts in a self-supervised manner without using
any 3D shape or multi-view supervision.

  * •

We develop Part-VAE to learn a latent prior over part shapes. We show that
training with geometric primitives can learn useful part embeddings, allowing
each part to faithfully represent object shape while constrained to have
simple geometry.

  * •

We conduct extensive experiments on both synthetic and natural images.
Qualitatively, our method produces more faithful and consistent object parts
compared to other part-based methods. Quantitatively, the discovered parts
improve whole-object reconstruction and achieve favorable accuracy against the
state-of-the-art techniques. In addition, our Part-VAE allows us to manipulate
object parts for various applications.

##  2 Related Work

3D Reconstruction. While 3D representation has been widely studied for
decades, the best and unified way to represent general objects remains
unclear. Voxel grids [4, 49, 47, 31], point clouds [8, 20, 28, 40, 6], and
meshes [23, 50, 14, 22, 34, 2, 18, 15, 32] are commonly used to represent
object shapes. Several recent methods [12, 38, 52, 16, 5, 11] explore the
possibilities to represent 3D shapes in a functional space. While fine-grained
voxels, point clouds, and local functions can represent complicated shapes,
the flexibility of representation demands strong 3D or multi-view supervision
for training. Meshes, on the other hand, are constrained to form a water-tight
surface and are convenient to render 2D images. By deforming from a simple
template mesh like sphere or cuboid, it is easier to apply shape
regularization and thus can be applied to reconstruction scenarios with weaker
supervision. One can learn single-view mesh reconstruction from multi-view or
single-view images using naive 2D projection or differentiable rendering [23,
34, 2]. For instance, Henderson  _et al_. [18] generate background images and
object rendering to learn textured mesh reconstructions from natural images.
Kato  _et al_. [22] propose view prior learning (VPL) to improve the
reconstructed shapes from unseen views. Although they are effective for
compact and deformable objects, a single mesh cannot represent complicated
shapes with holes or disconnected parts. In this work, we exploit multi-mesh
part representation which allows disconnected parts in an object while each
part can be well-regularized.

Part Discovery. Parts provide a mid-level representation that is robust to
appearance variations across objects in the same category. Hung  _et al_. [19]
learn 2D co-part segmentation on image collections with self-supervision.
Lathuilière  _et al_. [29] exploit motion cues in videos for part discovery.
In the 3D domain, Tulsiani  _et al_. [48] use volumetric cuboids as part
abstractions to learn 3D reconstruction. Li  _et al_. [33] assume known part
shapes and learn to assemble them given an input image. Using a point-cloud
representation, Mandikal  _et al_. [37] predict part-segmented 3D
reconstructions from a single image and Luo  _et al_. [36] learn to form
object parts by clustering 3D points. Paschalidou  _et al_. [41] propose
hierarchical part decomposition (HPD) by constraining 3D points with super-
quadratic functions [42]. These methods require 3D ground-truth shape of a
whole object or its parts as supervision. Furthermore, the part shapes in [48,
41] are limited by the expressiveness of their representations. Li  _et al_.
[32] leverage 2D semantic parts to improve single-view 3D reconstruction,
which, however, does not produce individual part shapes. To the best of our
knowledge, we propose the first 3D part reconstruction method without any part
annotations or ground-truth 3D shapes for training, and our latent part
representation enables each part to fit a given object shape faithfully.

Figure 2: Approach overview. (top) Our Part-VAE is trained with geometric
primitives. (bottom) Our reconstruction model shares the shape decoder with
Part-VAE and predicts object parts. We then composite the reconstructed parts
to form a 3D object. The prediction of part centroids and surface texture are
detailed in the supplemental material.

##  3 Approach

Reasoning 3D objects from single-view images is inherently ill-posed since the
reconstructed object can over-fit to a given view and be highly deformed in
the unseen parts. To address this, we propose LPD to represent an object with
multiple latent parts. Our intuition is that a complicated object shape can be
expressed by assembling simple and regularized parts. Consistent with some
recent approaches [22, 18], we propose a method under a weakly-supervised
setting where only a single-view image, 2D object silhouette, and its camera
viewpoint are available for each object. That is, we do not assume any 3D
shape or multi-view images to supervise part discovery or reconstruction. In
order to automatically discover underlying parts while reconstructing an
object, we propose to learn part embeddings with a variational auto-encoder
[26] (VAE) named Part-VAE. We train a reconstruction model that predicts part
embeddings which are then decoded into part meshes to compose the whole
object. Figure 2 illustrates the proposed method with two main modules: Part-
VAE and reconstruction network.

###  3.1 Learning Part Prior with Part-VAE

We propose Part-VAE to learn a latent shape prior for object parts. The
proposed method constrains parts with primitive shapes while allowing the
flexibility to fit real-world object parts. In addition, it enables smooth
part-interpolation and novel shape generation by random sampling in the latent
space. Figure 2 (top) illustrates the training process of the Part-VAE with
geometric primitives. We first collect a set of primitive shapes such as
ellipsoids, cylinders, cones, and cuboids, which are centered at origin but
with random scaling and rotation. The Part-VAE network consists of a shape
encoder and a shape decoder. The encoder transforms each given primitive shape
to a low-dimensional shape encoding, and the decoder reconstructs the input
shape by predicting the vertex deformations of a spherical template mesh. To
supervise the Part-VAE pre-training, we calculate the Chamfer distance between
the input and output vertices as the loss function since the point sets are
unordered and not densely corresponding. Given the vertices of an input shape
$Q$ and its reconstruction $P$, the Chamfer loss $\mathcal{L}_{c}$ can be
expressed as:

$\begin{split}\mathcal{L}_{c}(P,Q)=&\frac{1}{\|P\|}\sum_{p\in P}\hskip
2.84526pt\min_{q\in Q}\big{\|}p-q\big{\|}_{2}^{2}\hskip 2.84526pt+\\\
&\frac{1}{\|Q\|}\sum_{q\in Q}\hskip 2.84526pt\min_{p\in
P}\big{\|}p-q\big{\|}_{2}^{2}.\end{split}$ (1)

To encourage the continuity of latent shape distribution, we adopt a standard
KL divergence loss $\mathcal{L}_{kl}$ calculated between the shape embeddings
and a standard normal distribution $\mathcal{N}\sim(0,1)$. The overall
training loss of Part-VAE is:
$\mathcal{L}_{vae}=\mathcal{L}_{c}+\lambda_{kl}\hskip
2.84526pt\mathcal{L}_{kl}$ , where $\lambda_{kl}$ is a weight parameter.

Figure 3: Part and view adversarial learning. Given two images with different
objects, we randomly combine their reconstructed parts into a novel shape. The
novel shape is then rendered from a novel viewpoint, which we treat as a
‘fake’ sample. We train a discriminator to distinguish the fake and real
rendered images. By using a gradient reversal layer (GRL), the reconstruction
model learns to produce parts that can compose realistic novel shapes.

###  3.2 Part Discovery by Learning to Reconstruct

Figure 2 (bottom) illustrates our reconstruction model. Instead of directly
predicting an object mesh, we learn an image encoder that takes an input image
and predicts the 3D part centroid, latent shape encodings, and surface texture
for each part. The part encodings are then passed through the shape decoder of
Part-VAE to generate part meshes. To compose an entire object, we simply shift
the mesh vertices using the predicted part centroids and concatenate the
vertices and surfaces of each part. In the remainder of the paper, we denote
the reconstruction model as $R(\cdot)$, which takes an image as input and
outputs a part-composited mesh. The rendering function from viewpoint $v$ is
denoted as $G(\cdot,v)$, which produces a rendered image of the input mesh.
Note that each training image $I$ includes a silhouette channel $I_{s}$ and
RGB color channels $I_{c}$. Likewise, the rendering function $G$ can be
separated into $G_{s}$ and $G_{c}$ for silhouette projection and color
rendering, respectively.

Shape Reconstruction Losses. To supervise shape reconstruction, we enforce the
2D projection of a reconstructed shape to be close to the ground-truth
silhouette. In particular, we render the predicted meshes with input
viewpoints using a differentiable renderer [34], and calculate the
intersection-over-union (IoU) ratio between the rendered and ground-truth
silhouettes. Then the silhouette loss $\mathcal{L}_{sil}$ is computed as:

$\mathcal{L}_{sil}(I,v)=\frac{\lVert I_{s}\odot
G_{s}(R(I),v)\rVert_{1}}{\lVert I_{s}+G_{s}(R(I),v)\rVert_{1}-\lVert
I_{s}\odot G_{s}(R(I),v)\rVert_{1}},$ (2)

where $\odot$ denotes element-wise multiplication. We further apply Laplacian
regularization $\mathcal{L}_{lap}$ on the reconstructed mesh vertices:

$\mathcal{L}_{lap}(P)=\frac{1}{\|P\|}\sum_{p\in P}\Big{\|}\hskip
2.84526ptp-\frac{1}{\|\mathcal{N}(p)\|}\sum_{q\in\mathcal{N}(p)}q\hskip
2.84526pt\Big{\|}_{2}^{2},$ (3)

where $\mathcal{N}(p)$ denotes the neighboring vertices of vertex $p$. It aims
to smooth the mesh surfaces by pulling each vertex towards the center of its
neighboring pixels. Note that this regularization is applied on each part mesh
individually so discontinuity between part surfaces is allowed.

Color Reconstruction Loss. We further exploit the color information in input
images by generating textured reconstructions. Given the part encodings, our
model predicts texture flow to map the input image to a UV texture image. We
then color the mesh surfaces by sampling from the texture image using a pre-
defined UV mapping function. The texture flow is predicted per object part so
each part has more coherent texture. We denote the overall color rendering
process as $G_{c}$ and show more details in the supplemental material. The
color reconstruction loss $\mathcal{L}_{cr}$ is defined on the semantic
features of input and rendered images:

$\mathcal{L}_{cr}(I,v)={\lVert F(I_{c})-F(G_{c}(R(I),v))\rVert_{2}^{2}},$ (4)

where $F$ is the feature extractor of a fixed classification network. We use
AlexNet [27] pre-trained on the ImageNet dataset [27] and extract the output
of multiple convolutional layers as $F$. It encourages the rendered image to
be perceptually similar to the input at different levels.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
Input | VPL [22] | Free-form | Cuboids | Ellipsoids | LPD | LPD
| (1 part) | (3 parts) | (3 parts) | (3 parts) | (3 parts) | (6 parts)
Figure 4:  Qualitative results on the ShapeNet dataset [1]. LPD models (ours)
adopt the proposed Part-VAE and adversarial learning. The 3-part free-form
model reconstructs a whole object with three fully-deformable meshes without
any part prior. Compared to the baselines, our approach can produce more
faithful and consistent parts from diverse objects.

###  3.3 Part and View Adversarial Learning

Unlike the multi-view or 3D-supervised settings, single-view training requires
stronger regularization to produce realistic 3D shapes and to discover
meaningful parts. Based on the intuition that object parts are interchangeable
and they should look realistic from various viewpoints, we extend the view
adversarial learning in VPL [22] with part adversarial learning. As
illustrated in Figure 3, we assemble a novel shape by randomly combining parts
from different objects of the same class in a training batch, then render the
novel shape from a novel viewpoint. To make the novel shape realistic, we
treat the rendered images of novel shapes as fake examples and those of
original shapes as real ones. A discriminator is then trained to classify each
rendered image as real or fake. We train the discriminator with a binary
cross-entropy between the positive and negative samples:

$\begin{split}\mathcal{L}_{adv}(I,I^{\prime},v)=&-\text{log}(D(G(R(I),v)))\\\
&-\text{log}(1-D(G(R^{\prime}(I,I^{\prime}),v^{\prime}))),\end{split}$ (5)

where $I^{\prime}$ is a random image different from the input $I$,
$R^{\prime}(\cdot,\cdot)$ is the reconstruction model with random part
selection from two input images, $v^{\prime}$ is a random novel view, and $D$
is the discriminator. To apply adversarial training, we add a gradient
reversal layer (GRL) [9] before the discriminator. As a result, the
reconstruction model is trained to fool the discriminator by generating novel
objects with realistic shapes. Considering that different object classes may
have different view and shape prior, we condition the discriminator with input
class labels during training. Note that the class labels are not required
during inference. That is, for given data, we train a single part-
discovery/reconstruction model that operates across different object
categories. This part-based adversarial learning approach is proposed as a
semantic constraint to make the global part arrangements more feasible and
realistic.

Table 1: Ablative evaluations on the ShapeNet dataset [1]. The base model reconstructs an object shape with 3 meshes, each is fully-deformable as in SoftRas [34] (PP: part prior, VA: view adversarial learning, PA: part adversarial learning, CR: color reconstruction). PP | VA | PA | CR | Airplane | Bench | Dresser | Car | Chair | Display | Lamp | Speaker | Rifle | Sofa | Table | Phone | Vessel | All
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| ✓ | ✓ | ✓ | 56.7 | 34.4 | 56.0 | 68.3 | 43.1 | 34.8 | 47.2 | 59.9 | 50.6 | 48.5 | 41.1 | 42.7 | 53.4 | 49.0
✓ |  | ✓ | ✓ | 57.2 | 36.2 | 60.7 | 72.2 | 44.1 | 39.8 | 48.2 | 63.6 | 51.9 | 49.6 | 43.3 | 51.5 | 55.1 | 51.8
✓ | ✓ |  | ✓ | 57.1 | 35.8 | 61.4 | 73.7 | 45.1 | 39.4 | 48.5 | 63.7 | 52.7 | 49.3 | 43.9 | 52.5 | 54.9 | 52.2
✓ | ✓ | ✓ |  | 57.1 | 36.0 | 61.0 | 74.1 | 45.2 | 39.7 | 48.5 | 63.8 | 53.0 | 49.7 | 43.9 | 52.2 | 55.1 | 52.3
✓ | ✓ | ✓ | ✓ | 57.3 | 37.3 | 60.9 | 75.2 | 45.5 | 40.8 | 49.6 | 63.3 | 54.5 | 50.1 | 44.3 | 52.7 | 56.2 | 52.9

###  3.4 Model Training and Inference

We first pre-train the Part-VAE with primitive shapes to minimize the loss
$\mathcal{L}_{vae}$. Next, the Part-VAE and reconstruction network are jointly
trained using image collections together with primitive shapes. The overall
objective function of reconstruction network is given by:

$\quad\mathcal{L}_{sil}+\lambda_{lap}\hskip
2.84526pt\mathcal{L}_{lap}+\lambda_{cr}\hskip
2.84526pt\mathcal{L}_{cr}-\lambda_{adv}\hskip 2.84526pt\mathcal{L}_{adv},$ (6)

where ($\lambda_{lap}$, $\lambda_{cr}$, $\lambda_{adv}$) are weight
parameters. The discriminator is trained to minimize $\mathcal{L}_{adv}$,
whose gradients are reversed and back-propagated to the reconstruction network
to perform adversarial learning. Note that we still fine-tune the Part-VAE
with primitive shapes in this stage so that the shape decoder is regularized
while adapting to various part shapes in the training images. The Part-VAE,
reconstruction network, and discriminator are parametrized as deep neural
networks, and the weights are optimized by mini-batch gradient descent. In the
model inference phase, we discard the Part-VAE encoder and discriminator. An
input image is simply passed through the image encoder and Part-VAE decoder to
reconstruct the 3D parts. We represent each object part by a deformable mesh
with $N_{v}=642$ vertices and $N_{f}=1280$ faces. The size of texture image is
64$\times$64\. We implement the proposed method in PyTorch [43] framework and
use Adam optimizer [25] for training. The hyper-parameters are tuned on a
validation set. We present more model details in the supplemental material,
and the source code will be made available to the public.

##  4 Experiments and Analysis

Metrics and Baselines. Evaluating self-supervised part discovery can be
ambiguous and subjective as the discovered parts need not correspond to human-
annotated ones. Due to the lack of standard metrics or benchmark for 3D part
discovery, we qualitatively compare the discovered parts with other methods.
As a reference, we also quantitatively evaluate the reconstruction accuracy at
both object level and part level. We convert each predicted mesh into a volume
of $32^{3}$ voxels and calculate the intersection-over-union (IoU) ratio
between the voxelized object and the ground-truth voxels. We report results of
our model with $k=3$ parts and latent part embedding dimension of $d=64$ if
not specified otherwise. Since our work is the first to discover 3D parts
using single-view supervision, we mainly compare LPD against three part-based
baselines: cuboids, ellipsoids, and free-form meshes. We implement the cuboid
and ellipsoid models by reconstructing each object part with a scalable
cuboid/ellipsoid. The free-form model adopts fully-deformable meshes without
any part shape prior. For object-level reconstruction, we evaluate our method
against SoftRas [34] and VPL [22] as they adopt a similar training setting
with single-view images and known viewpoints. While there are several other
methods for single-view 3D reconstruction, we omit the comparison with them
since whole-object reconstruction is not the major focus of this work. We
experiment on the synthetic ShapeNet [1], PartNet [39], and real-world Pascal
3D+ [51] datasets.

|  |  |  |
---|---|---|---|---
|  |  |  |
Car | Lamp | Rifle | Table | Watercraft
Figure 5: Generalization across classes. We show sample inputs (top) and LPD
results (bottom) of different ShapetNet classes.

###  4.1 Results on ShapeNet

We first conduct experiments with the synthetic dataset provided by Kar  _et
al_. [21], which contains 43,784 objects in 13 classes from ShapeNet [1]. Each
sample includes a 3D CAD model, 20 camera viewpoints for rendering, and the
corresponding rendered images at a resolution of 224$\times$224 pixels. We use
the same training/validation/testing splits as the original dataset. The
training images are augmented by random shuffling the RGB channels and
horizontal flipping. We only use one view per object for training and evaluate
on all the 20 views in the test set (with independent single-view
reconstruction on each view). The ground-truth 3D shapes are only used for
testing. We show some qualitative results of our method and other baselines in
Figure 4. More part reconstruction results on car, lamp, rifle, table, and
watercraft samples are shown in Figure 5 to demonstrate that our method
generalizes well across diverse object classes.

Ablations on the Proposed Models. We perform ablation studies on the proposed
method by removing each component at a time. As shown in Table 1, removing
part prior learning causes a significant drop (3.9%) on the overall
reconstruction accuracy. It shows that the part prior provided by Part-VAE
effectively improves the generalization of the discovered parts. Without the
use of adversarial learning and color reconstruction, we also observe lower
voxel IoU by a clear margin (0.6-1.1%).

Table 2: Voxel IoU results on the ShapeNet dataset [1]. We compare our method with the state-of-the-art single-view supervised and 3D supervised approaches. Method | Supervision | Airplane | Car | Chair | All
---|---|---|---|---|---
SIF [12] | 3D shapes | 53.0 | 65.7 | 38.9 | 49.9
OccNet [38] | 3D shapes | 57.1 | 73.7 | 50.1 | 57.1
CvxNet [5] | 3D shapes | 59.8 | 67.5 | 49.1 | 56.7
HPD [41] | 3D shapes | 52.9 | 70.2 | 52.6 | 58.0
SoftRas [34] | Single-view | 52.2 | 65.7 | 40.4 | 46.9
VPL [22] | Single-view | 53.1 | 70.1 | 45.4 | 51.3
LPD (ours) | Single-view | 57.3 | 75.2 | 45.5 | 52.9

Comparisons with the State-of-the-arts. Table 2 shows performance comparisons
against the state-of-the-art methods. With the single-view training setting,
our method performs favorably against the existing approaches. When compared
to the 3D-supervised methods, our model achieves competitive results on many
object classes even though we use weaker single-view supervision. Among the
evaluated methods, SIF [12] and CvxNet [5] can subdivide an object into fine-
grained regions, and HPD [41] performs hierarchical part reasoning. However,
their shape representations require stronger supervision from 3D ground truths
to produce faithful part shapes. The qualitative results in Figure 4
demonstrate that our model discovers more faithful and consistent parts
compared to the baseline methods.

Ablations on Part Representations. We further compare 3D reconstruction
methods that use different part shape constraints. A majority of existing
methods represent objects with a single mesh [23, 50, 14, 22, 34, 18] and
allow each shape to be fully deformable by predicting the vertex deformations
directly. On the other end of the spectrum, most part-reasoning approaches
represent parts with primitive shapes like cuboids [48] or super-quadratic
surfaces [41]. Our method lies within these two extremes and enables variable
degree of freedom by adjusting the latent dimension of the part shape
embedding. Table 3 shows the quantitative comparisons between part
representations like free-form meshes, cuboid reconstructions, and our Part-
VAE embedding via different evaluation metrics. In addition to 3D voxel IoU,
we calculate the 2D re-projection IoU, 2D structural similarity (SSIM), and
Chamfer distance (CD) between the point sets sampled from 3D volumes. The
results show that LPD achieves a better trade-off between the degree of
deformation and shape regularization among the part-based methods. To observe
how our method adapts to different object classes, we perform single-class
training with different number of parts $k$ on airplane, car, and chair
images. As shown in Table 4, the optimal number of parts $k$ varies across
object classes. This suggests that each class have a distinct underlying part
configuration to optimally represent the object shapes. Note that LPD achieves
higher accuracy than other methods on all three classes with more than one
part. Our main reconstruction model is class-agnostic and we use the same
number of parts for all classes, and yet the performance could be further
improved if it is optimized for each class separately.

Table 3: Quantitative evaluations on the ShapeNet dataset [1] using different metrics. LPD allows part reasoning and achieves higher accuracy in terms of all metrics. Method | Part |  2D IoU $\uparrow$ |  SSIM $\uparrow$ |  CD $\downarrow$ |  Voxel IoU $\uparrow$
---|---|---|---|---|---
SoftRas [34] |  | 80.5 | 86.7 | 4.76 | 46.9
VPL [22] |  | 81.0 | 88.5 | 2.65 | 51.3
Free-form | $\checkmark$ | 81.1 | 87.9 | 3.83 | 48.1
Cuboids | $\checkmark$ | 72.5 | 67.3 | 6.12 | 39.7
LPD (ours) | $\checkmark$ | 83.6 | 91.0 | 2.37 | 52.9
Table 4: Voxel IoU results with different number of parts $k$. Note that the models are trained and tested on a single class. Method | $k$ | Airplane | Car | Chair
---|---|---|---|---
SoftRas [34] | 1 | 54.1 | 69.5 | 43.1
VPL [22] | 1 | 54.6 | 74.1 | 45.3
LPD (ours) | 1 | 54.5 | 74.3 | 45.0
LPD (ours) | 2 | 55.4 | 76.1 | 45.9
LPD (ours) | 3 | 55.6 | 75.5 | 46.6
LPD (ours) | 6 | 55.9 | 75.2 | 46.4

###  4.2 Results on PartNet

To evaluate the quality of the discovered parts, we compare our results with
the labeled parts in PartNet dataset. The dataset contains hierarchical part
annotations of several ShapeNet models. We collect 111 chair samples that are
in both the ShapeNet and PartNet testing sets, then combine the annotated part
models into chair back, seat, and base at the coarsest level. Note that we do
not use any 3D part supervision for training, so the PartNet annotations are
not ground-truths but a reference. Since the discovered parts are not
semantically labeled, we manually associate our parts to the closest
corresponding PartNet annotations. We report the quantitative voxel IoU in
Table LABEL:tab:partnet and the qualitative results in Figure 6. Compared to
the baseline without part prior and other representations, our method
discovers more faithful parts and achieves considerably higher IoU with
respect to PartNet annotations.

Table 5: Voxel IoU results on the PartNet [39] chair samples. Method | Back | Seat | Base | Avg
---|---|---|---|---
Free-form | 16.8 | 19.5 | 10.3 | 15.5
Cuboids | 22.3 | 23.4 | 10.7 | 18.8
LPD (ours) | 30.4 | 46.0 | 16.2 | 30.9
|  |  |
---|---|---|---
|  |  |
|  |  |
Input | Cuboids | LPD (ours) | Pseudo-GT
Figure 6:  Qualitative results on the PartNet dataset [39]. We show the
voxelized 3-part results of our method and a cuboid baseline. Each part is
specified with a color: chair back$\rightarrow$green, seat$\rightarrow$yellow,
base$\rightarrow$blue. Our method discovers faithful and consistent parts from
diverse objects that are relatively closer to the pseudo-GT part annotations
in PartNet.

###  4.3 Part Interpolation and Generation.

In addition to shape reconstruction, we demonstrate two applications of Part-
VAE: part-interpolation and random shape generation. Since the object parts
discovered by our models are consistent across instances, one can swap or
interpolate parts to create new 3D objects. In Figure 7, we linearly
interpolate the latent encodings $(u_{1},u_{2})$ of two objects from different
categories as: $u=\lambda u_{1}+(1-\lambda)u_{2}$. Compared to the baseline
without part prior, our method deforms each object part smoothly and results
in more realistic shapes. The Part-VAE can also be used as a generative model
to create novel shapes. Specifically, we fit a Gaussian Mixture Model (GMM)
with $k$ components on the latent shape vectors of class-specific images. By
sampling random vectors from individual GMM distribution, we can generate $k$
random parts and combine them into a new 3D shape. In Figure 8, we show some
randomly generated shapes of chairs and airplanes using the Part-VAE trained
on the ShapeNet dataset.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
$\lambda$ = 0.0 |  $\lambda$ = 0.25 |  $\lambda$ = 0.5 |  $\lambda$ = 0.75 |  $\lambda$ = 1.0
Figure 7: Cross-category interpolation. We perform interpolation on ShapeNet airplane-rifle, lamp-display, and chair-table. We show the VPL [22] results (mesh interpolation) in rows 1, 3, 5 and LPD results (latent interpolation) in rows 2, 4, 6. |  |  |
---|---|---|---
|  |  |
Figure 8: Random shape generation of chairs and airplanes. We fit a GMM model
on the latent shape vectors and generate random parts by sampling from
individual GMM components.

###  4.4 Results on Pascal 3D+

We also evaluate the proposed method on the real-world images from the Pascal
3D+ dataset [51] processed by Tulsiani  _et al_. [49]. It consists of images
in Pascal VOC [7], annotations of 3D models, silhouettes, and viewpoints in
Pascal 3D+ [51], and additional images in ImageNet [45] with silhouettes and
viewpoints automatically annotated by [30]. This dataset is more challenging
due to the complicated object shapes, image background, occlusions, and noisy
silhouette annotations. We train and evaluate our models with image resolution
of 224$\times$224\. The quantitative and qualitative results are shown in
Table LABEL:tab:pascal and Figure 9, respectively. Despite the challenges, our
method discovers consistent object parts and achieves higher reconstruction
accuracy than the state-of-the-art approaches.

Table 6: Voxel IoU results on the Pascal 3D+ dataset [51]. Method | Part | Aeroplane | Car | Chair | Avg
---|---|---|---|---|---
SoftRas [34] |  | 46.4 | 67.6 | 29.1 | 47.7
VPL [22] |  | 47.5 | 67.9 | 30.4 | 48.6
Free-form | ✓ | 47.0 | 68.5 | 28.7 | 48.0
Cuboids | ✓ | 37.1 | 60.7 | 18.9 | 38.9
LPD (ours) | ✓ | 48.2 | 69.1 | 31.0 | 49.4
|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Input | View 1 | View 2 | View 1 | View 2
Figure 9: Part and color reconstruction results on the Pascal 3D+ dataset
[51]. Despite that the dataset contains complicated 3D objects in a realistic
scene, our method is able to discover consistent parts and effectively
reconstruct the objects shapes.

##  5 Concluding Remarks

In this work, we propose LPD to discover 3D parts from single-view image
collections. By learning a part prior with Part-VAE, we demonstrate that each
part can be deformed to fit a realistic object shape while constrained to have
simple geometry. With the goal to compose an object with simple parts, our
reconstruction model automatically learns a latent part configuration. In
turn, the discovered parts can alleviate shape ambiguity and improve the
quality of full object reconstruction. Extensive experimental results show
that LPD can discover faithful parts from diverse object classes, and the
parts are consistent across different instances within a same category.
Furthermore, we achieve the state-of-the-art reconstruction accuracy in a
single-view training setting. Our work opens up the possibilities to learn,
infer, and manipulate object parts without the need for any ground-truth part
labels or 3D shape supervision.

## References

  * [1] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [2] Wenzheng Chen, Jun Gao, Huan Ling, Edward J Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  arXiv preprint arXiv:1908.01210, 2019.
  * [3] Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang.  Bae-net: Branched autoencoder for shape co-segmentation.  In ICCV, pages 8490–8499, 2019.
  * [4] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In ECCV, pages 628–644, 2016.
  * [5] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.  Cvxnet: Learnable convex decomposition.  In CVPR, pages 31–44, 2020.
  * [6] Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  Learning elementary structures for 3d shape generation and matching.  arXiv preprint arXiv:1908.04725, 2019.
  * [7] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 88(2):303–338, 2010.
  * [8] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In CVPR, pages 605–613, 2017.
  * [9] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  arXiv preprint arXiv:1409.7495, 2014.
  * [10] Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang.  Sdm-net: Deep generative network for structured deformable mesh.  TOG, 38(6):1–15, 2019.
  * [11] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape.  In CVPR, pages 4857–4866, 2020.
  * [12] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser.  Learning shape templates with structured implicit functions.  In ICCV, pages 7154–7164, 2019.
  * [13] Ross Girshick.  Fast r-cnn.  In ICCV, 2015.
  * [14] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In ICCV, pages 9785–9795, 2019.
  * [15] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik.  Shape and viewpoint without keypoints.  In ECCV, pages 88–104, 2020.
  * [16] Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, and Serge Belongie.  Dualsdf: Semantic shape manipulation using a two-level representation.  In CVPR, pages 7631–7641, 2020.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [18] Paul Henderson, Vagia Tsiminaki, and Christoph H Lampert.  Leveraging 2d data to learn textured 3d mesh generation.  In CVPR, pages 7498–7507, 2020.
  * [19] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz.  Scops: Self-supervised co-part segmentation.  In CVPR, pages 869–878, 2019.
  * [20] Eldar Insafutdinov and Alexey Dosovitskiy.  Unsupervised learning of shape and pose with differentiable point clouds.  In NeurIPS, pages 2802–2812, 2018.
  * [21] Abhishek Kar, Christian Häne, and Jitendra Malik.  Learning a multi-view stereo machine.  In NeurIPS, pages 365–376, 2017.
  * [22] Hiroharu Kato and Tatsuya Harada.  Learning view priors for single-view 3d reconstruction.  In CVPR, pages 9778–9787, 2019.
  * [23] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3d mesh renderer.  In CVPR, pages 3907–3916, 2018.
  * [24] Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada.  Neural star domain as primitive representation.  arXiv preprint arXiv:2010.11248, 2020.
  * [25] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [26] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, pages 1097–1105, 2012.
  * [28] K L Navaneet, Priyanka Mandikal, Varun Jampani, and Venkatesh Babu.  Differ: Moving beyond 3d reconstruction with differentiable feature rendering.  In CVPR Workshops, pages 18–24, 2019.
  * [29] Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe, et al.  Motion-supervised co-part segmentation.  arXiv preprint arXiv:2004.03234, 2020.
  * [30] Ke Li, Bharath Hariharan, and Jitendra Malik.  Iterative instance segmentation.  In CVPR, pages 3659–3667, 2016.
  * [31] Xiao Li, Yue Dong, Pieter Peers, and Xin Tong.  Synthesizing 3d shapes from silhouette image collections using multi-projection generative adversarial networks.  In CVPR, pages 5535–5544, 2019.
  * [32] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz.  Self-supervised single-view 3d reconstruction via semantic consistency.  In ECCV, pages 677–693, 2020.
  * [33] Yichen Li, Kaichun Mo, Lin Shao, Minhyuk Sung, and Leonidas Guibas.  Learning 3d part assembly from a single image.  In ECCV, pages 664–682, 2020.
  * [34] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  In CVPR, pages 7708–7717, 2019.
  * [35] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In ECCV, pages 21–37, 2016.
  * [36] Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su.  Learning to group: A bottom-up framework for 3d part discovery in unseen categories.  arXiv preprint arXiv:2002.06478, 2020.
  * [37] Priyanka Mandikal, Navaneet KL, and R Venkatesh Babu.  3d-psrnet: Part segmented 3d point cloud reconstruction from a single image.  In ECCV, pages 0–0, 2018.
  * [38] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, pages 4460–4470, 2019.
  * [39] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su.  Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding.  In CVPR, pages 909–918, 2019.
  * [40] KL Navaneet, Priyanka Mandikal, Mayank Agarwal, and R Venkatesh Babu.  Capnet: Continuous approximation projection for 3d point cloud reconstruction using 2d supervision.  In AAAI, pages 8819–8826, 2019.
  * [41] Despoina Paschalidou, Luc Van Gool, and Andreas Geiger.  Learning unsupervised hierarchical part decomposition of 3d objects from a single rgb image.  In CVPR, pages 1060–1070, 2020.
  * [42] Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger.  Superquadrics revisited: Learning 3d shape parsing beyond cuboids.  In CVPR, pages 10344–10353, 2019.
  * [43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, pages 8024–8035, 2019.
  * [44] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, pages 91–99, 2015.
  * [45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, 115(3):211–252, 2015.
  * [46] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [47] Shubham Tulsiani, Alexei A Efros, and Jitendra Malik.  Multi-view consistency as supervisory signal for learning shape and pose prediction.  In CVPR, pages 2897–2905, 2018.
  * [48] Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik.  Learning shape abstractions by assembling volumetric primitives.  In CVPR, pages 2635–2643, 2017.
  * [49] Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik.  Multi-view supervision for single-view reconstruction via differentiable ray consistency.  In CVPR, pages 2626–2634, 2017.
  * [50] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In ECCV, pages 52–67, 2018.
  * [51] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese.  Beyond pascal: A benchmark for 3d object detection in the wild.  In WACV, pages 75–82, 2014.
  * [52] Kohei Yamashita, Shohei Nobuhara, and Ko Nishino.  3d-gmnet: Learning to estimate 3d shape from a single image as a gaussian mixture.  arXiv preprint arXiv:1912.04663, 2019.