# Three-phase Majorana zero modes at tiny magnetic fields

Omri Lesser<EMAIL_ADDRESS>Department of Condensed Matter
Physics, Weizmann Institute of Science, Rehovot, Israel 7610001  Karsten
Flensberg  Center for Quantum Devices, Niels Bohr Institute, University of
Copenhagen, DK-2100 Copenhagen, Denmark  Felix von Oppen  Dahlem Center for
Complex Quantum Systems and Fachbereich Physik, Freie Universität Berlin,
Arnimallee 14, 14195 Berlin, Germany  Yuval Oreg  Department of Condensed
Matter Physics, Weizmann Institute of Science, Rehovot, Israel 7610001

###### Abstract

Proposals for realizing Majorana fermions in condensed matter systems
typically rely on magnetic fields, which degrade the proximitizing
superconductor and plague the Majoranas’ detection. We propose an alternative
scheme to realize Majoranas based only on phase-biased superconductors. The
phases (at least three of them) can be biased by a tiny magnetic field
threading macroscopic superconducting loops, focusing and enhancing the effect
of the magnetic field onto the junction, or by supercurrents. We show how a
combination of the superconducting phase winding and the spin-orbit phase
induced in closed loops (Aharonov-Casher effect) facilitates a topological
superconducting state with Majorana end states. We demontrate this scheme by
an analytically tractable model as well as simulations of realistic setups
comprising only conventional materials.

_Introduction.—_ The realization of robust Majorana zero modes (MZMs) at the
ends of quasi-one-dimensional (1D) $p$-wave superconductors (SCs) has been a
long-standing goal in contemporary condensed matter physics [1]. These exotic
quasiparticles, predicted to possess non-Abelian exchange statistics, signal
the appearance of a novel phase of matter: a topological superconductor.
Interest in realizing MZMs has been stimulated by the fundamental-physics
quest of discovering new phases of matter, as well as by potential
applications to topological quantum computation [2, 3].

Following the canonical toy model of a spinless $p$-wave SC chain [4], several
proposals for experimentally realizing MZMs have been put forward [5]. These
platforms include the surface of topological insulators proximity coupled to a
superconductor [6], hybrid semiconductor-superconductor nanowires [7, 8],
possibly current-biased [9], semiconductor-ferromagnet heterostructures [10,
11], quantum wells with an in-plane magnetic field [12], phase-biased
Josephson junctions [13, 14, 15, 16, 17], carbon nanotubes [18, 19, 20],
chains of magnetic adatoms on superconductors with strong spin-orbit coupling
[21, 22, 23], and full-shell proximitized nanowires [24, 25].

Generally, three ingredients are needed to realize topological
superconductivity in one dimension: proximity coupling to a conventional
$s$-wave superconductor (sufficiently thick to be free of phase fluctuations),
a spin-rotation mechanism, most commonly spin-orbit coupling (SOC), and a
source of time-reversal-symmetry breaking. With a proper combination of these
ingredients, the low-energy band becomes effectively spinless while remaining
susceptible to pairing, thus realizing a $p$-wave superconductor. Time-
reversal symmetry is usually broken by an external Zeeman field or by internal
magnetic phenomena, such as the exchange field of a nearby ferromagnet.

Realizations in which the proximitizing superconductor is subjected to a
magnetic field have the drawback of degrading superconductivity [26]. In
particular, all types of time-reversal-symmetry breakers – Zeeman field,
exchange field, magnetic flux in the presence of conventional impurities, or
magnetic impurities – lead to depairing of Cooper pairs and the formation of
in-gap states. In extreme cases, a gapless superconductor is formed [27]. This
makes MZMs fragile and renders their detection ambiguous. Moreover, one may
wonder why a Zeeman or exchange field is necessary at all. Indeed, several
proposals rely on controlling the phase of the SC order parameter only [6,
28]. Other proposals include on top of that the application of a weak magnetic
field [9, 14, 15, 29, 25, 30].

In this manuscript, we show that in the presence of a winding superconducting
phase, topological superconductivity arises without any Zeeman field or
magnetic flux penetrating the sample, using a conventional (non-topological)
semiconducting substrate with strong spin-orbit coupling. The distinction
between opposite spins is generated by closed electron trajectories (loops)
having gauge-invariant Aharonov-Casher phases [31]. Such gauge-invariant
phases arise when the loops encircle a net charge [31]. The winding can be
obtained when the phases of at least three superconductors form a polygon on
the unit circle surrounding the origin [32] [see Fig. 1(b)]. This alleviates
the need for a Zeeman field, an exchange field, magnetic fluxes [25], or
relatively large supercurrents [29]. The superconducting phases can be
controlled by macroscopic superconducting loops, which focus the time-
reversal-breaking element on the junction. Therefore, a tiny magnetic field,
of less than a micro-tesla for a micron-size loop, can be used to achieve
topological superconductivity. In this method, the superconductors remain free
of pair-breaking perturbations, in-gap states, and flux trapping, thereby
allowing even the use of type-II superconductors such as Nb.

Fu and Kane [6] studied the 2D surface of a 3D topological insulator in
proximity to a conventional $s$-wave superconductor, showing that a discrete
vortex associated with three phase-biased superconductors binds a MZM. This
MZM emerges as a result of two topological phases accumulated by the surface
Dirac fermions along loops encircling the vortex center. In addition to the
$\pi$ phase associated with the vortex, there is a $\pi$ Aharonov-Casher (or
Berry) phase [31], which originates from spin-momentum locking in the language
of surface Dirac electrons. Importantly, both phases are required for inducing
Majorana zero modes, even though the system is already topological without the
vortex. It would be very attractive to implement a similar scheme in 1D using
conventional materials, as inducing a discrete vortex requires only minimal
magnetic fields or supercurrents. However, it is not evident whether this is
possible. First, the 1D system will not already be topological by proximity
coupling to the conventional superconductor, and MZMs are in one-to-one
correspondence with the formation of a topological superconducting state.
Second, phase biasing does not directly introduce a Zeeman splitting, which is
typically required. Here, we show for explicit examples that Aharonov-Casher
phases in conjunction with a discrete vortex can stabilize topological
superconductivity in 1D systems using conventional materials. We believe that
this design principle can be highly beneficial in realizing topological
superconductivity, as it eliminates the severely detrimental effects of large
magnetic fields.

Our main result is the phase diagram in Fig. 1(b) for the three-phase system
depicted in Fig. 1(a). The phase diagram depends on the two phase differences
$\phi_{1}$ and $\phi_{2}$ ($\phi_{3}$ is set to zero), and periodically
repeats the unit cell indicated by the black square 111The phases are
conveniently visualized by plotting them on a unit circle, fixing the global
phase of the superconductors such that $\phi_{3}=0$. Then, the phases form a
triangle as shown in the inset of Fig. 1(b). When all $\phi_{n}=0$, the system
is obviously time-reversal symmetric. At the three additional time-reversal-
symmetric points $(\phi_{1},\phi_{2})=(0,\pi),(\pi,0),(\pi,\pi)$, the triangle
degenerates into a line through the center of the unit circle.. To highlight
the role of time-reversal symmetry, we plot the phase diagram as a function of
$\theta=\left(\phi_{1}-\phi_{2}\right)/2$ and
$\phi=\left(\phi_{1}+\phi_{2}\right)/2$. Then, similar to a single phase-
biased planar Josephson junction [14, 15], $\phi=\pi$ and $\theta=0$ is a
time-reversal-symmetric point (as are $\phi=\pi/2,\theta=\pi/2$ and
$\phi=3\pi/2,\theta=\pi/2$). In contrast to conventional Josephson junctions
where a Zeeman field is needed to break time-reversal symmetry and to drive
the system into a topological state, here this effect is achieved by the phase
difference $\theta$ between the superconductors.

_Coupled-wires model.—_ To demonstrate our approach in a tractable model, we
consider three spin-orbit-coupled wires in proximity to three $s$-wave
superconductors with pair potentials of magnitude $\Delta$ and phases
$\phi_{1},\phi_{2},\phi_{3}$, as illustrated in Fig. 1(a).

Figure 1: Coupled-wires model for topological superconductivity induced by
phase bias only. (a) Illustration of the system under study: three spin-orbit-
coupled wires, proximity coupled to SCs with three different phases. (b)
Topological phase diagram as a function of the SC phases $\phi_{1}$ and
$\phi_{2}$, with $\phi_{3}$ set to $0$ (see inset). The square in solid bold
lines repeats periodically. The solid thin lines correspond to the phase
boundaries at the optimal manifold where the critical value required for a
topological phase is $f_{\rm crit}=-1$, whereas the dashed lines correspond to
$f_{\rm crit}=-1.1$, away from the optimal manifold. The phase windings in the
central triangles realize a vortex ($+$) and an anti-vortex ($-$). Time-
reversal-invariant points are shown as red squares. Notice that the phase
diagram is mirror symmetric about the line $\theta=0$.

In the continuum limit, the topological properties are already encoded in the
spectrum for zero momentum along the wires, $k_{\parallel}=0$ [4]. In this
case, the Hamiltonian of the three-wire model takes the form

$\displaystyle\mathcal{H}\left(k_{\parallel}=0\right)=$
$\displaystyle\sum_{n=1}^{N}\sum_{s,s^{\prime}=\pm}\left[-\mu\delta^{ss^{\prime}}c_{n,s}^{\dagger}c_{n,s^{\prime}}\right.$
(1)
$\displaystyle+\left.\left(t_{\perp}\left(e^{i\lambda_{n}\sigma_{z}}\right)^{ss^{\prime}}c_{n,s}^{\dagger}c_{n+1,s^{\prime}}+{\rm
H.c.}\right)\right]$ $\displaystyle+\sum_{n=1}^{N}\left(\Delta
e^{i\phi_{n}}c_{n,\uparrow}^{\dagger}c_{n,\downarrow}^{\dagger}+{\rm
H.c.}\right),$

where $c_{n,s}$ annihilates an electron in wire $n$ with $k_{\parallel}=0$ and
spin projection $s$ along $z$, $t_{\perp}$ is the inter-wire hopping
amplitude, $\mu$ is the chemical potential, $\Delta$ is the induced SC pair
potential, and $\lambda_{n}$ is the SOC angle accumulated between the
neighboring wires $n$ and $n+1$. Here, we assume periodic boundary conditions,
$c_{N+1,s}=c_{1,s}$. As we will see, it is crucial that electrons acquire an
Aharonov-Casher phase [31], which will conspire with the SC phase winding to
eliminate one spin species at the Fermi level. Equation (1) is written for a
general number of wires $N$; for simplicity, we will focus on the minimal
value to create a phase winding, $N=3$. Notice that the gauge transformation
$c_{n,s}\rightarrow c_{n,s}e^{i\phi_{n}/2}$ eliminates the phases from the SC
terms and changes the hopping term to $t_{\perp}\rightarrow
t_{\perp}\exp\left(i\frac{\phi_{n+1}-\phi_{n}}{2}\right)$. This resembles but
is not equivalent to magnetic flux: unlike magnetic flux, the phases
$\phi_{n}$ can be gauged away when $\Delta=0$.

To identify phase transitions in the parameter space of our model, we search
for gap closures by equating the determinant of the Hamiltonian Eq. (1) to
zero:

$\displaystyle\det\mathcal{H}$
$\displaystyle\left(k_{\parallel}=0\right)=6\mu^{2}t_{\perp}^{2}\left(\Delta^{2}+\mu^{2}\right)-\left(\Delta^{2}+\mu^{2}\right)^{3}$
(2)
$\displaystyle-3t_{\perp}^{4}\left(\Delta^{2}+3\mu^{2}\right)-2f\Delta^{2}t_{\perp}^{2}\left(\Delta^{2}+\mu^{2}+t_{\perp}^{2}\right)$
$\displaystyle-4\mu
t_{\perp}^{3}\Lambda\left(f\Delta^{2}-\mu^{2}+3t_{\perp}^{2}\right)-4t_{\perp}^{6}\Lambda^{2}=0,$

where
$f=\cos\left(\phi_{1}-\phi_{2}\right)+\cos\left(\phi_{2}-\phi_{3}\right)+\cos\left(\phi_{3}-\phi_{1}\right)$
and $\Lambda=\cos\left(\lambda_{1}+\lambda_{2}+\lambda_{3}\right)$.

The SC phases appear in the determinant through a single parameter $-3/2\leq
f\leq 3$, which has a simple geometric interpretation: for $-3/2\leq f\leq-1$
the phases wind, i.e., when plotted as complex numbers
$\left\\{e^{i\phi_{n}}\right\\}$ on the unit circle, the triangle connecting
them contains the origin [34]. Solving the quadratic equation
$\det\mathcal{H}\left(k_{\parallel}=0\right)=0$ for $\Lambda$, we find that a
real solution is possible only for $f\leq-1$ [34], and therefore phase winding
is a necessary condition for the existence of a zero-energy state, in
agreement with the results of Ref. [32].

Assuming that the SC phase winds, we still have to determine the regions in
the three-dimensional parameter space spanned by $\mu$, $\Delta$, and
$\Lambda$ (choosing units such that $t_{\perp}=1$) for which the system is
topological. An optimal situation occurs when the values of the three
parameters are such that the determinant Eq. (2) is zero already for $f=-1$.
Then, the system is topological for the maximal range of $-3/2\leq f<-1$.
Setting $f=-1$ in Eq. (2) we find that the optimal situation occurs when
$\left(\mu,\Delta,\Lambda\right)$ are points on a circle ${\cal C}$
parametrized by $\left(\mu,\sqrt{1-\mu^{2}},\mu\right)$, see [34] and Fig. S1.
Setting $f=f_{\rm crit}$ with $-3/2\leq f_{\rm crit}<-1$ in Eq. (2) defines a
surface in the parameter space; when $\left(\mu,\Delta,\Lambda\right)$ lie on
this surface, topological superconductivity occurs for $-3/2\leq f<f_{\rm
crit}$, see Fig. 1(b). Hence we conclude that topological superconductivity is
obtained for all $\left(\mu,\Delta,\Lambda\right)$ points within the bulk of
the shape defined at $f_{\rm crit}=-3/2$ (see Fig. S1 of the Supplemental
Material [34]), with optimal values on the circle ${\cal C}$.

To find the energy gap in the topological state, we analyze the full spectrum
of the system away from $k_{\parallel}=0$. Belonging to symmetry class D [35,
36, 37], the full Hamiltonian is characterized by the $\mathbb{Z}_{2}$
topological invariant [4, 38]

$\mathcal{Q}={\rm sign}\left[{\rm
Pf}\left(\mathcal{P}\mathcal{H}(k_{\parallel}=0)\right){\rm
Pf}\left(\mathcal{P}\mathcal{H}(k_{\parallel}=\pi)\right)\right],$ (3)

where Pf is the Pfaffian and $\mathcal{P}$ is the particle-hole operator.
$\mathcal{Q}=1$ indicates the trivial phase, whereas $\mathcal{Q}=-1$ in the
topological phase, where the system supports MZMs 222In the continuum limit,
the parallel part of the Hamiltonian reads
$\mathcal{H}_{\parallel}=\left({k_{\parallel}^{2}}/{2m^{*}}+uk_{\parallel}\sigma_{x}\right)\tau_{z}$,
where $m^{*}$ is the effective electron mass, $u$ is the Rashba SOC parameter
along the wires, and the Pauli matrix $\tau_{z}$ acts in particle-hole space.
. The energy gap must be calculated for all values of $k_{\parallel}$.

The numerically calculated [40] topological phase diagram of the system is
shown in Fig. 2(a), for parameters chosen on the optimal manifold. Remarkably,
the model supports a topological phase with an excitation gap of about
$0.3\Delta$, with the application of only a phase difference and without any
applied Zeeman or orbital field in the sample. We note that at
$\left(\theta=\frac{\pi}{3},\phi=\pi\right)$ and
$\left(\theta=\frac{2\pi}{3},\phi=0\right)$ – perfect vortices with equal
phase differences forming an equilateral triangle – the system becomes
$C_{3}$-symmetric and turns out to be gapless, due to a non-topological gap
closure at finite $k_{\parallel}$, see Fig. 2(b). In Fig. 2(c), we demonstrate
that the gap opens when the $C_{3}$ symmetry is broken 333 This delicate gap
closure originates from the high symmetry of the system: For
$\lambda_{1}=\lambda_{2}=\lambda_{3}\equiv\lambda$, there is a $C_{3}$
rotation symmetry at these points which is removed by any symmetry-breaking
perturbation, such as adding a non-proximitized wire, depleting one of the
wires, or choosing different $\lambda_{n}$’s.. In addition, the topological
gap is bounded from above by the minimum of $\Delta$ (the induced SC gap) and
$\Delta_{\rm SO}\sim
t_{\perp}\sin^{2}\left(\frac{1}{2N}\sum_{n}\lambda_{n}\right)$ (the SOC
splitting energy).

Further confirmation for the existence of the topological phase is given in
Fig. 2(d), where we show the Majorana wavefunctions deep in the topological
phase. These are obtained by diagonalizing the Hamiltonian on a finite
lattice. The appearance of two localized Majorana modes with near zero energy
at the edges of the system signals its topological nature.

Figure 2: (a) Topological phase diagram of the coupled-wires model as a
function of the SC phase differences $\theta=\left(\phi_{1}-\phi_{2}\right)/2$
and $\phi=\left(\phi_{1}+\phi_{2}\right)/2$ (we set $\phi_{3}=0)$. The color
scale shows the $\mathbb{Z}_{2}$ invariant $\mathcal{Q}$, which is $+1$ ($-1$)
in the trivial (topological) phase, multiplied by the energy gap (normalized
by $\Delta$) . The dark blue regions correspond to a robust large-gap
topological phase. The phase boundaries (dashed lines) and Brillouin zone
boundaries (solid lines) are marked. The inset shows a cut at $\phi=\pi$. The
parameters are $t_{\perp}=1$, $\Delta=0.1$, $\mu=0.995$, $\lambda=0.033$ (on
the optimal manifold), $m=0.01$, $u=1$. (b) At the $C_{3}$-symmetric point
$\theta=\frac{\pi}{3}$, $\phi=\pi$, the system becomes gapless at finite
$k_{\parallel}$. (c) The gap closing becomes an avoided crossing when the
$C_{3}$ symmetry is broken, done here by changing the phases away from the
$C_{3}$-symmetric point. (d) Wavefunctions of (near) zero-energy Majorana
states in the topological phase, calculated for an open system discretized
with $L=600$ sites per wire, at $\left(\theta=\frac{5\pi}{6},\phi=0\right)$.

_Quantum-well model.—_ Having established the possibility of realizing a 1D
topological superconductor based on phase bias alone, we now turn to
exemplifying this concept in a realistic system comprising readily available
ingredients. Specifically, our proposal relies on a spin-orbit-coupled 2DEG
proximitized by three thick SCs. As we have seen, the topological transition
requires an Aharonov-Casher phase and thus, our proposal does not easily lend
itself to an all-planar geometry. Instead, we propose to use a 2DEG with two
(or more) layers giving rise to several subbands, see Fig. 3(a). If the Rashba
SOC parameter $\alpha$ is different in the two subbands, there are closed
loops in which electrons acquire a non-zero Aharonov-Casher phase, mimicking
the periodic boundary conditions in the simplified model we previously
studied.

The system is described by the continuum Hamiltonian

$\displaystyle\mathcal{H}$ $\displaystyle=$
$\displaystyle\left[-\frac{1}{2m^{*}}\left(\partial_{x}^{2}+\partial_{y}^{2}\right)-t_{\perp}\rho_{x}-\mu\right]\tau_{z}$
$\displaystyle+$ $\displaystyle
i\alpha\left(\sigma_{x}\partial_{y}-\sigma_{y}\partial_{x}\right)\tau_{z}\rho_{z}+\left[\Delta\left(x\right)\tau_{+}+\Delta^{*}\left(x\right)\tau_{-}\right]\rho_{\uparrow},$

where the Pauli matrices $\tau$, $\rho$ act in particle-hole and layer space,
respectively, $t_{\perp}$ is the interlayer hopping amplitude,
$\tau_{\pm}=\left(\tau_{x}\pm i\tau_{y}\right)/2$, and
$\rho_{\uparrow}=(\rho_{0}+\rho_{z})/2$.. We assume that the Rashba SOC
parameter is opposite in the two layers. To be specific, we consider an InSb
2DEG with $m^{*}=0.014m_{\rm e}$ and $\alpha=15\,{\rm meV\,nm}$ [42],
corresponding to a SOC length $\ell_{\rm SO}\approx 360\,{\rm nm}$. We take an
induced SC gap of $\Delta=1\,{\rm meV}$, appropriate for, e.g., Nb and Pb
[43], in only one layer. The widths of the SCs (normal regions between them)
are chosen to be $W_{\rm SC}=70\,{\rm nm}$ ($W_{\rm N}=40\,{\rm nm}$.) The
typical length $W$ is chosen roughly according to the relation $\ell_{\rm
SO}\Delta_{\rm SO}=W\Delta$. This rule of thumb, which is derived in the
Supplemental Material [34], provides a way to approximate favorable dimensions
of the system given the material’s parameters 444In our simulations we used
$\Delta=1\,{\rm meV}$, a large value which makes a topological phase
accessible in relatively small system. When using Al as the SC, the energy gap
is about ten times smaller and the system would have to be about ten times
larger..

The Hamiltonian Eq. (Three-phase Majorana zero modes at tiny magnetic fields)
was investigated by discretizing it on a lattice of spacing $a=10\,{\rm nm}$.
The topological phase diagram, calculated by the Pfaffian formula Eq. (3) (now
with $k_{\parallel}\,\widehat{=}\,k_{y}$), is shown in Fig. 3(b). The system
indeed becomes a topological superconductor in the relevant region of phases.
The topological phase constitutes 17% of the displayed $\theta$–$\phi$
section, compared to 25% on the optimal manifold of the coupled-wires model
[cf. Fig. 1(b)], implying that further optimization is possible. The maximal
topological gap is of order $\Delta_{\rm SO}$, which is reasonable: for the
chosen materials $\Delta_{\rm SO}$ is the smallest energy scale. Using
materials with larger $\Delta_{\rm SO}$ will lead to a larger topological gap.

Figure 3: Quantum-well model for topological superconductivity induced by
phase bias only. (a) Schematic of the experimentally available proposal: a
spin-orbit-coupled two-layer 2DEG is contacted by three SCs of width $W_{\rm
SC}$, separated by normal regions of width $W_{\rm N}$. The Rashba SOC
parameter $\alpha$ is assumed to be opposite in the two layers, and pairing is
only induced in one layer. The dashed gray line shows an example of a closed
trajectory that encircles an Aharonov-Casher phase and is affected by the SC
phase winding. (b) Topological phase diagram of the InSb quantum-well model as
a function of the SC phase differences
$\theta=\left(\phi_{1}-\phi_{2}\right)/2$ and
$\phi=\left(\phi_{1}+\phi_{2}\right)/2$ (setting $\phi_{3}=0)$. The color
scale shows the product of the $\mathbb{Z}_{2}$ invariant $\mathcal{Q}$, which
is $+1$ ($-1$) in the trivial (topological) phase, and the energy gap
(normalized by the SOC energy $\Delta_{\rm SO}$). Significant regions of
$\mathcal{Q}=-1$ with a large energy gap appear (dark blue), implying a robust
topological phase. The phase boundaries (dashed black lines), Brillouin zone
boundaries (solid black lines), and optimal phase boundaries (gray lines) are
marked. Parameters used: $\mu=108.9\,{\rm meV}$, $t_{\perp}=0.4\,{\rm meV}$,
corresponding to a density of $n=6.4\times 10^{11}\,{\rm cm}^{-2}$ and a Fermi
wavelength of $\lambda_{\rm F}=31\,{\rm nm}$.

As seen in Fig. 3(b), the gap is small compared to $\Delta_{\rm SO}$ in some
parts of the topological region. By inspecting the Bogoliubov-de-Gennes
spectrum, we find that the small gap originates from the presence of low-
energy high-$k_{\parallel}$ modes. Semiclassically, these modes result from
long trajectories that hardly encounter the SCs, which is a common problem in
such systems [29]. Perturbations that eliminate these trajectories, such as
non-standard geometries [29, 28] or disorder [45], lead to an increased
topological gap. We have verified that adding a chemical potential modulation
along the $x,y$ directions may significantly increase the topological gap.
Furthermore, in the Supplemental Material [34] we show that the topological
phase is robust to various perturbations in the model’s parameters.

_Discussion.—_ In contrast to the vast majority of previous schemes, the
topological phase in our proposal is induced solely by phase winding in the
SC, which is proximity coupled to semiconductors with strong spin-orbit
coupling such as InAs, InSb, or HgTe. SC phases can be manipulated using large
external loops, through which magnetic flux is threaded, or by application of
supercurrent. The applied magnetic field (or the supercurrent), being very
small and removed from the sample itself, should have only a mild effect on
the parent SC. Therefore in-gap states, which may mask the MZMs, are unlikely
to appear.

We illustrated our scheme by an analytically accessible toy model and
introduced a realistic setup in which these ideas can be implemented. Beyond
these settings, we expect that the concept presented here – relying
exclusively on SC phase bias and on the spin-dependent phase acquired in
closed loops (the Aharonov-Casher phase [31]) – may be harnessed in other
systems, as well. For example, it might be possible to realize the wire model
experimentally by contacting three of the six facets of an InAs nanowire with
three thick phase-biased SCs. The role of disorder deserves a separate
treatment. Disorder eliminates trajectories that do not encounter the
superconductors [46, 47, 48, 45] and therefore increases the topological gap.
We expect that under the right conditions it also gives rise to non-retro
Andreev reflection, thereby facilitating the existence of the relevant closed
trajectories.

Finally, a desirable goal for all Majorana platforms is an extension to
networks to implement quantum information processing or a two-dimensional
chiral phase [49]. In our proposal, the experimental challenge is to establish
control over a larger number of superconducting phases. At the same time,
engineering aspects may be significantly simplified by the absence of a need
for a Zeeman field, which requires careful alignment and induces harmful in-
gap states.

_Acknowledgment.—_ We are grateful to C. M. Marcus, N. Schiller, G. Shavit,
and A. Yacoby for fruitful discussions. K.F. acknowledges support from the
Danish National Research Foundation. F.v.O. is supported by Quantera-Grant
TOPOQUANT.

## References

  * Alicea [2012] J. Alicea, Reports on Progress in Physics 75, 076501 (2012).
  * Nayak  _et al._ [2008] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das Sarma, Reviews of Modern Physics 80, 1083 (2008).
  * Oreg and von Oppen [2020] Y. Oreg and F. von Oppen, Annual Review of Condensed Matter Physics 11, 397 (2020).
  * Kitaev [2001] A. Y. Kitaev, Physics-Uspekhi 44, 131 (2001).
  * Lutchyn  _et al._ [2018] R. M. Lutchyn, E. P. a. M. Bakkers, L. P. Kouwenhoven, P. Krogstrup, C. M. Marcus, and Y. Oreg, Nature Reviews Materials 3, 52 (2018).
  * Fu and Kane [2008] L. Fu and C. L. Kane, Physical Review Letters 100, 096407 (2008).
  * Lutchyn  _et al._ [2010] R. M. Lutchyn, J. D. Sau, and S. Das Sarma, Physical Review Letters 105, 077001 (2010).
  * Oreg  _et al._ [2010] Y. Oreg, G. Refael, and F. von Oppen, Physical Review Letters 105, 177002 (2010).
  * Romito  _et al._ [2012] A. Romito, J. Alicea, G. Refael, and F. von Oppen, Physical Review B 85, 020502 (2012).
  * Sau  _et al._ [2010] J. D. Sau, R. M. Lutchyn, S. Tewari, and S. Das Sarma, Physical Review Letters 104, 040502 (2010).
  * Vaitiekėnas  _et al._ [2020a] S. Vaitiekėnas, Y. Liu, P. Krogstrup, and C. M. Marcus, Nature Physics , 1 (2020a).
  * Alicea [2010] J. Alicea, Physical Review B 81, 125318 (2010).
  * Potter and Fu [2013] A. C. Potter and L. Fu, Physical Review B 88, 121109 (2013).
  * Hell  _et al._ [2017] M. Hell, M. Leijnse, and K. Flensberg, Physical Review Letters 118, 107701 (2017).
  * Pientka  _et al._ [2017] F. Pientka, A. Keselman, E. Berg, A. Yacoby, A. Stern, and B. I. Halperin, Physical Review X 7, 021032 (2017).
  * Ren  _et al._ [2019] H. Ren, F. Pientka, S. Hart, A. T. Pierce, M. Kosowsky, L. Lunczer, R. Schlereth, B. Scharf, E. M. Hankiewicz, L. W. Molenkamp, B. I. Halperin, and A. Yacoby, Nature 569, 93 (2019).
  * Fornieri  _et al._ [2019] A. Fornieri, A. M. Whiticar, F. Setiawan, E. Portolés, A. C. C. Drachmann, A. Keselman, S. Gronin, C. Thomas, T. Wang, R. Kallaher, G. C. Gardner, E. Berg, M. J. Manfra, A. Stern, C. M. Marcus, and F. Nichele, Nature 569, 89 (2019).
  * Sau and Tewari [2013] J. D. Sau and S. Tewari, Physical Review B 88, 054503 (2013).
  * Marganska  _et al._ [2018] M. Marganska, L. Milz, W. Izumida, C. Strunk, and M. Grifoni, Physical Review B 97, 075141 (2018).
  * Lesser  _et al._ [2020] O. Lesser, G. Shavit, and Y. Oreg, Physical Review Research 2, 023254 (2020).
  * Nadj-Perge  _et al._ [2013] S. Nadj-Perge, I. K. Drozdov, B. A. Bernevig, and A. Yazdani, Phys. Rev. B 88, 020407 (2013).
  * Pientka  _et al._ [2013] F. Pientka, L. I. Glazman, and F. von Oppen, Physical Review B 88, 155420 (2013).
  * Nadj-Perge  _et al._ [2014] S. Nadj-Perge, I. K. Drozdov, J. Li, H. Chen, S. Jeon, J. Seo, A. H. MacDonald, B. A. Bernevig, and A. Yazdani, Science 346, 602 (2014).
  * Stanescu  _et al._ [2018] T. D. Stanescu, A. Sitek, and A. Manolescu, Beilstein Journal of Nanotechnology 9, 1512 (2018).
  * Vaitiekėnas  _et al._ [2020b] S. Vaitiekėnas, G. W. Winkler, B. van Heck, T. Karzig, M.-T. Deng, K. Flensberg, L. I. Glazman, C. Nayak, P. Krogstrup, R. M. Lutchyn, and C. M. Marcus, Science 367, 10.1126/science.aav3392 (2020b).
  * Sabonis  _et al._ [2020] D. Sabonis, O. Erlandsson, A. Kringhøj, B. van Heck, T. W. Larsen, I. Petkovic, P. Krogstrup, K. D. Petersson, and C. M. Marcus, Physical Review Letters 125, 156804 (2020).
  * Tinkham [1996] M. Tinkham,  _Introduction to Superconductivity_ , 2nd ed., International Series in Pure and Applied Physics (McGraw-Hill, New York, 1996).
  * Melo  _et al._ [2019] A. Melo, S. Rubbert, and A. Akhmerov, SciPost Physics 7, 039 (2019).
  * Laeven  _et al._ [2020] T. Laeven, B. Nijholt, M. Wimmer, and A. R. Akhmerov, Physical Review Letters 125, 086802 (2020).
  * Kotetes [2015] P. Kotetes, Physical Review B 92, 014514 (2015).
  * Aharonov and Casher [1984] Y. Aharonov and A. Casher, Physical Review Letters 53, 319 (1984).
  * van Heck  _et al._ [2014] B. van Heck, S. Mi, and A. R. Akhmerov, Physical Review B 90, 155450 (2014).
  * Note [1] The phases are conveniently visualized by plotting them on a unit circle, fixing the global phase of the superconductors such that $\phi_{3}=0$. Then, the phases form a triangle as shown in the inset of Fig. 1(b). When all $\phi_{n}=0$, the system is obviously time-reversal symmetric. At the three additional time-reversal-symmetric points $(\phi_{1},\phi_{2})=(0,\pi),(\pi,0),(\pi,\pi)$, the triangle degenerates into a line through the center of the unit circle.
  * [34] See Supplemental Material for details on the necessity of SC phase winding, further properties of the optimal manifold in the coupled-wires model, an analysis of the bounds on the topological gap, and a stability analysis of the topological phase in the quantum-well model.
  * Altland and Zirnbauer [1997] A. Altland and M. R. Zirnbauer, Physical Review B 55, 1142 (1997).
  * Schnyder  _et al._ [2008] A. P. Schnyder, S. Ryu, A. Furusaki, and A. W. W. Ludwig, Physical Review B 78, 195125 (2008).
  * Kitaev [2009] A. Kitaev, AIP Conference Proceedings 1134, 22 (2009).
  * Lutchyn  _et al._ [2011] R. M. Lutchyn, T. D. Stanescu, and S. Das Sarma, Physical Review Letters 106, 127001 (2011).
  * Note [2] In the continuum limit, the parallel part of the Hamiltonian reads $\mathcal{H}_{\parallel}=\left({k_{\parallel}^{2}}/{2m^{*}}+uk_{\parallel}\sigma_{x}\right)\tau_{z}$, where $m^{*}$ is the effective electron mass, $u$ is the Rashba SOC parameter along the wires, and the Pauli matrix $\tau_{z}$ acts in particle-hole space.
  * Wimmer [2012] M. Wimmer, ACM Transactions on Mathematical Software 38, 30:1 (2012).
  * Note [3] This delicate gap closure originates from the high symmetry of the system: For $\lambda_{1}=\lambda_{2}=\lambda_{3}\equiv\lambda$, there is a $C_{3}$ rotation symmetry at these points which is removed by any symmetry-breaking perturbation, such as adding a non-proximitized wire, depleting one of the wires, or choosing different $\lambda_{n}$’s.
  * Kallaher  _et al._ [2010] R. L. Kallaher, J. J. Heremans, N. Goel, S. J. Chung, and M. B. Santos, Physical Review B 81, 075303 (2010).
  * Matthias  _et al._ [1963] B. T. Matthias, T. H. Geballe, and V. B. Compton, Reviews of Modern Physics 35, 1 (1963).
  * Note [4] In our simulations we used $\Delta=1\,{\rm meV}$, a large value which makes a topological phase accessible in relatively small system. When using Al as the SC, the energy gap is about ten times smaller and the system would have to be about ten times larger.
  * Haim and Stern [2019] A. Haim and A. Stern, Physical Review Letters 122, 126801 (2019).
  * Golubov and Kupriyanov [1988] A. A. Golubov and M. Y. Kupriyanov, Journal of Low Temperature Physics 70, 83 (1988).
  * Belzig  _et al._ [1996] W. Belzig, C. Bruder, and G. Schön, Physical Review B 54, 9443 (1996).
  * Pilgram  _et al._ [2000] S. Pilgram, W. Belzig, and C. Bruder, Physical Review B 62, 12462 (2000).
  * Alicea  _et al._ [2011] J. Alicea, Y. Oreg, G. Refael, F. von Oppen, and M. P. A. Fisher, Nature Physics 7, 412 (2011).

Supplemental Material

##  SI Relation between $f$ and winding

Here we prove the relation between $f\left(\phi_{1},\phi_{2},\phi_{3}\right)$,
that appears in Eq. (2) of the main text, and phase winding. Without loss of
generality, let us set $\phi_{3}=0$ and $\phi_{2}>\phi_{1}$, and examine

$f\left(\phi_{1},\phi_{2},0\right)=\cos\left(\phi_{1}\right)+\cos\left(\phi_{2}\right)+\cos\left(\phi_{2}-\phi_{1}\right).$
(S1)

It is instructive to factor this expression using trigonometric identities:

$\displaystyle f\left(\phi_{1},\phi_{2},0\right)$
$\displaystyle=2\cos\left(\frac{\phi_{1}+\phi_{2}}{2}\right)\cos\left(\frac{\phi_{2}-\phi_{1}}{2}\right)+2\cos^{2}\left(\frac{\phi_{2}-\phi_{1}}{2}\right)-1$
(S2)
$\displaystyle=2\cos\left(\frac{\phi_{2}-\phi_{1}}{2}\right)\left[\cos\left(\frac{\phi_{1}+\phi_{1}}{2}\right)+\cos\left(\frac{\phi_{2}-\phi_{1}}{2}\right)\right]-1$
$\displaystyle=4\cos\left(\frac{\phi_{2}-\phi_{1}}{2}\right)\cos\left(\frac{\phi_{1}}{2}\right)\cos\left(\frac{\phi_{2}}{2}\right)-1.$

The phases wind, i.e., the triangle connecting them encircles the origin, if
and only if

$0\leq\phi_{1}\leq\pi,\quad\pi\leq\phi_{2}\leq\pi+\phi_{1}.$ (S3)

It follows that

$\cos\left(\frac{\phi_{1}}{2}\right)>0,\,\cos\left(\frac{\phi_{2}}{2}\right)<0,\,\cos\left(\frac{\phi_{2}-\phi_{1}}{2}\right)>0.$
(S4)

Therefore, the first term in Eq. (S2) is non-positive, and thus $f\leq-1$. In
addition, it is straightforward to show that along the boundaries defined by
Eq. (S3), $f=-1$ exactly, and that $f\left(\phi_{1},\phi_{2},0\right)$ has
extrema only at the points
$\left(\phi_{1},\phi_{2}\right)=\left(\frac{2\pi}{3},\frac{4\pi}{3}\right),\left(0,\pi\right),\left(\pi,\pi\right),\left(\pi,2\pi\right)$.
This concludes the proof that phase winding occurs if and only if $f\leq-1$.

##  SII Further analysis of the phase diagram

In this section we provide further details of the phase diagram in the
coupled-wires model. The derivations are based on Eq. (2) of the main text,
which determines the phase boundaries.

We begin by setting $t_{\perp}=1$, and writing the solution of Eq. (2) which
is a quadratic equation for $\Lambda$:

$\Lambda_{\pm}=\frac{1}{2}\left[\mu\left(\Delta^{2}f+3-\mu^{2}\right)\pm\left|\Delta\right|\sqrt{\mu^{2}\left[f\left(\Delta^{2}f+4\right)+6-3\Delta^{2}\right]-\left[\Delta^{2}(\Delta^{2}+2f)+(2f+3)\left(1+\mu^{4}\right)\right]}\right].$
(S5)

In order to have a real solution, the argument of the square root must be non-
negative. Treating the argument of the square root as a second-order
polynomial in $\mu^{2}$, we find that the condition for a real solution is

$(f-3)(f+1)\left(\Delta^{2}(f+1)^{2}+8f+12\right)\geq 0.$ (S6)

Since $-3/2\leq f\leq 3$, the first factor is negative whereas the last factor
is positive. Therefore, this inequality is fulfilled only when $f\leq-1$. This
result, combined with our previous proof that $f\leq-1$ corresponds to phase
winding, is in agreement with Ref. [32].

Figure S1: Critical manifold, marking the topological phase boundaries, in
$\mu/t_{\perp}$, $\Delta/t_{\perp}$, $\Lambda$ space for (a) $f_{\rm
crit}=-1$, (b) $f_{\rm crit}=-1.1$, (c) $f_{\rm crit}=-1.5$, determined by the
solution given in Eq. (S5). As $f_{\rm crit}$ becomes more negative, the
manifold’s area increases. For a given value of $f_{\rm crit}$, the system is
topological at all points contained in the volume surrounded by the surface.
For $f_{\rm crit}=-1$ the manifold shrinks into a circle, as seen from Eq.
(S7). Alternatively, fixing a point in the parameter space, we can contain it
within the volume surrounded by the surface by changing $f$, leading to a
topological state.

The manifold in $\mu$, $\Delta$, $\Lambda$ parameter space defined by Eq. (S5)
is shown in Fig. S1 for several values of $f$. As explained in the main text,
the parameters $\mu,\Delta,\Lambda$ are said to be “optimal” if they support a
solution of Eq. (2) for $f=-1$, i.e., if the necessary condition $f\leq-1$ is
also sufficient. Solving again for $\Lambda$, we obtain

$\Lambda_{\pm}=\frac{1}{2}\left[\mu\left(3-\Delta^{2}-\mu^{2}\right)\pm\sqrt{-\Delta^{2}\left(1-\Delta^{2}-\mu^{2}\right)^{2}}\right].$
(S7)

The only way to make this expression real is demanding $\mu^{2}+\Delta^{2}=1$.
In this case the two solutions $\Lambda_{\pm}$ are identical and equal to
$\mu/t_{\perp}$. This gives us the optimal curve $\mathcal{C}$ – the circle
$\left(\mu,\sqrt{1-\mu^{2}},\mu\right)$.

It is worth noting that along this circle, one can make a simple connection to
the continuum description, thus finding an optimal condition for topological
superconductivity in experimental system parameters. If we take
$\Delta/t_{\perp}$ to be small (which means $\mu\approx t_{\perp}$), we obtain
$\lambda\approx\Delta/t_{\perp}$ (assuming
$\lambda_{1}=\lambda_{2}=\lambda_{3}=\lambda$). In our minimal three-wires
tight-binding description, the effective lattice spacing is the typical
distance between two superconductors $W$. Using a continuum description of the
tight-binding model along the circumference, we get $\Delta_{\rm SO}\approx
t_{\perp}\lambda^{2}$. Since $\lambda$ is the spin-dependent angle accumulated
when hopping between nearest neighbors, and $\ell_{\rm SO}$ is the distance
where a phase of $2\pi$ is accumulated, we have $W/\ell_{\rm
SO}\approx\lambda$. Therefore, we obtain the condition

$\frac{W}{\ell_{\rm SO}}\approx\frac{\Delta_{\rm SO}}{\Delta},$ (S8)

which expresses the ideal geometry as a function of the continuum parameters
only.

Let us exemplify the practical use of this relation. In the main text, we
assumed a superconducting gap of $\Delta=1\,{\rm meV}$, which is appropriate
for e.g. Nb and Pb [43]. Let us now take $\Delta=0.5\,{\rm meV}$, which is
appropriate for e.g. Sn and V [43]. Using the relation Eq. (S8) above, we
simulate a larger system compared to that of Fig. 3(b), with $W_{\rm
SC}=100\,{\rm nm}$, $W_{\rm N}=100\,{\rm nm}$. Fig. S2 shows the resulting
topological phase diagram, which indeed exhibits topological regions with a
topological gap comparable to $\Delta_{\rm SO}$, but the topological region is
smaller, indicating that further optimization might be necessary.

Figure S2: Topological phase diagram of the quantum-well model, same as in
Fig. 3(b), with $\Delta=0.5\,{\rm meV}$ and $W_{\rm SC}=100\,{\rm nm}$,
$W_{\rm N}=100\,{\rm nm}$, $\mu=109.5\,{\rm meV}$, $t_{\perp}=0.28\,{\rm
meV}$.

##  SIII Bounds on the topological gap

Here we discuss the bounds limiting the topological gap, in order to justify
the choice of comparing it to $\Delta_{\rm SO}$, which we made in Fig. 3(b) of
the main text.

To set the stage, we study the topological nanowire model [7, 8]

$H=\left(\frac{k^{2}}{2m^{*}}+uk\sigma_{z}-\mu\right)\tau_{z}-B\sigma_{x}+\Delta\tau_{x},$
(S9)

where $B$ is the applied Zeeman field and $u$ is the SOC parameter. For
simplicity, we focus on $\mu=0$ where the condition for a topological phase is
$B>\Delta$. The two relevant energy scales are $\Delta$ and
$\Delta_{\text{SO}}=mu^{2}/2$, and the question is whether or not they both
set a bound on the energy gap in the topological phase.

At finite $B$ and $\Delta$ there are two minima of the gap in the spectrum as
a function of the momentum $k$, one at $k=0$ and the other near the Fermi
momentum. The topological gap of the system is determined by the smallest of
the two, when $B>\Delta$. It is maximized at $B=B^{*}>\Delta$, for which the
gap at $k=0$ is equal to the gap near the Fermi momentum. A closed-form
expression for $B^{*}$ is hard to obtain, but it is straightforward to find it
numerically given the values of the other parameters.

Fig. S3(a) shows the maximal topological gap as a function of
$\Delta/\Delta_{\rm SO}$, normalized by $\Delta$ and by $\Delta_{\rm SO}$. For
InAs/InSb nanowires proximitized by Al, $\Delta$ and $\Delta_{\rm SO}$ are of
the same order of magnitude. However, for a InAs/InSb 2DEG such as the one we
studied, $\Delta\gg\Delta_{\rm SO}$ and therefore we analyze the asymptotic
behavior of the maximal topological gap in this limit—see the dashed lines in
Fig. S3(a). By fitting the asymptotes, which can be obtained numerically or
analytically, we find that the maximal topological gap in this limit is
$\sim\sqrt{2\Delta_{\text{SO}}\Delta}$. Therefore, the gap can be
parametrically larger than $\Delta_{\rm SO}$. However, it is evident and also
seen in Fig. S3(a) that the gap cannot exceed $\Delta$.

Figure S3: (a) Maximal topological gap for the nanowire model Eq. (S9) (see
also Refs. [7, 8]), as a function of $\Delta/\Delta_{\rm SO}$. The topological
gap is normalized by $\Delta$ (blue) and by $\Delta_{\rm SO}$ (orange), and
plotted in a log-log scale. The dashed lines are the asymptotic forms at
$\Delta\gg\Delta_{\rm SO}$, which is $\sqrt{2\Delta_{\rm SO}\Delta}$, i.e.,
the topological gap may be parametrically larger than $\Delta_{\rm SO}$. (b)
Same for the quantum-well model Eq. (Three-phase Majorana zero modes at tiny
magnetic fields), using the same parameters of Fig. 3(b). The maximal
topological gap is of order $\Delta_{\rm SO}$, and since $\Delta\gg\Delta_{\rm
SO}$ for the parameters we used, it is much smaller than $\Delta$.

The situation is qualitatively different for the quantum-well model studied
here, see Eq. (Three-phase Majorana zero modes at tiny magnetic fields) and
Fig. 3. We demonstrate this by using the same parameters as in Fig. 3(b), with
the phases optimally chosen, and vary the ratio $\Delta/\Delta_{\rm SO}$. The
results are shown in Fig. S3(b). It is clear from this figure that for our
system, the maximal gap in the topological region is of order $\Delta_{\rm
SO}$ (at the optimal configuration), but it is much smaller than $\Delta$.

##  SIV Stability to perturbations

In this section, we analyze the stability of the topological phase in the
quantum-well model to perturbations in the model’s parameters. We demonstrate
the robustness of the topological gap to various realistic imperfections,
which makes our proposal favorable for experiments.

The parameters used in Fig. S4 are the same as those of Fig. 3(b) of the main
text, with $\theta=0.55\pi$, $\phi=0.88\pi$, a representative point inside the
topological region. On top of these, we add perturbations as listed below. We
plot the topological invariant $\mathcal{Q}$ multiplied by the energy gap,
_without_ even changing the SC phases at all (which is probably the simplest
experimental knob).

In Fig. S4(a), the perturbation is a variation in the inter-layer hopping
amplitude $t_{\perp}$. In Fig. S4(b), the chemical potential in the two layers
is different: $\mu_{\rm top}=\mu+\delta\mu$, $\mu_{\rm bottom}=\mu-\delta\mu$.
In Fig. S4(c), the pair potential in the two layers is different: in the main
text we took $\Delta_{\rm top}=\Delta$, $\Delta_{\rm bottom}=0$, and now we
take $\Delta_{\rm top}=\Delta+\delta\Delta$, $\Delta_{\rm
bottom}=-\delta\Delta$. Finally, in Fig. S4(d) we add an inter-layer pair
potential $\Delta_{\rm inter}\tau_{x}\rho_{x}$.

Figure S4: Stability of the topological phase in the quantum-well model to
perturbations in (a) the inter-layer hopping $t_{\perp}$, (b) a difference
$\delta\mu$ in the chemical potential between the two layers, (c) a difference
$\delta\Delta$ in the pair potential between the two layers, and (d) inter-
layer pair potential $\Delta_{\rm inter}$. Plotted is the topological
invariant $\mathcal{Q}$ multiplied by the energy gap in units of the SOC
energy. The dashed red line marks the topological phase boundaries.

Under all these perturbations, the topological phase is robust in an
appreciable range of parameters. The important implication of this finding is
that no fine-tuning is required to drive the system into the topological
phase. We stress again that these reassuring results are obtained without
further tuning of the SC phases, which will likely increase the stability even
more.


# Make Bipedal Robots Learn How to Imitate

Vishal Kumar and Sinnu Susan Thomas Vishal Kumar and Sinnu S. Thomas are with
School of Computer Science and Engineering, Digital University Kerala (IIITMK)
Kerala 695317 India. e-mail: vishal.mi19<EMAIL_ADDRESS>
###### Abstract

Bipedal robots do not perform well as humans since they do not learn to walk
like we do. In this paper we propose a method to train a bipedal robot to
perform some basic movements with the help of imitation learning (IL) in which
an instructor will perform the movement and the robot will try to mimic the
instructor movement. To the best of our knowledge, this is the first time we
train the robot to perform movements with a single video of the instructor and
as the training is done based on joint angles the robot will keep its joint
angles always in physical limits which in return help in faster training. The
joints of the robot are identified by OpenPose architecture and then joint
angle data is extracted with the help of angle between three points resulting
in a noisy solution. We smooth the data using Savitzky-Golay filter and
preserve the Simulatore data anatomy. An ingeniously written Deep Q Network
(DQN) is trained with experience replay to make the robot learn to perform the
movements as similar as the instructor. The implementation of the paper is
made publicly available.

###### Index Terms:

Imitation Learning, Bipedal Robots, Pybullet Simulator, OpenAI.

##  I Introduction

Humans learn to perform any specific task through various supervised,
unsupervised, reinforcement, and semi-supervised techniques. Reinforcement
learning is the way of learning by past experiences, for example consider a
simulation environment in which a simulated agent will try to learn to play a
game of chess, agent will have all the knowledge of the way chess pieces move
and is called as action space for each iteration the agent will perform a move
and the get reward as +1 if he cuts opponents piece, -1 if he land into
dangerous situation and 0 if nothing happens based on the reward and
observation i.e. current position of all chess pieces agent will perform next
move and slowly learns how to play chess and win. This kind of learning is
more unsupervised, IL is a type of reinforcement learning where the learning
agent tries to mimic the teacher. Human beings prefer IL [1] as we
involuntarily employ it in our life irrespective of the success. We refer to
the human or trainable agents (if we talk in reinforcement learning domain)
gathering behaviour, expertise or way of completing any given task by
observing an instructor or any other skilled agent doing it, this method of
learning is getting popular because it eases teaching complex tasks without
exhausting the resources. Number of possible scenarios in a complex
application such as human-computer interaction [2], self-driving cars [3],
augmented reality [4], generative adversarial networks [5], and robot related
tasks is vast and insane. Robots are classified according to their movements
such as wheeled (uses wheels to move) [6], tracked (uses tank like tracks to
move) [7], quadruped (move on four legs) [8], and biped (move on two legs)
[9]. A lot of research is going into IL in bipedal robots [10, 11] due to the
reduced computational cost. If we consider an environment in which multiple
agents are trying to climb to the top of the mountain, let us consider a
random agent (Agent-22)with the ability of IL learn maneuvers that are likely
to be useful for the task. Agent-22 learns by the reinforcement learning
approach along with a referring database for agent to refer and adjust its
actions to learn the task faster without an exploration and exploitation
problem [12]. Bipedal robots utilize a human-like pair of legs to move around,
they are different from humanoid robots [13] as the bipedal may or may not
have a complete human like torso, but humanoid must have. Cassie [14] and
Atlas [15] are examples of bipedal and humanoid robots respectively used for
research. Making a bipedal learn any specific task is easy but the problem
increases with a higher number of tasks. The domain increases so the variables
on which the robot will depend would increase. Training a bipedal to walk with
human-like gait is a tedious task and we address this issue in this paper. The
contributions of the paper are as follows:

  * •

Extract joint angles for various movements of the bipedal robot using a single
video.

  * •

Train a DQN to predict the joint angles for the movement in OpenAI gym
environment

We minimize the data reliability and large duration for training while
extracting the joint angles data from a single video and then processing it to
train DQN. The organization of the paper is as follows: Section II discusses
the state-of-the-art approaches in the field of IL, Section III describes the
working of the proposed approach, Section IV shows the experimental results
and finally Section V concludes the paper.

##  II Related Works

Learning in humans is very diverse and subject to the task, we focus on
implying IL to the bipedal robot POPPY [16]. One of the areas kept in mind
while creating this bipedal is to provide researchers a low-cost machine to
test their experiments. Poppy is a 25-degree of freedom (DOF) humanoid robot
which is very less when compared to humans. It has much lower DOF than humans
who have 244 DOF but it mimics all the major joints a human will engage
whenever performing a movement such as walking, sprinting, kicking. Whenever
we humans move or perform a set of actions to complete a movement like
sprinting we engage a lot of muscles, activate a lot of joints, apply forces
and torque in varying manner to the body joints [17] there are many variables
that come into play, humans use active dynamics walking style i.e. while
activating muscles for movement we consume energy in form of ATP (Adenosine
triphosphate)[18] similarly in robots they consume power in form of
electricity from battery, fuel cells etc.

Bipedal robots utilise passive dynamic walking style [19] in which they do not
consume any power while walking while simply going down the slope. This
passive dynamic walking style saves energy but motion of bipedal is limited as
it can only move in a specific direction and can not avoid obstacles. Ohta et
al. [20] demonstrated a technique that can transform passive to active dynamic
walking providing better control. Simulations for training a bipedal to walk
is a tedious task, if we desire to make a bipedal learn to perform a movement
with the help of neural networks like a simple feed-forward network (FNN) [21]
it is a grueling task for the network. IL [1, 22] helps in this scenario as it
is a supervised learning. In traditional supervised learning training data
represents features and labels, whereas in IL training data demonstrate action
and state. The state represents the position with respect to the world and
velocities of the joints and action consist of a list of joint angles further
the subject will perform in complete duration of movement in the scenario we
are discussing. While training the agent mimics the movement performed by the
subject or demonstrator. Learning in IL can be performed through mainly two
classes behavioral cloning (BC) [23, 24] and inverse reinforcement learning
(IRL) [25].

In BC, computer programs reproduce human behaviour or sub-cognitive skills of
the human while performing a movement [26], demonstrator perform the movement
his or her action (joint angles and position) and the situation arise from
that action (states) are recorded are used as input to train the agent, the
agent will try to mimic the behaviour in learning process. BC is inefficient
in complex tasks. In the IRL model of IL a reward function is being devised
from the optimal behaviour of the demonstrator [26], the devised reward
function performance for training an agent will solely depend on how precise
and clean movement is performed by the demonstrator.

Humans use a lot of muscles while walking and mimicking but it is a difficult
task in robots hence researchers develop a small bipedal robot [27] and made
it walk in forward direction with a constant speed using a single condition
that robot gait must be efficient as human gait. Ames [28] utilise human
walking gait data to develop human-inspired control for walking of a bipedal
robot while 57 muscles are engaged that lead to highly non-linear dynamics and
forces acting on the muscles that sum up to hundreds of DOF. The authors
created human-inspired controllers that utilizes time-based joint angles as
output from human walking gait, processed and fed into a controller to control
the robot. Learning algorithms were not used in this approach. Schuitema et
al. [29] use a simple reinforcement learning algorithm to develop a controller
for a passive dynamic walking robot and to maintain the balance of the bipedal
while walking distorted due to disturbances. A feedback control loop is used
as a hip actuation mechanism and an imbalance of the bipedal calculates the
variables included and provided to the controller. First the algorithm is
tested while simulating and then the mechanical prototype is made and finally
tested.

Dadashi et al. [30] proposed an algorithm where the primal Wasserstein
distance between the expert and the agent state-action distributions is
minimized, Ho and Ermon [31] use Shannon-Jensen divergence, and Fu et al. [32]
use Kullback-Leibler divergence to match state-action distribution of agent to
the expert. Using similarity measures for IL shows promising results but the
practical implementation requires fine tuning of a lot of hyperparameters and
parameters approximations. Wang et al. [33] use Markov Decision Process [34]
for generation of pose prediction sequence of a human from an image. Ho and
Ermon [31] present a generative adversarial network to train the agent for IL.
The discriminator must be very fine tuned according to the problem in order to
improve the performance of the algorithm. Christiano et al. [35] propose a
deep neural network based reinforcement learning from human preferences
approach to perform the IL. This model can learn to imitate with minimal hours
of human training data. Authors tested the performance of this method on Atari
games and have satisfying results. Lepetit et al. [36] use Perspective-n-
points (PnP) to find camera-to-robot pose. This method works flawlessly for
articulated robots but does not work for bipedal or high DOF robots and Lee et
al. [37] propose an algorithm to find joint positions of an articulated robot
using a single image.

We propose to generate major joint angles with a single video of experts while
performing movement with help of OpenPose architecture and then training a DQN
to generate joint angles similar to the original input to make bot perform
movement similar to expert demonstrator. While training we consider the
physical limitations or joint limits of the robot so that the movement cannot
be clipped.

##  III Proposed Methodology

Fig. 1 explains the approach where the joint angles data are extracted with
the help of OpenPose [38], smoothed after that with the help of DQN [39], and
the robot is trained to perform the movement.

Figure 1: Flow chart of the proposed approach.

We use a color image as input to a network that produces 2D locations of the
joints or anatomical keypoints of the human body such as [shoulder, elbow,
knee, waist, head]. These points are connected to form a $2D$ vector that
represent limb with the help of PAFs (Part Affinity Fields) [40] that encode
the degree of union between points or joints. This is a bottom-up approach for
detecting human pose in which, first the body parts are detected and then they
are grouped together to form a complete human based on the PAFs. Fig. 2 shows
the neural architecture for the OpenPose

Figure 2: OpenPose Architecture

.

Branch $1$ network produces PAF’s $\textbf{P}^{1}=\theta^{1}(\textbf{IF})$
where $\theta^{1}$ refers to the neural network in branch 1 for summarizing in
stage 1. In each successive stage original image features IF and predicted
PAF’s from previous are chained to produce precise predictions,

$\textbf{P}^{i}=\theta^{i}(\textbf{IF},\textbf{P}^{i-1}),\forall\;1\leq
i\leq\textit{L}^{p}$ (1)

where $\textit{L}^{p}$ is total number of PAF stages in the network. After
producing precisely refined PAF’s prediction the process is repeated for
confidence map detection starting from the most updated PAF prediction.

$\textbf{C}^{i}=\theta^{i}(\textbf{IF},\textbf{P}^{L^{P}},\textbf{C}^{i-1}),\forall\;1\leq
i\leq\textit{L}^{p}+\textit{L}^{c}$ (2)

where $\textit{L}^{c}$ is the total number of confidence map stages. $L_{2}$
loss is calculated between the estimated predictions and the ground truth maps
and fields, loss functions for the PAF branch at stage $t_{p}$ and loss
function of confidence map branch at stage $t_{cm}$ are:

$\textit{F}^{t_{p}}_{PAF}=\sum^{I}_{i=1}\sum_{s}\textbf{B}(x,y).||\textbf{P}^{t_{p}}_{i}(x,y)-\textbf{P}^{*}_{i}(x,y)||^{2}_{2},$
(3)
$\textit{F}^{t_{cm}}_{CM}=\sum^{J}_{j=1}\sum_{s}\textbf{B}(x,y).||\textbf{C}^{t_{cm}}_{j}(x,y)-\textbf{C}^{*}_{j}(x,y)||^{2}_{2}$
(4)

where $\textbf{P}^{*}_{i}$ is ground truth PAF and $\textbf{C}^{*}_{j}$ is
ground truth confidence map and B is a binary mask with $\textbf{B}(x,y)=0$
when the label is not present in the ground truth, the loss function is used
to guide the network to predict the PAF’s of body in branch 1 and confidence
maps in branch 2 iteratively in precise manner. The overall loss function
becomes:

$\textit{F}=\sum^{L_{p}}_{l=1}\textit{F}^{t_{p}}_{PAF}+\sum^{L_{p}+L_{c}}_{l=L^{p}+1}\textit{F}^{t_{cm}}_{CM}$
(5)

PAF preserves location and orientation of the detected keypoints and predict
the connection between the limbs whether they are from same person or
different person, each PAF is a 2D vector field for each limb in this 2D
vector field a 2D vector encodes the direction of points from one point of
limb to another. After extracting the keypoints or anatomically important
joints of a person angle between the joints is calculated, output of the pose
estimation network is locations of joints of human in sequence ’nose’, ’neck’,
’r-shoulder’, ’r-elbow’, ’r-wrist’, ’l-shoulder’, ’l-elbow’, ’l-wrist’,
’r-hip’, ’r-knee’, ’r-ankle’, ’l-hip’, ’l-knee’, ’l-ankle’, ’r-eye’, ’l-eye’,
’r-ear’, ’l-ear’ and locations of these keypoints is defined as,

$\textbf{K}_{pt}=(x_{i},y_{i})$ (6)

where $x_{i}$ and $y_{i}$ are the pixel locations of the $\textbf{K}_{pt}$
keypoint from above mentioned joints. In this experiment we used ’r-shoulder’,
’r-elbow’, ’r-wrist’, ’l-shoulder’, ’l-elbow’, ’l-wrist’, ’r-hip’, ’r-knee’,
’r-ankle’, ’l-hip’, ’l-knee’, ’l-ankle’. Shoulder joint angle is angle between
a point vertically downwards to the shoulder joint and elbow joint for both
left and right shoulder, elbow joint angle is angle between shoulder and wrist
for both right and left arm, hip joint angle is angle between a point
vertically downwards hip joint and knee joint for both hip joints, knee joint
angle is angle between hip joint and ankle joint for both left and right legs.
We have three points $p_{1}=(x_{1},y_{1}),p_{2}=(x_{2},y_{2})$ and
$p_{3}=(x_{3},y_{3})$ and angle between three points is:

$\Theta=tan^{-1}\frac{m_{2}-m_{1}}{1+m_{1}m_{2}}$ (7)

where $m_{1}=(y_{2}-y_{1})/(x_{2}-x_{1})$ and
$m_{2}=(y_{3}-y_{2})/(x_{3}-x_{2})$ are slopes of two line formed by joining
$p_{1}$, $p_{2}$ and $p_{2}$, $p_{3}$ respectively. After all the major joint
angles are recorded and saved, the joint values are used as input to an OpenAI
gym [41] environment to make the bot learn and perform the movement. The
environment has the Poppy Humanoid robot [16] as the agent, initially the
shoulder joints of the robot are not in optimal standing human condition the
arms were stretching away from the torso. Hence, we replaced all the RPY
(roll, pitch and yaw) values to zero in the URDF ( Universal Robot Description
Format ) file of the robot. Bot joint positions with respect to the world and
joint forces are considered as state and the smoothed data of the joint angles
obtained from above mentioned method are considered as actions. Joint angles
data is smoothed by applying Savitzky-Golay [42] filter using local least
squares polynomial approximation without distorting signal tendency. Consider
joint angles $[y_{i}],i=1,\ldots,n$ of the right shoulder over a period of
$100$ milliseconds. The smoothed joint angle is given by the formula:-

$Y_{i}=\sum_{i=\frac{1-k}{2}}^{\frac{k-1}{2}}C_{z}y_{i+z},\qquad\frac{k-1}{2}\leq
i\leq n-\frac{k-1}{2}$ (8)

where $Y_{i}$ is smoothed data point, $k$ is the value of quadratic polynomial
used for smoothing if $k=7$ then 7-point quadratic polynomial is used and
$z=-3,-2,-1,0,1,2,3$ for $k=7$, $y_{i+z}$ is the data angle of index $i+z$
from original data and $C_{z}$ are convolutions coefficient and are the
elements of the matrix

$C=(U^{T}U)^{-1}U^{T}$ (9)

where $U$ is a vandermonde matrix in which $i^{th}$ row of $U$ has values
$1,z_{i},z_{i}^{2},...$. After smoothing the data we create a DQN that makes
the bipedal robot POPPY learn to imitate the instructor. In DQN states are
given as input to the neural network and Q-value of all possible action at
that time is generated as output, as in reinforcement learning we do not know
the target value there will be too much forking between the two if we use a
single network to calculate predicted value and target value. To solve this
issue we use two neural network one to approximate the Q-value functions and
other as target network to estimate the target, however the architecture of
target network is similar to the prediction network but with fixed weights the
weights of the target network changes after every b iterations (b\- a
hyperparameter) to train smoothly as it will keep the target value fixed for a
little while, weights are copied from the prediction network after b
iterations. Experience gained from the training is stored while training in
the form of (state, action, reward, next-state).

Number of Networks | 2
---|---
Number of layers in each network | 5
Activation Function | tanh/ReLU
Optimizer | Adam
Input Dimensions | 28
Output Dimensions | 25
Number of Hidden Layer Neurons | 768
Learning Rate | 0.001
Batch Size | 64
Number of Epochs | 500
TABLE I: Neural Network Architecture and Hyperparameters

Algorithm 1 shows DQN [43] to learn the task.

Algorithm 1 DQN algorithm for IL

1:Initialize Replay Memory M of certain length

2:Initialize prediction network with random weights $\Theta$

3:Initialize target network with weights $\Theta^{*}=\Theta$

4:for Z iterations do

5: for p iterations do

6: Feed state to DQN and get Q-values for all possible actions

7: Select action according to maximum Q-value

8: Execute action and Store Experience as state, action, reward, next state

9: Sample random batches from experience and calculate loss

10: Loss function =$\mathcal{L}(Q)\;=\;\|Q_{Target}-Q_{Predicted}\|_{2}^{2}$

11: Perform Gradient descent w.r.t network and minimize loss

12: end for

13: Copy $\Theta\quad to\quad\Theta^{*}$

14:end for

We use two types of neural networks CNN and FNN for training and testing
networks. The hyper parameters are given in Table I, activation functions used
are hyperbolic tan and ReLU by CNN and FNN respectively.

##  IV Experimental Results

We develop an OpenAI Gym [41] environment with the help of PyBullet [44]
physics simulator to simulate the real world scenario while training the bot.
Joint angles data is collected from a video of a person walking from Endless
Reference with the help of OpenPose. The full implementation of the
experiments in the paper is available here.

Figure 3: Detection of major joints and their natural limb connections with
OpenPose Network. Figure 4: Original and smoothed data of adult male normal
walk for all major joints.

Figure 5: First column shows 3-D plot of mean joint angles for all test cases
used as input to train, second column shows output of FNN for mean joint
angles for all test cases, third column gives output of CNN for mean joint
angles for all test cases and fourth column shows the plot of RMS error for
both the networks.

Fig. 3 shows the output of OpenPose with all major joints and connection
between them as a skeleton and provides a noisy collected data because of some
sudden movements of the instructor (person in video). We see absence of
detected keypoints from the video so the data must be smoothed out as this
data will lead to distortion in training of the agent. Savitzky-Golay filter
is being used to smoothen the data, this filter will remove the sudden spikes
in the data while keeping the data integrity as in Fig. 4.

Joint angles produced by these networks will be further known as F.Angles and
C.Angles for output of FNN and CNN respectively.

| F.Shoulder | F.Elbow | F.Thigh | F.Knee | C.Shoulder | C.Elbow | C.Thigh | C.Knee
---|---|---|---|---|---|---|---|---
Case-1 | -0.15 | -0.18 | 0.01 | 0.41 | 1.23 | 1.03 | 2.15 | 1.96
Case-2 | 0.17 | 0.21 | -0.14 | 0.36 | -1.36 | 2.00 | 1.98 | -0.75
Case-3 | 0.20 | -0.19 | 0.24 | 0.31 | 2.93 | 1.84 | 2.45 | 2.66
Case-4 | -0.13 | 0.16 | -0.19 | 0.53 | 2.88 | -1.53 | 2.26 | -2.98
TABLE II: Average Mean angle error of joints | F.Shoulder | F.Elbow | F.Thigh | F.Knee | C.Shoulder | C.Elbow | C.Thigh | C.Knee
---|---|---|---|---|---|---|---|---
Case-1 | 1.80 | 1.54 | 1.33 | 1.40 | 4.57 | 2.54 | 6.86 | 4.02
Case-2 | 2.08 | 1.92 | 3.47 | 2.12 | 12.90 | 12.81 | 9.51 | 11.28
Case-3 | 5.02 | 1.03 | 2.34 | 1.59 | 15.61 | 18.04 | 12.50 | 12.66
Case-4 | 6.03 | 1.96 | 2.17 | 2.53 | 22.88 | 21.03 | 21.26 | 22.98
TABLE III: Euclidean distance between original and produced joint angle

Fig. 5 shows four test cases to validate the proposed approach. We looked at
(1) Adult male walking in normal condition, (2) Adult female walking in normal
condition, (3) Adult male walking with long strides, and (4) Adult female
walking with long strides. First column of Fig. 5 shows the original input
joint angles used for training for all four test cases, second column shows
plot of F.Angles, third column shows plot of C.Angles and the last column
shows plots of root mean square (RMS). We conclude from Fig. 5 that the FNN
predicts joint angles more accurately than CNN. The plots in the second column
are smoother when compared to the plots in the third column and similar to the
original input. When we compare a small timestamp from both second and third
column plots we see that joint angle transitions are smooth and there are no
fluctuations whereas in third column plots joint angle transitions have
fluctuations which lead to shivers in output movement.

Average mean angle error is the difference of average of the predicted left
and right joint angles of the body and average of training input of left and
right joint angles. Positive average mean angle error depicts the predicted
joint angles are smaller or equal to the original joint angles and vice versa.
Table II shows average mean angle error of joints. While analysing the data in
Table II it seems that the values of F.Shoulder, F.Elbow, F.Thigh, F.Knee are
much closer to zero which means the joint angle prediction is approximately
equal, while values for C.Shoulder, C.Elbow, C.Thigh, C.Knee are far from zero
compared to F. values.

Table III shows the Euclidean distance between the predicted joint angles and
the original joint angles that is used for training for both the networks. F.
values have lower euclidean distance which means the input and predicted are
very much similar but for the C. values the euclidean distance has higher
values which shows that the input and predicted are distant from each other.

Figure 6: Right and Left side joint angles output from FNN network Figure 7:
Right and Left side joint angles output from CNN network

We consider joint angles as the sole variable to perform IL due to
unavailability of the desired data. Fig. 6 and Fig. LABEL:cnn_outpu shows the
plot of joint angles produced by FNN and CNN respectively after training for
both the left and right side of the body. The left part of the plot is the
input joint angles and right part of the graph is output joint angles. While
comparing these two figures it seems that the output of F.Angles are much
similar to the input angles used to train compared to the C.Angles as shown in
the bottom left plot of Fig. LABEL:cnn_outpu. Fig. 5, 6, and LABEL:cnn_outpu
shows that the joint angles are totally different from the input angles. The
prediction of F.Angles is comparatively similar to the original input data
than C.Angles. This also shows that FNN performs better than CNN in this
scenario. There will be some scenarios when we have to feed the input as an
image instead of joint angles data in those cases CNN may perform better than
FNN. Fig. 8 shows the mean squared error (MSE) and root mean squared error
(RMSE) for the training of both the networks, MSE loss is plotted on left side
of $y$ axis and RMSE loss is plotted on right side of $y$ axis due to range
variation of both losses.

Figure 8: Loss with respect to epochs for proposed approach.

The curves in Fig. 8 shows that the loss is decreasing over a number of epochs
and there is no sudden drop in loss which means the algorithm is learning
instead of remembering the demonstrated action.

##  V Conclusions

In this paper we made an agent learn to perform specific movement as
demonstrated by the instructor using IL, first data generation is done by
OpenPose, a video is fed to the OpenPose network frame by frame and the joint
angles are extracted and stored in text files after this step data smoothing
is done by Savitzky-Golay filter. Data is then fed to a DQN with a target
network made to predict the values of original joint angles data. We used two
networks for training and inferred that the performance of FNN performs better
to give joint angles similar to the original data. The proposed approach works
well for the slow movements such as walking and jogging in a computationally
cost effective manner using a single video as the training data. In future
with 3D-Data generation and other network architecture more complex movements
similar to the instructor can be imitated. The data generated would be 2D
joint angles and their position in 2D space. 3D data generation would
constitute joint angles, positions, and velocity of joints in 3D space giving
rise to more realistic training.

## References

  * [1] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters, “An Algorithmic Perspective on Imitation Learning,” _Foundations and Trends in Robotics_ , vol. 7, no. 1-2, p. 1–179, 2018.
  * [2] P. Booth, _An Introduction to Human-computer Interaction_. Erlbaum, 1989.
  * [3] M. Daily, S. Medasani, R. Behringer, and M. Trivedi, “Self-Driving Cars,” _Computer_ , vol. 50, no. 12, pp. 18–23, 2017.
  * [4] R. T. Azuma, “A Survey of Augmented Reality,” _Presence: Teleoperators and Virtual Environments_ , vol. 6, no. 4, pp. 355–385, 08 1997.
  * [5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” 2014.
  * [6] G. Klančar, A. Zdešar, S. Blažič, and I. Škrjanc, “Chapter 1 - Introduction to Mobile Robotics,” in _Wheeled Mobile Robotics_ , G. Klančar, A. Zdešar, S. Blažič, and I. Škrjanc, Eds., 2017, pp. 1–11.
  * [7] Z. Xu and P. Ma, “A Wall-Climbing Robot for Labelling Scale of Oil Tank’s Volume,” _Robotica_ , vol. 20, no. 2, p. 209–212, Mar. 2002.
  * [8] F. Gao, C. Qi, Q. Sun, X. Chen, and X. Tian, “A Quadruped Robot with Parallel Mechanism Legs,” in _ICRA_ , 2014, pp. 2566–2566.
  * [9] C. C. G. B. G. A. Y. Aoustin, _Bipedal Robots and Walking_. John Wiley & Sons, Ltd, 2009, ch. 1, pp. 1–45.
  * [10] S. Schaal, “Is Imitation Learning the Route to Humanoid Robots?” _Trends in Cognitive Sciences_ , vol. 3, no. 6, pp. 233–242, 1999.
  * [11] C. Nehaniv and K. Dautenhahn, _Imitation and Social Learning in Robots, Humans and Animals: Behavioural, Social and Communicative Dimensions_. Cambridge University Press, 01 2007.
  * [12] A. K. Gupta, K. G. Smith, and C. E. Shalley, “The Interplay Between Exploration and Exploitation,” _Academy of Management Journal_ , vol. 49, no. 4, pp. 693–706, 2006.
  * [13] S. Saeedvand, M. Jafari, H. S. Aghdasi, and J. Baltes, “A Comprehensive Survey on Humanoid Robot Development,” _The Knowledge Engineering Review_ , vol. 34, p. e20, 2019.
  * [14] Y. Gong, R. Hartley, X. Da, A. Hereid, O. Harib, J.-K. Huang, and J. Grizzle, “Feedback Control of a Cassie Bipedal Robot: Walking, Standing, and Riding a Segway,” _CoRR_ , vol. abs/1809.07279, 2018.
  * [15] G. Nelson, A. Saunders, and R. Playter, _The PETMAN and Atlas Robots at Boston Dynamics_. Springer Netherlands, 01 2019, pp. 169–186.
  * [16] M. Lapeyre, “Poppy: Open-source, 3D Printed and Fully-modular Robotic Platform for Science, Art and Education,” Ph.D. dissertation, University of Bordeaux, France, 2014.
  * [17] J. Schulkin, “Evolutionary Basis of Human Running and Its Impact on Neural Function,” _Frontiers in Systems Neuroscience_ , vol. 10, p. 59, 2016.
  * [18] A. Santanasto, P. Coen, N. Glynn, K. Conley, S. Jubrias, F. Amati, E. Strotmeyer, R. Boudreau, B. Goodpaster, and A. Newman, “The Relationship between Mitochondrial Function and Walking Performance in Older Adults with A Wide Range of Physical Function,” _Experimental Gerontology_ , vol. 81, 04 2016.
  * [19] T. McGeer, “Passive Dynamic Walking,” _The International Journal of Robotics Research_ , vol. 9, no. 2, pp. 62–82, 1990.
  * [20] H. Ohta, M. Yamakita, and K. Furuta, “From Passive to Active Dynamic Walking,” _International Journal of Robust and Nonlinear Control_ , vol. 11, no. 3, pp. 287–303, 2001.
  * [21] D. Dai, W. Tan, and H. Zhan, “Understanding the Feedforward Artificial Neural Network Model From the Perspective of Network Flow,” _CoRR_ , vol. abs/1704.08068, 2017.
  * [22] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation Learning: A Survey of Learning Methods,” _ACM Comput. Surv._ , vol. 50, no. 2, Apr. 2017\.
  * [23] F. Torabi, G. Warnell, and P. Stone, “Behavioral Cloning from Observation,” in _IJCAI_ , 2018, p. 4950–4957.
  * [24] N. Gavenski, J. Monteiro, R. Granada, F. Meneguzzi, and R. C. Barros, “Imitating Unknown Policies via Exploration,” in _BMVC_ , 2020.
  * [25] A. Y. Ng and S. Russell, “Algorithms for Inverse Reinforcement Learning,” in _ICML_ , 2000, pp. 663–670.
  * [26] C. Sammut, “Behavioral Cloning,” in _Encyclopedia of Machine Learning_. New York: Springer, 2010, pp. 93–97.
  * [27] S. H. Collins and A. Ruina, “A Bipedal Walking Robot with Efficient and Human-Like Gait,” in _ICRA_ , 2005, pp. 1983–1988.
  * [28] A. D. Ames, “Human-Inspired Control of Bipedal Walking Robots,” _IEEE Transactions on Automatic Control_ , vol. 59, no. 5, pp. 1115–1130, 2014.
  * [29] E. Schuitema, D. G. E. Hobbelen, P. P. Jonker, M. Wisse, and J. G. D. Karssen, “Using A Controller based on Reinforcement Learning for A Passive Dynamic Walking Robot,” in _ICHR_ , 2005, pp. 232–237.
  * [30] R. Dadashi, L. Hussenot, M. Geist, and O. Pietquin, “Primal Wasserstein Imitation Learning,” _CoRR_ , vol. abs/2006.04678, 2021.
  * [31] J. Ho and S. Ermon, “Generative Adversarial Imitation Learning,” in _NIPS_ , vol. 29, 2016.
  * [32] J. Fu, K. Luo, and S. Levine, “Learning Robust Rewards with Adversarial Inverse Reinforcement Learning,” _CoRR_ , vol. abs/1710.11248, 2018.
  * [33] B. Wang, E. Adeli, H. Chiu, D. Huang, and J. C. Niebles, “Imitation Learning for Human Pose Prediction,” in _ICCV_ , 2019, pp. 7123–7132.
  * [34] R. S. Sutton and A. G. Barto, _Introduction to Reinforcement Learning_ , 1st ed. MIT Press, 1998.
  * [35] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep Reinforcement Learning from Human Preferences,” in _NIPS_ , vol. 30, 2017.
  * [36] V. Lepetit, F. Moreno-Noguer, and P. Fua, “EPnP: An Accurate $\mathcal{O}(n)$ Solution to the PnP Problem,” _IJCV_ , vol. 81, 02 2009.
  * [37] T. E. Lee, J. Tremblay, T. To, J. Cheng, T. Mosier, O. Kroemer, D. Fox, and S. Birchfield, “Camera-to-Robot Pose Estimation from a Single Image,” in _ICRA_ , 2020.
  * [38] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” _IEEE TPAMI_ , vol. 43, no. 1, pp. 172–186, 2021.
  * [39] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari With Deep Reinforcement Learning,” in _NIPS Deep Learning Workshop_ , 2013.
  * [40] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields,” in _CVPR_ , 2017, pp. 1302–1310.
  * [41] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” _CoRR_ , vol. abs/1606.01540, 2016.
  * [42] A. Savitzky and M. J. E. Golay, “Smoothing and Differentiation of Data by Simplified Least Squares Procedures,” _Analytical Chemistry_ , vol. 36, no. 8, pp. 1627–1639, 1964.
  * [43] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Leibo, and A. Gruslys, “Deep Q-learning From Demonstrations,” in _AAAI_ , 2018, pp. 3223–3230.
  * [44] E. Coumans and Y. Bai, “PyBullet, A Python Module for Physics Simulation for Games, Robotics and Machine Learning,” http://pybullet.org, 2016–2019.