#  Spin-spin interaction mediated by chiral phonons

Takehito Yokoyama  Department of Physics, Tokyo Institute of Technology, Tokyo
152-8551, Japan

###### Abstract

We study interaction between two magnetic impurities on top of a two
dimensional insulator in the presence of chiral phonons by second-order
perturbation theory. We show that this exchange interaction arises from
angular momentum of phonons through spin-chiral phonon interaction of the form
of spin-orbit coupling. Analytical expressions of the interaction for acoustic
and optical phonons are obtained. The exchange interactions are always
positive due to the bosonic nature of phonons. We find that the exchange
interactions for acoustic and optical phonons show power-law decay with
respect to the distance between the two magnetic impurities and are
proportional to the temperature of the system at high temperature.

Chiral phonons are circularly polarized vibrational motions of ions in
solids.LZhang2014 ; LZhang2015 ; Chen2019 ; Komiyama ; Wang  Due to the
rotational motion of the ions, chiral phonons exhibit angular momentum and
magnetic moment.HZhu2018 ; Ishito2023 ; Grissonnanche2020 ; XTChen2019 ;
Li2019 ; Ueda2023  The rotation of the ions also induces effective magnetic
fields Nova2017 ; Juraschek2019 ; Geilhufe ; Juraschek2022 ; Xiong2022 ;
Luo2023 ; Hernandez2022 ; Chaudhary2023  which have been estimated as 0.01T in
telluriumXiong2022 , 100 T for cerium trichloride Juraschek2022 , and one T in
cerium fluoride Luo2023 . With these magnetic field, one can expect a coupling
between chiral phonons and other degrees of freedom such as spins which has
been extensively investigated in recent years. In fact, couplings between
spins and chiral phonons lead to conversion between electron spin and
microscopic atomic rotation HamadaPRR2020 , conversion of chiral phonons into
magnonsYao2023 , ultrafast demagnetizationTauchert2022 , magnetization
manipulation or reversalBasini ; Kahana2023 ; Davies2023 , light driven
spontaneous phonon chirality and magnetizationYafenRen2023 , interactions
between chiral phonons and eletric spinFransson2023  or orbitalRen2021
magnetizations, and spin current generated by chiral phononsKim2023 ; Li2022 ;
Yao ; Funato .

Recently, chiral phonon-mediated interlayer exchange interaction in
ferromagnetic metal-nonmagnetic insulator heterostructures has been
investigated experimentally. It has been reported that spins rotate as a
function of nonmagnetic insulating spacer thickness.Jeong2022  However,
although it is plausible that this interlayer exchange interaction stems from
chiral phonons,Jeong2022  its microscopic mechanism still remains to be
clarified. As for mechanisms of spin-spin interactions, electronic origins
have been mostly investigated.RKKY ; Bruno  Thus, one fundamental question is:
what is a signature of spin-spin interactions mediated by bosons?

To address these problems, in this paper, we study interaction between two
localized spins on top of a two dimensional insulator in the presence of
chiral phonons by second-order perturbation theory. We show that this
interaction is mediated by chiral phonons through spin-chiral phonon coupling
of the form of spin-orbit coupling. Analytical expressions of the interaction
for acoustic and optical phonons are obtained. The exchange interactions are
always positive due to the bosonic nature of phonons. We find that the
exchange interactions for acoustic and optical phonons show power-law decay
with respect to the distance between the two localized spins and are
proportional to the temperature of the system at high temperature.

We consider two localized spins on a two dimensional insulator in the presence
of chiral phonons as shown in Fig. 1. The Hamiltonian for phonons $H_{0}$
reads
${H_{0}}=\sum\limits_{{\bf{q}}\nu}{\omega_{\bf{q}\nu}}a_{\bf{q}}^{\nu{\dagger}}$
where $a_{\bf{q}}^{\nu{\dagger}}(a_{\bf{q}}^{\nu})$ is the creation
(annihilation) operator for a phonon with wave vector $\bf{q}$ and mode $\nu$
with the corresponding phonon dispersion $\omega_{\bf{q}\nu}$. We set
$k_{B}=\hbar=1$ throughout this paper.

Figure 1:  Schematic diagram of the model. Two magnetic impurities on top of a
two dimensional insulator interact with each other by chiral phonons.

We define the phonon Green function $D$ by

$\displaystyle
D({\bf{q}},\tau,\nu)=-\left\langle{{{\rm{T}}_{\tau}}\left[{{\varphi_{{\bf{q}}\nu}}(\tau)\varphi_{{\bf{q}}\nu}^{\dagger}}\right]}\right\rangle$
(1)

with
$\varphi_{{\bf{q}}\nu}^{\dagger}=\frac{1}{{\sqrt{2}}}\left({a_{\bf{q}}^{\nu}+a_{-{\bf{q}}}^{\nu{\dagger}},-i\left({a_{\bf{q}}^{\nu}-a_{-{\bf{q}}}^{\nu}{\dagger}}\right)}\right)$.
These two components correspond to the second quantization representation of
the displacement and momentum of atoms.

We perform a Fourier transform of the phonon Green function
$D({\bf{q}},\tau,\nu)$ for $H_{0}$ and obtain

$\displaystyle
D({\bf{q}},\omega_{n},\nu)=\frac{1}{{\omega_{n}^{2}+\omega_{{\bf{q}}\nu}^{2}}}\left({\begin{array}[]{*{20}{c}}{-\omega_{{\bf{q}}\nu}}&{\omega_{n}}\\\
{-\omega_{n}}&{-\omega_{{\bf{q}}\nu}}\end{array}}\right)$ (4)
$\displaystyle\equiv
D_{0}({\bf{q}},\omega_{n},\nu)\tau_{0}+D_{2}({\bf{q}},\omega_{n},\nu)i\tau_{2}$
(5)

with the Matsubara frequencies $\omega_{n}=2\pi nT$ and the Pauli matrices
$\tau_{0}$ and $\tau_{2}$. Here, $T$ and $n$ are the temperature and an
interger, respectively.

The angular momentum of phonons along $z$ direction, i.e., direction
perpendicular to the 2D insulator, can be expressed by the second quantization
representation of the displacement $u$ and momentum $p$ as Gao2023

$\displaystyle{L^{z}}=u^{x}p^{y}-u^{y}p^{x}=\frac{1}{{2}}\sum\limits_{{\bf{q}},\nu}{g_{\bf{q}}^{\nu,{\nu^{\prime}}}\left({a_{\bf{q}}^{\nu}+a_{-{\bf{q}}}^{\nu{\dagger}}}\right)}\left({a_{\bf{q}}^{\nu^{\prime}{\dagger}}-a_{-{\bf{q}}}^{\nu^{\prime}}}\right)$
(6)

with

$\displaystyle
g_{\bf{q}}^{\nu,{\nu^{\prime}}}=\sqrt{\frac{{\omega_{\bf{q}\nu^{\prime}}}}{{\omega_{\bf{q}\nu}}}}\xi_{{\bf{q}},{\nu^{\prime}}}^{\dagger}\left({\begin{array}[]{*{20}{c}}0&{-i}\\\
i&0\end{array}}\right){\xi_{{\bf{q}},\nu}}.$ (9)

Here, ${\xi_{{\bf{q}},\nu}}$ is the polarization vector. We assume doubly
degenerate acoustic and doubly degenerate optical modes which may be realized
in hexagonal lattices.LZhang2014  Each of circular polarized phonon modes is
labeled by $\nu=\pm$. Now, we assume
${\xi_{{\bf{q}},\nu}}=\frac{1}{{\sqrt{2}}}{\left({\begin{array}[]{*{20}{c}}1&{\nu
i}\end{array}}\right)^{t}}$. Then we have
$g_{\bf{q}}^{\nu,{\nu^{\prime}}}=\nu{\delta_{\nu,\nu^{\prime}}}$ and arrive at

$\displaystyle{L^{z}}=\frac{1}{{2}}\sum\limits_{{\bf{q}},\nu}{\nu\varphi_{{\bf{q}}\nu}^{\dagger}{\tau_{2}}\varphi_{{\bf{q}}\nu}}.$
(10)

Since we are interested in transfer of angular momenta between chiral phonons
and localized spins, we consider the interaction between them in the form of
spin-orbit coupling:

$\displaystyle
V=\lambda{L^{z}}\sum\limits_{i=1,2}\delta\left({{\bf{r}}-{{\bf{R}}_{i}}}\right)S_{i}^{z}$
(11)

where the localized spins at ${\bf{R}}_{1}$ and ${\bf{R}}_{2}$ are denoted by
$S_{1}$ and $S_{2}$, respectively. The parameter $\lambda$ represents the
coupling strength.

The spin-spin interaction can be calculated by second-order correction with
respect to $V$ to the free energyRKKY ; Kogan ; Oriekhov

$\displaystyle\delta{F}=\frac{{{\lambda^{2}}}}{8}S_{1}^{z}S_{2}^{z}T\sum\limits_{n,\nu}{{\mathop{\rm
tr}\nolimits}}\left[{{\tau_{2}}D\left({{{\bf{R}}_{1}}-{{\bf{R}}_{2}},{\omega_{n}},\nu}\right)}\right.\left.{{\tau_{2}}D\left({{{\bf{R}}_{2}}-{{\bf{R}}_{1}},{\omega_{n}},\nu}\right)}\right].$
(12)

We first consider acoustic phonons with $\omega_{\bf{q}\nu}={v_{a}}q$ and
$q=\left|{\bf{q}}\right|$. Then, the components of the phonon Green function
$D$ are obtained asPrudnikov

$\displaystyle{D_{0}}({\bf{R}},\omega_{n},\nu)=-\frac{1}{{2\pi{v_{a}}}}\left({\frac{1}{R}-\frac{{\pi\left|{\omega_{n}}\right|}}{{2{v_{a}}}}{I_{0}}\left({\frac{{\omega_{n}}}{{{v_{a}}}}R}\right)+\frac{{\pi{\omega_{n}}}}{{2{v_{a}}}}{L_{0}}\left({\frac{{\omega_{n}}}{{{v_{a}}}}R}\right)}\right),$
(13) $\displaystyle{D_{2}}({\bf{R}},\omega_{n},\nu)=\frac{{\omega_{n}}}{{2\pi
v_{a}^{2}}}{K_{0}}\left({\frac{{\left|{\omega_{n}}\right|}}{{{v_{a}}}}R}\right)$
(14)

with $R=\left|{\bf{R}}\right|$ and ${\bf{R}}={{\bf{R}}_{1}}-{{\bf{R}}_{2}}$.
Here, $I_{0}$ and $K_{0}$ are the modified Bessel functions and $L_{0}$ is the
Struve function.

Figure 2:  Plots of (a) $F(x)$ and (b) $f(x)$.

Therefore, we arrive at the expression of the correction to the free energy:
$\delta{F}=J_{z}^{a}S_{1}^{z}S_{2}^{z}$ with

$\displaystyle
J_{z}^{a}=\frac{{{\lambda^{2}}}}{{8{\pi^{2}}v_{a}{R^{3}}}}xF(x),\;F(x)=\sum\limits_{n}{f(2nx)},$
(15) $\displaystyle
f(x)={{\left({1-\frac{{\pi\left|x\right|}}{2}{I_{0}}\left(x\right)+\frac{{\pi
x}}{2}{L_{0}}\left(x\right)}\right)}^{2}}-{{\left({x{K_{0}}\left(x\right)}\right)}^{2}}$
(16)

with $x=\frac{T}{{{v_{a}}}}R$.

Figure 2 (a) shows the behavior of $F(x)$. It increases linearly with $x$ for
$x<1$ and saturates at $x\sim 1$. Therefore, for $x<1$, i.e., $R<\xi_{T}$
where $\xi_{T}=v_{a}/T$ is the thermal coherence length, the exchange coupling
$J_{z}^{a}$ behaves as

$\displaystyle J_{z}^{a}\sim T^{2}/R$ (17)

while for $x>1$ ($R>\xi_{T}$), it behaves as

$\displaystyle J_{z}^{a}\sim T/R^{2}.$ (18)

Since $F(x)>0$, we see that the exchange couplng is positive.

Figure 2 (b) shows the behavior of $f(x)$. It oscillates as a function of $x$
and goes to zero for $x\gg 1$. Thus, we find that the intergers $n$ which
satisfy $2nx\lesssim 3$ dominantly contribute to $J_{z}^{a}$ (see Eq.(10)).

Let us estimate the magnitude of the exchange interaction for acoustic
phonons. For $\lambda=10$meV/nm2, $T=300$K, ${v_{a}}=10^{4}$m/s, and $R=1$nm,
we have $x\sim 4$ and $J_{z}^{a}\sim 10^{-2}{\rm{eV}}$.

Figure 3:  Plots of (a) $(J_{1}(x)/x)^{2}$ which shows the dependence of
$J_{z}^{o}$ on $R$ and (b) $(x\sinh^{2}(1/x))^{-1}$ which exhibits the
dependence of $J_{z}^{o}$ on $T$.

Next, let us consider optical phonons with $\omega_{\bf{q}\nu}=\omega_{0}$
(constant). Then, the components of the phonon Green function $D$ are obtained
asPrudnikov

$\displaystyle{D_{0}}({\bf{R}},\omega_{n},\nu)=-\frac{1}{{2\pi}}\frac{{\omega_{0}}}{{\omega_{n}^{2}+\omega_{0}^{2}}}\frac{{\Lambda{J_{1}}\left({\Lambda
R}\right)}}{R},$ (19)
$\displaystyle{D_{2}}({\bf{R}},\omega_{n},\nu)=\frac{1}{{2\pi}}\frac{{\omega_{n}}}{{\omega_{n}^{2}+\omega_{0}^{2}}}\frac{{\Lambda{J_{1}}\left({\Lambda
R}\right)}}{R}$ (20)

where $\Lambda$ is the cutoff for $q$ and $J_{1}$ is the Bessel function.

Therefore, we obtain the expression of the correction to the free energy:
$\delta{F}=J_{z}^{o}S_{1}^{z}S_{2}^{z}$ with

$\displaystyle
J_{z}^{o}=\frac{{{\lambda^{2}}}}{{32{\pi^{2}}}}\frac{{{{\left({\Lambda{J_{1}}\left({\Lambda
R}\right)}\right)}^{2}}}}{{{R^{2}}T{{\sinh}^{2}}\left({\frac{{\omega_{0}}}{{2T}}}\right)}}.$
(21)

We find that $J_{z}^{o}$ is positive. From this expression, we find asymptotic
behaviors of the exchange interaction at $T\to 0$ and $T\to\infty$:

$\displaystyle J_{z}^{o}\sim\frac{{{e^{-\frac{{\omega_{0}}}{{T}}}}}}{T}\;(T\to
0),\;J_{z}^{o}\sim T\;(T\to\infty)$ (22)

and also those at $R\to 0$ and $R\to\infty$:

$\displaystyle J_{z}^{o}\sim{\rm{const.}}\;(R\to 0),$ (23) $\displaystyle
J_{z}^{o}\sim\frac{1}{{{R^{3}}}}{\cos^{2}}\left({\Lambda
R-\frac{3}{4}\pi}\right)\;(R\to\infty).$ (24)

This oscillating behavior for sufficiently large $R$ has a similarity to the
RKKY interaction. RKKY

To see the dependence of $J_{z}^{o}$ on $R$, we show the behavior of
$(J_{1}(x)/x)^{2}$ in Fig. 3 (a) (see also Eq.(16)). It is an oscillatory
decreasing junction of $x$. To see the dependence of $J_{z}^{o}$ on $T$, we
show the behavior of $(x\sinh^{2}(1/x))^{-1}$ in Fig. 3 (b). It is a
monotonically increasing function of $x$. The asymptotic behaviors in these
figures are consistent with Eqs.(17-19).

Let us estimate the magnitude of the exchange interaction for optical phonons.
For $\lambda=10$meV/nm2, $T=300$K, $\omega_{0}=10$meV, $\Lambda=10^{10}$/m,
and $R=1$nm, we have $J_{z}^{o}\sim 10^{-4}$ eV.

Here, we provide several discussions on the results obtained in this paper:

(i) For both acoustic and optical phonons, the exchange interactions become
zero at zero temperature and proportional to the temperature at high
temperature. This is because these phonons are thermally excited and reflects
the behaviors of the Bose distribution functions at these temperatures.

(ii) We have assumed that the coupling strength between chiral phonons and
spins $\lambda$ in Eq.(6) is a constant. In general, it depends on
temperature.Gao2023  In this case, the temperature dependences of the exchange
couplings are superimposed by that of $\lambda^{2}$.

(iii) We have found that the exchange interactions mediated by chiral phonons
show algebraic decay with respect to the distance between the two spins. On
the other hand, the exchange interaction mediated by electrons in insulators
shows exponential decay with the distance.Bruno ; Faure  Therefore, it is
possible to distinguish between the contribution from chiral phonons and that
from electrons by comparing the dependence of the exchange coupling on the
distance between the localized spins.

(iv) Let us compare the exchange interactions mediated by chiral phonons and
electrons in metals. As for the RKKY interaction, the exchange couplings are
proportional to trigonometric functions of $k_{F}R$ where $k_{F}$ is the Fermi
wavenumber. RKKY ; Kogan ; Oriekhov  These trigonometric functions lead to
oscillations of the exchange couplings accompanied by their sign change with
respect to $R$. In contrast, since phonons are bosons, the chemical potential
is zero (correponding to $k_{F}=0$ in the case of electrons), and therefore we
do not see such oscillations of the exchange couplings mediated by chiral
phonons as we have seen in this paper. The absence of oscillations accompanied
by sign change is expected to be a general feature of exchange couplings
mediated by bosons.

(v) Let us also discuss the coupling of chiral phonons with magnons based on
our results. Consider an effective Hamiltonian with
$H=J_{z}S_{1}^{z}S_{2}^{z}$ and $J_{z}=J_{z}^{a}$ or $J_{z}^{o}$. Now, let us
quantize the localized spins, assume antiferromagnetic states ($J_{z}>0$), and
use the Holstein-Primakoff transformation
$(-1)^{i}S_{i}^{z}=S-a_{i}^{\dagger}a_{i}(i=1,2)$ where $S$ denotes the
magnitude of the spins.Holstein  We then obtain, apart from a trivial constant
term,

$\displaystyle H=J_{z}S(a_{1}^{\dagger}a_{1}-a_{2}^{\dagger}a_{2}).$ (25)

Since the coupling $J_{z}$ stems from chiral phonons, this indicates that
magnons can be excited by chiral phonons.Yao2023

In summary, we have investigated interaction between two magnetic impurities
on top of a two dimensional insulator in the presence of chiral phonons by
second-order perturbation theory. We have shown that this interaction arises
from angular momentum of phonons through spin-chiral phonon interaction of the
form of spin-orbit coupling. Analytical expressions of the interaction for
acoustic and optical phonons are obtained. The exchange interactions are
always positive in contrast to the RKKY interactions due to the bosonic nature
of phonons. We have found that the exchange interactions for acoustic and
optical phonons show power-law decay with respect to the distance between the
two magnetic impurities and are proportional to the temperature of the system
at high temperature.

This work was supported by JSPS KAKENHI Grant No. JP30578216.

## References

  * (1) L. Zhang and Q. Niu, Phys. Rev. Lett. 112, 085503 (2014).
  * (2) L. Zhang and Q. Niu, Phys. Rev. Lett. 115, 115502 (2015).
  * (3) H. Chen, W. Zhang, Q. Niu, and L. Zhang, 2D Materials 6, 012002 (2019).
  * (4) H. Komiyama and S. Murakami, Phys. Rev. B 103, 214302 (2021).
  * (5) Q. Wang, S. Li, J. Zhu, H. Chen, W. Wu, W. Gao, L. Zhang, and S. A. Yang, Phys. Rev. B 105, 104301 (2022).
  * (6) H. Zhu, J. Yi, M.-Y. Li, J. Xiao, L. Zhang, C.-W. Yang, R.A. Kaindl, L.-J. Li, Y. Wang, and X. Zhang, Science 359, 579 (2018).
  * (7) X. Chen, X. Lu, S. Dubey, Q. Yao, S. Liu, A. Ataei, X. Wang, M. Dion, Q. Xiong, L. Zhang, and A. Srivastava, Nat. Phys. 15, 221 (2019).
  * (8) Z. Li, T. Wang, C. Jin, Z. Lu, Z. Lian, Y. Meng, M. Blei, M. Gao, T. Taniguchi, K. Watanabe, T. Ren, T. Cao, S. Tongay, D. Smirnov, L. Zhang, and S.-F. Shi, ACS Nano 13, 14107 (2019).
  * (9) G. Grissonnanche, S. Thériault, A. Gourgout, M.-E. Boulanger, E. Lefrançoiss, A. Ataei, F. Laliberté, M. Dion, J.-S. Zhou, S. Pyon, T. Takayama, H. Takagi, N. Dorion-Leyraud, and T. Taillefer, Nat. Phys. 16, 1108 (2020).
  * (10) K. Ishito, H. Mao, Y. Kousaka, Y. Togawa, S. Iwasaki, T. Zhang, S. Murakami, J.-I. Kishine, and T. Satoh, Nat. Phys. 19, 35 (2023).
  * (11) H. Ueda, M. García-Fernández, S. Agrestini, C. P. Romao, J. van den Brink, N. A. Spaldin, K.-J. Zhou, and U. Staub, Nature 618, 946 (2023).
  * (12) T. F. Nova, A. Cartella, A. Cantaluppi, M. Först, D. Bossini, R. V. Mikhaylovskiy, A. V. Kimel, R. Merlin, and A. Cavalleri, Nat. Phys. 13, 132 (2017).
  * (13) D. M. Juraschek and N. A. Spaldin, Phys. Rev. Mater. 3, 064405 (2019).
  * (14) R. M. Geilhufe, V. Juričić, S. Bonetti, J.-X. Zhu, and A. V. Balatsky, Phys. Rev. Res. 3, L022011 (2021).
  * (15) D. M. Juraschek, T. Neuman, and P. Narang, Phys. Rev. Res. 4, 013129 (2022).
  * (16) G. Xiong, H. Chen, D. Ma, and L. Zhang, Phys. Rev. B 106, 144302 (2022).
  * (17) J. Luo, T. Lin, J. Zhang, X. Chen, E. R. Blackert, R. Xu, B. I. Yakubson, and H. Zhu, Science 382, 698 (2023).
  * (18) F. G. G. Hernandez, A. Baydin, S. Chaudhary, F. Tay, I. Katayama, J. Takeda, H. Nojiri, A. K. Okazaki, P. H. O. Rappl, E. Abramof, M. Rodriguez-Vega, G. A. Fiete, and J. Kono, Sci. Adv. 9, eadj4074 (2023).
  * (19) S. Chaudhary, D. M. Juraschek, M. Rodriguez-Vega, and G. A. Fiete, arXiv:2306.11630 (2023).
  * (20) M. Hamada and S. Murakami, Phys. Rev. Res. 2, 023275 (2020).
  * (21) D. Yao and S. Murakami, J. Phys. Soc. Jpn. 93, 034708 (2024).
  * (22) S. R. Tauchert, M. Volkov, D. Ehberger, D. Kazenwadel, M. Evers, H. Lange, A. Donges, A. Book, W. Kreuzpaintner, U. Nowak, and P. Baum, Nature 602, 73 (2022).
  * (23) M. Basini, M. Pancaldi, B. Wehinger, M. Udina, T. Tadano, M. C. Hoffmann, A. V. Balatsky, and S. Bonetti, arXiv:2210.01690.
  * (24) C. S. Davies, F. G. N. Fennema, A. Tsukamoto, I. Razdolski, A. V. Kimel, and A. Kirilyuk, arXiv:2305.11551 (2023).
  * (25) T. Kahana, D. A. B. Lopez, and D. M. Juraschek, arXiv:2305.18656 (2023).
  * (26) Y. Ren, M. Rudner, and D. Xiao, Phys. Rev. Lett. 132, 096702 (2024).
  * (27) J. Fransson, Phys. Rev. Res. 5, L022039 (2023).
  * (28) Y. Ren, C. Xiao, D. Saparov, and Q. Niu, Phys. Rev. Lett. 127, 186403 (2021).
  * (29) K. Kim, E. Vetter, L. Yan, C. Yang, Z. Wang, R. Sun, Y. Yang, A. H. Comstock, X. Li, J. Zhou, L. Zhang, W. You, D. Sun, and J. Liu, Nat. Mater. 22, 322 (2023).
  * (30) X. Li, J. Zhong, J. Cheng, H. Chen, H. Wang, J. Liu, D. Sun, L. Zhang, and J. Zhou, Sci. China Phys. Mech. Astron. 67, 237511 (2024).
  * (31) D. Yao, M. Matsuo, and T. Yokoyama, Appl. Phys. Lett. 124, 162603 (2024).
  * (32) T. Funato, M. Matsuo, and T. Kato, arXiv:2401.17864.
  * (33) S. G. Jeong, J. Kim, A. Seo, S. Park, H. Y. Jeong, Y.-M. Kim, V. Lauter, T. Egami, J. H. Han, and W. S. Choi, Sci. Adv. 8, eabm4005 (2022).
  * (34) M. A. Ruderman and C. Kittel, Phys. Rev. 96, 99 (1954); T. Kasuya, Prog. Theor. Phys. 16, 45 (1956); K. Yosida, Phys. Rev. 106, 893 (1957).
  * (35) P. Bruno, B 49, 13231 (1994); Phys. Rev. B 52, 411 (1995).
  * (36) Y. Gao, Y. Pan, J. Zhou, and L. Zhang, Phys. Rev. B 108, 064510 (2023).
  * (37) E. Kogan, Phys. Rev. B 84, 115119 (2011).
  * (38) D. O. Oriekhov and V. P. Gusynin, Phys. Rev. B 101, 235162 (2020).
  * (39) A. P. Prudnikov, Yu. A. Brychkov, and O. I.Marichev, Integrals and Series. Special Functions (Nauka, Moskow, 1983).
  * (40) J. Faure-Vincent, C. Tiusan, C. Bellouard, E. Popova, M. Hehn, F. Montaigne, and A. Schuhl, Phys. Rev. Lett. 89, 107206 (2002).
  * (41) T. Holstein and H. Primakoff, Phys. Rev. 58, 1098 (1940).


vskip=0pt

# Habitat-Web: Learning Embodied Object-Search Strategies
from Human Demonstrations at Scale

Ram Ramrakhya1 Eric Undersander2 Dhruv Batra1,2 Abhishek Das2
1Georgia Institute of Technology 2Meta AI Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present a large-scale study of imitating human demonstrations on tasks that
require a virtual robot to search for objects in new environments – (1)
ObjectGoal Navigation (_e.g_. _‘find & go to a chair’_) and (2) Pick&Place
(_e.g_. _‘find mug, pick mug, find counter, place mug on counter’_). First, we
develop a virtual teleoperation data-collection infrastructure – connecting
Habitat simulator running in a web browser to Amazon Mechanical Turk, allowing
remote users to teleoperate virtual robots, safely and at scale. We collect
$80k$ demonstrations for ObjectNav and $12k$ demonstrations for Pick&Place,
which is an order of magnitude larger than existing human demonstration
datasets in simulation or on real robots. Our virtual teleoperation data
contains $29.3M$ actions, and is equivalent to $22.6k$ hours of real-world
teleoperation time, and illustrates rich, diverse strategies for solving the
tasks. Second, we use this data to answer the question – how does large-scale
imitation learning (IL) (which has not been hitherto possible) compare to
reinforcement learning (RL) (which is the status quo)? On ObjectNav, we find
that IL (with no bells or whistles) using $70k$ human demonstrations
outperforms RL using $240k$ agent-gathered trajectories. This effectively
establishes an ‘exchange rate’ – a single human demonstration appears to be
worth ${\sim}4$ agent-gathered ones. More importantly, we find the IL-trained
agent learns efficient object-search behavior from humans – it peeks into
rooms, checks corners for small objects, turns in place to get a panoramic
view – none of these are exhibited as prominently by the RL agent, and to
induce these behaviors via contemporary RL techniques would require tedious
reward engineering. Finally, accuracy  _vs_. training data size plots show
promising scaling behavior, suggesting that simply collecting more
demonstrations is likely to advance the state of art further. On Pick&Place,
the comparison is starker – IL agents achieve ${\sim}18\%$ success on episodes
with new object-receptacle locations when trained with $9.5k$ human
demonstrations, while RL agents fail to get beyond $0\%$. Overall, our work
provides compelling evidence for investing in large-scale imitation learning.
Project page: ram81.github.io/projects/habitat-web.

##  1 Introduction

Figure 1: a) Example ObjectNav 1) human demonstration, 2) agent trained on
human demonstrations, and 3) shortest path. Notice how humans demonstrate
sophisticated exploration behavior to succeed at this task in unseen
environments, which is hard to engineer into the right reward for an RL agent
and is unlikely to be captured in shortest path demonstrations. An agent
trained on human demonstrations learns this exploration and object-search
behavior. b) Success on the ObjectNav MP3D-val split  _vs_. no. of human
demonstrations for training.

General-purpose robots that can perform a diverse set of embodied tasks in a
diverse set of environments _have_ to be good at visual exploration. Consider
the canonical example of asking a household robot, _‘Where are my keys?’_. To
answer this (assuming the robot does not remember the answer from memory), the
robot would have to search the house, often guided by intelligent priors –
_e.g_. peeking into the washroom or kitchen might be sufficient to be
reasonably sure the keys are not there, while exhaustively searching the
living room might be much more important since keys are more likely to be
there. While doing so, the robot has to internally keep track of where all it
has been to avoid redundant search, and it might also have to interact with
objects,  _e.g_. check drawers and cabinets in the living room (but not those
in the washroom or kitchen!).

This example illustrates fairly sophisticated exploration, involving a careful
interplay of various implicit objectives (semantic priors, exhaustive search,
efficient navigation, interaction, _etc_.). Many recent tasks of interest in
the embodied AI community –  _e.g_. ObjectGoal Navigation [1, 2],
rearrangement [3, 4], language-guided navigation [5, 6] and interaction [7],
question answering [8, 9, 10, 11, 12] – involve some flavor of this visual
exploration. With careful reward engineering, reinforcement learning (RL)
approaches to these tasks have achieved commendable success [13, 14, 15, 16,
17]. However, engineering the ‘right’ reward function so that the learned
policy exhibits desired behavior is unintuitive and frustrating (even for
domain experts), expensive (requiring multiple rounds of retraining under
different rewards), and not scalable to new tasks or behaviors. For complex
tasks (_e.g_. object rearrangement or tasks specified in open-ended natural
language), RL from scratch may not even get off the ground.

In this work, we advance the alternative research agenda of imitation learning
[18] – _i.e_. collecting a large dataset of human demonstrations (that
implicitly capture intelligent behavior we wish to impart to our agents) and
learning policies directly from these human demonstrations.

First, we develop a safe scalable virtual teleoperation data-collection
infrastructure – connecting the Habitat simulator running in a browser to
Amazon Mechanical Turk (AMT). We develop this in way that enables collecting
human demonstrations for a variety of tasks being studied within the Habitat
[19, 20] ecosystem (_e.g_. PointNav [2], ObjectNav [1, 2], ImageNav [21], VLN-
CE [6], MultiON [22],  _etc_.).

We use this infrastructure to collect human demonstration datasets for $2$
tasks requiring visual search – 1) ObjectGoal Navigation (_e.g_. _‘find & go
to a chair’_) and 2) Pick&Place (_e.g_. _‘find mug, pick mug, find counter,
place on counter’_). In total we collect $92k$ human demonstrations, $80k$
demonstrations for ObjectNav and $12k$ demonstrations for Pick&Place. In
contrast, the largest existing datasets have $3$-$10k$ human demonstrations in
simulation [23, 24, 25] or on real robots [26, 27], an order of magnitude
smaller. This virtual teleoperation data contains $29.3M$ actions, which is
equivalent to $22,600$ hours of real-world teleoperation time assuming a
LoCoBot motion model from [28] (details in appendix (Sec. A.3)). The first
thing this data provides is a ‘human baseline’ with sufficiently tight error-
bars to be taken seriously. On the ObjectNav validation split, humans achieve
$93.7$$\pm 0.1$$\%$ success and $42.5$$\pm 0.5$$\%$ Success Weighted by Path
Length (SPL) [2] (_vs_. $34.6\%$ success and $7.9\%$ SPL for the 2021 Habitat
ObjectNav Challenge winner [15]). The success rate ($93.7\%$) suggests that
this task is largely doable for humans (but not $100\%$). The SPL ($42.5\%$)
suggests that even humans need to explore significantly.

Beyond scale, the data is also rich and diverse in the strategies that humans
use to solve the tasks. Fig. 1 shows an example trajectory of an AMT user
controlling a LoCoBot looking for a ‘plant’ in a new house – notice the
peeking into rooms, looping around the dining table – all of which is
(understandably) absent from the shortest path to the goal.

We use this data to answer the question – how does large-scale imitation
learning (IL) (which has not been hitherto possible) compare to large-scale
reinforcement learning (RL) (which is the status quo)? On ObjectNav, we find
that IL (with no bells or whistles) using only $70k$ human demonstrations
outperforms RL using $240k$ agent-gathered trajectories. This effectively
establishes an ‘exchange rate’ – a single human demonstration appears to be
worth ${\sim}4$ agent-gathered ones. More importantly, we find the IL-trained
agent learns _efficient object-search behavior_ – as shown in Fig. 1 and Sec.
7. The IL agent learns to mimic human behavior of peeking into rooms, checking
corners for small objects, turning in place to get a panoramic view – none of
these are exhibited as prominently by the RL agent. Finally, the accuracy
_vs_. training-data-size plot (Fig. 1b) shows promising scaling behavior,
suggesting that simply collecting more demonstrations is likely to advance the
state of art further. On Pick&Place, the comparison is even starker – IL-
agents achieve ${\sim}18\%$ success on episodes with new object-receptacle
locations when trained with $9.5k$ human demonstrations, while RL agents fail
to get beyond $0\%$.

On both tasks, we find that demonstrations from humans are essential;
imitating shortest paths from an oracle produces neither accuracy nor the
strategic search behavior. In hindsight, this is perfectly understandable –
shortest paths (_e.g_. Fig. 1(a3)) do not contain any exploration but the task
requires the agent to explore. Essentially, a shortest path is inimitable, but
imitation learning is invaluable. Overall, our work provides compelling
evidence for investing in large-scale imitation learning of human
demonstrations.

##  2 Related work

Embodied Demonstrations from Humans. Prior expert demonstration datasets for
embodied tasks combining vision and action (and optionally language) can be
broadly categorized into either consisting of shortest-path trajectories from
a planner with privileged information [7, 8, 29, 5], or consisting of human-
provided trajectories [24, 25, 23]. While some works in the former collect
natural language data from humans [7, 5], we contend that collecting
navigation data from humans is equally crucial. Datasets with human-provided
navigation trajectories are typically small. TEACh [23], CVDN [24] and WAY
[25] have ${<}10k$ episodes, while the EmbodiedQA [8] dataset has ${\sim}700$
human-provided episodes – all prohibitively small for training proficient
agents. A key contribution of our work is a scalable web-based infrastructure
for collecting human navigation and interaction demonstrations, that is easily
extensible to _any_ task situated in the Habitat [19] simulator, including
language-based tasks. We have collected ${\sim}13$x more demonstrations (in
total $92k$) compared to prior publicly available datasets. In a similar vein,
Abramson  _et al_. [30] study large-scale imitation learning on ${\sim}600k$
human demonstrations, but their dataset is not publicly available and the
environments used lack visual realism compared to Matterport3D [31].

Exploration. Learning how to explore an environment to gather sufficient
information for use in downstream tasks has a rich history [32]. Curiosity-
based approaches typically use reinforcement learning to maximize intrinsic
rewards that capture the surprise or state prediction error of the agent [33,
34, 35]. State visitation count rewards are also popular for learning
exploration [36, 37]. We refer the reader to Ramakrishnan  _et al_. [38] for a
review of exploration objectives for embodied agents. For improving
exploration in ObjectNav specifically, SemExp [17] made use of a modular
policy for semantic mapping and path planning, Ye  _et al_. [15] used time-
decaying state visitation count reward, and Maksymets  _et al_. [16] used area
coverage reward.

Most relatedly, Chen  _et al_. [39] used ${\sim}700$ human navigation
trajectories from the EmbodiedQA dataset [8] (ignoring the questions) to learn
task-independent exploration using imitation learning. We likewise train
agents via imitation learning on human demonstrations, but rather than
encouraging task-agnostic exploration, we consider human demonstrations to be
a rich _task-specific_ mix of exploration and efficient navigation, that
simple architectures without explicit mapping and planning modules can be
trained on.

##  3 Habitat-WebGL Infrastructure

Figure 2: Screenshot of our Amazon Mechanical Turk interface for collecting
ObjectNav demonstrations. Users are provided the agent’s first-person view of
the environment and an instruction such as “Find and go to chair”. They can
make the agent look around and move in the environment via keyboard controls,
and can submit the task upon successful navigation by clicking ‘Submit’.

To be able to train agents via imitation learning on human demonstrations, we
first need a reliable pipeline to collect human demonstrations at scale. To
this end, we develop a web-based setup to connect the Habitat simulator[19,
20] to AMT users, building on the work of Newman  _et al_. [40].

Interface. Fig. 2 shows a screenshot of the interface an AMT user interacts
with to complete a data collection task. This web application renders assets
from Habitat-Sim running on the user’s browser via WebGL. All data collection
in this work was done in Matterport3D [31] scans, but any Habitat-compatible
asset may be used in future. Users can see the agent’s first-person RGB view,
and can move around and grab / release objects using keyboard controls. On the
task page, users are provided an instruction and details about keyboard
controls to complete the task. For ObjectNav, we provide an instruction of the
form _‘Find and go to the <goal_object_category>’_. For tasks requiring
interaction with objects (_e.g_. Pick&Place), we highlight the object under
the user’s gaze by drawing a $3$D bounding box around it (pointed to by a
crosshair as in video games). In our initial pilots, we found this to improve
user experience when grabbing objects instead of users having to guess when
objects are available to be picked up. When an object is successfully grabbed,
it disappears from the first-person view and immediately appears in the
‘inventory’ area on the task interface. When a grabbed object is released, it
is dropped at the center of the user’s screen where the crosshair would be
pointing to. If the crosshair points to a distance, the object is dropped on
the floor from a height at a distance of $1m$ from the agent’s location. Upon
completion, users submit the task by clicking ‘Submit’. At this point, the
sequence of keyboard actions, agent, and object states are recorded in our
backend server.

Habitat simulator and PsiTurk. Our Habitat-WebGL application is developed in
Javascript, and allows us to access all C++ simulator APIs through Javascript
bindings. This lets us use the full set of simulation features available in
Habitat. To simulate physics, we use the physics APIs from Habitat $2.0$ [20],
including rigid body dynamics support (C++ APIs exposed as Javascript
bindings). Our interface executes actions entered by users every $50$ms
(rendering $20$ frames per second) and then steps physics for $50$ms in the
simulator. All of our tasks on AMT are served using PsiTurk and an NGINX
reverse proxy, and all data stored in a MySQL database. We use PsiTurk to
manage the tasks as it provides us with useful helper functions to log task-
related metadata, as well as launch and approve tasks.

See Section A.6 for details on how we validate human-submitted AMT tasks and
ensure data quality.

##  4 Tasks and Datasets

|  | ObjectNav |  | Pick-And-Place
---|---|---|---|---
|  | Human | Shortest Path |  | Human | Shortest Path
1) | Total Episodes | $80,217$ | $114,165$ |  | $11,955$ | $25,747$
2) | Success | $88.9\%$ | $100.0\%$ |  | $86.3\%$ | $100.0\%$
3) | SPL | $39.9\%$ | $94.9\%$ |  | $21.2\%$ | $90.9\%$
4) | Occupancy coverage | $17.9\%$ | $4.6\%$ |  | $26.5\%$ | $9.2\%$
5) | Sight coverage | $67.7\%$ | $33.2\%$ |  | $70.3\%$ | $42.5\%$
|  |  | (a) |  |  |

(b)

(c)

Figure 3: (a) Dataset statistics for human demonstrations  _vs_. shortest
paths for ObjectNav and Pick&Place. Coverage metrics are computed on subset of
$1000$ episodes. (b) Comparison of episode lengths and action histograms for
human demonstrations  _vs_. shortest paths. Human demonstrations are longer
and have a more uniform action distribution than shortest paths.

Using our web infrastructure, we collect demonstration datasets for two
embodied tasks – ObjectNav [1, 2] and Pick&Place, an instantiation of object
rearrangement [3].

###  4.1 ObjectGoal Navigation

In the ObjectGoal Navigation (ObjectNav) task, an agent is tasked with
navigating to an instance of a specified object category (_e.g_. ‘chair’) in
an unseen environment. The agent does not have access to a map of the
environment and must navigate using an RGBD camera and a GPS+Compass sensor
which provides location and orientation information relative to the start of
the episode. The agent also receives the goal object category ID as input. The
full action space is discrete and consists of move_forward ($0.25m$),
turn_left ($30^{\circ}$), turn_right ($30^{\circ}$), look_up ($30^{\circ}$),
look_down ($30^{\circ}$), and stop actions. For the episode to be considered
successful, the agent must stop within $1m$ Euclidean distance of the goal
object within a maximum of $500$ steps and be able to turn to view the object
from that end position [41].

Human Demonstrations (ObjectNav-HD). We collect $70k$ demonstrations on the
$56$ training scenes from Matterport3D [31] following the standard splits
defined in [31, 2]. For each scene, we collect ${\sim}59$ demonstration
episodes for each unique goal object category with a randomly set start
location of the human demonstrator for each episode. This amounts to an
average of ${\sim}1250$ demonstrations per scene. Additionally, we collect
$10k$ demonstrations on $25$ training scenes from Gibson [42]. For each Gibson
scene, we collect ${\sim}66$ demonstration episodes for each unique goal
object category. This amounts to ${\sim}396$ demonstrations per scene. Similar
to when training artificial agents, humans can view first-person RGB on the
task interface, but unlike artificial agents, humans do not get access to
Depth and GPS+Compass. We assume humans are sufficiently proficient at
inferring depth and odometry from vision, to the extent required to accomplish
the goal. In total, we collect $80k$ ObjectNav demonstrations amounting to
${\sim}19.5M$ steps of experience, each episode averaging $243$ steps.

Shortest Path Demonstrations. To compare against prior embodied datasets of
shortest paths [7, 8, 29, 5] and to demonstrate the unique advantage of human
demonstrations, we also generate a dataset of shortest paths. The analysis in
this section was performed on a subset of $35$k demonstrations of ObjectNav-HD
(collected in first phase). These demonstrations are generated by greedily
fitting actions to follow the geodesic shortest path to the nearest navigable
goal object viewpoint. Since shortest paths are (by design) shorter than human
demonstrations (average $67$  _vs_. $243$ steps per demonstration), we
compensate by generating a larger number of shortest paths to roughly match
the steps with $35k$ human demonstrations ($7.6M$ steps from $114k$ shortest
paths  _vs_. $8.4M$ steps from $35k$ human demonstrations).

Analysis. Table 3a reports statistics of our human and shortest path
demonstration datasets. Recall that an episode is considered a failure if the
target object is not found within $500$ navigation steps. Under this
definition, humans fail on $11.1\%$ training set episodes; they fail on $0\%$
episodes if we relax the step-limit. Surprisingly, SPL for humans is $39.9\%$
for training split episodes, significantly lower than $94.9\%$ for shortest
paths underscoring the difficulty in searching for objects in in unseen
environments.

We additionally report two metrics to demonstrate that the ObjectNav task
requires significant exploration. Occupancy Coverage (OC) measures percentage
of total area covered by the agent when navigating. To compute OC, we first
divide the map into voxel grids of $2.5m\times 2.5m\times 2.5m$ and increment
a counter for each visited voxel. Sight Coverage (SC) measures the percentage
of total navigable area visible to the agent in its field of view (FOV) during
an episode. To compute SC, we project a mask on the top-down map of the
environment using the agent’s FOV, that is iteratively updated at every step
to update the area seen by the agent. OC and SC metrics for human
demonstrations show that humans traverse $3\text{-}4$x and observe $2$x the
area of the environment when performing this task compared to shortest paths.

Fig. 3b,c show episode length and action histograms for human and shortest
path demonstrations. Human demonstrations are longer (average ${\sim}243$
_vs_. ${\sim}67$ steps per demonstration) and have a slightly more uniform
action distribution.

###  4.2 Object Rearrangement – Pick&Place

In the pick-and-place task (Pick&Place), an agent must follow an instruction
of the form _‘Place the <object> on the <receptacle>’_, without being told the
location of the <object> or <receptacle> in a new environment. The agent must
explore and navigate to the object, pick it up, explore and navigate to the
receptacle, and place the previously picked-up object on it. Similar to
ObjectNav, agents are not equipped with a map of the environment, and only
have access to an RGBD camera and a GPS+compass sensor. At a high level,
Pick&Place can be thought of as a natural extension of ObjectNav, performing
it twice in the same episode – once to find the specified object and again to
find the specified receptacle – delimited by grab and release actions. For
object interaction, we use the ‘magic pointer’ abstraction defined in [3]. If
the agent is not holding any object, the grab/release action will pick the
object pointed to by its crosshair (at the center of its viewpoint) if within
$1.5m$ of the object. If the agent is already holding an object, the
grab/release action will drop the object at the crosshair location. If there
is no drop-off point within $1.5m$ in the direction of the crosshair, the
object will be dropped on the floor $1m$ in front of the agent. The full
action space is discrete and consists of move_forward ($0.15m$), move_backward
($0.15m$), turn_left ($5^{\circ}$), turn_right ($5^{\circ}$), look_up
($5^{\circ}$), look_down ($5^{\circ}$), grab_release, no_op (step physics
$50ms$), and stop. For the episode to be considered successful, the agent must
place the object on top of the receptacle –  _i.e_. the object center should
be at a height greater than the receptacle center, and within $0.7m$ of the
receptacle object center – within $1500$ steps. We picked this $0.7m$
threshold distance between the object and receptacle based on pilots on AMT.
$0.7m$ was sufficiently strict for avoiding false positives in the collected
demonstrations where users are able to submit the task without necessarily
placing the object on top of the receptacle.

Human Demonstrations (Pick&Place-HD). We collect human demonstrations for
Pick&Place on $9$ scenes from Matterport3D [31]. In each episode, objects and
receptacles are instantiated by randomly sampling from $457$ possible object-
receptacle pairs. We initialize the object and receptacle at randomly sampled
locations in the environment, and collect $3$ demonstrations for each object-
receptacle pair. The agent, object, and receptacle locations are randomized
across all episodes (including the $3$ we collect for each object-receptacle
pair). In total, we have $457\times 3$ unique object-receptacle-agent position
initializations per scene, amounting to $457\times 3\times 9={\sim}12k$
demonstrations, which is ${\sim}11.5$M steps in experience, each episode
averaging $932$ steps.

Shortest Path Demonstrations. Similar to ObjectNav, we generate shortest path
demonstrations for Pick&Place. These demonstrations are generated by first
using the geodesic shortest-path follower to the object, then using a
heuristic action planner to face and pick up the object, then following the
geodesic shortest-path to the receptacle, and again using a heuristic action
planner to drop the object on the receptacle. We generated $25.7$k shortest
path demonstrations for Pick&Place, each averaging $342$ steps, amounting to a
total of ${\sim}8.8$ million steps of experience.

Analysis. Table 3a reports statistics for human and shortest path
demonstrations. Similar to ObjectNav, humans have significantly lower SPL, and
$2$x higher occupancy and sight coverage compared to shortest paths,
suggesting the need for exploration. Comparing episode lengths and action
histograms (see appendix (Sec. A.1.1) for figure), human demonstrations are
longer and make use of all $9$ actions. Interestingly, humans often use the
move_backward action to backtrack, which the shortest path agents do not use
(by design), instead of turning $180^{\circ}$ and moving forward. This
behavior does not appear in ObjectNav shortest path demonstrations because
there is just one target object, and so the geodesic shortest path would never
involve backtracking or making $180^{\circ}$ turns.

##  5 Imitation Learning from Demonstrations

(a) ObjectNav architecture

(b) Pick&Place architecture

| Method | Success $(\mathbf{\uparrow})$ | SPL $(\mathbf{\uparrow})$
---|---|---|---
6) | RL (ExploreTillSeen) [16] | $20.0\%$ | $6.5\%$
7) | RL (ExploreTillSeen + THDA) [16] | $28.4\%$ | $\mathbf{11.0}\%$
8) | RL (Red Rabbit) [15] | $34.6\%$ | $7.9\%$
9) | RL (EmbCLIP) [43] | $21.6\%$ | $8.7\%$
10) | IL w/ Shortest Paths | $4.4\%$ | $2.2\%$
11) | IL w/ $35k$ Human Demos (similar $\\#$steps as row 4) | $31.6\%$ | $8.5\%$
12) | IL w/ $50k$ Human Demos | $32.4\%$ | $9.1\%$
13) | IL w/ $50k$ Human Demos (includes $10k$ THDA [16]) | $33.2\%$ | $9.5\%$
14) | IL w/ $70k$ Human Demos | $\mathbf{35.4}\%$ | $10.2\%$
15) | IL w/ $70k$ Human Demos (includes $10k$ Gibson [42]) | $33.9\%$ | $9.7\%$
16) | IL w/ $80k$ Human Demos | $33.8\%$ | $9.9\%$
17) | Humans | $93.7\%$ | $42.5\%$
| (c) ObjectNav results on MP3D-val |  |

Figure 4: Our policy architectures for a) ObjectNav and b) Pick&Place. Both
are simple CNN+RNN networks that embed and concatenate all sensory inputs,
which are then fed into a GRU to predict actions. c) ObjectNav results on the
MP3D val split [2, 31].

We use behavior cloning to learn a policy from demonstrations. Let
$\pi_{\theta}(a_{t}\,|\,o_{t})$ denote a policy parametrized by $\theta$ that
maps observations $o_{t}$ to a distribution over actions $a_{t}$. Let $\tau$
denote a trajectory consisting of state, observation, human action tuples:
$\tau=\big{(}s_{0},o_{0},a_{0},\ldots,s_{T},o_{T},a_{T}\big{)}$ and
$\mathcal{T}=\big{\\{}\tau^{(i)}\big{\\}}_{i=1}^{N}$ denote a dataset of human
demonstrations. The learning problem can be summarized as:

$\theta^{*}=\text{arg\,min}_{\theta}\sum_{i=1}^{N}\sum_{(o_{t},a_{t})\in\tau^{(i)}}-\log\Big{(}\pi_{\theta}(a_{t}|o_{t})\Big{)}$
(1)

Inflection weighting introduced in Wijmans  _et al_. [9], adjusts the loss
function to upweight timesteps where actions change (_i.e_. $a_{t-1}\neq
a_{t}$). Specifically, the inflection weighting loss coefficient is computed
as total no. of actions in the dataset divided by the total no. of inflection
points, and this coefficient is multiplied with the loss at each inflection
timestep where $a_{t-1}\neq a_{t}$. This approach was found to be useful for
tasks like navigation with long sequences of the same actions, _e.g_. several
‘forward’ actions when navigating corridors [9]. We use inflection weighting
in all our experiments and found it to help over vanilla behavior cloning.

Our base policy is a simple CNN+RNN architecture. We first embed all sensory
inputs using feed-forward modules. For RGB, we use a randomly initialized
ResNet18 [44]. For depth, we use a ResNet50 that was pretrained on PointGoal
navigation using DD-PPO [13]. Then these RGB and depth features (and
optionally other task-specific features) are concatenated and fed into a GRU
[45] to predict a distribution over actions $a_{t+1}$. Task-specific
architectural choices over this base policy are described in the next
sections.

###  5.1 ObjectNav

Fig. 4(a) shows our ObjectNav architecture. Similar to Anand  _et al_. [46],
we feed in RGBD inputs of size $640\times 480$ passed through a 2x2-AvgPool
layer to reduce the resolution (performing low-pass filtering + downsampling).
The agent also has a GPS+Compass sensor, which provides location and
orientation relative to start of the episode. GPS+Compass inputs are pass
through fully-connected layers to embed them to $32$-d vectors. In addition to
RGBD and GPS+Compass, following Ye  _et al_. [15], we use two additional
semantic features – semantic segmentation (SemSeg) of the input RGB and a
‘Semantic Goal Exists’ (SGE) scalar which is the fraction of the visual input
occupied by the goal category. These semantic features are computed using a
pretrained and frozen RedNet [47] that was pretrained on SUN RGB-D [48] and
finetuned on $100k$ randomly sampled front-facing views rendered in the
Habitat simulator. Finally, we also feed in the object goal category embedded
into a $32$-d vector. All of these input features are concatenated to form an
observation embedding, and fed into a $2$-layer, $512$-d GRU at every
timestep. We train this policy for ${\sim}400$M steps ($={\sim}21$ epochs on
${\sim}70k$ demonstration episodes). We evaluate checkpoints at every
${\sim}15$M steps for the last $50$M steps of training, and report metrics for
checkpoints with the highest success on the validation split.

###  5.2 Pick&Place

Fig. 4(b) shows our Pick&Place architecture. We feed in RGBD inputs of size
$256\times 256$. In addition to RGBD observations, the policy gets as input
language instructions of the form _‘Place the <object> on the <receptacle>’_
encoded using a single-layer LSTM [49]. RGBD and instruction features are
concatenated to form an observation embedding, which is fed into a $2$-layer,
$512$-d GRU at every timestep. We train this policy for ${\sim}90$M steps
($={\sim}10$ epochs on ${\sim}9.5k$ demonstration episodes). We evaluate
checkpoints at every ${\sim}10$M steps during training, and report metrics for
checkpoints with the highest success on the validation split.

##  6 Experiments & Results

###  6.1 ObjectNav

Table 4c reports results on the MP3D val split for several baselines. First,
we compare our approach with two state-of-the-art RL approaches from prior
work. Maksymets  _et al_. [16] (row $1$) train their policy using a reward
structure that breaks ObjectNav into two subtasks – exploration and direct
navigation to goal object once it is spotted. This agent gets a positive
reward for maximizing area coverage until it sees the goal object. It then
receives a navigation reward to minimize distance-to-object. This policy
achieves $20.0\%$ success and $6.5\%$ SPL (row $1$). [16] then combine this
reward structure with Treasure Hunt Data Augmentation (THDA) – inserting
arbitrary $3$D target objects in the scene to augment the set of training
episodes. With THDA, this achieves $28.4\%$ success and $11.0\%$ SPL (row
$2$), $7.0\%$ worse and $0.8\%$ better respectively than behavior cloning on
$70k$ human demonstrations (row $9$). Ye  _et al_. [15] (row $3$) train their
policy with a combination of exploration and distance-based navigation
rewards, and their representations with several auxiliary tasks (_e.g_.
inverse dynamics and predicting map coverage). This achieves $34.6\%$ success
and $7.9\%$ SPL (row $3$), which is $0.8\%$ worse on success and $2.3\%$ worse
on SPL than our approach (row $9$). Khandelwal  _et al_. [43] (row $4$) train
a policy using CLIP [50] as a visual backbone with simple distance-based
navigation rewards. This achieves $21.6\%$ success and $8.7\%$ SPL (row $4$),
which is $13.8\%$ worse on success and $1.5\%$ worse on SPL than our approach
(row $9$). IL on a dataset of shortest paths achieves $4.4\%$ success and
$2.2\%$ SPL (row $5$), significantly worse than training on $35k$ human
demonstrations ($31.6\%$ success, $8.5\%$ SPL). Recall that comparison of
shortest path demonstrations was done with a subset of $35k$ ObjectNav-HD
demonstrations that were collected in the first phase of the project. Next, we
also collected $10k$ human demonstrations on ObjectNav episodes generated in
THDA fashion – _i.e_. asking humans to find randomly inserted objects. Notice
that this involves pure exhaustive search, since there are no semantic priors
that humans can leverage in this setting. An IL agent trained on $10k$ THDA
demonstrations combined with the original $40k$ demonstrations achieves
$33.2\%$ success and $9.5\%$ SPL (row $8$) which is $0.8\%$ better on success
and $0.4\%$ better on SPL than $50k$ non-THDA demonstrations (row $7$), _i.e_.
adding these THDA demonstrations with exhaustive search behavior helps. We
also collected $10k$ demonstrations on Gibson[42] ObjectNav episodes to
compare effect of different scene datasets. An agent trained on $10k$ Gibson
demonstrations combined with $60k$ MP3D demonstrations achieves $33.9\%$
success and $9.7\%$ SPL (row $11$), which is $1.5\%$ worse on success and
$0.5\%$ worse on SPL compared to when we use MP3D-only demonstrations (row
$9$).

Finally, we also benchmark human performance on the MP3D val split – $93.7\%$
success, $42.5\%$ SPL (row $12$).

Method | Success $(\mathbf{\uparrow})$ | SPL $(\mathbf{\uparrow})$
---|---|---
18) IL wo/ Vision | $0.0\%$ | $0.0\%$
19) IL wo/ Semantic Input | $22.7\%$ | $6.1\%$
20) IL w/ RGBD + Semantic Input | $31.6\%$ | $8.5\%$

Table 1: ObjectNav ablation results on the MP3D val split [2, 31].

ObjectNav Sensor Ablations. Table 1 reports results on the MP3D val split for
various ablations of our approach trained on $35k$ human demonstrations.
First, without any visual input (row $1$),  _i.e_. no RGBD and semantic
inputs, the agent fails to learn anything ($0\%$ success, $0\%$ SPL). Second,
without SemSeg and SGE features (and keeping only RGB and Depth features) to
the policy, performance drops by $8.9\%$ success and $2.4\%$ SPL (row $2$
_vs_. $3$).

Habitat ObjectNav Challenge Results. Table 2 compares our results with prior
approaches from the $2020$ and $2021$ Habitat Challenge leaderboards. Our
approach (IL w/ $70k$ demonstrations) achieves $27.8\%$ success and $9.9\%$
SPL (row $8$), outperforming prior RL-trained counterparts – $3.3\%$ better
success, $3.5\%$ better SPL than Red Rabbit (6-Act Base) [15] (row 5), and
$6.7\%$ better success, $1.1\%$ better SPL than ExploreTillSeen + THDA [16]
(row 7).

| Team / Method | Success $(\mathbf{\uparrow})$ | SPL $(\mathbf{\uparrow})$
---|---|---|---
21) | DD-PPO baseline [13, 15] | $6.2\%$ | $2.1\%$
22) | Active Exploration (Pre-explore) | $8.9\%$ | $4.1\%$
23) | SRCB-robot-sudoer | $14.4\%$ | $7.5\%$
24) | SemExp [51] | $17.9\%$ | $7.1\%$
25) | Red Rabbit (6-Act Base) [15] | $24.5\%$ | $6.4\%$
26) | Red Rabbit (6-Act Tether) [15] | $21.1\%$ | $8.1\%$
27) | ExploreTillSeen + THDA [16] | $21.1\%$ | $8.8\%$
28) | IL w/ $70k$ Human Demos | $\mathbf{27.8\%}$ | $\mathbf{9.9\%}$

Table 2: Results on Habitat ObjectNav Challenge test-std [52].

Performance  _vs_. Dataset size. To investigate scaling behavior, we plot val
success against the size of the human demonstrations dataset in Fig. 1b. We
created splits of the human demonstrations’ dataset of increasing sizes, from
$4k$ to $70k$, and trained models with the same set of hyperparameters on each
split. All hyperparameters were picked early in the course of the data
collection (on the $4k$ and $12k$ subsplits) and fixed for later experiments.
So val performance in the small-data regime may be an optimistic estimate and
in the large data regime a pessimistic estimate. True scaling behavior may be
even stronger. Increasing dataset size consistently improves performance and
has not yet saturated, suggesting that simply collecting more demonstrations
is likely to lead to further gains.

(a)

(b)

Figure 5: Comparing RL and IL on (a) val success  _vs_. no. of training steps,
and (b) val success  _vs_. no. of unique training steps. This distinguishes
between an IL agent that learns from a static dataset  _vs_. an RL agent that
gathers unique trajectories on-the-fly.

Sample Efficiency. Fig. 5 plots val success against no. of training steps of
experience (in millions) in Fig. 5(a) and against _unique_ steps of experience
in Fig. 5(b). Recall that IL involves ${\sim}21$ epochs on a static dataset of
${\sim}70k$ demos, while RL (from [15]) gathers unique agent-driven
trajectories on-the-fly. Fig. 5(a) shows that IL behaves like supervised
learning (as expected) with improvements coming from long training schedules;
unfortunately, this means that wall-clock training times are not lower than
RL. Fig. 5(b) shows that IL requires $7$x fewer unique steps of experience to
outperform success and is thus much more sample-efficient.

Zero-shot results on Gibson [42] are in Section A.2.

###  6.2 Pick&Place

Figure 6: Visualizations of different learnt agent behaviors. Best viewed in
the video at ram81.github.io/projects/habitat-web.

|  |  | val |  | test
---|---|---|---|---|---
|  | Method | Success $\%$ $(\mathbf{\uparrow})$ | SPL $\%$ $(\mathbf{\uparrow})$ |  | Success $\%$ $(\mathbf{\uparrow})$ | SPL $\%$ $(\mathbf{\uparrow})$
New Inits. $\begin{dcases}\\\ \\\ \\\ \end{dcases}$ |  |  |  |  |  |  |
29) | IL w/ Shortest Paths | $1.9$ | $1.8$ |  | $1.7$ | $1.6$
30) | IL w/ Human Demos |  $17.6$ $\pm 0.8$ |  $9.7$ $\pm 0.3$ |  | $17.5$ | $9.8$
| 31) | Humans | $87.2$ | $21.8$ |  | $89.1$ | $21.9$
New Instr. $\begin{dcases}\\\ \\\ \\\ \end{dcases}$ |  |  |  |  |  |  |
32) | IL w/ Shortest Paths | $1.3$ | $1.2$ |  | $1.1$ | $1.0$
33) | IL w/ Human Demos |  $15.9$ $\pm 0.2$ |  $8.4$ $\pm 0.4$ |  | $15.1$ | $8.3$
| 34) | Humans | $85.0$ | $21.0$ |  | $86.1$ | $20.5$
New Envs $\begin{dcases}\\\ \\\ \\\ \end{dcases}$ |  |  |  |  |  |  |
35) | IL w/ Shortest Paths | $-$ | $-$ |  | $0.2$ | $0.3$
36) | IL w/ Human Demos | $-$ | $-$ |  | $8.3$ | $4.1$
| 37) | Humans | $-$ | $-$ |  | $94.9$ | $20.5$

Table 3: Pick-and-place results on splits constructed with unseen
initializations in seen environments (1-3), with unseen instructions (4-6),
and with unseen environments (7-9).

| Method | OC (%) | SC (%) | GRTS (%) | Peeks (%) | PT (%) | Beeline (%) | ES (%)
---|---|---|---|---|---|---|---|---
38) | IL w/ shortest paths |  $4.2$$\pm 1.1$ |  $31.2$$\pm 3.2$ |  $20.5$$\pm 4.3$ |  $3.0$$\pm 1.9$ |  $0.0$$\pm 0.0$ |  $0.0$$\pm 5.2$ |  $10.1$$\pm 3.5$
39) | IL w/ human demos |  $21.4$$\pm 1.8$ |  $72.1$$\pm 3.5$ |  $22.4$$\pm 4.1$ |  $19.6$$\pm 4.4$ |  $4.3$$\pm 2.3$ |  $10.3$$\pm 5.1$ |  $55.3$$\pm 5.6$
40) | RL [15] |  $14.6$$\pm 1.6$ |  $66.6$$\pm 5.1$ |  $27.7$$\pm 8.5$ |  $9.7$$\pm 5.5$ |  $0.0$$\pm 0.0$ |  $0.1$$\pm 2.2$ |  $49.0$$\pm 7.0$
41) | Humans |  $15.4$$\pm 1.6$ |  $70.3$$\pm 3.4$ | - |  $13.8$$\pm 3.9$ |  $5.1$$\pm 2.4$ |  $23.6$$\pm 4.8$ |  $52.1$$\pm 5.6$

Table 4: Quantifying semantic exploration behaviors for IL agents trained on
shortest paths (row $1$) and human demonstrations (row $2$), the Red Rabbit RL
agent [15] (row $3$), and humans (row $4$).

Results. We report results in Table 3 across three evaluation splits. 1) New
Initializations: new locations of objects and receptacles. This tests
generalization to unseen locations in seen environments. 2) New Instructions:
compositionally novel object-receptacle combinations of objects and
receptacles individually seen during training. 3) New Environments:
generalization to $2$ scenes held out from training. Similar to ObjectNav and
as described in Section 4, we also report results with shortest paths. Again,
these paths are significantly shorter (average $342$  _vs_. $932$ steps per
demonstration) and hence, we generate a larger dataset of $25.7k$ episodes
roughly matching the cumulative steps of experience with human demonstrations
($8.8M$ shortest path steps  _vs_. $11.5M$ human steps). Training on $9.5k$
human demonstrations achieves $17.5\%$ success, $9.8\%$ SPL on new object-
receptacle initializations (row $2$). Across splits, training on shortest
paths hurts success by $8$-$16\%$. Going to new object-receptacle pairs,
success drops by $2.4\%$ (row $5$  _vs_. $2$), and then going to new
environments further hurts success by $6.8\%$ (row $8$  _vs_. $5$). We also
trained an RL policy with the exploration and distance-based rewards from
[16], but it failed to get beyond $0\%$ success on new object-receptacle
intializations. See the appendix (Sec. A.1.2) for training details.

Performance  _vs_. Dataset size. Similar to ObjectNav, we trained policies on
$2.5k$ to $9.5k$ subsets of our Pick&Place data, and found that performance
continues to improve with more data. Figure in appendix (Sec. A.1.3).

##  7 Characterizing Learned Behaviors

To characterize the behaviors learnt by our best IL agents, we first sample
$300$ validation ObjectNav episodes for each method and manually categorize
the behavior observed. A subset of observed behaviors are visualized in Fig.
6. Our agents demonstrate sophisticated object-search behaviors  _e.g_.
peeking into rooms to maximize sight coverage (SC), instead of occupancy
coverage (OC), checking corners of rooms for small objects, beelining to goal
object once seen, exhaustive search (ES), turning in place to get a panoramic
view (PT), and looping back to recheck some areas. Amusingly, unlike shortest
path / RL agents, these IL agents also stand idle and ‘look around’ _i.e_.
turn in place, like humans. Table 4 quantifies these behaviors. See appendix
(Sec. A.4) for details on how these were computed. Agents trained with IL on
human demonstrations have higher coverage (both occupancy and sight), peeking
behavior, panoramic turns, beelines, and exhaustive search than RL. RL-trained
agents achieve higher average Goal Room Time Spent (GRTS) –  _i.e_. time spent
in the room containing the target object – but also have significantly higher
variance in GRTS across scenes compared to IL agents. See appendix (Sec. A.4)
for a per-scene breakdown of GRTS as well as histograms of time spent in each
room (instead of just target room) when searching for a target object. We also
discuss limitations of our approach in the appendix Sec. A.7.

##  8 Conclusion

We developed the infrastructure to collect human demonstrations at scale and
using this, trained imitation learning (IL) agents on $92k+$ human
demonstrations for ObjectNav and Pick&Place. On ObjectNav, we found that IL
using $70k$ human demonstrations outperforms RL using $240k$ agent-gathered
trajectories, and on Pick&Place, IL agents get to ${\sim}18\%$ success while
RL fails to get beyond $0\%$. Qualitatively, we found that IL agents pick up
on sophisticated object-search behavior implicitly captured in human
demonstrations, much more prominently than RL agents. Overall, we believe our
work makes a compelling case for investing in large-scale imitation learning
of human demonstrations.

Acknowledgements. We thank Devi Parikh for help with brainstorming and
direction, Joel Ye for answering questions about his Red Rabbit [15],
Oleksandr Maksymets for the THDA [16] episode generation pipeline, and Erik
Wijmans for help with debugging DDP. The Georgia Tech effort was supported in
part by NSF, ONR YIP, and ARO PECASE. The views and conclusions contained
herein are those of the authors and should not be interpreted as necessarily
representing the official policies or endorsements, either expressed or
implied, of the U.S. Government, or any sponsor.

## References

  * [1] M. Savva, A. X. Chang, A. Dosovitskiy, T. Funkhouser, and V. Koltun, “MINOS: Multimodal indoor simulator for navigation in complex environments,” arXiv preprint arXiv:1712.03931, 2017.
  * [2] P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir, “On evaluation of embodied navigation agents,” arXiv preprint arXiv:1807.06757, 2018.
  * [3] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, M. Savva, and H. Su, “Rearrangement: A Challenge for Embodied AI,” arXiv preprint arXiv:2011.01975, 2020.
  * [4] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi, “Visual room rearrangement,” in CVPR, 2021.
  * [5] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in CVPR, 2018.
  * [6] J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environments,” in ECCV, 2020.
  * [7] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks,” in CVPR, 2020.
  * [8] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra, “Embodied Question Answering,” in CVPR, 2018.
  * [9] E. Wijmans, S. Datta, O. Maksymets, A. Das, G. Gkioxari, S. Lee, I. Essa, D. Parikh, and D. Batra, “Embodied Question Answering in Photorealistic Environments with Point Cloud Perception,” in CVPR, 2019.
  * [10] A. Das, F. Carnevale, H. Merzic, L. Rimell, R. Schneider, J. Abramson, A. Hung, A. Ahuja, S. Clark, G. Wayne, and F. Hill, “Probing emergent semantics in predictive agents via question answering,” in ICML, 2020.
  * [11] A. Das, Building agents that can see, talk, and act.  PhD thesis, Georgia Institute of Technology, 2020.
  * [12] L. Yu, X. Chen, G. Gkioxari, M. Bansal, T. L. Berg, and D. Batra, “Multi-target embodied question answering,” in CVPR, 2019.
  * [13] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames,” in ICLR, 2020.
  * [14] J. Ye, D. Batra, E. Wijmans, and A. Das, “Auxiliary tasks speed up learning pointgoal navigation,” in CoRL, 2020.
  * [15] J. Ye, D. Batra, A. Das, and E. Wijmans, “Auxiliary Tasks and Exploration Enable ObjectNav,” in ICCV, 2021.
  * [16] O. Maksymets, V. Cartillier, A. Gokaslan, E. Wijmans, W. Galuba, S. Lee, and D. Batra, “THDA: Treasure Hunt Data Augmentation for Semantic Navigation,” in ICCV, 2021.
  * [17] D. S. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov, “Object goal navigation using goal-oriented semantic exploration,” in NeurIPS, 2020\.
  * [18] S. Schaal, “Learning from demonstration,” in NIPS, 1996.
  * [19] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, et al., “Habitat: A platform for embodied AI research,” in ICCV, 2019.
  * [20] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra, “Habitat 2.0: Training home assistants to rearrange their habitat,” in NeurIPS, 2021.
  * [21] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in ICRA, 2017.
  * [22] S. Wani, S. Patel, U. Jain, A. X. Chang, and M. Savva, “MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation,” in NeurIPS, 2020.
  * [23] A. Padmakumar, J. Thomason, A. Shrivastava, P. Lange, A. Narayan-Chen, S. Gella, R. Piramuthu, G. Tur, and D. Hakkani-Tur, “TEACh: Task-driven Embodied Agents that Chat,” arXiv preprint arXiv:2110.00534, 2021.
  * [24] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer, “Vision-and-dialog navigation,” in CoRL, 2020.
  * [25] M. Hahn, J. Krantz, D. Batra, D. Parikh, J. M. Rehg, S. Lee, and P. Anderson, “Where are you? localization from embodied dialog,” arXiv preprint arXiv:2011.08277, 2020.
  * [26] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín, “What matters in learning from offline human demonstrations for robot manipulation,” in CoRL, 2021.
  * [27] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, “Bridge data: Boosting generalization of robotic skills with cross-domain datasets,” arXiv preprint arXiv:2109.13396, 2021\.
  * [28] J. Krantz, A. Gokaslan, D. Batra, S. Lee, and O. Maksymets, “Waypoint models for instruction-guided navigation in continuous environments,” in ICCV, 2021.
  * [29] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi, “IQA: Visual Question Answering in Interactive Environments,” in CVPR, 2018.
  * [30] J. Abramson, A. Ahuja, I. Barr, A. Brussee, F. Carnevale, M. Cassin, R. Chhaparia, S. Clark, B. Damoc, A. Dudzik, et al., “Imitating interactive intelligence,” arXiv preprint arXiv:2012.05672, 2020.
  * [31] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang, “Matterport3D: Learning from RGB-D Data in Indoor Environments,” in 3DV, 2017.  MatterPort3D dataset license: http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf.
  * [32] J. Schmidhuber, “A possibility for implementing curiosity and boredom in model-building neural controllers,” in Proc. of the international conference on simulation of adaptive behavior: From animals to animats, 1991\.
  * [33] B. C. Stadie, S. Levine, and P. Abbeel, “Incentivizing exploration in reinforcement learning with deep predictive models,” arXiv preprint arXiv:1507.00814, 2015.
  * [34] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” in ICML, 2017.
  * [35] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros, “Large-scale study of curiosity-driven learning,” arXiv preprint arXiv:1808.04355, 2018.
  * [36] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying count-based exploration and intrinsic motivation,” in NeurIPS, 2016.
  * [37] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel, “# exploration: A study of count-based exploration for deep reinforcement learning,” in NeurIPS, 2017.
  * [38] S. K. Ramakrishnan, D. Jayaraman, and K. Grauman, “An exploration of embodied visual exploration,” arXiv preprint arXiv:2001.02192, 2020.
  * [39] T. Chen, S. Gupta, and A. Gupta, “Learning exploration policies for navigation,” in ICLR, 2019.
  * [40] B. Newman, K. Carlberg, and R. Desai, “Optimal assistance for object-rearrangement tasks in augmented reality,” arXiv preprint arXiv:2010.07358, 2020.
  * [41] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans, “Objectnav revisited: On evaluation of embodied agents navigating to objects,” arXiv preprint arXiv:2006.13171, 2020.
  * [42] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: Real-world perception for embodied agents,” in CVPR, 2018.  Gibson dataset license agreement available at storage.googleapis.com/gibson_material/Agreement GDS 06-04-18.pdf.
  * [43] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, ‘‘Simple but Effective: CLIP Embeddings for Embodied AI,’’ in CVPR, 2022.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, ‘‘Deep Residual Learning for Image Recognition,’’ in CVPR, 2016.
  * [45] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, ‘‘Learning phrase representations using rnn encoder-decoder for statistical machine translation,’’ in EMNLP, 2014.
  * [46] A. Anand, E. Belilovsky, K. Kastner, H. Larochelle, and A. Courville, ‘‘Blindfold baselines for embodied qa,’’ arXiv preprint arXiv:1811.05013, 2018.
  * [47] J. Jiang, L. Zheng, F. Luo, and Z. Zhang, ‘‘RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation,’’ arXiv preprint arXiv:1806.01054, 2018.
  * [48] S. Song, S. P. Lichtenberg, and J. Xiao, ‘‘Sun RGB-D: A RGB-D scene understanding benchmark suite,’’ in CVPR, 2015.
  * [49] S. Hochreiter and J. Schmidhuber, ‘‘Long Short-Term Memory,’’ Neural Computation, 1997.
  * [50] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., ‘‘Learning transferable visual models from natural language supervision,’’ in ICML, 2021.
  * [51] D. S. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov, ‘‘Object goal navigation using goal-oriented semantic exploration,’’ in NeurIPS, 2020\.
  * [52] H. Team, ‘‘Habitat challenge, 2020.’’ https://aihabitat.org/challenge/2020, 2020.
  * [53] S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman, ‘‘PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning,’’ in CVPR, 2022.
  * [54] A. Mousavian, A.toshev, M. Fiser, J. Kosecka, A. Wahid, and J. Davidson, ‘‘Visual representations for semantic target driven navigation,’’ in ICRA, 2019.
  * [55] D. S. Chaplot, S. Gupta, D. Gandhi, A. Gupta, and R. Salakhutdinov, ‘‘Learning to explore using active neural mapping,’’ in ICLR, 2020.

##  Appendix A Appendix

###  A.1 Pick&Place

Recall that in the pick-and-place task (Pick&Place), an agent must follow an
instruction of the form _‘Place the <object> on the <receptacle>’_, without
being told the location of the <object> or <receptacle> in a new environment.
The agent must explore and navigate to the object, pick it up, explore and
navigate to the receptacle, and place the previously picked-up object on it.
In this section, we go over statistics of the human demonstrations dataset,
how our Pick&Place imitation learning (IL) agents scale as a function of
training dataset size, and details of our reinforcement learning baseline for
Pick&Place.

####  A.1.1 Dataset Stats

Fig. 8 compares the episode length and action histograms for human and
shortest path demonstrations for Pick&Place. Human demonstrations are longer
(average $932$ vs $342$ steps per demonstration) and have a more uniform
action distribution compared to shortest paths. Human demonstrations also make
use of all $9$ actions whereas shortest path demonstrations use only $6$
actions. Notice, humans also tend to stand idle and do nothing ($50$ms of idle
time is translated to a NO_OP action). They likely use this time to strategize
their next set of actions to explore the environment, which is not the case in
shortest path demonstrations (by design).

Figure 7: Dataset size vs performance plot for Pick&Place.

####  A.1.2 RL Baseline

Similar to the imitation learning baseline, our base policy is a simple
CNN+RNN architecture. We first embed all sensory inputs using feed-forward
modules. For RGB, we use a randomly initialized ResNet18 [44]. For depth, we
use a ResNet50 that was pretrained on PointGoal navigation using DDPPO [13].
In addition to RGBD observations, the policy gets as input language
instructions of the form _‘Place the <object> on the <receptacle>’_ encoded
using a single-layer LSTM [49]. RGBD and instruction features are concatenated
to form an observation embedding, which is fed into a $2$-layer, $512$-d GRU
at every timestep. We train this policy for ${\sim}100$M steps on ${\sim}9.5k$
episodes.

Rewards. The agent receives a sparse success reward $r_{success}$, a slack
reward $r_{slack}$ to motivate faster goal-seeking, an exploration reward
$r_{explore}$, an object seen reward $r_{seen}$, a grab/release success reward
$r_{grab\\_release}$, and a drop penalty reward $r_{drop\\_penalty}$ to
penalize dropping the object far from the receptacle. For incentivizing
exploration, we use a visitation-based coverage reward from Ye  _et al_. [15].
We first divide the map into a voxel grid of $2.5m\times 2.5m\times 2.5m$
voxels and reward the agent for visiting each voxel. Similar to [15], we
smooth $r_{explore}$ by decaying it by number of steps the agent has spent in
the voxel (visit count $v$). To ensure that the agent prioritizes Pick&Place
(and not just exploration), we decay $r_{explore}$ based on episode timestep
$t$ with a decay constant of $d=0.995$. The agent is provided a reward for
exploration until it sees the object. Once it sees the object, it receives a
significant positive reward $r_{seen}$, and then the reward switches to a
path-efficiency based navigation reward. In addition, the agent also receives
a significant positive reward when it successfully grabs a object or releases
the object close to the receptacle.

$\displaystyle r_{\text{total}}$
$\displaystyle=r_{\text{success}}+r_{\text{slack}}+r_{\text{explore}}+r_{\text{grab\\_release}}$
(2a) $\displaystyle+r_{\text{drop\\_penalty}}+r_{\text{seen}}$ (2b)
$\displaystyle r_{\text{success}}$
$\displaystyle=5.5\quad\quad\,\,\,\text{{\scriptsize on success}}$ (2c)
$\displaystyle r_{\text{slack}}$
$\displaystyle=-10^{-4}\quad\text{{\scriptsize per step}}$ (2d) $\displaystyle
r_{\text{seen}}$ $\displaystyle=1.5\quad\text{{\scriptsize First time object
seen}}$ (2e) $\displaystyle r_{\text{drop\\_penalty}}$
$\displaystyle=-3.5\quad\text{{\scriptsize Object dropped > $2m$ away from
receptacle}}$ (2f) $\displaystyle r_{\text{grab\\_release}}$
$\displaystyle=2.0\quad\text{{\scriptsize Grab / release success}}$ (2g)
$\displaystyle r_{\text{explore}}$
$\displaystyle=0.25\times\frac{d^{t}}{v}\quad\text{{\scriptsize Until object
seen}}$ (2h)

Results. A policy trained with this reward for $100$M steps fails to get
beyond $0\%$ success on the Pick&Place task. The agent learns to pick up the
object at the start of training if it sees the object while navigating but it
fails to search for the receptacle and place the object on top of receptacle.
Overall, throughout training, the agent doesn’t solve the task successfully
even once demonstrating the difficulty of the task and inadequacy of the above
reward structure.

####  A.1.3 Performance  _vs_. Dataset Size

Fig. 7 plots Val success of our IL agent  _vs_. the size of the Pick&Place
human demonstrations dataset. We trained policies on $2.5k$ to $9.5k$ subsets
of the data. Performance continues to improve with more data and has not
saturated.

Figure 8: Comparison of episode lengths and action histograms for human
demonstrations  _vs_. shortest paths for Pick&Place. Human demonstrations are
longer and have a more uniform action distribution than shortest paths.

###  A.2 Zero-shot ObjectNav results on Gibson

To test generalization of the IL agents trained on human demonstrations, we
report zero-shot results by transferring our policy trained on $40k$ human
demonstrations to the Gibson dataset [42] val split in Table 5. To enable
zero-shot transfer of semantic features, we remap the common goal categories
(chair, couch, potted plant, bed, toilet, TV, dining-table) from Matterport3D
[31] to Gibson goal category IDs. Our IL agent achieves $49.4\%$ success and
$16.4\%$ SPL (row 7) with no finetuning on Gibson dataset. Comparing our zero-
shot results to approaches trained on Gibson, our IL agent is $33.6\%$ better
on success and $11.5\%$ better on SPL than an RL baseline that takes RGBD +
Semantics as input (row 3  _vs_. row 7). Next, we compare our approach with
SemExp [51] which builds explicit semantic maps and learns a goal-oriented
semantic exploration policy which learns semantic priors for efficient
navigation. Our approach is $4.9\%$ worse on success and $3.5\%$ worse on SPL
compared to SemExp [51] (row 6  _vs_. row 7). [53] uses a modular framework
for ObjectNav by using a potential function conditioned on a semantic map
which is used to decide where to look for unseen object in an environment. Our
approach is $24.1\%$ worse on success and $24.6\%$ worse on SPL.

Method | Success $(\mathbf{\uparrow})$ | SPL $(\mathbf{\uparrow})$
---|---|---
42) Random | $0.4\%$ | $0.4\%$
43) RGBD+RL [19] | $8.2\%$ | $2.7\%$
44) RGBD+Semantics+RL [54] | $15.9\%$ | $4.9\%$
45) Classical Map + FBE | $40.3\%$ | $12.4\%$
46) Active Neural SLAM [55] | $44.6\%$ | $14.5\%$
47) SemExp [51] | $54.4\%$ | $19.9\%$
48) PONI [53] | $73.6\%$ | $41.0\%$
49) IL w/ $70k$ Human Demos (Zero-Shot) | $49.5\%$ | $16.4\%$

Table 5: ObjectNav results on the Gibson val split.

###  A.3 Estimating time using a LoCoBot motion model

To estimate the time a robot would take to execute the collected human
trajectories in the real world, we use the LoCoBot motion model from Krantz
_et al_. [28]. This model consists of a rotation function that maps turn angle
to time and a translation function that maps straight-line distance to time.
For estimating time required for grab/release actions, we replace them with
$0.15m$ forward steps and use the straight-line distance translation function.
We use the MoveBase controller from [28] for all our time estimates, with the
following rotation and translation equations:

$y^{rotate}=0.000358\phi^{2}+0.108\phi+2.23$ (3)

$y^{translate}=4.2x+0.362$ (4)

###  A.4 Characterizing Learnt Behaviors

In this section, we describe the metrics used to characterize the exploration
behavior exhibited by these agents in Sec. 7 in the main paper. These include
1) Occupancy Coverage (OC) and 2) Sight Coverage (SC) introduced in Sec. 4 in
the main paper, as well as 3) Goal Room Time Spent (GRTS) – the number of
steps as a fraction of total episode length an agent takes within the room
bounding box containing the target object, 4) Peeks – check if the agent steps
back into the last visited room after taking just ${\sim}10$ steps in another
room, 5) Panoramic Turn (PT) – whether the agent stands at one place and turns
left and right to get sweeping views, 6) Beeline – if the agent takes $10$
continuous forward actions before reaching the goal in the last $15$ steps, 7)
Exhaustive Search (ES) – $\geq 75\%$ sight coverage. To compute these metrics,
we use the semantic annotations in Matterport3D. These annotations provide
$3$D bounding box coordinates for each room category in an environment. We use
these bounding box coordinates to track the rooms an agent visits during an
episode. GRTS gives us a measure of how often the agent ends up reaching goal
object room but doesn’t successfully locate the object. A higher GTRS suggests
that the agent is at least good at reaching semantically meaningful locations
in search of the goal object. We find that RL agents have higher average GRTS
but also significantly higher variance in GRTS across scenes while our IL
agents have lower average GRTS but more consistently spend time in the target
room (see Fig. 9). To evaluate not just the final room the agent ends up at,
but all the rooms it visits through the course of an episode, we also plot
distributions of the time spent per room category for each goal object (see
Fig. 15) for human demonstrations  _vs_. IL agents trained on human
demonstrations  _vs_. RL agents.

Figure 9: Per scene breakdown of GRTS for IL and RL agent on MP3D val split.

###  A.5 Inter-human Variance in ObjectNav

To get a sense for the variance in ObjectNav human demonstrations, we
collected $20$ unique human-provided trajectories for the same initial
location and target object (‘cabinet’). This is visualized in Fig. 10. We see
that there is quite a bit of diversity in navigation trajectories across
humans. They often navigate to different instances of the goal object category
‘cabinet’, and even when multiple humans go to the same object instance, the
routes taken are different (red  _vs_. blue trajectory).

We also plot the average SPL per AMT user in our dataset in Fig. 11. We find
that human performance has a lot of variability, ranging from $25.2\%$ to
$68.2\%$ (Fig. 11(a)). The SPL range that has the most humans is ${\sim}50\%$.
The best-performing human annotator achieves an SPL of $68.2\%$ averaged over
$6$ episodes (Fig. 11(b)), which is particularly close to shortest paths and
arguably _super-human_.

Figure 10: Visualizing multiple human demonstrations for ObjectNav all
starting from the same start position and searching for ‘cabinet’.

(a)

(b)

Figure 11: a) Histogram of average SPL for each AMT user for ObjectNav. b)
Plot showing average SPL for AMT users for ObjectNav. These plots clearly
demonstrate some humans are better at solving the ObjectNav task than others.
Figure 12: Screenshot of our Amazon Mechanical Turk interface for collecting
Pick&Place demonstrations. Users are provided the agent’s first-person view of
the environment and an instruction such as "Pick the toy airplane and place it
on the the colored wood blocks". They can make the agent look around and move
in the environment via keyboard controls, and can submit the task upon
successful completion by clicking the ‘Submit’ button. Figure 13:
Visualizations of learnt agent behaviors for ObjectNav and Pick&Place. Best
viewed in videos at sites.google.com/view/object-search-supp. Figure 14:
Visualizations of learnt agent behaviors for ObjectNav and Pick&Place. Best
viewed in videos at ram81.github.io/projects/habitat-web.

###  A.6 AMT Interface

Fig. 12 shows a screenshot of our AMT interface for collecting Pick&Place
demonstrations. For the Pick&Place task, we provide humans with an instruction
of the form _‘Place the <object> on the <receptacle>’_, without being told the
location of the <object> or <receptacle> in a new environment, and they can
see agent’s first-person view of the environment. They can make the agent
move, look around, and interact with the environment using keyboard controls.
Once the AMT user completes the task they can submit the task by clicking the
‘Submit’ button. We then run task-specific validation checks to ensure only
successful tasks get submitted.

Validation. To ensure data quality, every submitted AMT task goes through a
set of validation checks. For ObjectNav, we use the same set of validation
checks as the Habitat challenge evaluation setup,  _i.e_. a task is considered
successful only when the user has moved the agent to within $1m$ of the goal
object. We do not limit the maximum number of steps to allow users on AMT to
explore the environment. This captures key human exploration behavior
necessary to succeed at these tasks. Similarly, for Pick&Place, a task is
considered successful when the target object is placed on a receptacle object.
Specifically, we check if the Euclidean distance between the centers of the
target and receptable objects is less than $0.7m$, and that the target object
is at a height greater than the receptacle center.

###  A.7 Limitations

Our approach is fundamentally limited by the limitations of imitation learning
as our approach uses vanilla behavior cloning with inflection weighting.
Additionally, these agents trained on human demonstrations exhibit some common
failure cases. Some examples of common failure cases are – reaching close to
the goal object but not within goal radius and ending episode early, trying to
move straight when agent is colliding and getting stuck, looping around
multiple instances of the goal object and as a result, exceeding maximum
episode steps, and exploring the environment and not finding the goal object.
Our approach is also limited by the amount of human demonstrations we can
gather and the agent architecture being trained on this dataset. Currently, we
use a vanilla CNN+RNN architecture to learn imitation learning policies but we
can build better architecture which make full use of the rich semantic
information these human demonstrations have.

(a) Humans

(b) IL on $40k$ Human demos

(c) RL

Figure 15: Comparison of per room time spent for all MP3D goal categories on
val split for human demonstrations  _vs_. IL agents trained on human demos
_vs_. RL agents. The plot shows the top 10 rooms ordered by the maximum time
spent in each room.

(a) Humans

(b) IL on $40k$ Human demos

(c) RL

Figure 16: Comparison of per room time spent for all MP3D goal categories on
val split for human demonstrations  _vs_. IL agents trained on human demos
_vs_. RL agents. The plot shows the top 10 rooms ordered by the maximum time
spent in each room.

(a) Humans

(b) IL on $40k$ Human demos

(c) RL

Figure 17: Comparison of per room time spent for all MP3D goal categories on
val split for human demonstrations  _vs_. IL agents trained on human demos
_vs_. RL agents. The plot shows the top 10 rooms ordered by the maximum time
spent in each room.

(a) Humans

(b) IL on $40k$ Human demos

(c) RL

Figure 18: Comparison of per room time spent for all MP3D goal categories on
val split for human demonstrations  _vs_. IL agents trained on human demos
_vs_. RL agents. The plot shows the top 10 rooms ordered by the maximum time
spent in each room.

(a) Humans

(b) IL on $40k$ Human demos

(c) RL

Figure 19: Comparison of per room time spent for all MP3D goal categories on
val split for human demonstrations  _vs_. IL agents trained on human demos
_vs_. RL agents. The plot shows the top 10 rooms ordered by the maximum time
spent in each room.

(a) Humans

(b) IL on $40k$ Human demos

(c) RL

Figure 20: Comparison of per room time spent for all MP3D goal categories on
val split for human demonstrations  _vs_. IL agents trained on human demos
_vs_. RL agents. The plot shows the top 10 rooms ordered by the maximum time
spent in each room.

(a) Humans

(b) IL on $40k$ Human demos

(c) RL

Figure 21: Comparison of per room time spent for all MP3D goal categories on
val split for human demonstrations  _vs_. IL agents trained on human demos
_vs_. RL agents. The plot shows the top 10 rooms ordered by the maximum time
spent in each room.