††thanks: These authors contributed equally to this work††thanks: These
authors contributed equally to this work

# Correlated Topological Mixed-Valence Insulators in Moiré Hetero-Bilayers

Juan Felipe Mendez-Valderrama  Sunghoon Kim  Debanjan Chowdhury  Department of
Physics, Cornell University, Ithaca, New York 14853, USA.

###### Abstract

Moiré transition metal dichalcogenide (TMD) materials provide an ideal
playground for studying the combined interplay of strong interactions and
band-topology over a range of electronic fillings. Here we investigate the
panoply of interaction-induced electronic phases that arise at a total
commensurate filling of $\nu_{T}=2$ in moiré TMD heterobilayers, focusing
specifically on their renormalized band-topology. We carry out a comprehensive
self-consistent parton mean-field analysis on an interacting mixed-valence
Hamiltonian describing AB-stacked MoTe2/WSe2 to highlight different
ingredients that arise due to “Mottness”, band-flattening, an enhanced
excitonic tendency, and band-inversion, leading to correlated topological
semi-metals and insulators. We also propose a possible route towards realizing
fractionalized insulators with emergent neutral fermionic excitations in this
and other closely related platforms.

Introduction.- Moiré transition metal dichalcogenide (TMD) materials have
brought together the fascinating interplay of tunable electronic correlations,
geometric frustration and band-topology in a highly controllable setting [1,
2, 3]. Using dual-gated untwisted heterobilayer and twisted homobilayer TMD
devices, a series of remarkable experiments have already demonstrated the
emergence of local moments in a Mott insulator [4], a sequence of Wigner-Mott
insulators at rational fillings [4, 5, 6, 7], bandwidth-tuned continuous
metal-insulator transitions at fixed fillings [8, 9], and the emergence of
Kondo-screening [10] as well as tunable Kondo-breakdown [11]. On the
topological front, the discovery of fractional quantum anomalous Hall
insulators [12, 13, 14, 15], as well as fractional quantum spin Hall
insulators [16] in the same setting has really helped establish this material
platform as a versatile quantum simulator of correlated many-body phenomena.
This manuscript is geared towards investigating the possible emergence of a
non-trivial and “interaction-renormalized” band-topology as a result of
hybridizing relatively dispersive bands with highly correlated and nearly flat
bands at commensurate fillings [17]. The primary goal here is to analyze the
stability of topological “mixed-valence” (and Kondo) insulating phases in
moiré TMD heterobilayers, and investigate the quantum phase transitions to
proximate phases.

The study of correlated mixed-valence materials in three-dimensional compounds
with strong valence (charge) fluctuations has a rich history [18, 19, 20, 21].
Recent years have re-energized the field as a result of experiments in a
number of strongly spin-orbit coupled compounds such as SmB6, YbB12, and
FeSb2, finding evidence for a high-temperature metallic state evolving into a
paramagnetic charge-insulator, with a metallic surface-state at low
temperature. First-principles based theory suggests that these insulators are
topological [22, 23, 24, 25], and there is already compelling experimental
indication that the surface states in SmB6 [26, 27, 28, 29, 30, 31] as well as
YbB12 [32, 33] are indeed topological. However, the insulating bulk in SmB6
and YbB12 shows striking quantum oscillations [34, 35, 36]; remarkably, the
electrically insulating bulk in YbB12 also acts as a metallic-like thermal
conductor, violating the Wiedemann-Franz ratio by orders of magnitude [37].
These observations suggest the intriguing possibility of their insulating bulk
hosting emergent, electrically neutral fermionic excitations forming a three-
dimensional Fermi surface [38, 39, 40, 41]. The possible origin for some of
these observations due to impurities and more conventional scenarios has been
discussed [42, 43, 44, 45, 46, 47], but a conclusive picture is yet to emerge.
The microscopic complexity associated with the traditional rare-earth-based
compounds, and the inability to tune across the interaction-induced insulating
phases readily has remained a roadblock towards resolving the key outstanding
challenges. Finding alternative ways to recreate similar phenomenology in
relatively simpler and tunable platforms such as the moiré bilayer TMD, and
probe their low-energy excitations using novel methods will be an important
milestone.

TMD heterobilayers.- We focus here on AB-stacked MoTe2/WSe2, where the lattice
mismatch gives rise to a moiré superlattice with lattice constant $a_{\rm
M}\sim 5$ nm. The wave functions of the topmost valence bands are centered at
the MM (M$\equiv$Metal) and XX (X$\equiv$Chalcogen) sites of the AB-stacking,
respectively, which constitute the two distinct sublattice sites in a low-
energy description in terms of a honeycomb lattice. The interlayer tunneling
(referred to as the “hybridization” henceforth) can be treated as a
perturbation relative to the decoupled layers due to the combined effects of a
strong spin-momentum locking and the AB-stacking. Additionally, the MoTe2
layer hosts a narrower bandwidth, and is more “correlated” relative to the
WSe2 layer; the former is the analog of the $f-$electron while the latter
represents the $c-$electron in the usual scheme of describing heavy fermion
materials. We will demonstrate that the bilayer is an ideal platform to
realize a topological mixed-valence insulator over a wide range of parameters,
but many of these considerations can be generalized to other layered setups.

Figure 1: Top: (Renormalized) electronic dispersions for valley $K$ with
$U=30$ meV and other parameters as introduced in main text. Bottom: Electronic
fermiology and Berry curvature distribution, $\Omega(k)$, for the lower bands.
(a) and (b) show results for $V=0$, which realizes a TSM. (c) and (d) show
results for $V=8$ meV, which realize a TMVI. The non-interacting bands are
shown in grey in the top panel and the renormalized bands in blue/red. The
representative values of $\Delta$ are chosen (see Fig. 2) to realize fillings
at and away from $\langle\nu_{\textnormal{Mo}}\rangle\approx 1$.

Previous experimental work in this setup has already revealed a topological
phase transition driven by the displacement field from a non-topological Mott
insulator (where only the MoTe2 layer is occupied) to a quantum anomalous Hall
insulator at fixed total moiré unit cell filling $\nu_{T}=1$ [48, 49]. A
series of subsequent theoretical works have investigated this physics [50, 51,
52, 53, 54, 55, 56, 57, 58, 59]. Doping away from this commensurate limit, as
itinerant holes are introduced to the WSe2 layer while the MoTe2 layer remains
a Mott insulator, over a wide range of fillings an effective Kondo lattice
model [60] can be simulated [10, 11]. Finally, at the commensurate total
filling $\nu_{T}=2$, a displacement-field induced topological phase transition
from a trivial band insulator to a ${\mathbb{Z}}_{2}$ topological insulator
has been observed [61]. Over the observed field range, the electrons mostly
reside in the MoTe2 layer, away from any regime of “Mottness”; it is unclear
the extent to which the strong correlation effects associated with a
correlated half-filled band need to be taken into consideration [53]. This
manuscript is devoted to the study of the insulating and semi-metallic phases,
both with and without non-trivial topology, that emerge at $\nu_{T}=2$ taking
into account the Mott-like correlations from the outset.

Model.- Instead of working in the Kondo-limit directly with
$\langle\nu_{\textnormal{Mo}}\rangle=1$, and treating the degrees of freedom
in the MoTe2 layer as $S=1/2$ local moments [60], we work with the full low-
energy lattice model including the charge fluctuations [62]. The model is
given by $H_{\textnormal{MV}}=H_{\rm{Mo}}+H_{\rm{W}}+H_{\rm{Mo-W}}$ with
$\ell=\\{\rm{Mo},\rm{W}\\}$,

$\displaystyle H_{\ell}$ $\displaystyle=$ $\displaystyle-
t_{\ell}\sum_{\sigma,\langle\langle
i,j\rangle\rangle\in\ell}(e^{-i\nu_{ij}\phi_{\ell}s_{\sigma}}c^{\dagger}_{i\sigma}c_{j\sigma}+\textnormal{h.c.})-\mu
N_{\ell}$ (1a)
$\displaystyle-\tau_{\ell}\frac{\Delta}{2}N_{\ell}+U_{\ell}\sum_{i\in\ell}n_{i\uparrow}n_{i\downarrow},$
$\displaystyle H_{\rm{Mo-W}}$ $\displaystyle=$ $\displaystyle-
t_{\rm{hyb}}\sum_{\langle
i,j\rangle}(c^{\dagger}_{i\sigma}c_{j\sigma}+\textnormal{h.c.})+V\sum_{\langle
i,j\rangle}n_{i}n_{j}.$ (1b)

Here $\nu_{ij}=-\nu_{ji}$ represents the direction-dependent factor for the
path connecting sites $i$ and $j$ that controls the pseudospin-dependent next-
nearest neighbor hopping with an additional sign $s_{\sigma}$, which ensures
that time-reversal symmetry is preserved between the microscopic valleys
labeled by $\sigma$ [51, 52, 56]. The non-interacting estimates for the
hopping amplitudes from DFT are $t_{\rm{Mo}}=4.5~{}$meV, $t_{\rm{W}}=9~{}$meV
[51, 52], while a simple estimate of the onsite interaction is $U\sim
e^{2}/\epsilon\epsilon_{0}a_{M}$ that can range from $U\sim 30-60~{}$meV for
dielectric constants $\epsilon\sim 10-5$. Due to lattice relaxation, different
spin flavors can mix within a single valley leading to a weak interlayer
tunneling that translates to a finite hybridization, $t_{\rm{hyb}}=2~{}$meV
[51]. In momentum space, the hybridization form-factor reads
$\Phi_{\bm{k}}=t_{\rm{hyb}}\sum_{j=1}^{3}\textnormal{exp}[i{\bm{k}}\cdot\bm{\delta}_{j}]$
with $\bm{\delta}_{j}=a_{\rm M}/\sqrt{3}(\sin(2\pi j/3),-\cos(2\pi j/3))$; the
form-factor plays a crucial role in the development of a topological gap. The
finite $t_{\rm{hyb}}$ leads to a single $U(1)$ conserved charge associated
with the total density, where the chemical potential, $\mu$, couples to
$N_{\rm{Mo}}+N_{\rm{W}}$. The effect of the displacement field is incorporated
via a sublattice potential, $\Delta$, that couples to both layers with
opposite signs $\tau_{\rm Mo}=-\tau_{\rm W}=1$, respectively.

The $C_{3}$ eigenvalues of the orbitals within DFT dictate that the MoTe2 and
WSe2 wannier functions pick up a phase of $\phi_{\rm Mo}\approx\phi_{\rm
W}\approx 2\pi/3$ [63, 56, 51]. Consequently, the band maxima of both the
MoTe2 and WSe2 bands reside near the $\kappa$ and $\kappa^{\prime}$ points of
the moiré Brillouin zone. These $C_{3}$ eigenvalues and the associated
location of the band maxima are crucially important for determining the band-
topology as a function of $\Delta$ for a finite $t_{\rm{hyb}}$ [52].
Additionally, the nearest-neighbor (interlayer) Coulomb interaction, $V$, can
lead to qualitatively new effects and is not expected to be small from a
purely microscopic perspective [57].

Figure 2: Self-consistent parton mean-field phase diagram at $\nu_{T}=2$ as a
function of $\Delta$ and $V$ for a $12\times 12$ system for (a)-(c) $U=30$ meV
and (d)-(f) $U=50$ meV, respectively. The color codings denote the (a) and (d)
charge-gap, (b) and (e) valley Hall conductivity, and (c) and (f) the boson
condensate associated with the W holes. The dispersions and topology
associated with the points marked in (a) are depicted in detail in Fig. 1 (a)
and (b), respectively.

Parton mean-field analysis.- The numerical estimates of the energy scales
places the system in the intermediate to strong-coupling regime. At filling
$\nu_{T}=2$, when the individual layers are at filling $(1\pm\delta)$, a
simple non-interacting or Hartree-Fock description might be insufficient to
capture many of the essential features of the problem. Therefore, we implement
a parton mean-field-based analysis, and obtain a theoretical phase diagram as
a function of various microscopic tuning parameters. This will pave the way
for a more sophisticated numerical analysis, which will also help guide future
experiments.

Consider the following representation for the electron operators,
$c_{i,\sigma}=b_{i}f_{i,\sigma}$, where the $b_{i}=e^{i\theta_{i}}$ represent
spinless charged bosonic (“rotor”) fields and the $f_{i,\sigma}$ denote
spinful neutral spinons [64, 65]. The parton representation entails an
enlarged Hilbert space, which can be brought back to the physical Hilbert
space by imposing the constraints $\langle
n_{i}^{\theta}\rangle+\langle\sum_{\sigma}n^{f}_{i,\sigma}\rangle=2$. Here,
$n_{i}^{\theta}$ is the rotor charge and $n^{f}_{i,\sigma}$ is the fermionic
occupation, respectively. In what follows, $\sigma$ represents the conserved
valley degree of freedom instead of physical spin. We begin by incorporating
the mean-field decomposition of the above Hamiltonian in a reasonably
conventional fashion, where the effect of the $U-$ term is included in the
bosonic (rotor) sector. On the other hand, the effect of the $V-$ term is
included in both the bosonic and fermionic sectors via variational weights,
$\alpha$ and $(1-\alpha)$, respectively [66].

The remaining variational parameters in the matter field sectors are the
correlators, $B_{ij}\equiv\langle
b^{\dagger}_{i}b^{\phantom{\dagger}}_{j}\rangle_{\theta}$ and
$\chi_{ij,\sigma}\equiv\langle
f_{i\sigma}^{\dagger}f^{\phantom{\dagger}}_{j\sigma}\rangle_{f}$,
respectively. Note that in the absence of the $t_{\textnormal{hyb}}$ term
(i.e. with enlarged $U(1)\times U(1)$ symmetry), a spontaneous expectation
value for $\chi_{ij,\sigma}$ for nearest-neighbors generates an interlayer
“excitonic” order parameter. In order to solve the quartic bosonic
Hamiltonian, we utilize a two-site cluster approximation for all pairs of
$\langle ij\rangle$ and $\langle\langle ij\rangle\rangle$ [65]; the self-
consistent computations are carried out until the energy converges to within a
small threshold [66]. We also note that the numerical computations do not
include the effects of emergent gauge-field fluctuations, but we comment on
their effects at the end.

Results.- We have performed our mean-field analysis as a function of $\Delta$
and $V$ for two representative values of $U$. We restrict our attention to
translation-invariant solutions, but allow for $C_{3}$ symmetry breaking. In
order to characterize the different phases, we have used a variety of
quantitative diagnostics, including the fermion gap ($E_{\textnormal{gap}}$),
the valley Hall conductivity ($\sigma_{\textnormal{vh}}$), and the bosonic
rotor condensate for the W holes ($\langle b_{\textnormal{W}}\rangle$). The
“Mottness” associated with the suppression of charge fluctuations is
characterized by the vanishing condensate. Recall that when the bosons are
condensed, the spinons become charged, and $E_{\textnormal{gap}}$
$(\sigma_{\textnormal{vh}})$ represents the electronic spectral-gap
(conductivity). On the other hand, when the bosons are gapped, the system can
potentially host fractionalized phases depending on the many-body state
associated with the neutral spinons.

Figure 3: Evolution of the fractionalized topological Kondo insulator (TKI∗;
marked by $\star$) for various hopping anisotropies across the two layers,
$t_{\ell}=6\pm\delta t$ (meV). The color-bar represents valley Hall
conductivity $\sigma_{\textnormal{vh}}$. The regime with $\langle
b_{\ell}\rangle\rightarrow 0$ appears to the right of the region demarcated by
the red dashed lines; only the TKI∗ phase is stable against confinement.

Let us analyze the renormalized electronic dispersions and Berry curvature at
two representative points in the phase diagram before a systematic scan over a
wider range of parameters. Consider first the $V=0$ limit, as in Figs. 1
(a)-(b), for two different values of $\Delta$. We note that the bare bands
(shown in gray) are strongly renormalized by the $U-$terms (colored bands in
Fig. 1 a-b), which enter via the $\langle b_{i}\rangle,~{}\langle
b^{\dagger}_{i}b^{\phantom{\dagger}}_{j}\rangle$ correlators in the parton
analysis. At zero displacement field, the band offset is larger than the
bandwidth of both of the Mo and W bands, and the system is a trivial band
insulator. With increasing displacement field, the WSe2 band approaches the
MoTe2 band, leading eventually to a band-inversion. The $p-$wave inter-layer
hybridization form-factor guarantees topological bands with a nonzero valley-
contrasting Chern number. However, generically, the many-body state is not an
insulator and is a compensated topological semimetal (TSM) instead, with
$E_{\textnormal{gap}}=0$ and finite $\sigma_{\textnormal{vh}}$ over a broad
range of $\Delta$. See Fig. 1 (a)-(b) for the fermiology associated with the
compensated electron and hole pockets, as well as the momentum distribution of
the Berry curvature associated with one of the valleys. It is worth
emphasizing that in a purely non-interacting computation at $\nu_{T}=2$ with
the bare bands and starting from the limit where $\nu_{\textnormal{Mo}}\approx
2$, the topological insulator survives in a rather small range of $\Delta$
[66].

Turning next to the effect of the nearest-neighbor repulsion, $V$, we find
that it generically helps realize and stabilize insulating phases. In part,
this is due to the enhanced excitonic tendency to bind the electrons and holes
near the compensated limit for the two layers, $\nu_{\ell}=1\pm\delta$ [57].
We find that for a critical $V\ll U$, a gap opening transition occurs (see
Fig. 2(c)-(d)) and the resulting insulator has a quantized
$\sigma_{\textnormal{vh}}$. Note that the insulator also exists over a wide
range of redistributed filling between the two layers (at the saddle-point
level), which extends to $\langle\nu_{\textnormal{Mo}}\rangle$ away from $1$.
We dub this state topological mixed valence insulator (TMVI). As an insulating
phase, the TMVI is smoothly connected to the non-interacting $\mathbb{Z}_{2}$
topological insulator discussed previously [61], but is crucially stabilized
by the effects of interactions, which plays a number of different roles as we
have seen above. To summarize, the main effect of $U$ is to drive band-
flattening, which is more prominent for the MoTe2 bands, and a small $V$ helps
drive a Lifshitz transition from the TSM to the TMVI.

Phase diagram.- We have extended the analysis over a much wider range of
parameters, as shown in the phase-diagrams for two different values of $U$;
see Fig. 2 (a), (d). The results in Fig. 1 for specific parameter sets are
marked with the special symbols in Fig. 2(a), respectively. Both the TMVI and
TSM host a non-trivial topological response, $\sigma_{\textnormal{vh}}\neq 0$
(Fig. 2 b, e) in a strongly renormalized electronic phase (i.e. without any
fractionalization and $\langle b\rangle\neq 0$; see Fig. 2c, f).

We find an additional distinct phase with increasing $V$, driven by the
competition between the TMVI and a topologically trivial “excitonic”
insulator; for $t_{\textnormal{hyb}}\neq 0$ the excitonic character is not
sharply defined. Over a range of $\Delta$, the underlying $C_{3}$ symmetry is
spontaneously broken and we dub the resulting state a nematic excitonic
insulator (NEI) [59]. The NEI is adiabatically connected to a dimerized
excitonic insulator in the strong-coupling limit, $V\rightarrow\infty$. We do
not find evidence of any single-particle gap closing transition between the
TMVI and the NEI, but the topological response $\sigma_{\textnormal{vh}}$
vanishes in the NEI. Other examples of a topological phase transition without
single-particle gap closing have been discussed previously [67, 48, 68].
Importantly, the transition above is accompanied by the disappearance of the
nodal points of the form-factor, $\Phi_{k}$. The NEI spans over a broader
extent in the phase diagram for larger $U$, as can be seen by comparing Figs.
2 (a) and (d), respectively.

It is useful to comment briefly on the role of gauge-field fluctuations beyond
the mean-field analysis at this point. Clearly, in the many-body phases with a
condensed $\langle b_{\ell}\rangle\neq 0$, and at $T=0$, the effects of the
gauge-field fluctuations are irrelevant. On the other hand, notice that there
is a range of values of $V,~{}\Delta$ in Fig. 2(f) where $\langle b\rangle=0$.
Unsurprisingly, in this part of the phase diagram, the individual fillings
associated with both layers self-consistently lock at
$\langle\nu_{\ell}\rangle=1$, and the bosonic sector enters into a Mott-
insulating phase. However, the spinons are fully gapped and are non-
topological in this entire region; the sliver arises as a subset of the NEI
phase. Therefore, in two spatial dimensions in the absence of any matter
fields and non-trivial topology, the gauge-field will confine and the ground
state will not exhibit any fractionalization [69].

Fractionalized Insulators.- The appearance of a region with $\langle
b\rangle=0$ and a nearly symmetric, self-organized “Mott-like” phase raises an
interesting possibility: Are there topologically non-trivial insulating phases
with fractionalized excitations in the bulk? To address this question, we
start from a theoretical limit of two identical layers, and perturb in a small
parameter that controls their deviation away from this symmetric limit. For
simplicity, we restrict our attention to $\Delta=0$. In the symmetric limit at
total filling $\nu_{T}=2$ and large $U/t$, we are guaranteed to obtain two
copies of a Mott insulator with the filling distributed as $\nu_{T}=1+1$. We
then map out the many-body phase diagram as a function of the increasing
hopping anisotropy $\delta t$, i.e. $t_{\ell}=6\pm\delta t$ (meV) over a range
of $U$ and $V$ (Fig. 3). For $\delta t=0$, there is a wide range of parameters
where $\langle b\rangle=0$, and the spinons remain gapped with a topological
valley-hall response (region labeled by $\star$ in Fig. 3a). We dub this phase
fractionalized topological Kondo insulator (TKI∗). With increasing $\delta
t\lesssim 1.5$, there exists a finite regime in parameter space where the TKI∗
continues to remain the ground-state within parton mean-field theory (Fig.
3b-e). Note that the state is stable to inclusion of gauge-field fluctuations.
For even larger $\delta t$, we find that the TKI∗ phase loses out the
competition to the NEI; the local repulsion needed to gap out the bosons is
large enough to flatten the fermionic bands and drive a nematic exitonic
transition of the neutral fermions, which is unstable to confinement.

Outlook.- This work has been concerned with the emergence of novel correlated
mixed-valence insulators with intertwined topological bands in TMD moiré
heterobilayers. Over an extended regime in the many-body phase-diagram at
total filling $\nu_{T}=2$, we find a strongly interaction-renormalized
topological mixed-valence insulator with quantized valley Hall response, lying
in close proximity to a topological semi-metal and trivial nematic “excitonic”
insulator. In the extended phase-diagram, we have also found the emergence of
an interesting insulating TKI∗ phase with a fully gapped bulk and electrically
neutral edge states. Finding a closely related tunable moiré system where this
competition between different ground states can be studied experimentally
remains a promising future direction.

Looking ahead, the moiré bilayers provide an exciting playground for
potentially realizing a number of closely related fractionalized phases. While
the TKI∗ hosts a fully gapped bulk, the possibility of realizing a compensated
topological semimetal of the electrically neutral spinons in the bulk (TSM∗)
as an exotic gapless state, which is reminiscent of a candidate gapless
ground-state for the three-dimensional mixed-valence insulators such as YbB12
[39], and quantum Hall bilayers [70, 71, 72] remains an exciting prospect.
While the parton-based methods here provide a key first step for exploring
this rich phenomenology, future studies that rely on more sophisticated and
numerically exact methods are needed to examine this physics using unbiased
approaches, including the tendency towards additional forms of complex
symmetry-breaking patterns [73].

Acknowledgements.- We are grateful to Zhongdong Han, Kin Fai Mak and Jie Shan
for numerous inspiring discussions related to their experimental data. We also
thank Thomas Kiely and Dan Mao for a number of fruitful discussions. This work
is funded in part by a Sloan research fellowship to DC from the Alfred P.
Sloan foundation.

Note added.- The day before this manuscript was being prepared for submission,
an independent theoretical study analyzed related physics in moire
heterobilayers [74].

## References

  * Wu  _et al._ [2018] F. Wu, T. Lovorn, E. Tutuc, and A. H. MacDonald, Hubbard model physics in transition metal dichalcogenide moiré bands, Phys. Rev. Lett. 121, 026402 (2018).
  * Wu  _et al._ [2019] F. Wu, T. Lovorn, E. Tutuc, I. Martin, and A. H. MacDonald, Topological insulators in twisted transition metal dichalcogenide homobilayers, Phys. Rev. Lett. 122, 086402 (2019).
  * Mak and Shan [2022] K. F. Mak and J. Shan, Semiconductor moiré materials, Nature Nanotechnology 17, 686–695 (2022).
  * Tang  _et al._ [2020] Y. Tang, L. Li, T. Li, Y. Xu, S. Liu, K. Barmak, K. Watanabe, T. Taniguchi, A. H. MacDonald, J. Shan, and K. F. Mak, Simulation of hubbard model physics in wse2/ws2 moiré superlattices, Nature 579, 353–358 (2020).
  * Regan  _et al._ [2020] E. C. Regan, D. Wang, C. Jin, M. I. Bakti Utama, B. Gao, X. Wei, S. Zhao, W. Zhao, Z. Zhang, K. Yumigeta, M. Blei, J. D. Carlström, K. Watanabe, T. Taniguchi, S. Tongay, M. Crommie, A. Zettl, and F. Wang, Mott and generalized wigner crystal states in wse2/ws2 moiré superlattices, Nature 579, 359–363 (2020).
  * Xu  _et al._ [2020a] Y. Xu, S. Liu, D. A. Rhodes, K. Watanabe, T. Taniguchi, J. Hone, V. Elser, K. F. Mak, and J. Shan, Correlated insulating states at fractional fillings of moiré superlattices, Nature 587, 214–218 (2020a).
  * Li  _et al._ [2021a] H. Li, S. Li, E. C. Regan, D. Wang, W. Zhao, S. Kahn, K. Yumigeta, M. Blei, T. Taniguchi, K. Watanabe, S. Tongay, A. Zettl, M. F. Crommie, and F. Wang, Imaging two-dimensional generalized wigner crystals, Nature 597, 650–654 (2021a).
  * Ghiotto  _et al._ [2021] A. Ghiotto, E.-M. Shih, G. S. S. G. Pereira, D. A. Rhodes, B. Kim, J. Zang, A. J. Millis, K. Watanabe, T. Taniguchi, J. C. Hone, L. Wang, C. R. Dean, and A. N. Pasupathy, Quantum criticality in twisted transition metal dichalcogenides, Nature 597, 345–349 (2021).
  * Li  _et al._ [2021b] T. Li, S. Jiang, L. Li, Y. Zhang, K. Kang, J. Zhu, K. Watanabe, T. Taniguchi, D. Chowdhury, L. Fu, J. Shan, and K. F. Mak, Continuous mott transition in semiconductor moiré superlattices, Nature 597, 350–354 (2021b).
  * Zhao  _et al._ [2023a] W. Zhao, B. Shen, Z. Tao, Z. Han, K. Kang, K. Watanabe, T. Taniguchi, K. F. Mak, and J. Shan, Gate-tunable heavy fermions in a moiré kondo lattice, Nature 616, 61–65 (2023a).
  * Zhao  _et al._ [2023b] W. Zhao, B. Shen, Z. Tao, S. Kim, P. Knüppel, Z. Han, Y. Zhang, K. Watanabe, T. Taniguchi, D. Chowdhury, J. Shan, and K. F. Mak, Emergence of ferromagnetism at the onset of moiré kondo breakdown (2023b), arXiv:2310.06044 [cond-mat.str-el] .
  * Cai  _et al._ [2023] J. Cai, E. Anderson, C. Wang, X. Zhang, X. Liu, W. Holtzmann, Y. Zhang, F. Fan, T. Taniguchi, K. Watanabe, Y. Ran, T. Cao, L. Fu, D. Xiao, W. Yao, and X. Xu, Signatures of fractional quantum anomalous hall states in twisted mote2, Nature 622, 63–68 (2023).
  * Zeng  _et al._ [2023] Y. Zeng, Z. Xia, K. Kang, J. Zhu, P. Knüppel, C. Vaswani, K. Watanabe, T. Taniguchi, K. F. Mak, and J. Shan, Thermodynamic evidence of fractional chern insulator in moiré mote2, Nature 622, 69–73 (2023).
  * Park  _et al._ [2023] H. Park, J. Cai, E. Anderson, Y. Zhang, J. Zhu, X. Liu, C. Wang, W. Holtzmann, C. Hu, Z. Liu, T. Taniguchi, K. Watanabe, J.-H. Chu, T. Cao, L. Fu, W. Yao, C.-Z. Chang, D. Cobden, D. Xiao, and X. Xu, Observation of fractionally quantized anomalous hall effect, Nature 622, 74–79 (2023).
  * Xu  _et al._ [2023] F. Xu, Z. Sun, T. Jia, C. Liu, C. Xu, C. Li, Y. Gu, K. Watanabe, T. Taniguchi, B. Tong, J. Jia, Z. Shi, S. Jiang, Y. Zhang, X. Liu, and T. Li, Observation of integer and fractional quantum anomalous hall effects in twisted bilayer ${\mathrm{mote}}_{2}$, Phys. Rev. X 13, 031037 (2023).
  * Kang  _et al._ [2024] K. Kang, B. Shen, Y. Qiu, Y. Zeng, Z. Xia, K. Watanabe, T. Taniguchi, J. Shan, and K. F. Mak, Evidence of the fractional quantum spin hall effect in moiré mote2, Nature 10.1038/s41586-024-07214-5 (2024).
  * [17] K. Mak, J. Shan, and Z. Han, Private communication.
  * Mott [1974] N. F. Mott, Rare-earth compounds with mixed valencies, The Philosophical Magazine: A Journal of Theoretical Experimental and Applied Physics 30, 403 (1974), https://doi.org/10.1080/14786439808206566 .
  * Varma [1976] C. M. Varma, Mixed-valence compounds, Rev. Mod. Phys. 48, 219 (1976).
  * Lawrence  _et al._ [1981] J. M. Lawrence, P. S. Riseborough, and R. D. Parks, Valence fluctuation phenomena, Reports on Progress in Physics 44, 1 (1981).
  * Riseborough [2000] P. S. Riseborough, Heavy fermion semiconductors, Advances in Physics 49, 257 (2000), https://doi.org/10.1080/000187300243345 .
  * Lu  _et al._ [2013] F. Lu, J. Zhao, H. Weng, Z. Fang, and X. Dai, Correlated topological insulators with mixed valence, Phys. Rev. Lett. 110, 096401 (2013).
  * Weng  _et al._ [2014] H. Weng, J. Zhao, Z. Wang, Z. Fang, and X. Dai, Topological crystalline kondo insulator in mixed valence ytterbium borides, Phys. Rev. Lett. 112, 016403 (2014).
  * Xu  _et al._ [2020b] K.-J. Xu, S.-D. Chen, Y. He, J. He, S. Tang, C. Jia, E. Y. Ma, S.-K. Mo, D. Lu, M. Hashimoto, T. P. Devereaux, and Z.-X. Shen, Metallic surface states in a correlated d-electron topological kondo insulator candidate fesb2, Proceedings of the National Academy of Sciences 117, 15409 (2020b), https://www.pnas.org/doi/pdf/10.1073/pnas.2002361117 .
  * Dzero  _et al._ [2016] M. Dzero, J. Xia, V. Galitski, and P. Coleman, Topological kondo insulators, Annual Review of Condensed Matter Physics 7, 249 (2016).
  * Neupane  _et al._ [2013] M. Neupane, N. Alidoust, S.-Y. Xu, T. Kondo, Y. Ishida, D. J. Kim, C. Liu, I. Belopolski, Y. J. Jo, T.-R. Chang, H.-T. Jeng, T. Durakiewicz, L. Balicas, H. Lin, A. Bansil, S. Shin, Z. Fisk, and M. Z. Hasan, Surface electronic structure of the topological kondo-insulator candidate correlated electron system smb6, Nature Communications 4, 2991 (2013).
  * Kim  _et al._ [2014] D. J. Kim, J. Xia, and Z. Fisk, Topological surface state in the kondo insulator samarium hexaboride, Nature Materials 13, 466 (2014).
  * Xu  _et al._ [2014] N. Xu, P. K. Biswas, J. H. Dil, R. S. Dhaka, G. Landolt, S. Muff, C. E. Matt, X. Shi, N. C. Plumb, M. Radović, E. Pomjakushina, K. Conder, A. Amato, S. V. Borisenko, R. Yu, H.-M. Weng, Z. Fang, X. Dai, J. Mesot, H. Ding, and M. Shi, Direct observation of the spin texture in smb6 as evidence of the topological kondo insulator, Nature Communications 5, 4566 (2014).
  * Eo  _et al._ [2018] Y. S. Eo, K. Sun, i. m. c. Kurdak, D.-J. Kim, and Z. Fisk, Inverted resistance measurements as a method for characterizing the bulk and surface conductivities of three-dimensional topological insulators, Phys. Rev. Appl. 9, 044006 (2018).
  * Lee  _et al._ [2019] S. Lee, V. Stanev, X. Zhang, D. Stasak, J. Flowers, J. S. Higgins, S. Dai, T. Blum, X. Pan, V. M. Yakovenko, J. Paglione, R. L. Greene, V. Galitski, and I. Takeuchi, Perfect andreev reflection due to the klein paradox in a topological superconducting state, Nature 570, 344 (2019).
  * Pirie  _et al._ [2020] H. Pirie, Y. Liu, A. Soumyanarayanan, P. Chen, Y. He, M. M. Yee, P. F. S. Rosa, J. D. Thompson, D.-J. Kim, Z. Fisk, X. Wang, J. Paglione, D. K. Morr, M. H. Hamidian, and J. E. Hoffman, Imaging emergent heavy dirac fermions of a topological kondo insulator, Nature Physics 16, 52 (2020).
  * Hagiwara  _et al._ [2016] K. Hagiwara, Y. Ohtsubo, M. Matsunami, S.-i. Ideta, K. Tanaka, H. Miyazaki, J. E. Rault, P. L. Fèvre, F. Bertran, A. Taleb-Ibrahimi, R. Yukawa, M. Kobayashi, K. Horiba, H. Kumigashira, K. Sumida, T. Okuda, F. Iga, and S.-i. Kimura, Surface kondo effect and non-trivial metallic state of the kondo insulator ybb12, Nature Communications 7, 12690 (2016).
  * Gupta  _et al._ [2023] A. Gupta, A. Weiser, L. Pressley, Y. Luo, C. Lygouras, J. Trowbridge, W. A. Phelan, C. L. Broholm, T. M. McQueen, and W. K. Park, Topological surface states in the kondo insulator $\mathrm{Yb}{\mathrm{b}}_{12}$ revealed via planar tunneling spectroscopy, Phys. Rev. B 107, 165132 (2023).
  * Tan  _et al._ [2015] B. S. Tan, Y.-T. Hsu, B. Zeng, M. C. Hatnean, N. Harrison, Z. Zhu, M. Hartstein, M. Kiourlappou, A. Srivastava, M. D. Johannes, T. P. Murphy, J.-H. Park, L. Balicas, G. G. Lonzarich, G. Balakrishnan, and S. E. Sebastian, Unconventional fermi surface in an insulating state, Science 349, 287 (2015), https://www.science.org/doi/pdf/10.1126/science.aaa7974 .
  * Hartstein  _et al._ [2018] M. Hartstein, W. H. Toews, Y.-T. Hsu, B. Zeng, X. Chen, M. C. Hatnean, Q. R. Zhang, S. Nakamura, A. S. Padgett, G. Rodway-Gant, J. Berk, M. K. Kingston, G. H. Zhang, M. K. Chan, S. Yamashita, T. Sakakibara, Y. Takano, J.-H. Park, L. Balicas, N. Harrison, N. Shitsevalova, G. Balakrishnan, G. G. Lonzarich, R. W. Hill, M. Sutherland, and S. E. Sebastian, Fermi surface in the absence of a fermi liquid in the kondo insulator smb6, Nature Physics 14, 166 (2018).
  * Xiang  _et al._ [2018] Z. Xiang, Y. Kasahara, T. Asaba, B. Lawson, C. Tinsman, L. Chen, K. Sugimoto, S. Kawaguchi, Y. Sato, G. Li, S. Yao, Y. L. Chen, F. Iga, J. Singleton, Y. Matsuda, and L. Li, Quantum oscillations of electrical resistivity in an insulator, Science 362, 65 (2018), https://www.science.org/doi/pdf/10.1126/science.aap9607 .
  * Sato  _et al._ [2019] Y. Sato, Z. Xiang, Y. Kasahara, T. Taniguchi, S. Kasahara, L. Chen, T. Asaba, C. Tinsman, H. Murayama, O. Tanaka, Y. Mizukami, T. Shibauchi, F. Iga, J. Singleton, L. Li, and Y. Matsuda, Unconventional thermal metallic state of charge-neutral fermions in an insulator, Nature Physics 15, 954 (2019).
  * Baskaran [2015] G. Baskaran, Majorana Fermi Sea in Insulating SmB6: A proposal and a Theory of Quantum Oscillations in Kondo Insulators, arXiv e-prints , arXiv:1507.03477 (2015), arXiv:1507.03477 [cond-mat.str-el] .
  * Chowdhury  _et al._ [2018] D. Chowdhury, I. Sodemann, and T. Senthil, Mixed-valence insulators with neutral fermi surfaces, Nature Communications 9, 1766 (2018).
  * Erten  _et al._ [2017] O. Erten, P.-Y. Chang, P. Coleman, and A. M. Tsvelik, Skyrme insulators: Insulators at the brink of superconductivity, Phys. Rev. Lett. 119, 057603 (2017).
  * Varma [2020] C. M. Varma, Majoranas in mixed-valence insulators, Phys. Rev. B 102, 155145 (2020).
  * Knolle and Cooper [2015] J. Knolle and N. R. Cooper, Quantum oscillations without a fermi surface and the anomalous de haas–van alphen effect, Phys. Rev. Lett. 115, 146401 (2015).
  * Zhang  _et al._ [2016] L. Zhang, X.-Y. Song, and F. Wang, Quantum oscillation in narrow-gap topological insulators, Phys. Rev. Lett. 116, 046404 (2016).
  * Shen and Fu [2018] H. Shen and L. Fu, Quantum oscillation from in-gap states and a non-hermitian landau level problem, Phys. Rev. Lett. 121, 026403 (2018).
  * Knolle and Cooper [2017] J. Knolle and N. R. Cooper, Excitons in topological kondo insulators: Theory of thermodynamic and transport anomalies in ${\mathrm{smb}}_{6}$, Phys. Rev. Lett. 118, 096604 (2017).
  * Panda  _et al._ [2022] A. Panda, S. Banerjee, and M. Randeria, Quantum oscillations in the magnetization and density of states of insulators, Proceedings of the National Academy of Sciences 119, e2208373119 (2022), https://www.pnas.org/doi/pdf/10.1073/pnas.2208373119 .
  * Skinner [2019] B. Skinner, Properties of the donor impurity band in mixed valence insulators, Phys. Rev. Mater. 3, 104601 (2019).
  * Li  _et al._ [2021c] T. Li, S. Jiang, B. Shen, Y. Zhang, L. Li, Z. Tao, T. Devakul, K. Watanabe, T. Taniguchi, L. Fu, J. Shan, and K. F. Mak, Quantum anomalous hall effect from intertwined moiré bands, Nature 600, 641–646 (2021c).
  * Tao  _et al._ [2024] Z. Tao, B. Shen, S. Jiang, T. Li, L. Li, L. Ma, W. Zhao, J. Hu, K. Pistunova, K. Watanabe, T. Taniguchi, T. F. Heinz, K. F. Mak, and J. Shan, Valley-coherent quantum anomalous hall state in ab-stacked ${\mathrm{mote}}_{2}/{\mathrm{w}\mathrm{s}\mathrm{e}}_{2}$ bilayers, Phys. Rev. X 14, 011004 (2024).
  * Xie  _et al._ [2022a] Y.-M. Xie, C.-P. Zhang, J.-X. Hu, K. F. Mak, and K. T. Law, Valley-polarized quantum anomalous hall state in moiré ${\mathrm{mote}}_{2}/{\mathrm{wse}}_{2}$ heterobilayers, Phys. Rev. Lett. 128, 026402 (2022a).
  * Zhang  _et al._ [2021] Y. Zhang, T. Devakul, and L. Fu, Spin-textured chern bands in ab-stacked transition metal dichalcogenide bilayers, Proceedings of the National Academy of Sciences 118, e2112673118 (2021), https://www.pnas.org/doi/pdf/10.1073/pnas.2112673118 .
  * Devakul and Fu [2022] T. Devakul and L. Fu, Quantum anomalous hall effect from inverted charge transfer gap, Phys. Rev. X 12, 021031 (2022).
  * Pan  _et al._ [2022] H. Pan, M. Xie, F. Wu, and S. Das Sarma, Topological phases in ab-stacked ${\mathrm{mote}}_{2}/{\mathrm{wse}}_{2}$: ${\mathbb{z}}_{2}$ topological insulators, chern insulators, and topological charge density waves, Phys. Rev. Lett. 129, 056804 (2022).
  * Su  _et al._ [2022] Y. Su, H. Li, C. Zhang, K. Sun, and S.-Z. Lin, Massive dirac fermions in moiré superlattices: A route towards topological flat minibands and correlated topological insulators, Phys. Rev. Res. 4, L032024 (2022).
  * Chang and Chang [2022] Y.-W. Chang and Y.-C. Chang, Quantum anomalous hall effect and electric-field-induced topological phase transition in ab-stacked ${\mathrm{mote}}_{2}/{\mathrm{wse}}_{2}$ moiré heterobilayers, Phys. Rev. B 106, 245412 (2022).
  * Rademaker [2022] L. Rademaker, Spin-orbit coupling in transition metal dichalcogenide heterobilayer flat bands, Phys. Rev. B 105, 195428 (2022).
  * Dong and Zhang [2023] Z. Dong and Y.-H. Zhang, Excitonic chern insulator and kinetic ferromagnetism in a ${\mathrm{mote}}_{2}/{\mathrm{wse}}_{2}$ moiré bilayer, Phys. Rev. B 107, L081101 (2023).
  * Xie  _et al._ [2022b] Y.-M. Xie, C.-P. Zhang, and K. Law, Topological $p\\_x+ip\\_y$ inter-valley coherent state in moir$\backslash$’e mote $\\_2$/wse $\\_2$ heterobilayers, arXiv preprint arXiv:2206.11666 (2022b).
  * Xie  _et al._ [2023] M. Xie, H. Pan, F. Wu, and S. Das Sarma, Nematic excitonic insulator in transition metal dichalcogenide moiré heterobilayers, Phys. Rev. Lett. 131, 046402 (2023).
  * Guerci  _et al._ [2023] D. Guerci, J. Wang, J. Zang, J. Cano, J. H. Pixley, and A. Millis, Chiral kondo lattice in doped mote 2 /wse 2 bilayers, Science Advances 9, 10.1126/sciadv.ade7701 (2023).
  * Zhao  _et al._ [2024] W. Zhao, K. Kang, Y. Zhang, P. Knüppel, Z. Tao, L. Li, C. L. Tschirhart, E. Redekop, K. Watanabe, T. Taniguchi, A. F. Young, J. Shan, and K. F. Mak, Realization of the haldane chern insulator in a moiré lattice, Nature Physics 20, 275–280 (2024).
  * Xie  _et al._ [2024] F. Xie, L. Chen, and Q. Si, Kondo effect and its destruction in heterobilayer transition metal dichalcogenides, Phys. Rev. Res. 6, 013219 (2024).
  * Li  _et al._ [2021d] H. Li, S. Li, M. H. Naik, J. Xie, X. Li, J. Wang, E. Regan, D. Wang, W. Zhao, S. Zhao, S. Kahn, K. Yumigeta, M. Blei, T. Taniguchi, K. Watanabe, S. Tongay, A. Zettl, S. G. Louie, F. Wang, and M. F. Crommie, Imaging moiré flat bands in three-dimensional reconstructed wse2/ws2 superlattices, Nature Materials 20, 945 (2021d).
  * Florens and Georges [2004] S. Florens and A. Georges, Slave-rotor mean-field theories of strongly correlated systems and the mott transition in finite dimensions, Phys. Rev. B 70, 035114 (2004).
  * Zhao and Paramekanti [2007] E. Zhao and A. Paramekanti, Self-consistent slave rotor mean-field theory for strongly correlated systems, Phys. Rev. B 76, 195101 (2007).
  * [66] See supplementary material for additional details on the model Hamiltonian, phase transitions in the non-interacting limit, and self-consistent mean-field calculations.
  * Ezawa  _et al._ [2013] M. Ezawa, Y. Tanaka, and N. Nagaosa, Topological phase transition without gap closing, Scientific Reports 3, 10.1038/srep02790 (2013).
  * Zhao  _et al._ [2023c] J. Zhao, P. Mai, B. Bradlyn, and P. Phillips, Failure of topological invariants in strongly correlated matter, Phys. Rev. Lett. 131, 106601 (2023c).
  * Polyakov [1987] A. M. Polyakov,  _Gauge fields and strings_ (Taylor & Francis, 1987).
  * Barkeshli  _et al._ [2018] M. Barkeshli, C. Nayak, Z. Papić, A. Young, and M. Zaletel, Topological exciton fermi surfaces in two-component fractional quantized hall insulators, Phys. Rev. Lett. 121, 026603 (2018).
  * Zaletel  _et al._ [2018] M. P. Zaletel, S. Geraedts, Z. Papić, and E. H. Rezayi, Evidence for a topological “exciton fermi sea” in bilayer graphene, Phys. Rev. B 98, 045113 (2018).
  * Zibrov  _et al._ [2017] A. A. Zibrov, C. Kometter, H. Zhou, E. M. Spanton, T. Taniguchi, K. Watanabe, M. P. Zaletel, and A. F. Young, Tunable interacting composite fermion phases in a half-filled bilayer-graphene landau level, Nature 549, 360 (2017).
  * [73] T. Kiely and D. Chowdhury, Under preparation.
  * Guerci  _et al._ [2024] D. Guerci, K. P. Lucht, V. Crépel, J. Cano, J. H. Pixley, and A. J. Millis, Topological kondo semimetal and insulator in ab-stacked heterobilayer transition metal dichalcogenides (2024), arXiv:2407.12912 [cond-mat.str-el] .
  * Hermele and Gurarie [2011] M. Hermele and V. Gurarie, Topological liquids and valence cluster states in two-dimensional su$(n)$ magnets, Phys. Rev. B 84, 174441 (2011).

Supplementary material for “Correlated Topological Mixed-Valence Insulators in
Moiré Heterobilayers”
Juan Felipe Mendez-Valderrama, Sunghoon Kim, Debanjan Chowdhury

##  I Description of the model

Supplemental Figure S1: Schematic of the interacting model defined on a
honeycomb superlattice. Red (Mo) and blue (W) dots belong to two distinct
sublattices of the honeycomb lattice.

In Fig. S1, we present a schematic of the model studied in the main text,
which illustrates all of the microscopic hopping and interaction parameters.

##  II Topological phase transitions in the non-interacting limit

Supplemental Figure S2: Non-interacting phase diagram in the
$(\nu_{\textnormal{Mo}},\nu_{\textnormal{W}})=(2-\delta,\delta)$ regime. (a)
Evolution of $E_{\textnormal{gap}}$, the valley Chern number of the lower
band, and $\langle\nu_{\textnormal{W}}\rangle$ as a function of $\Delta$. The
black dashed lines mark phase boundaries. (b-d) Fermionic band dispersions for
the 3 representative points shown in (a). The black dashed lines mark the
Fermi levels. The dispersions and the Chern numbers are obtained for valley
$K$.

In this section, we present the phase diagram at $\nu_{T}=2$ in the non-
interacting limit $(U=V=0)$. We focus on the $\nu_{\textnormal{Mo}}\approx 2$
regime near band inversion. At large $\Delta$, the system lies in a trivial
band insulator (BI), where $\langle\nu_{\textnormal{Mo}}\rangle=2$. As
$\Delta$ is decreased, the moiré gap decreases and eventually the band
inversion takes place. The resulting phase is a $\mathbbm{Z}_{2}$ topological
insulator (TI) with small gap. The TI is not as robust as in the interacting
case, and immediately gives way to a topological semimetal (TSM) phase. As
elaborated in the main text, strong $U$ and $V$ conspire to stabilize the
gapped phase all the way to the mixed-valence regime.

##  III Details of parton mean-field calculations

In this section, we provide details of our parton mean-field calculations [64,
65]. The mapping between the local Hilbert space of electrons and partons is
given by

$\displaystyle|0\rangle\equiv|2\rangle_{\theta}|0\rangle_{f},$
$\displaystyle|\uparrow\rangle\equiv|1\rangle_{\theta}|\uparrow\rangle_{f},$
$\displaystyle|\downarrow\rangle\equiv|1\rangle_{\theta}|\downarrow\rangle_{f},$
$\displaystyle|\uparrow\downarrow\rangle\equiv|0\rangle_{\theta}|\uparrow\downarrow\rangle_{f},$
(III.1)

Here, ket states on the right hand side are direct product states of rotor and
spinon ket states. The rotor ket states are labeled by rotor charge,
$n^{\theta}$, and the spinon ket states are labeled by the (pseudo) spin
degrees of freedom of the original electrons. For the parton representation,
we impose the constraints $\langle
n_{i}^{\theta}\rangle+\langle\sum_{\sigma}n^{f}_{i,\sigma}\rangle=2$.

As shown in the main text, the parton mean-field Hamiltonian is given by

$\displaystyle H$ $\displaystyle=$ $\displaystyle H^{f}+H^{\theta},$
$\displaystyle H^{f}$ $\displaystyle=$ $\displaystyle-\sum_{i\neq
j,\sigma}t_{ij,\sigma}B_{ij}f^{\dagger}_{i\sigma}f_{j\sigma}-\mu\sum_{i}n_{i}^{f}\pm\sum_{i}\frac{\Delta}{2}n_{i}^{f},$
$\displaystyle-$ $\displaystyle(1-\alpha)V\sum_{\langle
ij\rangle,\sigma}\chi_{ji,\sigma}f^{\dagger}_{i,\sigma}f_{j,\sigma},$
$\displaystyle H^{\theta}$ $\displaystyle=$ $\displaystyle-\sum_{i\neq
j}\sum_{\sigma}t_{ij,\sigma}\chi_{ij,\sigma}b^{\dagger}_{i}b_{j}+\frac{U}{2}\sum_{i}(2-n_{i}^{\theta})(1-n_{i}^{\theta})$
(III.2) $\displaystyle+$ $\displaystyle\alpha V\sum_{\langle
ij\rangle}(2-n_{i}^{\theta})(2-n_{j}^{\theta})-\sum_{i}\mu_{i}^{\theta}n^{\theta}_{i},$

where $\alpha$, $B_{ij}\equiv\langle b^{\dagger}_{i}b_{j}\rangle_{\theta}$ and
$\chi_{ij,\sigma}\equiv\langle f_{i\sigma}^{\dagger}f_{j\sigma}\rangle_{f}$
are variational parameters. The bosonic Hamiltonian is solved using two-site
cluster approximation [65]. To be specific, for every pair of sites $(i,j)$
with $|t_{ij,\sigma}|\neq 0$, we construct a cluster Hamiltonian. For
instance, a cluster Hamiltonian for interlayer pair $\langle ij\rangle$ reads

$\displaystyle H^{\theta}_{ij}$ $\displaystyle=$
$\displaystyle-\sum_{a\in\\{ij\\}}\left(\sum_{k\notin\\{ij\\}}\sum_{\sigma}t_{ak,\sigma}\chi_{ak,\sigma}b^{\dagger}_{a}\langle
b_{k}\rangle+\textnormal{h.c.}+\frac{U}{2}(2-n_{a}^{\theta})(1-n_{a}^{\theta})-\mu_{a}^{\theta}n^{\theta}_{a}\right)$
(III.3) $\displaystyle+$ $\displaystyle\alpha
V(2-n_{i}^{\theta})(2-n_{j}^{\theta})-\sum_{\sigma}t_{ij,\sigma}\chi_{ij,\sigma}b^{\dagger}_{i}b_{j}+\textnormal{h.c.},$

where $\langle b_{k}\rangle$ denotes the bosonic superfluid order parameter
for the site $k$ outside of the cluster. Interlayer Coulomb terms for sites
outside of the cluster are absorbed into the chemical potential term. For
generic numerical parameters $(\\{t_{ij,\sigma}\\},U,V,\Delta)$, we find that
choosing $\alpha\rightarrow 0$ minimizes the mean-field energy; see Fig. S3.

Supplemental Figure S3: Mean-field energy as a function of $\alpha$ for two
representative points

Self-consistent calculations are performed in the following fashion. We focus
on translationally-invariant solutions, while allowing for the $C_{3}$
symmetry breaking of correlators emanating from each site. We start by
constructing initial $H^{f}$ with an ansatz for the fermionic correlator
$\\{\chi_{ij,\sigma}^{(0)}\\}$. Using the initial fermionic ground state, we
compute $\\{\chi_{ij,\sigma}^{(1)}\\}$ and the initial fillings
$\langle\nu_{\textnormal{Mo}}\rangle=1+\delta$ and
$\langle\nu_{\textnormal{W}}\rangle=1-\delta$. We then construct initial
$H^{\theta}_{ij}$’s using $\\{\chi_{ij,\sigma}^{(1)}\\}$ and Lagrange
multipliers $\\{\mu_{i}^{\theta}\\}$, which are chosen to fix the bosonic
fillings to $\langle n_{\textnormal{Mo}}^{\theta}\rangle=1-\delta$ and
$\langle n_{\textnormal{W}}^{\theta}\rangle=1+\delta$. The Lagrange
multipliers can be obtained through an iterative procedure using the charge
compressibility [75]. We compute bosonic expectation values
$\\{B_{ij}^{(1)}\\},\\{\langle b_{i}\rangle^{(1)}\\}$ using the initial
$H^{\theta}_{ij}$’s, where $\\{\langle b_{i}\rangle^{(1)}\\}$ is obtained by
averaging over the results of all the cluster Hamiltonians involving the site
$i$. We repeat this series of procedures to obtain a new set of variational
parameters $\\{\chi_{ij,\sigma}^{(2)}\\},\\{B_{ij}^{(2)}\\},\\{\langle
b_{i}\rangle^{(2)}\\}$. The self-consistent calculations are performed until
the mean-field energy is converged within a threshold, i.e.
$|E^{(n+1)}_{\textnormal{mf}}-E^{(n)}_{\textnormal{mf}}|/|E^{(n)}_{\textnormal{mf}}|<10^{-4}$.
To obtain the mean-field phase diagram, we try $\sim 50$ random initial ansatz
$\\{\chi_{ij,\sigma}^{(0)}\\}$’, and choose the solution with a minimum mean-
field energy.

##  IV Phase diagram for different band parameters

In Fig. S4, we present the mean-field phase diagram obtained for a slightly
different set of band parameters with a larger anisotropy in the hopping
parameters than the one for the DFT parameters:
$(t_{\textnormal{Mo}},t_{\textnormal{W}},t_{\textnormal{hyb}},U)=(3,10,1,50)$
meV.

Supplemental Figure S4:  Mean-field phase diagram at $\nu_{T}=2$ as a function
of $\Delta$ and $V$ for system of size $12\times 12$. Band parameters are
chosen as
$(t_{\textnormal{Mo}},t_{\textnormal{W}},t_{\textnormal{hyb}},U)=(3,10,1,50)$
meV.


Soham<EMAIL_ADDRESS>Girish
<EMAIL_ADDRESS>C V<EMAIL_ADDRESS>Kohli Center for
Intelligent Systems, Center for Visual Information Technology, IIIT Hyderabad,
India. Flipkart Internet Private Limited,
Bangalore, India. Visual Relocalization using Anchor Points

# Improved Visual Relocalization by Discovering Anchor Points

###### Abstract

We address the visual relocalization problem of predicting the location and
camera orientation or pose (6DOF) of the given input scene. We propose a
method based on how humans determine their location using the visible
landmarks. We define anchor points uniformly across the route map and propose
a deep learning architecture which predicts the most relevant anchor point
present in the scene as well as the relative offsets with respect to it. The
relevant anchor point need not be the nearest anchor point to the ground truth
location, as it might not be visible due to the pose. Hence we propose a multi
task loss function, which discovers the relevant anchor point, without needing
the ground truth for it. We validate the effectiveness of our approach by
experimenting on Cambridge Landmarks (large scale outdoor scenes) as well as 7
Scenes (indoor scenes) using various CNN feature extractors. Our method
improves the median error in indoor as well as outdoor localization datasets
compared to the previous best deep learning model known as PoseNet (with
geometric re-projection loss) using the same feature extractor. We improve the
median error in localization in the specific case of Street scene, by over 8m.

##  1 Introduction

The visual relocalization problem is an essential component of many practical
systems like autonomous navigation, augmented reality, drone navigation etc.
Although GPS sensors could be used for localization in these applications, it
is often noisy and will not work in indoor environments. Hence an independent
source of location information is essential for safety as well as
applicability in a wide variety of environments. Though location information
could be accurately estimated using 3D point cloud data, gathering and
processing such data can be expensive and slow. The visual relocalization
problem is defined as estimating the location as well as camera pose, given
just the observed camera frame, without using any other sensor data.

Early solutions modeled this problem as an image retrieval problem. However,
these solutions required features for a large collection of images to be
stored. Also, a nearest neighbour search was needed at test time. Thus the
memory and runtime increased proportionately with the dataset size. The most
accurate approaches require 3D point cloud data of the region [Sattler et
al.(2017)Sattler, Leibe, and Kobbelt], which is expensive to gather and
process. PoseNet proposed to model this problem directly as a regression
problem and used a deep neural network as the image feature extractor [Kendall
et al.(2015)Kendall, Grimes, and Cipolla]. Extensions to this work aim at
modeling the uncertainty in estimating the location and pose by using a
Bayesian neural network [Kendall and Cipolla(2017)]. Furthermore, a spatial
LSTM based approach to regress the pose was found to improve performance
[Walch et al.(2016)Walch, Hazirbas, Leal-Taixé, Sattler, Hilsenbeck, and
Cremers]. A more recent approach has been to improve points through a
geometric loss function, which tries to minimize the reprojection error
[Kendall and Cipolla(2017)].

Figure 1:  An example of anchor point allocation for a sample road. The blue
points (A1-A4) denote the anchor points which are predefined uniformly. The
green triangle(I1) is the input frame which denotes the current location. The
coordinates of the anchor points are shown relative to the coordinate of the
current location (denoted as (0,0)). The nearest anchor point A3 or A2, might
not be visible properly from the current location. Our system can find the
most relevant anchor point and the relative offsets from it, giving a more
accurate prediction.

We propose a visual relocalization system inspired by how humans determine
location. We typically identify some landmarks which we deem as important. The
coordinates of these landmarks are already known. Then, we try to estimate our
position by determining the offset to our position relative to the landmark.
The landmark we choose need not be the nearest one, as it might not be
visible, due to the viewing angles. We refer to these landmarks as anchor
points. Inspired by this idea, we propose a system which assigns anchor points
relative to which the 6DOF (six degrees of freedom of the 3 spatial and 3 pose
coordinates) can be predicted accurately. We do this by modeling the problem
as a multi task problem, of classifying the input scene to an anchor point and
then finding the offsets relative to that anchor point. However, a direct
approach for training such a network requires ground truth of the anchor
points for each image. The datasets typically provide only the ground truths
of the 6-DOF and not for relative offsets from the anchor points. The anchor
point visible in the image need not be the nearest one, because of the pose.
Hence we propose a new loss function, which during training, automatically
finds the appropriate anchor point relative to which the offsets needs to be
regressed in an end to end fashion.

We benchmark our model on an outdoor dataset covering a large area (Cambridge
landmarks [Kendall et al.(2015)Kendall, Grimes, and Cipolla] covering few 1000
m2) as well as an indoor dataset (7 Scenes [Shotton et al.(2013b)Shotton,
Glocker, Zach, Izadi, Criminisi, and Fitzgibbon] covering few m2) suited for
small robot navigation. We improve the median error in all the scenes of both
the datasets, from the previous best model, PoseNet (with geometric
reprojection loss [Kendall and Cipolla(2017)]), when using the same GoogleNet
[Szegedy et al.(2015a)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan,
Vanhoucke, and Rabinovich] feature extractor. We achieve <1.5m and <4∘ in
localization performance in 4 out of the 6 outdoor scenes in Cambridge
Landmarks. In the specific case of Street scene, our method improves the
median error by over 8m. Furthermore, we localize to within <0.2m for all the
indoor scenes of the 7 Scenes dataset which is a significant improvement over
the previous deep learning based approaches. We also experiment with various
feature extractors like DenseNet [Huang et al.()Huang, Liu, Van Der Maaten,
and Weinberger] (giving better accuracy) and MobileNet [Howard et
al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang, Weyand, Andreetto, and Adam]
(giving better runtime performance, while maintaining accuracies).

##  2 Related Works

In this section, we briefly survey the literature. For a detailed survey see
[Piasco et al.(2018)Piasco, Sidibé, Demonceaux, and Gouet-Brunet].

#### Visual Place Recognition.

The visual place recognition task has been traditionally solved by modeling it
as an image retrieval problem [Arandjelović and Zisserman(2014)], [Cummins and
Newman(2008)], [Torii et al.(2013)Torii, Sivic, Pajdla, and Okutomi], [Shotton
et al.(2013a)Shotton, Glocker, Zach, Izadi, Criminisi, and Fitzgibbon]. This
enables Bag of Words (BOW) and VLAD [Jégou et al.(2012)Jégou, Perronnin,
Douze, Sánchez, Pérez, and Schmid] representations to be used in scalable
retrieval approaches. More recently, deep learning models have also been used
successfully for creating efficient image representations, which have been
combined with retrieval methods [Razavian et al.(2016)Razavian, Sullivan,
Carlsson, and Maki], [Gong et al.(2014)Gong, Wang, Guo, and Lazebnik],
[Babenko and Lempitsky(2015)], [Tolias et al.(2015)Tolias, Sicre, and Jégou]
[Babenko et al.(2014)Babenko, Slesarev, Chigorin, and Lempitsky]. However,
retrieval based solutions require image features for the entire dataset to be
stored. Also, a nearest neighbour search needs to be triggered at test time.
Thus, the memory and runtime increased proportionately with the data set size.
PlaNet [Weyand et al.(2016)Weyand, Kostrikov, and Philbin] modeled the problem
as a classification problem for localization at a world scale. However,
estimation of a fine-grained 6-DOF localization requires continuous output and
discrete methods have not worked so far for this task.

In contrast, metric localization (visual localization) techniques approach
this as a 2D-3D mapping between the 2D coordinate system of the image space to
the 3D coordinate system of the world space. This is typically done by
matching the representations of the learnt image [Li et al.(2010)Li, Snavely,
and Huttenlocher, Choudhary and Narayanan(2012), Sattler et al.(2012)Sattler,
Leibe, and Kobbelt, Svarm et al.(2014)Svarm, Enqvist, Oskarsson, and Kahl, Li
et al.(2016)Li, Snavely, Huttenlocher, and Fua, Sattler et al.(2017)Sattler,
Leibe, and Kobbelt] and adapting a nearest neighbour approach. The full 6-DOF
pose of the camera can be estimated quite precisely. These methods however
require a large database of features and efficient retrieval methods, which
makes them suffer at test time since most retrieval methods have an additional
feature matching latency.

#### PoseNet.

PoseNet [Kendall et al.(2015)Kendall, Grimes, and Cipolla] addressed this idea
by introducing a regression based deep neural network technique to estimate
the metric localization parameters. It achieves a combination of strengths of
place recognition and localization approaches and can localize without a prior
estimate on the pose. This solves the disadvantage of storing the features in
memory since they are learnt during training and estimated during testing.

Extensions to this work have been on RGB-D input images for localization [Li
et al.(2017)Li, Liu, Gui, Gu, and Hu], learning relative ego-motion of the
camera [Melekhov et al.(2017)Melekhov, Ylioinas, Kannala, and Rahtu],
improving the context of features used for localization [Szegedy et
al.(2015b)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and
Rabinovich], using video sequence information for localization [Kendall and
Cipolla(2016)], modeling the uncertainly in localization using Bayesian Neural
Networks [Kendall and Cipolla(2016)], exploring a number of loss functions for
learning camera pose which are based on geometry and scene re-projection error
[Kendall and Cipolla(2017)].

Although PoseNet and its extensions are robust and scalable, they lag behind
some of the traditional methods [Sattler et al.(2017)Sattler, Leibe, and
Kobbelt] in estimating the pose . In this work, we propose a novel discrete
anchor point assignment based regression technique which provides significant
improvement over these methods while maintaining the scalability and
robustness. We keep intact the advantages of PoseNet and its variants as well
as improve over the nearest neighbour based methods, where feature vectors and
required to be stored in memory entirely.

##  3 Approach

Figure 2: Block diagram of the proposed architecture. We experiment with
different CNN feature extractors including GoogleNet, DenseNet and MobileNet.
The CNN feature extractor is followed by a global average pooling layer to get
the final feature vector. The feature vector is fed to an anchor point
classifier, relative offset regressor and an absolute regressor for pose.

The visual relocalization problem involves inferring the 3D location
coordinates and the camera pose (given by 3 angles) from just the image. In
the previous works [Kendall and Cipolla(2016), Kendall and Cipolla(2017),
Kendall et al.(2015)Kendall, Grimes, and Cipolla], a deep neural network based
approach was proposed which directly predicts the 6-DOF by regressing against
the ground truth, relative to a global coordinate system. Here, we propose a
system which assigns anchor points relative to which the 6-DOF can be
predicted accurately. We do this by modeling the problem as a multi-task
problem. The first task is of classifying the input camera image to one of the
anchor points and the second task is of finding the offsets relative to that
anchor point. We use standard CNN feature extractors for image feature
extraction. The feature extractors are typically followed by a Global Average
Pooling (GAP) layer in order to get a single vector instead of a feature map.
This is illustrated in Figure 2. Our proposed method then branches this output
into 3 heads.

#### Anchor Point Classification.

Given a route map where visual relocalization is desired, we subdivide the
route into equally spaced intervals. The end points of the intervals are
designated as anchor points for the 2D spatial coordinates. Note that the
global coordinates of these anchor points are known, since they are fixed
a-priori. We then model the problem of finding the most relevant anchor point
as an image classification problem. Given an image, the predicted class should
be the most prominent anchor point in the image. Note that the most prominent
anchor point need not always be the one nearest to the location of the image.
Hence, we frame our problem in such a way that we need not have the ground
truth information about the prominent anchor point directly as part of the
training data set. We treat the predicted probabilities as a confidence score
for each of the anchor points. We combine this information in our loss with
the relative offsets. Furthermore, we experiment by considering the nearest
anchor point to the image, as the ground truth, and incorporate the
corresponding cross entropy loss in our overall loss function. We report
results for both cases.

The Anchor Point Classification head is obtained by mapping the global average
pooling output of the feature extractor to the number of predefined classes
using a Fully Connected (FC) layer. In our case, the number of classes is
equal to the number of anchor points defined during preprocessing. The output
of the FC layer is then subjected to a softmax classifier to get a probability
mapping. However, instead of using these values as probability predictions, we
use them as confidence scores in our experiments as will be explained in more
detail in Section 3.1.

#### Relative Offset Regression.

Along with anchor point classification, our system also produces the relative
offsets from the anchor point. We train these offsets through the regression
loss. However, we face the same problem of not having the ground truth for the
most relevant anchor point. Hence our system produces the relative offsets
with respect to all the anchor points and our model tries to regress these
relative offsets. Note that since the global coordinates of all the anchor
points are known, the relative offset ground truth information for all the
images can also be calculated from the global ground truth information. The
output of the GAP layer of the feature extractor is passed through a fully
connected neural network layer to obtain the regressing head whose task is to
predict the X,Y Coordinate offset values relative to each anchor point defined
previously. Thus if there were $N$ anchor points, say, the output of this head
would be a $2N$ dimensional vector.

#### Absolute Offset Regression for Z and Pose.

Our approach is motivated from human scene recognition through relative
identification. Hence, having the anchor points can be efficient only for the
X and Y coordinates and not for the Z and camera angle coordinates. Predicting
relative offsets for Z coordinate and pose is not carried out due to the
following reasons:

  1. 1.

The Z coordinate and the camera pose of a scene is independent of the previous
scene. It is thus counter intuitive and does not capture the continuity of the
scene like the X,Y coordinates do.

  2. 2.

Regressing relative offset for the Z coordinate as well as the pose
complicates the regression task further thereby leading to inferior results as
we found out in our experiments.

Hence, our model separately predicts the the remaining 4-DOF in accordance
with the global coordinate system. The third and final head is assigned the
task of predicting the absolute values of the Z coordinate and the remaining
DOF values for determining the camera pose. Therefore, the GAP layer of the
feature extractor is mapped to a 5 dimensional vector in this case. This head
also consists of a fully connected layer for regressing the absolute values of
the remaining DOF (i.e Z coordinate and pose).

###  3.1 Loss Function

We train our model with a loss function which is the sum of 3 components. Let
us denote the output of the anchor point classification head to be $\hat{C}$,
which gives confidence scores for each of the $N$ anchor points, for the
current input image. $X,Y$ denotes the vectors of dimension $N$ of ground
truth offsets of the frame in the horizontal plane, with respect to each of
the $N$ anchor points and $Z$, the vertical distance. The ground truths $X,Y$,
though not provided with the dataset, can easily be computed as a
preprocessing step, by changing the origin of the coordinate system to the
respective anchor points. $P$ denote the three angles of the camera. Let
$\hat{X},\hat{Y},\hat{Z},\hat{P}$, be the predictions of the model.

Since we do not have ground truths for the most relevant anchor point in the
image, we cannot train the anchor point classification head with a cross
entropy loss. However, we formulate a joint loss function that uses $\hat{C}$
and the offsets $\hat{X},\hat{Y}$ from each of anchor points. We weight the
squared loss of offsets with confidence scores of the respective anchor
points. The resultant loss function therefore looks like the following:

$\sum_{i}\left[\left(X_{i}-\hat{X}_{i}\right)^{2}+\left(Y_{i}-\hat{Y}_{i}\right)^{2}\right]\hat{C}_{i}$
(1)

This is motivated by the fact that the accuracy is determined by the offset of
the most relevant anchor point. If an anchor point is completely irrelevant
($\hat{C}_{i}=0$), then we do not need the corresponding offset predictions.

Now, we include the predictions of the absolute regressor which is responsible
for predicting the Z coordinate values and the remaining angular DOF for
determining the pose. In order to do this, add the following component to the
loss function:

$\sum_{i}\left[\left(Z_{i}-\hat{Z}_{i}\right)\right]^{2}+\left\|P_{i}-\frac{\hat{P_{i}}}{\left\|\hat{P_{i}}\right\|}\right\|^{2}$
(2)

Additionally, we also experiment with adding a cross entropy loss with the
ground truth being nearest anchor point.

$\text{Confidence loss}=H\left(C_{i},\hat{C}_{i}\right)$ (3)

All the 3 components of our loss function mentioned in equations 1, 2, 3 are
then assigned weights (hyperparameter) and the summed up to get the overall
loss for our task. Therefore, we now have the following resultant loss
function:

$\alpha_{1}H(C_{i},\hat{C}_{i})+\alpha_{2}\sum_{i}\left[\left(X_{i}-\hat{X}_{i}\right)^{2}+\left(Y_{i}-\hat{Y}_{i}\right)^{2}\right]C_{i}+\alpha_{3}\sum_{i}\left[\left(Z_{i}-\hat{Z}_{i}\right)\right]^{2}+\left\|P_{i}-\frac{\hat{P_{i}}}{\left\|\hat{P_{i}}\right\|}\right\|^{2}$
(4)

where $\alpha_{1},\alpha_{2},\alpha_{3}$ are weights assigned to the separate
components of the loss function.

##  4 Dataset and Experiments

We benchmark our method on an outdoor and as well as indoor localization
dataset. A summary of the datasets can be found in Table 1. We rescale all the
images to $224\times 224$ since our convolutional feature extractors are
trained on those dimensions.

As described in Section 3, we initialize our feature extractor (DenseNet
architecture) pretrained on the Imagenet 1K dataset trained on the image
classification task. This gives better performance for outdoor scenes than
using a network pre-trained on the Places dataset as shown in earlier work by
Kendall et.al [Kendall et al.(2015)Kendall, Grimes, and Cipolla].

Table 1: Summary of localization datasets used for benchmarking.

Dataset |  | Type
---
| Scale
---
| Imagery
---
| Scenes
---
| Train Images
---
| Test Images
---
| Spatial Area
---
Cambridge Landmarks [Kendall et al.(2015)Kendall, Grimes, and Cipolla] | Outdoor | Street | Mobile phone camera | 6 | 8,380 | 4,841 | 100 $\times$ 500 m
7 Scenes [Shotton et al.(2013b)Shotton, Glocker, Zach, Izadi, Criminisi, and Fitzgibbon] | Indoor | Room | RGB-D sensor (Kinect) | 7 | 26,000 | 17,000 | 4 $\times$ 3 m

As a pre-processing step, we first divide the scene space into several anchor
points by uniformly selecting every n-th frame from among all the frames in a
scene video. The distance of x,y coordinates for each image is then calculated
relatively from each of these anchor points. This is the ground truth
information for the Relative Offset Regressor. We illustrate some of the
selected anchor points for 3 scenes, King’s College, Shop Facade and Street in
Figure 4.

Our proposed loss function has 3 separate components. We assign separate
weights to each of these components, all of which are hyperparameters, and are
subject to be optimized through grid-search or any other hyperparameter
optimization technique. We found specific ranges for these weights for outdoor
scene localization task. The first part of the loss function is the cross-
entropy calculated between the classifier output and the nearest anchor point
to an input scene, which acts as its label. For this classification loss, we
use a weight value ranging between 1-3 across all the scenes. The second
component of the loss function is the Relative Offset (Translation) Regressor
which predicts the relative distance of the x,y coordinates of the input scene
from each of the anchor points. As a weight for this component, we use a value
ranging between 4-30. Finally, the pose regressor predicts the remaining 5-DOF
and a weight of 0.1-2 is assigned to it.

We use a learning rate varying from 0.00005 to 0.0005 for the scenes in the
dataset. The Adam optimizer is used for optimization with a scheduler to decay
the learning rate by half after every 30th epoch. The evaluation of accuracy
is a subjective procedure. We consider a prediction for the scene to be
accurate if the prediction for position is <2m and the corresponding pose is
<5∘. Other thresholds for accuracy in evaluation will naturally produce
variations in performance calculation.

##  5 Results and Discussions

For validating the effectiveness of our approach, we first compare the median
errors for 6-DOF pose in the spatial and angular dimensions using our
approach, as well as the previous best PoseNet model [Kendall et
al.(2015)Kendall, Grimes, and Cipolla] . For fairness of comparison, we
experiment with using the same GoogleNet [Szegedy et al.(2015a)Szegedy, Liu,
Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and Rabinovich] feature
extractor as the PoseNet result. Secondly, we also experiment with newer
feature extractors like DenseNet [Huang et al.()Huang, Liu, Van Der Maaten,
and Weinberger] and MobileNet [Howard et al.(2017)Howard, Zhu, Chen,
Kalenichenko, Wang, Weyand, Andreetto, and Adam] to drastically improve the
accuracy and runtime speed. This also demonstrates that our approach
generalizes to a variety of feature extractors. Furthermore, we provide a more
detailed analysis of anchor points as well as qualitative results.

#### Comparison with PoseNet on GoogleNet.

We compare our method with PoseNet. We use the same GoogleNet feature
extractor used in the PoseNet results, so that we can observe the improvement
due to our approach rather than using a improved feature extractor. We report
results without the cross entropy loss applied to the classification head
since letting the network discover relevant the anchor points gave better
results. We report the median error for all the scenes in the Cambridge
Landmarks and the 7 Scenes datasets in Table 2. Also, it can be seen from
Table 2, that we are doing better than the previous best method of PoseNet
(with geometric reprojection loss) [Kendall and Cipolla(2017)] in all the
scenes, and improve the localization median error by around 8m in the Street
scene. We also improve upon the previous best performance for each of the
indoor scenes in the 7 Scenes dataset.

Table 2: Comparison of median error for 6-DOF pose. As reported in the last
column, our model performs better than the best deep learning model PoseNet
[Kendall and Cipolla(2017)], making the gap with active search methods
[Sattler et al.(2017)Sattler, Leibe, and Kobbelt] (which uses 3D point cloud
data unlike us) lesser. Use of improved feature extractors like DenseNet
reduces the errors further.

Scene |  | Area or
---
Volume
| Active Search
---
(SIFT) [Sattler et al.(2017)Sattler, Leibe, and Kobbelt]
| Posenet
---
Spatial LSTM[Walch et al.(2016)Walch, Hazirbas, Leal-Taixé, Sattler,
Hilsenbeck, and Cremers]
| Posenet
---
sigma2 weight [Kendall and Cipolla(2017)]
| Posenet
---
Geom. Rep. [Kendall and Cipolla(2017)]
| Ours (DenseNet)
---
(cross entropy)
| Ours (DenseNet)
---
(w/o cross entropy)
| Ours (GoogleNet)
---
(w/o cross entropy)
Great Court | 8000$m^{2}$ | - | - | 7.00$m$, 3.65∘ | 6.83$m$, 3.47∘ | 5.85$m$, 3.61∘ | 4.64$m$, 3.42∘ | 5.89$m$, 3.53∘
King’s College | 5600$m^{2}$ | 0.42$m$, 0.55∘ | 0.99$m$, 3.65∘ | 0.99$m$, 1.06∘ | 0.88$m$, 1.04∘ | 0.55$m$, 0.97∘ | 0.57$m$, 0.88∘ | 0.79$m$, 0.95∘
Old Hospital | 2000$m^{2}$ | 0.44$m$, 1.01∘ | 1.51$m$, 4.29∘ | 2.17$m$, 2.94∘ | 3.20$m$, 3.29∘ | 1.45$m$, 3.16∘ | 1.21$m$, 2.55∘ | 2.11$m$, 3.05∘
Shop Facade | 875$m^{2}$ | 0.12$m$, 0.40∘ | 1.18$m$, 7.44∘ | 1.05$m$ 3.97∘ | 0.88$m$, 3.78∘ | 0.49$m$, 2.42∘ | 0.52$m$, 2.27∘ | 0.77$m$, 3.25∘
St. Mary’s Church | 4800$m^{2}$ | 0.19$m$, 0.54∘ | 1.52$m$, 6.68∘ | 1.49$m$, 3.43∘ | 1.57$m$, 3.32∘ | 1.12$m$, 2.84∘ | 1.04$m$, 2.69∘ | 1.22$m$, 3.02∘
Street | 50000$m^{2}$ | 0.85$m$, 0.83∘ | - | 20.7$m$, 25.7∘ | 20.3$m$, 25.5∘ | 8.19$m$, 25.5∘ | 7.86$m$, 24.2∘ | 11.8$m$, 24.3∘
Chess | 6$m^{2}$ | 0.04$m$, 1.96∘ | 0.24$m$, 5.77∘ | 0.14$m$, 4.50∘ | 0.13$m$, 4.48∘ | 0.06$m$, 3.95∘ | 0.06$m$, 3.89∘ | 0.08$m$, 4.12∘
Fire | 2.5$m^{2}$ | 0.03$m$, 1.53∘ | 0.34$m$, 11.9∘ | 0.27$m$, 11.8∘ | 0.27$m$, 11.3∘ | 0.16$m$, 10.4∘ | 0.15$m$, 10.3∘ | 0.16$m$, 11.1∘
Head | 1$m^{2}$ | 0.02$m$, 1.45∘ | 0.21$m$, 13.7∘ | 0.18$m$, 12.1∘ | 0.17$m$, 13.0∘ | 0.08$m$, 10.7∘ | 0.08$m$, 10.9∘ | 0.09$m$, 11.2∘
Office | 7.5$m^{2}$ | 0.09$m$, 3.61∘ | 0.30$m$, 8.08∘ | 0.20$m$, 5.77∘ | 0.19$m$, 5.55∘ | 0.11$m$, 5.24∘ | 0.09$m$, 5.15∘ | 0.11$m$, 5.38∘
Pumpkin | 5$m^{2}$ | 0.08$m$, 3.10∘ | 0.33$m$, 7.00∘ | 0.25$m$, 4.82∘ | 0.26$m$, 4.75∘ | 0.11$m$, 3.18∘ | 0.10$m$, 2.97∘ | 0.14$m$, 3.55∘
Red Kitchen | 18$m^{2}$ | 0.07$m$, 3.37∘ | 0.37$m$, 8.83∘ | 0.24$m$, 5.52∘ | 0.23$m$, 5.35∘ | 0.08$m$, 4.83∘ | 0.08$m$, 4.68∘ | 0.13$m$, 5.29∘
Stairs | 7.5$m^{2}$ | 0.03$m$, 2.22∘ | 0.40$m$, 13.7∘ | 0.37$m$, 10.6∘ | 0.35$m$, 12.4∘ | 0.13$m$, 10.1∘ | 0.10$m$, 9.26∘ | 0.21$m$, 11.9∘

#### Improved Accuracy using DenseNet and Abalation study.

We then experiment with the state of the art CNN feature extractors. DenseNet
[Huang et al.()Huang, Liu, Van Der Maaten, and Weinberger] is known to give
improved accuracies for image classification. In Table 2, we report the
results of our proposed method with the DenseNet feature extractor and compare
with all the previous methods. We report the results with and without the
cross entropy term (see Section 3.1). For the Cambridge Landmarks, 5 out of
the 6 scenes, the model performed better without the cross entropy loss,
validating our approach of discovering the appropriate anchor point relative
to which offsets needs to be calculated.

Table 3: We report the mean and median distances for each scene of the
Cambridge Landmarks dataset using the DenseNet feature extractor. The accuracy
is calculated by considering the orientation localization within 2m and the
pose localization with 5∘ to be a correct prediction.

Scene |  |  Mean
---
Distance
|  Median
---
Distance
|  Accuracy
---
(<2m,<5∘)
|  #Anchor
---
Points
Great Court | 10.48m | 5.85m | 69.51 % | 154
King’s College | 0.76m | 0.57m | 93.52 % | 122
Old Hospital | 2.93m | 1.45m | 85.94 % | 110
Shop Facade | 0.72m | 0.52m | 93.76 % | 33
| St. Mary’s
---
Church
1.62m | 1.12m | 88.95 % | 146
Street | 17.45m | 9.89m | 11.26 % | 201

In Table 3, we also report the accuracy, the mean and the median distance
localized to in orientation, and the median camera angle pose localization for
each of the 6 scenes of the Cambridge Landmarks dataset. It can be seen from
Table 3, that the accuracy for the Street scene is 11.26%. The Street scene is
the most challenging scene from among the used scenes for the visual
localization task since it covers more than $50000m^{2}$ in area and also
contains multiple landmarks. We achieve state-of-the-art performance for
translation and pose on the Street scene as well. Our method is able to
significantly improve the translation and orientation localization for this
particular scene from 20.3m to 7.86m (see Table 2) which is a considerable
improvement.

Next, we perform an experiment with DenseNet as the feature extractor
connected to a 6-DOF regressor directly. This is a sort of control for our
proposed method and we do not use an anchor point classifier here. We provide
a comparison of our proposed method and directly using the regressor in the
Cambridge Landmarks dataset. It is observed that our proposed method is able
to perform much better than simply regressing the DOF from the DenseNet
feature extractor. The results are summarized in Table 4. As can be seen our
method gives lesser errors. This verifies that the improvement in median
errors is indeed happening due to our approach and not due to the feature
extractors only.

Table 4: Comparison between the DenseNet feature extractor followed by a
simple regressor and our proposed method.

Scene |  | Median Dist.
---
(DenseNet + DOF
Regressor)
| Median Dist.
---
(Our method)
| Accuracy
---
(DenseNet +
DOF Regressor)
| Accuracy
---
(Our Method)
Shop Facade | 1.32m | 0.52m | 82.64% | 93.76%
King’s College | 1.45m | 0.57m | 81.80% | 93.52%

#### Runtime Analysis.

The MobileNet [Howard et al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang,
Weyand, Andreetto, and Adam] feature extractor is known to be significantly
faster while maintaining accuracies. We illustrate the results when using it
in Table 5. For some scenes in Cambridge Landmarks dataset, we empirically
compare the performance and efficiency of 3 popular feature extractors, namely
GoogleNet, DenseNet and MobileNet, when used with our proposed method. We
observe from Table 5 that MobileNet provide a considerable better runtime
performance, while still giving errors lesser than GoogleNet.

Table 5: Comparison of different feature extractors on the basis of
performance and FLOPs using our proposed anchor point based method for visual
relocalization on the Cambridge landmarks dataset.

Scene |  | DenseNet
---
(Feature Extractor)
| GoogleNet
---
(Feature Extractor)
| MobileNet
---
(Feature Extractor)
| Performance | FLOPs | Performance | FLOPs | Performance | FLOPs
Kings College | 0.57$m$, 0.88∘ | 5998 M | 0.79$m$, 0.95∘ | 760 M | 0.67$m$, 0.94∘ | 569 M
Shop Facade | 0.52$m$, 2.27∘ | 0.77$m$, 3.25∘ | 0.60$m$, 2.31∘

#### Analysis of Anchor Points and Qualitative Results.

An important hyperparameter for our approach is the number of frames between
consecutive anchor point which is required for assigning anchor points as a
preprocessing step. The outcome for this selection also determines the number
of classes the classifier should predict since it is equal to the number of
anchor points. We therefore plot the behavior of the median distance
localization for translation and the overall accuracy, with the varying frame
number selection (say $k$). This means that for a particular scene, we select
every $k^{\text{th}}$ frame. We observe in Figure 3, that we are able to get
an optimum accuracy for a specific frame number which in turn makes the task
easy of deciding the number of classes since it depends on $k$.

Figure 3: Plots showing how accuracy and median distance varies with different
choices of frames interval between anchor points. We chose the optimum frame
number for the dataset for anchor point selection.

Finally, we showcase some qualitative results for the Cambridge Landmark
scenes as well as for the 7 scenes dataset. It can demonstrated from those
results that not only is the learned anchor point different from the nearest
one but also better, in terms of camera angle and in avoiding occlusion.

Figure 4:  (Left) We contrast the nearest anchor point and the learned anchor
point for an input query for the Cambridge dataset. Note that for the Old
Hospital scene, the relevant anchor point learned is not blocked by a tree
while the nearest anchor point is. Generalizing, learned anchor points gives a
clear view of a landmark as compared to the nearest anchor point, validating
our approach of discovering the relevant anchor point. (Right) Learned anchor
points from the 7 Scenes dataset. In this case we observe a more zoomed in
version of the input image is learned as the reference anchor point.

##  6 Conclusion

We propose a novel approach for solving the visual relocalization problem,
inspired by how humans estimate their location, by observing suitable
landmarks. We model it as a multi task problem of classification and relative
offset regression. We propose a deep learning architecture and loss function
which automatically discovers the anchor point relative to which accurate
offset estimates can be predicted. We do not require each image to be tagged
with the relevant landmark to train the classification part. Through our
experiments, we show that our method achieves an improvement over PoseNet and
its extensions in all scenes of the Cambridge Landmarks dataset as well as the
indoor scenes of the 7 Scenes dataset. We achieve 1.5m and 4∘ in localization
performance in 4 out of the 6 outdoor scenes in Cambridge Landmarks and 0.2m
localization for the 7 indoor Scenes. Furthermore, our method outperforms
simple replacement of the feature extractor followed by regression which
further showcases the advantages of an anchor point classification and
relative offset regression based method for the visual localization task.

## References

  * [Arandjelović and Zisserman(2014)] Relja Arandjelović and Andrew Zisserman.  Dislocation: Scalable descriptor distinctiveness for location recognition.  In _ACCV_ , pages 188–204. Springer, 2014.
  * [Babenko and Lempitsky(2015)] Artem Babenko and Victor Lempitsky.  Aggregating local deep features for image retrieval.  In _ICCV_ , pages 1269–1277, 2015.
  * [Babenko et al.(2014)Babenko, Slesarev, Chigorin, and Lempitsky] Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky.  Neural codes for image retrieval.  In _ECCV_ , pages 584–599. Springer, 2014.
  * [Choudhary and Narayanan(2012)] Siddharth Choudhary and PJ Narayanan.  Visibility probability structure from sfm datasets and applications.  In _ECCV_ , pages 130–143. Springer, 2012.
  * [Cummins and Newman(2008)] Mark Cummins and Paul Newman.  Fab-map: Probabilistic localization and mapping in the space of appearance.  _The International Journal of Robotics Research_ , 27(6):647–665, 2008.
  * [Gong et al.(2014)Gong, Wang, Guo, and Lazebnik] Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik.  Multi-scale orderless pooling of deep convolutional activation features.  In _ECCV_ , pages 392–407. Springer, 2014.
  * [Howard et al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang, Weyand, Andreetto, and Adam] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _CoRR_ , abs/1704.04861, 2017.
  * [Huang et al.()Huang, Liu, Van Der Maaten, and Weinberger] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks. in cvpr, 2017.
  * [Jégou et al.(2012)Jégou, Perronnin, Douze, Sánchez, Pérez, and Schmid] Hervé Jégou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Pérez, and Cordelia Schmid.  Aggregating local image descriptors into compact codes.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 9(34):1704–1716, 2012.
  * [Kendall and Cipolla(2016)] Alex Kendall and Roberto Cipolla.  Modelling uncertainty in deep learning for camera relocalization.  In _Robotics and Automation, ICRA, 2016_ , pages 4762–4769. IEEE, 2016.
  * [Kendall and Cipolla(2017)] Alex Kendall and Roberto Cipolla.  Geometric loss functions for camera pose regression with deep learning.  In _CVPR_ , 2017.
  * [Kendall et al.(2015)Kendall, Grimes, and Cipolla] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In _ICCV_ , pages 2938–2946, 2015.
  * [Li et al.(2017)Li, Liu, Gui, Gu, and Hu] Ruihao Li, Qiang Liu, Jianjun Gui, Dongbing Gu, and Huosheng Hu.  Indoor relocalization in challenging environments with dual-stream convolutional neural networks.  _IEEE Transactions on Automation Science and Engineering_ , 2017.
  * [Li et al.(2010)Li, Snavely, and Huttenlocher] Yunpeng Li, Noah Snavely, and Daniel P Huttenlocher.  Location recognition using prioritized feature matching.  In _ECCV_ , pages 791–804. Springer, 2010.
  * [Li et al.(2016)Li, Snavely, Huttenlocher, and Fua] Yunpeng Li, Noah Snavely, Daniel P Huttenlocher, and Pascal Fua.  Worldwide pose estimation using 3d point clouds.  In _Large-Scale Visual Geo-Localization_ , pages 147–163. Springer, 2016.
  * [Melekhov et al.(2017)Melekhov, Ylioinas, Kannala, and Rahtu] Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu.  Relative camera pose estimation using convolutional neural networks.  In _International Conference on Advanced Concepts for Intelligent Vision Systems_ , pages 675–687. Springer, 2017.
  * [Piasco et al.(2018)Piasco, Sidibé, Demonceaux, and Gouet-Brunet] Nathan Piasco, Désiré Sidibé, Cédric Demonceaux, and Valérie Gouet-Brunet.  A survey on visual-based localization: On the benefit of heterogeneous data.  _Pattern Recognition, PR_ , 74:90–109, 2018.
  * [Razavian et al.(2016)Razavian, Sullivan, Carlsson, and Maki] Ali S Razavian, Josephine Sullivan, Stefan Carlsson, and Atsuto Maki.  Visual instance retrieval with deep convolutional networks.  _ITE Transactions on Media Technology and Applications_ , 4(3):251–258, 2016.
  * [Sattler et al.(2012)Sattler, Leibe, and Kobbelt] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Improving image-based localization by active correspondence search.  In _ECCV_ , pages 752–765. Springer, 2012.
  * [Sattler et al.(2017)Sattler, Leibe, and Kobbelt] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Efficient & effective prioritized matching for large-scale image-based localization.  _IEEE Transactions on Pattern Analysis & Machine Intelligence, TPAMI_, (9):1744–1756, 2017.
  * [Shotton et al.(2013a)Shotton, Glocker, Zach, Izadi, Criminisi, and Fitzgibbon] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon.  Scene coordinate regression forests for camera relocalization in rgb-d images.  In _CVPR_ , pages 2930–2937, 2013a.
  * [Shotton et al.(2013b)Shotton, Glocker, Zach, Izadi, Criminisi, and Fitzgibbon] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon.  Scene coordinate regression forests for camera relocalization in rgb-d images.  In _CVPR_ , pages 2930–2937, 2013b.
  * [Svarm et al.(2014)Svarm, Enqvist, Oskarsson, and Kahl] Linus Svarm, Olof Enqvist, Magnus Oskarsson, and Fredrik Kahl.  Accurate localization and pose estimation for large 3d models.  In _CVPR_ , pages 532–539, 2014.
  * [Szegedy et al.(2015a)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and Rabinovich] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _CVPR_ , pages 1–9, 2015a.
  * [Szegedy et al.(2015b)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and Rabinovich] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _CVPR_ , pages 1–9, 2015b.
  * [Tolias et al.(2015)Tolias, Sicre, and Jégou] Giorgos Tolias, Ronan Sicre, and Hervé Jégou.  Particular object retrieval with integral max-pooling of cnn activations.  _arXiv preprint arXiv:1511.05879_ , 2015.
  * [Torii et al.(2013)Torii, Sivic, Pajdla, and Okutomi] Akihiko Torii, Josef Sivic, Tomas Pajdla, and Masatoshi Okutomi.  Visual place recognition with repetitive structures.  In _CVPR_ , pages 883–890, 2013.
  * [Walch et al.(2016)Walch, Hazirbas, Leal-Taixé, Sattler, Hilsenbeck, and Cremers] Florian Walch, Caner Hazirbas, Laura Leal-Taixé, Torsten Sattler, Sebastian Hilsenbeck, and Daniel Cremers.  Image-based localization with spatial lstms.  _ICCV_ , 2016.
  * [Weyand et al.(2016)Weyand, Kostrikov, and Philbin] Tobias Weyand, Ilya Kostrikov, and James Philbin.  Planet-photo geolocation with convolutional neural networks.  In _ECCV_ , pages 37–55. Springer, 2016.