# Field line subdiffusion and cosmic ray perpendicular transport in isotropic
turbulence

Marco Kuhlen  Philipp Mertsch  Vo Hong Minh Phan  Institute for Theoretical
Particle Physics and Cosmology (TTK),
RWTH Aachen University, 52056 Aachen, Germany

###### Abstract

The transport of high-energy particles in the presence of small-scale,
turbulent magnetic fields is a long-standing issue in astrophysics. Analytical
theories disagree with numerical simulations at rigidities where the
particles’ gyroradii are slightly smaller than the correlation length of
turbulence. At the same time, extending the numerical simulations to lower
rigidities has proven computationally prohibitive. In this letter, we provide
a solution to the problem of perpendicular transport in isotropic turbulence
at both, high and low rigidities. We also clarify the relation between the
perpendicular diffusion of particles and the transport of magnetic field
lines. To this end, we have run a large suite of test particle simulations at
unprecedentedly low rigidities, making extensive use of graphical processing
units (GPUs). We have also developed an analytical model, based on (1) initial
particle transport along field lines, (2) the transport of field lines and (3)
the eventual decorrelation of particles from field lines. Our numerical
results exhibit a non-standard rigidity-dependence for the perpendicular
diffusion coefficient at intermediate rigidities. At the lowest rigidities,
the standard rigidity-dependence is recovered. The simulated diffusion
coefficients are nicely reproduced by our analytical model. We have traced the
non-standard rigidity-dependence to a subdiffusive phase in the field line
transport. Our study has important implications for the transport of Galactic
cosmic rays, acceleration at perpendicular shocks and for high-energy
particles in the heliosphere.

††preprint: TTK-22-36

Much of the universe is pervaded by magnetic fields, oftentimes turbulent, and
high-energy charged particles, cosmic rays (CRs). The transport of charged
particles in the presence of turbulent magnetic fields is a fundamental
question in astrophysics. The presence of a large-scale, coherent magnetic
field breaks the isotropy and leads to the differentiation of particle
transport along and across this large-scale field. Transport along the field
has been understood a while ago to be the result of initially ballistic
gyrations, followed by pitch-angle scattering due to resonant interactions
Jokipii (1966); Kennel and Engelmann (1966); Hall and Sturrock (1967).
Eventually, pitch-angle scattering results in spatial diffusion along the
large-scale magnetic field Hasselmann and Wibberenz (1970). Perpendicular
transport has proven much more difficult to model, but is generally believed
to be the combination of transport along field lines and transport of the
field lines Shalchi (2009, 2020).

Analytical models can be compared to so-called test particle simulations.
Those are based on solving the equations of motion of individual particles in
a turbulent magnetic field simulated on a computer. (See Ref. Mertsch (2020)
for a review.) At low reduced rigidities $r_{\text{g}}/L_{\text{c}}$ (where
$r_{\text{g}}$ is the gyroradius and $L_{\text{c}}$ is the correlation length
of turbulence Harari  _et al._ (2002)), particle trajectories have to be
followed for very long times before particle motion becomes diffusive.
Previously, simulations have only been possible at relatively large rigidities
of $r_{\text{g}}/L_{\text{c}}\gtrsim 10^{-2}$ Giacalone and Jokipii (1999);
Casse  _et al._ (2001); Qin  _et al._ (2002a, b); DeMarco  _et al._ (2007);
Giacinti  _et al._ (2012); Snodin  _et al._ (2016); Subedi  _et al._ (2017);
Giacinti  _et al._ (2018); Schlegel  _et al._ (2020); Reichherzer  _et al._
(2020, 2022). For parameters typical of the interstellar medium, e.g. an rms
field strength of $4\,\mu\text{G}$ and $L_{\text{c}}=30\,\text{pc}$, this
corresponds to rigidities $\mathcal{R}\simeq 1\,\text{PV}$.

Recently, it has become apparent that none of the analytical theories
developed over the last decades, e.g. Matthaeus  _et al._ (2003); Shalchi
(2010), can describe the results of test particle simulations for isotropic
turbulence Dundovic  _et al._ (2020). For instance, these theories typically
predict a rather weak, but falling dependence of the ratio of perpendicular
and parallel mean free paths, $\lambda_{\perp}/\lambda_{\parallel}$ on
$r_{\text{g}}/L_{\text{c}}$. Instead, in simulations, the ratio increases for
$r_{\text{g}}/L_{\text{c}}\ll 1$ and drops sharply for
$r_{\text{g}}/L_{\text{c}}\gg 1$ DeMarco  _et al._ (2007); Snodin  _et al._
(2016); Dundovic  _et al._ (2020). This failure of analytical theories is most
worrisome and casts doubt on our general understanding of perpendicular
transport. Furthermore, if the difference in the rigidity-scaling of
$\lambda_{\parallel}$ and $\lambda_{\perp}$ were confirmed, this would have
important consequences for phenomenological applications of such models, for
instance in modelling of Galactic cosmic rays and diffuse emission Evoli  _et
al._ (2012); Cerri  _et al._ (2017); Giacinti  _et al._ (2018); Reichherzer
_et al._ (2022).

We have run a large suite of test particle simulations with a field composed
of a large-scale, coherent magnetic field, taken to point in the $z$-direction
and a small-scale, static, turbulent magnetic field $\bm{\delta B}(\bm{r})$.
The turbulent modes are isotropically distributed and follow a Kolmogorov
Kolmogorov (1941) power spectrum $P(k)$ with an outer scale $L_{\text{max}}$,
i.e.
$P(k)\equiv\langle\bm{\delta\tilde{B}}(\bm{k})\bm{\delta\tilde{B}}^{*}(\bm{k})\rangle\propto
k^{-5/3}$ for $k>(2\pi)/L_{\text{max}}$. Here, angled brackets denote averages
over an ensemble of turbulent magnetic field. The correlation length
$L_{\text{c}}\simeq L_{\text{max}}/5$ and the overall turbulent strength is
parametrised by $\eta\equiv\delta B^{2}/(B_{0}^{2}+\delta B^{2})$ with $\delta
B^{2}\equiv 8\pi\int_{0}^{\infty}\mathrm{d}k\,k^{2}P(k)$. We have employed the
nested grid method Giacinti  _et al._ (2012, 2018); Mertsch (2020) for setting
up a large enough number of individual realisations of $\bm{\delta B}$ on a
computer. We have made use of graphical processing units (GPUs) that allow for
a significant speed up Tautz (2016); Kuhlen  _et al._ (2022a). From the
particle trajectories, $\\{x(t),y(t),z(t)\\}$, we have computed running
parallel and perpendicular diffusion coefficients Shalchi (2009),

$\displaystyle d_{\parallel}(t)$
$\displaystyle\equiv\frac{1}{2}\frac{\mathrm{d}}{\mathrm{d}t}\langle(z(t)-z(0))^{2}\rangle\,,$
(1) $\displaystyle d_{\perp}(t)$
$\displaystyle\equiv\frac{1}{4}\frac{\mathrm{d}}{\mathrm{d}t}\left(\langle(x(t)-x(0))^{2}\rangle+\langle(y(t)-y(0))^{2}\rangle\right)\,.$
(2)

Here, the angled brackets denote the average over initial particle directions
and realisations of the turbulent magnetic field.

We have also computed the magnetic field lines $\\{x(s),y(s),z(s)\\}$,
parametrised by their arc-length $s$, by solving the field line equations for
individual realisations of $\bm{\delta B}$. From this, we have computed their
perpendicular displacement $\langle(\Delta
r^{\text{FL}}_{\perp})^{2}(s)\rangle\equiv\left(\langle(x(s)-x(0))^{2}\rangle+\langle(y(s)-y(0))^{2}\rangle\right)/2$
and the running field line (FL) diffusion coefficient Shalchi (2009),

$d_{\text{FL}}(s)\equiv\frac{1}{2}\frac{\mathrm{d}}{\mathrm{d}s}\left(\langle\Delta
r^{2}_{\perp}\rangle\right)\,.$ (3)

In a companion paper Kuhlen  _et al._ (2022b), we provide a detailed
discussion of the method and possible systematic errors as well as the time-
dependent running diffusion coefficients.

Our analytical model is based on the same principles as earlier works Bieber
and Matthaeus (1997); Matthaeus  _et al._ (2003); Qin  _et al._ (2002b);
Shalchi  _et al._ (2004); Shalchi (2019), but with a crucial difference. (1)
Initially, particles follow magnetic field lines, at first ballistically,
later diffusively. (2) The transport of field lines is also transitioning from
an early ballistic phase to an eventual diffusive regime. At variance with
earlier studies, however, we allow for the possibility of a subdiffusive
episode in field line transport where $\langle(\Delta
r^{\text{FL}}_{\perp})^{2}(s)\rangle\propto s^{\alpha}$ with $\alpha<1$. (3)
Eventually, as the particle displacement in the perpendicular direction
exceeds about a correlation length, particles can be considered decoupled from
individual field lines and start to diffuse Shalchi (2019).

We employ the above definition of the running perpendicular diffusion
coefficient, eq. (2), and the running FL diffusion coefficient, eq. (3).
Integrating eq. (3),

$\langle(\Delta
r^{\text{FL}}_{\perp})^{2}(z)\rangle=2\int_{0}^{z}\mathrm{d}z^{\prime}\,d_{\text{FL}}(z^{\prime})\,,$
(4)

identifying $\Delta r^{\text{CR}}_{\perp}(t)=\Delta
r^{\text{FL}}_{\perp}(z(t))$ and substituting eq. (4) into eq. (2) leads to

$d_{\perp}(t)=\frac{\mathrm{d}}{\mathrm{d}t}\int_{0}^{z(t)}\mathrm{d}z^{\prime}d_{\text{FL}}(z^{\prime})\,.$
(5)

Next, we evaluate $z$ with $\sqrt{\langle z^{2}\rangle}$,

$\displaystyle
d_{\perp}(t)\\!=\\!\frac{\mathrm{d}}{\mathrm{d}t}\\!\int_{0}^{\sqrt{\langle
z^{2}\rangle}}\\!\\!\\!\mathrm{d}z^{\prime}d_{\text{FL}}(z^{\prime})\\!=\\!\frac{d_{\text{FL}}(\\!\sqrt{\langle(\Delta
z)^{2}\rangle})}{2\sqrt{\langle(\Delta z)^{2}\rangle}}d_{\parallel}(t),$ (6)

with the running parallel diffusion coefficient

$d_{\parallel}(t)\\!=\\!\frac{1}{2}\frac{\mathrm{d}}{\mathrm{d}t}\langle(\Delta
z)^{2}(t)\rangle\Leftrightarrow\langle(\Delta
z)^{2}(t)\rangle\\!=\\!2\int_{0}^{t}\\!\\!\\!\mathrm{d}t^{\prime}d_{\parallel}(t).$
(7)

It is easy to see that this will lead to subdiffusive behaviour for the
running perpendicular diffusion coefficient $d_{\perp}(t)\propto t^{-1/2}$ for
times $t\gg\tau_{\text{d}}$ where particle transport in the parallel direction
is diffusive, $\langle(\Delta z)^{2}\rangle\propto t\Leftrightarrow
d_{\parallel}=\kappa_{\parallel}$ and $z>L_{\text{c}}$ such that field line
transport is also diffusive, that is $d_{\text{FL}}(z)=\kappa_{\text{FL}}$.
However, once particles have travelled more than $\sim L_{\text{c}}$ in the
perpendicular direction they experience the transverse complexity, decouple
from individual field lines and their perpendicular motion becomes diffusive
Shalchi (2019), with the asymptotic diffusion coefficient
$d_{\perp}(\tau_{c})=\kappa_{\perp}=\text{const.}$ The time $\tau_{\text{c}}$
is implicitly defined through

$d_{\perp}(\tau_{\text{c}})\\!=\\!\frac{1}{2}\frac{\mathrm{d}}{\mathrm{d}t}\langle(\Delta
r_{\perp})^{2}(t)\rangle\Big{|}_{t\to\tau_{\text{c}}}\\!\\!\\!\\!\sim\\!\frac{\langle(\Delta
r_{\perp})^{2}(\tau_{\text{c}})\rangle}{2\tau_{\text{c}}}\\!\sim\\!\frac{L_{\text{c}}^{2}}{2\tau_{\text{c}}}.$
(8)

In order to evaluate $\kappa_{\perp}=d_{\perp}(\tau_{c})$ with eqs. (6) and
(8), we need to specify the functional forms for the running parallel and
field line diffusion coefficients, $d_{\parallel}$ and $d_{\text{FL}}$. We
parametrise the time-dependence of $d_{\parallel}$ with a broken power law,

$d_{\parallel}(t)=\kappa_{\parallel}\left(1+\left(t/\tau_{\text{s}}\right)^{-1/s_{d}}\right)^{-s_{d}}$
(9)

where $\tau_{\text{s}}=3\kappa_{\parallel}/v^{2}$ and $s_{d}=1/2$. At early
times, $t\ll\tau_{\text{s}}$, parallel particle transport is ballistic,
$d_{\parallel}(t)\simeq\kappa_{\parallel}t/\tau_{\text{s}}=v^{2}t/3$; at late
times, $t\gg\tau_{\text{s}}$, it turns diffusive,
$d_{\parallel}(t)\simeq\kappa_{\parallel}$.

As our focus here is not on parallel transport, we model the asymptotic
diffusion coefficient $\kappa_{\parallel}$ as a broken power law in
$r_{\text{g}}/L_{\text{c}}$,

$\kappa_{\parallel}(r_{\text{g}}/L_{\text{c}})=A(r_{\text{g}}/L_{\text{c}})^{1/3}\left(1+\left(\frac{r_{\text{g}}/L_{\text{c}}}{\rho_{*}}\right)^{5/(3s_{\kappa})}\right)^{s_{\kappa}}\,.$
(10)

At small rigidities, $r_{\text{g}}/L_{\text{c}}\ll\rho_{*}$, CRs diffuse due
to resonant interactions with turbulent fields and the rigidity-scaling
$\kappa_{\parallel}\propto(r_{\text{g}}/L_{\text{c}})^{1/3}$ is characteristic
for the Kolmogorov turbulence assumed. At large rigidities,
$r_{\text{g}}/L_{\text{c}}\gg\rho_{*}$, no resonant interactions are possible;
instead, particles diffuse in the small-angle scattering limit
$\kappa_{\parallel}\propto(r_{\text{g}}/L_{\text{c}})^{2}$ Gruzinov (2018).
For each turbulence level $\eta$, there are thus three free parameters for
$d_{\parallel}$: $A$, $\rho_{*}$, and $s_{\kappa}$ which are determined by
fitting eq. (10) to the simulated
$\kappa_{\parallel}=(3/v)\lambda_{\parallel}$.

Figure 1: Asymptotic parallel and perpendicular mean free paths
$\lambda_{\parallel}$ and $\lambda_{\perp}$ as a function of reduced rigidity
$r_{\text{g}}/L_{\text{c}}$ for different turbulence levels $\eta$. The dots
and squares are the results from test particle simulations. The dot-dashed and
solid lines show the predictions from our model that includes a subdiffusive
phase in the running field-line diffusion coefficient $d_{\text{FL}}$. The
perpendicular mean-free path without a subdiffusive phase in $d_{\text{FL}}$
is indicated by the dotted lines.

As far as $d_{\text{FL}}$ is concerned, our simulations have shown that field-
line transport not just transitions from ballistic to diffusive behaviour
Kuhlen  _et al._ (2022b). Instead, there is a subdiffusive phase where
$d_{\text{FL}}$ drops. We model such a behaviour with a double-broken power
law:

$d_{\text{FL}}(z)=z\frac{\delta
B_{x}^{2}}{B_{0}^{2}}\left(1+\left(\frac{z}{\kappa_{\text{FL}}\sigma}\frac{\delta
B_{x}^{2}}{B_{0}^{2}}\right)^{1/s_{\text{FL}}}\right)^{-s_{\text{FL}}}\left(1+\left(\frac{z}{\alpha}\right)^{1/s_{\text{FL}}}\right)^{-s_{\text{FL}}}\left(1+\left(\frac{z}{\alpha\sigma}\right)^{1/s_{\text{FL}}}\right)^{s_{\text{FL}}}\,.$
(11)

For small distances, $z/(\kappa_{\text{FL}}\sigma)(\delta
B_{x}^{2}/B_{0}^{2})\ll 1$, field-line transport is ballistic,
$d_{\text{FL}}(z)\simeq z(\delta B_{x}^{2}/B_{0}^{2})$ in line with analytical
models of ballistic field line transport Shalchi (2020). For
$\kappa_{\text{FL}}\sigma/(\delta B_{x}^{2}/B_{0}^{2})\ll z\ll\alpha$, there
is a plateau, $d_{\text{FL}}(z)\simeq\kappa_{\text{FL}}\sigma$. For $\alpha\ll
z\ll(\alpha\sigma)$, field line transport is subdiffusive,
$d_{\text{FL}}(z)\propto 1/z$. Finally, for $z\gg(\alpha\sigma)$, field lines
diffuse again, $d_{\text{FL}}(z)\simeq\kappa_{\text{FL}}$.

The above requirement for transverse complexity is only approximate. Therefore
we replace $L_{\text{c}}$ in eq. (8) with another free parameter,
$L_{\text{c},\perp}$. All in all, there are five free parameters for
$d_{\text{FL}}(z)$ once we have fixed $(\delta
B_{x}^{2}/B_{0}^{2})=(1/3)\eta/(1-\eta)$, however, we have chosen fixed values
of $\kappa_{\text{FL}}=0.0125,0.05,0.2$ for $\eta=0.2,0.5$ and $0.8$,
respectively, in line with the field-line random walk limit Shalchi (2020).
Finally, we have found that $L_{\text{c},\perp}/L_{\text{c}}=0.4,0.5,0.6$ for
$\eta=0.2,0.5$ and $0.8$, respectively gives satisfactory results. The
remaining three parameters (for each $\eta$) are determined by fitting
$\kappa_{\perp}=d_{\perp}(\tau_{c})$ with eqs. (6) and (8) to the simulated
$\lambda_{\perp}$ for all simulated values $r_{\text{g}}/L_{\text{c}}$.

Figure 2: Ratio $\lambda_{\perp}/\lambda{\parallel}$ of the perpendicular and
parallel mean free paths as a function of reduced rigidity
$r_{\text{g}}/L_{\text{c}}$ for different turbulence levels $\eta$. The dots
show the results of test particle simulations. Our model predictions including
a subdiffusive phase in the running field-line diffusion coefficient
$d_{\text{FL}}$ is shown by the solid lines. A model prediction without a
subdiffusive phase in $d_{\text{FL}}$ is indicated by the dotted lines.

Figure 3:  Running field line diffusion coefficients $d_{\text{FL}}$ for the
different turbulence levels $\eta$. The solid lines show the running field-
line diffusion coefficient $d_{\text{FL}}$ (eq. (11)) including a subdiffusive
phase with parameters fitted to the asymptotic perpendicular mean free path
$\lambda_{\perp}$. The dotted line is $d^{\prime}_{\text{FL}}$ (eq. (12))
without a subdiffusive phase. The grey dashed line and shaded band indicate
the results of simulations for field line transport for $\eta=0.5$.

The first main result of this letter are the parallel and perpendicular mean
free paths, $\lambda_{\parallel}$ and $\lambda_{\perp}$ from our test particle
simulations, shown in Fig. 1 by the dots and squares, respectively. The
different shades of blue indicate different turbulence levels $\eta$. The
parallel mean free path follows the form of eq. (10), indicated by the dot-
dashed lines. The perpendicular mean free path, however, exhibits a more
complicated rigidity-dependence: At low rigidities, it has the same scaling as
the parallel one, $\lambda_{\perp}\propto(r_{\text{g}}/L_{\text{c}})^{1/3}$,
but at larger rigidities, the dependence is closer to
$(r_{\text{g}}/L_{\text{c}})^{1/2}$. Eventually, for
$r_{\text{g}}/L_{\text{c}}\gg 1$, $\lambda_{\perp}$ becomes constant. In Fig.
3, we also show the ratio of simulated perpendicular and parallel mean-free
paths in order to better bring out the different rigidity scalings.

Indications of the $(r_{\text{g}}/L_{\text{c}})^{1/2}$-scaling had been
observed already earlier DeMarco  _et al._ (2007); Giacinti  _et al._ (2018);
Dundovic  _et al._ (2020), however, understanding of its origin was still
lacking. Specifically, Ref. Dundovic  _et al._ (2020) compared their
simulation results with a few analytical theories Matthaeus  _et al._ (2003);
Shalchi (2010) and concluded that none could reproduce the non-standard
scaling at intermediate rigidities. To the best of our knowledge, we are the
first to report the recovery of the standard scaling at lower rigidities. In
the following, we will contrast our numerical results with those of our
analytical model.

$\eta$ | $A$ | $\rho_{*}$ | $s_{\kappa}$ | $s_{\text{FL}}$ | $\sigma$ | $\alpha$
---|---|---|---|---|---|---
0.2 | 3.81 | 0.755 | 0.796 | 0.234 | 1.83 | 5.38
0.5 | 1.06 | 0.627 | 0.906 | 0.128 | 2.11 | 2.36
0.8 | 0.331 | 0.428 | 1.22 | 0.0935 | 7.47 | 0.321
Table 1:  Model parameters for different turbulence levels $\eta$, determined
by fitting to the simulated $\kappa_{\parallel}$ and $\lambda_{\perp}$. $A$,
$\rho_{*}$ and $s_{\kappa}$ refer to the rigidity-dependence of the asymptotic
parallel diffusion coefficient $\kappa_{\parallel}$, see eq. (10).
$s_{\text{FL}}$, $\sigma$ and $\alpha$ parametrise the subdiffusive phase of
the running field line diffusion coefficient $d_{\text{FL}}$, see eq. (11).

In Fig. 1, we also show the predictions of our analytical model, the second
main result of this letter. For the parallel mean free path, indicated by the
dot-dashed line, this is the parametrisation of eq. (10) with the fitted
parameters given in Tbl. 1. For the perpendicular mean free path, indicted by
the solid line, this is $\kappa_{\perp}=d_{\perp}(\tau_{c})$ with eqs. (6) and
(8); the fitted parameters are given in Tbl. 1.

Overall, the model reproduces the mean free paths very well. In particular,
the perpendicular mean free path $\lambda_{\perp}$ reproduces the complex
rigidity-dependence found in the simulations.

The cause of the non-standard rigidity-dependence for intermediate rigidities
is the sub-diffusive phase in the running field-line diffusion coefficient.
The fact that without such a subdiffusive phase, the non-standard rigidity-
dependence for intermediate rigidities could not be reproduced is illustrated
by the dotted lines in Figs. 1 and 3. Here, we have assumed a running field
line diffusion coefficient _without_ a subdiffusive phase,

$d^{\prime}_{\text{FL}}(z)=z\frac{\delta
B_{x}^{2}}{B_{0}^{2}}\left(1+\left(\frac{z}{\kappa_{\text{FL}}}\frac{\delta
B_{x}^{2}}{B_{0}^{2}}\right)^{1/s_{\text{FL}}}\right)^{-s_{\text{FL}}}\,.$
(12)

With this $d^{\prime}_{\text{FL}}$, the $\lambda_{\perp}$ does not exhibit a
non-standard rigidity-dependence for intermediate rigidities, but transitions
almost smoothly between a $(r_{\text{g}}/L_{\text{c}})^{1/3}$-scaling and a
constant.

While the sub-diffusive nature of the field-line diffusion coefficient nicely
reproduces the simulated parallel and perpendicular mean-free paths, one might
wonder whether our parametrisation eq. (11) with the parameters of Tbl. 1 is
actually in agreement with the transport of field lines from simulations. We
provide a comparison of the model running field line diffusion coefficients
with the simulation for the turbulence level $\eta=0.5$ where such simulation
data are available Kuhlen  _et al._ (2022b). The model agrees very well with
the data up to $z\simeq 10\,L_{\text{c}}$. The ballistic phase is reproduced
by definition, but also the subdiffusive episode between $z\simeq
2\,L_{\text{c}}$ and $6\,L_{\text{c}}$ is reproduced. Similar trends could
have already been observed in the simulations of Ref. Sonsrettee  _et al._
(2016). At larger $z$, we note that the simulations exhibit a recovery of
$d_{\text{FL}}$ to a larger value than our asymptotic value which we fixed by
hand, see above. We could have also included this through another series of
breaks in $d_{\text{FL}}$, but for the reproduction of the perpendicular mean
free paths, this is not necessary as particles will have already decorrelated
from field lines for $z\lesssim 10\,L_{\text{c}}$.

To summarise, we have shown the results of a large suite of simulations of the
transport of high-energy particles and field lines in isotropic Kolmogorov-
like turbulence. The perpendicular diffusion coefficient in particular scales
like $(r_{\text{g}}/L_{\text{c}})^{1/3}$ for low rigidities,
$(r_{\text{g}}/L_{\text{c}})^{1/2}$ at intermediate rigidities and becomes
constant for $r_{\text{g}}/L_{\text{c}}\gg 1$. This behaviour is nicely
reproduced in our analytical model and is ultimately due to a subdiffusive
phase in the running field line diffusion coefficient.

Our study has provided a number of crucial insights: First, we have confirmed
the non-standard rigidity-scaling of the perpendicular mean free path at
intermediate rigidities, but also found that it transitions back to the
standard $(r_{\text{g}}/L_{\text{c}})^{1/3}$-dependence at the lowest
rigidities. Second, we have for the first time provided a consistent
analytical model that reproduces the non-standard behaviour. Our parametrised
running and asymptotic diffusion coefficients are easy to implement in
phenomenological studies or for comparison with future test particle
simulations. Third, our model largely confirms the interpretation of
perpendicular transport as due to particle transport along field lines and
field line transport in the perpendicular direction, but with the need for a
subdiffusive phase in the running field line diffusion coefficient. Finally,
this understanding has given us renewed confidence in the standard scaling
$\lambda_{\perp}\propto(r_{\text{g}}/L_{\text{c}})^{1/3}$ of the perpendicular
diffusion coefficient at low rigidities. We provide a discussion of
applications to a number of astrophysical problems in the companion paper
Kuhlen  _et al._ (2022b).

###### Acknowledgements.

The authors would like to thank Alexandre Marcowith for helpful discussions.
This project was funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) – project number 426614101.

## References

  * Jokipii (1966) J. R. Jokipii, ApJ 146, 480 (1966).
  * Kennel and Engelmann (1966) C. F. Kennel and F. Engelmann, Physics of Fluids 9, 2377 (1966).
  * Hall and Sturrock (1967) D. E. Hall and P. A. Sturrock, Physics of Fluids 10, 2620 (1967).
  * Hasselmann and Wibberenz (1970) K. Hasselmann and G. Wibberenz, ApJ 162, 1049 (1970).
  * Shalchi (2009) A. Shalchi,  _Nonlinear Cosmic Ray Diffusion Theories_, Vol. 362 (Springer Berlin, Heidelberg, 2009).
  * Shalchi (2020) A. Shalchi, Space Sci. Rev. 216, 23 (2020).
  * Mertsch (2020) P. Mertsch, Ap&SS 365, 135 (2020), arXiv:1910.01172 [astro-ph.HE] .
  * Harari  _et al._ (2002) D. Harari, S. Mollerach, E. Roulet, and F. Sánchez, Journal of High Energy Physics 2002, 045 (2002), arXiv:astro-ph/0202362 [astro-ph] .
  * Giacalone and Jokipii (1999) J. Giacalone and J. R. Jokipii, ApJ 520, 204 (1999).
  * Casse  _et al._ (2001) F. Casse, M. Lemoine, and G. Pelletier, Phys. Rev. D 65, 023002 (2001), arXiv:astro-ph/0109223 [astro-ph] .
  * Qin  _et al._ (2002a) G. Qin, W. H. Matthaeus, and J. W. Bieber, Geophys. Res. Lett. 29, 1048 (2002a).
  * Qin  _et al._ (2002b) G. Qin, W. H. Matthaeus, and J. W. Bieber, ApJ 578, L117 (2002b).
  * DeMarco  _et al._ (2007) D. DeMarco, P. Blasi, and T. Stanev, J. Cosmology Astropart. Phys 2007, 027 (2007), arXiv:0705.1972 [astro-ph] .
  * Giacinti  _et al._ (2012) G. Giacinti, M. Kachelrieß, D. V. Semikoz, and G. Sigl, J. Cosmology Astropart. Phys 2012, 031 (2012), arXiv:1112.5599 [astro-ph.HE] .
  * Snodin  _et al._ (2016) A. P. Snodin, A. Shukurov, G. R. Sarson, P. J. Bushby, and L. F. S. Rodrigues, MNRAS 457, 3975 (2016), arXiv:1509.03766 [astro-ph.HE] .
  * Subedi  _et al._ (2017) P. Subedi, W. Sonsrettee, P. Blasi, D. Ruffolo, W. H. Matthaeus, D. Montgomery, P. Chuychai, P. Dmitruk, M. Wan, T. N. Parashar, and R. Chhiber, ApJ 837, 140 (2017), arXiv:1612.09507 [physics.space-ph] .
  * Giacinti  _et al._ (2018) G. Giacinti, M. Kachelrieß, and D. V. Semikoz, J. Cosmology Astropart. Phys 2018, 051 (2018), arXiv:1710.08205 [astro-ph.HE] .
  * Schlegel  _et al._ (2020) L. Schlegel, A. Frie, B. Eichmann, P. Reichherzer, and J. B. Tjus, ApJ 889, 123 (2020), arXiv:1907.09934 [astro-ph.HE] .
  * Reichherzer  _et al._ (2020) P. Reichherzer, J. Becker Tjus, E. G. Zweibel, L. Merten, and M. J. Pueschel, MNRAS 498, 5051 (2020), arXiv:1910.07528 [astro-ph.HE] .
  * Reichherzer  _et al._ (2022) P. Reichherzer, L. Merten, J. Dörner, J. Becker Tjus, M. J. Pueschel, and E. G. Zweibel, SN Applied Sciences 4, 15 (2022), arXiv:2104.13093 [astro-ph.HE] .
  * Matthaeus  _et al._ (2003) W. H. Matthaeus, G. Qin, J. W. Bieber, and G. P. Zank, ApJ 590, L53 (2003).
  * Shalchi (2010) A. Shalchi, ApJ 720, L127 (2010).
  * Dundovic  _et al._ (2020) A. Dundovic, O. Pezzi, P. Blasi, C. Evoli, and W. H. Matthaeus, Phys. Rev. D 102, 103016 (2020), arXiv:2007.09142 [astro-ph.HE] .
  * Evoli  _et al._ (2012) C. Evoli, D. Gaggero, D. Grasso, and L. Maccione, Phys. Rev. Lett. 108, 211102 (2012), arXiv:1203.0570 [astro-ph.HE] .
  * Cerri  _et al._ (2017) S. S. Cerri, D. Gaggero, A. Vittino, C. Evoli, and D. Grasso, J. Cosmology Astropart. Phys 2017, 019 (2017), arXiv:1707.07694 [astro-ph.HE] .
  * Kolmogorov (1941) A. Kolmogorov, Akademiia Nauk SSSR Doklady 30, 301 (1941).
  * Tautz (2016) R. C. Tautz, New A 45, 1 (2016).
  * Kuhlen  _et al._ (2022a) M. Kuhlen, V. H. M. Phan, and P. Mertsch, ApJ 927, 110 (2022a), arXiv:2111.06419 [astro-ph.HE] .
  * Kuhlen  _et al._ (2022b) M. Kuhlen, P. Mertsch, and V. H. M. Phan, preprint (2022b).
  * Bieber and Matthaeus (1997) J. W. Bieber and W. H. Matthaeus, ApJ 485, 655 (1997).
  * Shalchi  _et al._ (2004) A. Shalchi, J. W. Bieber, W. H. Matthaeus, and G. Qin, ApJ 616, 617 (2004).
  * Shalchi (2019) A. Shalchi, ApJ 881, L27 (2019), arXiv:1908.00694 [astro-ph.SR] .
  * Gruzinov (2018) A. Gruzinov, (2018), arXiv:1808.00041 [astro-ph.HE] .
  * Sonsrettee  _et al._ (2016) W. Sonsrettee, P. Subedi, D. Ruffolo, W. H. Matthaeus, A. P. Snodin, P. Wongpan, P. Chuychai, G. Rowlands, and S. Vyas, ApJS 225, 20 (2016).

## Supplemental Material

In Fig. 4 we compare the running diffusion coefficients from the test particle
simulations with those of our model.

Figure 4:  Running parallel and perpendicular diffusion coefficients
$d_{\parallel}$ and $d_{\perp}$ for different reduced rigidities
$r_{\text{g}}/L_{\text{c}}$. The grey lines and bands indicate the means and
standard mean errors from the test particle simulations. The dashed (solid)
line shows the running parallel (perpendicular) diffusion coefficient from the
simulations. The asymptotic values are indicated by the dashed and dot-dashed
lines, respectively. Finally, the dotted grey line indicates
$L_{\text{c}}^{2}/t$, that is the asymptotic perpendicular asymptotic
diffusion coefficient $\kappa_{\perp}$ is attained once the running
perpendicular diffusion coefficient $d_{\perp}$ intersects this line.


# PoseMatcher: One-shot 6D Object Pose Estimation by Deep Feature Matching

Pedro Castro
Imperial College London
<EMAIL_ADDRESS>Tae-Kyun Kim
Imperial College London, KAIST
<EMAIL_ADDRESS>
###### Abstract

Estimating the pose of an unseen object is the goal of the challenging one-
shot pose estimation task. Previous methods have heavily relied on feature
matching with great success. However, these methods are often inefficient and
limited by their reliance on pre-trained models that have not be designed
specifically for pose estimation. In this paper we propose PoseMatcher, an
accurate model free one-shot object pose estimator that overcomes these
limitations. We create a new training pipeline for object to image matching
based on a three-view system: a query with a positive and negative templates.
This simple yet effective approach emulates test time scenarios by cheaply
constructing an approximation of the full object point cloud during training.
To enable PoseMatcher to attend to distinct input modalities, an image and a
pointcloud, we introduce IO-Layer, a new attention layer that efficiently
accommodates self and cross attention between the inputs. Moreover, we propose
a pruning strategy where we iteratively remove redundant regions of the target
object to further reduce the complexity and noise of the network while
maintaining accuracy. Finally we redesign commonly used pose refinement
strategies, zoom and 2D offset refinements, and adapt them to the one-shot
paradigm. We outperform all prior one-shot pose estimation methods on the
Linemod and YCB-V datasets as well achieve results rivaling recent instance-
level methods. The source code and models are available at
github.com/PedroCastro/PoseMatcher.

##  1 Introduction

Accurately retrieving the relative position and orientation of an object is
the first step for any task that requires interaction with objects in the real
world. Estimating the pose of an object is an indispensable step for robotic
manipulation as well as in VR/AR applications. It is imperative for the pose
estimation to be accurate and robust to external obstacles such as occlusion,
illumination and symmetries. Existing methods excel at retrieving the pose of
known objects to a very high accuracy standard [42, 7, 2]. Some methods can
even produce pose estimation at high throughput [42, 4], be specially robust
to symmetries [10] and can even surpass synthetic to real domain gaps [41,
36]. However, most share a very large limitation: the target objects must be
known. This constraint is severely limiting as it necessitates model
retraining for each object addition. Retraining new objects on existing models
might lead to catastrophic forgetting if not handled properly [19]. Category
level approaches [46, 43] try to generalize up to a category, where each
target object can be obtained with a simple deformation of a canonical model
and all share semantic keypoints (ex. handle on a mug or the cap of a bottle).
Nonetheless, much of the same one-shot domain problems still remain: objects
category must have been seen before and the target object must not lie outside
of the scope of the category for the estimation to be possible. In order to
overcome these problems we must tackle object pose estimation in the one-shot
paradigm.

Figure 1: Illustrative diagram of PoseMatcher training diagram. For each
training instance query we sample two templates. From these templates, we
reconstruct a partial point cloud simulating the object point cloud at test
time.

By one-shot pose estimation, we refer to estimating the pose of an novel
object based on information seen only at test-time, without the object,or its
category, being present in the training dataset. Older methods have some very
hard constraints such as needing masking and depth maps at test-time [13], a
full colored 3D model of the target object [22], need to be fine-tuned on
images of the same dataset for difficulties of overcoming domain gap [25] or
rely on an extremely expensive test-time optimization through renderers [47,
21, 28]. More recently, feature matching methods have been shown to achieve
impressive results. Particularly, OnePose [38] and OnePose++ [12] make use of
existing state of the art pre-trained descriptor extractors on top of which a
pose estimation pipeline is built. However, by relying on a fixed pre-trained
model to extract descriptive keypoints from templates it fails to capture the
optimal keypoint object descriptions. Ideally, the template extraction model
should be the same as the query model and should be jointly optimized.

We rethink the approach to the problem and introduce a three-view pipeline
that allows us to jointly train the template and query extractors. Just by
redesigning the training pipeline, we can improve OnePose++[12] without
additional changes to its methodology. We also introduce a new efficient image
to object attention layer which we call IO-Layer, which reduces both
parameters when compared to the modules used by OnePose++ [12] and also
separates the two input modalities, an image and a pointcloud, allowing the
model to optimize weights specifically for either image or pointcloud
keypoints. We also found that working with a full object at all stages of the
feature matching is redundant and reduces pose estimation accuracy which lead
us to introduce a novel iterative matching based pointcloud prunning. On top
of these changes, we employ a 3D-refinement technique based on zoom refinement
used on instance-level pose estimation [22, 4, 20].

In summary our contributions are as follows:

  * •

We redesign the training pipeline for one-shot object-to-image matching. Our
three-view training approach allows us to train from scratch PoseMatcher, a
pose estimation method leveraging detection free keypoint matching.

  * •

We introduce a more efficient image to object attention layer we call IO-
Layer, specifically built to accommodate the two input modalities, the image
and the object pointcloud.

  * •

We propose a layered pruning of the target object point cloud. We improve
runtime by reducing the amount of attented keypoints while reducing noisy
matching, leading to an improvement in accuracy.

  * •

We create a fine level 3D based refinement where we directly estimate relative
2D and depth positions, replacing the 2D keypoint refinement used by prior
one-shot approaches.

##  2 Literature

Figure 2: Diagram of PoseMatcher. At training time we sample from the Google
Scanned Objects[8] dataset and from two opposing views templates we construct
a partial point cloud. At test time, a novel unseen object is used instead. We
extract local features from both the query and the templates. Using our novel
IO-Layer, we compute dense correspondences between the image and the object.
We then take the matches (yellow star) and further refine them using 2D and 3D
techniques.

Fully supervised pose estimation. Instance-level pose estimation is designed
to support a single object. Due to this task’s narrowed scope, the accuracy of
instance-level methods are becoming increasingly more impressive. Early 6D
pose estimation works focused on recovering the 2D position of specific
keypoints, mainly the 3D bounding box of the target [33, 27, 39]. PVNet [30]
found that choosing keypoints that lie within the object’s silhouette would
yield better results. This idea has become a mainstay among current keypoint
based methods [15, 4]. An alternative to chosing a limited amount of features
was to estimate the coordinate of the surface of the object at each pixel [43,
48, 29]. GDR[42] and SO-Pose[7] approximated PnP through a small neural
network and optimizing directly through pose errors. Direct pose estimation
with a combination of a dense intermediate representation makes up most of the
state of the art methods as per the BOP challenge [14]. Other ideas have
focused on designing textures as learnable features that can be more easily
estimated [10, 16] and as a result, better correspondences are found. In order
to decrease the need for real data, Sock _et al_. [36] proposed a
differentiable pipeline where learning is done through comparison with a
rendering of the estimation which is done using keypoints, where Self6D [41]
directly outputs pose.

One-shot pose estimation. Until recent versions of the BOP challenge [14],
Point Pair Features [13] held the top spot on its leaderboard. It relies on
selecting geometrical relevant keypoints from an existing 3D and matching
these to a depth map. This method relies on capturing information with a depth
sensor at inference time which is not commonly available. Extending the prior
instance-level approaches to category-level allows for object deformation and
texture change within a narrow category [49, 9, 43]. The main idea from NOCS
[43], although used for instance-level by a wide range of works [42, 7, 48],
was initially to predict pose of slightly deformable objects within a
category, as it was inspired by its human reconstruction contemporary
DenseBody [1] . Recent advancements have been made that allow for training for
in the wild [9]. However, these are sill limited by the similarity to objects
of the same category. Our aim is to overcome this limitation.

Methods such as Pitteri  _et al_. [31] and CorNet[32] rely on training with a
subset of similar objects of the same dataset and/or scene, which includes
biases towards illuminations, noise, background and shape. Recent attempts
have shown researchers are keen on tackling one-shot pose estimation. OSOP
[35] introduces a global template matching method where a given query is
matched to the closest viewpoint from a database of pre-processed synthetic
viewpoints, which requires the object model, an assumption we do not make.
Gen6D [25] works in a similar fashion by matching a query to a small amount of
real annotated images and then refining the pose. However, it is susceptible
to poor pose initialization and poor 2D bounding box detections. The closest
work to our own is OnePose++, an improved version of OnePose[38], where the
feature extraction is replaced by the detector free feature matcher LoFTR
[37]. However, OnePose++[12] adapts a pre-trained LoFTR to one-shot pose
estimation without taking consideration the different modalities of image and
point cloud, self-occlusion redundancy and the possibility of using 6D pose
based refinement.

##  3 PoseMatcher

The goal of the PoseMatcher is to establish matches between the two sets of
keypoints. We establish correspondence between the 2D keypoints features
extracted from the query image $\mathcal{I}^{Q}$ and the object point cloud
$\\{\mathcal{P}_{O}\\}$. From those matches we can apply PnP and recover full
6D pose $\zeta$. A diagram of PoseMatcher can be seen in Fig. 2.

###  3.1 Template Based Training

We adopt a new training methodology that allows us to design a one-shot
object-image detector free feature matching model from scratch, specifically
aimed at pose estimation task. We therefore remove the need for pre-trained
descriptor extractors used by OnePose and OnePose++ [38, 12].

However, a single viewpoint will result in a partial reconstruction of the
object. At inference time, both the visible and occluded regions of the object
will be represented in the template point-cloud. Therefore, a single template
is not enough to emulate the conditions at test-time.

To address these limitations, we sample an additional template image we refer
to as the negative template $\mathcal{I}^{-}$. This template shares low co-
visibility area with the anchor image. The keypoints extracted from
$\mathcal{I}^{-}$ serve to generate nearly complete reconstruction of the
target object, with the visible sections being sampled from $\mathcal{I}^{+}$
while the self-occluded ones from $\mathcal{I}^{-}$. This 2-template paradigm
simulates the full object point-cloud available at the inference time.

In order to optimize through the matching task we apply the differentiable
dual-softmax operator as proposed by LoFTR [37] and subsequently used by both
OnePose and OnePose++ [38, 12].

In order to output matches, we start by using a local feature extractor, such
as a Resnet [11] to extract coarse and fine level feature maps,
$\hat{\mathcal{F}}_{\mathcal{I}^{Q}}\in\mathbb{R}^{HxWx\hat{C}}$ and
$\tilde{\mathcal{F}}_{\mathcal{I}^{Q}}\in\mathbb{R}^{HxWx\tilde{C}}$ from the
query image. We repeat the same process for both $\mathcal{I}^{+}$ and
$\mathcal{I}^{-}$ and sample $M$ points from both templates. We make sure that
the template positions lie within the objects mask such that:

$\begin{split}p^{+}=\\{p_{k}^{+}|~{}k\in\mathcal{M}^{+}\\},\\\
p^{-}=\\{p_{k}^{-}|~{}k\in\mathcal{M}^{-}\\},\end{split}$ (1)

where $\mathcal{M}$ refers to the segmentation mask. We backproject, using the
a depth map only available at training time, and generate a template point
${\mathcal{P_{\mathcal{T}}}^{3D}}\in\mathbb{R}^{2Mx3}$ with its corresponding
extracted features
${\hat{\mathcal{F}_{\mathcal{T}}}}\in\mathbb{R}^{2Mx\hat{C}}$ and
${\tilde{\mathcal{F}_{\mathcal{T}}}}\in\mathbb{R}^{2Mx\tilde{C}}$, where
$\mathcal{T}$ refers to the combined positive and negative templates.

At this point, we perform dense matching between the coarse level query
features $\hat{\mathcal{F}}_{\mathcal{I}^{Q}}$ and the extracted point cloud
${\hat{\mathcal{F}_{\mathcal{T}}}}$. We now aim to globally match each pixel
of the query image that contains the object to its respective point cloud
keypoint. We make use of positional embeddings to encode positional
information into each feature point. For 2D keypoints we use the sinusoidal
fixed version used by DETR [3] while for 3D positional encoding we use a
simple 3-layer MLP as used by SurfEmb [10] and OnePose++ [12].

We flatten both feature maps and apply self and cross attention layers
following other feature matching papers [34, 37, 38, 12, 44] to generate more
easily separable features on each set. Our objective is to construct a matrix
$\mathds{P}^{HWx2M}$ that reflects the level of confidence of the
correspondence between $\mathcal{P}^{2D}$ and $\mathcal{P}^{3D}$. A score
matrix $\mathcal{S}$ is computed by measuring the cosine similarity between
the two sets of transformed features in a contrastive manner. We apply dual-
softmax [40] over $\mathcal{S}$ to calculate the correspondence confidence
matrix $\mathds{P}$:

$\mathds{P}=softmax(\mathcal{S}(i,\cdot))_{c}~{}\cdot~{}softmax(\mathcal{S}(\cdot,c))_{i},$
(2)

Figure 3: IO-Layer architecture. We improve on the attention modules used by
LoFTR and OnePose++ [37, 38]. We have modality specific projections for image
features embedded with 2D positional encoding and for object template features
with its 3D positional encoding. We found that separating the two modalities
improves the final pose accuracy.

$\tau$ being a temperature hyperparameter, with $i$ and $c$ being the indices
of the flattened image pixels and point cloud, respectively. The image to
object correspondences $\mathcal{C}$ are established by choosing only those
that exceed a certain level of confidence, represented by the threshold value
$\theta$, and meet the mutual nearest neighbor (MNN) constraint to remove
false matches:

$\mathcal{C}=\\{(i,c)|\forall(i,c)\in
MNN(\mathcal{P}_{i}^{2D},\mathcal{P}_{c}^{3D}),\mathds{P}_{i,c}\geq\theta\\}.$
(3)

The coarse loss $\mathcal{L}_{c}$ used to optimize $\mathcal{C}$ uses focal
loss [24] as suggested by LoFTR [37].

###  3.2 IO-Layer: An object-image attention layer

Image to image feature matching has seen its effectiveness increase with the
introduction of self and cross attention mechanisms [44, 34, 37, 26]. These
layers are particularly efficient for this task seeing as both inputs are from
the same modality and share the same spatial structure. For this reason, self
attention layers can share weights and cross attention only requires swapping
the query and template image, as you can see in Fig 3. OnePose++ [12] has
suggested using the same approach for image to object matching however in this
scenario we are working with two different modalities, a 2D image and a 3D
point cloud.

We postulate that mixing modalities has an adverse effect on learning
distinctive features. The positional encoding for each input is structurally
different which means that attention layers will have difficulty separating
the features of each modality. OnePose++ [12] shares a sign of this problem
when they observe that adding 3D positional encoding results in a small
($<1\%$) improvement. A simple ad-hoc solution would be to have modality
specific weights, however that would double the amount of parameters and
subsequently the necessary amount of training data and compute. In standard
self attention layers an input sequence is linearly projected into a query,
key and value: $Q_{i}$,$K_{i}$,$V_{i}$,stemming from the same input $i$. For
cross attention, one would linearly project $Q_{i}$, $K_{j}$, $V_{j}$ where
$i$ and $j$ are different inputs. Usually, in attention mechanisms, the
encoded message is parsed to the decoder and is not returned. However, since
it is important for the image features to be distinct from the object’s and
vice-versa, the message must be passed bi-directionally and therefore requires
careful redesign.

We propose a simple layer, specially designed for Image to Object matching we
call IO-Layer. ($Q_{i}$,$K_{i}$,$V_{i}$) and ($Q_{o}$,$K_{o}$,$V_{o}$), the
inputs to the attention mechanism for inputs and object respectively, are
computed only once and used for both self and cross attention. Therefore cross
attention is computed as $softmax(Q_{i}K_{o}^{T}/\sqrt{d})K_{o}$ and
$softmax(Q_{o}K_{i}^{T}/\sqrt{d})K_{i}$ for image to object and object to
image attention respectively, where we only have to project each sequence
once. We modify self and cross attention accordingly to support Linear
Attention [17]. Much like the layers used in LoFTR and OnePose++, these can be
stacked together. We exemplify the IO-Layer on Fig.3.

###  3.3 Object Pruning

OnePose++ proposes sampling and using a point cloud template with over 15k
keypoints per object. Performing attention over such a higher number of
keypoints is expensive even if using more efficient mechanisms such as Linear
Attention [17]. The set of keypoints that are not in visible in the query
image should be quickly identifiable in early matching. Removing these
keypoints from the point cloud allows for better separability of the features
of the remaining visible keypoint, which leads to a better match. As a
peripheral advantage, by removing a set of keypoints from the template point
cloud, we reduce the complexity of the following attention layers. During
pruning, we do not impose a hard threshold on the confidence of the matches
but rather but rather we select the keypoints with higher confidences of
existing in the image. We extend the matching operation in Eq. 3 where we sum
the keypoint confidences over every pixel and prune the lowest ones. We insert
a pruning step after each IO-Layer. The amount of pruning is subjected to
ablation studies in Section 4.5.1.

###  3.4 Pose Refinement

We add a fine level 2D-refinement post-processing step as described by LoFTR
[37] and OnePose++[12]. For each match, we crop a window at the match location
and perform simple matching w.r.t. the matched keypoints. We supervise it by
minimizing the Euclidean distance between the center of the grid and the
projected keypoints. More details about 2D refinement are available in LoFTR
[37].

However, its impact is limited as we show in the experimental section. We
believe this is due to the use of PnP following the keypoint position
refinement. Prior 6D object pose literature has shown that estimating
translation through PnP leads to poor results [23, 42]. An alternative
approach, first introduced by CDPN [23], is to directly estimate the
translation as the 2D offset of the object centroid as well as bounding box
relative depth estimation. Directly applying this step to PoseMatcher cannot
be done seeing as the objects are not known.

In order to perform translation estimation, we adopt translation refinement,
which will refer to as 3D-refinement, a strategy used by iterative methods [4,
20, 22]. Instead of estimating the translations directly, we estimate the
translation errors given an initial pose. Intuitively, we estimate the
necessary 2D translation and zoom necessary to align the initial pose to the
target pose.

In order to generate the initial pose $\zeta^{0}$, we perform PnP over the
coarse matching keypoints. In the 2D-refinement step, the initial position of
the matched keypoints are the image grid default positions. However, if we are
performing a refinement over an initial pose, this alignment might be broken
due to erroneous matching, pose estimation or even sub-pixel positioning.
Therefore we project the matched keypoints using the initial pose and use
those 2D locations as the center for the fine-sampling grid crops
$F^{i}_{crop}$. We perform self and cross attention over the grids and the
corresponding 3D fine feature $T_{f}^{i}$ using a single IO-Layer. We compute
the 2D expectation and supervise its output using the same methodology as
described for the 2D-refinement.

We propose a lightweight CNN similar to the one used for learned Patch-PnP in
GDR-Net [42]. We build our pose representation as a pixel-wise map where at
each coarse pixel we collect each matched keypoint refinement prediction (i.e.
refinement direction). This CNN outputs the 3D-refinement in the form of
$\Delta T$ and $\Delta Z$, the 2D location and zoom offsets respectively.
Moreover, in order to supervise the refinement step, LoFTR [37] proposes
computing the total variance of the matching heatmap in order to penalize more
confident but erroneous estimations. We propose reusing this variance to
introduce a confidence measure to each keypoint refinement output and append
to each input its corresponding refinement.

Type | Fully Supervised | Self-Supervised | One-Shot
---|---|---|---
Method | PVNet [30] | GDR [42] | SO-Pose [7] | Self6D [41] | Sock  _et al_. [36] | Gen6D [25] | OnePose [38] | OnePose++ [12] | PoseMatcher
Ape | 43.6 | 85.9 | - | 38.9 | 37.6 | - | 11.8 | 31.2 | 59.2
Benchvise | 99.9 | 99.8 | - | 75.2 | 78.6 | 62.1 | 92.6 | 97.3 | 98.1
Camera | 86.9 | 96.5 | - | 36.9 | 65.6 | 45.6 | 88.1 | 88.0 | 93.4
Can | 95.5 | 99.3 | - | 65.6 | 65.6 | - | 77.2 | 89.8 | 96.0
Cat | 79.3 | 93.0 | - | 57.9 | 52.5 | 40.9 | 47.9 | 70.4 | 88.0
Driller | 96.4 | 100. | - | 67.0 | 48.8 | 48.8 | 74.5 | 92.5 | 98.4
Duck | 52.6 | 65.3 | - | 19.6 | 35.1 | 16.2 | 34.2 | 42.3 | 54.1
Eggbox* | 99.2 | 99.9 | - | 99.0 | 89.2 | - | 71.3 | 99.7 | 97.8
Glue* | 95.7 | 98.1 | - | 94.1 | 64.5 | - | 37.5 | 48.0 | 91.5
Holepuncher | 81.9 | 73.4 | - | 16.2 | 41.5 | - | 54.9 | 69.7 | 73.4
Iron | 98.9 | 86.9 | - | 77.9 | 80.9 | - | 89.2 | 97.4 | 97.9
Lamp | 99.3 | 99.6 | - | 98.2 | 70.7 | - | 87.6 | 97.8 | 98.1
Phone | 92.4 | 86.3 | - | 50.1 | 60.5 | - | 60.6 | 76.0 | 92.1
Average | 86.3 | 91.0 | 96.0 | 58.9 | 60.6 | - | 63.6 | 76.9 | 87.5
|  |  |  |  |  |  |  |  |

Table 1: Comparison study on Linemod. We present the results for ADD(-S)
metric and compare them to state of the art. While Linemod is close to
saturated for fully supervised methods, one-shot pose estimation is still
challenging. PoseMatcher achieves the best results for all objects for the
one-shot category, surpassing self-supervised methods and close to fully
supervised methods. Best results for one-shot are bolded. ∗ denotes symmetric
objects.

We model the zoom as a classification problem and discrete the possible zooms
in $K$ classes [2]. We output the estimated zoom $\delta_{z}$ as the
expectation over all classes and 2d translation $\delta_{2d}$ necessary to
align $\zeta^{0}$ with $\zeta^{GT}$ as following:

$\begin{cases}\zeta^{GT}_{z}=\zeta^{GT}_{z}\cdot\delta_{z},\\\
p(\zeta^{GT})=p(\zeta^{GT})~{}\cdot~{}\delta_{2d},\\\ \end{cases}$ (4)

where $p(P)=\mathcal{K}\zeta[0,0,0]^{T}$ is the 2D projection location of the
object’s centroid adjusted for the crop bounding box and $K$ are the camera
intrinsic parameters.

We supervise the zoom and 2D location offset directly with a sparse loss:

$\begin{cases}\mathcal{L}_{z}=||\delta_{z}-\varepsilon_{z}||_{1}\\\
\mathcal{L}_{2d}=||\delta_{2d}-\varepsilon_{2d}||_{1}\\\ \end{cases}$ (5)

where $\varepsilon$ is noise introduced at training time.

The full loss with our improvements becomes:

$\mathcal{L}=\mathcal{L}_{c}+\mathcal{L}_{f}+\alpha~{}(\mathcal{L}_{z}+\mathcal{L}_{2d}).$
(6)

##  4 Experiments.

###  4.1 Data preparation

In order to train PoseMatcher from scratch we must use a sufficiently large
dataset to encompass a large number of shapes and textures. We use the Google
Scanned Objects[8] dataset with 1023 household objects. We use the renderings
provided by [25] for fair comparison where each object is rendered at 250
different viewpoints. We use an additional set of 500 keypoints from ShapeNet
[5] with renderings provided by [9]. For each instance we find a close
viewpoint, $\mathcal{T}^{+}$, sampled within $[5^{\circ},25^{\circ}]$
orientation to ensure high co-visibility while being sufficiently disparate.
$\mathcal{T}^{-}$ is sampled randomly from a set of the 5 farthest viewpoints
from the query, which ensures enough data variety. We apply the standard color
and noise augmentations [10, 42, 4] as well as bounding box zoom-ins as
proposed by CDPN [23]. We also perform object level augmentations by changing
the object’s canonical reference frame. Although only 1500 objects are used,
by providing strong augmentations we are able to avoid overfitting.

###  4.2 Implementation Details.

For fair comparison with OnePose++, we use 3 IO-Layers. We set the contrastive
temperature term $\tau=0.1$. For 3D-refinement we set $K=100$ and
$\alpha=100$. During training we sample $M=2048$ from each templates for a
total of $4096$. For refinement, we use the top 512 matches with confidences
up to $\theta=0.1$. If not enough matches during training, we pad the
pointcloud with groundtruth matches. At test time, we sample 16k keypoints
from the available training images. We prune $50\%$ of the point cloud after
each IO-Layer, to 8k and 4k after the first two layers, respectively.

For all experiments, we use AdamW [18] with cosine learning rate decay and we
linearly warmup the learning rate for 5 epochs. We train PoseMatcher for 50
epochs with a batch size of 8 and an intial learning rate of 0.0001. At each
epoch, we sample 10 different viewpoint queries for each of the 1500 objects.

###  4.3 RGB vs Grayscale images

Interest point descriptor models are mostly used over grayscale images as it
has been shown to improve generalization. OnePose and OnePose++ [38, 12] are
applied to grayscale images because they make use of pre-trained descriptor
models trained solely on grayscale, SuperPoint [6] and LoFTR respectively
[37]. PoseMatcher does not rely on prior methods thus is not limited to
grayscale. To provide a comprehensive analysis and ensure thoroughness we
train our method on both RGB and grayscale input. Intuitively, RGB inputs
should provide more information and allow PoseMatcher to better separate
textured regions of an object as well as from the background. Surprisingly, we
found that using RGB data improves our method. Grayscale only reaches an
ADD-(S) accuracy of $84.1\%$ on Linemod while RGB inputs reach an accuracy
$87.5\%$. All our experiments and further ablations use RGB inputs.

###  4.4 Evaluation Results

Our results on Linemod and the YCB-V datasets use the standard 2D bounding
boxes provided by the BOP Challenge [14]. To note these are trained using
synthetic versions of the target datasets which is done in order for a fair
comparison to other methods that rely on the same detections. To measure the
performance of PoseMatcher, we employ the same metrics used by prior methods.
The ADD(-S) metric considers a pose correct if the average distance to the
groundtruth falls below a threshold, usually $10\%$ of the objects diameter,
with a slightly modified version for symmetric objects [45]. We also measure
the accuracy of the predicted translation under a range of threshold to
support our 3D-refinement step.

We outperform every existing one-shot method by a significant margin,
specially on more complicated objects such as the Ape or the Cat. We have a
large advantage over DeepIM [22] which needs a strong pose initialization from
PoseCNN [45] whereas PoseMatcher does not require any type of pose
initialization or the 3D mesh model. It is interesting to observe that we also
outperform PVNet[30], an instance-level pose estimator.

###  4.5 Ablation Studies

Three-View | IO-Layer | Pruning | 2D-Ref | 3D-Ref | Grayscale | Linemod | YCB-V
---|---|---|---|---|---|---|---
|  |  | ✓ |  |  | 76.9 | -
✓ |  |  |  |  |  | 81.1 | 22.1
✓ |  |  | ✓ |  |  | 81.4 | 22.5
✓ | ✓ |  |  |  |  | 83.4 | 24.1
✓ | ✓ |  | ✓ |  |  | 83.4 | 24.2
✓ | ✓ | ✓ | ✓ |  |  | 83.9 | 25.8
✓ | ✓ | ✓ |  | ✓ |  | 87.5 | 31.3
✓ | ✓ | ✓ |  | ✓ | ✓ | 84.1 | 27.4
|  |  |  |  |  |  |

Table 2: Ablation studies for each component. We present the results on
Linemod and YCB-V measured by ADD-(S). The first line are the results gathered
from OnePose++ [38] The model used for the second and third rows refer is
identical to the OnePose++ [12] model trained with our three-view based
pipeline.

Our new training pipeline elevates the results from OnePose++ [12]. By
learning meaningful descriptors specifically trained for pose estimation
instead of pretrained ones, we are able increase the quality of the feature
matching thereby improving the pose accuracy by $5\%$. This confirms that
carefully designing a training pipeline can yield better results.

The IO-Layer, our novel attention based layer, yields an improvement of $2\%$
on both datasets. While it does not increase runtime performance when compared
to OnePose++ [38], we can see an improvement stemming from the use of
specialized modality weights.

Replacing 2D based refinement with pose specific 3D-Refinement improved our
results significantly. We found improvements in both Linemod and the YCB-V
datasets. The improvement cannot be solely observed by measuring the ADD-(S)
since the average threshold for Linemod is $15cm$. We provide further
information for Linemod on Fig. 5 where the error curves for each object in
Linemod can be seen. The 3D-refinement has a much larger impact on lower
thresholds accuracies than higher ones. An addition like 3D-refinement is
invaluable for applications that require high precision.

020406080100% keypoints prunned20406080ADD-(S)LMYCB-V

Figure 4: Pruning ablations. With this ablation, we show that PoseMatcher is
not reliant on a large number of keypoints. Although significant decreases in
performance occur when using 8 keypoints, we see small differences for larger
amounts, with the optimal number of keypoints being 64 for both datasets.

We found 2D-refinement to result in no significant improvements. This step
does not correct significant matching errors and PnP seems to overcome small
imperfections stemming from resolution errors. PnP is also subject to biasing
for translation as observed by prior works [42, 23]. Our results on
3D-refinement show a large increase in the translation accuracy when we use
3D-refinement over 2D-refinement, which is also reflected on ADD-(S).

####  4.5.1 Object pruning ablations

.

As discussed before, pruning an object provides two advantages. Firstly, it
eliminates self-occluded regions of the object which removes possible noise
during feature matching. Secondly, it reduces the complexity of the attention
mechanism by decreasing the amount of keypoints needed to be attented, which
leads to faster runtime. However, pruning a large amount of keypoints leads to
poorer results as it excludes essential keypoints. We perform inference time
ablation studies to help us determine the optimal pruning percentage and plot
the results of percentages ranging from $10\%$ to $90\%$ after each IO-Layer
in Fig 4 and measure the ADD-(S) accuracy on Linemod. If a low amount of
keypoints are removed, the performance is not highly affected. However, we see
a small drop at $75\%$ with a large drop when crop $90\%$ of the existing
pointcloud. The latter might be due to the number of initial keypoints as the
number of keypoints are just 150 points. Although increasing the number of
pruned keypoints leads to higher throughput, the peak accuracy results is at
$50\%$.

####  4.5.2 Number of Templates

For all our experiments on Linemod, we have been using all the available
training images in order to sample template keypoints. Due to our training
pipeline, PoseMatcher is robust to incomplete and noisy templates. We perform
an ablation study over the number of templates needed. Since each object in
Linemod contains different number of training images, we present the
percentage of training images used. To sample which images to use, we sample
from the available viewpoints using furthest point sampling [30] w.r.t.
orientations in order to cover the widest range of viewpoints. PoseMatcher
almost achieves the same level of accuracy as OnePose++ [12] using only $75\%$
of the available training images.

$\%$ | Ape | BW | Camera | Can | Cat | Driller | Duck | EB* | Glue* | HP | Iron | Lamp | Phone | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
10 | 2.1 | 15.1 | 8.8 | 0.0 | 1.9 | 6.4 | 0 | 25 | 2.5 | 0 | 12.4 | 2.1 | 12.4 | 6.8
25 | 5.9 | 24.1 | 15.2 | 12.4 | 13.1 | 12.4 | 1.2 | 45.1 | 8.3 | 0 | 24.4 | 12.6 | 29.1 | 15.7
50 | 12.4 | 59.4 | 35.6 | 19.7 | 19.3 | 34.7 | 9.4 | 78.9 | 41.9 | 5.7 | 34.1 | 21.8 | 57.3 | 33.1
75 | 39.9 | 78.9 | 81.2 | 76.6 | 57.3 | 65.1 | 27.1 | 92.1 | 72.1 | 46.8 | 87.1 | 79. | 88.5 | 68.6
100 | 59.2 | 98.1 | 93.4 | 96 | 88.0 | 98.4 | 54.1 | 97.8 | 91.5 | 73.4 | 97.9 | 98.1 | 92.1 | 87.5
|  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 3: Number of templates. We present the results on Linemod for $\%$ of
training images used. Each object has around 180 images therefore $10\%$ is
only 18 templates to use.

05101520Threshold/mm020406080100ADD-(S)

(a)

05101520Threshold/mm020406080100ADD-(S)ApeBenchviseCameraCanCatDrillerDuckEggboxGlueHolepuncherIronLampPhone

(b)

Figure 5: Difference between refinement operations. We can see that for all
objects in Linemod, the 3D-refinement has a big impact on lower thresholds.

Figure 6: Visualization of matching keypoints. Here we show PoseMatcher output
$\mathcal{C}$, where we assigned a normalized coordinate of the highest
confidence keypoint to the corresponding pixel. The last two rows are examples
from the YCB-V dataset.

##  5 Conclusions

We proposed PoseMatcher, a novel model free one-shot pose estimator based on
deep feature matching. Given a sequence of template images, we can reconstruct
a feature point cloud and extract matches from a query image. In order to
avoid using pre-trained descriptor models, we introduce a new training
pipeline that allows us to train from scratch. With this simple addition, we
show that we can improve OnePose++ [12] without any additional changes. We
build on top of OnePose++ by designing a new attention layer IO-Layer,
designed specifically for image to object matching. Additionally, we propose
improvements to the pipeline including object pruning and 3D based refinement.

Limitations. Unfortunately we found that MatchPose has limited domain
adaptation capability. If there is a large domain gap between the template
domain (ex. synthetic renderings) and queries MatchPose is unable to correctly
match keypoints, even coarsely. We specifically find this on YCB-V where each
scene contains different levels of sensor noise and illumination. We also
found symmetric objects particularly difficult, even though object pruning did
show improvements.

## References

  * [1] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  DensePose: Dense human pose estimation in the wild.  In CVPR, 2018.
  * [2] Dingding Cai, Janne Heikkilä, and Esa Rahtu.  Sc6d: Symmetry-agnostic and correspondence-free 6d object pose estimation.  arXiv preprint arXiv:2208.02129, 2022.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [4] Pedro Castro and Tae-Kyun Kim.  Crt-6d: Fast 6d object pose estimation with cascaded refinement transformers.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5746–5755, 2023.
  * [5] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [6] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018.
  * [7] Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, and Federico Tombari.  So-pose: Exploiting self-occlusion for direct 6d pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [8] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke.  Google scanned objects: A high-quality dataset of 3d scanned household items.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2553–2560. IEEE, 2022.
  * [9] Yang Fu and Xiaolong Wang.  Category-level 6d object pose estimation in the wild: A semi-supervised learning approach and a new dataset.  arXiv preprint arXiv:2206.15436, 2022.
  * [10] Rasmus Laurvig Haugaard and Anders Glent Buch.  Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6749–6758, 2022.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [12] Xingyi He, Jiaming Sun, Yuang Wang, Di Huang, Hujun Bao, and Xiaowei Zhou.  Onepose++: Keypoint-free one-shot object pose estimation without CAD models.  In Advances in Neural Information Processing Systems, 2022.
  * [13] Stefan Hinterstoisser, Vincent Lepetit, Naresh Rajkumar, and Kurt Konolige.  Going further with point pair features.  CoRR, abs/1711.04061, 2017.
  * [14] Tomáš Hodaň, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiří Matas, and Carsten Rother.  Bop: Benchmark for 6d object pose estimation.  ECCV, 2018.
  * [15] Yinlin Hu, Joachim Hugonot, Pascal Fua, and Mathieu Salzmann.  Segmentation-driven 6d object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [16] Shun Iwase, Xingyu Liu, Rawal Khirodkar, Rio Yokota, and Kris M. Kitani.  Repose: Fast 6d object pose refinement via deep texture rendering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [17] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020.
  * [18] Diederick P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations (ICLR), 2015\.
  * [19] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [20] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic.  Cosypose: Consistent multi-view multi-object 6d pose estimation.  In European Conference on Computer Vision, 2020.
  * [21] Fu Li, Hao Yu, Ivan Shugurov, Benjamin Busam, Shaowu Yang, and Slobodan Ilic.  Nerf-pose: A first-reconstruct-then-regress approach for weakly-supervised 6d object pose estimation.  arXiv preprint arXiv:2203.04802, 2022.
  * [22] Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox.  Deepim: Deep iterative matching for 6d pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [23] Zhigang Li, Gu Wang, and Xiangyang Ji.  Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation.  In ICCV, 2019.
  * [24] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, 2017.
  * [25] Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, and Wenping Wang.  Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 298–315. Springer, 2022.
  * [26] Xiaoyong Lu, Yaping Yan, Bin Kang, and Songlin Du.  Paraformer: Parallel attention transformer for efficient feature matching.  arXiv preprint arXiv:2303.00941, 2023.
  * [27] Markus Oberweger, Mahdi Rad, and Vincent Lepetit.  Making deep heatmaps robust to partial occlusions for 3D object pose estimation.  In ECCV, 2018.
  * [28] Keunhong Park, Arsalan Mousavian, Yu Xiang, and Dieter Fox.  Latentfusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10710–10719, 2020.
  * [29] Kiru Park, Timothy Patten, and Markus Vincze.  Pix2pose: Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation.  In ICCV, 2019.
  * [30] Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao.  Pvnet: Pixel-wise voting network for 6dof pose estimation.  In CVPR, 2019.
  * [31] Giorgia Pitteri, Aurélie Bugeau, Slobodan Ilic, and Vincent Lepetit.  3d object detection and pose estimation of unseen objects in color images with local surface embeddings.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [32] Giorgia Pitteri, Slobodan Ilic, and Vincent Lepetit.  Cornet: generic 3d corners for 6d pose estimation of new objects without retraining.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [33] Mahdi Rad and Vincent Lepetit.  Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth.  In ICCV, 2017.
  * [34] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  * [35] Ivan Shugurov, Fu Li, Benjamin Busam, and Slobodan Ilic.  Osop: A multi-stage one shot object pose estimation framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6835–6844, 2022.
  * [36] Juil Sock, Guillermo Garcia-Hernando, Anil Armagan, and Tae-Kyun Kim.  Introducing pose consistency and warp-alignment for self-supervised 6d object pose estimation in color images.  In 2020 International Conference on 3D Vision (3DV), pages 291–300, 2020.
  * [37] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  LoFTR: Detector-free local feature matching with transformers.  CVPR, 2021.
  * [38] Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou.  OnePose: One-shot object pose estimation without CAD models.  CVPR, 2022.
  * [39] Bugra Tekin, Sudipta N Sinha, and Pascal Fua.  Real-Time seamless single shot 6D object pose prediction.  In CVPR, 2018.
  * [40] Michał Tyszkiewicz, Pascal Fua, and Eduard Trulls.  Disk: Learning local features with policy gradient.  Advances in Neural Information Processing Systems, 33:14254–14265, 2020.
  * [41] Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, and Federico Tombari.  Self6D: Self-supervised monocular 6d object pose estimation.  In European Conference on Computer Vision (ECCV), pages 108–125, 2020.
  * [42] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji.  Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [43] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [44] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen.  Matchformer: Interleaving attention in transformers for feature matching.  In Asian Conference on Computer Vision, 2022.
  * [45] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes.  Robotics: Science and Systems (RSS), 2018.
  * [46] Yang Xiao, Xuchong Qiu, Pierre-Alain Langlois, Mathieu Aubry, and Renaud Marlet.  Pose from shape: Deep pose estimation for arbitrary 3d objects.  arXiv preprint arXiv:1906.05105, 2019.
  * [47] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin.  inerf: Inverting neural radiance fields for pose estimation.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1323–1330. IEEE, 2021.
  * [48] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic.  Dpod: 6d pose object detector and refiner.  In ICCV, 2019.
  * [49] Kaifeng Zhang, Yang Fu, Shubhankar Borse, Hong Cai, Fatih Porikli, and Xiaolong Wang.  Self-supervised geometric correspondence for category-level 6d object pose estimation in the wild.  arXiv preprint arXiv:2210.07199, 2022.


11institutetext: Center for Complex Systems and Brain Sciences and Center for
Future Mind, Florida Atlantic University, Florida, USA
11email<EMAIL_ADDRESS>
# The Role of Bio-Inspired Modularity in General Learning††thanks: Supported
by Machine Perception and Cognitive Robotics Labs

Rachel A. StClair 11 William Edward Hahn 11 Elan Barenholtz 11

###### Abstract

One goal of general intelligence is to learn novel information without
overwriting prior learning. The utility of learning without forgetting (CF) is
twofold: first, the system can return to previously learned tasks after
learning something new. In addition, bootstrapping previous knowledge may
allow for faster learning of a novel task. Previous approaches to CF and
bootstrapping are primarily based on modifying learning in the form of
changing weights to tune the model to the current task, overwriting previously
tuned weights from previous tasks. However, another critical factor that has
been largely overlooked is the initial network topology, or architecture.
Here, we argue that the topology of biological brains likely evolved certain
features that are designed to achieve this kind of informational conservation.
In particular, we consider that the highly conserved property of modularity
may offer a solution to weight-update learning methods that adheres to the
learning without catastrophic forgetting and bootstrapping constraints. Final
considerations are then made on how to combine these two learning objectives
in a dynamical, general learning system.

###### Keywords:

Architecture General Learning Modularity

##  1 Introduction

###  1.1 Knowledge Preservation

A basic goal of most approaches to artificial general intelligence is the
ability to learn multiple tasks without overwriting what has previously been
learned (i.e. catastrophic forgetting, or CF). The potential utility of
learning without forgetting, or ’knowledge preservation’, is twofold: first,
the system can return to previously learned tasks after learning something
new. In addition, bootstrapping previous knowledge may allow for faster
learning of a novel task through transfer learning. Both of these potential
benefits of preserving learned information may serve to conserve resources.
Allowing the same computational hardware to be ‘multiplexed’ for another task,
rather than requiring allocation of distinct resources which reduces the
required energy resources involved in the learning process. In the former,
multiplexed information is stored in such a way that the system can switch
between already learned tasks while retaining performance. In the latter case,
information is accessed for learning new tasks in a contextually relevant
manner. Multiplexed information is thus stored and retrieved in a way that
facilitates the conservation and use of prior learning, broadening the ability
of a learning system to multiple task domains.

###  1.2 Previous Approaches

A significant body of recent work has explored approaches to CF and
bootstrapping in relation to multi-layer neural networks or deep learning (DL)
[41]. Learning in these systems consists of updating the weights between nodes
according to their respective partial derivatives calculated from a loss
function (i.e. backpropagation). With the recent implementation of auto-
differentiation and parallel computing devices, progress in backpropagation-
based DL has rapidly emerged as the dominant approach in AI [27]. Previous
approaches to overcoming CF in DL models have involved strategies with regard
to the learning process itself. Most commonly in the case of bootstrapping,
ensemble networks are re-trained on resampled datasets with little improvement
[23]. Transfer learning is also used as a bootstrapping method in which
previously-learned features are imported as the starting weights for later
update [39]. CF approaches are more diverse, mostly involving modification to
the weight update methods. Notably, Elastic Weight Consolidation has shown
promising results in overcoming CF in sequences of Atari tasks by slowing down
updates on previously encoded weights [15, 1]. However, these approaches have
not been widely adopted, likely because the techniques do not demonstrate
knowledge preservation on a level necessary for general intelligence. We
suspect this is in part due to the fundamental nature of backpropagation in DL
models to rewrite learning; where weights are updated most heavily towards
current learning at the expense of weights learned in more distant tasks.

###  1.3 Network Topology

Another potential approach that has thus far been largely neglected is to
consider the topology of the learning system — that is, its ‘architecture’.
While there is a significant body of work on architecture optimization for
other metrics of learning [21, 4, 7, 36, 20, 40, 28], to date there has been
little work considering architectural choices in relation to CF and/or
bootstrapping. Fortunately, hundreds of million years of evolution has
generated highly conserved, vertebrate and mammalian brain architectures that
have been optimized with regard to intelligent functions including, it is
reasonable to assume, the preservation of learning without CF.

One potentially promising property of biological nervous system architectures,
which may serve to overcome the pitfalls of CF, is the presence of modular
architectures such as the cortex and subcortical structures. Modularity is a
highly conserved property that is present across many organisms with general
learning abilities. Complex neural architectures are thought to have emerged
in the Cambrian explosion, 540 million years ago, with records of both camera
eyes and the emergence of animals with complex behaviors [33]. In contemporary
species, the four basic brain structures (Telencephalon, Cerebellum,
Diencephalon, Mesencephalon) have been preserved across nearly all mammalian
species and most avian, reptile, shark brains, and some fish brains [24, 42,
16, 19, 2]. Insects have their own highly preserved modular nervous-system
structures as well [14].

In humans, modularity is highly pervasive. For example, the cortex processes
incoming sensory information while the basal ganglia controls motor movements
in behavioral learning [25, 31]. These structures are specialized not only
with regard to the types of information they processed (e.g. photon patterns
for V1 stream vs. layers 5/6 cortical output from cortex), but also with
regard to their neuron types as well as their morphological structures.
Communication between the two structures occurs in specialized. limited
routes, cortico-basal ganglia networks, which serve to integrate information
between the two regions [18]. The result of this particular organization is a
cortex that is optimized for sensory information, while the basal ganglia is
optimized to interpret cortical information for motor movement implementation
[3].

Here, we propose that such modular architectures may also serve to preserve
learning during the update process by separating different types of
computation (specialization) and storing the learning across un-updatable
boundaries (segregation) which communicate through specific upstream pathways
(integration). In artificial systems, this modularity may be modeled in the
form of segregation of weight updates based on local update rules within a
module. Previous work on artificial systems has looked at the effects of
modularity for generalization, but using different learning metrics [9, 8,
22]. Promising results on the effects of modularity for generalizing to out-
of-distribution data has been shown in the Badger architecture, which relies
on nested self-sufficient computational units which communicate for a shared
objective [26]. Badger is an agent based model in the sense that each agent,
or small self-sufficient network, updates in an internal loop, while the outer
loop optimizes the collection of homogeneous agents. However, badger does not
incorporate within-agent modularity which we believe is essential to solving
the knowledge preservation problem.

While we have attempted to make the specific case for modularity as an example
of a potentially important design principle in natural systems, we propose
that knowledge preservation should serve as a general guiding framework in
architecture design that may give rise to dynamical learning systems capable
of some level of generality. The key idea is modeled from the human brain
where it is likely that network topology is perhaps at least, if not more
important than later learning algorithms in achieving this goal. This allows
us to avoid ending up with a system that is doing the bulk of the learning
process after it has been configured. In a system that prioritizes initial
connectivity, this manifests as selecting appropriate modularity over a
variety of different domain tasks. Rewriting existing learning prevents
sharing of computational resources for novel tasks. This is often the problem
in designing AGI from theoretical systems that don’t take resource constraints
into account, such as universal Turing machines and AIXI [35, 13].

##  2 Possible Implementation Approaches

Many methods could be conceived for discovering the inductive biases that
modularity has to offer learning system architectures. Crucially, there is a
tradeoff between initial connectivity and later weight modification. A strong
dynamical learning system would utilize initial topology search as a prior
learning experience to later be exploited during the weight modification
learning phase.

Consider regions separated by un-updatable boundaries that delineate the
extent of backpropagation. These modules could optimize without overwriting
each other, while still being able to share resources when relevant (i.e.
bootstrapping). This approach also limits rewriting prior learning for the
sake of the current learning, where modules are activated at different times
depending on their relevance to the current tasks. Furthermore, modules
dedicated to computational tasks, instead of behavioral tasks (e.g. memory,
sensory, self-activation, etc.) may serve as a more biologically inspired
approach, since this is largely what we observe between cortical and
subcortical structures.

Evolutionary searches are particularly useful in discovering unique solutions
to optimization problems, which can be extended here. Although computationally
extravagant, one promising candidate would be an evolutionary approach that
couples network topology and embodiment search to different learning
environments that also evolve over time. Derivatives of this approach would be
useful, especially those focusing on modularity and learning. The advantage to
this method is that appropriate submodules will likely be paired with the
particular sensory receptors given to the agent and the types of learning that
needs to be delegated throughout the system in ways that may not be currently
known.

Other approaches include intelligently designed architectures where
architecture is random, learned, selected by an operator, and most commonly in
combination. Random networks, such as [5], have seen interesting results and
may offer some potential insights into the core properties of learning,
however they will likely have to be harnessed in a constructive way to get
desired behavior, as in [21]. Similar to random learning, methods exist for
crystallizing order from disorder in unorganized networks as in the case of
[10, 12], leading to module-like structures. Still, there are other
conceivable approaches involving learning functions that select sequences of
computational architecture pathways and more. More extensive studies could
iteratively search through all possible configurations of N-node networks to
discover what modularity benefits particular learning environments. Even hand-
designed simple models that start to test the questions of module design and
learning without forgetting would be useful in answering the questions
presented here.

##  3 Benchmarking Generality

In order to assess a model’s knowledge preservation, it must be able to
demonstrate that it can learn a sequence of tasks in a variety of domains and
return to those tasks with improvements in computational costs and/or learning
time. Below we describe initial tests to measure generality in terms of
learning without forgetting and bootstrapping. However, as generality scales,
more advanced testing is likely necessary.

####  3.0.1 Knowledge Preservation Metrics

Learning without catastrophic forgetting is typically measured by performance
on a sequence of tasks. Here, we propose it is imperative to test on sequences
in different domains (e.g. vision classification and time-series forecasting)
to demonstrate generality over narrow-learning.

####  3.0.2 Boostrapping A-Priori Knowledge Metrics

Measuring bootstrapping a-priori knowledge is slightly more challenging due to
the effects seen in transfer learning. The goal here is to see an increased
speed in learning task B after learning task A. Again, in trying to capture
generality, it is useful to compare tasks from different domains so the
distribution of features in the data is different enough that it’s possible to
measure using contextual knowledge instead of the re-use of features. This
later point is what occurs in transfer learning. As in the case of the popular
technique of using pre-trained ResNet on Imagenet and vision based tasks, the
transferred weights capture many features which would have to have been
learned in the secondary network either way [39]. Here, we also want to be
able to use learned features to aid in learning a new task, but we want to
ensure that these are general features that can be maximally shared across all
tasks the system may be expected to learn. Teasing out this difference is
aided in investigating how data flows through different parts of the system’s
topology. For example, the Fusiform Face Area is shown to activate when
distinguishing between faces and also in bird experts for distinguishing
between birds [32]. This multi-use activation of the same subnetwork is one
example of bootstrap learning. We might expect similar submodules to be
redeployed across different kinds of learning domains.

####  3.0.3 Challenges

We recognize that adding modularity alone will probably not multiplex
information in a way that provides human-level general intelligence. The works
of Coward, Buszaki, and Turing suggest algorithms employed by the human brain
may give rise to this generality based on eliciting sequences of computation
across different modules [3, 11, 34]. Furthermore, special message passing
algorithms between the regions would benefit from that of publish and
subscribe models; that regions in the system can communicate with each other
when necessary.

##  4 Conclusion

We have highlighted topology as an important topic for future research, as
well as the need for algorithms that allow for learning without rewriting
previous learning. We suspect modularity in biological intelligent systems
plays a crucial role in achieving this need while conserving resources. This
paper also contributes to discussions on AGI agent architectures by suggesting
some useful considerations on how to test for generality in regards to CF and
bootstrapping. Works from [38, 26, 17, 30, 1] have attempted to tackle these
issues separately, while others [29, 37] have come from the angle of single-
use systems, all of which overwrite prior learning to a unsatisfactory degree.
Still, some works have come from the position of down-scaling theoretical
models without resource constraints [6, 13, 35]. While these approaches are
useful, we find the proposed approach is more appropriate for the pursuit of
general intelligence. We believe this paper provides a framework for
investigating computationally tractable general learning systems. Promising
directions involve evolutionary and intelligent topology design that utilizes
modularity to minimize rewriting prior learning. We hope this paper serves as
a tool for those interested in benchmarking early progress in generality.
There is room for many future works to provide insight into the approach
presented here as well as how to measure generality in terms of bootstrapping
and learning new without forgetting old. We have tried to clarify how
modularity may offer a method of learning without rewriting and the role it
plays in topology design. Much work needs to be done to improve that
understanding and find ways to appropriately incorporate modularity in AGI
frameworks.

## References

  * [1] Aich, A.: Elastic weight consolidation (ewc): Nuts and bolts. arXiv preprint arXiv:2105.04093 (2021)
  * [2] Bruce, L.L.: Evolution of the Brain in Reptiles, pp. 1295–1301. Springer Berlin Heidelberg, Berlin, Heidelberg (2009)
  * [3] Coward, L.A.: A system architecture approach to the brain: From neurons to consciousness. Nova Publishers (2005)
  * [4] Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. The Journal of Machine Learning Research 20(1), 1997–2017 (2019)
  * [5] Gershenson, C.: Introduction to random boolean networks. arXiv preprint nlin/0408006 (2004)
  * [6] Goertzel, B., Pennachin, C.: Artificial general intelligence, vol. 2. Springer (2007)
  * [7] Goudarzi, A., Teuscher, C., Gulbahce, N., Rohlf, T.: Emergent criticality through adaptive information processing in boolean networks. Physical review letters 108(12), 128702 (2012)
  * [8] Goyal, A., Bengio, Y.: Inductive biases for deep learning of higher-level cognition. arXiv preprint arXiv:2011.15091 (2020)
  * [9] Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., Schölkopf, B.: Recurrent independent mechanisms. arXiv preprint arXiv:1909.10893 (2019)
  * [10] Gutowitz, H.: Cellular automata: Theory and experiment. MIT press (1991)
  * [11] György Buzsáki, M.: The brain from inside out. Oxford University Press (2019)
  * [12] Hebb, D.O.: The organisation of behaviour: a neuropsychological theory. Science Editions New York (1949)
  * [13] Hutter, M.: A gentle introduction to the universal algorithmic agent aixi (2003)
  * [14] Ito, K., Shinomiya, K., Ito, M., Armstrong, J.D., Boyan, G., Hartenstein, V., Harzsch, S., Heisenberg, M., Homberg, U., Jenett, A., Keshishian, H., Restifo, L.L., Rössler, W., Simpson, J.H., Strausfeld, N.J., Strauss, R., Vosshall, L.B.: A systematic nomenclature for the insect brain. Neuron 81(4), 755–765 (2014). https://doi.org/https://doi.org/10.1016/j.neuron.2013.12.017, https://www.sciencedirect.com/science/article/pii/S0896627313011781
  * [15] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)
  * [16] Kotrschal, A., Kotrschal, K.: Fish Brains: Anatomy, Functionality, and Evolutionary Relationships, pp. 129–148. Springer International Publishing, Cham (2020)
  * [17] Koutník, J., Cuccu, G., Schmidhuber, J., Gomez, F.: Evolving large-scale neural networks for vision-based reinforcement learning. In: Proceedings of the 15th annual conference on Genetic and evolutionary computation. pp. 1061–1068 (2013)
  * [18] Lalo, E., Thobois, S., Sharott, A., Polo, G., Mertens, P., Pogosyan, A., Brown, P.: Patterns of bidirectional communication between cortex and basal ganglia during movement in patients with parkinson disease. Journal of Neuroscience 28(12), 3008–3016 (2008)
  * [19] Larsell, O.: The cerebellum of reptiles: lizards and snake. Journal of Comparative Neurology 41(1), 59–94 (1926)
  * [20] Lee, D.S.: Evolution of regulatory networks towards adaptability and stability in a changing environment. Physical Review E 90(5), 052822 (2014)
  * [21] Lukoševičius, M., Jaeger, H., Schrauwen, B.: Reservoir computing trends. KI-Künstliche Intelligenz 26(4), 365–371 (2012)
  * [22] Werle van der Merwe, A.: Investigating the evolution of modularity in neural networks. Ph.D. thesis, Stellenbosch: Stellenbosch University (2020)
  * [23] Nixon, J., Lakshminarayanan, B., Tran, D.: Why are bootstrapped deep ensembles not better? In: ”I Can’t Believe It’s Not Better!”NeurIPS 2020 workshop (2020)
  * [24] Nomura, T., Izawa, E.I.: Avian brains: Insights from development, behaviors and evolution. Development, Growth & Differentiation 59(4), 244–257 (2017). https://doi.org/https://doi.org/10.1111/dgd.12362, https://onlinelibrary.wiley.com/doi/abs/10.1111/dgd.12362
  * [25] Ring, H., Serra-Mestres, J.: Neuropsychiatry of the basal ganglia. Journal of Neurology, Neurosurgery & Psychiatry 72(1), 12–21 (2002)
  * [26] Rosa, M., Afanasjeva, O., Andersson, S., Davidson, J., Guttenberg, N., Hlubuček, P., Poliak, M., Vítku, J., Feyereisl, J.: Badger: Learning to (learn [learning algorithms] through multi-agent communication). arXiv preprint arXiv:1912.01513 (2019)
  * [27] Schmidhuber, J.: Deep learning in neural networks: An overview. Neural networks 61, 85–117 (2015)
  * [28] Schölkopf, B., Locatello, F., Bauer, S., Ke, N.R., Kalchbrenner, N., Goyal, A., Bengio, Y.: Toward causal representation learning. Proceedings of the IEEE 109(5), 612–634 (2021)
  * [29] Silver, D., Singh, S., Precup, D., Sutton, R.S.: Reward is enough. Artificial Intelligence p. 103535 (2021)
  * [30] Stanley, K.O., Clune, J., Lehman, J., Miikkulainen, R.: Designing neural networks through neuroevolution. Nature Machine Intelligence 1(1), 24–35 (2019)
  * [31] Sutherland, R., Whishaw, I., Kolb, B.: Contributions of cingulate cortex to two forms of spatial learning and memory. Journal of Neuroscience 8(6), 1863–1872 (1988)
  * [32] Tong, M.H., Joyce, C.A., Cottrell, G.W.: Why is the fusiform face area recruited for novel categories of expertise? a neurocomputational investigation. Brain research 1202, 14–24 (2008)
  * [33] Trestman, M.: The cambrian explosion and the origins of embodied cognition. Biological Theory 8(1), 80–92 (2013)
  * [34] Turing, A.: Intelligent machinery. 1948. The Essential Turing p. 395 (1969)
  * [35] Turing, A.M.: Computing machinery and intelligence. In: Parsing the turing test, pp. 23–65. Springer (2009)
  * [36] Vanschoren, J.: Meta-learning: A survey. arXiv preprint arXiv:1810.03548 (2018)
  * [37] Wang, J., Elfwing, S., Uchibe, E.: Modular deep reinforcement learning from reward and punishment for robot navigation. Neural Networks 135, 115–126 (2021)
  * [38] Wang, R., Lehman, J., Clune, J., Stanley, K.O.: Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 (2019)
  * [39] Weiss, K., Khoshgoftaar, T.M., Wang, D.: A survey of transfer learning. Journal of Big data 3(1), 1–40 (2016)
  * [40] Williams, S., Yaeger, L.: Evolution of neural dynamics in an ecological model. Geosciences 7(3), 49 (2017)
  * [41] Xie, Z., He, F., Fu, S., Sato, I., Tao, D., Sugiyama, M.: Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. Neural Computation pp. 1–30 (2020)
  * [42] Yopak, K.E., Lisney, T.J., Darlington, R.B., Collin, S.P., Montgomery, J.C., Finlay, B.L.: A conserved pattern of brain scaling from sharks to primates. Proceedings of the National Academy of Sciences 107(29), 12946–12951 (2010). https://doi.org/10.1073/pnas.1002195107, https://www.pnas.org/content/107/29/12946