# Design rules for active control of narrowband thermal emission using phase-
change materials

Maxime Giteau  Mitradeep Sarkar  Maria Paula Ayala  Michael T. Enders  Georgia
T. Papadakis<EMAIL_ADDRESS>ICFO-Institut de Ciencies Fotoniques,
The Barcelona Institute of Science and Technology, Castelldefels, Barcelona
08860, Spain

###### Abstract

We propose an analytical framework to design actively tunable narrowband
thermal emitters at infrared frequencies. We exemplify the proposed design
rules using phase-change materials (PCM), considering dielectric-to-dielectric
PCMs (e.g. GSST) and dielectric-to-metal PCMs (e.g. $\mathrm{VO_{2}}$). Based
on these, we numerically illustrate near-unity ON-OFF switching and
arbitrarily large spectral shifting between two emission wavelengths,
respectively. The proposed systems are lithography-free and consist of one or
several thin emitter layers, a spacer layer which includes the PCM, and a back
reflector. Our model applies to normal incidence, though we show that the
behavior is essentially angle-independent. The presented formalism is general
and can be extended to any mechanism that modifies the optical properties of a
material, such as electrostatic gating or thermo-optical modulation.

The ability to control the spectrum, direction, and polarization of thermal
emission is critical for applications including infrared (IR) sources [1, 2],
thermal camouflage [3], radiative cooling [4, 5] and energy conversion [6]. In
particular, the possibility of generating spectrally narrowband IR emission
has been the object of a very rich literature [7, 8, 9, 10, 11, 12, 13, 14].
While most architectures involve in-plane patterning and/or a large number of
layers to reduce the emission bandwidth, surface phonon polaritons (SPhPs)
offer naturally narrowband resonances owing to their large material quality
factors [15]. In particular, it has been recently shown that few-monolayer
SPhP-based emitters used in a Salisbury screen configuration [16, 17] (a
3-layer structure consisting of an emitter, a dielectric spacer, and a back
reflector, forming a Fabry-Perot cavity whose resonance wavelength matches
that of the emitter) can achieve strong narrowband emission [18].

Another active area of nanophotonics is the active control of optical
properties [19, 20]. Such dynamic modulation can take the form of electrical
gating [21, 22, 23, 24, 2, 13], optical biasing [25] or applied strain [26].
In the mid-IR region, the spectral range of interest for thermal emission, a
popular approach for active tuning is phase-change materials (PCMs). These
materials show a dramatic reversible and (for some of them) non-volatile
change in their optical properties upon heating, leading to a very different
spectral response [27, 28, 29, 30, 31]. Several techniques have been developed
to induce the phase change beyond simple thermal heating, which is slow and
can result in significant hysteresis [32]. For volatile PCMs, the phase change
can be triggered by laser heating, with a characteristic switching time of a
few tens of nanoseconds [33]. Applying an electrical current can also result
in ultrafast switching in a few nanoseconds [34]. In the case of non-volatile
PCMs, crystallization and amorphization are typically triggered by short (in
the order of nanoseconds, depending on the film thickness) laser pulses [35].

Two classes of materials emerge from this description: those switching from
one dielectric phase to another, with different refractive indices, such as
some GeSbTe (GST) compounds [36, 37, 38, 35], and those switching from a
dielectric to a metallic phase, such as $\mathrm{VO_{2}}$ [39, 32]. PCMs have
been studied for various applications including non-volatile optical switching
[36, 40, 41, 42, 43, 44], beam switching and bifocal lensing [45], homeostasis
[39], radiative cooling [32] and thermal camouflage [46, 47]. They are
particularly relevant for spectrally-tunable narrowband sources [48, 36, 49,
14, 50], with applications in spectroscopy as well as thermophotovoltaics.
However, simple, lithography-free structures tend to have relatively broadband
emissivity [28]. Tunable narrowband sources have been achieved only for more
complex structures, with an emissivity that is usually not unitary over the
whole range of operation [48, 50, 14]. Furthermore, all these devices have
limited spectral tunability as they rely on the temperature dependence of a
single material’s resonance wavelength.

Figure 1: Ideal narrowband spectral emissivity upon the phase transition of a
PCM for two configurations: (a) ON-OFF switching and (b) Spectral shifting
from one resonance wavelength to another.

In this work, we propose a simple framework combining SPhPs-based resonances
and PCMs in a Salisbury screen configuration to design lithography-free
narrowband IR emitters with different properties upon phase transition of the
PCM. We first derive analytical conditions for unitary and zero emissivity. We
then apply this versatile framework to two configurations. In the first, the
emissivity of a single-resonance emitter is turned ON and OFF upon phase
transition (Fig. 1(a)). In the second, which considers two arbitrary emission
wavelengths, the emission peak switches from one wavelength to the other (Fig.
1(b)). In both cases, we quantify the performance of optimized devices,
considering both idealized and real materials. Finally, we show that the
emissivity of these structures shows very little angular dependence, making
them relevant for spectrally-tunable diffuse narrowband thermal emission.

We consider a two-layer stack consisting of an emitter with complex refractive
index $n_{e}$ and thickness $d_{e}$ on top of a spacer with a real refractive
index $n_{s}$ and thickness $d_{s}$. It is surrounded by two semi-infinite
media: an upper medium with real refractive index $n_{i}$ and a back reflector
with a complex refractive index $n_{b}$. The general architecture is
illustrated in Fig. 2(a). The emitter layer supports a SPhP resonance at
wavelength $\lambda_{e}$, and its relative permittivity is
$\varepsilon_{e}=n_{e}^{2}$. We also define the optical thickness of the
spacer as $\Delta_{s}=n_{s}d_{s}$. We restrict our analysis to normal
incidence, and discuss its extension for oblique angles towards the end of the
manuscript. Using Kirchhoff’s law of thermal radiation, we describe emissivity
as $\epsilon=1-R$, where $R=|r|^{2}$ is the reflectivity, $r$ being the
Fresnel reflection coefficient of the system. We consider the incident medium
to be air ($n_{i}=1$) and the back medium to be a perfect reflector
($1/n_{b}\to 0$). The following results are derived and generalized in the
Supplemental Material [51], starting from ref. [52]. Assuming
$\Im(\varepsilon_{e}(\lambda_{e}))\gg 1$ ($\Im$ meaning the imaginary part),
the emitter thickness required to achieve unitary emissivity (resonance) at
wavelength $\lambda_{e}$ is:

$d_{e}=\frac{\lambda_{e}}{2\pi\ \Im(\varepsilon_{e}(\lambda_{e}))},$ (1)

whereas the spacer’s optical thickness is:

$\Delta_{s}^{\epsilon=1}=\left(m+\frac{1}{2}\right)\frac{\lambda_{e}}{2},\
m\in\mathbb{N},$ (2)

where $\mathbb{N}$ is the set of all natural numbers, including zero. On the
contrary, imposing the condition of zero emissivity and considering the same
emitter thickness (Eq. 1), the spacer’s optical thickness becomes:

$\Delta_{s}^{\epsilon=0}=m\frac{\lambda_{e}}{2},\ m\in\mathbb{N}$ (3)

Therefore, if $\Im(\varepsilon_{e})\gg 1$, then $d_{e}\ll\lambda_{e}$ (Eq. 1),
hence we can neglect the impact of the front layer on the resonance
conditions, leading to the resonance and anti-resonance wavelengths of a
simple Fabry-Perot cavity (Eqs. 2-3).

Figure 2:  (a) Salisbury screen configuration considered for unitary and zero
emissivity: a thin emitter is placed on top of a dielectric spacer, above a
back reflector. (b-c) Two ways to modify the optical thickness of the spacer.
(b) When the PCM has a dielectric-to-dielectric transition, it can be used as
the spacer. (c) When the PCM has a dielectric-to-metal transition, it can be
used as a reflector hiding a second spacer which only plays a role in the
dielectric phase.

The emissivity of such a system can be actively tuned between 0 and 1 by
modifying the optical thickness of the spacer, $\Delta_{s}=n_{s}d_{s}$. This
can be achieved with PCMs. In particular, a PCM that switches from one
dielectric phase to another dielectric phase, with different refractive
indices $n_{s}^{\alpha}$ and $n_{s}^{\beta}$, can be used directly as the
spacer. The index change upon phase transition modulates the optical thickness
and thus the emissivity of the system (Fig. 2(b)). Alternatively, a PCM that
switches from a dielectric phase to a metallic phase can be inserted between
two spacer layers such that the thickness of the cavity itself changes upon
phase transition (Fig. 2(c)). In its metallic phase, the PCM behaves like a
perfect reflector, and the spacer thickness is $d_{s}^{\alpha}$, while in its
dielectric phase, the spacer becomes the combination of the two spacers and
the PCM, with a total thickness $d_{s}^{\beta}$. Note that for the two-layer
model (Fig. 2(a)) to be analytically valid, the PCM should either be extremely
thin or have the same refractive index as the spacer when the PCM is in its
dielectric phase. The model allows for unitary emissivity at the resonance
wavelength to be achieved either in phase $\alpha$ or $\beta$ (depending on
the thickness of the spacer(s)).

In the following, we illustrate this framework with two examples, the first
one pertaining to ON-OFF switching of thermal emission at a certain wavelength
upon phase transition, and the second one regarding spectral shifting between
two frequencies, where the emission wavelength is arbitrarily tuned. In both
cases, we present the results obtained with both idealized and real materials.
The simulations are performed using an in-house transfer matrix method [53].
The refractive index spectra used for all materials considered are represented
in the Supplemental Material [51]. The materials considered in this work, with
the exception of the PCMs, show negligible dependence in their permittivity
with temperature.

As a first illustration, we design a system with narrowband unitary emissivity
in phase $\alpha$ and zero emissivity in phase $\beta$. The figure of merit
for evaluating the performance of the switch can be defined as the difference
in emissivity between the two phases, at the resonance wavelength
$\lambda_{e}$ of the emitter:

$\varphi_{switch}=\epsilon^{\alpha}(\lambda_{e})-\epsilon^{\beta}(\lambda_{e}).$
(4)

For ideal materials, $\varphi_{switch}$ should be unity. By considering Eqs.
2-3 in this particular system, we derive the ON and OFF conditions,
respectively, for the optical thickness of the PCM spacer:

$\displaystyle\Delta_{s}^{\alpha}$
$\displaystyle=\left(m_{\alpha}+\frac{1}{2}\right)\frac{\lambda_{e}}{2},\
m_{\alpha}\in\mathbb{N},$ (5) $\displaystyle\Delta_{s}^{\beta}$
$\displaystyle=m_{\beta}\frac{\lambda_{e}}{2},\ m_{\beta}\in\mathbb{N},$ (6)

which imposes

$\frac{\Delta_{s}^{\alpha}}{\Delta_{s}^{\beta}}=\frac{2m_{\alpha}+1}{2m_{\beta}}.$
(7)

Note that, to minimize the thickness of the spacer, the indices $m_{\alpha}$
and $m_{\beta}$ should be as small as possible.

Here, we consider the configuration where the PCM is used as the spacer (Fig.
2(b)), as it is the simpler configuration. Based on Eq. 7, the refractive
index ratio should correspond to:

$\frac{n_{s}^{\alpha}}{n_{s}^{\beta}}=\frac{2m_{\alpha}+1}{2m_{\beta}}.$ (8)

Such a configuration is only constrained by the refractive indices of the PCM.
One promising material for this application is
$\mathrm{Ge_{2}Sb_{2}Se_{4}Te_{1}}$ (GSST), which shows a large refractive
index ratio, close to $3/2$, in the IR frequency range between its crystalline
phase ($n^{s}_{\alpha}\approx 4.60$) and its amorphous phase
($n^{s}_{\beta}\approx 3.19$) [54]. Therefore, from Eq. 8, we consider
$m_{\alpha}=m_{\beta}=1$. For the sake of illustration, here, we consider a
SiC [55, 56] emitter with a resonance wavelength $\lambda_{e}=12.6\
\mathrm{\mu m}$. Its thickness, determined from Eq. 1, is $d_{e}=3.7\
\mathrm{nm}$.

Figure 3: Spectral emissivity for both PCM phases, shown with the blue and red
curves, in the ON-OFF switching configuration, using a 3.7 nm-thick SiC
emitter with a dielectric-to-dielectric PCM. Dashed line: Assuming an
idealized PCM and a perfect back reflector. Solid line: Considering a GSST
spacer and a silver back reflector.

For the ideal configuration, we consider a dispersionless and non-absorptive
GSST with refractive indices $n_{s}^{\alpha}=4.60$ and $n_{s}^{\beta}=3.19$ on
top of a perfect reflector, so as to satisfy the refractive index ratio of Eq.
8 for $m_{\alpha}=m_{\beta}=1$. The figure of merit is maximized for a spacer
thickness $d_{s}=2.015\ \mathrm{\mu m}$, corresponding to near-ideal ON-OFF
switching, with $\varphi_{switch}=0.996$ (Fig. 3). This spacer thickness is
the average of the values predicted by eqs. 5-6 ($2.054\ \mathrm{\mu m}$ and
$1.975\ \mathrm{\mu m}$ , respectively). The width of the resonance is
determined by that of the SPhP mode in SiC. We note that even with an
imperfect refractive index ratio, we can achieve a figure of merit extremely
close to unity, demonstrating the imperfection tolerance of the approach.

For a realistic configuration, we consider the complex refractive indices of
GSST [54] and silver [57], for the PCM layer and back reflector, respectively.
The PCM thickness is set to $d_{s}=2.035\ \mathrm{\mu m}$ (very close to the
ideal case), as it maximizes the figure of merit. As can be seen in Fig. 3,
the ON-OFF switching remains extremely effective, with
$\varphi_{switch}=0.980$. Nonetheless, we note that emissivity in the
crystalline phase is significantly broadened due to parasitic absorption in
the PCM and the Ag back reflector (3). The origin of this broadening and
strategies to mitigate it are discussed in the Supplemental Material [51].

The framework discussed above can be extended to several thin emitters stacked
on top of each other (or an emitter with several SPhP resonances) to achieve a
spectral shift in the emissivity in the two PCM phases. This can be achieved
as long as there is no significant spectral overlap between the SPhP modes of
the two thin emitters. Here, we consider a system with two emitters, referred
to as $e_{1}$ and $e_{2}$ henceforth. These emitters support a SPhP resonance
at wavelengths $\lambda_{e_{1}}$ and wavelength $\lambda_{e_{2}}$,
respectively. The aimed operation is to achieve unitary emission at wavelength
$\lambda_{e_{1}}$ in phase $\alpha$ and unitary emission at wavelength
$\lambda_{e_{2}}$ in phase $\beta$ (Fig. 1(b)). We define a figure of merit to
quantify the quality of the spectral shift, which should approach unity in the
ideal case:

$\varphi_{shift}=\frac{1}{2}\left[\left(\epsilon^{\alpha}(\lambda_{e_{1}})-\epsilon^{\beta}(\lambda_{e_{1}})\right)+\left(\epsilon^{\beta}(\lambda_{e_{2}})-\epsilon^{\alpha}(\lambda_{e_{2}})\right)\right]$
(9)

We assume the spacer is dispersionless, such that
$n_{s}^{\alpha/\beta}(\lambda_{e_{1}})=n_{s}^{\alpha/\beta}(\lambda_{e_{2}})=n_{s}^{\alpha/\beta}$.
Based on Eqs. 2-3, the system must satisfy at once the following four
equations:

$\displaystyle\Delta_{s}^{\alpha}$
$\displaystyle=\left(m_{1}+\frac{1}{2}\right)\frac{\lambda_{e_{1}}}{2},\
m_{1}\in\mathbb{N},$ (10) $\displaystyle\Delta_{s}^{\alpha}$
$\displaystyle=m_{2}\frac{\lambda_{e_{2}}}{2},\ m_{2}\in\mathbb{N},$ (11)
$\displaystyle\Delta_{s}^{\beta}$
$\displaystyle=\left(m_{3}+\frac{1}{2}\right)\frac{\lambda_{e_{2}}}{2},\
m_{3}\in\mathbb{N},$ (12) $\displaystyle\Delta_{s}^{\beta}$
$\displaystyle=m_{4}\frac{\lambda_{e_{1}}}{2},\ m_{4}\in\mathbb{N},$ (13)

which imposes a wavelength ratio:

$\frac{\lambda_{e_{2}}}{\lambda_{e_{1}}}=\frac{2m_{1}+1}{2m_{2}}=\frac{2m_{4}}{2m_{3}+1}$
(14)

It is impossible to find four integers ($m_{1}$-$m_{4}$) that satisfy Eq. 14
exactly. Nonetheless, one can achieve a spectral shift close to optimal if the
two integer ratios in Eq. 14 are close to each other. We emphasize that this
approach allows arbitrarily large spectral shifts in emissivity, provided the
availability of materials with the desired resonance frequencies, the
transparency of the spacer material in that spectral range, and the bandwidth
of blackbody radiation. This is in contrast to previous works that rely on
tuning the resonance frequency of a single emitter material, therefore
generally leading to rather limited frequency shifts [48, 50, 14].

To illustrate this case, we consider a system where the PCM has a dielectric-
to-metal phase transition (Fig. 2(c)). The advantage of this configuration is
that any optical thickness ratio can be achieved by adjusting the relative
thickness of the spacers, relaxing the number of constraints. We consider two
emitters: hexagonal boron nitride (hBN), which supports a SPhP mode near
$\lambda_{e_{1}}\approx 7.3\ \mu m$ [58, 59] and $\rm{\alpha}$-SiC [55, 56],
which supports a SPhP mode near $\lambda_{e_{2}}\approx 12.6\ \mu m$, with
thicknesses $2.3$ nm and $3.7$ nm, respectively (calculated from Eq. 1). Such
small thicknesses are particularly attractive for low-dimensional layered
materials, such as hBN, which can easily be exfoliated into few-monolayer
flakes. Nevertheless, the principle of operation of the device does not change
for thicker emitter layers, other than slightly shifting the position of the
central resonance (Eq. 1).

Figure 4: Spectral shifting between two wavelengths using two SPhP emitters
and a PCM with a phase transition from metal to dielectric surrounded by two
dielectric spacers. Dashed line: Assuming a perfect dielectric-to-metallic PCM
and a perfect reflector Solid line: Considering $\mathrm{VO_{2}}$ as the PCM
and a silver back reflector.

The wavelength ratio $\frac{\lambda_{e_{2}}}{\lambda_{e_{1}}}\approx 1.73$ can
be well approximated by considering the set of integers $m_{1}=3$, $m_{2}=2$,
$m_{3}=3$ and $m_{4}=6$. This leads to $\Delta_{s}^{\alpha}\approx 12.7\
\mathrm{\mu m}$ and $\Delta_{s}^{\beta}\approx 22.0\ \mathrm{\mu m}$. For the
spacer, we consider KBr which is transparent and dispersionless in the
spectral region of interest, with $n_{s}=1.52$.

In the ideal case, we consider a perfect back reflector. We assume that the
PCM has a negligible thickness so that it can be ignored in the dielectric
phase, while it behaves like a perfect reflector in the metallic phase (very
large refractive index). We obtain a near-ideal spectral shift, with a figure
of merit $\varphi_{shift}=0.992$ by considering a top spacer thickness
$d_{s}^{\alpha}=8.36\ \mathrm{\mu m}$ and a bottom spacer thickness of 6.03
µm, for a total spacer thickness $d_{s}^{\beta}=14.4\ \mathrm{\mu m}$ (Fig.
4). This is consistent with the model predictions $d_{s}^{\alpha}\in[8.29\
\mathrm{\mu m},8.40\ \mathrm{\mu m}]$ and $d_{s}^{\beta}\in[14.4\ \mathrm{\mu
m},14.5\ \mathrm{\mu m}]$ (eqs. 10-13).

For the realistic configuration, we consider Ag [57] as the back reflector,
and $\mathrm{VO_{2}}$ as the PCM [60]. We optimize the thickness of
$\mathrm{VO_{2}}$ as well as that of both spacers to maximize the figure of
merit, leading to $\varphi_{shift}=0.840$ with a top spacer thickness of 8.30
µm, a bottom spacer thickness of 5.62 µm and a $\mathrm{VO_{2}}$ thickness of
170 nm. Compared to the ideal case, the $\mathrm{VO_{2}}$ layer also
contributes to the optical path in the dielectric phase, explaining the lower
thickness of the bottom spacer. The $\mathrm{VO_{2}}$ thickness thus obtained
roughly corresponds to its skin depth, ensuring a sufficient reflectivity in
the metallic phase while minimizing parasitic absorption in the dielectric
phase. Such $\mathrm{VO_{2}}$ thicknesses have been deposited on various
materials including Si [61, 44].

Compared to the ideal case, $\mathrm{VO_{2}}$ leads to significant parasitic
emission, as can be seen in Fig. 4. In its metallic phase, $\mathrm{VO_{2}}$
is significantly less reflective than noble metals, thus broadening the
emissivity. In its dielectric phase, the imaginary part of the refractive
index is not negligible, and increases significantly above 10 µm, leading to
parasitic emission. Using materials with a more drastic dielectric-to-metallic
transition, such as $\mathrm{In_{3}SbTe_{2}}$ [62], may enable a behavior
closer to the idealized case.

We comment here on the angular dependence of thermal emission for these
architectures. The effective optical thickness of the spacer depends on the
angle $\theta$ with respect to normal incidence, following [63]:

$\Delta_{s}(\theta)=d_{s}\sqrt{n_{s}^{2}-\sin^{2}(\theta)}.$ (15)

Therefore, if the refractive index of the spacer is large ($n_{s}\gtrsim 2$),
the optical thickness will depend weakly on the angle. As a result, the
emission should be diffuse (assuming the angular dependence of front
reflectivity plays a minor role). Indeed, we confirm in the Supplemental
Material [51] that the behavior of the realistic ON-OFF switching structure
(Fig. 3) is essentially angle-independent up to 40 degrees, owing to the large
refractive index of GSST. As a result, this structure offers a diffuse
narrowband IR source that can be turned ON and OFF through control of the PCM
phase.

Finally, we discuss the resilience of the system to thickness variations. The
emitters considered in this work have a very high quality factor, enabling
narrowband thermal emission and leading to very thin optimal thicknesses.
Since hBN is a 2D-layered material with a c-axis lattice constant of 0.67 nm
[64], a 4-monolayer emitter would have a thickness of 2.7 nm. Compared to the
optimal thickness of 2.3 nm, this makes no difference in device performance.
For SiC, the peak absorptivity starts decreasing for emitters thicker than 5
nm, which might be challenging to achieve homogeneously via chemical or
physical vapor deposition. Considering emitters with lower quality factors
such as $\mathrm{SiO_{2}}$ can yield to more practical thicknesses around 100
nanometers, at the expense of emission bandwidth. Regarding the spacer
thickness, the tolerance is in the order of 50 nm for each layer to maintain
optimal performance. Although small, this is within the precision achievable
using carefully calibrated thermal evaporation or sputtering.

We have proposed a simple framework to design lithography-free narrowband IR
thermal sources which can change their behavior upon the phase transition of a
PCM layer. We suggested different implementations depending on the nature of
the phase transition. We applied it first to ON-OFF emission switching with a
single emitter, then to spectral shifting with two emitters, only one emitting
in each phase. With ideal, non-absorptive materials, it is easy to achieve
near-ideal performance. Devices for ON-OFF switching with very high contrast
at all angles should currently be manufacturable, although the relatively
thick GSST layers required make it challenging to ensure complete and
reversible phase transition [54]. More complex functions will require further
developments in materials to operate at peak performance, mostly to eliminate
the parasitic absorption in the PCM. Additional degrees of control can be
considered in order to improve the response and enable new functionalities,
for instance by considering dispersive spacers or by adjusting the thickness
of the emitters and their position within the stack [53]. The switch
configuration is particularly promising for thermophotovoltaic systems,
enabling on-demand energy generation through control of the PCM phase. In
addition, the demonstrated diffuse thermal emission behavior is attractive for
optimizing thermophotovoltaic energy conversion [65, 66]. More generally,
these architectures and the associated formalism should be valuable not only
for thermal emission applications including infrared sources and thermal
camouflage but more broadly for mid-IR photonic devices such as multi-spectral
photodetectors.

## acknowledgments

The authors declare no competing financial interest. G. T. P. acknowledges
funding from ”la Caixa” Foundation (ID 100010434), from the
PID2021-125441OA-I00 project funded by MCIN /AEI /10.13039/501100011033 /
FEDER, UE, from the TED2021-129841A-I00 funded by MCIN/AEI/
10.13039/501100011033 and by the European Union “NextGenerationEU”/PRTR, and
from the European Union’s Horizon 2020 research and innovation programme under
the Marie Sklodowska-Curie grant agreement No 847648. The fellowship code is
LCF/BQ/PI21/11830019. M.G. acknowledges financial support from the Severo
Ochoa Excellence Fellowship. M.E. acknowledges ayuda PRE2020-094401 financiada
por MCIN/AEI/ 10.13039/501100011033 y FSE "El FSE invierte en tu futuro". This
work is part of the R&D project CEX2019-000910-S, funded by MCIN/
AEI/10.13039/501100011033/, from Fundació Cellex, Fundació Mir-Puig, and from
Generalitat de Catalunya through the CERCA program.

## References

  * Schuller  _et al._ [2009] J. A. Schuller, T. Taubner, and M. L. Brongersma, Optical antenna thermal emitters, Nature Photonics 3, 658 (2009).
  * Chen  _et al._ [2020] C. Chen, X. Lu, B. Deng, X. Chen, Q. Guo, C. Li, C. Ma, S. Yuan, E. Sung, K. Watanabe, T. Taniguchi, L. Yang, and F. Xia, Widely tunable mid-infrared light emission in thin-film black phosphorus, Science Advances 6, eaay6134 (2020).
  * Li  _et al._ [2018] Y. Li, X. Bai, T. Yang, H. Luo, and C.-W. Qiu, Structured thermal surface for radiative camouflage, Nature Communications 9, 273 (2018).
  * Rephaeli  _et al._ [2013] E. Rephaeli, A. Raman, and S. Fan, Ultrabroadband Photonic Structures To Achieve High-Performance Daytime Radiative Cooling, Nano Letters 13, 1457 (2013).
  * Raman  _et al._ [2014] A. P. Raman, M. A. Anoma, L. Zhu, E. Rephaeli, and S. Fan, Passive radiative cooling below ambient air temperature under direct sunlight, Nature 515, 540 (2014).
  * Liu  _et al._ [2011] X. Liu, T. Tyler, T. Starr, A. F. Starr, N. M. Jokerst, and W. J. Padilla, Taming the Blackbody with Infrared Metamaterials as Selective Thermal Emitters, Physical Review Letters 107, 045901 (2011).
  * Baranov  _et al._ [2019] D. G. Baranov, Y. Xiao, I. A. Nechepurenko, A. Krasnok, A. Alù, and M. A. Kats, Nanophotonic engineering of far-field thermal emitters, Nature Materials 18, 920 (2019).
  * Greffet  _et al._ [2002] J.-J. Greffet, R. Carminati, K. Joulain, J.-P. Mulet, S. Mainguy, and Y. Chen, Coherent emission of light by thermal sources, Nature 416, 61 (2002).
  * Celanovic  _et al._ [2005] I. Celanovic, D. Perreault, and J. Kassakian, Resonant-cavity enhanced thermal emission, Physical Review B 72, 075127 (2005).
  * Inoue  _et al._ [2015] T. Inoue, M. D. Zoysa, T. Asano, and S. Noda, Realization of narrowband thermal emission with optical nanostructures, Optica 2, 27 (2015).
  * Sakurai  _et al._ [2019] A. Sakurai, K. Yada, T. Simomura, S. Ju, M. Kashiwagi, H. Okada, T. Nagao, K. Tsuda, and J. Shiomi, Ultranarrow-Band Wavelength-Selective Thermal Emission with Aperiodic Multilayered Metamaterials Designed by Bayesian Optimization, ACS Central Science 5, 319 (2019).
  * Wang  _et al._ [2020] Z. Wang, J. K. Clark, Y.-L. Ho, S. Volz, H. Daiguji, and J.-J. Delaunay, Ultranarrow and Wavelength-Tunable Thermal Emission in a Hybrid Metal–Optical Tamm State Structure, ACS Photonics 7, 1569 (2020).
  * Duan  _et al._ [2022] J. Duan, F. J. Alfaro-Mozaz, J. Taboada-Gutiérrez, I. Dolado, G. Álvarez Pérez, E. Titova, A. Bylinkin, A. I. F. Tresguerres-Mata, J. Martín-Sánchez, S. Liu, J. H. Edgar, D. A. Bandurin, P. Jarillo-Herrero, R. Hillenbrand, A. Y. Nikitin, and P. Alonso-González, Active and Passive Tuning of Ultranarrow Resonances in Polaritonic Nanoantennas, Advanced Materials 34, 2104954 (2022).
  * Ma  _et al._ [2022] B. Ma, Y. Huang, W. Zha, B. Qin, R. Qin, P. Ghosh, S. Kaur, M. Qiu, and Q. Li, Narrowband diffuse thermal emitter based on surface phonon polaritons, Nanophotonics 11, 4115 (2022).
  * Caldwell  _et al._ [2015] J. D. Caldwell, L. Lindsay, V. Giannini, I. Vurgaftman, T. L. Reinecke, S. A. Maier, and O. J. Glembocki, Low-loss, infrared and terahertz nanophotonics using surface phonon polaritons, Nanophotonics 4, 44 (2015).
  * Salisbury [1952] W. W. Salisbury, Absorbent body for electromagnetic waves (U.S. patent US2599944A, 1952).
  * Fante and McCormack [1988] R. Fante and M. McCormack, Reflection properties of the Salisbury screen, IEEE Transactions on Antennas and Propagation 36, 1443 (1988).
  * Zhao  _et al._ [2021] B. Zhao, J.-H. Song, M. Brongersma, and S. Fan, Atomic-Scale Control of Coherent Thermal Radiation, ACS Photonics 8, 872 (2021).
  * Picardi  _et al._ [2022] M. F. Picardi, K. N. Nimje, and G. T. Papadakis, Dynamic modulation of thermal emission – a Tutorial, 10.48550/arXiv.2210.01587 (2022).
  * Fan  _et al._ [2022] K. Fan, R. D. Averitt, and W. J. Padilla, Active and tunable nanophotonic metamaterials, Nanophotonics 11, 3769 (2022).
  * De Zoysa  _et al._ [2012] M. De Zoysa, T. Asano, K. Mochizuki, A. Oskooi, T. Inoue, and S. Noda, Conversion of broadband to narrowband thermal emission through energy recycling, Nature Photonics 6, 535 (2012).
  * Jang  _et al._ [2014] M. S. Jang, V. W. Brar, M. C. Sherrott, J. J. Lopez, L. Kim, S. Kim, M. Choi, and H. A. Atwater, Tunable large resonant absorption in a midinfrared graphene Salisbury screen, Physical Review B 90, 165409 (2014).
  * Inoue  _et al._ [2014] T. Inoue, M. D. Zoysa, T. Asano, and S. Noda, Realization of dynamic thermal emission control, Nature Materials 13, 928 (2014).
  * Wang  _et al._ [2015] H. Wang, Y. Yang, and L. Wang, Infrared frequency-tunable coherent thermal sources, Journal of Optics 17, 045104 (2015).
  * Chen  _et al._ [2008] H.-T. Chen, J. F. O’Hara, A. K. Azad, A. J. Taylor, R. D. Averitt, D. B. Shrekenhamer, and W. J. Padilla, Experimental demonstration of frequency-agile terahertz metamaterials, Nature Photonics 2, 295 (2008).
  * Papadakis  _et al._ [2021] G. T. Papadakis, C. J. Ciccarino, L. Fan, M. Orenstein, P. Narang, and S. Fan, Deep-Subwavelength Thermal Switch via Resonant Coupling in Monolayer Hexagonal Boron Nitride, Physical Review Applied 15, 054002 (2021).
  * Tittl  _et al._ [2015] A. Tittl, A.-K. U. Michel, M. Schäferling, X. Yin, B. Gholipour, L. Cui, M. Wuttig, T. Taubner, F. Neubrech, and H. Giessen, A Switchable Mid-Infrared Plasmonic Perfect Absorber with Multispectral Thermal Imaging Capability, Advanced Materials 27, 4597 (2015).
  * Du  _et al._ [2017] K.-K. Du, Q. Li, Y.-B. Lyu, J.-C. Ding, Y. Lu, Z.-Y. Cheng, and M. Qiu, Control over emissivity of zero-static-power thermal emitters based on phase-changing material GST, Light: Science & Applications 6, e16194 (2017).
  * Cao  _et al._ [2018] T. Cao, X. Zhang, W. Dong, L. Lu, X. Zhou, X. Zhuang, J. Deng, X. Cheng, G. Li, and R. E. Simpson, Tuneable Thermal Emission Using Chalcogenide Metasurface, Advanced Optical Materials 6, 1800169 (2018).
  * Kalantari Osgouei  _et al._ [2021] A. Kalantari Osgouei, H. Hajian, B. Khalichi, A. E. Serebryannikov, A. Ghobadi, and E. Ozbay, Active Tuning from Narrowband to Broadband Absorbers Using a Sub-wavelength VO2 Embedded Layer, Plasmonics 16, 1013 (2021).
  * Zheng  _et al._ [2022] Z. Zheng, Y. Luo, H. Yang, Z. Yi, J. Zhang, Q. Song, W. Yang, C. Liu, X. Wu, and P. Wu, Thermal tuning of terahertz metamaterial absorber properties based on VO2, Physical Chemistry Chemical Physics 24, 8846 (2022).
  * Ko  _et al._ [2021] B. Ko, T. Badloe, and J. Rho, Vanadium Dioxide for Dynamically Tunable Photonics, ChemNanoMat 7, 713 (2021).
  * Ryckman  _et al._ [2013] J. D. Ryckman, K. A. Hallman, R. E. Marvel, R. F. Haglund, and S. M. Weiss, Ultra-compact silicon photonic devices reconfigured by an optically induced semiconductor-to-metal transition, Optics Express 21, 10753 (2013).
  * Markov  _et al._ [2015] P. Markov, R. E. Marvel, H. J. Conley, K. J. Miller, R. F. J. Haglund, and S. M. Weiss, Optically Monitored Electrical Switching in VO2, ACS Photonics 2, 1175 (2015).
  * Abdollahramezani  _et al._ [2020] S. Abdollahramezani, O. Hemmatyar, H. Taghinejad, A. Krasnok, Y. Kiarashinejad, M. Zandehshahvar, A. Alù, and A. Adibi, Tunable nanophotonics enabled by chalcogenide phase-change materials, Nanophotonics 9, 1189 (2020).
  * Michel  _et al._ [2014] A.-K. U. Michel, P. Zalden, D. N. Chigrin, M. Wuttig, A. M. Lindenberg, and T. Taubner, Reversible Optical Switching of Infrared Antenna Resonances with Ultrathin Phase-Change Layers Using Femtosecond Laser Pulses, ACS Photonics 1, 833 (2014).
  * Wuttig  _et al._ [2017] M. Wuttig, H. Bhaskaran, and T. Taubner, Phase-change materials for non-volatile photonic applications, Nature Photonics 11, 465 (2017).
  * Cao and Cen [2019] T. Cao and M. Cen, Fundamentals and Applications of Chalcogenide Phase-Change Material Photonics, Advanced Theory and Simulations 2, 1900094 (2019).
  * Kats  _et al._ [2013] M. A. Kats, R. Blanchard, S. Zhang, P. Genevet, C. Ko, S. Ramanathan, and F. Capasso, Vanadium Dioxide as a Natural Disordered Metamaterial: Perfect Thermal Emission and Large Broadband Negative Differential Thermal Emittance, Physical Review X 3, 041004 (2013).
  * Li  _et al._ [2016] P. Li, X. Yang, T. W. W. Maß, J. Hanss, M. Lewin, A.-K. U. Michel, M. Wuttig, and T. Taubner, Reversible optical switching of highly confined phonon–polaritons with an ultrathin phase-change material, Nature Materials 15, 870 (2016).
  * Stegmaier  _et al._ [2017] M. Stegmaier, C. Ríos, H. Bhaskaran, C. D. Wright, and W. H. P. Pernice, Nonvolatile All-Optical 1 × 2 Switch for Chipscale Photonic Networks, Advanced Optical Materials 5, 1600346 (2017).
  * Zhang  _et al._ [2018] Q. Zhang, Y. Zhang, J. Li, R. Soref, T. Gu, and J. Hu, Broadband nonvolatile photonic switching based on optical phase change materials: beyond the classical figure-of-merit, Optics Letters 43, 94 (2018).
  * Chen  _et al._ [2022] X. Chen, Y. Xue, Y. Sun, J. Shen, S. Song, M. Zhu, Z. Song, Z. Cheng, and P. Zhou, Neuromorphic Photonic Memory Devices Using Ultrafast, Non-volatile Phase-change Materials, Advanced Materials , 2203909 (2022).
  * Wan  _et al._ [2022] C. Wan, D. Woolf, C. M. Hessel, J. Salman, Y. Xiao, C. Yao, A. Wright, J. M. Hensley, and M. A. Kats, Switchable Induced-Transmission Filters Enabled by Vanadium Dioxide, Nano Letters 22, 6 (2022).
  * Yin  _et al._ [2017] X. Yin, T. Steinle, L. Huang, T. Taubner, M. Wuttig, T. Zentgraf, and H. Giessen, Beam switching and bifocal zoom lensing using active plasmonic metasurfaces, Light: Science & Applications 6, e17016 (2017).
  * Buhara  _et al._ [2021] E. Buhara, A. Ghobadi, B. Khalichi, H. Kocer, and E. Ozbay, Mid-infrared adaptive thermal camouflage using a phase-change material coupled dielectric nanoantenna, Journal of Physics D: Applied Physics 54, 265105 (2021).
  * King  _et al._ [2022] J. L. King, A. Shahsafi, Z. Zhang, C. Wan, Y. Xiao, C. Huang, Y. Sun, P. J. Roney, S. Ramanathan, and M. A. Kats, Wavelength-by-Wavelength Temperature-Independent Thermal Radiation Utilizing an Insulator–Metal Transition, ACS Photonics 9, 2742 (2022).
  * Cao  _et al._ [2013] T. Cao, L. Zhang, R. E. Simpson, and M. J. Cryan, Mid-infrared tunable polarization-independent perfect absorber using a phase-change metamaterial, JOSA B 30, 1580 (2013).
  * Chen  _et al._ [2015] Y. Chen, X. Li, X. Luo, S. A. Maier, and M. Hong, Tunable near-infrared plasmonic perfect absorber based on phase-change materials, Photonics Research 3, 54 (2015).
  * Xiong  _et al._ [2022] L. Xiong, H. Ding, Y. Lu, and G. Li, Extremely Narrow and Actively Tunable Mie Surface Lattice Resonances in GeSbTe Metasurfaces: Study, Nanomaterials 12, 701 (2022).
  * [51] See Supplemental Material at [URL] for a general derivation of the conditions for zero and unitary emissivity, the refractive index of the materials considered in this study, a discussion on resonance broadening due to parasitic absorption and the impact of the angle of incidence.
  * Sarkar  _et al._ [2022] M. Sarkar, M. Giteau, M. Enders, and G. T. Papadakis, Lithography-free directional control of thermal emission, 10.48550/arXiv.2210.01026 (2022).
  * Giteau  _et al._ [2022] M. Giteau, Y. Oteki, K. Kitahara, N. Miyashita, R. Tamaki, and Y. Okada, Resonant absorption for multilayer quantum well and quantum dot solar cells, Journal of Photonics for Energy 12, 022203 (2022).
  * Zhang  _et al._ [2019] Y. Zhang, J. B. Chou, J. Li, H. Li, Q. Du, A. Yadav, S. Zhou, M. Y. Shalaginov, Z. Fang, H. Zhong, C. Roberts, P. Robinson, B. Bohlin, C. Ríos, H. Lin, M. Kang, T. Gu, J. Warner, V. Liberman, K. Richardson, and J. Hu, Broadband transparent optical phase change materials for high-performance nonvolatile photonics, Nature Communications 10, 4279 (2019).
  * Spitzer  _et al._ [1959] W. G. Spitzer, D. Kleinman, and D. Walsh, Infrared Properties of Hexagonal Silicon Carbide, Physical Review 113, 127 (1959).
  * Le Gall  _et al._ [1997] J. Le Gall, M. Olivier, and J.-J. Greffet, Experimental and theoretical study of reflection and coherent thermal emissionby a SiC grating supporting a surface-phonon polariton, Physical Review B 55, 10105 (1997).
  * Yang  _et al._ [2015] H. U. Yang, J. D’Archangel, M. L. Sundheimer, E. Tucker, G. D. Boreman, and M. B. Raschke, Optical dielectric function of silver, Physical Review B 91, 235137 (2015).
  * Cai  _et al._ [2007] Y. Cai, L. Zhang, Q. Zeng, L. Cheng, and Y. Xu, Infrared reflectance spectrum of BN calculated from first principles, Solid State Communications 141, 262 (2007).
  * Kumar  _et al._ [2015] A. Kumar, T. Low, K. H. Fung, P. Avouris, and N. X. Fang, Tunable Light–Matter Interaction and the Role of Hyperbolicity in Graphene–hBN System, Nano Letters 15, 3172 (2015).
  * Wan  _et al._ [2019] C. Wan, Z. Zhang, D. Woolf, C. M. Hessel, J. Rensberg, J. M. Hensley, Y. Xiao, A. Shahsafi, J. Salman, S. Richter, Y. Sun, M. M. Qazilbash, R. Schmidt-Grund, C. Ronning, S. Ramanathan, and M. A. Kats, Optical properties of thin-film vanadium dioxide from the visible to the far infrared, Annalen der Physik 531, 1900188 (2019).
  * Morsy  _et al._ [2020] A. M. Morsy, M. T. Barako, V. Jankovic, V. D. Wheeler, M. W. Knight, G. T. Papadakis, L. A. Sweatlock, P. W. C. Hon, and M. L. Povinelli, Experimental demonstration of dynamic thermal regulation using vanadium dioxide thin films, Scientific Reports 10, 13964 (2020).
  * Heßler  _et al._ [2021] A. Heßler, S. Wahl, T. Leuteritz, A. Antonopoulos, C. Stergianou, C.-F. Schön, L. Naumann, N. Eicker, M. Lewin, T. W. W. Maß, M. Wuttig, S. Linden, and T. Taubner, In3SbTe2 as a programmable nanophotonics material platform for the infrared, Nature Communications 12, 924 (2021).
  * Yariv [1991] A. Yariv,  _Optical Electronics_ (Saunders College Publishing, Philadelphia, PA, 1991).
  * Lynch and Drickamer [1966] R. W. Lynch and H. G. Drickamer, Effect of High Pressure on the Lattice Parameters of Diamond, Graphite, and Hexagonal Boron Nitride, The Journal of Chemical Physics 44, 181 (1966).
  * Pfiester and Vandervelde [2017] N. A. Pfiester and T. E. Vandervelde, Selective emitters for thermophotovoltaic applications, physica status solidi (a) 214, 1600410 (2017).
  * Wang  _et al._ [2022] Z. Wang, D. Kortge, Z. He, J. Song, J. Zhu, C. Lee, H. Wang, and P. Bermel, Selective emitter materials and designs for high-temperature thermophotovoltaic applications, Solar Energy Materials and Solar Cells 238, 111554 (2022).


# Bootstrapped model learning and error correction for planning with
uncertainty in model-based RL ††thanks: A diagram of the architecture, list of
hyperparameters and source code are located in
https://github.com/aovalle/bootstrapped-transition-functions

Alvaro Ovalle  School of Electronic Engineering and Computer Science
Queen Mary University of London
London, UK
<EMAIL_ADDRESS>Simon M. Lucas  School of Electronic Engineering and
Computer Science
Queen Mary University of London
London, UK
<EMAIL_ADDRESS>
###### Abstract

Having access to a forward model enables the use of planning algorithms such
as Monte Carlo Tree Search and Rolling Horizon Evolution. Where a model is
unavailable, a natural aim is to learn a model that reflects accurately the
dynamics of the environment. In many situations it might not be possible and
minimal glitches in the model may lead to poor performance and failure. This
paper explores the problem of model misspecification through uncertainty-aware
reinforcement learning agents. We propose a bootstrapped multi-headed neural
network that learns the distribution of future states and rewards. We
experiment with a number of schemes to extract the most likely predictions.
Moreover, we also introduce a global error correction filter that applies
high-level constraints guided by the context provided through the predictive
distribution. We illustrate our approach on Minipacman. The evaluation
demonstrates that when dealing with imperfect models, our methods exhibit
increased performance and stability, both in terms of model accuracy and in
its use within a planning algorithm.

###### Index Terms:

model based reinforcement learning, deep learning, rolling horizon evolution,
forward models, planning, uncertainty, error-correction, bootstrapping.

##  I Introduction

In recent years model-based reinforcement learning (RL) has experienced a
revival boosted by advances in deep learning. Compared to a reactive strategy,
possessing an internal model (i.e. forward model) and being able to encode
knowledge about the dynamics of the environment offers several advantages.
Having a model appears to be a key component enabling the capacity to forecast
events, anticipate consequences, evaluate possibilities or visualise different
perspectives.

Beyond these positive aspects, model-based RL could also potentially lessen
some of the technical issues that afflict model-free approaches, such as
sample inefficiency and lack of generalisation. Ideally if an agent acquires a
model, it gains the ability to simulate experiences to plan or to learn a
policy without the necessity of interacting directly with the environment.
This implies a considerable advantage in robotic control tasks or those
requiring safe exploration. Furthermore, by encoding environmental dynamics
the knowledge can be reused and leveraged for different tasks.

However acquiring a forward model that is accurate enough to support planning
comes with its own set of challenges. The agent does not only have to learn a
policy as in model-free RL, but also a model on which to learn it, introducing
an additional source of error. Crucially since the model approximates the
dynamics, even small errors can be catastrophic, making the model essentially
useless as they accumulate over the rollout. Nonetheless, very promising
research has focused on how to make models more reliable through various
techniques. For example by learning and executing them completely in latent
space [1, 2], by adding noise to increase their robustness [3] or through a
combination of multiple techniques [4]. Alternative approaches sideline
learning state transitions altogether, and focus instead on modelling other
aspects such as value functions and future policy [5]. Despite these advances
it is almost unavoidable, that for any complex task, a model will be
inherently incomplete. There will be an epistemic component related to a gap
in knowledge that could decrease by further learning or new data. However
there will also be an aleatoric aspect associated to the stochasticity of the
process that generates the data itself [6].

In this paper we are concerned with the importance it has for an agent to have
access to its uncertainty. Different from [1], we are not specifically
interested in finding better state representation mechanisms intended to learn
a more accurate model. Instead it is assumed the agent must deal with the
faulty or premature models it currently has, and yet, it must plan and act
under uncertainty. The method we propose employs bootstrapping techniques to
approximate the predictive distribution of state transitions by learning an
ensemble of models through a single multi-headed architecture. We devise and
test three different schemes to integrate the data from the distribution. In
this sense our objective is closer to that pursued in [7], where the goal is
to assist the decision making process by aggregating information from a group
of inaccurate models. In addition we also try to demonstrate how having access
to a predictive distribution enables the construction of error-correcting
routines, increasing the reliability of the predictions. Concretely, we
consider an environment with multiple interacting elements, whose presence or
absence follows a consistent set of rules serving as constraints. If the
initial prediction of an agent does not satisfy those constraints, alternative
predictions contained in the distribution can inform how to adjust the
original prediction. We test our approach on a video game control task where
it exhibits increased performance compared to a non-probabilistic agent.

##  II Background and Previous Work

###  II-A Model-Based Reinforcement Learning

In RL we denote state $s_{t}\in\mathcal{S}$, action $a_{t}\in\mathcal{A}$ and
a reward function $r(s_{t},a_{t})$. The environment is governed by a
transition function or model $f(s_{t},a_{t})=s_{t+1}$ providing the next
state. Thus in the model-based case a mapping to $f$, which we refer to as
$f_{\theta}$, is either provided or learned from observations. For stochastic
environments $f(s_{t+1}|s_{t},a_{t})=Pr(s_{t+1}|s_{t},a_{t})$. It is possible
to express a simulated rollout of action-state sequences
$\tau=(s_{1},a_{1},\dots,s_{T},a_{T})$ with respect to a policy $\pi$ as a
distribution:

$p_{\theta,\phi}(\tau)=p_{\theta}(s_{1})\prod_{t}\pi_{\phi}(a_{t}|s_{t})p_{\theta}(s_{t+1}|s_{t},a_{t})$
(1)

The goal of a model-based RL agent is to find a policy $\pi$ that maximises
the rewards along this simulated trajectory subjected to
$\max_{\theta,\phi}\mathbb{E}_{\tau\sim
p_{\theta,\phi}}\big{[}\sum_{t}r(s_{t},a_{t})\big{]}$.

###  II-B The Bootstrap Method

The bootstrap method is an statistical technique to assess the accuracy of an
statistic or estimator of an unknown population parameter [8]. In some cases
the variability of the estimator of interest is established by repeatedly
sampling from the population itself. However, sometimes this may not be
feasible and the only measurements at our disposal come from a single observed
pool of sample data $D$. The underlying principle behind the bootstrap method
relies on the assumption that the sample data is representative of the true
population. Thus we can simulate the process of randomly sampling from the
population by treating the sample data as if it were the population. If the
number of samples is large enough, it can be possible to obtain a distribution
that approximates the sampling distribution of the estimator. The process
consists of the following steps:

  * •

From a sample data $D$ of size $N$ resample with replacement to obtain a
bootstrap sample $\tilde{D}$ of the same size. This process is repeated to get
$K$ bootstrap samples.

  * •

Compute an statistic or estimator $\chi^{*}_{k}$ for each of the bootstrap
samples.

  * •

Build a bootstrap distribution with the $K$ estimators.

From the bootstrap distribution then it is possible to perform inference. For
example, by calculating confidence intervals or obtaining the standard error
to assess the variability across estimates.

SampleDataEstimator$\chi_{0}$$\chi^{*}_{2}$$\chi^{*}_{1}$$\chi^{*}_{k}$⋮BootstrapSamplesBootstrapEstimatorsBootstrapDistribution

Figure 1: The bootstrap method. The process consists in resampling with
replacement from an original group of observations. For each new sample pool
an estimator is computed and then used to build the bootstap distribution.

###  II-C Bootstrapping in Reinforcement Learning and Randomised Prior
Functions

The idea behind the bootstrap method has found application in RL to estimate
uncertainty and to facilitate more principled and efficient exploration
strategies. [9] proposed the bootstrapped DQN to approximate the posterior of
a Q-function. Compared to the original DQN [10] they introduced a multi-headed
architecture where each head is trained on a different group of experiences
from a replay buffer as determined by a masking distribution. Therefore each
head models its own approximation of the state-action values. Initially during
training due to the discrepancy in the approximations, the agent chooses
different actions depending on the head doing the decision making, which
drives exploration in a natural manner. As training progresses the heads tend
to converge to an estimate. After training, the agent can assess its
confidence about the state-action value of an observation by sending it
through each head and obtaining $K$ different predictions (i.e. the bootstrap
distribution)

A shortcoming of the bootstrap method is that by itself it does not include a
mechanism to reflect uncertainty beyond what is coming from the observed data.
The authors pointed out that this issue is mitigated by relying on the
diversity that is induced by the random initialisation of the network which
acts as a form of prior. However this is further addressed in [11] with the
inclusion of randomised prior functions (RPF). The approach consists in
explicitly specifying a prior via an additional fixed and randomly initialised
network. This can be described as:

$Q_{\theta_{k}}(x)=f_{\theta_{k}}(x)+\beta p_{k}$ (2)

Where $Q_{\theta_{k}}$ corresponds to head $k$ and it is modelled by a
function approximator $f_{\theta_{k}}$ with tunable parameters $\theta_{k}$
and a non-trainable prior $p_{k}$ scaled by $\beta$.

The bootstrapped DQN has also been extended to model-based RL [12], helping in
the identification of state-action values with a larger level of uncertainty
to guide search in look-ahead trees [13].

##  III Method

###  III-A Bootstrapped Transition Functions

In this work we also take the idea behind bootstrapped DQN, but instead of
applying it to Q-functions we turn our interest towards environment
transitions. Thus we take a probabilistic interpretation to the problem of
estimating the next set of future observations and rewards.

Formally we can represent a neural network as a conditional model
$p_{\theta}(y|x)$ parameterised by $\theta$, where $x$ is the input and $y$ is
the output. In the RL context, consider an agent that learns
$p_{\theta}(s_{t+1},r_{t}|s_{t},a_{t})$ as it observes data
$D=\\{s_{i},a_{i},r_{i},s_{i+1}\\}_{i}^{N}$. That is, it learns the set of
assumptions of the model, encoded in the parameters $\theta$, that explains
$D$ as good as possible. The posterior distribution $p(\theta|D)$ is usually
intractable and often the parameters $\theta$ are learned as point estimates
by calculating the maximum likelihood or the maximum a posteriori when
introducing some form of regularisation.

If we require to work with a distribution, a popular approach consists in
finding the network parameters $\phi$ of a candidate distribution $q$ that is
as similar as possible to the true posterior distribution (i.e.
$q_{\phi}(\theta|D)\approx p(\theta|D)$) using variational inference to
minimise $D_{KL}[q_{\phi}(\theta|D)||p(\theta|D)]$ [14, 15, 16]. In our method
instead of learning $q_{\phi}$, we assume that the parameters on each head
$k$, plus those in the part of the network that is shared amongst the heads,
are a sample $\theta_{k}\sim p(\theta|D)$. Together all these samples form an
ensemble distribution $q_{e}$ that approximates the true posterior $p$. Then
we can use $q_{e}$ to obtain an estimate of the predictive distribution of
next state $s_{t+1}$ and reward $r_{t}$ given a new state-action input:

$p(s_{t+1},r_{t}|s_{t},a_{t})\approx\mathbb{E}_{\theta\sim
q_{e}(\theta|D)}[p_{\theta}(s_{t+1},r_{t}|s_{t},a_{t})]$ (3)

The predictive distribution corresponds to the bootstrap distribution reviewed
in sections II-B and II-C. Every head outputs a prediction of the next state
and reward and can be interpreted as estimators
$\chi^{*}_{s^{\prime}_{i},r_{i}}$. From this distribution we can quantify the
uncertainty and compute other statistical metrics of interest.

Any behavioural policy can be followed to learn the multi-headed environment
models. To increase the stability and generalisation, the models are not
learned online but from a minibatch sampled from a replay buffer and trained
to minimise the cross-entropy loss between predictions and actual observations
or rewards. In addition to vanilla bootstrapping of transition functions, we
were also interested in comparing the effects of adding RPFs. As per equation
1 when a prior was included the predicted next state an reward on head $k$
corresponded respectively to:

$\displaystyle\begin{split}T_{\theta_{k}}(s_{t},a_{t})=f^{t}_{\theta_{k}}(s_{t},a_{t})+\beta
p_{k}\\\ r_{\theta_{k}}(s_{t},a_{t})=f^{r}_{\theta_{k}}(s_{t},a_{t})+\beta
p_{k}\end{split}$ (4)

Input: Policy $\pi$, number of heads $K$

Initialise: Replay buffer $D$, parameters $\theta$

1 for  _$t\dots T$_ do

2 Sample $a_{t}\sim\pi(a_{t}|s_{t})$;

3 Observe $s_{t+1},r_{t}\sim p(s_{t+1},r_{t}|s_{t},a_{t})$;

4 if  _done_ then

5 $env.reset()$;

6

7 end if

8 Generate masks $m\sim Ber(K,0.5)$;

9 $D\leftarrow D\cup\\{s,a,r,s^{\prime},m\\}$;

10 Sample $s_{m},a_{m},r_{m},s^{\prime}_{m},m_{m}\sim D$;

11 $\hat{y}=\\{\hat{s}_{t+1},\hat{r}_{t}\\}=f_{\theta}(s_{m},a_{m})$;

12 Update model by minimising $\mathcal{L}_{\theta}(\hat{y},y)$,
backpropagating according to $m$ via SGD;

13

14 end for

15

Algorithm 1 Bootstrapped Transition Functions

###  III-B Rollout strategy

Regardless of the level of accuracy contained in a learned model of the
environment. An agent harnesses it to simulate spatio-temporal prospects and
contingencies. We use Rolling Horizon Evolution (RHE)[17] for integrating the
knowledge encoded in the model with the capacity to manipulate it in order to
imagine potential outcomes.

RHE is a real-time control and planning algorithm that uses a forward model to
search the space of trajectories in order to maximise certain utility. We can
define a policy $\pi$ as an action sequence $a_{0:\mathcal{T}}$ of length
$\mathcal{T}$, starting from a current state $s_{0}$ to $s_{\mathcal{T}}$ (or
an earlier terminal state). Thus RHE attempts to find a policy that satisfies,
within certain budget, the following:

$\pi(s)=\operatorname*{argmax}_{a_{0:\mathcal{T}}}\mathbb{E}\Bigg{[}r(s_{0},a_{0})+\sum^{\mathcal{T}}_{t=1}r(\hat{s}_{t},a_{t})\Bigg{]}$
(5)

Where $r(s,a)$ is the reward function. It has to be noted that only $s_{0}$ is
an actual observed state while $\hat{s}_{t}$ are states simulated via a
forward model.

A major advantage of RHE is its simplicity while retaining competitive
performance against other well established planning algorithms such as MCTS
[17, 18, 19]. In its most basic formulation, RHE works by mutating an initial
random action sequence to get a population of size $\mathcal{P}$. The fitness
of the action sequences is determined by evaluating them using the forward
model. The action sequence with the highest total reward in a rollout is
selected, and the first action of the sequence is carried out in the real
environment. The process is described in algorithm 2.

Several enhancements intended to improve the performance of RHE have been
proposed [20, 21, 22]. For the moment we focus on the vanilla version of RHE
that we have just briefly summarised. The only enhancement we have added to
this standard version is the shift buffer technique which aims to facilitate
information transfer from the past [20]. This consists in seeding the new
population with the previous fittest action sequence, by shifting it one time
step to the left and appending a new random action at the end of the sequence
to preserve its length.

RHE has traditionally been applied to perfect simulators but the basic
mechanics of the algorithm remain unchanged when using imperfect forward
models. However one must establish how to obtain the imagined states $\hat{s}$
on which RHE will operate. Taking into account that the network has $k$ heads
and that each of them generates its own predictions, it is necessary to
determine how to tackle the discrepancy arising from the uncertainty of the
forward model to consolidate the outputs into a single unified prediction. We
considered three different criteria for next state and reward prediction.

  * •

Average: computes the mean of the output values in the last layer of each head
and retrieves the class with the largest value:

$\hat{s}_{t+1},\hat{r}_{t}=\operatorname*{argmax}_{c}\Bigg{[}\dfrac{1}{K}\sum_{k}\sigma\Big{(}\mathbf{u}^{(L)}\Big{)}_{k}\Bigg{]}_{c}$
(6)

Where $\mathbf{u}^{(L)}$ is the vector of logits in the last layer $L$ of head
$k$ and $c\in\\{0,\dots,C\\}$.

  * •

Majority Voting: takes the most common prediction by obtaining the mode over
the output of each head:

$\hat{s}_{t+1},\hat{r}_{t}=mode\big{(}f_{\theta_{1}},\dots,f_{\theta_{K}}\big{)}$
(7)

  * •

Sampling: selects a prediction from the predictive distribution formed by the
output of each head:

$\hat{s}_{t+1},\hat{r}_{t}\sim
p\big{(}s_{t+1},r_{t}|s_{t},a_{t};[\theta_{k}]_{k=1}^{K}\big{)}$ (8)

Input: Forward model $p_{\theta}$, number of actions $A$, sequence length
$\mathcal{T}$, population size $\mathcal{P}$, mutation rate $\mu$

1 while  _true_ do

2 if  _not shift buffer_ then

3 $\pi\leftarrow a_{t},\dots,a_{\mathcal{T}}\sim Cat(A)$;

4

5 else

6 Shift $\pi$ to the left and add $a_{\mathcal{T}}\sim Cat(A)$ at the end of
the sequence;

7

8 end if

9 for  _$i\dots\mathcal{P}$_ do

10 if  _$i >0$_ then

11 $\pi_{i}\leftarrow$ Mutate $\pi$ with rate $\mu$

12 end if

13 for  _$h\dots\mathcal{T}$_ do

14 Predict $\hat{s}_{h+1}$ and $\hat{r}_{h}$ using eq. 6, 7 or 8 via
$p_{\theta}$;

15 if  _error-correction_ then

16 $\hat{s}_{h+1}\leftarrow correct(\hat{s}_{h+1})$;

17

18 end if

19 Save or update current return
$R(\pi_{i}^{(0:h)})=r(s_{t},a_{0})+\sum^{h}_{k=1}r(\hat{s}_{k},a_{k})$;

20

21 end for

22

23 end for

24 Select fittest action sequence $\pi\leftarrow max_{\pi}R(\pi)$;

25 Perform first action in the sequence $a\leftarrow\pi^{(0)}$;

26 Observe $s_{t+1},r_{t}\sim p(s_{t+1},r_{t}|s_{t},a)$;

27

28 end while

Algorithm 2 Rolling Horizon Evolution with inaccurate models and error-
correction

###  III-C Error-Correction

It is widely acknowledged that learning a forward model is challenging in
complex and noisy tasks. Compound error may render a model impractical for
planning. However it is less understood the different ways the forward models
can be wrong in their predictions. There is currently no comprehensive
classification on the type of errors we may encounter.

A particular type of error is the lack of alignment between predictions done
locally and how they scale up at a higher level. Even if local predictions are
sensible, it does not necessarily imply they will be consistent once all of
them are integrated as a whole. For instance in visual domains where a frame
can be modelled by several distributions operating at lower levels such as
pixels or objects. Consider the example of a video game frame where a neural
network outputs predictions for each section of the frame. The network learns
about the existence of various objects in the environment, for example as
clouds, trees or animals. If the network predicts a cloud at a given frame
section, this prediction by itself and in isolation, might not seem
surprising. However if we look at the frame and observe a cloud displayed
below a tree, then the predicted frame has a low likelihood of actually
occurring in the environment. Simulating further into the future by building
upon inaccurate predictions may lead to quick degeneration. Thus in some cases
it could be useful to think in terms of the higher level constraints that need
to be satisfied by a prediction.

In this work we focus on two kinds of constraints intended to regulate the
amount of objects of certain class that should be present in the predicted
frame. Furthermore here we describe how the ensemble can leverage the
uncertainty in the estimates via the bootstrap distribution in order to make
predictions that are more robust, reliable and that have a higher chance of
conforming to the expectations of what is realistically possible in an
environment.

We start from the assumption that some of these constraints are discrete and
known to the agent at the beginning of a simulated rollout. Any correction
made to the frames is based on what is immediately accessible to the agent:
(1) its current observation and/or (2) its predictions. For the latter, having
a bootstrap distribution becomes crucial for error-correcting. By having not
only a single point estimate but several candidate predictions, we can
establish mechanisms that search, compare and integrate observations gathered
from the heads into new revised predictions that are more likely to satisfy
the criteria demanded by the constraints. Note that we will refer to an
element as any generic section in the frame such as a pixel, a cell or an
object. The two error-correction mechanisms are the following:

Input: Unified frame prediction $f_{u}$, Multi-headed predictions
$f_{\theta_{k}}$, element $e$, constraint $c$, last observed or imagined
position $lp$

1 if  _$e$ in $f_{u}<c$_ then

2 Find positions in each $f_{\theta_{k}}$ where $e$ is and store in p;

3 if  _p is not empty_ then

4 if  _sampling_ then

5 $p^{\prime}\sim\textbf{p}$;

6

7 else if  _average or voting_ then

8 $p^{\prime}\leftarrow mode(p_{0},\dots,p_{N})$;

9

10 end if

11

12 else

13 $p^{\prime}\leftarrow lp$;

14

15 end if

16 Insert $e$ in new position $p^{\prime}$ in unified frame $f_{u}$;

17 $lp\leftarrow p^{\prime}$;

18

19 end if

Algorithm 3 Error-correction for missing elements

  * •

Missing element: this type of verification applies to situations where the
unified frame prediction does not include the presence of a particular element
in any of the cells even though it would be expected. To correct the frame,
every head is inspected for the element. If a head has one or multiple
predictions containing the element, the positions where the element is located
are stored in a shared vector along with the locations coming from other
heads. After all heads have been examined, the frame is corrected by inserting
the missing element in a position selected from the shared location vector.
The selection is done either by taking the mode of the location vector or by
sampling from it, depending on the criteria used by the network to predict
next state and reward. If the location vector is empty because none of the
heads predict the existence of the element then the position is taken from the
previous frame. Only for the first step in a simulated rollout, the previous
frame really corresponds to the last observation gathered from the environment
whereas for the rest of the steps it is taken from the previously imagined
frame. The pseudo-code of this mechanism is outlined in algorithm 3.

  * •

Additional element: the other situation considered here is when the prediction
includes more elements of a certain type than those anticipated. First we find
the locations where the element is present in the unified predicted frame and
store them in a vector a. Then we locate the positions of the element in every
head of the ensemble and store them in a shared vector p. Similar to the
unified frame prediction, a head can also be predicting the existence of the
element more times than those specified by the constraints. The two vectors
are then compared to identify and separate the locations in the shared vector
p that occur in a as well. From this resulting vector we select the position
of the elements that will be preserved in the unified prediction. The
selection is done by either taking the mode or by sampling from the vector.

After it has been decided what elements are preserved, then it has to be
determined what should substitute those that have not been chosen. This
substitution can also be assisted by the predictions contained in the heads.
First by gathering the location of those elements in the unified frame and
then by extracting the predictions done by each head in those locations, but
considering only the predictions that are different from the element that
should be excluded. Once they have been gathered the selection is again made
computing the mode or by sampling one of the elements. Similar to the missing
element routine when a decision cannot be made due to an empty vector, the
rectification will default to the element occurring in the last frame, whether
real or imagined.

Input: Unified frame prediction $f_{u}$, Multi-headed predictions
$f_{\theta_{k}}$, element $e$, constraint $c$, last imagined or observed
element $le$

Initialise: Vector v that will hold the elements that replace the copies

1 if  _$e$ in $f_{u}>c$_ then

2 Find positions in $f_{u}$ where $e$ is and store in a;

3 Find positions in each $f_{\theta_{k}}$ where $e$ is and store in p;

4 Check which values in a are in p and store in g;

5 if  _g is not empty_ then

6 if  _sampling_ then

7 $g^{\prime}\sim\textbf{g}$;

8

9 else if  _average or voting_ then

10 $g^{\prime}\leftarrow mode(g_{0},\dots,g_{N})$;

11

12 end if

13 Remove $g^{\prime}$ from g;

14 for  _$g_{i}\in\textbf{g}$_ do

15 For each $f_{\theta_{k}}$ extract the element located at $g_{i}$ and store
in v;

16 if  _v is not empty_ then

17 if  _sampling_ then

18 $v^{\prime}\sim\textbf{v}$;

19

20 else if  _average or voting_ then

21 $v^{\prime}\leftarrow mode(v_{0},\dots,v_{N})$;

22

23 end if

24

25 else

26 $v^{\prime}\leftarrow le$

27 end if

28 Insert $v^{\prime}$ in position $g_{i}$ in unified frame $f_{u}$;

29 $le\leftarrow v^{\prime}$;

30

31 end for

32

33 end if

34

35 end if

Algorithm 4 Error-correction for additional elements

##  IV Experimental Setup

Figure 2: Minipacman. The green cell corresponds to the agent, red to the
ghost, turquoise to the power pills, blue to the food still left in the maze
and black to a cell where the food has been eaten.

###  IV-A Environment

We demonstrate the effects of the bootstrapped transition functions with and
without error correction mechanisms in Minipacman 111Our version has been
modified and adapted to the task. It was originally taken from
https://github.com/higgsfield/Imagination-Augmented-Agents [7] (fig. 2). The
environment is a reduced version of Pacman and provides a simple minimal
discrete control benchmark. The game consists in an agent that navigates
through a maze eating food in the corridors. The agent is chased by a ghost
moving semi-randomly. This element of stochasticity arising from the behaviour
of the ghost is an additional challenge for learning a forward model, as it
deals not only with the epistemic factors of the environment but also the
aleatoric. In addition, the game contains power pills that when eaten by the
agent allows it for a fixed number of steps the possibility to eat the ghost.
The agent has a repertoire of five actions: up, down, left, right and no
operation. It also receives a reward depending on specific game events. When
the agent eats food it gets a reward of 1, if it does not move or goes to a
section of the corridor with no food it does not receive a reward. It gets a
reward of 3 for the power pills and 6 if it manages to eat the ghost whilst
under the effects of the power pills. If the ghost kills the agent the reward
is -1 and the game instance terminates.

Besides the reward, the agent also receives an observation that reflects the
current state of the game. The format of the observation is an array of
$15\times 19\times 3$, where the last number is the amount of channels. Thus
the observation contains a total of $15\times 19$ cells. Each of them is
associated with a different object either an agent, food, power pills, a
ghost, eaten and inaccessible cells. The fact that we can treat a cell as an
object allows us to evaluate our error-correcting strategy as proof of
concept, since unsupervised object detection for control tasks is still an
nascent area of research [23, 24, 25].

###  IV-B Error-Correction in Minipacman

The algorithms described in section III-C offer a general strategy. A starting
point for error-correcting a simulated rollout. To test their performance we
focused on the most fundamental elements of the game: the agent and the ghost.
Predicted frames are checked against the constraints established for the game.
The constraints are simply the number of agents or ghosts counted from the
last real observation (or the last imagined observation). If there are less or
more than the number of elements expected, the routines in algorithm 3 and
algorithm 4 are invoked respectively. There is however a small element-
specific adjustment done for algorithm 3. If the predicted frame initially
contains more than a single pacman, then from the pool of copies to be
discarded, we verify if one of them corresponds to the position where it was
observed in the last frame. If the conditon is met then the cell is turned
black, symbolizing that the food there has been eaten. Otherwise they follow
the original outline and take their values from the last observed or imagined
frame.

###  IV-C Architecture

For learning the forward model of the environment we implemented a
convolutional neural network (CNN) with an additional fully connected last
layer. The input consists of a concatenation of the last frame and a
broadcasted one-hot representation of the previous action. The CNN attempts to
capture long range temporal dependencies through pool-and-inject layers (we
refer the reader to [7] for more details). The output of the network is the
categorical representation of every cell in the next predicted frame and the
predicted reward. The frame and the reward are transformed into their original
format and passed on to the agent to continue the cycle.

Accordingly, for those forward models parameterised by bootstrapped multi-
headed networks the last layer produces $K$ frames and rewards. As outlined in
sections II-B and III-A, RPFs require a prior in the form of an additional
neural network with fixed parameters. The prior network maintains the same
structure, inputs and outputs of the bootstrapped network.

##  V Results

Figure 3: Comparison of uncertainty-aware BTF agents against a non-
probabilistic agent with a single-headed model. The plots show the performance
in Minipacman with different planning horizons.

###  V-A Model learning

Accuracy _Single Head_ _Average_ _Majority Voting_ _Sampling_ _Boot_ _RPF_
_Boot_ _RPF_ _Boot_ _RPF_ Fruit 0.99583 0.99627 0.99617 0.99627 0.99638
0.99617 0.99617 Eaten Cell 0.99598 0.99545 0.99261 0.99564 0.9926 0.99506
0.99174 Pacman 0.90284 0.99899 0.99899 0.99919 0.99939 0.99878 0.99777 Pacman
(EC) - 0.99939 0.99979 0.99939 0.99979 0.99939 0.99959 Ghost 0.22980 0.36043
0.47513 0.35595 0.46303 0.34722 0.40053 Ghost (EC) - 0.49955 0.50134 0.49955
0.50179 0.49865 0.50179 Reward 0.9557 0.97356 0.97376 0.97519 0.97437 0.97112
0.97397 Frame 0.9962 0.99661 0.9966 0.99663 0.99661 0.9966 0.99658 Constraint
Fulfilment Fruit 0.86594 0.96433 0.98296 0.95127 0.98566 0.9548 0.97173 Pacman
0.90499 0.95564 0.99768 0.99654 0.99584 0.97429 0.99558 Pacman (EC) - 1.0 1.0
1.0 1.0 1.0 1.0 Ghost 0.31941 0.63119 0.85846 0.63407 0.86749 0.51388 0.7348
Ghost (EC) - 0.92741 0.9197 0.93 0.95096 0.94887 0.92379

Table I: Model learning. Top: Average next step accuracy by category and type
of forward model. Bottom: The proportion of frames that satisfied exactly the
constraints as observed in the environment.

The first empirical evaluation considers the capacity to predict future state
and reward. We compared a non-probabilistic single-headed agent (SH) against
six variations of a 10-headed CNN architecture as described in the previous
section. Three of the agents operated with a forward model trained through
bootstrapped transition functions (BTF) while the other three also included a
random prior (BTF+RPF). For both groups we tested the three mechanisms to
consolidate predictions defined by eq. 6, 7 and 8. All CNNs were trained for
50,000 steps using a random policy and then tested on their next step
predictions over 100 episodes. The accuracy was measured by verifying the
number of cells (or reward class) that matched with the ground truth. If we
look only at the average accuracy in the entire frame, all models appear to be
deceivingly close. However, decomposing the accuracy by element category it is
easier to appreciate the differences that arise when predicting the two most
dynamic elements of the game, namely the ghost and pacman (upper section table
I). Uncertainty-aware agents are consistently able to predict the motor
consequences of their actions to a larger extent than when using a SH model.
However all agents are less adept at predicting the semi-stochastic behaviour
of the ghost. Activating the error-correcting routines has a positive effect
for both categories. They allow the agents to predict correctly the ghost
movement up to 50% of the time.

Error-correcting mechanisms however provided something considerably more
critical. From the observations we had gathered on model prediction it was not
rare to find frames where pacman was missing or cloned. This was also a common
occurrence with the ghost. The bottom section of table I refers to the
proportion of frames that complied exactly with the element constraints
imposed by the environment. That is, the predicted frames that did not miss or
included extra elements compared to the real observation. There we can observe
that error-correcting mechanisms rightly included pacman and increased
significantly the likelihood of removing or including the exact number of
ghosts that appear in the next time step.

###  V-B Planning and game performance

We measured the performance of the agent in the game to analyse the impact
that the probabilistic models have on planning with simulated temporal
sequences. The models were tested in combination with RHE for sequences of 1,
5, 10 and 20 steps into the future, a population size of 10, a mutation rate
of 0.9 and evaluated for 10 episodes. We observed that in general all
bootstrapped models, given a choice of sequence length, offered better
performance than the SH model. At their peak, bootstrapped models without
error-correction achieved a 2x improvement and above 3x when enhanced with
error-correction. All models degraded in performance as horizon depth
increases, however error-correcting models tended to benefit more from longer
sequences and managed to retain proficiency even with 20-step sequences. For
reference, RHE using a perfect simulator achieves a mean score of 134.54.

We also noticed that BTF+RPF exhibits more stable performance than BTF alone.
Supporting these observations with the results in model learning, we could
draw parallels with [11] and speculate that the prior is indeed acting as a
regularisation mechanism. This could imply that during training the ensemble
is more diverse in its predictions and therefore less susceptible to premature
overconfident estimations.

##  VI Discussion and Future Work

The original objective behind this work was to investigate potential benefits
in performance when incorporating uncertainty estimation into the forward
model of an agent. Although in the literature there has been a large emphasis
on producing more accurate models, our results support the conclusions reached
in previous work regarding the deficiency of pursuing blindly this approach
[26]. Producing highly accurate immediate predictions may still lead to
catastrophic failure if global accuracy measures and loss functions fail to
capture how the essential components of the environment are modelled. Thus
there is a necessity to study further which additional criteria should assist
in evaluating the viability of a forward model. We focused on structural
consistency in the predictions as one of these criteria, by relating them to
the specific environmental constraints they should satisfy. We explored this
notion by introducing post-processing error-correcting mechanisms and showed
that it is possible to take advantage of the predictions from an ensemble to
guide the corrections.

The results demonstrated that training a forward model capable of producing
uncertainty estimates leads to improved performance and that enhancing them
with error-correction not only improves it further but also increases
stability in longer horizons. On a conceptual level this may suggest that
authenticity or frame coherence could be complementary to (granular) accuracy
for correctly planning in uncertain environments. Because we were interested
in the implications of uncertainty estimation with inaccurate forward models
some of the experimental choices were intended to study the robustness of our
approach. For example, using a high mutation rate to produce a widely diverse
set of sequences to test the model or planning with an unbiased RHE. Thus it
is possible that the agents could perform better on the task with adequate
hyperparameter tuning and we could establish comparisons against similar
methods such as I2A [7].

Although BTF by itself can be generically applicable to multiple domains, BTF
with error-correction is provided as a starting point and it is still at this
moment limited in scope. There are several directions for future work. For
instance, we have assumed prior knowledge of the constraints. One could
imagine learning these constraints directly from data by predicting the
presence of elements and passing on this prediction to the error-correcting
routines. Another future challenge is to scale the error-correcting capacity
beyond grid-based games as well as the possibility of developing other
mechanisms able to operate in latent space. Lastly, a future study could
consider the inclusion of a density estimator or a novelty detector for an
additional assessment of the structural consistency and realism of a
prediction.

## Acknowledgement

This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL
Research-IT. http://doi.org/10.5281/zenodo.438045

## References

  * [1] L. Buesing, T. Weber, S. Racaniere, S. M. A. Eslami, D. Rezende, D. P. Reichert, F. Viola, F. Besse, K. Gregor, D. Hassabis, and D. Wierstra, “Learning and Querying Fast Generative Models for Reinforcement Learning,” Feb. 2018.
  * [2] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning Latent Dynamics for Planning from Pixels,” in International Conference on Machine Learning, pp. 2555–2565, May 2019.
  * [3] D. Ha and J. Schmidhuber, “Recurrent World Models Facilitate Policy Evolution,” in Advances in Neural Information Processing Systems 31 (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), pp. 2450–2462, Curran Associates, Inc., 2018.
  * [4] K. Asadi, D. Misra, S. Kim, and M. L. Littman, “Combating the Compounding-Error Problem with a Multi-step Model,” arXiv:1905.13320 [cs, stat], May 2019.
  * [5] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver, “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model,” arXiv:1911.08265 [cs, stat], Feb. 2020.
  * [6] A. Kendall and Y. Gal, “What uncertainties do we need in Bayesian deep learning for computer vision?,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, (Long Beach, California, USA), pp. 5580–5590, Curran Associates Inc., Dec. 2017.
  * [7] S. Racanière, T. Weber, D. Reichert, L. Buesing, A. Guez, D. Jimenez Rezende, A. Puigdomènech Badia, O. Vinyals, N. Heess, Y. Li, R. Pascanu, P. Battaglia, D. Hassabis, D. Silver, and D. Wierstra, “Imagination-Augmented Agents for Deep Reinforcement Learning,” in Advances in Neural Information Processing Systems 30 (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), pp. 5690–5701, Curran Associates, Inc., 2017.
  * [8] B. Efron, “Bootstrap Methods: Another Look at the Jackknife,” The Annals of Statistics, vol. 7, pp. 1–26, Jan. 1979.
  * [9] I. Osband, C. Blundell, A. Pritzel, and B. V. Roy, “Deep exploration via bootstrapped DQN,” in Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, (Barcelona, Spain), pp. 4033–4041, Curran Associates Inc., Dec. 2016.
  * [10] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” arXiv:1312.5602 [cs], Dec. 2013.
  * [11] I. Osband, J. Aslanides, and A. Cassirer, “Randomized Prior Functions for Deep Reinforcement Learning,” in Advances in Neural Information Processing Systems 31 (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), pp. 8617–8629, Curran Associates, Inc., 2018.
  * [12] W. Huang, J. Zhang, and K. Huang, “Bootstrap Estimated Uncertainty of the Environment Model for Model-Based Reinforcement Learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 3870–3877, July 2019.
  * [13] G. Farquhar, T. Rocktäschel, M. Igl, and S. Whiteson, “TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning,” in International Conference on Learning Representations, Feb. 2018.
  * [14] G. E. Hinton and D. van Camp, “Keeping the neural networks simple by minimizing the description length of the weights,” in Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT ’93, (Santa Cruz, California, USA), pp. 5–13, Association for Computing Machinery, Aug. 1993.
  * [15] A. Graves, “Practical Variational Inference for Neural Networks,” in Advances in Neural Information Processing Systems 24 (J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, eds.), pp. 2348–2356, Curran Associates, Inc., 2011.
  * [16] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight Uncertainty in Neural Network,” in International Conference on Machine Learning, pp. 1613–1622, June 2015.
  * [17] D. Perez, S. Samothrakis, S. Lucas, and P. Rohlfshagen, “Rolling horizon evolution versus tree search for navigation in single-player real-time games,” in Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO ’13, (Amsterdam, The Netherlands), pp. 351–358, Association for Computing Machinery, July 2013.
  * [18] J. Liu, D. Pérez-Liébana, and S. M. Lucas, “Rolling Horizon Coevolutionary planning for two-player video games,” in 2016 8th Computer Science and Electronic Engineering (CEEC), pp. 174–179, Sept. 2016.
  * [19] B. Santos, H. Bernardino, and E. Hauck, “An Improved Rolling Horizon Evolution Algorithm with Shift Buffer for General Game Playing,” in 2018 17th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pp. 31–316, Oct. 2018.
  * [20] R. D. Gaina, S. M. Lucas, and D. Perez-Liebana, “Rolling horizon evolution enhancements in general video game playing,” in 2017 IEEE Conference on Computational Intelligence and Games (CIG), pp. 88–95, Aug. 2017.
  * [21] R. D. Gaina, S. M. Lucas, and D. Pérez-Liébana, “Population seeding techniques for Rolling Horizon Evolution in General Video Game Playing,” in 2017 IEEE Congress on Evolutionary Computation (CEC), pp. 1956–1963, June 2017.
  * [22] X. Tong, W. Liu, and B. Li, “Enhancing Rolling Horizon Evolution with Policy and Value Networks,” in 2019 IEEE Conference on Games (CoG), pp. 1–8, Aug. 2019.
  * [23] T. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih, “Unsupervised Learning of Object Keypoints for Perception and Control,” arXiv:1906.11883 [cs], Nov. 2019.
  * [24] N. Watters, L. Matthey, M. Bosnjak, C. P. Burgess, and A. Lerchner, “COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration,” arXiv:1905.09275 [cs], Aug. 2019.
  * [25] A. Anand, E. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. Hjelm, “Unsupervised State Representation Learning in Atari,” arXiv:1906.08226 [cs, stat], Jan. 2020.
  * [26] E. Talvitie, “Model regularization for stable sample rollouts,” in Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI’14, (Quebec City, Quebec, Canada), pp. 780–789, AUAI Press, July 2014.