#  On the importance of illustration for mathematical research

Rémi Coulon  Univ Rennes, CNRS, IRMAR - UMR 6625, 35000 Rennes, France
Gabriel Dorfsman-Hopkins  Department of Mathematics, Statistics, and Computer
Science, St. Lawrence University, Canton, NY  Edmund Harriss  Department of
Mathematics and School of Art, University of Arkansas  Martin Skrodzki
Computer Graphics and Visualization, TU Delft, Netherlands  Katherine E.
Stange  Department of Mathematics, University of Colorado Boulder  Glen
Whitney  Prison Math Project

## Abstract

Mathematical understanding is built in many ways. Among these, illustration
has been a companion and tool for research for as long as research has taken
place. We use the term _illustration_ to encompass any way one might bring a
mathematical idea into physical form or experience, including hand-made
diagrams or models, computer visualization, 3D printing, and virtual reality,
among many others. The very process of illustration itself challenges our
mathematical understanding and forces us to answer questions we may not have
posed otherwise. It can even make mathematics an experimental science, in
which immersive exploration of data and representations drive the cycle of
problem, conjecture, and proof. Today, modern technology for the first time
places the production of highly complicated models within the reach of many
individual mathematicians. Here, we sketch the rich history of illustration,
highlight important recent examples of its contribution to research, and
examine how it can be viewed as a discipline in its own right.

## Introduction

In the last decade, it has become increasingly possible for researchers to
augment their experience of abstract mathematics with sensory exploration:
3D-printed or CNC-milled models, the ability to walk through seemingly
impossible physical spaces with virtual reality, and the potential to explore
high-dimensional mathematical spaces through computer visualisation all
provide such opportunities, among other media. Now much more than simply an
aid to understanding, these tools have reached a level of sophistication that
makes them indispensable to many frontiers of mathematical research. To
preview one particular case recounted below, the tantalizing structure visible
in Figure 1 (and many others like it) led to conjectures and proofs that would
likely otherwise have been inaccessible.

Figure 1: Detail of a $\mathbb{Z}^{2}$-lattice sandpile stabilized from a
starting configuration with 500 million grains at the origin. Whenever any
position has at least 4 grains, it distributes one to each neighbour. Nodes in
the stable configuration have 0, 1, 2 or 3 grains (indicated by colour). Image
by Stange.

The list of examples of research driven by illustration is rapidly expanding
in recent years. We use the term _illustration_ to encompass any way one might
bring a mathematical idea into physical form or experience, including hand-
made diagrams or models, computer visualization, 3D printing, or virtual
reality, among many others. We will discuss instances of this interplay
including breakthroughs in such areas as representation theory (such as
Knutson and Tao’s honeycombs [honeycombAMS, honeycomb1, honeycomb2]), self-
organized criticality (work of Levine, Pegden, Smart and others on Abelian
sandpiles [LPS17] as illustrated in Figure 1), geometry (Coiculescu and
Schwartz on Thurston’s _Sol_ geometry [coiculescu2022sol]), and most recently
the solution of the einstein problem with the hat monotile
[smith2023aperiodic] and its chiral version, the spectre [smith2023chiral].
These solutions came after many years of physical exploration by Dave Smith,
which revealed especially interesting properties of the hat tile. Note that
these papers have already sparked several follow-up works
[reitebuch2023direct, baake2023dynamics, socolar2023quasicrystalline]).

Illustration is beginning to find a home at programs like the special semester
in _Illustrating Mathematics_ in Fall 2019 at the Institute for Computational
and Experimental Research in Mathematics (ICERM) and the Institute for
Advanced Study (IAS)/Park City Math Institute (PCMI) virtual program in Summer
2021,111See http://illustratingmath.org/ for links to these two programs,
along with many other resources. and a community is forming around many modern
tools. Of course, the importance of illustration to research is not new:
abstraction was linked to plane diagrams in the work of the ancients,
including Euclid’s _Elements_ or the Chinese treatise _The Nine Chapters on
the Mathematical Art_ (九章算術). Precise three-dimensional models were produced
by skilled artisans in the 19th century, notable examples of which remain in
the collections at the Institute Henri Poincaré222https://patrimoine.ihp.fr/
[Villani:OM] or Göttingen University333https://sammlungen.uni-
goettingen.de/sammlung/slg_1017/, among many other institutions. When computer
visualization first became widely available in the 1980’s, the Geometry Center
was founded at the University of Minnesota, with a mission to exploit these
new tools on behalf of mathematics. But we are now at another cusp: modern
technological tools have recently made 3D models and virtual reality widely
available, and computation and computer visualization is more accessible and
more powerful than ever. We can now collect huge mathematical datasets and
examples, and it has become urgent to develop intuitive ways to interact with
this data.

Making full use of modern tools is not without its challenges: beyond the
obvious technical challenges and software learning curves, there are important
questions about how an illustration, much like a statistic or an experiment,
can subtly mislead the researcher, or miss the essential mathematical pattern
sought. Researchers often individually reinvent the necessary skill sets as
they seek to advance their own projects, and these projects are pushing the
boundaries of the possible. But by building a discipline around this
enterprise, we can develop its full potential to advance mathematical
research.

## Some highlights from the history of mathematical illustration

Figure 2: An abstract ochre drawing on silica from the 73,000 year old layers
of the Blombos Cave, studied in [Henshilwood:ADF7LBCSA]. Image by Craig Foster
courtesy of Professor Christopher Henshilwood.

Illustration of mathematics goes back as far as mathematical ideas themselves.
In fact, some of the earliest evidence we have for abstract thinking comes
from human-made designs, for example the cross-hatched carvings in Blombos
Caves in South Africa, potentially from 73,000 years ago
[Henshilwood:ADF7LBCSA]. A little more recently, the middle-eastern tradition
of geometry presented in Euclid’s _Elements_ [Euclid:EE] provides a structural
link between statements deduced from axioms, and figures made with straight
edge and compass. These two tools provide a physical realization of the two
key objects (straight lines and circles) described by the axioms. Euclid’s
diagrams give a map to help follow (or discover!) the chain of deduction in a
proof. Conversely, the proof validates the image (which could otherwise
mislead by error or the selection of a non-generic example). This approach
leads at the conclusion of Book 1 to a proof of the Pythagorean theorem; see
Figure 3.

Figure 3: The proof of Book 1 Proposition 47, half of Pythagoras’ theorem, in
an Oliver Byrne’s edition from 1847. The figure shown here had appeared in
printed copies of the _Elements_ since the fifteenth century, but Byrne’s
rendition links it tightly and visually to the proof alongside. Image Credit:
Harriss and University of Arkansas Library.

In Chinese mathematics, this theorem is the 勾股 (Gougu) theorem. In the classic
_Nine Chapters on the Mathematical Art_ (九章算術) [Shen:NCMCC] , it plays a key
role in applying the arithmetical mathematics of the text to geometric
problems, for example in measuring altitude. The Chinese tradition also gives
an elegant visual proof of the result by rearranging triangles, as in Figure
4.

Figure 4: Two pages from the Arithmetical Classic on the Gnomon and the
Circular Paths of Heaven (周髀算經), a Chinese work on astronomy and mathematics
showing a proof of the right triangle 勾股 (Gougu) theorem. Image credit: Zhou
bi suan jing : shang xia juan, volume 1 / Zhao, Junqing; Zhen, Luan ; Li,
Chunfeng et al.; Hu, Zhenting; Bao, Shan = 周髀算經 : 上卷 /
趙君卿注,甄鸞重述,李淳風等注釋,胡震亭提辭,鮑山較, circa 1621-1627. Smith Chinese C-13, folio 4-5.
Smith-Plimpton East Asian Collection. Rare Book and Manuscript Library,
Columbia University Library.

Although the Chinese proof is not considered rigorous by modern standards,
Euclid was also criticized by Bertrand Russell when he wrote “A valid proof
retains its demonstrative force when no figure is drawn, but very many of
Euclid’s earlier proofs fail before this test.” [Russell:TE]. This criticism
reveals one of the challenges of mathematical illustration. A powerful example
comes from a well-known “proof,” apparently originally due to W. W. Rouse
Ball, that all triangles are equilateral.444For a modern presentation of this
fallacy and a lively discussion of its flaws, see Joel Hamkins’ essay at
https://jdh.hamkins.org/all-triangles-are-isosceles/. In this case a subtly
misleading diagram leads to a false conclusion following an otherwise entirely
correct argument. Briefly, starting with no constraints on the triangle $ABC$,
Ball produces points $O$, $E$, and $F$ by a construction depicted in Figure 5.
He establishes (without error) that $AF=AE$ and $FB=EC$. The diagram then
makes it “clear” that $AB=AF-FB$ must equal $AC=AE-EC$. Hence the triangle is
isosceles, and since the labels of $A$, $B$, and $C$ were arbitrary, it “must”
in fact be equilateral!

Disallowing these particular subtle errors requires axioms capturing the
meaning of “between,” that took considerable work by David Hilbert to
formulate [Hilbert:FG, Lee:AG]. Armed with these axioms, one can show that,
except when the triangle _is_ isosceles, Ball’s construction perfectly
executed will produce either $E$ within segment $AC$ or $F$ in segment $AB$
(but not both), destroying the final subtraction step.

Figure 5: This diagram from Ball’s 1882 _Mathematical Recreations and Essays_
subtly misleads the reader. In reality, either $E$ lies between $A$ and $C$ or
$F$ lies between $A$ and $B$. Image credit: University of Illinois
Library/Internet Archive.

A related pitfall – when a good illustration, overused, can become a pair of
blinders – is illustrated by the following example. In the _Elements_ , the
concept of number is based on the concept of length. So the squares in the
Pythagorean theorem are actual squares (the area of which are equal), not
squared numbers. In the 11th century algebra treatise of Omar Khayyam,
although he gives solutions to equations with higher powers than three, he
also states: “Square-square, which, to the algebraists, is the product of the
square by itself, has no meaning in continuous objects. This is because how
can one multiply a square, which is a surface, by itself? Since the square is
a two-dimensional object … and two-dimensional by two-dimensional is a four
dimensional object. But solids cannot have more than three dimensions.”
[Khayyam:AOK]. The relation between number and length was also an important
factor in the European reluctance to consider negative numbers. A line, after
all, cannot have negative length. In contrast, negative quantities are used
freely in the _Nine Chapters_ , where arithmetic is the foundational idea,
with geometry built from it. In Europe the development of the number line,
starting with John Wallis, gave an alternative illustration of number (see
Figure 6) with the capacity to include negative quantities as numbers in their
own right [Schubring:CBGRI].

Figure 6: An early depiction of the now-familiar number line, from Wallis’
1685 _A Treatise of Algebra_ ; image credit: Max Planck Institute for the
History of Science, Library.

Powers and negative numbers are but two examples of a productive pattern of
mathematics developing from the tension between illustration and symbolic
idea. The study of complex numbers advanced significantly with the concept of
the complex plane, and then allowed a new algebraic approach to the geometry
of the plane. Both quaternions and matrices were developed to try to extend
that understanding to higher dimensions [Knott:HVM, Grattan-Guiness:MT]. In
the case of real numbers, although the symbolic ideas would refine the
illustrations needed, it was not until the late nineteenth century when fully
symbolic definitions were developed, such as Dedekind cuts and Cauchy
sequences. At that point the need for illustrations as foundational objects
was removed, although the potential for developing intuition and challenging
what might be done with the concepts remained [Fowler:DT].

Projective geometry, first developed (as perspective) by artists as a tool to
create realistic images, provided one such challenge. These ideas were
explored mathematically by Johannes Kepler, Gérard Desargues and Blaise
Pascal. In the early nineteenth century, perspective was developed by Gaspard
Monge into “descriptive geometry” for the training of engineers in
constructing forts and later developed and axiomatized in the foundational
work by Jean-Victor Poncelet [Andersen:GA, Kemp:SOTWFBS]. In turn this work
would be key in establishing models for non-euclidean geometry, explored
axiomatically by Nikolai I. Lobachevsky and János Bolyai [Shenitzer:HNGECGS].
In this case it was such models, themselves illustrations, that convinced
mathematicians of the existence and interest of the non-euclidean geometries.

Projective geometry also spurred the study of algebraic geometry. In the late
nineteenth century an industry emerged to reveal the surfaces constructed in
this field and their properties, such as cone singularities and embedded
straight lines. One pioneer was Alexander Brill, a student of Alfred Clebsch
with a degree in architecture. Following the work of Peter Henrici (another
student of Clebsch), Brill made sliceform paper models of surfaces. He later
worked with Felix Klein in Munich to set up a laboratory for the design and
production of mathematical objects. This lab grew into a company that, when it
was taken over by Martin Schilling in 1911, had a catalogue of over 400
models. His work combined deep mathematical understanding with a knowledge of
printing and construction from his family business [Polo-Blanco:THGM,
OConnor:AWB, Friedman:MMF12C].

Figure 7: There can be two parallels through a point not on a line, in the
(crocheted) hyperbolic plane. Photo credit: Taimiņa.

The need to combine mathematical knowledge with fabrication techniques is also
highlighted by a story of missed opportunity: how to make physical patches of
hyperbolic planes. In addition to his disk model (often called the Poincaré
disk model), Eugenio Beltrami also attached together strips of paper to
approximate the surface. Other examples used paper polygons connected to make
a sort of hyperbolic “soccer ball.” These paper models are often fragile, and
the rigidity of the paper means that it cannot change its local geometry; thus
such models are crude approximations. Roughly a century later, Daina Taimiņa
realised that crocheting could produce far more resilient surfaces, with local
stretching that meant the negative curvature was more smoothly distributed
[Taimina:CAWHP]. An example of this medium of representation is shown in
Figure 7. In fact, similar techniques had been used to create ruffles in
scarves and skirts for decades. If the methods of fiber arts had earlier been
considered seriously and not dismissed as “work for women,” researchers could
have had the opportunity to handle robust hyperbolic planes far sooner.

## The incredible potential for mathematical illustration

Figure 8: Set of integer superharmonic matrices in the space of all symmetric
real matrices. Image by Stange.

Turning to recent developments, the work of Lionel Levine, Wesley Pegden, and
Charles K. Smart provides an excellent example of the value of illustration as
a research tool. Their _Annals of Mathematics_ paper _The Apollonian structure
of integer superharmonic matrices_ [LPS17] was motivated by the study of
Abelian sandpiles on $\mathbb{Z}^{2}$: place a large number $N$ of sand grains
at the origin, and allow any position with at least four grains to distribute
those grains equally to its four neighbours. The stable configuration that
results from this simple system displays impressive large-scale structure that
can be discovered through computer visualization (see Figure 1). Especially
striking is the vivid visual impression that the structure continues to refine
at larger $N$ toward a continuum scaling limit, which was proven earlier by
Pegden and Smart [PS13]. To describe the PDE governing this process, the
individual periodic tilings in the regions of the limit must be understood.
They are each governed by an _integer superharmonic matrix_. Levine, Pegden,
and Smart generated a picture of the set of integer superharmonic matrices,
and were astonished to see the familiar fractal structure of an Apollonian
circle packing (Figure 8). Each circle of the packing was associated to a
periodic pattern appearing in the scaling limit. Through extensive computer
investigation, the authors were able to determine the intricate recursive
relationships between the patterns for circles generated from one another
(‘ancestors’ overlap and merge to form ‘descendent’ patterns according to
complicated rules). These recursions led to a difficult inductive proof that
the set did indeed have the Apollonian structure evident in experiments. The
development of these results provide a perfect example of the role
illustration can play in the cycle of conjecture, theorem, and proof. Without
the data available through large-scale computer experimentation and the
ability to explore it visually, the question of the scaling limit may not have
been raised at all, and the recursive proof of their main result would likely
not have been discovered.

Figure 9: In-space view of a finite-volume hyperbolic $3$-manifold lit by a
single white light. Image by Coulon, Matsumoto, Segerman and Trettel
[Coulon:2020ac].

Another area where research is intertwined with illustration is in the study
of William Thurston’s geometrization conjecture [geoConjecture], proved by
Grigori Perelman [perelman1, perelman2, perelman3]. This key tool in our
understanding of $3$-manifolds implies, for instance, the famous Poincaré
conjecture. Geometrization states that any compact _topological_ $3$-manifold
can be cut into finitely many pieces, each of which carries a _geometric_
structure. There are eight possible such structures, known as _Thurston
geometries_. Some of them are rather familiar to mathematicians, such as the
3-dimensional Euclidean and hyperbolic spaces or the $3$-sphere.

Despite the fact that Thurston’s geometries have been intensively studied, the
more exotic geometries such as _Nil_ and _Sol_ still defy our “Euclidean-
grown” spatial intuition. Keeping in mind the well-established power of our
physical and visual intuition to aid geometrical research, Rémi Coulon,
Elisabetta Matsumoto, Henry Segerman, and Steve Trettel developed virtual
reality software to immerse the user in any of the eight Thurston geometries
[Coulon:2020ac] (see Figure 9). Besides building much-needed intuition for
these spaces, the development of the software itself raised mathematical
questions. To build their virtually rendered Thurston geometries, these
researchers use raymarching techniques which require computation of distances
between objects. But, for example, there is no closed formula for the distance
in _Nil_ or _Sol_! Thus, the development of the algorithms themselves becomes
a mathematical result in its own right.

Work on Thurston’s geometries has very often been closely tied with
illustration. For example, the study of _Spheres in Sol_ by Matei P.
Coiculescu and Rich Schwartz in _Geometry and Topology_ (positively) answers
an old open question, whether metric spheres in _Sol_ are homeomorphic to
$S^{2}$ [coiculescu2022sol]. Each step of the proof was found after numerous
graphical experiments, and 3D printing brings yet another perspective (see
Figure 10).

Figure 10: 3D printed models of the spheres in Sol produced during the ICERM
program “Illustrating Mathematics” in Fall 2019. Models by Coulon, Image
Credit: Harriss. Figure 11: The Bruhat-Tits tree of $\mathbb{Q}_{2}$, with
geodesics [HMMS17]*Figure 1.

For an example at the intersection of algebraic geometry and number theory, a
few key illustrations have helped drive developments in the field of $p$-adic
analytic geometry. At the same time, illustrating the $p$-adic analogs of
complex analytic manifolds presents unique challenges, not the least of which
is the fact that the $p$-adic numbers themselves are topologically a Cantor
set. Nevertheless, clever and meaningful illustrations of $p$-adic analogs to
the complex upper half-plane and complex unit disk have proved incredibly
fruitful. An illustration of Vladimir Drinfeld’s $p$-adic upper half plane as
tubular neighborhoods of Bruhat-Tits trees (Figure 11) clarified the behavior
of the action of $\operatorname{GL}_{2}(\mathbb{Q}_{p})$ by Möbius
transformations. Understanding this action was instrumental in the work of
David Mumford [mum72], Drinfeld [drin76], in the construction of $p$-adic
analytic uniformization of elliptic curves (reflecting the famous complex
analytic uniformization of elliptic curves as quotients of the complex upper
half plane). Similarly, Peter Scholze’s illustrations of the adic unit ball
(Figure 12) provide access to the foundational geometric construction in his
theory of perfectoid spaces [sch12]. The act of illustrating the central
geometric objects of $p$-adic analysis has proven both beneficial and uniquely
challenging, demanding a systematic and critical approach.

Figure 12: An early illustration of the adic unit ball by Scholze
[sch12]*Example 2.20. Figure 13: A honeycomb [honeycombAMS]*Figure 1.

An example arising somewhat further afield of geometry is the work of Allen
Knutson, Terence Tao, and Christopher Woodward in representation theory
[honeycombAMS, honeycomb2]. Knutson and Tao introduced the notion of
_honeycombs_ (subsets of the plane as in Figure 13) to solve a longstanding
open problem: Alfred Horn’s conjectured shape of the polyhedral cone
(sometimes called the Littlewood-Richardson cone) of triples of eigenvalue
spectra $(\lambda,\mu,\nu)$ for Hermitian matrices $A,B,C$ which satisfy
$A+B+C=0$ [horn62].

Figure 14: Knutson manufactured puzzles to study the sum-of-Hermitian-matrices
problem. Image credit: Knutson.

This _sum-of-Hermitian-matrices_ problem has applications to perturbation
theory, quantum measurement theory, and the spectral theory of self-adjoint
operators. Knutson and Tao were able to show that there exist such Hermitian
matrices with the specified spectra if and only if there exist honeycombs with
a specified boundary. They used this correspondence to prove Horn’s
conjecture. The honeycomb formalism also led naturally to a polynomial time
algorithm to decide whether a triple of spectra can be realized by Hermitian
matrices. In a follow-up, Knutson, Tao, and Woodward extended the study of
honeycombs to define _puzzles_ (Figure 14), which they described as replacing
the Schubert calculus in past approaches to the Hermitian matrices problem,
and used geometric arguments to give a complete characterization of the facets
of the cone [honeycomb2]. Puzzles and honeycombs provide an example of the
power of rephrasing an algebraic problem as one about visual objects, where we
can draw on other types of intuition. In what circumstances can we expect
these sort of insightful geometric versions to exist for algebraic problems?
When a geometric analog exists, it naturally exhibits additional features –
can we then find new corresponding objects in the original problem? For
example, what do the vertices of a honeycomb actually represent?

There are, of course, many more examples. Among these, the most famous may be
the computer exploration of the Mandelbrot set and fractal geometry in the
1980’s [mand82] (Figure 15). In the 1990’s, Jeffrey Weeks created SnapPea
(which now exists as SnapPy under the guidance of Marc Culler and Nathan
Dunfield555http://snappy.computop.org) as part of his doctoral thesis,
[Weeks93], to explore the cusp structures of hyperbolic 3-manifolds. Its use
inspired David Gabai, Robert Meyerhoff, and Peter Milley to invent _mom
structures_ to answer questions of the volumes of hyperbolic 3-manifolds
[momTech]. In the same decade, the _Geometry Center_ founded by Al Marden was
focused on the use of computer visualization in
mathematics.666http://www.geom.uiuc.edu/ It hosted mathematicians such as
Eugenio Calabi, John Horton Conway, Donald E. Knuth, Mumford, and Thurston,
among others, and produced the GeomView software [GeomView] used to create
some famous early computer visualizations, including the sphere
eversion777Outside In, (1994), http://www.geom.uiuc.edu/docs/outreach/oi/ and
illustrations for knot theory.888Not Knot, (1991),
http://www.geom.uiuc.edu/video/NotKnot/ Illustration has shown its importance
in virtually all areas of mathematics, from random tilings in combinatorics,
[aztec], to diagrammatic approaches to algebra, [TLDiagrams], to Apollonian
circle packings and Schmidt arrangements in number theory, [Mar19, Sta18,
Fuchs13], and their higher dimensional analogs, [spherePackings], to mention
just a few.

Figure 15: The first graph (in ASCII) of the Mandelbrot set. Used with
permission of Princeton University Press, from Brooks and Matelski, “The
Dynamics of 2-Generator Subgroups of PSL(2, $\mathbb{C}$),” 1980; permission
conveyed through Copyright Clearance Center, Inc. [BM81]*Figure 2.

The examples above focus on pure mathematics, which is poised to join a great
many other areas of scientific endeavour embracing illustration. In applied
mathematics, illustration has already made great strides. Consider for
instance the process of Alan H. Schoen, when describing the _gyroid_ decades
before it was mathematically proven to be a minimal surface. He worked with
both a sculpture of the surface and various models in _Computer-Aided Design /
Modelling_ (CAD/CAM), which ultimately led to the structure being found in
various lipid and liquid crystalline systems [schoen2012reflections]. Other
fields, like _mathematical geometry processing_ rely equally on quantitative
measures and qualitative visualizations for judging the quality of their
results [lavoue2010comparison]. Still, a back-and-forth between the
development of mathematical procedures and their application to real-world
data yields results that are well-grounded in mathematical quality guarantees,
yet efficient and relevant for their applications; consider for instance
[alexa2022super, sawhney2022gridfree, sharp2022spelunking]. In the field of
_exploratory data analysis_ , visualizations even form the main tool for
finding research results. Here, large, possibly high-dimensional, data sets
are investigated for patterns by embedding them, e.g., as 2D scatter plots
that can then be inspected by domain experts. With this technique, in 2020, a
novel type of anti-tumor cell was discovered [vries2020highdimensional]. None
of these research results would have been possible without the utilization of
illustrations. Furthermore, this last example utilized non-linear
dimensionality reduction techniques for the visualization of high-dimensional
data. These techniques were themselves the result of research driven by the
desire for better illustrations.

The very closely allied field of _computation_ in mathematics is a little
ahead of illustration in its maturity as a tool for mathematical research. To
give just one significant example in number theory, much recent activity has
centered around the multi-million-dollar _Simons Collaboration on Arithmetic
Geometry, Number Theory, and Computation_
,999https://simonscollab.icerm.brown.edu/ whose mission states: “Our common
perspective is that advances in computational techniques accelerate research
in arithmetic geometry and number theory, both as a source of data and
examples, and as an impetus for effective results. The dynamic interplay
between experiment, theory, and computation has historically played a pivotal
role in the development of number theory.” The work supported by the
collaboration is rapidly expanding the L-Functions and Modular Forms
Database,101010http://www.lmfdb.org an online database of mathematical objects
(including visualizations) that is at the center of much modern progress in
number theory.111111See the extensive list of publications arising from the
collaboration: https://simonscollab.icerm.brown.edu/publications/. The
discipline of mathematical computation is supported by a number of
journals121212Consider for instance “Advances in Computational Mathematics”,
https://www.springer.com/journal/10444, the “Journal for Computational and
Applied Mathematics”, https://www.sciencedirect.com/journal/journal-of-
computational-and-applied-mathematics, or the “Journal of Computational
Mathematics”, https://www.jstor.org/journal/jcompmath. and has engendered
areas of research in their own right, such as _computational geometry_.
Illustration appears to be following a similar trajectory. As it becomes more
accessible and pervasive it demands rigorous and careful study, leading to the
development of mathematical illustration as a discipline in its own right.

## Illustration as a discipline

Thurston once said, “mathematicians usually have fewer and poorer figures in
their papers and books than in their heads” [thur94]. Although the power of
good illustrations to advance mathematical knowledge is clear, they are not
simple to produce.

The challenges to creating powerful and trustworthy illustrations come on many
levels. On the one hand, some challenges are technical and concern rather
practical questions regarding the production of mathematical illustrations.
Especially with newer technologies like virtual reality or 3D modeling, the
learning curves are steep and while there are general tutorials available,
just a handful target issues specific to the illustration of
mathematics.131313A noteworthy example for introductory material, aimed at
illustration of mathematics, is the _Processing_ tutorial of Roger Antonsen,
to be found online: https://rant.codes/pcmi/. Consider for instance [Taalman]
for a nice discussion of some of the challenges of 3D printing for
mathematical illustration.

On the other hand, there are challenges within the mathematics itself. The
objects to be illustrated do not necessarily come with a description that
lends itself to a suitable illustration. Thus, a necessary initial step is the
translation of the underlying mathematical object into a form that allows
illustration in the first place. However, this transformation is usually not
enough by itself. Subsequent steps aim at making the illustration effective,
which can entail bridging the gap between the theoretical and the
computational, crafting a responsive and immersive experience, or ensuring the
illustration actually imparts the desired aspects of the mathematical object.
In particular the last part implies important theoretical considerations: What
exactly do we want to illustrate? And how do we do so faithfully, i.e.,
without creating wrong impressions of the mathematical object illustrated?

Mathematics is not the first field of research to tackle these difficulties.
There are parallels to be found in the development of the scientific method
and statistical methods for the natural sciences: Which experimental designs
and statistics can be relied upon for developing conjectures and conclusions?
Cornerstones of the scientific methods were laid down, such as the important
notion of falsifiability of a scientific theory. Similarly, statistical
methods amplified their usefulness and trustworthiness when expanded from pure
descriptive statistics to inference statistics and statistical tests to assert
the validity of results. So in fact, all scientific fields have progressed by
examining head-on some of the questions raised by their methodologies.

The question of _illustrating well_ has been asked in statistics and data
visualization, as explored in Darrell Huff’s best-selling book _How to Lie
with Statistics_ , which became a standard college text. The pioneering and
richly illustrated books of Edward Tufte and Tamara Munzner on data
visualization established that field in its own right. Every year, new
research in data visualization is discussed at various venues, such as the
Institute for Electrical and Electronics Engineers (IEEE) VIS meeting or the
EuroVis conference, and published in outlets like the IEEE Transactions on
Visualization and Computer Graphics. As it matures, the data visualization
community addresses meta-questions on its research, such as where “the value
of visualization” lies [van2005value] or “Are we making progress in
visualization research?” [correll2022we].

Thus, the example of data visualization provides a pattern of development that
the field of mathematical illustration might follow. However, in comparison,
mathematical illustration is just taking the first steps on its journey
towards being a research field. It is still facing basic challenges with
regard to creating and evaluating the illustrations it produces.

Figure 16: Roots of polynomials around $i$. Each dot is sized by
$\frac{1}{\sqrt[d]{\Delta}}$, where $\Delta$ is the discriminant of the
polynomial and $d$ is the degree. All images show the region of the complex
plane around $i$, with real values between $-.3$ and $.3$ and imaginary values
between $.7$ and $1.3$. (a) shows all cubic roots with coefficients between
$-10$ and $10$. (b) shows all cubic roots where the sum of the absolute value
of the coefficients is less than or equal to $26$. (c) shows all roots of
polynomials up to degree $13$, with coefficients $\pm 1$ [HST22] Images by
Harriss, Stange, and Trettel.

As an example of these challenges, consider the images in Figure 16 showing
polynomial roots near $i$ in the complex plane. Figure 16(a) is an image of
all roots of polynomials of degree $3$ with integer coefficients between $N$
and $-N$, where here $N=10$ [HST22]. Figure 16(c) is an image of all roots of
polynomials with coefficients from $\\{-1,1\\}$ and degree no more than $D$,
where in this case $D=13$. In both, in the region around $i$, there appears to
be a hole shaped like two ellipses overlapping at right angles.

How to interpret this shape? It turns out that in Figure 16(a) it is very much
an artifact of the algorithm for creating these images. If you consider the
picture as an approximation of all cubic roots (by allowing $N$ to tend to
infinity), there are infinitely many such polynomials. By limiting $N$, we are
looping through them in a growing hypercube in the coefficient space. The
corners of this cube are the corners jutting in toward $i$, and as the cube
expands in the coefficient space, this hole will get filled in. If instead of
looping through coefficient space in a growing cube, we choose a different
ordering, the apparent shape of the void changes. This phenomenon is shown in
Figure 16(b).

In Figure 16(c), however, the limiting shape turns out to be refined but not
substantially changed or filled in as $D$ increases from $13$ to infinity. So
this hole ‘really exists’ in the picture! The shapes one sees at the
boundaries of the limiting set of roots are explained in terms of fractal
geometry and certain symmetries of this set.141414These features are
beautifully described by John Baez on his personal website:
https://math.ucr.edu/home/baez/roots/.

Figure 17:  On the left: extrinsic view of some geodesics in Nil identified as
a set with $\mathbb{R}^{3}$. On the right: in-space view of a sphere in Nil
seen from a long distance, where light is assumed to travel along geodesics.
Images by Coulon, Matsumoto, Segerman and Trettel.

As another example of the challenges discussed above, the virtual reality
versions of Thurston’s geometries of [Coulon:2020ac] are a profound way to
experience these spaces, but can feel overwhelming and nearly psychedelic, as
our brains struggle to make sense of what we are seeing. As an alternative,
for several of the geometries, it is possible to place the geodesics of the
geometry into familiar euclidean space as curves (see Figure 17). The
interplay between these two methods of illustration can be much more
enlightening than either one alone. The mathematical arguments that are
developed to explain how one view can predict the other can end up as the
basis of a mathematical proof. Conjectures and mathematical arguments about
the space can quickly be evaluated by predicting their effect on these
illustrations.

## Looking forward

Illustrations have been used both historically and in recent state-of-the-art
research projects to expand the boundaries of knowledge in pure mathematics.
Other fields of research, such as statistics and microbiology, have
systematized visualization, and studied it in its own right.

However, as our gallery of examples shows, the quality of illustrations in
pure mathematics varies, and there is no common framework to create, discuss,
or evaluate them. To further the possibilities that illustrations provide,
there needs to be a dedicated community to tackle the next important problems.
These include, among others:

  1. 1.

How to identify illustrations that have rich potential to provide insight?

  2. 2.

How to identify (and mitigate) the ways that illustrations can mislead and
distract?

  3. 3.

How to measure the fidelity of an illustration; are perceived patterns a
result of its construction or the underlying mathematics?

  4. 4.

How can we harness the processing power and pattern-recognition capabilities
of the human visual system?

  5. 5.

How can we empower a next generation of mathematical illustrators to create
and leverage sophisticated illustrations?

  6. 6.

And how do we increase professional recognition of the illustration of
mathematics?

Exploring these questions will lay the foundation of a discipline built around
the illustration of mathematics, providing powerful tools for the advancement
of mathematical research.

### Acknowledgment

The authors would like to thank the two referees for their valuable feedback
and suggestions. The illustrating mathematics group also provided insightful
discussions on the topics presented here.

## References


# A Novel Approach to Curiosity and Explainable Reinforcement Learning via
Interpretable Sub-Goals

Connor van Rossum, Candice Feinberg, Adam Abu Shumays, Kyle Baxter, Benedek
Bartha Manuscript received April 13, 2021. (_All authors contributed equally
to this work_)The authors are with BrightMind, Bright Media Inc., California
USA (email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>benedek.bartha@wolfson.ox.ac.uk).Digital Object
Identifier ….

###### Abstract

Two key challenges within Reinforcement Learning involve improving (a) agent
learning within environments with sparse extrinsic rewards and (b) the
explainability of agent actions. We describe a curious subgoal focused agent
to address both these challenges. We use a novel method for curiosity produced
from a Generative Adversarial Network (GAN) based model of environment
transitions that is robust to stochastic environment transitions.
Additionally, we use a subgoal generating network to guide navigation. The
explainability of the agent’s behavior is increased by decomposing complex
tasks into a sequence of interpretable subgoals that do not require any manual
design. We show that this method also enables the agent to solve challenging
procedurally-generated tasks that contain stochastic transitions above other
state-of-the-art methods.

##  I Introduction

Deep Reinforcement Learning (DRL) has been tremendously successful. Agents
have been trained to defeat a champion Go player [1], outperform humans in
Atari games [2, 3] and to perform a variety of difficult robotic tasks [4, 5].
Two of the major challenges in DRL are the problem of sparse rewards and the
lack of explanablity provided by the algorithms. Random exploration in
environments with extremely sparse rewards, which is the core exploration
method in many RL approaches, is unlikely to receive a sufficient reward
signal to successfully train the agent [6]. The Deep Neural Networks (DNNs)
utilized by DRL also lack explainablity of their decision making. Several
works have focused on investigating methods for local interpretability, most
notably attribution methods, primarily in supervised learning [7, 8, 9]. These
methods generally show which inputs were most influential in producing the
DNN’s output as per the algorithms criteria. Similar methods have been
utilized in DRL in addition to reward decomposition [10] and action influence
[11].

To improve upon the tools available to increase exploration and explainability
with DRL, we present a novel method for goal orientated environments in which
the agent learns to propose subgoals. The agent is composed of a subgoal
generating network, a navigator network that learns to move towards the
current subgoal, and a Generative Adversarial Network (GAN) tasked with next
state generation. The GAN serves to build an understanding of object
interaction and the effects of actions within the environment. Subgoals are
set relative to the observation space, where a goal value for one element of
the space is selected by the subgoal generator. The navigator network takes
actions within the environment with the intent of fulfilling the chosen
subgoal. The decomposition of a complex task into a sequence of simpler
subgoals increases the human interpretability of the agent’s policies and
decisions. Exploration efficiency and performance is also increased because of
the additional feedback the subgoals provide.

Many authors have pointed out that agents that quantify novelty through
prediction errors tend to get attracted to transitions that are stochastic in
nature [12, 13, 14] decreasing the effectiveness of curiosity driven
exploration. We address this by implementing a modification of BicycleGAN [15]
as our prediction network. BicycleGAN is a multimodal image generator, able to
generate a variety of possible images from the same input. With BicycleGAN,
our agent predicts possible next states and determines impossible transitions
removing the need to make a single deterministic prediction.

We evaluate our model on procedurally generated environments, which are
naturally challenging as agents need to deal with a parameterized family of
tasks, resulting in large observation spaces and making memorizing
trajectories infeasible. It is in these environments that the agent must learn
to generalize its policies.

##  II Related Work

Intrinsic motivation has been a popular method to improve the exploration
abilities of agents within environments yielding sparse rewards. Novelty and
curiosity are two of the most prominent sources of intrinsic motivation. Where
novelty is primarily driven by reaching unfamiliar states, curiosity seeks to
better understand or predict environment dynamics. One of the methods to power
curiosity is next state prediction, where future states that have greater
prediction errors receive greater rewards to encourage the agent to take
actions within states that it is not yet able to accurately predict the result
of.

Multi-Goal Hierarchical Reinforcement learning (MGHL) [16] by Xing utilizes a
manager network, which sets auxiliary control tasks as subgoals that direct a
worker network. The manager primarily sets subgoals where the worker is
rewarded for either changing the selected pixel blocks or changing high level
features where the magnitude of the reward is proportional to the magnitude of
the change irrespective of the direction. We contrast this to our work where
our agent generates subgoals by selecting one observation element and one
particular value where the resulting subgoal is to change the chosen
observation element to the chosen value.

Another similar sub-task orientated work is the Modular Multitask
Reinforcement Learning with Policy Sketches [17]. The approach uses a high
level policy to select sub-tasks that a worker network learns to carry out. In
contrast to our work, the high level policy requires hand crafted sub tasks
for each unique environment and does not include any predictive models of the
environment.

Adversarially Motivated Intrinsic Goals (AMIGo) [18] sees a teacher network
generate goal positions for a student network to learn to reach. The teacher
network is rewarded by generating goals that the student is able to reach; but
not for those reached too quickly. In contrast to our model which only uses
the agent’s partial view, the teacher network sets goals relative to the full
observation space. Additionally, the primary purpose of the teacher network is
to drive exploration with a single goal increasing in difficulty rather than
increase the explainability of the agent through a sequence of subgoals.

##  III Curious Sub-Goal focused Agent

Our Curious Sub-Goal focused agent (CSG) is composed of three subsystems: a
next state generator powered by a GAN, the agent navigation network, and a
subgoal generating network.

###  III-A Next State Generation

Previous works on next state prediction primarily focus on producing a
deterministic prediction and comparing this prediction to the actual next
state to produce a loss function, with an optimizer using the loss function to
improve future predictions. This method of prediction does poorly in
environments with a lot of noise present or if there are stochastic elements
present. By using a GAN to generate the next state, the network is not
predicting the next state but instead producing one possible next state. In
taking this approach the agent is able to learn which transitions are possible
and which are impossible, and in doing so gain robustness to noise and
stochasticity.

In order to be able to predict a variety of possible transitions we learn a
low-dimensional latent space $\mathbf{z}\in\mathbf{R}^{\mathbf{Z}}$ which
encapsulates the stochastic aspects of the transition. We then learn a
deterministic mapping
$G(\mathbf{s}_{t},a_{t},\mathbf{z}_{t})\to\mathbf{\hat{s}}_{t+1}$. Where
$a_{t}$ is the action at time step $t$, $\mathbf{s}_{t}$ is the state at time
step $t$, and $\mathbf{\hat{s}}_{t+1}$ is a generated configuration of what
the state could be at time step $t+1$. We draw the latent vector
$\mathbf{z}_{t}$ from a prior distribution $p(\mathbf{z}_{t})$; in this work
we use a standard Gaussian distribution $\mathcal{N}(0,1)$. A discriminator
network, $D$, is trained to detect synthetic transitions generated by $G$:
$D(\mathbf{s}_{t},a_{t},\mathbf{u}_{t+1})\to d_{t}$, where
$\mathbf{u}_{t+1}\in\\{\mathbf{s}_{t+1},\mathbf{\hat{s}}_{t+1}\\}$, and
$d_{t}\in(0,1)$ corresponds to the probability that $\mathbf{u}_{t+1}$ is
synthetic ($\mathbf{u}_{t+1}=\mathbf{\hat{s}}_{t+1}$).

We use the generator $G$ and discriminator $D$ as a source for an intrinsic
curiosity reward similar to those produced from next state prediction error.
In order to produce in informative curiosity reward, we force the generator
$G$ to attempt to generate a prediction that matches the true next state
$\mathbf{s}_{t+1}$. We directly map $\mathbf{s}_{t+1}$ to the latent space
$\mathbf{z}$ using an encoding function,
$E(\mathbf{s}_{t+1})\to\mathbf{z}_{t}^{\text{enc}}$. The generator $G$ then
uses a sample of the encoded latent space $\mathbf{z}_{t}^{\text{enc}}$, the
state $\mathbf{s}_{t}$ and action $a_{t}$ to synthesize the desired output
$\mathbf{\hat{s}}_{t+1}^{\text{enc}}$. This process can be likened to the
reconstruction of $\mathbf{s}_{t+1}$ using an auto-encoder. We define the
reconstruction error of $\mathbf{s}_{t+1}$ to be
$||\mathbf{s}_{t+1}-\mathbf{\hat{s}}_{t+1}^{\text{enc}}||$, which is used as a
component of the curiosity reward function.

We define the curiosity based reward utilizing the reconstruction error and
the discriminator’s output $d_{t}$ predicting a synthetic transition:

$r_{t}^{c}=\alpha||\mathbf{s}_{t+1}-\mathbf{\hat{s}}_{t+1}^{\text{enc}}||d_{t}$
(1)

where $\alpha\in[0,\infty)$ is a scaling hyper parameter.

###  III-B Navigation Network

We consider the traditional RL framework of a Markov Decision Process with a
state space $S$, a set of actions $A$ and a transition function
$p(\mathbf{s}_{t+1}|\mathbf{s}_{t},a_{t})$ which specifies the distribution
over next states given a current state and action. At each time-step $t$, the
agent in state $\mathbf{s}_{t}\in S$ takes an action $a_{t}\in A$ by sampling
from a goal-conditioned stochastic policy
$\pi(a_{t}|\mathbf{s}_{t},g_{t};\theta_{\pi})$ represented as a neural network
with parameters $\theta_{\pi}$ where $g_{t}$ is the goal. We assume that some
goal verification function $v$ can be specified such that
$v(\mathbf{s}_{t},g_{t})=1$, if the state $\mathbf{s}_{t}$ satisfies the goal
$g_{t}$, and 0 otherwise. The undiscounted intrinsic goal reward $r^{g_{t}}$
is defined to be:

$r^{g_{t}}_{t}=\begin{cases}r&\text{if }v(\mathbf{s}_{t},g_{t})=1\\\
0&\text{otherwise}\end{cases}$ (2)

where $r>0$ is a hyper-parameter. The navigation network is trained to
maximize the discounted expected reward, $R^{Nav}_{t}$:

$R^{Nav}_{t}=\mathbf{E}[\sum_{k=0}^{T}\gamma^{k}(r^{g_{t}}_{t+k}+r^{c}_{t+k})]$
(3)

where $\gamma\in[0,1)$ is the discount factor and $r^{c}$ is the curiosity
reward.

###  III-C Sub-Goal Generation

The subgoal generator neural network is defined as
$SG(\mathbf{s}_{t},g_{0})\to g_{t}$, where $g_{t}$ is the current goal, and
$g_{0}$ is the initial goal set by the environment. Possible goals are set
relative to the agent’s observation space. It selects an input position and a
goal value for that position. The navigator network is then tasked with
altering the selected input to be the goal value. A new subgoal can be
proposed at the start of an episode, once the previous one is reached or the
previous one is abandoned. The agent abandons subgoals if it exceeds a hyper-
parameter that sets the step limit to attempt to reach the goal. Finally, the
network can also choose to set no subgoal, which is taken when the network
selects the current goal as the subgoal.

The subgoal generator network is trained to maximize the discounted expected
reward:

$R^{SG}_{t}=\mathbf{E}[\sum_{k=0}^{T}\gamma^{k}(r^{c}_{t+k}+\frac{v(\mathbf{s}_{t+k},g_{t+k})+\beta}{1+\beta}r^{e}_{t+k})]$
(4)

where $\gamma\in[0,1)$ is the discount factor, $r^{c}$ is the curiosity
reward, $r^{e}$ is the extrinsic reward within the environment,
$v(\mathbf{s}_{t},g_{t})$ is the goal verification function, and
$\beta\in[0,\infty)$ is a scaling parameter which determines the penalty
applied to the extrinsic reward when the goal does not align with the reward.

##  IV Experiments

We use a modified version of the TorchBeast implementation of IMPALA [19]
(https://github.com/facebookresearch/torchbeast) as the framework for our
code.

We establish our proof of concept within the challenging and scaleable
environment of minigrid (https://github.com/maximecb/gym-minigrid). This
provides a good test bed for the agent as the observations are symbolic rather
than high dimensional allowing for clear and interpretable subgoals.

###  IV-A Environments

We use a modified version of Door & Key environments within the minigrid
package (Figure 1). The environment randomizes the position of the dividing
wall, door, and key, as well as the agent’s starting position and orientation.
We alter the environment to also randomize the location of the agent’s target
position within the second room. The randomization adds increased difficulty
by increasing the number of possible trajectories and adds stochastic
transitions into the environment.

The environments are comprised of $N\times N$ tiles, where each tile contains
at most one of the following objects: door, key, wall, goal. If it contains
none, it is the ‘empty’ object. The agent has limited vision, seeing within a
$M\times M$ square with the agent positioned on a center edge of the square.
The agent cannot see behind walls or closed doors even if they fall within the
vision range. Such objects are assigned as ‘unseen’ rather than their true
values. All objects and their attributes are transformed using an embedding
layer unique for each network before being fed into the remainder of the
network. The navigator is able to perform the following actions: turn left,
turn right, move forward, pick up an object, drop an object, or toggle
(interact with objects such as opening a door).

(a) $5\times 5$ Door & Key

(b) $6\times 6$ Door & Key

(c) $8\times 8$ Door & Key

(d) $10\times 10$ Door & Key

Figure 1: Examples of Door & Key environments. To complete the environment the
agent (red triangle) must proceed through the locked door (yellow box) to the
green goal zone in the adjacent room. Vision is only in the direction the
agent faces (red triangle) and expands from a $5\times 5$ box in (1(a), 1(b),
1(c)) to a $9\times 9$ box in (1(d)). Objects within the agent’s vision are
indicated by a lighter gray shading.

###  IV-B Implementations

The navigator and subgoal generator have a network architecture consisting of
a fully connected layer with rectified linear units, two LSTM layers, followed
by a final fully connected layer. They are trained using the TorchBeast [20]
implementation of IMPALA [19].

The GAN’s encoder simply consists of three fully connected layers interleaved
with a hyperbolic tangent activation function. Similarly, both the generator
and discriminator consist of four fully connected layers interleaved with a
hyperbolic tangent activation function. In addition, the singular output from
the final layer of the discriminator is activated by a sigmoid function. We
train the GAN using the same methods as BicycleGAN [15], with one exception.
As BicylceGAN is designed for continuous data sets, we replace the standard
generator loss function with the MaliGAN [21] gradient estimator.

For comparison, we implement 3 alternate methods. We use IMPALA without any
intrinsic motivation as a standard deep RL baseline. Additionally, we include
two alternative methods utilizing intrinsic motivation. Random Network
Distillation Exploration (RND) [22] which uses next state feature prediction
error as a curiosity based form of intrinsic motivation, where features are
produced from a random neural network. AMIGO is also trained where the teacher
and student network architectures match those described in the AMIGO paper
[18] and accompanying code https://github.com/facebookresearch/adversarially-
motivated-intrinsic-goals. The AMIGO networks receive the full absolute view
of the environment, as opposed to the agent centric partial view the other
methods receive. Both RND and IMPALA share the same network architecture as
our navigator and subgoal generator networks. All three alternate methods are
trained using the TorchBeast framework.

Model | Door & Key Mean Extrinsic Reward ($N\times N$ size) |
---|---|---
$5\times 5$ |  $6\times 6$ |  $8\times 8$ |  $10\times 10$
CSG |  $0.85\pm 0.01$ |  $0.89\pm 0.00$ |  $0.88\pm 0.01$ |  $0.92\pm 0.01$
IMPALA |  $0.91\pm 0.00$ |  $0.90\pm 0.00$ |  $0.94\pm 0.01$ |  $0.00\pm 0.00$
RND |  $0.87\pm 0.01$ |  $0.86\pm 0.00$ |  $0.89\pm 0.00$ |  $0.82\pm 0.01$
AMIGO |  $0.88\pm 0.01$ |  $0.86\pm 0.00$ |  $0.87\pm 0.02$ |  $0.81\pm 0.02$
TABLE I: Comparison of mean extrinsic reward at the end of training. $5\times
5$ was run for 5,000,000 steps, $6\times 6$ for 30,000,000 steps, $8\times 8$
for 60,000,000 steps, and $10\times 10$ for 90,000,000 steps.

###  IV-C Results and Discussion

We use the mean extrinsic reward at the end of training to evaluate the
performance of each agent. Where the extrinsic reward defined in the
environment is $r^{e}_{t}=1-(0.9t)/t_{\text{max}}$. We summarize our results
in Table I.

IMPALA is unable to reach the goal once the environment space becomes too
large given the lack of intrinsic motivation for exploration. However it
performs better in the low dimensional spaces due to the lack of reward noise
the other methods utilize. RND and AMIGO perform comparatively across all
environment dimensions. Our agent, CSG, performs better than both RND and
AMIGO in the highest dimensional environment.

(a) Time step $0$.
Subgoal: Pick up key

(b) Time step $5$.
New subgoal: Go to door.

(c) Time step $10$.
New subgoal: Go to goal.

(d) Time step $17$.
Final step before episode ends

Figure 2: Trained Agent completing the $8\times 8$ environment. Frames (a) -
(c) show a new subgoal generation, in addition (d) shows the last frame before
the episode ends.

The novel subgoal generation has the added benefit of increasing the agent’s
explainablity. Figure 2 highlights an example set of goals used to direct the
navigator network in order to solve the environment. The subgoals listed are
the human interpretation of the subgoal generator output. In figure 2 the
agent receives a $5\times 5$ field of view:

$\begin{bmatrix}\mathbf{b}_{11}&\mathbf{b}_{12}&\mathbf{b}_{13}&\mathbf{b}_{14}&\mathbf{b}_{15}\\\
\mathbf{b}_{21}&\mathbf{b}_{22}&\mathbf{b}_{23}&\mathbf{b}_{24}&\mathbf{b}_{25}\\\
\mathbf{b}_{31}&\mathbf{b}_{32}&\mathbf{b}_{33}&\mathbf{b}_{34}&\mathbf{b}_{35}\\\
\mathbf{b}_{41}&\mathbf{b}_{42}&\mathbf{b}_{43}&\mathbf{b}_{44}&\mathbf{b}_{45}\\\
\mathbf{b}_{51}&\mathbf{b}_{52}&\mathbf{b}_{53}&\mathbf{b}_{54}&\mathbf{b}_{55}\\\
\end{bmatrix}$

where $\mathbf{b}_{ij}$ is an object. The field of view moves with the agent
such that the agent is always positioned at $\mathbf{b}_{35}$, with
$\mathbf{b}_{34}$ directly in front. At time step $0$, in Figure 2(a), the
subgoal generator selects $\mathbf{b}_{35}$ to be changed to the object index
value corresponding to the yellow key, which can be interpreted as the agent
holding the key, and hence the subgoal is to pick up the key. Similarly, at
time step $5$, in Figure 2(b), the subgoal generator selects $\mathbf{b}_{34}$
to be changed to the object index value corresponding to the locked yellow
door, which we interpret as the agent setting going to the door as its
subgoal. Finally, at time step $10$, in Figure 2(c), $\mathbf{b}_{35}$ is
selected to be changed to the object index value corresponding to the green
goal zone as its subgoal, which is also the overarching goal set by the
environment.

The ability to generate explicit subgoals has two benefits: the first is to
developers, as it gives a window into the training process, showing stages
along the way to the agent’s final functionality, that are accessible to human
language, logic and reasoning. Therefore, more interventions are possible,
giving the ability to craft and debug the training process at a finer grain
than conventional methods. The second benefit is to end users of the agent.
These users will be able to understand in human terms how and why the agent
made the decisions it did, leading to greater trust in the agent. Along this
track there is a future long-term benefit to end users in that their feedback
can be integrated as reward to train the agent, giving them the ability to
teach the agent according to their preferences. Users will get a greater
measure of control and insight into the agent’s reasoning and decision
criteria.

##  V Conclusion

In this work we propose a method for developing a curious subgoal focused
agent that has a robust method for driving curiosity and that decomposes
complex tasks using interpretable subgoals increasing both exploration and
explainability. In future work we would like to expand the testing
environments to cement CSG as a new state of the art hard exploration method.

In this implementation of the CSG, applications are limited. They are
restricted to discrete observation spaces with basic dynamics. However, it may
be easily incorporated to run with any modern DRL agent operating within these
constraints as it needs no hand-crafted features or demonstrations. In future
work we also would like to include abstract concepts and representations as
possible subgoals. For example if the pixel input version of gridworld were
used, we would like to enable concepts for ‘key’ and ‘door’ created by the
agent to be used rather than any specific pixel.

## References

  * [1] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, “Mastering the game of go without human knowledge,” _Nature_ , vol. 550, no. 7676, pp. 354–359, Oct 2017. [Online]. Available: http://www.nature.com/articles/nature24270
  * [2] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver, “Mastering atari, go, chess and shogi by planning with a learned model,” _arXiv_ , pp. 1–21, 2019. [Online]. Available: https://arxiv.org/pdf/1911.08265v2.pdf
  * [3] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, D. Guo, and C. Blundell, “Agent57: Outperforming the atari human benchmark,” _arXiv_ , 2020. [Online]. Available: https://arxiv.org/pdf/2003.13350v1.pdf
  * [4] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” _33rd International Conference on Machine Learning, ICML 2016_ , vol. 3, pp. 2001–2014, 2016. [Online]. Available: https://arxiv.org/pdf/1604.06778v3.pdf
  * [5] A. Raffin and F. Stulp, “Generalized State-Dependent Exploration for Deep Reinforcement Learning in Robotics,” _arXiv_ , pp. 1–21, 2020. [Online]. Available: https://arxiv.org/pdf/2005.05719v1.pdf
  * [6] M. J. Mataric, “Reward Functions for Accelerated Learning,” W. W. Cohen and H. B. T. M. L. P. . Hirsh, Eds. San Francisco (CA): Morgan Kaufmann, 1994, pp. 181–189. [Online]. Available: http://www.sciencedirect.com/science/article/pii/B9781558603356500301
  * [7] M. T. Ribeiro, S. Singh, and C. Guestrin, “”Why should i trust you?” Explaining the predictions of any classifier,” _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , vol. 13-17-Augu, pp. 1135–1144, 2016. [Online]. Available: https://arxiv.org/pdf/1602.04938.pdf
  * [8] M. Ancona, C. Öztireli, and M. Gross, “Explaining deep neural networks with a polynomial time algorithm for Shapley values approximation,” _36th International Conference on Machine Learning, ICML 2019_ , vol. 2019-June, pp. 400–409, 2019. [Online]. Available: https://arxiv.org/pdf/1903.10992.pdf
  * [9] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” _34th International Conference on Machine Learning, ICML 2017_ , vol. 7, pp. 5109–5118, 2017. [Online]. Available: https://arxiv.org/pdf/1703.01365.pdf
  * [10] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez, “Explainable Reinforcement Learning via Reward Decomposition,” _Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence_ , pp. 47—-53, 2019\. [Online]. Available: https://web.engr.oregonstate.edu/{~}erwig/papers/ExplainableRL{\\_}XAI19.pdf
  * [11] P. Madumal, T. Miller, L. Sonenberg, and F. Vetere, “Explainable Reinforcement Learning Through a Causal Lens,” 2019. [Online]. Available: http://arxiv.org/abs/1905.10958
  * [12] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” _34th International Conference on Machine Learning, ICML 2017_ , vol. 6, pp. 4261–4270, 2017. [Online]. Available: https://arxiv.org/pdf/1705.05363.pdf
  * [13] N. Bougie and R. Ichise, “Skill-based curiosity for intrinsically motivated reinforcement learning,” _Machine Learning_ , vol. 109, no. 3, pp. 493–512, Mar 2020. [Online]. Available: http://link.springer.com/10.1007/s10994-019-05845-8
  * [14] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros, “Large-scale study of curiosity-driven learning,” _arXiv_ , 2018. [Online]. Available: https://arxiv.org/pdf/1808.04355.pdf
  * [15] J. Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman, “Toward multimodal image-to-image translation,” _Advances in Neural Information Processing Systems_ , vol. 2017-Decem, no. 1, pp. 466–477, 2017. [Online]. Available: https://arxiv.org/pdf/1711.11586.pdf
  * [16] L. Xing, “Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning,” _arXiv_ , 2019. [Online]. Available: https://arxiv.org/pdf/1905.05180.pdf
  * [17] J. Andreas, D. Klein, and S. Levine, “Modular multitask reinforcement learning with policy sketches,” _34th International Conference on Machine Learning, ICML 2017_ , vol. 1, pp. 229–239, 2017. [Online]. Available: https://arxiv.org/pdf/1611.01796.pdf
  * [18] A. Campero, R. Raileanu, H. Küttler, J. B. Tenenbaum, T. Rocktäschel, and E. Grefenstette, “Learning with AMIGo: Adversarially Motivated Intrinsic Goals,” 2020. [Online]. Available: http://arxiv.org/abs/2006.12122
  * [19] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, B. Yotam, F. Vlad, H. Tim, I. Dunning, S. Legg, and K. Kavukcuoglu, “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures,” _35th International Conference on Machine Learning, ICML 2018_ , vol. 4, pp. 2263–2284, 2018. [Online]. Available: https://arxiv.org/pdf/1802.01561.pdf
  * [20] H. Küttler, N. Nardelli, T. Lavril, M. Selvatici, V. Sivakumar, T. Rocktäschel, and E. Grefenstette, “TorchBeast: A PyTorch platform for distributed RL,” _arXiv_ , pp. 1–10, 2019. [Online]. Available: https://arxiv.org/pdf/1910.03552.pdf
  * [21] T. Che, Y. Li, R. Zhang, R. Devon Hjelm, W. Li, Y. Song, and Y. Bengio, “Maximum-likelihood augmented discrete generative adversarial networks,” _arXiv_ , 2017. [Online]. Available: https://arxiv.org/pdf/1702.07983.pdf
  * [22] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” _arXiv_ , pp. 1–17, 2018. [Online]. Available: https://arxiv.org/pdf/1810.12894.pdf