# Why are NLP Models Fumbling at Elementary Math?
A Survey of Deep Learning based Word Problem Solvers

Sowmya S Sundaram  L3S Research Center, Hannover, Germany  Sairam Gurajada
IBM Research, Almaden, USA111Work done while author was here Marco Fisichella
L3S Research Center, Hannover, Germany  Deepak P  Queen’s University, Belfast,
UK  Savitha Sam Abraham  University of Örebro, Sweden

###### Abstract

From the latter half of the last decade, there has been a growing interest in
developing algorithms for automatically solving mathematical word problems
(MWP). It is a challenging and unique task that demands blending surface level
text pattern recognition with mathematical reasoning. In spite of extensive
research, we are still miles away from building robust representations of
elementary math word problems and effective solutions for the general task. In
this paper, we critically examine the various models that have been developed
for solving word problems, their pros and cons and the challenges ahead. In
the last two years, a lot of deep learning models have recorded competing
results on benchmark datasets, making a critical and conceptual analysis of
literature highly useful at this juncture. We take a step back and analyse
why, in spite of this abundance in scholarly interest, the predominantly used
experiment and dataset designs continue to be a stumbling block. From the
vantage point of having analyzed the literature closely, we also endeavour to
provide a road-map for future math word problem research.

## Introduction

Natural language processing has been one of the most popular and intriguing
AI-complete sub-fields of artificial intelligence. One of the earliest systems
arguably was the PhD Thesis on automatically solving arithmetic word problems
Bobrow (1964). The challenge lay on two fronts (a) analysing unconstrained
natural language, and (b) mapping intricate text patterns onto a small
mathematical vocabulary, for usage within its reasoning framework.

Right up until 2010, there has been prolific exploration of MWP solvers, for
various domains (such as algebra, percentages, ratio etc). These solvers
relied heavily on hand-crafted rules for bridging the gap between language and
the corresponding mathematical notation. As can be surmised, these approaches,
while being effective within their niches, did not generalise well to address
the broader problem of solving MWPs. Moreover, due to the lack of well
accepted datasets, it is hard to measure the relative performance across
proposed systems Mukherjee and Garain (2008).

Input |  Kevin has 3 books. Kylie has 7 books. How many books do they have together?
---|---
Answer |  10
Table 1: Typical Example

The pioneering work by Kushman et al. (2014) employed statistical methods to
solve word problems, which set the stage for the development of automatic MWP
solvers using traditional machine learning methods. The work also introduced
the first dataset, popularly referred to as Alg514, that had multiple linear
equations associated with a problem. The machine learning task was to map the
coefficients in the equation to the numbers in the problem. The dataset
comprises data units with a triplet structure: natural language question,
equation set, and the final answer.

Mirroring recent trends in NLP, there has been an explosion of deep learning
models for MWP. Some of the early ones Wang et al. (2017); Ling et al. (2017)
modeled the task of converting the text to equation as a sequence-to-sequence
(seq2seq, for short) problem. In this context, increasingly complex models
have been proposed to capture semantics beyond the surface text. Some have
captured structural information (pertaining to input text, domain knowledge,
output equation structure) in the form of graphs and used advances in graph
neural networks (Li et al. (2020), Zhang et al. (2020c), etc.). Others have
utilised the benefits of transformers in their modelling (Liang et al. (2021),
Piękos et al. (2021), etc.). We will explore these models in detail.

Since this is a problem that has consistently attracted steady (arguably, slow
and steady) attention, ostensibly right from the birth of the field of NLP, a
survey of the problem solving techniques offers a good horizon for
researchers. The authors collected 30+ papers on deep learning for word
problem solving, published over the last three years across premier NLP
avenues. Each paper has its own unique intuitive basis, but most achieve
comparable empirical performance. The profusion of methods has made it hard to
crisply point out the state-of-the-art, even for fairly general word problem
solving settings. Hence, a broad overview of the techniques employed gives a
good grounding for further research. Similarly, understanding the source,
settings and relevance of datasets is often important. For example, there are
many datasets that are often referred to by multiple names at different points
in time. Also, the finer aspects of problem scenario varies across systems
(whether multiple equations can be solved, whether it is restricted to algebra
or more domains etc.). In this survey, we systematically analyse the models,
list the benchmark datasets and examine word problem solving literature using
a critical analysis perspective.

### Related Surveys

There are two seminal surveys that cover word problem solving research. One,
Mukherjee and Garain (2008), has a detailed overview of the symbolic solvers
for this problem. The second, more recent one Zhang et al. (2020a), covers
models proposed up until 2020. In the last two years, there has been a sharp
spike in algorithms developed, that focus on various aspects of deep learning,
to model this problem. Our survey is predominantly based on these deep
learning models. The differentiating aspects of our survey from another
related one, Faldu et al. (2021) are: the usage of a critical perspective to
analyze deep learning models, which enables us to identify robustness
deficiencies in the methods analytically, and also to trace them back to model
design and dataset choice issues. We will also include empirical performance
values of various methods on popular datasets, and deliberate on future
directions.

## Symbolic Solvers

We begin our discussion with traditional solvers that employ a rule-based
method to convert text input to a set of symbols. Early solvers within this
family such as STUDENT Bobrow (1964) and other subsequent ones (Fletcher
(1985), Dellarosa (1986)), the dominant methodology was to map natural
language input to an underlying pre-defined schema. This calls for a mechanism
to distil common expectations of language, word problems and the corresponding
mathematical notation, to form bespoke rulesets that will power the
conversion. This may be seen as setting up a slot-filling mechanism that map
the main entities of the word problem to a slots within a set of equation
templates.

An example of a schema for algebraic MWP is shown in Table 2.

Problem |  John has 5 apples. He gave 2 to Mary. How many does he have now?
---|---
Template |  [Owner1] has [X] [obj].
|  [Owner1] [transfer] [Y] [obj] to [Owner2].
|  [Owner1] has [Z] [obj].
|  Z = X - Y
Slot-Filling |  [John] has [5] [apple].
|  [John] [give] [2] [apple] to [Mary].
|  [Mary] has [Z] [apple].
|  Z = 5 - 2
Answer |  Z = 3
Table 2: Workflow of Symbolic Solvers

The advantage is that these systems are robust in handling irrelevant
information, with expert-authored rulesets enabling focus towards pertinent
parts of the problem. To further enhance the practical effectiveness within
applications focusing niche domains, research focused on tailoring these
symbolic systems for target domains Mukherjee and Garain (2008). As one can
observe, the rules would need to be exhaustive to capture the myriad nuances
of language. Thus, they did not generalise well across varying language
styles. Since each system was designed for a particular domain, comparative
performance evaluation was hindered by the unavailability of cross-domain
datasets.

## Statistical Solvers

As with many tasks in natural language processing, statistical machine
learning techniques to solve word problems started dominating the field from
2014. The central theme of these techniques has been to score a number of
potential solutions (may be equations or expression trees as we will see
shortly) within an optimization based scoring framework, and subsequently
arrive at the correct mathematical model for the given text. This may be
thought of as viewing the task as a structure prediction challenge Zhang et
al. (2020a).

$P(y|x;\theta)=\frac{e^{\theta.\phi(x,y)}}{\sum_{y^{\prime}\in
Y}e^{\theta.\phi(x,y^{\prime})}}$ (1)

As with optimization problems, Equation 1 refers to the problem of learning
parameters $\theta$, which relate to the feature function $\phi$. Consider
labeled dataset $D$ consisting of $n$ pairs $(x,y,a)$ where $x$ is the natural
language question, $y$ is the mathematical expression and $a$ is the numerical
answer. The task is to score all possible expressions $Y$, and maximise the
choice of the labelled $y$ through an optimisation setting. This is done by
modifying the parameters $\theta$ of the feature function $\phi(x,y)$.
Different models propose different formulations of $\phi$. In practise, beam
search is used as a control mechanism. We grouped the prolific algorithms that
were developed, based on the type of mathematical structure $y$ \- either as
equation templates or expression trees. Equation templates were mined from
training data, much like the slot filling idea of symbolic systems. However,
they became a bottleneck to generalizability, if the word problem at inference
time, was from an unseen equation template. To address this issue, expression
trees, with unambiguous post-fix traversals, were used to model equations.
Though they restricted the complexity of the systems to single equation
models, they offered wider scope for generalizability.

### Equation Templates

Equation templates extract out the numeric coefficients and maintain the
variable and operator structure. This was used as a popular representation of
mathematical modelling. To begin with, Kushman et al. (2014), used structure
prediction to score both equation templates and alignment of the numerals in
the input text to coefficients in the template. Using a state based
representation, Hosseini et al. (2014) modelled simple elementary level word
problems with emphasis on verb categorisation. Zhou et al. (2015) enhanced the
work done by Kushman et al. (2014) by using quadratic programming to increase
efficiency. Upadhyay and Chang (2017) introduced a sophisticated method of
representing derivations in this space.

### Expression Trees

Expression trees are applicable only to single equation systems. The single
equation is represented as a tree, with leaves of the tree being numbers and
the internal nodes being operators as illustrated in Koncel-Kedziorski et al.
(2015).

Expression tree based methods converge faster, understandably due to the
diminished complexity of the model. Some solvers (such as Roy and Roth (2015))
had a joint optimisation objective to identify relevant numbers and populating
the expression tree. On the other hand, Koncel-Kedziorski et al. (2015); Mitra
and Baral (2016) used domain knowledge to constrain the search space.

## Neural Solvers

Among the major challenges for the solvers we have seen so far was that of
converting the input text into a meaningful feature space to enable downstream
solving; the main divergences across papers seen across the previous sections
has been based on the technological flavour and methodology employed for such
text-to-representation conversion.

for tree= draw, text width=1.4cm, align=center , forked edges, [Automatic
Word
Problem
Solvers [Symbolic
Solvers] [Statistical
Solvers [Expression
Trees] [Equation
Templates] ] [Neural
Solvers [Seq2Seq] [Graph-Based] [Transformers] [Contrastive] [Knowledge
Distillation] ] ] [draw, fit=(current bounding box.south east) (current
bounding box.north west)] ;

Figure 1: Types of Word Problem Solvers

The advent of distributed representations for text Le and Mikolov (2014);
Peters et al. (2018); Pennington et al. (2014); Devlin et al. (2018), marked a
sharp departure in the line of inquiry towards solving math word problems,
focusing on the details of the learning architecture rather than feature-space
modelling. There have even been domain specific distributed representation
learners for word problems Sundaram et al. (2020). As an example of solvers,
Ling et al. (2017) designed a seq2seq model that incorporated learning a
program as an intermediate step. This and other early works made it
fashionable to treat the word problem solving task as a language translation
task, i.e., translating from the input natural language text to a sequence of
characters representing either the equation or a sequence of predicates. This
design choice, however, has its limitations, which are sometimes severe in
terms of the restrictions they place on math problems that can be admitted
within such architectures Patel et al. (2021). A few of these linguistic vs.
math structure understanding challenges, especially for neural solvers, are
illustrated in Table 3. As an important example, equation systems that involve
solving multiple equations are not straightforward to address within such a
framework. A notable exception to this is the popular baseline MathDQN Wang et
al. (2018), which employs deep reinforcement learning. We consider different
families of deep learning solvers within separate sub-sections herein.

Input |  Kevin has 3 books. Kylie has 7 books and 3 pencils. How many books do they have together?
---|---
Mathematical Structure |  3 \+ 7
Linguistic Structure |  (Person1) has (X) (object1). (Person2) has (Y) (object1) and (Z) (object2).
Challenges |  (1) Order of X and Y does not matter in addition (2) multiple equations do not make a sequence (3) Similar objects need to be grouped together
Table 3: Typical Challenges

### Seq2Seq Solvers

The ubiquitous Seq2Seq Sutskever et al. (2014) architecture is widely popular
for automatic word problem solving. From early direct use of LSTMs Hochreiter
and Schmidhuber (1997) / GRUs Cho et al. (2014) in Seq2Seq models (Huang et
al. (2017), Wang et al. (2017)) to complex models that include domain
knowledge Ling et al. (2017); Qin et al. (2020); Chiang and Chen (2019); Qin
et al. (2021)), diverse formulations of this basic architecture have been
employed.

Input sequenceOutput sequenceMath ProblemEquationWord EmbeddingsCharacter/Word
Embeddings Figure 2: General Seq2Seq Formulations

The initial set of models used Seq2Seq as is, with small variations in the
usage of LSTM or GRUs or with simple heuristics (for example, Huang et al.
(2016) used retrieval to enhance the results). Significant improvements were
made by including some mathematical aspects. This, once again, demonstrates
that the task is not merely that of language translation. Ling et al. (2017)
converted the word problem to a text containing the explanation or rationale.
This was done through an intermediate step of generating a step-by-step
program on a large dataset. Though the accuracy values reported were low, the
domains spanned anywhere between probability to relative velocity, and the
unified framework demonstrated performing meaningful analysis through
qualitative illustrations. This was improved upon by Amini et al. (2019),
which enhanced the dataset and added domain information through a label on the
category. The SAU-Solver Qin et al. (2020) introduced a tree like
representation with semantic elements that align to the word problem. As seen
in Table 6, this is a formidable contender. In Chiang and Chen (2019), a novel
way of decomposing the equation construction into a set of stack operations -
such that more nuanced mapping between language and operators can be learned -
was designed. There is a burgeoning section of the literature that is invested
in using neuro-symbolic reasoning to bridge this gap between perception level
tasks (language understanding) and cognitive level tasks (mathematical
reasoning). An example of this is Qin et al. (2021). With this discussion, it
is clear that adding some form of domain knowledge benefits an automatic
solver.

### Graph-based Solvers

With the advent of graph modeling Xia et al. (2019) and enhanced interest in
multi-modal processing, the graph data structure became a vehicle for adding
knowledge to solvers. One way of enabling this has been to simply model the
input problem as a graph Feng et al. (2021); Li et al. (2020); Yu et al.
(2021); Hong et al. (2021). This incorporates domain knowledge of (a) language
interactions pertinent to mathematical reasoning, or (b) quantity graphs
stating how various numerals in the text are connected. Another way is to
model the decoder side to accept graphical input of equations Xie and Sun
(2019); Lin et al. (2021); Zaporojets et al. (2021); Cao et al. (2021); Liu et
al. (2019); Wu et al. (2021b). Another natural pathway that has been employed
towards leveraging graphs is to use graph neural networks for both encoder and
decoder Zhang et al. (2020c); Wu et al. (2020, 2021a); Shen and Jin (2020).

Graph InputOutput SequenceInput SequenceGraph OutputGraph InputGraph Output
Figure 3: General Graph based Formulations

Graphs are capable of representing complex relationships. With the time-tested
success of graph neural networks (GNNs) Wu et al. (2021c), they fit easily
into the encoder-decoder architecture. Intuitively, when graphs are used on
the input side, we can model complex semantic relationships in the linguistic
side of the task. When graphs are used on the decoder side, relationships
between the numerical entities or an intermediate representation of the
problem can be captured. Analogously, graph-to-graph modelling enables
matching the semantics of both language and math. This does not necessarily
imply graph-to-graph outperforms all the other formulations. There are unique
pros and cons of each of the graph-based papers, as both language and
mathematical models are hard to (a) model separately and (b) model the
interactions. The interesting observation as seen in Table 6, graph based
models are both popular and powerful. Unlike sequences, when the input text is
represented as a graph, the focus is more on relevant entities rather than a
stream of text. Similarly quantity graphs or semantics informed graphs,
eliminate ordering ambiguities in equations. This formulation, however, still
does not address the multiple equation problem.

### Transformers

Transformers Vaswani et al. (2017) have lately revolutionised the field of
NLP. Word problem solving has been no exception. Through the use of BERT
Devlin et al. (2018) embeddings or through transformer based encoder-decoder
models, some recent research has leveraged concepts from transformer models
Liu et al. (2019); Kim et al. (2020). The translation has been modeled
variously, such as from text to explanation Piękos et al. (2021); Griffith and
Kalita (2020), or from text to equation Shen et al. (2021); Liang et al.
(2021).

When moving from Word2Vec Mikolov et al. (2013) vectors to BERT embeddings
Devlin et al. (2018), massive gains were expected due to (a) greater
incorporation of context level information and (b) automatic capturing of
relevant information as BERT is essentially a Masked Language Model.
Interestingly, the gains do not have as large a margin as seen in other
language tasks such as question answering or machine translation Devlin et al.
(2018). BERT is a large model that needs to be fine tuned with domain specific
information. The small gains point towards low quality of word problem
datasets, which is in line with the fact that the datasets are either quite
small by deep learning standards or that they have high lexical overlap,
effectively suggesting that the set of characteristic word problems are small.

### Contrastive Solvers

With the widespread usage of Siamese networks Koch et al. (2015), the idea of
building representations that contrast between vectorial representations
across classes in data has seen some interest. In the context of word problem
solving, a few bespoke transformer based encoder-decoder models Li et al.
(2021b); Hong et al. (2021) have been proposed; these seek to effectively
leverage contrastive learning Le-Khac et al. (2020).

This is a relatively new paradigm and more research needs to emerge to
ascertain definite trends. One of the main stumbling blocks of word problem
solving is that two highly linguistically similar looking word problems may
have entirely different mathematical structure. Since contrastive learning is
built on the principle that similar input examples lead to closer
representations, it allows one to use the notion of similarity and
dissimilarity to overcome this bottleneck and consciously design semantically
informed intermediate representations, such that the similarity is built not
only from the language vocabulary, but also from the mathematical concepts.

### Teacher-Student Solvers

The paradigm of knowledge distillation, in the wake of large, generic end-to-
end models, has become popular in NLP Li et al. (2021a). The underlying idea
behind this is to distill smaller task-specific models from a generic large
pre-trained or generic model. Since word problem datasets are of comparatively
smaller size, it is but logical that large generic networks can be fine-tuned
for downstream processing of word problem solving, as favourably demonstrated
by Zhang et al. (2020b) and Hong et al. (2021).

Once again, this is an emerging paradigm. Similar to the discussion we
presented with transformer based models, the fact that the presence of pre-
trained language models alone is not sufficient for this task has bolstered
initial efforts in this direction. Knowledge distillation enables a model to
focus the learnings of one generic model on to a smaller, more focussed one,
especially with less datapoints. Hence, the method of adding semantic
information through the usage of knowledge distillation algorithms is
promising and one to look out for.

## Domain-Niche Solvers

Some research, encompassing families of statistical solvers and deep models,
focus on the pertinent characteristics of a particular domain in mathematics,
such as probability word problems Dries et al. (2017); Suster et al. (2021);
Tsai et al. (2021), number theory word problems Shi et al. (2015), geometry
word problems Seo et al. (2015); Chen et al. (2021) and age word problems
Sundaram and Abraham (2019).

## Datasets

Datasets used for math word problem solving are listed in Table 4 with their
characteristics. The top section of the table describes datasets with
relatively fewer data objects ($\leq 1k$, to be specific). The bottom half
consists of more recent datasets that are larger and more popularly used
within deep learning methodologies.

Dataset | Type | Domain | Size | Source
---|---|---|---|---
Alg514 | Multi-equation | (+,-,*,/) | 514 | Kushman et al. (2014)
(SimulEq-S) |  |  |  |
AddSub | Single-equation | (+,-) | 340 | Hosseini et al. (2014)
(AI2) |  |  |  |
SingleOp | Single-equation | (+,-,*,/) | 562 | Roy et al. (2015)
(Illinois, IL) |  |  |  |
SingleEq | Single-equation | (+,-,*,/) | 508 | Koncel-Kedziorski et al. (2015)
MAWPS | Multi-equation | (+,-,*,/) | 3320 | Koncel-Kedziorski et al. (2016)
MultiArith | Single-equation | (+,-,*,/) | 600 | Roy and Roth (2015)
(Common Core, CC) |  |  |  |
AllArith | Single-equation | (+,-,*,/) | 831 | Roy and Roth (2017)
Perturb | Single-equation | (+,-,*,/) | 661 | Roy and Roth (2017)
Aggregate | Single-equation | (+,-,*,/) | 1492 | Roy and Roth (2017)
DRAW-1k | Multi-equation | (+,-,*,/) | 1k | Upadhyay and Chang (2017)
AsDIV-A | Single-equation | (+,-,*,/) | 2373 | Miao et al. (2020)
SVAMP | Single-equation | (+,-,*,/) | 1000 | Patel et al. (2021)
Dolphin18k | Multi-equation | (+,-,*,/) | 18k | Huang et al. (2016)
AQuA-RAT | Multiple-choice | - | 100k | Ling et al. (2017)
Math23k* | Single-equation | (+,-,*,/) | 23k | Huang et al. (2017)
MathQA | Single-equation | (+,-,*,/) | 35k | Amini et al. (2019)
HMWP* | Multi-equation | (+,-,*,/) | 5k | Qin et al. (2020)
Ape210k* | Single-equation | (+,-,*,/) | 210k | Liang et al. (2021)
GSM8k | Single-equation | (+,-,*,/) | 8.5k | Cobbe et al. (2021)
CM17k* | Multi-equation | (+,-,*,/) | 17k | Qin et al. (2021)
Table 4: Datasets
(*Chinese Datasets)

### Small Datasets

The pioneering work in solving word problems Kushman et al. (2014), introduced
a classical dataset (Alg514) of 514 word problems, across various domains in
algebra (such as percentages, mixtures, speeds etc). This dataset was
annotated with multiple equations per problem. AddSub was introduced in
Hosseini et al. (2014), with simple addition/subtraction problems, exhibiting
limited language complexity. SingleOp Roy et al. (2015) and MultiArith Roy and
Roth (2015) were proposed such that there is a control over the operators
(single operator in the former and two operators in the latter). SingleEq
Koncel-Kedziorski et al. (2015) is unique in incorporating long sentence
structures for elementary level school problems. AllArith Roy and Roth (2017)
is a subset of the union of AddSub, SingleEq and SingleOp. "Perturb" is a set
of slightly perturbed word problems of AllArith, whereas Aggregate is the
union of AllArith and Perturb. MAWPS (A Math Word Problem Solving Repository)
Koncel-Kedziorski et al. (2016) is a curated dataset (with deliberate template
overlap control) that comprises all proposed datasets till that date. A single
equation subset of MAWPS (AsDIV-A) Miao et al. (2020) has been studied , for
diagnostic analysis of solvers. Similarly, the critique offered by Patel et
al. (2021) was demonstrated using their newly proposed dataset SVAMP. In
SVAMP, minutely perturbed word problems from the popular dataset AsDIV-A. This
particular subset is used to demonstrate that, while high values of accuracy
can be obtained on AsDIV-A easily, SVAMP poses a formidable challenge to most
solvers, as it captures nuances in the relationship between similar language
formation and dissimilar equations. All aforementioned datasets incorporate an
annotation of both the equation and the answer. Given the subset-superset
relationships between some of these datasets, empirical usage of these
datasets would need to ensure careful sampling to creating subsets for
training, testing and cross-validation.

### Large Datasets

Dolphin18k Huang et al. (2016) is an early proprietary dataset that was
evaluated primarily with the statistical solvers. AQuA-RAT Ling et al. (2017)
introduced the first large crowd-sourced dataset for word problems with
rationales or explanations. This makes the setting quite different from the
aforementioned datasets, not only with respect to size, but also in the wide
variety of domain areas (spanning physics, algebra, geometry, probability
etc). Another point of difference is that the annotation involves the entire
textual explanation, rather than just the equations. MathQA Amini et al.
(2019) critically analysed AQuA-RAT and selected the core subset and annotated
it with a predicate list, to widen the remit of its usage. Once again,
researchers must be mindful of the fact that MathQA is a subset of AQuA-RAT.
GSM8k Cobbe et al. (2021) is a recent single-equation dataset, that is the
large scale version of AsDIV-A Miao et al. (2020). Math23K is a popular
Chinese dataset for single equation math word problem solving. A recent
successor is Ape210k Liang et al. (2021).

## Evaluation Measures

The most popular metric is answer accuracy, which evaluates the predicted
equation and checks whether it is the same as the labelled one. The other
metric is equation accuracy, which predominantly does string matching and
assesses the match between the produced equation and the equation from the
annotation label.

## Performance of Deep Models

In this section, we describe the performance of neural solvers towards
providing the reader with a high-level view of the comparative performance
across the several proposed models.

We have listed the performance of the deep models in Table 6, on two major
datasets - Math23K and MAWPS.

Some of these deep models report scores on other datasets as well. For
conciseness, we have chosen the most popular datasets for deep models. We see
that, in general, the models achieve around 70-80 percentage points on answer
accuracy. Shen et al. (2021) outperforms all other models on Math23k whereas
RPKHS Yu et al. (2021) is the best model for MAWPS till date. As mentioned
before, graph based models are both popular and effective. A note of caution
is that, as inferred from the discussion on datasets, (a) both Math23k and
MAWPS are single equation datasets and (b) though some lexical overlap has
been performed in the design of these two datasets, the semantic quality of
these datasets are quite similar. This aspect has also been experimented and
explored in Patel et al. (2021). Hence, though we present the best performing
algorithms in this table, more research is required to design a suitable
metric or a suitable dataset, such that one can conclusively compare these
various algorithms.

Model | Type | AQuA-RAT | MathQA | Source
---|---|---|---|---
AQuA | Seq2Seq | 36.4 | - | Ling et al. (2017)
Seq2Prog | Seq2Seq | 37.9 | 57.2 | Amini et al. (2019)
BERT-NPROP | Transformer | 37.0 | - | Piękos et al. (2021)
Graph-To-Tree | Graph-based | - | 69.65 | Li et al. (2020)
Table 5: Performance on Large Multi-Domain Datasets

Apart from these algebraic datasets, multi-domain datasets MathQA and AquA are
also of special interest. This is described in Table 5. The interesting
takeaway is that, the addition of BERT modelling to AQuA Piękos et al. (2021),
still performed slightly worse than the Seq2Prog Amini et al. (2019) model,
which is a derivative of the Seq2Seq paradigm.

Model Name | Type | Math23k | MAWPS | Source
---|---|---|---|---
GTS | Graph-based | 74.3 | - | Xie and Sun (2019)
SAU-SOLVER | Graph-based | 74.8 | - | Chiang and Chen (2019)
Group-att | Transformer | 69.5 | 76.1 | Li et al. (2019)
Graph2Tree | Graph-based | 77.4 | - | Li et al. (2020)
KA-S2T | Graph-based | 76.3 | - | Wu et al. (2020)
NS-Solver | Seq2Seq | 75.67 | - | Qin et al. (2020)
Graph-To-Tree | Graph-based | 78.8 | - | Li et al. (2020)
TSN-MD | Teacher Student | 77.4 | 84.4 | Zhang et al. (2020b)
Graph-Teacher | Graph & Teacher | 79.1 | 84.2 | Liang and Zhang (2021)
NumS2T | Graph-based | 78.1 | - | Wu et al. (2020)
Multi-E/D | Graph-based | 78.4 | - | Shen and Jin (2020)
EPT | Transformer | - | 84.5 | Kim et al. (2020)
Seq2DAG | Graph-based | 77.1 | - | Cao et al. (2021)
EEH-D2T | Graph-based | 78.5 | 84.8 | Wu et al. (2021a)
Generate and Rank | Graph-based | 85.4 | 84.0 | Shen et al. (2021)
HMS | Graph-based | 76.1 | 80.3 | Lin et al. (2021)
RPKHS | Graph-based | 83.9 | 89.8 | Yu et al. (2021)
CL | Contrastive Learning | 83.2 | - | Li et al. (2021b)
GTS+RODA | Graph-based | 77.9 | - | Liu et al. (2022)
Table 6: Answer Accuracy of Deep Models

## Analysis of Deep Models

In this section of the paper, we analyze the pros and cons of applying deep
learning techniques to solve word problems automatically. At the outset, two
layers of understanding are imperative: (i) linguistic structures that
describe a situation or a sequence of events and (ii) mathematical structures
that govern these language descriptions. Though deep learning models have
rapidly scaled and demonstrated commendable results for capturing these two
characteristics, a closer look reveals much potential for further exploration.
The predominant modus-operandus is to create a deep model that converts the
input natural language to the underlying equation. In some cases, the input is
converted into a set of predicates Amini et al. (2019) or explanations Ling et
al. (2017).

### What Shortcuts are being Learned?

Shortcut Learning Geirhos et al. (2020) is a recently well-studied phenomenon
of deep neural networks. It describes how deep learning models learn patterns
in a shallow way and fall prey to questionable generalizations across datasets
(an example is an image being classified as sheep if there was grass alone;
due to peculiarities in the dataset).This is a function of the low-level input
we provide to such models (pixels, word embeddings etc.). In the context of
word problems, Patel et al. (2021) exposed how removing the question and
simply passing the situational context, leads to the correct equation being
predicted. This suggests two things, issues with model design as well as
issues with dataset design. The datasets have high equation template overlap,
as well as text overlap. Word problem solving is a hard because two otherwise
identical word problems, with a small word change (say changing the word give
to take), would completely change the equation. Hence high lexical similarity
does not translate to corresponding similarity in the mathematical realm Patel
et al. (2021); Sundaram et al. (2020), and attention to key aspects within the
text is critical.

### Is Language or Math being Learned?

Problem | Solved?
---|---
John has 5 apples. Mary has 2 apples more than John. How many apples does Mary have? | Yes
John has 5 apples. Mary has 2 apples more than John. Who has less apples? | No
What should be added to two to make it five? | No
Table 7: Behaviour of Baseline BERT Model

The question that looms large is whether adequate mapping of language to math
has been modelled, whether linguistic modelling has been unfavourably
highlighted or that the mathematical aspects have been captured succinctly. We
observe that there are opportunities to refine the modelling of both language
and math aspects of word problems. Apart from the perturbations experiment
done by SVAMP Patel et al. (2021), which exposes that the mapping between
linguistic and mathematical structures is not captured, we suggest two more
experimental analysis frameworks that illustrate deficiencies in linguistic
and mathematical modelling. The first one involves imposing a question
answering task on top of the word problem as a probing test. For example, a
baseline BERT model that converts from input language to equation (Table 7),
trained on MAWPS, can solve a simple word problem such as "John has 5 apples.
Mary has 2 apples more than John. How many apples does Mary have?", but cannot
answer the following allied question "John has 5 apples. Mary has 2 apples
more than John. Who has less apples?". One reason is of course, dataset
design. The governing equation for this problem is "X = 5-2". However, the
text version of this, "What should be added to two to make it five?", cannot
be solved by the baseline model. Similarly, many solvers wrongly output
equations such as "X = 2 - 5" Patel et al. (2021), which suggests mathematical
modelling of subtraction of whole numbers could potentially be improved by
simply embedding more basic mathematical aspects. Hence, we observe, that deep
translation models neither model language, nor the math sufficiently.

### Is Accuracy Enough?

As suggested by the discussion above, a natural line of investigation is to
examine the evaluation measures, and perhaps the error measures for the deep
models, in order to bring about a closer coupling between syntax and
semantics. High accuracy of the models to predicting the answer or the
equation suggests a shallow mapping between the text and the mathematical
symbols. This is analogous to the famously observed McNamara
fallacy222https://en.wikipedia.org/wiki/McNamara_fallacy, which cautions
against the overuse of a single metric to evaluate a complex problem. One
direction of exploration is data augmentation with a single word problem
annotated with multiple equivalent equations. Metrics that measure the
soundness of the equations generated, the robustness of the model to simple
perturbations (perhaps achieved using a denoising autoencoder) and the ability
of the model to discern important entities in a word problem (perhaps using an
attention analysis based metric), are the need of the future. An endeavour has
been done by Kumar et al. (2021), where adversarial examples have been
generated and utilised to evaluate SOTA models.

### Are the Trained Models Accessible?

Most of the SOTA systems come with their own, well-documented repositories.
Though an aggregated toolkit Lan et al. (2021) (open-source MIT License) is
available, running saved models in inference mode, to probe the quality of the
datasets, proved to be a hard task, with varying missing hyper-parameters or
missing saved models. This, however, interestingly suggests that API’s that
can take a single word problem as input and computes the output, would be
highly useful for application designers. This has been done in the earlier
systems such as Roy and Roth (2018) and Wolfram (2015).

## Analysis of Benchmark Datasets

In this section of the paper, we explore the various dimensions of the popular
datasets (Table 4) with a critical and constructive perspective.

### Low Resource Setting

Compared to usual text related tasks, the available datasets are quite small
in size. They also suffer from a large lexical overlap Amini et al. (2019).
This taxes algorithms, that now have to generalise from an effectively small
dataset. The fact that the field of word problem solving is niche, where we
cannot simply lift text from generic sources like Wikipedia, is one of the
primary reasons why these datasets are small. Language precision is required,
while maintaining mathematical sense. Hence, language generation is also a
hard task.

### Annotation Cost

The datasets currently have little to no annotation costs involved as they are
usually scraped from homework websites. There are some exceptions that involve
crowd-sourcing Ling et al. (2017) or intermediate representations apart from
equations Amini et al. (2019).

### Template Overlap

Many studies Zhang et al. (2020a) have demonstrated that there is a high
lexical and mathematical overlap between the word problems in popular
datasets. While lexical overlap is desirable in a principled fashion, as
demonstrated by Patel et al. (2021), it often limits the diversity and thus
utility of the datasets. Consequently, many strategies have been adopted to
mitigate such issues. Early attempts include controlling linguistic and
equation template overlap (Koncel-Kedziorski et al. (2016), Miao et al.
(2020)). Later ideas revolve around controlled design and quality control of
crowd-sourcing Amini et al. (2019).

## Road Ahead

In this section, we describe exciting frontiers of research for word problem
solving algorithms.

### Semantic Parsing

As rightly suggested by Zhang et al. (2020a), the closest natural language
task for word problem solving is that of semantic parsing, and not translation
as most of the deep learning models have modelled. The mapping between
extremely long chunks of text to short equation sentences has the advantage of
generalising on the decoder side, but equally has the danger of overloading
many involved semantics into a simplistic equation model. To illustrate, an
equation may be derived after applying a sequence of steps that is lost in a
simple translation process. A lot of efforts have already been employed in
adding such nuances in the modelling. One way is to model the input
intelligently (for e.g., Liang et al. (2021)) Here, sophisticated embeddings
are learned from BERT based models, using the word problem text as a training
bed. The intermediate representations include simple predicates Roy and Roth
(2018), while others involve a programmatic description (Ling et al. (2017),
Amini et al. (2019)). Yet another way is to include semantic information in
the form of graphs as shown in (Huang et al. (2018), Chiang and Chen (2019),
Qin et al. (2020), Li et al. (2020), etc.)).

### Informed Dataset Design

As most datasets are sourced from websites, there is bound to be repetition.
Efforts invested in modelling things such as the following could help aiding
word problem research: (a) different versions of the same problem, (b)
different equivalent equation types, (c) semantics of the language and the
math. A step in this direction has been explored by Patel et al. (2021), which
provides a challenge dataset for evaluating word problems, and Kumar et al.
(2021) where adversarial examples are automatically generated.

#### Dataset Augmentation

A natural extension of dataset design, is dataset augmentation. Augmentation
is a natural choice when we have datasets that are small and focused on a
single domain. Then, linguistic and mathematical augmentation can be automated
by domain experts. While template overlap is a concern in dataset design, it
can be leveraged in contrastive designs as in Sundaram et al. (2020); Li et
al. (2021b). A principled approach of reversing operators and building
equivalent expression trees for augmentation has been explored here Liu et al.
(2022).

#### Few Shot Learning

This is useful if we have a large number of non-annotated word problems or if
we can come up with complex annotations (that capture semantics) for a small
set of word problems. In this way few shot learning can generalise from few
annotated examples.

### Knowledge Aware Models

We propose that word problem solving is more involved than even semantic
parsing. From an intuitive space, we learn language from examples and
interactions but we need to be explicitly trained in math to solve word
problems Marshall (1996). This suggests we need to include mathematical models
into our deep learning models to build generalisability and robustness. As
mentioned before, a common approach is to include domain knowledge as a graph
Chiang and Chen (2019); Wu et al. (2020); Qin et al. (2020, 2021).

## Conclusion

In this paper, we surveyed the existing math word problem solvers, with a
focus on deep learning models. Deep models are predominantly modeled as
encoder-decoder models, with input as text and decoder output as equations. We
listed several interesting formulations of this paradigm - namely Seq2Seq
models, graph-based models, transformer-based models, contrastive models and
teacher-student models. In general, graph based models tend to capture complex
structural elements that can benefit both linguistic and mathematical aspects.
We then explored in detail the various datasets in use. Subsequently, we
analysed the various approaches of modelling word problem solving, followed by
the characteristics of the popular datasets. We saw an overwhelming trend that
paying heed to the mathematical modelling and tying to the linguistic aspects
reaped rich dividends. We concluded that the brittleness of the SOTA models
was due to: (a) modelling decisions, and (b) dataset design. This is intended
as a comprehensive survey, but the authors acknowledge that there may be
methods that have escaped their attention. We also caution that the analysis
provided could be subjective and opinionated, and there could be legitimate
disagreements with the perspectives put forward. Finally, we mentioned few
avenues of further exploration such as the use of semantically rich models,
informed dataset design and incorporation of domain knowledge.

## References

  * Amini et al. (2019) Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H. (2019).  MathQA: Towards interpretable math word problem solving with operation-based formalisms.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357–2367, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Bobrow (1964) Bobrow, D. G. (1964).  A question-answering system for high school algebra word problems.  In Proceedings of the October 27-29, 1964, fall joint computer conference, part I, pages 591–614. ACM.
  * Cao et al. (2021) Cao, Y., Hong, F., Li, H., and Luo, P. (2021).  A bottom-up dag structure extraction model for math word problems.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(1):39–46.
  * Chen et al. (2021) Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E., and Lin, L. (2021).  GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 513–523, Online. Association for Computational Linguistics.
  * Chiang and Chen (2019) Chiang, T.-R. and Chen, Y.-N. (2019).  Semantically-aligned equation generation for solving and reasoning math word problems.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2656–2668, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Cho et al. (2014) Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014).  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Cobbe et al. (2021) Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021).  Training verifiers to solve math word problems.  CoRR, abs/2110.14168.
  * Dellarosa (1986) Dellarosa, D. (1986).  A computer simulation of children’s arithmetic word-problem solving.  Behavior Research Methods, Instruments, & Computers, 18(2):147–154.
  * Devlin et al. (2018) Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018).  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805.
  * Dries et al. (2017) Dries, A., Kimmig, A., Davis, J., Belle, V., and De Raedt, L. (2017).  Solving probability problems in natural language.  Proceedings Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 3981–3987.
  * Faldu et al. (2021) Faldu, K., Sheth, A. P., Kikani, P., Gaur, M., and Avasthi, A. (2021).  Towards tractable mathematical reasoning: Challenges, strategies, and opportunities for solving math word problems.  CoRR, abs/2111.05364.
  * Feng et al. (2021) Feng, W., Liu, B., Xu, D., Zheng, Q., and Xu, Y. (2021).  GraphMR: Graph neural network for mathematical reasoning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3395–3404, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Fletcher (1985) Fletcher, C. R. (1985).  Understanding and solving arithmetic word problems: A computer simulation.  Behavior Research Methods, Instruments, & Computers, 17(5):565–571.
  * Geirhos et al. (2020) Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020).  Shortcut learning in deep neural networks.  Nature Machine Intelligence, 2(11):665–673.
  * Griffith and Kalita (2020) Griffith, K. and Kalita, J. (2020).  Solving arithmetic word problems using transformer and pre-processing of problem texts.  In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 76–84, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
  * Hochreiter and Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J. (1997).  Long short-term memory.  Neural Comput., 9(8):1735–1780.
  * Hong et al. (2021) Hong, Y., Li, Q., Ciao, D., Huang, S., and Zhu, S.-C. (2021).  Learning by fixing: Solving math word problems with weak supervision.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(6):4959–4967.
  * Hosseini et al. (2014) Hosseini, M. J., Hajishirzi, H., Etzioni, O., and Kushman, N. (2014).  Learning to solve arithmetic word problems with verb categorization.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523–533.
  * Huang et al. (2017) Huang, D., Shi, S., Lin, C.-Y., and Yin, J. (2017).  Learning fine-grained expressions to solve math word problems.  Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 805–814.
  * Huang et al. (2016) Huang, D., Shi, S., Lin, C.-Y., Yin, J., and Ma, W.-Y. (2016).  How well do computers solve math word problems? large-scale dataset construction and evaluation.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887–896, Berlin, Germany. Association for Computational Linguistics.
  * Huang et al. (2018) Huang, D., Yao, J.-G., Lin, C.-Y., Zhou, Q., and Yin, J. (2018).  Using intermediate representations to solve math word problems.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 419–428, Melbourne, Australia. Association for Computational Linguistics.
  * Kim et al. (2020) Kim, B., Ki, K. S., Lee, D., and Gweon, G. (2020).  Point to the Expression: Solving Algebraic Word Problems using the Expression-Pointer Transformer Model.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3768–3779, Online. Association for Computational Linguistics.
  * Koch et al. (2015) Koch, G., Zemel, R., and Salakhutdinov, R. (2015).  Siamese neural networks for one-shot image recognition.  In ICML Deep Learning Workshop, volume 2.
  * Koncel-Kedziorski et al. (2015) Koncel-Kedziorski, R., Hajishirzi, H., Sabharwal, A., Etzioni, O., and Ang, S. D. (2015).  Parsing algebraic word problems into equations.  Transactions of the Association for Computational Linguistics, 3:585–597.
  * Koncel-Kedziorski et al. (2016) Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. (2016).  Mawps: A math word problem repository.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157.
  * Kumar et al. (2021) Kumar, V., Maheshwary, R., and Pudi, V. (2021).  Adversarial examples for evaluating math word problem solvers.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2705–2712, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kushman et al. (2014) Kushman, N., Artzi, Y., Zettlemoyer, L., and Barzilay, R. (2014).  Learning to automatically solve algebra word problems.  ACL (1), pages 271–281.
  * Lan et al. (2021) Lan, Y., Wang, L., Zhang, Q., Lan, Y., Dai, B. T., Wang, Y., Zhang, D., and Lim, E.-P. (2021).  Mwptoolkit: An open-source framework for deep learning-based math word problem solvers.  arXiv preprint arXiv:2109.00799.
  * Le and Mikolov (2014) Le, Q. and Mikolov, T. (2014).  Distributed representations of sentences and documents.  In International conference on machine learning, pages 1188–1196.
  * Le-Khac et al. (2020) Le-Khac, P. H., Healy, G., and Smeaton, A. F. (2020).  Contrastive representation learning: A framework and review.  IEEE Access, 8:193907–193934.
  * Li et al. (2019) Li, J., Wang, L., Zhang, J., Wang, Y., Dai, B. T., and Zhang, D. (2019).  Modeling intra-relation in math word problems with different functional multi-head attentions.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6162–6167, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2021a) Li, L., Lin, Y., Ren, S., Li, P., Zhou, J., and Sun, X. (2021a).  Dynamic knowledge distillation for pre-trained language models.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2020) Li, S., Wu, L., Feng, S., Xu, F., Xu, F., and Zhong, S. (2020).  Graph-to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2841–2852, Online. Association for Computational Linguistics.
  * Li et al. (2021b) Li, Z., Zhang, W., Yan, C., Zhou, Q., Li, C., Liu, H., and Cao, Y. (2021b).  Seeking patterns, not just memorizing procedures: Contrastive learning for solving math word problems.  CoRR, abs/2110.08464.
  * Liang et al. (2021) Liang, Z., Zhang, J., Shao, J., and Zhang, X. (2021).  MWP-BERT: A strong baseline for math word problems.  CoRR, abs/2107.13435.
  * Liang and Zhang (2021) Liang, Z. and Zhang, X. (2021).  Solving math word problems with teacher supervision.  In Zhou, Z.-H., editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3522–3528. International Joint Conferences on Artificial Intelligence Organization.  Main Track.
  * Lin et al. (2021) Lin, X., Huang, Z., Zhao, H., Chen, E., Liu, Q., Wang, H., and Wang, S. (2021).  Hms: A hierarchical solver with dependency-enhanced understanding for math word problem.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4232–4240.
  * Ling et al. (2017) Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. (2017).  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  arXiv preprint arXiv:1705.04146.
  * Liu et al. (2022) Liu, Q., Guan, W., Li, S., Cheng, F., Kawahara, D., and Kurohashi, S. (2022).  Roda: Reverse operation based data augmentation for solving math word problems.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:1–11.
  * Liu et al. (2019) Liu, Q., Guan, W., Li, S., and Kawahara, D. (2019).  Tree-structured decoding for solving math word problems.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2370–2379, Hong Kong, China. Association for Computational Linguistics.
  * Marshall (1996) Marshall, S. P. (1996).  Schemas in Problem Solving.  Cambridge University Press.
  * Miao et al. (2020) Miao, S.-y., Liang, C.-C., and Su, K.-Y. (2020).  A diverse corpus for evaluating and developing English math word problem solvers.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online. Association for Computational Linguistics.
  * Mikolov et al. (2013) Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013).  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * Mitra and Baral (2016) Mitra, A. and Baral, C. (2016).  Learning to use formulas to solve simple arithmetic problems.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2144–2153, Berlin, Germany. Association for Computational Linguistics.
  * Mukherjee and Garain (2008) Mukherjee, A. and Garain, U. (2008).  A review of methods for automatic understanding of natural language mathematical problems.  Artificial Intelligence Review, 29(2):93–122.
  * Patel et al. (2021) Patel, A., Bhattamishra, S., and Goyal, N. (2021).  Are NLP models really able to solve simple math word problems?  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C. (2014).  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  * Peters et al. (2018) Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018).  Deep contextualized word representations.  In Proc. of NAACL.
  * Piękos et al. (2021) Piękos, P., Malinowski, M., and Michalewski, H. (2021).  Measuring and improving BERT’s mathematical abilities by predicting the order of reasoning.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 383–394, Online. Association for Computational Linguistics.
  * Qin et al. (2021) Qin, J., Liang, X., Hong, Y., Tang, J., and Lin, L. (2021).  Neural-symbolic solver for math word problems with auxiliary tasks.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5870–5881, Online. Association for Computational Linguistics.
  * Qin et al. (2020) Qin, J., Lin, L., Liang, X., Zhang, R., and Lin, L. (2020).  Semantically-aligned universal tree-structured solver for math word problems.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3780–3789, Online. Association for Computational Linguistics.
  * Roy and Roth (2015) Roy, S. and Roth, D. (2015).  Solving general arithmetic word problems.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal. Association for Computational Linguistics.
  * Roy and Roth (2017) Roy, S. and Roth, D. (2017).  Unit dependency graph and its application to arithmetic word problem solving.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 3082–3088. AAAI Press.
  * Roy and Roth (2018) Roy, S. and Roth, D. (2018).  Mapping to declarative knowledge for word problem solving.  Transactions of the Association of Computational Linguistics, 6:159–172.
  * Roy et al. (2015) Roy, S., Vieira, T., and Roth, D. (2015).  Reasoning about quantities in natural language.  Transactions of the Association for Computational Linguistics, 3:1–13.
  * Seo et al. (2015) Seo, M., Hajishirzi, H., Farhadi, A., Etzioni, O., and Malcolm, C. (2015).  Solving geometry problems: Combining text and diagram interpretation.  Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1466–1476.
  * Shen et al. (2021) Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., and Liu, Q. (2021).  Generate & rank: A multi-task framework for math word problems.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2269–2279, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Shen and Jin (2020) Shen, Y. and Jin, C. (2020).  Solving math word problems with multi-encoders and multi-decoders.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 2924–2934, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Shi et al. (2015) Shi, S., Wang, Y., Lin, C., Liu, X., and Rui, Y. (2015).  Automatically solving number word problems by semantic parsing and reasoning.  Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1132–1142.
  * Sundaram and Abraham (2019) Sundaram, S. S. and Abraham, S. S. (2019).  Semantic representation for age word problems with schemas.  New Generation Computing, 37(4):429–452.
  * Sundaram et al. (2020) Sundaram, S. S., P, D., and Abraham, S. S. (2020).  Distributed representations for arithmetic word problems.  Thirty-Fourth AAAI Conference on Artificial Intelligence.
  * Suster et al. (2021) Suster, S., Fivez, P., Totis, P., Kimmig, A., Davis, J., de Raedt, L., and Daelemans, W. (2021).  Mapping probability word problems to executable representations.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3627–3640, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sutskever et al. (2014) Sutskever, I., Vinyals, O., and Le, Q. V. (2014).  Sequence to sequence learning with neural networks.  In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 3104–3112, Cambridge, MA, USA. MIT Press.
  * Tsai et al. (2021) Tsai, S.-h., Liang, C.-C., Wang, H.-M., and Su, K.-Y. (2021).  Sequence to general tree: Knowledge-guided geometry word problem solving.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 964–972, Online. Association for Computational Linguistics.
  * Upadhyay and Chang (2017) Upadhyay, S. and Chang, M.-W. (2017).  Annotating derivations: A new evaluation strategy and dataset for algebra word problems.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 494–504, Valencia, Spain. Association for Computational Linguistics.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017).  Attention is all you need.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  * Wang et al. (2018) Wang, L., Zhang, D., Gao, L., Song, J., Guo, L., and Shen, H. T. (2018).  Mathdqn: Solving arithmetic word problems via deep reinforcement learning.  Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
  * Wang et al. (2017) Wang, Y., Liu, X., and Shi, S. (2017).  Deep neural solver for math word problems.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845–854.
  * Wolfram (2015) Wolfram, S. (2015).  Wolfram|alpha.  On the WWW. URL http://www. wolframalpha. com.
  * Wu et al. (2020) Wu, Q., Zhang, Q., Fu, J., and Huang, X. (2020).  A knowledge-aware sequence-to-tree network for math word problem solving.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7137–7146, Online. Association for Computational Linguistics.
  * Wu et al. (2021a) Wu, Q., Zhang, Q., and Wei, Z. (2021a).  An edge-enhanced hierarchical graph-to-tree network for math word problem solving.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1473–1482, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wu et al. (2021b) Wu, Q., Zhang, Q., Wei, Z., and Huang, X. (2021b).  Math word problem solving with explicit numerical values.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5859–5869, Online. Association for Computational Linguistics.
  * Wu et al. (2021c) Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Yu, P. S. (2021c).  A comprehensive survey on graph neural networks.  IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24.
  * Xia et al. (2019) Xia, M., Huang, G., Liu, L., and Shi, S. (2019).  Graph based translation memory for neural machine translation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7297–7304.
  * Xie and Sun (2019) Xie, Z. and Sun, S. (2019).  A goal-driven tree-structured neural model for math word problems.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5299–5305. International Joint Conferences on Artificial Intelligence Organization.
  * Yu et al. (2021) Yu, W., Wen, Y., Zheng, F., and Xiao, N. (2021).  Improving math word problems with pre-trained knowledge and hierarchical reasoning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3384–3394, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zaporojets et al. (2021) Zaporojets, K., Bekoulis, G., Deleu, J., Demeester, T., and Develder, C. (2021).  Solving arithmetic word problems by scoring equations with recursive neural networks.  Expert Systems with Applications, 174:114704.
  * Zhang et al. (2020a) Zhang, D., Wang, L., Zhang, L., Dai, B. T., and Shen, H. T. (2020a).  The gap of semantic parsing: A survey on automatic math word problem solvers.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(9):2287–2305.
  * Zhang et al. (2020b) Zhang, J., Lee, R. K.-W., Lim, E.-P., Qin, W., Wang, L., Shao, J., and Sun, Q. (2020b).  Teacher-student networks with multiple decoders for solving math word problem.  In Bessiere, C., editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4011–4017. International Joint Conferences on Artificial Intelligence Organization.  Main track.
  * Zhang et al. (2020c) Zhang, J., Wang, L., Lee, R. K.-W., Bin, Y., Wang, Y., Shao, J., and Lim, E.-P. (2020c).  Graph-to-tree learning for solving math word problems.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3928–3937, Online. Association for Computational Linguistics.
  * Zhou et al. (2015) Zhou, L., Dai, S., and Chen, L. (2015).  Learn to solve algebra word problems using quadratic programming.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 817–822, Lisbon, Portugal. Association for Computational Linguistics.


# Bonding states underpinning structural transitions in IrTe2 observed with
micro-ARPES

C. W. Nicholson<EMAIL_ADDRESS>University of Fribourg and
Fribourg Centre for Nanomaterials, Chemin du Musée 3, CH-1700 Fribourg,
Switzerland  Fritz-Haber-Institut der Max-Planck-Gesellscahft, Faradayweg 4-6,
D-14195 Berlin, Germany  M. D. Watson  Diamond Light Source Ltd, Harwell
Science and Innovation Campus, Didcot, Oxfordshire, OX110DE, U.K.  A.
Pulkkinen  University of Fribourg and Fribourg Centre for Nanomaterials,
Chemin du Musée 3, CH-1700 Fribourg, Switzerland  New Technologies-Research
Center, University of West Bohemia, Plzeň 301 00, Czech Republic  M. Rumo
University of Fribourg and Fribourg Centre for Nanomaterials, Chemin du Musée
3, CH-1700 Fribourg, Switzerland  G. Kremer  University of Fribourg and
Fribourg Centre for Nanomaterials, Chemin du Musée 3, CH-1700 Fribourg,
Switzerland  Institut Jean Lamour, UMR 7198, CNRS-Université de Lorraine,
Campus ARTEM, 2 allée André Guinier, BP 50840, 54011 Nancy, France  K. Y. Ma
Max Planck Institute for Chemical Physics of Solids, Nöthnitzer Str. 40, 01187
Dresden, Germany  F. O. von Rohr  Department of Quantum Matter Physics,
University of Geneva, 24 Quai Ernest-Ansermet, CH-1211 Geneva, Switzerland  C.
Cacho  Diamond Light Source Ltd, Harwell Science and Innovation Campus,
Didcot, Oxfordshire, OX110DE, U.K.  C. Monney<EMAIL_ADDRESS>University of Fribourg and Fribourg Centre for Nanomaterials, Chemin du Musée
3, CH-1700 Fribourg, Switzerland

###### Abstract

Competing interactions in low-dimensional materials can produce nearly
degenerate electronic and structural phases. We investigate the staircase of
structural phase transitions in layered IrTe2 for which a number of potential
transition mechanisms have been postulated. The spatial coexistence of
multiple phases on the micron scale has prevented a detailed analysis of the
electronic structure. By exploiting micro-ARPES obtained with synchrotron
radiation we extract the electronic structure of the multiple structural
phases in IrTe2 in order to address the mechanism underlying the phase
transitions. We find direct evidence of lowered energy states that appear in
the low-temperature phases, states previously predicted by ab initio
calculations and extended here. Our results validate a proposed scenario of
bonding and anti-bonding states as the driver of the phase transitions.

The diversity of electronic and magnetic phenomena occurring in quasi-2D
transition metal dichalcogenides (TMDs) can largely be traced back to the
presence of a partially filled $d$ valence band [1]. The relatively narrow
$d$-bands, with widths on the same order as the Coulomb interaction, produce
ideal conditions for correlated phenomena such as superconductivity [2, 3, 4],
metal-insulator transitions [5], and charge density waves beyond the Peierls
paradigm [6, 7, 8].

Here we focus on the unusual structural transitions in the well-known TMD
IrTe2 involving Ir atoms formally in a 5$d^{6}$ electronic configuration.
Already at 280 K the crystal structure of IrTe2 transitions from a trigonal
1$T$ phase ($P$-3$m$1) to a monoclinic ($P$-1) 5x1x5 superstructure in a
first-order fashion [9]. In the bulk this is followed by a second transition
at 180 K into an 8x1x8 phase [10]. Low-temperature (LT) measurements have
revealed a 6x1x6 ground state [11, 12], which can also be stabilized in Se
doped [13] and uniaxially strained crystals [14]. Although the resistivity is
reduced in the LT phases, IrTe2 remains metallic at all temperatures. Each of
the phases below 280 K exhibits a shortening of some of the Ir-Ir distances
within the unit cell into effective dimers [15]. The number of these Ir dimers
within a unit cell is characteristic to each of the LT phases [10, 16] (see
Fig. 1a and b). Furthermore, in monolayer samples a novel insulating 2x1
behavior has been observed [17]. Determining the mechanism of the structural
transitions is therefore an important task.

Figure 1: a) Side view of the crystal structure of 1$T$-IrTe2 in the high
temperature (HT) 1x1 phase, and b) in the various LT phases (top row 5x1,
middle row 8x1, bottom row 6x1). c) Schematic bonding energy level diagram in
the HT and LT phases, highlighting the formation of bonding states due to the
dimer bonds. d) Overview image of the sample obtained with micro-ARPES with a
pixel size of 5$\mu$m. Blue color corresponds to the background signal from
the epoxy used to glue the sample.

Initially proposed weak-coupling scenarios based on Fermi nesting [9] appear
to be incompatible with the dramatic 20 percent reduction in Ir bond lengths
later observed in the LT phases [15]. Alternative scenarios include a
$J_{\mathrm{eff}}=1/2$ spin-orbit Mott state [10], similar to that observed in
Sr2IrO4 [18], proposed to arise due to a possible Ir 5$d^{5}$ (Ir4+)
configuration in the LT phases. A number of works have put forward ideas based
on a molecular-type picture involving either depolymerisation between the
layers [13], or an effective Ir dimerization that produces bonding and anti-
bonding 5$d$ states [15, 19] (see Fig. 1c) as part of a more complex multi-
centre bond [20]. Further support for this scenario was obtained by recent
high-pressure diffraction measurements, where an observed modification of the
Ir-Te angle provides conditions compatible with a ring-shaped bond across
multiple sites [21]. In both the LT and high-pressure calculations, the
predicted bonding states appear distinctly split-off to higher binding
energies from the density of states (DoS) of the remaining undimerized Ir
atoms. Direct experimental confirmation of such states has not been obtained.

Validation or falsification of this scenario therefore requires clear
electronic structure data e.g. from angle-resolved photoemission spectroscopy
(ARPES). Numerous studies have provided insights into the electronic structure
of IrTe2 [22, 23, 24, 10, 25, 26, 16, 14, 27, 28, 29]. However, extraction of
the intrinsic electronic structure is complicated by the spatial coexistence
of a ladder of nearly degenerate phases with periodicity $3n+2$
($n=1,2,3,...$) at the surface with 5x1, 8x1 and 6x1 the dominant phases in
observations [11]. Typical domain sizes ($\sim 10~{}\mu$m [27]) are beyond the
reach of most ARPES setups, with the result that only the 6x1 phase has been
unambiguously isolated with strain-ARPES [14]. An outstanding issue is
therefore to address the electronic structure in the intrinsic LT structures
without extraneous influence.

Figure 2: a) XPS spectra of the 4$f_{7/2}$ core level obtained from specific
domains with a single phase (the individual positions are indicated by markers
in (c)). b) Exemplary fit of the Ir 4$f_{7/2}$ core-level in the 5x1 phase.
The fit function comprises of the sum of two Voigt peaks on top of a linear
background and a constant offset. Individual component curves and the
background are shown. The spectral weight under each peak is extracted and
used to plot the ratio of peaks in c). c) XPS map of the region highlighted
(black rectangle) in Fig. 1d obtained at 160 K and 130 eV photon energy. The
color scale encodes the proportion ($R$) of Ird relative to the total
population of Ir atoms, $R$ = Ird/(Irm+Ir${}_{d})$. The observed $R$ values
for the three spectra shown in (a) are: 0.64 (star), 0.50 (circle) and 0.41
(triangle).

We overcome this challenge by applying micro-ARPES and micro-XPS (X-ray
Photoemission Spectroscopy) with synchrotron radiation to IrTe2 in order to
isolate the electronic structure of the surface 5x1, 8x1 and 6x1 phases.
Individual phases are identified by their characteristic Ir 4$f$ core level
spectra, allowing us to determine the spatial extent of domains on the
micrometer scale, and to extract the intrinsic band structure of individual
phases with ARPES. We observe split-off states in the DoS of the LT phases,
clearly supporting a bonding mechanism for the phase transition.

Single crystals of IrTe2 were grown using the self-ﬂux method [30, 31] and
were characterized by magnetic susceptibility and resistivity measurements
[16]. Micro-ARPES and micro-XPS measurements were carried out on the nano-
branch of the I05 beamline at Diamond Light Source with a hemispherical
analyzer in a base pressure of low 10-10 mbar. For these measurements we
selected beam sizes produced by a micro-focussing capillary in order to
achieve a spot size of around 6 $\mu$m (FWHM) with high photon flux on the
sample. This configuration was chosen to allow efficient probing of the
typical domain sizes [27]. All data were obtained with $p$-polarized light.
The ARPES measurements shown here were carried out between 70 – 90 eV, while
XPS was obtained at a photon energy of 130 eV due to the increased Ir 4$f$
cross-section. All crystals were cleaved in vacuum using a ceramic top-post.

The density functional theory DoS calculations were performed using the Vienna
ab-initio simulation package (VASP)[32, 33, 34, 35] within the projector
augmented wave method [36] and the generalized gradient approximation using
the Perdew-Burke-Ernzerhof functional [37]. The kinetic energy cutoff was set
to 400 eV and the Brillouin zone was sampled with the tetrahedron method [38]
and a $\Gamma$-centered $k$-point grid with spacing $0.08\>\mathrm{\AA^{-1}}$.
Spin-orbit interaction was included in all calculations.

A large scale micro-ARPES map of an IrTe2 sample obtained at 50 K with 5
$\mu$m step size is shown in Fig. 1d. The image is produced by integrating the
ARPES intensity over the full detector, and represented according to the color
scale shown. We focus on the area highlighted by the black rectangle in order
to perform more detailed mapping, as multiple phases are found within the scan
area. We initially carried out an XPS map in order to determine the
distribution of phases at a sample temperature of 160 K i.e. below the nominal
5x1 - 8x1 transition. Since the ratio of dimer (Ird) to monomer (Irm) Iridium
atoms is different in each LT phase, each phase has a characteristic ratio $R$
= Ird/(Irm+Ir${}_{d})$ of the integrated intensity under the XPS peak
corresponding to those two separate Ir environments [16] e.g. in the different
Ir 4$f_{7/2}$ peaks. Thus the ratio is $2/5$ (0.4) for the 5x1 phase, $4/8$
(0.5) for the 8x1 phase, and $4/6$ (0.67) in the 6x1 phase (see Fig. 1c).
Exemplary individual XPS spectra of the 4$f_{7/2}$ core-level for each LT
phases are presented in Fig. 2a, revealing the strong increase of the dimer
signal at 61.5 eV in the 8x1 and 6x1 phases, in comparison to the 5x1 phase.
The ratio is obtained from the XPS data by fitting with a two-peak Voigt
function on a linear background and a constant offset (see Fig. 2b). The
resulting spatial ratio plot of $R$ is shown in Fig. 2c. From the color scale
it is evident that all three phases coexist at this temperature, which is
somewhat surprising given that in the bulk the 5x1 and 8x1 phases do not
coexist [10]. Nevertheless, this is in good agreement with our previous
photoemission study indicating coexistence of different phases at low
temperature at the surface [16]. The XPS spatial map is dominated by regions
with $R$ values near 0.4 and 0.5, indicating dominantly 5x1 and 8x1 phases.
The gradual variation in the color scale near domain boundaries suggests a
partial mixing of phases within the selected spatial resolution.

The spatial distribution of phases obtained from the XPS data also allows us
to probe the individual LT phases with ARPES; the results obtained at 160 K
along the H-A-H direction and the $K-\Gamma$-K direction of the bulk Brillouin
zone of the HT phase are shown in Fig. 3. A global similarity can be observed
between all phases, including the 1x1 phase obtained at room temperature (Fig.
3a). Notable features in the 1x1 phase that are carried over to the LT phase
include the intense band dispersing from -2 eV at the zone boundary to the
Fermi level, and the topological surface state [39] with an energy of -1 eV at
the zone centre, which is shifted to higher binding energies in the LT phases
[16].

However, distinct differences are observed across the energy window in the LT
phases, particularly in the region close to the Fermi level. In the 5x1 phase
(Fig. 3b) mirror symmetry is broken (the corresponding low-symmetry space
group is $P$-1), as seen from the states off-centre around -0.3 eV at
$k_{HAH}=\pm 0.6$ $\mathrm{\AA}^{-1}$ (see arrows). In addition, in each of
the LT phases, the bottom of the bulk Dirac states, unoccupied in the 1x1
phase, are now shifted into the occupied states at the zone centre (see the
horizontal arrows in panels b,c,d). This feature is similar in the 5x1 and 8x1
(Fig. 3b,c) phases, but the hyperbolic dispersion only becomes evident in the
6x1 phase (Fig. 3d) as previously observed [14]. A clear difference between
the 5x1 and 8x1 phases is observed close to the zone centre. Between the Dirac
states and the intense band dispersing to -2 eV, only a single band is
observed dispersing to the Fermi level in the 5x1 phase. In contrast the 8x1
phase shows two states dispersing to the Fermi level, which may be a result of
the broken inversion symmetry of this phase that is expected to lift the spin
degeneracy [15]. For completeness, the 6x1 phase appears identical to that
obtained in strained samples [14]. In contrast to a previous study [10] we do
not find evidence for the appearance of a flat band at $\Gamma$ below 280 K
and instead find a dispersing band in the expected region along the high
symmetry direction $K\Gamma K$ (Fig. 3f,g,h). The corresponding ARPES data for
the 1x1 phase is shown in Fig. 3e. Since a non-dispersing state is expected
for the spin-orbit Mott state, we therefore conclude that a Mott phase in
IrTe2 is unlikely. This difference compared to the previous work may be due to
the specially chosen spatial resolution used in the current study, removing
any influence from a mixture of domains with different orientations or from
other phases.

As mentioned above, the energy of the topological surface state is shifted to
higher binding energies compared with the 1x1 phase. The value of the shift is
-0.37 eV (5x1), -0.34 eV (8x1), and -0.46 eV (6x1) for the three LT phases.
These values are slightly larger, but with the same trend, as previously
determined values employing larger beam sizes in ARPES measurements [16]. This
shift of the surface state implies a lowered energy configuration in the LT
states, which foreshadows the observation of bulk bonding states discussed in
more detail below.

Figure 3: Experimental band structure at the bulk A-point (90 eV photon
energy) along the high-symmetry direction $HAH$ of the HT Brillouin zone in
the a) 1x1, b) 5x1, c) 8x1, and d) 6x1 phase. Arrows mark the key features
described in the text. Experimental band structure at the bulk $\Gamma$-point
(70 eV photon energy) along the high-symmetry direction $K\Gamma K$ of the HT
Brillouin zone in the e) 1x1, f) 5x1, g) 8x1, and h) 6x1 phase. All spectra
are obtained at the points marked in Fig. 2, and at a temperature of 160 K.

Given the suggestive behavior of the surface states, we investigated the
split-off bonding DoS predicted by bulk DFT calculations in the region of -3
eV binding energy [15]. In order to better compare the momentum-resolved
experimental data with the predicted DoS, we have calculated the orbital-
resolved DoS within a small momentum range close to the zone centre
($\Gamma$-point) to avoid including the highly dispersive states at higher
momenta present at all temperatures. We use a pseudocubic axis system with the
$x,y$ axes pointing approximately along the Ir-Te bond directions, such that
the Ir $d_{xy}$ orbitals are directed along the dimer bonds. The 1x1 phase is
dominated by the Ir $d$-orbitals as can be seen in Fig. 4a. In the following,
we focus on the $d_{xy}$ orbital in the 5x1 phase, since the $d_{xy}$ orbitals
overlap directly in the formation of the Ir dimers, resulting in large binding
energy shifts, as previously addressed by Pascut $et~{}al.$ [15]. The 5x1
phase is chosen due its smaller unit cell compared with the 8x1, and the fact
that structural data for the intrinsic 6x1 phase is not available [13, 15,
21]. The three curves in Fig. 4b reflect the three inequivalent Ir sites in
the 5x1 phase: dimerized atoms, and two species of monomers depending on their
proximity to the dimers. A distinct shift of the dimer DoS to 0.5 eV lower
binding energy compared with the monomers is observed. We note here that a
similar behavior is observed in the $d_{yz}$ and $d_{xz}$ orbitals. These
split-off states are the fingerprints of bonding state formation as proposed
by previous works [15, 20]. Corresponding anti-bonding states are predicted
above the Fermi level at around 1.5 eV, see Ref. [20] and Fig. 4b. Our result
is similar to the previous calculations, although it displays a clearer effect
due to the selective momentum range applied. We note that similar split-off
states are expected also at other binding energies below the range currently
under investigation [20].

To compare directly with the calculated DoS, we extract an energy slice at the
zone centre ($\Gamma$-point) from ARPES measurements, presented in Fig. 4c.
The energy range is chosen to encompass the DoS states highlighted in Fig. 4b
which have already been shown to be of most relevance. In the 1x1 data (black
curve) the tail of the DoS extends to $-3$ eV, but no peaks are observed in
this region. In contrast, in the 5x1 phase a clear peak is evident at $-3.05$
eV, with a difference in energy of $-0.46$ eV compared with the lowest energy
peak in the 1x1 phase. This peak becomes even more pronounced in the 6x1
phase, with a double peak structure forming that mirrors that of the DoS
calculation. The ARPES data therefore provides direct evidence for states
split-off from the 1x1 DoS associated with the LT phases.

Figure 4: a) Calculated partial DoS in the 1x1 phase for the all atoms, for Te
atoms only, for Ir atoms only and for the $d$-orbitals of the Ir atoms only. A
pseudocubic axis system was used with the $x,y$ axes pointing approximately
along the Ir-Te bond directions, such that the Ir $d_{xy}$ orbitals are
directed along the dimer bonds. b) Calculated partial DoS in the 5x1 phase for
the $d_{xy}$ orbital for the different Ir environments. c) Experimental
spectra extracted at $k_{K\Gamma K}=0$ and normalized to the intensity at -3.5
eV in the 1x1, 5x1 and 6x1 phases. Data are obtained at 70 eV, corresponding
to the bulk $\Gamma$-point. d) Magnification of the region shown in c) for the
calculated partial DoS.

The observation of split-off states in the LT phases, at energies well
separated from the 1x1 DoS, is in excellent agreement with the prediction from
our momentum-selective calculations, and fully compatible with the previous
momentum-integrated calculations. The close correspondence of experiment and
theory therefore provides direct evidence for bonding state formation tied to
the structural transition, validating a scenario of strongly bonded atoms as
the driver of the structural phase transition [15, 20]. Indeed, such a
scenario provides a natural alternative to the formation of an electronic gap,
which is not observed in IrTe2 and where a metallic, albeit reduced, DoS at
the Fermi level is always observed in the LT phases. The large energetic
shifts of 0.5 eV in IrTe2 are comparable but larger than the typical
electronic energy gains in quasi-2D CDW materials [8].

The observed split-off states are clearly associated with bulk Ir 5$d$ states,
and are far-removed from the surface states in energy. This confirms that the
electronic structure differences observed here with ARPES are representative
of the bulk transition. We note that this situation may not hold in very thin
samples, where a number of additional phenomena are observed, including an
enhanced transition temperature for the 5x1 phase [40] and a coexistence of
structural 5x1 order and superconductivity [41]. These observations further
imply that the out-of-plane bonding is involved in the phase transition.
Indeed, the role of the Te-Te bond was already discussed in early works to
explain the anomalously small $c/a$ ratio [30, 42], and the results of Se-
doping [13]. In fact, a reduction of interlayer bond strength is an additional
consequence of the multi-centre bond formed in the LT phases [20], and is
supported by the reduced interlayer coupling observed in ARPES of the 6x1
phase [14]. This highlights that the picture of the dimer motif forming
bonding states provides a coherent description of the observed data for the
structural transitions in IrTe2 .

In summary, we used synchrotron micro-ARPES to separate the intrinsic
electronic structure of the micro-scale LT phases in IrTe2. The rich
structures unveiled display mirror symmetry breaking in the 5x1 phase, and
possible features of inversion symmetry breaking in the 8x1 phase. The
structure of the 6x1 phase appears to be equivalent to that stabilized under
uniaxial strain. Disentangling the various phases allowed us to resolve split-
off bonding states that appear in the LT phases at around -3 eV binding
energy, as previously predicted based on structural models. Our DoS
calculations confirms the appearance of these states in terms of $d_{xy}$,
$d_{yz}$, $d_{xz}$ and $d_{x^{2}-y^{2}}$ orbitals. Our results provide
experimental evidence for a mechanism based on short dimer bonds as the
energetic driver of the structural transitions in layered IrTe2.

##  I Acknowledgments

This project was supported by the Swiss National Science Foundation (SNSF)
Grant No. $P00P2\\_170597$. A.P. would like to thank the QM4ST project with
Reg. No. CZ.02.01.01$/$00$/$22_008$/$0004572, cofunded by the ERDF as part of
the MŠMT. We thank Diamond Light Source for acces to beamline I05 under
proposal no. SI25906.

## References

  * Wilson and Yoffe [1969] J. A. Wilson and A. D. Yoffe, The transition metal dichalcogenides discussion and interpretation of the observed optical, electrical and structural properties, Advances in Physics 18, 193 (1969).
  * Revolinsky  _et al._ [1963] E. Revolinsky, E. P. Lautenschlager, and C. H. Armitage, Layer Structure Superconductor, Solid State Communications 1, 59 (1963).
  * Morosan  _et al._ [2006] E. Morosan, H. W. Zandbergen, B. S. Dennis, J. W. G. Bos, Y. Onose, T. Klimczuk, A. P. Ramirez, N. P. Ong, and R. J. Cava, Superconductivity in CuxTiSe2, Nature Physics 2, 544 (2006).
  * Sipos  _et al._ [2008] B. Sipos, A. F. Kusmartseva, A. Akrap, H. Berger, L. Forró, and E. Tutis, From Mott state to superconductivity in 1T-TaS2, Nature Materials 7, 960 (2008).
  * Fazekas and Tosatti [1979] P. Fazekas and E. Tosatti, Electrical, structural and magnetic properties of pure and doped 1T-TaS2, Philosophical Magazine B 39, 229 (1979).
  * Wilson [1979] J. A. Wilson, Bands, bonds, and charge-density waves in the NbSe3 family of compounds, Physical Review B 19, 6456 (1979).
  * Cercellier  _et al._ [2007] H. Cercellier, C. Monney, F. Clerc, C. Battaglia, L. Despont, M. G. Garnier, H. Beck, and P. Aebi, Evidence for an Excitonic Insulator Phase in 1T-TiSe2, Phys. Rev. Lett. 99, 146403 (2007).
  * Rossnagel [2011] K. Rossnagel, On the origin of charge-density waves in select layered transition-metal dichalcogenides, J. Phys.: Condens. Matter 23, 213001 (2011).
  * Yang  _et al._ [2012] J. J. Yang, Y. J. Choi, Y. S. Oh, A. Hogan, Y. Horibe, K. Kim, B. I. Min, and S. W. Cheong, Charge-orbital density wave and superconductivity in the strong spin-orbit coupled IrTe 2:Pd, Physical Review Letters 108, 116402 (2012).
  * Ko  _et al._ [2015] K. T. Ko, H. H. Lee, D. H. Kim, J. J. Yang, S. W. Cheong, M. J. Eom, J. S. Kim, R. Gammag, K. S. Kim, H. S. Kim, T. H. Kim, H. W. Yeom, T. Y. Koo, H. D. Kim, and J. H. Park, Charge-ordering cascade with spin-orbit Mott dimer states in metallic iridium ditelluride, Nature Communications 6, 7342 (2015).
  * Hsu  _et al._ [2013] P.-j. Hsu, T. Mauerer, M. Vogt, J. J. Yang, Y. S. Oh, S.-W. Cheong, M. Bode, and W. Wu, Hysteretic Melting Transition of a Soliton Lattice in a Commensurate Charge Modulation, Phys. Rev. Lett. 111, 266401 (2013).
  * Takubo  _et al._ [2018] K. Takubo, K. Yamamoto, Y. Hirata, H. Wadati, T. Mizokawa, R. Sutarto, F. He, K. Ishii, Y. Yamasaki, H. Nakao, Y. Murakami, G. Matsuo, H. Ishii, M. Kobayashi, K. Kudo, and M. Nohara, Commensurate versus incommensurate charge ordering near the superconducting dome in Ir1-xPtxTe2 revealed by resonant x-ray scattering, Physical Review B 97, 1 (2018).
  * Oh  _et al._ [2013] Y. S. Oh, J. J. Yang, Y. Horibe, and S. Cheong, Anionic Depolymerization Transition in IrTe 2, Phys. Rev. Lett. 110, 127209 (2013).
  * Nicholson  _et al._ [2021] C. W. Nicholson, M. Rumo, A. Pulkkinen, G. Kremer, B. Salzmann, M.-l. Mottas, B. Hildebrand, T. Jaouen, T. K. Kim, S. Mukherjee, K. Ma, M. Muntwiler, F. O. V. Rohr, C. Cacho, and C. Monney, Uniaxial strain-induced phase transition in the 2D topological semimetal IrTe2, Communications Materials 2, 25 (2021).
  * Pascut  _et al._ [2014a] G. L. Pascut, K. Haule, M. J. Gutmann, S. A. Barnett, A. Bombardi, S. Artyukhin, T. Birol, D. Vanderbilt, J. J. Yang, S. W. Cheong, and V. Kiryukhin, Dimerization-induced cross-layer quasi-two-dimensionality in metallic IrTe2, Physical Review Letters 112, 086402 (2014a).
  * Rumo  _et al._ [2020] M. Rumo, C. W. Nicholson, A. Pulkkinen, B. Hildebrand, G. Kremer, B. Salzmann, M. Mottas, K. Y. Ma, E. L. Wong, M. K. L. Man, K. M. Dani, B. Barbiellini, M. Muntwiler, T. Jaouen, F. O. V. Rohr, and C. Monney, Examining the surface phase diagram of IrTe2 with photoemission, Phys. Rev. B 101, 235120 (2020).
  * Hwang  _et al._ [2022] J. Hwang, K. Kim, C. Zhang, T. Zhu, C. Herbig, S. Kim, B. Kim, Y. Zhong, M. Salah, M. M. El-Desoky, C. Hwang, Z. X. Shen, M. F. Crommie, and S. K. Mo, Large-gap insulating dimer ground state in monolayer IrTe2, Nature Communications 13, 906 (2022).
  * Kim  _et al._ [2008] B. J. Kim, H. Jin, S. J. Moon, J. Y. Kim, B. G. Park, C. S. Leem, J. Yu, T. W. Noh, C. Kim, S. J. Oh, J. H. Park, V. Durairaj, G. Cao, and E. Rotenberg, Novel jeff=1/2 mott state induced by relativistic spin-orbit coupling in Sr2IrO4, Physical Review Letters 101, 076402 (2008).
  * Pascut  _et al._ [2014b] G. L. Pascut, T. Birol, M. J. Gutmann, J. J. Yang, S. W. Cheong, K. Haule, and V. Kiryukhin, Series of alternating states with unpolarized and spin-polarized bands in dimerized IrTe2, Physical Review B 90, 195122 (2014b).
  * Saleh and Artyukhin [2020] G. Saleh and S. Artyukhin, First-Principles Theory of Phase Transitions in IrTe2, Journal of Physical Chemistry Letters 11, 2127 (2020).
  * Ritschel  _et al._ [2021] T. Ritschel, Q. Stahl, M. Kusch, J. Trinckauf, G. Garbarino, V. Svitlyk, M. Mezouar, J. Yang, S. W. Cheong, and J. Geck, Local mechanism of valence bond formation in IrTe2, arXiv , 1 (2021), arXiv:2112.04584 .
  * Ootsuki  _et al._ [2013] D. Ootsuki, S. Pyon, K. Kudo, M. Nohara, M. Horio, T. Yoshida, T. Oshida, A. Fujimori, M. Arita, H. Anzai, H. Namatame, M. Taniguchi, N. L. Saini, and T. Mizokawa, Electronic Structure Reconstruction by Orbital Symmetry Breaking in IrTe 2, Journal of the Physical Society of Japan 82, 093704 (2013).
  * Ootsuki  _et al._ [2017] D. Ootsuki, H. Ishii, K. Kudo, M. Nohara, M. Takahashi, M. Horio, A. Fujimori, T. Yoshida, M. Arita, H. Anzai, H. Namatame, M. Taniguchi, N. L. Saini, and T. Mizokawa, A Novel One-Dimensional Electronic State at IrTe 2 Surface, J. Phys. Soc. Japan 86, 123704 (2017).
  * Qian  _et al._ [2014] T. Qian, H. Miao, Z. J. Wang, X. Shi, Y. B. Huang, Z. P, N. Xu, L. K. Zeng, M. J. Z, R. P, M. Shi, G. Xu, X. Dai, Z. Fang, A. F. Fang, N. L. Wang, and H. Ding, Structural phase transition associated with van Hove singularity in 5d transition metal compound IrTe 2, New Journal of Physics 16, 123038 (2014).
  * Blake  _et al._ [2015] S. F. Blake, M. D. Watson, A. McCollam, S. Kasahara, R. D. Johnson, A. Narayanan, G. L. Pascut, K. Haule, V. Kiryukhin, T. Yamashita, D. Watanabe, T. Shibauchi, Y. Matsuda, and A. I. Coldea, Fermi surface of IrTe2 in the valence-bond state as determined by quantum oscillations, Physical Review B 91, 2 (2015).
  * Lee  _et al._ [2017] H. Lee, K. T. Ko, K. Kim, B. G. Park, J. Yang, S. W. Cheong, and J. H. Park, Electronic reconstruction on dimerized IrTe2, European Physics Letters 120, 47003 (2017).
  * Bao  _et al._ [2021] C. Bao, H. Zhang, Q. Li, S. Zhou, H. Zhang, K. Deng, K. Zhang, L. Luo, W. Yao, C. Chen, J. Avila, M. C. Asensio, Y. Wu, and S. Zhou, Spatially-resolved electronic structure of stripe domains in IrTe2 through electronic structure microscopy, Communications Physics 4, 1 (2021).
  * Rumo  _et al._ [2021] M. Rumo, A. Pulkkinen, B. Salzmann, G. Kremer, B. Hildebrand, K. Y. Ma, F. O. Von Rohr, C. W. Nicholson, T. Jaouen, and C. Monney, Insensitivity of the striped charge orders in IrTe2 to alkali surface doping implies their structural origin, Physical Review Materials 5, 1 (2021), arXiv:2106.08642 .
  * Mizokawa  _et al._ [2022] T. Mizokawa, A. Barinov, V. Kandyba, A. Giampietri, R. Matsumoto, Y. Okamoto, K. Takubo, K. Miyamoto, T. Okuda, S. Pyon, H. Ishii, K. Kudo, M. Nohara, and N. L. Saini, Domain Dependent Fermi Arcs Observed in a Striped Phase Dichalcogenide, Advanced Quantum Technologies , 2200029 (2022).
  * Jobic  _et al._ [1991] S. Jobic, P. Deniard, R. Brec, and J. Rouxel, Crystal and Electronic Band Structure of IrTe2: Evidence of Anionic Bonds in a CdI2-like Arrangement, Z. anorg. allg. Chem. 598/599, 199 (1991).
  * Fang  _et al._ [2013] A. F. Fang, G. Xu, T. Dong, P. Zheng, and N. L. Wang, Structural phase transition in IrTe2: A combined study of optical spectroscopy and band structure calculations, Scientific Reports 3, 1153 (2013), arXiv:1203.4061 .
  * Kresse and Hafner [1993] G. Kresse and J. Hafner, Ab initio molecular dynamics for liquid metals, Phys. Rev. B 47, 558 (1993).
  * Kresse and Hafner [1994] G. Kresse and J. Hafner, Ab initio molecular-dynamics simulation of the liquid-metalamorphous- semiconductor transition in germanium, Physical Review B 49, 14251 (1994).
  * Kresse and Furthmüller [1996a] G. Kresse and J. Furthmüller, Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set, Computational Materials Science 6, 15 (1996a).
  * Kresse and Furthmüller [1996b] G. Kresse and J. Furthmüller, Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set, Physical Review B 54, 11169 (1996b).
  * Kresse and Joubert [1999] G. Kresse and D. Joubert, From ultrasoft pseudopotentials to the projector augmented-wave method, Physical Review B 59, 1758 (1999).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized gradient approximation made simple, Physical Review Letters 77, 3865 (1996).
  * Blöchl [1994] P. E. Blöchl, Projector augmented-wave method, Phys. Rev. B 50, 17953 (1994).
  * Bahramy  _et al._ [2018] M. S. Bahramy, O. J. Clark, B. J. Yang, J. Feng, L. Bawden, J. M. Riley, I. Markovic, F. Mazzola, V. Sunko, D. Biswas, S. P. Cooil, M. Jorge, J. W. Wells, M. Leandersson, T. Balasubramanian, J. Fujii, I. Vobornik, J. E. Rault, T. K. Kim, M. Hoesch, K. Okawa, M. Asakawa, T. Sasagawa, T. Eknapakul, W. Meevasana, and P. D. King, Ubiquitous formation of bulk Dirac cones and topological surface states from a single orbital manifold in transition-metal dichalcogenides, Nature Materials 17, 21 (2018).
  * Song  _et al._ [2021] Y. Song, F. Meng, T. Ying, J. Deng, J. Wang, X. Han, Q. Zhang, Y. Huang, J. G. Guo, and X. Chen, Spatially Separated Superconductivity and Enhanced Charge-Density-Wave Ordering in an IrTe2 Nanoflake, Journal of Physical Chemistry Letters 12, 12180 (2021).
  * Park  _et al._ [2021] S. Park, S. Y. Kim, H. K. Kim, M. J. Kim, T. Kim, H. Kim, G. S. Choi, C. J. Won, S. Kim, K. Kim, E. F. Talantsev, K. Watanabe, T. Taniguchi, S. W. Cheong, B. J. Kim, H. W. Yeom, J. Kim, T. H. Kim, and J. S. Kim, Superconductivity emerging from a stripe charge order in IrTe2 nanoflakes, Nature Communications 12, 1 (2021), arXiv:2009.12578 .
  * Jobic  _et al._ [1992] S. Jobic, R. Brec, and J. Rouxel, Anionic Polymeric Bonds in Transition Metal Ditellurides, Journal of Solid State Chemistry 96, 169 (1992).