tcb@breakable

# TACT: Advancing Complex Aggregative Reasoning
with Information Extraction Tools

Avi Caciularuγ Alon Jacoviγ Eyal Ben-Davidγ Sasha Goldshteinγ
Tal Schusterδ Jonathan Herzigγ Gal Elidanγ Amir Globersonγ
γGoogle Research δGoogle DeepMind
<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) often do not perform well on queries that require
the aggregation of information across texts. To better evaluate this setting
and facilitate modeling efforts, we introduce TACT—Text And Calculations
through Tables, a dataset crafted to evaluate LLMs’ reasoning and
computational abilities using complex instructions. TACT contains challenging
instructions that demand stitching information scattered across one or more
texts, and performing complex integration on this information to generate the
answer. We construct this dataset by leveraging an existing dataset of texts
and their associated tables. For each such tables, we formulate new queries,
and gather their respective answers. We demonstrate that all contemporary LLMs
perform poorly on this dataset, achieving an accuracy below 38%. To pinpoint
the difficulties and thoroughly dissect the problem, we analyze model
performance across three components: table-generation, Pandas command-
generation, and execution. Unexpectedly, we discover that each component
presents substantial challenges for current LLMs. These insights lead us to
propose a focused modeling framework, which we refer to as IE as a tool.
Specifically, we propose to add “tools” for each of the above steps, and
implement each such tool with few-shot prompting. This approach shows an
improvement over existing prompting techniques, offering a promising direction
for enhancing model capabilities in these tasks. The dataset is available at
tact-benchmark.github.io.

Figure 1: An illustration of the annotated components within the TACT dataset.
The answer is short and objective, but requires advanced reasoning and
information integration skills. The intermediate artifacts are useful for
analyzing the reasoning process of LLMs, and for designing the IE as a tool
method. The relevant spans in the text and the instruction are underlined for
presentation.

##  1 Introduction

Large Language Models (LLMs) have shown exceptional capabilities across a wide
range of natural language tasks. However, they still face significant
challenges in solving complex problems that require reasoning over data
presented in non-mathematical formats, such as word and algebraic problems [2,
15]. Interestingly, research indicates that these types of problems pose
difficulties not only for LLMs but also for humans [12, 16]. This difficulty
mirrors a broader observation about LLM reasoning capabilities: the process of
transforming linguistic or graphical inputs into solvable mathematical
equations is often more challenging than performing the calculations
themselves [49, 13]. This overarching issue is particularly evident when LLMs
attempt tasks that involve the aggregation of information from either single
or multiple texts. These models frequently underperform in tasks that require
counting, comparing, or processing similar events or entities within texts [8,
3, 34]. This highlights a fundamental limitation: while LLMs can handle
isolated data points effectively, their ability to integrate and interpret
information across contexts remains a significant hurdle.

A first step towards advancing the capabilities of LLMs on complex reasoning,
is to have a high-quality benchmark for evaluating and analyzing their
performance in this setting. To this end, we introduce TACT—Text And
Calculations through Tables. TACT instances were created by NLP and data
science experts who wrote aggregative queries over texts. The experts were
instructed to use tables (from the InstructIE dataset [21]) as the basis for
writing the instructions, as they consolidate dispersed information from
source texts into a structured format, enabling comprehensive aggregation
across the text. The resulting TACT instances consist of the original text,
the written instruction, and a gold answer, all requiring advanced text
comprehension and reasoning (illustrated in Figure 1). Through TACT, the model
is implicitly required to address information extraction (IE) challenges such
as coreference resolution [33, 31, 32], multi-hop reasoning [37, 15, 60],
summarization [59, 20, 50], and multi-document processing [7, 23, 9, 63]. In
Section 2, we describe our methodology for constructing this benchmark—by
layering our new expert annotations over InstructIE instances—and the measures
we took to ensure its difficulty and robustness.

Having evaluated LLMs on TACT and observed the challenges it presents to them
(Section 4), we aim to understand the root of these difficulties and explore
potential modeling improvements. We propose dissecting the problem into three
tasks: table-generation, Pandas command-generation, and command-execution.
Leveraging TACT’s ground-truth tables and Pandas commands curated by experts,
we analyze the LLM performance of each step in Section 5. Our findings reveal
significant performance headroom in each task, implying that with targeted
few-shot prompting, models can considerably enhance their individual task
performance. Building on these results, we propose a focused modeling strategy
termed the IE as a tool framework, which specifically addresses each phase
independently (see an illustration in Figure 4 and more details in Section 3).
This approach has shown to be superior to existing prompting techniques, as
detailed in Section 4. The demonstrated improvements suggest a promising
direction for enhancing LLM capabilities in complex reasoning tasks, aligning
with our initial findings of untapped potential in each dissected component of
the task.

Our contributions are summarized as follows:

  * •

TACT: An expert-curated, diverse evaluation dataset that challenges LLMs on
following aggregative queries, requiring information extraction and complex
reasoning.

  * •

A rigorous analysis of LLM performance on decomposed TACT tasks, revealing
model strengths and weaknesses in table-generation, Pandas command-generation,
and execution.

  * •

Introduction of the IE as a tool framework, leveraging the aforementioned sub-
tasks as discrete tools, demonstrating up to 12% improvement over conventional
prompting techniques.

##  2 Dataset

This section introduces TACT—Text And Calculations through Tables—a novel
challenge set designed to evaluate and improve the capability of LLMs on
complex queries that require integration of information. The data is derived
from the InstructIE [30] test set using new expert-annotated labels, as
described below. In this section, we detail the data labeling methodology
employed to create TACT, highlighting the steps taken to ensure the
reliability and validity of the labeled data. We first introduce the
InstructIE benchmark creation methodology (Section 2.1), then we introduce our
TACT dataset (Section 2.2), and finally, we explore the properties and conduct
an analysis of TACT (Section 2.3).

###  2.1 Background: The InstructIE Dataset

InstructIE [30] is a dataset that includes texts alongside corresponding
tables, which summarize the textual content. These tables effectively organize
the extracted information into sets of triples—subjects, relations, and
objects—derived from the texts. To compile the tables and texts in the test
set of InstructIE, which we employed for creating TACT, human annotators first
defined the table topics and columns using real-world texts from the web.
These texts were then utilized to craft tables that summarize them through a
process combining automatic extraction and manual validation.

The primary components of InstructIE that we utilized are: Text– the
accompanying document or collection of short documents, Table– a structured
representation of the extracted information, where the first row serves as the
table header and the subsequent rows contain the extracted data. See an
illustrative example in Figure 7 in Appendix C.2, and Jiao et al. [30] for
additional details and descriptions of other components that were not included
in our study. While InstructIE provides a good setup for information
extraction, it does not directly test the models’ abilities to aggregate the
extracted information, which we target in TACT.

###  2.2 The TACT Dataset

Our goal is to evaluate the capabilities of LLMs in addressing aggregative,
information-seeking queries, that require both text comprehension and complex
reasoning. Using tables as the basis for creating such queries is highly
effective, since they consolidate the essential information from their source
texts into a structured format. Thus, performing an aggregation on these
tables is equivalent to executing an aggregation across the entire text. We
leverage and extend the use of the InstructIE dataset, which already contains
structured information in table format (see above). We introduce the Text And
Calculations through Tables (TACT) challenge set, aimed at verifying the
capabilities of LLMs in handling complex numerical instructions (the resulting
TACT dataset and its components are compared to InstructIE in Figure 7 in
Appendix C.2).

TACT was created by NLP and data science experts, who employed a rigorous
annotation process to transform InstructIE instances into a format suitable
for aggregative instruction following. Creating the data includes the steps of
assessing the text and the table, then formulating a query in natural
language, and finally translating the query into a Pandas command, and
executing it on the table. See the data creation guidelines, summarization of
the annotation process, and more details about the data creation in Appendix
C). The full steps are:

Initial Review and Relevance Vetting: A comprehensive review of the InstructIE
dataset, focusing on texts and tables out of InstructIE’s test set containing
numerical data. Experts identified tables and text segments where numerical
data was present and suitable for quantitative instructions. Tables were
vetted for numerical integrity and alignment with the text to ensure data
quality. For the remaining examples, the experts were tasked to convert the
Markdown-formatted tables from InstructIE into the CSV table format (for
convenient parsing into the Pandas dataframe format).

Numerical Aspect Identification: Experts identified specific numerical aspects
within the text and tables—such as years, currencies, population counts, and
temperatures—that enable quantitative operations like counting, calculation,
and aggregation. This step identifies which aspect of the table data should be
incorporated into the instruction.

Natural Language Instruction Formulation: Based on the identified numerical
aspects, experts formulated clear and precise natural language instructions
over the text that result in a single numerical value. These instructions
targeted the numerical aspects with a focus on aggregation functions like sum,
mean, and filtering.

Natural Language Query Over the Table: After formulating the natural language
instructions, experts verbalized them into corresponding natural language
queries over the tables (see Figure 1). These queries refined the focus on the
numerical data within the table, minimized ambiguity, and helped to prepare
the Pandas command.

Translation to Pandas Commands and Gold Response Extraction: Next, experts
translated the previous natural language query over the table into a Pandas
command. Then, they extracted the gold response by executing the formulated
Pandas commands over the tables.

Command Execution and Validation: Finally, the extracted responses were
manually verified against the expected outcomes derived from the formulated
instructions and texts. This validation step ensured that the results were
consistent with the intended instructions and the underlying data from both
the text and the tables.

Each instance in the dataset consists of (see illustrative example in Figure
1):

  1. 1.

Original Text and Table: Sourced from the InstructIE dataset, these elements
contain the foundational data and numerical information relevant to the query.
The text provides context, while the table offers structured numerical data
aligned with the text content.

  2. 2.

Natural Language Question: A clearly formulated query in natural language that
targets specific numerical aspects identified in the text and table. These
questions focus on computational tasks like sum, mean, and filtering to
challenge the models’ understanding and processing capabilities.

  3. 3.

Natural Language Query Over the Table: After formulating the natural language
question, a corresponding natural language query over the table is developed.
This step refines the focus on the numerical data within the table, ensuring
that the essential information for the computation is precisely delineated and
consistent with the intent of the initial question.

  4. 4.

Pandas Command: A precise translation of the natural language question into a
Pandas command. This command is designed to replicate the expected
computational process using the original column names from the table, ensuring
the accuracy and consistency of the data manipulation.

  5. 5.

Expected Result: The correct numerical answer derived from executing the
Pandas command, serving as a benchmark to validate the models’ responses
against the ground truth.

The resulting TACT benchmark contains 124 examples,111Recent studies, through
empirical validation, found that 100 examples are sufficient to conduct a
high-quality evaluation of LLMs [35, 47]. as well as additional 4 examples
that serve as few-shot examples.222These examples are sourced from the
validation set of InstructIE, but follow the same process for constructing the
TACT, as elaborated in Section 2.2. For evaluating performance on TACT, we
employ exact match for the final answer, since it is a single-span (number).
For intermediate steps available in TACT, such as table- and command-
generation, we utilize a both similarity metrics (e.g., ROUGE [36]) and
execution-based metrics (e.g., accuracy of the generated command’s output) as
described in Section 5.

###  2.3 Exploring the Numerical Challenges in TACT

In this section, we delve into the characteristics of the TACT dataset. TACT
offers a diverse range of tasks, primarily focusing on two types of
instructions–“Calculate” and “Count”:

“Calculate” Instructions: Out of the 124 examples, 63 are categorized under
“Calculate” instructions. These tasks require the execution of basic
mathematical operations to solve the instance. As depicted in Figure 3, the
operations include addition, subtraction, multiplication, division, and other
arithmetic functions. The distribution of these operations, such as summation
(28.3%), mean calculation (10.4%), and power functions (6.1%), highlights the
varied complexity and the need for precise computational understanding by the
models.

“Count” Instructions: The remaining 61 examples fall under “Count”
instructions, where the primary objective is to identify specific types or
categories within the attached text and perform a simple counting operation.
This task challenges the model’s ability to accurately parse and interpret
textual data, identifying relevant entities or events, and perform the proper
counting.

The composition of the TACT dataset, with a balanced mix of “Calculate” and
“Count” instructions, ensures a comprehensive evaluation of models across
different dimensions of numerical reasoning. The operations, as detailed in
the pie chart (Figure 3), further emphasize the diversity and scope of
numerical challenges that TACT presents.

Figure 2: The TACT Dataset pandas different tokens’ distribution.

Figure 3: TACT’s Pandas commands’ length vs. the total number of cells in
their corresponding tables.

In Figure 3, we present a comparison of the total number of cells in a table
against the lengths of the corresponding Pandas commands. The figure reveals a
wide distribution of data points, illustrating that the length of Pandas
commands, quantified in Gemini tokens [52], does not correlate directly with
the total number of cells in the tables. The varied spread of points across
the graph indicates that additional factors, such as the complexity of
arithmetic operations required or the specific data arrangement within the
tables, might play a more significant role in determining the length of the
commands than simply the volume of data.

In Table 1, we present an illustration of the diverse range of implicit tasks
incorporated within TACT, which are specifically designed to test advanced
text comprehension and numerical reasoning. Each task is tied to text spans
that underline the specific data points or contextual clues necessary for task
completion, ranging from multi-document summarization to date and time
numerical reasoning. For example, one task leverages coreference resolution,
requiring the model to understand and connect information spread across
different parts of the text. Another task tests the model’s capability for
lexical matching, identifying specific words within a context. Complex
arithmetic operations are also present, demanding a high level of numerical
literacy to interpret numerical and financial concepts. This highlights the
interplay between linguistic understanding and numerical computations which
examines can handle a wide spectrum of real-world tasks, from simple counting
to handling complex, multi-step mathematical operations embedded in textual
data.

Table 1: An overview of implicit tasks in TACT, split by their types of
’Count’ and ’Calc.’ (Calculate) instruction types, along with textual
instructions, accompanying texts and their corresponding sub-tasks. The sub-
tasks’ related aspects in the text are underlined.

Operation |  Instruction |  Relevant text spans |  Implicit task
---|---|---|---
Count | Calc.
| ✓ |  Calculate the sum of the years that "To Kill a Mockingbird" was published in, and the year that it won a prize according to the text. |  $\ldots$ It was published in 1960 $\ldots$ A year after its release, it won the Pulitzer Prize $\ldots$ |  Coreference resolution
✓ |  |  Count the number of achievements that include instructions. |  1. We present FLAIR $\ldots$ 2. How well can NLP models perform? $\ldots$ 3. Pretrained language models have become increasingly prominent $\ldots$ |  Multi-document summarization
| ✓ |  Calculate the sum of squares of the stock price increases in the text. |  $\ldots$ The S&P 500 rose 1.45% $\ldots$ the Nasdaq Composite popped 1.07% $\ldots$ The Dow Jones Industrial Average led gains, rising 2.12% $\ldots$ |  Complex arithmetics
✓ |  |  Count the number of weather forecasts that include temperatures between 50 and 91 degrees. |  $\ldots$ with temps currently ranging from the upper 80s to low 90s $\ldots$ Overnight, seasonal temps in the upper 60s to low 70s continue $\ldots$ |  Numerical range entailment
✓ |  |  Count the number of wars in the text that have "Indian" within their names. |  $\ldots$ The American Indian Wars, also known as the $\ldots$ and the Indian Wars $\ldots$ |  Lexical matching
✓ |  |  Count the number of cases where the delivery date was later than June 2, 2023 and the travel time was more than 2 hours in this text. |  $\ldots$ 2\. On June 10, 2023, XYZ Shipping’s Truck 789, $\ldots$ Departing at 8:30 AM and arriving at 2:00 PM $\ldots$ |  Date and time numerical reasoning

##  3 IE as a Tool

As our experiments in the following sections will show, current models
struggle with TACT. current models struggle with the TACT task. To address
this, we propose an approach called IE as a tool. See Figure 4 for an
illustration. The key idea is to solve TACT instructions through the
sequential invocation of two tools: one that generates a table from the text
and the instruction and one that generates a corresponding Pandas command. The
model then executes the command, along with the original instruction and text,
to produce the final answer. We argue that this sequence is a natural strategy
for solving TACT’s aggregative queries. Implementation of these tools can use
various approaches. While these tools could be implemented in various ways, we
opted for a simple few-shot prompting approach (see the prompt templates in
Appendix D.3). Our experimental results demonstrate that IE as a tool leads to
a 12% improvement in TACT performance compared to traditional prompting
techniques (see Section 4).

Figure 4: Possible setups for solving TACT with LLMs. (a) Typical Approach:
the large language model (LLM) directly generates the answer based on the
provided query and text, but without the aid of any external tools. (b) With
IE as a Tool: This approach utilizes a three-step process. First, an
information extraction tool generates a structured table from the text and
query. Next, another tool formulates an appropriate Pandas command based on
this table, the text and the query. Finally, all this information is fed into
the LLM, which then generates the answer; or into a code interpreter that can
run the Pandas command over the table.

##  4 Experimental Setup and TACT Results

We assess the performance of several models on the TACT dataset, including
Gemini-1.0-Ultra [52], Gemini-1.5-Pro [48], Llama-2-13b-chat, Llama-2-70b-chat
[54], Gemma-7b-it (v1.1) [53], Mistral-7b-instruct (v0.2) [27], and Mixtral
8x7B [28]. Their capabilities are tested using both standard prompting
techniques and our IE as a tool method (detailed in Section 3). Subsequent
analysis focuses on the specific sub-tasks of table generation (Section 5.1)
and Pandas command generation (Section 5.2), providing a comprehensive
evaluation of each model’s performance and identifying potential areas for
improvement in each sub-task.

We measure performance by averaging over four combinations of few-shot
examples (by sampling the order and examples), following the procedure
described in Jacovi et al. [26], and report the results for {0, 2, 4}-shot,
using dedicated examples from the validation set of InstructIE (see more
details in Appendix D).

We evaluate the LLMs’ performance on the TACT task, measuring exact match,
where the models are provided a numerical instruction and a text, and are
tasked to produce the correct numerical answer. We report 4-shot results for
Gemma-7b-it, Mistral-7b-instruct, and Mixtral 8x7B, while zero-shot results
for the remaining larger models, given the results on table and Pandas query
generation in the following sections, and the observation that few-shot
demonstrations yield saturated performance for most of the tasks for the
larger models. We propose the following experimental setups for each LLM:

  * •

Generic: The baseline setting where the LLM receives a TACT instruction and a
text passage, and is tasked with directly generating the answer.

  * •

Chain-of-Thought (CoT): This setting is akin to the baseline, with the
enhancement of adding “Let’s think step-by-step” to the prefix (input). This
encourages the model to generate a detailed, step-by-step CoT reasoning before
producing the final answer.

  * •

In-context IE: This method adopts a chain-of-thought-like approach, where the
LLM first generates a table from the text, then creates a Pandas query, and
finally, with all this context provided, the model generates the answer, all
within the same prompt.

  * •

IE as a tool: Here, the model can utilize generated tables and Pandas queries
to answer the instruction, like the previous setup, however, in this
variation, we employ few-shot prompted LLMs as separate tools (the number of
shots applies for these tools). See Section 3 for more details.

  * •

IE as a tool (Gold): This configuration is similar to the previous one but
utilizes gold-standard (i.e., ground truth) tables (Gold Table) and/or Pandas
commands (Gold Table+Pandas) from TACT, rather than relying on outputs
generated by the tools. This baseline serves as an upper bound for the
performance of the IE as a tool approach.

Table 2: Exact match accuracy evaluation results of different models on TACT,
evaluated across different experimental setups, including Generic, Chain-of-
Thought (CoT), In-context IE, IE as a tool, and IE as a tool (Gold). The best-
performing results are highlighted in bold.

Model | Generic | CoT | In-context IE | IE as a tool | IE as a tool (Gold Table) | IE as a tool (Gold Table+Pandas)
---|---|---|---|---|---|---
Gemma-7b (4-shot) | 17.1 | 25.4 | 26.2 | 28.9 | 33.3 | 45.1
Mistral-7b (4-shot) | 2.4 | 2.6 | 2.6 | 3.5 | 4.1 | 10.9
Llama-2-13b (4-shot) | 8.5 | 8.0 | 8.3 | 8.5 | 9.2 | 14.3
Mixtral 8x7B (4-shot) | 4.4 | 4.2 | 5.2 | 6.1 | 6.3 | 12.6
Llama-2-70b (0-shot) | 3.6 | 4.0 | 4.2 | 7.6 | 12.9 | 22.4
Gemini-Pro (0-shot) | 28.4 | 34.7 | 12.3 | 41.9 | 48.5 | 72.1
Gemini-Ultra (0-shot) | 25.4 | 37.3 | 36.6 | 41.4 | 49.8 | 72.3

The results, which are depicted in Table 2, show that all models experience
substantial benefits when using the IE as a tool approach, and this
particularly clear in the larger models (Gemini-Ultra and Gemini-Pro), which
excel on this task. This is evident from the consistent performance
improvement across different models when comparing the IE as a tool setup with
the Generic baseline and other approaches. On the other hand, smaller models
showed more moderate improvements when using IE as a tool. The gap between the
performance in IE as a tool and the Gold variant implies a significant
potential for enhancing the overall task effectiveness through the refinement
of IE tools. Specifically, the larger gap for smaller models suggests that
their capabilities can be dramatically increased by improving the accuracy and
reliability of the generated tables and commands. This points towards the
critical importance of future work on optimizing IE tools to maximize the end-
task performance for complex reasoning tasks.

##  5 Performance Analysis via TACT Decomposition

To understand the factors contributing to the suboptimal performance of
current LLMs on TACT, we decompose the problem into two constituent tasks: 1)
table generation from text based on a TACT instruction and the corresponding
text (Section 5.1), and 2) Pandas query generation based on the corresponding
table (either gold or previously generated), instruction, and text (Section
5.2). Successful execution of these two tasks would naturally result in
accurate TACT results. We assess current LLM capabilities on each task using
gold outputs, revealing substantial headroom and a potential for improvement.
This observation directly motivated our design of IE as a Tool, a method that
demonstrably improved TACT performance.

###  5.1 Evaluating the Accuracy of Table Generation

We assess the capabilities of LLMs to generate the appropriate tables given a
TACT instruction and its corresponding text, tasking the model to construct
the correct table based on the specified instruction. Note that the model
should infer the correct table from the TACT instruction, which only
implicitly points towards the relevant information to extract. Each model is
provided with TACT instructions and corresponding texts. The task requires
generating tables that accurately reflect the data described in the text, and
helps to seek the correct information given the instruction.

We follow the evaluation protocol from Jiao et al. [30] and adopt a soft
matching strategy [29] by using SentenceT5-Large [43] to calculate the cosine
similarity (multiplied by 100) as the semantic similarity score between the
generated table and the gold table, as table contents reflect the quality of
extraction. Additionally, we use the ROUGE-L F1 score [36] to evaluate the
lexical similarity of the generated table to the gold one. We also report the
Table Validity rate, where we were able to parse a syntactically correct CSV
table from the generated content.

Table 3: Evaluation Results of Different Models on TACT table generation,
measuring semantic similarity, ROUGE-L F1 (lexical matching), and table
validity between the generated tables and the gold tables. The best-performing
results are highlighted in bold.

| Semantic Similarity | ROUGE-L F1 | Table Validity Rate (%)
---|---|---|---
Model | 0-shot | 2-shot | 4-shot | 0-shot | 2-shot | 4-shot | 0-shot | 2-shot | 4-shot
Gemma-7b | 68.6 | 69.1 | 69.4 | 6.5 | 6.6 | 7.1 | 0.6 | 23.1 | 25.5
Mistral-7b | 73.5 | 72.8 | 72.8 | 4.9 | 6.4 | 7.0 | 1.2 | 30.9 | 34.4
Llama-2-13b | 73.4 | 72.7 | 73.1 | 4.3 | 5.2 | 5.5 | 42.7 | 43.5 | 47.4
Mixtral 8x7B | 72.3 | 71.7 | 71.8 | 4.5 | 7.1 | 7.2 | 9.5 | 39.9 | 24.3
Llama-2-70b | 72.3 | 72.4 | 72.7 | 3.9 | 5.1 | 4.9 | 92.5 | 73.4 | 73.8
Gemini-Pro | 78.4 | 78.2 | 78.3 | 18.8 | 21.0 | 22.9 | 81.5 | 90.2 | 93.3
Gemini-Ultra | 78.6 | 78.6 | 79.3 | 18.7 | 21.1 | 24.8 | 81.3 | 89.9 | 94.1

The evaluation of various LLMs on their ability to generate accurate tables
based on TACT instructions is shown in Table 3. Gemini-Ultra consistently
outperforms others across all metrics. Gemini-Pro also shows strong
performance but varies significantly across different shots, indicating
potential instability in its output quality. Llama-2-13b, Llama-2-70b, and
Mistral-7b exhibit moderate performance, with Mistral-7b achieving a higher
semantic similarity but lower table validity rates. Gemma-7b and Mixtral 8x7B
show comparatively lower performance, particularly in table validity. Notably,
the smaller models like Gemma-7b and Mistral-7b benefit significantly from
few-shot learning, demonstrating that small models are incapable of solving
this task without any aid.

While the semantic similarity between the generated tables of Gemini-Ultra and
the gold standard tables is relatively high, lexical similarity remains low.
However, a qualitative analysis suggests that the generated tables contain key
information that addresses the instructions, despite their differences from
the gold tables. This observation supports the use of semantic similarity as a
more appropriate metric for evaluating table generation in this context [30].

###  5.2 Evaluating the Accuracy of Pandas Command Generation

The ability to accurately generate Pandas commands is a key intermediate step
in solving TACT queries, and evaluates how well LLMs can comprehend the TACT
instruction and the table at once. Thus, we next evaluate the ability of LLMs
to generate Pandas commands, when provided with the TACT instruction, the
associated text, as well as a table extracted from the text. We consider two
cases: one where the provided table is the gold one, and one where it is the
one generated by the model. To assess the quality of the generated Pandas
queries, we execute them using a Python interpreter, and compare the output to
the gold answer.

Table 4: Evaluation Results of Different Models on TACT Pandas command
generation on the generated/gold table, measured by the accuracy after
executing the command with a Python interpreter. The best-performing results
are highlighted in bold.

Model | 0-shot (Generated/Gold) | 2-shot (Generated/Gold) | 4-shot (Generated/Gold)
---|---|---|---
Gemma-7bt | 0/0.4 | 1.4/2.3 | 1.9/2.4
Mistral-7b | 0.0/0.1 | 0.4/0.6 | 0.5/0.9
Llama-2-13b | 0.0/0.3 | 0.1/0.6 | 0.3/1.2
Mixtral 8x7B | 1.2/1.8 | 1.3/2.1 | 1.5/2.9
Llama-2-70b | 2.5/3.4 | 3.1/4.1 | 3.2/4.3
Gemini-Pro | 3.4/3.6 | 3.0/4.8 | 7.3/8.0
Gemini-Ultra | 1.1/1.9 | 4.8/5.0 | 7.7/8.4

Table 4 shows the results. It can be seen that Gemini-Ultra consistently
outperforms the other models. Gemini-Pro and Llama-2-70b also demonstrate
relatively solid performance but with some variability across different shots.
Interestingly, as in the previous experiment, the smaller models, such as
Gemma-7b and Mistral-7b showed relatively low performance, but significant
improvements with few-shot learning, indicating their ability to leverage
additional examples effectively. While Llama-2-13b and Mixtral 8x7B exhibited
moderate performance, they lag behind the larger models. We note that the
overall numbers in Table 4 are quite low even compared to the results in Table
2, and even for gold tables. This might be surprising. Our inspection revealed
that the reason is that Pandas commands are often with wrong syntax or other
errors that result in errors when executing them. However, in the results in
Table 2, Python is not called for execution, which results in more robust
interpretation of the commands, and better answers as a result.

##  6 Related Work

#### Information Extraction (IE) and Text-to-Table

IE is the process of automatically extracting structured information from
unstructured text, involving sub-tasks like named entity recognition, relation
extraction, and event extraction. Many works have leveraged large language
models (LLMs) to provide effective solutions for IE [39, 38, 62]. Recently, Wu
et al. [55] presented the concept of text-to-table, and Jiao et al. [30]
introduced InstructIE, a benchmark that includes triplets of an IE
instruction, their associated text, and the relevant content in a tabular
format (see Section 2). We employ a labeling methodology on top of InstructIE
to distill an aggregative instruction following challenge set. Yuan et al.
[57] presented an effort with a similar focus on numerical tasks but with a
narrower scope, limited to financial data. Another related research realm is
open information extraction, which aims to extract information without
predefined schemas, typically focusing on simple structures from short texts
[4, 40, 51, 58].

#### Numerical and math reasoning

A persistent challenge for LLMs lies in their ability to solve numerical
problems, particularly those involving mathematical calculations [19, 24, 10,
1]. Some works propose to evaluate models on such tasks, by presenting
challenge datasets based on financial data [11, 57]. DROP [15] and IIRC [17],
two reading comprehension benchmarks involving reasoning, showcase the
complexity of comprehensive numerical reasoning. DROP focuses on discrete
reasoning over paragraphs, requiring models to perform operations like
addition, counting, and sorting, while IIRC evaluates the ability to handle
incomplete contexts and locate additional sources of information. TACT focuses
on a more practical and common use-case: reasoning over texts given natural
language instructions, necessitating the integration of information scattered
throughout the text.

Our IE as a tool method also aligns with previous research on semantic parsing
[45, 56, 41, 6], as we utilize executable Pandas command generation over texts
and tables, demonstrating how LLMs can interpret and convert complex
instructions into executable code operations.

#### Multi-step Reasoning and LLM Tools

Our research is closely related to various methodologies that utilize LLM
tools for task resolution, as explored in recent studies [44, 42, 49, 22, 46].
These methods train LLMs to utilize APIs independently during inference,
contrasting with our IE as a tool approach, which employs a static strategy
for addressing complex reasoning tasks. Furthermore, prior research has
emphasized enhancing task resolution through multi-step processes [5, 14, 61,
18]. Unlike these approaches, which apply general multi-step reasoning or tool
triggering across various domains, IE as a tool specifically concentrates on
constructing tables and executing commands for numerical reasoning, thereby
targeting a more focused application of multi-step numerical reasoning.

##  7 Conclusion

In this paper, we introduced TACT—Text And Calculations through Tables, a
dataset designed to assess the reasoning capabilities of Large Language Models
(LLMs) through complex, aggregative instructions. TACT features numerical
instructions that require processing and integrating information dispersed
across one or more texts for producing the correct answer. By leveraging the
InstructIE dataset [30], experts annotated and transformed instances into a
format suitable for aggregative instruction following, ensuring high precision
and relevance. To better understand the performance of LLMs on TACT, we
provide further analysis that evaluates performance on two distinct sub-tasks
that are likely to be relevant for solving TACT (table-generation and Pandas
command-generation). We also provide a modeling scheme, IE as a tool, that is
based on this decomposition, and show that it improves performance on TACT.
Future work could focus on further enhancing the performance of LLMs on TACT
by developing and integrating new, more sophisticated tools that are
specifically designed for handling complex, aggregative instructions over
text.

## Acknowledgements

We thank Jonathan Berant, Yonatan Bitton, Mor Geva, and Eran Ofek for their
valuable feedback and constructive suggestions, and Ayelet Shasha Evron for
her assistance in designing the figures for this paper.

## References

  * Ahn et al. [2024] J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin.  Large language models for mathematical reasoning: Progresses and challenges.  In N. Falk, S. Papi, and M. Zhang, editors, _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024: Student Research Workshop, St. Julian’s, Malta, March 21-22, 2024_ , pages 225–237. Association for Computational Linguistics, 2024.  URL https://aclanthology.org/2024.eacl-srw.17.
  * Amini et al. [2019] A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi.  MathQA: Towards interpretable math word problem solving with operation-based formalisms.  In J. Burstein, C. Doran, and T. Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2357–2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1245.  URL https://aclanthology.org/N19-1245.
  * Amouyal et al. [2023] S. Amouyal, T. Wolfson, O. Rubin, O. Yoran, J. Herzig, and J. Berant.  QAMPARI: A benchmark for open-domain questions with many answers.  In S. Gehrmann, A. Wang, J. Sedoc, E. Clark, K. Dhole, K. R. Chandu, E. Santus, and H. Sedghamiz, editors, _Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)_ , pages 97–110, Singapore, Dec. 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.gem-1.9.
  * Banko et al. [2007] M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni.  Open information extraction from the web.  In M. M. Veloso, editor, _IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6-12, 2007_ , pages 2670–2676, 2007.  URL http://ijcai.org/Proceedings/07/Papers/429.pdf.
  * Berant et al. [2014] J. Berant, V. Srikumar, P.-C. Chen, A. Vander Linden, B. Harding, B. Huang, P. Clark, and C. D. Manning.  Modeling biological processes for reading comprehension.  In A. Moschitti, B. Pang, and W. Daelemans, editors, _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1499–1510, Doha, Qatar, Oct. 2014. Association for Computational Linguistics.  doi: 10.3115/v1/D14-1159.  URL https://aclanthology.org/D14-1159.
  * Bogin et al. [2023] B. Bogin, S. Gupta, P. Clark, and A. Sabharwal.  Leveraging Code to Improve In-context Learning for Semantic Parsing.  _CoRR_ , abs/2311.09519, 2023.  doi: 10.48550/ARXIV.2311.09519.  URL https://doi.org/10.48550/arXiv.2311.09519.
  * Caciularu et al. [2021] A. Caciularu, A. Cohan, I. Beltagy, M. Peters, A. Cattan, and I. Dagan.  CDLM: Cross-document language modeling.  In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2648–2662, Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-emnlp.225.  URL https://aclanthology.org/2021.findings-emnlp.225.
  * Caciularu et al. [2022] A. Caciularu, I. Dagan, J. Goldberger, and A. Cohan.  Long context question answering via supervised contrastive learning.  In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2872–2879, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.207.  URL https://aclanthology.org/2022.naacl-main.207.
  * Caciularu et al. [2023] A. Caciularu, M. Peters, J. Goldberger, I. Dagan, and A. Cohan.  Peek across: Improving multi-document modeling via cross-document question-answering.  In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1970–1989, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.110.  URL https://aclanthology.org/2023.acl-long.110.
  * Chang et al. [2024] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, W. Ye, Y. Zhang, Y. Chang, P. S. Yu, Q. Yang, and X. Xie.  A Survey on Evaluation of Large Language Models.  _ACM Trans. Intell. Syst. Technol._ , 15(3), mar 2024.  ISSN 2157-6904.  doi: 10.1145/3641289.  URL https://doi.org/10.1145/3641289.
  * Chen et al. [2021] Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T. Huang, B. R. Routledge, and W. Y. Wang.  Finqa: A dataset of numerical reasoning over financial data.  In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 3697–3711. Association for Computational Linguistics, 2021.  doi: 10.18653/V1/2021.EMNLP-MAIN.300.  URL https://doi.org/10.18653/v1/2021.emnlp-main.300.
  * Cummins et al. [1988] D. D. Cummins, W. Kintsch, K. Reusser, and R. Weimer.  The Role of Understanding in Solving Word Problems.  _Cognitive Psychology_ , 20(4):405–438, 1988.  ISSN 0010-0285.  doi: https://doi.org/10.1016/0010-0285(88)90011-4.  URL https://www.sciencedirect.com/science/article/pii/0010028588900114.
  * Das et al. [2024] D. Das, D. Banerjee, S. Aditya, and A. Kulkarni.  MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning.  _arXiv preprint arXiv:2402.17231_ , 2024.
  * Drozdov et al. [2023] A. Drozdov, N. Schärli, E. Akyürek, N. Scales, X. Song, X. Chen, O. Bousquet, and D. Zhou.  Compositional semantic parsing with large language models.  In _The Eleventh International Conference on Learning Representations (ICLR)_ , 2023.  URL https://openreview.net/forum?id=gJW8hSGBys8.
  * Dua et al. [2019] D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In J. Burstein, C. Doran, and T. Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2368–2378. Association for Computational Linguistics, 2019.  doi: 10.18653/V1/N19-1246.  URL https://doi.org/10.18653/v1/n19-1246.
  * Elliott [2023] E. C. Elliott.  Why Word Problems are Hard for High School Math Students: Problem Formulation and Disciplinary Literacy.  2023\.
  * Ferguson et al. [2020] J. Ferguson, M. Gardner, H. Hajishirzi, T. Khot, and P. Dasigi.  IIRC: A dataset of incomplete information reading comprehension questions.  In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 1137–1147. Association for Computational Linguistics, 2020.  doi: 10.18653/V1/2020.EMNLP-MAIN.86.  URL https://doi.org/10.18653/v1/2020.emnlp-main.86.
  * Fu et al. [2023] Y. Fu, H. Peng, A. Sabharwal, P. Clark, and T. Khot.  Complexity-based prompting for multi-step reasoning.  In _The Eleventh International Conference on Learning Representations (ICLR)_ , 2023.  URL https://openreview.net/forum?id=yf1icZHC-l9.
  * Geva et al. [2020] M. Geva, A. Gupta, and J. Berant.  Injecting numerical reasoning skills into language models.  In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 946–958, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.89.  URL https://aclanthology.org/2020.acl-main.89.
  * Goyal et al. [2022] T. Goyal, J. J. Li, and G. Durrett.  News Summarization and Evaluation in the Era of GPT-3.  _arXiv preprint arXiv:2209.12356_ , 2022.
  * Gui et al. [2023] H. Gui, J. Zhang, H. Ye, and N. Zhang.  InstructIE: A Bilingual Instruction-based Information Extraction Dataset.  _ArXiv_ , abs/2305.11527, 2023.  URL https://api.semanticscholar.org/CorpusID:258823375.
  * Hao et al. [2023] S. Hao, T. Liu, Z. Wang, and Z. Hu.  ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings.  In _Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.  URL https://openreview.net/forum?id=BHXsb69bSx.
  * Hirsch et al. [2021] E. Hirsch, A. Eirew, O. Shapira, A. Caciularu, A. Cattan, O. Ernst, R. Pasunuru, H. Ronen, M. Bansal, and I. Dagan.  iFacetSum: Coreference-based interactive faceted summarization for multi-document exploration.  In H. Adel and S. Shi, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 283–297, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-demo.33.  URL https://aclanthology.org/2021.emnlp-demo.33.
  * Imani et al. [2023] S. Imani, L. Du, and H. Shrivastava.  Mathprompter: Mathematical reasoning using large language models.  In S. Sitaram, B. B. Klebanov, and J. D. Williams, editors, _Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics: Industry Track, ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 37–42. Association for Computational Linguistics, 2023.  doi: 10.18653/V1/2023.ACL-INDUSTRY.4.  URL https://doi.org/10.18653/v1/2023.acl-industry.4.
  * Jacovi et al. [2023a] A. Jacovi, A. Caciularu, O. Goldman, and Y. Goldberg.  Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks.  In H. Bouamor, J. Pino, and K. Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5075–5084, Singapore, Dec. 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.308.  URL https://aclanthology.org/2023.emnlp-main.308.
  * Jacovi et al. [2023b] A. Jacovi, A. Caciularu, J. Herzig, R. Aharoni, B. Bohnet, and M. Geva.  A Comprehensive Evaluation of Tool-Assisted Generation Strategies.  In H. Bouamor, J. Pino, and K. Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 13856–13878, Singapore, Dec. 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.926.  URL https://aclanthology.org/2023.findings-emnlp.926.
  * Jiang et al. [2023] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Jiang et al. [2024] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ , 2024.
  * Jiao et al. [2022] Y. Jiao, S. Li, Y. Xie, M. Zhong, H. Ji, and J. Han.  Open-Vocabulary Argument Role Prediction For Event Extraction.  In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5404–5418, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-emnlp.395.  URL https://aclanthology.org/2022.findings-emnlp.395.
  * Jiao et al. [2023] Y. Jiao, M. Zhong, S. Li, R. Zhao, S. Ouyang, H. Ji, and J. Han.  Instruct and extract: Instruction tuning for on-demand information extraction.  In H. Bouamor, J. Pino, and K. Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 10030–10051, Singapore, Dec. 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.620.  URL https://aclanthology.org/2023.emnlp-main.620.
  * Joshi et al. [2019] M. Joshi, O. Levy, L. Zettlemoyer, and D. Weld.  BERT for coreference resolution: Baselines and analysis.  In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5803–5808, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1588.  URL https://aclanthology.org/D19-1588.
  * Kirstain et al. [2021] Y. Kirstain, O. Ram, and O. Levy.  Coreference resolution without span representations.  In C. Zong, F. Xia, W. Li, and R. Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 14–19, Online, Aug. 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-short.3.  URL https://aclanthology.org/2021.acl-short.3.
  * Lee et al. [2017] K. Lee, L. He, M. Lewis, and L. Zettlemoyer.  End-to-end neural coreference resolution.  In M. Palmer, R. Hwa, and S. Riedel, editors, _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 188–197, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1018.  URL https://aclanthology.org/D17-1018.
  * Li et al. [2023] J. Li, M. Wang, Z. Zheng, and M. Zhang.  LooGLE: Can Long-Context Language Models Understand Long Contexts?  _arXiv preprint arXiv:2311.04939_ , 2023.
  * Liang et al. [2023] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. A. Cosgrove, C. D. Manning, C. Re, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. WANG, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. A. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda.  Holistic evaluation of language models.  _Transactions on Machine Learning Research (TMLR)_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=iO4LZibEqW.  Featured Certification, Expert Certification.
  * Lin [2004] C.-Y. Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * Lin et al. [2018] X. V. Lin, R. Socher, and C. Xiong.  Multi-hop knowledge graph reasoning with reward shaping.  In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3243–3253, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1362.  URL https://aclanthology.org/D18-1362.
  * Lu et al. [2023] D. Lu, S. Ran, J. Tetreault, and A. Jaimes.  Event extraction as question generation and answering.  In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1666–1688, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-short.143.  URL https://aclanthology.org/2023.acl-short.143.
  * Ma et al. [2023] Y. Ma, Y. Cao, Y. Hong, and A. Sun.  Large language model is not a good few-shot information extractor, but a good reranker for hard samples!  In H. Bouamor, J. Pino, and K. Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10572–10601, Singapore, Dec. 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.710.  URL https://aclanthology.org/2023.findings-emnlp.710.
  * Mausam et al. [2012] Mausam, M. Schmitz, S. Soderland, R. Bart, and O. Etzioni.  Open language learning for information extraction.  In J. Tsujii, J. Henderson, and M. Paşca, editors, _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 523–534, Jeju Island, Korea, July 2012. Association for Computational Linguistics.  URL https://aclanthology.org/D12-1048.
  * Mekala et al. [2023] D. Mekala, J. Wolfe, and S. Roy.  ZEROTOP: Zero-shot task-oriented semantic parsing using large language models.  In H. Bouamor, J. Pino, and K. Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5792–5799, Singapore, Dec. 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.354.  URL https://aclanthology.org/2023.emnlp-main.354.
  * Mialon et al. [2023] G. Mialon, R. Dessi, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Roziere, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, E. Grave, Y. LeCun, and T. Scialom.  Augmented language models: a survey.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=jh7wH2AzKK.  Survey Certification.
  * Ni et al. [2022] J. Ni, G. Hernandez Abrego, N. Constant, J. Ma, K. Hall, D. Cer, and Y. Yang.  Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models.  In S. Muresan, P. Nakov, and A. Villavicencio, editors, _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1864–1874, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.146.  URL https://aclanthology.org/2022.findings-acl.146.
  * Parisi et al. [2022] A. Parisi, Y. Zhao, and N. Fiedel.  Talm: Tool augmented language models.  _arXiv preprint arXiv:2205.12255_ , 2022.
  * Pasupat and Liang [2015] P. Pasupat and P. Liang.  Compositional semantic parsing on semi-structured tables.  In C. Zong and M. Strube, editors, _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1470–1480, Beijing, China, July 2015. Association for Computational Linguistics.  doi: 10.3115/v1/P15-1142.  URL https://aclanthology.org/P15-1142.
  * Patil et al. [2023] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_ , 2023.
  * Polo et al. [2024] F. M. Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin.  tinyBenchmarks: evaluating LLMs with fewer examples.  _arXiv preprint arXiv:2402.14992_ , 2024.
  * Reid et al. [2024] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_ , 2024.
  * Schick et al. [2023] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom.  Toolformer: Language Models Can Teach Themselves to Use Tools.  In _Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.  URL https://openreview.net/forum?id=Yacmpz84TH.
  * Slobodkin et al. [2023] A. Slobodkin, A. Caciularu, E. Hirsch, and I. Dagan.  Don’t add, don’t miss: Effective content preserving generation from pre-selected text spans.  In H. Bouamor, J. Pino, and K. Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 12784–12800, Singapore, Dec. 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.852.  URL https://aclanthology.org/2023.findings-emnlp.852.
  * Stanovsky et al. [2018] G. Stanovsky, J. Michael, L. Zettlemoyer, and I. Dagan.  Supervised open information extraction.  In M. Walker, H. Ji, and A. Stent, editors, _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 885–895, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1081.  URL https://aclanthology.org/N18-1081.
  * Team et al. [2023] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Team et al. [2024] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_ , 2024.
  * Touvron et al. [2023] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Wu et al. [2022] X. Wu, J. Zhang, and H. Li.  Text-to-table: A new way of information extraction.  In S. Muresan, P. Nakov, and A. Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 2518–2533. Association for Computational Linguistics, 2022.  doi: 10.18653/V1/2022.ACL-LONG.180.  URL https://doi.org/10.18653/v1/2022.acl-long.180.
  * Yoran et al. [2022] O. Yoran, A. Talmor, and J. Berant.  Turning tables: Generating examples from semi-structured tables for endowing language models with reasoning skills.  In S. Muresan, P. Nakov, and A. Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6016–6031, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.416.  URL https://aclanthology.org/2022.acl-long.416.
  * Yuan et al. [2024] Z. Yuan, K. Wang, S. Zhu, Y. Yuan, J. Zhou, Y. Zhu, and W. Wei.  FinLLMs: A Framework for Financial Reasoning Dataset Generation with Large Language Models.  _CoRR_ , abs/2401.10744, 2024.  doi: 10.48550/ARXIV.2401.10744.  URL https://doi.org/10.48550/arXiv.2401.10744.
  * Zhan and Zhao [2020] J. Zhan and H. Zhao.  Span model for open information extraction on accurate corpus.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 9523–9530. AAAI Press, 2020.  doi: 10.1609/AAAI.V34I05.6497.  URL https://doi.org/10.1609/aaai.v34i05.6497.
  * Zhang et al. [2020] J. Zhang, Y. Zhao, M. Saleh, and P. Liu.  PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization.  In H. D. III and A. Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 11328–11339. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/zhang20ae.html.
  * Zhao et al. [2022] Y. Zhao, Y. Li, C. Li, and R. Zhang.  MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data.  In S. Muresan, P. Nakov, and A. Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6588–6600, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.454.  URL https://aclanthology.org/2022.acl-long.454.
  * Zhou et al. [2023] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. V. Le, and E. H. Chi.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations (ICLR)_ , 2023.  URL https://openreview.net/forum?id=WZH7099tgfM.
  * Zhou et al. [2024] W. Zhou, S. Zhang, Y. Gu, M. Chen, and H. Poon.  UniversalNER: Targeted distillation from large language models for open named entity recognition.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=r65xfUb76p.
  * Zhu et al. [2024] A. Zhu, A. Hwang, L. Dugan, and C. Callison-Burch.  FanOutQA: Multi-Hop, Multi-Document Question Answering for Large Language Models.  _arXiv preprint arXiv:2402.14116_ , 2024.

##  Appendix A Limitations

While our work and the introduced tools specifically target numerical complex
reasoning tasks, they are not designed to address the full spectrum of natural
language processing challenges. Consequently, their application to non-
numerical tasks may not yield optimal results. Another significant limitation
is the sequential use of tools, which can introduce and propagate errors
through the processing stages, potentially compromising the accuracy and
reliability of the final response. This propagation of errors underscores the
need for careful handling and validation at each step to mitigate compounding
inaccuracies. Future work should aim to develop more versatile tools and
methodologies that can handle a broader range of tasks while minimizing the
risk of error propagation.

##  Appendix B License and Intended Use

The TACT benchmark, including templates and instructions, is licensed under
the Creative Commons Attribution NoDerivs 4.0 International License (CC BY ND
4.0). Contributions derived from InstructIE [30], such as texts and tables,
are provided under the terms of this license. Users must assume responsibility
for their use in accordance with the obligations to the creators of
InstructIE. The intended use of these materials is for the improvement and
evaluation of large language models (LLMs), and we assume full responsibility
for any potential violations of rights and confirm adherence to the licensing
agreements associated with the data used in this study.333The data, the full
licence details are all available in
https://huggingface.co/datasets/google/TACT, and the TACT metadata is
available in https://huggingface.co/api/datasets/google/TACT/croissant.

We emphasize the strict use of the TACT dataset exclusively for evaluation
purposes, prohibiting its inclusion in NLP model training datasets to mitigate
potential biases and contamination. We implement measures to prevent data
contamination as outlined by Jacovi et al. [25], and we require that any
future redistribution or use of the data adheres to these same guidelines.
Additionally, redistribution of any part of the dataset is advised against
without robust measures to block web-crawler access. To facilitate the tracing
and management of potential data contamination within web-crawled corpora, a
distinct 64-character identifier string is appended to each dataset instance.

##  Appendix C TACT Data Creation and Guidelines

In this section, we include additional details and material regarding the data
creation process. We provide the official guidelines that were given to the
experts for creating the data (Appendix C.1) along with the concluded data
creation process (Appendix C.2).

###  C.1 Guidelines

The data creation process was guided by a comprehensive set of guidelines,
prepared to equip the participants with the necessary skills and knowledge for
the task. These prerequisites included familiarity with basic Python,
proficiency in the Pandas library, and an understanding of data aggregation
concepts such as sum, mean, and filtering. The original instructions are
depicted in Figure 5.

Figure 5: The data creation guidelines for the TACT Dataset. This figure
presents a comprehensive set of guidelines designed to assist annotators in
extracting and computing numerical aspects from provided text and table data
using the Pandas library. The guidelines include steps for reviewing and
assessing the relevance of data, identifying numerical aspects, formulating
natural language instructions and queries, translating these into precise
Pandas commands, and validating the results. An example is provided to
demonstrate the process, from text and table review to the execution and
verification of the computed result.

###  C.2 Creation Process

In the creation of TACT, we engaged NLP and data science experts (with a PhD
degree specializing in NLP), each with a minimum of four years of experience,
to ensure high-quality data curation. These experts reported that labeling
each example typically required between 16 to 20 minutes. Approximately 70% of
this time was dedicated to carefully reading the provided tables and
accompanying texts, and formulating challenging numerical questions that draw
on the data. The remaining time was allocated to writing the queries,
including one in Pandas and two in natural language, and executing the Pandas
query on the table to verify that the intended results were achieved. This
meticulous process guarantees that our dataset can both challenge LLMs and
accurately reflects realistic scenarios where mathematical reasoning is
essential. The summarized data creation process with an accompanying example,
following the guidelines above and the description in Section 2.2, is
illustrated in Figure 6. The resulting dataset and its components are compared
to InstructIE in Figure 7. [30]

Figure 6: The summarized data creation process for the TACT benchmark. This
illustration outlines the systematic guidelines employed for annotating
numerical data derived from textual and tabular content within the TACT
framework. It elaborates on the sequential steps necessary for annotators to
effectively review text and tables, identify numerical data, formulate and
translate these into natural language instructions and corresponding Pandas
queries, and finally, execute and validate these commands. An example
accompanies the instructions to showcase the entire process from initial
review to the successful execution and verification of the result.
Figure 7: TACT (above) and typical Information Extraction (IE) (below). TACT
includes an instruction, and allows various intermediate outputs before the
answer generation, while IE focuses on table generation. The TACT instruction,
the Pandas command, the query over the table, and the answer, were all created
by NLP experts.

##  Appendix D Additional Experimental Details

In this section, we include additional experimental details, including the
Large Language Models’ (LLMs’) settings we used in our experiments (Appendix
D.1), the few-shot prompting setup (Appendix D.2), as well as the templates we
used (Appendix D.3).

###  D.1 Models and Inference Details

In our experiments, all large language models (LLMs) were configured to
operate with a temperature setting of 0.8. This parameter choice was aimed at
striking an optimal balance between diversity and coherence in the model’s
responses, facilitating more varied yet still plausible outputs for the
generation of the final answer as well as the tools. For tasks involving table
generation and Pandas command generation, we parse the model’s response to
extract the desired output from the generated text. Conversely, for tasks
requiring a numerical answer, we identify and consider the final number in the
generated text as the definitive response.

For the inference of open-source models in our study –
Llama-2-13b-chat444huggingface.co/meta-llama/Llama-2-13b-chat-hf,
Llama-2-70b-chat555huggingface.co/meta-llama/Llama-2-70b-chat-hf [54],
Gemma-7b-it666huggingface.co/google/gemma-1.1-7b-it [53],
Mistral-7b-instruct777huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 [27],
and Mixtral 8x7B888huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 [28],
we utilized 8 H100 GPUs. For the rest of the evaluated models,
Gemini-1.0-Ultra, Gemini-1.0-Pro [52], we used the developers
API999ai.google.dev.

###  D.2 Few-shot Prompting

In the few-shot evaluation settings, prompts were constructed by randomly
choosing few-shot examples out of a pool of 4 demonstrations (that were
manually curated from the validation set of InstructIE). These prompts were
adjusted to include the maximum number of demonstrations that, along with the
query, stayed within the context length limit for all the evaluated models. We
ended up with 4 different prompts for each example, to maintain uniformity in
the number of shots. For zero-shot examples we ran the same prompt, but the
temperature sampling maintained randomness. This approach resulted in
effectively 496 examples for each performance result report in Sections 4 and
5. The robustness of this methodology allowed us to achieve statistically
significant results, demonstrating the reliability and efficacy of our few-
shot evaluation framework in mimicking real-world analytical tasks.

###  D.3 Prompt Templates

Next, we detail the template structures employed for our few-shot prompts.
It’s important to note that we tested multiple templates and settings to
ensure that substantial effort was devoted to reporting the strongest baseline
prompt implementations. Our approach included the creation of prompts designed
for the specific tasks in this paper, each complemented by examples that
demonstrate their goal. The Generic TACT task prompt is depicted in Figure 8,
and the Table Generation task prompt, Table Generation task prompt, and the
overall task prompt are illustrated in Figures 9, 10, and 11, respectively.
Figure 11 matches both the In-context IE and IE as a tool setups, as for the
latter we insert the generated table and Pandas command as the outputs from
the tools in Figures 9 and 10 (see Section 4 for the setups’ details). Note
that for the Chain-of-Thought (CoT) setups, the “Let’s think step-by-step”
suffix was added.

basicstyle=,numbers=left,numberstyle=, xleftmargin=2.8em,xrightmargin=1.5em,
showstringspaces=false, showspaces=false, showtabs=false, tabsize=2,
breaklines=true, flexiblecolumns=true, escapeinside=<@@>,
breakatwhitespace=true

` `

Figure 8: Example of the Generic TACT task prompt provided to our evaluated
models. The prompt consists of basic instructions, several few-shot in-context
examples, and the instance input. The examples demonstrate how to generate the
answer to the instruction, based on given instruction and text. Note that the
examples illustrated are slightly modified from the original TACT, for
simplicity.

basicstyle=,numbers=left,numberstyle=, xleftmargin=2.8em,xrightmargin=1.5em,
showstringspaces=false, showspaces=false, showtabs=false, tabsize=2,
breaklines=true, flexiblecolumns=true, escapeinside=<@@>,
breakatwhitespace=true

` `

Figure 9: Example table generation prompt provided to our evaluated models.
The prompt consists of basic instructions, several few-shot in-context
examples, and the instance input. The examples demonstrate how to create a CSV
table based on given instructions and text. Note that the examples illustrated
are slightly modified from the original TACT, for simplicity.

basicstyle=,numbers=left,numberstyle=, xleftmargin=2.8em,xrightmargin=1.5em,
showstringspaces=false, showspaces=false, showtabs=false, tabsize=2,
breaklines=true, flexiblecolumns=true, escapeinside=<@@>,
breakatwhitespace=true

` `

Figure 10: Example Pandas command generation prompt provided to our evaluated
models. The prompt consists of basic instructions, several few-shot in-context
examples, and the instance input. The examples demonstrate how to create a
Pandas command based on given instructions, text, and table. Note that the
examples illustrated are slightly modified from the original TACT, for
simplicity.

basicstyle=,numbers=left,numberstyle=, xleftmargin=2.8em,xrightmargin=1.5em,
showstringspaces=false, showspaces=false, showtabs=false, tabsize=2,
breaklines=true, flexiblecolumns=true, escapeinside=<@@>,
breakatwhitespace=true

` `

Figure 11: Example of the TACT task prompt provided to our evaluated models,
using tables and Pandas commands. The prompt consists of basic instructions,
several few-shot in-context examples, and the instance input. The examples
demonstrate how to generate the answer to the instruction, based on given
instruction, text, table, and Pandas command, and should generate the computed
answer. Note that the examples illustrated are slightly modified from the
original TACT, for simplicity.