George Datseris

# Agents.jl: A performant and feature-full agent based modelling software of
minimal code complexity

George Datseris11affiliationmark:  Ali R. Vahdati22affiliationmark:  and
Timothy C. DuBois33affiliationmark:  11affiliationmark:  Max Planck Institute
for Meteorology, Germany
22affiliationmark:  Department of Anthropology, University of Zurich, Zurich,
Switzerland
33affiliationmark:  Stockholm Resilience Centre, Stockholm University,
Stockholm, Sweden<EMAIL_ADDRESS>
###### Abstract

Agent based modelling is a simulation method in which autonomous agents
interact with their environment and one another, given a predefined set of
rules. It is an integral method for modelling and simulating complex systems,
such as socio-economic problems. Since agent based models are not described by
simple and concise mathematical equations, code that generates them is
typically complicated, large, and slow. Here we present Agents.jl, a Julia-
based software that provides an ABM analysis platform with minimal code
complexity. We compare our software with some of the most popular ABM software
in other programming languages. We find that Agents.jl is not only the most
performant, but also the least complicated software, providing the same (and
sometimes more) features as the competitors with less input required from the
user. Agents.jl also integrates excellently with the entire Julia ecosystem,
including interactive applications, differential equations, parameter
optimization, and more. This removes any “extensions library” requirement from
Agents.jl, which is paramount in many other tools.

###### keywords:

Agent based modelling, ABM, software, framework, Julia, NetLogo, Mesa, MASON,
complex systems

##  1 Introduction

Many processes in biology, ecology, sociology, and economics are characterized
by interactions between their constituent parts (1; 2; 3; 4; 5; 6; 7; 8; 9).
The large number of interactions leads to numerous possible states within each
system. Such systems, with many interacting components are complex: where a
single component cannot generally determine system behavior. Each component
may have a negligible effect in isolation, but a significant effect when
interacting with other components.

To model and analyze complex systems, bottom-up approaches such as agent-based
simulations are common, and sometimes the only feasible approach. Agent-based
models (ABMs) consist of autonomous agents or individuals that behave
according to a set of predefined rules. The rules specify how agents interact
with one another, as well as with their environment.

ABMs differ from other analytical models, such as differential equations.
Analytical models use variables that characterize the whole system, they are
top-down. ABMs use variables that describe the components of a system, rather
than the behavior of the whole system. A modeler chooses ABM variables based
on an understanding of the system, but not to fit some expectation of outcome.
The outcome emerges (10) from all these lower-level interactions, which are
often nonlinear and cannot be captured by aggregating them. By incorporating
spatial and temporal heterogeneity, each agent may only interact with a local
neighborhood. Such heterogeneity allows for more realistic models that can
show behaviors not captured in top-down approaches (11).

An agent-based modeling framework helps define a general structure for ABMs.
Reducing the amount of code needed to write an ABM, and providing a
standardized model template, makes it easier for model developers to define
models, explore parameters, and collect data; as well as enabling the target
audience to better understand, compare, reproduce, and modify models (Figure
1). This is especially important at present, since increasingly complex models
are being developed in collaboration, where each party focuses on a single
component of the model. A well-defined and simple framework fosters mutual
understanding between collaborators. ABMs can be computationally heavy
programs, and implementing one from scratch that ‘works’ is seldom ‘fast’ the
first time around. A well designed agent-based simulation framework has taken
care of the largest performance bottlenecks one may encounter as much as
possible. Such a framework also separates the tasks of defining a model from
running it, collecting and merging model outputs, as well as analysis of
results.

Figure 1: Flow chart representation of the Agents.jl framework.

We developed an agent-based simulation framework: Agents.jl (12), that
fulfills the aforementioned tasks. Various agent-based simulation frameworks
exists in different programming languages (13). Notable examples include Swarm
(14), NetLogo (15), MASON (16), Repast (17) and Mesa (18) (for a comprehensive
review, see (19)). These frameworks differ in their capabilities, scope,
learning curve, amount of code needed to develop a model, speed of execution,
as well as data collection and visualization features. Our framework is
written purely in the Julia language. This programming language choice brings
advantages over other frameworks: quick and intuitive model development, fast
model execution, and easy integration with many analytical tools in the Julia
ecosystem (removing the need for plugins or extensions). Finally, because
Julia is a general purpose language, coding skills developed while learning
and working with Agents.jl are directly transferable to other situations
outside ABMs (while this is in contrast with e.g. NetLogo, which is mostly a
GUI-based, isolated piece of software).

Here we discuss Agents.jl version 4, with many more features and improvements
over the initial release (20). Specifically, it supports three additional
space types (continuous space, directed graphs and OpenStreetMap), better
visualization functions, more flexible data collection, simpler source code,
automatic parameter exploration, as well as interactive model execution &
visualization. We show the advantages of Agents.jl through a detailed
comparison with three other commonly used frameworks: Mesa, NetLogo and MASON
(Tables LABEL:tab:features and LABEL:tab:performance and comparison section).
We also demonstrate its integration with other Julia packages to create
powerful applications: include differential equations in ABMs, optimize model
parameters and construct novel space types.

##  2 Simulations with Agents.jl

The design of Agents.jl separates a simulation into simple components,
following the philosophy of giving as much freedom to the user as possible,
whilst also minimising the usage complexity. Each of these components
integrates with each other through the help of the Agents.jl API, as
illustrated in Figure 1. In this section we will describe the design of
Agents.jl, by going through a typical workflow of an Agents.jl simulation,
referencing all aspects of Figure 1. Our goal here is not to highlight the
full list of features of Agents.jl (for this please see the comparison section
and the online documentation), but instead to highlight the simplicity of
using Agents.jl.

We will use the Schelling segregation model as an example (a fully detailed
version of this model is available in our documentation, the example herein is
provided solely to outline the basic principles of Agents.jl). Below we will
be including code snippets that implement the Schelling model in Agents.jl.
These code snippets are typically stored in a single script, but could also be
inputted interactively into a Julia console or separated into multiple files.
All code snippets are based on standard, generic Julia functions, as Agents.jl
can be used like any other Julia package. This is in contrast to requiring you
to code in a specific environment (NetLogo), defaulting to using a dedicated
“server” (Mesa) or distributing model files in a binary format (MASON). This
makes models from Agents.jl easier to share and reproduce, but also easier to
integrate with the Julia ecosystem and therefore easier to learn.

###  2.1 Model creation

In Agents.jl, an agent based model is represented by a bundle called
`AgentBasedModel`, that contains all currently alive agents, the space they
reside in, and other model-level parameters. To create such an
`AgentBasedModel`, the user must provide:

  1. 1.

the type of agents the model will contain (but not the agents themselves)

  2. 2.

the properties of the space they can occupy

  3. 3.

the order the agents will activate (optional)

  4. 4.

model-level parameters (optional)

The agent type is defined via a Julia mutable `struct`, which in principle is
a container of arbitrary data (in the case of a mixed-agent models, one
`struct` for each agent type needs to be provided). Such a `struct` must
always have a field `id` and `pos` (for position). For our Schelling model,
the struct looks like:

⬇

mutable struct Schelling <: AbstractAgent

id::Int

pos::Dims{2}

mood::Bool

group::Int

end

Notice that the fields of such a struct (besides the mandatory fields `id,
pos`) can be any possible data structure supported by the Julia language.
Their value can be altered at any point during the simulation. Rather than
writing this out manually, Agents.jl also provides an `@agent` macro that
simplifies this process. Next, the user creates a space structure which can be
populated by agents. Agents.jl currently provides four spaces: grid, graph,
continuous, and open street map. A grid space (for example) is initialized by

⬇

dims = (10, 10)

space = GridSpace(dims; periodic = false)

All spaces have their appropriate set of configuration options.

The final setup step is to choose model-level parameters and agent activation
order. In Agents.jl, agents activate sequentially, according to a dynamically
determined order (arbitrary user-defined function which can include arbitrary
events at arbitrary times). In this example the activation order does not
matter and we use the default (random) activation. After creating a model-
parameter container, we instantiate the `AgentBasedModel` with

⬇

properties = Dict(:min_to_be_happy => 3)

schelling = ABM(Schelling, space;

properties)

(where `ABM` is an alias of `AgentBasedModel`). In the Julia console, the
output of the above command would be:

    AgentBasedModel with 0 agents of type
    Schelling
     space: GridSpace with size (10, 10),
     metric=chebyshev and periodic=false
     scheduler: fastest
     properties: Dict(:min_to_be_happy => 3)

One can populate the model immediately now, by taking advantage of the API of
Agents.jl, and functions like `add_agent!` or `fill_space!`, but we skip this
step here for brevity.

Before actually running a simulation, the user must also define the dynamics
of the model. This is done by providing two functions (which of course
themselves can be composed by simpler parts). First, an agent-stepping
function which decides what happens when each agent is activated, and second,
a model-step function which is called either before or after every scheduled
agent has performed their operations, and acts on the model as a whole (all
agents are still accessible by the model if needed). Both functions are
optional, depending on the simulation’s requirements. The user creates these
two functions by taking advantage of the API of Agents.jl. For example, the
Schelling model has the rules that

  1. 1.

Agents belong to one of two groups (0 or 1).

  2. 2.

If an agent is in a location with at least three neighbors of the same group,
then it is happy.

  3. 3.

If an agent is unhappy, it keeps moving to new locations until it is happy.

This can be implemented with the function shown in listing 1.

⬇

function agent_step!(agent, model)

agent.mood == true && return # do nothing if already happy

minhappy = model.min_to_be_happy

neighbor_positions = nearby_positions(agent, model)

count_neighbors_same_group = 0

# For each neighbor, get group and compare to current agent’s group

# and increment count_neighbors_same_group as appropriately.

for neighbor in nearby_agents(agent, model)

if agent.group == neighbor.group

count_neighbors_same_group += 1

end

end

# After counting the neighbors, decide whether or not to move the agent.

# If count_neighbors_same_group is at least the min_to_be_happy, set the

# mood to true. Otherwise, move the agent to a random position.

if count_neighbors_same_group >= minhappy

agent.mood = true

else

move_agent_single!(agent, model)

end

end

Listing 1: Agent stepping function for the Schelling model

This function uses several functions from the API of Agents.jl. Specifically:

  * •

`model.x` returns the model-level property called `x` (`agent.x` behaves in
the same manner).

  * •

`nearby_agents(agent, model)` returns an iterator of agents nearby the given
`agent`.

  * •

`move_agent_single!(agent, model)` moves the `agent` to a random, but empty
location (if possible).

In a similar manner, one defines a model step function. A full list of
functions available from the API is described in our documentation.

###  2.2 Simulation run, data collection

Once the aforementioned structures and functions have been defined, the model
can be evolved for one step by simply doing

⬇

step!(model, agent_step!)

which internally takes care of scheduling agents, activating them one by one,
and applying the given rules to them. The full form of `step!` is

⬇

step!(model, agent_step!, model_step!, n)

where `n` is either an integer (step for `n` steps), or an arbitrary Julia
function `n(model, s)` with `s` the current step number. In this case,
evolution goes on until `n` returns `true`. Model evolution is in a sense
interactive (since Julia is an interactive language, and all data structures
involved in Agents.jl are mutable). Thus, after stepping the model, the
contained agents and/or model parameters have changed values according to
model rules.

Data collection in Agents.jl is also as simple and as general as constructing
a model. This is accomplished via a two step process. First, the user decides
which data should be collected, which can be any combination of:

  1. 1.

Agent properties

  2. 2.

Aggregated agent properties

  3. 3.

Aggregated agent properties, conditional on a user-defined filter

  4. 4.

Model properties

This is done by providing vectors of appropriate entries for data collection.
For example, if the user wanted to collect data for the property `mood` and
position of the agents, they would define

⬇

adata = [:mood, :pos]

It is also possible to collect arbitrary data from an agent by providing a
function, e.g.

⬇

f(agent) = agent.pos[2]- agent.pos[1]

adata = [:mood, f]

This process works identically for model parameters.

As noted above, it is also possible to aggregate agent data during data
collection. For example, while getting the ‘mood‘ of each individual agent as
data is sometimes desired, other scenarios may only require an aggregated
result. We can achieve this by modifying the `adata` vector above, so that its
entries are `(:value, aggregation_function)` instead of just `:value`. For
example,

⬇

using Statistics # access ‘mean‘

right(a::Schelling) = a.pos[1] > 5

adata = [(:mood, sum),

(f, mean),

(:mood, sum, right)]

would sum the `mood` property (and thus in our example count how many agents
are happy), provide the average value of the `f` function, and finally the
number of agents that are happy, provided they are in the right side of the
space.

Once the user has defined `adata` (and `mdata` for model parameters), they can
simply call

⬇

run!(model, agent_step!, model_step!, n;

adata, mdata)

The `run!` function evolves the model in the same manner as `step!`, but
collects data in addition. It provides the results in the form of a
`DataFrame`: the most common Julia tabular data format. An example output of
the executable version of the Schelling model (from our documentation) is

    |  step | sum_mood | maximum_x |
    | Int64 |    Int64 |     Int64 |
    |-------|----------|-----------|
    |     0 |        0 |        20 |
    |     1 |      219 |        20 |
    |     2 |      278 |        20 |
    |     3 |      299 |        20 |
    |     4 |      312 |        20 |
    |     5 |      313 |        20 |

###  2.3 Visualization

Visualization follows the same principles as data collection. The user
provides a few simple functions which decide how an agent should be
represented. These user-defined functions are then given to the main plotting
function `abm_plot` that is provided by InteractiveDynamics.jl (a package
providing visualization and interactive applications for the packages of the
JuliaDynamics organization).

Using the current Schelling example, we can define two functions for the color
and shape of the agents as follows

⬇

# access plotting functions & backend

using InteractiveDynamics, GLMakie

groupcolor(a) =

ifelse(a.group == 1, :blue, :orange)

groupmarker(a) =

ifelse(a.group == 1, :circle, :rect)

fig, _ abm_plot(model; ac = groupcolor,

am = groupmarker, as = 4)

fig # display figure

The keywords `ac, am, as` decide the agent color, marker type and size
respectively. The output of the above code block (for the documentation
version of the Schelling model) is an image like Figure 2.

Figure 2: An example plot of an implementation of the Schelling segregation
model in Agents.jl.

Changing `abm_plot` to `abm_video` will instead produce a video of the time
evolution of the ABM using same visualization without any extra effort from
the user. If the model posses some property that has a value at every part of
the space (for example, the amount of grass), it is trivial to visualize this
property as a heatmap below the agents simply by providing the argument
`heatarray = :grass` to the plotting functions. Agents.jl will also
automatically animate changes in the property by changing the color of the
heatmap. Making composable animations with multiple sub-plots is also
straightforward, we refer to the “Sugarscape” example in our online
documentation for additional details.

Figure 3: An interactive application of an agent based model. Controls on the
bottom left are created automatically and tune the simulation speed. Red
vertical lines in the timeseries of the collected data denote when the “reset”
button was pressed. Here it was pressed after the slider of the parameter
“minimum to be happy” was changed from 3 to 6, and then to 8.

###  2.4 Interactive applications

By adding only a couple of lines of code to the existing simple interface for
data collection and plotting within Agents.jl, we can immediately explore an
ABM in an interactive application that looks like Figure 3. The data-
collection flags `adata` and `mdata` are re-used to make the timeseries plot
in the right side of the window. The arguments `ac, am, as` of the function
`abm_plot` are re-used as-is. Lastly, the user can choose some model-level
parameters to vary interactively during the simulation, by providing a
dictionary that maps parameter names to value ranges. All in all, the only
extra lines of code the user has to write can be expressed as (we continue
with the Schelling example used throughout the article)

⬇

using InteractiveDynamics, GLMakie

parange = Dict(:min_to_be_happy => 0:8)

alabels = ["happy", "avg. x"]

fig, adf, mdf = abm_data_exploration(

model, agent_step!, dummystep, parange;

ac=groupcolor, am=groupmarker, as=10,

adata, alabels

)

The only _new_ thing the user had to define was the `parange` and `alabels`
variables, where the latter only affects the shown labels of the timeseries.
This is in striking contrast to the user-defined input necessary by, for
example Mesa, which requires much more user input for the same level of
interaction.

##  3 Framework Comparison

ABMs have had a long history, with many tools which enabled their construction
along the way. In Table LABEL:tab:features we compare the software Agents.jl
with three current and popular ABM softwares: Mesa, NetLogo and MASON, to
assess where Agents.jl excels and also may need some future improvement. This
assessment is quantitative where it can be, although many aspects of the
comparison are qualitative by nature. To keep the table as objective as
possible, we only consider features that are directly available from the
exported API of each software, and do not consider things a user could do with
the software with an arbitrary amount of effort, as this is subjective and
also depends on the user’s level of expertise. We categorise our results first
by having either Poor/None (red color), Basic (yellow color), or Good
functionality (green color). If there is a clear category winner, this is
labelled as Current Best (blue color).

Our major goal in this paper is to highlight that Agents.jl is a framework
that is simple and easy-to-use (something hard to showcase in a comparison
table, but already illustrated in the “Simulations with Agents.jl” section).
Regardless, even though Agents.jl is a new-comer ABM software (development
started Dec. 2018 (20)), it becomes clear from Table LABEL:tab:features that
we already match the main functionality of decades-old competitors (all of
which are under active development), most of the time exceeding it, with only
a few aspects being available in the competitors and not in Agents.jl (e.g.
GIS integration).

Table 2: Benchmarks of four model types across four ABM frameworks. Run-times
are normalised against the Agents.jl time, thus a value of 2x means it took
twice as long to complete the benchmark in the respective framework. Lines of
code (LOC) are provided for each model implementation. NetLogo stores
configuration data in the GUI, so we provide the model stepping LOC together
with the complete file LOC in parenthesis.

|  |  |  |
---|---|---|---|---
|  Agents.jl 4.4 |  Mesa 0.8 |  NetLogo 6.2 |  Mason 20.0
| Objective property comparisons.
Continuous Space |  Yes |  Yes |  Yes |  Yes
Graph Space |  Yes, and mutable |  Only undirectional |  Link Agents |
(not a Space) |  Networks (not a Space) |  |  |
Grid Space |  Yes |  Yes (+Hexagonal) |  Yes |  Yes (+Hexagonal, Triangular)
OpenStreetMap Space |  Yes |  No |  No |  No
Dimensionality |  Any1 |  2D |  2D & 3D (separate applications) |  2D & 3D (complicated install for 3D)
License permissiveness |  MIT |  Apache v2.0 |  GPL v2 |  Academic Free License
Mixed-agent |  |  |  |
models |  Yes |  Yes |  Yes |  Yes
Simulation |  |  |  |
termination |  After ‘n’ steps or user-provided boolean condition of model state |  Explicitly written user loop |  Manually by pressing a button on the interface, stop command in code |  When Schedule is empty, or user provided custom finish function
Parameter types |  Anything |  Anything |  Float64, Lists |
Hashtables and Assoc. Arrays in the Table extension |  Anything |  |  |
Modeling and Analysis in the same language |  Yes, Julia v1.5+ |  Yes, Python v3+ |  No |  Yes, Java but designed to work within the console or GUI of the applet
Maximum memory capacity |  Hardware limits |  Hardware limits |  1 GB, manually expanded by increasing JVM heap |  1 GB, manually expanded by increasing JVM heap
Distributed computing2 |  Yes |  No. BatchRunnerMP is only multithreaded |  No. BehaviorSpace is only multithreaded |  Yes
Interop with external libraries |  Yes, also couples to anything in Python / R / C / C++ seamlessly. |  Yes, modular design. |  Partial, via the Extensions API. JVM languages (Scala, Clojure) and Python |  Partial. Extensions in the ‘contrib’ directory. No simple user API
Language ecosystem integration |  By Design. Examples: black box optimization, differential equations |  Any of Python’s analytical tools can be used |  Complex. Must create plugins or use Control API |  Warned against (e.g. Random), provides custom types in place of Java primitives
Browser-based online ABM execution |  No |  No |  Yes (NetLogo Web) |  No
Data collection |  Any chosen parameter / property or function mapped over them. Aggregating and filtered aggregate functions |  Any chosen parameter / property. Aggregating functions. No conditional options |  boolean, number, string and lists of these types. |  Inspectors track & chart any parameter / property. Entire model saved to disk via checkpointing, no custom export
Checkpoints (model IO) |  Yes |  No |  Yes |  Yes
Scheduling |  As added, by property, by type, filtered, random, custom function |  As added, random, staged |  Custom function |  Custom function
Finding nearest neighbors |  Same API for all spaces, custom ranges |  Covers all spaces |  Covers graphs, cardinal directions and city blocks on grids and continuous space |  Cardinal, city block, Von Neumann and radial types. No 3D search in continuous space
Adding agents to space |  Specified position, random, random empty, fill |  Specified position |  Specified position |  Specified position
Automatic agent creation and addition from given attributes |  Yes |  No |  Yes |  No
Moving agents |  Unified API for all space types. Also move along pre-planned routes. |  Unified API for all space types. |  Specify position, only Turtle Agents move |  Specify position, move with mouse in GUI
Killing agents |  Individual, all, specified by function |  Individual, all |  Individual, all, specified by function |  Individual, all
Random number distributions |  Any |  Any |  Normal, Poisson, Exponential, Gamma |  Uniform, Gaussian
Can use COLT library but not recommended |  |  |  |
Agent sample & replacement3 |  Yes |  No |  No |  No
GIS data |  No |  No |  GIS Extension |  GeoMason Extension
Parameter scanning |  Yes |  Yes |  Yes |  Yes
New space types API4 |  Yes |  No |  No |  No
Advanced API for continuous space |  Yes |  No |  No |  No
Path-finding |  Yes |  No |  No |  No
Data collection low-level API |  Yes |  No |  Yes |  Yes, but only via checkpointing
GUI for simulation setup5 |  No |  User implemented |  Yes |  User implemented
| Subjective property comparisons. $*$ LOC: Lines of code
Ease of Installation |  One-click for Julia, one command for package |  One-click for Python, one command for package |  One-click JRE install |
Run jar file |  One-click JRE, install libraries, complex Java3D install |  |  |
Run jar file |  |  |  |
Documentation quality6 |  Short, with tutorials, 15+ executable examples, API listings & integration examples. |  Short, has a tutorial but no hosted run examples online. Space documentation does not exist. |  Extensive, split over website and GitHub wiki (hard to search). Community adoption covers up for it. |  Extensive, over 350 pages in pdf and a developer dump of class properties. Hard to navigate.
Code complexity of the model (see Ref. (19)) |  Simple |  Moderate |  Simple |  High
Complexity of visualization |  Simple API for both plotting & interaction (5 LOC∗) |  Simple API for plotting, complex for interaction |  Simple, function based. Extend agent properties and plot |  Complex API, many LOC

Table 2: Benchmarks of four model types across four ABM frameworks. Run-times are normalised against the Agents.jl time, thus a value of 2x means it took twice as long to complete the benchmark in the respective framework. Lines of code (LOC) are provided for each model implementation. NetLogo stores configuration data in the GUI, so we provide the model stepping LOC together with the complete file LOC in parenthesis. |  |  |  |
---|---|---|---|---
|  Agents.jl 4.4 |  Mesa 0.8 |  NetLogo 6.2 |  Mason 20.0
Flocking (continuous) implementation |  1 (normalised) |  |  |
62 LOC |  26.8x |  |  |
102 LOC |  10.3x |  |  |
82 (689) |  2.1x |  |  |
369 LOC |  |  |  |
Wolf-Sheep-Grass (grid) implementation |  1 (normalised) |  |  |
122 LOC |  31.9x |  |  |
227 LOC |  10.3x |  |  |
137 (871) LOC |  No Implementation Available |  |  |
Forest Fire (grid) implementation |  1 (normalised) |  |  |
23 LOC |  125.6x |  |  |
35 LOC |  53.0x |  |  |
43 (545) LOC |  No Implementation Available |  |  |
Schelling (grid) implementation |  1 (normalised) |  |  |
31 LOC |  24.9x |  |  |
56 LOC |  8.0x |  |  |
60 (743) LOC |  14.3x |  |  |
248 LOC |  |  |  |

Clarifications mapping to the superscript numbers in Table LABEL:tab:features
are given below:

1\. We score Dimensionalty “Current Best” for Agents.jl since it provides true
N-dimensional spaces with higher order search functions and grouping
utilities. In addition, the Battle Royale example in our documentation
showcases a novel application of this capability. An N-dimensional space, with
a 2D spatial grid and the higher order dimensions representing agent
categories. While agent categories can be represented as standard agent
properties, using additional “spatial” dimensions for them instead allows
finding nearest neighbors along these dimensions, which would become
cumbersome to do via the property approach.

2\. Julia is known to provide tools for easily achieving excellent performance
through parallelization. Agents.jl contains a documentation page dedicated to
model performance and parallelization tips instructing users to appropriate
sources. Furthermore we provide automatic distributed computing (i.e. across
multiple CPUs) for ensemble simulations or parameter scanning. Notice that in-
model parallelization is outside the control of Agents.jl as it depends on the
actual model operations. This stems from the nature of ABMs, where same-
memory-location modifications are done all the time by killing and/or adding
agents and is an active concern for all ABM frameworks.

3\. Agent sampling is one of the many unique features of Agents.jl. It is the
ability to select randomised subsets of the model population based on certain
properties. Useful in biological applications (for example).

4\. Our design of space types allows fundamentally new spaces to be created
with relatively low effort. Specifically, a new space can be created by
defining a new Julia `struct` and extending only 5 methods (i.e. defining 5
functions). The resulting space then integrates with all of the Agents.jl API
as any other space would. For example, the entire implementation of our graph
space is only 75 lines of code.

5\. Notice that the lack of a GUI present for simulation set up in Agents.jl
is not really a lack of feature but rather a more natural consequence of the
integration of Agents.jl with the entire Julia ecosystem. Julia is run in
several diverse settings, from the console (REPL), to many standard IDEs like
VSCode, to Jupyter and Pluto notebooks. It is also extremely straightforward
to run any of these on a remote server via ssh. A GUI would not be able to run
in any of these scenarios, which is in fact a downside of NetLogo and MASON
(headless scripts accompany both we concede, but are considered an
afterthought and have little support or flexibility). The fact that Agents.jl
run in any environment that Julia can run, allows it integrate in a
straightforward manner into larger decision-support systems as only a sub-
component of the whole process.

6\. While the actual documentation of NetLogo is not on par with that of
Agents.jl (according to our comparison), NetLogo has several associated
external resources, like books introducing ABMs based on NetLogo as well as
educational courses. These resources however cannot be considered part of the
software (which is what we compare here), as they are not under its license.
Nevertheless, they make it easier to learn.

Table LABEL:tab:performance provides benchmarks for four standard agent based
models:

  * •

Flocking, a `ContinuousSpace` model, chosen over other models to include a
MASON benchmark. Agents must move in accordance with social rules over the
space.

  * •

Wolf Sheep Grass, a `GridSpace` model, which requires agents to be added,
removed and moved; as well as identify properties of neighbouring positions.

  * •

Forest Fire, provides comparisons for cellular automata type ABMs (i.e. when
agents do not move and every location in space contains a single agent).

  * •

Schelling, an additional `GridSpace` model to compare with MASON. Simpler
rules than Wolf Sheep Grass.

The results are characterised in two ways: how long it took each model to
perform the same scenario (initial conditions, grid size, run length etc. are
the same across all frameworks), and how many lines of code (LOC) it took to
describe each model and its dynamics. We use this result as a metric (albeit
imperfect) to represent the complexity of learning and working with a
framework.

Time taken is presented in normalised units, measured against the runtime of
Agents.jl. In other words: the results do not depend on any computers specific
hardware.

For LOC, we use the following convention: code is formatted using standard
practices & linting for the associated language. Documentation strings and in-
line comments (residing on lines of their own) are discarded, as well as any
benchmark infrastructure. NetLogo is assigned two values since its files have
a code base section and an encoding of the GUI. Since many parameters live in
the GUI, we must take this into account. Thus 375 (785) in a NetLogo count
means 375 lines in the code section, 785 lines total in the file. An
additional complication to this value in NetLogo is that it stores plotting
information (colours, shapes, sizes) as agent properties, and as such the
number outside of the bracket may be slightly inflated.

Analysing performance between the frameworks was difficult, since each system
implements example models in their own unique manner. This highlights the lack
of standardised bench-marking models, perhaps stemming from the lack of
communication between the ABM communities. Since the Wolf-Sheep-Grass model
requires frameworks to utilise most of the common machinery (multiple agent
types, adding, deleting and moving agents, etc.), we would appreciate if the
MASON community (and ABM communities as a whole) could provide an
implementation of this model for future comparisons. From the analysis we
present here, Agents.jl is a clear winner in performance, most of the time by
an order of magnitude. Since typical ABM simulations can cover hours of run
time, even a 2x speed up is a large gain.

JuliaDynamics hosts the ABM Framework Comparisons repository (21) for anyone
who wishes to validate these results, improve implementations or add new
comparisons.

##  4 Ecosystem interaction examples

In this section we want to showcase how easily Agents.jl interacts with the
rest of the Julia ecosystem. This is possible for two reasons: first, the
minimal design of Agents.jl, as well as the support it provides for low-level
interfaces. Second, the design of the core of the Julia language itself, which
allows straightforward inter-package communication. Notice that the examples
we showcase here have fully detailed documentation online, explaining
precisely how they work. Our goal here is to highlight how easy it is for
Agents.jl to “communicate” with other Julia packages, removing any need for a
plugin or extension ecosystem and thus making the user experience smoother.

###  4.1 ODEs with DifferentialEquations.jl

Coupling a set of differential equations (DE) to an ABM has historically led
to a complex set of validation and sensitivity tests (22), which stem from
discretizing a DE in some manner (predominantly via the forward Euler method)
to conform with the step function of the ABM framework. The tests outlined in
Martin et al. (22) concerning sensitivity can be handled automatically by
integrating Agents.jl with DifferentialEquations.jl (23).

To demonstrate this, our documentation (under the “Ecosystem Integration”
section) describes a small fishery model where fish stocks are managed on a
yearly basis. A number of fishers, with differing competence at catching fish,
work in a common catchment. This is managed by some agency that makes sure the
catchment is not over-fished. The fish population in the catchment is modelled
via a logistic function

$\frac{\mathrm{d}s}{\mathrm{d}t}=s\left(1-\frac{s}{120}\right)-h,$ (1)

where $s$ is the fish stock with some maximum carry capacity ($120$ here) and
a harvest rate $h$.

The status-quo method to implement such a hybrid dynamical system-agent based
model is to discretize this equation to

$s_{t+1}=s_{t}+s_{t}(1-s_{t}/120)-h$ (2)

with a timestep of $1$ normalised unit initially. To validate this result, it
would be important to undertake a step size analysis as a bare minimum, and to
be thorough, use a scheme such as the one outlined in Martin et al. (22).
Thankfully, the issues caused by discretization do not need to exist within an
Agents.jl model, as we can couple our model with a continuous implementation
of the DE from DifferentialEquations.jl.

We can see in Figure LABEL:fig:diffeq that a forward Euler method with no step
size optimisation performed (or further sensitivity checks as discussed above)
will yield an average discrepancy of 30 fish. Integration with
DifferentialEquations.jl has provided us with a stable, valid solution—with an
added bonus of efficiency. Since the chosen solver (in this case `Tsit5`)
required less allocations and computations to obtain the result, we achieved a
6x speedup for this model.

###  4.2 Agents on Open Street Maps

With our new space type API, building ABMs on novel spaces is no-longer a
months-long development process. An `OpenStreetMapSpace` has been introduced
in Agents.jl 4.0, which is a continuous space that constrains agents onto
roads and streets of any provided real-world map obtained from Open Street
Map. We leverage the OpenStretMapX.jl package and build methods specific to
agent navigation and neighbor searching, which culminates in incredibly
simple, yet powerful map based models.

Our Zombie Outbreak example (see documentation online) explains how a simple
agent constructor:

⬇

@agent Zombie OSMAgent begin

infected::Bool

end

coupled with 8 lines of movement dynamics can depict a city in chaos after a
zombie infection (Figure LABEL:fig:zombies).

###  4.3 Parameter optimization

Describing the logic of an ABM is usually not complicated, even when ABMs have
a large number of heterogeneous agents (24). However, exploring the effect of
model parameters has the possibility to become infeasible. ABMs are often
computationally more expensive than analytical models, and brute force
algorithms do not suit parameter exploration since the size of the parameter
space of a simple model with 10 parameters and 10 possible values per
parameter is $10^{10}$. Even if each simulation takes only one second,
exploring the entire parameter space would take more than 300 years.
Additionally, each parameter setting needs to be run multiple times and an
average taken, since most ABMs are stochastic. Machine learning algorithms
handle the large parameter space by differentiation. ABMs, however, are not
(universally) differentiable.

We must resort to optimization strategies for non-differentiable functions.
One such strategy is evolutionary algorithms (25). They are inspired from how
living organisms evolve in a constantly changing environment and with large
parameter spaces, similar to how ABMs often need to explore large parameter
spaces.

The Agents.jl documentation demonstrates how an epidemiological model can be
optimized with evolutionary algorithms using the BlackBoxOptim.jl package. We
optimize a number of parameters of a SIR111SIR stands for Susceptible-
Infected-Recovered and is a simple model for infection dynamics commonly used
in ABMs. model explicitly accounting for multiple cities/regions. Specifically
we tune transmission rates, death rate, migration rate, infection and
detection times, and reinfection probability to minimize the number of
infections. We note that to optimize the ABM, the simulation code does not
need to be changed. All we need is a cost function that takes model parameters
as input, runs the model one or more times, and returns one or more numbers as
the objectives that need to be minimized (here, the number of infected
individuals and the negative of the number of individuals). With our initial
values, 94% of the population gets the infection. The optimization finds that
reducing the transmission rate is enough for reducing death rate and
infections to 0.3% and 0.04% of the population, respectively. That is despite
increasing the reinfection probability, migration rate, and death rate.
Accessibility of optimization tools in the Julia ecosystem and their easy
integration with Agents.jl makes ABM analysis much easier.

##  5 Conclusions and Future Work

We have presented an overview of the capabilities of Agents.jl, showing the
simplicity and power of this framework compared to long-established frameworks
(NetLogo, MASON), as well as contemporaries (Mesa). From our perspective, the
biggest take-away of this paper is that Agents.jl is a framework that is
simple to use, requiring small amount of written code from the user and
overall easy to learn. Despite this, our comparison shows that Agents.jl
always exceeds other frameworks in performance, and often also in capability.
An added bonus is how simple it is for a user to incorporate other parts of
the already large, and constantly evolving, Julia ecosystem into their model.
With this, we hope to motivate more users to try out Agents.jl, which will
enable them to extend the frontier of possibilities in the world of ABMs, due
to faster prototyping and faster code execution.

Several possible future directions already exist for Agents.jl, some planned
by the developers and others requested by users. A useful new feature would be
crowd dynamics and obstacle avoidance, as well as a new type of grid space
based on hexagonal grids, rather than the existing rectangular. The ODD
protocol (26) is a formal description of ABMs, aiming to make models more
understandable and less subject to criticism for being irreproducible. Whilst
Agents.jl models are reproducible by design, a planned feature will leverage
Julia’s strong macro language capability to pre-fill many aspects of the
standard ODD template. Integration into the greater Julia ecosystem is useful
to highlight as well: one upcoming integration will target Bayesian inference
for decision making. A performance issue Agents.jl currently has is regarding
multi-agent models (even though it is still the fastest software in this
regard). In the future we plan to re-work our multi-agent internals from
scratch to lead to more performant, but not more complicated, design.

Given that Agents.jl is an open source project, we welcome new users to add to
the wish-list of functionalities by opening a new issue in our GitHub
repository, or even better: to contribute new features via a pull request.

We acknowledge all users of Agents.jl that contributed in the form of
reporting bugs, suggesting new features, and even contributing code directly
via pull requests.

T.C.D. acknowledges funding from the EU project LimnoScenES (2017–2018 Belmont
Forum and BiodivERsA joint call under the BiodivScen ERA-Net COFUND with
funding from the Swedish Research Council FORMAS).

###  Author Contribution Statement

G.D. provided direction to the team, refactored and optimised much of the
current codebase, oversaw critical design decisions regarding the
representation of spaces and plotting, and served as lead developer from v2.0
until v4.0. A.R.D. is the original author of Agents.jl and has been
continuously active in development since. T.C.D. is the current lead developer
of Agents.jl, has been active in development since version v2.0, implementing
and optimising a large portion of the framework and contributing several new
features and examples.

A.R.D. drafted the introduction section, G.D. drafted the usage section and
the outline of the comparison table. Mesa comparisons were compiled by A.R.D.,
all other frameworks by T.C.D. Benchmarks were run and listed by T.C.D. T.C.D.
and A.R.D. wrote the Ecosystem Integration. All authors contributed to draft
revisions and editing.

## References

  * (1) V. Grimm and S. F. Railsback, “Agent-Based Models in Ecology: Patterns and Alternative Theories of Adaptive Behaviour,” in Agent-Based Computational Modelling (F. C. Billari, T. Fent, A. Prskawetz, and J. Scheffran, eds.), pp. 139–152, Heidelberg: Physica-Verlag, 2006.
  * (2) I. Politopoulos, “Review and analysis of agent-based models in biology,” University of Liverpool.—2007, no. September, pp. 1–14, 2007.
  * (3) J. D. Farmer and D. Foley, “The economy needs agent-based modelling,” Nature, vol. 460, pp. 685–686, Aug. 2009.
  * (4) S. Heckbert, T. Baynes, and A. Reeson, “Agent-based modeling in ecological economics: Agent-based modeling in ecological economics,” Annals of the New York Academy of Sciences, vol. 1185, pp. 39–53, Jan. 2010.
  * (5) A. J. McLane, C. Semeniuk, G. J. McDermid, and D. J. Marceau, “The role of agent-based models in wildlife ecology and management,” Ecological Modelling, vol. 222, pp. 1544–1556, Apr. 2011.
  * (6) T. Lekvam, B. Gambäck, and L. Bungum, “Agent-based modeling of language evolution,” in Proceedings of the 5th Workshop on Cognitive Aspects of Computational Language Learning (CogACLL), (Gothenburg, Sweden), pp. 49–54, Association for Computational Linguistics, 2014.
  * (7) J. Schulze, B. Müller, J. Groeneveld, and V. Grimm, “Agent-Based Modelling of Social-Ecological Systems: Achievements, Challenges, and a Way Forward,” Journal of Artificial Societies and Social Simulation, vol. 20, no. 2, p. 8, 2017.
  * (8) Ş. Bora and S. Emek, “Agent-Based Modeling and Simulation of Biological Systems,” in Modeling and Computer Simulation (D. Cvetković, ed.), IntechOpen, Apr. 2019.
  * (9) M. Lippe, M. Bithell, N. Gotts, D. Natalini, P. Barbrook-Johnson, C. Giupponi, M. Hallier, G. J. Hofstede, C. Le Page, R. B. Matthews, M. Schlüter, P. Smith, A. Teglio, and K. Thellmann, “Using agent-based modelling to simulate social-ecological systems across scales,” GeoInformatica, vol. 23, pp. 269–298, Apr. 2019.
  * (10) J. O. Dada and P. Mendes, “Multi-scale modelling and simulation in systems biology,” Integrative Biology, vol. 3, no. 2, p. 86, 2011.
  * (11) S. F. Railsback, Agent-Based and Individual-Based Modeling: A Practical Introduction.  Princeton, NJ: Princeton University Press, 2nd edition ed., 2019.
  * (12) G. Datseris, A. Vahdati, and T. C. DuBois, “Agents.jl online repository and documentation.” https://github.com/JuliaDynamics/Agents.jl.
  * (13) S. F. Railsback, S. L. Lytinen, and S. K. Jackson, “Agent-based Simulation Platforms: Review and Development Recommendations,” SIMULATION, vol. 82, pp. 609–623, Sept. 2006.
  * (14) H. Iba, Agent-Based Modeling and Simulation with Swarm.  Chapman and Hall/CRC, zeroth ed., June 2013.
  * (15) L. Wilensky, “Netlogo.” Center for Connected Learning and Computer-Based Modeling, Northwestern University, 1999.
  * (16) S. Luke, C. Cioffi-Revilla, L. Panait, K. Sullivan, and G. Balan, “MASON: A Multiagent Simulation Environment,” SIMULATION, vol. 81, pp. 517–527, July 2005.
  * (17) M. J. North, N. T. Collier, J. Ozik, E. R. Tatara, C. M. Macal, M. Bragen, and P. Sydelko, “Complex adaptive systems modeling with Repast Simphony,” Complex Adaptive Systems Modeling, vol. 1, p. 3, Dec. 2013.
  * (18) D. Masad and J. Kazil, “Mesa: An Agent-Based Modeling Framework,” in Python in Science Conference, (Austin, Texas), pp. 51–58, 2015.
  * (19) S. Abar, G. K. Theodoropoulos, P. Lemarinier, and G. M. O’Hare, “Agent Based Modelling and Simulation tools: A review of the state-of-art software,” Computer Science Review, vol. 24, pp. 13–33, May 2017.
  * (20) A. Vahdati, “Agents.jl: Agent-based modeling framework in Julia,” Journal of Open Source Software, vol. 4, p. 1611, Oct. 2019.
  * (21) T. C. DuBois, “JuliaDynamics ABM framework comparison repository.” https://github.com/JuliaDynamics/ABM_Framework_Comparisons.
  * (22) R. Martin and M. Schlüter, “Combining system dynamics and agent-based modeling to analyze social-ecological interactions—an example from modeling restoration of a shallow lake,” Frontiers in Environmental Science, vol. 3, Oct. 2015.
  * (23) C. Rackauckas and Q. Nie, “Differentialequations. jl–a performant and feature-rich ecosystem for solving differential equations in julia,” Journal of Open Research Software, vol. 5, no. 1, 2017.
  * (24) T. Terano, H. Deguchi, and K. Takadama, eds., Meeting the Challenge of Social Problems via Agent-Based Simulation.  Springer Japan, 2003.
  * (25) P. A. Vikhar, “Evolutionary algorithms: A critical review and its future prospects,” in 2016 International conference on global trends in signal processing, information computing and communication (ICGTSPICC), pp. 261–265, IEEE, 2016.
  * (26) V. Grimm, S. F. Railsback, C. E. Vincenot, U. Berger, C. Gallagher, D. L. DeAngelis, B. Edmonds, J. Ge, J. Giske, J. Groeneveld, A. S. Johnston, A. Milles, J. Nabe-Nielsen, J. G. Polhill, V. Radchuk, M.-S. Rohwäder, R. A. Stillman, J. C. Thiele, and D. Ayllón, “The odd protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism,” Journal of Artificial Societies and Social Simulation, vol. 23, no. 2, p. 7, 2020.


# Fine-tuning a Subtle Parsing Distinction Using a Probabilistic Decision
Tree: the Case of Postnominal _"that"_ in Noun Complement Clauses vs. Relative
Clauses

Author 1  Author n
Address line
…
Address line  Zineddine Tighidet
Université Paris Cité
F-75013 Paris, France
<EMAIL_ADDRESS>
&Nicolas Ballier
CLILLAC-ARP and LLF, Université Paris Cité
& CNRS F-75013 Paris France
<EMAIL_ADDRESS>

###### Abstract

In this paper we investigated two different methods to parse relative and noun
complement clauses in English and resorted to distinct tags for their
corresponding that as a relative pronoun and as a complementizer. We used an
algorithm to relabel a corpus parsed with the GUM Treebank using Universal
Dependency. Our second experiment consisted in using TreeTagger, a
Probabilistic Decision Tree, to learn the distinction between the two
complement and relative uses of postnominal _“that”_. We investigated the
effect of the training set size on TreeTagger accuracy and how representative
the GUM Treebank files are for the two structures under scrutiny. We discussed
some of the linguistic and structural tenets of the learnability of this
distinction.

Fine-tuning a Subtle Parsing Distinction Using a Probabilistic Decision Tree:
the Case of Postnominal _"that"_ in Noun Complement Clauses vs. Relative
Clauses

Zineddine Tighidet Université Paris Cité F-75013 Paris, France
<EMAIL_ADDRESS>Nicolas Ballier CLILLAC-ARP and LLF, Université
Paris Cité & CNRS F-75013 Paris France<EMAIL_ADDRESS>

##  1 Introduction

English has relative clauses (the man that I saw) and noun complement clauses
(the fact that I saw a man) that may have similar surface representations
(often the definite article, a noun, often immediately followed by that) but
different structural properties Ballier (2004). For POS-tagging systems based
on trigrams, the distinction between these constructions can be challenging,
not to mention the case of ambiguous sentences such as "the suggestion that he
was advancing was ridiculous" Huddleston (1984).This is an issue for
information retrieval, as conceptual argumentation makes heavy uses of noun
complement clauses Ballier (2007), the governors of these noun complement
clauses being "shell nouns" Schmid (2000). Complement taking nouns Bowen
(2005) are crucial for the expression of stance Charles (2007) in documents,
which is why this distinction may matter more than is usually assumed.

The Penn Treebank Marcus et al. (1993) tagset Santorini (1990) does not make
strict distinctions between the part-of-speech (POS) tag of _"that"_ when used
as a relative pronoun (WDT) or when used as a conjunction when complementizing
nouns: it uses IN when complementizing verbs or nouns. Even though the CLAWS8
111CLAWS, the Constituent Likelihood Automatic Word-tagging System, is the
name of the tagset and of the POS-tagging software for English text, CLAWS
Garside (1987). University Centre for Computer Corpus Research on Language
(1995-2004) tagset encodes this distinction with the _CST_ 222 _”that”_ as a
conjunction and _WPR_ 333 _”that”_ as a relative pronoun tags, this tagger is
not free and remains the property of the University Centre for Computer Corpus
Research on Language (UCREL). To the best of our knowledge, the precision and
recall of these two tags (and their corresponding syntactic structures) have
not been reported.

Admitting POS-tagging systems have reached an overall satisfactory precision
rate for standard English tagsets, we claim that this is not necessarily the
case for tags that reflect such a subtle distinction which may have very
similar surface representations. Discussing such POS-tags involves parsing
issues of the _that_ -clause that follows the noun. Our research question is
mostly based on the ability of a system to identify noun complement clauses as
apposed to (restrictive) relative clauses, but this can be addressed by
analysing dependency relation labels (parsing) or distinct tags that encode
this syntactic distinction (POS-tagging). We present the two strategies in two
experiments, exploring whether such specific Universal Dependency labels can
be learnt. In this paper, we only investigate overt complementizers as we are
also investigating how that is tagged and do take into account noun complement
clauses with zero complementizer, like in the example Plus the fact I’m a
coward from the British National Corpus Consortium et al. (2007).

The rest of the paper is structured as follows. Section 2 details the data we
used for our experiments. Section 3 analyses the Universal Dependency (UD) GUM
Treebank for English in terms of precision for the dependency labels of these
two structures as well as their distribution across the training, testing and
development sets. We describe an experiment replicating one of the specific
features of the GUM Treebank. Section 4 details an experiment based on
algorithm adapting the UD annotation generated with GUM. Section 5 explains
how Treetagger can be used to learn distinct tags for that used as a relative
pronoun (WPR) or as a complementizer (CST). Section 6 discusses our results
and section 7 outlines our future research.

##  2 Material and Methods

###  2.1 Test Sets

For our validation procedure, we used two test sets NCCtest and RCtest, one
including 194 noun complement clauses (NCC), the other one included 189
relative clauses (RC). As language is complex, some sentences included other
syntactic realisations, and a couple of "distractors" representative of the
alternate structure were therefore included in our two test sets. We specify
in Table 3 the expected (gold) label counts for each test set. Two annotators
agreed on these gold labels of these two test sets ($\kappa=1$).

###  2.2 Brown Corpus

We used the Brown corpus Kucera et al. (1967), which is rather small with its
1 M tokens by contemporary standards, but well-balanced and freely available.
Its current distribution in the NLTK python library Bird (2006) has been POS-
tagged with the Penn Treebank, this is the substrate we used for our re-
annotation experiment with TreeTagger. Treetagger is a probabilistic tagger
which uses decision trees for probability transitions, which is robust for its
retraining and claims accuracy above 96 % Schmid (1994).

###  2.3 Universal Dependency Annotation with UDPipe

UDPipe Straka (2018) is a pipeline that takes as input a text file and renders
a CoNLL-U444https://universaldependencies.org/format.html file which contains
the language-specific part-of-speech tag (_XPOS_), lemma or stem, the _DEPREL_
(universal dependency relation) etc.
A file annotated in Universal Dependency contains among other columns the
_XPOS_ (part of speech) for each token and the dependency relation,
_acl:relcl_ for relative clauses and (just) _acl_ for noun complement clauses,
though this more general category (_acl_ corresponds to clausal modifier of
noun, adnominal clause) also includes non-finite clause.

Clausal modifier of noun (_acl_)
_acl_ stands for finite and non-finite clauses that modify a noun. The
governor (head) of the _acl_ dependency relation is the noun that is modified,
and the dependent is the predicate of the clause that modifies the noun. In
Figure 1 the finite clause “as he sees them” modifies the noun “the issues”.

Figure 1: Example of clause modifier of noun (_acl_).

As evidenced by this example taken from the UD documentation, acl is a label
that encompasses more than that noun complement clauses.

Relative clause modifier (acl:relcl)
A relative clause modifier of a noun is a clause that modifies the antecedent.
The _acl:relcl_ relation points from the governor (the antecedent) head of the
modified nominal to the dependent (verb) of the relative clause. In Figure 2
the relative clause “which you bought” modifies the nominal “the book”.

Figure 2: Example of relative clause modifier (_acl:relcl_).

Several treebanks for English are available
555https://universaldependencies.org/treebanks/en-comparison.html for the
Universal dependency annotation McDonald et al. (2013). We focused on the GUM
Treebank Zeldes (2017), based on the Georgetown University Multilayer (GUM)
Corpus 666https://gucorpling.org/gum/ as its CoNLL-U format 777an adaptation
of the CoNLL-X format, see Buchholz and Marsi (2006),
https://universaldependencies.org/format.html contains a specific column that
reports the dependency relation and the governor. Our next section analyses
the accuracy of these two tags when labelling noun complement clauses and
relative clauses in the development (DEV), training (TRAIN) and testing (TEST)
sets of the GUM Treebank based on the GUM corpus Levine, Lauren and Zeldes,
Amir (2017).

##  3 Revisiting the GUM Treebank

We noticed some debatable annotations for some cases where ellipsed such and
so led some that-clauses expressing consequence to be labelled as acl as in
"As a result, wikiHow is still at the size that every editor eventually gets
to know other editors". We computed the proportion of Relative Clauses (RC) in
relation to noun complement clauses (NCC).

###  3.1 Frequency of RC and NCC in the GUM Treebank

deprel | Train | Dev | Test
---|---|---|---
acl:that | 65 (0.513) | 13 (0.65) | 14 (0.69)
acl:relcl | 1419 | 258 | 216
| (11.21) | (12.92) | (10.70)
Table 1: Frequency of "acl:relcl" and "acl:that" in the GUM Treebank files raw
(normalized per 1000 tokens).

As can be seen in Table 1, there are at least 15 times more relative clauses
(RC) than noun complement clauses (NCC) in the GUM Treebank.

One of the benefits of the GUM Treebank is that it contains extra information,
the ninth column conflates the dependency relation (acl) and that for noun
complement clauses, we have tried to exploit this acl:that tag by building a
UDPipe model based on this treebank and by trying to recapture this
information by an algorithm.

###  3.2 Replicating the GUM Ninth Column

In the ninth column of the GUM corpus, we were specifically interested in the
"acl:relcl" and "acl:that" annotations to improve the detection of noun
complement clauses, since the standard deprel (dependency relation) column
only provides the "acl" label and does not distinguish between finite and non
finite uses of adnominal clauses. We trained a UDPipe model using the
training, development and test sets of the GUM Treebank on
Github888https://github.com/UniversalDependencies/UD_EnglishGUM. However, once
we applied the model on the same unannotated corpus, the ninth column was
empty. It seems that UDPipe only captures the standard columns of the
treebanks.

###  3.3 Emulating the Ninth column

We were therefore interested in reconstructing this column by implementing a
heuristic. Once the acl:relcl have been copied from the deprel column, the
algorithm consists in exploiting the seventh (Head of the current word) and
eighth (Universal dependency relation to the HEAD) columns such that:

Algorithm 1 : Heuristic to emulate acl:that labels in the ninth column

for each $sentence\in corpus$ do

for each $token\in sentence$ do

  1. 1.

Combine the seventh and eighth columns of the $token$ that were generated by
the previously trained UDPipe model.

  2. 2.

If "that" is right after the word to which the seventh column of the $token$
points to, then add "that" to the ninth column.

end for

end for

##  4 Learning to tag with TreeTagger

This retagging experiment Gaillat et al. (2014) relies on the ability of
TreeTagger Schmid (1994) to be used not only as a POS-tagger but as a tool
which can be trained to learn how to tag, provided a specific tagset and
sample data are provided. We used samples from the Brown corpus in its NLTK
distribution and modified the Penn Treebank tagset to distinguish that as WPR
(relative pronoun) and that as CST (complementizer). In the learning phase,
TreeTagger sees a vocabulary file and tokens associated to their tags and
generates a .par model file to be used for POS-tagging. This section describes
how we modified the tags to train the system 999The Python implementation is
available in this GitHub repository: https://github.com/Zineddine-
Tighidet/Relative-Complement-That-Annotator. After the annotation of the Brown
corpus by UDPipe, a heuristic was applied on the results in order to introduce
the _WPR_ and _CST_ tags which are not previously used in the tagset. To do
that, the _DEPREL_ label was used, so our method assumes that the UDPipe
trained with the English GUM corpus provides a sufficiently correct _DEPREL_
label for noun complement clauses:

Algorithm 2 : Heuristic for Brown re-annotation

for each $sentence\in corpus$ do

for each $token\in sentence$ do

  * •

If the $token$ is a verb (i.e. XPOS = VB) and is a clausal modifier of noun
(i.e. DEPREL = acl) then go steps before that $token$ to see if there is any
_"that"_ , if so, label it as _CST_.

  * •

If the $token$ is a verb (i.e. XPOS = VB) and is part of a relative clause
(i.e. DEPREL = acl:relcl) then go steps before that $token$ to see if there is
any _"that"_ , if so, label it as _WPR_.

end for

end for

The aim of this experiment is to see how the TreeTagger accuracy increases as
a function of the training set size. To do this, the TreeTagger received
different proportions of a training set as input. To be more specific, there
are 500 training files representing the annotated Brown corpus, for the first
training the first 10 files were used, and then the 30, 100, 200, 300, 400 and
finally the 500 training files. For each training a _.par_ file that
corresponds to the model was returned.

##  5 Results

###  5.1 Emulating the Ninth column

To assess this algorithm that selects only that\- (finite) clauses among the
acl clauses, we tested it with the GUM treebank, comparing our results in our
reconstructed column with the original data. The heuristic gave good results
for the annotations of relative clauses "acl:relcl" with an accuracy that
exceeds 90% (see table 2).

deps column | Train | Dev | Test
---|---|---|---
acl:that | 0.78 | 0.76 | 0.71
acl:relcl | 0.92 | 0.92 | 0.94
Table 2: Accuracy of acl:relcl and acl:that annotations in the "deps" column
recreated by combining the "head" and "deprel" columns for each of the GUM
Treebank files.

Nevertheless, the algorithm works less well for "acl:that", this is partly due
to some coordinated NCC clauses and to multi-word-units (like quid pro quo).

###  5.2 Re-annotating with TreeTagger

We used our specifically designed testing files that contain respectively 189
_"that"_ as _WPR_ and 194 _"that"_ as _CST_. The first one named RCtest
(Relative Clause) was used to compute the accuracy for _WPR_ and the second
one named NCCtest (Noun Complement Clause) for _CST_ (see Figure 3 and 4). We
used these specific files because they are manually annotated and each one of
them contains a majority of the two tags we are interested in, which makes it
convenient for our experiments.

Figure 3: TreeTagger accuracy curve for _WPR_ tag (computed using the RCtest
data). Figure 4: TreeTagger accuracy curve for _CST_ tag (computed on the
NCCtest data).

As shown in Figures 3 and 4 the TreeTagger accuracy increases with the number
of training files for the RCtest data. This is a natural behaviour from a
probabilistic model such as the TreeTagger, the probability increases as the
weight of Relative clauses increases in the data it has. However, there is a
drastic decrease in the _CST_ curve around 100 training files, and the
TreeTagger did not perform very well annotating the _"that"_ with _CST_ tag,
as shown with Figure 4 the accuracy is very low, and as the number of training
files increases, the accuracy goes down. Whatever technique is used, the
detection of noun complement clauses is more challenging than for relative
clauses.

##  6 Discussion

Two approaches for explaining the results obtained for the CST tag can be
either statistically or linguistically motivated. Starting with the first
approach, as shown in Table 3, as the number of training files goes up, the
number of other tags increases, especially for the IN tag, which in this case
represents a confusion by the TreeTagger to annotate with the right tag, in
fact IN is not a specific tag but rather a generic tag as it also corresponds
to verbal that-clauses, therefore, this shows that the TreeTagger generated
noise due to a confusion on the annotation of "that" (this is better
illustrated in the figure 9 especially for the graph that represents the IN
tag in blue.) The second approach consists in analysing the competing labels
for that.

###  6.1 Accuracy in relation to other categories

The Penn Treebank tagset Santorini (1990), even though it does not acknowledge
the whole complex range of functional realisations of that, e.g. adverbial,
proform vs deictic uses, see Ballier et al. (2022) can help visualise the
complex interaction of the learning process of the identification of the
different functional uses of that. As the training data increases, the
variable proportions of the different functional realisations of that probably
changes, so that a probabilistic tagger generates models variable in their
results for this tagging task. The tagger has to learn the different competing
tags for that. Our two test datasets allow us to monitor the evolution of the
training phase as the size of the training data increases. Whereas we tried to
train Treetagger to learn CST for NCC that and WPR for relative pronouns, we
also computed the distribution of other tags that "that" may take, such as
"WDT" (that when used as a relative pronoun, but also "WH"-determiners such as
which), "DT" (Determiners), and "IN" (Subordinating conjunction, whether for
nouns or for verbs) for each of the RC and NCC corpus. Table 3 recaps the
changes observed when we evaluated the labels with our two testing sets
(RCtest and NCCtest). For each testing set, we indicate the expected count of
each label in the columns RCtest GOLD and NCCtest GOLD.

| RCtest | RCtest GOLD | NCCtest | NCCtest GOLD
---|---|---|---|---
10 training files |  |  |  |
_WPR_ | 107 | 189 | 20 | 17
_CST_ | 22 | 26 | 10 | 194
_IN_ | 95 | 0 | 183 | 0
_DT_ | 7 | 15 | 16 | 14
30 training files |  |  |  |
_WPR_ | 146 | 189 | 28 | 17
_CST_ | 5 | 26 | 3 | 194
_IN_ | 72 | 0 | 189 | 0
_DT_ | 8 | 15 | 9 | 14
100 training files |  |  |  |
_WPR_ | 158 | 189 | 25 | 17
_CST_ | 2 | 26 | 3 | 194
_IN_ | 65 | 0 | 194 | 0
_DT_ | 6 | 15 | 7 | 14
200 training files |  |  |  |
_WPR_ | 156 | 189 | 27 | 17
_CST_ | 1 | 26 | 1 | 194
_IN_ | 66 | 0 | 196 | 0
_DT_ | 8 | 15 | 0 | 14
300 training files |  |  |  |
_WPR_ | 157 | 189 | 22 | 17
_CST_ | 2 | 26 | 0 | 194
_IN_ | 67 | 0 | 202 | 0
_DT_ | 5 | 15 | 5 | 14
400 training files |  |  |  |
_WPR_ | 159 | 189 | 21 | 17
_CST_ | 2 | 26 | 0 | 194
_IN_ | 64 | 0 | 199 | 0
_DT_ | 6 | 15 | 7 | 14
500 training files |  |  |  |
_WPR_ | 158 | 189 | 23 | 17
_CST_ | 1 | 26 | 4 | 194
_IN_ | 65 | 0 | 188 | 0
_DT_ | 7 | 15 | 7 | 14

Table 3: Statistics about _WPR_ , _CST_ , _IN_ and _DT_ tags obtained for each
of the 7 models (i.e. trained with 10, 30, 100, 200, 300, 400 and 500 files).
Figure 5: Evolution of IN, CST, WPR and DT tags with training files in the
NCCtest corpus. Figure 6: Evolution of IN, CST, WPR and DT tags with training
files in the RCtest corpus.

Here is an example of these potential mishaps in the POS-tagging: "that
meeting that|IN [vs DT] morning was about a public case that|IN [vs WPR] we
might make". The first deictic that was properly labelled, the second one was
erroneously labelled as a subordinating conjunction and for the third
occurrence, the relative pronoun was tagged as a subordinating conjunction
(see additional examples in the Appendix).

###  6.2 Weakness of the TreeTagger-based heuristic

We re-annotated a corpus initially tagged with the Penn Treebank, which means
that we modified some IN tags to CST and some IN tags to WPR for relative
pronouns but the Brown corpus data retained some WDT labels. As shown in Table
3, there are many _WDT_ tags, this is simply because the _WDT_ tag is both an
older and more general version of the _WPR_ tag, and seemingly the TreeTagger
kept the older version. So the _WDT_ and _WPR_ tags are likely labels for
relative pronouns considered as equivalent in the computing of the metrics,
even though strictly speaking some WDT tokens in the Brown corpus may
correspond to WH-determiners such as which. The main objection to our method
is that we only relabelled a portion of the IN tags, so that the system has to
learn a WPR versus CST distinction while still being fed with some examples of
IN. In this sense, we can only partially monitor the behaviour of Treetagger
when subjected to more examples. Figure 5 and Figure 6 plot the evolution of
the tagging of the NCCtest and RCtest sets (respectively) as the corpus size
increases. We expect the system to learn to relabel IN as either WPR or CST
but this is hardly the case for CST. It should be noted that we did not
control the input of the respective number of examples with CST and with WPR
when increasing the data size of the training data. We only report the total
counts of the tags assigned to that, we did monitor the individual behaviour
of the tagging system for each occurrence of that.

###  6.3 Long-Distance Dependencies

As already pointed out, noun complement clauses can follow a relative clause
for the same noun (but not the other way round). That-relative clauses tend to
be adjacent to their antecedents (and are often restrictive relative clauses)
whereas (that-) noun complement clauses can be separated from their governor.
So we explored a simple metric which is the distance (i.e. number of tokens)
separating a _"that"_ (annotated either with _CST_ or _WPR_) and the last noun
before it. As shown in the boxplots in Figure 7 there is a tendency showing
that the _"that"_ tagged with _CST_ using a verb with a _DEPREL_ = _acl_ have
a higher distance separating them from the last noun before them. This can
probably cause some ambiguity due to the higher distance. However, as we can
see for the _"that"_ tagged with _WPR_ using a verb with a _DEPREL_ =
_acl:relcl_ the distance with the last noun is smaller, and there are less
misclassifications (i.e. less noise) for the _"that"_ used as _WPR_.

Figure 7: Distribution of the distance (number of tokens) separating a
_"that"_ and the last noun before it for each of the _WPR_ and _CST_ _"that"_.

This is just a statistical approach to see if there is any bias that can
explain why the heuristic produces a lot of noise.

Our metric is rather crude but head nouns of NCCs need not be adjacent to the
that-clauses, so that an inventory of structures in-between could be taken
into account. The distance between the governor and the that-clause of these
long-distance dependencies Osborne (2019) could be more systematically
investigated.

###  6.4 Relevance of UD Deprel Labels for NCC?

It should be noted that UD changed the dependency label for noun complement
clauses, as explained on the UD website: "In earlier versions of SD/USD,
complement clauses with nouns like fact or report were also analyzed as ccomp
[clausal complement]. However, we now analyze them as acl. Hence, ccomp does
not appear in nominals. This makes sense, since nominals normally do not take
core arguments." We may challenge this view since ccomp implies a "clausal
complement" and nouns may require a "core argument", even more so than for
adjectives.101010For a similar argumentation see Osborne and Gerdes (2019).
One of the unfortunate consequences is that adverbs like now in the sentence
"Now that the world is in the age where lighting seems to be a daily
necessity" are labelled as a governor of the "adnominal" clause. It maybe the
case that acl is a debatable label, also used after verbs as for that verbal
complement clauses ("if this seems incredibly far-fetched, comfort yourself
that double chute failure in modern times is also extremely unlikely, and that
you have already beaten worse odds"). Consequently, the (SUD) Surface-
Syntactic Universal Dependencies Gerdes et al. (2018) has suggested
alternative labels for acl. Another approach might be to restrict noun
complement clauses to a subcategory of acl specific to noun complement clauses
(possibly labelled as acl:ncl).

##  7 Further Research

###  7.1 Quality Monitoring of the Training Phase

We have only estimated the accuracy of the annotation on our testing sets but
we have not monitored the qualitative aspect of the annotation. Are some
sentences systematically mislabelled or can we observe some changes during the
training phase? For example, this NCC gets to be interpreted as a relative
clause:  “O’Neill had an emotional reaction that [tagged as WPR] the level of
corruption was too high to do serious projects in Russia,” Deripaska recalls.
Some configurations seem to remain challenging for parsing, and qualitative
monitoring of the accuracy should take into account these sentences for which
labelling improves or not. Controlling for frequency of exposure in the
training data should prove to be very fruitful to maybe detect thresholds in
frequency (or proportions) in the training data for accurate tagging. For
example, an example in our appendix seems to suggest that a trigram sequence
no/N/that (and corresponding identification of noun complement clauses) seems
to be learned after exposure to the 100 training files (36 occurrences). As
some of the examples of mislabellings in the Appendix also suggest, it is
likely that our relabelling algorithm for WPR is too greedy, and a more
elaborate version should filter out alternative relative pronouns that should
inhibit the relabelling process. We should also apply stricter conditions on
the type of that which can be re-tagged. Assuming the DT label is correct,
only IN labels should be re-tagged.

###  7.2 More data?

More training files from the Brown corpus have been manually annotated and
given to the TreeTagger, and an improvement in the _CST_ accuracy was observed
(see Figure 8). Though a plateau seems to be observed for the tag CST (that
for NCC complementizer), one may wonder if more examples of NCCs in the
training data would alter this curve. We have only analysed the GUM Treebank
for the UD analysis, but no less than six treebanks are available on github
for the Universal Dependency analysis of English.

Figure 8: TreeTagger accuracy for _"that"_ annotated with _CST_ in red and
_WPR_ in green with more training files.

###  7.3 Learnability and Dispersion

Our monitoring of the learning curve of the tag distinction in our TreeTagger
experiment could be finer-grained: we did not control for genre types within
the Brown corpus and the relative distributions of the two structures. If
relative clauses seem to be more frequent than NCCs in the GUM Treebank, NCCs
are more likely to be more frequent in argumentative texts Ballier (2007). Our
experiment only reported the effect of the number of the Brown files in the
training data, not the specific distribution of the two structures across the
different registers of the Brown corpus. The dispersion of these linguistic
structures in the training data could be monitored across the corpus subparts
using adequate dispersion measures Gries (2020) or by comparing the vocabulary
growth curves Evert and Baroni (2007) of the two constructions across the
Brown corpus files. Our Figure 9 crudely plots the distribution of the
different tags in the training data as the size of the corpus increases
(measured in number of files, but not with the corresponding text genres).
Increasing the size of the corpus may require more attention to a
frequency/textual diversity trade off.

Figure 9: Evolution of the number of different tags for the re-annotated Brown
corpus file groups (10, 30, 100, 200, 300, 400, 500 files.)

##  8 Conclusion

In this study, we have experimented two methods to detect noun complement
clauses, either by using the universal dependency GUM treebank or by retagging
the Brown corpus with specific _WPR_ and _CST_ tags. We also explored an
automated way to do this annotation using a specific heuristic. We have
evidenced the longer distance between the noun and the that-clause for noun
complement clauses. The detection of relative clauses does seem to be much
more robust than for noun complement clauses, which remains a problem for
information retrieval as text genres could be interestingly classified with
this criterion. The difference in frequency and in adjacency may account for
such a discrepancy in the accuracy of the identification of the clause type.
We have only begun to explore the parameters of the learnability of these tags
corresponding to such a subtle linguistic distinction.

## Acknowledgements

We thank the three reviewers for their careful comments on a preliminary
version of this paper. Thanks are due to Université Paris Cité MSc in Machine
Learning for Data Science, which triggered this joint paper. Part of this
research was carrried out on a CNRS research leave at the Laboratoire de
Linguistique Formelle CNRS research lab, for which grateful thanks are
acknowledged. We thank Issa Kanté for his collection of examples for the test
datasets, partially reflecting his PhD data Kanté (2017).

## Appendix

###  8.1 Example of a noun complement clause where that gets properly tagged
after 100 files in the training data (containing 36 occurrences of the no N
that sequence)

"However, there is no guarantee that[tagged as CST] only the genuine repentant
will produce works of value to the society."

###  8.2 Examples of remaining errors in our test sets

We include examples of persistent mislabelling in our test data. After 500
training files, 6 sentences with that in noun complement clauses are still
tagged as if they were relative pronouns (with WPR).

  * •

_The statement that|WPR the tribunal has made an "error of law" means no more
or less than that|CST the construction placed upon the term by the court is
preferred to that|DT of the tribunal._

  * •

_There was no dispute that|WPR Bunn throughout acted with the authority of the
bank._

  * •

_This included a commitment that|WPR “if one of the two states should become
the target of aggression, then the other side will give the aggressor no
military aid or other support”._

  * •

_We have received information that|WPR today, between 1400 and 1500, there was
an explosion at the residence of Seyed Ali Khamenei._

  * •

_Recently there was the illusion that|WPR Hamas, while not a perfect partner,
was at least a group that could implement decisions,” he said._

  * •

_Where there is a contract for the sale of goods by description, there is an
implied condition that|WPR the goods shall correspond with the description._

####  8.2.1 Example from our test sets that has been annotated with DT rather
than with CST

We illustrate the complexity of the polyfunctionality of "that" by showing an
example of overfitting for the deictic/pronominal uses of "that".

"A high-ranking official in the Clinton administration expressed shock
that[tagged as DT rather than CST] “the kids” in the White House “did not
stand up when the president entered the room."

####  8.2.2 Examples from the RC test set that have been annotated with IN
rather than with WPR

  * •

_High death rates among children reduce the value that |IN parents place on
education; and so on._

  * •

_The distinction that |IN matters is from that of ’patronage’, which itself,
as we shall see, is highly varied._

####  8.2.3 Examples from the NCCtest set that have been annotated with IN
rather than with CST

  * •

_They’re living proof that asthma can be passed from generation to
generation._

  * •

_Where there is a contract for the sale of goods by description, there is an
implied condition that the goods shall correspond with the description._

###  8.3 An example of false positives for the Brown relabelling heuristic

  * •

_"… But one does not have to affirm the existence of an evil order
irredeemable in that[tagged as WPR] sense, or a static order in which no
changes will take place in time, to be able truthfully to affirm the following
fact: there has never been justitia imprinted in social institutions and
social relationships except in the context of some pax-ordo preserved by
clothed or naked force …"_ (it should be _DT_ rather than _WPR_). The relative
clause is with WHICH, not with THAT.

## References

  * Ballier (2004) Nicolas Ballier. 2004.  _Praxis métalinguistiques et ontologie des catégories_.  Habilitation thesis in linguistics, Université de Paris X-Nanterre.
  * Ballier (2007) Nicolas Ballier. 2007.  La complétive du nom dans le discours des linguistes.  _D. Banks, La coordination et la subordination dans le texte de spécialité, Paris, L’Harmattan_ , pages 55–76.
  * Ballier et al. (2022) Nicolas Ballier, Antonio Balvet, Taylor Arnold, and Thomas Gaillat. 2022.  _Some metalinguistic assumptions behind tagsets for English: evidence from that in different versions of the Brown corpus_ , pages 27–81. Peter Lang, Bern.
  * Bird (2006) Steven Bird. 2006.  NLTK: the natural language toolkit.  In _Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions_ , pages 69–72.
  * Bowen (2005) Rhonwen Bowen. 2005.  Noun complementation in english : a corpus-based study of structural types and patterns.
  * Buchholz and Marsi (2006) Sabine Buchholz and Erwin Marsi. 2006.  CoNLL-X shared task on multilingual dependency parsing.  In _Proceedings of the tenth conference on computational natural language learning (CoNLL-X)_ , pages 149–164.
  * Charles (2007) Maggie Charles. 2007.  Argument or evidence? disciplinary variation in the use of the noun that pattern in stance construction.  _English for Specific Purposes_ , 26(2):203–218.
  * Consortium et al. (2007) BNC Consortium et al. 2007.  British National Corpus XML edition.  _Oxford Text Archive. http://hdl.handle.net/20.500_ , 12024:2554.
  * Evert and Baroni (2007) Stefan Evert and Marco Baroni. 2007.  zipfR: Word frequency distributions in R.  In _Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions_ , pages 29–32.
  * Gaillat et al. (2014) Thomas Gaillat, Pascale Sébillot, and Nicolas Ballier. 2014.  Automated classification of unexpected uses of this and that in a learner corpus of english.  _Language and Computers_ , 78:309–324.
  * Garside (1987) Roger Garside. 1987.  The CLAWS word-tagging system.  _The Computational analysis of English: A corpus-based approach. London: Longman_ , pages 30–41.
  * Gerdes et al. (2018) Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2018.  SUD or Surface-Syntactic Universal Dependencies: An annotation scheme near-isomorphic to UD.  In _Universal dependencies workshop 2018_.
  * Gries (2020) Stefan Th. Gries. 2020.  Analyzing Dispersion.  In _A Practical Handbook of Corpus Linguistics_ , pages 99–118, Cham. Springer International Publishing.
  * Huddleston (1984) Rodney Huddleston. 1984.  _Introduction to the Grammar of English_.  Cambridge University Press.
  * Kanté (2017) Issa M Kanté. 2017.  Étude sémantico-syntaxique de la complétive nominale en anglais et en français.  _Étude sémantico-syntaxique de la complétive nominale en anglais et en français_.
  * Kucera et al. (1967) Henry Kucera, Henry Kučera, and Winthrop Nelson Francis. 1967.  _Computational analysis of present-day American English_.  Brown University Press.
  * Levine, Lauren and Zeldes, Amir (2017) Levine, Lauren and Zeldes, Amir. 2017.  GUM: The Georgetown University multilayer corpus.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * McDonald et al. (2013) Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. 2013.  Universal dependency annotation for multilingual parsing.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 92–97.
  * Osborne (2019) Timothy Osborne. 2019.  _A dependency grammar of English: An introduction and beyond_.  John Benjamins Publishing Company.
  * Osborne and Gerdes (2019) Timothy Osborne and Kim Gerdes. 2019.  The status of function words in dependency grammar: A critique of Universal Dependencies (UD).  _Glossa: a journal of general linguistics_ , 4(1):1–28.
  * Santorini (1990) Beatrice Santorini. 1990.  Part-of-speech tagging guidelines for the penn treebank project (3rd revision).  _Technical Reports (CIS)_ , page 570.
  * Schmid (2000) Hans-Jörg Schmid. 2000.  _English abstract nouns as conceptual shells_.  De Gruyter Mouton, Berlin.
  * Schmid (1994) Helmut Schmid. 1994.  TreeTagger-a language independent part-of-speech tagger.  _http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/_.
  * Straka (2018) Milan Straka. 2018.  UDPipe 2.0 prototype at CoNLL 2018 UD Shared Task.  In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , pages 197–207, Brussels, Belgium. Association for Computational Linguistics.
  * University Centre for Computer Corpus Research on Language (1995-2004) University Centre for Computer Corpus Research on Language. 1995-2004.  Constituent Likelihood Automatic Word-tagging System 8.
  * Zeldes (2017) Amir Zeldes. 2017.  The GUM corpus: Creating Multilayer Resources in the Classroom.  _Language Resources and Evaluation_ , 51(3):581–612.