# Extracting Fine-Grained Knowledge Graphs of Scientific Claims:
Dataset and Transformer-Based Results

Ian H. Magnusson♠♡ Scott E. Friedman♠
♠SIFT, Minneapolis, MN, USA
♡Northeastern University, Boston, MA, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Recent transformer-based approaches demonstrate promising results on
relational scientific information extraction. Existing datasets focus on high-
level description of how research is carried out. Instead we focus on the
subtleties of how experimental associations are presented by building
SciClaim, a dataset of scientific claims drawn from Social and Behavior
Science (SBS), PubMed, and CORD-19 papers. Our novel graph annotation schema
incorporates not only coarse-grained entity spans as nodes and relations as
edges between them, but also fine-grained attributes that modify entities and
their relations, for a total of 12,738 labels in the corpus. By including more
label types and more than twice the label density of previous datasets,
SciClaim captures causal, comparative, predictive, statistical, and
proportional associations over experimental variables along with their
qualifications, subtypes, and evidence. We extend work in transformer-based
joint entity and relation extraction to effectively infer our schema, showing
the promise of fine-grained knowledge graphs in scientific claims and beyond.

##  1 Introduction

Using relations as edges to connect nodes consisting of extracted entity
mention spans produces expressive and unambiguous knowledge graphs from
unstructured text. This approach has been applied to diverse domains from
moral reasoning in social media Friedman et al. (2021b) to qualitative
structure in ethnographic texts Friedman et al. (2021a), and is particularly
useful for reasoning about scientific claims, where several experimental
variables in a sentence may have differing relations. Scientific information
extraction datasets such as SciERC Luan et al. (2018) use relations for
labeling general scientific language. Utilizing the advances of SciBERT
Beltagy et al. (2019) in scientific language modeling, SpERT Eberts and Ulges
(2020)—a transformer-based joint entity and relation extraction model—advanced
the state of the art on SciERC.

Figure 1: SciClaim knowledge graph with entities (nodes), relations (edges),
and attributes (parentheticals) connecting an independent variable via arg0 to
distinct correlations with dependent variables via arg1.

To extend relational scientific information extraction to specifically target
scientific claims, we annotate SciClaim,111Dataset available at
https://github.com/siftech/SciClaim. a dataset of 12,738 annotations on 901
sentences from expert identified claims in Social and Behavior Science (SBS)
papers Alipourfard et al. (2021), detected causal language in PubMed papers Yu
et al. (2019), and claims and causal language heuristically identified from
CORD-19 abstracts Wang et al. (2020).

Figure 2: This SciClaim graph captures the chaining together of associations
and uncovers a mediating factor in the qualitative proportionality (q+)
between the "subliminal prime" and "perceived likelihood of occurrence."

For annotation, we developed a novel graph schema that reifies claimed
associations as entity spans with fine-grained attributes and extracts factors
as additional entities connected with relations to one or more associations in
which they are involved. In Figure 1, two association entities relate two
pairs of dependent factors to an independent factor, while attributes and
additional relations delimit the scope and qualitative proportionalities of
the claim. Inspired by semantic role labeling, attributes modify associations
and the roles of their arguments, allowing us to represent claims of causal,
comparative, predictive, statistical, and proportional associations along with
their qualifications, subtypes, and evidence.

We adapt SpERT to model this additional multi-label attribute task and
demonstrate that extraction of our highly expressive knowledge graphs is
within reach of present methods.

|  | Entities | Relations | Attributes/Corefs | Total Labels
---|---|---|---|---|---
Dataset | Words | Count | Per Word | Count | Per Word | Count | Per Word | Count | Per Word
SciREX | 2512806 | 157680 | 6.27% | 9198 | 0.37% | - | - | 166878 | 06.64%
SemEval2017 | 84010 | 9946 | 11.84% | 672 | 0.79% | - | - | 10618 | 12.64%
SemEval2018 | 58144 | 7483 | 12.87% | 1595 | 2.74% | - | - | 9078 | 15.61%
SciERC | 65334 | 8089 | 12.38% | 4716 | 7.21% | 2752 | 4.21% | 15557 | 23.81%
SciClaim | 20070 | 5548 | 27.64% | 5346 | 26.64% | 1844 | 9.19% | 12738 | 63.47%
Table 1: Our SciClaim dataset contains the highest label densities per word
and comparable label counts to other scientific information extraction
datasets except SciREX, which uses distant supervision and noisy automatic
labeling. Our dataset contains fine-grained attributes as additional labels,
while SciERC contains coreference links.

##  2 Related Work

Many previous datasets for relational scientific information extraction—such
as SemEval 2017 task 10 and 2018 task 7, SciERC, and SciREX Augenstein et al.
(2017); Gábor et al. (2018); Luan et al. (2018); Jain et al. (2020)—have
annotated corpora from NLP, computer science, or similar engineering-oriented
fields. As such their annotation schemas have emphasized the description of
how research was carried out, by extracting categories of entities such as
methods, tasks, metrics, and datasets as well as relations mostly describing
their intrinsic properties such as uses, composition, and hyponymy. Two of
these datasets Luan et al. (2018); Gábor et al. (2018) contain associative
relations that directly link entities being compared or producing a result.
Our work extends further in this direction by examining not only which
entities are associated, but also how the presentation of the associations is
nuanced by the assertion of fine-grained attributes such as causality or
proportionality.

SciClaim provides the largest number of fine-grained label types among
comparable datasets. Table 1 shows SciClaim’s remarkable label densities per
word. SciClaim also contains 81.88% as many total labels as SciERC and more
total labels than SemEval 2017 task 10 and 2018 task 7. On the other hand,
SciREX utilizes distant supervision from an existing knowledge base and noisy
automatic labeling trained on SciERC to provide an order of magnitude more
labels and annotate complete documents. This is one example of how smaller yet
more densely and directly labeled datasets like SciERC and SciClaim can enable
and compliment larger, higher-level corpora.

Meanwhile, our dataset also focuses on scientific claims. Some previous work
identifies claims within scientific texts Wadden et al. (2020); Gelman et al.
(2021), but does not extract the relations and factors within the claims
themselves. Other recent symbolic semantic NLP systems do model relational
representations of scientific claims (e.g., Friedman et al., 2017), but these
approaches rely on rule-based engines with hand tuning, which require NLP
experts to maintain and adapt to new domains. Instead we modify SpERT Eberts
and Ulges (2020), a transformer-based method that has been shown to
effectively extract relational scientific information on SciERC Luan et al.
(2018). We extend this model to accommodate our additional multi-label
attributes and apply it to our claim graph extraction task.

##  3 Approach

###  3.1 Problem Definitions

SciClaim defines the multi-attribute knowledge graph extraction task as
follows: for a sentence $\mathcal{S}$ of $n$ tokens $s_{1},...,s_{n}$, and
sets of entity types $\mathcal{T}_{e}$, attribute types $\mathcal{T}_{a}$, and
relation types $\mathcal{T}_{r}$, predict the set of entities $\langle
s_{j},s_{k},t\in\mathcal{T}_{e}\rangle\in\mathcal{E}$ ranging from tokens
$s_{j}$ to $s_{k}$, where $1\leq j\leq k\leq n$, the set of relations over
entities $\langle
e_{head}\in\mathcal{E},e_{tail}\in\mathcal{E},t\in\mathcal{T}_{r}\rangle\in\mathcal{R}$
where $e_{head}\neq e_{tail}$, and the set of attributes over entities
$\langle e\in\mathcal{E},t\in\mathcal{T}_{a}\rangle\in\mathcal{A}$. This
defines a directed multi-graph without self-cycles, where each unique span can
be represented by at most one entity node with zero to $|\mathcal{T}_{a}|$
attributes.

###  3.2 Dataset Construction

To create SciClaim, two NLP researchers annotated 901 sentences from several
sources: 336 from papers in Social and Behavior Science (SBS) with expert
identified claims Alipourfard et al. (2021), 411 filtered for causal language
in PubMed papers Yu et al. (2019), 135 containing claims and causal language
identified from CORD-19 abstracts Wang et al. (2020) with heuristic keywords,
and 19 manual perturbations included only in training data.

To aid in the labeling of these densely annotated sentences, we iteratively
trained on already collected data and utilized the predictions of the
partially trained model on new training examples as suggestions in our
labeling interface. We disabled these model suggestions on our 100 example
test set to ensure that this did not bias our evaluation.

Due to the dense and potentially overlapping span annotations, small decisions
about what tokens to include in a span frequently influence the span
boundaries of several other entities in a sentence. However, most of these
decisions have negligible impact on the meaningfulness of the annotation (e.g.
the decision to include a determiner in span), rendering exact match agreement
ineffective. Instead to promote consistency and domain relevance we employed
iterative schema design sessions in consultation with a subject matter expert
in reproducibility of SBS experiments and a process of consensus, schema re-
development, and re-annotation on 250 examples where annotators overlapped.

Table 1 contrasts SciClaim’s label counts and density with the other
relational scientific information extraction datasets discussed in Section 2,
and precise counts for each label type are provided in Table 3. Further
details are in Appendix A.

###  3.3 Graph Schema

The SciClaim graph schema is designed to capture associations between factors
(e.g., causation, comparison, prediction, statistics, proportionality),
monotonicity constraints across factors, epistemic status, subtypes, and high-
level qualifiers.

Figure 3: Comparison attributes modify arguments to account for a (sometimes
implicit) frame of reference.

Entities are labeled text spans. The same exact span cannot correspond to more
than one entity type, but two entity spans can overlap. Entities comprise the
nodes of SciClaim graphs upon which attributes and relations are asserted. Our
schema includes six entity types: Factors are variables that are tested or
asserted within a claim (e.g., “sleep quality” in Figure 1). Associations are
explicit phrases associating one or more factors (e.g., “higher” Figure 3).
Magnitudes are modifiers of an association indicating its likelihood,
strength, or direction (e.g., “significantly” and “negatively” in Figure 1).
Evidence is an explicit mention of a study, theory, or methodology supporting
an association (e.g., “our SEIR model”). Epistemics express the belief status
of an association, often indicating whether something is hypothesized,
assumed, or observed (e.g., “predicted” in Figure 2). Qualifiers constrain the
applicability or scope of an assertion (e.g., “for medical staff” in Figure
1).

Attributes are multi-label fine-grained annotations (visualized in
parentheses), where zero or more may apply to any given entity. Our schema
includes the following attributes, all of which apply solely to Association
entities: Causation expresses cause-and-effect over its constituent factors
(e.g., both “increase” spans in Figure 2). Correlation expresses
interdependence over its constituent factors (e.g., both “associated with”
spans in Figure 1). Comparison expresses an association with a frame of
reference (as in the “higher” statement of Figure 3). Sign+ and Sign-
expresses high/low or increased/decreased factor value (e.g., “correlates more
closely with” or “shortened” respectively). Test expresses statistical
measurements (e.g., “ANOVA”). Indicates expresses a predictive relationship
(e.g., “prognostic factors for”).

Relations are directed edges between labeled entities in SciClaim graphs. They
are critical for expressing what-goes-with-what over the set of entities. Note
that in Figures 1 and 2 the unlabeled arrows are all modifier relations, left
blank to avoid clutter. We encode six relations: arg0 relates an association
to its cause, antecedent, subject, or independent variable (e.g., “levels of
social support” in Figure 1). arg1 relates an association to its result or
dependent variable (e.g., “self-efficacy” and “stress” in Figure 1). comp_to
is an explicit frame of reference in a comparative association (e.g., “control
group” in Figure 3). subtype relates a head entity to a subtype tail (e.g.,
“stillbirth” as a subtype of “pregnancy outcome”). modifier relates
associations to qualifiers, magnitudes, epistemics, and evidence (e.g., all
unlabeled arrows in Figure 1 and Figure 2). q+ and q- indicate positive and
negative qualitative proportionality, respectively, where increasing the head
factor increases or decreases the tail factor, respectively (e.g., “levels of
social support” is q+ to “sleep quality” and q- to “stress” in Figure 1).

###  3.4 Model Architecture

Figure 4: Our extension of SpERT components (a, b, and c) with multi-label attributes (d) and attention-based entity span representations (e). |  | Entities | Attributes | Relations
---|---|---|---|---
Data | Model | P | R | F1 | P | R | F1 | P | R | F1
SciERC | SpERT | 70.87 | 69.79 | 70.33 | - | - | - | 53.40 | 48.54 | 50.84
SciClaim | SpERT-attrs-as-ents | 90.13 | 88.63 | 89.37 | 92.35 | 82.13 | 86.94 | 77.59 | 74.34 | 75.92
SciClaim | SpERT-modified | 89.81 | 87.87 | 88.83 | 91.89 | 82.62 | 87.01 | 76.43 | 73.72 | 75.05
Table 2: Micro Precision, Recall, and F1 averaged over 5 runs on SciClaim with SciERC for comparison. | Label | P | R | F1 | S
---|---|---|---|---|---
Entities | factor | 91.28 | 89.97 | 90.62 | 2756
evidence | 88.80 | 93.33 | 90.96 | 230
epistemic | 91.21 | 72.17 | 80.52 | 299
association | 92.45 | 88.20 | 90.27 | 1290
magnitude | 87.71 | 88.38 | 88.02 | 613
qualifier | 75.86 | 78.33 | 77.02 | 360
Attributes | causation | 38.15 | 60.00 | 46.20 | 342
comparison | 86.69 | 80.00 | 83.19 | 329
indicates | 84.79 | 76.25 | 80.20 | 84
sign+ | 97.27 | 88.31 | 92.58 | 542
sign- | 91.97 | 72.86 | 81.28 | 202
correlation | 98.42 | 84.41 | 90.88 | 320
Relations | arg0 | 79.53 | 75.03 | 77.19 | 1325
arg1 | 79.92 | 77.57 | 78.71 | 1384
comp_to | 65.86 | 60.00 | 62.78 | 187
modifier | 77.71 | 76.21 | 76.95 | 1582
subtype | 40.00 | 33.33 | 36.00 | 156
q+ | 65.53 | 67.61 | 66.50 | 504
q- | 70.70 | 53.00 | 60.09 | 208
Table 3: High Precision, Recall, and F1 across most types relative to total
Support in SciClaim, using SpERT-modified averaged over 5 runs.

In order to model the additional multi-label task in SciClaim, we extend SpERT
Eberts and Ulges (2020) with an attribute classifier. SpERT provides
components (Figure 4 a–c) for joint entity and relation extraction and permits
the overlapping spans in our data. These classifiers utilize a span
representation that combines the SciBERT Beltagy et al. (2019) contextual
embeddings of all tokens in the span through maxpooling, along with a context
representation and learned width embedding. SpERT classifies entities first
and only infers relations on pairs of identified entities.

Instead of maxpool we adopt an attention-based span representation (Figure 4
e) inspired by Lee et al. (2017). This produces scalars $\alpha_{i,t}$ for
each SciBERT token vector $\mathbf{h}_{t}$ in a span $i$ using learned
parameters $\mathbf{w}$ and $b$:

$\displaystyle\alpha_{i,t}={\exp(\mathbf{w}\cdot\mathbf{h}_{t}+b)\over\sum_{k=START(1)}^{END(i)}\exp(\mathbf{w}\cdot\mathbf{h}_{k}+b)}$
(1)

These attention weights are used to make a span representation
$\mathbf{\hat{h}}_{i}$ with the following weighted sum:

$\displaystyle\mathbf{\hat{h}}_{i}=\sum_{t=START(1)}^{END(i)}\alpha_{i,t}\mathbf{h}_{t}$
(2)

We use the same cascaded inference strategy and input the span representations
of identified entities $\mathbf{x}^{a}$ to an attribute classifier (Figure 4
d) with weights $\mathbf{W}^{a}$ and bias $\mathbf{b}^{a}$. A pointwise
sigmoid $\sigma$ yields seperate confidence scores $\mathbf{\hat{y}}^{a}$ for
each attribute:

$\displaystyle\mathbf{\hat{y}}^{a}=\sigma(\mathbf{W}^{a}\mathbf{x}^{a}+\mathbf{b}^{a})$
(3)

We train the attribute classifier with a binary cross entropy loss
$\mathcal{L}_{a}$ summed with the SpERT entity and relation losses,
$\mathcal{L}_{e}$ and $\mathcal{L}_{r}$, for a joint loss:

$\displaystyle\mathcal{L}=\mathcal{L}_{e}+\mathcal{L}_{r}+\mathcal{L}_{a}$ (4)

##  4 Evaluation

In Table 2 we report micro performance metrics on the SciClaim test set
averaged over 5 runs.

In addition to the modified SpERT (detailed in Section 3.4), we also test a
variant attrs-as-ents where all attribute labels on an entity span are
collapsed into a single combined annotation, allowing unmodified SpERT to
process attributes. Precisely, we collapse $\mathcal{T}_{e}$ entity types with
all combinations of ${\mathcal{T}_{a}}$ attribute types into
$\\{\mathcal{T}_{e}\times{{\mathcal{T}_{a}}\choose k}:0\leq
k\leq|\mathcal{T}_{a}|\\}$ multi-class entity labels. We hypothesized that the
combinatorially larger number of labels required by attrs-as-ents would lower
performance on rarely occurring combinations. Surprisingly the variants get
almost identical results, suggesting that—at least for our data—a single layer
classifier can infer the attributes of a span simultaneously just as well as
doing so independently. We tested other model variants that also produced
changes $\sim$1% F1 and thus are relegated to Appendix B.

To our knowledge no previous models exists that can run directly on all three
tasks in our dataset due to the presence of both overlapped entity spans and
multi-label attributes. For comparison we include SpERT’s state-of-the-art
performance on SciERC, the dataset closest to ours in terms of label density.
The high performance of our adapted SpERT on SciClaim demonstrates the
practicality of extracting our novel graph schema with present methods despite
its fine-grained approach.

The per-class evaluations for our main model are reported in Table 3. With few
exceptions performance is good, and generally follows support for the label in
the dataset. The _Causation_ attribute metrics may be influenced by noise from
anomalously low representation in the test set (only 5 instances compared to
59 instances of _Correlation_). Likewise the _Test_ attribute unfortunately
does not appear in the test set at all, but receives validation F1 of 95.95%
despite only appearing 25 times in the corpus. Another outlier, the _subtype_
relation, is particularly challenging, especially with its low rate of
occurrence, due to it being one of the few relation types occurring directly
between factors rather than mediated through a reified association span. The
_q+_ /_q-_ relations are likewise expressed as direct links between factors.
Although these require complex reasoning about the qualitative
proportionalities of factors (e.g., Figure 2), they nonetheless receive
promising results. The attributes _Sign+_ /_Sign-_ serve a similar role and
provide partial redundancy for _q+_ /_q-_ labels, allowing downstream
reasoning to back off to these less precise, more robust attributes when the
qualitative proportionalities are not extracted.

##  5 Conclusion

Previous scientific information extraction crafts useful high-level
representation of papers, going as far as document level relations spanning
thousands of words in Jain et al. (2020). Complimentary to these efforts, we
propose fine-grained and densely annotated scientific information extraction
that captures not just what is said but how it is presented and argued.
SciClaim applies this approach to associative claims and demonstrates that
existing models such as SpERT Eberts and Ulges (2020) can be modified to
accurately extract fine-grained knowledge graphs ripe for downstream
reasoning.

## Acknowledgements

This material is based upon work supported by the Defense Advanced Research
Projects Agency (DARPA) and Army Research Office (ARO) under Contract No.
W911NF-20-C-0002. Any opinions, findings and conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the Defense Advanced Research Projects Agency (DARPA) and
Army Research Office (ARO). We thank the reviewers for their helpful feedback.

## References

  * Alipourfard et al. (2021) Nazanin Alipourfard, Beatrix Arendt, Daniel M Benjamin, Noam Benkler, Michael M Bishop, Mark Burstein, Martin Bush, James Caverlee, Yiling Chen, Chae Clark, and et al. 2021.  Systematizing confidence in open research and evidence (score).
  * Augenstein et al. (2017) Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017.  SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 546–555, Vancouver, Canada. Association for Computational Linguistics.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Eberts and Ulges (2020) Markus Eberts and Adrian Ulges. 2020.  Span-based joint entity and relation extraction with transformer pre-training.  _24th European Conference on Artificial Intelligence_.
  * Friedman et al. (2017) Scott Friedman, Mark Burstein, David McDonald, Alex Plotnick, Laurel Bobrow, Rusty Bobrow, Brent Cochran, and J Pustejovsky. 2017.  Learning by reading: Extending and localizing against a model.  _Advances in Cognitive Systems_ , 5:77–96.
  * Friedman et al. (2021a) Scott E. Friedman, Ian H. Magnusson, and Sonja M. Schmer-Galunder. 2021a.  Extracting qualitative causal structure with transformer-based nlp.  In _QR2021 @ IJCAI_.
  * Friedman et al. (2021b) Scott E. Friedman, Ian H. Magnusson, Sonja M. Schmer-Galunder, Ruta Wheelock, Jeremy Gottlieb, Pooja Patel, and Christopher Miller. 2021b.  Toward Transformer-Based NLP for Extracting Psychosocial Indicators of Moral Disengagement.  In _Annual Meeting of the Cognitive Science Community (CogSci)_.
  * Gábor et al. (2018) Kata Gábor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang QasemiZadeh, Haïfa Zargayouna, and Thierry Charnois. 2018.  SemEval-2018 task 7: Semantic relation extraction and classification in scientific papers.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 679–688, New Orleans, Louisiana. Association for Computational Linguistics.
  * Gelman et al. (2021) Ben Gelman, Chae Clark, Scott Friedman, Ugur Kuter, and James Gentile. 2021.  Toward a robust method for understanding the reproducibility and replicability of research.  _AAAI Workshop on Scientific Document Understanding_.
  * Jain et al. (2020) Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020.  SciREX: A challenge dataset for document-level information extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7506–7516, Online. Association for Computational Linguistics.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3219–3232, Brussels, Belgium. Association for Computational Linguistics.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Online. Association for Computational Linguistics.
  * Wang et al. (2020) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex Wade, Kuansan Wang, Nancy Xin Ru Wang, Chris Wilhelm, Boya Xie, Douglas Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020.  Cord-19: The covid-19 open research dataset.
  * Yu et al. (2019) Bei Yu, Yingya Li, and Jun Wang. 2019.  Detecting causal language use in science findings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4664–4674, Hong Kong, China. Association for Computational Linguistics.

|  | Entities | Attributes | Relations
---|---|---|---|---
Model | Avg Val F1 | P | R | F1 | P | R | F1 | P | R | F1
SpERT-attrs-as-ents | 80.45 | 90.13 | 88.63 | 89.37 | 92.35 | 82.13 | 86.94 | 77.59 | 74.34 | 75.92
SpERT-modified | 80.66 | 89.81 | 87.87 | 88.83 | 91.89 | 82.62 | 87.01 | 76.43 | 73.72 | 75.05
SpERT-modified-maxpool | 80.22 | 90.32 | 88.54 | 89.42 | 92.00 | 80.90 | 86.09 | 76.11 | 75.92 | 75.99
SpERT-modified-unfiltered | 79.99 | 89.28 | 88.03 | 88.64 | 91.65 | 80.74 | 85.84 | 75.62 | 73.98 | 74.78
Table 4: Micro Precision, Recall, and F1 averaged over 5 runs on the SciClaim
test set as well as F1 averaged over the 3 tasks on 5 runs of SciClaim
validation data (Avg Val F1).

##  Appendix A Claims Dataset

Our English language dataset SciClaim consists of 901 examples sentences
divided into a training set of 721 sentences, a validation set of 80
sentences, and a test set of 100 sentences. The training and validation data
contain examples that were labeled from corrected suggestions from a partially
trained model, while the test set was labeled from scratch without any model
suggestions as a starting point.

Our data from CORD-19 Wang et al. (2020) is sampled with the following
keywords as a heuristic identification of claims and causal language similar
to our expert identified data from PubMed and Social and Behavioral Science
(SBS) papers: associated with, reduce, increase, leads to, led to, our result,
greater, less, more, cause, demonstrate, show, improve.

200 of our sentences (50 from PubMed and 150 from SBS) were selected to
intentionally minimize the likelihood of claims and causal language. This
includes sentences that discuss factors and other entities present in our
schema but either do not contain associations or frame associations in unusual
ways such as rhetorical questions. We intend for this data to encourage
robustness that maintains correct labels for partial graph extractions rather
than simply hallucinating associations in all sentences. We employ the
following heuristics to identify this data: We sample 50 PubMed sentences from
Yu et al. (2019) that are identified as having low causal content. We sample
100 titles from SBS paper present in Alipourfard et al. (2021), as titles
contain factors but rarely contain explicit associations and may be present in
input data from automatically extracted text from PDFs. Finally we sample 50
first lines of SBS papers from Alipourfard et al. (2021), as these lines
frequently introduce topics or rhetorical questions which either lack
associations or present highly hypothetical associations unlike those in our
main corpus.

Each filtered data source was sampled chronologically.

We utilized the following procedure for labeling: The annotators undertook
extensive, iterative schema design sessions in consultation with a subject
matter expert in reproducibility of SBS experiments. After the schema was
settled on pilot examples, a lead annotator established the annotation
standards on several hundred examples through a process of relabeling and
retraining the suggestion model. Once the suggestion model became effective,
the lead annotator and model suggestions guided the other annotator in
adopting the annotation standards. The lead annotator reviewed and corrected
the 250 overlapping examples in a consensus process with the other annotator.

##  Appendix B Variants and Hyperparameters

###  B.1 Variants

We experiment with several variants none of which substantially outperformed
the others. SpERT-modified-maxpool contains our modifications but simply uses
SpERT’s original maxpooling span representation instead of the attention-based
representations inspired by Lee et al. (2017). SpERT-modified-unfiltered
forgoes cascading inferences and instead classifies all possible spans for
attributes. Full test and averaged validation results are presented in Table
4.

###  B.2 Hyperparameters

The following hyperparameters and settings were selected using manual tuning
of 10-fold cross validation on the training set and optimizing for average
micro-f1 performance over all 3 tasks: language model SciBERT uncased, mini
batch size 8, epochs 40, optimizer AdamW, linear scheduling, warm up 0.05,
learning rate 5e-5, learning rate warm up 0.1, weight decay 0.01, max grad
norm 1.0, size embedding dimension 25, dropout probability 0.1, maximum span
size 20, attribute filter threshold 0.55, relation filter threshold 0.4.

We ran 32 trials on 5 RTX 2080 ti GPUs, where each trial takes roughly 20
minutes. Our model contains 110 million parameters.

We explored the following hyperparameter bounds: language model $\in$ {BERT,
SciBERT, SpanBERT, SciBERT tuned on SciERC }, epochs $\in$ {20, 40, 80}, batch
size $\in$ {4, 8, 16}, learning rate $\in$ {1e-5, 5e-5, 1e-4}, scheduling
$\in$ {linear, cyclic }, warm up $\in$ {0.0, 0.05, 0.1, 0.15. 0.2, 0.25, 0.3},
attribute filter threshold $\in$ {0.4, 0.5, 0.55, 0.6}, relation filter
threshold $\in$ {0.35, 0.4, 0.5, 0.6}. The remaining settings we inherit from
SpERT as initial experimentation on early datasets revealed little impact.


# On Creativity and Open-Endedness

Lisa B Soros1, Alyssa M Adams3, Stefano Kalonaris2, Olaf Witkowski3-6
Christian Guckelsberger7,8

1Department of Computer Science, Barnard College, USA 2RIKEN AIP, Japan 3Cross
Labs, Cross Compass Ltd., Japan
4Graduate School of Arts and Sciences, University of Tokyo, Japan
5Earth-Life Science Institute, Tokyo Institute of Technology, Japan 6Institute
for Advanced Study, USA
7Department of Computer Science, Aalto University, Finland
8School of Electronic Engineering and Computer Science, Queen Mary University
of London, UK
<EMAIL_ADDRESS>
###### Abstract

ALife (ALife) as an interdisciplinary field draws inspiration and influence
from a variety of perspectives. Scientific progress crucially depends, then,
on concerted efforts to invite cross-disciplinary dialogue. The goal of this
paper is to revitalize discussions of potential connections between the fields
of CC (CC) and ALife, focusing specifically on the concept of OE (OE); the
primary goal of CC is to endow artificial systems with creativity, and ALife
has dedicated much research effort into studying and synthesizing OE and
artificial innovation. However, despite the close proximity of these concepts,
their use so far remains confined to their respective communities, and their
relationship is largely unclear. We provide historical context for research in
both domains, and review the limited work connecting research on creativity
and OE explicitly. We then highlight specific questions to be considered, with
the eventual goals of (i) decreasing conceptual ambiguity by highlighting
similarities and differences between the concepts of OE, (ii) identifying
synergy effects of a research agenda that encompasses both OE and creativity,
and (iii) establishing a dialogue between ALife and CC research.

## Introduction

The engineering of both CC (CC) and OE (OE) have equally been advocated as the
“final frontier” [Colton et al., 2012], and the “last grand challenge”
[Stanley et al., 2017] of AI (AI) research, respectively. These claims,
together with an immense and growing body of related work, highlight the
magnitude of, and desire toward accomplishing, these two research challenges.
They constitute central and enduring research themes found in creativity,
ALife, and AI that are addressed, amongst others, by the CC and ALife (ALife)
communities independently. We note that neither challenge is close to being
fully solved, but both overlap in their overarching purpose. We consequently
ask: Are CC and OE entirely different concepts, and does their synthesis
represent disconnected research endeavors? Are the concepts within both
mutually exclusive or do they overlap (and to what extent)?

There is much reason to believe that this is not the case. Creativity, more
broadly, is understood as a hallmark feature of human and (with nuances)
animal cognition, while OE is considered foundational for the evolution and
development of advanced lifeforms. OE is often used to characterize tasks and
tests that assess human creativity, and as a challenge to motivate the
advancement of CC algorithms. Both creativity and OE have been proposed
together as features of (especially evolutionary) algorithms across both
disciplines.

We argue that these similarities signal opportunity: there is potential for
the CC and ALife communities to learn about each other’s research heritage and
challenges, and, especially in close interaction, use it to advance research
on either front. Making use of this opportunity is important, since research
involving both creativity and OE has unresolved ambiguities in the involved
concepts, including their definitions and measurements. In a way, this also
marks an (unfortunate) commonality between the two areas.

The premise of our paper is threefold: (1) to ask questions on specific
connections between CC and OE that have the potential to promote further work,
(2) to identify synergy effects of a research agenda that encompasses both,
and (3) to consequently spark discussion and collaboration between the CC and
ALife communities. To this end, we bring together a team of CC and ALife
researchers to make two distinct contributions: we (a) summarise connections
from isolated, existing work on both concepts and (b) uncover new connections
based on our expert discussions. While our goal is ultimately to deepen our
understanding of each concept and the requirements to reproduce them in
artificial systems, we do not pretend to resolve these connections
conclusively within this limited space and team of authors. Instead, we ask
provoking questions on the nature and constituents of these connections to
establish a starting point for additional deep, dedicated interdisciplinary
investigations.

By doing so, we continue and revitalize the discussion proposed in Soros and
Stanley [2016], asking “Are there nontrivial differences between creativity
and OEE, or are they fundamentally the same concept?” This questions whether
creativity and OEE have nontrivial differences or are fundamentally the same
concept. Existing work in CC focuses on OE as a property of tasks / exercises
/ problems [e.g. Epstein, 2011] or environments [e.g. Guckelsberger et al.,
2016], but not as a characteristic of the process or the resulting product
which ALife seeks to reproduce. Vice versa, ALife researchers have drawn
connections from OE to creativity, but articulated only as sidelines.
Instances of work spanning both fields and research agendas [Boden, 2009;
Soros and Stanley, 2016] are promising, but few and isolated.

While this paper was written primarily with the goals and understanding of CC
and ALife researchers in mind, creativity and OE are becoming dramatically
more relevant beyond the boundaries of these communities. We hope that our
insights in this paper can inspire and advance AI more broadly.

## Background

The concepts of OE and creativity (more broadly) are used in a multitude of
fields and contexts, such as educational research & psychology, games & play,
software engineering, and design, just to name a few. Here, we focus on their
use specifically within the fields of ALife (ALife) and CC (CC), separate from
adjacent fields such as machine learning and AI research more generally. At
the same time, we acknowledge that the boundaries between fields are fluid and
evolving111Cf. e.g. the workshops on “Agent Learning in Open-Endedness” (ALOE)
and “Intrinsically Motivated Open-ended Learning” (IMOL) at NeurIPS, which
also feature ALife research.. As a first notable difference between both
concepts, they play a very different role in the definition of the respective
discipline’s identity.

#### (Computational) Creativity

CC has been defined as “the art, science, philosophy, and engineering of
computational systems which, by taking on particular responsibilities, exhibit
behaviors that unbiased observers would deem to be creative” [Colton et al.,
2012]. This places creativity right at the core of the discipline’s identity.
Researchers follow two broad goals: “understanding human creativity” and
“producing machine creativity” [Boden, 2004; Pérez y Pérez, 2018]. The latter
can be differentiated into co-creative systems [Kantosalo and Toivonen, 2016],
taking on some creative responsibility [Colton et al., 2012] in interaction
with others, and fully autonomous creative systems [Colton, 2008].

CC’s conceptualization of creativity is mostly adopted from (psychological)
creativity studies, the long-standing joint effort of psychology, sociology,
philosophy, and other non-computational disciplines to investigate creativity.
CC’s ontological contribution then consists of re-evaluating and extending the
scope and attribution of creativity in the context of (typically
computational) artificial systems. Due to its adoption from creativity
studies, CC follows a strongly anthropocentric agenda. As a result, creativity
is almost exclusively understood in terms of human creativity.

Creativity is a very old concept, and our understanding today mixes two
historical traditions [Still and d’Inverno, 2016]. In the older, Pagan
tradition, creativity concerns the unfolding and dissolution of natural
processes, such as the birth of a child or the growth of a plant. Here,
_create_ translates to “having an impact through natural forces” [ibid] and is
different from _facere_ , which corresponds to “make out of available
materials” [ibid]. The younger, Christian tradition used _creare_ as “bringing
about by making” [ibid], thus mixing the two prior Pagan notions. It is
typically considered an individual power, denoting “creation out of nothing”
[ibid], and has mostly been attributed to the genius of artists who bring
forth novel ideas, rather than exercising skill alone.

Following a surge in creativity-related research starting in the 1960s,
definitions of creativity proliferated. Already in 1988, 50 different
definitions had been proposed over the previous five decades. This development
prompted researchers to lament that either creativity has “almost ceased to
mean anything” [Batey and Furnham, 2006] or is an essentially contested
concept [Jordanous and Keller, 2016].

Runco and Jaeger [2012] have helped mitigate this situation by abstracting
common underlying factors in existing studies and definitions, arriving at a
two-component “standard definition” of creativity which requires a product or
process to be both _novel_ and _valuable_ to be deemed creative. Novelty can
be minimal and is often not considered sufficient; the definition is extended
by a third component– _surprise_ [Boden, 2004]. With the surge of more
powerful creative AI systems, this “standard definition” has recently been
updated to fence human from artificial creativity via the features of
authenticity and intentionality [Runco, 2023]. Crucially though, this proposal
can be argued to lack behind CC research which has long embraced both features
as part of its central challenge [cf. Ventura, 2016; Guckelsberger et al.,
2017; Colton et al., 2018; McCormack et al., 2019, e.g.].

Unfortunately, these definitions only resolve some of the ambiguity while
introducing more. While the meaning of authenticity and intentionality remains
subject of philosophical debate, the meanings of _value_ , _novelty_ , and
_surprise_ vary depending on who evaluates the creativity, of what, and in
which context [Silvia, 2018; Puryear and Lamb, 2020]. As a consequence of this
conceptual ambiguity, domain-, observer- and perspective-sensitivity, no
consensus exists on standardised measures to assess CC [Lamb et al., 2018],
despite various efforts [e.g. Ritchie, 2019].

#### Artificial Life, Open-Ended Evolution, and Eventual Open-Endedness

As a whole, the field of ALife aims to recreate aspects of real and
hypothetical living systems from scratch, i.e. artificially. These endeavors
fall under one or more of the following three categories: wet experiments
(such as chemical evolution), hardware experiments (such as robotics), and
software experiments (such as evolutionary models).

It is important to note that the concept of open-ended _evolution_ (OEE),
arising from the ALife community predates the generalization of broad open-
endedness (OE). Foundational descriptions of OEE (OEE) originate from many
pre-millennium works. Specifically, Bedau [1998] finds the phrase “open-ended
evolution” previously used by Lindgren [1991] and Ray [1996]. Bedau et al.
[2000] then establishes determining “what is inevitable in the open-ended
evolution of life” as a grand challenge for ALife. After nearly two decades of
work on the problem, Packard et al. [2019] note that evolution’s dynamic
production of diversity “has led life’s creative productivity to be called
Open-Ended Evolution (OEE) in the field”. The typical approach towards
understanding and synthesizing the phenomenon has been to create virtual
worlds imbued with features of the natural world thought to be key to
unbounded generation (of whatever things, be it code snippets or more
physically embodied agents), and then performing systemic experiments probing
exactly how long the generative process lasts and for what reasons it might
fail. Notable systems establishing the foundation for current research include
Tierra [Ray, 1992], Avida [Ofria and Wilke, 2004], PolyWorld [Yaeger, 1994],
Geb [Channon and Damper, 2000], Cosmos Taylor [2004], Division Blocks [Spector
et al., 2007], Evosphere [Miconi and Channon, 2005], and Chromaria [Soros and
Stanley, 2014]. It should be noted that this list is not exhaustive.

Throughout these decades, many definitions of OEE have been proposed. To only
name a few, it has been defined as:

  * •

a system in which the “number of possible types by far exceeds the number of
individuals (copies, sequences, etc.) in a plausible (realistic) population”
[Maynard-Smith and Szathmary, 1995],

  * •

the “on-going and indefinitely creative production of significantly new kinds
of adaptive responses to significantly new kinds of adaptive challenges and
opportunities” [Bedau, 1998],

  * •

“a system in which components continue to evolve new forms continuously,
rather than grinding to a halt when some sort of ‘optimal’ or stable position
is reached” [Taylor, 1999], as well as

  * •

“a process in which there is the possibility for an indefinite increase in
complexity” [Ruiz-Mirazo et al., 2008]

Dolson et al. [2015] turn the question of what OEE _is_ (and, accordingly, how
it should be measured) on its head and ask instead what OEE _is not_. Key
indicators include a lack of change in the population, a lack of novelty, a
lack of increasing complexity, a lack of ecosystem diversity, and a lack of
major transitions. This account then admits a pluralistic framing of the
phenomenon, allowing us to consider a wide variety of systems ranging from
definitely-not-open-ended to very-strongly-open-ended. However, the steps in
this gradient (and the gradient in general) remain unclear and difficult to
categorize at present.

The idea of OE as a property not exclusive to evolution has been a topic of
exploration in the years bridging the gap between the field’s inception and
its more recent diffusion into adjacent fields. Lehman et al. [2008] establish
“a link between the challenge of solving highly ambitious problems in machine
learning and the goal of reproducing the dynamics of open-ended evolution in
artificial life”. They refer to “open-endedness in natural evolution”, hinting
at the possibility that other kinds of open-endedness might exist apart from
the biological or Darwinian kind. Pattee and Sayama [2019] note that “open-
endedness is often considered a prerequisite property of the whole
evolutionary system and its dynamical behaviors”, explicitly separating the
concepts of OE and evolution.

Much like CC, the idea of OE is enveloped in ambiguous concepts (like novelty
and unboundedness, for example) that are difficult to measure quantitatively
and are nearly impossible to untangle from context. Even so, researchers in
ALife seem to agree on a broad definition of OE, even if the details vary. In
general, OE is defined as the ability to continually create new, innovative
structures and evolve under new, innovative dynamics, all while continually
increasing in complexity [Banzhaf et al., 2016].

Similar to the case of creativity, beyond conceptual clarity, there is a
distinct lack of consensus for “gold-standard” quantitative measures for open-
endedness [Taylor et al., 2016]. This makes it difficult to interpret the
resulting behaviors/dynamics of new, innovative ALife models that agree with a
field-wide consensus. Because of this, there is a strong push from within the
ALife community to further explore these concepts [Song, 2022]. Without this
consensus, it remains an open challenge to 1) develop a theory for open-
endedness that easily translates to measures and algorithmic design in silico
and 2) imitate and recreate real open-ended systems like biological evolution
artificially.

## Early Connections: OE & Creativity

As our first contribution, we provide an up-to-date review of existing
connections between OE and creativity drawn in related work. First though, we
highlight which work has been excluded from this review.

### Excluded Literature

Explicit discussions on the relationship of creativity and OE are scarce and
we exclude any work that merely draws implicit connections from this review.
For instance, Stanley et al. [2017] do not directly compare creativity and OE,
but on two occasions amalgamate the concepts into “open-ended creativity ”.
Open-endedness is framed as a modality of a process (creativity) that could
just as well be bounded, akin to traditional problem-solving. Similarly,
[2020] asserts that evolution is inherently creative, but lacks the foresight
and intentionality of human creativity. Lehman et al. [2020] and Wiggins et
al. [2015] draw connections between creativity and evolution, but neither of
these investigations connects creativity with OE as a property of evolutionary
processes. As a final example, Boden [2015], who has had a major influence on
the development of CC as a research endeavor, finds parallels between three
types of creative processes (combinatory, exploratory, and transformational)
and the research matter of ALife, but does not relate these to OE
specifically. This has later been mended by Taylor [2019, 2021], as discussed
below.

We also exclude work that focuses on OE as a property of
tasks/exercises/problems – which have more than one right answer, solution or
outcome and can be completed in different ways [e.g. Epstein, 2011].
Similarly, we exclude work on open-ended _environments_ [e.g. Guckelsberger et
al., 2016] within which such tasks can be established and solved. Open-ended
_tasks_ are often cited as a necessary condition for the emergence of
creativity. Such tasks can be completed in different ways, and are distinct
from OE as characteristic of _processes_ addressed in ALife (see Background).
This exclusion rests on our understanding that connections between creativity
and OE as a subject of CC and ALife research can best be drawn by focusing on
the fields’ effort to synthesize a creative or open-ended _process_. This
process binds agent embodiment and environment together in one system and is
potentially constrained by a task or giving rise to tasks (cf. creativity as
problem-finding). In this view, agent embodiment, environment, and task
represent variables to e.g. benchmark the generality of this synthesized
process against. We leverage literature on creativity and OE as a process to
anchor our investigation and discuss the systems perspective on creativity and
OE as a conceptual commonality below. Notably, this excludes literature from
educational and creativity research investigating the potentiality of
different tasks and environments to assess and exercise human creativity.

### A (Very) Short History of Early Connections

Boden [2015], mentioned previously, provides a detailed conceptual connection
between CC and ALife. First, “psychological creativity” (involving human
thought and activity) is contrasted with “biological creativity”, which is
described as “the ability to generate new cells, organs, organisms, or
species. This includes the (phylogenetic) creativity of evolution, the
(ontogenetic) creativity of morphogenesis, and the (autopoietic) creation of
individual cells”. Then, as mentioned previously, creativity in the broader
sense is classified into three types: “the different sorts of surprise that we
experience on encountering a creative idea correspond to different mechanisms
for producing novelty. Those mechanisms mark three kinds of creativity:
combinational, exploratory, and transformational.”

Soros and Stanley [2016] additionally drew connections soon thereafter,
focusing more specifically on OEE. To this end, they highlight a potential
connection between Taylor et al. [2016]’s primary categories of hallmarks of
open-endedness (ongoing adaptive novelty, and ongoing growth of complexity)
and Boden’s concepts of exploratory and transformational creativity [Boden,
2004]. Additionally, they raise three open questions about the relationship
between OEE and creativity theory: 1) Are there nontrivial differences between
creativity and OEE or are they fundamentally the same concept? 2) Is there any
utility in formally pursuing the nature of the relationship between evolution
and creativity? 3) Do there exist metrics and practices for studying
computational creativity (or creativity in general) that could help identify
and quantify hallmarks of open-ended evolution? They thus appeal to the
struggles in both field to define and measure creativity and OE, respectively.

Taylor [2019] discusses three different kinds of novelties described by
[Banzhaf et al., 2016], based on three different types of creativity
(combinatorial, exploratory, and transformational) proposed by [Boden, 2004,
2015]: exploratory, expansive, and transformational novelty. Combinatorial
creativity refers to the creation of new concepts or artifacts by combining
features of existing ones. Exploratory creativity corresponds to traversing a
space of such concepts or artifacts to construct or discover new and valued
instances. Finally, transformational creativity is defined as the reaching of
new points that only become accessible by altering the rules defining the
space itself “so that thoughts are now possible which previously (…) were
literally inconceivable” [Boden, 2004, p. 6].

Many process theorists consider transformational creativity as the most
profound [cf. Lamb et al., 2018], including Boden herself, who notes that only
transformational creativity allows for the most radical forms of surprise. She
considers surprise to be a defining feature of creativity, next to novelty and
value.

Taylor notes that, since Banzhaf et al.’s [2016] types are defined in terms of
whether or not the novelty results in a change in the model or meta-model of a
system, it closely resembles the three different kinds of creativity as
outlined by [Boden, 2004]. Exploratory novelties result from applying the
current dynamics of a system (summarised as a model) to multiple possible
states. Expansive novelties result from applying similar models that share
concepts present in the current meta-model. Lastly, transformational novelties
can only result from models made of entirely new concepts, which necessitates
a change in the meta-model description of the system’s dynamics. Taylor
directly translates these three types of novelties into three types of open-
ended processes that produce these novelties. In particular, the difference
between I-creativity (previously P-creativity, when an idea, action, or object
is new to an individual [Boden, 2015]) and H-creativity (when an idea, action,
or object is new to historically [Boden, 2004]) is compared to the difference
between transformational and expansive novelties, respectively.

Taylor [2021] argues that systems that produce expansive and transformational
novelties and innovations can be generalized further into transdomain bridges
and non-additive compositional systems [Taylor, 2021]. By characterizing a
dynamical system by its parameters, spatial arrangement, and dynamics
(Parameter Space–Organisation Space–Action Space (POA)), all evolutionary
innovations, including the three mentioned in [Taylor, 2019], are either a
physical phenomenon that had previously been unseen, or resulting from new,
persistent systems constructed from within the environment. This last point is
related to the concept from creativity research that involves transforming the
space in which innovations can be explored. By explicitly considering an
environment that can host new physical systems– and therefore new dynamics and
new spaces of possibility– the resulting non-additive compositional systems
make it possible for evolutionary systems to expand their search spaces. By
doing so, Taylor suggests that evolutionary systems can evolve in ways that
are expansive and transformational, not just exploratory.

Song [2022] recites these previous connections and highlights a “loose
[conceptual] equivalence” between creativity (as defined by Boden [2004] in
terms of surprise, novelty, and usefulness) and OE (as defined in terms of
novelty, diversity, and complexity). Song grounds this equivalence in
Schmidhuber’s (2008) theory of beauty-as-compressibility. Compressibility in
turn is often used to estimate complexity in OE [e.g. Earle et al., 2021].
Notably, Schmidhuber’s theory has received arguably little approval, and
Song’s attempt at an equivalence misses instances of creativity that are
disconnected from beauty. This leaves the exact relationship between
compressibility and these three components of creativity open. Song also notes
that requirements for OE are weaker than for creativity according to Boden’s
definition, in that OE “requires one of novelty, diversity, and complexity,
while having the contingency for the other qualities”. Our goal is to go
beyond statements of “loose equivalence” and illuminate the exact nature of
this match, based on the wide array of different conceptualisations of OE and
creativity.

## New Connections

To this end, as our second contribution, we put forward a preliminary list of
questions, each of which proposes a potential connection between OE and
creativity. We invite researchers to dig into these proposals to identify
similarities, differences and potential extensions to the concepts and
measurement of OE and creativity. These questions were identified by the
authors, who are from the ALife (initials anonymized) and CC (initials
anonymized) communities, respectively, of which at least two (initials
anonymized) have actively contributed to research in both fields. In eight
remote sessions, we collaboratively constructed an affinity diagram, capturing
the connections between the two concepts. Affinity diagrams represent a
qualitative data analysis method used to group data into emergent categories
[Sharp et al., 2019]. Akin to self-organized focus groups [Morgan, 1996], we
proposed, explored, and discussed potential connections between OE and
creativity along several dimensions, such as agency, timescale, substrate, and
subjectivity.

This process was informed by the authors’ detailed knowledge of each field’s
research heritage. As a foundation for our investigation, we first reviewed
existing surveys of OE and (computational) creativity, as well as existing
work on their connections, as discussed throughout this paper. To summarize
these connections, we translate each into an open research question for future
research directions. We refer to existing work to detail and refine these
questions:

  * •

Could _generative creativity_ [Bown, 2012] or the Pagan creativity tradition
[Still and d’Inverno, 2016] provide the most universal similarity to OE since
it requires novelty (but not value), and its inspiration from natural
processes?

  * •

Could the notion of _adaptive creativity_ [Bown, 2012; Guckelsberger et al.,
2017] provide a closer fit to cases of OE that comprise a notion of value as
adaptive success?

  * •

Value is mostly considered an essential component of creativity [Runco and
Jaeger, 2012]; are any conceptualisations of value exclusive to creativity or
OE and thus deny a strict equivalence match between both concepts?

  * •

Do the modeling approaches of dynamic processes in CC and OE differ in their
requirement for continuity? More specifically, would CC researchers consider a
system to be creative even if it quickly exhausted its potential of generating
novel and valuable artifacts? To this end, what does the scarce work on
continuous CC [Cook, 2021] betray?

  * •

Could Boden’s [2004] three types of creative processes and Taylor’s [2021]
ALife interpretation be used as diagnostics for OE? Would a system be
considered open-ended even if it only briefly exhibited transformational-, but
then mostly produced explorative novelties?

  * •

Is creative autonomy [Jennings, 2010; Saunders, 2012; Guckelsberger et al.,
2017] a necessary requirement for closing the gap between creativity and OE?

  * •

If creative autonomy is defined (amongt others) through independence of a
programmer’s instructions[Jennings, 2010; Saunders, 2012; Guckelsberger et
al., 2017], then what is the reference point that a system should be
independent of to be considered open-ended?

  * •

Can research on OE and CC inspire the viewpoint from which the respective
phenomenon is driven and evaluated? Most CC systems are evaluated w.r.t. the
goals of an external observer (potentially injected into the system), and
creativity is driven from the inside-out, rather than through the system’s
surroundings.

  * •

What is the trade-off between OE and CC, and controllability in a dynamical
system? How are these two concepts related to an observer’s ability to control
a system’s dynamics? Can the relationship of either one of the first two
concepts illuminate the role of the other?

  * •

While evolution is explicitly related to the broader notion of open-endedness,
in CC it is usually used as a metaphor for algorithm design[McCormack, 2019],
and to inspire studies of creativity in evolution [Wiggins et al., 2015].
Could OE research highlight another connection between creativity and
evolution to inspire work in CC?

## The Road Ahead: Bridging Communities

The dual pursuits of synthesizing creativity in CC and OE in ALife involve
understanding the rich histories of each research agenda, as briefly
summarized in this short paper. Our goal has not been to move toward a
complete coalescence of creativity and OE but, through intentional dialogue
about both the similarities and differences in approaching each challenge, we
hope to arrive at a clearer vision of how best to further our individual aims
by understanding as tightly interconnected with those of CC and ALife as
neighbouring communities. To this end, we suggest a few questions that, if
attended, would likely help push both communities toward a fuller
understanding of these respective fields:

  * •

Can we translate research facets and goals from CC to OE research, and vice
versa? For instance, an interesting exercise might be to imagine “creativity
as it might be”.

  * •

Can engagement with the other disciplines’ research heritage draw our
attention to missing pieces in our present, own conceptualisation of
creativity and OE?

  * •

In light of these recent expansions of OE into more mainstream AI research,
can the lens of creativity theory help contextualize OE as primarily distinct
from optimization?

  * •

Can the language of creativity and open-endedness be used to inspire novel
research in fields outside of CC and ALife communities? Language about open-
endedness, as noted, has been lately spreading to the broader AI/ML spheres–
at least in a limited fashion.

A long history of exchange between ALife and CC [Saunders, 2012; Bown, 2012;
McCormack, 2012; Guckelsberger et al., 2017; McCormack, 2019] bears witness of
this endeavour’s inspirational and transformational potential.

## Conclusions and Future Work

The main goal of this paper is to revisit and collect several connections
between CC (CC) and OE (OE) from the literature and to recommend new avenues
of interdisciplinary directions for future research in creativity research, AI
(AI), and ALife (ALife). Since CC and OE have been identified as the “final
frontier” and “the last grand challenge” of AI research, respectively, we
stress that there is opportunity for both communities to learn from each
other’s research heritage, progress, and challenges.

To this end, we provide a short, comprehensive, and historical context for
progress in both research areas. Special attention is placed on existing
literature that connects concepts from creativity and OE explicitly. We
discuss potentially fertile questions for both fields to consider, such as
whether there exist any nontrivial differences between creativity and OE or
whether they are fundamentally the same concept. We also identify possible
synergy effects of a cross-community research agenda that encompasses both,
including potential strategies to reduce ambiguity in defining and measuring
each concept.

While this provides context on much of the progress made on the intersection
of both areas, we most importantly sound the call for further discussion and
collaboration between the (computational) creativity and ALife open-endedness
communities. It is crucial to establish a dialogue between them, since doing
so might provide additional context, insight, and directions moving forward.

While we do not intend to resolve these connections conclusively within this
limited space and team of authors, we provide a starting point for dedicated
interdisciplinary investigations. We hope that such endeavors will deepen our
understanding of both research challenges and further elucidate the
requirements to reproduce them in artificial systems– including reducing open
conceptual ambiguities and fostering collaborative research between these two
communities.

## Acknowledgements

We thank our anonymous reviewers for their excellent feedback and support for
this paper’s mission.

## References

  * Banzhaf et al. [2016] Wolfgang Banzhaf, Bert Baumgaertner, Guillaume Beslon, René Doursat, James A Foster, Barry McMullin, (…), and Roger White.  Defining and simulating open-ended novelty: Requirements, guidelines, and challenges.  Theory in Biosciences, pages 1–31, 2016.
  * Batey and Furnham [2006] Mark Batey and Adrian Furnham.  Creativity, Intelligence, and Personality: A Critical Review of the Scattered Literature.  Genetic, Social, and General Psychology Monographs, 132(4):355–429, 2006.
  * Bedau et al. [2000] Mark A Bedau, John S McCaskill, Norman H Packard, Steen Rasmussen, Chris Adami, David G Green, Takashi Ikegami, Kunihiko Kaneko, and Thomas S Ray.  Open problems in artificial life.  Artificial life, 6(4):363–376, 2000.
  * Bedau [1998] Mark A Bedau.  Four puzzles about life.  Artificial life, 4(2):125–140, 1998.
  * Boden [2004] Margaret A Boden.  The creative mind: Myths and mechanisms.  Routledge, 2004.
  * Boden [2009] M.A. Boden.  Creativity and artificial evolution.  In J. Copeland and R. A. Brooks, editors, Creativity: the mind, machines, and mathematics. MIT / Templeton Press, 2009.
  * Boden [2015] Margaret A Boden.  Creativity and ALife.  Artificial Life, 21(3):354–365, 2015.
  * Bown [2012] Oliver Bown.  Generative and adaptive creativity: A unified approach to creativity in nature, humans and machines.  In Computers and creativity, pages 361–381. Springer, 2012.
  * Channon and Damper [2000] A. D. Channon and R. I. Damper.  Towards the evolutionary emergence of increasingly complex advantageous behaviours.  International Journal of Systems Science, 31(7):843–860, 2000.
  * Colton et al. [2012] Simon Colton, Geraint A Wiggins, et al.  Computational creativity: The final frontier?  In Ecai, volume 12, pages 21–26, 2012.
  * Colton et al. [2018] Simon Colton, Alison Pease, and Rob Saunders.  Issues of authenticity in autonomously creative systems.  In International Conference on Computational Creativity 2018, pages 272–279. Association for Computational Creativity (ACC), 2018.
  * Colton [2008] Simon Colton.  Creativity versus the perception of creativity in computational systems.  In AAAI spring symposium: creative intelligent systems, volume 8, page 7, 2008.
  * Cook [2021] Michael Cook.  A vision for continuous automated game design.  Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 13(2):54–60, Jun. 2021.
  * Dolson et al. [2015] Emily Dolson, Anya Vostinar, and Charles Ofria.  What’s holding artificial life back from open-ended evolution?, 2015\.
  * Earle et al. [2021] Sam Earle, Julian Togelius, and Lisa B Soros.  Video games as a testbed for open-ended phenomena.  In 2021 IEEE Conference on Games (CoG), pages 1–9. IEEE, 2021.
  * Epstein [2011] R. Epstein.  Exercises.  In Mark A. Runco and Steven R. Pritzker, editors, Encyclopedia of Creativity (Second Edition), pages 480–487. Academic Press, 2011.
  * Guckelsberger et al. [2016] Christian Guckelsberger, Christoph Salge, Rob Saunders, and Simon Colton.  Supportive and antagonistic behaviour in distributed computational creativity via coupled empowerment maximisation.  In International Conference on Computational Creativity 2016, pages 9–16, 2016.
  * Guckelsberger et al. [2017] Christian Guckelsberger, Christoph Salge, and Simon Colton.  Addressing the ”why?” in computational creativity: A non-anthropocentric, minimal model of intentional creative agency.  In Proc. Int. Conf. on Computational Creativity (ICCC)., 2017.
  * Jennings [2010] Kyle E Jennings.  Developing creativity: Artificial barriers in artificial intelligence.  Minds and Machines, 20:489–501, 2010.
  * Jordanous and Keller [2016] Anna Jordanous and Bill Keller.  Modelling creativity: Identifying key components through a corpus-based approach.  PloS one, 11(10):e0162959, 2016.
  * Kantosalo and Toivonen [2016] Anna Kantosalo and Hannu Toivonen.  Modes for creative human-computer collaboration: Alternating and task-divided co-creativity.  In Proc. Int. Conf. Computational Creativity, pages 77–84, 2016\.
  * Lamb et al. [2018] Carolyn Lamb, Daniel G Brown, and Charles LA Clarke.  Evaluating computational creativity: An interdisciplinary tutorial.  ACM Computing Surveys (CSUR), 51(2):1–34, 2018.
  * Lehman et al. [2008] Joel Lehman, Kenneth O Stanley, et al.  Exploiting open-endedness to solve problems through the search for novelty.  In ALIFE, pages 329–336, 2008.
  * Lehman et al. [2020] Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, (…), and Jason Yosinski.  The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities.  Artificial Life, 26(2):274–306, 05 2020.
  * Lindgren [1991] Kristian Lindgren.  Evolutionary phenomena in simple dynamics.  Artificial life II, 10:295–312, 1991.
  * Maynard-Smith and Szathmary [1995] J. Maynard-Smith and E. Szathmary.  The Major Transitions in Evolution.  Oxford University Press, New York, NY, 1995.
  * McCormack et al. [2019] Jon McCormack, Toby Gifford, and Patrick Hutchings.  Autonomy, authenticity, authorship and intention in computer generated art.  In International conference on computational intelligence in music, sound, art and design (part of EvoStar), pages 35–50. Springer, 2019\.
  * McCormack [2012] Jon McCormack.  Creative ecosystems.  In Computers and creativity, pages 39–60. Springer, 2012.
  * McCormack [2019] Jon McCormack.  Creative systems: A biological perspective.  Computational creativity: The philosophy and engineering of autonomously creative systems, pages 327–352, 2019.
  * Miconi and Channon [2005] Thomas Miconi and Alastair Channon.  A virtual creatures model for studies in artificial evolution.  In The 2005 IEEE Congress on Evolutionary Computation, volume 1, pages 565–572. IEEE, 2005.
  * Morgan [1996] David L Morgan.  Focus groups as qualitative research, volume 16.  Sage publications, 1996.
  * Ofria and Wilke [2004] Charles Ofria and Claus O Wilke.  Avida: A software platform for research in computational evolutionary biology.  Artificial life, 10(2):191–229, 2004.
  * Packard et al. [2019] Norman Packard, Mark Bedau, Alastair Channon, Takashi Ikegami, Steen Rasmussen, Kenneth Stanley, and Tim Taylor.  Open-ended evolution and open-endedness: Editorial introduction to the open-ended evolution i special issue.  Artificial Life, 25(1):1–3, 2019.
  * Pattee and Sayama [2019] Howard H Pattee and Hiroki Sayama.  Evolved open-endedness, not open-ended evolution.  Artificial Life, 25(1):4–8, 2019.
  * Pérez y Pérez [2018] Rafael Pérez y Pérez.  The computational creativity continuum.  In Proc. Int. Conf. on Computational Creativity (ICCC)., 2018.
  * Puryear and Lamb [2020] Jeb S. Puryear and Kristen N. Lamb.  Defining creativity: How far have we come since plucker, beghetto, and dow?  Creativity Research Journal, 32(3):206–214, 2020.
  * Ray [1992] Tom S. Ray.  An approach to the synthesis of life.  In Proc. of Artificial Life II, pages 371–408, 1992.
  * Ray [1996] Thomas S Ray.  An approach to the synthesis of life.  In The Philosophy of Artificial Life. Oxford University Press, 1996\.
  * Ritchie [2019] Graeme Ritchie.  The evaluation of creative systems.  In Tony Veale and F. Amílcar Cardoso, editors, Computational Creativity: The Philosophy and Engineering of Autonomously Creative Systems, pages 159–194. Springer International Publishing, Cham, 2019.
  * Ruiz-Mirazo et al. [2008] Kepa Ruiz-Mirazo, Jon Umerez, and Alvaro Moreno.  Enabling conditions for ’open-ended evolution’.  Biology and Philosophy, 23(1):67–85, 2008.
  * Runco and Jaeger [2012] Mark A Runco and Garrett J Jaeger.  The standard definition of creativity.  Creativity research journal, 24(1):92–96, 2012.
  * Runco [2023] Mark A Runco.  Updating the standard definition of creativity to account for the artificial creativity of ai.  Creativity Research Journal, pages 1–5, 2023.
  * Saunders [2012] Rob Saunders.  Towards autonomous creative systems: A computational approach.  Cognitive Computation, 4:216–225, 2012.
  * Schmidhuber [2008] Jürgen Schmidhuber.  Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes.  In Workshop on anticipatory behavior in adaptive learning systems, pages 48–76. Springer, 2008.
  * Sharp et al. [2019] Helen Sharp, Jenny Preece, and Yvonne Rogers.  Interaction design: beyond human-computer interaction, 2019.
  * Silvia [2018] Paul J Silvia.  Creativity is Undefinable, Controllable, and Everywhere.  In Robert J. Sternberg and James C. Kaufman, editors, The Nature of Human Creativity, pages 291–301. Cambridge University Press, 2018\.
  * Song [2022] Asiiah Song.  A little taxonomy of open-endedness.  In ICLR Workshop on Agent Learning in Open-Endedness, 2022.
  * Soros and Stanley [2014] L. B. Soros and Kenneth O. Stanley.  Identifying minimal conditions for open-ended evolution through the artificial life world of chromaria.  In Proceedings of the Fourteenth International Conference on the Synthesis and Simulation of Living Systems, pages 793–800, Cambridge, MA, 2014\. MIT Press.
  * Soros and Stanley [2016] Lisa B Soros and Kenneth O Stanley.  Is evolution fundamentally creative?  In Proceedings of the Second Workshop on Open-Ended Evolution at the 15th International Conference on the Synthesis and Simulation of Living Systems (ALIFE XV), 2016.
  * Spector et al. [2007] Lee Spector, Jon Klein, and Mark Feinstein.  Division blocks and the open-ended evolution of development, form, and behavior.  In Proc. of the 9th annual conf. on Genetic and evolutionary computation, pages 316–323. ACM, 2007.
  * Stanley et al. [2017] Kenneth O Stanley, Joel Lehman, and Lisa Soros.  Open-Endedness: The Last Grand Challenge You’ve Never Heard Of, 2017\.
  * Still and d’Inverno [2016] Arthur Still and Mark d’Inverno.  A history of creativity for future AI research.  In Proceedings of the International Conference on Computational Creativity, 2016.
  * Taylor et al. [2016] Tim Taylor, Mark Bedau, Alastair Channon, David Ackley, Wolfgang Banzhaf, Guillaume Beslon, Emily Dolson, Tom Froese, Simon Hickinbotham, Takashi Ikegami, et al.  Open-ended evolution: Perspectives from the oee workshop in york.  Artificial life, 22(3):408–423, 2016.
  * Taylor [1988] Calvin W Taylor.  Various Approaches to and Definitions of Creativity.  In Robert J. Sternberg, editor, The Concept of Creativity: Contemporary Psychological Perspectives, pages 99–121. Cambridge University Press, 1988.
  * Taylor [1999] Tim Taylor.  From Artificial Evolution to Artificial Life.  PhD thesis, University of Edinburgh, 1999.
  * Taylor [2004] Tim Taylor.  Niche construction and the evolution of complexity.  In Artificial Life IX: Proceedings of the Ninth International Conference on the Simulation and Synthesis of Artificial Life, pages 375–380, Cambridge, MA, 2004. MIT Press.
  * Taylor [2019] Tim Taylor.  Evolutionary innovations and where to find them: Routes to open-ended evolution in natural and artificial systems.  Artificial life, 25(2):207–224, 2019.
  * Taylor [2021] Tim Taylor.  Evolutionary Innovation Viewed as Novel Physical Phenomena and Hierarchical Systems Building, 2021.
  * Ventura [2016] Dan Ventura.  Mere generation: Essential barometer or dated concept.  In Proceedings of the Seventh International Conference on Computational Creativity, pages 17–24. Sony CSL, 2016.
  * Wiggins et al. [2015] Geraint A Wiggins, Peter Tyack, Constance Scharff, and Martin Rohrmeier.  The Evolutionary Roots of Creativity: Mechanisms and Motivations.  Philosophical Trans. of the Royal Society B: Biological Sciences, 370(1664):1–9, 2015.
  * Yaeger [1994] Larry Yaeger.  PolyWorld: Life in a new context.  Proc. Artificial Life, 3:263–263, 1994.

  *[ALife]:
  *[CC]:
  *[OE]:
  *[AI]:
  *[OEE]: