#  Revisiting the Prepositional-Phrase Attachment Problem
Using Explicit Commonsense Knowledge

Yida Xin
<EMAIL_ADDRESS>
&Henry Lieberman
<EMAIL_ADDRESS>
&Peter Chin
<EMAIL_ADDRESS>
###### Abstract

We revisit the challenging problem of resolving prepositional-phrase (PP)
attachment ambiguity. To date, proposed solutions are either rule-based, where
explicit grammar rules direct how to resolve ambiguities; or statistical,
where the decision is learned from a corpus of labeled examples [Hamdan and
Khan, 2018]. We argue that explicit commonsense knowledge bases can provide an
essential ingredient for making good attachment decisions. We implemented a
module, named Patch-Comm, that can be used by a variety of conventional
parsers, to make attachment decisions. Where the commonsense KB does not
provide direct answers, we fall back on a more general system that infers
“out-of-knowledge-base” assertions in a manner similar to the way some NLP
systems handle out-of-vocabulary words. Our results suggest that the
commonsense knowledge-based approach can provide the best of both worlds,
integrating rule-based and statistical techniques. As the field is
increasingly coming to recognize the importance of explainability in AI, a
commonsense approach can enable NLP developers to better understand the
behavior of systems, and facilitate natural dialogues with end users.

##  1 Introduction

Many applications and social scenarios nowadays are eager to see reliable and
trustworthy Artificial Intelligence (AI) systems. But, in order to really
establish trust, it is not enough to have computers only be able to
communicate the end results of their “thought process.” Instead, computers,
from the ground up, should be expected to use explainable processes to obtain
their results, because that makes it easier — and in some cases, possible at
all — for people to think about what reasoning processes the computers must
have employed to get to those results. An even better solution is to build
computers that can actually tell us, in natural language, what their reasoning
and thinking processes are.

Today’s neural-network based language models (LM) — such as Transformers and
its descendants [Vaswani et al., 2017, Devlin et al., 2018, Liu et al., 2019]
— learn implicit knowledge from enormous corpora, but their representations,
such as neural network weights, may only make sense to the systems themselves.
Work has been done on constructing _post-facto_ explanations for opaque
inference processes [Petroni et al., 2019, Da and Kasai, 2019, Shwartz et al.,
2020], but it is questionable how general the models and explanations really
are. They must grapple with a tradeoff between faithfulness to the original
process, and understandability by people. On the other hand, if we build a
language-understanding system based on explicit commonsense knowledge and
inference, from the ground up, reporting the commonsense involved is a good
way to provide explanation to users. But it is hard to acquire, explicitly,
sufficient knowledge to achieve wide coverage. So knowledge acquisition needs
to be automated, which requires the use of Machine Learning mechanisms.

LMs have been proven exceptionally powerful at what they do. Recently, more
empirical findings have validated our intuition about these systems: They are
great knowledge learners for the domains that they are evaluated in [Sakaguchi
et al., 2019, Shwartz et al., 2020]. It seems unwarranted not to consider
using their great power, just because they are somewhat opaque.

That leads us to the statement of our research agenda: We make language
understanding explainable by using explicit commonsense knowledge, and we make
the knowledge acquirable by using Machine Learning mechanisms, even if they
may be somewhat opaque.

As a first step, we focus on a small but notoriously challenging problem in
the realm of language understanding: the prepositional-phrase (PP) attachment
problem [Hamdan and Khan, 2018]. We also focus on a particular knowledge base
and a particular neural-network based architecture for extending the knowledge
contained in that knowledge base.

##  2 Background and Related Work

Hamdan and Khan [Hamdan and Khan, 2018] wrote a comprehensive survey that
highlights many historical attempts at the PP attachment problem. Very roughly
speaking, there are two approaches: (1) PP attachment can either be treated as
a classification/ranking problem where one classifies/ranks the place in the
sentence that makes the most sense to associate with the said PP; (2) PP
attachment can also be dealt with by writing specific grammars and rules for a
parser, where such grammars and rules tell the parser where, by default, the
PP is to be attached within the sentence.

In this paper, we work with a rule-based parser that has its own default way
of always attaching PPs to the highest head in the parse tree. The parser is
part of a question-answering system, and that heuristic was deemed to be often
appropriate in that application. But recognizing that the default way is not
always desirable, we implement a system that makes informed decisions for the
parser, so that the parser can consult with our system. And by informed, we
mean that we have our module querying for knowledge from a knowledge base.

###  2.1 The START Parser

The parser we work with is the START parser [Katz, 1997]. The first reason
that we use START is that it is a syntactic parser, whose nested-triples
representation allows for additions of semantic knowledge. As an example of
how the START parser works, suppose we request START to parse the sentence

“John is playing the guitar with one finger,” (1)

START will output

                        [John         play        guitar]
                        [play         with        finger]
                        [finger   has_quantity      1   ]

where [John play guitar] and [play with finger] are the results of syntactic
knowledge, and [finger has_quantity 1] is semantic knowledge.

The second reason we use the START parser is that it is the parser used by the
Genesis story-understanding system [Winston and Holmes, 2018]. Our long-term
research agenda, which we will articulate in section 4, is to scale up the
language-processing capability of existing story-understanding systems like
Genesis.

###  2.2 The ConceptNet Knowledge Base

The knowledge base we use is the ConceptNet commonsense knowledge base [Speer
et al., 2017]. The first reason that we use ConceptNet is that it is very
large: It contains crowdsourced “pure” commonsense knowledge input directly by
people, implicit knowledge from behavior in games, and other sources. It is
then combined with more “factual” knowledge from WordNet, Wiktionary, and
others. See Speer et al. [Speer et al., 2017].

The second reason is that there have been successful attempts at using
ConceptNet’s knowledge to reason about out-of-knowledge concepts, using
Machine Learning methods [Colon-Hernandez et al., 2019]. We would like to take
advantage of that research result to give ConceptNet a form of knowledge-
acquisition capability, thus advancing our research agenda.

Each piece of knowledge in ConceptNet is represented as an assertion that
looke like:

                      [<relation>, <concept1>, <concept2>]

A concrete example could be:

                    [/r/HasA/, /c/en/guitar/, /c/en/strings/]

where HasA is the relation (/r/) that connects the concept (/c/) guitar to the
concept strings, and en means English. Note that this assertion can only be
interpreted as guitar HasA strings and not as strings HasA guitar.

Since the work on AnalogySpace [Speer et al., 2008], every assertion in
ConceptNet now comes with pre-learned numerical weights that indicate its
confidence level. In the example here, the assertion guitar HasA strings has a
a weight of $2.83$, indicating that ConceptNet is fairly confident that a
guitar has strings, provided that $1.0$ is the default value that indicates
the mere existence of an assertion in ConceptNet.

###  2.3 Problems with Commonsense Knowledge Bases

Working closely with ConceptNet has helped us think about commonsense
knowledge bases — as well as the notion of commonsense itself — in much depth.

####  2.3.1 The Inference Problem

One problem with ConceptNet is what we call the inference problem. Here is an
illuminating example: The assertion

              [/r/IsA/, /c/en/guitar/, /c/en/string_instrument/]

exists in ConceptNet, with a weight of $6.32$, which is much higher than the
previous $2.83$. But, querying ConceptNet with just guitar (as well as its
plural form guitars) and strings (as well as its singular form string), we
cannot get to guitar IsA string_instrument, because no assertion that connects
string with string_instrument is explicitly given, and ConceptNet alone does
not know how to infer an assertion to connect the two concepts.

The inference problem points to an even more fundamental problem with all
commonsense knowledge bases: They merely store knowledge are just data, but
they do not know the meanings of those data. So, ironically, commonsense
knowledge bases lack commonsense.

####  2.3.2 The Insufficiency Problem

The other problem is the insufficiency problem, which says that ConceptNet
simply does not have all the knowledge in and about the world. In fact, all
knowledge bases and all corpora suffer from the insufficiency problem, to
various degrees. To address insufficiency, we utilize the RetroGAN-DRD system,
introduced by Colon-Hernandez et al. [Colon-Hernandez et al., 2019], that
infers “out-of-knowledge-base” assertions in a manner similar to the way some
NLP systems handle out-of-vocabulary words. We will see that, by using
RetroGAN-DRD instead of just ConceptNet, we are guaranteed to always get an
answer, which ConceptNet alone cannot guarantee. The answers align with our
own commonsense and intuition about language, and they inform START’s PP
attachment decisions in just that manner.

In addition to RetroGAN-DRD, there are such representative works as
ATOMIC/COMET [Sap et al., 2019, Bosselut et al., 2019] in commonsense
knowledge base completion and extension. There, the Transformers-based neural
networks mechanisms learn both to fill in the gaps of knowledge bases
(completion) and to generate and add new knowledge into the knowledge bases
(expansion).

Both RetroGAN-DRD [Colon-Hernandez et al., 2019] and COMET [Bosselut et al.,
2019] achieve a level of neural-symbolic integration. They ensure that a user
interacts with their symbolic, understandable information, and that a
developer debugs the underlying Machine Learning mechanisms more easily by
using the symbolic information as guidance.

##  3 Implementation Details

We call our system Patch-Comm, and we will use this name for the remainder of
this paper.

###  3.1 A Simple Case, Using Only ConceptNet

Suppose we request START to parse the sentence:

“John is playing the guitar with one string.” (2)

Then, by default, START alone would output:

                    [John          play          guitar]
                    [play          with          string]
                    [strings   has_quantity        1   ]

Figure 1: START’s output for sentence (2), without using Patch-Comm.

Evidently for us, guitar, instead of play, should be the more sensible
attachment for string. So clearly, START needs help, and we will explicitly
use ConceptNet’s commonsense knowledge to help START to make the more sensible
parsing decision.

To do so, we probe START’s partial parsing information. Specifically, START
begins and continues parsing just like before, until it reaches a point of
ambiguity. But this time, START consults with Patch-Comm for a parsing
decision by sending its latest partial parse to Patch-Comm:

                (:ambig-PP
                    (PP :prep "with" :det "one" :noun "string")
                 :possible-attachments
                    ((V :verb "play")
                     (NP :det "the" :noun "guitar")))

Figure 2: START’s partial parse for sentence (2).

Patch-Comm receives it, and takes out the keyword concepts from both the
ambiguous prepositional phrase (:ambig-pp) and from each of the possible
attachments.111A future work is to try to utilize the other information
available in this partial parse, such as the preposition itself. For example,
we can treat the querying of ConceptNet for the most sensible relations
between keyword concepts as a first loop; and then, we can build a second loop
where Patch-Comm compares the queried relations against the prepositions, and
use linguistic information to decide whether, and to what extent, those
relations make sense with respect to the prepositions. Namely, Patch-Comm
extracts the concept string (and plural strings) as well as the concepts play
and guitar (and plural guitars).222Note that we do not inflect the verb,
because in ConceptNet verb concepts are registered in their lemmatized forms.

We query ConceptNet with a pair of concepts, and ask for the highest-weighted
assertion that relates them.333We did not implement multi-hop searches between
the two concepts. One reason is that, at the commonsense level, it is hard to
know what to make of multiple chained-together assertions with potentially
wildly different relations, only to try to justify some sort of connection
between those two concepts. So we decided to stick with the most
straightforward knowledge possible — just the assertions that already exist.
Specifically, for each extracted concept (as well as its lemmatized or
inflected version) from the possible attachments, we pair it up with the
extracted concept from the ambiguous PP. In the example here, Patch-Comm
extracts six pairs of concepts, queries ConceptNet with each such pair for a
total of six times, and obtains the following information:

Concept Pair | Relation | Weight
---|---|---
(string, play) | — | —
(strings, play) | — | —
(string, guitar) | AtLocation | 2.0
(string, guitars) | — | —
(strings, guitar) | HasA | 2.8284
(strings, guitars) | — | —
Table 1: ConceptNet’s symbolic knowledge about the ambiguous PP from sentence
(2). The concept that appear to the left of a relation is in blue, and the
concept to the right in red.

Evidenced by the four empty rows in Table 1, we see that, even after taking
into account the singulars and plurals of the nouns, we are still running into
the disconnected-knowledge problem (section 2.3.1). Still, to agree with
lemmatized nouns in the partial parse, we compress Table 1 into Table 2:

Concept Pair | Relation | Weight
---|---|---
(string, play) | — | —
(string, guitar) | HasA | 2.8284
Table 2: ConceptNet’s symbolic knowledge about the ambiguous PP from sentence
(2), refined. The concept that appear to the left of a relation is in blue,
and the concept to the right in red.

Note that we indeed only preserve the highest-weighted relation overall.

Having obtained the information in Table 2, Patch-Comm makes the decision, on
behalf of START, that the ambiguous PP (ambig-PP “with one string”) is to be
attached to the noun phrase (NP “the guitar”) rather than to the verb (V
“play”). Note, crucially, that this decision contradicts START’s default
decision. Our decision is then sent back to START, so that START can continue
and complete its parse result:

                    [John          play          guitar]
                    [guitar        with          string]
                    [string    has_quantity        1   ]

Figure 3: START’s output for sentence (2), using Patch-Comm querying
ConceptNet.

Although the only difference between Figure 3 and Figure 1 is changing the
word “play” to the word “guitar” in the second row, it is a big difference.

###  3.2 A Less Simple Case, Using Only ConceptNet

As a more complex example, suppose we have now:

“John is playing the guitar with one string with one finger.” (3)

What makes sentence (3) more complex is that it has two, instead of one,
ambiguous PPs. But just like before, we request START to parse sentence (3).
This time, START pauses as soon as it meets the first ambiguity, and sends
Patch-Comm the partial parse:

                (:ambig-PP
                    (PP :prep "with" :det "one" :noun "finger")
                 :possible-attachments
                    ((V :verb "play")
                     (NP :det "the" :noun "guitar")
                     (PP :prep "with" :det "one" :noun "string")))

Figure 4: START’s 1st partial parse for sentence (3).

And then, just like before, Patch-Comm extracts concepts, queries ConceptNet
with the concepts, and obtains a refined table of information:

Concept Pair | Relation | Weight
---|---|---
(finger, play) | RelatedTo | 1.0
(finger, guitar) | — | —
(finger, string) | — | —
Table 3: ConceptNet’s knowledge about the 1st ambiguous PP from sentence (2),
refined.

This time with more blank rows in the table, we are also seeing the
insufficient-knowledge problem (section 2.3.2) with ConceptNet, in addition to
the disconnected-knowledge problem. Recall that a weight of $1.0$ is just an
indication that an assertion exists in ConceptNet [Speer et al., 2008], so the
information we are receiving from ConceptNet in this particular case is pretty
weak. But nonetheless, we will still make use of that weak information. Patch-
Comm goes ahead and decides that the ambiguous PP should be attached to the
play, and sends that decision back to START.

START resumes its parsing process, only to encounter the second ambiguity. So
it sends Patch-Comm its partial parse for a second time:

                (:ambig-PP
                    (PP :prep "with" :det "one" :noun "string")
                 :possible-attachments
                    ((V :verb "play")
                     (NP :det "the" :noun "guitar")))

Figure 5: START’s 2nd partial parse for sentence (3).

We notice that this is exactly the same scenario as the one described in
section 3.1, so everything proceeds in exactly the same way as if they are for
sentence (2). At the end, START completes its parse result:

                    [John          play          guitar]
                    [play          with          finger]
                    [guitar        with          string]
                    [string    has_quantity        1   ]
                    [finger    has_quantity        1   ]

Figure 6: START’s output for sentence (3), using Patch-Comm querying
ConceptNet.

###  3.3 Using RetroGAN-DRD, A System That Goes Beyond ConceptNet

The final part of our implementation, as promised, addresses ConceptNet’s
disconnected-knowledge and insufficient-knowledge problems (section 2.3). The
solution we have come up with is incorporating the Deep Relationship Discovery
(RetroGAN-DRD) system , a developing system that facilitates commonsense
learning and reasoning [Colon-Hernandez et al., 2019].

There are in fact two subsystems: A RetroGAN system, followed by a DRD system.
The RetroGAN system utilizes the CycleGAN [Zhu et al., 2017] idea to train a
target-domain generator that mimicks word vectors that already carry
ConceptNet semantics implicitly [Speer and Chin, 2016]. Specifically, RetroGAN
takes pre-trained fastText [Bojanowski et al., 2017] embeddings in its source
domain and retrofitted ConceptNet emsemble [Speer and Chin, 2016] in its
target domain, and post-specializes [Vulić et al., 2018] out-of-knowledge,
unseen concepts by learning a generalizable mapping from seen fastText
concepts and the retrofitted ConceptNet ensemble. The DRD system consists of
trained relation-predicting models; each such model corresponds to a
ConceptNet relation. Essentially, DRD serves to take a pair of post-
specialized concepts from RetroGAN, test it on all the relation models, and
output the predicted weights for all the models (i.e., relations).

We query the RetroGAN-DRD system in basically the same way that we query
ConceptNet itself: Provide two concepts, and ask for the relation that carries
the highest weight.

Taking into account both statistical sensitivity and the fact that RetroGAN-
DRD is under development, we took some care to ensure better experimental
outcomes. The first thing we did was, for each of the two concepts, we also
introduce $5$ of its neighbors to make a neighborhood of $6$ concepts,
represented as a $6\times 300$ tensor. This was done to ensure that “true”
concepts are not left out due to possible statistical instability of
vectorized knowledge. After drawing two neighborhoods for both concepts, we
match up the entire neighborhood of the first concept with that of the second
concept, making a total of $6\times 6=36$ pairs. Each such pair then gets fed
through the RetroGAN-DRD system, and the system outputs the highest-weighted
relation along with the weight. Overall, we get $36$ relations and $36$
weights.

The second thing we did was that, once all these $36$ weights were collected
into a list, we treat the highest $3$ weights and the lowest $3$ weights,
along with their corresponding relations, as outliers. We remove these
outliers to obtain a remainder of $30$ relations and $30$ weights, and we
compute the following:

$\displaystyle
w^{*}=\log\left\\{\max_{i=1}^{30}\left\\{\text{SOFTMAX}(\exp{(\vec{w})})\right\\}\right\\}$

where $\vec{w}=\begin{bmatrix}w_{1}&\dotsi&w_{30}\end{bmatrix}$ is the vector
of those $30$ weights, and

$\displaystyle\exp{(\vec{w})}=\begin{bmatrix}\exp{(w_{1})}&\dotsi&\exp{(w_{30})}\end{bmatrix}.$

This way, $w^{*}$ is a statistically less sensitive measure for how the
original pair of concepts drawn from the partial parse should relate.

By way of demonstration, when using RetroGAN-DRD on sentences (1) and (2), we
have

(1) John is playing the guitar with one finger  Concept Pair Relation $w^{*}$
(finger, play) SymbolOf -3.26578 (finger, guitar) ObstructedBy -3.33881

(a) RetroGAN-DRD’s reasoning about the ambiguous PP from sentence (1),
refined.

(2) John is playing the guitar with one string  Concept Pair Relation $w^{*}$
(string, play) ObstructedBy -3.38202 (string, guitar) UsedFor -3.37002

(b) RetroGAN-DRD’s reasoning about the ambiguous PP from sentence (2),
refined.

Table 4: RetroGAN-DRD’s reasoning about ambiguous PPs.

When Patch-Comm uses the information in Tables 4(a) and 4(b), it will still
decide that sentence (1) should have “plays with one finger” and that sentence
(2) should have “guitar with one string.” The only difference is that, by
using RetroGAN-DRD, we are now able to overcome both the disconnected-
knowledge problem and the insufficient-knowledge problem, at least to some
extent.

And finally, START’s parse results in this case are:

                    [John          play          guitar]
                    [play          with          finger]
                    [string    has_quantity        1   ]

Figure 7: START’s output for sentence (1), using Patch-Comm querying RetroGAN-
DRD.

                    [John          play          guitar]
                    [guitar        with          string]
                    [string    has_quantity        1   ]

Figure 8: START’s output for sentence (2), using Patch-Comm querying RetroGAN-
DRD.

##  4 Some Results

We evaluated our system on section 23 of Belinkov et al.’s English
dataset444We used the publicly available data set from
https://github.com/boknilev/pp-attachment/tree/master/data/pp-data-english.
[Belinkov et al., 2014]. Each data point consists of a preposition, its child,
and a set of candidate heads to which the PP, consisted of the preposition and
the child, is to be attached.

We only evaluated on the preposition “with” to be consist with our
demonstration examples, and we selected only the data points that have at most
3 candidate heads. Overall, there are 17 of them with 2 candidate heads and 31
of them with 3 candidate heads, for a total of 48. The statistics of our
results are summarized in the following table:

Baseline | Accuracy |  $w^{*}$ within $0.1$ | Best
---|---|---|---
39.2% | 45.8% | 27.1% |  88.7%
Table 5: Our results on a part of the data set developed by Belinkov et al.
Baseline accuracy is computed by assigning $50\%$ to the 17 data points with 2
candidates and $33.333\%$ to the 31 data points with 3 candidates. Accuracy is
computed as we got 22 correct out of 48. $w^{*}$ within $0.1$ means, among the
data points we got wrong, the $w^{*}$ score for the answer that Patch-Comm
picked differs is no more than $0.1$ from the ground-truth answer. The last
column is the best result from the paper, obtained from evaluations on their
entire data set.

These results suggest that the scalability of Patch-Comm awaits.

##  5 Discussion and Future Works

This work is an immediate result of our decision to take on the challenge of
scaling up the language-processing capacity of story-understanding systems.
Scaling-up is important, because all story-understanding systems to date —
with Genesis [Winston and Holmes, 2018] being perhaps the most prominent one —
can only read and analyze toy stories, but we want them to start reading and
analyzing natural-language stories. Scaling-up is a challenge, because
natural-language understanding (NLU) is hard.

It soon becomes clear to us that, despite of the seeming bifurcation of
language and story in our setup, both language understanding and story
understanding require commonsense knowledge, but perhaps language knowledge
and story knowledge should reside on different “levels.” Story knowledge is
really global information about the story at hand, and what we mean by
language knowledge is essentially local, sentence-level information.
Therefore, we think it makes sense in principle for story knowledge to have
the ability to override sentence-level language knowledge. For example, if
there is a story context that tells us that John is a three-year-old kid who
plays with all kinds of strings all the time, then all of a sudden, it seems
all the more likely that John is actually using a piece of string to play the
guitar.

We realize that it is actually important to incorporate story-level knowledge,
which a story-understanding system already has, to scale up its sentence-level
language understanding. Consequently, we have, on our research agenda, the
goal of employing global, story-level knowledge to resolve local, sentence-
level ambiguities, and to compile the refined local interpretations of all
sentences into a global interpretation of the story. In fact, a story-
understanding system like Genesis is already able to offer global
interpretations for toy stories.

Next up, we want to extend our commonsense knowledge-based approach to other
challenging linguistic problems, such as co-reference resolution (a.k.a.,
pronoun disambiguation). Our hope for co-reference resolution is to go beyond
the state-of-the-art benchmarks, such as the series of works on the Winograd
Schema Challenge [Levesque et al., 2012, Sakaguchi et al., 2019, Kocijan et
al., 2020], and build programs that can resolve more global co-references,
like the ones in the following children’s story:

Robbie and Susie were going to Marvin’s birthday party. One of them wanted to
buy a kite. “But he already has one,” he said. “He will make you take it
back.”

Last but not least, the Holy Grail of NLU is building computer systems that
understand discourses. To that effect, story-understanding systems such as the
Genesis system are built with the intention of enabling us to gain insights
into our own human intelligence, through the lens of story understanding
[Winston and Holmes, 2018]. We hope to scale up to a point where we will gain
a sufficient understanding of our own intelligence by looking at the
underlying story-understanding processes that are and will continue to be
transparent. And when we understand and can explain ourselves that much
better, we will hopefully have raised our own conception of trust to a higher
level within society.

## Acknowledgments

We would like to thank Sue Felshin for sharing with us her valuable experience
with and resources of the START natural-understanding system, as well as our
in-depth discussions on this project. We would also like to thank Pedro Colón-
Hernández for his contibuting the RetroGAN-DRD system as well as numerous
insights into the problem of computer language understanding.

## References

  * [Belinkov et al., 2014]  Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson.  2014\.  Exploring compositional architectures and word vector representations for prepositional phrase attachment.  Transactions of the Association for Computational Linguistics, 2\.
  * [Bojanowski et al., 2017]  Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  2017\.  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5:135–146.
  * [Bosselut et al., 2019]  Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi.  2019\.  Comet: Commonsense transformers for automatic knowledge graph construction.  In ACL.
  * [Colon-Hernandez et al., 2019]  Pedro Colon-Hernandez, Henry Lieberman, and Catherine Havasi.  2019\.  Does a dog desire cake?-expanding knowledge base assertions through deep relationship discovery.  In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.
  * [Da and Kasai, 2019]  Jeff Da and Jungo Kasai.  2019\.  Cracking the contextual commonsense code: Understanding commonsense reasoning aptitude of deep contextual representations.  In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, Hong Kong, China, November. Association for Computational Linguistics.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 (cs.CL).
  * [Hamdan and Khan, 2018]  Mohammed Hussein Hamdan and Imtiaz Hussain Khan.  2018\.  An analysis of prepositional-phrase attachment disambiguation.  International Journal of Computational Linguistics Research, 9(2), June.
  * [Katz, 1997]  Boris Katz.  1997\.  Annotating the world wide web using natural language.  In Computer-Assisted Information Searching on Internet, RIAO ’97, page 136–155, Paris, FRA.
  * [Kocijan et al., 2020]  Vid Kocijan, Thomas Lukasiewicz, Ernest Davis, Gary Marcus, and Leora Morgenstern.  2020\.  A review of winograd schema challenge datasets and approaches.  arXiv preprint arXiv: 2004.13831 (cs.CL).
  * [Levesque et al., 2012]  Hector J. Levesque, Ernest Davis, and Leora Morgenstern.  2012\.  The winograd schema challenge.  In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, page 552–561. AAAI Press.
  * [Liu et al., 2019]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv: 1907.11692 (cs.CL).
  * [Petroni et al., 2019]  Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel.  2019\.  Language models as knowledge bases?  ArXiv, abs/1909.01066.
  * [Sakaguchi et al., 2019]  Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  2019\.  Winogrande: An adversarial winograd schema challenge at scale.  ArXiv, abs/1907.10641.
  * [Sap et al., 2019]  Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi.  2019\.  Atomic: An atlas of machine commonsense for if-then reasoning.  In AAAI.
  * [Shwartz et al., 2020]  Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  2020\.  Unsupervised commonsense question answering with self-talk.  arXiv preprint arXiv: 2004.05483 (cs.CL).
  * [Speer and Chin, 2016]  Robyn Speer and Joshua Chin.  2016\.  An ensemble method to produce high-quality word embeddings.  arXiv preprint arXiv:1604.01692.
  * [Speer et al., 2008]  Robyn Speer, Catherine Havasi, and Henry Lieberman.  2008\.  Analogyspace: Reducing the dimensionality of common sense knowledge.  In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 1, AAAI’08, page 548–553. AAAI Press.
  * [Speer et al., 2017]  Robyn Speer, Joshua Chin, and Catherine Havasi.  2017\.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In Thirty-First AAAI Conference on Artificial Intelligence.
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  arXiv preprint arXiv:1706.03762 (cs.CL).
  * [Vulić et al., 2018]  Ivan Vulić, Goran Glavaš, Nikola Mrkšić, and Anna Korhonen.  2018\.  Post-specialisation: Retrofitting vectors of words unseen in lexical resources.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 516–527. Association for Computational Linguistics, June.
  * [Winston and Holmes, 2018]  Patrick Henry Winston and Dylan Holmes.  2018\.  The genesis enterprise: Taking artificial intelligence to another level via a computational account of human story understanding.  Technical report, MIT DSpace, Computational Models of Human Intelligence Community, CMHI Report Number 1.
  * [Zhu et al., 2017]  Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  2017\.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  2017 IEEE International Conference on Computer Vision (ICCV), pages 2242–2251.