# StreamSide: A Fully-Customizable Open-Source Toolkit for
Efficient Annotation of Meaning Representations

Jinho D. Choi
Computer Science
Emory University
Atlanta GA 30322 USA
<EMAIL_ADDRESS>
&Gregor Williamson
Computer Science
Emory University
Atlanta GA 30322 USA
<EMAIL_ADDRESS>

###### Abstract

This demonstration paper presents StreamSide, an open-source toolkit for
annotating multiple kinds of meaning representations. StreamSide supports
frame-based annotation schemes e.g., Abstract Meaning Representation (AMR) and
frameless annotation schemes e.g., Widely Interpretable Semantic
Representation (WISeR). Moreover, it supports both sentence-level and
document-level annotation by allowing annotators to create multi-rooted graphs
for input text. It can open and automatically convert between several types of
input formats including plain text, Penman notation, and its own JSON format
enabling richer annotation. It features reference frames for AMR predicate
argument structures, and also concept-to-text alignment. StreamSide is
released under the Apache 2.0 license, and is completely open-source so that
it can be customized to annotate enriched meaning representations in different
languages (e.g., Uniform Meaning Representations). AllStreamSide resources are
publicly distributedthrough our open source project at:
https://github.com/emorynlp/StreamSide.

##  1 Introduction

Abstract Meaning Representation (AMR) encodes the meaning of a natural
language sentence (or document) as a direct acyclic graph, in which terminal
nodes correspond to concepts and edges represent semantic relations between
the concepts (Banarescu et al., 2013). Below is an AMR in Penman notation
(Matthiessen and Bateman, 1991):

(w / want-01
---
(w :ARG0 (b / boy)
(w :ARG1 (b2 / believe-01
(w :ARG1 (b2 :ARG0 (g / girl)
(w :ARG1 (b2 :ARG1 b))
Figure 1: An abstract meaning representation for the sentence, “the boy wants
the girl to believe him”.

At the time of writing, there is one existing tool for producing AMR
annotation in Penman notation; the online AMR editor of Hermjakob
(2013).111Editor: https://amr.isi.edu/editor.html Despite its various
functionalities, and its contribution to the early-stage development of AMR,
there are a few downsides to this editor.Firstly, it is not customizable
except by administrators and thus has not kept up with the latest advancements
in the field, as it was last updated in 2018. Next, the way that the editor
builds graphs is rigidly top-down and therefore does not give any flexibility
for annotators to create sub-graphs which can later be connected. This means
that the annotator must figure out the entire structure of the final graph
before they start building it.Most significantly, it is only usable as a guest
without a login and, at present, logins must be requested from the
administrators. Finally, annotators need to be online in order to use the
editor.

With these drawbacks in mind, we introduce StreamSide, an offline editor for
annotating graph-based meaning representations which is open-source and fully
customizable. It is Python-based and so can be easily updated to reflect
developments in the field as this is the preferred programming language for
computational linguists without an engineering background. Everyone is
encouraged to add further functionality to the toolkit as we plan to build a
community of users, expanding the project to various domains in multiple
languages.

##  2 Installation

StreamSide can be installed through the Python Package Index (PyPi) using the
pip command, and is compatible with Python versions $\geq$ 3.7:

    pip install streamside

Detailed instructions on updating pip and installing StreamSide in a virtual
environment can be found at https://pypi.org/project/streamside/.

Figure 2: The StreamSide GUI with a completed annotation in the workspace.

##  3 Workflow

###  3.1 Launching Annotation GUI

The StreamSide graph annotator is launched from the terminal. When entering
the launch command, you can specify the annotator ID (\--annotator or -a), the
annotation scheme (\--scheme or -s), and the resource directory (\--resources
or -r):

    python -m streamside.annotator
       -a ANNOTATOR_ID
      [-s ANNOTATION_SCHEME]
      [-r RESOURCE_DIR]

The -s and -r are optional. Currently, StreamSide has annotation support for
two schemes: amr for Abstract Meaning Representation and wiser for Widely
Interpretable Semantic Representation, which is a frameless representation
similar to the Penman Sentence Plan Language Kasper (1989). If -s is not
specified, wiser is chosen by default.

The resource directory contains two JSON files, concepts.json and
relations.json, that define concept IDs and relation labels used for the
annotation, respectively. StreamSide includes the necessary resources for
English AMR and WISeR
annotation.222https://github.com/emorynlp/StreamSide/tree/master/streamside/resources/
For customizability, it also allows you to create your own resource files and
specify them using the -r option, in which case, the -s option will be ignored
and StreamSide will adapt to the scheme provided in the resource files.

###  3.2 Opening and Saving

Annotation batches should be formatted in a plain text file, with each
sentence starting on a new line. Figure 2 shows the StreamSide GUI, where the
first sentence (starting with ID 0) is annotated in AMR. The annotator can
navigate through sentences using the Navigate dropdown menu or the
corresponding shortcut keys that allow you to move to the previous and next
sentence, or jump to a specific one. StreamSide also has the capacity to read
annotations in Penman format, which allows users to open annotations from
existing AMR corpora and revise them using its comprehensive GUI.

In-progress annotation batches are automatically saved as a JSON file with the
annotator ID included in the filename and can be continued or revised at
alater time if needed. If a returning annotator opens a text/penman file that
has been already claimed byone’s annotator ID, StreamSide will open the
corresponding JSON file instead. The JSON format used by StreamSide, as well
as its benefits over Penman notation, are explained in Section 4.6.

(a) Creating a concept with live search results.

(b) Creating a concept with the AMR/PropBank frames.

Figure 3: Creating a concept using the Concept Dialog.

##  4 Annotation

###  4.1 Word-to-Concept Alignment

StreamSide allows you to align text spans to concepts or attributes you
create. To do this, you first select text spans using the Select dropdown menu
or pressing x (Figure 4). Disjoint spans can be created by selecting the
relevant sub-spans sequentially. These are then aligned to a concept/attribute
upon its creation (section 4.2). Once a concept/attribute is created,
additional text spans can be added or removed from the alignment as
appropriate. For proper alignment, input should be tokenized although
alignment is optional, and annotation can be produced without it.333As text-
alignment is non-trivial in certain cases (e.g., named entity, co-reference),
guidelines are provided at
https://github.com/emorynlp/StreamSide/blob/master/docs/streamside_guidelines.md.

Figure 4: Selecting a span of text in the GUI

Research shows that transducer-based parsing models tend to perform better at
producing AMR graphs in Penman notation when the order of arguments correspond
more closely to the linear sentence order (Konstas et al., 2017; Bevilacqua et
al., 2021). StreamSide orders arguments according to the order of appearances
of the aligned text spans. As with any feature of StreamSide, the default
argument ordering can be easily overridden according to the needs of your
annotation project.

###  4.2 Creating Concepts and Attributes

Using the Edit menu, annotators can create a concept, attribute, or relation.
When creating a concept, a popup appears featuring a search bar, which is
filled with the selected span of text (Figure 3(a)). The live search results
may consist of concepts which are used by AMR, including predicate senses from
PropBank (Palmer et al., 2005; Bonial et al., 2014) and standard concepts such
as amr-unknown. If a predicate sense associated with an AMR frame is chosen
(e.g., want-01), then its argument structure will appear in the description
box, as shown in Figure 3(b). Annotators are thus able to verify that the
selected sense is the desired one.

Constants such as numbers (e.g., 23), names(e.g., "California"), politeness
and polarity attributes (i.e., +/-), and mode attributes for sentential mood
(i.e., expressive and imperative) are created using Create Attribute under the
Edit menu. Attributes are displayed in the StreamSide workspace with a
variable ID and parentheses so that they can be aligned with the text and
interact with other nodes, be deleted, or be updated. When the output of
StreamSide is converted to Penman notation, however, attributes will appear
without parentheses or a variable ID.

(a) The workspace populated with concepts. The parent node and its aligned
text are highlighted in red, and the child node and its aligned text are in
green.

(b) Creating a relation with the referent box automatically checked in the
Relation Dialog.

Figure 5: Creating a relation using the main workspace and the Relation
Dialog.

An advantage of StreamSide over existing editors such as the online AMR editor
(Hermjakob, 2013) is the degree of flexibility annotators have when creating
graphs. The original AMR editor builds graphs in a strictly top-down fashion,
while StreamSide allows for both top-down and bottom-up graph construction, as
well as a combination of the two. For instance, the annotator can populate the
workspace with the concepts that will be used prior to creating the relations
between them (see Figure 5(a)). Moreover, since StreamSide workspace can
accommodate more than one root, annotators can build multiple sub-graphs
before combining them into the final graph. This amount of freedom when
figuring out how to build a graph is particularly useful if the graph has a
complex structure.

###  4.3 Creating Relations

To create a relation between two nodes, annotators first highlight and select
the parent node using the Select menu before selecting the child node. This
will leave the parent node and its associated span of text highlighted in red,
and the child and its associated span in green, as shown in Figure 5(a).

The annotator then creates a relation using the Edit menu. If the parent node
is associated with an AMR frame, the corresponding argument structure will be
displayed in the description box. When creating a relation, two check boxes
for handling reentrancies are available to annotators (Figure 5(b)). The first
is the referent box which should be checked for instances of co-reference.
This box will be checked automatically if a concept is already in a relation
with another. The second check box is the -of box which creates an inverse
relation.

###  4.4 Deleting and Updating

All created concepts, attributes, and relations can be deleted or updated
using the Edit menu. For concepts and attributes, first highlight their IDs
and choose an appropriate action from the menu.For relations, highlight their
labels instead and then choose an action.

###  4.5 Keyboard shortcuts

Although StreamSide can be used in a point-and-click fashion, annotators are
encouraged to use the available keyboard shortcuts to increase annotation
speed. Shortcuts are included for opening/saving files, navigating sentences,
selecting spans and nodes, creating, deleting, or updating concepts,
attributes, or relations, as well as checking the referent and inverse boxes
when creating relations.When keyboard shortcuts are utilized, the use of mouse
is only needed for highlighting text spans.

###  4.6 Annotation Output

StreamSide saves annotation files in a JSON format.Figure 6 shows the AMR
graph from Figures 1 in our JSON format. These JSON files can be converted
into Penman files using StreamSide’s built-in JSON-to-Penman converter, where
IN_PATH is the path to a directory containing input JSON files and OUT_PATH is
the directory path to save output Penman files (optional):

    python
      -m streamside.json_to_penman
      -i IN_PATH [-o OUT_PATH]

The alignment information is also represented in Penman files created using
the JSON-to-Penman converter. However, the alignment notation used is
different from that in the AMR 3.0 release (Knight et al., 2020), and is
designed to be more intuitive. Once a sufficiently large corpus of aligned
annotations is produced, models can be trained on aligned data for improved
parsing results.

{
---
"graphs":[{
"tid":"demo.1",
"annotator":"ID",
"last_saved":"04/17/2021 11:23:42",
"tokens":["The", "boy", "wants", "the",
"girl", "to", "believe", "him"],
"concepts":{
"c0":{
"name":"want-01",
"token_ids":[2],
"attribute":false,
"first_token_id":2},
"c1":{
"name":"boy",
"token_ids":[1, 7],
"attribute":false,
"first_token_id":1},
"c2":{
"name":"girl",
"token_ids":[4],
"attribute":false,
"first_token_id":4},
"c3":{
"name":"believe-01",
"token_ids":[6],
"attribute":false,
"first_token_id":6}
},
"relations":{
"r0":{
"parent_id":"c0",
"child_id":"c1",
"label":"ARG0",
"referent":false},
"r1":{
"parent_id":"c0",
"child_id":"c3",
"label":"ARG1",
"referent":false},
"r2":{
"parent_id":"c3",
"child_id":"c2",
"label":"ARG0",
"referent":false},
"r3":{
"parent_id":"c3",
"child_id":"c1",
"label":"ARG1",
"referent":true}
},
"covered_token_ids":[1, 2, 4, 6, 7],
"_concept_id":4,
"_relation_id":4
}]
}

Figure 6: The JSON format of the StreamSide annotation for the AMR annotation
in Figure 1.

There are several advantages to storing graphs in the JSON format. Firstly, it
is straightforwardly machine-readable, and does not require familiaritywith
additional APIs. Secondly, this format is capable of saving more meta
information than the Penman notation, and it contains annotation-to-text
alignment for attributes. This is particularly important for the negative
particle not which is often aligned with the polarity attribute ‘-’. In Penman
notation, this alignment information is lost since the attribute is not
associated with a variable name. Finally, although graph similarity can be
evaluated for Penman graphs using the Smatch metric (Cai and Knight, 2013),
the JSON format is already stored as lists of triples which means that
evaluating features of graph similarity beyond those traditionally considered
by Smatch can be done easily. Likewise, researchers can quickly produce a
fine-grained Smatch breakdown like that proposed by Damonte et al. (2017) by
isolating the F-score for specific (groups of) roles, concepts, or attributes.

Finally, StreamSide can convert from Penman to JSON using the following
command, where IN_PATH is the path to a directory containing input Penman
files and OUT_PATH is the directory path to save output JSON files (optional):

    python
      -m streamside.penman_to_json
      -i IN_PATH [-o OUT_PATH]

This allows annotators to edit existing annotations and add alignment data to
existing corpora using the StreamSide GUI.

##  5 Customization

StreamSide is open source and released under the Apache 2.0 license, which
gives a freedom for developers to customize it for their own projects. As
such, future improvements and additional functionality can be updated
collaboratively by researchers and annotators.

The StreamSide directory consists of a resources folder in which frames,
roles, and descriptions to aid annotation are stored in JSON formats. These
can be modified as needed. This has the advantage that researchers can add
novel annotation schemes to StreamSide as they are developed. For instance,
UMR (Unfiform Meaning Representation; Van Gysel et al., 2021) features
document-level representations for cross sentential dependencies. Because
StreamSide can accommodate multi-rooted graphs,it can be used to annotate more
than one sentences simultaneously in addition to a document level
representation. All resources including the source codes for StreamSide are
publicly available at https://github.com/emorynlp/StreamSide.

##  6 Annotation Statistics

The StreamSide annotation tool has been used to annotate a dialogue corpus of
1,000 sentences from a number of sources including EmpatheticDialogues Rashkin
et al. (2018), DailyDialog Li et al. (2017), Boston English Centre444900
English Conversational Sentences from Boston English Centre:
https://youtu.be/JP5LYRTZtjw, and PersonaChat Gu et al. (2020). The
annotations were completed in both AMR and WISeR and are available at the
GitHub repository.555At the time these corpora were produced text-to-
annotation alignment was not standardized across annotators. However, we are
working on standardizing the alignment data post hoc. The average sentence
length of this corpus is 8.35 tokens. First-time annotators ($n=\text{6}$)
were timed when annotating a subset of these sentences. The average annotation
time per sentence is 131 seconds in AMR and 126 seconds in WISeR,
demonstrating that StreamSide is an effective annotation tool for producing
rapid, gold standard annotations.

##  7 Conclusions

This paper presents StreamSide, an open source toolkit for annotating meaning
representations. This toolkit has a number of features which make it ideal for
annotation projects: it can build graphs flexibly, supporting both top-down
and bottom-up construction; it supports optional annotation-to-text alignment;
it can annotate at the sentence-level as well as the document-level; and it
can read and write files in a number of standard formats. Most importantly,
StreamSide is completely open source.As such, we hope that researchers and
annotators may collaborate to develop this toolkit both in terms of annotation
capability and functionality.

## Acknowledgements

We gratefully acknowledge the support from the ETRI (Electronics and
Telecommunications Research Institute).666 ETRI: https://www.etri.re.kr Any
opinions, findings, and conclusions or recommendations expressed in this
material are those of the authors and do not necessarily reflect the views of
ETRI.

## References

  * Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013\.  Abstract Meaning Representation for sembanking.  In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_ , pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
  * Bevilacqua et al. (2021) Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021.  One SPRING to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , AAAI’21.
  * Bonial et al. (2014) Claire Bonial, Julia Bonn, Kathryn Conger, Jena D. Hwang, and Martha Palmer. 2014\.  PropBank: Semantics of new predicate types.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 3013–3019, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Cai and Knight (2013) Shu Cai and Kevin Knight. 2013.  Smatch: an evaluation metric for semantic feature structures.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics.
  * Damonte et al. (2017) Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017.  An incremental parser for Abstract Meaning Representation.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 536–546, Valencia, Spain. Association for Computational Linguistics.
  * Gu et al. (2020) Jia-Chen Gu, Zhen-Hua Ling, Xiaodan Zhu, and Quan Liu. 2020.  Dually Interactive Matching Network for Personalized Response Selection in Retrieval-Based Chatbots.
  * Hermjakob (2013) Ulf Hermjakob. 2013.  AMR editor: A Tool to Build Abstract Meaning Representations.  Technical report, ISI.
  * Kasper (1989) Robert T. Kasper. 1989.  A flexible interface for linking applications to Penman’s sentence generator.  In _Speech and Natural Language: Proceedings of a Workshop Held at Philadelphia, Pennsylvania, February 21-23, 1989_.
  * Knight et al. (2020) Kevin Knight, Bianca Badarau, Laura Banarescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, and Nathan Schneider. 2020.  Abstract Meaning Representation (AMR) Annotation Release 3.0.
  * Konstas et al. (2017) Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017.  Neural AMR: Sequence-to-sequence models for parsing and generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 146–157, Vancouver, Canada. Association for Computational Linguistics.
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017.  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset.  _CoRR_ , abs/1710.03957.
  * Matthiessen and Bateman (1991) Christian M. I. M. Matthiessen and John A Bateman. 1991.  _Text generation and systemic-functional linguistics: experiences from English and Japanese_.  Pinter.
  * Palmer et al. (2005) Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005.  The Proposition Bank: An annotated corpus of semantic roles.  _Computational Linguistics_ , 31(1):71–106.
  * Rashkin et al. (2018) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018.  I Know the Feeling: Learning to Converse with Empathy.  _CoRR_ , abs/1811.00207.
  * Van Gysel et al. (2021) Jens EL Van Gysel, Meagan Vigus, Jayeol Chun, Kenneth Lai, Sarah Moeller, Jiarui Yao, Tim O’Gorman, Andrew Cowell, William Croft, Chu-Ren Huang, et al. 2021.  Designing a uniform meaning representation for natural language processing.  _KI-Künstliche Intelligenz_ , pages 1–18.