# Replication and Verifiability in
Requirements Engineering: the NLP for RE Case

Sallam Abualhaija<EMAIL_ADDRESS>University of LuxembourgLuxembourg
,  F. Başak Aydemir  Boğaziçi UniversityTürkiye<EMAIL_ADDRESS>,
Fabiano Dalpiaz  Utrecht UniversityThe Netherlands<EMAIL_ADDRESS>,  Davide
Dell’Anna  Utrecht UniversityThe Netherlands<EMAIL_ADDRESS>,  Alessio
Ferrari  CNR-ISTIItaly<EMAIL_ADDRESS>,  Xavier Franch  Technical
University of CataloniaSpain<EMAIL_ADDRESS>and  Davide Fucci  Blekinge
Institute of TechnologySweden<EMAIL_ADDRESS>
(2018; Date; Date; Date)

###### Abstract.

[Context] Study replication is essential for theory building and empirical
validation. [Problem] Despite its empirical vocation, requirements engineering
(RE) research has given limited attention to study replication, threatening
thereby the ability to verify existing results and use previous research as a
baseline. [Solution] In this perspective paper, we – a group of experts in
natural language processing (NLP) for RE – reflect on the challenges for study
replication in NLP for RE. Concretely: (i) we report on hands-on experiences
of replication, (ii) we review the state-of-the-art and extract replication-
relevant information, and (iii) we identify, through focus groups, challenges
across two typical dimensions of replication: data annotation and tool
reconstruction. NLP for RE is a research area that is suitable for study
replication since it builds on automated tools which can be shared, and
quantitative evaluation that enable direct comparisons between results.
[Results] Replication is hampered by several factors, including the context
specificity of the studies, the heterogeneity of the tasks involving NLP, the
tasks’ inherent hairiness, and, in turn, the heterogeneous reporting
structure. To address these issues, we propose an ID card whose goal is to
provide a structured summary of research papers, with an emphasis on
replication-relevant information. [Contribution] We contribute in this study
with: (i) a set of reflections on replication in NLP for RE, (ii) a set of
recommendations for researchers in the field to increase their awareness on
the topic, and (iii) an ID card that is intended to primarily foster
replication, and can also be used in other contexts, e.g., for educational
purposes. Practitioners will also benefit from the results since replications
increase confidence on research findings.

Requirements Engineering (RE), Natural Language Processing (NLP), Replication,
Tool Reconstruction, Annotation, ID Card

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Software and its engineering
Reusability††ccs: Software and its engineering Requirements analysis††ccs:
Computing methodologies Natural language processing

##  1\. Introduction

The requirements engineering (RE) community has long been interested in
applying natural language processing (NLP) technologies for analyzing
requirements, as system requirements are primarily written in natural language
(NL) (Kassab et al., 2014; Fernández et al., 2017). The emerging research
strand, namely NLP4RE, has received a lot of attention from both industry
practitioners and academic researchers. NLP4RE tools make use of and adapt NLP
techniques, which in turn rely on a variety of algorithms—recently, machine
learning (ML) and deep learning (DL) are prominent—to process textual
information.  In 2018, the research community has launched the NLP4RE workshop
series as a regular meeting place for researchers who are working on theory
and application of NLP to address a variety of RE problems (Dalpiaz et al.,
2018a).

The landscape of NLP4RE research has been explored in a recent systematic
mapping (Zhao et al., 2021). The study shows that NLP4RE is dominated by
solution proposal studies ($\approx$67.1%), followed by validation
($\approx$17.3%) and evaluation ($\approx$7.2%). The evaluation methods
applied to NLP4RE studies include primarily laboratory experiments with
software as objects, where novel software systems are compared with existing
ones. This implies being able to faithfully replicate previous studies, to
enable sound comparisons with novel proposals.

Replication is an important aspect of empirical evaluation that involves
repeating the experiment under similar conditions using a different subject
population (Wohlin et al., 2012; Shull et al., 2008). Replicability is
regarded as a major quality attribute in software engineering research. The
ACM badge system was introduced at the end of the
2010s111https://www.acm.org/publications/policies/artifact-review-and-badging-
current to reward replicable research. Several major conferences, e.g., ICSE,
ESEC/FSE, ASE, and RE today include an Artifact Evaluation Track, awarding
tailored versions of the ACM badges. Two mapping studies, from Da Silva et al.
(Da Silva et al., 2014) and from Cruz et al. (Cruz et al., 2019), cover
replications in software engineering from 1994 to 2010, and from 2013 to 2018,
respectively. In both studies, the RE field appears to be among the ones in
which replication is most common. However, replication does not seem commonly
practiced in the field of NLP4RE  as most of the replications concern
experiments with human subjects.

With a few exceptions, (e.g., (Dalpiaz et al., 2019a)), replications in NLP4RE
are rare, despite their importance for increasing the confidence placed in
existing NLP4RE tools and facilitating the empirical evaluation of new
solutions. This, in turn, can facilitate industrial adoption of tools, which
is also limited (Zhao et al., 2021).

This paper aims to investigate what are the challenges associated to the
replication of NLP4RE research, and aims to propose a solution to address
them. Our solution takes the form of an artifact that we call ID card, to be
associated to NLP4RE papers, specifically designed to facilitate some aspects
of replication, and that can be used by authors, newcomers to the field,
reviewers, and students (see Table 3). The ID Card is a template composed of
47 questions concerning replication-relevant information, divided into seven
topics. These topics characterize: the RE task addressed in the study for
which the ID Card is filled in; the NLP task(s) used to support the RE task;
information about data, data sets and data annotation; implementation-related
information; and information related to the evaluation of the proposed
solution.  As observed, e.g., by Shepperd et al. (Shepperd et al., 2018),
incomplete reporting of the studies is a major obstacle for replication, and
an ID card can include additional information to overcome this obstacle. We
perform two hands-on replication experiences, followed by focus groups to
discuss the challenges encountered in practice and on a selective review of
representative NLP4RE literature, aimed to extract replication-relevant
information. Based on these activities, we define and refine the ID card. The
hands-on experiences concern two replication scenarios: (i) building a
benchmark for anaphoric ambiguity detection, which involves replicating a
solution from the state-of-the-art; and (ii) replicating a widely-used
solution for classifying requirements into functional or non-functional.

Figure 1. Overview of the terminology used in this paper. The figure focuses
on reuse in the context of replication, although practitioners or researchers
may also reuse artifacts for other purposes.

Terminology (overview in Fig. 1). The two main terms used in this paper are
reuse and replication. Reuse refers to using a tool or dataset provided by
other authors in another study. Replication refers to the attempt, executed by
an independent team of researchers, to obtain the same results of a specific
study. Reuse is important for replication: in an NLP4RE replication study,
researchers reuse NLP4RE artifacts that were proposed by other researchers.
Replication can be exact when one follows the original procedure as closely as
possible, or differentiated when one adjusts the experimental procedures to
fit the replication context (Juristo and Vegas, 2011). Our experiences and the
review of NLP4RE papers reveal that exact replication in most of the cases is
impossible. For example, implementation details are often incomplete in the
original papers, leading to critical decisions to be made during replication.
Thus, we focus this work on differentiated replication. We refer to the
information that can be extracted from the original paper for the purpose of
replication as replication-relevant information. We further use solution and
tool interchangeably, to indicate the automated procedure described in a paper
to solve a particular problem. We use the term reconstruction to denote the
(re)-implementation of a solution as explained in the original paper.

Contributions.  This paper makes the following contributions:

  1. (1)

Building on our first-hand experience, we identify practical challenges for
the replication of NLP4RE studies. We do so by reconstructing two NLP4RE
solutions and via an in-depth study of existing papers in NLP4RE. As
replications in NLP4RE often requires creating a dataset, we differentiate
between the challenges related to dataset (re-)annotation—aimed to determine
the gold standard against which the solution is evaluated—versus tool
reconstruction. The final list of challenges is presented in Section 4.

  2. (2)

To overcome these challenges, we devise an ID card that summarizes in a
structured way the replication-relevant information. We apply the ID card to
46 research papers by filling in the diverse fields about replication-relevant
information. We assess the ID card on 16 papers both internally and externally
with the authors of the original papers. As we discuss in Section 5, the
results indicate that the ID card can be used as a complementary source to the
original paper for facilitating replication. We make the validation material
publicly available in online annex (Authors, 2022).

Structure.  Section 2 discusses the research method and research questions.
Section 3 provides the context on the two replication scenarios considered in
our work. Section 4 describes the challenges of annotation and tool
reconstruction. Section 5 presents the ID card. Section 6 discusses the
challenges in relation with the ID card. Section 7 addresses the validity
considerations. Finally, Section 8 concludes the paper and sketches future
directions.

##  2\. Research Questions and Method

We deal with the practical challenges of replicating studies in NLP4RE.
Replication implies empirical verifiability, as the verification of the
results of a study can be ultimately achieved through the full replication of
the study itself.

We propose two main dimensions that characterise replication in the context of
NLP4RE: (i) the datasets with their annotations, as research in NLP inherently
relies on datasets for evaluation as well as for training machine learning
(ML) solutions; and (ii) the reconstruction of the proposed tools, as most
NLP4RE studies describe an automated NLP-based solution to an RE problem (Zhao
et al., 2021), which, as observed earlier, typically needs to be
reconstructed.

Accordingly, the research goal of this work, defined using the Goal-Question-
Metric (GQM) template (Van Solingen et al., 2002), is as follows:

Analyse:

NLP4RE studies

for the purpose:

of characterizing their challenges

with respect to:

dataset annotation and tool reconstruction

from the viewpoint:

of researchers

in the context:

of replications of RE research.

To achieve this goal, we devise a set of research questions (RQs).

  1. RQ1:

What are the challenges of annotating datasets for training and evaluating
NLP4RE tools?
Nevertheless, as stressed by previous studies (Zhao et al., 2021; Dalpiaz et
al., 2018a; Ferrari et al., 2017a), the number of annotated (i.e., labelled)
datasets in NLP4RE is limited. Sound dataset annotation is relevant to ensure
both reuse and replication. Indeed, if one wants to reuse a certain tool, and
wants to check whether the tool works on other data (e.g., industrial ones),
these need to be annotated following the same annotation guidelines adopted
for the original data. If other authors want to replicate an entire NLP4RE
study, they need to also re-apply the annotation procedure adopted. If this is
not well defined, results are likely to be dissimilar. Comment1-1 We
investigate the challenges of developing annotated requirements datasets,
while making them available to and reusable by the research community. We also
consider the case in which an existing dataset needs to be re-annotated to
support different NLP4RE tasks.

  2. RQ2:

What are the challenges in reconstructing NLP4RE tools?
We study what are the challenges that third-party researchers222Researchers
who are not directly involved with the design and development of the tool.
face when using tools and build upon them in their research. Given that many
NLP4RE tools are unavailable (Zhao et al., 2021), we pay attention to the
process of reconstructing (i.e., developing) a tool using only the
descriptions of the algorithms provided in the research papers presenting the
tool. Thus, the notion of reconstruction we use in this research refers to
cases in which the NLP4RE solution described in the research paper is not
complemented with source code. This can be regarded as an extreme, yet common,
situation of tool reuse. Therefore, we expect to identify a set of problems
that include also the ones encountered in less challenging replication
contexts.

As will become visible later in this paper, the set of challenges derived from
RQ1 and RQ2 highlight the difficulties with extracting replication-relevant
information from NLP4RE papers. As a step toward mitigating this issue, after
the identification of these challenges, we introduced a third research
question:

  1. RQ3:

How to support the extraction of replication-relevant information from NLP4RE
papers?
In this paper, we focus on the development of an artifact that complements
NLP4RE research papers to facilitate the replication process. This artifact
takes the form of an identity card (ID card) for NLP4RE papers, and it is
presented in Section 5.

To answer RQ1–RQ3, we follow the research method sketched in Fig. 2. The
figure also serves as a reading guide for the various sections of this paper.

Figure 2. Overview of the research method of this paper.

① Reconstruction & Annotation Cases. To gain first-hand experience on the
subject matter, we conducted the reconstruction of two established NLP4RE
approaches whose code was not available (Yang et al., 2011; Kurtanović and
Maalej, 2017). These experiences also required the annotation of datasets. The
case of Yang et al. (Yang et al., 2011) concerns ambiguity detection, a long-
standing, highly researched problem in NLP4RE (Zhao et al., 2021). The case of
Kurtanovic and Maalej (Kurtanović and Maalej, 2017) is about requirements
classification, a task that was chosen for the dataset challenge of the RE’17
conference. The two cases were approached by two subgroups of our research
team.

Table 1. Moderators, leads, and participants of the focus groups.  | Moderator | Leads | Participants
---|---|---|---
Focus Group RQ1 | A2 | A1, A3, A4, A6 | A5, A7
Focus Group RQ2 | A5 | A1, A2, A4, | A3, A6, A7
Focus Group ID Card | A6 | A1, A3 | A2, A4, A5, A7

② Focus group RQ1. Building on the collective expertise of the research team,
and based on the case studies, we conducted a first focus group among the
authors of this paper  that focused on the challenges in annotating datasets.
The idea is that the cases and the collected data act as a trigger for
reflection, so that the participants can brainstorm about specific challenges
they encountered, recall previously faced problems, and compare their
viewpoints with their peers. The following process, which follows the
guidelines by Breen (Breen, 2006), was employed. The same process was adopted
for two other focus groups, aimed at addressing questions RQ2 (reconstruction)
and RQ3 (extraction of replication-relevant information through the ID card).
Table 1 reports the moderators, lead participants, and regular participants
for the three focus groups. All authors are experienced in the annotation and
tool reconstruction tasks. .

Focus Group Preparation.:

This task includes the following steps:

  1. (1):

A moderator is appointed among the authors who did not participate in the
annotation (focus group 1) or reconstruction cases (focus group 2), or ID card
definition (focus group 3).

  2. (2):

Lead participants are selected among those who participated in the annotation,
reconstruction, and ID card definition cases. Regular participants are those
who are not lead participants or moderators.

  3. (3):

A meeting is organised in which the moderator is informed about the results by
the lead participants, and is given an overview of the collected data.

  4. (4):

The moderator defines a schedule for the focus group and organises it.

  5. (5):

Before each focus group, the lead participants are required to independently
prepare a list of challenges or relevant discussion topics about the
experience. These are shared during the focus group execution.

Focus Group Execution.:

The focus group session is planned for 90 minutes: 40 minutes for free
discussion, 40 minutes for focused discussion, and finally 10 minutes for
wrapping up the session. The moderator starts the session by introducing the
main question to be addressed by the focus group, i.e., the RQ associated to
the focus group. Following this, the lead participants present their
experience to the others in two slides prepared in advance prior to the actual
session. The moderator then collects a list of challenges mentioned by the
lead participants in one slide and moderated the discussion among the
participants in free-form manner. Further questions were progressively added
and addressed within eight minutes each during the focused discussion. After
the focused discussion, the moderator summarizes the discussion and concludes
the session.

Qualitative Data Collection and Analysis.:

The moderator of the focus group analyses the tape recording, and extracts the
themes that were discussed the most and perceived as the most important. This
was done by the moderator to ensure neutrality, as the moderator was someone
who did not directly participated in the activity discussed in each focus
group.

Review of Qualitative Data.:

The themes are proposed to the participants in a wrap-up meeting. This is
conducted two weeks after each focus group, lasts 9 minutes, and involves all
participants. The participants can complete, enrich or amend the themes to
come to an agreed set of challenges around the specific RQ.

③ Focus group RQ2. A similar focus group to that of Step ② was conducted for
the challenges related to reconstructing NLP4RE tools, following the same
process described earlier. The two focus groups led to a set of challenges
related to RQ1 and RQ2.

④ Development of the NLP4RE ID card. To support researchers who are interested
in replicating existing research, we developed an NLP4RE ID card that is meant
to highlight the replication-relevant information in a structured manner. This
is an iterative activity that required multiple rounds of design and internal
assessment among the authors and their close colleagues. The development of
the ID card was paired with a review of representative NLP4RE papers, with the
objective of assessing the completeness, cohesiveness, and coverage of the ID
card. Furthermore, for the final design iteration, we have provided the NLP4RE
ID card to a number of authors of NLP4RE papers, and asked them to fill in the
ID card, and to evaluate their own experience with the card. This led to 15
filled in ID cards from 15 distinct authors.

⑤ Focus group ID card. The researchers organised a focus group in which they
discussed the ID card and its use. Also in this case, the focus group was
moderated by one of the authors (A6) who did not participate in the
development of the ID card. This step, led to a set of challenges for the ID
card.

⑥ Reviewing literature and past experience. A longitudinal activity that took
place for the whole duration of our research. The authors of this paper have
read and analyzed numerous NLP4RE papers; combined with their prior experience
in the field, this provided important insights in the various activities. In
order to systematically analyze the breadth of the NLP4RE literature, we
organized the papers using a similar classification structure as that by Zhao
et al. (Zhao et al., 2021): (i) detection of language issues (e.g., ambiguous
and vague expressions), (ii) extraction of domain abstractions (e.g., glossary
terms), (iii) requirements classification (e.g., functional vs. non-
functional), (iv) modelling (e.g., extracting domain models from requirements
documents), (v) tracing and relating (e.g., identifying trace links between
different artifacts), and (vi) search and retrieval (e.g., retrieving
requirements from repositories). The considered literature was used in the
development and assessment of the ID card.

To facilitate reading, the method is presented here linearly. However, the
process was conducted iteratively. For example, the development of the ID card
started right after the initial informal discussions that followed the
reconstruction and annotation cases.

The research method of Fig. 2 aligns with Wieringa’s design science research,
particularly with the design cycle (Wieringa, 2014) that comprises problem
investigation, treatment design, and treatment validation. In particular,
problem investigation is conducted in Steps ① through ③, wherein the
researchers conduct reconstruction and annotation cases. These activities,
complemented by the analysis of the literature conducted in Step ⑥, and
consolidated by two focus groups, led to the identification of a set of
challenges for RQ1 and RQ2. These challenges inform the treatment design—the
NLP4RE ID card (Step ④). Treatment validation of the NLP4RE ID card has been
conducted via internal assessment within the authors and with the help of
external experts. This led to improved versions of the ID card: in essence,
multiple iterations of the design cycles were conducted (Step ④ has a cyclic
nature in Fig. 2). The last validation activity, conducted via a focus group
(Step ⑤), has a set of challenges as a tangible output.

##  3\. Reconstruction Cases of NLP4RE Solutions

We discuss our hands-on experience in replicating two state-of-the-art
solutions from the NLP4RE literature: Step ① in Fig. 2. The first solution
(Sec. 3.1) is an early work by Yang et al. (Yang et al., 2011) for detecting
anaphoric ambiguity in NL requirements, whereas the second (Sec. 3.2) is a
more recent work by Kurtanovic and Maalej (Kurtanović and Maalej, 2017) about
non-functional requirements classification. Authors A1, A3, A5, A6, and A7
were involved in the former reconstruction case, while authors A2, A3, and A4
were involved in the latter.  We have chosen these topics because defect
detection and requirements classification are the most common tasks in NLP4RE
(Zhao et al., 2021). Anaphoric ambiguity detection is a specific type of
defect detection. The original paper has 150+ citations to date, according to
Google Scholar, and also recent papers address the same problem, e.g., the
work by Ezzini and colleagues (Ezzini et al., 2022). We arguably deem these
topics and associated papers as highly representative of the NLP4RE field.

###  3.1. Anaphoric Ambiguity Detection

Motivation and overview. Ambiguity in natural-language (NL) requirements is a
long-standing research topic in RE (Kamsties and Peach, 2000; Berry et al.,
2003; Kiyavitskaya et al., 2008; Dalpiaz et al., 2018b; Gervasi et al., 2019;
Ezzini et al., 2021b). A common source of ambiguity is anaphora, defined as
“the use of a word referring back to a word used earlier in a text or
conversation” (Anaphora, 2022). In particular, pronominal anaphoric ambiguity
occurs when a pronoun can refer to multiple preceding noun phrases (also
referred to as antecedents). For example, the requirement: “R1. Administration
functions include soliciting and negotiating submission agreements with
producers, auditing submissions to ensure that they meet archive standards.”,
contains anaphoric ambiguity since the pronoun “they” can refer to one of the
following antecedents: “administration functions”, “submission agreements”,
“producers”, and “submissions”.

As part of the 2020 edition of the NLP4RE workshop (Abualhaija et al., 2021),
we designed a contest for addressing anaphoric ambiguity. The aim was to
create a benchmark for this problem and thus help the community further build
on and compare with existing solutions. To properly assess the participating
approaches, we introduced a baseline from the state-of-the-art that is
dedicated to both resolving anaphora and detecting anaphoric ambiguity. This
baseline is adapted from the work of Yang et al. (Yang et al., 2011). Since
neither the dataset nor the source code were to be found publicly, and the
original tool was developed over 10 years earlier using now outdated
libraries, e.g., the Genia tagger for performing constituency parsing, we
decided to reconstruct the work using the details in the original paper and
adapt it to the needs of our contest. This annotation activity acted as a
trigger for the overall study reported in this paper.  Next, we discuss the
annotation and dataset creation, then we describe the automated solution
highlighting the discrepancies between our reconstructed tool and the original
one.

Annotation and Dataset Creation. To create a labeled dataset, we first looked
for a public source of industrial NL requirements. To this end, we used the
PURE (PUblic REquirements) dataset (Ferrari et al., 2017b). We randomly
selected a subset of 200 requirement statements from seven different domains
such as railway and space. Each requirement statement constituted one
sentence, typically using the ‘shall’ format.  The rationale behind our
selection was that the resulting set should be manageable, time- and effort-
wise, for annotation within the time frame of our contest. Further, covering
diverse domains could be advantageous to assess the performance of a given
solution across domains.

Prior to annotating the requirements, we exchanged some guidelines about the
ambiguity task. More specifically, the task was to decide whether a pronoun
occurrence in a given requirement is ambiguous or not by investigating the
relevant antecedents. Each pronoun occurrence was analyzed by two annotators
from the participants.  Our annotation process was then performed in multiple
rounds. After each round, we had a session to discuss our findings and some
problematic issues. The annotation process resulted in 103 ambiguous
requirements (i.e., containing an ambiguous pronoun occurrence). For all
remaining requirements which were marked as unambiguous, the annotators were
asked to identify the antecedent they deemed correct. The outcome of our
annotation process is a dataset where each pronoun occurrence is labeled as
ambiguous if it is (i) marked as such by at least one annotator, or (ii)
interpreted differently by the two annotators. Otherwise, when two annotators
agreed on the same interpretation, the pronoun occurrence is labeled as
unambiguous. We computed the pair-wise inter-rater agreement using Cohen’s
Kappa (Landis and Koch, 1977) on a subset of $\approx$8% from our dataset (16
requirements randomly selected). We obtained an average Kappa of 0.69,
suggesting “substantial agreement” between the annotators. Following common
practices, we used the disagreements as indicators of ambiguity.

Compared to Yang et al.’s work, our dataset creation differs in two ways.
First, our dataset contained a set of one-sentence requirements randomly
selected from multiple documents, where each requirement contained at least
one pronoun occurrence irrespective of where the pronoun occurs. In contrast,
the dataset from Yang et al.’s (i) maintained the sequential order of
requirements coming from the same document and (ii) contained either one or
two sentences according to where the pronoun occurs. For the cases when the
pronoun occurs at the beginning of a sentence, the preceding sentence was also
included in the dataset to provide enough context for analyzing the anaphoric
ambiguity of that pronoun. Otherwise, only the sentence in which the pronoun
occurs was considered.  Second, to decide whether the pronoun is ambiguous or
not, each pronoun in our work was analyzed by two annotators from the
participants, while a pronoun was analyzed by 13 annotators in Yang et al.’s
work. Given the time limitations, we believe that two judges per pronoun were
sufficient for the purpose of creating a benchmark.

Tool Reconstruction. The solution proposed by Yang et al. (Yang et al., 2010,
2011) combines ML and NLP technologies to detect anaphoric ambiguity in
requirements. The input is a set of textual requirements and the output is a
decision for each pronoun occurrence whether it is ambiguous or not. Our
reconstruction, described next, has been recently reconstructed and publicly
released (Ezzini et al., 2022).

The reconstructed version of the approach is composed of four components,
namely text preprocessing, pronoun-antecedent pairs identification,
classification, and anaphoric ambiguity detection. The first component (text
preprocessing) parses the textual content of the requirements document and
applies an NLP pipeline consisting of four modules: (i) a tokenizer for
separating out words from the running text, (ii) a sentence splitter for
breaking up the text into separate sentences, (iii) a part-of-speech (POS)
tagger for assigning a part of speech to each word in the text (e.g., noun,
verb, and pronoun), and (iv) a chunker (or a constituency parser) to delineate
phrases boundaries (e.g., noun phrases – NPs). For operationalizing the NLP
pipeline, the original tool applies libraries that are no longer available. We
adapted our reconstruction by replacing these libraries with equivalent ones
from currently available NLP toolkits.

The second component (pronoun-antecedent pairs identification) extracts all
pronouns occurring from the input requirements, identifies a set of likely
antecedents for each pronoun, and finally generates a set of pronoun-
antecedent pairs. In the original paper, this component further applies an
antecedent classifier which is trained to combine the antecedents that are
referring to the same entity, e.g., “the received parameters” and “these
parameters”. This task is called coreference resolution in the current NLP
literature. For the purpose of simplifying our reconstruction scenario and
since coreference resolution was not directly relevant to our contest, we left
it out.

The third component (classification) builds a ML-based classifier to classify
a given pair of a pronoun $p$ and a relevant antecedent $a$ into YES when $p$
refers to $a$, NO when $p$ does not refer to $a$, or QUESTIONABLE when it is
unclear whether $p$ refers to $a$. The classifier is trained over a set of
manually-crafted language features which characterize the relation between $p$
and $a$, e.g., whether $p$ and $a$ agree in number or gender. The original
paper lists 17 features divided in three categories: 11 syntactic and semantic
features, two document-based, and three corpus-based features. Detailed
descriptions of the learning features can be found in the original paper (Yang
et al., 2011). In our reconstruction scenario, we disregarded the learning
features that are computed using a proprietary library in order to improve the
re-usability of our reconstructed solution. We further disregarded features
related to sequential and structural information since they are not applicable
on our labeled dataset. Dropping these learning features relaxes the pre-
assumptions about the input. We therefore believe that the reconstructed tool
would be applicable to any dataset irrespective of whether the dataset
consists of sequential requirements or simply a random set of requirements.

Finally, the last component (ambiguity detection) applies a set of rules over
the predictions produced by the ML-based classifier to distinguish the
ambiguous cases from unambiguous ones. The original paper reports two
thresholds for detecting correct and incorrect antecedents for unambiguous
cases. In our reconstructed solution, we applied the same rules but redefined
these thresholds with empirically optimized values for our dataset. A
different procedure was followed for finding the optimal values of the
thresholds. The original paper relied on the number of annotators marking a
pronoun as ambiguous or unambiguous, since each pronoun had been annotated by
13 annotators. In the reconstruction scenario, each pronoun was annotated by
only two annotators (not enough for concluding a threshold value). Thus, the
thresholds were empirically tuned on the training dataset. This subtle
difference shows clearly that the thresholds in the original paper were very
specific to the dataset used in their work, while the thresholds in the
reconstructed tool could be generalized beyond the dataset. This argument is
supported by empirical evidence in the paper which reports the reconstruction
(Ezzini et al., 2022).  The reconstructed code in available in an online
repository (Ezzini et al., 2021a).

###  3.2. Functional and Non-functional Requirements Classification

Motivation and Overview. Motivated by the importance of identifying quality
aspects in a requirements specification starting from the early stages of
software engineering, ML and NLP techniques have been widely used in proposing
solutions to several related problems (Montgomery et al., 2022), such as
classifying functional and non-functional requirements (Cleland-Huang et al.,
2007). Despite the variety of existing requirements classification approaches,
as of 2019, the most effective reported classifiers (Kurtanović and Maalej,
2017) relied on a characterization of the requirements at the word level,
e.g., via text n-grams or Part-of-Speech (POS) n-grams, resulting in a large
number of low-level features (100 or 500 in (Kurtanović and Maalej, 2017)).
Moreover, the great majority of the literature in requirements classification
focused, for evaluating the classifiers, on the PROMISE NFR data set (Cleland-
Huang et al., 2006), a collection of 625 requirements from 15 projects,
created and classified by graduate students.

Aiming at a more interpretable solution than those relying on a large number
of low-level (word-level) features, which make it hard for analysts to
understand why the classifier performs well or poorly, and why requirements
are classified in a certain way, in 2019 we proposed a new NLP approach to the
classification of functional and non-functional requirements (Dalpiaz et al.,
2019a). In proposing and evaluating the new solution, we (i) manually
annotated a set of 1,500+ requirements consisting of 8 different projects,
(ii) reconstructed the tool from Kurtanovic and Maalej (Kurtanović and Maalej,
2017) to compare the proposed solution with the state-of-the-art.

Dataset and Annotation. We manually annotated 1,500+ requirements that consist
of the PROMISE dataset (Cleland-Huang et al., 2006), which we re-annotated, as
well as requirements from seven industrial projects. For the annotation, we
followed an approach based on Li et al.’s taxonomy (Li et al., 2014), which
allows a requirement to possess both functional and quality aspects, as
opposed to the original tagging, which allowed a single label per requirement.
In particular, we annotated a requirement as possessing functional aspects (F)
if it included either a functional goal or a functional constraint, while we
annotated a requirement as possessing quality aspects (Q) if it included a
quality goal or a quality constraint. The decision on the functional aspect
was independent from the decision on the quality aspect; thus, a requirement
could possess only F aspects, only Q aspects, both aspects (F+Q), or none. In
the last case, we considered the requirement as denoting information (Winkler
and Vogelsang, 2016).

Each dataset was tagged by two authors independently. Then, the taggers
organized reconciliation meetings to go over the disputes in tagging. If the
taggers failed to convince each other, a third author was consulted for the
final tag. The authors went over all disputes and managed to resolve them.

Tool Reconstruction. To provide insights about how our approach faired against
the state-of-the-art, we selected a relatively recent approach by Kurtanovic
and Maalej (Kurtanović and Maalej, 2017) which is described extensively in the
original paper and shows excellent performance. The original solution consists
of characterizing the training (PROMISE NFR) dataset by means of a number of
low-level word features, such as n-grams, or POS n-grams. In particular, in
the original publications the authors consider two cases with the top (most
discriminant) 100 and 500 features. The requirements, labeled with the top
features, are then used to train a Support Vector Machine (SVM) model with a
linear kernel, and use the trained model to classify previously unseen
requirements as functional or non-functional.

Since the original classifier was not publicly available, and the authors
could not provide us with the code as a revised version was being developed as
part of a research project, we reconstructed it from the details provided in
the original publication. We also complemented such information with a code of
another classifier related to app reviews (Kurtanović and Maalej, 2018) that
is partially available online and developed by the same research group.

During reconstruction, we applied a few minor modifications to the original
version (details in (Dalpiaz et al., 2019a)). For example, to build parse
trees, we used a different better performing library (Berkeley’s benepar
(Kitaev and Klein, 2018)) than that used in the original publication (the
Stanford parser (Chen and Manning, 2014)). We could not reproduce one of the
features of the classifier, since the explanation in the original paper was
insufficient for us to make a correct re-implementation. We could not use a
dataset that the authors took from Amazon software reviews and used to
artificially balance the minority class of NFRs, as it was not publicly
released.

Finally, we released the reconstructed code in an online repository as two
Jupyter Notebooks (Dalpiaz et al., 2019b).

##  4\. Replication Challenges of NLP4RE Studies

In this section, we present a list of challenges, outlined in Table 2, which
were derived through our focus groups related to dataset annotation (RQ1, Step
②) and tool reconstruction (RQ2, Step ③) complemented by the experience gained
from our two reconstruction cases described in Section 3. The details of the
focus groups are provided in an online annex (Authors, 2022).

Table 2. Challenges pertinent to Replication of NLP4RE studies.

(a) Annotation Challenges (RQ1) ID |  Description
---|---
Ann1 |  Some RE-specific categorization tasks lack solid theories that can guide the annotation process.
Ann2 |  Besides annotation experience and theoretical knowledge, the lack of domain knowledge can limit the accuracy of the annotations.
Ann3 |  The annotation activity is time consuming due to factors such as language barriers, different individuals’ background, and fatigue.
Ann4 |  The annotation protocol can evolve and thereby necessitate the re-annotation of the data which might, again, cause additional time and effort.
Ann5 |  Theoretical and practical training resources and opportunities are limited and not adequate for training novice annotators who are often trained during the annotation task by more experienced annotators.
Ann6 |  The lack of benchmarks entails that limited annotated datasets are available which contain representative industrial requirements and further enable comparison against the state-of-the-art.
Ann7 |  Available imbalanced datasets pose the challenge of both understanding the minority class and consequently the annotation of new examples thereof.
Ann8 |  Determining the right amount of context to be shared alongside the annotation raw data is essential and can significantly affect the annotation results.
Ann9 |  Motivating the annotators poses another challenge since an immediate observation of the impact of a given annotation task is not always possible.
Ann10 |  Annotators are often not experienced in managing the social aspects or resolving conflicts originating from power, authority, or other social relations.

(b) Tool Reconstruction Challenges (RQ2) ID |  Description
---|---
Rec1 |  The reconstruction-relevant information and implementation details of the original approach can be ambiguous, imprecise, and incomplete.
Rec2 |  If a tool was partially or fully developed and/or evaluated using proprietary data, then there is no guarantee that the reconstructed tool would be identical to the original one since the used data cannot be accessed for reconstruction purposes.
Rec3 |  Communication with the original authors is not always useful since the actual information sources may not be available anymore.
Rec4 |  The continuous evolution of the NLP ecosystem entails that some libraries become outdated, unavailable, or not maintained anymore.
Rec5 |  Tool reconstruction per se is not much appreciated by the community which oftentimes limit the publications to tools that are not easy to reconstruct or not maintained over time.

###  4.1. Challenges of Dataset Annotation and Re-annotation (RQ1)

During our focus group for RQ1, we identified a total of 10 challenges which
we organize into four categories: (a) theoretical foundation (Ann1 and Ann2),
(b) annotation process (Ann3 – Ann5), (c) dataset-related (Ann6 – Ann8), and
(d) the human and social aspects-related (Ann9 and Ann10).

(a) Theoretical Foundation Challenges:

Annotation Challenge 1: Some RE-specific categorization tasks lack solid
theories that can guide the annotation process. ptAnn1ptAnn1

2em0 Unlike typical NLP tasks which rely on solid theories from linguistics,
we lacked a common and agreed-upon theory defining different classes when
facing RE categorization tasks. For example, researchers have proposed
numerous definitions and taxonomies for classifying functional and non-
functional requirements (Glinz, 2007; Li et al., 2014). Accordingly, the
requirement “The system shall backup the customer data every 2 hours.” can be
labeled according to the definition by Glinz (Glinz, 2007) as non-functional
since it conveys a specific quality concern (i.e., reliability). Conversely,
the same requirement can be labeled according to the definition by Li et al.
(Li et al., 2014) as both functional and non-functional since it has a
functional part (i.e., to backup customer data) as well as a quality that
qualifies it in terms of performance (i.e., every 2 hours). Since neither of
these two definitions are widely followed in industry, the annotation output
can differ depending on the annotator’s background.

Annotation Challenge 2: Besides annotation experience and theoretical
knowledge, the lack of domain knowledge can limit the accuracy of the
annotations. ptAnn2ptAnn2

2em0 For example, not knowing regulations, standards, and data types can
result in confusing compatibility with compliance requirements. Furthermore,
the annotators often use existing datasets without having experience in that
domain or interactions with those who created the datasets.

(b) Annotation Process Challenges:

Annotation Challenge 3: The annotation activity is time consuming due to
factors such as language barriers, different individuals’ background, and
fatigue. ptAnn3ptAnn3

2em0 An annotation task in NLP4RE often deals with manually analyzing several
requirements artifacts and assigning labels as required by the respective
task. However, the following factors, among others, can cause delays:  (i)
Language barriers: Most requirements texts are written in English. Since
English is not necessarily the mother tongue of neither those who write the
requirements and those who read them, handling requirements in English may
lead to grammatical errors in the text and/or different interpretations.  (ii)
Reconciliation of multiple annotations for the same sample: Individuals’
background might be a barrier to reaching full agreement. even when the
different annotators see each others’ perspectives, sometimes they still
insist on their own. (iii) Fatigue: The amount of effort spent on the
annotation task causes considerable fatigue which can affect the performance
of the annotators. Defining clear fatigue thresholds and fatigue-mitigation
rules is challenging as it requires finding a balance between throughput and
quality of the resulting annotations.

Annotation Challenge 4: The annotation protocol can evolve and thereby
necessitate the re-annotation of the data which might, again, cause additional
time and effort. ptAnn4ptAnn4

2em0 Changes in the annotation protocol may compromise the quality of the
annotation. Any change would immediately raise the question about whether to
re-annotate the previous annotations and thus improve the quality or continue
without doing so due to the considerations of time.

Annotation Challenge 5: Theoretical and practical training resources and
opportunities are limited and not adequate for training novice annotators who
are often trained during the annotation task by more experienced annotators.
ptAnn5ptAnn5

2em0 In an ideal scenario, annotators should be trained prior the annotation
task, because (i) annotators are expected to meet a minimum qualification
level (e.g., domain knowledge, motivation, availability, seriousness, etc.),
and (ii) annotators will be more attentive when they learn about the inter-
rater agreement. However, proper training is often impractical and appropriate
training resources are often difficult to determine. An alternative is to
prepare guidelines and/or have small iterations (pilots to train/test people).
Written guidelines, despite useful, may still be ambiguous. Hence, the same
results are unlikely to be obtained by different annotators even if the same
written guidelines are provided. More details about annotation can be found in
(Pustejovsky and Stubbs, 2012).

(c) Dataset-related Challenges:

Annotation Challenge 6: The lack of benchmarks entails that limited annotated
datasets are available which contain representative industrial requirements
and further enable comparison against the state-of-the-art.  ptAnn6ptAnn6

2em0 For the requirements classification case, the most widely adopted dataset
for the classification of functional and non-functional requirements is
PROMISE dataset. This is a benchmark with questionable quality, since it is a
collection of requirements which were written and classified by students. More
recent work in the NLP4RE literature has provided a re-annotated version of
that dataset (Li et al., 2014; Dalpiaz et al., 2019a), which clearly shows
quite some discrepancy with the original annotated dataset.

Annotation Challenge 7: Available imbalanced datasets pose the challenge of
both understanding the minority class and consequently the annotation of new
examples thereof.  ptAnn7ptAnn7

2em0 The different labels are often unevenly distributed among the data. Given
that there are few examples of the minority data, annotators have less
examples to learn about that label. The distributions of functional and non-
functional requirements, for example, were quite diverse and often skewed
across the different annotated datasets leaving the annotators with doubts
when it comes to the minority class.

Annotation Challenge 8: Determining the right amount of context to be shared
alongside the annotation raw data is essential and can significantly affect
the annotation results. ptAnn8ptAnn8

2em0 We observed in both reconstruction cases that not knowing the wider
context of a given requirement to be annotated can complicate the annotation
process. In the case of referential ambiguity, the context of a requirement
may clarify the ambiguity. Similarly, in the case of non-functional
requirements classification, the location of the requirement in the document
or the surrounding requirements may hint the intended class.

(d) Human and Social Aspects-related Challenges:

Annotation Challenge 9: Motivating the annotators poses another challenge
since an immediate observation of the impact of a given annotation task is not
always possible. ptAnn9ptAnn9

2em0 Inviting annotators from industry is thus difficult and sometimes not
possible. Annotation per se has no direct application (or interest) in
industry since industry practitioners do not need annotated data, rather a
working solution. In addition, merely annotation tasks are not always
appreciated in the research community and are thus hard to publish.  We note
that the majority of the focus group participants do not find annotation as a
fulfilling task.

Annotation Challenge 10: Annotators are often not experienced in managing the
social aspects or resolving conflicts originating from power, authority, or
other social relations. ptAnn10ptAnn10

2em0 The human aspects of an annotation process (e.g., the role of persuasion)
are typically not covered by the annotation protocols and training. However,
the social aspect is often necessary both to improve understanding of each
other perspectives and to reach an agreement. For any annotation task, social
challenges cover different aspects including the decision of objectively
accounting for persuasion, the best schema to decide on consensus (e.g.,
consensus seeking, majority votes, senior one makes the final decision),
motivating the annotators, power and authority relations, and the impact of
peer-pressure.

###  4.2. Challenges of Tool Reconstruction (RQ2)

During our focus group for RQ2, we identified a total of five challenges
concerning (a) information availability (Rec1 – Rec3), (b) technological
divergence (Rec4), and (c) motivation (Rec5).

(a) Information Availability Challenges:

Reconstruction Challenge 1: The reconstruction-relevant information and
implementation details of the original approach can be ambiguous, imprecise,
and incomplete. ptRec1ptRec1

2em0 The original paper is normally written to communicate a study rather than
the goal of facilitating tool reconstruction. Therefore, details about the
architecture are frequently left out. While omitting implementation details
can make a paper more readable, it comes with the cost of complicating
reconstruction. Examples of such implementation details include parameters’
choice, seed values for the experiments, configuration options, features
detailed descriptions, and (the exact versions of) libraries. Missing such
details, on the other hand, complicates the reconstruction. Even when the
source code is available to ease the reconstruction, the accompanying
documentation is unclear impeding thereby reconstruction.

2em0 To illustrate, we provide the following example from our experience with
requirements classification case. One of the major challenges we encountered
was related to understanding the learning features. The authors of the
original approach applied 11 different types of learning features, including
text n-grams, POS n-grams, the fractions of nouns, verbs, adjectives, adverbs
in the requirement, the length of the requirement, the height of the syntax
tree characterizing he requirement, etc. While some of these features are
standard and well-known, other features were not known to us, and their
corresponding descriptions in the original paper were inadequate for an exact
re-implementation. To give a concrete example, we could not reproduce the
learning feature “CP features” described (in the original paper) as “unigrams
of part of speech (POS) tags on the clause and phrase level (CP)”. At the end,
this feature was not implemented in the reconstructed tool.

Reconstruction Challenge 2: If a tool was partially or fully developed and/or
evaluated using proprietary data, then there is no guarantee that the
reconstructed tool would be identical to the original one since the used data
cannot be accessed for reconstruction purposes.  ptRec2ptRec2

2em0 Private or unpublished datasets can be sometimes used in developing (some
steps of) the original tool or during validation. For example, In our
experience with requirements classification, the authors of the original
approach have used, in addition to the public PROMISE dataset, a second
dataset from Amazon software reviews to balance the minority class. As we did
not have access to this additional dataset, we had to resort to a different
solution for balancing the minority class. We note that the use of a second
dataset was applied for selecting the final set of most relevant learning
features. Different training data for ML-based approaches could lead to
selecting different relevant features, hence a different best-performing
classifier. In addition, verifying the correctness of the reconstructed tool
would not be possible when the evaluation set contains proprietary data.

Reconstruction Challenges 3: Communication with the original authors is not
always useful since the actual information sources may not be available
anymore. ptRec3ptRec3

2em0 It is fundamental to communicate with the authors of the original tool
for assistance with the reconstruction. We observed to this end limited
responsiveness, in particular when the authors were contacted about very
specific details (e.g., explanation of learning features). Ideally, the
authors of the original tool should help validate the correctness of the
reconstructed tool.  However, in practice, the developers of a tool are often
PhD of MSc students, who may have left the organization, taking away in-depth
knowledge about the tool.

(b) Technological Divergence Challenges:

Reconstruction Challenges 4: The continuous evolution of the NLP ecosystem
entails that some libraries become outdated, unavailable, or not maintained
anymore. ptRec4ptRec4

2em0 An existing approach from the NLP4RE research often contains text
preprocessing steps based on available NLP technologies (e.g., stopwords
removal or lemmatization). Outdated NLP libraries do not always have
equivalent replacements. Based on our experience, older libraries were applied
for example in the case of requirements classification to construct syntax
trees or to undersample the majority class. Some of these old libraries became
deprecated, incompatible with new versions of libraries doing other tasks, or
performed worse than their respective new versions. Updating libraries
therefore might introduce a discrepancy between what is expected and what is
achieved for the better or worse. More concretely, to more accurately create
the parse trees we opted for Berkley’s benepar library instead of Stanford
parser used in the original paper. This challenge is even more notable in the
case of ambiguity detection Since 2011 (the publication year of the original
paper), the NLP field has witnessed an evolutionary leap in the underlying
technologies. As a consequence, we had to update the libraries used in the
original paper while ensuring that the reconstructed tool yields the same
desired functionality.

(c) Motivation-related Challenges:

Reconstruction Challenges 5: Tool reconstruction per se is not much
appreciated by the community which oftentimes limit the publications to tools
that are not easy to reconstruct or not maintained over time. ptRec5ptRec5

2em0 The activity of reconstructing a tool is not sufficiently rewarding from
a community standpoint. While some venues accept replications in the form of
scientific evaluation paper types, which may require tool reconstruction, such
papers could be harder to publish because novelty aspects are, in the opinion
of the focus group’s participants, rated higher than consolidation of
knowledge.

###  4.3. Take-aways

To mitigate the above-listed challenges or reduce their effect, we recommend
the following:

  * •

Authors should make the annotation task more rigorous by (i) conducting pilot
annotations to set the protocol and to avoid later changes, (ii) maintaining
written guidelines, including examples, to guide the annotators as well as for
future reference, (iii) performing reconciliation sessions to resolve
disagreements, and (iv) reminding the annotators about their ethical
responsibilities: the annotation should be as reliable as possible, as it may
be reused by other researchers in other studies.

  * •

The research community should consider rewarding both the annotation and tool
reconstruction tasks by creating more venues and events where such activities
can be published and shared with the community.

  * •

The natural language description of a tool that is present in a paper is often
insufficient and too informal for other researchers to unambiguously
reconstruct the tool. Therefore, when sharing their tool is not possible,
researchers should allocate sufficient effort in clearly describing how the
tool can be reconstructed from scratch.

  * •

Flexibility is essential and often necessary to properly reconstruct existing
tools. This includes simplifying or adapting existing tools to be applicable
in the reconstruction scenario, e.g., using different data, filling in missing
implementation details, replacing proprietary or legacy libraries.

  * •

The rigor of a reconstruction depends on the goal of the endeavor. When
building a baseline to compare a new solution, more rigor is necessary than
for reconstructing a tool for practical use in a company.

##  5\. Replication-relevant Information Extraction from NLP4RE Literature
(ID-Card, RQ3)

Motivated by addressing the challenges that are discussed in Section 4, we
conducted Step ④ of our research method and propose the concept of ID card.
The ID Card is a template that is composed of 47 questions, divided into seven
topics, concerning information relevant for the reconstruction of a tool
proposed in a research paper.

###  5.1. ID Card Concept

The ID card provides a structured summary of an NLP4RE empirical study,
typically documented in one paper, with respect to reconstruction relevant
information.

The ID Card serves a number of purposes to different actors as summarized in
Table 3. To NLP4RE newcomers (e.g., researchers entering in the field, PhD
students starting their thesis), the ID card is an effective instrument to get
acquainted of the state of the art. Experienced researchers may then use the
ID card when conducting their NLP4RE studies at several stages, namely design,
evaluation and reporting. In the frequent case that the study is presented in
a paper, researchers may submit the ID card to a public repository as
accompanying material. This way, reviewers of the paper have access to more
detailed information, which helps in providing an informed evaluation of the
paper. Last, we foresee the ID card as an useful source of information for
educators when preparing NLP4RE-related material.

Table 3. Actors using the ID card as support for a number of activities.  Actor |  Activity |  Use of the ID Card
---|---|---
NLP4RE newcomer |  Learn state of the art |  Having a quick introduction to NLP4RE tasks using the ID card structure
|  |  Knowing the characteristics of existing solutions using the ID card contents
Study author |  Design the study |  Learning the relevant aspects to be covered in the study using as checklist the ID card structure for the target type of NLP4RE task
|  Evaluate the study |  Searching and filtering state-of-the-art tools that act as baseline
|  |  Reusing or reconstructing those tools
|  |  Comparing results with respect to baseline in a systematic way
|  Report the study |  Summarizing the detailed characteristics and results of the study
Paper author |  Submit paper |  Submitting the ID card as accompanying artefact for the paper
Paper reviewer |  Read paper |  Using the ID card contents to complement the paper contents
|  Evaluate paper |  Using the ID card structure as a checklist for the evaluation
|  Write review |  Using the ID card structure to structure the review
Educator |  Prepare course materials |  Organizing course materials using the ID card structure
|  |  Summarizing the state of the art using the ID card contents

###  5.2. ID Card Design

We designed the structure of the ID card following an iterative method
composed of three steps: 1) Preliminary Definition; 2) Internal Assessment;
and 3) External Assessment.

Preliminary Definition. In the first step, we outlined a list of information
items needed for reconstructing an NLP4RE solution based on our own experience
in reconstructing the two examples introduced in Section 3. The researchers
worked in pairs, and each pair drafted a set of questions and possible answers
for a specific dimension. The considered dimensions were: task, dataset,
annotators & annotation process, tool, and evaluation. The dimensions were
selected based on previous experience and brainstorming between the
researchers. After the pairs drafted the questions, a 1-hour group meeting was
carried out to consolidate them. At this stage, the researchers listed 56
questions. Then, other two additional iterations were carried out, to reach a
stable set of 47 questions. The process was driven by the need to make the
questions understandable by possible target readers.

Internal Assessment. We analysed 46 papers from the NLP4RE literature (5 of
which co-authored by at least one of the authors of this paper), and extracted
replication-relevant data according to the information items from the first
step. We included in our analysis also the two papers which we used in our
replication scenarios (see Section 3). To so, we selected five papers from
each of the nine categories inspired by the classification of Zhao et al.
(Zhao et al., 2021) including requirements classification, tracing, defect
detection, model generation, test generation, requirements retrieval,
information extraction from both legal documents and requirements documents,
and app review analysis. The selection of papers was performed by pairs of
researchers, trying to balance recent papers and highly-cited papers from the
list by Zhao et al. (Zhao et al., 2021), plus 155 papers published after 2019,
and retrieved using a similar strategy as in (Zhao et al., 2021).

Each individual researcher used the preliminary ID card (56 closed questions)
to extract information from a subset of the papers. At least one researcher
extracted information from each of the 46 papers. The goal of this activity
was to assess the applicability of the ID card, possibly specialising it based
on the specific category. After this initial application of the ID card, the
researchers had a plenary discussion to share problems and decide on possible
adjustments to the ID card. The outcome of this meeting was an updated list of
47 questions, which included a combination of 32 closed and 15 open questions.

External Assessment. The ID card was then sent to the authors of the original
papers, and we asked them to provide their own answers (47 questions). We
contacted 32 authors, avoiding to contact the same research group for multiple
papers, and received filled-in ID cards from 15 authors (each author covered a
different paper). For each paper, we analyzed the discrepancies between our
answers and those of the original authors and provided explicit notes listing
the points of disagreement and reflections on possible motivations. A plenary
discussion of 1.5 hours followed, in which the identified discrepancies were
discussed based on the researchers’ notes, to identify major sources of
disagreement. The goal was not a fine-grained analysis of the agreement, but
rather to identify elements that could lead to misunderstanding, gather
external viewpoints on the developed card, and discuss possible residual
issues. After this analysis, the phrasing of part of the questions was
adjusted, so as to avoid the misunderstandings that led to the observed
disagreements.

###  5.3. ID Card Description

The ID card resulting from the iterations consists of 47 questions, divided
into seven topics, described in the following. The complete card is reported
in an online annex (Authors, 2022) alongside its application on the two
reconstruction scenarios described in Section 3, and all the material that we
use in our validation (discussed in Section 5).

I. RE Task.:

It identifies the RE task addressed by the study, e.g., classification,
tracing, defect detection, out of a total of 10 tasks (the ID card allows
adding new type of unanticipated RE tasks). The user can select one option
only. Although the majority of RE studies address one main RE task, this is
not always the case. A typical example is a pipeline with multiple RE and NLP
tasks, using different methods and datasets and developing multiple tools. In
this case, study authors shall fill a separate ID card for each of these
tasks, which simplifies the overall design of the ID card (avoiding e.g.
reiteration of certain questions for each RE task).

II. NLP Task(s).:

It specifies the NLP tasks used to support the RE task, i.e., classification,
translation, information extraction and information retrieval (again,
unanticipated NLP tasks may be specified). NLP tasks are distinguished from RE
tasks as some NLP tasks could be applied for different RE purposes. For
example, classification can be used to organize requirements into different
categories, but also to detect defects, or identify trace links. The user
could select more than one NLP task, as NLP tasks can be used in
combination—e.g., in model generation, one uses information extraction plus
translation.

III. NLP Task Details.:

They identify the input granularity for the RE task addressed (document, word,
paragraph, etc.), and the output type. The options for output type are
different for each type of NLP task. For example, for classification one needs
to specify whether the output is binary-single label (e.g., ambiguous XOR not
ambiguous), multi-class multi-label (e.g., feature request OR bug OR praise),
or other options in between. Also, the user needs to specify what the possible
labels are. For translation, the output can be text, but also tests or models,
and the user needs to specify the cardinality of the translation, i.e., one
input to one output (e.g., one document to one diagram) many input to many
output (e.g., from many use case descriptions to one class diagram and
multiple other diagrams). Examples are provided for each case, to ease
understanding.

IV. Data and Dataset.:

This category characterizes the dataset used in the study. In particular, it
asks the user to specify: the size of the dataset; the year; the source of the
data (e.g., proprietary industrial data, regulatory documents, user generated
content—eight options plus “other”); the level of abstraction of the data
(e.g., user-level, business-level, system-level). Multiple options are
allowed, as with the term data are collectively identified both possible input
and output data of the NLP task(s) considered. This choice was driven by the
need to keep the ID card simple. The same section also asks the user to
specify the format of the data (use case, shall requirements, diagrams, etc.),
the degree of rigor (unconstrained NL, restricted grammar, etc.), and the
actual language, if applicable. Additional information is asked about
heterogeneity of the dataset—in terms of domains considered and number of
sources—, and also about data licensing and URL to access the data.

V. Annotators and Annotation Process.:

This section includes information to characterize the annotators, in terms of
background knowledge, number, and level of bias—if annotation was carried out
on the data. In addition, information is collected about the annotation scheme
adopted, if any, and the process to compute and resolve disagreement.

VI. Tool.:

This section collects information about any implementation provided in the
paper, e.g., scripts, executable programs, APIs, collectively designated with
the term tool. The user specifies the type of NLP solution adopted (e.g.,
machine learning, rule-based), what has been released (e.g., binary file,
source code, etc.), and additional information concerning documentation,
licensing, dependencies and other information that can help the user to access
and execute the tool.

VII. Evaluation.:

This provides information about the evaluation carried out in the study.
Specifically, the metrics used (precision/recall, AUC, etc.), the type of
validation process (cross-validation, train-test split, etc.), and the type of
baselines used for comparison.

The ID card is displayed in Table 6. The card contains questions that either
attempt to fully or partially address the challenges introduced in Section 4
or aim to provide metadata about the research paper such as questions I.1 and
II.1. We recall that the classification of the RE tasks was adopted from the
recent systematic literature review by Zhao et al. (Zhao et al., 2021) and the
NLP task classification was created to fit in the RE tasks. We believe that
using the ID card on the long term helps address/mitigate the replication
challenges discussed earlier, if not directly then by being informative and
emphasizing these challenges. We discuss in the following the rationale for
majority of the questions in the ID card in relation with the challenges
described in Table 2.

The ID card partially addresses the annotation challenges as follows:

  * •

To address challenge Ann1, questions V.5 and V.6 are introduced in the ID card
to highlight the necessity for establishing a clear annotation protocol prior
to the annotation process and further making it publicly available afterwards.
Question V.10 also points out that communicating among annotators is required
for achieving a high-quality dataset. Once the annotation protocol is agreed
on, the likelihood that it frequently evolves (Ann4) is more limited.

  * •

While including domain experts in the annotation process is not always
possible, questions V.1 – V.4 in the ID card inform the researchers interested
in reconstructing a dataset about whether the domain expertise was involved or
not.

  * •

With regard to Ann3, question V.8 indicates the need to mitigate fatigue.

  * •

Questions IV.10 – IV.13 are concerned with the dataset. The questions
spotlight the publicly available datasets alongside the licenses under which
these datasets are released. By leveraging such datasets in future similar
annotation tasks, one can address the challenges Ann5 – Ann7.

  * •

Question V.7 gives insights about the context that is shared with the
annotators during the annotation process. The ID card can be used to create a
common practice to assist researchers in designing new annotation tasks. Thus,
addressing the challenge Ann8.

With regard to the tool-reconstruction challenges, the ID card demands for
more details and precise information concerning the implementation of the
tool. Specifically, questions VI.6 and VI.8 are about the libraries employed
in the implementation, question VI.5 is about the available documentation
related to the tool, and questions VI.3 and VI.9 are about what has been
publicly released.

Table 4. ID Card for NLP4RE Research Papers.  I. RE Task
---
I.1. |  What RE Task is your study addressing? $\Circle$ Requirements Classification $\Circle$ Requirements tracing $\Circle$ Requirements defect detection $\Circle$ Model generation $\Circle$ Test generation $\Circle$ Requirements retrieval $\Circle$ Information extraction from legal documents $\Circle$ App review analysis $\Circle$ Dependency and relation extraction $\Circle$ Information extraction from requirements $\Circle$ Other $\ldots$
II. NLP Task(s)
II.1. |  What types of NLP task is your study tackling? $\Square$ Classification (choose among classes) $\Square$ Translation (models, tests, etc.) $\Square$ Information $\Square$ Extraction (glossary, terms) $\Square$ Information Retrieval (search / rank) $\Square$ Other $\ldots$
III. NLP Task Details
III.1. |  What is the input of your NLP task? $\Square$ Document $\Square$ Paragraphs $\Square$ Sentences $\Square$ Phrases $\Square$ Words $\Square$ Structured/tabular text $\Square$ Other $\ldots$
III.2. |  [Classification] What type of classification is the study about? $\Square$ Binary-Single label $\Square$ Binary-Multi label $\Square$ Multi class-Single label $\Square$ Multi class-Multi label $\Square$ Other $\ldots$
III.3. |  [Classification] What are the labels that can be assigned? $\ldots$
III.4. |  [Information Extraction] What is the level of granularity of the extracted elements? $\Square$ Sentences $\Square$ Phrases $\Square$ Words $\Square$ Other $\ldots$
III.5. |  [Information Extraction] What is the type of the extracted elements? $\ldots$
III.6. |  [Translation] What is the type of output? $\Square$ Text $\Square$ Table $\Square$ Graphical diagram $\Square$ Executable model $\Square$ Test cases $\Square$ Other $\ldots$
III.7. |  [Translation] What is the translation mapping cardinality between initial input and final output? $\Square$ 1 to 1 $\Square$ 1 to many $\Square$ Many to 1 $\Square$ Many to many $\Square$ Other $\ldots$
III.8. |  [Information Retrieval] What are the types of elements retrieved with the query? $\Square$ Document $\Square$ Paragraphs $\Square$ Sentences $\Square$ Phrases $\Square$ Words
III.9. |  [Other NLP Task] What is the type of output? $\ldots$
IV. Data and Dataset
IV.1. |  How many data items do you process? $\ldots$
IV.2. |  In which year or interval of years was the data produced? $\ldots$
IV.3. |  What is the source of the data? $\Square$ Industrial project, proprietary data $\Square$ Industrial project, publicly available data $\Square$ Community-based open source projects $\Square$ Textbook examples or cases $\Square$ Student projects $\Square$ Toy requirements $\Square$ Legal/regulatory documents $\Square$ User generated content $\Square$ Other $\ldots$
IV.4. |  What is the level of abstraction of the data (not limited to requirements)? $\Square$ User-level $\Square$ Normative-level $\Square$ Business-level $\Square$ System-level $\Square$ Module-level $\Square$ Code-level $\Square$ Other $\ldots$
IV.5. |  What is the format of the data? $\Square$ “Shall”-requirements $\Square$ User stories $\Square$ Use caes $\Square$ User views $\Square$ Social media posts $\Square$ Bug/defect reports $\Square$ Messages in user forums
IV.6. |  How rigorous is the format of the data? $\Square$ Unconstrained natural language (NL) $\Square$ Template-based controlled NL $\Square$ Restricted grammar based controlled NL $\Square$ Semantically-augmented NL $\Square$ Other $\ldots$
IV.7. |  What is the natural language of the data (if applicable)? $\ldots$
IV.8. |  Please list which domains your data belongs to: [comma separated] $\ldots$
IV.9. |  From how many different sources does your data come from? $\ldots$
Table 5. ID Card for NLP4RE Research Papers (Continued). IV.10. |  Is the dataset publicly available (also from other authors)? $\Circle$ Fully $\Circle$ Partially $\Circle$ Upon request $\Circle$ No $\Circle$ Other $\ldots$
---|---
IV.11. |  What license has been used? $\Circle$ No license $\Circle$ Reuse only for non-commercial purposes $\Circle$ Reuse for any purpose $\Circle$ Modification only for non-commercial purposes $\Circle$ Modification for any purpose $\Circle$ Other $\ldots$
IV.12. |  Where is the dataset stored? $\Circle$ On a private/corporate website $\Circle$ In a repository like GitHub $\Circle$ In a persistent platform with DOI $\Circle$ Other $\ldots$
IV.13. |  Provide a URL to the dataset, if available, or to the original paper that proposed the dataset $\ldots$
V. Annotators and Annotation
V.1. |  How many annotators have been involved? $\ldots$
V.2. |  How are the entries annotated? $\Circle$ Multiple annotator per entry $\Circle$ One annotator per entry (no quality control) $\Circle$ One annotator per entry (quality control) $\Circle$ Partly multiple annotators, partly one annotator per entry $\Circle$ Other $\ldots$
V.3. |  What is the average level of application domain experience of the annotators? $\Circle$ None or unknown $\Circle$ Informed Outsider $\Circle$ Domain expert $\Circle$ Other $\ldots$
V.4. |  Who are the annotators? $\Circle$ The designer of the technique/tool $\Circle$ People who have direct contact with designers $\Circle$ Independent annotators (e.g., crowd workers) $\Circle$ Other $\ldots$
V.5. |  How was the annotation scheme established among the annotators? $\Circle$ Only via labels $\Circle$ Oral agreement among the annotators $\Circle$ Written guidelines with label definitions $\Circle$ Written guidelines with definitions and examples $\Circle$ Other $\ldots$
V.6. |  Did the authors make the written guidelines public? $\Circle$ YES $\Circle$ NO
V.7. |  Did the authors share other information that could support the annotators other than the elements to annotate? $\Circle$ NO $\Circle$ Surrounding context $\Circle$ Entire document $\Circle$ Other $\ldots$
V.8. |  Did the authors employ techniques to mitigate fatigue effects during the annotation sessions? $\Circle$ YES $\Circle$ NO
V.9. |  What are the metrics used to measure intercoder reliability? $\Circle$ Cohen’s Kappa $\Circle$ Krippendorf’s Alpha $\Circle$ Fleiss’ Kappa $\Circle$ Other $\ldots$
V.10. |  How were conflicts resolved? $\Square$ Majority voting $\Square$ Discussion among annotators $\Square$ Resolution by authors $\Square$ Resolution by independent expert (not an annotator) $\Square$ Disagreements were disregarded $\Square$ Not resolved $\Square$ Other $\ldots$
V.11. |  What is the measured agreement? $\ldots$
VI. Tool
VI.1. |  What is the type of proposed solution? $\Square$ Supervised machine learning (ML) $\Square$ Unsupervised ML $\Square$ Supervised deep learning (DL) $\Square$ Unsupervised DL $\Square$ Rule-based $\Square$ Other $\ldots$
VI.2. |  What algorithms are used in the tool? $\ldots$
VI.3. |  What has been released? $\Square$ Standalone Tool $\Square$ Service on the web $\Square$ Library/API $\Square$ Source code $\Square$ Executable notebook $\Square$ Pre-trained models $\Square$ No tool has been released $\Square$ Other $\ldots$
VI.4. |  What needs to be done for running the tool? $\Circle$ No installation is needed $\Circle$ Import and integrate into your own code $\Circle$ Compile and run $\Circle$ Virtual machine/Docker container $\Circle$ Reproduce the tool from the explanation of the paper $\Circle$ Other $\ldots$
VI.5. |  What type of documentation has been provided alongside the tool? $\Square$ README file $\Square$ Pseudocode/illustration in the paper $\Square$ Wiki or dedicated website $\Square$ Tutorial $\Square$ Ready-to-use examples $\Square$ An academic paper $\Square$ No documentation $\Square$ Other $\ldots$
Table 6. ID Card for NLP4RE Research Papers (Continued). VI.6. |  What type of dependencies does the tool have? $\Square$ None $\Square$ Open-source libraries/software $\Square$ Proprietary libraries/software $\Square$ Specific operating system $\Square$ Specific hardware $\Square$ External knowledge-bases $\Square$ Other $\ldots$
---|---
VI.7. |  How is the tool released? $\Circle$ On a private/corporate website $\Circle$ In a repository like GitHub $\Circle$ In a persistent platform with DOI $\Circle$ Other $\ldots$
VI.8. |  What license has been used? Other $\Circle$ No license $\Circle$ Reuse only for non-commercial purposes $\Circle$ Reuse for any purpose $\Circle$ Modification only for non-commercial purposes $\Circle$ Modification for any purpose $\Circle$ Other $\ldots$
VI.9. |  Where is the tool released? [url] $\ldots$
VII. Evaluation
VII.1. |  What metrics are used to evaluate the approach(es)? $\Square$ Precision/Recall $\Square$ F-score $\Square$ Accuracy $\Square$ AUC $\Square$ MAP $\Square$ LAG $\Square$ Word Error Rate (WER) $\Square$ NIST – METEOR – ROUGE – BLEU $\Square$ Other $\ldots$
VII.2. |  What is the validation procedure? $\Circle$ Train-test split $\Circle$ Cross-validation $\Circle$ Cross-project validation $\Circle$ Entire dataset $\Circle$ Other $\ldots$
VII.3. |  What baseline do you compare against? $\Square$ Existing tool or algorithm $\Square$ Reconstructed tool from other research papers $\Square$ Automated, but self-defined $\Square$ Theoretical/Conceptual $\Square$ Human baseline $\Square$ None $\Square$ Other $\ldots$
VII.4. |  Please provide more details about the baseline you compare against, if any. $\ldots$

###  5.4. Evaluation of the ID Card

We disseminated a short survey (four Likert-scale questions and two open-ended
questions) among the authors who filled in the ID Card, asking them to provide
explicit feedback on it. Inspired by the questionnaire from the Technology
Acceptance Model (Davis, 1989), we included survey items regarding the ID Card
perceived ease of use and intended use in three different use cases: reuse and
replication, literature surveys (as typical instrument that NLP4RE newcomers
use for learning the state of the art) and education (Fig. 3). The respondents
agreed that the ID Card has the potential to be used in all of them.
Nevertheless, opinions were mixed regarding the card’s ease of use. In the
open feedback section of the survey, they indicated a preference for the
support of multiple datasets and a more interactive format which support
conditional sections based on previous answers.

(a) NLP4RE ID Card perceived ease of use.

(b) NLP4RE Card ID intention to use for different use cases.

Figure 3. Results of TAM assessment for the NLP4RE ID Card.

We also evaluated the respondents’ perception about the ID Card
appropriateness and level of details for the three different use cases (see
Fig. 4). Although the vast majority agrees that the use of the ID Card is
appropriate across the three use cases, they perceived that more details may
be necessary to further facilitate reuse and replication of existing studies.
None of the respondents indicated which details should be included, but the
necessity to strike a balance between details and usability was recognised, as
one of the respondent pointed out: “For replication purposes, I think more
details would be required. On the other hand, putting in more detail will make
the ID card creation more time-consuming (potentially defeating the purpose).
There is no easy answer here.”

The respondents indicated other possible usages, such as leveraging the
information contained in the card for searching and filtering previous studies
(e.g., when selecting a baseline to compare in their own studies) and as a
checklist to follow during the planning phase of a new study.

(a) NLP4RE ID Card perceived appropriateness for different use cases.

(b) NLP4RE ID Card perceived level of details.

Figure 4. Assessment of the NLP4RE ID Card appropriateness and level of
details.

We plan to complement this initial empirical assessment with other studies.
For instance, we would like to evaluate whether The ID card may facilitate the
graduate courses where the students read and discuss papers. Preliminary
versions of the ID card have been used to support the weekly discussions of a
graduate course on advanced requirements engineering. The students remarked
that before reading a paper, the ID card provided them the important points
that they need to look for in the paper. When they filled the ID card, it
acted as their structured notes on the paper which they can quickly consult
during in-class discussions and when they study for the exams. However, these
remarks come from a small group of student who worked with earlier versions of
the ID card and further studies are needed to confirm or refute them.

We further envisage that the ID card can be used to create and archive
summaries of papers which one has to review at a time and come back to them
again in a later stage, e.g., for PhD students who are required to write
literature review about papers they read long time ago.

Finally, the ID card can be used as a means to assess the adequacy for
reconstruction of a particular study without having to read the entire paper
(e.g., incomplete description, missing implementation details). In fact, the
mere existence of an ID card increases the chances of reconstructing a paper.
In the future, ID cards can be collected from the authors during paper
submission/review process and stored in a public repository to be publicly
accessible by the RE community. Authors as well as reviewers can also use the
ID card as a checklist of all the information that should be reported in the
paper concerning reconstruction and re-usability. Thus, the ID card can create
more awareness of including such necessary information.

##  6\. Challenges of ID Card design (RQ3)

To identify the challenges related to the ID card design (RQ3, Step ⑤ of our
research), we conducted the last focus group to discuss our findings with
respect to extracting replication information extraction from NLP4RE
literature. The details of the focus group are provided in an online annex
(Authors, 2022).

###  6.1. Challenges of ID Card Design

We list below a total of six challenges concerning the ID card, summarized in
Table 7. We organize the challenges into two categories: (a) Card Design
Challenges (Card1 – Card 4) describe the challenges which we encountered
during the design process of the ID card, and (b) Card-Tagging Challenges333By
tagging we intend filling the ID card based on the information extracted from
the papers.  (Card 5 and Card6) describe the challenges from the tagger’s
perspective and how to approach them.

(1) Card Design Challenges:

The level of details required by the card should cover sufficient information
about the solution without merely repeating what is in the paper.
ptCard1ptCard1

2em0 The ID card aims to summarize an NLP4RE paper for the purpose of
replicating the solution and/or dataset. Ensuring that the ID card is capable
of covering a wide spectrum of the papers in the NLP4RE literature is
important yet challenging. There is a trade-off between the coverage of the ID
card to the paper versus the level of details required to achieve a
comprehensive description of the paper. Asking about many details in the ID
card entails more time and effort in filling in the card and this can be
discouraging. To address this challenge, we opted to design the card at a
generic-enough level to be applicable to many papers of different research
focus. The ID card does not substitute the original paper, but it should help
researchers decide whether or not a paper is useful for their reconstruction
goal. For example, the ID card can be used to report the details of a dataset,
without necessarily reporting details related to any automated solution.

To account for clarity and consistency on one side and coverage on the other
side, the card has to include between different types of questions.
ptCard2ptCard2

2em0 Open-ended questions allow free text answers and permit the respondent to
use their own words without being restricted by some choices, while close-
ended questions are helpful in providing guidance to the researchers about
possible answers. The challenge grows with the latter since such questions
have to be defined in a way that is not confusing and easy to understand. We
addressed this challenge by including a combination of the two question types,
to achieve the two goals of flexibility and guidance.

Conventional inter-rater agreement measures do not apply for validating the
answers provided by multiple taggers on the same paper (e.g., multiple
students are tagging the card in an educational scenario). ptCard3ptCard3

2em0 Unlike traditional inter-rater agreements, the agreement between taggers
of the ID card depends on their own understanding of the card, the effort they
put into reading the accompanying notes, and their knowledge of the tool or
paper that is reported upon. Further, not all the answers are commensurable.
As we explain in Sec. 5.4, we measured usefulness and understandability of the
ID card using external tagging. Some of the external tagging in our work has
been conducted by the authors of the original papers considered in our study.

For different reasons, research papers can Missing details that are relevant
to reconstruction must be properly accounted for and incorporated in the ID
card. ptCard4ptCard4

2em0 We address this challenge by adding a free text option alongside each
question to give the possibility to elaborate on the reason or remarks related
to missing details. We believe that providing a justification can give hints
to the reader of the ID card about what to do with these missing details. For
example, if the evaluation uses the entire dataset instead of cross
validation, the reason might be that the tool does not require training. In
this case, details about the proportion of training data might be missing but
also not needed.

(2) Challenges of ID Card Tagging:

For meaningful tagging of the ID card, the tagger should first map the
proposed NLP solution in a given paper to the underlying RE task.
ptCard5ptCard5

2em0 Ultimately, one has to establish a 1-to-1 mapping between an NLP solution
and the RE task that the original paper tackles. However, many papers discuss
multiple NLP solutions (e.g., extraction and classification) in one pipeline
for solving the same RE task (e.g., solving cross-referencing in requirements
as part of detecting anaphoric ambiguity). Tagging such papers with a single
ID card may fall short. To improve re-usability and foster reconstruction, we
recommend to tag the ID card by focusing on one RE sub-task which is likely
the main task being solved in the paper. The ID card in its current status can
be theoretically filled multiple times, each time for an RE sub-task.

To identify the reconstruction-relevant information (and consequently fill in
the ID card) it might not be clear for the tagger to what extent such
information should be sought beyond the original paper. ptCard6ptCard6

2em0 The source of reconstruction-relevant information is often the original
paper, unless the tagger is also an author who has considerably more knowledge
than that reported in the paper. Although having all possible details is the
best option when it comes to reconstruction, this poses the pragmatic
challenges related to the additional required time that it takes for the
respondents to fill in the ID card, which could be discouraging. We believe
that the tagger should decide on addressing this challenge in line with the
motivation of filling in the card. For example, the author of a paper might be
motivated to increase visibility of the original paper and thus would
compensate the effort needed to add all required details.

Table 7. Challenges pertinent to Designing and Tagging the ID Card ID |  Description
---|---
Card1 |  The level of details required by the card should cover sufficient information about the solution without merely repeating what is in the paper.
Card2 |  To account for clarity and consistency on one side and coverage on the other side, the card has to include between different types of questions.
Card3 |  Conventional inter-rater agreement measures do not apply for validating the answers provided by multiple taggers on the same paper (e.g., multiple students are tagging the card in an educational scenario).
Card4 |  Missing details that are relevant to reconstruction must be properly accounted for and incorporated in the ID card.
Card5 |  For a meaningful tagging of the ID card, the tagger should first map the proposed NLP solution in a given paper to the underlying RE task.
Card6 |  To identify the reconstruction-relevant information (and consequently fill in the ID card) it might not be clear for the tagger to what extent such information should be sought beyond the original paper.

##  7\. Limitations

This section identifies the limitations of our study and discusses how we
mitigate them. We group the limitations distinguishing those related to the
identification of the challenges, and to the construction of the ID card.

The problems of requirements ambiguity detection and requirements
classification are classical and widely studied topics in NLP4RE, as shown by
the survey of Zhao et al. (Zhao et al., 2021).  However, the specific cases
discussed during the focus groups are selected based on the experiences of the
authors, so personal _bias_ might threaten the validity of the results. To
mitigate personal bias, experts in different NLP4RE tasks participated in the
focus groups and shared their views. The participants also identified
commonalities among the challenges identified in different focus group
studies, which indicates that the challenges were not identified based on the
personal bias of the participants. In order to prevent any threats due to
_social relations and persuasion_ , each focus group was moderated by a
different person, and had distinct discussion leaders. During the discussion,
each participant was given the opportunity, and encouraged, to freely speak
up. Nevertheless, some participants were more vocal than others. During the
focus groups, identified challenges may not be mapped to the same in the
cognitive state of the participants, so _understandability_ of the challenges
may issue a threat. To mitigate this, each moderator took notes during the
focus group and shared those notes in real-time so the other participants
could read and clarify them when necessary. Each moderator summarized the
findings at the end of each question during the focused discussion phase and
also at the end of the focus group. _Completeness_ of the challenges
identified in the focus groups is another threat. To help achieving
completeness, we did not enforce time limitations to the focus groups,
allowing the participants to deeply discuss the cases and to add new points as
they see fit. To ensure _conciseness_ , we iterated over the identified
challenges to check for redundancy and reduced the redundancy. The challenges
are identified based on two distinct, yet representative, NLP4RE cases. It is
worth remarking that the two cases were used as triggers for the discussion,
and the actual challenges elicited are based on the analysis of 46 papers, and
on the substantial expertise of the participants of the focus groups, who have
participated in several artifacts tracks, in NLP4RE tool design, and in study
replications. Further experimentation is however needed to ensure _generality_
of our findings. Some challenges may apply to the field of natural language
processing for software engineering (NLP4SE), while others may be more general
to SE. Nevertheless, as the scope of this study is strictly NLP4RE, more
studies are needed to support these claims.Comment1-2

Two main concerns for the ID card are _completeness_ and _pragmatic usability_
, which may be in conflict with each other. We tried to find a healthy balance
of the two by means of multiple iterations of application and refinement of
the ID card, although we expect that lightweight versions of the card may be
created for specific tasks and uses, e.g., for those listed in Table 3. These
also increased _understandability_ of the card, as we applied the ID card to
multiple articles, and after each iteration we discussed the results to
confirm that a shared understanding was reached. Understandability was further
assessed by asking the authors of the paper to fill the ID card. To prevent
the personal _bias_ of single authors affecting the construction of the ID
card, multiple experts of the different NLP4RE tasks participated in its
construction. Issues related to _coverage_ of the ID card in terms of types of
alternative answers for each question was mitigated by assessing the card on
46 papers specifically selected to cover the spectrum of typical RE tasks,
based on the taxonomy by Zhao et al. (Zhao et al., 2021), and considering both
widely cited seminal papers, and relevant recent ones.

##  8\. Conclusions and Outlook

Replication, covering both data (re-)annotation and tool reconstruction, is an
important strategy in experimentation and empirical evaluation. In this paper,
addressing the field of NLP4RE Comment1-2, we identified, by means of focus
groups, a set of 11 challenges concerning data annotation and six challenges
for reconstructing tools. To do so, we reflected on our first-hand experience
in replicating two NLP4RE state-of-the-art solutions on anaphoric ambiguity
detection and requirements classification, and further analyzed 46 papers
covering a wide spectrum of NLP4RE landscape.

Annotation can be challenging at different levels. Theories specific to RE
tasks are not always available to build the annotation task on, e.g., concrete
definitions of non-functional requirements or nocuous ambiguity. Additional
time and effort should be anticipated due to potentially evolving annotation
protocols. Further issues such as dataset imbalance and the lack of domain
knowledge have to be considered. Most of the challenges for reconstructing
tools arise from missing details in the original papers, e.g., the exact
library version, or the application of outdated NLP tools that are no longer
available. Some of these challenges are specific to NLP4RE, while part of them
can be considered applicable to other software engineering fields. Though we
refrain from generalising our findings as they stem from an NLP4RE context, we
encourage other authors to further investigate our list of challenges, and
possible adapt it to their contexts. Artifact assessment, reuse,
replicability, and verifiability are themes that are dear to the whole
software engineering community, and a comprehensive manifesto of these
challenges and of possible mitigation strategies would be beneficial to
improve the maturity of the research community. Comment1-2

To reduce the effect of these challenges on replicating studies in the context
of NLP4RE, we proposed introducing an ID card as a complementary source to
original papers for summarizing replication-relevant information. The ID card
covers seven essential topics captured via a total of 47 questions covering
the description of both RE and NLP tasks, inputs, outputs, annotation process,
tool, and evaluation details. We assessed the ID card both internally and
externally (with the authors of original papers) by cross-tagging the same
paper. Though filling the ID card requires time and effort (which should be
marginal for the authors of a paper), the ID card provides a useful starting
point to facilitate replication.

In the future, we would like to foster the ID card as part of paper submission
at different RE venues. The goal is to help reviewers assess the replication
consideration of the paper and increase the overall awareness of replication-
relevant information. Furthermore, we plan to investigate extending the ID
card to cover other software engineering-related areas. This is highly needed
as artifact evaluation is not sufficiently mature, and needs further
improvement (Hermann et al., 2020; Winter et al., 2022). Though the ID card
mainly aims to improve artifact evaluation, this is not its only purpose, as
it can be used for other objectives, such as literature reviews, as also
remarked by the participants in our external assessment. These are
hypothetical scenarios of usage, which can evolve after the ID card is used
and possibly adapted by the community.

## References

  * (1)
  * Abualhaija et al. (2021) Sallam Abualhaija, Fabiano Dalpiaz, Alessio Ferrari, Xavier Franch, and Davide Fuccie. 2021.  ReqEval: The shared task on anaphora ambiguity detection and disambiguation.  https://github.com/frieden84/nlp4re-reqeval
  * Anaphora (2022) Anaphora. 2022.  Definition of anaphora.  https://www.lexico.com/definition/anaphora Oxford University Press. Last accessed: 16 May 2022.
  * Authors (2022) Authors. 2022.  _“Online Annex”_.  Available at https://figshare.com/s/a7ca0228c7a98ab882ca,.
  * Berry et al. (2003) D. Berry, E. Kamsties, and M. Krieger. 2003.  From Contract Drafting to Software Specification: Linguistic Sources of Ambiguity, A Handbook.  http://se.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdf
  * Breen (2006) Rosanna L Breen. 2006\.  A practical guide to focus-group research.  _Journal of geography in higher education_ 30, 3 (2006), 463–475.
  * Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014.  A Fast and Accurate Dependency Parser using Neural Networks. In _Proceedings of the 18th Conference on Empirical Methods in Natural Language Processing (EMNLP’14)_.
  * Cleland-Huang et al. (2006) Jane Cleland-Huang, Raffaella Settimi, Xuchang Zou, and Peter Solc. 2006.  The detection and classification of non-functional requirements with application to early aspects. In _14th IEEE International Requirements Engineering Conference_. 39–48.
  * Cleland-Huang et al. (2007) Jane Cleland-Huang, Raffaella Settimi, Xuchang Zou, and Peter Solc. 2007.  Automated classification of non-functional requirements.  _Requirements engineering_ 12, 2 (2007), 103–120.
  * Cruz et al. (2019) Margarita Cruz, Beatriz Bernárdez, Amador Durán, Jose A Galindo, and Antonio Ruiz-Cortés. 2019\.  Replication of studies in empirical software engineering: A systematic mapping study, from 2013 to 2018.  _IEEE Access_ 8 (2019), 26773–26791.
  * Da Silva et al. (2014) Fabio QB Da Silva, Marcos Suassuna, A César C França, Alicia M Grubb, Tatiana B Gouveia, Cleviton VF Monteiro, and Igor Ebrahim dos Santos. 2014.  Replication of empirical studies in software engineering research: a systematic mapping study.  _Empirical Software Engineering_ 19, 3 (2014), 501–557.
  * Dalpiaz et al. (2019a) Fabiano Dalpiaz, Davide Dell’Anna, Fatma Basak Aydemir, and Sercan Çevikol. 2019a.  Requirements Classification with Interpretable Machine Learning and Dependency Parsing. In _Proceedings of the 27th IEEE International Requirements Engineering Conference, RE 2019_. 142–152.  https://doi.org/10.1109/RE.2019.00025
  * Dalpiaz et al. (2019b) Fabiano Dalpiaz, Davide Dell’Anna, Fatma Başak Aydemir, and Sercan Çevikol. 2019b.  explainable-re/RE-2019-Materials v0.  https://doi.org/10.5281/zenodo.3309669
  * Dalpiaz et al. (2018a) Fabiano Dalpiaz, Alessio Ferrari, Xavier Franch, and Cristina Palomares. 2018a.  Natural language processing for requirements engineering: The best is yet to come.  _IEEE software_ 35, 5 (2018), 115–119.
  * Dalpiaz et al. (2018b) Fabiano Dalpiaz, Ivor Schalk, and Garm Lucassen. 2018b.  Pinpointing Ambiguity and Incompleteness in Requirements Engineering via Information Visualization and NLP. In _Proceedings of the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ’18)_.
  * Davis (1989) Fred D. Davis. 1989\.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology.  _MIS Quarterly_ 13, 3 (1989), 319–340.  http://www.jstor.org/stable/249008
  * Ezzini et al. (2021a) Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2021a.  taphsir v6.  https://doi.org/10.5281/zenodo.6009790
  * Ezzini et al. (2022) Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2022.  Automated Handling of Anaphoric Ambiguity in Requirements: A Multi-solution Study. In _2022 IEEE/ACM 44th International Conference on Software Engineering_.
  * Ezzini et al. (2021b) Saad Ezzini, Sallam Abualhaija, Chetan Arora, Mehrdad Sabetzadeh, and Lionel C Briand. 2021b.  Using domain-specific corpora for improved handling of ambiguity in requirements. In _2021 IEEE/ACM 43rd International Conference on Software Engineering_.
  * Fernández et al. (2017) D Méndez Fernández, Stefan Wagner, Marcos Kalinowski, Michael Felderer, Priscilla Mafra, Antonio Vetrò, Tayana Conte, M-T Christiansson, Des Greer, Casper Lassenius, et al. 2017\.  Naming the pain in requirements engineering.  _Empirical software engineering_ 22, 5 (2017), 2298–2338.
  * Ferrari et al. (2017a) Alessio Ferrari, Felice Dell’Orletta, Andrea Esuli, Vincenzo Gervasi, and Stefania Gnesi. 2017a.  Natural Language Requirements Processing: A 4D Vision.  _IEEE Softw._ 34, 6 (2017), 28–35.
  * Ferrari et al. (2017b) Alessio Ferrari, Giorgio Oronzo Spagnolo, and Stefania Gnesi. 2017b.  PURE: A Dataset of Public Requirements Documents. In _2017 IEEE 25th International Requirements Engineering Conference_. 502–505.  https://doi.org/10.1109/RE.2017.29
  * Gervasi et al. (2019) Vincenzo Gervasi, Alessio Ferrari, Didar Zowghi, and Paola Spoletini. 2019.  Ambiguity in Requirements Engineering: Towards a Unifying Framework.  In _From Software Engineering to Formal Methods and Tools, and Back_. Springer.
  * Glinz (2007) Martin Glinz. 2007\.  On non-functional requirements. In _15th IEEE international requirements engineering conference (RE 2007)_. IEEE, 21–26.
  * Hermann et al. (2020) Ben Hermann, Stefan Winter, and Janet Siegmund. 2020\.  Community expectations for research artifacts and evaluation processes. In _Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering_. 469–480.
  * Juristo and Vegas (2011) Natalia Juristo and Sira Vegas. 2011.  The role of non-exact replications in software engineering experiments.  _Empirical Software Engineering_ 16, 3 (2011), 295–324.
  * Kamsties and Peach (2000) Erik Kamsties and Barbara Peach. 2000.  Taming ambiguity in natural language requirements. In _Proceedings of the 13th International Conference on Software and Systems Engineering and Applications (ICSSEA’00)_.
  * Kassab et al. (2014) Mohamad Kassab, Colin Neill, and Phillip Laplante. 2014\.  State of practice in requirements engineering: contemporary data.  _Innovations in Systems and Software Engineering_ 10, 4 (2014), 235–241.
  * Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018.  Constituency Parsing with a Self-Attentive Encoder. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, 2676–2686.  https://doi.org/10.18653/v1/P18-1249
  * Kiyavitskaya et al. (2008) Nadzeya Kiyavitskaya, Nicola Zeni, Luisa Mich, and Daniel Berry. 2008.  Requirements for tools for ambiguity identification and measurement in natural language requirements specifications.  _Requirements Engineering_ 13, 3 (2008).
  * Kurtanović and Maalej (2017) Zijad Kurtanović and Walid Maalej. 2017.  Automatically classifying functional and non-functional requirements using supervised machine learning. In _2017 IEEE 25th International Requirements Engineering Conference (RE)_. Ieee, 490–495.
  * Kurtanović and Maalej (2018) Zijad Kurtanović and Walid Maalej. 2018.  On user rationale in software engineering.  _Requirements Engineering_ 23, 3 (2018), 357–379.
  * Landis and Koch (1977) J. Richard Landis and Gary G. Koch. 1977.  An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers.  _Biometrics_ 33, 2 (1977).
  * Li et al. (2014) Feng-Lin Li, Jennifer Horkoff, John Mylopoulos, Renata SS Guizzardi, Giancarlo Guizzardi, Alexander Borgida, and Lin Liu. 2014.  Non-functional requirements as qualities, with a spice of ontology. In _2014 IEEE 22nd International Requirements Engineering Conference (RE)_. IEEE, 293–302.
  * Montgomery et al. (2022) Lloyd Montgomery, Davide Fucci, Abir Bouraffa, Lisa Scholz, and Walid Maalej. 2022.  Empirical research on requirements quality: a systematic mapping study.  _Requirements Engineering_ (2022), 1–27.
  * Pustejovsky and Stubbs (2012) James Pustejovsky and Amber Stubbs. 2012.  _Natural Language Annotation for Machine Learning: A guide to corpus-building for applications_.  O’Reilly Media, Inc.
  * Shepperd et al. (2018) Martin Shepperd, Nemitari Ajienka, and Steve Counsell. 2018\.  The role and value of replication in empirical software engineering results.  _Information and Software Technology_ 99 (2018), 120–132.
  * Shull et al. (2008) Forrest J Shull, Jeffrey C Carver, Sira Vegas, and Natalia Juristo. 2008.  The role of replications in empirical software engineering.  _Empirical software engineering_ 13, 2 (2008), 211–218.
  * Van Solingen et al. (2002) Rini Van Solingen, Vic Basili, Gianluigi Caldiera, and H Dieter Rombach. 2002.  Goal Question Metric (GQM) approach.  _Encyclopedia of Software Engineering_ (2002).
  * Wieringa (2014) Roel J Wieringa. 2014\.  _Design science methodology for information systems and software engineering_.  Springer.
  * Winkler and Vogelsang (2016) Jonas Winkler and Andreas Vogelsang. 2016.  Automatic Classification of Requirements Based on Convolutional Neural Networks. In _24th IEEE International Requirements Engineering Conference, RE 2016, Beijing, China, September 12-16, 2016_. IEEE Computer Society, 39–45.  https://doi.org/10.1109/REW.2016.021
  * Winter et al. (2022) Stefan Winter, Christopher S Timperley, Ben Hermann, Jurgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer. 2022\.  A retrospective study of one decade of artifact evaluations. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 145–156.
  * Wohlin et al. (2012) Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012.  _Experimentation in software engineering_.  Springer Science & Business Media.
  * Yang et al. (2011) Hui Yang, Anne de Roeck, Vincenzo Gervasi, Alistair Willis, and Bashar Nuseibeh. 2011.  Analysing anaphoric ambiguity in natural language requirements.  _Requirements Engineering_ 16, 3 (may 2011), 163.  https://doi.org/10.1007/s00766-011-0119-y
  * Yang et al. (2010) Hui Yang, Anne de Roeck, Alistair Willis, and Bashar Nuseibeh. 2010\.  A Methodology for Automatic Identification of Nocuous Ambiguity. In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_. Coling 2010 Organizing Committee, Beijing, China, 1218–1226.  https://aclanthology.org/C10-1137
  * Zhao et al. (2021) Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J Letsholo, Muideen A Ajagbe, Erol-Valeriu Chioasca, and Riza T Batista-Navarro. 2021\.  Natural Language Processing for Requirements Engineering: A Systematic Mapping Study.  _ACM Computing Surveys (CSUR)_ 54, 3 (2021), 1–41.