11institutetext: Bauhaus-Universität Weimar, Germany
11email<EMAIL_ADDRESS>

# Drawing the Same Bounding Box Twice? Coping Noisy Annotations in Object
Detection with Repeated Labels

David Tschirschwitz  0000-0001-5344-4172
Christian Benz  0000-0001-9915-0057
Morris Florek  0009-0008-8425-5161
Henrik Norderhus  0009-0006-2613-2572
Benno Stein  0000-0001-9033-2217
Volker Rodehorst  0000-0002-4815-0118

###### Abstract

The reliability of supervised machine learning systems depends on the accuracy
and availability of ground truth labels. However, the process of human
annotation, being prone to error, introduces the potential for noisy labels,
which can impede the practicality of these systems. While training with noisy
labels is a significant consideration, the reliability of test data is also
crucial to ascertain the dependability of the results. A common approach to
addressing this issue is repeated labeling, where multiple annotators label
the same example, and their labels are combined to provide a better estimate
of the true label. In this paper, we propose a novel localization algorithm
that adapts well-established ground truth estimation methods for object
detection and instance segmentation tasks. The key innovation of our method
lies in its ability to transform combined localization and classification
tasks into classification-only problems, thus enabling the application of
techniques such as Expectation-Maximization (EM) or Majority Voting (MJV).
Although our main focus is the aggregation of unique ground truth for test
data, our algorithm also shows superior performance during training on the
TexBiG dataset, surpassing both noisy label training and label aggregation
using Weighted Boxes Fusion (WBF). Our experiments indicate that the benefits
of repeated labels emerge under specific dataset and annotation
configurations. The key factors appear to be (1) dataset complexity, the (2)
annotator consistency, and (3) the given annotation budget constraints.

###### Keywords:

Object Detection Instance Segmentation Robust Learning.

##  1 Introduction

Figure 1: Comparison between different ground truth aggregation methods,
exemplary on the VinDr-CXR dataset [23]. Left: the original image with the
repeated labels indicated by the different line types. Right: the four smaller
images from top left to bottom right are, MJV+$\cap$, LAEM+$\mu$, LAEM+$\cup$
and WBF.

Data-driven machine learning systems are expected to operate effectively even
under "difficult" and unforeseen circumstances. Consider safety-relevant
domains such as autonomous driving, medical diagnosis, or structural health
monitoring, where system failure sets lives at risk. Robust systems $-$ those
capable of reliable operation in unseen situations $-$ may encounter several
challenges, including domain shifts [25, 36, 16, 5], adversarial attacks [43,
42], degrading image quality [21, 7, 41] and noisy or uncertain labels [18, 9,
10, 37]. Past studies [33] indicate that noisy labels can cause more harm than
the three aforementioned sources of input noise. Given this context, our study
concentrates on addressing the issue of noisy labels, specifically within
noisy test data. Without a unique ground truth, evaluation is unattainable.
Therefore, to enhance robustness against label noise, it will be pivotal to
first devise methods tailored towards annotation aggregation, which lays the
groundwork for potential future integration with multi-annotator learning
methods.

The creation of annotated data for supervised learning is a costly endeavor,
particularly in cases where experts such as medical professionals or domain
experts are needed to annotate the data. To mitigate this issue, crowd-
sourcing has emerged as a cost-effective means of generating large datasets,
albeit with the disadvantage of potentially lower quality annotations that may
contain label noise [30, 44, 47]. Although the reduced costs of crowd-sourced
annotations often justifies their use, deep neural networks have the capacity
to memorize noisy labels as special cases, leading to a declining performance
and overfitting towards the noisy labeled data [44]. Notably, even expert
annotated data is susceptible to label noise, given the difficulty of the data
to annotate. A survey by Song et. al.[33] revealed that the number of corrupt
labels in real-world datasets ranges between 8.0% to 38.5%. The authors
demonstrate that reducing label noise and creating cleaned data can improve
the accuracy of models. To address the issue of noisy labels, an approach
known as “repeated-labeling” has been proposed. Repeated-labeling means to
obtain annotations from multiple annotators/coders for the same data entry,
such as an image. More specifically: For a set of images
$\\{x_{i}\\}^{N}_{i=1}$ multiple annotators create noisy labels
$\\{\tilde{y}_{i}^{r}\\}^{r=1,\dots,R}_{i=1,\dots,N}$, with
$\tilde{y}_{i}^{r}$ being the label assigned from annotator $r$ to image
$x_{i}$, but without a ground truth label $\\{y_{i}\\}_{i=1,\dots,N}$ [34].

Figure 2: Left: Original input image featuring three separate annotations by
distinct annotators. Center: Application of the LAEM aggregation method to the
three annotations, yielding an approximate ground truth. Right: Aggregated
ground truth utilized during the training process.

Methods for mitigating the negative effect of label noise via repeated
labeling can be divided into two categories [18, 34]: (a) _two-stage_
approaches [39, 8] and (b) _one-stage_ or _simultaneous_ approaches [15, 12,
11]. Two-stage approaches aim to approximate the ground truth prior to
training, a process known as ground truth estimation or ground truth inference
[45], as depicted in Figure 2; a straightforward approach is to compute a
majority vote. Following label aggregation, the model is trained in a regular
fashion. Two-stage approaches offer the benefit of being compatible with
commonly used model architectures. On the other hand, simultaneous approaches
attempt to integrate repeated labels directly into the training process. In
any case, the primary objective of both strategies is to achieve robust and
accurate results by leveraging the repeated labeled data to the fullest extent
possible. Doing so is crucial to justify the additional annotation efforts.
Lastly, to enable the use of established performance metrics, such as those
employed in the COCO object detection dataset (mAP) [20], a ground truth
estimation step is essential for the validation and test sets. While
simultaneous approaches can more effectively utilize repeated labels, they are
not intended to execute the necessary aggregation step required to generate
the unique ground truth estimate [34]. Consequently, reliable approximation
methods are indispensable for evaluation purposes.

Object detection and instance segmentation require both localization and
classification, which means that existing methods for repeated labels that are
used for classification tasks such as image classification or named entity
recognition are not applicable [28]. That is, the available selection of
ground truth inference methods is limited. Furthermore, the creation of
bounding box or polygonal annotations is expensive [10] and reduces the number
of datasets with repeated labels available for evaluating ground truth
inference methods [35, 23]. However, we deliberately avoid using synthetic
data and focus on real datasets. Our contributions are as follows:

  1. 1.

We propose a localization algorithm that enables the use of existing ground
truth estimation methods such as majority voting or expectation maximization
for instance-based recognition tasks and evaluate it extensively with existing
methods [32, 18].

  2. 2.

We introduce a comparative analysis of ground truth inference methods that
highlights their properties and limits.

  3. 3.

We conduct ablation studies to analyze the costs associated with creating
repeated annotations, and what to do when the amount of available annotated
data is limited.

  4. 4.

We introduce an extension for the TexBiG dataset [35] in the form of a test
subset, wherein each of the 200 test images has been annotated by five expert
annotators. Utilizing our aggregation method, we establish a unique
approximation of the ground truth, which will serve as the unknown reference
standard on an evaluation server. This approach allows the TexBiG dataset to
be used for evaluation of robust learning methods addressing the challenge of
noisy labels.

Once released, the link to the evaluation server will be posted on the GitHub
repository where the code is hosted: https://github.com/Madave94/gtiod.

##  2 Related Work

To approximate the ground truth, estimation methods make assumptions about the
data and task properties as well as the annotation process. Majority Voting
(MJV) [14, 29, 26] assumes correct labels for the majority of training samples
and aggregates the labels accordingly:

$\tilde{y_{i}}=\begin{cases}\text{$1$ if $(1/R)\sum^{R}_{r}=y_{i}^{r}>0.5$}\\\
\text{$0$ if $(1/R)\sum^{R}_{r}=y_{i}^{r}<0.5$}\end{cases}$ (1)

In case of a tie, the label is chosen randomly between the tied ones or
selected by a super-annotator. On data with high inter-annotator agreement,
majority voting can be a straightforward approach to obtain ground truth
estimates reasonable quality.

Numerous methods for inferring ground truth rely on the Expectation-
Maximization (EM) approach, first introduced by Dawid and Skene [8]. This
approach estimates annotator confidence and integrates it into a weighted
voting procedure for determining the true label. By considering annotator
performance, these methods address the limitations of majority voting, thereby
avoiding potential outliers. One notable advancement in this area is the GLAD
[39] method, which not only attempts to identify the most probable class but
also assesses image difficulty, additional to the annotator confidence. It
should be noted, however, that this approach is limited to binary
classification tasks [31].

In addition to classification tasks, pixel-wise classification (semantic
segmentation) also has existing ground truth inference methods, such as STAPLE
[38], SIMPLE [17], and COLLATE [1]. Recent developments in this field have led
to approaches that incorporate data difficulty into the estimation process, as
seen in a newly developed simultaneous method [11]. Although there are
numerous variations of ground truth estimation methods for classification and
segmentation tasks, this discussion will focus on methods applicable to object
detection and instance segmentation, rather than diving deeper into this area.

For instance-based recognition tasks like object detection and instance
segmentation, there is an additional issue to consider – the localization
step. During training, methods consisting of a combination of thresholds and
non-maximum suppression are used to solve the localization problem and then
focus on classification accuracy. While this may work during training,
repeated labeling is likely to have more than just a prediction and ground
truth pair to match, since multiple annotators might have created multiple
labels. Hence, existing methods are not applicable. An existing approach to
aggregate annotations for object detection is called Weighted Boxes Fusion
(WBF) [32, 18], which was used for the VinDr-CXR dataset [23]. WBF focuses on
the weighted aggregation within each class, ignoring inter-class disagreements
and also not discarding any annotations even with low agreement. This is
beneficial in cases where missing a possible case is far more severe then
finding too many, such as a task that requires high recall. Apart from this
single existing instance-based recognition approach, we are not aware of any
other aggregation methods for object detection or instance segmentation.

##  3 Method

In the following section we introduce a novel adaptation of the EM algorithm,
localization-aware expectation maximization (LAEM), for instance-based
recognition tasks. The same localization algorithm can also be used with
majority voting, which therefore functions as a baseline. Additionally, we
expand the existing weighted boxes fusion technique to encompass weighted mask
fusion, which enables its use in instance segmentation and facilitates
benchmarking on a broader range of datasets. As extending weighted boxes
fusion is not our core contribution it can be found in Appendix 1.

###  3.1 Localization-Aware Expectation-Maximization

Our novel approach adds a localization stage to existing methods like majority
voting and expectation maximization, enabling the use of these established
methods for instance-based recognition tasks. Thus, the proposed label
aggregation process consists of two stages: (1) localization-stage and (2)
classification-stage. Assuming that $R$ annotators have created noisy
instance-based labels $\tilde{y}_{ij}^{r}$ for image $x_{i}$. Subscript
$j=0,...,M_{r}$ refers to the single instances annotated by annotator $r$ for
image $x_{i}$. $M_{r}$ can be zero if no instances were labeled by $r$. Each
instance contains a class $c\in C$ denoted $\tilde{y}_{ijc}^{r}$. Furthermore,
$\tilde{y}_{ijb}^{r}$ refers to the respective bounding box and
$\tilde{y}_{ijs}^{r}$ to the optional pixel-wise segmentation mask.

Algorithm 1 Outline of the localization algorithm used for LAEM

$X=\\{x_{i}\\}_{i=1,\dots,N}$ $\triangleright$ Set of images

$\tilde{Y}=\\{\tilde{Y}_{i}\\}_{i=1,\dots,N}$ $\triangleright$ Set of noisy
labels per image

$S=\\{S_{i}\\}_{i=1,\dots,N}$ $\triangleright$ Set of annotators per image

$\theta$ $\triangleright$ IoU threshold

for $i\in X$ do $\triangleright$ Loop over images

$\tilde{Y}_{i}=\\{\tilde{y}^{1}_{i1},\tilde{y}^{1}_{i2},\dots,\tilde{y}^{1}_{iM_{1}},\tilde{y}^{2}_{i1},\tilde{y}^{2}_{i2},\dots,\tilde{y}^{2}_{iM_{2}},\dots,\tilde{y}^{R}_{i1},\tilde{y}^{R}_{i2},\dots,\tilde{y}^{R}_{iM_{R}}\\}$

$\tilde{Y}_{i}^{\text{LAEM}}=\emptyset$

$Q$ =
$\\{U_{k}|U_{k}\in\mathcal{P}(S_{i})\wedge|U_{k}|\geq|U_{k+1}|\wedge\lceil|S_{i}|/2\rceil\leq|U_{k}|\\}$
$\triangleright$ Ordered set of annotator combinations

for $U\in Q$ do $\triangleright$ Loop over annotator combinations

L =
$\\{\tilde{Y}_{i:}^{k_{1}}\times\cdots\times\tilde{Y}_{i:}^{k_{n}}|k_{1},\dots,k_{n}\in
U\wedge n=|U|\\}$ $\triangleright$ Possible combinations of labels

$F=\\{u_{k}|u_{k}\in L\wedge\theta\leq IoU(u_{k})\wedge IoU(u_{k})\geq
IoU(u_{k+1})\\}$ $\triangleright$ Filtered and ordered L

for $k\in N$ do $\triangleright$ Loop over label combinations

$K=\\{k_{1},k_{2},\dots,k_{n}\\}$

if $K\cap\tilde{Y}_{i}=\emptyset$ then $\triangleright$ Check for label
availability

$\tilde{Y}_{i}=\tilde{Y}_{i}\setminus K$ $\triangleright$ Remove labels from
available labels

$\tilde{Y}_{i}^{\text{LAEM}}=\tilde{Y}_{i}^{\text{LAEM}}\cup\emph{aggregate}(K)$
$\triangleright$ Add aggregated label to accepted labels

end if

end for

end for

end for

Algorithm 1 outlines the LAEM approach. The algorithm requires image set $X$,
a set of noisy labels $\tilde{Y}$, a set of annotators $S$, and a threshold
$\theta$. Looping over the images of the dataset, the power set
$\mathcal{P(S)}$ over the annotators is computed. Subsets containing less than
half the number of annotators are removed and a descending order is enforced
onto the set. It subsequently iterates through the remaining ordered subsets
of annotators and computes the Cartesian product between the respective
annotators. Each tuple is then ordered and filtered according to threshold
$\theta$ based on the intersection over union in its generalized form:

$IoU=\dfrac{\bigcap_{r=1}^{R}\tilde{y}_{ijb}^{r}}{\bigcup_{r=1}^{R}\tilde{y}_{ijb}^{r}}$
(2)

The remaining set of tuples ordered by descending IoU forms the set of
candidate solutions $F$. In case all labels from a candidate tuple are still
available, they are aggregated according to an aggregation function and added
to the inferred solutions $\tilde{Y}_{i}^{\text{LAEM}}$. The aggregation
function comprises two steps: (1) all classes contained in the original tuple
$\tilde{y}^{r}_{ijc}$ are appended to a list serving as input for expectation
maximization or majority voting and (2) the areas of the different candidates
$\tilde{y}^{r}_{ijb}$ are combined according to the union, intersection, or
average area of all boxes involved. The average operation is based on the WBF
algorithm [32] with uniform weights. If available, the same procedure (for
details cf. Appendix 1) is applied to the segmentation masks
$\tilde{y}^{r}_{ijs}$. This concludes the localization stage. In the
subsequent classification-stage existing ground truth inference methods can be
applied such as majority voting or expectation maximization [8].

###  3.2 Algorithmic Design Choices

Our algorithm is designed in a divide-and-conquer manner. Firstly, we
prioritize localization, effectively reducing the problem to a classification
task for each matched instance after localization. This strategy consequently
facilitates the application of established methods for ground truth inference
developed in other fields. We always prefer a localization match with more
annotators to maximize consensus. If a localization match involving all
available annotators cannot be found given the threshold value $\theta$, we
ensure successive reduction to potentially prefer the next largest number of
annotators. This approach first guarantees localization quality, and only upon
establishing matched areas based on their localizations do we aggregate the
classes. The algorithm is parameterized by the threshold value $\theta$, which
can be adjusted to enforce stricter localization quality and also control the
order in which instances are matched. Though this heuristic solution may not
provide an optimal outcome for larger problem sizes (e.g., numerous instances
on a single image), when an image exhibits high agreement among annotators, a
consensus area can be aggregated, and the class of this area can be
unambiguously determined.

One advantage of the Expectation-Maximization (EM) approach is that assignment
is unambiguous. The confidence calculated during the EM algorithm serves as a
tie-breaker, a benefit not present with Majority Voting (MJV). Furthermore,
fitting the EM algorithm is efficient; following localization matching, no
further areas are calculated, and only the solutions
$\tilde{Y}_{i}^{\text{LAEM}}$ are considered along with their classes.

While localization fusion functions, such as union or intersection, are
available and applicable for training data, the intended use for test data
within the context of LAEM (Localization-Aware Expectation-Maximization)
primarily involves the averaging fusion function. This approach enables a
balanced aggregation of areas across different annotators. Additionally, this
method is also utilized to aggregate test data as required for the TexBiG
dataset [35].

###  3.3 Comparative Analysis

Methods |  LAEM |  MJV |  WBF |  WBF+EM
---|---|---|---|---
Assignment |  1) Localization 2) Classification |  1) Localization 2) Classification |  Localization only |  Localization only
Low agreement |  Discard annotation |  Discard annotation |  Keep annotation |  Keep annotation
Edge cases |  Use confidence |  Randomized | $-$ | $-$
Localization fusion |  Union / average / intersection |  Union / average / intersection |  Averaging |  Weighted averaging
Annotator confidence | ✓ | $\times$ | $\times$ | ✓
Handling missing data | ✓ | ✓ | ✓ | $\checkmark$
Dataset characteristic |  Balanced |  Precision oriented |  Recall oriented |  Recall oriented
Data dependence | $\times$ | $\times$ | $\times$ | $\times$
Table 1: Comparison table for the characteristics and properties of the
different ground truth inference methods. MJV and LAEM both use the novel
localization algorithm.

In Table 1, we present a comparative analysis of the four available ground
truth inference methods for instance-based recognition tasks, distinguished by
their respective characteristics and properties. Each method is described
based on eight distinct features relevant to ground truth estimation methods.
A noteworthy difference between LAEM and MJV, as compared to WBF and its
adaptation (detailed in Appendix 2), is the handling of instances that lack
consensus among annotators. Figure 1 and Appendix 3 illustrates the
aggregation processes of MJV, LAEM, and WBF on a few specific images, serving
as practical examples. This illustrations reveal that MJV and LAEM tend to
find consensus instances, resulting in a final image that appears as if a
single annotator has labelled the image. In contrast, the WBF image is
relatively cluttered with overlapping instances. This discrepancy arises
because WBF merges areas of the same class that significantly overlap but does
not discard any annotation. This is also the case, for instances where two
annotators found the same instance area but disagreed on the class, resulting
in more instances overall. Although this property might be beneficial for a
high-recall scenario $-$ where missing an instance is more detrimental than
detecting multiple false positives $-$ it is not ideal for many applications.
It’s important to note that none of the current methods incorporate data
dependence, a feature described in several state-of-the-art ground truth
estimation methods for semantic segmentation [17, 38, 1].

##  4 Experimental Results

In our preliminary experiment, we scrutinized the influence of annotation
budget size by exploring scenarios in which repeated labels might be preferred
over single labels. This ablation study was designed to determine the optimal
use of a restricted annotation budget, i.e., whether it is more beneficial to
train with a larger volume of noisy labels or a reduced set of refined labels.
This experimental analysis was conducted using two separate datasets.

In our subsequent investigation, we assessed the effect of annotator selection
on model performance by deliberately excluding certain annotators from the
labeling process. This enabled us to emulate the potential impact of annotator
selection on model performance and to probe the influence of proficient and
suboptimal annotators on model output.

Our final experiment, which is detailed in Appendix 2, was actually conducted
first since it influenced the choice of the aggregation method used for the
training data. However, this experiment is not the main focus of this
publication.

###  4.1 Set-Up

To the best of our knowledge, there are only two datasets available that
contain repeated labels for object detection and instance segmentation,
respectively: the VinDr-CXR [23, 18] dataset and the TexBiG [35] dataset. We
focus solely on these two datasets and do not make use of any synthetic data.

VinDr-CXR dataset. This dataset comprises 15,000 training and 3,000 test chest
X-ray images, each of which was annotated by three annotators and five
annotators, respectively. With a total of 36,096 instances in the training
dataset, the dataset can be considered sparsely annotated, with in average
$0.8$ instances per image111Computed by dividing the number of instances by
the product of the number of images and the number of annotators.. The dataset
consists of 14 different classes that were annotated by 17 radiologists [22].
Using the agreement evaluation method presented in [35] describing the data
quality, the K-$\alpha$ (Krippendorff’s alpha) is $0.79$. However, since only
$29.3\%$ of the images in the dataset contains any annotations at all, another
K-$\alpha$ was calculated for this reduced subset, resulting in a K-$\alpha$
value of $0.29$. This indicates that while annotators largely agree in cases
where no anomaly is present, there is significant disagreement in cases that
contain instances.

TexBiG dataset. Recently published [35], the TexBiG provides labels for
document layout analysis, similar to the PubLayNet [46] or DocBank [19]
datasets. It covers 19 classes for complex document layouts in historical
documents during a specific time period, and in the version used here, the
training data contains 44,121 instances, the validation data 8,251 instances
and the test data 6,678 instances. While the total number of instances is
larger as in the VinDr-CXR dataset, there are only 2,457 images in total,
1,922 in the training set, 335 in the validation set and 200 in the test set.
Due to the iterative creation process of the dataset, the number of repeated
labels is different depending on the sample. An agreement value was used per
image to evaluate which samples were to be annotated again. For each image,
two annotators were assigned, and in case the agreement value was low after
the first iteration, an additional annotator was added to that specific
sample. This was done until a maximum of four annotators per sample. In the
combined validation and training set, 34 images were annotated by 4
annotators, 336 by at least 3 annotators (including the 34 from before), and
2,257 by at least 2 annotators. We created an additional test set with 5
annotators for 200 newly selected images from the same domain, in accordance
with the guideline provided by the authors [35]. We plan to publish this test-
set for benchmarking purposes on an evaluation server. The TexBiG dataset is
more densely annotated, with $10.7$ instances per image, which is $13$ time
more than the VinDr-CXR dataset. Furthermore, the K-$\alpha$ for the TexBiG
training dataset is higher with $0.93$.

Comparing the two datasets we find that they represent two opposing marginal
cases with one dataset having high-agreement and dense annotations, while the
other one has a low-agreement and sparse annotations. However, a more balanced
dataset is missing.

Architecture choice. Regarding the architecture choice, we aimed to find a
well-performing and stable choice, rather than aiming for state-of-the-art
results since we wanted to focus on comparing the ground truth inference
methods and ablation studies on different tasks. For the VinDr-CXR dataset, we
tested various architectures including different anchor-based two-stage
detectors like Faster R-CNN [27], Cascade R-CNN [2] and Double Head R-CNN
[40], and additionally, the transformer-based Detection Transformer (DETR)
[3]. After thorough investigation, we found that the Double Head R-CNN
performs stably and with reasonable results. Therefore, we selected this
architecture for our experiments. On the TexBiG dataset, we tried several
instance segmentation models like Mask R-CNN [13], Cascade Mask R-CNN [2] and
DetectorRS [24], as well as the Mask2Former [6] as a transformer-based
architecture. In this case, DetectoRS yielded the most stable performance, and
we continued our experiments with this model. We extended MMDetection [4] with
our implementation, and the code is available on GitHub under
https://github.com/Madave94/gtiod.

###  4.2 Annotation Budget Ablation

When working on a deep learning project, data is often a limiting factor, and
resources must be carefully allocated to create task-specific data. Even when
there are ample resources, maximizing the value of those resources is
important. In the context of human-annotated data, this concept is referred to
as the “annotation budget”, which represents the available number of images or
instances that can be labeled by a pool of annotators within their available
time. The question then becomes, “How can a limited annotation budget be best
utilized?” One approach is to prioritize annotating as many different images
as possible to cover a broad range of cases within the application domain.
However, this approach comes with the risk of introducing more noisy labels
due to the inherent variability in annotator performance. Alternatively,
creating repeated labels may be more beneficial to improve the quality of the
annotations. Ultimately, the decision between prioritizing quantity versus
quality of labels must be carefully weighed and considered in the context of
the project goals and available resources.

Split | Budget | Averaged | Maximum
---|---|---|---
| rel. | abs. | AP | APbb | AP | APbb
1922$\times$2 | 100% | 3844 | 41.9 | 47.5 | 43.3 | 48.7
966$\times$2 966$\times$1 | 75% | 2883 | 42.4 | 47.9 | 42.8 | 48.4
1922$\times$1 | 50% | 1922 | 42.7 | 48.3 | 43.4 | 48.8
641$\times$2 640$\times$1 | 50% | 1922 | 42.4 | 47.7 | 43.9 | 48.8
966$\times$2 | 50% | 1922 | 41.1 | 46.2 | 42.8 | 47.8
30$\times$4 243$\times$3 1073$\times$1 | 50% | 1922 | 41.9 | 47.3 | 43.0 | 48.2
30$\times$4 243$\times$3 536$\times$2 1$\times$1 | 50% | 1922 | 39.7 | 45.0 | 41.0 | 46.5
Table 2: Ablation study on the TexBiG dataset using a limited annotation
budget. The results are in $mAP@[.5:.95]$, show that multi annotator learning
methods are required to justify repeated labels. However, even without multi
annotator methods the performance loss using repeated annotations is marginal.

Split | Budget |  Avg. AP |  Max. AP
---|---|---|---
rel. | abs.
15,000$\times$2 | 66.6% | 30,000 | 14.8 | 15.0
10,000$\times$3 | 66.6% | 30,000 | 14.7 | 14.9
15,000$\times$1 | 33.3% | 15,000 | 13.4 | 13.9
| 10,000$\times$1
---
2,500$\times$2
33.3% | 15,000 | 13.6 | 14.1
7,500$\times$2 | 33.3% | 15,000 | 13.5 | 13.8
| 3,000$\times$3
---
3,000$\times$2
33.3% | 15,000 | 13.6 | 14.3
5,000$\times$3 | 33.3% | 15,000 | 13.4 | 14.0
Table 3: Ablation study on the VinDr-CXR dataset using a limited annotation
budget. The results are in $mAP_{40}$ as provided by the leaderboard.

In the two ablation studies presented in Table 3 and 3, we compare the
performance of different annotation budgets, which refer to the available
number of images or instances that can be labeled with the pool of annotators
and their available time. The splits used in the studies represent three
different cases: (1) Only single annotated labels are available, which are
more prone to label noise. (2) A mix of repeated labels and single annotated
labels is available. Multiple splits may have this property. (3) Maximum label
repetition, where no or very few single annotated labels are available,
resulting in significantly less training data. To reduce randomization
effects, we create five different random versions for each split and compute
their mean and maximum results.

Our results show that the TexBiG dataset quickly reached a data saturation
point, suggesting potential benefits from employing multi-annotator learning
methods to better utilize repeated labels. Conversely, the VinDr-CXR dataset
showed improved performance with higher budgets, indicating that more data
helps performance in scenarios with noisy, low-agreement labels.

Both datasets demonstrate that moderate inclusion of repeated labels does not
adversely impact performance, with mixed splits achieving peak results at
their lowest budgets. These findings highlight the value of repeated
annotations, which not only increase label reliability, but also allow for
efficient use of multi-annotator learning methods. Remarkably, the opportunity
costs for creating such repeated labels seem negligible.

Our findings suggest that higher fragmentation in annotator splits could lead
to reduced performance, possibly due to enhanced intracoder consistency.
Moreover, the influence of split distribution appears prominent only when the
annotation budget is limited. Identifying a systematic relationship between
split distribution and performance, thereby suggesting optimal splits before
the annotation process, could be a promising future research direction.

The overall takeaway is that multiple annotations may not always yield
significant advantages, yet in scenarios with a constrained annotation budget,
they could prove beneficial. Determining which cases fall into each category
remains an open challenge.

###  4.3 Leave-One-Out Annotator Selection

Table 4 displays the results of a final experiment conducted on the TexBiG
dataset. To create four groups of annotators, each group consisting of one to
three individuals, annotations were distributed unevenly among them, resulting
in groups of different sizes. Subsequently, each group was left out of the
training process, while the remaining three groups were used to train the
model. This approach led to a smaller training set. Surprisingly, the
experiment showed that when the largest group, denoted as B, was excluded,
leaving only $61.6\%$ of the annotations available, the model’s performance
reached its peak. This outcome underscores the importance of selecting precise
annotators in the training process, since less precise ones may introduce
noisy labels that can hinder performance. However, it is challenging to
identify precise annotators before the annotation process, as there is no data
available to determine their level of precision.

Left out group |  | Left out
---
images
| Left out
---
annotations
| Perfor-
---
mance
rel. | abs. | rel. | abs. | AP | APbb
Group A | 25.1% | 1,040 | 26.8% | 11,810 | 42.4 | 47.1
Group B | 29.5% | 1,225 | 38.4% | 16,932 | 44.1 | 49.8
Group C | 25.7% | 1,067 | 18.2% | 8,017 | 42.6 | 48.1
Group D | 19.7% | 815 | 16.7% | 7,362 | 43.1 | 48.3
Table 4: Choosing the right annotator? If annotators are not in the group what
would happen to the results? Splits are unequal, due to the annotation
distribution.

##  5 Conclusion

Our results indicate the potential benefits of repeated labels which seem to
be contingent on several factors. The identified key factors are the balance
between (1) the complexity or variation in the dataset and its corresponding
task difficulty, (2) the variability in annotation depending on inter-
annotator consistency and annotator proficiency, and (3) the constraints of
the annotation budget. This interaction suggests the existence of an ‘optimal
range’ for image annotation strategy. For instance, datasets with high
variance and low annotator consistency may benefit from multiple annotations
per image, while in cases with low image variation and high annotator
consistency, many images annotated once might suffice. This balancing act
between data and annotation variation could guide decisions when choosing
between single or multiple annotators per image, given a fixed annotation
budget.

However, the utility of repeated labels is substantially hampered due to the
lack of multi-annotator-learning approaches for object detection and instance
segmentation. Thus, future work should concentrate on developing methods that
bridge this gap between these areas and other computer vision domains like
image classification or semantic segmentation.

Lastly, a significant challenge remains regarding the availability of suitable
datasets. With limited datasets in the domain and disparities among them, our
findings’ generalizability remains constrained to the two domains covered in
this study. A larger dataset with repeated labels and balanced agreement would
be valuable for future research. Synthetic data could be beneficial but pose
the risk that models trained on these data may only learn the distribution
used to randomly create repeated labels from the original annotations. Thus,
creating a suitable dataset remains a formidable task.

####  5.0.1 Acknowledgments.

This work was supported by the Thuringian Ministry for Economy, Science and
Digital Society / Thüringer Aufbaubank (TMWWDG / TAB).

## References

  * [1] Asman, A.J., Landman, B.A.: Robust statistical label fusion through consensus level, labeler accuracy, and truth estimation (collate). IEEE transactions on medical imaging 30(10), 1779–1794 (2011)
  * [2] Cai, Z., Vasconcelos, N.: Cascade r-cnn: High quality object detection and instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence p. 1–1 (2019). https://doi.org/10.1109/tpami.2019.2956516, http://dx.doi.org/10.1109/tpami.2019.2956516
  * [3] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. pp. 213–229. Springer (2020)
  * [4] Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C.C., Lin, D.: MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)
  * [5] Chen, Y., Li, W., Sakaridis, C., Dai, D., Van Gool, L.: Domain adaptive faster r-cnn for object detection in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3339–3348 (2018)
  * [6] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1290–1299 (2022)
  * [7] Cheng, Y., Zhu, J., Jiang, M., Fu, J., Pang, C., Wang, P., Sankaran, K., Onabola, O., Liu, Y., Liu, D., et al.: Flow: A dataset and benchmark for floating waste detection in inland waters. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10953–10962 (2021)
  * [8] Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics) 28(1), 20–28 (1979)
  * [9] Feng, D., Wang, Z., Zhou, Y., Rosenbaum, L., Timm, F., Dietmayer, K., Tomizuka, M., Zhan, W.: Labels are not perfect: Inferring spatial uncertainty in object detection. IEEE Transactions on Intelligent Transportation Systems (2021)
  * [10] Gao, J., Wang, J., Dai, S., Li, L.J., Nevatia, R.: Note-rcnn: Noise tolerant ensemble rcnn for semi-supervised object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9508–9517 (2019)
  * [11] Gao, Z., Sun, F.K., Yang, M., Ren, S., Xiong, Z., Engeler, M., Burazer, A., Wildling, L., Daniel, L., Boning, D.S.: Learning from multiple annotator noisy labels via sample-wise label fusion. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV. pp. 407–422. Springer (2022)
  * [12] Guan, M., Gulshan, V., Dai, A., Hinton, G.: Who said what: Modeling individual labelers improves classification. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)
  * [13] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
  * [14] Karimi, D., Dou, H., Warfield, S.K., Gholipour, A.: Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical image analysis 65, 101759 (2020)
  * [15] Khetan, A., Lipton, Z.C., Anandkumar, A.: Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577 (2017)
  * [16] Khodabandeh, M., Vahdat, A., Ranjbar, M., Macready, W.G.: A robust learning approach to domain adaptive object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 480–490 (2019)
  * [17] Langerak, T.R., van der Heide, U.A., Kotte, A.N., Viergever, M.A., Van Vulpen, M., Pluim, J.P.: Label fusion in atlas-based segmentation using a selective and iterative method for performance level estimation (simple). IEEE transactions on medical imaging 29(12), 2000–2008 (2010)
  * [18] Le, K.H., Tran, T.V., Pham, H.H., Nguyen, H.T., Le, T.T., Nguyen, H.Q.: Learning from multiple expert annotators for enhancing anomaly detection in medical image analysis. arXiv preprint arXiv:2203.10611 (2022)
  * [19] Li, M., Xu, Y., Cui, L., Huang, S., Wei, F., Li, Z., Zhou, M.: Docbank: A benchmark dataset for document layout analysis. arXiv preprint arXiv:2006.01038 (2020)
  * [20] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [21] Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A.S., Bethge, M., Brendel, W.: Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484 (2019)
  * [22] Nguyen, D.B., Nguyen, H.Q., Elliott, J., KeepLearning, Nguyen, N.T., Culliton, P.: Vinbigdata chest x-ray abnormalities detection (2020), https://kaggle.com/competitions/vinbigdata-chest-xray-abnormalities-detection
  * [23] Nguyen, H.Q., Lam, K., Le, L.T., Pham, H.H., Tran, D.Q., Nguyen, D.B., Le, D.D., Pham, C.M., Tong, H.T., Dinh, D.H., et al.: Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations. Scientific Data 9(1), 429 (2022)
  * [24] Qiao, S., Chen, L.C., Yuille, A.: Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10213–10224 (2021)
  * [25] Ramamonjison, R., Banitalebi-Dehkordi, A., Kang, X., Bai, X., Zhang, Y.: SimROD: A Simple Adaptation Method for Robust Object Detection. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3550–3559. IEEE, Montreal, QC, Canada (Oct 2021). https://doi.org/10.1109/ICCV48922.2021.00355, https://ieeexplore.ieee.org/document/9711168/
  * [26] Raykar, V.C., Yu, S., Zhao, L.H., Jerebko, A., Florin, C., Valadez, G.H., Bogoni, L., Moy, L.: Supervised learning from multiple experts: whom to trust when everyone lies a bit. In: Proceedings of the 26th Annual international conference on machine learning. pp. 889–896 (2009)
  * [27] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015)
  * [28] Rodrigues, F., Pereira, F.: Deep learning from crowds. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)
  * [29] Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 614–622 (2008)
  * [30] Sheng, V.S., Zhang, J.: Machine learning with crowdsourcing: A brief summary of the past research and future directions. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 9837–9843 (2019)
  * [31] Sinha, V.B., Rao, S., Balasubramanian, V.N.: Fast dawid-skene: A fast vote aggregation scheme for sentiment classification. arXiv preprint arXiv:1803.02781 (2018)
  * [32] Solovyev, R., Wang, W., Gabruseva, T.: Weighted boxes fusion: Ensembling boxes from different object detection models. Image and Vision Computing 107, 104117 (2021)
  * [33] Song, H., Kim, M., Park, D., Shin, Y., Lee, J.G.: Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems (2022)
  * [34] Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D.C., Silberman, N.: Learning from noisy labels by regularized estimation of annotator confusion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11244–11253 (2019)
  * [35] Tschirschwitz, D., Klemstein, F., Stein, B., Rodehorst, V.: A dataset for analysing complex document layouts in the digital humanities and its evaluation with krippendorff’s alpha. In: DAGM German Conference on Pattern Recognition. pp. 354–374. Springer (2022)
  * [36] Wang, X., Huang, T.E., Liu, B., Yu, F., Wang, X., Gonzalez, J.E., Darrell, T.: Robust object detection via instance-level temporal cycle confusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9143–9152 (2021)
  * [37] Wang, Z., Li, Y., Guo, Y., Fang, L., Wang, S.: Data-uncertainty guided multi-phase learning for semi-supervised object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4568–4577 (2021)
  * [38] Warfield, S.K., Zou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (staple): an algorithm for the validation of image segmentation. IEEE transactions on medical imaging 23(7), 903–921 (2004)
  * [39] Whitehill, J., Wu, T.f., Bergsma, J., Movellan, J., Ruvolo, P.: Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22 (2009)
  * [40] Wu, Y., Chen, Y., Yuan, L., Liu, Z., Wang, L., Li, H., Fu, Y.: Rethinking classification and localization for object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10186–10195 (2020)
  * [41] Wu, Z., Suresh, K., Narayanan, P., Xu, H., Kwon, H., Wang, Z.: Delving into robust object detection from unmanned aerial vehicles: A deep nuisance disentanglement approach. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1201–1210 (2019)
  * [42] Xie, C., Wang, J., Zhang, Z., Zhou, Y., Xie, L., Yuille, A.: Adversarial examples for semantic segmentation and object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 1369–1378 (2017)
  * [43] Zhang, H., Wang, J.: Towards Adversarially Robust Object Detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 421–430. IEEE, Seoul, Korea (South) (Oct 2019). https://doi.org/10.1109/ICCV.2019.00051, https://ieeexplore.ieee.org/document/9009990/
  * [44] Zhang, Z., Zhang, H., Arik, S.O., Lee, H., Pfister, T.: Distilling effective supervision from severe label noise. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9294–9303 (2020)
  * [45] Zheng, Y., Li, G., Li, Y., Shan, C., Cheng, R.: Truth inference in crowdsourcing: Is the problem solved? Proceedings of the VLDB Endowment 10(5), 541–552 (2017)
  * [46] Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1015–1022. IEEE (2019)
  * [47] Zhu, X., Wu, X.: Class noise vs. attribute noise: A quantitative study. The Artificial Intelligence Review 22(3), 177 (2004)

## Appendix 1

This section presents our adaptation of the weighted box fusion (WBF)
technique, tailored specifically for instance segmentation as a weighted mask
fusion (WMF).

In their study, [18] propose a method for combining annotations from multiple
annotators using a weighted box fusion [32] approach. In this method, bounding
boxes are matched greedily only with boxes of the same class, and no
annotations are discarded. The WBF algorithm fuses boxes that exceed a
specified overlap threshold, resulting in new boxes that represent the
weighted average of the original boxes. The approach also allows for inclusion
of box confidence scores and prior weights for each annotator.

To extend the WBF method for instance segmentation, we introduce an option to
fuse segmentation masks, which involves four steps: (1) calculating the
weighted area and weighted center points from the different masks, (2) compute
the average center point and average area from the selected masks, (3)
determining the closest center point of the original masks to the weighted
center point and selecting this mask, and (4) dilating or eroding the chosen
mask until the area is close to the averaged area. The resulting mask is used
as the aggregated segmentation mask and is also used as the averaging
operation during the aggregation for LAEM and MJV with uniform weight.

Moreover, we integrate the WBF approach with LAEM, yielding WBF+EM. This
integration involves assessing annotator confidence using LAEM, and
subsequently incorporating it into the WBF method to produce weighted average
areas instead of simply averaged areas. While the differences between LAEM and
WBF might seem subtle, WBF+EM offers a more thorough approach to annotator
fusion. This modification is relatively minor, and its impact is modest, as
corroborated by our experiments delineated in Appendix 2.

## Appendix 2

DetectoRS | Training
---|---
TexBiG | RL | MJV | LAEM | WBF
$\cup$ | $\mu$ | $\cap$ | $\cup$ | $\mu$ | $\cap$ | base | EM
Test | MJV | $\cup$ | AP |  32.5 |  34.5 |  30.4 |  25.5 |  35.1 |  29.1 |  28.4 |  30.2 |  30.7
| APbb |  34.7 |  36.6 |  34.0 |  29.9 |  37.5 |  32.7 |  33.4 |  34.4 |  33.5
$\mu$ | AP |  41.9 |  43.9 |  39.9 |  35.8 |  43.6 |  40.8 |  35.0 |  41.5 |  41.9
APbb |  45.6 |  48.2 |  44.6 |  41.4 |  47.9 |  44.5 |  40.2 |  45.5 |  46.2
$\cap$ | AP |  44.2 |  41.7 |  46.6 |  45.0 |  43.2 |  45.3 |  45.2 |  46.0 |  44.9
APbb |  49.6 |  47.9 |  51.4 |  49.9 |  49.3 |  50.6 |  49.5 |  51.0 |  49.2
LAEM | $\cup$ | AP |  31.5 |  34.9 |  44.4 |  25.9 |  33.6 |  30.5 |  26.8 |  29.8 |  31.1
APbb |  33.6 |  36.5 |  48.9 |  30.5 |  35.8 |  33.8 |  31.4 |  33.0 |  34.6
$\mu$ | AP |  41.1 |  42.5 |  40.5 |  34.9 |  43.6 |  40.2 |  35.9 |  40.7 |  40.4
APbb |  44.8 |  46.8 |  44.6 |  41.9 |  47.7 |  44.3 |  41.7 |  45.5 |  45.1
$\cap$ | AP |  43.5 |  40.8 |  43.0 |  45.0 |  41.6 |  44.1 |  44.0 |  45.0 |  45.1
APbb |  49.8 |  46.0 |  48.5 |  49.5 |  46.9 |  48.9 |  48.0 |  49.5 |  50.3
WBF | base | AP |  36.1 |  38.0 |  34.8 |  33.4 |  37.3 |  30.3 |  32.7 |  34.9 |  36.9
APbb |  38.8 |  41.6 |  38.0 |  37.4 |  40.2 |  33.6 |  37.8 |  38.5 |  40.3
EM | AP |  38.1 |  39.9 |  34.9 |  32.4 |  39.8 |  36.5 |  32.8 |  36.3 |  35.9
APbb |  40.6 |  42.8 |  38.3 |  36.4 |  42.8 |  40.2 |  37.6 |  40.0 |  39.3
Mean | AP |  38.6 |  39.5 |  39.3 |  34.7 |  39.7 |  37.1 |  35.1 |  38.1 |  38.4
APbb |  42.2 |  43.3 |  43.5 |  39.6 |  43.5 |  41.1 |  40.0 |  42.2 |  42.3
Table 5: Cross-Validation of ground truth inference combinations between
training and test data, for the DetectoRS with a ResNet-50 backbone on the
TexBiG dataset. Showing the $mAP@[.5:.95]$ for instance masks and bounding
boxes. Union is represented by $\cup$, intersection by $\cap$ and averaging by
$\mu$. RL denotes training conducted on un-aggregated noisy labels. The two
rows on the bottom show how the training methods perform on average.

In this experiment, we carried out a comparative analysis of different ground
truth inference methods. To do this, we separated the annotations for training
and testing, and created various combinations of train-test datasets using the
available ground truth estimation methods. Afterward, a model was trained on
these combinations. The results from this experiment reveal how aggregation
methods can impact the performance of the trained models and show how these
outcomes can vary based on the specific combination of training and testing
aggregation used.

Tables 5 and 6 present the application of various ground truth estimation
methods on repeated labels. In the TexBiG dataset, each method is employed to
aggregate the labels of both training and test data, and all possible train-
test combinations are learned and tested to perform a cross comparison of the
different ground truth inference methods, as shown in Table 5. The
hyperparameter for the area combination is denoted as $\cup$ for union, $\mu$
for averaging and $\cap$ for intersection. Additionally, the plain repeated
labels, without any aggregation, are compared with the different aggregated
test data. Our findings reveal that on a high-agreement dataset, weighted
boxes fusion does not perform well. This could be attributed to the inclusion
of most annotations by WBF, whereas in cases with high agreement, it is more
desirable to exclude non-conforming instances. Majority voting and
localization-aware expectation maximization perform similarly; however, LAEM
provides a more elegant solution for addressing edge cases. Calculating the
annotator confidence, as performed in LAEM, is highly advantageous. However,
in rare cases, spammer annotators could potentially circumvent annotation
confidence by annotating large portions of simple examples correctly but
failing at hard cases. Such cases would result in a high confidence level for
the spammer, potentially outvoting the correct annotators on challenging and
crucial cases.

Double Head R-CNN VinDr-CXR | Training
---|---
RL | MJV | LAEM | WBF
$\cup$ |  $\mu$ | $\cap$ |  $\cup$ |  $\mu$ | $\cap$ |  base |  EM
private LB |  16.2 |  15.2 |  14.6 |  14.3 |  14.9 |  15.0 |  14.7 |  13.7 |  14.3
Table 6: Comparing results with the private Kaggle leaderboard [22] for the
VinDr-CXR dataset using the double headed R-CNN at $mAP_{40}$. Union is
represented by $\cup$, intersection by $\cap$ and averaging by $\mu$. RL
denotes training conducted on un-aggregated noisy labels.

The main performance differences between MJV and LAEM arise due to the
application of the three different combination operations – union, averaging,
and intersection. Combining areas by taking their union results in larger
areas, making it easier for a classifier to identify the respective regions.
Analysis of the mean results of the training methods reveals that both
MJV+$\cup$ and LAEM+$\cup$ exhibit the highest performance across various test
configurations. On the contrary, methods parameterized with intersection
$\cap$ yield the lowest mean results. Training with repeated labels without
any aggregation yields results similar to training with aggregated labels.
However, while it may be generally feasible to train with noisy labels, the
performance is slightly dampened. Since the test data aggregation method is
LAEM-$\mu$ as described in Section 3.2, the best performing training method
LAEM-$\cup$ is choosen as the aggregation method for the training data in the
experiments shown in Section 4.2 and 4.3.

For the VinDr-CXR dataset, a smaller, similar experiment is performed as shown
in Table 6. As the Kaggle leaderboard already provides an aggregated ground
truth and labels are unavailable, only the training data are aggregated. Our
findings indicate that training with plain repeated labels leads to higher
results. Given the low agreement of the dataset, training with repeated labels
may be seen as a form of “label augmentation.” Interestingly, the methods used
to aggregate the test data, such as WBF, do not outperform the other methods.
However, ground truth estimation methods are not designed to boost performance
but rather to provide a suitable estimation for the targeted outcome. Based on
these results, the following experiments on VinDr-CXR will be run with the
repeated labels for training.

## Appendix 3

This section shows three more comparisons between different ground truth
aggregation methods, exemplary on the VinDr-CXR dataset [23]. All of them
follow the same structure. Left: the original image with the repeated labels
indicated by the different line types. Right: the four smaller images from top
left to bottom right are, MJV+$\cap$, LAEM+$\mu$, LAEM+$\cup$ and WBF.

Figure 3: Qualitative results on three test images from the VinDr-CXR. Left:
the original image with the repeated labels indicated by the different line
types. Right: the four smaller images from top left to bottom right are,
MJV+$\cap$, LAEM+$\mu$, LAEM+$\cup$ and WBF.


# PromptChainer: Chaining Large Language Model Prompts
through Visual Programming

Tongshuang Wu<EMAIL_ADDRESS>University of WashingtonUSA ,  Ellen
Jiang<EMAIL_ADDRESS>Google ResearchUSA ,  Aaron Donsbach
<EMAIL_ADDRESS>Google ResearchUSA ,  Jeff Gray<EMAIL_ADDRESS>Google ResearchUSA ,  Alejandra Molina<EMAIL_ADDRESS>Google
ResearchUSA ,  Michael Terry<EMAIL_ADDRESS>Google ResearchUSA and
Carrie J. Cai<EMAIL_ADDRESS>Google ResearchUSA

(2022)

###### Abstract.

While LLMs have made it possible to rapidly prototype new ML functionalities,
many real-world applications involve complex tasks that cannot be easily
handled via a single run of an LLM. Recent work has found that chaining
multiple LLM runs together (with the output of one step being the input to the
next) can help users accomplish these more complex tasks, and in a way that is
perceived to be more transparent and controllable. However, it remains unknown
what users need when authoring their own LLM chains – a key step to lowering
the barriers for non-AI-experts to prototype AI-infused applications. In this
work, we explore the LLM chain authoring process. We find from pilot studies
that users need support transforming data between steps of a chain, as well as
debugging the chain at multiple granularities. To address these needs, we
designed _PromptChainer_ , an interactive interface for visually programming
chains. Through case studies with four designers and developers, we show that
PromptChainer supports building prototypes for a range of applications, and
conclude with open questions on scaling chains to even more complex tasks, as
well as supporting low-fi chain prototyping.

††journalyear: 2022††copyright: rightsretained††conference: CHI Conference on
Human Factors in Computing Systems Extended Abstracts; April 29-May 5, 2022;
New Orleans, LA, USA††booktitle: CHI Conference on Human Factors in Computing
Systems Extended Abstracts (CHI ’22 Extended Abstracts), April 29-May 5, 2022,
New Orleans, LA, USA††doi: 10.1145/3491101.3519729††isbn:
978-1-4503-9156-6/22/04††ccs: Human-centered computing Empirical studies in
HCI††ccs: Human-centered computing Interactive systems and ools††ccs:
Computing methodologies Machine learning

##  1\. Introduction

Large language models (LLMs) have introduced new possibilities for prototyping
with AI (Yang et al., 2020). Pre-trained on a large amount of text data,
models like GPT-3 (Brown et al., 2020) and Jurassic-1 (Lieber et al., 2021)
encode enough information to support _in-context learning_ : they can be
easily customized at run time (without any re-training needed) to handle new
tasks, simply by taking in natural language instructions called _prompts_. For
example, a user could customize a pre-trained, general purpose LLM to create
an ad-hoc search engine for musicians by giving it the prompt string: “Genre:
Jazz; Artist: Louis Armstrong. Genre: Country; Artist: ”. An LLM would likely
continue the prompt with the name of a country artist, e.g. “Garth Brooks.”
Beyond this toy example, non-ML experts have used prompting to achieve various
ML functionalities in real-time, including code generation, question
answering, creative writing, etc. (Swanson et al., 2021; Mishra et al., 2021;
Brown et al., 2020). Recent work on prompt-based prototyping (Jiang et al.,
2022) found that, with LLMs’ fluid adaptation to natural language prompts,
non-ML experts (e.g. designers, product managers, front-end developers) can
now prototype custom ML functionality with lower effort and less time, as they
bypass the otherwise necessary but expensive process of collecting data and
training models upfront (Jiang et al., 2022; Bommasani et al., 2021).

Figure 1.  The PromptChainer interface. (A) The _Chain View_ visualizes the
chain structure with node-edge diagrams (enlarged in Figure 3), and allows
users to edit the chain by adding, removing, or reconnecting nodes. (B) The
_Node View_ supports implementing, improving, and testing each individual
node, _e.g.,_ editing prompts for LLM nodes. PromptChainer also supports
running the chain end-to-end (C).

[The figure shows an overview of the PromptChainer interface.]The figure shows
an overview of the PromptChainer interface. On the left, we have a Chain View
which visualizes the chain structure with node-edge diagrams. On the right, we
have a Node View that supports implementing, improving, and testing each
individual node.

Despite the demonstrated versatility of LLMs, many real-world applications
involve complex or multi-step tasks that are nontrivial for a single run of an
LLM. For example, a music-oriented chatbot (which we build in Figure 3) may
require an AI to first determine a user’s query type (_e.g.,_ find artists by
genre as shown above, or find songs given artists, etc.), before generating a
response based on the query type. As a result, designers and developers may
struggle to prototype realistic applications with only a single LLM prompt. In
response, we previously proposed _Chaining_ multiple LLM runs together (Wu et
al., 2022), _i.e.,_ decomposing an overarching task into a series of highly
targeted sub-tasks, mapping each to a distinct LLM step, and using the output
from one step as an input to the next. They observed that people can
effectively _use_ Chaining: they could complete more complex tasks in a more
transparent and controllable way. However, it remains an open question how to
support users in _authoring_ their own LLM chains. For designers and
developers to apply chaining to their own prototypes, they need to not only
prompt within each individual LLM step, but also design the overarching task
decomposition. Such a process requires targeted tooling, akin to end-user
programming (Burnett et al., 2004; Burnett, 2010).

In this work, we examine the user experience of _authoring_ LLM chains.
Through formative studies, we distill three unique challenges that emerge from
the extreme versatility of LLMs: (1) the overhead of fully utilizing LLM
capabilities, (2) the tendency of inadvertently introducing errors to the
chain when prompting, and (3) the cascading errors caused by blackbox and
unstable LLM generations. Addressing these challenges, we propose
PromptChainer, a chain authoring interface that provides scaffolds for
building a mental model of LLM’s capabilities, handling arbitrary LLM data
formats, defining a “function signature” for each LLM step, and debugging
cascading errors. We conduct case studies with four designers and developers,
who proposed and built chains for their own realistic application ideas
(_e.g.,_ chatbots, writing assistants, etc.) Our qualitative analysis reveals
patterns in how users build and debug chains: (1) users build chains not only
for addressing LLM limitations, but also for making their prototypes
extensible; (2) some users constructed one step of a chain at a time, whereas
others sketched out abstract placeholders for all steps before filling them
in; (3) the interactions between multiple LLM prompts can be complex,
requiring both local and global debugging of prompts.

We also observed some additional open challenges, and conclude with discussion
on future directions: First, how can we scale chains to tasks with high
interdependency or logical complexity, while still preserving global context
and coherence? Second, how can we find a “sweet spot” for prompting such that
users can quickly low-fi prototype _multiple_ alternative chains, without
investing too much time designing any single prompt?

##  2\. Background: Large Language Models, Prompting and Chaining

A generative language model is designed to continue its input with plausible
output (_e.g.,_ given a prompt “I went to the”, it might auto-complete with
“coffee shop”). However, when pre-trained on billions of samples from the
Internet, recent LLMs can be adapted on-the-fly to support user-defined use
cases like code generation, question answering, creative writing, etc. (Brown
et al., 2020; Swanson et al., 2021). To invoke the desired functionalities,
users need to write _prompts_ (Lu et al., 2021; Betz et al., 2021; Liu et al.,
2021) that are appropriate for the task. The most common patterns for
prompting are either zero-shot or few-shot prompts. Zero-shot prompts directly
describe what ought to happen in a task. For example, we can enact
_Classification_ in Figure 4 with a prompt such as “Is the statement: ‘hey
there, what’s up’ about music?” In contrast, few-shot prompts show the LLM
what pattern to follow by feeding it examples of desired inputs and outputs:
“[Dialog] Play some music I like. [Class] is_music [Dialog] hey there, what’s
up [Class]”. Given this prompt, the LLM may respond with “not_music” (full
prompt in Figure 4).

While a single LLM enables people to prototype specific ML features (Jiang et
al., 2022), their inherent limitations (_e.g.,_ lack of multi-step reasoning
capabilities) make them less capable of prototyping complex applications. In
response, we previously proposed proposed the notion of chaining multiple LLM
prompts together, and demonstrated the utilities of _using_ chains (Wu et al.,
2022). We follow up on their work to explore how users can _author_ effective
chains for prototyping.

Figure 2.  An example chain for prototyping music chatbot, modified from a
pilot user’s chain (its overview is in Figure 1). We provide primary input-
output examples, and annotate the node functionalities are annotated inline.

The figure an example chain for prototyping music chatbot, modified from a
pilot user’s chain.

Node Type |  Description |  Example in Figure 3
---|---|---
LLM |  Generic LLM |  Use the LLM output directly as the node output. |  4 6 9
LLM Classifier |  Use LLM output to filter and branch out inputs. |  2 3
Helper |  Evaluation |  Filter or re-rank LLM outputs by human-designed criteria, _e.g.,_ politeness. |  10 Toxicity classifier111From Tensorflow.js https://github.com/tensorflow/tfjs-models/tree/master/toxicity
Processing |  Pre-implemented JavaScript functions for typical data transformation. |  5 Split by number
Generic JavaScript |  User-defined JS functions, in case pre-defined helpers are insufficient. |  8 Format the query
Comm. |  Data Input |  Define the input to a chain. |  1
User Action |  Enables external (end user) editing on intermediate data points. |  (Figure 5 11)
API Call |  Call external functions to connect professional services with LLMs. |  6 Call YouTube API

Figure 3. A summary of node types, including the core LLM nodes, helper nodes
for data transformation and evaluation, and communication nodes for exchanging
LLM data with external users or services.

The table summarizes all the node types in detail. It includes the core LLM
nodes, helper nodes for data transformation and evaluation, and communication
nodes for exchanging LLM data with external users or services.

##  3\. PromptChainer: Interface Requirement Analysis & Design

###  3.1. Requirement Analysis

Figure 4.  An expansion of Figure 3, is about music: (A) Node visualization:
the node has an status icon ($a_{1}$), a list of named input ($a_{2}$) and
output handles ($a_{3}$), as well as detailed data previews ($a_{4}$). (B)
Implementation: the handle names are synchronized with the underlying prompt
template ($b_{1}$). (C) We can debug the node at multiple levels.

To inform the design of PromptChainer, we conducted a series of formative
studies with a team of two software engineers and three designers over the
course of three months. These formative studies included sessions reflecting
on their experiences authoring LLM chains without any assistance, as well as
iterative design sessions where they proposed and built their chains-of-
interest in early prototypes for several rounds, reporting back their pain
points (the chains from pilot studies are in Appendix A). We summarize our
observations into the following authoring challenges:

  1. C.1

The versatility of LLMs and need for data transformations: The versatility of
LLMs means that users need to develop a mental model of their capabilities.
LLMs also produce outputs in arbitrary string formats, making it nontrivial to
transform the output of upstream LLM steps to be compatible with the input to
downstream steps.

  2. C.2

The instability of LLM function signatures: LLMs also lack stable _function
signatures_ , _i.e.,_ the semantic types of their outputs easily vary with LLM
prompts. This complicates local chain iterations: for example, if a user’s
edits on a prompt unintentionally make an LLM step output _numbered lists_
instead of _short phrases_ , this would introduce input errors to the
following steps, and thereby break the entire chain.

  3. C.3

The likelihood of cascading errors: The black-box nature of LLMs means that
sub-optimal or even unsafe output in a single step could potentially lead to
cascading errors across an LLM chain (see (Sculley et al., 2014) for a similar
observation in traditional machine learning pipelines).

###  3.2. Interface Design

We design the interface in Figure 1 in response to the challenges, with a
_Chain View_ (Figure 1A) for authoring the chain structure, a _Node View_
(Figure 1B) for authoring a single step (node) of the chain, and support for
chain debugging.

The Chain View is the visual panel for building and viewing chains. As in
Figure 1A, each rectangle depicts a node, which represents a single step in
the chain, with the edges between them denoting how these nodes are
_connected_ , or how the output of one node gets used as the input to the
next.

_Node visualization._ As shown in Figure 4 (a zoomed-in node of Figure 3 2),
each node can have one or more named inputs ($a_{2}$) and outputs ($a_{3}$),
which are used to connect nodes. Inspired by several existing node-edge-based
visual programming platforms222 _e.g.,_ Maya:
https://www.autodesk.com/products/maya/overview; Node-RED:
https://nodered.org/, we provide node previews to increase chaining
transparency, including a status icon highlighting whether the node contains
errors (Figure 4$a_{1}$), as well as inline and detailed data views ($a_{3}$
and $a_{4}$).

_Node Types._ As summarized in Figure 3, we define several types of nodes to
cover diverse user needs. At its core are two types of LLM nodes: Generic LLM
nodes and LLM Classifier nodes (See the node definitions and examples in
Figure 3). Users can implement these nodes by providing a natural language
prompt, call an LLM with the prompt as input, and use the LLM outputs
accordingly. PromptChainer also provides helper nodes that address common data
transformation (C.1 in Section 3.1) and evaluation needs (C.3), or to allow
users to implement their own custom JavaScript (JS) nodes. Finally, to support
users in prototyping AI-infused applications, PromptChainer provides
communication nodes for exchanging data with the external world (_e.g.,_
external API calls).

_Example gallery._ To address the versatility challenge of LLMs (C.1),
PromptChainer provides examples of frequently composed (sub-)chains, to help
users develop a mental model of which capabilities are likely to be useful.
These examples also serve as a soft nudge towards a set of prompting patterns,
such that users’ prompts are more likely to be compatible with predefined
processing nodes. For example, Figure 3 4 is forked from an _Ideation_ example
that returns numbered lists “1) Garth Brooks 2) George Strait…”, which is
parsable with the provided 5 Split by number node.

The Node View allows users to inspect, implement, and test individual nodes
(Figure 1B). When a node is selected, the panel changes in accordance with the
Node Type. PromptChainer automatically parses the input names of a node based
on the LLM prompt for that node (or, for JavaScript helper nodes, based on the
function signature). For example, in Figure 4, the input handle $a_{1}$ “user”
is synchronized with the bolded placeholder string [[user]] ($b_{1}$) in its
corresponding prompt template, meaning that the input to $a_{1}$ will be used
to fill in $b_{1}$ in the prompt. If a user changes $b_{1}$ to _e.g.,_
[[sentence]], $a_{1}$ would get renamed to “sentence,” such that there will be
no outdated handles. As such, PromptChainer automatically updates the global
chain to be consistent with users’ local edits (addressing C.2).

Interactive debugging functionalities. To address the cascading error
challenge (C.3), PromptChainer supports chain debugging at various levels of
granularity: First, to _unit test_ each node, users can use the provided
_testing block_ (Figure 4$c_{1}$) to test each node, with examples independent
of the remaining chain. Second, to perform _end-to-end assessment_ , users can
run the entire chain and _log the outputs per node_ , such that the ultimate
chain output is easy to retrieve (Figure 4$c_{2}$). Third, to help users map
global errors to local causes, PromptChainer supports _breakpoint debugging_
(Figure 4$c_{3}$), and allows users to directly edit the output of a node
before it is fed into the next node. By fixing intermediate node outputs,
users can test a subset of downstream nodes independent of earlier errors.

##  4\. User Feedback Sessions

We conducted a preliminary study to understand what kinds of chains would
users want to build, the extent to which PromptChainer supports their needs,
and what additional challenges users face.

###  4.1. Study design

Because Chain authoring goes beyond single prompts to chaining multiple
prompts together, we recruited four participants (3 designers, 1 developer
within Google) who had at least some prior experience with non-chained
prompts: P1 and P2 had prior experience writing prompts, and P3 had seen some
LLM demos. We personally reached out to these participants in different
product teams, to prioritize interests and experience in a wide range of
domains. Before the study session, participants spent 30 minutes on
preparation: They watched a 10-minute tutorial on interface features. They
were also asked to prepare a task beforehand that they believed would require
multiple runs of the LLM, envision the LLM call for each step, and draft each
of those prompts. This way, the study could focus on chaining prompts together
rather than writing the initial prompts. In the hour-long actual study,
participants loaded their prompts and authored their envisioned Chain while
thinking aloud. To observe the extent to which PromptChainer could support
iteration, we asked participants to describe deficiencies (if any) in their
Chain, and modify their Chains. We observed and recorded their entire task
completion sessions, and later transcribed their comments for qualitative
analysis. In total, participant spent approximately 90 minutes, and received a
$75 gift credit for their time.

Underlying LLM. All of our experiments (including pilot study) rely on the
same underlying LLM called LaMDA (Thoppilan et al., 2022)333We used a non-
dialog version of the model.: a 137 billion parameter, general-purpose
language model. This model is roughly equivalent to the GPT-3 model in terms
of size and capability: it is trained with more than 1.5T words of text data,
in an auto-regressive manner using a decoder-only Transformer structure. It
has comparable performances with GPT-3 on a variety of tasks, and behaves
similarly in its ability to follow prompts.

Figure 5.  Four different chains built by user study participants. P1 and P2’s
chains used parallel branching logic , whereas P3 and P4’s chains depict
iterative content processing. The full details are in Figure 6, Appendix A.

###  4.2. Study Results

We analyze our study to answer the three questions listed below.

Q: What kinds of chains would users want to build?
A: Users proposed diverse tasks, some that used branching logic, and some that
iterated on content. They used chaining not only to address single-prompt
limitations, but also to make their prototypes extensible.

Chaining patterns. All participants successfully built their desired chains
during the study session, with the chains containing on average $5.5\pm 0.9$
nodes. As shown in Figure 5, participants built chains for various kinds of
tasks, ranging from music chatbot, to ads generator, to writing assistants.
Their chain structures reflect different high-level patterns: (1) P1 and P2
built chains with parallel logic branches, similar to decision trees (Geng et
al., 2018). For example, P2’s chain sought to generate specialized
descriptions for different kinds of product review attributes. They first
classified whether attributes were “high end”, “discount”, or “generic” with
4, which determined which downstream node (specialized LLM description
generator) would run. (2) P3 and P4 built chains that incrementally iterate on
content. These chains usually take a divide-and-conquer strategy (Kittur et
al., 2011). For example, P4 wrote one story by first generating a list of
story outlines 10, then generating paragraphs per point 12, and finally
merging them back 13.

Chaining rationales. While we primarily expected participants to author chains
for the purpose of combating LLM limitations (as we previously observed (Wu et
al., 2022)), interestingly, participants also inserted chaining steps to make
their prototypes more generalizable. For example, though P3 knew they could
directly build an ideation node for “summer vacation activities”, they instead
chose to build a more general ideator 6 combined with a summer activity
classifier 7, such that they could _“switch the classifier for any other time
or variables, for variability and flexibility.”_ Further, P4 mentioned the
desire for taking intermediate control. Because he noticed the top-1 LLM
generations were not necessarily their favorite, he built a chain to make
active interventions: He first _over-generated_ three candidate story spines
in 10, from which he would select and refine one of them in 11 for subsequent
paragraph expansions.

Q: To what extent does PromptChainer support users in iteratively authoring
and improving their chains?
A: PromptChainer supported a variety of chain construction strategies and
enabled multi-level debugging.

#### _Chain construction._

Participants applied various chain construction strategies. P1 performed a
top-down approach, _i.e.,_ they connected blank, placeholder nodes first to
illustrate the imaginary task decomposition before filling in the prompts. In
contrast, the other three participants worked on each node one at a time,
before moving on to the next node: whereas P2 carefully ran and tested each
node, others created “rough drafts”, starting with basic prompts and deferring
detailed refinement of prompts until after a draft chain was finished (P3: _“I
should probably move on with this, I want to fine-tune my LLM prompt
later.”_). These varied chain construction strategies indicate that
PromptChainer can support different pathways for chain prototyping, but that a
common user tendency may be to work one node at a time.

To further characterize the node utilities, we analyzed the node distributions
in both the user study chains and those from pilot users (8 in total). We
found that _pre-defined helpers could cover most of the chaining needs_ :
participants used three times as many pre-defined helpers (13 in total) as
customized JS nodes (4). One author further codified all the LLM nodes
according to the primitive LLM operations previously identified (Wu et al.,
2022), and found that out of the 27 LLM nodes, 7 were for categorizing inputs,
13 for sourcing information from the LLM, and 7 for re-organizing the input.
This variety in utilization may have resulted from the PromptChainer’s example
galleries. For example, P4 successfully created their own LLM classifier by
forking a simple default example, even though they were less familiar with
prompting.

Chain debugging. When they completed constructing their chains, all
participants ran the chain end-to-end (Figure 4$C_{2}$) as an “initial
debugging strategy” (P1 and P4). Afterwards, they usually attributed chain
failure to particular LLM nodes (P1: _“easy to pinpoint unexpected outputs
from the data preview”_), and performed local debugging. P1 appreciated the
_breakpoint functionality_ (Figure 4$C_{3}$), as they did not need to take the
chain apart in order to debug one of the nodes; P3, on the other hand, relied
on the independent _testing block_ (Figure 4$c_{1}$) when debugging the
Description Generator 9, as a way to avoid expensive executions on multiple
inputs coming from prior nodes.

Interestingly, most participants made some non-trivial refinements to their
pre-built prompts _in the interface_ , even though they had spent a fair
amount of time doing prompt engineering before the study. We hypothesize that
being able to observe the interaction effects _between_ nodes affected their
understanding and expectations of each local node. For example, when
constructing the story creation chain, P4 wanted to add a final ending, “The
End”, to the generated story. They first tried to always generate “The End” as
the final bulletpoint in the story outline (“Story Spine”) in 10, but realized
that this would cause paragraph generator 12 to produce a paragraph repeating
“The End The End The End.” They therefore removed this generation from 10, and
instead (with some help from the study facilitator) made a final JavaScript
helper node 13 for appending the text “The End”. This suggests that
PromptChainer can help users _discover_ errors, though future research is
needed in supporting them to _resolve_ identified problems through alternative
solutions.

Q: What are remaining challenges in chain authoring?
A: Ensuring coherence between interdependent sub-tasks; tracking chains with
complex logic.

Chains with interdependent parallel tasks can lead to decreased coherence.
Because P4’s story writing chain independently generated a paragraph for each
point in the outline, the final essay lacked coherence: though the first
several sentences followed the input description (“Morris…hates eating
flies”), the final sentence instead hinted that Morris _likes_ flies (“none of
them tasted as good as flies”). One pilot study user faced a similar
challenge, and created another input node to manually track previous outputs.
In the future, it may be worthwhile to further investigate methods that
consider inter-dependency between parallel sub-tasks (Swanson et al., 2021).

Chains that involve more complex decomposition can be overwhelming to track.
In P1’s music chatbot chain, the extractor node 2 produces _a list of_
candidate entities per input. Thus, it became unclear how the entities fed
into the classifier 3 mapped to the original input node 1. We hope to enhance
the tracing capabilities of PromptChainer. For example, future work could
enable customized chain grouping: Instead of running one or all of the nodes,
participants can explicitly select a subset to run. We may also add execution
visualizations (_e.g.,_ taking inspiration from Python
Tutor444https://pythontutor.com/), to highlight the mapping from the original
input all the way to the final output.

###  4.3. Discussion and Limitations

Because participants were asked to pre-create some LLM prompts for their
desired sub-tasks prior to the study, this may have unintentionally led to
participants feeling invested in their prompts and their particular chain
decomposition, making them less inclined to consider other chain structures or
scrap the prompts they had already created. Yet, prior work in prototyping
indicates that concurrently considering multiple alternatives (_e.g.,_
parallel prototyping (Dow et al., 2010)) can lead to better outcomes. Thus,
future work could explore how to encourage low-fi prototyping of _multiple_
possible chains: in other words, how can users create half-baked prompts for
each step, such that the feasibility of an entire chain can be rapidly tested,
without investing too much time designing each prompt? For example,
PromptChainer could perhaps encourage users to start with only one or two
examples in a few-shot prompt, or start with a very simple zero-shot prompt
(even if they don’t initially perform reliably) to reduce initial time
invested in each prompt.

Given time constraints in the study, users may have also picked tasks that
were naturally easy to decompose. In the future, it would be worthwhile to
explore task decomposition strategies for even larger and more complex tasks.
For example, PromptChainer could help encourage users to further decompose a
node in the chain into more nodes, if extensive prompting efforts appear
unsuccessful.

##  5\. Conclusion

We identified three unique challenges for LLM chain authoring, brought on by
the highly versatile and open-ended capabilities of LLMs. We designed
PromptChainer, and found that it helped users transform intermediate LLM
output, as well as debug the chain when LLM steps had interacting effects. Our
study also revealed interesting future directions, including supporting more
complex chains, as well as more explicitly supporting “half-baked” chain
construction, so that users can easily sketch out a chain structure without
investing too much time prompting upfront.

## References

  * (1)
  * Betz et al. (2021) Gregor Betz, Kyle Richardson, and Christian Voigt. 2021\.  Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2.  _ArXiv preprint_ abs/2103.13033 (2021).  https://arxiv.org/abs/2103.13033
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the Opportunities and Risks of Foundation Models.  arXiv:2108.07258 [cs.LG]
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * Burnett (2010) Margaret Burnett. 2010\.  End-user software engineering and why it matters.  _Journal of Organizational and End User Computing (JOEUC)_ 22, 1 (2010), 1–22.
  * Burnett et al. (2004) Margaret M. Burnett, Curtis R. Cook, and Gregg Rothermel. 2004\.  End-user software engineering.  _Commun. ACM_ 47 (2004), 53 – 58.
  * Dow et al. (2010) Steven P Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L Schwartz, and Scott R Klemmer. 2010.  Parallel prototyping leads to better design results, more divergence, and increased self-efficacy.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 17, 4 (2010), 1–24.
  * Geng et al. (2018) Baocheng Geng, Qunwei Li, and Pramod K Varshney. 2018.  Decision tree design for classification in crowdsourcing systems. In _2018 52nd Asilomar Conference on Signals, Systems, and Computers_. IEEE, 859–863.
  * Jiang et al. (2022) Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J. Cai. 2022.  Prompt-based Prototyping with Large Language Models. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_.
  * Kittur et al. (2011) Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011.  CrowdForge: Crowdsourcing Complex Work. In _Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology_ (Santa Barbara, California, USA) _(UIST ’11)_. Association for Computing Machinery, New York, NY, USA, 43–52.  https://doi.org/10.1145/2047196.2047202
  * Lieber et al. (2021) Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021\.  _Jurassic-1: Technical Details And Evaluation_.  Technical Report. AI21 Labs.
  * Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021\.  What Makes Good In-Context Examples for GPT-$3$?  _ArXiv preprint_ abs/2101.06804 (2021).  https://arxiv.org/abs/2101.06804
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021.  Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity.  _ArXiv preprint_ abs/2104.08786 (2021).  https://arxiv.org/abs/2104.08786
  * Mishra et al. (2021) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021.  Cross-Task Generalization via Natural Language Crowdsourcing Instructions.  _ArXiv preprint_ abs/2104.08773 (2021).  https://arxiv.org/abs/2104.08773
  * Sculley et al. (2014) D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, and Michael Young. 2014\.  Machine Learning: The High Interest Credit Card of Technical Debt. In _SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop)_.
  * Swanson et al. (2021) Ben Swanson, Kory Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu. 2021.  Story Centaur: Large Language Model Few Shot Learning as a Creative Writing Tool. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 244–256.  https://aclanthology.org/2021.eacl-demos.29
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022\.  LaMDA: Language Models for Dialog Applications.  _ArXiv preprint_ abs/2201.08239 (2022).  https://arxiv.org/abs/2201.08239
  * Wu et al. (2022) Tongshuang Wu, Michael Terry, and Carrie J Cai. 2022\.  AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3491102.3517582
  * Yang et al. (2020) Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020.  _Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design_.  Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376301

Figure 6.  The full details of user study chains.

##  Appendix A Sample Chains from Pilot Users

Figure 7.  The chains built by pilot users.