:
225 2023 LEAVE UNSET LEAVE UNSET Machine Learning for Health (ML4H) 2023

# LLMs Accelerate Annotation for Medical Information Extraction

Akshay Goel<EMAIL_ADDRESS>
Almog Gueta11footnotemark: 1<EMAIL_ADDRESS>
Omry Gilon11footnotemark: 1<EMAIL_ADDRESS>
Chang Liu<EMAIL_ADDRESS>
Sofia Erell<EMAIL_ADDRESS>
Lan Huong Nguyen<EMAIL_ADDRESS>
Xiaohong Hao<EMAIL_ADDRESS>
Bolous Jaber<EMAIL_ADDRESS>
Shashir Reddy<EMAIL_ADDRESS>
Rupesh Kartha<EMAIL_ADDRESS>
Jean Steiner<EMAIL_ADDRESS>
Itay Laish<EMAIL_ADDRESS>
Amir Feder<EMAIL_ADDRESS>
Google Research These authors contributed equally.

###### Abstract

The unstructured nature of clinical notes within electronic health records
often conceals vital patient-related information, making it challenging to
access or interpret. To uncover this hidden information, specialized Natural
Language Processing (NLP) models are required. However, training these models
necessitates large amounts of labeled data, a process that is both time-
consuming and costly when relying solely on human experts for annotation. In
this paper, we propose an approach that combines Large Language Models (LLMs)
with human expertise to create an efficient method for generating ground truth
labels for medical text annotation. By utilizing LLMs in conjunction with
human annotators, we significantly reduce the human annotation burden,
enabling the rapid creation of labeled datasets. We rigorously evaluate our
method on a medical information extraction task, demonstrating that our
approach not only substantially cuts down on human intervention but also
maintains high accuracy. The results highlight the potential of using LLMs to
improve the utilization of unstructured clinical data, allowing for the swift
deployment of tailored NLP solutions in healthcare.

###### keywords:

Medical NLP, Large Language Models, Data Annotation

##  1 Introduction

Medical NLP holds promise for transforming healthcare by extracting valuable
insights from vast amounts of unstructured clinical data (Feder et al., 2022;
Zhang et al., 2022; Liu et al., 2023). These NLP insights can improve data
organization and highlight critical clinical information for users (Landolsi
et al., 2022). However, a significant impediment to this transformation is the
challenge of obtaining high-quality labeled data for training and evaluating
machine learning models specific to medical NLP tasks (Hakala and Pyysalo,
2019). The crux of the issue resides in the human labeling process. Manual
annotation, especially when it necessitates the expertise of medical
professionals, is labor-intensive and expensive. Given the intricacies of
medical data, achieving a comprehensive and accurate set of text spans
corresponding to specific medical entities often requires multiple annotators
or iterative rounds of annotation.

Figure 1: Evaluation overview comparing annotation efficiency between LLM-
assisted (left) and human-only (right) annotation workflows using the
identical document set. Duration and metrics of each phase are benchmarked
against i2b2 ground-truth labels.

To alleviate this bottleneck, in this work, we test the potential of LLMs to
streamline the labeling process (Ding et al., 2022; Liu et al., 2022). LLMs
have recently gained immense popularity due to their generalizability across
various tasks (Zhou et al., 2023b). _Prompt engineering_ is the optimization
of natural language instructions to effectively condition an LLM (Reynolds and
McDonell, 2021). When incorporating examples of the task, this is termed _few-
shot learning_ (Brown et al., 2020). Notably, _prompt engineering_ has shown
resilience against overfitting (Akinwande et al., 2023). Although LLMs offer
the allure of automating data labeling, they have two main limitations: their
propensity for generating unconstrained outputs and their inability to
guarantee expert-level performance (Agrawal et al., 2022).

Acknowledging this, we introduce a two-step method that combines the
capabilities of LLMs with human expertise. Initially, the LLM, conditioned in
a _few-shot learning_ setup, generates Base Annotations. Subsequently, these
annotations are refined by medical annotation experts to produce Refined
Annotations.

To validate the efficacy of our approach, we compare it against a standard
annotation pipeline, which utilizes a round of human Base Annotations followed
by expert human Refinement Annotations. Figure 1 provides an overview of our
evaluation, comparing an LLM-assisted annotation process to a human-only
annotation process. Our empirical evaluation, focused on the medication-
extraction task, highlights the potential of our approach to significantly
reduce the human annotation workload while maintaining expert-level high-
quality annotations. Through this study, we aim to establish a more efficient
route for labeling data to deploy robust medical NLP solutions, ultimately
driving advancements in patient care and healthcare operations.

Our contributions are:

  1. 1.

We empirically test the utility of LLMs in annotating clinical narratives, a
core bottleneck in medical NLP (see Figure 1).

  2. 2.

We present a method that significantly accelerates the annotation process
while maintaining expert-level quality.

  3. 3.

We produce a set of labels for medication extraction using a public medical
dataset (MIMIC-IV-Note) (Johnson et al., 2023).111The labels can be accessed
at PhysioNet (Goel et al., 2023; Goldberger et al., 2000 (June 13)).

The paper is structured as follows. In Section 2, we discuss prior work on
clinical information extraction and the challenges these systems face. In
Section 3, we present the data and task with which we are experimenting.
Subsequently, in Section 4 and Section 5, we describe the human annotators and
the LLM pipeline that supports them, respectively. Using these annotators, we
then present in Section 6 our empirical findings and discuss the efficiency
gains from using our annotation approach. Finally, in Section 7 we discuss
paths forward for utilizing LLMs to enhance medical NLP while ensuring
reliability.

##  2 Related Work

#### Information Extraction from Clinical Notes.

With the rise of Electronic Health Records (EHRs), there is a pressing need
for efficient clinical information extraction systems (Dash et al., 2019;
Birkhead et al., 2015). Extracting patient information from medical notes is
pivotal in clinical-NLP, with many standardized datasets and competitions
(Uzuner, 2009; Savova et al., 2010; Jensen et al., 2012; Ford et al., 2016;
Zhu et al., 2018). Modern solutions often employ the transformer architecture
for these tasks (Vaswani et al., 2017; Peng et al., 2019; Yadav and Bethard,
2019; Si et al., 2019; Lee et al., 2020). These approaches have shown promise
in tasks like biomedical Named Entity Recognition (NER) but are constrained by
their reliance on extensive domain-specific labels (Hakala and Pyysalo, 2019).

#### LLMs in Medical NLP.

LLMs have transformed medical NLP, excelling in tasks such as medical
question-answering and unsupervised clinical information extraction (Driess et
al., 2023; Singhal et al., 2022; Agrawal et al., 2022). Their adaptability
allows for extension across various medical domains, from oncology to
telemedicine, and their ability to be distilled into task-specific models,
such as UniversalNER, highlights their versatility (Zhou et al., 2023a).

#### Active Learning in Medical NLP.

Active learning has emerged as a solution to reduce manual labeling in medical
NLP. For instance, Kholghi et al. (2015) utilized active learning for medical
concept extraction, while Wei et al. (2019) introduced a cost-aware method.
Feder et al. (2020) combined BERT (Devlin et al., 2019) with active learning,
emphasizing iterative manual refinement. While promising, these methods
require explicit model parameter tuning to fit a data distribution and task.

In this work, we propose an LLM-based approach that significantly enhances
medical information extraction over traditional methods, without necessitating
model parameter tuning.

##  3 Data and Task

###  3.1 i2b2 Dataset

Our experiment utilizes the $2009$ i2b2 Workshop on NLP Challenges dataset
(Uzuner et al., 2010), consisting of de-identified clinical discharge
summaries (documents). Each document contains labels of medication fields,
which includes the medication’s name, dosage, mode (route) of administration,
frequency, duration of administration, and the reason for
administration.222Uzuner et al. (2010) note a narrative field, distinguishing
list structure from narrative text in discharge summaries. This was omitted as
our focus is classification, not extraction. Additionally, the reason field
was excluded due to its high variance in human labeling (Uzuner et al., 2010).
The fields of a single medication are linked together to create a medication
entry.

###  3.2 Annotation Task

The i2b2 dataset provides high-quality labels for a medication-extraction
task, divided into two sub-tasks: Named Entity Recognition (NER) for
medication fields and Relationship Extraction (RE) to associate these fields
with specific medication entries. This involves a two-step process:

  1. 1.

NER Task: Identifies the textual spans corresponding to each medication field,
marking the start and end token indices.

  2. 2.

RE Task: Forms medication entries by linking related fields for each
medication mention.

A visual representation of the labels and task can be seen in Figure 2.

Figure 2: Example of a medication-extraction annotation, showcasing two
medication entries underlined in yellow. Field types are highlighted by color.

###  3.3 Data Splits & Pre-Processing

The dataset, comprising $261$ documents, was randomly divided into three
subsets: the ‘pilot set’ ($16$ documents, $6\%$), used to ensure human
annotators’ understanding of the guidelines and task and to select expert
raters for the refinement task (see §4); the ‘development set’ ($29$
documents, $11\%$), for selecting optimal prompts and parameters for the LLM
pipeline; and a ‘test set’ ($216$ documents, $83\%$), for measuring the
performance of both annotation methods, with human or LLM Base Annotations.
Table 1 presents the number of occurrences of each field in each dataset
subset.

| Name | Dose | Mode | Frequency | Duration
---|---|---|---|---|---
Total | $9,237$ | $4,646$ | $3,491$ | $4,182$ | $562$
Pilot | $511$ | $300$ | $180$ | $282$ | $38$
Development | $1,123$ | $523$ | $437$ | $452$ | $57$
Test | $7,603$ | $3,823$ | $2,874$ | $3,448$ | $467$

Table 1: Number of occurrences for each labeled field across all data subsets
of the i2b2 2009 dataset.

Minimal preprocessing was performed on the dataset to remove leading and
trailing punctuation, spaces, and tabs from label spans. In addition,
discontinuous medication fields were infrequent, constituting less than 1% of
the data, and were therefore excluded to streamline the preprocessing stage.

##  4 Human Annotations

Human annotations in the medical domain often necessitate medical expertise
and extensive training, making the process both time-consuming and costly.
Annotation typically involves marking spans of text in lengthy documents,
requiring high focus to avoid errors. Rater fatigue is thus a significant
concern.

#### Annotation Workflow:

To mitigate the above challenges, our annotation process comprises two
sequential phases:

  1. 1.

Base Annotation Phase: Initial labels are added to blank documents by a rater.

  2. 2.

Refinement Annotation Phase: The base labels are refined and corrected by
specialized _expert raters_.

#### Annotation Platform and Guidelines

Annotations were carried out on a text labeling platform, configured to
support the medication-extraction schema detailed in §3.2. The platform
consists of a viewing panel to display raw medical documents and a question
panel to guide the annotation process. Special cases like ambiguity between
frequency and duration were handled according to specific rules based on the
original _i2b2_ guidelines.

#### Raters Team Composition and Expertise

Our annotation team was composed of seven raters, each with a medical
background and an extensive experience in medical NLP tasks. Their medical
training ranged from medical assistant to attending physician.

A pilot task was conducted to select the _expert raters_ based on their
performance in annotating a pilot set of documents, resulting in the selection
of three expert raters. To optimize the use of rater time and expertise, the
more exact Refinement Annotations were performed solely by _expert raters_.
Additionally, both _expert raters_ and _non-expert raters_ contributed to Base
Annotations. This setup allowed us to perform a subgroup analysis on
expertise-level for the annotation pipeline.

#### Rater Tasks

Raters were responsible for three types of tasks, where documents were
randomly allocated to ensure no rater annotated a document twice:

  1. 1.

Base Rater Label: Base Annotation of medical documents (labeling documents
from scratch). Performed by _all raters_.

  2. 2.

Refined Rater Label: Refinement Annotation of labeled documents produced by
another rater. Performed by _expert raters_.

  3. 3.

Refined LLM Label: Refinement Annotation of labeled documents produced by the
LLM pipeline. Performed by _expert raters_.

##  5 LLM Annotations

To assess the efficiency gains from LLM annotation in medication extraction,
we devised an LLM labeling pipeline that consists of three key components: (1)
an LLM, (2) a task-specific prompt with an input chunk of text, and (3) a
post-processing Resolver Module that converts the generative outputs into NER-
RE structured objects.

The steps for the LLM medication-extraction annotation pipeline are:

  1. 1.

Input document is segmented into text chunks, which are delineated by textual
breaks such as new lines and end-of-sentence markers.

  2. 2.

Each text chunk is incorporated into a task-specific _few-shot prompt_
designed for medication extraction.

  3. 3.

The LLM performs inference on this prompt, generating textual outputs.

  4. 4.

Generative textual outputs are converted into structured medication-extraction
objects by the resolver module. Each object encapsulates the relevant
medication fields along with their associated RE information.

  5. 5.

Optionally, human reviewers can refine these annotations to enhance their
quality.

Figure 3 provides an illustration of the pipeline.

Figure 3: Overview of the LLM medication-extraction annotation pipeline. Our
labeling pipeline consists of a task-specific prompt with an input chunk of
text, an LLM, and a post-processing Resolver Module that converts the
generative outputs into NER-RE structured objects.

#### PaLM 2 Language Model.

To construct the LLM medical-extraction annotation pipeline, we utilized
Google’s latest-generation language model, PaLM 2 (Anil et al., 2023). This
model has achieved state-of-the-art performance across a broad spectrum of
tasks. While employing a fine-tuned version of PaLM 2 for the NER-RE task
might yield even better results, our primary objective was to demonstrate a
proof-of-concept using an LLM, without explicit tuning.

###  5.1 Prompt Structure

To apply a generalized LLM to the specific medication-extraction task, we
utilize _in-context_ or _few-shot learning_ (Brown et al., 2020). This
approach leverages prompts containing: contextual information, a description
of the task, and one or more _few-shot examples_ with input-output pairs.
During inference, the input chunk text is integrated into the prompt, as
illustrated in Figure 4. (See appendix F for example prompt implementations).

Structure of Prompt Contextual Information:
“You are a medical assistant…”
Task Description:
“Tag groups of entities related to medications using YAML format. Include drug
names, dose, etc…”
Examples:
Question: “Patient is on Prozac 20 mg…”
Answer:

    entities:
      - group: 1
        medication_name:
          text: Prozac
        ...

Question/Answer Pairs Continued…
Question: {input_chunk_text}
Answer: Figure 4: Prompt structure for LLM annotation: includes contextual
information, task description, and curated examples to illustrate task
intricacies.

###  5.2 Medication-Extraction Prompts Schema

To leverage in-context learning for medication extraction, the task should be
precisely described and defined within the prompt. This information shapes the
generative outputs from the LLM. We evaluated two prompt schemas, and
generated annotations of both of which were ultimately ensembled to achieve
optimal performance. The first schema, IOB-Token, utilizes a modified version
of the Inside-Outside-Beginning (IOB) token-tagging technique (Ramshaw and
Marcus, 1999), using medication fields as possible classes. The IOB method
classifies each token into a specific class. Our modification introduces an
additional group tag to retain RE information, as depicted in Figure 5.

IOB-Token Annotation Chunk Input: “Patient takes Prozac 20 mg and Zoloft 50
mg…” Token IOB Tag-Label Group Tag ‘Patient’ O None ‘takes’ O None ‘Prozac’
B-MEDICATION ‘entity_1’ ‘20’ B-DOSE ‘entity_1’ ‘mg’ I-DOSE ‘entity_1’ ‘and’ O
None ‘Zoloft’ B-MEDICATION ‘entity_2’ ‘50’ B-DOSE ‘entity_2’ ‘mg’ I-DOSE
‘entity_2’ Figure 5: Example of IOB-token annotations for medication
information extraction. In the prompt implementation, the tokenized text, IOB
tag, and group entity are comma-delimited.

The second schema, Direct Chunk, directly produces the fields labels and
grouping for the input chunk of text which may contain multiple words (e.g.,
‘20 mg’), without requiring intermediate tokens. Figure 6 provides an
illustration of this schema.

Direct Chunk Annotation Chunk Input: “Patient takes Prozac 20 mg and Zoloft 50
mg…” Field Type Entity Text Position (start-end) Group: 1 MEDICATION: Prozac
18-24 DOSE: 20 mg 25-30 FREQUENCY: - DURATION: - REASON: - MODE: - Group: 2
MEDICATION: Zoloft 35-41 DOSE: 50 mg 42-47 FREQUENCY: - DURATION: - REASON: -
MODE: - Figure 6: Example of Direct Chunk annotations, demonstrating the
extraction of medication fields from the input by groups. In the prompt
implementation, each group is represented in YAML format.

Once generative outputs are produced, the Resolver Module transforms
unstructured output data into medication-extraction objects. The object
definitions of the output are consistent with the human labeling pipeline,
facilitating seamless integration between LLM and human annotations. Both the
IOB-Token and Direct Chunk schemas have a dedicated resolver module. For IOB-
Token resolver, lines are parsed using a comma as the delimiter, while the
Direct Chunk resolver parses output using a Python YAML library. See section
A.1 for further details.

###  5.3 Prompt Engineering & Ensembling

With an established prompt structure and schema in place, we performed
iterative experiments to identify the optimal LLM prompts. To identify areas
for prompt improvement, we evaluated errors in the ‘development set’. The most
significant error patterns were corrected and incorporated as instructions and
examples into subsequent prompt versions (see section A.2 for further
details). We ultimately developed 10 versions of the IOB-Token schema prompt
and 5 versions of the Direct Chunk prompt. See appendix B for details on LLM
document inference time.

We also evaluated other key parameters: Our early observations indicated that
a temperature of $0.0$ offered the best performance. Max-decoding-steps above
$1024$ tokens had minimal impact, so this value was fixed. Chunk sizes of
$250$ and $180$ offered the best performance for IOB-Token and Direct Chunk,
respectively. See section A.3 for further discussion.

We placed a higher priority on annotation recall over precision because our
empirical experience suggests that deleting annotations is less time consuming
than producing them from scratch. To select for a weighted balance of recall
over precision, we used an F2 score. Section D.2 quantifies the benefit of
prioritizing recall over precision, showing the lower human time cost for
deleting a label compared to adding or modifying one.

Lastly, we evaluated the performance of an ensemble that combines generated
annotations from the best configurations from both prompt schemas. Pseudocode
for the ensemble procedure is presented in section A.4. The ensemble had the
highest F2 score and was therefore chosen for the LLM Base Annotations on the
‘test set’. See section A.5 for experiment results of best performing
configurations.

##  6 Results

We focus on three key aspects: time savings, label quality, and the
performance differentiation between expert and non-expert raters for the Base
Annotations phase. Our main findings are as follows:

  * •

Time Efficiency: Using the LLM pipeline to replace the human rater in the
initial Base Annotations phase reduces human annotation total time by 58% on
average.

  * •

Label Quality: Comparative evaluations between human raters and the LLM
pipeline show that the label quality at the Base Annotations phase is
comparable. Additionally, the expert Refinement phase achieves the same
quality level regardless of the source of Base Annotations — whether from
human raters or the LLM pipeline.

  * •

Expert vs. Non-Expert Performance: Subgroup analysis focusing on _expert
raters_ demonstrates that LLMs improve Base Annotations efficiency by 26% on
average, underlining the utility of LLMs even within exceptionally skilled
raters. Furthermore, the benefit of utilizing LLMs is even more pronounced
compared to _non-expert raters_.

In addition, to substantiate these findings, we have employed statistical
tests for the main experiments. The details of which can be found in Table 4.

Label Type | Vertical | Horizontal | Time Cost
---|---|---|---
| Recall | Precision | F1 | Recall | Precision | F1 | Mean | Median
Base Rater | 0.789 | 0.893 | 0.838 | 0.734 | 0.821 | 0.775 | 17.60 | 11.93
Base LLM | 0.850 | 0.762 | 0.804 | 0.810 | 0.703 | 0.752 | n/a | n/a
Base Rater + Refinement | 0.912 | 0.907 | 0.910 | 0.887 | 0.879 | 0.883 | 26.67 | 19.18
Base LLM + Refinement | 0.921 | 0.893 | 0.907 | 0.892 | 0.860 | 0.876 | 11.32 | 7.27
Table 2: Labeling Quality and Time metrics for the entire Test Set (n=216
documents). Quality metrics are calculated at the phrase-level. _Time Cost_
refers to the total time expended by raters, measured in minutes per document.
Refinement methods demonstrate significantly higher quality, with LLM +
Refinement method reducing annotation time by $57.6\%$, while compromising
only $1\%$ in quality metrics.  Figure 7: Total human time cost for producing
refined labels with and without LLM. Using LLMs reduces $58\%$ of the
annotation time ($11.3$ vs. $26.7$). Figure 8: Human time cost at the two
labeling phases- Base Annotations and Refinement. Refining annotations reduces
time costs by $42\%$ on average compared to starting from scratch.

###  6.1 Efficiency Metrics.

Figure 7 and Table 2 illustrate that the integration of LLMs leads to a
substantial decrease in the human time cost for the production of _expert-
level_ labels. Using LLMs, the mean time required to produce refined labels
for a document decreased by 58%, from $26.67$ minutes to $11.32$ minutes
($p<1\times 10^{-18}$). Figure 8 provides a more granular view of the time
costs of each annotation phase. The _Base Rater_ has the highest time cost,
with a mean of $17.6$ minutes per document, compared to the _Refinement over
Base LLM_ and _Refinement over Base Rater_ , which have means of $11.3$ and
$9.1$ minutes per document, respectively.

###  6.2 Quality Metrics.

To evaluate label quality, we utilize phrase-level evaluation333Token-level
evaluation shows similar results and is presented in Appendix E.2. based on
vertical and horizontal metrics defined by (Uzuner et al., 2010). _Phrase-
level_ refers to the entire text span of a field, as opposed to _Token-level_
which refers to each token in the span, e.g. ‘20 mg’ vs. [‘20’, ‘mg’].
_Vertical metrics_ evaluate NER with respect to fields, while _horizontal
metrics_ also evaluate RE between fields and the medication entry. Pseudocode
for these metrics is presented in appendix C. Table 2 outlines the label
quality across the two phases of labeling, with and without the LLM.

#### Quality of Base Annotations Phase.

Comparing the Quality of _Base Rater Labels_ and _Base LLM Labels_ in Table 2
indicates that the Base rater outperforms in F1 score ($0.838$ vs. $0.804$ in
vertical F1 score, and $0.775$ vs. $0.752$ in horizontal F1 score), due to the
dominance in precision difference, while the _Base LLM_ obtained a higher
recall rates, as we intended (see Sec 5.3).

Label Type | Vertical | Horizontal | Time Cost
---|---|---|---
| Recall | Precision | F1 | Recall | Precision | F1 | Mean | Median
Base Expert Rater | 0.908 | 0.893 | 0.900 | 0.882 | 0.859 | 0.871 | 16.17 | 11.13
Base Expert Rater + Refinement | 0.923 | 0.900 | 0.911 | 0.898 | 0.873 | 0.886 | 22.25 | 15.22
Base LLM | 0.847 | 0.762 | 0.802 | 0.815 | 0.706 | 0.757 | n/a | n/a
Base LLM + Refinement | 0.914 | 0.885 | 0.899 | 0.896 | 0.858 | 0.876 | 11.97 | 7.55
Base Non-Expert Rater | 0.712 | 0.892 | 0.792 | 0.637 | 0.789 | 0.705 | 18.67 | 12.36
Base Non-Expert Rater + Refinement | 0.905 | 0.913 | 0.909 | 0.880 | 0.882 | 0.881 | 30.41 | 21.06
Base LLM | 0.853 | 0.763 | 0.805 | 0.806 | 0.700 | 0.750 | n/a | n/a
Base LLM + Refinement | 0.925 | 0.899 | 0.912 | 0.890 | 0.861 | 0.875 | 10.83 | 7.10

Table 3: Comparison of Quality and Time metrics in Base Annotations by Experts
and Non-Experts. Top: Test documents with Base Annotations performed by
experts (n=93). Bottom: Test documents with Base Annotations performed by non-
experts (n=123). Base LLM and Base LLM + Refinement are included for
completeness. Note: Refinements are exclusively expert-performed. _Time Cost_
denotes the total time spent by raters, in minutes per document. Notably, the
quality of Base annotations by experts without refinement is comparable to
that of Base LLM + Refinement, but requires 26% more time.

Label Comparison | Vertical | Horizontal | Time Cost
---|---|---|---
| $\Delta_{\textbf{Recall}}$ | $\Delta_{\textbf{Precision}}$ | $\Delta_{\textbf{F1}}$ | $\Delta_{Recall}$ | $\Delta_{\textbf{Precision}}$ | $\Delta_{\textbf{F1}}$ | $\Delta_{\textbf{Median}}$
Base LLM+Refin. $-$ Base Rater+Refin. | $0.009^{**}$ | $-0.014^{**}$ | $-0.003$ | $0.006$ | $-0.018^{**}$ | $-0.007^{*}$ | $-11.90^{**}$
Base LLM+Refin. $-$ Base Expert Rater | $0.006^{**}$ | $-0.008^{**}$ | $-0.001$ | $0.013^{**}$ | $-0.001$ | $0.006^{*}$ | $-3.58^{**}$
Base LLM $-$ Base Non-expert Rater | $0.141^{**}$ | $-0.129^{**}$ | $0.013^{**}$ | $0.169^{**}$ | $-0.089^{**}$ | $0.045^{**}$ | $-12.36^{**}$

Table 4: Comparing Quality and Time metrics across labeling methods on the
Test Set. ‘Refin.’ denotes Refinement. Differences ($\Delta$) are calculated
by subtracting the second method’s values from the first. ** and * signify
statistical significance at $p<0.01$ and $p<0.05$. See Appendix D for analysis
details.

#### Quality of Refinement Annotations Phase.

Upon transitioning to the Refinement phase, a striking improvement in label
quality is evident from the results presented in Table 2. Both _Base LLM +
Refinement_ and _Base Rater + Refinement_ exhibit substantial advancements in
their F1 scores. Specifically, the F1 score for _Base LLM + Refinement_
reaches $0.907$ in vertical metrics and $0.876$ in horizontal metrics. These
scores closely mirror the high quality of the _Base Rater + Refinement_ ,
which achieves F1 scores of $0.910$ and $0.883$ for vertical and horizontal
metrics, respectively. An analysis of the precision and recall metrics
demonstrates similar alignment.

Results suggest a noteworthy convergence in label quality between the _Base
LLM + Refinement_ Labels and _Base Rater + Refinement_ Labels. The findings
also highlight the value of expert rater involvement in elevating both LLM-
generated and average-rater annotations to produce high-quality labels.

###  6.3 Impact of Rater Expertise

To further evaluate the quality and efficiency metrics between expert raters
and non-expert raters, we evaluated two subgroups of test documents, differing
based on the rater’s skill level for the Base Annotation phase.

Table 3 shows that the time cost for producing Base Annotations is similar
regardless of the level of expertise. The mean time is 16.17 minutes for
experts and 18.67 minutes for non-experts.444A Figure displaying the time cost
for experts versus non-experts, as well as the time cost per rater, are shown
in section E.3.

Evaluating quality, Table 3 also underscores the strong relationship between
rater expertise and labeling quality. _Base expert raters_ labels achieve high
F1 scores—$0.900$ in vertical and $0.871$ in horizontal metrics—signifying
that these labels seldom require refinement. However, the efficiency aspect
should not be overlooked. The table demonstrates that even _expert raters_
realize significant time savings when assisted by LLMs. Specifically, the
average time for labeling reduces from $16.17$ minutes with _Base Expert Rater
Labels_ to $11.97$ minutes with _Base LLM + Refinement Labels_ , improving
efficiency by 26% ($p<0.01$). Lastly, for the _non-expert raters_ subgroup,
the benefits of using LLMs are even more pronounced, as can be seen in Table
3.

##  7 Discussion

Our study demonstrates that LLMs can significantly accelerate the process of
medication information extraction, achieving baseline accuracy comparable to
that of a trained medical NLP annotator, with superior recall at the expense
of precision. Remarkably, the results were achieved solely through prompt
engineering, without direct model parameter tuning. Furthermore, we show that
when integrated into a human-in-the-loop process, LLMs can facilitate the
generation of expert-level annotations while saving considerable human time.
This is particularly noteworthy given the traditionally high costs and time
investments required for generating high-quality medical NLP labels through
human annotation.

In future work, we plan to explore fine-tuning LLMs for specific tasks to
improve performance, and the integration of constrained decoding to ensure
outputs adhere to predefined schemas. As the field continues to advance
rapidly, we expect to see further improvements in performance and utility.

In addition to speeding up the annotation process, our study reveals the
importance of expert-level human annotators in achieving high-quality results.
We quantified the time-efficiency gains from using LLMs in the annotation
pipeline and found that optimizing for recall is more time-efficient as
deleting annotations is easier than adding new annotations. Although our study
focused on medical information extraction, the flexibility of LLMs suggests
their applicability across various information extraction tasks.

However, our study is not without its limitations. The focus on a single
domain, namely medication extraction, may limit the generalizability of our
findings. As more diverse and validated medical NLP datasets become available,
future studies could broaden this scope. While our study concentrated on the
feasibility of LLM-based labeling, a key area for further investigation
involves directly comparing an LLM annotation pipeline with other ML methods,
including active learning. Lastly, our cost analysis solely considers a basic
metric of human time cost. It is important to differentiate between expert-
level and average raters in future assessments, and to explore a cost-aware
approach.

We believe that integrating LLMs into data labeling workflows has potential to
change the field, especially in areas where high-quality annotations are
required but resources are limited. As LLMs continue to evolve, they are
poised to make the challenge of data acquisition increasingly manageable.

The authors extend their sincere gratitude to Dem Gerolemou for substantial
contributions to the visuals, and to Wei-Hung Weng for offering insightful
feedback during the manuscript’s review process.

## References

  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag.  Large language models are few-shot clinical information extractors.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1998–2022, 2022.
  * Akinwande et al. (2023) Victor Akinwande, Yiding Jiang, Dylan Sam, and J. Zico Kolter.  Understanding prompt engineering may not require rethinking generalization, 2023.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Birkhead et al. (2015) Guthrie S Birkhead, Michael Klompas, and Nirav R Shah.  Uses of electronic health records for public health surveillance to advance public health.  _Annual review of public health_ , 36:345–359, 2015.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Dash et al. (2019) Sabyasachi Dash, Sushil Kumar Shakyawar, Mohit Sharma, and Sandeep Kaushik.  Big data in healthcare: management, analysis and future prospects.  _Journal of Big Data_ , 6(1):54, Jun 2019.  ISSN 2196-1115.  10.1186/s40537-019-0217-0.  URL https://doi.org/10.1186/s40537-019-0217-0.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, 2019.
  * Ding et al. (2022) Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li.  Is gpt-3 a good data annotator?  _arXiv preprint arXiv:2212.10450_ , 2022.
  * Driess et al. (2023) Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  Palm-e: An embodied multimodal language model, 2023.
  * Feder et al. (2020) Amir Feder, Danny Vainstein, Roni Rosenfeld, Tzvika Hartman, Avinatan Hassidim, and Yossi Matias.  Active deep learning to detect demographic traits in free-form clinical notes.  _Journal of Biomedical Informatics_ , 107:103436, 2020.  ISSN 1532-0464.  https://doi.org/10.1016/j.jbi.2020.103436.  URL https://www.sciencedirect.com/science/article/pii/S1532046420300642.
  * Feder et al. (2022) Amir Feder, Itay Laish, Shashank Agarwal, Uri Lerner, Avel Atias, Cathy Cheung, Peter Clardy, Alon Peled-Cohen, Rachana Fellinger, Hengrui Liu, Lan Huong Nguyen, Birju Patel, Natan Potikha, Amir Taubenfeld, Liwen Xu, Seung Doo Yang, Ayelet Benjamini, and Avinatan Hassidim.  Building a clinically-focused problem list from medical notes.  In _Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)_ , pages 60–68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics.  10.18653/v1/2022.louhi-1.8.  URL https://aclanthology.org/2022.louhi-1.8.
  * Ford et al. (2016) Elizabeth Ford, John A Carroll, Helen E Smith, Donia Scott, and Jackie A Cassell.  Extracting information from the text of electronic medical records to improve case detection: a systematic review.  _Journal of the American Medical Informatics Association_ , 23(5):1007–1015, 2016.
  * Goel et al. (2023) Akshay Goel, Almog Gueta, Omry Gilon, Sofia Erell, and Amir Feder.  Medication extraction labels for mimic-iv-note clinical database.  PhysioNet, 2023.  URL https://doi.org/10.13026/6d3r-w470.
  * Goldberger et al. (2000 (June 13) A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley.  PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals.  _Circulation_ , 101(23):e215–e220, 2000 (June 13).  Circulation Electronic Pages: http://circ.ahajournals.org/content/101/23/e215.ful PMID:1085218; doi: 10.1161/01.CIR.101.23.e215.
  * Hakala and Pyysalo (2019) Kai Hakala and Sampo Pyysalo.  Biomedical named entity recognition with multilingual BERT.  In _Proceedings of the 5th Workshop on BioNLP Open Shared Tasks_ , pages 56–61, Hong Kong, China, November 2019. Association for Computational Linguistics.  10.18653/v1/D19-5709.  URL https://aclanthology.org/D19-5709.
  * Jensen et al. (2012) Peter B Jensen, Lars J Jensen, and Søren Brunak.  Mining electronic health records: towards better research applications and clinical care.  _Nature Reviews Genetics_ , 13(6):395–405, 2012\.
  * Johnson et al. (2023) Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark.  Mimic-iv-note: Deidentified free-text clinical notes, 2023.
  * Kholghi et al. (2015) Mahnoosh Kholghi, Laurianne Sitbon, Guido Zuccon, and Anthony Nguyen.  Active learning: a step towards automating medical concept extraction.  _Journal of the American Medical Informatics Association_ , 23(2):289–296, 08 2015.  ISSN 1067-5027.  10.1093/jamia/ocv069.  URL https://doi.org/10.1093/jamia/ocv069.
  * Landolsi et al. (2022) Mohamed Yassine Landolsi, Lobna Hlaoua, and Lotfi Romdhane.  Information extraction from electronic medical documents: state of the art and future research directions.  _Knowledge and Information Systems_ , 65, 11 2022.  10.1007/s10115-022-01779-1.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240, 2020.
  * Liu et al. (2022) Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi.  Wanli: Worker and ai collaboration for natural language inference dataset creation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6826–6847, 2022.
  * Liu et al. (2023) Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel.  Large language models are few-shot health learners.  _arXiv preprint arXiv:2305.15525_ , 2023.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu.  Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets.  _arXiv preprint arXiv:1906.05474_ , 2019.
  * Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus.  Text chunking using transformation-based learning.  In _Natural language processing using very large corpora_ , pages 157–176. Springer, 1999.
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell.  Prompt programming for large language models: Beyond the few-shot paradigm, 2021.
  * Savova et al. (2010) Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute.  Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications.  _Journal of the American Medical Informatics Association_ , 17(5):507–513, 2010.
  * Si et al. (2019) Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts.  Enhancing clinical concept extraction with contextual embeddings.  _Journal of the American Medical Informatics Association_ , 26(11):1297–1304, 2019.
  * Singhal et al. (2022) Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan.  Large language models encode clinical knowledge, 2022.
  * Uzuner (2009) Özlem Uzuner.  Recognizing obesity and comorbidities in sparse data.  _Journal of the American Medical Informatics Association_ , 16(4):561–570, 2009.
  * Uzuner et al. (2010) Özlem Uzuner, Imre Solti, and Eithon Cadag.  Extracting medication information from clinical text.  _Journal of the American Medical Informatics Association_ , 17(5):514–518, 2010.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wei et al. (2019) Qiang Wei, Yukun Chen, Mandana Salimi, Joshua C Denny, Qiaozhu Mei, Thomas A Lasko, Qingxia Chen, Stephen Wu, Amy Franklin, Trevor Cohen, and Hua Xu.  Cost-aware active learning for named entity recognition in clinical text.  _Journal of the American Medical Informatics Association_ , 26(11):1314–1322, 07 2019.  ISSN 1527-974X.  10.1093/jamia/ocz102.  URL https://doi.org/10.1093/jamia/ocz102.
  * Yadav and Bethard (2019) Vikas Yadav and Steven Bethard.  A survey on recent advances in named entity recognition from deep learning models.  _arXiv preprint arXiv:1910.11470_ , 2019.
  * Yeh (2000) Alexander Yeh.  More accurate tests for the statistical significance of result differences.  _arXiv preprint cs/0008005_ , 2000.
  * Zhang et al. (2022) Fan Zhang, Itay Laish, Ayelet Benjamini, and Amir Feder.  Section classification in clinical notes with multi-task transformers.  In _Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)_ , pages 54–59, 2022.
  * Zhou et al. (2023a) Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon.  Universalner: Targeted distillation from large language models for open named entity recognition, 2023a.
  * Zhou et al. (2023b) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.  Large language models are human-level prompt engineers, 2023b.
  * Zhu et al. (2018) Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi.  Clinical concept extraction with contextual word embedding.  _arXiv preprint arXiv:1810.10566_ , 2018.

##  Appendix A LLM Labeling Pipeline Implementation Details

Here we describe implementation details for our LLM pipepline, including our
resolver module, configurations and hyper-parameters and error identification
and improvement.

###  A.1 Resolver Module: Parsing Generative Outputs from Unconstrained LLMs

Parsing generative outputs from an unconstrained LLM can present various edge
cases that are specific to the annotation schema. In the case of the IOB-Token
schema, the outputs from the LLMs were formatted so that each token was
separated by a new line. As such, the outputs were processed one line at a
time. The individual components of each token were then parsed using a comma-
delimited structure and regular expressions. Occasionally, the LLM produced
outputs that did not conform to the expected structure, such as an extra comma
or a missing new line. When the output structure could not be resolved, it was
discarded and logged for further analysis. The ‘Group Tag’ information is
employed to group fields into medication entries.

Similarly, for the Direct-Chunk implementation, the LLM output is produced in
YAML format and delimited by backticks (‘‘‘). This output is isolated and then
sent to the standard Python YAML parsing library to generate a dictionary
representation of the output. If an edge case leads to a YAML parsing error,
the output is logged for further analysis. Once the output has been parsed
into a dictionary, each entity is mapped back to the original text. Since the
fields for a specific medication are all contained within a single YAML
output, individual grouping of fields is unnecessary.

During the phase where outputs are matched to the original text, additional
considerations come into play, such as applying a “fuzzy match.” This allows
for some minor differences in characters and helps to improve entity recall
(i.e., matching a correctly spelled medication generated by the LLM with a
misspelled medication in the raw text). When the outputs could not be resolved
to fit the target structure or mapped back to the original text, they were
logged for analysis but ultimately not used as labels. This can contribute to
potential recall errors.

###  A.2 Error Identification and Prompt Improvement

Our experiments with the LLM pipeline revealed various types of errors, which
have been categorized and addressed to improve in-context learning. Errors
spanned from missing annotations to spurious annotations and field-type
confusions.

Missing Annotations: The model had difficulty in identifying less frequent or
differently formatted fields. We incorporated additional examples to address
these omissions, though it led to a trade-off in overall precision.

Spurious Annotations: The model sometimes incorrectly annotated entities that
did not belong to medication information. To mitigate this, we added negative
examples in the prompt to refine its identification capabilities.

Confusion between Fields: Occasionally, the model confused similar field
types. This was particularly evident in distinguishing between ‘mode’ and
‘frequency’. Adjustments were made in the prompt to improve the model’s
accuracy in this aspect.

Nuanced Errors: Many errors exhibited a reasonable basis, pointing to the
complexities in the common definitions used in the medical domain. For
example, the model classifying ‘oxygen’ as a medication could be attributed to
the overlap between the common definitions of chemical medications and
therapeutic treatments like oxygen therapy. Examples of these nuanced errors
were also incorporated into the prompt.

###  A.3 Configurations and Hyper-Parameters

In addition to experimenting with different prompts, we also examined various
hyper-parameters, including chunk size, maximum decoding steps, and
temperature. We tested multiple values for each parameter and selected those
that performed best. It is worth noting that we conducted a partial grid
search and drew our conclusions based on these results. To determine the best
configuration we used a balanced F2 score defined as
$F_{2}=\frac{5}{{\frac{4}{\text{{recall}}}+\frac{1}{\text{{precision}}}}}$.
This provided a higher emphasis on recall, which we believed would be more
important for the annotation task. For all best-performing models, 1024
maximum decoding steps yielded the best performance. The optimal chunk size
varied between YAML-based and IOB-based models, with chunks of 180 and 250
characters performing best, respectively. Different optimal parameters between
the two schema were expected given their different approaches. The YAML-based
models directly produce labels and groups for the input chunk which may
benefit from shorter chunks, whereas IOB-based models have an intermediate
step of outputting per token before grouping them. For all selected models, a
temperature of 0.0 proved optimal, supporting that a deterministic setting is
optimal for an information extraction task.

Type | Chunk Size | F1 Score | F2 Score | Recall | Precision
---|---|---|---|---|---
IOB Token | 250 | 82.0 | 84.0 | 85.5 | 78.7
Direct Chunk | 180 | 80.9 | 80.8 | 80.7 | 81.0
Ensemble | - | 81.8 | 84.9 | 87.1 | 77.1
Table 5: Performance of top experiments on development set for both prompt
schemas. F2 score was used to select top experiments. The ensemble version,
highlighted in gray, was the final version used for LLM base annotations on
the test set.

###  A.4 Pseudocode for the Ensemble Procedure to Combine Annotations

Figure 9 presents pseudocode for the ensemble procedure that combines
generated annotations from both prompt schemas. As described in Section 5.3,
the ensemble achieved the highest F2 score and was therefore chosen for the
LLM Base Annotations on the ‘test set’.

Figure 9: Pseudocode for the ensemble procedure.

###  A.5 Choosing the Final Model for LLM Base Annotations

Table 5 displays the performance of the best-performing models on the
development set. We observe that the ensemble of the two top models achieves
the best F2 score, indicating the best balance of precision and recall for our
objective. Consequently, we selected the ensemble model for the LLM Base
Annotations.

##  Appendix B Inference Time of LLM Annotations

As the inference process can be scaled across multiple documents
simultaneously with appropriate infrastructure, a direct comparison of LLM
inference time to human annotation time is not applicable. Nonetheless,
performing inference using PaLM 2 on a TPU 8x8 pod, applied individually to
each document in a random subset comprising 15% of the documents, revealed an
average inference time of 1.5 minutes per 1,000 characters, with a standard
deviation of 0.77 minutes. This results in an average of 9.49 minutes per
document, with a standard deviation of 5.55 minutes.

##  Appendix C Pseudocode for Vertical and Horizontal Metrics

Figures 10 and 11 present pseudocode for the Vertical metrics and Horizontal
metrics, respectively. Both are based on the metrics defined by (Uzuner et
al., 2010). As described in Section 6.2, _Vertical metrics_ evaluate NER with
respect to fields, while _horizontal metrics_ also evaluate RE between fields
and the medication entry.

Figure 10: Pseudocode for Vertical metrics. Figure 11: Pseudocode for
Horizontal metrics.

##  Appendix D More Details on Statistical Analysis

Here we provide more details on the statistical analysis conducted in Section
6.

###  D.1 Statistical Significance Test

To compare the performance metrics (precision, recall and F1 score) of various
labeling methods, we use the approximated randomization test in (Yeh, 2000),
which is more powerful and precise than the method implemented in i2b2. For
two different labeling methods $A$ and $B$, denote their output sets as
$O_{A}$ and $O_{B}$ and the performance metrics as $f_{A}$ and $f_{B}$. Then
$\Delta=f_{A}-f_{B}$ is the observed performance difference, $Q=O_{A}\cap
O_{B}$ is the set of common outputs shared by both methods, and
$P_{A}=O_{A}\setminus O_{B}$ and $P_{B}=O_{B}\setminus O_{A}$ are the sets of
unique outputs only generated from either method $A$ or method $B$. When the
two methods are similar (null hypothesis), the outputs in $P_{A}$ and $P_{B}$
should be exchangeable. The implementation details of our approximated
randomization test are as follows:

  1. 1.

Gather unique outputs generated from either $A$ or $B$, i.e., $P=P_{A}\cup
P_{B}$.

  2. 2.

Randomly assign each output in $P$ to new sets $P^{\prime}_{A}$ or
$P^{\prime}_{B}$ with equal probability, and create the set of psedo-outputs
for $A$ and $B$

$O^{\prime}_{A}=P^{\prime}_{A}\cup Q,\>O^{\prime}_{B}=P^{\prime}_{B}\cup Q.$

  3. 3.

Compute performance metrics on $O^{\prime}_{A}$ and $O^{\prime}_{B}$, and
calculate their difference $\Delta^{\prime}$.

  4. 4.

Repeat Steps 2-3 for $n=1,000$ times, and generate a sampling distribution of
$\Delta^{\prime}$.

If the observed metric difference $\Delta$ is above the $95\%$ percentile of
the sampling distribution, we can claim that method $A$ performs significantly
better than $B$ with metric difference $\Delta$.

###  D.2 Time cost and types of refinement

We conducted further analysis to investigate what type of errors in pre-labels
/ what type of corrections has greatest impact on time cost. Focusing on text
spans only, there are three types of corrections a human refiner can do:
addition (adding a missing text span), modification (modify an existing text
span) and deletion (delete a spurious text span). First, we look into the
distribution of these three types of corrections for refinement based on human
pre-labels and refinement based on LLM pre-labels Figures 12, 13, and 14.
Refinement on LLM pre-labels usually requires significant more deletions and
modifications, while the human + refinement method involves more additions
(all p-values $<0.5$). By fitting a regression model on time cost using the
number of corrections of each type as predictors, the amount of time needed
for adding a missing text span and modify an existing text span are similar,
while it is much faster to delete a text span; see regression results in Table
6.

Figure 12: Distribution of additions in the refinement process, LLM vs. rater base generated labels. Figure 13: Distribution of modifications in the refinement process, LLM vs. rater base generated labels. Figure 14: Distribution of deletions in the refinement process, LLM vs. rater base generated labels. Variable | Coefficient | Std. Error | z score
---|---|---|---
Intercept | 4.327** | 0.735 | 5.888
# Modified X | 0.351** | 0.076 | 4.618
# Added | 0.288** | 0.031 | 9.158
# Deleted | -0.013 | 0.235 | -0.056
Table 6: Linear mixed effect model regression results: time cost vs. type of corrections. ** denotes $p<0.01$. Method | Medication | Dose | Frequency | Mode | Duration
---|---|---|---|---|---
Base LLM | 0.784 | 0.883 | 0.787 | 0.848 | 0.420
Base Rater | 0.776 | 0.903 | 0.883 | 0.907 | 0.384
Refined Rater | 0.908 | 0.927 | 0.914 | 0.952 | 0.466
Refined LLM | 0.907 | 0.941 | 0.890 | 0.947 | 0.540
Table 7: F1 scores for different types of labels across each medication attribute. The attributes include medication name, dosage, frequency, mode, and duration. Label Type | Vertical | Horizontal
---|---|---
| Recall | Precision | F1 | Recall | Precision | F1
Base Rater | 0.820429 | 0.924383 | 0.869309 | 0.723582 | 0.756489 | 0.739669
Base LLM | 0.874945 | 0.769727 | 0.818970 | 0.800494 | 0.678708 | 0.734587
Refined Rater | 0.922830 | 0.925182 | 0.924005 | 0.873006 | 0.869832 | 0.871416
Refined LLM | 0.939259 | 0.900770 | 0.919612 | 0.886027 | 0.843396 | 0.864186
Table 8: _Token-level_ labeling Quality and Time Metrics for the Entire Test
Set.

##  Appendix E Additional Results

For brevity, we provide here additional results that support the main findings
in Section 6.

###  E.1 Supplemental Results showing performance across each field type

In Table 7, we compare all of the tested approaches in terms of $F1$ scores
for each entity type.

###  E.2 Supplemental Results showing token-level metrics

Table 8 presents token-level labeling quality in terms of horizontal and and
vertical metrics on the test set. Results are similar to the ones of phrase-
level metrics.

###  E.3 Time Cost Evaluations

In Figure 15 and Figure 16, we show the time cost for individual raters and
based on their level of expertise, respectively.

Figure 15: Base labeling time cost of individual raters. Measurements are in
minutes per note. Figure 16: Base labeling time cost: expert raters vs non-
expert raters. Measurements are in minutes per note.

##  Appendix F  Example Prompts

Note: The following examples are synthetic and inspired by, but not directly
taken from, the _i2b2_ dataset. Only a single question-answer example is
provided in each prompt for the sake of brevity.

###  F.1 IOB-Token Prompt Example

⬇

description: "You are a helpful medical knowledge assistant for Beginning-
Inside-Outside (BIO) tagging of medications’ information. Your task is to
create Medical Named Entity Recognition (NER) annotations using the Beginning-
Inside-Outside (BIO) format. Label the given sentence (question) using BIO
tags from these classes ONLY: MEDICATION (medication name), DOSE, FREQUENCY,
DURATION, REASON, MODE. For tokens not belonging to these classes, tag as
O-tag. For each token you MUST start a new line. Punctuations should be
skipped in tokens. Each line must include: (1) word or token, (2) BIO tag, and
(3) entity number for B-tags and I-tags ONLY. Use ’<None>’ for O-tags.
Maintain the correct format. Example: ’token’, BIO-tag, ’entity_1’."

examples: {

question: "’CURRENT MEDICATIONS: Ibuprofen as needed, and diclofenac for one
month as needed, for abdominal discomfort.’"

answer:

"’CURRENT’, O, ’<None>’\n"

"’MEDICATIONS’, O, ’<None>’\n"

"’Ibuprofen’, B-MEDICATION, ’entity_1’\n"

"’as’, B-FREQUENCY, ’entity_1’\n"

"’needed’, I-FREQUENCY, ’entity_1’\n"

"’and’, O, ’<None>’\n"

"’diclofenac’, B-MEDICATION, ’entity_2’\n"

"’for’, B-DURATION, ’entity_2’\n"

"’one’, I-DURATION, ’entity_2’\n"

"’month’, I-DURATION, ’entity_2’\n"

"’as’, I-DURATION, ’entity_2’\n"

"’needed’, I-DURATION, ’entity_2’\n"

"’for’, O, ’<None>’\n"

"’abdominal’, B-REASON, ’entity_1|entity_2’\n"

"’discomfort’, I-REASON, ’entity_1|entity_2’\n"

}

###  F.2 Direct-Chunk Prompt Example

⬇

description:

"You are a medical assistant with expertise in document processing.\n"

"\n"

"- Your task is to tag groups of entities related to medications. (Some groups
may have overlap as two medications can be used for the same reason.)\n"

"- Each medication is a SPECIFIC reference to the drug’s name used by the
patient. This should not contain other data already within a group.\n"

"- The mode of administration, such as ’oral’ or ’intravenous’, should be
tagged SEPARATELY from the medication’s name. \n"

"- Text should contain the EXACT characters in the raw input, including
punctuation, and typos.\n"

"- Use YAML as the output format."

examples: {

question: "’CURRENT MEDICATIONS: Ibuprofen as needed, and diclofenac for one
month as needed, for abdominal discomfort.’"

answer:

"‘‘‘yaml\n"

"entities:\n"

" - group: 1\n"

" MEDICATION: \n"

" text: Ibuprofen\n"

" start_pos: 20\n"

" end_pos: 29\n"

" DOSE: \n"

" text: ’’\n"

" start_pos: \n"

" end_pos: \n"

" FREQUENCY: \n"

" text: as needed\n"

" start_pos: 30\n"

" end_pos: 39\n"

" DURATION: \n"

" text: ’’\n"

" start_pos: \n"

" end_pos: \n"

" REASON: \n"

" text: abdominal discomfort\n"

" start_pos: 81\n"

" end_pos: 100\n"

" MODE: \n"

" text: ’’\n"

" start_pos: \n"

" end_pos: \n"

" - group: 2\n"

" MEDICATION: \n"

" text: diclofenac\n"

" start_pos: 45\n"

" end_pos: 55\n"

" DOSE: \n"

" text: ’’\n"

" start_pos: \n"

" end_pos: \n"

" FREQUENCY: \n"

" text: as needed\n"

" start_pos: 68\n"

" end_pos: 77\n"

" DURATION: \n"

" text: for one month\n"

" start_pos: 56\n"

" end_pos: 68\n"

" REASON: \n"

" text: abdominal discomfort\n"

" start_pos: 81\n"

" end_pos: 100\n"

" MODE: \n"

" text: ’’\n"

" start_pos: \n"

" end_pos:\n"

"‘‘‘\n"

}


# RawArray: A Simple, Fast, and Extensible Archival Format for Numeric Data

David S<EMAIL_ADDRESS>GitHub: https://github.com/davidssmith
222This work was funded by NCI/NIH K25 CA176219.

(Institute of Imaging Science, Vanderbilt University Medical Center,
Nashville, TN, USA)

###### Abstract

Raw data sizes are growing and proliferating in scientific research, driven by
the success of data-hungry computational methods, such as machine learning.
The preponderance of proprietary and shoehorned data formats make computations
slower and make it harder to reproduce research and to port methods to new
platforms. Here we present the RawArray format: a simple, fast, and extensible
format for archival storage of multidimensional numeric arrays on disk.

The RawArray file format is a simple concatenation of a header array and a
data array. The header comprises seven or more 64-bit unsigned integers. The
array data can be anything. Arbitrary user metadata can be appended to an
RawArray file if desired, for example to store measurement details, color
palettes, or geolocation data.

We present benchmarks showing a factor of 2–3$\times$ speedup over HDF5 for a
range of array sizes and a speedup of up to 20$\times$ in reading the common
deep learning datasets MNIST and CIFAR10.

##  1 Introduction

Our requirements for data storage were that the format should be S.A.F.E.:
Simple, Archival, Fast, and Extensible.

  * •

_Simple_ : easily implemented, easily audited for bugs, and mirror the data
layout in memory for memory mapping and security;

  * •

_Archival_ : contain all of the information needed to read the data contained
in the file with minimal external libraries or code available; subject to
introspection through widely available command-line tools;

  * •

_Fast_ : to minimize input and output latency and speed up codes that do heavy
input/output and minimize the carbon footprint of the increasing amount of
data analysis being done in the world; and

  * •

_Extensible_ : easily improved without a committee and without breaking
backward compatibility.

None of the existing scientific data formats met these requirements.

Hierarchical Data Format v5 (HDF5333https://www.hdfgroup.org/HDF5/) is
burdensome for the casual user to install and to manipulate, raising the
barrier to reproducible research. HDF5 also doesn’t natively support complex
floats, which is mind boggling for a scientific data format. Julia’s JLD2
format is also based on HDF5, unfortunately.

Matlab’s MAT format is widespread and relatively easy to manipulate, but the
older version 5 doesn’t support arrays over 2 GB, and later versions are based
on HDF5.

The Nearly Raw Raster Data (NRRD)
format444http://teem.sourceforge.net/nrrd/format.html is closest to meeting
our requirements, but has some problems. First, it has a flawed type
specification. By not separating the numeric bits type (int, float, etc.) from
the width of the storage (bits per datum), it is less future proof. For
example, the newer float16 formats popular in deep learning or very large
(512-bit) wide AVX data types are already supported out of the box in RawArray
without additional programming. If completely new types come along, we can add
additional type codes to RawArray without breaking backward compatibility.
Second, NRRD is overly complicated because it implements both raw data and
text formats, as well as internal compression, which we prefer to handle
externally as part of archives. Nevertheless, NRRD is a strong competitor and
may be preferred for some applications.

Numpy’s NPY format is quite fast, but not so simple and is not widely
implemented in other languages, making it less portable. But it has, depending
on your perspective, an advantage over our format in that it can “pickle”
derived types and store them automatically. With RawArray, the user is
responsible for writing an array of derived types (structs/objects)
themselves. Since that is usually as simple as passing a pointer to the array
of objects and the total size of the array, we don’t consider this to be a
significant issue.

Our vision for archiving raw data is metadata as human-readable markup (e.g.
YAML, JSON, TOML, INI), raw data in RawArray files, organized by a file system
directory structure, and packaged in the user’s preferred archive file format,
such as tar, zip, etc. This combination is simple, proven, and widely
available with battle-tested tools on every platform.

Coincidentally, Microsoft appears to agree: the latest Office document formats
(.docx, .xlsx, .pptx) are actually zip archives of a directory structure
containing embedded resources and XML files that describe the document.

##  2 Format

Table 1: RawArray File Structure. Note that u64 $\equiv$ 64-bit unsigned
integer, u8 $\equiv$ 8-bit unsigned integer

Offset (bytes) | Type | Length | Description
---|---|---|---
0 | u64 | 8 | magic number
8 | u64 | 8 | flags
16 | u64 | 8 | element type code
24 | u64 | 8 | element size (bytes)
32 | u64 | 8 | data length (bytes)
40 | u64 | 8 | number of dimensions
48 | u64[] | 8$\times$ndims | dimension values
⋮ | ⋮ | ⋮ | ⋮
48 $+$ $8\times$ndims | u8[] | data_length | array data
⋮ | ⋮ | ⋮ | ⋮
48 $+$ 8$\times$ndims | u8[] |  | optional metadata
$+$ data_length |  |  |

The first part of the header is a magic number that identifies the file as
being a RawArray file. The magic number for RawArray was chosen as the ASCII
representation of the character string “rawarray” (796172726177617216). This
also happens to be eight bytes, so that the entire header can cleanly consist
of 64-bit unsigned integers.

The second header field is a bit field that specifies options. This is where
byte order and optional or future processing differences can be specified,
such as compression or encryption.

Table 2: Currently supported elemental type codes

Type Code | Description
---|---
0 | user-defined struct
1 | signed integer
2 | unsigned integer
3 | floating point (IEEE-754)
4 | complex float (float tuples)
5+ | reserved for future use

The third and fourth fields specify the elemental data type, specifically the
representation code and the number of bytes per datum. Elemental sizes are
arbitrary, so quad (128-bit) floats are defined by elemental type 3 and
elemental size 16, while the new half (16-bit) floats (useful for deep
learning) would be type 3, size 2.

The fifth field is the total size in bytes of the data segment. This is
technically redundant, since it could be computed from the elemental size and
the array dimensions, but providing it allows a sanity check and simplifies
reading and memory mapping at the cost of just 8 bytes of additional file
size.

The sixth and seventh fields are the number of dimensions of the array (e.g. 2
for an image) and a vector that describes the shape of the array. The length
of the shape vector should match the number of dimensions specified
immediately prior.

Finally, the array data is stored in binary format. Row and column major do
not matter, because that is a detail of the implementation language. As long
as the data get read and written in the correct order, everything is portable.

All data is up front in the file in a linear layout immediately after the
purely numerical header to facilitate immediate memory mapping with minimal
file parsing. This confers a speed advantage on operating systems that support
it. Note that this does not preclude the use of stream compression, such as
LZ4, because that can be used to pre-compress the data before storage and
uncompress the data when reading. (Although user compression will obfuscate
the data within the file and prevent command line tools from easily
manipulating the file, thus reducing the “archival” quality of the data.)

Checksumming was deliberately omitted because it is difficult to checksum a
file containing its checksum. Some formats, such as tar, zero out the checksum
field and then checksum the rest of the file, but this requires special
software that understands the format, so standard command-line checksum tools
won’t work. Finally, checksum algorithms come and go, and ideally scientific
data formats would be future proof, without having to modify the archive
itself, so we would prefer checksums be computed using the tools of the day
and stored external to the data file. To compute an external checksum an
RawArray file, simple run your local checksum command. For example, on Linux:

    $ md5sum examples/test.ra
    1dd9f98a0d57ec3c4d8ad50343bd20cd  examples/test.ra

Time stamping was also omitted because almost all file systems already provide
it. Adding a time stamp that changes upon rewrite or access also foils
checksums. HDF5 files are difficult to properly checksum for this reason. Two
RawArray files are defined as identical if and only if they contain identical
contents, no matter when they were created or accessed last.

##  3 Example Usage

###  3.1 Applications

Using RawArray files in C is extremely simple from the user’s perspective. If
given a RawArray file called mydata.ra, the C code to read the file into a
RawArray struct, modify it, and write it back out would be simply

    ra_t mydata;
    ra_read(&mydata, "mydata.ra");
    ... do some stuff ...
    ra_write(&mydata, "mydata.ra");

The process for the Python, Julia, Matlab, and Rust implementations are
similar In Python, Julia, and Matlab, the RawArray object doesn’t even need to
be wrapped in a derived type because the arrays in the language contain
sufficient information to reconstruct it when writing the file. For example,
to read an image in Python that has been stored as a RawArray (something will
we demonstrate in the benchmarks below), double the first pixel, then write it
back out, you’d simply write

    import ra
    img = ra.read(’airplane.ra’)
    img[0, 0] *= 2
    ra.write(img, ’airplane.ra’)

In some languages, such as C and Rust, run time type polymorphism is strongly
discouraged or even impossible. In those languages, the complied code must
know what data type you are trying to import before reading the RawArray file.
For example, in Rust, writing a float 32 array and then reading it back would
look like

    use rawarray::RawArray;
    use std::io;
    fn main() -> io::Result<()> {
        let vec1: Vec<f32> = vec![1.0, 2.0, 3.0, 4.0];
        let ra: RawArray<f32> = vec1.clone().into();
        ra.write("myarray.ra")?;

        let vec2: Vec<f32> = RawArray::<f32>::read("myarray.ra")?.into();
        assert_eq!(vec1, vec2);
        Ok(())
    }

Notice the type parameters on the RawArray struct that tells the compiler how
to monomorphize the methods. This is not ideal, but in practice doesn’t seem
to present a problem and allows for better optimization.

###  3.2 Command Line File Introspection

To get a better handle on the format of an RawArray file, let’s look inside
one. If you are on a Unix system or have Cygwin installed on Windows, you can
examine the contents of an RawArray file using command line tools. For this
section, we will use the test.ra file provided in the data/ subdirectory of
the main repository and the ubiquitous octal dump command, od.

First, let’s pretend you don’t know the dimensionality of the array. Then

    > od -t uL -N 48 test.ra
    0000000              8746397786917265778              0
    0000020              4                                8
    0000040              96                               2
    0000060

shows the dimension (2) as the second number on the third line. The command is
extracting the first 48 bytes and formatting them as 64-bit unsigned integers.
The ridiculous number listed first is the magic number indicating that this is
an RawArray file. A slightly different command illuminates that:

    > od -a -N 16 test.ra
    0000000    r   a   w   a   r   r   a   y nul nul nul nul nul nul nul nul
    0000020

One can already see the shell scripting potential of this format. Armed with
the knowledge that the array is 2D, we know that the header must be 48 + 2*8 =
64 bytes long. The command to skip the header and view only the data would be:

    > od -j 64 -f test.ra
    0000100     0.000000e+00            -inf    1.000000e+00   -1.000000e+00
    0000120     2.000000e+00   -5.000000e-01    3.000000e+00   -3.333333e-01
    0000140     4.000000e+00   -2.500000e-01    5.000000e+00   -2.000000e-01
    0000160     6.000000e+00   -1.666667e-01    7.000000e+00   -1.428571e-01
    0000200     8.000000e+00   -1.250000e-01    9.000000e+00   -1.111111e-01
    0000220     1.000000e+01   -1.000000e-01    1.100000e+01   -9.090909e-02
    0000240

Here we are using -j to skip the first 64 bytes and -f to format the byte data
as single-precision floats. Note od doesn’t understand complex numbers, but
the complex data is stored as real and imaginary float pairs that are
contiguous on disk. This means that each line of the output is showing two
complex numbers with columns 1 and 3 the real parts and columns 2 and 4 the
imaginary parts. Notice that it correctly renders the negative infinity.

These examples show how easy it is to manipulate RawArray files on the command
line, which improves the accessibility and archival potential of the format.

##  4 Benchmarks

To demonstrate the efficiency gains of RawArray, we ran a series of benchmarks
typical of common scientific computing workloads. All benchmarks were run on a
Linux server with a RAID5 array of Seagate IronWolf Pro 7200 RPM SATA drives
with 256 MB cache and 6 Gb/s transfer speeds (Model ST10000VN0004-1Z).

The first benchmark compared RawArray to HDF5 in reading and writing arrays of
a range of strides. The results are shown in Figs. 1 and 2. First test was the
time to write 100,000 tiny vectors of length 10, the second test was reading
and writing 10,000 small images of size 10 $\times$ 10, and the third test was
reading and writing a single large matrix of dimension 10 $\times$ 100,000.
The total data size of all tests were kept the same to tease out the overhead
of individual operations. Across all tests, RawArray was consistently
2–3$\times$ faster than HDF5.

Figure 1: Time to write and then read back one million 32-bit floats in a
range of configurations. Vectors are length 10, images are $10\times 10$, and
the matrix is $10\times 100,000$. The proposed format was $\sim 2$–$3\times$
faster than HDF5. The total data size read and written was the same; only the
number of I/O calls changed. Note the log scale. Figure 2: Transfer speed to
write and then read back one million 32-bit floats in a range of
configurations. Vectors are length 10, images are $10\times 10$, and the
matrix is $10\times 100,000$. The proposed format was $\sim 2$–$3\times$
faster than HDF5. Note the log scale.

The second benchmark compared reading images as RawArrays or as the common
image format PNG. In applications with large image datasets, such as deep
learning, the images are often stored in common image formats, such as PNG.
Furthermore, instead of storing the data in one large archive, it is common
for the data to be split among thousands or even millions of small files.

Here we compare the performance of RawArray and PNG in reading and writing two
of the most common deep learning data sets:
MNIST555http://yann.lecun.com/exdb/mnist/ and CIFAR10.666A. Krizhevsky and G.
Hinton, Learning multiple layers of features from tiny images, tech. rep.,
Citeseer, 2009 MNIST comprises 60,000 28$\times$28 pixel grayscale images,
while CIFAR10 comprises 60,000 36$\times$36 pixel 3-channel color images. Fig.
3 shows the striking results of using RawArray over PNG. Even though the PNG
data is compressed, while the RawArray is uncompressed, so that more data must
be read from disk in the RawArray case, our proposed format is 600% faster
reading MNIST and 1800% faster reading CIFAR. The practical implications are
that data loading times during neural network training would be dramatically
reduced if the files were read from RawArray files.

Figure 3: Time to read 50,000 small images, typical of the MNIST and CIFAR
datasets. The MNIST images are 28$\times$28 single-channel, 8-bit images,
while the CIFAR images are $36\times 36$ 8-bit RGB images. Reading the images
in RawArray format was 600% faster for MNIST and 1800% faster in the CIFAR
dataset.

##  5 Discussion and Conclusions

RawArray is designed to be simple, fast, and highly extensible, while
maintaining good qualities for an archival data format.

An example of extensibility is compression or encryption. If at some point in
the future, it is decided to add either of these, that can easily be
implemented via a new header flag to maintain backward compatibility. Older
file versions with the flag naturally set to false will not be affected. Up to
64 different flags could be defined without altering the header structure.

RawArray could also serve as a file container by using composite types. We
think containing files is better left to the file system, however, which is
already designed to hierarchically organize heterogeneous objects. But—in
theory—you could store RawArray files in a RawArray file, so the format is
fractal in that sense. This would be implemented as a user-defined type, where
the type is a struct that matches the layout of RawArray file.

RawArray currently handles file sizes up to 18 exabytes ($2^{64}$), but the
implementation could be adapted in the future to go beyond that without
modification to the format and still retain backward compatibility by
exploiting the variable-length list of dimensions.

Reference implementations exist currently for C, Python, and
Matlab777https://github.com/davidssmith/ra, as well as
Julia888https://github.com/davidssmith/RawArray.jl and
Rust999https://github.com/davidssmith/rawarray-rust. Contributors and beta
testers are welcome, as are helpful suggestions.