# Assessing the Efficacy of Grammar Error Correction: A Human Evaluation
Approach in the Japanese Context

###### Abstract

In this study, we evaluated the performance of the state-of-the-art sequence
tagging grammar error detection and correction model (SeqTagger) using
Japanese university students’ writing samples. With an automatic annotation
toolkit, ERRANT, we first evaluated SeqTagger’s performance on error
correction with human expert correction as the benchmark. Then a human-
annotated approach was adopted to evaluate Seqtagger’s performance in error
detection using a subset of the writing dataset. Results indicated a precision
of 63.66% and a recall of 20.19% for error correction in the full dataset. For
the subset, after manual exclusion of irrelevant errors such as semantic and
mechanical ones, the model shows an adjusted precision of 97.98% and an
adjusted recall of 42.98% for error detection, indicating the model’s high
accuracy but also its conservativeness. Thematic analysis on errors undetected
by the model revealed that determiners and articles, especially the latter,
were predominant. Specifically, in terms of context-independent errors, the
model occasionally overlooked basic ones and faced challenges with overly
erroneous or complex structures. Meanwhile, context-dependent errors, notably
those related to tense and noun number, as well as those possibly influenced
by the students’ first language (L1), remained particularly challenging.

Keywords: Grammar error correction, Grammar error detection, Japanese
university students, Human evaluation, Language learning

Assessing the Efficacy of Grammar Error Correction: A Human Evaluation
Approach in the Japanese Context

Qiao Wang1 and Zheng Yuan2
---
1 Center for English Language Education, Faculty of Science and Engineering
(CELESE),
Waseda University, Tokyo, Japan
2 Department of Informatics, King’s College London, U.K.
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Grammatical error correction (GEC) is the task of automatically detecting and
correcting grammatical errors in text, which has important pedagogical
applications that can benefit both students and teachers in classroom, as well
as online language learning. GEC has often been considered a sequence-to-
sequence translation task, where systems learn to “translate” an ungrammatical
input sentence to a grammatical output one (Yuan and Briscoe, 2016; Kiyono et
al., 2019; Yuan et al., 2019; Lichtarge et al., 2020). More recently however,
Omelianchuk et al. (2020) proposed a sequence tagging approach for GEC that
can perform both error detection and correction, and benefit from fast
inference compared to sequence-to-sequence GEC models. Following this trend,
several sequence tagging GEC systems have been developed (Mesham et al.,
2023), and state-of-the-art results on public benchmark datasets have been
reported using automatic GEC evaluation metrics like ERRor ANnotation Toolkit
(ERRANT) (Bryant et al., 2017), M2 Scorer (Dahlmeier and Ng, 2012) and GLEU
(Napoles et al., 2015). However, much less work has been done on human
evaluation of the efficacy of such GEC models in the context of language
education with a linguistic focus.

In this study, we employed the state-of-the-art
SeqTagger111https://github.com/StuartMesham/gector_experiment_public GEC model
from Mesham et al. (2023), which introduced new transformation tags to help
simplify the sequence tagging problem, as well as improve the generalisation
of the GEC system. A fully annotated learner dataset with Japanese as the
first language (L1) has been created to help evaluate the model’s efficacy. A
new human evaluation scheme has been proposed where a human-annotated approach
was adopted to highlight agreement and disagreement between the errors
detected by both human experts and GEC models. Further thematic analysis was
employed to identify errors that are still challenging for state-of-the-art
GEC models. The new dataset and human evaluation data is made publicly
available to facilitate future
research.222https://doi.org/10.6084/m9.figshare.25295743.v1

| No. of sentence pairs | TP | FP | FN | P | R | F0.5
---|---|---|---|---|---|---|---
Full dataset | 1,577 | 820 | 468 | 3,242 | 63.66 | 20.19 | 44.50
Human evaluation dataset | 300 | 242 | 5 | 321 | 97.98 | 42.98 | 78.01
Table 1: Performance metrics of SeqTagger with human corrections as the
reference

## 2\. Dataset

The Japanese university students’ writing sample dataset was sourced from 71
sophomore students majoring in science and engineering at a Japanese
university. Their English proficiency levels ranged from higher B1 to higher
B2 according to the CEFR framework.333https://www.coe.int/en/web/common-
european-framework-reference-languages These participants were distributed
across two distinct classes and were enrolled in a year-long Academic Reading
course444https://celese.jp/courses/undergraduate/ar1/ spanning two semesters.
As a course requirement, students were tasked with composing two reaction
papers each semester. In these papers, students provided summaries and
critical reflections on science news articles of their choosing. In total, 261
writing samples were gathered.

## 3\. Grammar Error Analysis

To assess the efficacy of the state-of-the-art SeqTagger GEC model, we first
engaged human experts alongside SeqTagger to detect and correct grammatical
errors in the provided writing samples. Subsequently, an automated comparative
analysis was conducted between the corrections made by human experts and those
generated by the model. In addition, it is crucial to recognize that multiple
correction methods can address a single grammar error. Therefore, we also
looked specifically at how the model agreed with human experts in error
detection through human annotation.

### 3.1. Human and model correction

Two native-speaking teachers meticulously examined each writing sample,
selecting sentences with grammar errors (“original sentences”). They were
provided with examples differentiating between grammar and semantic errors,
and were instructed to avoid choosing sentences based on semantic ambiguity.
Then they were asked to correct grammatical mistakes in the chosen sentences
with minimal syntactic alterations. Examples of revision were also given. Each
teacher was assigned half the 261 samples.

Recognizing the inherent subjectivity in grammatical judgments (Bryant et al.,
2023), we acknowledged that even experts might differ on certain grammar
interpretations. To ensure alignment, the teachers submitted the first 100
sentences for feedback from the research team before continuing. After they
finished selecting the sentences and correcting the errors, they cross-checked
each other’s work, noting disagreements or overlooked errors. They then
integrated each other’s feedback and submitted their revised selections and
corrections to the researchers. A third expert was then engaged to review and
finalize the sentences and corrections (“corrected sentences”). Ultimately,
1577 sentence pairs, consisting of original and corrected sentences, were
produced.

The automated correction phase involved inputting the same original sentences
into the SeqTagger model. This produced an equivalent number of machine-
corrected sentence pairs. Consequently, two distinct datasets emerged: the
human dataset and the machine dataset.

### 3.2. Comparative analysis scheme

For both datasets, we used ERRANT to align the original and corrected
sentences and tag the errors. Output files in M2 format were created, locating
every correction made in each sentence pair. The following is an example from
the machine dataset:

Original: It is surprising to know that the latest games have high
technologies which enable to connect games and physical activities.

Corrected: It is surprising to know that the latest games have advanced
technologies which enable us to connect games and physical activities.

Output M2:555Here, lines starting with ‘S’ represent original sentences, while
those starting with ‘A’ indicate ERRANT-identified edits. Each ‘A’ line
delineates the beginning and end token offsets of the modification, the type
of error, and the suggested correction in tokenized form. The subsequent
fields are historically retained. When no correction is made, the output is a
“noop” edit signifying the system found no necessary changes to the original
sentence (Dahlmeier and Ng, 2012).

    S It is surprising to know that the latest
    games have high technologies which enable
    to connect games and physical activities.

    A 10 11|||R:ADJ|||advanced|||REQUIRED|||-NONE-|||0
    A 14 14|||M:PRON|||us|||REQUIRED|||-NONE-|||0

To evaluate the model’s performance in error correction, with the human
dataset as the benchmark, we quantified the model’s true positives (TF), false
positives (FP), false negatives (FN), precision (P), recall (R), and F0.5
scores under each error category based on the M2 files.

The evaluation of the models’ performance in error detection required human
annotation of agreement and disagreement between human and model detection.
Anticipating an intensive workload for the in-depth qualitative analysis, a
decision was made to limit the evaluation to a subset of the complete dataset,
which included 300 sentences. Two annotators put the ERRANT data for both
subsets side by side to ascertain whether an error detected by humans was
identified or overlooked by the model, and vice versa. To offer a
comprehensive analysis, categories and subcategories were introduced.
Disagreements between human and model detection were further annotated with
comments, which were used in subsequent thematic analysis to provide insights
into the limitations of the model, as well as those of human detection.

## 4\. Results

ERRANT automatic evaluation results for the performance metrics of the model
(SeqTagger) with human corrections as the reference on the full dataset,
including true positives (TPs), false positives (FPs), false negatives (FNs),
precision (P), recall (R) and the F0.5 score are presented in Table 1. The
state-of-the-art SeqTagger GEC model achieved an F0.5 score of 44.50% on our
Japanese university students’ writing sample dataset.666We notice that the
results here are worse than those reported on public benchmarks (e.g. BEA-dev
and BEA-test (Bryant et al., 2019)) by Mesham et al. (2023), as the model has
not been trained or tuned for Japanese L1.

For the selected 300 sentence pairs for human annotation, based on our
analysis scheme, errors were deductively categorized into two main groups:
“Agreed Detections” and “Disagreed Detections”. Additionally, we introduced a
category called “Excluded Items” to account for cases that didn’t fit within
the primary two classifications. Agreed Detections encompasses errors
identified by both human annotators and the model, while Disagreed Detections
represents errors detected exclusively by either entity. We further created
categories and subcategories for a more nuanced understanding of the natures
of errors. Three subcategories were also generated under Excluded Items: no
corrections, repeated corrections and collateral corrections, and they have
been excluded from the analysis to avoid overestimation.777No corrections: 18
instances; Repeated corrections: 30 instances; Collateral corrections: 108
instances In the following, we present the results for agreed and disagreed
detections in detail.

### 4.1. Human-model agreement

Four major categories under agreed detections between human experts and the
model were generated, including context-independent grammar errors, context-
dependent grammar errors, semantic errors and mechanical errors. Table 2 shows
the results for agreed detections and the detailed explanations of the
categories and subcategories are laid out below.

Categories | Subcategories | Instances
---|---|---
Context-independent grammar errors | Same revisions | 202
| Different revisions | 39
Context-dependent grammar errors |  | 1
Subtotal of grammar errors | 242
Semantic errors |  | 1
Mechanical errors |  | 2
Total | 245

Table 2: Agreed Detections

#### Context-independent grammar errors

are those that are inherently incorrect, irrespective of their surrounding
context. This category is further divided into “same revisions” and “different
revisions”. The former refers to situations where both human experts and the
model converged on identical corrections. Meanwhile, the latter refers to
cases where the model and human experts adopted distinct corrections that are
both acceptable for an agreed detection.

#### Context-dependent grammar errors

hinge on context for accurate judgment of grammaticality. Without the specific
context, they might pass as grammatically accurate.

#### Semantic errors

target the essence or meaning of words without referencing grammatical
accuracy, e.g. both humans and the model changed “scene” to “scenery”.

#### Mechanical errors

were punctuation and spelling errors.

### 4.2. Human-model disagreement

Categories | Subcategories | Undetected Instances (S) | Undetected Instances (M)
---|---|---|---
Context-independent grammar errors |  | 196 | 3
Context-dependent grammar errors |  | 125 | 1
Subtotal of grammar errors | 321 | 4
Unnecessary corrections | Original correct | 23 | 4
Debatable corrections | 29 | 1
Subtotal of unnecessary corrections | 52 | 5
Semantic errors |  | 113 | 11
Mechanical errors |  | 45 | 6
Total | 531 | 26

Table 3:  Disagreed Detections

Annotation results of human-model disagreement resembled the categories formed
under agreement, namely: context-independent grammar errors, context-dependent
grammar errors, semantic errors, and mechanical errors. However, upon closer
inspection, we also observed instances where disagreements were caused by
unnecessary corrections made either by humans or the model. Table 3 shows the
annotation results. In the table, the column labeled “Undetected Instances
(S)” quantifies the occasions where the model failed to identify an error
detected by human experts. Conversely, the column “Undetected Instances (M)”
counts the scenarios where human experts overlooked errors that the model
successfully detected.

#### Unnecessary corrections

were further sub-divided into “original correct” and “debatable corrections”.
The “original correct” subcategory pertains to situations where an original
sentence was accurate upon closer inspection, but was corrected extraneously
by either humans or the model. The “debatable corrections” subcategory
captures errors deemed borderline: while they might violate strict grammatical
conventions, they align with evolving language norms.

### 4.3. Adjusted performance metrics on the human annotation subset

For the complete dataset, the performance metrics were calculated based on
error correction. In other words, a true positive is an error identified and
corrected exactly in the same way as the human experts have done. For the
human annotation subset, as mentioned earlier, the focus is on detection.
Thus, upon obtaining the quantitative data reflecting agreements and
disagreements in terms of detection, we adjusted the model’s metrics,
accounting for both context-dependent and context-independent grammatical
errors. Results show that the model achieved an F0.5 score of 78.01%, with
97.98% precision and 42.98% recall for error detection, strikingly different
from the metrics based on error correction (see Table 1).

## 5\. Discussions

### 5.1. Quantitative results

Based on results regarding semantic errors and unnecessary correction, it can
be seen that human experts often incorporate semantic adjustments to
complement grammatical corrections, evidenced by the significant count of
semantic corrections (n=113 out of 531). Contrarily, the model infrequently
detect these semantic nuances. In addition, human experts occasionally “over-
correct” or introduce unnecessary grammatical adjustments (n=52 out of 531).

Perhaps most importantly, the model exhibits a high precision in error
detection, as almost every error detected by the model was a true error.
However, the recall is less than 50%, caused by the model’s notable deficiency
in detecting both context-dependent and context-independent grammar errors. In
particular, context-dependent errors remain a challenging domain for model
detection, with only 1 successfully detected by the model and 125 went
unnoticed.

The impact of precision and recall varies between students and teachers. For
students, a high precision is preferred to avoid misleading them with false
detection. This indicates that the current model is a readily applicable tool
for Japanese students. However, for teachers, a higher recall is preferable,
as it reduces the time needed for error identification, allowing them to focus
more on assessing the correctness of detected errors. Therefore, our next step
will be to improve the model’s recall to better support teachers in providing
thorough feedback.

### 5.2. Thematic analysis on grammar errors undetected by the model

For grammar errors undetected by the model, including both context-dependent
and context-independent ones, we added comments to each of such cases. The
following themes emerged from the comments:

#### Determiners and Articles

The model consistently underperforms in the domain of determiners and, more
specifically, articles. A potential reason is the significant challenge that
English articles present to Japanese students Izumi et al. (2003) due to the
lack of equivalent constructs in their native language Snape et al. (2016).
This poses great challenges to the model as it was not trained specifically
for the population. Another reason may be the inherent intricacy of article
usage, with studies underscoring that even native speakers occasionally
display disagreements in this domain Chodorow et al. (2010). The difficulty in
automatically detecting article errors was also highlighted in previous
studies Nagata et al. (2005).

#### Noun Number

Errors in noun numbers often intertwine with those of articles and
determiners, as converting a singular noun to plural can sometimes rectify the
absence of an article or determiner preceding a singular noun Berend et al.
(2013). However, a deeper issue might stem from the Japanese language’s lack
of obligatory plural marking for nouns Snape et al. (2016), and the resulting
frequent errors among Japanese students were beyond the detection capability
of the model. Further complicating matters are nouns that can be both
countable and uncountable depending on their sense or metonymic usage, such as
“coffee” which denotes an uncountable substance or a countable cup of the
beverage Berend et al. (2013). Discriminating between these contexts is a
challenging task, demanding human judgment or sophisticated NLP techniques.

#### Tense

Tense errors were heavily context-dependent and conspicuous in their absence
from the model’s detections. The context-dependent nature of such errors was
supported by Granger (1999), who conducted a comprehensive analysis of two
error-tagged learner corpora on tense use and found that there would be no
tense error if the segment was taken out of context. Further, Granger’s study
revealed that the simple present and simple past tenses accounted for 32.5%
and 29% of all verb tense errors, also echoing the findings of this study.

#### Miscellaneous

Though not a major theme, subject-verb agreement also caught our attention, as
some cases are straightforward and should have been easily detected by the
model. For instance, in the sentence “If it occur human, it is very
terrible.”, the verb “occur” should conform to third person singular agreement
following the pronoun “it”. However, the model missed this basic error.

Additionally, there are cases where the intricacy or elongated nature of a
sentence seems to impede detection. Consider the example, “The orbiting line
that is caused the objects are sucked seriously look like noodles, so
scientists say it spaghettification.” Here, “look” should indeed be “looks”,
but given the sentence’s complexity and presence of other errors, the model
overlooked the subject-verb agreement issue.

## 6\. Conclusion

In this study, we compared human and model detection of grammar errors in
Japanese university students’ writing samples and conducted qualitative
analysis on the errors undetected by the state-of-the-art SeqTagger GEC model.
Quantitative results show a high adjusted precision (97.98%) but a relatively
low adjusted recall (42.98%) of the model in error detection, indicating an
accurate but conservative approach. A significant revelation from our thematic
analysis is the model’s deficiency in detecting issues with determiners and
particularly articles in both context-dependent and context-independent
errors. In terms of context-independent errors, basic ones such as subject-
verb agreement remain areas where the model can benefit from further
refinement. There were also instances where original sentences were laden with
errors or possessing elongated structures, leading the model to struggle in
identifying these errors. More importantly, context-dependent errors pose a
substantial challenge, such as tense and noun number. GEC models like
SeqTagger, lacking the innate human cognitive ability to infer context,
struggle to match human performance in this area.

## Ethical Considerations

Subjectivity in determing grammar errors is a major ethical consideration in
this study. While measures were taken to ensure a consistent grammar error
detection process, subsequent analysis by the researchers highlighted
instances of corrections that were deemed unnecessary. This suggests that
personal biases in comprehending and interpreting grammar are not just
possible but inevitable, even among experts.

For data privacy, every precaution was taken to ensure the data’s anonymity,
making sure no personally identifiable information was retained. Moreover, it
is worth noting that students had provided consent for their writings to be
utilized, albeit only in fragmented sentences and not in their entirety.

## Limitations

A salient limitation in this research is the inevitable subjectivity
associated with grammar judgments. By its very nature, grammar is
interpretative, leading even seasoned experts to possibly diverge in their
evaluations or opinions about certain grammatical aspects. The research did
involve a thorough cross-examination between the two experts, and the
inclusion of a third expert to provide insights. Yet, this doesn’t completely
overcome the subjective nature intrinsic to grammatical evaluations. This is
also the reason why we did not report inter-rater agreement between the two
experts.

Furthermore, no thematic analysis was conducted on the instances of agreement
between the model and human experts. We did not compare the detected and
undetected cases that may fall under the same theme to examine why the model
sometimes identifies and at other times fails to detect the same type of
error. This oversight might conceal certain patterns or insights regarding the
model’s performance.

The sample size is another limitation of the study. With the full dataset
comprising only 261 writing samples and a smaller subset of 300 sentences
selected for human annotation, the scope of our findings may have been
restricted. This limitation stems from the extensive time required for
thorough human evaluation. Moving forward, we aim to expand our research by
incorporating a larger collection of writing samples. To facilitate this
expansion and alleviate the burden on human evaluators, we plan to employ
large language models as an auxiliary tool in the validation process.

Another aspect to consider is the generalizability of the study. This study
used writing sample from a single population, the Japanese student cohort.
Thus, it might not fully encapsulate linguistic nuances inherent to various
cultural or linguistic backgrounds.

## Bibliographical References

##

  * Berend et al. (2013) Gábor Berend, Veronika Vincze, Sina Zarrieß, and Richárd Farkas. 2013.  Lfg-based features for noun number and article grammatical errors.  Association for Computational Linguistics.
  * Bryant et al. (2019) Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019.  The BEA-2019 shared task on grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 52–75, Florence, Italy. Association for Computational Linguistics.
  * Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017.  Automatic annotation and evaluation of error types for grammatical error correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  * Bryant et al. (2023) Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023.  Grammatical Error Correction: A Survey of the State of the Art.  _Computational Linguistics_ , pages 1–59.
  * Chodorow et al. (2010) Martin Chodorow, Michael Gamon, and Joel Tetreault. 2010.  The utility of article and preposition error correction systems for english language learners: Feedback and assessment.  _Language Testing_ , 27(3):419–436.
  * Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012.  Better evaluation for grammatical error correction.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 568–572, Montréal, Canada. Association for Computational Linguistics.
  * Granger (1999) Sylviane Granger. 1999.  Uses of tenses by advanced efl learners: evidence from an error-tagged computer corpus.  _Language and Computers_ , 26:191–202.
  * Hinrichs et al. (2015) Lars Hinrichs, Benedikt Szmrecsanyi, and Axel Bohmann. 2015.  "which"-hunting and the standard english relative clause.  _Language_ , 91(4):806–836.
  * Izumi et al. (2003) Emi Izumi, Kiyotaka Uchimoto, Toyomi Saiga, Thepchai Supnithi, and Hitoshi Isahara. 2003.  Automatic error detection in the Japanese learners’ English spoken data.  In _The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics_ , pages 145–148, Sapporo, Japan. Association for Computational Linguistics.
  * Kiyono et al. (2019) Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019.  An empirical study of incorporating pseudo data into grammatical error correction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1236–1242, Hong Kong, China. Association for Computational Linguistics.
  * Lichtarge et al. (2020) Jared Lichtarge, Chris Alberti, and Shankar Kumar. 2020.  Data weighted training strategies for grammatical error correction.  _Transactions of the Association for Computational Linguistics_ , 8:634–646.
  * Mesham et al. (2023) Stuart Mesham, Christopher Bryant, Marek Rei, and Zheng Yuan. 2023.  An extended sequence tagging vocabulary for grammatical error correction.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1608–1619, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Nagata et al. (2005) Ryo Nagata, Takahiro Wakana, Fumito Masui, Atsuo Kawai, and Naoki Isu. 2005.  Detecting article errors based on the mass count distinction.  In _Second International Joint Conference on Natural Language Processing: Full Papers_.
  * Napoles et al. (2015) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015.  Ground truth for grammatical error correction metrics.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 588–593, Beijing, China. Association for Computational Linguistics.
  * Omelianchuk et al. (2020) Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020.  GECToR – grammatical error correction: Tag, not rewrite.  In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 163–170, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * Snape et al. (2016) Neal Snape, Mari Umeda, John Wiltshier, and Noriaki Yusa. 2016.  Teaching the complexities of english article use and choice for generics to l2 learners.  In _Proceedings of the 13th Generative Approaches to Second Language Acquisition Conference (GASLA 2015)_ , pages 208–222. Cascadilla Proceedings Project.
  * Yuan and Briscoe (2016) Zheng Yuan and Ted Briscoe. 2016.  Grammatical error correction using neural machine translation.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 380–386, San Diego, California. Association for Computational Linguistics.
  * Yuan et al. (2019) Zheng Yuan, Felix Stahlberg, Marek Rei, Bill Byrne, and Helen Yannakoudakis. 2019.  Neural and FST-based approaches to grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 228–239, Florence, Italy. Association for Computational Linguistics.


Embedding algorithms are increasingly used to represent clinical concepts in healthcare for improving machine learning tasks such as clinical phenotyping and disease prediction. Recent studies have adapted state-of-the-art bidirectional encoder representations from transformers (BERT) architecture to structured electronic health records (EHR) data for the generation of contextualized concept embeddings, yet do not fully incorporate temporal data across multiple clinical domains. Therefore we developed a new BERT adaptation, CEHR-BERT, to incorporate temporal information using a hybrid approach by augmenting the input to BERT using artificial time tokens, incorporating time, age, and concept embeddings, and introducing a new second learning objective for visit type. CEHR-BERT was trained on a subset of  clinical data from Columbia University Irving Medical Center-York Presbyterian Hospital, which includes 2.4M patients, spanning over three decades, and tested using 4-fold evaluation on the following prediction tasks: hospitalization, death, new heart failure (HF) diagnosis, and HF readmission. Our experiments show that CEHR-BERT outperformed existing state-of-the-art clinical BERT adaptations and baseline models across all 4 prediction tasks in both ROC-AUC and PR-AUC. CEHR-BERT also demonstrated strong few-shot learning capability, as our model trained on only 5% of data outperformed comparison models trained on the entire data set. Ablation studies to better understand the contribution of each time component showed incremental gains with every element, suggesting that CEHR-BERT’s incorporation of artificial time tokens,  time/age embeddings with concept embeddings, and the addition of the second learning objective represents a promising approach for future BERT-based clinical embeddings.

Representation learning, Electronic Health Records, Pre-training

§ INTRODUCTION

Embedding algorithms, widely used for obtaining low dimensional vector representations of words in natural language processing (NLP) applications, have been increasingly adapted for the representation of clinical concepts in healthcare to improve the development of clinical phenotypes and disease prediction  [Glicksberg et al., 2018]. Recent advances in contextualized representations such as Bidirectional encoder representations from transformers (BERT) have revolutionized the NLP field, achieving state of the art performance on all benchmark tasks [Devlin et al., 2019, Peters et al., 2018]. However, there have been few efforts to apply BERT to structured electronic health record (EHR) data for generating contextualized concept embeddings despite promising results from early BERT adaptations demonstrating improved performance compared to classic embedding algorithms such as word2vec and GloVe  [Rasmy et al., 2021, Li et al., 2020].

Despite the differences between structured EHR and text data, a common practice used in the aforementioned BERT adaptations and other classic embedding algorithms is to treat a patient’s medical history as a text document where medical concepts are treated as words and ordered chronologically  [Beam et al., 2020, Choi et al., 2016, Xiang et al., 2019]. Although  this  representation  could  capture the rich contextual information of a patient’s medical history, the  temporal  intervals  between  medical concepts or visits are not preserved; as a consequence, BERT models trained using this patient representation cannot fully leverage temporal information, limiting their performance in downstream prediction tasks. Another challenge is that BERT’s second learning objective – Next Sentence Prediction (NSP) [Devlin et al., 2019] does not apply in the context of structured EHR data as the entire patient history is treated as a single sentence. A common approach adopted by others is to disable NSP and pre-train BERT only with Masked Language Modeling (MLM) [Li et al., 2020], yet there is abundant information in structured EHR data that could be leveraged for designing a new secondary learning objective to improve BERT’s performance in downstream prediction tasks.

In this paper, we focus on adapting the original BERT architecture for structured EHR data in order to improve disease predictions. We propose a new BERT architecture called CEHR-BERT, where we combine two approaches for encoding temporal information of the structured EHR data by:
1) modifying the patient sequence representation through the insertion of artificial tokens between visits to indicate the time intervals; 2) concatenating both age embeddings and time embeddings to concept embeddings to form temporal concept embeddings. Additionally, we designed a second learning objective – Visit Type Prediction (VTP) for CEHR-BERT that leverages heterogeneous EHR data to further boost the performance of BERT.

§ RELATED WORK

A small number of recent studies have sought to adapt BERT for structured EHR data and demonstrated significant performance improvements in their respective evaluations. However, these studies were often limited to a single clinical domain, single visit, or limited in their consideration of time, without fully utilizing the richness of a patient's full medical history.

Li et al., 2020 described the first BERT adaptation for structured EHR data named  BEHRT, which pioneered the idea of utilizing multiple types of embeddings to represent patient history, including concept embeddings, visit segment embeddings, age embeddings, and positional embeddings. In addition, the authors inserted  SEP tokens between visits to indicate the boundaries of visits and enable the execution of BERT as-is on structured EHR data without any modification of the encoder. However, this study only included diagnosis codes in a patient sequence and excluded other clinical domains such as procedures and medications that contain valuable contextual information. In addition, their evaluation focused on diagnosis code prediction instead of disease prediction based on phenotypes.

Another BERT adaptation named G-BERT [Shang et al., 2019] extended BERT to incorporate a graph neural network (GNN). The key idea was to leverage prior knowledge from well crafted medical ontologies to guide the learning of concept embeddings. However, G-BERT was only tailored to medication recommendation; its input data was limited to single visits, and the dataset used for pre-training was relatively small, containing 20K patients.

The latest BERT adaptation, MedBert [Rasmy et al., 2021], had a similar patient representation as BEHRT except that it did not include any temporal information in their model. MedBert excluded age embeddings and visit segment embeddings, along with excluding the  SEP token inserted between visits in favor of including more concept codes. MedBert was trained on EHR data from 20 million patients and introduced a new second learning objective to predict whether the patient had a prolonged length of stay (defined as inpatient visit longer than 7 days). The authors fine-tuned for two disease prediction tasks using a different data source to demonstrate the potential of few-shot learning and showed improved performance. Though a large training data set used and the study showed improvement in the prediction tasks, the lack of temporal information may have limited its potential performance.

Finally, there are two studies that attempted to incorporate temporal information from structured EHR data [Peng et al., 2019, Che et al., 2018]. These studies adopted a similar strategy of incorporating time intervals between neighboring clinical events into their models (e.g. two neighboring visits or lab values in time-series data). In this work, we take a  different approach of incorporating time with the introduction of artificial time tokens in CHER-BERT, which will be described in the following section.

§ DATA AND PREPROCESSING

§.§ Data

EHR data from Columbia University Irving Medical Center-New York Presbyterian Hospital (CUIMC-NYP) was converted into Observational Medical Outcomes Partnership (OMOP), a common data model used to support observational studies and managed by the Observational Health Data Science and Informatics (OHDSI) open-science community [Hripcsak et al., 2015]. The CUIMC-NYP OMOP instance includes numerous data and clinical domains, including visits, conditions, procedures, medications, lab tests, vital signs, and problem lists, among others. Data spans from the early 1980s to present day. We used the CUIMC-NYP OMOP to generate training data and downstream prediction cohorts. To pre-train and fine-tune BERTs, we limited the data to three OMOP domains - conditions, procedures, and medications.

§.§ Data processing and patient representation

For each patient, all medical codes were aggregated from three domains and constructed into a sequence chronologically. In order to incorporate temporal information, we inserted an artificial time token (ATT) between two neighboring visits based on their time interval. The following logic was used for creating ATTs based on the following time intervals between visits: 1) if less than 28 days, ATTs take on the form of $W_n$ where n represents the week number ranging from 0-3 (e.g. $W_1$); 2) if between 28 days and 365 days, ATTs are in the form of $M_n$ where n represents the month number ranging from 1-11 e.g $M_{11}$; 3) beyond 365 days then a $LT$ (Long Term) token is inserted. In addition, we added two more special tokens — $VS$ and $VE$ to represent the start and the end of a visit to explicitly define the visit segment, where all the concepts associated with the visit are subsumed by $VS$ and $VE$. Conceptually, a patient can be represented as a list of visits,

\begin{equation*}
\begin{aligned}
P = \{ & VS, \;\; v_1, \;\;  VE, \;\; ATT, \\
& VS, \;\; v_2, \;\; VE, \;\; ATT, \\
& VS, \;\; v_3, \;\; VE, \;\; ATT, \\
&  …, \\
& VS, \;\; v_i, \;\; VE \}
\end{aligned}
\end{equation*}

where $v_i$ represents the $ith$ visit, and each visit consists of a list of medical concepts $v_i = \{c_{i1}, c_{i2}, c_{i3}, \ldots, c_{ij}\}$. We will refer to EHR patient data representation as patient sequences in the rest of the paper.

§ METHODS

fig:bert_architecture_final shows a high level overview of our adapted BERT architecture.  We used multiple sets of embeddings to represent a patient history including concept embeddings, visit segment embeddings, time embeddings and age embeddings. Concept embeddings were used to capture the numeric representations of the concept codes based on underlying co-occurrence statistics, whereas visit segment embeddings were used to indicate the boundaries of visits (values alternating between A and B). Unlike the previous work, we decided to encode both absolute time (time embeddings) and relative time with respect to visits (age embeddings), due to the finding that certain conditions follow a more seasonal pattern (e.g. flu) while other conditions are more age related (e.g. type 2 diabetes). However, because time and age are numeric values that cannot be directly encoded using standard procedures, we therefore followed the methodology proposed by time2vec [Kazemi et al., 2019]. A fourier transform was applied to decompose a sequence of time points into a series of sine functions, which are controlled by learnable parameters in order to adapt to specific training data. We concatenated concept, time and age embeddings together, then fed it through a fully connected (FC) layer to bring it back to the original dimension, which became the temporal concept embeddings input for the BERT architecture.

Overview of our BERT architecture on structured EHR data. To distinguish visit boundaries, visit segment embeddings are added to concept embeddings. Next, both visit embeddings and concept embeddings go through a temporal transformation, where concept, age and time embeddings are concatenated together. The concatenated embeddings are then fed into a fully connected layer. This temporal concept embedding becomes the input to BERT. We used the BERT learning objective Masked Language Model as the primary learning objective and introduced an EHR specific secondary learning objective visit type prediction.

For pre-training, we used the core learning objective MLM and followed the standard procedure described in the original BERT paper [Devlin et al., 2019]. In addition, we designed a second learning objective named Visit Type Prediction (VTP) to improve BERT’s performance in downstream prediction tasks. VTP was developed based on the observation that different medical concepts are associated with different visit types, and therefore incorporating such domain knowledge may allow BERT to capture additional contextual information. Conceptually, VTP can be thought of as a language translation task, where a sequence of medical concepts are translated to a sequence of visit types (e.g. Inpatient visit v.s. Outpatient visit). To realize this idea, we added a single decoder layer to the BERT architecture to perform VTP. The decoder setup can be summarized as follows: 1) 50% of tokens in the visit sequence are randomly masked; 2) the visit type sequence undergoes the same temporal transformation as the concept sequence to generate the temporal visit type embeddings; 3) in the decoder, the temporal visit type embeddings and contextualized concept embeddings are combined using multi-headed attention to produce the contextualized visit embeddings; 4) contextualized visit embeddings are used to predict the original visit types for those masked positions in the visit sequence. This second learning objective was trained together with the primary learning objective, MLM. It should be emphasized that the visit sequence is not a list of visits that a patient experienced in the past, but rather, it is a list of visit types constructed from the corresponding concepts in the patient sequence.

§ EXPERIMENTS AND RESULTS

§.§ Experiment setup

CEHR-BERT was pre-trained using EHR data from a Columbia University Irving Medical Center-New York Presbyterian Hospital (CUIMC-NYP). Patients who had at least one visit and more than 5 data points in their medical history were included, resulting in 2.4M patients and 184.7M clinical data points across OMOP domains - condition, procedure, and medication. The data characteristics can be found in tab:summery-stats. For pre-training BERT, we used 5 encoders and 8 heads with a dropout rate of $0.1$, along with embedding and hidden dimensions of $128$. The context window of 300 tokens rather than the standard 512 was used to construct the patient sequence because 300 is enough to capture more than 90% of patients' entire medical histories. For those patients who have a sequence of more than 300 codes, we randomly sliced a subset of their sequence (patient history) for pre-training, while patients with less than 300 codes were post-padded with the PAD token. We trained the BERT model for 5 epochs using the Adam optimizer [Kingma and Ba, 2014] with a batch size of 32. The learning rate was initially set to $2e^{-4}$ and the Cosine Annealing LR was used to decay learning rate after every epoch. All training and testing for this study was done on a Linux based server with 768 GB memory, dual Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz processors, and two RTX 2080ti GPUs.

Summary statistics of the CUIMC-NYP OMOP instance

No. of visits per patient     No. of records per patient

mean     14     76

std     29     196

min     1     5

25%     2     10

50%     5     24

75%     14     68

max     1106     31226

§.§ Experiments

To evaluate the model, we followed previously published standards where we fine-tuned BERT together with a Bi-LSTM layer for a set of disease prediction tasks [Rasmy et al., 2021]. Then, we conducted a few-shot learning experiment using the same setup except that a subset of the training data was used for fine-tuning and the model performance was evaluated using the full validation set [McDermott et al., 2021]. In these experiments, we included both BEHRT and MedBert as BERT comparators to better understand the relative performance of our model. To understand the contribution of pre-training, we also included a non pre-trained version of CHER-BERT, which will be referred to as R-BERT in the rest of the paper. In addition, several baseline models were evaluated, including logistic regression (LR), XG-boost, and Bi-LSTM with pre-trained time attention concept embeddings [Xiang et al., 2019]. Finally, we conducted ablation studies to understand how different temporal components in our adaption could impact the model performance.

§.§.§ Disease prediction

Disease prediction is the likelihood of a patient experiencing a condition (disease) in a given time window. tab:summery-predictions shows the 4 prediction tasks including demographics and patient outcomes. Full definitions can be found in section <ref>. For feature extraction, a one-year observation window was used prior to the entry of the target cohort by default unless stated otherwise in the prediction definition.

Definitions and cohort characteristics of prediction tasks

HF readmission      Discharge home Death     T2DM HF     Hospitalization

2.2ex Cohort size     $97758$     $207919$     $114564$     $590578$

2.2ex Median age     $72$     $49$     $61$     $45$

2.2ex Male     $50.30\%$     $33.23\%$     $49.66\%$     $37.46\%$

2.2ex Female     $49.71\%$     $66.77\%$     $50.34\%$     $62.54\%$

2.2ex Outcome     $24.16\%$     $4.85\%$     $9.38\%$    $10.90\%$

4.2ex Description      30 days all-cause readmission in HF patients [Golas et al., 2018], see <ref> for details.     Mortality within 1 year since discharged to home, see <ref> for details     Life time heart failure prediction since the initial diagnosis of type 2 diabetes mellitus [Rasmy et al., 2021], the medical codes used can be found in <ref>     2 year risk of hospitalization starting from the $3rd$ year since the inital entry into the EHR system [Zhang et al., 2018], see <ref> for details

Extracting features for sequence models (e.g. LSTM and BERTs) was straightforward, where medical concepts were organized in a sequence using chronological order; and artificial tokens and the SEP token were inserted between neighboring visits for CEHR-BERT and BEHRT respectively. The pre-truncate and post-padding strategy was utilized to standardize the size of the inputs for patients who had more/less concepts than the size of the context window. For LR and XG-boost, frequency-based features were constructed by counting the number of occurrences of medical concepts in the observation window. Additionally, medical concepts were rolled up using  ontological hierarchies to reduce dimensionality [Ng et al., 2016], for which a detailed explanation could be found in section <ref>

For performance evaluation, the 4-fold evaluation was utilized, where in each fold split the data was split into three different sets (75:10:15 split for train/val/test). The sequence models were trained for 10 epochs with early stopping to monitor validation loss with the patience set to 1. On the other hand, since frequency based models (LR and XGB) were not fine-tuned for hyper-parameters, $85\%$ of the data was used for training and the remaining $15\%$ was used for testing directly. The python library sklearn was utilized for training with the default configuration. At the end of each fold, the area under the receiver operating characteristics curve (AUC) was calculated using the test set. In addition, PR-AUC (Precision-Recall) was reported because some of the prediction cohorts had imbalanced outcomes e.g. Discharge Home Death.

tab:main-results shows the average AUC and PR-AUC for all models across 4 prediction tasks, where the best value is highlighted in bold. Overall, sequence models outperformed frequency based models, and BERTs performed better than bi-LSTM. Among all models evaluated, CEHR-BERT achieved the best performance in both AUC and PR-AUC across all tasks. In particular, CEHR-BERT is the only model that achieved an AUC of $80$ in t2dm hf, and its PR-AUC exceeded the second best performing model MedBert by more than $10$% (from 0.274 to 0.323). The second and third top performing models were MedBert and BEHRT, whose performances were consistently worse than that of CEHR-BERT. However, these relative performances did not follow the same trend in hospitalization, where Bi-LSTM was the second best performing model after CEHR-BERT and outperformed MedBERT and BEHRT. Finally, R-BERT was performing consistently worse than CHER-BERT, suggesting that pre-training played an important role in improving the downstream prediction tasks.

Average AUC and PR-AUC values and standard deviations for three baseline models and three BERT based models across 4 prediction tasks
LR     XGB     LSTM     R-BERT     BEHRT     MedBert     CEHR-BERT

2*t2dm hf     PR     24.8±0.8%     24.8±0.5%     25.8±0.7%     28.1±1.5%     27.1±1.7%     27.37±0.6%      32.3±1.0%

AUC     76.7±0.3%     76.5±0.5%     77.4±0.5%     78.0±0.9%     77.5±0.5%     78.19±0.1%     80.7±0.6%

2*hf readmit     PR     36.3±0.7%     37.6±1.3%     33.3±0.5%     37.1±0.6%     37.4±0.9%     38.0±0.5%     38.6±0.1%

AUC     64.2±0.7%     64.0±0.3%     61.7±0.2%     65.0±0.5%     65.1±0.4%     65.8±0.2%     66.3±0.2%

home death     PR     46.7±0.7%     48.5±0.8%     49.1±1.8%     46.7±1.2%     50.7±0.5%     51.4±0.5%     52.7±0.4%

AUC     93.4±0.1%     93.5±0.3%     93.8±0.2%     93.6±0.2%     94.0±0.1%     94.2±0.1%     94.6±0.1%

2*hospitalization     PR     26.9±0.5%     29.0±0.3%     30.0±0.4%     28.0±0.5%     29.4±0.2%     29.5±0.3%     31.1±0.4%

AUC     72.9±0.1%     74.0±0.1%     75.1±0.2%     74.4±0.3%     74.7±0.1%     74.6±0.1%     75.9±0.1%

§.§.§ Performance of few-shot learning

One of the advantages of using pre-trained models is the ability to leverage prior knowledge captured from millions of training examples during the pre-training phase. As a result, BERT could be fine-tuned for downstream tasks using a small training set to achieve decent performance. The setup for few-shot learning was similar to the 4-fold evaluation except that we used a subset of the training data in each fold. Specifically, we predefined a list of training percentages (5%, 10%, 20%, 40%, and 80%) with respect to 75% of the training data in each fold in the experiment, and then iterated through the percentages to conduct a separate 4-fold evaluations, in which a subset of the training data was randomly selected based on the training percentage. Apart from randomly sampling the training data, the other configurations including disease phenotypes, baseline models, the process for train/val/test split, and reporting metrics were identical to the disease prediction tasks. fig:transfer_learning shows that CEHR-BERT is the best performing model in terms of AUC and PR-AUC at different training percentages for t2dm hf. Following CEHR-BERT,  MedBert and BEHRT were other top performers compared to LR, XGB and BI-LSTM. CEHR-BERT outperformed the second best model MedBert by the same margin throughout the course of this few-show learning experiment. Noticeably, CEHR-BERT fine-tuned with 5% of the training data could achieve an AUC of nearly $0.78$ and the PR-AUC of almost $0.29$, whereas other models only achieved the AUCs between 0.60 and 0.76 and PR-AUCs between 0.12 and 0.26 despite using up to 80% of the training data. Furthermore, the same trend can be observed for other prediction tasks as well, shown in fig:discharge_home_death_apd_fig,fig:hospitalization_apd_fig,fig:t2dm_hf_apd_fig. The only exception is in hospitalization, where CEHR-BERT and LSTM were the first and second best performing models; whereas, other BERTs marginally improved the performance compared to frequency based models.

§.§ Ablation studies

To better understand the contributions of time tokens, time/age embeddings and VTP, we conducted a number of ablation studies. We pre-trained several variations of BERT by excluding one component at a time, then conducted the evaluation described in disease_prediction, and reported results for each variation in tab:ablation-studies-results. The best value is highlighted in bold in each row. We discuss the contribution of each component separately in the following sections. We also provided the size of each BERT network and the pre-training time in tab:parameter_counts so the practitioners could choose the appropriate model based on the use-case.

Average AUC and PR-AUC values and standard deviations for ablation studies

M-BERT     B-BERT     NS-BERT     NT-BERT     ALT-BERT     V-BERT     CEHR-BERT

2*t2dm hf     PR     29.9±1.0%     30.6±0.5%     31.8±1.3%     28.2±0.2%     29.3±0.5%     28.6±0.8%     32.3±1.0%

AUC     79.2±0.3%     79.5±0.5%     80.2±0.6%     78.5±0.4%     76.7±0.2%     78.6±0.1%     80.7±0.6%

2*hf readmit     PR     34.1±0.6%     37.4±0.9%     38.3±0.6%     39.3±0.7%     33.3±0.8%     38.6±0.4%     38.6±0.1%

AUC     62.6±0.3%     65.1±0.2%     65.8±0.1%     66.4±0.3%     61.6±0.7%     65.9±0.2%     66.3±0.2%

home death     PR     53.1±0.5%     52.0±1.0%     52.0±0.7%     52.5±0.8%     31.6±3.0%     52.6±1.3%     52.7±0.4%

AUC     94.4±0.3%     94.4±0.1%     94.3±0.1%     94.2±0.3%     87.3±0.9%     94.4±0.1%     94.6±0.1%

2*hospitalization     PR     30.0±0.7%     30.4±0.5%     31.3±0.7%     30.8±0.6%     23.4±0.5%     30.6±0.4%     31.1±0.4%

AUC     74.9±0.3%     75.3±0.3%     76.1±0.2%     75.2±0.2%     69.2±0.4%     75.3±0.3%     75.9±0.1%

[] M-BERT: CEHR-BERT trained on MedBert patient representation

[] B-BERT: CEHR-BERT trained on BEHRT patient representation

[] NS-BERT: CEHR-BERT without the second learning objective

[] NT-BERT: CEHR-BERT without the time/age embeddings

[] ALT-BERT: modified CEHR-BERT where concept, time and age embeddings are summed together

[] V-BERT: modified CEHR-BERT where VS and VE are removed from the patient sequence

§.§.§ Assessment of time tokens

To understand the effectiveness of time tokens, we compared our patient representation described in patient_representation to the existing ones adopted by BEHRT and MedBert. MedBert used a patient representation that only contained the medical concepts and did not include any other artificial tokens. BEHRT used a variation of the M-BERT patient representation, where the SEP token was inserted between neighboring visits. fig:patient_representation_comparison shows an example of these different EHR representations for a patient's medical history.
To perform a fair comparison, we applied the same architecture (described in sec:methods) to pre-train using these two patient representations, and then we tested the model performances following the same evaluation procedure described in disease_prediction. We will refer to these two BERTs as B-BERT (trained using the BEHRT representation) and M-BERT (trained using the MedBert representation) to distinguish them from others. tab:ablation-studies-results shows that CEHR-BERT improved the performances in all tasks as compared to the other patient representations except that M-BERT performed slightly better in terms of PR-AUC in discharge home death. Our results indicate that embedding ATTs can effectively capture the temporal information of structured EHR data. One plausible explanation is that ATTs may be treated like any other tokens such that when BERT utilizes them in the self-attention mechanism, those tokens function like a place holder for preserving temporal information, which is then propagates through to the last encoder. In addition, we trained another CEHR-BERT variation where we removed VS and VE tokens and only kept the ATT tokens, (which we will be refer to as V-BERT). The performance of V-BERT dropped compared to CEHR-BERT, and was comparable to that of B-BERT (SEP token inserted between visits), suggesting that VS and VE play an important role in explicitly defining the boundary of a visit, so that ATT tokens would just function like a regular SEP token without their presence.

§.§.§ Assessment of visit type prediction

VTP was designed in this study as a substitute for the original second learning objective NSP to leverage the unique characteristics of structured EHR data.
To understand its contribution, we excluded VTP and pre-trained BERT, and then followed the standard evaluation. This BERT will be referred to as NS-BERT. The comparison between NS-BERT and CEHR-BERT in tab:ablation-studies-results demonstrates the improved performances across all prediction tasks attributed to VTP with the exception of hospitalization, where NS-BERT slightly outperformed CEHR-BERT but both remained largely similar. Such minor differences may be due to random variation of 4-fold evaluation. The underlying assumption of VTP was that concepts associated with different visit types follow different distributions. Therefore, incorporating VTP into BERT could help learn the representation of concepts. To better understand this, we reported the top 10 most frequent concepts stratified by visit type and domain in tab:top_10_most_frequent_concepts_part1 in apd:second, which shows that the concept distributions are indeed distinct between visit types. For example, the most frequent concepts associated with inpatient visits relate to childbirth or severe conditions that require immediate hospitalization, while the top outpatient concepts normally pertain to chronic conditions such as hypertension and type 2 diabetes that may not require immediate medical attention but long-term management. Currently, the threshold for masking visit tokens is set to 50%, but we plan to investigate different thresholds to optimize the performance of BERT.

§.§.§ Assessment of time/age embeddings

At the input layer, concept, time, and age embeddings are concatenated together, and fed into a FC layer to form temporal concept embeddings, which is then used as the input for CEHR-BERT. To understand the impact of this transformation, we pre-trained a BERT named ALT-BERT, where we summed time embeddings, age embeddings and concept embeddings to form temporal concept embeddings instead of using a FC layer. tab:ablation-studies-results shows that the performance of ALT-BERT was lower compared to that of CEHR-BERT, suggesting that our temporal concept embeddings transformation is more effective than the summation of time/age/concept embeddings. This confirms existing empirical evidence that summing is more rigid than feeding a concatenated product into another the FC layer in terms of fitting the model to the training data.

Furthermore, we wanted to know the impact of using time/age embeddings. Therefore we pre-trained another BERT named NT-BERT, where we disabled all components related to time/age embeddings and only used the concept embeddings. We added the positional encoder (used by Rasmy et al., 2021, Li et al., 2020) to give the model a basic sense of temporality. tab:ablation-studies-results shows that
without time/age embeddings, NT-BERT did perform slightly better in hf readmit than CEHR-BERT; however, CEHR-BERT outperformed NT-BERT in all other tasks. In particular, PR-AUC for NT-BERT in t2dm hf dropped by 4% compared to CEHR-BERT (from 32.3±1.0% to 28.2±0.2%). Therefore, the benefit of performing the transformation for generating temporal concept embeddings likely outweighs the cost.

§ DISCUSSION

§.§ Understanding artificial time tokens

To understand the functional role of ATTs, we extracted the base embeddings of ATTs and computed 2D features using PCA for visualization. fig:time_token_visualization shows that ATTs were arranged from right to left in increasing order of time intervals, where the rightmost and leftmost tokens represent the shortest and longest time intervals respectively. Specifically, the ATT week tokens (including $W_0$, $W_1$, and $W_2$) formed a cluster on the right bottom, and month tokens (in the form of $M_n$ e.g. $M_1$) seemed to present a linear relationship. In addition, the VS and VE tokens were located far away from time tokens because they represent the start and the end of a visit rather than time intervals, corroborating that their semantics are fundamentally different from time tokens. The analysis supported our assumption that BERT can derive semantics of ATTs and show meaningful relationships between them, thereby learning the hidden knowledge that different time tokens are associated with different groups of concepts.

§.§ Inflated patient sequence

Due to the use of ATTs, we artificially inflated the length of patient sequences, and as a result involuntarily cut-off some records from patient sequences. Whereas, in other sequence models (M-BERT, and Bi-LSTM) we used patient sequences as-is. We calculated the median length (defined as the median number of concepts) as well as 95% sequence length (defined as the number of concepts at 95% percentile) of the patient sequences for different BERT representations across 4 disease prediction tasks, which is shown in tab:inflated-patient-sequences. Among all tasks, patient sequences for t2dm hf and hospitalization patients have the shortest patient sequences; whereas, hf readmission has the richest medical history. Among all patient representations, CEHR-BERT has the longest sequence due to the use of ATTs, and B-BERT has the second longest sequence due to the use of SEP. This raised the question of whether cutting off early records in patient sequences could have contributed to the performance boost in evaluations. To address this concern, we re-ran the hf readmission analysis using a shorter observation window of $180$ days for all models except CEHR-BERT. The motivation was to simulate a scenario where early records were omitted and only the latest records were included. The performance of this modified evaluation was reported in tab:hf_readmission_180_days_results. It showed that model performances did not improve but dropped when compared to the original evaluation in tab:main-results and cutting off the early records in patient sequences did not improve the performance. This comparison suggests the performance gain by CEHR-BERT is attributed to the use of ATT tokens.

§ FUTURE WORK

In this work, we used a context window of 300 for pre-training and conducting experiments. Although we can increase the context window to the BERT sequence limit of 512, patients may have more clinical concepts in their medical histories than the sequence length supported by BERT. To address this limitation, we plan to employ a sliding window strategy, in which we can apply BERT to extract the contextualized embedding representations for each region scanned by the sliding window. Then the regional representations could be combined via a 1D convolution layer or LSTM to generate the final patient representation. Although VTP seems to have improved the results by a small margin, it is not clear whether the improvement could be attributed to random variation, so we need to investigate VTP in the follow-up analysis. Furthermore, we want to include labs to the patient sequence as labs offer an extensive amount of useful information. However, embedding the lab data would require a different strategy as labs are continuous features unlike the discrete data types included in this study. Finally, we will investigate whether or not our methods of incorporating time could be generalized for other models as well.

§ CONCLUSION

To the best of our knowledge, this is the first study that focuses on incorporating time into BERT for use on structured EHR data, including multiple elements for representing temporal information and leveraging multiple domains of clinical care. CEHR-BERT outperforms existing state of the art BERT-based approaches across a number of different prediction tasks. Based on our results, incorporating time tokens into the patient sequence and combining time/age embeddings with concept embeddings seem to synergistically boost the performance more than any individual modification alone. Therefore, including both into the final CEHR-BERT architecture seems to be the most effective way of capturing the temporal information of structured EHR data. This study was developed using the OMOP common data model, and as a result can be expanded to run on the OHDSI network in order to replicate these results beyond a single database.

This work was sponsored by the National Center for Advancing Translational Sciences grant 1U01TR002062 and the Director’s Office of National Institutes of Health grants 5U2COD023196 and 3UG3OD023183.

[Beam et al., 2020]
Andrew L. Beam, Benjamin Kompa, Allen Schmaltz, Inbar Fried, Grin Weber, Nathan
Palmer, Xu Shi, Tianxi Cai, and Isaac S. Kohane.
Clinical concept embeddings learned from massive sources of
multimodal medical data.
In Pacific Symposium on Biocomputing, 2020.

[Che et al., 2018]
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu.
Recurrent Neural Networks for Multivariate Time Series with Missing
Scientific Reports 2018 8:1, 80 (1):0 1–12,
apr 2018.
ISSN 2045-2322.
URL <https://www.nature.com/articles/s41598-018-24271-9>.

[Choi et al., 2016]
Youngduck Choi, Chill Yi-I Chiu, and David Sontag.
Learning Low-Dimensional Representations of Medical Concepts.
AMIA Summits on Translational Science Proceedings,
2016:0 41, 2016.
URL </pmc/articles/PMC5001761/

[Devlin et al., 2019]
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova.
BERT: Pre-training of deep bidirectional transformers for language
In NAACL HLT 2019 - 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies - Proceedings of the Conference, 2019.
ISBN 9781950737130.

[Glicksberg et al., 2018]
Benjamin S. Glicksberg, Riccardo Miotto, Kipp W. Johnson, Khader Shameer,
Li Li, Rong Chen, and Joel T. Dudley.
Automated disease cohort selection using word embeddings from
Electronic Health Records.
Pacific Symposium on Biocomputing. Pacific Symposium on
Biocomputing, 230 (212669):0 145, 2018.
URL </pmc/articles/PMC5788312/

[Golas et al., 2018]
Sara Bersche Golas, Takuma Shibahara, Stephen Agboola, Hiroko Otaki, Jumpei
Sato, Tatsuya Nakae, Toru Hisamitsu, Go Kojima, Jennifer Felsted, Sujay
Kakarmath, Joseph Kvedar, and Kamal Jethwani.
A machine learning model to predict the risk of 30-day readmissions
in patients with heart failure: a retrospective analysis of electronic
medical records data.
BMC Medical Informatics and Decision Making, 180 (1),
jun 2018.
URL </pmc/articles/PMC6013959/

[Hripcsak et al., 2015]
George Hripcsak, Jon D. Duke, Nigam H. Shah, Christian G. Reich, Vojtech Huser,
Martijn J. Schuemie, Marc A. Suchard, Rae Woong Park, Ian Chi Kei Wong,
Peter R. Rijnbeek, Johan Van Der Lei, Nicole Pratt, G. Niklas Norén,
Yu Chuan Li, Paul E. Stang, David Madigan, and Patrick B. Ryan.
Observational Health Data Sciences and Informatics (OHDSI):
Opportunities for Observational Researchers.
In Studies in Health Technology and Informatics, volume 216,
pages 574–578. IOS Press, 2015.
ISBN 9781614995630.

[Kazemi et al., 2019]
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet
Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus
Time2Vec: Learning a Vector Representation of Time.
jul 2019.
URL <https://arxiv.org/abs/1907.05321v1>.

[Kingma and Ba, 2014]
Diederik P. Kingma and Jimmy Ba.
Adam: A Method for Stochastic Optimization.
3rd International Conference on Learning Representations, ICLR
2015 - Conference Track Proceedings, dec 2014.
URL <https://arxiv.org/abs/1412.6980v9>.

[Li et al., 2020]
Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine,
Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza
BEHRT: Transformer for Electronic Health Records.
Scientific Reports 2020 10:1, 100 (1):0 1–12,
apr 2020.
ISSN 2045-2322.
URL <https://www.nature.com/articles/s41598-020-62922-y>.

[McDermott et al., 2021]
Matthew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter
Szolovits, and Marzyeh Ghassemi.
A comprehensive EHR timeseries pre-training benchmark.
ACM CHIL 2021 - Proceedings of the 2021 ACM Conference on
Health, Inference, and Learning, pages 257–278, apr 2021.

[Ng et al., 2016]
Kenney Ng, Steven R. Steinhubl, Christopher DeFilippi, Sanjoy Dey, and
Walter F. Stewart.
Early Detection of Heart Failure Using Electronic Health Records:
Practical Implications for Time before Diagnosis, Data Diversity, Data
Quantity and Data Density.
Circulation. Cardiovascular quality and outcomes, 90
(6):0 649, nov 2016.
URL </pmc/articles/PMC5341145/

[Peng et al., 2019]
Xueping Peng, Guodong Long, Tao Shen, Sen Wang, Jing Jiang, and Michael
Temporal Self-Attention Network for Medical Concept Embedding.
Proceedings - IEEE International Conference on Data Mining,
ICDM, 2019-November:0 498–507, sep 2019.
URL <https://arxiv.org/abs/1909.06886v1>.

[Peters et al., 2018]
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark,
Kenton Lee, and Luke Zettlemoyer.
Deep contextualized word representations.
NAACL HLT 2018 - 2018 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language Technologies
- Proceedings of the Conference, 1:0 2227–2237, feb 2018.
URL <https://arxiv.org/abs/1802.05365v2>.

[Rasmy et al., 2021]
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi.
Med-BERT: pretrained contextualized embeddings on large-scale
structured electronic health records for disease prediction.
npj Digital Medicine 2021 4:1, 40 (1):0 1–13,
may 2021.
ISSN 2398-6352.
URL <https://www.nature.com/articles/s41746-021-00455-y>.

[Shang et al., 2019]
Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun.
Pre-training of Graph Augmented Transformers for Medication
IJCAI International Joint Conference on Artificial
Intelligence, 2019-August:0 5953–5959, jun 2019.
URL <https://arxiv.org/abs/1906.00346v2>.

[Xiang et al., 2019]
Yang Xiang, Jun Xu, Yuqi Si, Zhiheng Li, Laila Rasmy, Yujia Zhou, Firat
Tiryaki, Fang Li, Yaoyun Zhang, Yonghui Wu, Xiaoqian Jiang, Wenjin Jim Zheng,
Degui Zhi, Cui Tao, and Hua Xu.
Time-sensitive clinical concept embeddings learned from large
electronic health records.
BMC Medical Informatics and Decision Making, 2019.
ISSN 14726947.

[Zhang et al., 2018]
Jinghe Zhang, Kamran Kowsari, James H. Harrison, Jennifer M. Lobo, and Laura E.
Patient2Vec: A Personalized Interpretable Deep Representation of the
Longitudinal Electronic Health Record.
IEEE Access, 6:0 65333–65346, oct 2018.
URL <https://arxiv.org/abs/1810.04793v3>.

§ PREDICTION TASKS

A prediction task can be phrased as the following, “among a particular group of people, who will go on and experience some event”. One can think of this problem as defining a target cohort that represents the initial group of people, and an outcome cohort that represents the subset of the initial group who will experience a particular event, e.g. among the type 2 diabetes patients, who will go on and develop heart failure. Both target and outcome cohorts can be defined as a group of people who satisfy certain inclusion criteria for a certain period of time. Typically, a cohort definition includes a cohort entry event and a set of inclusion criteria (an exclusion criterion can be thought of as an inclusion criterion with 0 occurrence). Specifically, the cohort entry event defines the index date, at which the patients enter the cohort, and the inclusion criteria add more constraints to the cohort if applicable, such as the requirements of certain diagnosis, medications, procedures or temporal relationships among criteria, and etc. In addition, a prediction window needs to be specified for generating the ground truth labels for the given target and outcome cohorts, if the outcome index date falls between the index date of the target cohort and the prediction window, we will declare the case to be positive, and otherwise negative.

In terms of prediction time range, we use observation window, hold-off window and prediction window to collect data in different time periods. One patient's medical history is built on an event sequence. Each event represents one medical engagement within a visit. It could be condition occurrence, medication exposure, procedure occurrence and measurement etc. In this paper, we only focus on the first three types of events. We extract features using data from the observation window followed by a hold-off window to avoid same target concepts being included into feature construction. The prediction window is right after the index event. And the goal is to predict the occurrence of any outcome event. With the datasets, we could train, test and validate the model. fig:cohort_definition_prediction_window visualizes the cohort definition that is constructed based on all events in patient medical history.

Cohort Definition and Prediction Windows

We validate our model by applying it to 4 downstream prediction problems.  tab:cohort_definition_parameters lists the cohort definition parameters for the prediction tasks.

Cohort Definition Parameters

cohort name     index event     inclusion criteria     outcome event     observation window     hold-off window     prediction window

t2dm hf     Type 2 Diabetes Mellitus condition occurrence or medication exposure     No pre-existing diabetes, type 1 diabetes, diabetes insipidus, gestational diabetes, secondary diabetes, neonatal diabetes     Heart Failure     unbounded     0     unbounded

hf readmit     Heart Failure condition occurrence     At least one HF treatment, lab test or medication exposure     Readmission (In-patient visit)     360 days     0     30 days

discharge home death     Discharge to home     Following an in-patient visit     Death     360 days     0     360 days

hospitalization     EHR start     Number of visits between 2 and 30     In-patient visit     540 days     180 days     720 days

§.§ Feature Engineering

Frequency based features were used for LR and XG-boost,  and generated as the following 1) International Classification of Diseases (ICD) ICD-9 or ICD-10 codes were rolled up to 2 or 3 digit codes for procedure and diagnosis depending on where the dot character is in the code e.g. 44.98 and I50.0; 2) for procedure records encoded by Current Procedural Terminology (CPT), they were rolled up to the second level from the root; 3) all medications were rolled up to the ingredient level; 4) for the codes that couldn't be rolled up, the original codes were kept; 5) the frequencies of the rolled-up codes were calculapted within the observation window.

§.§ Type 2 Diabetes Mellitus patients who developed Heart Failure

The target cohort consists of patients who had T2DM (type 2 diabetes mellitus) in the medical history. The index event of T2DM is patient encounters with condition concept ids or medication exposures of anti-diabetes medications. We also exclude any patients with pre-existing T2DM, type 1 diabetes, diabetes insipidus, gestational diabetes, secondary diabetes and neonatal diabetes.

The outcome cohort is a subset of the target cohort who developed heart failure during the prediction window. We defined Heart Failure cohort as patients who were diagnosed with heart failure, at least one lab test with high BNP results, received any treatment including mechanical circulatory support, artificial heart associated procedure, diuretic agent, vasoactive agent or dialysis procedure.

For each of the criteria, we construct a concept set with a group of OMOP concept ids. In tab:t2dm_cohort_related_concepts & tab:heart_failure_related_concepts, we list all related concept sets and concept ids from OMOP vocabulary.

T2DM Cohort Related Concepts

1lConcept Set
1lOMOP concept ids

7*Condition     1L6cmT2DM     1L7cm443238, 201820, 442793, 4016045
1L6cmpre-existing T2DM     1L7cm40769338,43021173,42539022, 46270562
1L6cmtype 1 diabetes     1L7cm201254, 4019513, 40484648
1L6cmdiabetes insipidus     1L7cm30968, 438476
1L6cmgestational diabetes     1L7cm4058243
1L6cmsecondary diabetes     1L7cm195771
1L6cmneonatal diabetes     1L7cm193323
19*Medication     1L6cmMetformin     1L7cm1503297
1L6cmChlorpropamide     1L7cm1594973
1L6cmGlimepiride     1L7cm1597756
1L6cmGlyburide     1L7cm1559684
1L6cmGlipizide     1L7cm1560171
1L6cmTolbutamide     1L7cm1502855
1L6cmTolazamide     1L7cm1502809
1L6cmPioglitazone     1L7cm1525215
1L6cmRosiglitazone     1L7cm1547504
1L6cmSitagliptin     1L7cm1580747
1L6cmSaxagliptin     1L7cm40166035
1L6cmAlogliptin     1L7cm43013884
1L6cmLinagliptin     1L7cm40239216
1L6cmRepaglinide     1L7cm1516766
1L6cmNateglinide     1L7cm1502826
1L6cmMiglitol     1L7cm1510202
1L6cmLinagliptin     1L7cm40239216
1L6cmAcarbose     1L7cm1529331
1L6cmInsulin     1L7cm35605670, 35602717, 1516976, 1502905, 46221581, 1550023, 35198096, 42899447, 1544838, 1567198, 35884381, 1531601, 1588986, 1513876, 19013951, 1590165, 1596977, 1586346, 19090204, 1513843, 1513849, 1562586, 19090226, 19090221, 1586369, 19090244, 19090229, 19090247, 19090249, 19090180, 19013926, 19091621, 19090187

Heart Failure Cohort Related Concepts

1lConcept Set
1lOMOP concept ids

1L6cmHeart Failure     1L7cm316139
1L6cmDiuretic Agent     1L7cm4186999, 956874, 942350, 987406, 932745, 1309799, 970250, 992590, 907013
1L6cmVasoactive Agent     1L7cm1942960
1L6cmHigh B-type Natriuretic Peptide (BNP) $>$ 500 pg/mL     1L7cm4307029
1L6cmNT-proBNP $>$ 2000 pg/mL     1L7cm1594973
1L6cmMechanical Circulatory Support     1L7cm45888564, 4052536, 4337306, 2107514, 45889695, 2107500, 45887675, 43527920, 2107501, 45890116, 40756954, 4338594, 43527923, 40757060, 2100812
1L6cmArtificial Heart Associated Procedure
1L7cm4144390, 4150347, 4281764, 725038, 725037, 2100816, 2100822, 725039, 2100828, 4337306, 4140024, 4146121, 4060257, 4309033, 4222272, 4243758, 4241906, 4080968, 4224193, 4052537, 4050864

§.§ Heart Failure patients who were readmitted within 30 days

The target cohort contains patients who were admitted into hospital due to heart failure . The index event is an inpatient visit with a heart failure diagnosis (316139). Patients in the target cohort who were readmitted into hospital within 30 days will be in the outcome cohort. The concept_ids of inpatient visits are 9201 and 262. In this case, the prediction window is 30 days.

§.§ Patients who were discharged and died within one year

The target cohort is patients who had an inpatient visit and were discharged to home. The outcome cohort is patients who died within one year after being discharged. The index event is inpatient visit with visit concept_id as 9201 or 262 and discharge_to_concept_id is home or other nursing facilities. The outcome event is death. The prediction window is 360 days.

§.§ Hospitalization

The hospitalization cohort has a special structure showed in fig:hospitalization_cohort_definition_prediction_window than the generalized cohort structure. The index event is when the patient had the first visit in the hospital. And the observation window is post the index event. The outcome event is an inpatient visit. We only include patients who had visit occurrences between 2 to 30 during the observation window to make sure patients had enough data points and also remove any outliers.

Hospitalization Cohort Definition and Prediction Windows

§ ADDITIONAL FIGURES AND ANALYSES

EHR Representation of patient medical history

2d visualization of the Artificial Time Tokens added to CEHR-BERT. The base embeddings of those tokens were extracted from CEHR-BERT, and PCA was run to extract the 2d features for visualization.

AUC and PR-AUC at different training percentages for all the models for few-shot learning task for hf readmission are plotted against

AUC and PR-AUC at different training percentages for all the models for few-shot learning task for discharge home death are plotted against

AUC and PR-AUC at different training percentages for all the models for few-shot learning task for hospitalization are plotted against

AUC and PR-AUC at different training percentages for all the models for few-shot learning task for hf readmission are plotted against

Average AUC and PR-AUC values for LR, XGB, LSTM, BEHRT, and MedBert for hf readmission using the 180-day observation window

LR     XGB     LSTM     BEHRT     MedBert

PR-AUC     36.6±0.7%     37.1±0.6%     36.3±0.1%     37.2±0.3%     31.7±0.4%

AUC     64.7±0.3%     64.1±0.7%     64.2±0.4%     65.0±0.3%     59.5±0.1%

Trainable and Non trainable parameters counts for M-BERT, B-BERT, NS-BERT, ALT-BERT, V-BERT and CEHR-BERT.

M-BERT     B-BERT     NS-BERT     ALT-BERT     V-BERT     CEHR-BERT

Trainable Parameters     9083060     9083060     8811380     9044596     9085108     9085364

NonTrainable Parameters     0     0     0     0     0     0

Time to pre-train per epoch     8.5h     8.5h     7.5h     8.5h     8.5h     8.5h

Patient sequence lengths for 3 BERT representations across 4 disease predictions

HF readmission      Discharge home Death     T2DM HF     Hospitalization

3*median length     CEHR-BERT     123     64     21     19

b-bert     98     47     15     12

m-bert     86     38     13     9

3*95% length     CEHR-BERT       608     330     366     108

b-bert      520     257     246     78

m-bert      481     223     189     66

Top 10 most frequent condition concepts associated with each visit type

1p3cmVisit Type
1p6cmCondition Concepts
1p2cmPatient Count
1p2cmPercentage of the Visit Type Patients

23cmOutpatient Visit
1L0.8cm1     1L6cmEssential hypertension     1L2cm310794     1L2cm13.00%
1L0.8cm2     1L6cmAbdominal pain     1L2cm172393     1L2cm7.21%
1L0.8cm3     1L6cmChest pain     1L2cm164499     1L2cm6.88%
1L0.8cm4     1L6cmHyperlipidemia     1L2cm160068     1L2cm6.70%
1L0.8cm5     1L6cmFinding related to pregnancy     1L2cm140044     1L2cm5.86%
1L0.8cm6     1L6cmJoint pain     1L2cm117134     1L2cm4.90%
1L0.8cm7     1L6cmDyspnea     1L2cm114580     1L2cm4.79%
1L0.8cm8     1L6cmLow back pain     1L2cm114031     1L2cm4.77%
1L0.8cm9     1L6cmPure hypercholesterolemia     1L2cm113437     1L2cm4.75%
1L0.8cm10     1L6cmUnplanned pregnancy     1L2cm110584     1L2cm4.63%
23cmInpatient Visit
1L0.8cm1     1L6cmSingle live birth     1L2cm279820     1L2cm35.01%
1L0.8cm2     1L6cmEssential hypertension     1L2cm150002     1L2cm18.77%
1L0.8cm3     1L6cmFinding related to pregnancy     1L2cm121545     1L2cm15.21%
1L0.8cm4     1L6cmPostpartum finding     1L2cm100345     1L2cm12.55%
1L0.8cm5     1L6cmDelivery normal     1L2cm66666     1L2cm8.34%
1L0.8cm6     1L6cmLate effect of medical and surgical care complication     1L2cm63224     1L2cm7.91%
1L0.8cm7     1L6cmCongestive heart failure     1L2cm55631     1L2cm6.96%
1L0.8cm8     1L6cmCoronary arteriosclerosis     1L2cm53763     1L2cm6.73%
1L0.8cm9     1L6cmHyperlipidemia     1L2cm51459     1L2cm6.44%
1L0.8cm10     1L6cmDiabetes mellitus without complication     1L2cm51069     1L2cm6.39%
23cmEmergency Room and Inpatient Visit
1L0.8cm1     1L6cmEssential hypertension     1L2cm64914     1L2cm30.09%
1L0.8cm2     1L6cmFinding related to pregnancy     1L2cm43878     1L2cm20.34%
1L0.8cm3     1L6cmSingle live birth     1L2cm41606     1L2cm19.29%
1L0.8cm4     1L6cmHyperlipidemia     1L2cm39051     1L2cm18.10%
1L0.8cm5     1L6cmAcute renal failure syndrome     1L2cm34763     1L2cm16.11%
1L0.8cm6     1L6cmPostpartum finding     1L2cm30382     1L2cm14.08%
1L0.8cm7     1L6cmAnemia     1L2cm24841     1L2cm11.51%
1L0.8cm8     1L6cmUrinary tract infectious disease     1L2cm23831     1L2cm11.05%
1L0.8cm9     1L6cmChest pain     1L2cm22768     1L2cm10.55%
1L0.8cm10     1L6cmDehydration     1L2cm22402     1L2cm10.38%

23cmHome Visit
1L0.8cm1     1L6cmEssential hypertension     1L2cm144     1L2cm45.71%
1L0.8cm2     1L6cmMalaise     1L2cm97     1L2cm30.79%
1L0.8cm3     1L6cmConstipation     1L2cm89     1L2cm28.25%
1L0.8cm4     1L6cmMajor depression, single episode     1L2cm61     1L2cm19.37%
1L0.8cm5     1L6cmCough     1L2cm52     1L2cm16.51%
1L0.8cm6     1L6cmDementia     1L2cm43     1L2cm13.65%
1L0.8cm7     1L6cmDementia with behavioral disturbance     1L2cm42     1L2cm13.33%
1L0.8cm8     1L6cmDisorder due to infection     1L2cm42     1L2cm13.33%
1L0.8cm9     1L6cmSlow transit constipation     1L2cm41     1L2cm13.02%
1L0.8cm10     1L6cmHyperlipidemia     1L2cm40     1L2cm12.70%

1p3cmVisit Type
1p6cmCondition Concepts
1p2cmPatient Count
1p2cmPercentage of the Visit Type Patients

23cmOffice Visit
1L0.8cm1     1L6cmEssential hypertension     1L2cm62118     1L2cm5.83%
1L0.8cm2     1L6cmHyperlipidemia     1L2cm36152     1L2cm5.76%
1L0.8cm3     1L6cmGastroesophageal reflux disease without esophagitis     1L2cm26766     1L2cm5.61%
1L0.8cm4     1L6cmVitamin D deficiency     1L2cm26442     1L2cm4.65%
1L0.8cm5     1L6cmCough     1L2cm25727     1L2cm4.48%
1L0.8cm6     1L6cmPure hypercholesterolemia     1L2cm21321     1L2cm4.38%
1L0.8cm7     1L6cmObesity     1L2cm20555     1L2cm4.23%
1L0.8cm8     1L6cmChronic pain     1L2cm20091     1L2cm4.07%
1L0.8cm9     1L6cmDyspnea     1L2cm19387     1L2cm4.01%
1L0.8cm10     1L6cmFatigue     1L2cm18656     1L2cm3.73%

23cmHealth examination
1L0.8cm1     1L6cmChronic pain     1L2cm369     1L2cm16.83%
1L0.8cm2     1L6cmShoulder joint pain     1L2cm248     1L2cm11.31%
1L0.8cm3     1L6cmMusculoskeletal finding     1L2cm172     1L2cm7.84%
1L0.8cm4     1L6cmLow back pain     1L2cm158     1L2cm7.20%
1L0.8cm5     1L6cmInterstitial lung disease     1L2cm126     1L2cm5.75%
1L0.8cm6     1L6cmPain in right knee     1L2cm124     1L2cm5.65%
1L0.8cm7     1L6cmPain in left knee     1L2cm110     1L2cm5.02%
1L0.8cm8     1L6cmPostoperative state     1L2cm103     1L2cm4.70%
1L0.8cm9     1L6cmDifficulty walking     1L2cm93     1L2cm4.24%
1L0.8cm10     1L6cmLumbago with sciatica     1L2cm89     1L2cm4.06%

23cmEmergency Room Visit
1L0.8cm1     1L6cmAbdominal pain     1L2cm153538     1L2cm13.77%
1L0.8cm2     1L6cmEssential hypertension     1L2cm123530     1L2cm11.08%
1L0.8cm3     1L6cmChest pain     1L2cm99274     1L2cm8.91%
1L0.8cm4     1L6cmViral disease     1L2cm95402     1L2cm8.56%
1L0.8cm5     1L6cmHeadache     1L2cm94671     1L2cm8.49%
1L0.8cm6     1L6cmFever     1L2cm92601     1L2cm8.31%
1L0.8cm7     1L6cmCough     1L2cm89758     1L2cm8.05%
1L0.8cm8     1L6cmAcute upper respiratory infection     1L2cm88679     1L2cm7.96%
1L0.8cm9     1L6cmAcute pharyngitis     1L2cm74356     1L2cm6.67%
1L0.8cm10     1L6cmAsthma     1L2cm73301     1L2cm6.58%