#  NAACL’21
Improving Faithfulness in Abstractive Summarization
with Contrast Candidate Generation and Selection

Sihao Chen1 Fan Zhang2 Kazoo Sone2 Dan Roth1∗
1University of Pennsylvania 2Google
{sihaoc<EMAIL_ADDRESS>{zhanfan<EMAIL_ADDRESS>
Most of the work done while the authors were at Google.

###### Abstract

Despite significant progress in neural abstractive summarization, recent
studies have shown that the current models are prone to generating summaries
that are unfaithful to the original context. To address the issue, we study
contrast candidate generation and selection as a model-agnostic post-
processing technique to correct the extrinsic hallucinations (i.e. information
not present in the source text) in unfaithful summaries. We learn a
discriminative correction model by generating alternative candidate summaries
where named entities and quantities in the generated summary are replaced with
ones with compatible semantic types from the source document. This model is
then used to select the best candidate as the final output summary. Our
experiments and analysis across a number of neural summarization systems show
that our proposed method is effective in identifying and correcting extrinsic
hallucinations. We analyze the typical hallucination phenomenon by different
types of neural summarization systems, in hope to provide insights for future
work on the direction.

##  1 Introduction

Abstractive Summarization is the task of producing a concise and fluent
summary that is salient and faithful to the source document(s). Data-driven,
neural methods Rush et al. (2015); Nallapati et al. (2016); See et al. (2017),
and the more recent, pretrained transformer language models Vaswani et al.
(2017); Devlin et al. (2019); Liu and Lapata (2019), have shown improvements
in the fluency and salience of generated summaries.

However, less progress has been made on improving the faithfulness of the
generated summaries, that is, producing a summary that is entailed by the
information presented in the source document. Despite the increased level of
performance under automatic metrics such as Rouge Lin (2004) or BertScore
Zhang et al. (2020), current state of the art models Liu and Lapata (2019);
Lewis et al. (2020) produce summaries that suffer from intrinsic and extrinsic
hallucinations – the fabrication of untruthful text spans containing
information either present or absent from the source Maynez et al. (2020).

Source: He was re-elected for a second term by the UN General Assembly,
unopposed and unanimously, on 21 June 2011, with effect from 1 January 2012.
Mr. Ban describes his priorities as mobilising world leaders to deal with
climate change, economic upheaval, pandemics and increasing pressures
involving food, energy and water…
---
Unfaithful Summary: The United Nations Secretary-General Ban Ki-moon was
elected for a second term in 2007.
Our Summary: The United Nations Secretary-General Ban Ki-moon was elected for
a second term in 21 June 2011.
Table 1: An example unfaithful summary. It suffers from extrinsic
hallucination, where information not present in the source document was
generated. Our method attempts to correct the unfaithful summary by replacing
"2007" with an entity from the source with compatible semantic type (i.e.
Date).

Table 1 shows an example of such summary, generated by BART Lewis et al.
(2020), an auto-regressive, transformer-based sequence-to-sequence model. The
article describes an event where the former UN-Secretary-General Ban Ki-Moon
was re-elected for a second term. The model hallucinates "2007", which never
appears in the source document, leading to inconsistency with the correct date
of the event presented.

In this work, we focus on the problem of correcting such hallucinations as a
post processing step111Our code and data is available at
http://cogcomp.org/page/publication_view/938. A post processing correction
step allows us to rely on the fluency of the text generated by SOTA systems,
that gain from huge pretrained models and large fine-tuning datasets, and
correct it using small amounts of automatically generated training data.

Under the setting where a large fraction of ground truth summarization data is
hallucinated, as we show in Table 2, we study the method of contrast candidate
generation and selection. In the generation step, we replace named entities in
a potentially hallucinated summary with ones with compatible semantic types
that are present in the source, and create variants of candidate summaries. In
the selection step, we rank the generated candidates with a discriminative
model trained to distinguish between faithful summaries and synthetic negative
candidates generated given the source. We experiment on a range of RNN- and
transformer-based abstractive summarization models. Our preliminary results on
the XSum corpus Narayan et al. (2018a), which contains substantial presence of
hallucinated ground truth examples, show the effectiveness of our method in
correcting unfaithful summaries with extrinsic hallucinations.

Our main contributions are as follows. First, our work is the first to study
the effectiveness of contrast candidate generation and selection as a model-
agnostic method for correcting hallucinations, under the setting where a large
fraction of ground truth summarization data suffers from hallucinations.
Second, we validate our method on various neural summarization systems trained
on XSum, and provide detailed analysis on the typical types of hallucinations
from each system.

##  2 Contrast Candidate Generation & Selection

Our proposed method is built on the observation that a large fraction of
extrinsic hallucinations happen on named entities and quantities. Table 2
shows the human analysis by Maynez et al. (2020) on the hallucinations of
$500$ randomly sampled gold summaries from the XSum corpus . We break down
each category and annotate the proportion of hallucinations that happen on
entity and number/quantity spans.

Type | % | Ent. % | Num. %
---|---|---|---
Faithful | $23.1$ | - | -
Ex. Hallucination | 73.1 | 35.9 | 18.2
In. Hallucination | 7.4 | 1.9 | 0.5
Table 2: Frequency of extrinsic and intrinsic hallucinations in 500 ground
truth summary of the XSum corpus. The “%” column shows the $\%$ of intrinsic
and extrinsic hallucinations annotated by Maynez et al. (2020). We analyzed
the $\%$ of hallucinations on entities and numbers/quantities, and show the
$\%$ out of all 500 summaries in the right two columns.

As Maynez et al. (2020) further show that the hallucinations in training data
translate to similar issues for the generated outputs across different
summarization models, we want to study a model-agnostic, post-processing
method that can correct such entity and quantity hallucinations. We frame the
problem as a correction task and make it conceptually a less complex problem
than summarization. Modeling correction as a standalone task would require
less training data, which becomes crucial when a large proportion of ground
truth summarization data suffer from hallucinations, and inherit the fluency
of data intensive SOTA models.

###  2.1 Contrast Candidate Generation

From a model-generated summary, we first identify any potentially hallucinated
entities or quantities by checking whether entities with similar surface forms
have appeared in the source document. We use a neural Named Entity Recognition
(NER) system from the Stanza NLP toolkit Qi et al. (2020) trained on the
OntoNotes corpus Weischedel et al. (2013) to extract named entities of
different semantic types from the source document and summary. Each named
entity present in the summary is replaced with a different entity present in
the document with the same NER label. This gives us different variants of the
original summary with the same level of fluency , but not necessarily
faithful.

###  2.2 Contrast Candidate Selection

For the candidate selection step, we want to identify the best candidate among
the variants generated in the previous step as the final output summary. As
the contrast candidates vary in no more than a few tokens from the original
summary, it requires a model with more delicate local decision boundaries
Gardner et al. (2020) to select the correct candidate. For example, we observe
that MNLI models Williams et al. (2018) fail to produce satisfactory results.

To create training data for that purpose, we sample examples from the XSum
training set where all entities in the ground truth summary appear in the
source document. We then follow the same procedure in the generation step, and
produce unfaithful variants from the ground truth summary by replacing
entities with others that have the same semantic type but different surface
form in the source text. With the ground truth and synthetic negative
summaries, we train a text classifier with a discriminative objective to score
and rank the variants of the summaries.

We use BART Lewis et al. (2020) plus a linear layer as our classification
model. We adopt a similar learning objective to contrastive learning Khosla et
al. (2020). For each pair of positive and negative summary candidate, we use
cross entropy loss $\mathcal{L}_{\textsc{XE}}$ to handle the correctness of
the label predictions. We add a margin ranking loss term
$\mathcal{L}_{\textsc{Rank}}$ to encourage the model to assign higher
probability to the positive than the negative candidate. The margin $\gamma$
is a tunable hyperparameter in training.

$\displaystyle\mathcal{L}=\mathcal{L}_{\textsc{XE}}(\hat{y_{+}},1)$
$\displaystyle+\mathcal{L}_{\textsc{XE}}(\hat{y_{-}},0)+\mathcal{L}_{\textsc{Rank}}(\hat{y_{+}},\hat{y_{-}})$
$\displaystyle\mathcal{L}_{\textsc{Rank}}$
$\displaystyle=max(0,\hat{y_{-}}-\hat{y_{+}}+\gamma)$

During test time, we use the trained model to score the generated contrast
candidate summaries, as well as the original version generated by the
summarization model. We take the candidate with the highest score as the final
summary.

##  3 Experiments

Full XSum Test Set
---
Method | RougeL | Bert | Feqa (%)
BARTlarge | 36.95 | 91.57 | -
\+ correct | 36.70 | 91.50 | -
Changed Summary Only (13.3%)
BARTlarge | 38.63 | 91.61 | 22.50
\+ correct | 36.62 | 91.10 | 25.62
Table 3: Evaluation with automatic metrics on the summaries generated by the
baseline BARTlarge model, plus our post-processing correction method. We
report $F_{\beta=1}$ scores with Rouge and BertScore, plus the macro-averaged
percentage of questions answered correctly for each summary with Feqa, a QA-
based metric for summary faithfulness proposed by Durmus et al. (2020).

Our experiments focus on the aforementioned XSum corpus, where the target
summary is highly abstractive and likely hallucinated. We first consider the
summaries generated by a BART model trained on the XSum corpus. By applying
our method, we are able to change $13.3\%$ of all model generated summaries.
For $38.4\%$ of all summaries, the original summary does not have a
hallucinated entity, or there is no entity with compatible type in the source
text. Our model decides to keep the original summary in the rest $48.3\%$.

###  3.1 Rouge and BertScore Evaluation

We first verify that our method does not hurt the fluency and salience of the
generated summaries, for which we assume Rouge Lin (2004) and BertScore Zhang
et al. (2020) are suitable metrics. We report the results in Table 3. We
observe though both the baseline and our method do well in both Rouge and
BertScore, our method trails behind in both metrics slightly. This is due to
the existence of extrinsic hallucinations in the ground truth summary, and the
model manages to generate a part of the hallucinations, and gets incorrectly
rewarded by such.

###  3.2 Faithfulness Evaluation

Method | Faith. % | Ex. % | In. %
---|---|---|---
BART | 23.8$\pm{9.6}$ | 71.7$\pm{11.2}$ | 1.7$\pm{3.5}$
\+ correct | 59.5$\pm{12.4}$ | 9.2$\pm{7.3}$ | 29.1$\pm{11.6}$
Table 4: Percentage of examples human annotator judged as “faithful” (Faith.),
“extrinsically hallucinated” (Ex.), and “intrinsically hallucinated” (In.)
among the 95 randomly sampled corrected summaries. The $95\%$ confidence
intervals are estimated with bootstrap resampling (Appendix. C).

To test whether our correction method can improves the faithfulness of the
summaries, we evaluate the summaries with Feqa Durmus et al. (2020), a QA-
based metric for summary faithfulness. Given a summary, Feqa automatically
generates questions on noun phrase and named entity spans in the summary, and
uses a pretrained QA model to verify if the answer derived from the source
document exact-matches the span in the summary.

We run Feqa and compute the macro-averaged percentage of questions answered
correctly for each of the $1510$ summaries that our system made corrections
to, and report the results in Table 3. The results suggest that the corrected
summaries present statistically significant improvements over the original
ones ($p<0.001$, with a two-tailed, paired t-test).

Good Corrections
---
Type |  System |  Original Summary and Our Change
Correcting NE Hallucination |  BertS2S |  Tranmere Rovers have signed midfielder [Alfreton]${}_{\textsc{PER}}$ $\rightarrow$ [Mooney]${}_{\textsc{PER}}$ on loan until the end of the season.
Correcting Number Hallucination |  Bart |  A judge has ruled that the [$9.6bn (£5.03bn)]${}_{\textsc{Money}}$ $\rightarrow$ [$7.8bn (£5.03bn)]${}_{\textsc{Money}}$ oil spill compensation fund is not fraudulent.
Typical Mistakes
No correct replacement exists in source |  Bart |  Helmut Kohl, who has died at the age of [87]${}_{\textsc{Cardinal}}$ $\rightarrow$ [39]${}_{\textsc{Cardinal}}$, was one of the driving forces behind Germany’s reunification in 1990.
Wrong type of NE in summary |  TranS2S |  [Andrew Marr]${}_{\textsc{PER}}$ $\rightarrow$ [Venter]${}_{\textsc{PER}}$ is one of the most important scientific discoveries in human life.
Not explicit in source, but can be inferred |  BertS2S |  [Three]${}_{\textsc{Cardinal}}$ $\rightarrow$ [Two]${}_{\textsc{Cardinal}}$ fugitives have been arrested and charged with attempting to smuggle drugs into the country.
Table 5: Examples of corrections and typical mistakes made by our proposed method on generated summaries by different summarization models. The original and replaced entities in each summary are highlighted, and are colored by their faithfulness categories (Red: Extrinsic Hallucation; Orange: Intrinsic Hallucation; Blue: Faithful)  System | $P$ | $R$ | $F_{1}$ | Ent. %
---|---|---|---|---
PtGen | 79.86 | 58.38 | 67.45 | 65.48
TConvS2S | 87.76 | 61.87 | 72.57 | 64.27
TranS2S | 81.81 | 57.35 | 67.44 | 80.15
BertS2S | 80.54 | 37.82 | 51.47 | 56.85
Table 6: The selection model’s precision, recall and $F_{1}$ on identifying
hallucinated output from four different summarization systems. The Ent. %
column shows the $\%$ of hallucinations on entities and quantities among all
hallucinated summaries by each system.

Table 4 shows the human evaluation results on the 95 randomly sampled subset
of changed summaries. Two expert annotators assign each summary into three
faithfulness categories and adjudicate the decisions. Additional annotations
from a third expert is then used to calculate the inter-annotator agreement.
As the results show, our model is able to improve the faithfulness of the
summaries, but at the cost of incurring intrinsic hallucinations on mistakes,
which we will discuss more in detail in section 4.2.

##  4 Analysis and Discussion

###  4.1 Identifying Hallucination Across Systems

Table 6 shows our selection model’s performance when measuring P, R, F1 w.r.t
all the hallucinated instances. We use the test set from Maynez et al. (2020),
who have annotated hallucination categories of generated summaries from four
neural summariazaiton models: PtGen See et al. (2017) TConvS2S Narayan et al.
(2018a), BertS2S and TranS2S Rothe et al. (2020). Our system achieves
consistently high level of precision across models. The system achieves high
relative recall with respect to the $\%$ of entity and quantity hallucinations
among all hallucinations. As our method only targets entities and quantities,
the overall recall varies by the typical type of hallucinations each
summarization system makes. We also observe while our method achieves high
recall on models with lower Rouge and BertScore, the recall drops on
pretrained models such as BertS2S. This is potentially due to the decreased
percentage of entity/quantity hallucinations exist in generated summaries from
the models with pretraining.

###  4.2 Intrinsic vs. Extrinsic Hallucinations Trade-off

As our method detects and corrects extrinsic-hallucinated entities, naturally
any entities replaced wrong would introduce intrinsic hallucinations in the
changed summary, as indicated by the results in Table 4. To speculate why the
mistakes happen, we analyzed the typical mistakes by the model, and listed a
few representative examples in Table 5. For example, our method could not find
the correct replacement for a hallucinated entity when no such one exists in
the source text. We observe that the models with pretraining, such as BertS2S,
Rothe et al. (2020) and Bart, suffer from the issue by most, as they tend to
be affected by artifacts/priors from the pretraining process.

###  4.3 Entity Faithfulness $\subsetneqq$ Summary Faithfulness

From the observation that models often hallucinate entities with no correct
replacement in the source, we suspect that solving entity faithfulness alone
does not guarantee the faithfulness of the summary. In the last example from
Table 5, the BertS2S system correctly identifies that three fugitives are
involved in the event described by the source text, even though the number
"three" has never been explicitly mentioned in the source context in any
surface forms. Furthermore, statistics provided by Maynez et al. (2020) show
that abstractive summarization models often produces factual statements, i.e.
verifiable in the real world independent of the source text. Such findings
imply that identifying hallucinations often requires more complex objectives
such as commonsense reasoning and knowledge retrieval. The solution we propose
here that focuses only on entites and quantities would likely be insufficient
to solve the entire problem.

##  5 Related Work

There have been growing interests in quantitatively measuring the faithfulness
of text generation models. Most widely-adopted evaluation metrics for text
generation, such as ROUGE Lin (2004) and BERTScore Zhang et al. (2020),
correlate poorly with the human perceived faithfulness of the generated text
Kryscinski et al. (2019); Durmus et al. (2020). Recent studies explore
categorical, content-based analysis for measuring the faithfulness of
summaries Goyal and Durrett (2020); Deutsch and Roth (2020). Narayan et al.
(2018b); Deutsch et al. (2020); Durmus et al. (2020) propose to use question
answering to test the consistency of summary content to the information
presented in the source text.

There have been efforts to study pre- or post- processing methods to improving
faithfulness of generated summaries. Falke et al. (2019) attempt to use
textual entailment models to re-rank the summary candidates generated from
beam search or different neural systems. As Maynez et al. (2020) highlight the
existence of hallucinations in training data, truncating potentially
unfaithful gold summaries during training is an effective strategy Kang and
Hashimoto (2020); Filippova (2020). Kryscinski et al. (2020) take similar
apporach as in this work to identify the hallucinations in summary. A
concurrent study to this work Cao et al. (2020) uses similar strategies as in
this paper on a dataset with a very small fraction of hallucinations present.
Our study instead focuses on the more challenging setting Goyal and Durrett
(2021) where a large part of training data suffers from extrinsic and
intrinsic hallucinations, and provides cross-system analysis on the both
hallucinations categories.

##  6 Conclusion

We study contrast candidate generation and selection as a method to apply
post-hoc fixes to extrinsically hallucinated summary on entities and
quantities, under the setting where the summarization dataset suffers from
intrinsic and extrinsic hallucinations. We conduct our experiments on the XSum
dataset, and show that our method is able to correct extrinsic hullucinations,
but incurs a small fraction of intrinsic hallucinations on mistakes. We also
provide detailed analysis and discussions on the capabilities and limitations
of our method. We hope our findings in the paper will provide insights to
future work in this direction.

## Acknowledgments

We thank Sunita Verma and Sugato Basu for valuable input and feedback on
drafts of the paper. This work was supported in part by a Focused Award from
Google, a gift from Tencent, and by Contract FA8750-19-2-1004 with the US
Defense Advanced Research Projects Agency (DARPA). The views expressed are
those of the authors and do not reflect the official policy or position of the
Department of Defense or the U.S. Government.

## References

  * Cao et al. (2020) Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020.  Factual error correction for abstractive summarization models.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251–6258, Online. Association for Computational Linguistics.
  * Deutsch et al. (2020) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2020.  Towards question-answering as an automatic metric for evaluating the content quality of a summary.  arXiv preprint arXiv:2010.00490.
  * Deutsch and Roth (2020) Daniel Deutsch and Dan Roth. 2020.  Understanding the extent to which summarization evaluation metrics measure the information quality of summaries.  arXiv preprint arXiv:2010.12495.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055–5070, Online. Association for Computational Linguistics.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Filippova (2020) Katja Filippova. 2020.  Controlled hallucinations: Learning to generate faithfully from noisy data.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 864–870, Online. Association for Computational Linguistics.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating models’ local decision boundaries via contrast sets.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics.
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating factuality in generation with dependency-level entailment.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592–3603, Online. Association for Computational Linguistics.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In Proceedings of the 2021 Proceedings Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  * Kang and Hashimoto (2020) Daniel Kang and Tatsunori Hashimoto. 2020.  Improved natural language generation via loss truncation.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718–731, Online. Association for Computational Linguistics.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  Proceedings of the 34th Conference on Neural Information Processing Systems.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721–3731.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar GuÌ‡lçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Narayan et al. (2018a) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018a.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807.
  * Narayan et al. (2018b) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b.  Ranking sentences for extractive summarization with reinforcement learning.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747–1759, New Orleans, Louisiana. Association for Computational Linguistics.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A Python natural language processing toolkit for many human languages.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
  * Rothe et al. (2020) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020.  Leveraging pre-trained checkpoints for sequence generation tasks.  Transactions of the Association for Computational Linguistics, 8:264–280.
  * Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389.
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In Advances in Neural Information Processing Systems, pages 5998–6008.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013.  Ontonotes release 5.0.  Linguistic Data Consortium, Philadelphia, PA, 23.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  BERTScore: Evaluating Text Generation with BERT.  In International Conference on Learning Representations.

##  Appendix A Candidate Selection Model

For our contrast candidate selection model, we use a pretrained BART base
model. We add a linear layer over the max pooled embedding, and the
classification model is expected to output a label between ["Faithful",
"Hallucinated"].
For all our experiments, we use the following set of hyper-parameters:
$r=1e-5$, margin $\gamma=0$, number of training epoch$=3$.

##  Appendix B Complete Rouge and BertScore Results

Full XSum Test Set
---
Method | $R_{1}$ | $R_{2}$ | $R_{L}$ | Bert
BARTlarge | 45.10 | 21.86 | 36.95 | 91.57
\+ correct | 44.82 | 21.49 | 36.70 | 91.50
Changed Summary Only (13.3%)
BART | 46.73 | 23.51 | 38.63 | 91.61
\+ correct | 44.35 | 20.70 | 36.62 | 91.10
Table 7: Rouge{1,2,L} and BertScore evaluation results (in $F_{1}$) of
summaries generated by the baseline BARTlarge model, plus the corrected
summaries with our post-processing method, on the test set of the XSum corpus.

##  Appendix C Estimating Confidence Interval for Human Evaluation

We use bootstrapping to estimate the confidence interval for the expert
annotation presented in Table 4. For each faithfulness category on the two
systems, we regard the adjudicated annotation as ground truth, and label the
individual instance as the true positive (TP), false negative (FN), true
negative (TN) and false positive (FP) respectively according the annotations
from the third expert. We re-sample the 95 instances with replacement for
1,000 times. We estimate the adjusted mean and $95\%$ confidence interval from
the mean and standard deviation of the sampled distribution of (TP \+ FN).


# Generalizing over Long Tail Concepts for Medical Term Normalization

Beatrice Portelli1,2∗ Simone Scaboro1∗ Enrico Santus3†
Hooman Sedghamiz3 Emmanuele Chersoni4 Giuseppe Serra1
1 University of Udine, Italy 2 University of Naples Federico II, Italy 3 DSIG
- Bayer Pharmaceuticals, New Jersey, USA 4 The Hong Kong Polytechnic
University, Hong Kong<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Medical term normalization consists in mapping a piece of text to a large
number of output classes. Given the small size of the annotated datasets and
the extremely long tail distribution of the concepts, it is of utmost
importance to develop models that are capable to generalize to scarce or
unseen concepts. An important attribute of most target ontologies is their
hierarchical structure. In this paper we introduce a simple and effective
learning strategy that leverages such information to enhance the
generalizability of both discriminative and generative models. The evaluation
shows that the proposed strategy produces state-of-the-art performance on seen
concepts and consistent improvements on unseen ones, allowing also for
efficient zero-shot knowledge transfer across text typologies and datasets.

**footnotetext: Equal contribution††footnotetext: The author was affiliated
with Bayer Pharmaceuticals at the time of the experiments, and is currently
affiliated with Bloomberg.

##  1 Introduction

Term normalization is the task of mapping a variety of natural language
expressions to specific concepts in a dictionary or an ontology. It is a key
component for information processing systems, and it is extensively used in
the medical domain. In this context, term normalization is often used to map
reported adverse events (AEs) related to a drug to a medical ontology, such as
MedDRA (Brown et al., 1999). This is a challenging task, due to the high
variability of natural language input (i.e., from the informality of social
media and conversational transcripts to the formality of medical and legal
reports) and the high cardinality and long tail distribution of the output
concepts. AEs are usually mappable to different levels of the same ontology:
low-level concepts, which are closer to layman terms, and higher level
concepts, which encompass the meaning of multiple low-level concepts. In
MedDRA,111MedDRA is a five-level hierarchy https://www.meddra.org/how-to-
use/basics/hierarchy, but in this work we mainly focus on two of the levels:
PT and LLT. these two sets of concepts are called Lowest Level Terms (LLT),
and Preferred Terms (PT) respectively; both of them have a very high
cardinality (48,713 for LLT and 24,571 for PT, in MedDRA version 23.1). The
following are examples of AEs, with their corresponding LLTs and PTs:

AE | LLT | PT
---|---|---
feel like crap | feeling unwell | malaise
weak knees | weakness | asthenia
zap me of all energy | loss of energy | asthenia

Currently this problem is addressed with large pretrained language models
(Gonzalez-Hernandez et al., 2020), finetuned on medical term normalization
datasets, such as SMM4H (Gonzalez-Hernandez et al., 2020) or CADEC (Karimi et
al., 2015). However, these datasets contain maximum 5,000 samples, distributed
on a few PT/LLT classes, and with a long tail distribution (see Figure 1). Due
to the size and distribution of these datasets, the resulting models usually
perform well on examples that are seen in the training, but struggle to
generalize on rare or unseen samples.

Figure 1: Long tail distribution of PTs in the datasets used for this paper.

To improve the generalization capabilities of the models on long tail
concepts, in this paper, we suggest to leverage the hierarchical nature of the
medical ontology to enrich the large language models with domain knowledge
before finetuning on a given training set. Extensive experimental evaluation
on three different datasets shows that the proposed strategy can be
successfully applied to various model typologies, and it consistently
outperforms other mainstream learning strategies, showing generalization
capabilities not only across the long tail distribution, but also across text
typologies and datasets. The code and resources needed to replicate our
experiments and test our learning strategy are publicly
available222https://github.com/AilabUdineGit/ontology-pretraining-code.

##  2 Related Work

Medical term normalization is generally regarded as either a classification or
a ranking problem (Yuan et al., 2022). In the former case, a neural
architecture encodes the term into a hidden representation and outputs a
distribution over the classes (Limsopatham and Collier, 2016; Tutubalina et
al., 2018; Niu et al., 2019), but this is difficult to scale to ontologies
containing thousands of concepts, due to the absence of comprehensive
datasets. In the ranking approach, on the other hand, the goal is to rank
concepts by their similarity to the input term (Leaman et al., 2013; Li et
al., 2017; Sung et al., 2020): a system is trained on binary classification,
where terms and matching concepts are the positive samples, while terms and
non-matching concepts are the negative ones. The raw output of the model is
then used to rank the concepts.

Recent work successfully combined the two approaches. Ziletti et al. (2022)
presented a system mixing a BERT-based classifier (Devlin et al., 2019) and a
zero/few-shot learning method to incorporate label semantics in the input
instances (Halder et al., 2020), showing improved performance in single model
and in ensemble settings.

Finally, systems like CODER (Yuan et al., 2022) and SapBERT (Liu et al., 2021)
introduced novel contrastive pretraining strategies that leverage UMLS to
improve the medical embeddings of BERT-based models. While SapBERT leverages
self-alignment methods, CODER maximizes the similarities between positive
term-term pairs and term-relation-term triples and it claimed state-of-the-art
results on several tasks, including zero-shot term normalization. Another
recent work by Zhang et al. (2021) introduced an even more extensive
pretraining procedure, based on self-supervision and a combination of the
traditional masked language modelling with contrastive losses . The strategy
proved to be extremely effective for medical entity linking, a kind of term
normalization which makes use of the full original context (instead of using
only the AE).

##  3 Proposed Learning Strategy: OP+FT

Dataset |  Total Samples |  Train Samples |  Test Samples |  %out Samples |  Unique PTs
---|---|---|---|---|---
SMM4H | 1,442 | 868 ±06 | 574 ±06 | 12.49 ±0.94 | 274
CADEC | 5,866 | 3,540 ±21 | 2,326 ±21 | 4.65 ±0.41 | 488
PROP | 4,453 | 2,658 ±65 | 1,796 ±65 | 10.02 ±0.79 | 634

Table 1: Dimensions of the datasets, reporting the average figures over the
three train/test splits (± std), as well as the number of unique PT terms
contained in each dataset.

Let’s consider a target ontology (e.g. MedDRA v23.1) containing two sets of
concepts $\mathit{PT}=\\{p_{i}\\}$ and the $\mathit{LLT}=\\{\ell_{i}\\}$. The
ontology is structured so that every $\ell_{i}$ has only one parent $p_{j}$:
$\mathit{parent}(\ell_{i})=p_{j}$, but each $p_{j}$ can be parent of many
$\ell_{i}$. Given a set of Adverse Events $\mathit{AE}=\\{a_{i}\\}$, every
$a_{i}$ can be univocally mapped to a $p_{j}$: $\mathit{norm}(a_{i})=p_{j}$.

Our objective is to train a large language model $\mathcal{M}$ to encode
$\mathit{norm}$: given a sample $(a_{i},p_{j})$, such that
$\mathit{norm}(a_{i})=p_{j}$, we want $\mathcal{M}(a_{i})=p_{j}$.

We propose a learning strategy based on the hierarchical structure of the
ontology, composed of two steps: Ontology Pretraining and Finetuning.

During the first step, we expose the language model $\mathcal{M}$ to all
possible output classes $p_{j}$ by leveraging the intrinsic hierarchical
relation between LLT and PT. Specifically, we use the $\mathit{parent}$
relation to create a new set of training samples from the ontology, pairing
each $\ell_{i}$ to its parent concept $p_{j}$. In the case of MedDRA, the new
set of samples contains 48,713 $(\ell_{i},p_{j})$ pairs, where each $p_{j}$
appears multiple times, associated with different $\ell_{i}$. For example the
PT “asthenia” will appear in the samples (weakness, asthenia), and (loss of
energy, asthenia). As LLTs are more informal than PTs, the language model
$\mathcal{M}$ can be pretrained on this new set of data to gain general
knowledge about all the target classes. This pretraining set is highly similar
to our target dataset ($\mathcal{M}(a_{i})=p_{j}$), increasing the model
transfer capability. We call this process “Ontology Pretraining” (OP).

The second step consists in finetuning (FT) an OP model on a specific term
normalization dataset, which maps every AE $a_{i}$ to the corresponding PT
$p_{j}$. This step is crucial because the OP model lacks specific knowledge
about the natural language style of real-world samples. Finetuning will also
exploit the dataset sample distribution to boost the model’s accuracy on the
specific set of $p_{j}$ in the training set. Note that the FT step can also be
applied to a regular model without OP, resulting in a regular finetuning.

We hypothesize that the combination of OP+FT to a discriminative or a
generative language model $\mathcal{M}$ will improve its performance on seen
concepts in the training set, while making it more generalizable to long tail
and unseen concepts.

##  4 Experimental Setting

###  4.1 Datasets

To investigate the performance of our learning strategy, we used three English
datasets for MedDRA term normalization, with different writing styles.

SMM4H (Gonzalez-Hernandez et al., 2020). Public dataset for the challenge
SMM4H 2020 - Task 3, AE normalization. It contains 2,367 tweets, 1,212 of
which report AEs with highly informal language, mapped to a PT/LLT.

CADEC (Karimi et al., 2015). Public dataset containing 1,250 posts from the
health forum “AskaPatient”, containing user-reported AEs mapped to a PT/LLT.
The language is informal, but still more medically precise than SMM4H.

PROP. Proprietary dataset provided by Bayer Pharmaceuticals, containing 2,010
transcripts of phone-calls with healthcare professionals reporting their
patients’ AEs, mapped to PTs. The language is more formal and medically
accurate.

###  4.2 Data Preparation

All datasets were preprocessed to obtain samples containing only
$(a_{i},p_{j})$ pairs. The samples in CADEC and SMM4H which were labelled with
an $\ell_{i}\in\mathit{LLT}$ were re-labelled with
$\mathit{parent}(\ell_{i})=p_{j}\in\mathit{PT}$, obtaining a uniform output
space for all datasets containing only PT concepts.

Since the focus of this work is on the generalization capabilities of the
models, it is important to test the models on different sets of unseen labels.
For this reason, we created three random splits of train/test samples using a
60:40 proportion, instead of using the public fixed train/test split. Given a
train and a test set, every test sample with label $p_{j}$ falls into one of
the following categories:
– in, if $p_{j}$ is present in the training set;
– out, if $p_{j}$ is not present in the training set.

The most important set of samples to measure the generalization capabilities
of the models is out.

Table 1 reports figures for the resulting datasets. CADEC and PROP contain the
largest number of samples (5,866 and 4,453 respectively), while SMM4H is
sensibly smaller, with only 1,442 samples. The largest datasets also contain
the largest number of PTs: 488 for CADEC and 634 for PROP. SMM4H only contains
274 PTs instead. Most of the PTs are unique to one of the three datasets and
do not appear in the other ones, making it impossible to gain a substantial
advantage by combining them (see Appendix C). We observe that the percentage
of out samples varies from 5% to 12%, with SMM4H being the most challenging
dataset. The standard deviation is low, showing that the presence of 5–12% out
samples is a characteristic of the specific dataset, resulting from its long
tail PT distribution. Note also that the smaller the dataset, the higher the
percentage of out samples in the test set.

###  4.3 Models

To test the proposed strategy and observe how it affects generalization, we
selected different kinds of widely-adopted models. In particular, we compare
PubMedBERT (Gu et al., 2020), Sci5 (Phan et al., 2021), GPT-2 (Radford et al.,
2019), CODER (Yuan et al., 2022) and SapBERT (Liu et al., 2021).

PubMedBERT (PMB). It was chosen as an example of a BERT-based classifier due
to its medical pretraining (PubMed articles) and strong performance in other
medical tasks (Gu et al., 2020; Portelli et al., 2021; Scaboro et al., 2021,
2022).

GPT-2 and Sci5. GPT-2 was selected as an example of a general-purpose
autoregressive language model for text generation, while Sci5 was chosen for
its medical pretraining, performed on the same kind of texts as PMB. The
models were trained to generate a PT, given an input prompt containing the
adverse event.

CODER and SapBERT (SapB). To the best of our knowledge, CODER and SapBERT are
some of the best dataset-agnostic models for medical term embeddings. They
were both trained on the UMLS ontology (Bodenreider, 2004), which is a super-
set of MedDRA, and tested on several term normalization datasets, showing
promising results. Following both original papers, we use CODER and SapBERT to
generate embeddings for $a_{i}$ and for all $p_{j}\in\mathit{PT}$. We then
select as prediction the $p_{j}$ that minimizes the cosine similarity with
$a_{i}$.

We also trained both models according to our proposed strategy. Both models
were trained using the contrastive settings described in their paper and the
respective codebases333CODER: https://github.com/GanjinZero/CODER
SapBERT: https://github.com/cambridgeltl/sapbert .

See Appendix A for training details for all models and B for more details on
the contrastive training of CODER and SapBERT.

Performance is assessed with the Accuracy metric, but we also report the F1
Score in Appendix D, as it can give more insights when classes are unbalanced.

##  5 Experimental Results

In an ablation-study fashion, we compare the OP+FT learning strategy with its
two components: OP and FT. Figure 2 contains the results for all the tested
models and training strategies, and is organized as follows. We display a plot
for each dataset, reporting the accuracy of the models on in samples
($\circ$), out samples ($\diamond$) and the whole test set ($\bullet$). The
first column shows the performance of a basic CODER and SapBERT model without
any additional training. We consider their accuracy on out ($\diamond$) as our
generalization goal, and plot them as solid lines across the chart. The
following three columns display the performance of all the models, trained
with one of the learning strategies (FT, OP and OP+FT respectively). For
tabular results, see Appendix D.

CODER and SapBERT on their own proved to be strong baselines across the three
datasets. Looking at the first column, they reach 40–50% accuracy on CADEC and
SMM4H (overall, in and out, see solid lines), and around 15–20% overall
accuracy ($\bullet$) on PROP.

All learning strategies seemed to be ineffective on CODER: its performance
(gray markers) remains roughly the same across all strategies (FT, OP or
OP+FT). A possible explanation for this behaviour is that CODER embeddings are
already in an optimal state according to the training objectives, as they have
been trained on a very similar task. In fact, CODER generates predictions
using the similarity between the embeddings, and the stable performance
indicates that there were no drastic changes in the structure of the embedding
space.

A clearer effect of the training strategies can be seen on SapBERT (lilac
markers), although it is still limited when compared with the other models.
SapBERT embeddings are probably more subject to adjustments compared to CODER
because the latter was trained for significantly more steps and using more
objective functions, leading to less-adaptable embeddings.

The following observations apply the other three models: PMB, GPT-2 and Sci5.

Figure 2: Accuracy of all models on the three datasets on in ($\circ$), out
($\diamond$) and all ($\bullet$) samples.

The FT strategy (second column), as expected, works really well for in
($\circ$) samples: on CADEC the in accuracy of all models is over to 80%,
while it is close to 50% for the other two datasets. However, the out accuracy
($\diamond$) is lower than 20% in all cases (significantly lower than the
solid line), and reaches 0% for PMB, showing that finetuning alone is not
sufficient for classifiers to generalize on out samples in this setting.

The OP strategy (third column), brings the out accuracy of all models on pair
with the CODER/SapBERT baselines, while the in/overall accuracy matches or
surpasses them. Comparing OP with FT, we see that the overall accuracy
($\bullet$) of the model is generally lower for OP. However, the performance
on out ($\diamond$) samples doubles for generative models, and jumps from 0 to
40% for PMB. This shows that the first step of our proposed learning strategy
has the desired effect, as it improves the models’ understanding of all the
output classes.

Finally, looking at the models trained with the OP+FT strategy (fourth
column), we see that they outperform the FT ones on overall and in accuracy.
The effect is particularly strong on the SMM4H dataset (cf. PMB FT, 44% and
PMB OP+FT, 70%). At the same time, the performance on out ($\diamond$) samples
remains similar to the OP models and close to the CODER baseline (gray solid
line). The only exception is CADEC, where the performance on out is in-between
the baseline and the accuracy with FT only. This shows that the proposed OP+FT
learning strategy can successfully improve the generalization capabilities of
various language models, while also improving their overall performance.

Figure 3: Cross-dataset accuracy for GPT-2 (FT and OP+FT) on out ($\diamond$)
and all ($\bullet$) samples. One plot for each test dataset; the x-axis
reports the training dataset.

We further test the generalization of OP+FT models in zero-shot, cross-dataset
term normalization, normalizing the terms of each dataset with models that
have been trained on one of the other two. Figure 3 shows the accuracy of
GPT-2, with a plot for each test dataset, different training datasets on the
x-axis, and one column for each learning strategy (FT or OP+FT). The behaviour
of the other models is similar (see Appendix D). In all columns, we observe a
drop in overall accuracy ($\bullet$) between the first data point and the
following ones (e.g., cf. CADEC trained on CADEC and CADEC trained on SMM4H).
However, this drop is larger for FT models than OP+FT ones (e.g., 35 vs. 20
points on CADEC). In addition, the out accuracy of OP+FT models remains high
regardless of the training set. This shows that OP+FT models generalize better
than FT models across-dataset. Note that generalization is still challenging
when moving from a dataset with highly informal language to a formal one (see
PROP trained on SMM4H).

##  6 Conclusions

In this paper, we shed some light on the importance of generalization for
medical term normalization models. We showed that AE normalization models
trained with traditional finetuning, despite showing high accuracy on
leaderboards, have low generalization capabilities due to the long tail
distribution of the target labels. Our proposed training strategy (OP+FT),
which leverages the hierarchical structure of the ontology, outperforms
traditional models, while also obtaining state-of-the-art results in
generalization on out samples. This was also demonstrated in a zero-shot
normalization setting. OP+FT showed improvements on discriminative and
generative language models, while it seems to be less effective on models
trained with contrastive losses. This promising technique could also be
applied to other tasks with massive output spaces organized in a hierarchical
manner.

## Limitations

The proposed learning strategy was tested only for the task of medical term
normalization (from adverse events to MedDRA concepts). However, it would be
interesting to test its effectiveness on other term normalization tasks beyond
MedDRA mapping and outside of the medical domain.

Even restricting the problem to medical term normalization, and using datasets
with different text styles, we only focused on English texts. Medical
ontologies such as MedDRA and UMLS are released in multiple languages, and the
research community is moving towards multi-lingual approaches. In the future,
we plan to extend this strategy to other languages (such as Spanish and
Chinese) and to test the models’ capacity to perform crosslingual transfer in
zero-shot scenarios.

## Acknowledgements

The authors thank Juergen Dietrich, Senior Lead Data Scientist at Bayer
Pharmaceuticals, for the help in the creation and annotation of the PROP
dataset. Thanks also to the three anonymous reviewers for their insightful
comments.

## References

  * Bodenreider (2004) Olivier Bodenreider. 2004.  The Unified Medical Language System (UMLS): Integrating Biomedical Terminology.  _Nucleic Acids Research_ , 32:D267–70.
  * Brown et al. (1999) Elliot G Brown, Louise Wood, and Sue Wood. 1999.  The Medical Dictionary for Regulatory Activities (MedDRA).  _Drug Safety_ , 20(2):109–117.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of NAACL_.
  * Gonzalez-Hernandez et al. (2020) Graciela Gonzalez-Hernandez, Ari Z. Klein, Ivan Flores, Davy Weissenbacher, Arjun Magge, Karen O’Connor, Abeed Sarker, Anne-Lyse Minard, Elena Tutubalina, Zulfat Miftahutdinov, and Ilseyar Alimova. 2020.  Proceedings of the COLING Social Media Mining for Health Applications Workshop & Shared Task.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing.  _arXiv preprint arXiv:2007.15779_.
  * Halder et al. (2020) Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020.  Task-aware Representation of Sentences for Generic Text Classification.  In _Proceedings of COLING_.
  * Karimi et al. (2015) Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chenchen Wang. 2015.  Cadec: A Corpus of Adverse Drug Event Annotations.  _Journal of Biomedical Informatics_ , 55:73–81.
  * Leaman et al. (2013) Robert Leaman, Rezarta Islamaj Doğan, and Zhiyong Lu. 2013.  DNorm: Disease Name Normalization with Pairwise Learning to Rank.  _Bioinformatics_ , 29(22):2909–2917.
  * Li et al. (2017) Haodi Li, Qingcai Chen, Buzhou Tang, Xiaolong Wang, Hua Xu, Baohua Wang, and Dong Huang. 2017.  CNN-based Ranking for Biomedical Entity Normalization.  _BMC Bioinformatics_ , 18(11):79–86.
  * Limsopatham and Collier (2016) Nut Limsopatham and Nigel Collier. 2016.  Normalising Medical Concepts in Social Media Texts by Learning Semantic Representation.  In _Proceedings of ACL_.
  * Liu et al. (2021) Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021\.  Self-Alignment Pretraining for Biomedical Entity Representations.  pages 4228–4238.
  * Niu et al. (2019) Jinghao Niu, Yehui Yang, Siheng Zhang, Zhengya Sun, and Wensheng Zhang. 2019.  Multi-task Character-level Attentional Networks for Medical Concept Normalization.  _Neural Processing Letters_ , 49(3):1239–1256.
  * Phan et al. (2021) Long N. Phan, James T. Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. 2021.  SciFive: A Text-to-text Transformer Model for Biomedical Literature.  _arXiv preprint arXiv:2106.03598_.
  * Portelli et al. (2021) Beatrice Portelli, Edoardo Lenzi, Emmanuele Chersoni, Giuseppe Serra, and Enrico Santus. 2021.  BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection.  In _Proceedings of EACL_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.  _OpenAI Blog_.
  * Scaboro et al. (2021) Simone Scaboro, Beatrice Portelli, Emmanuele Chersoni, Enrico Santus, and Giuseppe Serra. 2021.  NADE: A Benchmark for Robust Adverse Drug Events Extraction in Face of Negations.  In _Proceedings of the EMNLP Workshop on Noisy User-Generated Text_.
  * Scaboro et al. (2022) Simone Scaboro, Beatrice Portelli, Emmanuele Chersoni, Enrico Santus, and Giuseppe Serra. 2022.  Increasing Adverse Drug Events Extraction Robustness on Social Media: Case Study on Negation and Speculation.  _arXiv preprint arXiv:2209.02812_.
  * Sung et al. (2020) Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020.  Biomedical Entity Representations with Synonym Marginalization.  In _Proceedings of ACL_.
  * Tutubalina et al. (2018) Elena Tutubalina, Zulfat Miftahutdinov, Sergey Nikolenko, and Valentin Malykh. 2018\.  Medical Concept Normalization in Social Media Posts with Recurrent Neural Networks.  _Journal of Biomedical Informatics_ , 84:93–102.
  * Yuan et al. (2022) Zheng Yuan, Zhengyun Zhao, Haixia Sun, Jiao Li, Fei Wang, and Sheng Yu. 2022.  CODER: Knowledge-infused Cross-lingual Medical Term Embedding for Term Normalization.  _Journal of Biomedical Informatics_ , 126:103983.
  * Zhang et al. (2021) Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Knowledge-Rich Self-Supervised Entity Linking.  _CoRR_ , abs/2112.07887.
  * Ziletti et al. (2022) Angelo Ziletti, Alan Akbik, Christoph Berns, Thomas Herold, Marion Legler, and Martina Viell. 2022.  Medical coding with biomedical transformer ensembles and zero/few-shot learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track_ , pages 176–187, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.

##  Appendix A Training Specifications

Table 2 contains the specifics for the number of Ontology Pretraining (OP) and
Finetuning (FT) used for all the selected models.

Model |  OP epochs |  FT epochs |  OP+FT epochs
---|---|---|---
PMB | 30 | 10 | 30 + | 5
GPT-2 | 30 | 15 | 30 + | 10
Sci5 | 40 | 15 | 40 + | 8
CODER | 50 | 20 | 50 + | 15
SapBERT | 30 | 15 | 30 + | 10
Table 2: Further details about training parameters

Other model-related details:

  * •

PMB A classification head (24,571 output classes) was added to the base model.

  * •

GPT-2 Given a sample $(a_{i},p_{j})$, the input prompt for the model was
"INPUT: $a$\nMEANING:". The model was trained to complete the sentence with
$p_{j}$.

  * •

Sci5 Given a sample $(a_{i},p_{j})$, the input prompt for the model was
"normalize: $a$". The model was trained to respond with a string containing
$p_{j}$.

  * •

CODER/ SapBERT Following their original papers, we use CODER/SapBERT to
normalize an AE $a$ as follows:

$\hat{p}=\mathop{\mathrm{argmax}}_{p\in
P}\mathit{sim}(\mathcal{C}(a),\mathcal{C}(p))$

where $\mathit{sim}$ is a similarity measure (cosine, in our case), and
$\mathcal{C}(\cdot)$ is the result of embedding a term with CODER/SapBERT.
$\hat{p}$ is the predicted PT, which is compared with the actual one to
evaluate the model.

##  Appendix B Sample Creation for Contrastive Training

###  B.1 CODER

CODER leverages on term-term pairs and term-relation-term triples for its
contrastive training strategy. We create positive/negative samples for the
term-term pairs using the AEs having equal/different PT, and term-relation-
term triples connecting AEs whose PTs have the same $\mathit{parent}$.

For example, let’s consider the following $(a_{i},p_{j})$ samples, for which
we also report $\mathit{parent}(p_{j})$:

$a_{i}$ | $p_{j}$ | $\mathit{parent}(p_{j})$
---|---|---
feel like crap | malaise | Asthenic conditions
weak knees | asthenia | Asthenic conditions
zap me of all energy | asthenia | Asthenic conditions

This will generate the following training samples for CODER:

  * •

positive term-term:
(weak knees, zap me of all energy)
because they share the same $p_{j}$ “asthenia”

  * •

negative term-term:
(weak knees, feel like crap) and
(zap me of all energy, feel like crap)
because they are labelled with a different $p_{j}$ (“asthenia” vs “malaise”)

  * •

positive term-relation-term:
(weak knees, RO, feel like crap) and
(zap me of all energy, RO, feel like crap),
because their $p_{j}$ share the same $\mathit{parent}$ “Asthenic conditions”.
RO stands for “Related Other”, one of the standard term relations defined in
the UMLS ontology, and we use it to encode the relation “same granparent”.

This sample generation procedure is repeated for all samples in the three
datasets (SMM4H, CADEC and PROP), as well as for the additional samples
generated from MedDRA for the OP strategy.

###  B.2 SapBERT

SapBERT leverages on term-term synonym pairs, where the positive pairs belong
to the same upper-level concept.

The finetuning script present in the GitHub repository requires a list of term
pairs belonging to the same concept. In the the case of the three datasets
(SMM4H, CADEC and PROP) we generate the terms pairs as $(\ell_{i},a_{j})$,
where $\mathit{parent}(\ell_{i})=\mathit{norm}(a_{j})$. For the OP strategy,
the samples are all possible pairs $(\ell_{i},\ell_{j})$, where
$\mathit{parent}(\ell_{i})=\mathit{parent}(\ell_{j})$.

##  Appendix C Dataset Comparison

Most of the PTs present in the three datasets are unique to a specific
dataset, making it really challenging to perform transfer learning from one to
the other without dealing with long-tail and unseen concepts. The Venn diagram
in Figure 4 shows the number of PT concepts in common between all the
datasets. 706 PTs are unique to one of the three datasets, 276 are shared
among at least two datasets, and only 98 appear in all three of them. Out of
all the PTs in PROP, 64% are unique (410 out of 634) to this dataset alone,
making it the most challenging to perform cross-dataset normalization on. The
following most challenging datasets are CADEC (41% unique PTs) and SMM4H (34%
unique PTs).

Figure 4: Venn diagram of the shared/unique PT concepts for the three
datasets.

##  Appendix D Complete Results

Tables 3, 4 and 5 include the full results of all tested models (both Accuracy
and F1 Score).

Tables 6, 7, 8, 9 and 10 report the accuracy for all the cross-dataset
experiments (one table for each model).

SMM4H

| Accuracy
---|---
|  | in | out | overall
| CODER | 32.34 ±1.22 | 37.47 ±2.07 | 32.98 ±1.24
SAPB | 42.56 ±0.50 | 45.98 ±3.60 | 43.00 ±0.77
FT | CODER | 33.20 ±1.53 | 38.91 ±1.97 | 33.90 ±1.58
SAPB | 42.76 ±1.58 | 45.02 ±4.84 | 43.05 ±1.96
PMB | 50.26 ±0.95 | 00.00 ±0.00 | 44.12 ±1.24
GPT2 | 72.31 ±1.55 | 10.02 ±2.21 | 64.68 ±0.98
SCI5 | 34.73 ±2.05 | 12.63 ±3.33 | 32.06 ±1.84
OP | CODER | 34.20 ±0.62 | 42.25 ±1.86 | 35.19 ±0.65
SAPB | 44.29 ±1.91 | 51.25 ±6.04 | 45.15 ±2.31
PMB | 51.33 ±1.15 | 44.25 ±2.10 | 50.46 ±1.18
GPT2 | 41.17 ±0.52 | 44.48 ±4.90 | 41.60 ±1.05
SCI5 | 40.90 ±0.80 | 40.56 ±6.90 | 40.90 ±1.48
OP+FT | CODER | 34.33 ±0.80 | 43.24 ±1.40 | 35.43 ±0.83
SAPB | 45.75 ±0.66 | 47.76 ±7.09 | 46.03 ±1.36
PMB | 74.10 ±2.55 | 37.46 ±4.10 | 69.64 ±1.80
GPT2 | 73.71 ±1.29 | 31.25 ±3.37 | 68.52 ±1.30
SCI5 | 60.76 ±0.91 | 37.87 ±3.21 | 57.98 ±0.97
F1 Score
---
in | out | overall
13.52 ±0.96 | 23.67 ±1.42 | 16.34 ±1.14
16.91 ±0.57 | 29.93 ±3.32 | 20.23 ±0.20
14.12 ±1.22 | 24.67 ±1.16 | 17.14 ±1.34
16.80 ±0.78 | 29.58 ±4.68 | 20.10 ±1.17
21.43 ±1.91 | 00.00 ±0.00 | 13.68 ±1.43
11.51 ±0.49 | 07.09 ±1.68 | 10.43 ±0.48
13.77 ±0.93 | 05.27 ±2.61 | 11.83 ±0.42
14.25 ±0.81 | 27.30 ±1.20 | 17.93 ±0.83
17.96 ±1.03 | 35.12 ±4.84 | 22.27 ±1.45
25.87 ±1.21 | 27.95 ±0.84 | 27.51 ±1.41
17.50 ±0.89 | 24.29 ±4.18 | 19.63 ±1.47
17.13 ±1.05 | 22.70 ±2.60 | 18.99 ±1.37
14.40 ±0.64 | 28.19 ±1.02 | 18.18 ±0.65
18.43 ±1.17 | 32.04 ±5.22 | 21.92 ±1.66
55.59 ±3.08 | 22.49 ±2.73 | 45.62 ±2.51
52.52 ±2.61 | 19.59 ±2.53 | 42.20 ±2.68
32.25 ±2.85 | 23.50 ±2.83 | 30.08 ±1.77

Table 3: Full metrics (accuracy and F1 score) of all tested models on the
SMM4H dataset.

CADEC

|  | Accuracy
---|---|---
|  | in | out | overall
| CODER | 35.44 ±0.40 | 44.79 ±3.35 | 35.89 ±0.42
SAPB | 39.57 ±1.05 | 51.34 ±4.67 | 40.14 ±1.14
FT | CODER | 39.13 ±1.54 | 45.96 ±2.15 | 39.46 ±1.45
SAPB | 48.04 ±4.03 | 46.48 ±4.49 | 47.97 ±3.70
PMB | 82.47 ±0.27 | 00.00 ±0.00 | 78.49 ±0.10
GPT2 | 84.70 ±0.42 | 07.40 ±0.13 | 80.97 ±0.45
SCI5 | 72.80 ±0.12 | 15.94 ±1.37 | 70.05 ±0.17
OP | CODER | 41.98 ±0.65 | 48.06 ±3.12 | 42.26 ±0.65
SAPB | 69.49 ±0.37 | 50.53 ±1.40 | 68.58 ±0.37
PMB | 73.64 ±0.35 | 47.05 ±0.50 | 72.36 ±0.40
GPT2 | 68.35 ±0.22 | 43.29 ±2.47 | 67.14 ±0.22
SCI5 | 65.99 ±0.58 | 44.73 ±2.22 | 64.96 ±0.45
OP+FT | CODER | 42.08 ±0.80 | 48.34 ±2.86 | 42.38 ±0.75
SAPB | 62.02 ±2.06 | 48.44 ±1.74 | 61.38 ±1.91
PMB | 87.18 ±0.33 | 23.88 ±1.79 | 84.12 ±0.28
GPT2 | 86.08 ±0.78 | 23.92 ±1.09 | 83.08 ±0.69
SCI5 | 80.96 ±0.40 | 37.21 ±4.32 | 78.85 ±0.52
F1 Score
---
in | out | overall
17.78 ±0.43 | 26.81 ±2.20 | 19.89 ±0.58
17.44 ±0.63 | 31.62 ±3.29 | 20.20 ±0.56
18.72 ±0.54 | 27.53 ±1.46 | 20.88 ±0.38
20.25 ±1.50 | 27.02 ±3.67 | 22.11 ±0.62
47.64 ±1.29 | 00.00 ±0.00 | 33.32 ±1.44
30.97 ±0.50 | 08.31 ±0.83 | 25.00 ±0.40
37.02 ±1.63 | 07.11 ±3.29 | 29.84 ±1.39
24.40 ±1.78 | 27.12 ±3.61 | 25.25 ±1.95
27.48 ±0.81 | 32.38 ±1.54 | 29.39 ±0.32
38.46 ±1.36 | 32.21 ±1.40 | 38.47 ±0.65
28.76 ±1.20 | 28.75 ±1.23 | 29.66 ±1.35
27.10 ±1.09 | 27.27 ±1.49 | 27.98 ±0.86
20.77 ±0.66 | 29.52 ±1.66 | 23.31 ±0.71
27.20 ±0.82 | 29.45 ±1.59 | 28.68 ±0.54
70.43 ±1.00 | 15.74 ±1.15 | 55.78 ±0.36
64.17 ±2.89 | 15.07 ±0.67 | 51.05 ±1.04
50.09 ±2.36 | 22.66 ±2.23 | 44.36 ±2.11

Table 4: Full metrics (accuracy and F1 score) of all tested models on the
CADEC dataset.

PROP

|  | Accuracy
---|---|---
|  | in | out | overall
| CODER | 12.58 ±0.44 | 28.44 ±3.02 | 14.19 ±0.71
SAPB | 20.28 ±0.59 | 36.54 ±3.61 | 21.91 ±0.81
FT | CODER | 19.88 ±0.89 | 28.39 ±3.81 | 20.76 ±1.20
SAPB | 33.77 ±2.57 | 36.34 ±3.71 | 34.02 ±2.63
PMB | 60.20 ±0.34 | 00.00 ±0.00 | 54.09 ±0.76
GPT2 | 65.21 ±0.73 | 05.75 ±1.22 | 59.18 ±1.12
SCI5 | 45.70 ±0.77 | 12.71 ±1.13 | 42.34 ±0.61
OP | CODER | 15.29 ±0.24 | 30.62 ±1.97 | 16.84 ±0.38
SAPB | 24.10 ±0.76 | 43.80 ±2.54 | 26.09 ±0.85
PMB | 30.99 ±1.00 | 37.92 ±1.66 | 31.69 ±1.04
GPT2 | 24.05 ±0.77 | 32.89 ±2.23 | 24.94 ±0.86
SCI5 | 24.71 ±0.78 | 32.71 ±4.80 | 25.49 ±1.12
OP+FT | CODER | 19.64 ±0.25 | 32.88 ±2.51 | 20.98 ±0.28
SAPB | 39.88 ±3.20 | 42.68 ±3.66 | 40.16 ±3.23
PMB | 68.89 ±1.17 | 25.60 ±1.91 | 64.50 ±1.48
GPT2 | 66.90 ±0.61 | 21.75 ±2.16 | 62.31 ±1.10
SCI5 | 54.07 ±0.54 | 27.42 ±4.77 | 51.34 ±0.81
F1 Score
---
in | out | overall
08.02 ±0.56 | 16.79 ±1.30 | 10.79 ±0.83
10.43 ±0.33 | 22.50 ±2.83 | 13.67 ±0.83
10.18 ±0.77 | 15.85 ±2.01 | 12.23 ±0.84
12.20 ±0.77 | 22.41 ±2.96 | 15.35 ±1.14
25.56 ±0.66 | 00.00 ±0.00 | 14.66 ±0.96
17.50 ±0.50 | 07.90 ±0.63 | 13.91 ±0.31
20.80 ±1.26 | 07.76 ±0.49 | 17.31 ±1.51
09.61 ±0.66 | 17.79 ±0.70 | 12.35 ±0.51
12.71 ±0.83 | 27.30 ±2.13 | 16.67 ±0.93
20.89 ±1.88 | 23.60 ±1.39 | 22.75 ±1.66
11.36 ±0.78 | 19.54 ±3.04 | 13.38 ±1.18
10.86 ±0.66 | 19.73 ±1.98 | 13.10 ±0.85
11.38 ±0.32 | 19.27 ±1.27 | 14.22 ±0.51
16.11 ±1.88 | 26.86 ±2.79 | 19.61 ±2.09
54.19 ±2.05 | 16.35 ±1.59 | 39.66 ±2.77
49.29 ±1.26 | 13.41 ±1.28 | 35.12 ±2.52
33.36 ±1.16 | 17.15 ±2.70 | 27.89 ±2.35

Table 5: Full metrics (accuracy and F1 score) of all tested models on the PROP
dataset.

CODER FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 39.13 | 35.80 | 31.84
SMM4H | 37.78 | 33.20 | 40.44
PROP | 37.83 | 35.08 | 19.88

Test (out)
---
CADEC | SMM4H |  PROP
45.96 | 35.79 | 08.79
31.87 | 38.91 | 09.39
35.73 | 32.56 | 28.39

Test (overall)
---
CADEC | SMM4H |  PROP
39.46 | 35.82 | 15.25
36.00 | 33.90 | 14.25
37.39 | 33.85 | 20.76

CODER OP+FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 42.08 | 35.12 | 33.56
SMM4H | 44.90 | 34.33 | 43.95
PROP | 44.25 | 38.62 | 19.64

Test (out)
---
CADEC | SMM4H |  PROP
48.34 | 35.63 | 11.34
36.72 | 43.24 | 12.81
37.77 | 32.21 | 32.88

Test (overall)
---
CADEC | SMM4H |  PROP
42.38 | 35.25 | 17.59
42.42 | 35.43 | 17.69
42.85 | 35.49 | 20.98

Table 6: Cross-dataset accuracy for CODER FT and CODER OP+FT on the three
datasets.

SapBERT FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 42.69 | 35.09 | 30.80
SMM4H | 39.14 | 39.58 | 39.92
PROP | 32.21 | 31.07 | 26.69

Test (out)
---
CADEC | SMM4H |  PROP
34.47 | 35.32 | 8.73
23.79 | 32.42 | 10.02
29.77 | 34.97 | 25.81

Test (overall)
---
CADEC | SMM4H |  PROP
42.30 | 35.12 | 14.94
34.53 | 38.69 | 14.71
31.71 | 32.98 | 26.60

SapBERT OP+FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 63.96 | 45.93 | 49.29
SMM4H | 70.71 | 46.28 | 62.73
PROP | 68.69 | 49.77 | 41.33

Test (out)
---
CADEC | SMM4H |  PROP
47.76 | 34.72 | 14.63
61.31 | 48.43 | 17.84
48.09 | 28.82 | 39.81

Test (overall)
---
CADEC | SMM4H |  PROP
63.19 | 42.65 | 24.39
67.89 | 46.54 | 24.90
64.24 | 39.45 | 41.17

Table 7: Cross-dataset accuracy for SapBERT FT and SapBERT OP+FT on the three
datasets.

PMB FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 82.47 | 54.00 | 51.74
SMM4H | 38.82 | 50.26 | 31.43
PROP | 47.76 | 39.83 | 60.20

Test (out)
---
CADEC | SMM4H |  PROP
00.00 | 00.00 | 00.00
00.00 | 00.00 | 00.00
00.00 | 00.00 | 00.00

Test (overall)
---
CADEC | SMM4H |  PROP
78.49 | 38.00 | 14.55
27.16 | 44.12 | 04.93
37.48 | 20.22 | 54.09

PMB OP+FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 87.18 | 61.05 | 64.95
SMM4H | 79.84 | 74.10 | 79.24
PROP | 75.33 | 63.33 | 68.89

Test (out)
---
CADEC | SMM4H |  PROP
23.88 | 18.76 | 11.64
53.14 | 37.46 | 17.35
39.84 | 23.86 | 25.60

Test (overall)
---
CADEC | SMM4H |  PROP
84.12 | 48.54 | 26.64
71.82 | 69.64 | 27.07
67.72 | 43.88 | 64.50

Table 8: Cross-dataset accuracy for PMB FT and PMB OP+FT on the three
datasets.

GPT-2 FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 84.70 | 60.86 | 60.52
SMM4H | 65.73 | 72.31 | 69.88
PROP | 56.68 | 52.43 | 65.21

Test (out)
---
CADEC | SMM4H |  PROP
07.40 | 03.78 | 01.51
06.54 | 10.02 | 01.84
07.41 | 03.45 | 05.75

Test (overall)
---
CADEC | SMM4H |  PROP
80.97 | 43.99 | 18.10
47.94 | 64.68 | 12.52
46.08 | 28.31 | 59.18

GPT-2 OP+FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 86.08 | 62.50 | 62.56
SMM4H | 78.23 | 73.71 | 77.98
PROP | 72.75 | 58.96 | 66.90

Test (out)
---
CADEC | SMM4H |  PROP
23.92 | 34.21 | 08.90
26.72 | 31.25 | 11.02
27.65 | 18.72 | 21.75

Test (overall)
---
CADEC | SMM4H |  PROP
83.08 | 54.06 | 24.00
62.74 | 68.52 | 21.54
63.09 | 39.15 | 62.31

Table 9: Cross-dataset accuracy for GPT-2 FT and GPT-2 OP+FT on the three
datasets.

Sci5 FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 72.80 | 44.44 | 46.20
SMM4H | 35.28 | 34.73 | 44.05
PROP | 29.23 | 28.44 | 45.70

Test (out)
---
CADEC | SMM4H |  PROP
15.94 | 06.86 | 03.07
10.42 | 12.63 | 03.30
15.44 | 11.14 | 12.71

Test (overall)
---
CADEC | SMM4H |  PROP
70.05 | 33.33 | 15.19
27.81 | 32.06 | 09.71
26.24 | 19.93 | 42.34

Sci5 OP+FT

| Test (in)
---|---
|  | CADEC | SMM4H |  PROP
Train | CADEC | 80.96 | 56.07 | 60.35
SMM4H | 71.96 | 60.76 | 72.82
PROP | 63.63 | 47.74 | 54.07

Test (out)
---
CADEC | SMM4H |  PROP
37.21 | 36.86 | 13.95
52.17 | 37.87 | 17.19
40.12 | 26.95 | 27.42

Test (overall)
---
CADEC | SMM4H |  PROP
78.85 | 50.34 | 27.02
66.02 | 57.98 | 25.92
58.52 | 37.45 | 51.34

Table 10: Cross-dataset accuracy for Sci5 FT and Sci5 OP+FT on the three
datasets.


# Complete study of four top quark production at the LHC in the SMEFT

Hesham El Faham

###### Abstract

We present a study of four top quark production at the LHC in the Standard
Model Effective Field Theory (SMEFT). The analysis is performed at the tree-
level, including all terms from the mixed QCD-EW-coupling cross-section
expansion and all the relevant dimension-six SMEFT operators. We find several
cases in which formally subleading terms give rise to significant
contributions. Moreover, we obtain limits on the SMEFT Wilson coefficients
(WCs), including different collider setups, FCC-hh and HL-LHC, using a
simplified chi-square fit. Through the latter, we also assessed the importance
of including differential information in the fit.

##  1 Introduction

The four top quark production at the LHC is characterised by a tiny SM cross-
section at 13 TeV of about 12 fb. Despite this, $t\bar{t}t\bar{t}$ signatures
are distinctive, leading to an affluent and energetic final state rendering
four top signatures a remarkable opportunity for probing new physics.

The complete NLO predictions, including all possible QCD and electroweak (EW)
orders, have been calculated in Ref. [1]. The latter’s results show a peculiar
and unforeseen interplay between EW and QCD contributions, with significant
contributions arising from formally subleading terms. In the SM,
representative diagrams of purely QCD-induced four-top production and through
the inclusion of EW vertices are shown in fig. 1. QCD-induced diagrams
typically provide the leading contribution.

$\bar{t}$$g$$\bar{t}$$g$$t$$t$$\bar{t}$$g$$t$$g$$\bar{t}$$t$$Z$$\gamma^{*}$

Figure 1: Representative diagrams of four top production at
$\mathscr{O}(\alpha_{s}^{2})$ (_left_) and at
$\mathscr{O}(\alpha_{s}\alpha_{\mathrm{w}})$ (_right_), at the LHC.

However, Ref. [1] shows that in the SM at $\sqrt{s}=13$ TeV, the contributions
arising from the non-purely-QCD diagrams are significant, reaching more than a
third of the leading tree-level QCD contributions. Such a peculiar cross-
section behaviour motivates a detailed study of new physics effects,
particularly in the SMEFT framework.

##  2 Four-fermion operators and cross-section expansion

This work considers all possible contributions from the SM and SMEFT,
including all dimension-six operators of the latter [2]. We follow the
notation and operator conventions of Refs. [3, 4] and consider all possible
dimension-six SMEFT operators contributing to four top quark production.
However, in this note, we mainly focus on the 4-heavy four-fermion operators
(we refer the interested reader to Ref. [2] for the detailed study):

$\\{\mathcal{O}_{\scriptscriptstyle QQ}^{\scriptscriptstyle
1},\mathcal{O}_{\scriptscriptstyle QQ}^{\scriptscriptstyle
8},\mathcal{O}_{\scriptscriptstyle tt}^{\scriptscriptstyle
1},\mathcal{O}_{\scriptscriptstyle Qt}^{\scriptscriptstyle
1},\mathcal{O}_{\scriptscriptstyle Qt}^{\scriptscriptstyle 8}\\}.$ (1)

The decomposition of the partonic differential cross-section up to
$\mathscr{O}(\Lambda^{-4})$ in SMEFT reads

$\displaystyle d\sigma=d\sigma_{\rm SM}+\frac{1}{\Lambda^{2}}d\sigma_{\rm
int}+\frac{1}{\Lambda^{4}}\big{(}d\sigma_{\rm quad}+d\sigma_{\rm
dbl}+d\sigma_{\rm d8}\big{)},$ (2)

where the leading SMEFT contribution, $d\sigma_{\rm int}$, is the linear
interference between the SM amplitudes and ones with a single insertion of a
dimension-six operator. The $d\sigma_{\rm quad}$ and $d\sigma_{\rm dbl}$ are
the squared single-insertion and double-insertion contributions, respectively.
The interference cross-section expansion for the $gg$-initiated four top quark
production in the presence of four-fermion SMEFT operators can therefore be
written as follows:

$\displaystyle d\sigma_{{\rm int},gg,\texttt{[4F]}}$
$\displaystyle=\alpha_{s}^{3}\,d\sigma^{(3,0,0)}_{{\rm
int},gg}+\alpha_{s}^{2}\left(\,\alpha\,d\sigma^{(2,1,0)}_{{\rm
int},gg}+\alpha_{t}\,d\sigma^{(2,0,1)}_{{\rm int},gg}\right),$ (3)

with $\alpha_{s}$, $\alpha$ and $\alpha_{t}$ denoting the strong, gauge and
top-Yukawa couplings, respectively.

##  3 Inclusive predictions

This section presents the complete LO SMEFT inclusive predictions for
$t\bar{t}t\bar{t}$ production at $\sqrt{s}=13$ TeV at the LHC. Predictions are
shown for the 4-heavy four-fermion operators, ones connecting four heavy quark
lines, as displayed in the _left_ diagram of fig. 2 where the shaded blob
depicts the SMEFT insertion. For more results on the 2-heavy 2-light four-
fermion operators (depicted in the _right_ diagram of fig. 2), as well as the
two-fermion and purely bosonic operators, we remind the reader to consult Ref.
[2], where also the differential predictions are presented.

$\bar{t}$$g$$\bar{t}$$g$$t$$t$$\bar{t}$$q$$t$$\bar{q}$$\bar{t}$$t$

Figure 2: Representative diagrams of four top quark production with blobs
representing the one dimension-six EFT insertion, in the $gg$-initiated
production mode (_left_) and in the $q\bar{q}$-initiated production mode
(_right_).

The computations were performed via MadGraph5_aMC@NLO [5, 6] with the use of
the SMEFTatNLO model [4]. The interference cross-section of the 4-heavy four-
fermion operators is presented in fig. 3 where the total inclusive
interference cross-section, $\sigma_{\rm incl.}$/INCL, is split as follows:
$\sigma_{\rm incl.}=\sigma_{3}+\sigma_{2}+\sigma_{1}+\sigma_{0}$, with
$\sigma_{i}$ with $i=3,2,1,0$ denoting the different contributions arising
from terms with order $\alpha_{s}^{i}$ in line with eq. 3.

Figure 3:  Depiction of the interference strength of the 4-heavy operators.
The columns denote the WCs in their UFO notation. The SM LO cross-section at
$\sqrt{s}=13$ TeV is presented for reference. WCs were individually set to
unity, and the scale of new physics $\Lambda$ is fixed to 1 TeV.

For _all_ 4-heavy operators, the dominant interference arises from formally
subleading orders, $\sigma_{2}$. This observation conflicts with the “naive”
expectation that purely QCD-induced terms deliver the highest contribution to
the cross-section (through $\sigma_{3}$). Such conflict emphasises the
significance of the EW $tt\to tt$ scattering present in $t\bar{t}t\bar{t}$
production at LO.

##  4 Toy fits

This section presents limits on the WCs from a simplified
${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}^{2}$
individual fit in different collider scenarios: LHC, FCC-hh and HL-LHC.
Through such exercise, we explored the effects of differential information and
the different collider energies on the WCs bounds.

### Impact of differential information

Figure 4 displays the individual limits presented (I) when only QCD-induced
(leading) terms are considered and (II) when contributions to the cross-
section from all tree-level terms in the mixed QCD-EW expansion are included.
Furthermore, it compares the use of only inclusive information from
$\sigma_{tttt}$ to when also adding differential information in the fit from
$m_{tttt}$. The exercise is performed for the HL-LHC at $\sqrt{s}=14$ TeV with
3 $\mathrm{ab}^{-1}$ of integrated luminosity. The SM prediction calculated at
LO is $\sigma_{tttt}^{\rm HL}=9.0$ fb, with a 20% theoretical uncertainty.
Experimental measurement is assumed to be that of SM within the expected 28%
experimental total uncertainty [7].

Figure 4: 95%CL limits on the 4-heavy operators’ coefficients at the HL-LHC
scenario from a
${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}^{2}$
fit. The limits are shown for when using inclusive and differential
information from $\sigma_{tttt}$ and $m_{tttt}$, respectively.

Our results indicate that differential information improves the sensitivity
and should be used whenever possible.

### Comparison of different collider setups

Finally, in fig. 5, we compare the results from current LHC measurements to
the FCC-hh bounds when only using inclusive information. For the LHC, we use
the SM prediction at NLO in QCD of Ref. [1], and we fit the theoretical
predictions to the inclusive ATLAS [8] and CMS [9] measurements.

Figure 5: Limits on all the relevant operators used in the full study of Ref.
[2] obtained from the
${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}^{2}$
fit.

The results from this fit show the significant constraining power the FCC-hh
will be capable of providing on the SMEFT.

##  5 Summary and conclusions

We emphasise the importance of our results in summarising the four top quark
production SMEFT predictions and analysing them in each order of the QCD-EW
expansion. Our observations underline the significance of subleading terms for
the 4-heavy quark operators. Moreover, within all the dimension-six SMEFT
operators, we have shown that four top quark production provides strong
constraints on the 4-heavy coefficients. We also showed that including
differential information further improves the bounds on the SMEFT
coefficients.

## Acknowledgments

I thank the organisers of ICHEP2022 for the invitation to share my work and
for delivering a successful conference. This work was supported from the
European Union’s Horizon 2020 research and innovation program as part of the
Marie Skłodowska-Curie Innovative Training Network MCnetITN3 (grant agreement
no. 722104), the F.R.S.-FNRS under the “Excellence of Science” EOS be.h
project numbers 30820817 and 40005600, the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(grant agreement No. 949451), and from a Royal Society University Research
Fellowship through grant URF/R1/201553.

## References

  * [1] R. Frederix, D. Pagani and M. Zaro, _Large NLO corrections in $t\bar{t}W^{\pm}$ and $t\bar{t}t\bar{t}$ hadroproduction from supposedly subleading EW contributions_, _JHEP_ 02 (2018) 031 [1711.02116].
  * [2] R. Aoude, H. El Faham, F. Maltoni and E. Vryonidou, _Complete SMEFT predictions for four top quark production at hadron colliders_ , _JHEP_ 10 (2022) 163 [2208.04962].
  * [3] D. Barducci et al., _Interpreting top-quark LHC measurements in the standard-model effective field theory_ , 1802.07237.
  * [4] C. Degrande, G. Durieux, F. Maltoni, K. Mimasu, E. Vryonidou and C. Zhang, _Automated one-loop computations in the standard model effective field theory_ , _Phys. Rev. D_ 103 (2021) 096024 [2008.11743].
  * [5] J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer et al., _The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations_ , _JHEP_ 07 (2014) 079 [1405.0301].
  * [6] R. Frederix, S. Frixione, V. Hirschi, D. Pagani, H.S. Shao and M. Zaro, _The automation of next-to-leading order electroweak calculations_ , _JHEP_ 07 (2018) 185 [1804.10017].
  * [7] P. Azzi et al., _Report from Working Group 1: Standard Model Physics at the HL-LHC and HE-LHC_ , _CERN Yellow Rep. Monogr._ 7 (2019) 1 [1902.04070].
  * [8] ATLAS collaboration, _Evidence for $t\bar{t}t\bar{t}$ production in the multilepton final state in proton–proton collisions at $\sqrt{s}=13$ TeV with the ATLAS detector_, _Eur. Phys. J. C_ 80 (2020) 1085 [2007.14858].
  * [9] CMS collaboration, _Search for production of four top quarks in final states with same-sign or multiple leptons in proton-proton collisions at $\sqrt{s}=$ 13 TeV_, _Eur. Phys. J. C_ 80 (2020) 75 [1908.06463].