# Impact of Domain-Adapted Multilingual Neural Machine Translation in the
Medical Domain ††thanks: Citation: Work presented at Translating and the
Computer - TC44, 24 November 2022, Luxembourg.
https://asling.org/tc44/confirmed-presentations/#Rios-Gaona

Miguel Rios, Raluca-Maria Chereji, Alina Secară, Dragos Ciobanu
Centre for Translation Studies, University of Vienna
{miguel.angel.rios.gaona, raluca-maria.chereji,
alina.secara<EMAIL_ADDRESS>

###### Abstract

Multilingual Neural Machine Translation (MNMT) models leverage many language
pairs during training to improve translation quality for low-resource
languages by transferring knowledge from high-resource languages. We study the
quality of a domain-adapted MNMT model in the medical domain for English-
Romanian with automatic metrics and a human error typology annotation which
includes terminology-specific error categories. We compare the out-of-domain
MNMT with the in-domain adapted MNMT. The in-domain MNMT model outperforms the
out-of-domain MNMT in all measured automatic metrics and produces fewer
terminology errors.

_K_ eywords Multilingual machine translation $\cdot$ Medical machine
translation $\cdot$ Error typology annotation

##  1 Introduction

Current state-of-the-art Neural Machine Translation (NMT) models have shown
promising results on low-resource language pairs, particularly for non-
specialised domains (Araabi and Monz, 2020). However, in a high-risk and low-
resource domain, like the medical domain, the accurate translation of
terminology is crucial for exchanging information across international
healthcare providers or users (Skianis et al., 2020). Multilingual NMT (MNMT)
models leverage many language pairs and millions of segments during training
(Johnson et al., 2017). The inclusion of many language pairs helps to improve
the translation quality for low-resource languages by transferring knowledge
from high-resource languages. Moreover, domain adaptation techniques are used
to adapt MNMT models into new domains (Bérard et al., 2020). However,
evaluation studies of MNMT models are focused on automatic metrics without
providing insights into the quality of the translation of specialised
terminology.

In this paper, we study the quality of a pre-trained MNMT model in the medical
domain for a low-resource language pair (English-Romanian). Our goal is to
compare an out-of-domain MNMT with a fine-tuned in-domain MNMT in terms of
automatic metrics and terminology translation. We use a pre-trained model
based on MBart (Liu et al., 2020) and fine-tune it with a medical in-domain
parallel corpus.

We test the models on the English-Romanian language pair with a corpus of
medical paper abstracts (Neves et al., 2018). We evaluate both models with
automatic metrics, and a terminology error typology annotation performed by
in-house human annotators (Haque et al., 2019). The fine-tuned MBart model
outperforms MBart on the automatic metrics. In addition, the error analysis
based on a terminology-based error typology (Haque et al., 2019) shows that
the fine-tuned model also produces fewer errors than the MBart model.

##  2 Background

MNMT models are based on transferring parameters or information across
multiple languages, where low-resource languages benefit from the high-
resource languages. The MNMT model shares a common word representation (i.e.,
arrays of numbers) across language pairs. During training, the MNMT model
clusters words with similar contexts from the high- and low-resource segments
(Johnson et al., 2017). The low-resource pairs learn meaningful word
representations given the access to a large number of similar contexts from
the high-resource language pairs. Moreover, an MNMT model allows to translate
across multiple languages by using only one translation system. The multiple
languages are processed jointly by indicating the target translation direction
on each segment of the multilingual corpora in the input training data by
using an artificial token (label <2target>). For example, an English-Romanian
segment pair would be labelled as follows:

<2ro> It is noted that in some cases increase of blood pressure was
documented. $\rightarrow$ Se remarcă faptul că, în unele cazuri, s-a
înregistrat cresterea tensiunii arteriale.

MNMT models outperform standard bilingual baselines on translation quality for
low-resource languages (Johnson et al., 2017), but they require a high amount
of computational resources to process the millions of parallel multilingual
segments.

In particular, MBart is a sequence-to-sequence model pre-trained on
monolingual data from 25 languages based on a text reconstruction learning
objective for MNMT (Liu et al., 2020). MBart incorporates a monolingual
training step before the multilingual MT training for a better initialisation
of the translation model. In other words, MBart first learns an improved
individual representation of each language with monolingual data. After that,
MBart continues with the multilingual translation training based on parallel
data. MBart shows a better translation quality compared to previous MNMT
models.

However, most MNMT models are general-purpose systems trained with web crawled
corpora (Liu et al., 2020; Verma et al., 2022), and they struggle with
specialised domains (e.g. medical). Domain adaptation aims to improve the
translation performance in specialised domains, where fine-tuning is a low-
cost and common technique. Fine-tuning consists of resuming the training of an
out-of-domain resource-rich MT model with a poor-resourced in-domain corpus
(Chu and Wang, 2018). The resulting model is adapted to work with an in-domain
language pair, instead of re-training the MNMT model from scratch (Verma et
al., 2022).

MT models are usually evaluated with automatic metrics that take into account
fluency and adequacy, by comparing the machine translation output against one
or more human reference translations (Papineni et al., 2002). Metrics produce
a corpus-level score or a segment-level score for a given MT model (Rei et
al., 2020). However, automatic metrics are not designed to identify
translation errors in MT outputs, for example, errors in terminology (Haque et
al., 2019). On the other hand, error typology evaluation frameworks, such as
the Multidimensional Quality Metrics (MQM) (Lommel et al., 2013), are based on
manually classifying and annotating errors using predefined categories. Haque
et al. (2019) propose an error typology with a focus on terminology: human
evaluators identify an error in the MT output, select a category out of the
eight available, and assign a severity score.

##  3 Experiments

For fine-tuning, we use the English-Romanian section from the EMEA parallel
corpus (CLARIN:EL, 2015). The EMEA corpus consists of PDF documents from the
European Medicines Agency. We split the corpus into $775,904$ training, and
$7,837$ validation segments. We evaluate the MNMT models with the test dataset
of abstracts from scientific publications from Medline (Neves et al., 2018)
which contains $291$ segments.

We use BLEU (Papineni et al., 2002; Post, 2018), chrF (Popović, 2015), and
COMET (Rei et al., 2020) for automatic evaluation. For human evaluation, we
use (Haque et al., 2019) which contains eight terminology-related error
categories - Partial error, Source term copied, Inflectional error, Reorder
error, Disambiguation issue in target, Incorrect lexical selection, Term drop,
and Other error -, and three severity levels - Minor, Major and Critical.

We continue training MBart with the EMEA corpus to adapt it into the medical
domain, and we perform model selection using BLEU on the validation split. We
performed our experiments with Fairseq (Ott et al., 2019) using an open-source
pre-trained model for
MBart111https://dl.fbaipublicfiles.com/fairseq/models/mbart/mbart.cc25.ft.enro.tar.gz.
The settings for the fine-tuned MBart are as follows: Adam with learning rate
$3\mathrm{e}{-5}$, inverse square root scheduler, $2,500$ warm-up updates,
$40,000$ updates, dropout $0.3$, attention dropout $0.1$, label smoothing
$0.2$, batch size $2048$ tokens ($256$ maximum tokens per batch, and $8$
batches for gradient accumulation), and memory efficient fp16 training. We
used a 16GB Tesla T4 GPU from the Google Cloud platform for training222The
scripts for our experiments are available at:
https://github.com/mriosb08/medical-NMT-HAITrans. The fine-tuning process took
38 hours to complete.

###  3.1 Results

We define general MBart (out-of-domain data), and fine-tuned MBart (in-domain
medical data). Table 1 shows the automatic metrics scores for both models.
Fine-tuned MBart outperforms the general model on all the metrics. The BLEU
score is statistically significant $p=0.001$ based on bootstrap resampling
with $1,000$ iterations.

| BLEU $\uparrow$ | chrF $\uparrow$ | COMET $\uparrow$
---|---|---|---
MBart | 21.9 | 51.5 | 0.556
fine-tuned MBart | 25.8 | 54.9 | 0.663
Table 1: Automatic metrics for MBart and fine-tuned MBart.

Furthermore, we performed an analysis of the COMET segment level scores. We
use MT-Telescope (Rei et al., 2021) to compare both systems. Figure 1 shows
the percentage of segments divided into four quality bins. Each bin is defined
by a default threshold from the COMET scores, from green (residual errors) to
red (critical errors). The red bin has translations lower than $0.10$ score,
the yellow bin has translations between $0.10$ and $0.30$ score, the light
green has translations between $0.30$ and $0.70$ score, and the dark green has
translations greater than $0.70$ score. MBart is System X and fine-tuned MBart
is System Y. The fine-tuned MBart has the highest number of high scores
compared to the original general model.

Figure 1: COMET scores for segments divided into quality bins for MBart
(System X) and fine-tuned MBart (System Y).

Figure 2 shows visually the difference of COMET scores between the two systems
for each segment. The size and colour of a bubble is the difference in the
COMET score between systems for the same segment. Moreover, systems are
different when the bubbles are far from each other along the axis (x_score
MBart, and y_score fine-tuned MBart), and from the centre of the plot. Both
systems MBart and fine-tuned MBart are different in terms of COMET scores, and
fine-tuned MBart has a higher COMET score. If both models produce different
translations, in this case, it means that the fine-tuned model is learning to
generate MT outputs close to the medical domain.

Figure 2: Bubble plot of segment level COMET scores for MBart (x_score) and
fine-tuned MBart (y_score).

###  3.2 Error Analysis

To gain insights into the specific terminology errors produced by the two
models, we show a sample of $12$ abstracts with a total of $75$ segments to
three annotators. The annotators are native Romanian speakers with in-house
and freelance translation experience; moreover, one of the annotators also has
in-house and freelance medical translation experience. The annotators had
access to the source, the reference, and the output of the two MT systems to
annotate each MT segment with error categories (Klubička et al., 2017) using
(Haque et al., 2019). The annotators annotated the abstracts collaboratively
(Esperança-Rodier et al., 2019) – the motivation for the joint in-person
annotation is to increase agreement for identifying possible terms and errors.

To perform the annotation, we set up a translation project in Trados Studio
$2021$333https://www.trados.com/products/trados-studio/ and import the source,
reference and MT output files as bilingual .xlsx files. We install the freely-
available Qualitivity 444https://community.rws.com/product-groups/trados-
portfolio/rws-appstore/w/wiki/2251/qualitivity plugin integrated into Studio
using an API key; this serves as the environment in which the annotators
record any identified errors, their severity level and proposed corrections,
along with explanatory comments. At the end of the annotation process, we
export a report from Qualitivity containing the full annotation data for the
reference texts, and MBart and fine-tuned MBart outputs.

The total number of terminology-related errors for general-model MBart and
fine-tuned MBart are $98$ and $64$ respectively, demonstrating the improvement
brought about by the fine-tuning process with in-domain data. Table 2 shows
the number of errors for each category present in the abstracts.

Error Type | MBart $\downarrow$ | fine-tuned MBart $\downarrow$
---|---|---
Partial error | 41 | 23
Source term copied | 22 | 9
Inflectional error | 2 | 4
Reorder error | 1 | 3
Disambiguation issue in target | 14 | 6
Incorrect lexical selection | 9 | 6
Term drop | 0 | 0
Other error | 9 | 13
Table 2: Total errors for each terminology-related category.

The fine-tuned MBart model produces fewer errors than the general model on
most of the categories. However, the fine-tuned model fails in the following
categories: Inflectional error, Reorder error, and Other. Moreover, we show
annotated examples of random segments for each error category to further
understand the cause of the errors. In Table 3 we show a random selection of
source and MT output for each error category, except Other, and highlight the
annotated errors for each category for fine-tuned MBart.

Category | Source | Target (fine-tuned MBart)
---|---|---
Partial error |  The DX-OSA score may be useful for identifying obese patients with significant OSA who require CPAP (continuous positive airway pressure) treatment, and CPAP could be commenced without the need for polysomnography, therefore, without delaying surgery. |  Scorul DX-OSA poate fi util pentru identificarea pacienţilor obezi cu OSA semnificativă care necesită tratament cu CPAP (tensiune arterială continuă pozitivă [instead of presiune pozitivă continuă în căile aeriene]), iar CPAP poate fi început fără a fi necesară polisomnografie, prin urmare, fără a întârzia intervenţia chirurgicală.
Source term copied |  The objectives of this study were to reveal possible relations between antioxidant therapy and a number of serum biochemical variables (ALT, AST, APPT, LDH, urea, leukocytes, platelets), the length of mechanical ventilation, the time spent in the ICU, and the mortality rate in major trauma patients. |  Obiectivul acestui studiu a fost să evidentieze posibilele relatii dintre tratamentul cu antioxidanti si o serie de variabile biochimice serice (ALT, AST, APPT [instead of APTT], LDH, uree, leucocite, trombocite), durata ventilatiei mecanice, timpul petrecut în ICU si rata mortalitătii la pacientii cu traumatisme majore.
Inflectional error |  Two of these drugs, duloxetine and venlafaxine, are used also in chronic pain management. |  Două dintre aceste medicamente, duloxetină şi venlafaxină [instead of duloxetina si venlafaxina], sunt utilizate şi în tratamentul durerii cronice.
Reorder error |  Although not statistically significant, MODS and ARDS incidences were higher in the DCO shock group: MODS (41.7% versus 22.6% and 20%; p = 0.08/0.17), ARDS (29.2% versus 17% and 20%; p = 0.22/0.53). |  Deşi nu au fost semnificative statistic, incidenţele MODS şi ARDS au fost mai mari în grupul cu şoc DCO [instead of grupul DCO cu soc]: MODS (41,7% faţă de 22,6% şi 20%; p = 0,08/0,07), ARDS (29,2% faţă de 17% şi 20%; p = 0,22/0,53).
Disambiguation issue in target |  The drug’s efficacy results from its modulating effect on the descending inhibitory pain pathways and the inhibition of the nociceptive input. |  Eficacitatea medicamentului rezultă din efectul său de modulare asupra căilor de durere inhibatoare descendente [instead of căilor descendente inhibitorii ale durerii] si inhibarea contributiei nociceptive.
Incorrect lexical selection |  These results correlate with a higher trauma score in these patients, more serious lesions requiring several damage control procedures. |  Aceste rezultate sunt corelate cu un scor traumatic [instead of gravitatea traumatismelor] mai mare la acesti pacienti, leziunile mai grave necesitând mai multe proceduri de control al leziunilor.
Table 3: Fine-tuned MBart annotated examples for each error category (except
Other error).

Table 4 shows all the examples for the Other error category for the fine-tuned
MBart. As the fine-tuned model underperformed in terms of Other errors - $13$
to $9$ -, we investigate this further and list all the annotated errors within
the Other category in Table 4. We identify two phenomena regarding the
treatment of English borrowings and acronyms, and evidence of hallucination.
The first phenomenon observed is that source terms are translated, even where
a borrowing from English would be the correct translation strategy. For
example, Early Total Care and Damage Control Orthopaedics lead to translations
based on erroneous lexical selection: metode de control al daunelor, and
principii de îngrijire în primii ani de viaţă, respectively, instead of
retaining the original source terms in English. Moreover, for burst and burst
(suppression), the fine-tuned model produces the translations arsură and
(supresie) pulmonară belonging to the lexical fields of burn and bust,
pointing to challenges with the setup of the Byte pair encoding (BPE)
vocabulary in NMT (Araabi et al., 2022; Lignos et al., 2019). Secondly, when
acronyms should have been maintained as per the EN source, for instance MODS,
DCO, ARDS, and OS, they are instead randomly recomposed as SMO, COD, SRA, and
SSO. Acronyms corresponding to terms with a translation into Romanian are also
randomly recomposed, for example FR is translated as RF rather than RL.
Finally, there is also an example of a hallucination, the English
intramedullary (nailing) is erroneously translated by adding a Romanian
inflection at the end: (nailing) intramedullar, instead of tijă
centromedulară.

Source | Target (fine-tuned MBart)
---|---
The aim of this study was to evaluate the frontal intracortical connectivity during deep anaesthesia (burst-suppression). |  Scopul acestui studiu a fost să evalueze conectivitatea intracorticală frontală în timpul anesteziei profunde (supresie pulmonară).
Rats were maintained in deep level anaesthesia (burst-suppression). |  Ratii s-au menţinut în anestezie profundă (supresie pulmonară).
The global cortical connectivity (0.5-100 Hz) was 0.61 ± 0.078 during the burst periods compared to 0.55 ± 0.032. |  Conectivitatea corticală globală (0,5-100 Hz) a fost de 0,61 ± 0,078 în timpul perioadelor de arsură comparativ cu 0,55 ± 0,032.
The global cortical connectivity increased during the burst periods. |  Conectivitatea corticală globală a crescut în timpul perioadelor de arsură.
Once the "two event model" was accepted, it became clear that patients although initially resuscitated, but in a vulnerable condition, have a high risk that a secondary aggression (for example, surgical interventions) would precipitate a state of hyperinflammation and early multiple organ dysfunction syndrome (MODS). |  Odată ce „modelul celor două evenimente” a fost acceptat, a devenit clar că pacienţii, deşi iniţial resuscitaţi, dar aflaţi într-o stare vulnerabilă, prezintă un risc crescut ca o agresivitate secundară (de exemplu intervenţii chirurgicale) să precipite o stare de hiper inflamaţie şi sindrom de disfuncţie multiplă precoce (SMO).
This is a retrospective study of severe polytrauma patients with femoral shaft fractures admitted to the intensive care unit of the Emergency clinical Hospital of Bucharest and treated from an orthopaedic point of view by either Damage Control Orthopaedics (DCO) or Early Total Care (ETC) principles. |  Acesta este un studiu retrospectiv la pacienţi cu politraum sever, cu fracturi ale căilor femurale, internaţi în unitatea de terapie intensivă a Spitalului clinic de urgenţă din Bucureşti şi trataţi din punct de vedere ortopedic, fie conform principiilor de control al deteriorării (DCO), fie conform principiilor de îngrijire totală precoce (ETC).
This is a retrospective study of severe polytrauma patients with femoral shaft fractures admitted to the intensive care unit of the Emergency clinical Hospital of Bucharest and treated from an orthopaedic point of view by either Damage Control Orthopaedics (DCO) or Early Total Care (ETC) principles. |  Acesta este un studiu retrospectiv la pacienţi cu politraum sever, cu fracturi ale căilor femurale, internaţi în unitatea de terapie intensivă a Spitalului clinic de urgenţă din Bucureşti şi trataţi din punct de vedere ortopedic, fie conform principiilor de control al deteriorării (DCO), fie conform principiilor de îngrijire totală precoce (ETC).
Using patients files we recorded the following data: 30 day mortality, development of acute respiratory distress syndrome (ARDS) and MODS, local infectious complications (LIC), intensive care unit length of stay (ICU LOS), days of mechanical ventilation (MV), units of red blood cells units/48 h (RBC). |  Utilizând dosarele pacienţilor, am înregistrat următoarele date: mortalitate cu durata de 30 zile, apariţia sindromului de detresă respiratorie acută (SRA) şi MODS, complicaţii infecţioase locale (LIC), durata de şedere la unitatea de terapie intensivă (ICU LOS), zile de ventilaţie mecanică (MV), unităţi de celule roşii în sânge/48 ore (RBC).
We decided to analyze results in three groups - DCO group with shock on admission, DCO group without shock and ETC group. |  Am hotărât să analizăm rezultatele în trei grupuri - grupul cu COD cu soc la admitere, grupul cu COD fără soc si grupul cu ETC.
In the other two groups (DCO without shock and ETC) all outcomes were similar. |  În celelalte două grupuri (COD fără şoc şi ETC), toate rezultatele au fost similare.
In patients who are not in a very severe condition (shock), the choice for femoral shaft stabilization by intramedullary nailing represents a safe option. |  La pacienţii care nu sunt într-o afecţiune foarte severă (şoc), opţiunea stabilizării căilor femurale prin nailing intramedullar reprezintă o opţiune sigură.
The biochemical processes of bioproduction of free radicals (FR) are significantly increasing in polytrauma patients. |  Procesele biochimice de bioproducţie a radicalilor liberi (RF) cresc semnificativ la pacienţii cu politrauma.
Decreased plasma concentrations of antioxidants, correlated with a disturbance of the redox balance are responsible for the installation of the phenomenon called oxidative stress (OS). |  Scăderea concentraţiilor plasmatice de antioxidanti, corelată cu o tulburare a echilibrului redox, este responsabilă de instalarea fenomenului numit stres oxidativ (SSO).
Table 4: Fine-tuned MBart annotated examples for the Other error category. The
additional errors present in these examples have not been highlighted in this
table.

##  4 Conclusions and Future Work

We quantified the impact of domain adaptation on MBart in the medical domain
for English-Romanian. The fine-tuned MBart outperforms the general model with
automatic metrics and produces fewer errors ($\downarrow$ $34.7$%) related to
terminology in the relatively small sample ($75$ segments belonging to $12$
medical article abstracts) annotated by our annotators. While lower numbers of
errors were recorded in the Partial error, Source term copied, Disambiguation
issue in target, Incorrect lexical selection, and Term drop, in the three
remaining categories the fine-tuned MBart actually contained more errors than
general MBart: Inflectional error, Reorder error, and Other error.

Of these three categories, the Inflectional error, and Other error items
present in the fine-tuned MBart output we evaluated are related to the Byte
pair encoding (BPE) vocabulary. In future work, we plan to extend the BPE
vocabulary in MBart (Berard, 2021) to cope with in-domain terminology, and to
quantify the impact of the fine-tuning on other error types present in the MQM
Core. Moreover, we noticed further examples of hallucinations, but they were
not within the area of terminology translation, and we will leave them as
future work, alongside the additional types of errors noticed in the general
MBart and fine-tuned MBart outputs, but also in the reference translations,
which were by no means error-free.

More generally, it is essential to raise the awareness of machine translation
post-editors, as well as clients, regarding how these error categories are
still manifested in MT output even after fine-tuning. NMT output errors remain
difficult to identify due to the apparent fluency of the output, and even
subject-matter experts can miss some of them. The alert revision and
correction of MT output (which has been misleadingly called “postediting” for
the past $60$ years (Pierce et al., 1966) as if it were a monolingual task,
not a bilingual one) carries important risks in some settings if assigned to
only one person working under high time pressure and using the same text-based
revision environments created in the 1990s to accommodate translation
memories.

## Acknowledgments

The GPU used for this research was sponsored by the Google Cloud Research
Credits Program.

## References

  * Araabi and Monz [2020] A. Araabi and C. Monz.  Optimizing Transformer for Low-Resource Neural Machine Translation.  Technical Report arXiv:2011.02266, arXiv, Nov. 2020.  URL http://arxiv.org/abs/2011.02266.
  * Araabi et al. [2022] A. Araabi, C. Monz, and V. Niculae.  How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?, Aug. 2022\.  URL http://arxiv.org/abs/2208.05225.  Number: arXiv:2208.05225 arXiv:2208.05225 [cs].
  * Berard [2021] A. Berard.  Continual Learning in Multilingual NMT via Language-Specific Embeddings.  Technical Report arXiv:2110.10478, arXiv, Oct. 2021.  URL http://arxiv.org/abs/2110.10478.
  * Bérard et al. [2020] A. Bérard, Z. M. Kim, V. Nikoulina, E. L. Park, and M. Gallé.  A Multilingual Neural Machine Translation Model for Biomedical Data.  Technical Report arXiv:2008.02878, arXiv, Aug. 2020.  URL http://arxiv.org/abs/2008.02878.
  * Chu and Wang [2018] C. Chu and R. Wang.  A Survey of Domain Adaptation for Neural Machine Translation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1304–1319, Santa Fe, New Mexico, USA, Aug. 2018. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/C18-1111.
  * CLARIN:EL [2015] CLARIN:EL.  EMEA Corpus, 2015.  URL http://hdl.handle.net/11500/ATHENA-0000-0000-25DB-0.
  * Esperança-Rodier et al. [2019] E. Esperança-Rodier, F. Brunet-Manquat, and S. Eady.  ACCOLÉ: A Collaborative Platform of Error Annotation for Aligned Corpora.  In _Translating and the computer 41_ , Londres, United Kingdom, Nov. 2019.  URL https://hal.archives-ouvertes.fr/hal-02363208.
  * Haque et al. [2019] R. Haque, M. Hasanuzzaman, and A. Way.  Investigating Terminology Translation in Statistical and Neural Machine Translation: A Case Study on English-to-Hindi and Hindi-to-English.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_ , pages 437–446, Varna, Bulgaria, Sept. 2019. INCOMA Ltd.  doi: 10.26615/978-954-452-056-4_052.  URL https://aclanthology.org/R19-1052.
  * Johnson et al. [2017] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean.  Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation.  _Transactions of the Association for Computational Linguistics_ , 5:339–351, 2017.  doi: 10.1162/tacl_a_00065.  URL https://aclanthology.org/Q17-1024.
  * Klubička et al. [2017] F. Klubička, A. Toral, and V. M. Sánchez-Cartagena.  Fine-grained human evaluation of neural versus phrase-based machine translation.  _The Prague Bulletin of Mathematical Linguistics_ , 108(1):121–132, June 2017.  ISSN 1804-0462.  doi: 10.1515/pralin-2017-0014.  URL http://arxiv.org/abs/1706.04389.
  * Lignos et al. [2019] C. Lignos, D. Cohen, Y.-C. Lien, P. Mehta, W. B. Croft, and S. Miller.  The Challenges of Optimizing Machine Translation for Low Resource Cross-Language Information Retrieval.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3497–3502, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1353.  URL https://aclanthology.org/D19-1353.
  * Liu et al. [2020] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742, Dec. 2020.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00343.  URL https://direct.mit.edu/tacl/article/96484.
  * Lommel et al. [2013] A. R. Lommel, A. Burchardt, and H. Uszkoreit.  Multidimensional quality metrics: a flexible system for assessing translation quality.  page 7, 2013.
  * Neves et al. [2018] M. Neves, A. Jimeno Yepes, A. Névéol, C. Grozea, A. Siu, M. Kittner, and K. Verspoor.  Findings of the WMT 2018 Biomedical Translation Shared Task: Evaluation on Medline test sets.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 324–339, Belgium, Brussels, Oct. 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-6403.  URL https://aclanthology.org/W18-6403.
  * Ott et al. [2019] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling, Apr. 2019.  URL http://arxiv.org/abs/1904.01038.  arXiv:1904.01038 [cs].
  * Papineni et al. [2002] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu.  Bleu: a Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.  doi: 10.3115/1073083.1073135.  URL https://aclanthology.org/P02-1040.
  * Pierce et al. [1966] J. R. Pierce, J. B. Carroll, E. P. Hamp, D. G. Hays, C. F. Hockett, A. G. Oettinger, and A. Perlis.  Language and Machines: Computers in Translation and Linguistics.  Technical report, National Academy of Sciences/National Research Council, Washington, DC, USA, 1966.
  * Popović [2015] M. Popović.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics.  doi: 10.18653/v1/W15-3049.  URL https://aclanthology.org/W15-3049.
  * Post [2018] M. Post.  A Call for Clarity in Reporting BLEU Scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium, Oct. 2018\. Association for Computational Linguistics.  doi: 10.18653/v1/W18-6319.  URL https://aclanthology.org/W18-6319.
  * Rei et al. [2020] R. Rei, C. Stewart, A. C. Farinha, and A. Lavie.  COMET: A Neural Framework for MT Evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online, Nov. 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.213.  URL https://aclanthology.org/2020.emnlp-main.213.
  * Rei et al. [2021] R. Rei, A. C. Farinha, C. Stewart, L. Coheur, and A. Lavie.  MT-Telescope: An interactive platform for contrastive evaluation of MT systems.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 73–80, Online, Aug. 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-demo.9.  URL https://aclanthology.org/2021.acl-demo.9.
  * Skianis et al. [2020] K. Skianis, Y. Briand, and F. Desgrippes.  Evaluation of Machine Translation Methods applied to Medical Terminologies.  In _Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis_ , pages 59–69, Online, Nov. 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.louhi-1.7.  URL https://aclanthology.org/2020.louhi-1.7.
  * Verma et al. [2022] N. Verma, K. Murray, and K. Duh.  Strategies for Adapting Multilingual Pre-training for Domain-Specific Machine Translation.  In _Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 31–44, Orlando, USA, Sept. 2022. Association for Machine Translation in the Americas.  URL https://aclanthology.org/2022.amta-research.3.


# ALICE ITS 3: the first truly cylindrical inner tracker

Domenico Colella

###### Abstract

The high integration density of MAPS, with silicon sensor and readout
electronics implemented in the same device, allows very thin structures with a
greatly reduced material budget. Thicknesses of $\mathcal{O}$(50 $\mu$m),
values at which silicon chips become flexible, are readily used in many
applications. In addition, MAPS can be produced in sensors of wafer size by a
process known as stitching. This in turn allows to build detector elements
that are large enough to cover full tracker half-layers with single bent
sensors. The ALICE ITS 3 project is planning to build a new vertex tracker
based on truly cylindrical wafer-scale sensors, with <0.05% X/X0 per layer and
located as close as 18 mm to the interaction point. R&D on all project aspects
(including mechanics for bent wafer-scale devices, test beams of bent MAPS,
design of stitched sensors) is rapidly progressing with the aim for
installation during LHC long shutdown 3 (2025–2027). This contribution
summarises the project motivation, its R&D schedule, and will show selected
highlights of recently accomplished project milestones, including full-scale
engineering prototypes with dummy chips and small-scale, fully functional
assemblies of functional, bent MAPS.

##  1 ITS 3 project and R&D activities

The ALICE Collaboration [1] has just completed detector upgrades for data-
taking during LHC Run 3 [2]. During this major enterprise the Inner Tracking
System (ITS) used during LHC Run 1 and Run 2 has been replaced by a new
detector, the ITS 2, based on Monolithic Active Pixel Sensors (MAPS) [3]. To
achieve the pointing resolution needed for the defined ALICE physics program
during LHC Run 3 and Run 4, the detector has been designed to have very high
intrinsic spatial resolutions (5 $\mu$m), to be placed close to the
interaction point (23 mm for the innermost layer), and to feature low material
budget (0.35% X/X0 for the inner layers). Details of the angular distribution
of the material budget for two adjacent modules of the innermost layer of the
ITS 2 can be seen in Figure 1 (left). The following considerations can be
made: i) material budget is, to large extent, formed by passive components
such as water cooling, carbon and kapton support structures, and aluminum
wires; ii) a lot of irregularities, e.g. due to the overlap of adjacent
modules (needed to grant hermeticity) or due to the presence of the water
cooling pipes, are visible; iii) silicon contributes only about 15% of the
total material budget. Based on these observations, ALICE has started a new
R&D programme towards next-generation inner-tracker layers, ITS 3, that aims
to reduce the material budget to the minimum by removing the passive
components and keeping just the silicon layer. Such a detector, designed to
replace the three innermost layers of the ITS 2, would achieve an
unprecedented low material budget of about 0.05% X/X0 per layer [4].

Figure 1: Left: Material budget angular distribution of two adjacent modules
in the ITS 2 configuration. Right: proposed design for the inner barrel of the
ITS in Run 4.

The construction of the detector would require (i) to reduce the power
consumption below 20 mW/cm2, in order to cool down the sensors only by
airflow, (ii) to integrate power and data buses on the chip, in order to
remove any other flex covering sensitive region, and (iii) to rely on the
stiffness of large size, bent silicon wafers, in order to remove mechanical
support structures. The first two points require the design of a new chip
while the last point needs R&D activities for sensor bending and light support
development. The composition of the new detector is shown in Figure 1 (right).
Each layer is made up of two large dimension silicon chips (largest one having
a surface area of 280 $\times$ 94 mm2) held in place using carbon foam
structures and electrically connected only from a single side in a way that
makes them compatible with the existing infrastructure of ITS 2. The
installation of a new beam-pipe with smaller radius (inner radius 16 mm) and
thickness (500 $\mu$m), will allow to place the layer even closer to the first
interaction point (18 mm). Performance studies indicate an improvement by a
factor 2 of the pointing resolution and of the stand-alone tracking efficiency
(for $p_{\rm T}<$ 100 MeV/c) with respect to ITS 2. The improvement of the
vertexing performance and the reduction of material budget will have a
dramatic impact on the measurement of charm and beauty hadrons at low
transverse momentum as well as on the measurement of low-mass and low-$p_{\rm
T}$ dielectrons.

###  1.1 Detector integration

A 3-point bending test, made with ALPIDE [5] chips (MAPS sensors used for the
assembly of the ITS 2) at four different thicknesses (between 100 and 30
$\mu$m), measuring the displacement from flat position as a function of the
applied force, demonstrates that ITS 3 target radii are easily reachable with
silicon thickness below 50 $\mu$m. As demonstrated with ALPIDE chips, the
power consumption of the pixel matrix can stay quite low (40 mW/cm2),
concentrating most of it in the periphery where the readout electronics are
integrated. A combination of air flow and contact dissipation with carbon foam
support structures is a viable option to cool-down the sensors, removing the
cooling pipes and coolant in the active volume. An extensive study established
the best commercially available carbon foam materials to be used for
mechanical supports, targeting an excellent thermal conductivity with a low
density. A full size mechanical prototype, using large dimension (50 $\mu$m
thick) blank wafer, has been assembled, allowing the development of bending
tools and verification of carbon foam support structures. In Figure 2 one can
see how the chips are held in place using carbon foam wedges, whose material
budget contribution is negligible. The assembly is now being characterised for
its mechanical properties and shape imperfections, and serves as reference for
further optimisation of the procedure.

Figure 2: Left: Large dimension chips bending tool. Right: ITS 3 mechanical
prototype assembly.

###  1.2 Bent sensor characterisation

The ALPIDE sensor characterised in the laboratory showed no effect of bending
on the performance; the noise level and number of dead pixels remained
unchanged, and the difference in pixel threshold was negligible [6]. The
detection efficiency of a single bent ALPIDE sensor was measured at the DESY
test beam facility, using a 5 GeV electron beam. In Figure 3 (left), it can be
observed that below 100 e- the hit inefficiency is generally lower than 10-4,
independent of the incident angle or the position on the chip. An assembly,
made of 6 ALPIDE sensors bent around cylinders of ITS 3 target radii, called
$\mu$ITS3 (Figure 3, right), and with open windows in correspondence to the
sensors, has been tested under beam to verify tracking and vertexing
capabilities and the data analysis is currently ongoing.

Figure 3: Left: hit inefficiency as a function of threshold for different rows
and incidence angles as measured in a bent ALPIDE. Right: micro-ITS 3
assembly.

###  1.3 Wafer-sensor design

The required low value for the power consumption can be achieved by moving the
sensor periphery, including the serial link, to the edge of the chip and by
the usage of the 65 nm CMOS technology (ALPIDE was fabricated in 180 nm),
allowing also implementation of smaller pixels. Tower Semiconductor is able to
provide such technology on 300 mm-large wafers (180 nm is done on 200 mm
wafers), allowing to produce sensors with the needed dimensions. Still this
requires the application of a technique, known as stitching, that allows to
overcome the chip mask or reticle size limitations (typically of the order of
2 $\times$ 3 cm2). New chip design in 65 nm reached its first milestone in
June 2021 with the production of the multi-layer reticle 1 (MLR1) including
first test structures like: transistor test structures, analog building
blocks, various diode matrices and digital test matrices. Characterisation of
these structures in laboratory and under beam is ongoing. Next important
milestone will be the first engineer run including the first implementation of
stitching technique.

## References

  * [1] The ALICE Collab., _The ALICE experiment at the CERN LHC_ , _JINST_ 3 (2008) S08002.
  * [2] The ALICE Collab., _Upgrade of the ALICE Experiment: Letter Of Intent_ , _J. Phys. G_ 41 (2014) 087001.
  * [3] The ALICE Collab., _Technical Design Report for the Upgrade of the ALICE Inner Tracking System_ , _J. Phys. G_ 41 (2014) 087002.
  * [4] The ALICE Collab., _Letter of Intent for an ALICE ITS Upgrade in LS3_ , CERN-LHCC-2019-018.
  * [5] G. Aglieri Rinella (on behalf of ALICE Collaboration), _The ALPIDE pixel sensor chip for the upgrade of the ALICE Inner Tracking System_ , Nucl. Instrum. Methods Phys. Res. A 845, 583 (2017).
  * [6] The ALICE ITS Project, _First demonstration of in-beam performance of bent Monolithic Active Pixel Sensors_ , arXiv:2105.13000v2.