Enhancing Neural Machine Translation of Low-Resource Languages:

Corpus Development, Human Evaluation and Explainable AI Architectures

Séamus Lankford MSc, MBA, BEng

Supervised by Prof. Andy Way
and Dr Haithem Afli (Munster Technological University)

A Thesis Submitted for the Award of Doctor of Philosophy

School of Computing
Dublin City University

January 2024

## Declaration

I hereby certify that this material, which I now submit for assessment on the
programme of study leading to the award of Doctor of Philosophy (PhD) is
entirely my own work, and that I have exercised reasonable care to ensure that
the work is original, and does not to the best of my knowledge breach any law
of copyright, and has not been taken from the work of others save and to the
extent that such work has been cited and acknowledged within the text of my
work.

Signed:
ID No.: 20216607
Date: January 2nd, 2024

## Dedication

Letty agus Cormac atá caillte. I ndorn Dé go raibh a n-anamacha dílse.

## Acknowledgements

I wish to thank my PhD supervisors, Professor Andy Way and Dr Haithem Afli for
their unwavering support. Right throughout, Andy has been an exceptional
mentor in my career development as a researcher. Despite a busy schedule, he
was always available for our scheduled meetings and I was given a great deal
of autonomy in how my research should be directed. Furthermore, Andy stepped
in to steer me in the right direction whenever needed. Given his decades of
experience, Andy helped in identifying research gaps and opportunities. It is
a testament to Andy’s professionalism that he has retained his drive and
interest in developing his students throughout this career and I consider
myself fortunate that our paths have crossed. Likewise, I count myself lucky
to have worked with Haithem who not only adopted the role of supervisor but
also became an important ally in research projects not directly linked to my
PhD. Haithem always exudes a positive attitude which helped greatly when the
going got tough. The funding received from MTU was very much appreciated as
was the support from Dr Teresa Lynn in the early days of my PhD journey. I
must also thank my wife, Helen, who as always was steadfast in her support
despite the countless hours I put in. My children, Darragh and Julie must also
be acknowledged, not only for their support but also for their love of the
Irish language which has been, in part, an inspiration for this work. Táim
faoi chomaoin acu.

## Motivation

The seeds of my interest in the fields of deep learning (DL) and natural
language processing (NLP) were sown while taking an MSc in Artificial
Intelligence during the Covid-19 pandemic. The decision to pursue such an MSc
was motivated by the rise of artificial intelligence (AI) which can be used
both as a force for positive and negative change. Motivated by how AI could be
used to impact society positively, I signed up for the MSc programme which
culminated with the publication of my mini-thesis: Automatic Neural
Architecture using AutoML, Swarm Intelligence and Ensemble methods.

I grew up in a family where Irish was spoken with ease as a second language
which created an awareness that language is a matter of inheritance, in the
same way our place and family are. In my case, my second language was an
inheritance from two generations previously, for I had the privilege of
knowing and interacting with bilingual grandparents. They had acquired their
early speech patterns in the 1890s, when vernacular in Ireland was still
transitioning from Irish to English.

Given my background as an Irish language speaker, it was clear the
foundational principles which I learned throughout the MSc could be applied in
the context of a PhD study focused on the neural machine translation (NMT) of
low-resource languages. Initially, the research questions were not immediately
clear. However, it soon emerged that enhancing NMT of low-resource languages
should concentrate on three key pillars namely corpus development, improved
human evaluation and the development of transparent and easily understood
open-source NMT architectures.

## List of Publications

  1. 1.

Lankford, S., Afli, H. and Way, A., 2023. adaptMLLM: Fine-Tuning Multilingual
Language Models on Low-Resource Languages with integrated LLM playgrounds. In:
Information 14.12, issn: 2078-2489, doi: 10.3390/info14120638, url:
https://www.mdpi.com/2078-2489/14/12/638.

  2. 2.

Lankford, S., Afli, H. and Way, A., 2023. adaptNMT: an open-source, language-
agnostic development environment for neural machine translation. In: Language
Resources and Evaluation, pp.1-26. url:
https://doi.org/10.1007/s10579-023-09671-2

  3. 3.

Lankford, S., Afli, H. and Way, A., 2023. Design of an Open-Source
Architecture for Neural Machine Translation. In: Proceedings of the 1st
Workshop on Open Community-Driven Machine Translation. Tampere, Finland:
European Association for Machine Translation, pp. 15–20.
url: https://aclanthology.org/2023.crowdmt-1.2.

  4. 4.

Lankford, S., Afli, H. and Way, A., 2022. Human Evaluation of English-Irish
Transformer-Based NMT. In: Information 13.7, issn: 2078-2489. doi:
10.3390/info13070309. url: https://www.mdpi.com/2078-2489/13/7/309.

  5. 5.

Lankford, S., Afli, H., Ní Loinsigh Ó., Way, A., 2022. “gaHealth: An
English–Irish Bilingual Corpus of Health Data”. In: Proceedings of the
Thirteenth Language Resources and Evaluation Conference. Marseille, France:
European Language Resources Association, pp. 6753–6758.
url: https://aclanthology.org/2022.lrec-1.727.

  6. 6.

Lankford, S., Afli, H., and Way, A. 2021. “Transformers for Low-Resource
Languages: Is Féidir Linn!” In: Proceedings of Machine Translation Summit
XVIII: Research Track. Virtual: Association for Machine Translation in the
Americas, pp. 48–60. url: https://aclanthology.org/2021.mtsummit-research.5.

  7. 7.

Lankford, S., Afli, H., and Way, A. 2021. “Machine Translation in the Covid
domain: an EN$\leftrightarrow$GA case study for LoResMT 2021”. In: Proceedings
of the 4th Workshop on Technologies for MT of Low Resource Languages
(LoResMT2021). Virtual: Association for Machine Translation in the Americas,
pp. 144–150. url: https://aclanthology.org/2021.mtsummit-loresmt.15

###### Contents

  1. 1 Introduction
    1. 1.1 Neural Networks: an overview
    2. 1.2 Hyperparameters and Subword Models
    3. 1.3 Neural Machine Translation
    4. 1.4 Research Questions
    5. 1.5 Thesis Outline
    6. 1.6 Research Contributions
    7. 1.7 Publications
  2. 2 Transformers for Low-Resource Languages
    1. 2.1 Context
    2. 2.2 Abstract
    3. 2.3 Introduction
    4. 2.4 Background
      1. 2.4.1 Irish Language
      2. 2.4.2 Hyperparameter Optimisation
      3. 2.4.3 Subword Models
    5. 2.5 Proposed Approach
      1. 2.5.1 Architecture Tuning
      2. 2.5.2 Subword Models
    6. 2.6 Empirical Evaluation
      1. 2.6.1 Experimental Setup
      2. 2.6.2 Results
    7. 2.7 Environmental Impact
    8. 2.8 Discussion
    9. 2.9 Conclusion
  3. 3 MT in the Covid domain: Shared Task for LoResMT2021
    1. 3.1 Context
    2. 3.2 Abstract
    3. 3.3 Introduction
    4. 3.4 Background
      1. 3.4.1 Transformer
      2. 3.4.2 Domain Adaptation
    5. 3.5 Proposed Approach
      1. 3.5.1 Architecture Tuning
    6. 3.6 Empirical Evaluation
      1. 3.6.1 Experimental Setup
      2. 3.6.2 Results
    7. 3.7 Discussion
    8. 3.8 Conclusion and Future Work
  4. 4 gaHealth: EN$\leftrightarrow$GA Bilingual Corpus of Health Data
    1. 4.1 Context
    2. 4.2 Abstract
    3. 4.3 Introduction
    4. 4.4 Related work
      1. 4.4.1 Transformer
      2. 4.4.2 Transformer Hyperparameter Optimisation
      3. 4.4.3 Neural MT
    5. 4.5 Proposed Approach
      1. 4.5.1 Sources for gaHealth Development
      2. 4.5.2 Toolchain used for gaHealth Development
      3. 4.5.3 Guidelines
      4. 4.5.4 Transformer Architecture
    6. 4.6 Empirical Evaluation
      1. 4.6.1 Infrastructure
      2. 4.6.2 Metrics
      3. 4.6.3 Results: Automatic Evaluation
    7. 4.7 Discussion
    8. 4.8 Conclusion and Future Work
  5. 5 Human Evaluation of EN$\leftrightarrow$GA Transformer-Based NMT
    1. 5.1 Context
    2. 5.2 Abstract
    3. 5.3 Introduction
    4. 5.4 Background
      1. 5.4.1 Hyperparameter Optimisation
      2. 5.4.2 SentencePiece
      3. 5.4.3 Human Evaluation
    5. 5.5 Proposed Approach
      1. 5.5.1 Architecture Tuning
      2. 5.5.2 Subword Models
      3. 5.5.3 Human Evaluation of NMT
    6. 5.6 Empirical Evaluation
      1. 5.6.1 Experimental Setup
      2. 5.6.2 Automatic Evaluation Results
      3. 5.6.3 Human Evaluation Results
    7. 5.7 Environmental Impact
    8. 5.8 Discussion
      1. 5.8.1 Inter-Annotator Reliability
      2. 5.8.2 Performance of Models Relative to Google
      3. 5.8.3 Linguistic Observations
      4. 5.8.4 Limitations of the Study
    9. 5.9 Conclusions and Future Work
  6. 6 Design of an Open-Source Architecture for NMT
    1. 6.1 Context
    2. 6.2 Abstract
    3. 6.3 Credits
    4. 6.4 Introduction
    5. 6.5 Related Work
      1. 6.5.1 NMT
      2. 6.5.2 NMT Tools
    6. 6.6 Architecture of adaptNMT
      1. 6.6.1 adaptNMT
      2. 6.6.2 Infrastructure
    7. 6.7 Discussion
    8. 6.8 Conclusion and Future Work
  7. 7 adaptNMT: Open-Source Neural Machine Translation
    1. 7.1 Context
    2. 7.2 Abstract
    3. 7.3 Graphical abstract
    4. 7.4 Introduction
    5. 7.5 Neural Networks for MT
      1. 7.5.1 Recurrent Neural Network Architecture
      2. 7.5.2 Transformer Architecture
      3. 7.5.3 Attention
      4. 7.5.4 NMT
      5. 7.5.5 Subword Models
      6. 7.5.6 NMT Tools
      7. 7.5.7 Hyperparameter Optimisation
    6. 7.6 Architecture of adaptNMT
      1. 7.6.1 adaptNMT
      2. 7.6.2 serverNMT
    7. 7.7 Empirical Evaluation
      1. 7.7.1 Infrastructure
      2. 7.7.2 Metrics
      3. 7.7.3 Results: Automatic Evaluation
      4. 7.7.4 Environmental Impact
      5. 7.7.5 Stochastic Nuances
    8. 7.8 Discussion
    9. 7.9 Conclusion and Future Work
  8. 8 adaptMLLM: Fine-Tuning Multilingual Language Models
    1. 8.1 Context
    2. 8.2 Abstract
    3. 8.3 Graphical abstract
    4. 8.4 Introduction
    5. 8.5 Related Work
      1. 8.5.1 Transformer Architecture
      2. 8.5.2 Multilingual Language Models - NLLB
      3. 8.5.3 Large Language Models
      4. 8.5.4 DeepSpeed
      5. 8.5.5 HuggingFace
      6. 8.5.6 Human Evaluation
    6. 8.6 Datasets
      1. 8.6.1 Language Pairs
      2. 8.6.2 Shared Task Datasets
    7. 8.7 Approach
      1. 8.7.1 Initialisation and Pre-processing
      2. 8.7.2 Modes of Operation
      3. 8.7.3 Fine-tuning and Visualisation
      4. 8.7.4 Deployment
      5. 8.7.5 Green Report
      6. 8.7.6 MLLM Translation and Evaluation
      7. 8.7.7 LLM Playgrounds
    8. 8.8 Empirical Evaluation
      1. 8.8.1 Infrastructure and Hyperparameters
      2. 8.8.2 Results: Automatic Evaluation
      3. 8.8.3 Human Evaluation Results
      4. 8.8.4 Environmental Impact
    9. 8.9 Discussion
      1. 8.9.1 Performance of adaptMLLM Relative to Google Translate
      2. 8.9.2 Linguistic Observations
    10. 8.10 Conclusion and Future Work
    11. 8.11 Limitations of the Study
  9. 9 Conclusion
    1. 9.1 Research contributions
      1. 9.1.1 HPO for Low-resource Languages
      2. 9.1.2 Corpus Development and Guidelines
      3. 9.1.3 Human Evaluation of Low-resource Languages
      4. 9.1.4 Explainable AI Architectures
    2. 9.2 Research Impact
    3. 9.3 Lessons Learnt
    4. 9.4 Future Work
      1. 9.4.1 Hyperparameter Tuning
      2. 9.4.2 Corpus Development
      3. 9.4.3 Human Evaluation
      4. 9.4.4 Explainable AI Architectures
    5. 9.5 Final Remarks
  10. A No Language Left Behind

###### List of Figures

  1. 1.1 Simplified feed-forward NN
  2. 1.2 Timeline of publications and mapping to thesis chapters.
  3. 2.1 Proposed approach of Transformers for low-resource languages
  4. 2.2 BLEU performance for all model architectures
  5. 2.3 TER performance for all model architectures
  6. 2.4 Training DGT Transformer baseline
  7. 2.5 Training DGT Transformer 16k BPE
  8. 3.1 Proposed approach for LoResMT2021 Shared Task
  9. 3.2 Proposed approach of MT in Covid domain
  10. 3.3 Translation performance using Transformers with 2 heads
  11. (a) BLEU
  12. (b) TER
  13. 4.1 Corpus development process
  14. 4.2 gaHealth en2ga* system: training EN$\rightarrow$GA model
  15. 4.3 adapt covid_extended system:training EN$\rightarrow$GA model
  16. 4.4 gaHealth ga2en system: training GA$\rightarrow$EN model
  17. 5.1 Proposed approach to evaluate RNN and Transformer architectures
  18. 5.2 The core set of error categories proposed by the MQM guidelines.
  19. 5.3 BLEU performance for all model architectures
  20. 5.4 TER performance for all model architectures
  21. 5.5 Transformer baseline.
  22. 5.6 Transformer 16k BPE subword model
  23. 6.1 Proposed architecture for adaptNMT
  24. 7.1 Graphical abstract summarising the adaptNMT system
  25. 7.2 Transformer architecture using an encoder-decoder
  26. 7.3 Multi-head attention in the decoder
  27. 7.4 Neurons within an RNN
  28. 7.5 Encoder-decoder architecture
  29. 7.6 Beam Search Algorithm
  30. 7.7 Proposed architecture for adaptNMT
  31. 7.8 adaptNMT and serverNMT
  32. (a) Overview of adaptNMT. Key areas include initialisation, pre-processing, environment setup, visualisation, auto and custom NMT, training of subword model, training of main model, evaluation and deployment (cf. Section 7.6.1).
  33. (b) Overview of serverNMT. Highlighted cells include initialisation, environment setup, Anvil server, API functions, translation, model building, adaptNMT and running the server (cf. Section 7.6.2).
  34. 7.9 adapt covid_extended system
  35. 7.10 gaHealth en2ga* system: training EN$\rightarrow$GA model
  36. 7.11 gaHealth ga2en system: training GA$\rightarrow$EN model
  37. 8.1 Fine-tuned MLLM approach of adaptMLLM for EN$\leftrightarrow$GA translation
  38. 8.2  Fine-tuned MLLM approach of adaptMLLM for EN$\leftrightarrow$MR translation
  39. 8.3 Graphical abstract summarising the adaptMLLM system
  40. 8.4 Proposed architecture for adaptMLLM: a system for fine-tuning MLLMs
  41. 8.5 Overview of adaptMLLM
  42. A.1 Overview of the NLLB approach [32]

###### List of Tables

  1. 2.1 Hyperparameter optimisation for Transformer models
  2. 2.2 RNN performance on DGT dataset of 52k lines
  3. 2.3 RNN performance on PA dataset of 88k lines
  4. 2.4 Transformer performance on 52k DGT dataset
  5. 2.5 Transformer performance on 88k PA dataset
  6. 2.6 Samples of human reference translations
  7. 2.7 Transformer model compared with Google Translate
  8. 3.1 Datasets used in proposed approach for MT in Covid domain
  9. 3.2 Hyperparameter optimisation for Transformer models
  10. 3.3 Comparison of Transformer performance with 2 attention heads
  11. 3.4 Comparison of Transformer performance with 8 attention heads
  12. 4.1 Extracts from the gaHealth corpus
  13. 4.2 Sources used in corpus development
  14. 4.3 Hyperparameter optimisation for Transformer models
  15. 4.4 EN$\rightarrow$GA training, validation and test datasets
  16. 4.5 GA$\rightarrow$EN training, validation and test datasets
  17. 4.6 EN$\rightarrow$GA gaHealth system compared with LoResMT2021 systems.
  18. 4.7 GA$\rightarrow$EN gaHealth system compared with LoResMT2021 systems.
  19. 5.1 Transformer HPO using a random search approach
  20. 5.2 SQM levels explained
  21. 5.3 Description of error categories within the core MQM framework
  22. 5.4 Inter-annotator agreement using Cohen values
  23. 5.5 RNN performance on DGT dataset of 52k Lines
  24. 5.6 Transformer performance on 52k DGT dataset
  25. 5.7 Total errors found by each annotator using the MQM metric.
  26. 5.8 Transformer and RNN approach compared
  27. 5.9 Random samples of human reference translations from the test dataset.
  28. 5.10 Transformer model compared with Google Translate
  29. 5.11 Linguistic analysis of system outputs
  30. 5.12 Transformer approach compared to the RNN approach
  31. 7.1 Key features differentiating adaptNMT from Joey NMT
  32. 7.2 Hyperparameter optimisation for Transformer models
  33. 7.3 EN$\rightarrow$GA training, validation and test dataset distributions
  34. 7.4 GA$\rightarrow$EN training, validation and test dataset distributions
  35. 7.5 EN$\rightarrow$GA gaHealth system compared with LoResMT 2021
  36. 7.6 GA$\rightarrow$EN gaHealth systems compared with LoResMT2021
  37. 7.7 Stochastic differences between EN$\rightarrow$GA systems
  38. 7.8 Stochastic differences between GA$\rightarrow$EN systems
  39. 8.1 Hyperparameter Optimisation
  40. 8.2 EN${\rightarrow}$GA: adaptMLLM systems compared with LoResMT2021
  41. 8.3 GA${\rightarrow}$EN: adaptMLLM systems compared with LoResMT2021
  42. 8.4 EN${\rightarrow}$MR: adaptMLLM systems compared with LoResMT2021
  43. 8.5 MR${\rightarrow}$EN: adaptMLLM systems compared with LoResMT2021
  44. 8.6 SQM levels explained [40].
  45. 8.7 Annotator SQM scores for adaptMLLM systems
  46. 8.8 Description of error categories within the core MQM framework
  47. 8.9 System errors found by each annotator using the MQM metric
  48. 8.10 Fine-grained analysis with concatenated errors across both annotators
  49. 8.11 Inter-annotator agreement using Cohen values
  50. 8.12 Energy consumption during MLLM fine-tuning experiments
  51. 8.13 EN${\rightarrow}$GA test dataset of LoResMT2021: samples of human reference translations.
  52. 8.14 EN${\rightarrow}$GA fine-tuned MLLM model compared with Google Translate.
  53. 8.15 Samples of human reference translations from EN${\rightarrow}$MR LoResMT2021
  54. 8.16 EN${\rightarrow}$MR fine-tuned MLLM model compared with Google Translate
  55. 8.17 Linguistic analysis of EN$\rightarrow$GA system output

Enhancing Neural Machine Translation of Low-Resource Languages: Corpus
Development, Human Evaluation and Explainable AI Architectures

Séamus Lankford

Abstract

In the current machine translation (MT) landscape, the Transformer
architecture stands as the gold standard, especially for high-resource
language pairs. This research delves into its efficacy for low-resource
language pairs including both the English$\leftrightarrow$Irish and
English$\leftrightarrow$Marathi language pairs. Notably, the study identifies
the optimal hyperparameters and subword model type to significantly improve
the translation quality of Transformer models for low-resource language pairs.

The scarcity of parallel datasets for low-resource languages can hinder MT
development. To address this, we developed gaHealth, the first bilingual
corpus of health data for the Irish language. Focusing on the health domain,
models developed using this in-domain dataset exhibited very significant
improvements in BLEU score when compared with models from the LoResMT2021
Shared Task. A subsequent human evaluation using the multidimensional quality
metrics error taxonomy showcased the superior performance of the Transformer
system in reducing both accuracy and fluency errors compared to an RNN-based
counterpart.

Furthermore, this thesis introduces adaptNMT and adaptMLLM, two open-source
applications streamlined for the development, fine-tuning, and deployment of
neural machine translation models. These tools considerably simplify the setup
and evaluation process, making MT more accessible to both developers and
translators. Notably, adaptNMT, grounded in the OpenNMT ecosystem, promotes
eco-friendly natural language processing research by highlighting the
environmental footprint of model development. Fine-tuning of MLLMs by
adaptMLLM demonstrated advancements in translation performance for two low-
resource language pairs: English$\leftrightarrow$Irish and
English$\leftrightarrow$Marathi, compared to baselines from the LoResMT2021
Shared Task.

Feabhas a chur ar Mheaisínaistriúchán Néarach Teangacha Ísealacmhainne:
Forbairt Chorpais, Meastóireacht Dhaonna agus Ailtireacht IS Inmhínithe

Séamus Lankford

Achoimre

Sa tírdhreach meaisínaistriúcháin (MA) reatha, seasann ailtireacht an
Trasfhoirmeora mar an caighdeán is airde, go háirithe do phéirí teanga
ardacmhainne. Déanann an taighde seo tochailt isteach ar a éifeachtúlacht le
haghaidh péirí teanga nach bhfuil mórán acmhainní acu lena n-áirítear péirí
teanga Béarla$\leftrightarrow$Gaeilge agus Béarla$\leftrightarrow$Maratais. Go
suntasach, aithníonn an staidéar na hipear-pharaiméadair agus an cineál múnla
fho-fhocal chun feabhas suntasach a chur ar chaighdeán aistriúcháin mhúnlaí an
Trasfhoirmeora le haghaidh péirí teanga ísealacmhainne.

Féadtar leis an easnamh de thacair shonraí chomhthreomhara constaic a chur
roimh fhorbairt MA. Chun aghaidh a thabhairt air seo, d’fhorbraíomar gaHealth,
an chéad chorpas sláinte dátheangach do theanga na Gaeilge. Ag díriú ar an
bhfearann sláinte, léirigh múnlaí a forbraíodh ag baint úsáid as an tacar
sonraí seo feabhsuithe suntasacha i scór BLEU nuair a chuirtear iad i
gcomparáid le múnlaí ón Tasc Roinnte LoResMT2021.

Léirigh meastóireacht dhaonna ina dhiaidh sin a bhain úsáid as meádracha
ilghnéitheacha cáilíochta earráide tascsanmaíochta ardfheidhmíocht chóras an
Trasfhoirmeora maidir le hearráidí cruinnis agus líofachta araon a laghdú i
gcomparáid le macasamhail RNN-bunaithe.

Chomh maith leis sin, tugann an tráchtas seo adaptNMT agus adaptMLLM isteach,
dhá fheidhm fhoinse-oscailte sruthaithe d’fhorbairt, mionchoigeartú agus úsáid
mhúnlaí meaisínaistriúcháin néaraigh. Simplíonn na huirlisí an leagan amach
agus an próiseas meastóireachta go mór, rud a dhéanann MA níos inrochtana le
haghaidh forbróirí agus aistritheoirí araon.

Rud atá le tabhairt faoi ndeara ná go gcuireann adaptNMT, atá bunaithe i
ngnáthóg an OpenNMT, taighde próiseála teanga nádúrtha éiceabháiche chun cinn
trí achar timpeallachta an mhúnla fhorbartha a léiriú. Léirigh mionchoigeartú
MMLManna ag adaptMLLM forbairtí i leith feidhmíochta aistriúcháin le haghaidh
dhá phéire teanga ísealacmhainne: Béarla$\leftrightarrow$Gaeilge agus
Béarla$\leftrightarrow$Maratais, i gcomparáid le bonnlínte ón Tasc Roinnte
LoRESMT2021.

##  Chapter 1 Introduction

The digital age has ushered in a plethora of innovations in machine learning
(ML) and natural language processing (NLP), delivering solutions to difficult
problems. Machine translation (MT) is among the prominent areas that have
experienced a significant transformation over the years. While MT has made
commendable strides in various language pairs, challenges remain in deploying
these advanced models for low-resource languages. One such low-resource
language pair that presents unique challenges is the English-to-Irish pair
(EN$\leftrightarrow$GA).

The EU flags the importance of digital services being available in all
languages to ensure a level playing field and full access to services for
citizens, companies and governments (EU Digital Single Market 2018). This has
been reinforced by a 2018 EU Parliament decision endorsing language equality
in the digital
age111https://www.europarl.europa.eu/doceo/document/A-8-2018-0228_EN.html and
is a particularly acute issue for Ireland in fostering and protecting the
Irish language as the European Commission derogation of Irish as an official
EU language expired at the end of 2021.

A major part of this research involves developing applications and methods to
address the challenges of low-resource language technology while also
addressing the problem of data scarcity affecting deep learning (DL) for
digital engagement. Harnessing the full potential of neural machine
translation (NMT) for low-resource languages like EN$\leftrightarrow$GA is
multifaceted. This research helps deliver parity in support for Irish and
other less-resourced languages.

###  1.1 Neural Networks: an overview

To grasp the concept of a neural network (NN), we can refer to the simplified
diagram in Figure 1.1. While real-world NNs used for translating one language
to another are considerably more complex, this basic representation helps
offer a general insight into NMT. The input layer depicts the source language
words, whereas the output layer showcases potential translations suggested by
the NN. These interconnections have specific weights, which might be initially
set at random.

When the network proposes a translation, its accuracy is gauged by comparing
it to a known human expert translation. If there is a difference (or error)
between the predicted translation and the reference, as perhaps assessed by
BLEU [88], this error is fed back into the NN using back-propagation [97].
This process adjusts the connection weights and retrains the network in a
subsequent cycle, involving a complete forward and backward pass through the
NN.

Ideally, the network’s translation suggestions should come closer to the human
reference with each iteration, indicating improved weights. If errors persist,
the NN adjusts its weights, re-evaluates, and the cycle continues until no
improvements in the BLEU score are observed. After repeatedly refining the
network using a comprehensive set of examples from the training dataset, the
final model is set. This optimised NN can then be introduced to new sentences
it has not encountered before, marking the commencement of the actual
translation process.

Figure 1.1: Simplified representation of a feed-forward NN with 3 layers. A
matrix of weights, i-w1 to i-w6 connects the input and hidden layers whereas a
separate matrix h-w1 to h-w6 connects the hidden and output layers.

###  1.2 Hyperparameters and Subword Models

In the context of ML, hyperparameters are distinct from the typical model
parameters. While model parameters are adjusted during training (e.g., weights
and biases in a NN), hyperparameters are set before training begins. Common
examples of hyperparameters include the learning rate, which governs how much
model parameters are adjusted during training, batch size, which determines
the number of data points considered in each iteration, and epochs, specifying
how many times the algorithm will run through the entire training dataset.
Regularisation parameters, which can help in adding penalties on the magnitude
of parameters to prevent overfitting, are another set of important
hyperparameters.

The significance of hyperparameters cannot be overstated. Their proper
selection and tuning can dramatically influence the performance of the model.
A suitable learning rate, for instance, can be the difference between a model
that converges efficiently and one that does not converge at all. In scenarios
with limited data, such as with low-resource languages, hyperparameter choices
become even more critical. It is paramount to extract as much value as
possible from the scarce data available, and well-tuned hyperparameters
facilitate that.

Languages are diverse and rich, with some exhibiting significant morphological
variations. This variability becomes a challenge when training language
models, especially for languages that are under-represented in terms of
digitally available data. Subword models offer a solution by breaking words
into smaller, more manageable units. These units can range from individual
characters to larger subword chunks. For example, the word “unhappiness” could
be segmented into “un-”, “-happi-”, and “-ness” using a subword tokenization
approach.

For low-resource languages, the use of subword models brings multiple
advantages. Firstly, it helps in managing the vocabulary size. Instead of
handling every possible word form, the model deals with a more limited set of
subword units which makes training of models faster. This approach also
provides a mechanism for understanding and generating words the model has not
explicitly seen during training. Instead of relegating unfamiliar words to a
generic “unknown” category, the model can represent and process them using its
known subword units. Furthermore, in the realm of transfer learning, where
knowledge is transferred from data-rich languages to low-resource ones,
subword models perform well. The shared subword units across languages make
knowledge transfer more seamless.

Both hyperparameters and subword models play pivotal roles in language
modelling, particularly for languages with limited data. While hyperparameters
ensure the efficient and effective training of models, subword models offer a
practical approach to tokenization that accounts for the morphological
richness and data scarcity inherent in many of the world’s languages.

###  1.3 Neural Machine Translation

NMT has achieved impressive performance for many language pairs, especially
those with very large amounts of parallel data available (e.g., English to
French). However, for low-resource languages, which are languages with limited
available parallel data, there are challenges:

  * •

Data Scarcity: The primary challenge is the lack of substantial parallel
corpora. Neural models, especially the DL models used for NMT, require large
amounts of data to train effectively and generalise well to unseen inputs [58,
121, 109].

  * •

Noisy Training Data: To overcome the data scarcity problem, sometimes
researchers use web-crawled data or other less reliable sources to augment the
training set. Such data can introduce noise due to incorrect alignments or
low-quality translations [99, 87].

  * •

Overfitting: Due to the limited data, NMT models can overfit easily, which
means they might memorise the training data rather than generalising from it.
As a result, their performance on unseen data might be suboptimal [114, 18].

  * •

Morphological Complexity: Many low-resource languages are morphological rich,
meaning they have a large number of word forms due to inflection, derivation,
or compounding. This complexity can pose challenges for NMT models which might
not see enough examples of each morphological variant in the training data [9,
102].

  * •

Limited Pre-trained Models or Embeddings: For popular languages, there are
often pre-trained models or word embeddings available that can be fine-tuned
for specific tasks. For many low-resource languages, these might not exist,
forcing researchers to start from scratch [96].

  * •

Domain Mismatch: The available data for low-resource languages might be
restricted to specific domains (e.g., religious texts or legal documents).
Training on such a restricted domain can limit the extent to which a model can
generalise to other domains.

  * •

Evaluation Challenges: Even if a model is trained for a low-resource language,
evaluating its performance can be problematic due to the lack of standard
benchmarks, reference translations, or even native speakers who can assess the
translations’ quality.

  * •

Cultural and Contextual Nuances: All languages convey cultural and contextual
nuances. With limited data, the model might miss these subtleties, leading to
translations that might be technically correct but culturally or contextually
inappropriate or odd.

To address these challenges, researchers are exploring various methods, such
as using transfer learning, building multilingual models, applying data
augmentation techniques and developing new corpora. Translating low-resource
languages with high accuracy remains an active area of research. A core
objective of our NMT work is to address some of the challenges outlined above
by posing and answering the research questions in the next section. An in-
depth discussion of NMT architecture and the mathematical first principles
governing its operation are covered in Section 7.5.

###  1.4 Research Questions

Having introduced the concept of NMT in the context of low-resource languages,
we will now discuss the specific research questions this PhD thesis seeks to
address.

  * •

RQ1 \- How can hyperparameters and subword models be optimised for
Transformer-based MT in low-resource language settings?

  * •

RQ2 \- What is the benefit and impact of using in-domain datasets in improving
the performance of MT models for low-resource language pairs, and how can
these datasets be effectively developed and utilised?

  * •

RQ3 \- How do NMT systems, specifically Transformer and recurrent neural
network (RNN) based models, differ in terms of accuracy and fluency errors
when evaluated using a human evaluation technique such as the Multidimensional
Quality Metrics (MQM) error taxonomy?

  * •

RQ4 \- How can the process of NMT development, evaluation, and deployment be
streamlined for both developers and translators, while also considering
environmental sustainability?

##### RQ1 How can hyperparameters and subword models be optimised for
Transformer-based MT in low-resource language settings?

The effectiveness of fine-tuning the Transformer model for the low-resource
EN$\leftrightarrow$GA language pair was investigated in the paper submitted to
the MT Summit conference in 2021. Hyperparameter optimisation (HPO) and the
impact of different subword models on translation performance were evaluated.
The observations made as part of this study were used to choose the
hyperparameters for developing an EN$\rightarrow$GA model entered into the
LoResMT2021 Shared Task222https://machinetranslate.org/loresmt2021 [64]. The
focus of the shared task was to develop MT models for translating Covid-
related data. The experimental findings from this shared task were the basis
of a separate paper. This paper highlighted the role of careful selection of
Transformer hyperparameters and demonstrated that choosing a 16k BPE
SentencePiece submodel yielded high-performing translation models in a low-
resource setting [61].

The research question was also explored as part of the human evaluation of
EN$\leftrightarrow$GA Transformer-Based NMT where the impact of modifying
hyperparameter settings and regularisation techniques on the performance of
NMT models was evaluated [66].

The theme is further addressed in Chapters 7 and 8 which cover the research
papers on open-source architectures for fine-tuning NMT models [68] and MLLM
models [67].

##### RQ2 \- What is the impact of using small in-domain datasets in improving
the performance of MT models for low-resource language pairs, and how can
these datasets be effectively developed and utilised?

The fine-tuning experiments conducted as part of the winning LoResMT2021
Shared Task entry demonstrate that augmenting in-domain data, even by
relatively modest amounts, for EN$\leftrightarrow$GA translations in the Covid
domain leads to much better performance when compared with other techniques.

Given the success of the initial approach of using augmented data for fine-
tuning Covid models, a separate study was conducted that resulted in the
development of an in-domain health dataset. The development of the gaHealth
corpus333https://github.com/seamusl/gaHealth showcases the process and
benefits of focusing on specific domains for low-resource languages [63].

To build a bilingual corpus of health data, we selected multiple sources of
professionally translated documents from within the Irish government, all of
which are publicly available. In particular, the bilingual strategy statements
and annual reports of the Irish Department of Health since 2010 were chosen.

Furthermore, a dataset of Covid-related data, developed for a previous study
[64], was incorporated into a larger health dataset. This amalgamated corpus,
gaHealth, consists of 16,201 lines of parallel text.

The main contribution of this work is to present an ongoing translation
project that aims to build a parallel corpus, gaHealth, of health data for the
Irish language by fully utilising freely available parallel documents.

Due to the issues encountered during the conversion of PDF documents, we
developed guidelines for researchers developing low-resource corpora to aid in
the conversion process [63]. In addition to developing the gaHealth corpus, we
trained and evaluated translation models for in-domain health data.

There is no such corpus available according to the best of our knowledge, so
gaHealth has become a useful resource in the NLP community, especially for
those working with the Irish language domain. Addressing this research
question has helped to increase the resources available for a low-resource
language. The outcome of the work was published at the LREC 2022 conference in
Marseille, France.

##### RQ3 \- How do MT systems, specifically Transformer and RNN-based models,
differ in terms of accuracy and fluency errors when evaluated using a human
evaluation technique such as the Multidimensional Quality Metrics (MQM) error
taxonomy?

In addressing this research question, a quantitative fine-grained manual
evaluation was conducted which compared the performance of MT systems. Using
the MQM [75] error taxonomy, a human evaluation of the error types generated
by an RNN-based system and a Transformer-based system was conducted.
Furthermore, Scalar Quality Metrics (SQM) [40] were also used so a combined
approach helped end users to understand translation quality from a human,
rather than a solely automatic perspective.

An integral part of the human evaluation was to highlight the linguistic
weaknesses associated with certain MT architectures which were discovered as
part of the linguistic observations. Our findings show the best-performing
Transformer system significantly reduces both accuracy and fluency errors when
compared with an RNN-based model.

##### RQ4 \- How can the process of NMT development, evaluation, and
deployment be streamlined for both developers and translators, while also
considering environmental sustainability?

To address this research question, two separate open-source tools were
developed and journal papers describing them in detail were written [68, 67].
The manner in which the features of each tool addressed this research question
is outlined below.

  1. 1.

Streamlining for both developers and translators:

     * •

The applications are designed for both technical and non-technical users in
the field of MT.

     * •

The setup of the development environment and the creation of training,
validation, and test splits are simplified.

     * •

HPO is made user-friendly through an intuitive interface.

  2. 2.

Evaluation and Deployment:

     * •

Models can be evaluated using BLEU, TER [105] and ChrF [90].

     * •

Deployment is facilitated as a translation service within the application
itself.

  3. 3.

Environmental Sustainability:

     * •

The application contains a green report that flags power consumption and kgCO2
emissions generated during model development. The inclusion of this feature
creates an awareness of the environmental cost amongst developers and
translators. This helps in addressing the environmental sustainability aspect
of the research question.

###  1.5 Thesis Outline

The outline of the thesis provides an overview of the papers, the principal
motivation for the work and the key research contributions. The adopted
approach dedicates a chapter to each published paper which focuses on the
outlined research questions. At the start of each chapter, a context section
outlines the motivation for the research work and illustrates the flow in how
the research questions were addressed. The timeline of the paper publications
and their mapping to chapters within the thesis is illustrated in Figure 1.2.

Chapter 2: Transformers for Low-Resource Languages: This chapter examines the
effectiveness of the Transformer model, particularly in MT scenarios where
limited training data is available, such as with the EN$\leftrightarrow$GA
language pair. By testing model configuration parameters, the study found
significant performance improvements. Notably, the choice of subword model was
crucial, with SentencePiece models using both unigram and BPE methods being
examined. Overall, this paper demonstrates that Transformers can be efficient
for a low-resource language pair.

Chapter 3: Machine Translation in the Covid domain: an EN$\rightarrow$GA case
study for LoResMT2021 Shared Task. Focused on the specific challenge of
translating EN$\rightarrow$GA Covid-related data, this chapter describes the
paper which explored different domain adaptation techniques. The study also
introduced a unique EN$\leftrightarrow$GA dataset concerning health and
education in the Covid context. Remarkably, just by adding 5k lines to an 8k
in-domain baseline dataset, the BLEU score improved by a massive 27 points,
with the Transformer architecture delivering the best performance.

Chapter 4: gaHealth: An EN$\leftrightarrow$GA Bilingual Corpus of Health Data:
Highlighting the challenges and potential of MT for low-resource languages,
this chapter discusses the motivation for gaHealth, a bilingual corpus geared
towards health data in the EN$\leftrightarrow$GA language pair. The research
showcased how using a specific in-domain dataset, in this case for health,
could significantly improve translation performance. By leveraging the
gaHealth corpus, BLEU scores improved by a substantial 22.2 points when
stacked against models from the LoResMT2021 Shared Task. The paper also shared
linguistic guidelines for developing such datasets.

Chapter 5: Human Evaluation of EN$\leftrightarrow$GA Transformer-Based NMT:
The previous work focused on automatic evaluation whereas the ultimate
validation of the research effort can only be determined by a human analysis
of the MT output. Therefore the chapter focuses on human evaluation of
Transformer-based NMT for the EN$\leftrightarrow$GA pair. The paper found that
Transformer-optimised models significantly outperformed RNN models. A fine-
grained manual evaluation using the MQM error taxonomy identified that the
Transformer models considerably cut down on both accuracy and fluency errors.
The linguistic observations noted by our translators also highlight the
typical errors which occur in EN$\leftrightarrow$GA MT. Suggestions on how
these errors could potentially be mitigated are outlined in the future work
section of Chapter 9.

Chapter 6: Design of an Open-Source Architecture for Neural Machine
Translation: The paper presented at the inaugural CrowdMT workshop in Tampere
is discussed in this chapter. As a workshop paper, the research objective was
to present a high-level view of the adaptNMT architecture. A more in-depth
analysis of the tool is provided in the LREV journal paper which is covered in
Chapter 7.

Chapter 7: adaptNMT: an open-source, language-agnostic development environment
for Neural Machine Translation: This chapter includes the paper which
introduced adaptNMT, an integrated open-source application designed for both
newcomers and experts in MT. It simplifies various processes, from setting up
the environment to model training, while also enabling hyperparameter
customisation through a user-friendly interface. The application integrates
graphing for training progress visualisation and uses SentencePiece for
subword segmentation. One useful feature is the green report that brings to
light the environmental cost associated with model development in terms of
power consumption and kgCO2 emissions, thus encouraging researchers to choose
an alternative greener infrastructure.

Chapter 8: adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource
Languages with integrated LLM playgrounds: Continuing the theme of open-source
tools, this chapter describes the paper on adaptMLLM which describes the tool
designed to fine-tune MLLMs. This application is user-friendly, streamlining
the process of setting up the development environment and customising
hyperparameters. Tested on two low-resource language pairs,
EN$\leftrightarrow$GA and English-Marathi (EN$\leftrightarrow$MR), it
demonstrated significant performance improvements over the bespoke NMT system
which won the LoResMT2021 Shared Task.

Chapter 9: Conclusion: The concluding chapter revisits the research questions
and highlights the research contributions before finally discussing the
lessons learnt as well as avenues for future research. In the future work
section, roadmaps are laid out for future development work on the adaptNMT and
adaptMLLM applications. While the performance of MT systems developed through
adaptMLLM is higher, it also demands an infrastructure with a higher
specification. Therefore, it is anticipated there will be separate roles for
both applications going forward.

###  1.6 Research Contributions

The transformative potential of the Transformer model in MT is undeniable, as
its adoption has heralded state-of-the-art (SOTA) results for many language
pairs. However, as illustrated in our paper “Transformers for Low-Resource
Languages - is féidir linn!”,444The title of the paper was inspired by Obama’s
2011 presidential visit to Ireland where he adapted his campaign slogan “Yes
we can!” to its Gaelic equivalent: “Is Féidir Linn!”. when faced with limited
training data, even advanced NMT models like the Transformer may fall short of
expectations. This limitation underscores the necessity to fine-tune models
and optimise hyperparameters for better translation performance, particularly
for low-resource language pairs like EN$\leftrightarrow$GA. The pivotal role
of subword models, especially in determining MT translation performance in
low-resource scenarios was emphasised. In addressing RQ1, the Transformer
architecture was optimised yielding impressive gains in BLEU scores and other
performance metrics, thus bridging the gap posed by insufficient amounts of
training data.

The relevance and potential of domain-specific translation cannot be
overstated, especially in the current global landscape marked by unpredictable
challenges. The recent Covid-19 pandemic highlighted the urgent need for
accurate translation within the health domain, especially for languages that
lack large resources. In the paper, “MT in the Covid domain: an
EN$\leftrightarrow$GA case study for LoResMT2021 Shared Task”, cf. Chapter 3,
the efficacy of domain adaptation techniques was explored to address RQ2.
Through this research, the significance of developing specialised datasets for
domain-specific translation was highlighted. Augmenting in-domain data with a
modest amount of domain-specific data yielded MT models which obtained 1st
place in the LoResMT2021 Shared Task. Such findings emphasise the potential
that lies in domain-focused MT models and drive advancements in domain-
specific translation.

The idea of domain specificity was further developed by creating a corpus
specifically for the health sector. As highlighted in the paper “gaHealth: An
English–Irish Bilingual Corpus of Health Data”, cf. Chapter 4, there is a
marked absence of parallel datasets tailored for low-resource languages. While
the rush to build larger, more encompassing datasets is understandable, the
merits of focused, in-domain datasets have often been overshadowed. The
development of an in-domain corpus and subsequent results from training an MT
system with the gaHealth corpus demonstrate the tangible benefits of deploying
in-domain datasets for translation tasks. Moreover, in addressing RQ2, this
study has provided a valuable resource for the wider NLP community.

In addressing the research question posed by RQ3, a detailed human evaluation
was carried out to validate the translation outputs of RNN and Transformer
models for a low-resource language pair. The approach taken and the findings
are described in the paper, ‘‘Human Evaluation of English–Irish Transformer-
Based NMT”, cf. Chapter 5. We utilised the MQM error framework to investigate
the types of errors produced by both RNN and Transformer-based systems. Native
Irish speakers were collaborated with for this assessment, and one of the
research contributions was to establish how a human evaluation could be
conducted with limited resources. As such, we adopted a simplified MQM error
taxonomy, pairing it with an SQM. By engaging two native speakers and
analysing 25 reference translations, we were able to validate the outputs from
both the RNN and Transformer models. Our results indicate the top-performing
Transformer system notably minimises errors in both accuracy and fluency when
compared with its RNN counterpart. A significant takeaway from our study is
the linguistic insights shared by our translators, which are elaborated on in
the related paper (cf. Section 5.8.3).

These practices were combined into a new framework which was subsequently
applied when evaluating the output from MLLMs in our later work. Furthermore,
in a separate contribution, this human evaluation component was integrated
into the adaptNMT and adaptMLLM applications effectively bridging RQ3 with
RQ4.

The research question identified by RQ4 was addressed in the subsequent
journal paper, “adaptNMT: an open-source, language-agnostic development
environment for Neural Machine Translation”, cf. Chapter 7. The application
introduces an open-source application designed to simplify the NMT development
process. adaptNMT is a representation of the increasing demand for tools that
are intuitive, powerful, and user-friendly. One of the key features of this
tool is its commitment to sustainable NLP research, evident from its green
report, which calculates the environmental costs of model training. By
highlighting the environmental cost of training models, users are encouraged
to think about the carbon footprint of their research activities and are
consequently incentivised to use a greener infrastructure or pre-trained
models, where possible. Furthermore, adaptNMT’s role in the broader NLP
community is underlined by its open-source nature, inviting collaborative
enhancement from researchers worldwide.

Expanding on the foundation set by adaptNMT, RQ4 was further developed in the
paper “adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource
Languages with integrated LLM playgrounds”, cf. Chapter 8. Recognising the
transformative potential of MLLMs and LLMs, the paper showcases the very
significant gains in translation quality achieved by fine-tuning MLLMs and
LLMs for low-resource language pairs. The adaptMLLM tool democratises cutting-
edge NLP technologies, demonstrating that SOTA results can be achieved without
the backing of colossal research infrastructures. Moreover, its open-source
nature mirrors the ethos of adaptNMT, advocating for community-driven
advancements in the field.

Figure 1.2: Timeline of publications and mapping to thesis chapters. The
illustration highlights the initial set of papers on automated machine
learning (AutoML), followed by the MT papers and culminating in a book on the
topic of MT and automation [82].

###  1.7 Publications

My PhD research has culminated in the publication of eleven peer-reviewed
papers, and presentations both virtual and physical at conferences in Dublin,
Kyoto, Macau, Orlando and Tampere. Best presentation awards were received at
the ACM conferences in Kyoto (2020)555http://www.adip.org/2020.html and Macau
(2021).666http://www.aibc.org/2021.html Furthermore, I developed a translation
system for LoResMT2021 which obtained first place in the EN$\rightarrow$GA
shared task [85].

All of the recent work, comprising seven publications, focuses on advancements
in NMT and the development of user-friendly tools. These MT-related papers
have been included as individual chapters in this thesis. The full set of
peer-reviewed MT publications is outlined below.

  1. 1.

Lankford, S., Afli, H. and Way, A., 2023. adaptMLLM: Fine-Tuning Multilingual
Language Models on Low-Resource Languages with integrated LLM playgrounds. In:
Information 14.12, issn: 2078-2489, doi: 10.3390/info14120638, url:
https://www.mdpi.com/2078-2489/14/12/638.

  2. 2.

Lankford, S., Afli, H. and Way, A., 2023. adaptNMT: an open-source, language-
agnostic development environment for neural machine translation. In: Language
Resources and Evaluation, pp.1-26.
url: https://doi.org/10.1007/s10579-023-09671-2

  3. 3.

Lankford, S., Afli, H. and Way, A., 2023. Design of an Open-Source
Architecture for Neural Machine Translation. In: Proceedings of the 1st
Workshop on Open Community-Driven Machine Translation. Tampere, Finland:
European Association for Machine Translation, pp. 15–20.
url: https://aclanthology.org/2023.crowdmt-1.2.

  4. 4.

Lankford, S., Afli, H. and Way, A., 2022. Human Evaluation of English-Irish
Transformer-Based NMT. In: Information 13.7, issn: 2078-2489. doi:
10.3390/info13070309. url: https://www.mdpi.com/2078-2489/13/7/309.

  5. 5.

Lankford, S., Afli, H., Ní Loinsigh Ó., Way, A., 2022. “gaHealth: An
English–Irish Bilingual Corpus of Health Data”. In: Proceedings of the
Thirteenth Language Resources and Evaluation Conference. Marseille, France:
European Language Resources Association, pp. 6753–6758. url:
https://aclanthology. org/2022.lrec-1.727.

  6. 6.

Lankford, S., Afli, H., and Way, A. 2021. “Transformers for Low-Resource
Languages: Is Féidir Linn!” In: Proceedings of Machine Translation Summit
XVIII: Research Track. Virtual: Association for Machine Translation in the
Americas, pp. 48–60. url: https://aclanthology.org/2021.mtsummit-research.5.

  7. 7.

Lankford, S., Afli, H., and Way, A. 2021. “Machine Translation in the Covid
domain: an EN$\leftrightarrow$GA case study for LoResMT 2021”. In: Proceedings
of the 4th Workshop on Technologies for MT of Low Resource Languages
(LoResMT2021). Virtual: Association for Machine Translation in the Americas,
pp. 144–150. url: https://aclanthology.org/2021.mtsummit-loresmt.15

##  Chapter 2 Transformers for Low-Resource Languages

###  2.1 Context

The motivation for this research stems from the recognition that Transformer
models are now SOTA in the MT arena, offering remarkable performance on well-
resourced language pairs. However, such models often struggle when dealing
with low-resource language pairs due to the scarcity of training data. This
limitation has led to relatively few experiments and advancements in using
Transformers for such language pairs. As a result, my motivation was to
address RQ1 by exploring and optimising Transformer models specifically for
translating low-resource languages such as the EN$\leftrightarrow$GA language
pair.

The key motivations and objectives of this research can be summarised as
follows:

  * •

Addressing Low-Resource Challenges: The research seeks to tackle the
challenges posed by low-resource language pairs, where conventional NMT models
tend to perform poorly due to insufficient training data. The motivation is to
bridge the performance gap and enhance translation quality for
underrepresented language pairs.

  * •

Hyperparameter Optimisation: The study focuses on hyperparameter optimisation
for Transformer models. This is motivated by the understanding that the
performance of MT models is heavily influenced by the choices made in
configuring the model’s hyperparameters. The goal is to identify the most
effective set of hyperparameters for EN$\leftrightarrow$GA translation.

  * •

Subword Model Selection: Another key motivation is to determine the optimal
subword model for low-resource translation. This choice is critical as it
impacts the model’s ability to handle the linguistic nuances of the target
language. The research evaluates different subword models, including
SentencePiece models with unigram and BPE approaches.

  * •

Model Architecture Exploration: Various aspects of the Transformer
architecture were investigated to enhance translation performance. This
included modifying the number of layers, testing regularisation techniques,
and evaluating the ideal number of attention heads. The goal is to identify
the best architectural configurations for the given language pair.

  * •

Benchmarking and Comparison: To assess the effectiveness of the optimised
Transformer models, the research benchmarks them against existing translation
systems, including Google Translate. This comparison provides empirical
evidence of the improvements achieved by their proposed approach.

  * •

Reducing Post-Editing Effort: By improving translation quality, the research
aims to reduce the post-editing effort required for low-resource language
pairs. This has practical implications for making MT more accessible and cost-
effective in scenarios where human expertise is required for quality control.

In summary, the motivation for this paper is to address the challenges faced
by low-resource language pairs in MT by optimising Transformer models. The
study explores hyperparameter selection, subword modelling, and architectural
modifications to achieve substantial performance improvements, ultimately
making MT more effective and accessible for EN$\leftrightarrow$GA translation.

Transformers for Low-Resource Languages: Is Féidir Linn!

Séamus Lankford
Haithem Afli
Andy Way

Proceedings of Machine Translation Summit XVIII

Research Track. Virtual: Association for Machine Translation in the Americas
August 16 - 20, 2021

Florida, USA

ADAPT Centre

Dublin City University

Ireland

https://aclanthology.org/2021.mtsummit-research.5.pdf

###  2.2 Abstract

The Transformer model is state-of-the-art in MT. However, in general, neural
translation models often underperform on language pairs with insufficient
training data. As a consequence, relatively few experiments have been carried
out using this architecture on low-resource language pairs. In this study,
hyperparameter optimisation of Transformer models in translating the low-
resource English-Irish language pair is evaluated. We demonstrate that
choosing appropriate hyperparameters leads to considerable performance
improvements. Most importantly, the correct choice of subword model is shown
to be the biggest driver of translation performance. SentencePiece models
using both unigram and BPE approaches were appraised. Variations on model
architectures included modifying the number of layers, testing various
regularisation techniques and evaluating the optimal number of heads for
attention. A generic 55k DGT corpus and an in-domain 88k public admin corpus
were used for evaluation. A Transformer-optimised model demonstrated a BLEU
score improvement of 7.8 points when compared with a baseline RNN model.
Improvements were observed across a range of metrics, including TER,
indicating a substantially reduced post-editing effort for Transformer-
optimised models with 16k BPE subword models. Benchmarked against Google
Translate, our translation engines demonstrated significant improvements. The
question of whether or not Transformers can be used effectively in a low-
resource setting of English-Irish translation has been addressed. Is féidir
linn - yes we can.

###  2.3 Introduction

The advent of neural machine translation (NMT) has heralded an era of high-
quality translations. However, these improvements have not been manifested in
the translation of all languages. Large datasets are a prerequisite for high-
quality NMT. This works well in the context of well-resourced languages where
there is an abundance of data. In the context of low-resource languages which
suffer from a sparsity of data, alternative approaches must be adopted.

An important part of this research involves developing applications and models
to address the challenges of low-resource language technology. Such technology
incorporates methods to address the data scarcity affecting deep learning for
digital engagement of low-resource languages.

It has been shown that an out-of-the-box NMT system, trained on English-Irish
(EN$\leftrightarrow$GA) data, achieves a lower translation quality compared
with using a tailored SMT system (Dowling et al, 2018). It is in this context
that further research is required in the development of NMT for low-resource
languages and the Irish language in particular.

Most research on choosing subword models has focused on high-resource
languages [36, 43]. In the context of developing models for
EN$\leftrightarrow$GA translation, there are no clear recommendations on the
choice of subword model types. One of the objectives of this study is to
identify which type of subword model performs best in this low-resource
scenario.

###  2.4 Background

Native speakers of low-resource languages are often excluded from useful
content since, more often than not, online content is not available to them in
their language of choice. Such a digital divide and the resulting social
exclusion experienced by second-language speakers, such as refugees living in
developed countries, has been well documented in the research literature [78,
4].

Research on machine translation (MT) in low-resource scenarios directly
addresses this challenge of exclusion via pivot languages [72], and
indirectly, via domain adaptation of models [42]. Breakthrough performance
improvements in the area of MT have been achieved through research efforts
focusing on NMT [10, 28]. Consequently, state-of-the-art (SOTA) performance
has been attained on multiple language pairs [20, 21].

####  2.4.1 Irish Language

The Irish language is a primary example of such a low-resource language that
will benefit from this research. NMT involving Transformer model development
will improve the performance in specific domains of low-resource languages.
Such research will address the end of the Irish language derogation in the
European Commission in 2021111http://amtaweb.org/wp-
content/uploads/2020/11/MT-in-EU-Overview-with-Voiceover-Andy-Way-
KEYNOTE-K1.pdf helping to deliver parity in support for Irish in online
digital engagement.

####  2.4.2 Hyperparameter Optimisation

Hyperparameters are employed to customise machine learning models such as
translation models. It has been shown that machine learning performance may be
improved through HPO rather than just using default settings [98].

The principle methods of HPO are grid search [81] and random search [16]].
Grid search is an exhaustive technique which evaluates all hyperparameter
permutations. However, as the number of features grows, the amount of data
permutations grows exponentially making optimisation expensive in the context
of developing long-running translation models.

An effective, and less computationally intensive, alternative is to use random
search which samples random configurations.

##### Recurrent Neural Networks

Recurrent neural networks (RNNs) are often used for the tasks of natural
language processing, speech recognition and MT. RNN models enable previous
outputs to be used as inputs while having hidden states. In the context of MT,
such neural networks were ideal due to their ability to process inputs of any
length. Furthermore, the model sizes do not necessarily increase with the size
of its input. Commonly used variants of RNN include bidirectional (BRNN) and
deep (DRNN) architectures. However, the problem of vanishing gradients coupled
with the development of attention-based algorithms often leads to Transformer
models performing better than RNNs.

##### Transformer

The greatest improvements have been demonstrated when either the RNN or the
CNN architecture is abandoned completely and replaced with an attention
mechanism creating a much simpler and faster architecture known as Transformer
[114]. Transformer models use attention to focus on previously generated
tokens. The approach allows models to develop a long memory which is
particularly useful in the domain of language translation. Performance
improvements to both RNN and CNN approaches may be achieved through the
introduction of such attention layers in the translation architecture.

Experiments in MT tasks show such models are better in quality due to greater
parallelisation while requiring significantly less time to train.

####  2.4.3 Subword Models

Translation, by its nature, requires an open vocabulary and the use of subword
models aims to address the fixed vocabulary problem associated with NMT. Rare
and unknown words are encoded as sequences of subword units. By adapting the
original Byte Pair Encoding (BPE) algorithm [41], the use of BPE submodels can
improve translation performance [103, 60].

Designed for NMT, SentencePiece, is a language-independent subword tokenizer
that provides an open-source C++ and a Python implementation for subword
units. An attractive feature of the tokenizer is that SentencePiece trains
subword models directly from raw sentences [61].

##### Byte Pair Encoding compared with Unigram

BPE and unigram language models are similar in that both encode text using
fewer bits but each uses a different data compression principle (dictionary
vs. entropy). In principle, we would expect the same benefits with the unigram
language model as with BPE. However, unigram models are often more flexible
since they are probabilistic models that output multiple segmentations with
their probabilities.

Figure 2.1: Proposed approach of Transformers for low-resource languages Hyperparameter | Values
---|---
Learning rate | 0.1, 0.01, 0.001, 2
Batch size | 1024, 2048, 4096, 8192
Attention heads |  2, 4, 8
Number of layers | 5, 6
Feed-forward dimension | 2048
Embedding dimension | 128, 256, 512
Label smoothing |  0.1, 0.3
Dropout | 0.1, 0.3
Attention dropout | 0.1
Average Decay | 0, 0.0001
Table 2.1: Hyperparameter optimisation for Transformer models. Optimal
hyperparameters are highlighted in bold. The highest performing model trained
on the 55k DGT corpus uses 2 attention heads whereas the best model trained
with the larger 88k PA dataset uses 8 attention heads.

###  2.5 Proposed Approach

HPO of RNN models in low-resource settings has previously demonstrated
considerable performance improvements. The extent to which such optimisation
techniques may be applied to Transformer models in similar low-resource
scenarios is evaluated as part of this study. Evaluations included modifying
the number of attention heads, the number of layers and experimenting with
regularisation techniques such as dropout and label smoothing. Most
importantly, the choice of subword model type and the vocabulary size are
evaluated.

To test the effectiveness of our approaches, optimisation was carried out on
two EN$\leftrightarrow$GA parallel datasets: a general corpus of 52k lines
from the Directorate General for Translation (DGT) and an in-domain corpus of
88k lines of Public Administration (PA) data. With DGT, the test set used 1.3k
lines and the development set comprised 2.6k lines. In the case of the PA
dataset, there were 1.5k lines of test data and 3k lines of validation. All
experiments involved concatenating source and target corpora to create a
shared vocabulary and a shared SentencePiece subword model. The impact of
using separate source and target subword models was not explored.

The approach adopted is illustrated in Figure 2.1. Two baseline architectures,
RNN and Transformer, are evaluated. On evaluating the hyperparameter choices
for Transformer models, the values outlined in Table 2.1 were tested using a
random search approach. A range of values for each hyperparameter was tested
using short cycles of 5k training steps. Once an optimal value, within the
sampled range was identified, it was locked in for tests on subsequent
hyperparameters.

####  2.5.1 Architecture Tuning

Given the long training times associated with NMT, it is difficult and costly
to tune systems using a conventional grid search approach. Therefore a random
search approach was adopted in the HPO of our transformer models.

With low-resource datasets, the use of smaller and fewer layers has previously
been shown to improve performance [5]. Performance of low-resource NMT has
also been demonstrated to improve in cases where shallow Transformer models
are adopted [17]. Guided by these findings, configurations were tested which
varied the number of neurons in each layer and modified the number of layers
used in the Transformer architecture.

The impact of regularisation, by applying varying degrees of dropout to
Transformer models, was evaluated. Configurations using smaller (0.1) and
larger values (0.3) were applied to the output of each feed-forward layer.

####  2.5.2 Subword Models

It has become standard practice to incorporate word segmentation approaches,
such as Byte-Pair-Encoding (BPE) when developing NMT models. Previous work
shows that subword models may be particularly beneficial for low-resource
languages since rare words are often a problem. Reducing the number of BPE
merge operations resulted in substantial improvements of 5 BLEU points
(Sennrich and Zhang 2019) when tested on RNN models.

In the context of EN$\leftrightarrow$GA translation, there is no clear
agreement as to what constituted the best approach. Consequently, as part of
this study, subword regularisation techniques, involving BPE and unigram
models were evaluated to determine the optimal hyperparameters for maximising
translation performance. BPE models with varying vocabulary sizes of 4k, 8k,
16k and 32k were tested.

###  2.6 Empirical Evaluation

####  2.6.1 Experimental Setup

##### Datasets

The performance of the Transformer and RNN approaches is evaluated on
EN$\leftrightarrow$GA parallel datasets. Two datasets were used in the
evaluation of our models namely the publicly available DGT dataset which may
be broadly categorised as generic and an in-domain dataset which focuses on
public administration data.

The DGT, and its Joint Research Centre, have made available all Translation
Memory (TM; i.e. sentences and their professionally produced translations)
which cover all official European Union languages [107].

Data provided by the Department of Tourism, Culture, Arts, Gaeltacht, Sport
and Media in Ireland formed the majority of the data in the public
administration dataset. This includes staff notices, annual reports, website
content, press releases and official correspondence.

Parallel texts from the Digital Corpus of the European Parliament (DCEP) and
the DGT are included in the training data. Crawled data, from sites of a
similar domain are included. Furthermore, a parallel corpus collected from
Conradh na Gaeilge (CnaG), an Irish language organisation that promotes the
Irish language, was included. The dataset was compiled as part of a previous
study which carried out a preliminary comparison of SMT and NMT models for the
Irish language [37].

##### Infrastructure

Models were developed using a lab of machines each of which has an AMD Ryzen 7
2700X processor, 16GB memory, a 256GB SSD and an NVIDIA GeForce GTX 1080 Ti.
Rapid prototype development was enabled through a Google Colab Pro
subscription using NVIDIA Tesla P100 PCIe 16GB graphic cards and up to 27GB of
memory when available [19].

Our MT models were trained using the Pytorch implementation of OpenNMT 2.0, an
open-source toolkit for NMT [56].

##### Metrics

As part of this study, several automated metrics were used to determine the
translation quality. All models were trained and evaluated on both the DGT and
PA datasets using the BLEU [88], TER [105] and ChrF [90] evaluation metrics.
Case-insensitive BLEU scores, at the corpus level, are reported. Model
training was stopped once an early stopping criteria of no improvement in
validation accuracy for 4 consecutive iterations was recorded.

####  2.6.2 Results

##### Performance of subword models

The impact on translation accuracy when choosing a subword model is
highlighted in Tables 2.2 \- 2.5. In training both RNN and Transformer
architectures, incorporating any submodel type led to improvements in model
accuracy. This finding is evident when training either the smaller generic DGT
dataset or the larger in-domain PA dataset.

Using an RNN architecture on DGT, as illustrated in Table 2.2, the best-
performing model with a 32k unigram submodel, achieved a BLEU score 7.4%
higher than the baseline. With the PA dataset using an RNN, as shown in Table
2.3, the model with the best BLEU, TER and ChrF3 scores again used a unigram
submodel.

Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
dgt-rnn-base | 52.7 | 0.42 | 0.71 | 75k | 4.47 | 0
dgt-rnn-bpe8k | 54.6 | 0.40 | 0.73 | 85k | 5.07 | 0
dgt-rnn-bpe16k | 55.6 | 0.39 | 0.74 | 100k | 5.58 | 0
dgt-rnn-bpe32k | 55.3 | 0.39 | 0.74 | 95k | 4.67 | 0
dgt-rnn-unigram | 55.6 | 0.39 | 0.74 | 105k | 5.07 | 0
Table 2.2: RNN performance on DGT dataset of 52k lines Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
pa-rnn-base | 40.4 | 0.47 | 0.63 | 60k | 2.13 | 0
pa-rnn-bpe8k | 41.5 | 0.46 | 0.64 | 110k | 4.16 | 0
pa-rnn-bpe16k | 41.5 | 0.46 | 0.64 | 105k | 3.78 | 0
pa-rnn-bpe32k | 41.9 | 0.47 | 0.64 | 100k | 2.88 | 0
pa-rnn-unigram | 41.9 | 0.46 | 0.64 | 95k | 2.75 | 0
Table 2.3: RNN performance on PA dataset of 88k lines

There are small improvements in BLEU scores when the RNN baseline is compared
with models using a BPE submodel of either 8k, 16k or 32k words, as
illustrated in Tables 2.2 and 2.3. The maximum BLEU score improvement of 1.5
points (2.5%) is quite modest in the case of the public admin corpus. However,
there are larger gains with the DGT corpus. A baseline RNN model, trained on
DGT, achieved a BLEU score of 52.7 whereas the highest-performing BPE variant,
using a 16k vocab, recorded an improvement of nearly 3 points with a score of
55.6.

Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
dgt-trans-base | 53.4 | 0.41 | 0.72 | 55k | 14.43 | 0.81
dgt-trans-bpe8k | 59.5 | 0.34 | 0.77 | 200k | 24.48 | 1.38
dgt-trans-bpe16k | 60.5 | 0.33 | 0.78 | 180k | 26.90 | 1.52
dgt-trans-bpe32k | 59.3 | 0.35 | 0.77 | 100k | 18.03 | 1.02
dgt-trans-unigram | 59.3 | 0.35 | 0.77 | 125k | 21.95 | 1.24
Table 2.4: Transformer performance on 52k DGT dataset. The highest-performing model uses 2 attention heads. All other models use 8 attention heads. Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
pa-trans-base | 44.1 | 0.44 | 0.66 | 20k | 5.97 | 0.34
pa-trans-bpe8k | 46.6 | 0.40 | 0.68 | 160k | 20.1 | 1.13
pa-trans-bpe16k | 47.1 | 0.41 | 0.68 | 100k | 14.22 | 0.80
pa-trans-bpe32k | 46.8 | 0.41 | 0.68 | 70k | 12.7 | 0.72
pa-trans-unigram | 46.6 | 0.42 | 0.68 | 75k | 13.34 | 0.75
Table 2.5: Transformer performance on 88k PA dataset. All models use 8
attention heads.

In the context of Transformer architectures, highlighted in Table 2.4 and
Table 2.5, the use of subword models delivers significant performance
improvements for both the DGT and public admin corpora. The performance gains
for Transformer models are far greater than RNN models. Baseline DGT
Transformer models achieve a BLEU score of 53.4 while a Transformer model,
with a 16k BPE submodel, has a score of 60.5 representing a BLEU score
improvement of 13% at 7.1 BLEU points.

For translating into a morphologically rich language, such as Irish, the ChrF
metric has proven successful in showing a strong correlation with human
translation [106]. In the context of our experiments, it worked well in
highlighting the performance differences between RNN and Transformer
architectures.

Figure 2.2: BLEU performance for all model architectures

##### Transformer performance compared with RNN

The performance of RNN models is contrasted with the Transformer approach in
Figure 2.2 and Figure 2.3. Transformer models, as anticipated, outperform all
their RNN counterparts. It is interesting to note the impact of choosing the
optimal vocabulary size for BPE submodels. Both datasets demonstrate that
choosing a BPE vocabulary of 16k words yields the highest performance.

Furthermore, the TER scores highlighted in Figure 2.3 reinforce the findings
that using 16k BPE submodels on Transformer architectures leads to better
translation performance. The TER score for the DGT Transformer 16k BPE model
is significantly better (0.33) when compared with the baseline performance
(0.41).

Figure 2.3: TER performance for all model architectures Figure 2.4: Training
DGT Transformer baseline Figure 2.5: Training DGT Transformer 16k BPE

###  2.7 Environmental Impact

Motivated by the findings of Stochastic Parrots [15], energy consumption
during model development was tracked. Prototype model development used Colab
Pro, which as part of Google Cloud is carbon neutral [62]. However, longer
running Transformer experiments were conducted on local servers using 324 gCO2
per kWh222https://www.seai.ie/publications/Energy-in-Ireland-2020.pdf [100].
The net result was just under 10 kgCO2 created for a full run of model
development. Models developed during this study will be reused for ensemble
experiments in future work.

###  2.8 Discussion

Validation accuracy, and model perplexity, in developing the baseline and
optimal models for the DGT corpus are illustrated in Figure 2.4 and Figure
2.5. Rapid convergence was observed while training the baseline model such
that little accuracy improvement occurred after 20k steps. Including a subword
model led to much slower convergence and there were only marginal gains after
60k steps. Furthermore, it is observed that training the DGT model, with a 16k
BPE submodel, boosted validation accuracy by over 8% compared with its
baseline.

With regard to the key metric of perplexity, it is shown to rise after
training for 15k steps in the baseline models. PPL was observed to rise at
later stages, typically after 40k steps in models developed using subword
models. Perplexity (PPL), shows how many different, equally probable words can
be produced during translation. As a metric for translation performance, it is
important to keep low scores so the number of alternative translations is
reduced. Therefore, for future model development, it may be worthwhile to set
PPL as an early stopping hyperparameter.

On examining the PPL graphs of Figure 2.4 and Figure 2.5, it is clear that a
lower global minimum is achieved when the Transformer approach is used with a
16k BPE submodel. The PPL global minimum (2.7) is over 50% lower than the
corresponding PPL for the Transformer base model (5.5). Such a finding
illustrates that choosing an optimal submodel delivers significant performance
gains.

Translation engine performance was benchmarked against Google Translate’s
333https://translate.google.com/ EN$\leftrightarrow$GA translation service
which is freely available on the internet. Four random samples were selected
from the English source test file and are presented in Table 2.6. Translation
of these samples was carried out on the optimal DGT Transformer model and
using Google Translate. Case insensitive, sentence level BLEU scores were
recorded and are presented in Table 2.7. The results are encouraging and
indicate well-performing translation models on the DGT dataset.

The optimal hyperparameters selected in this discovery process are identified
in bold in Table 2.1. A higher initial learning rate of 2 coupled with an
average decay of 0.0001 led to longer training times but more accurate models.
Despite setting an early stopping hyperparameter, many of the Transformer
builds continued for the full cycle of 200k steps over periods of 20+ hours.

Training transformer models with a reduced number of attention heads led to a
marginal improvement in translation accuracy with a smaller corpus. Our best-
performing model on a 55k DGT corpus, with 2 heads and a 16k BPE submodel,
achieved a BLEU score of 60.5 and a TER score of 0.33. By comparison, using 8
heads with the same architecture and dataset yielded 60.3 for the BLEU and
0.34 for the TER. In the case of a larger 88k PA corpus, all transformer
models using 8 heads performed better than equivalent models using just 2
heads.

Source Language (English) |  Reference Human Translation (Irish)
---|---
A clear harmonised procedure, including the necessary criteria for disease–free status, should be established for that purpose. |  Ba cheart nós imeachta comhchuibhithe soiléir, lena n-áirítear na critéir is gá do stádas saor ó ghalar, a bhunú chun na críche sin.
the mark is applied anew, as appropriate. |  déanfar an mharcáil arís, mar is iomchuí.
If the court decides that a review is justified on any of the grounds set out in paragraph 1, the judgment given in the European Small Claims Procedure shall be null and void. |  Má chinneann an chúirt go bhfuil bonn cirt le hathbhreithniú de bharr aon cheann de na forais a leagtar amach i mír 1, beidh an breithiúnas a tugadh sa Nós Imeachta Eorpach um Éilimh Bheaga ar neamhní go hiomlán.
households where pet animals are kept; |  teaghlaigh ina gcoimeádtar peataí;
Table 2.6: Samples of human reference translations Transformer (16 kBPE) |  BLEU $\uparrow$ |  Google Translate |  BLEU $\uparrow$
---|---|---|---
Ba cheart nós imeachta soiléir comhchuibhithe, lena n-áirítear na critéir is gá maidir le stádas saor ó ghalair, a bhunú chun na críche sin. |  61.6 |  Ba cheart nós imeachta comhchuibhithe soiléir, lena n-áirítear na critéir riachtanacha maidir le stádas saor ó ghalair, a bhunú chun na críche sin. |  70.2
go gcuirtear an marc i bhfeidhme, de réir mar is iomchuí. |  21.4 |  cuirtear an marc i bhfeidhm as an nua, de réir mar is cuí. |  6.6
Má chinneann an chúirt go bhfuil bonn cirt le hathbhreithniú ar aon cheann de na forais a leagtar amach i mír 1, beidh an breithiúnas a thugtar sa Nós Imeachta Eorpach um Éilimh Bheaga ar neamhní. |  77.3 |  Má chinneann an chúirt go bhfuil údar le hathbhreithniú ar aon cheann de na forais atá leagtha amach i mír 1, beidh an breithiúnas a thugtar sa Nós Imeachta Eorpach um Éilimh Bheaga ar neamhní |  59.1
teaghlaigh ina gcoimeádtar peataí; |  100 |  teaghlaigh ina gcoinnítear peataí; |  30.2
Table 2.7: Transformer model compared with Google Translate using random
samples from the DGT corpus. Full evaluation of Google Translate on the DGT
test set, with 1.3k lines, generated a BLEU score of 46.3 and a TER score of
0.44. Comparative scores on the test set using our Transformer model, with 2
attention heads and 16k BPE submodel realised 60.5 for BLEU and 0.33 for TER.

Standard Transformer hyperparameters for batch size (2048) and the number of
encoder and decoder layers (6) were both observed to perform well on the DGT
and PA corpora. Reducing hidden neurons to 256 and increasing regularisation
dropout to 0.3 improved translation performance and were chosen when building
all Transformer models.

###  2.9 Conclusion

In our paper, we demonstrated that a random search approach to hyperparameter
optimisation leads to the development of high-performing translation models.

We have shown that choosing subword models, in our low-resource scenarios, is
an important driver for the performance of MT engines. Moreover, the choice of
vocabulary size leads to varying degrees of performance. Within the context of
low-resource EN$\rightarrow$GA translation, we achieved optimal performance,
on a 55k generic corpus and an 88k in-domain corpus, when a Transformer
architecture with a 16k BPE submodel was used. The importance of selecting
hyperparameters in training low-resource Transformer models was also
demonstrated. By reducing the number of hidden layer neurons and increasing
dropout, our models performed significantly better than baseline models and
Google Translate.

Performance improvement of our optimised Transformer models, with subword
segmentation, was observed across all key indicators namely a higher
validation accuracy, a PPL achieved at a lower global minimum, a lower post-
editing effort and a higher translation accuracy.

##  Chapter 3 MT in the Covid domain: Shared Task for LoResMT2021

###  3.1 Context

Figure 3.1: Proposed approach for LoResMT2021 Shared Task

In refining the Transformer model with a custom dataset, we investigated the
research objectives outlined in RQ1 and RQ2. A significant contribution of
this study was showcasing that hyperparameters specified in “Transformers for
Low-Resource Languages - is féidir linn!” could be employed to excel in a
shared task targeting low-resource languages. Therefore, the hyperparameters
identified in the HPO experiments of Table 2.1 were employed. Another key
contribution was to demonstrate the notable improvement in translation quality
when the shared task dataset was augmented by a compact bespoke in-domain
dataset.

The approach I adopted in developing the model for the LoResMT2021 Shared Task
is illustrated in Figure 3.1. As the winning entry in the EN$\rightarrow$GA
direction, the system achieved 36.0 BLEU, 0.531 TER and 0.6 ChrF. By combining
the baseline LoResMT2021 5k line dataset with a bespoke 8k line Covid dataset,
an amalgamated Covid-specific corpus of 13k lines was created. Using this
dataset, a Transformer model was trained from scratch. Our system won by a
wide margin with the second-placed team achieving a 25.8 BLEU.

Machine Translation in the Covid domain: an English-Irish case study -
LoResMT2021

Séamus Lankford
Haithem Afli
Andy Way

Proceedings of the 4th Workshop on Technologies for

MT of Low-Resource Languages (LoResMT2021)

Virtual: Association for Machine Translation in the Americas

August 16 - 20, 2021

Florida, USA

1st place in EN$\rightarrow$GA Shared Task

ADAPT Centre

Dublin City University

Ireland

https://aclanthology.org/2021.mtsummit-loresmt.15.pdf

###  3.2 Abstract

Translation models for the specific domain of translating Covid data from
English to Irish were developed. Domain adaptation techniques, using a Covid-
adapted generic 55k corpus from the Directorate General of Translation, were
applied. Fine-tuning, mixed fine-tuning and combined dataset approaches were
compared with models trained on an extended in-domain dataset. As part of this
study, an English-Irish dataset of Covid related data, from the Health and
Education domains, was developed. The highest-performing model used a
Transformer architecture trained with an extended in-domain Covid dataset. In
the context of this study, we have demonstrated that extending an 8k in-domain
baseline dataset by just 5k lines improved the BLEU score by 27 points.

###  3.3 Introduction

Neural machine translation (NMT) has routinely outperformed statistical
machine translation (SMT) when large parallel datasets are available [33,
119]. Furthermore, Transformer-based approaches have demonstrated impressive
results in moderate low-resource scenarios [65]. NMT involving Transformer
model development will improve the performance in specific domains of low-
resource languages [5]. However, the benefits of NMT are less clear when using
very low-resource machine translation (MT) on in-domain datasets of less than
10k lines.

The Irish language is a primary example of a low-resource language that will
benefit from such research. This paper reports the results of the MT system
developed for the English–to-Irish (EN$\rightarrow$GA) Shared Task at
LoResMT2021 [85]. Relevant work is presented in the background section
followed by an overview of the proposed approach. The empirical findings are
outlined in the results section. Finally, the key findings are presented and
discussed.

###  3.4 Background

####  3.4.1 Transformer

A novel architecture called Transformer was introduced in the paper ‘Attention
Is All You Need’ [114]. Transformer is an architecture for transforming one
sequence into another with the help of an encoder and decoder without relying
on recurrent neural networks (RNNs).

Transformer models use attention to focus on previously generated tokens. This
approach allows models to develop a long memory which is particularly useful
in the domain of language translation.

####  3.4.2 Domain Adaptation

Domain adaptation is a proven approach to addressing the paucity of data in
low-resource settings. Fine-tuning an out-of-domain model by further training
with in-domain data is effective in improving the performance of translation
models (Freitag and Al-Onaizan, 2016; Sennrich et al., 2016). With this
approach, an NMT model is initially trained using a large out-of-domain
corpus. Once fully converged, the out-of-domain model is further trained by
fine-tuning its parameters with a low-resource in-domain corpus.

A modification to this approach is known as mixed fine-tuning [29]. With this
technique, an NMT model is trained on out-of-domain data until fully
converged. This serves as a base model which is further trained using the
combined in-domain and out-of-domain datasets.

###  3.5 Proposed Approach

Figure 3.2: Proposed approach of MT in the Covid domain. Optimal
hyperparameters are applied to Transformer models which are trained using one
of several possible approaches. The training dataset composition is determined
by the chosen approach. Models are subsequently evaluated using a suite of
metrics.

Hyperparameter optimisation of RNN models in low-resource settings has
previously demonstrated considerable performance improvements [104]. The
extent to which such optimisation techniques may be applied to Transformer
models in similar low-resource scenarios was evaluated in a previous study
[65]. Evaluations included modifying the number of attention heads, the number
of layers and experimenting with regularisation techniques such as dropout and
label smoothing. Most importantly, the choice of subword model type and the
vocabulary size were evaluated.

To test the effectiveness of our approaches, models were trained using three
EN$\rightarrow$GA parallel datasets: a general corpus of 52k lines from the
Directorate General for Translation (DGT) and two in-domain corpora of Covid
data (8k and 5k lines). All experiments involved concatenating source and
target corpora to create a shared vocabulary and a shared SentencePiece [61]
subword model. The impact of using separate source and target subword models
was not explored.

The approach adopted is illustrated in Figure 3.2 and the datasets used in
evaluating this approach are outlined in Table 3.1. All models were developed
using a Transformer architecture.

Approach | Source | Lines |
---|---|---|---
Covid baseline | Baseline | 8k |
Covid extended | Baseline + Covid_DCU | 13k |
Out-of-domain | DGT | 52k |
Fine-tuned | Baseline + Covid_DCU + DGT | 65k |
Mixed fine-tuned | Baseline + Covid_DCU + DGT | 65k |
Combined domains | Baseline + Covid_DCU + DGT | 65k |
Table 3.1: Datasets used in proposed approach for MT in Covid domain

####  3.5.1 Architecture Tuning

Long training times associated with NMT make it costly to tune systems using
conventional grid search approaches. A previous study identified the
hyperparameters required for optimal performance [65]. Reducing the number of
hidden layer neurons and increasing dropout led to significantly better
performance. Furthermore, within the context of low-resource EN$\rightarrow$GA
translation, using a 16k BPE submodel resulted in the highest-performing
models. The Transformer hyperparameters, chosen in line with these findings,
are outlined in Table 3.2.

Hyperparameter | Values
---|---
Learning rate | 0.1, 0.01, 0.001, 2
Batch size | 1024, 2048, 4096, 8192
Attention heads |  2, 4, 8
Number of layers | 5, 6
Feed-forward dimension | 2048
Embedding dimension | 128, 256, 512
Label smoothing |  0.1, 0.3
Dropout | 0.1, 0.3
Attention dropout | 0.1
Average Decay | 0, 0.0001
Table 3.2: Hyperparameter optimisation for Transformer models. Optimal
parameters are highlighted in bold. The highest-performing model trained on
the 55k DGT corpus uses 2 attention heads [65].

###  3.6 Empirical Evaluation

####  3.6.1 Experimental Setup

##### Datasets

The performance of the Transformer approach is evaluated on EN$\rightarrow$GA
parallel datasets in the Covid domain. Three datasets were used in the
evaluation of our models. These consisted of a baseline Covid dataset (8k)
provided by MT Summit 2021 [85], an in-domain Covid dataset (5k) developed at
DCU and a publicly available out-of-domain dataset (52k) provided by DGT
[107].

##### Infrastructure

Models were developed using a lab of machines each of which has an AMD Ryzen 7
2700X processor, 16GB memory, a 256GB SSD and an NVIDIA GeForce GTX 1080 Ti.
Rapid prototype development was enabled through a Google Colab Pro
subscription using NVIDIA Tesla P100 PCIe 16GB graphic cards and up to 27GB of
memory when available [19].

Our MT models were trained using the Pytorch implementation of OpenNMT 2.0, an
open-source toolkit for NMT [56].

##### Metrics

Automated metrics were used to determine the translation quality. All models
were trained and evaluated using the BLEU [88], TER [105] and ChrF [90]
evaluation metrics. Case-insensitive BLEU scores, at the corpus level, are
reported. Model training was stopped once an early stopping criteria of no
improvement in validation accuracy for 4 consecutive iterations was recorded.

####  3.6.2 Results

Experimental results achieved using a Transformer architecture, with either 2
or 8 attention heads, are summarised in Table 3.3 and in Table 3.4. Clearly,
in the context of our low-resource experiments, it can be seen there is little
performance difference using Transformer architectures with a differing number
of attention heads. The largest difference occurs when using a fine-tuned
approach (2.1 BLEU points). However, the difference between a 2-head and an
8-head approach is less than 1 BLEU point for all other models. The highest-
performing approach uses the extended Covid dataset (13k) which is a
combination of the MT summit Covid baseline and a custom DCU Covid dataset.
This Transformer model, with 2 heads, performs well across all key translation
metrics (BLEU: 36.0, TER: 0.63 and ChrF3: 0.32).

(a) BLEU

(b) TER

Figure 3.3: Translation performance using Transformers with 2 heads System | Heads | Lines | Steps |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---|---|---|---
Covid baseline | 2 | 8k | 35k | 9.0 | 0.89 | 0.32
Covid extended | 2 | 13k | 35k | 36.0 | 0.63 | 0.54
Out-of-domain | 2 | 52k | 200k | 13.9 | 0.80 | 0.41
Fine-tuned | 2 | 65k | 35k | 22.9 | 0.64 | 0.42
Mixed fine-tuned | 2 | 65k | 35k | 18.2 | 0.71 | 0.42
Combined domains | 2 | 65k | 35k | 32.2 | 0.59 | 0.55
Table 3.3: Comparison of Transformer performance with 2 attention heads System | Heads | Lines | Steps |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---|---|---|---
Covid baseline | 8 | 8k | 35k | 9.6 | 0.91 | 0.33
Covid extended | 8 | 13k | 35k | 35.7 | 0.61 | 0.55
Out-of-domain | 8 | 52k | 200k | 13.0 | 0.80 | 0.40
Fine-tuned | 8 | 65k | 35k | 25.0 | 0.63 | 0.43
Mixed fine-tuned | 8 | 65k | 35k | 18.0 | 0.71 | 0.42
Combined domains | 8 | 65k | 35k | 32.8 | 0.59 | 0.57
Table 3.4: Comparison of Transformer performance with 8 attention heads

The worst-performing model uses the Covid baseline which is not surprising
given that only 8k lines are available. The performance of the higher
resourced models (out-of-domain, fine-tuned, mixed fine-tuned and combined
domains) all lag that of the Covid extended model. In particular, the out-of-
domain model, using the DGT dataset, performs very poorly with a BLEU score of
just 13.9 on a Transformer model with 2 heads.

The BLEU and TER scores for all approaches are illustrated in Figure 3.3(a)
and Figure 3.3(b). As expected, there is a high level of inverse correlation
between BLEU and TER. Well-performing models, with high BLEU scores, also
required little post-editing effort as indicated by their lower TER scores.

###  3.7 Discussion

Standard Transformer parameters identified in a previous study were observed
to perform well [65]. Reducing hidden neurons to 256 and increasing
regularisation dropout to 0.3 improved translation performance and these
hyperparameters were chosen when building all Transformer models. Furthermore,
a batch size of 2048 and using 6 layers for the encoder and decoder were
chosen throughout.

The results demonstrate that translation performance for specific domains is
driven by the amount of data which is available for that specific domain. It
is noteworthy that an in-domain dataset of 13k lines (Covid extended), trained
for just 35k steps outperformed by 22.1 BLEU points the corresponding out-of-
domain 52k dataset (DGT) which was trained for 200k steps.

###  3.8 Conclusion and Future Work

In our paper, we demonstrated that a high-performing in-domain translation
model can be built with a dataset of 13k lines. Developing a small in-domain
dataset, of just 5k lines, improved the BLEU score by 27 points when models
were trained with the combined Covid baseline and custom Covid dataset.

Following on from our previous work, careful selection of Transformer
hyperparameters, and using a 16k BPE SentencePiece submodel, enabled rapid
development of high-performing translation models in a low-resource setting.

Within the context of our research in low-resource EN$\rightarrow$GA
translation, we have shown that augmenting in-domain data, by a small amount,
performed better than approaches which incorporate fine-tuning, mixed fine-
tuning or the combination of domains.

As part of our future work, we plan to develop EN$\leftrightarrow$GA MT models
trained on a dataset derived from the health domain (cf. Chapter 4). Domain
adaptation, through fine-tuning such models with the Covid extended dataset
may further improve Covid MT performance.

##  Chapter 4 gaHealth: EN$\leftrightarrow$GA Bilingual Corpus of Health Data

###  4.1 Context

While MT has made significant strides in high-resource language pairs, there
is a noticeable scarcity of parallel data for low-resource languages,
particularly in specialised domains like health. An example of a high-resource
language pair that has near-perfect MT in a specific domain can be seen in the
English$\leftrightarrow$French pair. Weather forecasting in Canada, which is
provided both in English and French, has led to one of the best MT systems
ever produced, namely the METEO system [111]. Having been perfected over
decades, the only errors made by the system are due to erroneous human input.

With regard to low-resource languages, the existing efforts often prioritise
quantity over domain specificity. The focus of this research is to address, in
part, this imbalance and the key motivations for this research are driven by
RQ2. A separate contribution from the research was the development of a set of
guidelines for the pre-processing, alignment and validation of a corpus for a
low-resource language pair. An important motivating factor for answering this
research question is that key information first comes out in “major” languages
before it does in less well-resourced languages. Consequently, people who
would prefer to access information in their preferred language are forced to
operate in a language they otherwise would not choose. In particular, this
paper deals with several aspects of the research question which are
highlighted below:

  * •

Addressing Low-Resource Language Challenges: The scarcity of parallel data for
low-resource languages hinders the development of effective MT models. This
scarcity is particularly acute in specialised domains like health. The study
aims to alleviate this issue by creating a focused, in-domain dataset.

  * •

Importance of In-Domain Data: The benefits of using in-domain data for
training MT models are highlighted. In contrast to generic datasets, in-domain
datasets are tailored to a specific domain (in this case, health), which can
lead to more accurate and contextually relevant translations for that domain.

  * •

Development of gaHealth Corpus: The research introduces the gaHealth corpus,
which is the first bilingual corpus of health data for the Irish language.
This corpus is designed to be a valuable resource for the NLP community,
especially those working within the Irish language domain. The importance of
open access is recognised and the gaHealth corpus has been shared online,
inviting further research and contributions from the wider community.

  * •

Empirical Demonstration of Benefits: The study provides empirical evidence to
support the use of in-domain data, showing significant improvements in
translation performance. The comparison with top-performing models from a
shared task (LoResMT2021) highlights the substantial gains achieved through
the use of the gaHealth corpus.

  * •

Guidelines for Corpus Development: Recognising the challenges encountered
during the conversion of PDF documents, guidelines were established to
facilitate the conversion process. This contribution not only aids in the
development of the gaHealth corpus but also provides valuable insights for
similar projects.

  * •

Future Directions: The research outlines a clear roadmap for future work. This
includes expanding the gaHealth corpus as more Irish language documents become
available, refining the models, and conducting a deep linguistic investigation
to understand the nuances of model performance. Additionally, it is planned to
extend the effort to other key domains such as Education and Finance.

In summary, the research addresses a critical gap in the availability of
specialised, in-domain data for low-resource languages, particularly in the
context of health-related content for the Irish language. By creating the
gaHealth corpus and demonstrating its effectiveness in improving translation
models, the study provides a valuable resource for the NLP community and lays
the foundation for future advancements in this domain. More importantly, it
has contributed to high-quality MT systems which can assist Irish-language
speakers in accessing health information in their preferred language.

gaHealth: An EN$\leftrightarrow$GA Bilingual Corpus of Health Data

Séamus Lankford
Haithem Afli
Órla Ní Loinsigh
Andy Way

Proceedings of the Thirteenth Language Resources and Evaluation Conference

European Language Resources Association

20-25 June 2022

Marseille, France

ADAPT Centre

Dublin City University

Ireland

https://aclanthology.org/2022.lrec-1.727.pdf

###  4.2 Abstract

Machine translation is a mature technology for many high-resource language
pairs. However, in the context of low-resource languages, there is a paucity
of parallel data datasets available for developing translation models.
Furthermore, the development of datasets for low-resource languages often
focuses on simply creating the largest possible dataset for generic
translation. The benefits and development of smaller in-domain datasets can
easily be overlooked. To assess the merits of using in-domain data, a dataset
for the specific domain of health was developed for the low-resource English-
to-Irish language pair. Our study outlines the process used in developing the
corpus and empirically demonstrates the benefits of using an in-domain dataset
for the health domain. In the context of translating health-related data,
models developed using the gaHealth corpus demonstrated a maximum BLEU score
improvement of 22.2 points (40%) when compared with top-performing models from
the LoResMT2021 Shared Task. Furthermore, we define linguistic guidelines for
developing gaHealth, the first bilingual corpus of health data for the Irish
language, which we hope will be of use to other creators of low-resource data
sets. gaHealth is now freely available online and is ready to be explored for
further research.

###  4.3 Introduction

Improvements in performance in natural language processing (NLP) tasks are
typically to be seen when deep learning models are used. However, deep
learning requires large amounts of data for model training. Consequently, the
availability of large amounts of textual data has become fundamental to the
success of NLP applications, such as language modelling [24] and neural
machine translation (NMT) [10, 101].

A popular method of developing such corpora for machine translation (MT) tasks
is to crawl and parse bilingual web pages to general parallel corpora.
However, given the nature of low-resource languages, there are often
insufficient websites available in both the languages of study. Accordingly, a
lack of web content typically hinders the development of NLP applications for
low-resource languages.

The motivation underpinning our present work comes from the challenges we
faced in developing high-performing MT models in low-resource settings [2]. In
this work, we developed the first bilingual corpus of health data for the
English$\leftrightarrow$Irish (EN$\leftrightarrow$GA) language pair. A
procedure was created to extract, clean and select appropriate sentences to
build a bilingual corpus. In addition, we built a high-performing MT model for
translating in-domain health data.

###  4.4 Related work

####  4.4.1 Transformer

Transformer [114] is an architecture for transforming an input sequence into
an output sequence via an encoder and decoder without relying on recurrent
neural networks. Transformer models use attention to focus on previously
generated tokens. This approach allows models to develop a long memory which
is particularly useful in the domain of language translation.

####  4.4.2 Transformer Hyperparameter Optimisation

Hyperparameter optimisation of Transformer models in translating the low-
resource English-Irish (EN$\rightarrow$GA) language pair has been evaluated in
previous studies [65]. Carefully selecting the appropriate subword model has
been shown to be an important driver of translation performance. A Transformer
architecture, using a 16k BPE SentencePiece subword model, demonstrated
optimal performance.

####  4.4.3 Neural MT

Using large bilingual corpora, NMT approaches require the training of neural
networks to learn a statistical model for MT. The technique has demonstrated
state-of-the-art translation performance on many benchmarks. However, one of
the key factors in enabling the development of high-performing NMT models is
the availability of large amounts of parallel data [58, 104].

###  4.5 Proposed Approach

####  4.5.1 Sources for gaHealth Development

To build a bilingual corpus of health data, we selected multiple sources of
professionally translated documents from within the Irish government, all of
which are publicly available. In particular, the bilingual strategy statements
and annual reports of the Irish Department of Health since 2010 were chosen.

Furthermore, a dataset of Covid-related data, developed for a previous study
[64], was incorporated into a larger health dataset. Given the pace at which
Covid-19 data was being published, translated Irish website content often
lagged behind the English-language counterpart. Website snapshots taken by the
WayBack Machine111https://archive.org/web/ [6] proved particularly useful in
creating good parallel data from unaligned parallel websites. Extracts from
the corpus are illustrated in Table 4.1.

Type |  Sentence
---|---
EN-1 |  The Programme for Government makes strong commitments to strengthen community-based care, including primary care and social care, and sees this as fundamental to advancing Sláintecare reforms.
GA-1 |  Tá gealltanais láidre sa Chlár Rialtais maidir le cúram pobalbhunaithe a neartú, cúram príomhúil agus cúram sóisialta san áireamh, agus breathnaítear air sin mar chuid bhunriachtanach d’athchóirithe Sláintecare a chur ar aghaidh.
EN-2 |  The Hepatitis C Compensation Tribunal (Amendment) Act 2006 established a statutory scheme to address insurance difficulties experienced by persons infected with Hepatitis C and HIV through the administration within the State of blood and blood products.
GA-2 |  Faoin Acht um Binse Cúitimh i ndáil le Heipitíteas C (Leasú) 2006, bunaíodh scéim reachtúil chun dul i ngleic le deacrachtaí árachais a bhí ag daoine a bhí ionfhabhtaithe le Heipitíteas C agus VEID trí fhuil agus táirgí fola a tugadh dóibh laistigh den Stát.
EN-3 |  To provide virological evidence on the presence and extent of undetected community transmission of covid-19 and monitor positivity rates among individuals presenting ill or acute respiratory tract infections to primary care.
GA-3 |  Fianaise víreolaíoch a sholáthar maidir le láithreacht agus méid an tarchuir pobail anaithnid de covid-19 agus monatóireacht a dhéanamh ar rátaí dearfacha i measc daoine aonair a bhfuil ionfhabhtuithe ili nó conaire riospráide géarmhíochaine orthu chuig cúram príomhúil.
Table 4.1: Extracts from the gaHealth corpus are illustrated in this table.
The sentences, EN-1 / GA-1, are drawn from strategy statements, EN-2 / GA-2
are taken from annual reports whereas EN-3 / GA-3 are from Covid sources.

This amalgamated corpus, gaHealth, consists of 16,201 lines of parallel text
files. The combined English and Irish vocabulary size is 19,269 unique words.
The constituent elements of the dataset, prior to applying the toolchain, are
outlined in Table 4.2.

Documents | Source | Lines |
---|---|---|---
Strategy Statement 2020 | HSE | 3k |
Strategy Statement 2017 | HSE | 2.5k |
Strategy Statement 2015 | HSE | 3k |
Annual Report 2020 | HSE | 2k |
Annual Report 2019 | HSE | 2k |
Annual Report 2017 | HSE | 2k |
Website (Covid) | Citizen’s Advice | 4k |
Publications (Covid) | HSE | 4k |
Table 4.2: Sources used in corpus development

####  4.5.2 Toolchain used for gaHealth Development

The HSE PDF and Word documents were pre-processed using a toolchain currently
under development as part of the Irish Language Resource Infrastructure
project (ILRI), funded by the Department of the Gaeltacht. This toolchain has
been written to accept primarily data that originates in public administration
organisations, i.e. relatively formal text for which the translation quality
is assumed to be high, the structure and formatting to be reasonably
consistent, and the potential for noise to be low.

The source material consisted of a combination of twelve input files: six in
English (all PDF) and six in Irish (five PDF, one Word); all PDFs had a text
layer. They ranged from relatively short (30 to 40 A4 pages) to more
substantial (ca. 200 pages). PDFs in particular can be problematic for
creating high-quality corpora for a variety of reasons. In other words, while
the quality of the input content in this case can be said to be high, the
quality of the input medium is low.

The process used for developing the corpus is illustrated in Figure 4.1. The
toolchain consists of a set of components run in sequence over a set of input
documents, to convert it from raw content to a sentence-aligned corpus.
Several of the components listed below have different implementations
depending on the source type and intended output.

Figure 4.1: Corpus development process. In developing the corpus, the key
steps of data collection, pre-processing, alignment and validation were
followed. The role of the toolchain at various stages is highlighted.

###### Text extractors

These are wrappers around external components that extract text based on input
type; for the gaHealth file types, wrappers were written for LibreOffice and
pdftotext.

###### Unicode normalizer

This is used to achieve Unicode equivalence, and optionally to substitute
certain (e.g. corrupted) characters.

###### Language detector

In this toolchain, the language detector is a wrapper around langdetect,
itself a port from language-detection by Nakatani Shuyo [83]. The wrapper was
written to allow this to be run conveniently either on a string or on an
entire file.

###### Sentence splitters

These are custom-written components to reconstruct sentence boundary
information. For editable file types like plain text and Word, this process is
relatively straightforward. However, PDFs present particular challenges in
this regard. Along with ordering issues, the absence of sentence boundary
information is one of the biggest reasons why it is so difficult to construct
a high-quality corpus from PDFs. A custom sentence splitter was used to
determine sentence boundaries from text extracted from PDFs specifically,
primarily using capitalisation and language-specific lists of abbreviations to
determine where sentences should be broken.

###### Document aligner

This aligns sets of files whose languages have been identified. A wrapper was
written around an external component called FaDA [73] to adapt it to the
toolchain. As FaDA always names an alignment for each input file, sometimes
even mapping two different files to the same one, it was necessary to put some
selection logic in here, as well as a mechanism for determining when there is
no appropriate mapping. Constraints may be put on the relative size of the
files to accept an alignment, and the process may be re-run multiple times,
with previously rejected files being run again.

###### Sentence aligner

This aligns pairs of files at the sentence level. This was a wrapper around
the external component hunalign [45]. No special parameters were used, as
default settings produced results of high quality.

###### Text cleaners

These remove sentence pairs that are believed to be incorrect alignments, such
as empty segments and those with obviously mismatched content.

####  4.5.3 Guidelines

With the above considerations in mind, the following set of rules was decided
upon when processing the gaHealth dataset. Many of these could be specified as
parameters to the toolchain, while others were hard-coded into the system.

  1. 1.

Unicode standard: normalise all characters to Unicode UTF-8 NFC. Remove any
byte order marks.

  2. 2.

Whitespacing and capitalisation: merge sequences of whitespace characters into
a single space. Do not perform tokenization or truecasing.

  3. 3.

File language detection: scan the first 50 lines, and then every 100th line.

  4. 4.

Document alignment: assume that specific patterns like a line beginning with a
single letter in parentheses or a number followed by a full stop indicate a
sentence break from the previous line. Ensure each document is 0.75-1.33 times
the size of the document it is being aligned with. Run for a maximum of three
iterations.

  5. 5.

Sentence alignment: allow one-to-many alignments.

  6. 6.

Cleaning: remove any pairs where source or target:

     * •

is empty

     * •

contains no non-alphabetical characters

     * •

is of an incorrect language. This will remove most untranslated segments. The
language is only to be detected for segments that have at least 40 characters

####  4.5.4 Transformer Architecture

All EN$\leftrightarrow$GA models, trained with the gaHealth corpus, were
developed using a Transformer architecture. A reduction in the number of
hidden layer neurons and increasing dropout significantly improved
performance. Furthermore, within the context of low-resource EN$\rightarrow$GA
translation, using a 16k BPE submodel resulted in the highest-performing
models. Optimal hyperparameters were selected in line with these findings from
our previous and are outlined in Table 4.3.

Hyperparameter | Values
---|---
Learning rate | 0.1, 0.01, 0.001, 2
Batch size | 1024, 2048, 4096, 8192
Attention heads |  2, 4, 8
Number of layers | 5, 6
Feed-forward dimension | 2048
Embedding dimension | 128, 256, 512
Label smoothing |  0.1, 0.3
Dropout | 0.1, 0.3
Attention dropout | 0.1
Average Decay | 0, 0.0001
Table 4.3: Hyperparameter optimisation for Transformer models. Optimal
parameters are highlighted in bold [65].

###  4.6 Empirical Evaluation

In addition to developing the gaHealth corpus, the effectiveness of the
dataset was evaluated by training models for EN$\leftrightarrow$GA translation
in the Health domain. All experiments involved concatenating source and target
corpora to create a shared vocabulary and a shared SentencePiece [61] subword
model. The impact of using separate source and target subword models was not
explored.

To benchmark the performance of our models, the EN$\rightarrow$GA and
GA$\rightarrow$EN test datasets from the LoResMT2021 Shared Task [85] were
used. These test datasets enabled the evaluation of the gaHealth corpus, and
associated models since this shared task focused on an application of the
health domain i.e. the translation of Covid-related data. Furthermore, using a
shared task test dataset enables the comparison of gaHealth models’
performance with models entered by other teams.

The results from the IIITT [92] and UCF [27] teams are included in Tables 4.6
and 4.7 so the performance of the gaHealth models can be easily compared with
the findings of LoResMT2021. IIITT fine-tuned an Opus MT model from Helsinki
NLP on the training dataset. UCF [27] used transfer learning, unigram and
subword segmentation methods for EN$\leftrightarrow$GA translation.

####  4.6.1 Infrastructure

Rapid prototype development was enabled through a Google Colab Pro
subscription using NVIDIA Tesla P100 PCIe 16GB graphic cards and up to 27GB of
memory when available [19]. Our MT models were trained using the Pytorch
implementation of OpenNMT 2.0, an open-source toolkit for NMT [56].

####  4.6.2 Metrics

Automated metrics were used to determine the translation quality. All models
were trained and evaluated using the BLEU [88], TER [105] and ChrF [90]
evaluation metrics. Case-insensitive BLEU scores, at the corpus level, are
reported. Model training was stopped after 40k training steps or once an early
stopping criteria of no improvement in validation accuracy for four
consecutive iterations was recorded.

####  4.6.3 Results: Automatic Evaluation

The hyperparameters used for developing the models are outlined in Table 4.3.
The details of the training, validation and test sets used by our NMT models
are outlined in Table 4.4 and Table 4.5. In all cases, 502 lines were used
from the LoResMT2021 validation dataset whereas the test dataset used 502
lines for EN$\rightarrow$GA translation and 250 lines for GA$\rightarrow$EN
translation. Both were independent health-specific Covid test sets which were
provided by LoResMT2021. There was one exception, due to a data overlap
between test and training data, a reduced test set was used when testing the
gaHealth en2ga* system.

Team | System | Train | Dev | Test
---|---|---|---|---
adapt | covid_extended | 13k | 502 | 500
adapt | combined_domains | 65k | 502 | 500
IIITT | en2ga-b | 8k | 502 | 500
UCF | en2ga-a | 8k | 502 | 500
gaHealth | en2ga | 24k | 502 | 500
gaHealth | en2ga* | 24k | 502 | 338
Table 4.4: EN$\rightarrow$GA training, validation and test dataset distributions. The baseline gaHealth system was augmented with an 8k Covid dataset provided by LoResMT2021. A smaller test set was used when evaluating gaHealth en2ga* due to an overlap with the training data. An alternative approach of removing the overlap from the gaHealth corpus, before training, was also carried out to produce the gaHealth en2ga system. Team | System | Train | Dev | Test
---|---|---|---|---
IIITT | ga2en-b | 8k | 502 | 250
UCF | ga2en-b | 8k | 502 | 250
gaHealth | ga2en | 24k | 502 | 250
Table 4.5: GA$\rightarrow$EN training, validation and test dataset
distributions. The baseline gaHealth system was augmented with an 8k Covid
dataset provided by LoResMT2021. All overlaps were removed from the gaHealth
corpus prior to training the gaHealth ga2en model.

Figure 4.2: gaHealth en2ga* system: training EN$\rightarrow$GA model with
combined 16k gaHealth corpus and 8k LoResMT2021 covid corpus achieving a max
validation accuracy of 38.5% and perplexity of 111 after 40k steps. BLEU
score: 37.6.

Figure 4.3:  adapt covid_extended system: training EN$\rightarrow$GA model
with 8k LoResMT2021 covid corpus achieving a max validation accuracy of 30.0%
and perplexity of 354 after 30k steps. BLEU score: 36.0.

Figure 4.4: gaHealth ga2en system: training GA$\rightarrow$EN model with
combined 16k gaHealth corpus and 8k LoResMT2021 Covid corpus achieving a max
validation accuracy of 39.5% and perplexity of 116 after 40k steps. BLEU
score: 57.6.

Experimental results achieved using a Transformer architecture, are summarised
in Table 4.6 and Table 4.7. In the LoResMT2021 Shared Task, the highest-
performing EN$\rightarrow$GA system was submitted by the ADAPT team. The
system uses an extended Covid dataset (13k, which is a combination of the MT
summit Covid baseline and a custom DCU Covid dataset. This Transformer model,
with 2 heads, performs well across all key translation metrics (BLEU: 36.0,
TER: 0.531 and ChrF3: 0.6).

Team | System |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---|---
UCF | en2ga-b | 13.5 | 0.756 | 0.37
IIITT | en2ga-b | 25.8 | 0.629 | 0.53
adapt | combined | 32.8 | 0.590 | 0.57
gaHealth | en2ga | 33.3 | 0.604 | 0.56
adapt | covid_extended | 36.0 | 0.531 | 0.60
gaHealth | en2ga* | 37.6 | 0.577 | 0.57
Table 4.6: EN$\rightarrow$GA gaHealth system compared with LoResMT2021
systems.

Validation accuracy, and model perplexity, in developing the gaHealth models
are illustrated in Figure 4.2 and Figure 4.4 whereas Figure 4.3 illustrates
model training on just the covid_extended dataset. Rapid convergence was
observed while training the gaHealth models such that little accuracy
improvement occurs after 30k steps. Only marginal gains were achieved after
this point and it declined in the case of the system trained using the
covid_extended dataset.

Perplexity (PPL) shows how many different, equally probable words can be
produced during translation. As a metric for translation performance, it is
important to keep low scores so the number of alternative translations is
reduced.

Of the models developed by the ADAPT team, the worst-performing model uses a
larger 65k dataset. This is not surprising given the dataset is from a generic
domain of which only 20% is health-related. The performance of this higher-
resourced 65k line model lags the augmented gaHealth model which was developed
using just 24k lines.

Team | System |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$
---|---|---|---|---
UCF | ga2en-b | 21.3 | 0.711 | 0.45
IIITT | ga2en-b | 34.6 | 0.586 | 0.61
gaHealth | ga2en | 57.6 | 0.385 | 0.71
Table 4.7: GA$\rightarrow$EN gaHealth system compared with LoResMT2021
systems.

For GA$\rightarrow$EN translation, the best-performing model for the
LoResMT2021 Shared Task was developed by IIITT with a BLEU of 34.6, a TER of
0.586 and ChrF3: 0.6. This effectively serves as the baseline by which our
GA$\rightarrow$EN model, developed using the gaHealth corpus, can be
benchmarked. The performance of the gaHealth model offers an improvement
across all metrics with a BLEU score of 57.6, a TER of 0.385 and a CHrF3
result of 0.71. In particular, the 40% improvement in BLEU score is very
significant.

###  4.7 Discussion

Although the main objective of this work is to develop the first bilingual
corpus of EN$\leftrightarrow$GA data, we conduct initial experiments on the
effectiveness of such datasets in training MT models. We have used our
gaHealth dataset to train an MT model on test data from the LoResMT2021 Shared
task to evaluate how the system performs translating EN$\leftrightarrow$GA
health data. Our systems, developed using the gaHealth corpus achieved
significantly higher scores.

###  4.8 Conclusion and Future Work

The main contribution of this work is to present an ongoing translation
project that aims at building the first ever parallel corpus of health data
for the Irish language – gaHealth – by fully utilising freely available
parallel documents.

Due to the issues encountered during the conversion of PDF documents, we
developed guidelines to aid in the conversion process. In addition to
developing the gaHealth corpus, we trained and evaluated translation models
for in-domain health data.

In our experiments, the models achieved a BLEU score of 37.6 (Table 4.6) for
translating EN$\rightarrow$GA test data and 57.6 (Table 4.7) for
GA$\rightarrow$EN translation, which is encouraging performance given this is
the beginning of our work on gaHealth. There is no such corpus available
according to the best of our knowledge, so gaHealth will become a useful
resource in the NLP community, especially for those working with the Irish
language domain.

For future work, we intend to extend the corpus, as more Irish language
documents become available. Upon extension, we will refine our models. One
important aspect which needs further investigation is to understand why the
EN$\rightarrow$GA model (gaHealth en2ga), tested with the full test set,
performed worse than the model (gaHealth en2ga*) which was tested with the
reduced test set. A deep linguistic investigation involving a sentence-level
BLEU analysis will be conducted as part of a future study.

In addition, we aim to build in-domain datasets for other key domains such as
Education and Finance. We will also apply deep learning techniques to further
refine our in-domain models. We have released the gaHealth corpus
online222https://github.com/seamusl/gaHealth to facilitate further research on
this data set.

##  Chapter 5 Human Evaluation of EN$\leftrightarrow$GA Transformer-Based NMT

###  5.1 Context

In addressing RQ3, we employed a rigorous quantitative human evaluation of the
translation outputs of RNN and Transformer models for a specific low-resource
language pair. These methods and results are detailed in the paper “Human
evaluation of English–Irish Transformer-based NMT”. Even though a different,
human-centric approach was adopted, there is continuity with our previous
research efforts. In particular, we sought a human validation of Transformer
HPO for low-resource languages whereas in our previous work, “Transformers for
low-resource languages - is féidir linn!”, the focus was on automatic metrics.

For our assessment, we employed an MQM error taxonomy complemented by an SQM
to pinpoint the error types in both RNN and Transformer systems. Collaborating
with native Irish speakers, our study devised a method for human evaluation
under limited resources. Engaging native speakers to assess reference
translations, we verified the outputs of both models. Our data reveals the
Transformer system substantially reduces errors in accuracy and fluency. A
pivotal aspect of our research is the linguistic feedback from our annotators.

Human Evaluation of EN$\leftrightarrow$GA Transformer-Based NMT

Séamus Lankford
Haithem Afli
Andy Way

Information 13.7, MDPI

January 10th, 2023

ADAPT Centre

Dublin City University

Ireland

https://www.mdpi.com/2078-2489/13/7/309

###  5.2 Abstract

In this study, a human evaluation is carried out on how hyperparameter
settings impact the quality of Transformer-based neural machine translation
for the low-resource English-Irish language pair. SentencePiece models using
both byte pair encoding (BPE) and unigram approaches were appraised.
Variations in model architectures included modifying the number of layers,
evaluating the optimal number of heads for attention and testing various
regularisation techniques. The greatest performance improvement was recorded
for a Transformer-optimised model with a 16k BPE subword model. Compared with
a baseline recurrent neural network (RNN) model, a Transformer-optimised model
demonstrated a BLEU score improvement of 7.8 points. When benchmarked against
Google Translate, our translation engines demonstrated significant
improvements. Furthermore, a quantitative fine-grained manual evaluation was
conducted which compared the performance of machine translation systems. Using
the multidimensional quality metrics error taxonomy, a human evaluation of the
error types generated by an RNN-based system and a Transformer-based system
was explored. Our findings show the best-performing Transformer system