# Transformers for Low-Resource Languages:
Is Féidir Linn!

Séamus Lankford<EMAIL_ADDRESS>
ADAPT Centre, Department of Computing, Dublin City University, Dublin,
Ireland. Haithem Afli<EMAIL_ADDRESS>
ADAPT Centre, Department of Computer Science, Munster Technological
University, Ireland. Andy Way<EMAIL_ADDRESS>
ADAPT Centre, School of Computing, Dublin City University, Dublin, Ireland

###### Abstract

The Transformer model is the state-of-the-art in Machine Translation. However,
in general, neural translation models often under perform on language pairs
with insufficient training data. As a consequence, relatively few experiments
have been carried out using this architecture on low-resource language pairs.
In this study, hyperparameter optimization of Transformer models in
translating the low-resource English-Irish language pair is evaluated. We
demonstrate that choosing appropriate parameters leads to considerable
performance improvements. Most importantly, the correct choice of subword
model is shown to be the biggest driver of translation performance.
SentencePiece models using both unigram and BPE approaches were appraised.
Variations on model architectures included modifying the number of layers,
testing various regularisation techniques and evaluating the optimal number of
heads for attention. A generic 55k DGT corpus and an in-domain 88k public
admin corpus were used for evaluation. A Transformer optimized model
demonstrated a BLEU score improvement of 7.8 points when compared with a
baseline RNN model. Improvements were observed across a range of metrics,
including TER, indicating a substantially reduced post editing effort for
Transformer optimized models with 16k BPE subword models. Bench-marked against
Google Translate, our translation engines demonstrated significant
improvements. The question of whether or not Transformers can be used
effectively in a low-resource setting of English-Irish translation has been
addressed. Is féidir linn - yes we can.

##  1 Introduction

The advent of Neural Machine Translation (NMT) has heralded an era of high-
quality translations. However, these improvements have not been manifested in
the translation of all languages. Large datasets are a prerequisite for high
quality NMT. This works well in the context of well-resourced languages where
there is an abundance of data. In the context of low-resource languages which
suffer from a sparsity of data, alternative approaches must be adopted.

An important part of this research involves developing applications and models
to address the challenges of low-resource language technology. Such technology
incorporates methods to address the data scarcity affecting deep learning for
digital engagement of low-resource languages.

It has been shown that an out-of-the-box NMT system, trained on English-Irish
data, achieves a lower translation quality compared with using a tailored SMT
system (Dowling et al, 2018). It is in this context that further research is
required in the development of NMT for low-resource languages and the Irish
language in particular.

Most research on choosing subword models has focused on high resource
languages (Ding et al.,, 2019; Gowda and May,, 2020). In the context of
developing models for English to Irish translation, there are no clear
recommendations on the choice of subword model types. One of the objectives in
this study is to identify which type of subword model performs best in this
low resource scenario.

##  2 Background

Native speakers of low-resource languages are often excluded from useful
content since, more often than not, online content is not available to them in
their language of choice. Such a digital divide and the resulting social
exclusion experienced by second language speakers, such as refugees living in
developed countries, has been well documented in the research literature
(MacFarlane et al.,, 2008; Alam and Imran,, 2015).

Research on Machine Translation (MT) in low-resource scenarios directly
addresses this challenge of exclusion via pivot languages (Liu et al.,, 2018),
and indirectly, via domain adaptation of models (Ghifary et al.,, 2016).
Breakthrough performance improvements in the area of MT have been achieved
through research efforts focusing on NMT (Bahdanau et al.,, 2014; Cho et al.,,
2014). Consequently, state-of-the-art (SOA) performance has been attained on
multiple language pairs (Bojar et al.,, 2017, 2018).

###  2.1 Irish Language

The Irish language is a primary example of such a low-resource language that
will benefit from this research. NMT involving Transformer model development
will improve the performance in specific domains of low-resource languages.
Such research will address the end of the Irish language derogation in the
European Commission in 2021 111amtaweb.org/wp-content/uploads/2020/11/MT-in-
EU-Overview-with-Voiceover-Andy-Way-KEYNOTE-K1.pdf (Way,, 2020) helping to
deliver parity in support for Irish in online digital engagement.

###  2.2 Hyperparameter Optimization

Hyperparameters are employed in order to customize machine learning models
such as translation models. It has been shown that machine learning
performance may be improved through hyperparameter optimization (HPO) rather
than just using default settings (Sanders and Giraud-Carrier,, 2017).

The principle methods of HPO are Grid Search (Montgomery,, 2017) and Random
Search (Bergstra and Bengio,, 2012)]. Grid search is an exhaustive technique
which evaluates all parameter permutations. However, as the number of features
grows, the amount of data permutations grows exponentially making optimization
expensive in the context of developing long running translation models.

An effective, and less computationally intensive, alternative is to use random
search which samples random configurations.

####  2.2.1 Recurrent Neural Networks

Recurrent neural networks are often used for the tasks of natural language
processing, speech recognition and MT. RNN models enable previous outputs to
be used as inputs while having hidden states. In the context of MT, such
neural networks were ideal due to their ability to process inputs of any
length. Furthermore, the model sizes do not necessarily increase with the size
of its input. Commonly used variants of RNN include Bidirectional (BRNN) and
Deep (DRNN) architectures. However, the problem of vanishing gradients coupled
with the development of attention-based algorithms often leads to Transformer
models performing better than RNNs.

####  2.2.2 Transformer

The greatest improvements have been demonstrated when either the RNN or the
CNN architecture is abandoned completely and replaced with an attention
mechanism creating a much simpler and faster architecture known as Transformer
(Vaswani et al.,, 2017). Transformer models use attention to focus on
previously generated tokens. The approach allows models to develop a long
memory which is particularly useful in the domain of language translation.
Performance improvements to both RNN and CNN approaches may be achieved
through the introduction of such attention layers in the translation
architecture.

Experiments in MT tasks show such models are better in quality due to greater
parallelization while requiring significantly less time to train.

###  2.3 Subword Models

Translation, by its nature, requires an open vocabulary and the use of subword
models aims to address the fixed vocabulary problem associated with NMT. Rare
and unknown words are encoded as sequences of subword units. By adapting the
original Byte Pair Encoding (BPE) algorithm (Gage,, 1994), the use of BPE
submodels can improve translation performance (Sennrich et al.,, 2015; Kudo,,
2018).

Designed for NMT, SentencePiece, is a language-independent subword tokenizer
that provides an open-source C++ and a Python implementation for subword
units. An attractive feature of the tokenizer is that SentencePiece trains
subword models directly from raw sentences (Kudo and Richardson,, 2018).

####  2.3.1 Byte Pair Encoding compared with Unigram

BPE and unigram language models are similar in that both encode text using
fewer bits but each uses a different data compression principle (dictionary
vs. entropy). In principle, we would expect the same benefits with the unigram
language model as with BPE. However, unigram models are often more flexible
since they are probabilistic models that output multiple segmentations with
their probabilities.

Figure 1: Proposed Approach Hyperparameter | Values
---|---
Learning rate | 0.1, 0.01, 0.001, 2
Batch size | 1024, 2048, 4096, 8192
Attention heads |  2, 4, 8
Number of layers | 5, 6
Feed-forward dimension | 2048
Embedding dimension | 128, 256, 512
Label smoothing |  0.1, 0.3
Dropout | 0.1, 0.3
Attention dropout | 0.1
Average Decay | 0, 0.0001
Table 1: Hyperparameter Optimization for Transformer models. Optimal
parameters are highlighted in bold. The highest performing model trained on
the 55k DGT corpus uses 2 attention heads whereas the best model trained with
the larger 88k PA dataset uses 8 attention heads.

##  3 Proposed Approach

HPO of RNN models in low-resource settings has previously demonstrated
considerable performance improvements. The extent to which such optimization
techniques may be applied to Transformer models in similar low-resource
scenarios is evaluated as part of this study. Evaluations included modifying
the number of attention heads, the number of layers and experimenting with
regularization techniques such as dropout and label smoothing. Most
importantly, the choice of subword model type and the vocabulary size are
evaluated.

In order to test the effectiveness of our approaches, optimization was carried
out on two English-Irish parallel datasets: a general corpus of 52k lines from
the Directorate General for Translation (DGT) and an in-domain corpus of 88k
lines of Public Administration (PA) data. With DGT, the test set used 1.3k
lines and the development set comprised of 2.6k lines. In the case of the PA
dataset, there were 1.5k lines of test data and 3k lines of validation. All
experiments involved concatenating source and target corpora to create a
shared vocabulary and a shared SentencePiece subword model. The impact of
using separate source and target subword models was not explored.

The approach adopted is illustrated in Figure 1. Two baseline architectures,
RNN and Transformer, are evaluated. On evaluating the hyperparameter choices
for Transformer models, the values outlined in Table 1 were tested using a
random search approach. A range of values for each parameter was tested using
short cycles of 5k training steps. Once an optimal value, within the sampled
range was identified, it was locked in for tests on subsequent parameters.

###  3.1 Architecture Tuning

Given the long training times associated with NMT, it is difficult and costly
to tune systems using a conventional Grid Search approach. Therefore a Random
Search approach was adopted in the HPO of our transformer models.

With low-resource datasets, the use of smaller and fewer layers has previously
been shown to improve performance (Araabi and Monz,, 2020). Performance of
low-resource NMT has also been demonstrated to improve in cases where shallow
Transformer models are adopted (Van Biljon et al.,, 2020). Guided by these
findings, configurations were tested which varied the number of neurons in
each layer and modified the number of layers used in the Transformer
architecture.

The impact of regularization, by applying varying degrees of dropout to
Transformer models, was evaluated. Configurations using smaller (0.1) and
larger values (0.3) were applied to the output of each feed forward layer.

###  3.2 Subword Models

It has become standard practise to incorporate word segmentation approaches,
such as Byte-Pair-Encoding (BPE), when developing NMT models. Previous work
shows that subword models may be particularly beneficial for low-resource
languages since rare words are often a problem. Reducing the number of BPE
merge operations resulted in substantial improvements of 5 BLEU points
(Sennrich and Zhang 2019) when tested on RNN models.

In the context of English to Irish translation, there is no clear agreement as
to what constituted the best approach. Consequently, as part of this study,
subword regularization techniques, involving BPE and unigram models were
evaluated to determining the optimal parameters for maximising translation
performance. BPE models with varying vocabulary sizes of 4k, 8k, 16k and 32k
were tested.

##  4 Empirical Evaluation

###  4.1 Experimental Setup

####  4.1.1 Datasets

The performance of the Transformer and RNN approaches is evaluated on English
to Irish parallel datasets. Two datasets were used in the evaluation of our
models namely the publicly available DGT dataset which may be broadly
categorised as generic and an in-domain dataset which focuses on public
administration data.

The DGT, and its Joint Research Centre, has made available all Translation
Memory (TM; i.e. sentences and their professionally produced translations)
which cover all official European Union languages (Steinberger et al.,, 2013).

Data provided by the Department of Tourism, Culture, Arts, Gaeltacht, Sport
and Media in Ireland formed the majority of the data in the public
administration dataset. This includes staff notices, annual reports, website
content, press releases and official correspondence.

Parallel texts from the Digital Corpus of the European Parliament (DCEP) and
the DGT are included in the training data. Crawled data, from sites of a
similar domain are included. Furthermore a parallel corpus collected from
Conradh na Gaeilge (CnaG), an Irish language organisation that promotes the
Irish language, was included. The dataset was compiled as part of a previous
study which carried out a preliminary comparison of SMT and NMT models for the
Irish language (Dowling et al.,, 2018).

####  4.1.2 Infrastructure

Models were developed using a lab of machines each of which has an AMD Ryzen 7
2700X processor, 16 GB memory, a 256 SSD and an NVIDIA GeForce GTX 1080 Ti.
Rapid prototype development was enabled through a Google Colab Pro
subscription using NVIDIA Tesla P100 PCIe 16 GB graphic cards and up to 27GB
of memory when available (Bisong,, 2019).

Our MT models were trained using the Pytorch implementation of OpenNMT 2.0, an
open-source toolkit for NMT (Klein et al.,, 2017).

####  4.1.3 Metrics

As part of this study, several automated metrics were used to determine the
translation quality. All models were trained and evaluated on both the DGT and
PA datasets using the BLEU (Papineni et al.,, 2002), TER (Snover et al.,,
2006) and ChrF (Popović,, 2015) evaluation metrics. Case-insensitive BLEU
scores, at the corpus level, are reported. Model training was stopped once an
early stopping criteria of no improvement in validation accuracy for 4
consecutive iterations was recorded.

###  4.2 Results

####  4.2.1 Performance of subword models

The impact on translation accuracy when choosing a subword model is
highlighted in Tables 2 \- 5. In training both RNN and Transformer
architectures, incorporating any submodel type led to improvements in model
accuracy. This finding is evident when training either the smaller generic DGT
dataset or the larger in-domain PA dataset.

Using an RNN architecture on DGT, as illustrated in Table 2, the best
performing model with a 32k unigram submodel, achieved a BLEU score 7.4%
higher than the baseline. With the PA dataset using an RNN, as shown in Table
3, the model with the best BLEU, TER and ChrF3 scores again used a unigram
submodel.

Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
dgt-rnn-base | 52.7 | 0.42 | 0.71 | 75k | 4.47 | 0
dgt-rnn-bpe8k | 54.6 | 0.40 | 0.73 | 85k | 5.07 | 0
dgt-rnn-bpe16k | 55.6 | 0.39 | 0.74 | 100k | 5.58 | 0
dgt-rnn-bpe32k | 55.3 | 0.39 | 0.74 | 95k | 4.67 | 0
dgt-rnn-unigram | 55.6 | 0.39 | 0.74 | 105k | 5.07 | 0
Table 2: RNN performance on DGT dataset of 52k lines Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
pa-rnn-base | 40.4 | 0.47 | 0.63 | 60k | 2.13 | 0
pa-rnn-bpe8k | 41.5 | 0.46 | 0.64 | 110k | 4.16 | 0
pa-rnn-bpe16k | 41.5 | 0.46 | 0.64 | 105k | 3.78 | 0
pa-rnn-bpe32k | 41.9 | 0.47 | 0.64 | 100k | 2.88 | 0
pa-rnn-unigram | 41.9 | 0.46 | 0.64 | 95k | 2.75 | 0
Table 3: RNN performance on PA dataset of 88k lines

There are small improvements in BLEU scores when the RNN baseline is compared
with models using a BPE submodel of either 8k, 16k or 32k words, as
illustrated in Tables 2 and 3. The maximum BLEU score improvement of 1.5
points (2.5%) is quite modest in the case of the public admin corpus. However,
there are larger gains with the DGT corpus. A baseline RNN model, trained on
DGT, achieved a BLEU score of 52.7 whereas the highest-performing BPE variant,
using a 16k vocab, recorded an improvement of nearly 3 points with a score of
55.6.

Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
dgt-trans-base | 53.4 | 0.41 | 0.72 | 55k | 14.43 | 0.81
dgt-trans-bpe8k | 59.5 | 0.34 | 0.77 | 200k | 24.48 | 1.38
dgt-trans-bpe16k | 60.5 | 0.33 | 0.78 | 180k | 26.90 | 1.52
dgt-trans-bpe32k | 59.3 | 0.35 | 0.77 | 100k | 18.03 | 1.02
dgt-trans-unigram | 59.3 | 0.35 | 0.77 | 125k | 21.95 | 1.24
Table 4: Transformer performance on 52k DGT dataset. Highest performing model uses 2 attention heads. All other models use 8 attention heads. Architecture |  BLEU $\uparrow$ |  TER $\downarrow$ |  ChrF3 $\uparrow$ | Steps |  | Runtime
---
(hours)
kgCO2
pa-trans-base | 44.1 | 0.44 | 0.66 | 20k | 5.97 | 0.34
pa-trans-bpe8k | 46.6 | 0.40 | 0.68 | 160k | 20.1 | 1.13
pa-trans-bpe16k | 47.1 | 0.41 | 0.68 | 100k | 14.22 | 0.80
pa-trans-bpe32k | 46.8 | 0.41 | 0.68 | 70k | 12.7 | 0.72
pa-trans-unigram | 46.6 | 0.42 | 0.68 | 75k | 13.34 | 0.75
Table 5: Transformer performance on 88k PA dataset. All models use 8 attention
heads.

In the context of Transformer architectures, highlighted in Table 4 and Table
5, the use of subword models delivers significant performance improvements for
both the DGT and public admin corpora. The performance gains for Transformer
models are far greater than RNN models. Baseline DGT Transformer models
achieve a BLEU score of 53.4 while a Transformer model, with a 16k BPE
submodel, has a score of 60.5 representing a BLEU score improvement of 13% at
7.1 BLEU points.

For translating into a morphologically rich language, such as Irish, the ChrF
metric has proven successful in showing strong correlation with human
translation (Stanojević et al.,, 2015). In the context of our experiments, it
worked well in highlighting the performance differences between RNN and
Transformer architectures.

Figure 2: BLEU performance for all model architectures

####  4.2.2 Transformer performance compared with RNN

The performance of RNN models is contrasted with the Transformer approach in
Figure 2 and Figure 3. Transformer models, as anticipated, outperform all
their RNN counterparts. It is interesting to note the impact of choosing the
optimal vocabulary size for BPE submodels. Both datasets demonstrate that
choosing a BPE vocabulary of 16k words yields the highest performance.

Furthermore, the TER scores highlighted in Figure 3 reinforce the findings
that using 16k BPE submodels on Transformer architectures leads to better
translation performance. The TER score for the DGT Transformer 16k BPE model
is significantly better (0.33) when compared with the baseline performance
(0.41).

Figure 3: TER performance for all model architectures

Figure 4: Training DGT Transformer baseline Figure 5: Training DGT Transformer
16k BPE

##  5 Environmental Impact

Motivated by the findings of Stochastic Parrots (Bender et al.,, 2021), energy
consumption during model development was tracked. Prototype model development
used Colab Pro, which as part of Google Cloud is carbon neutral (Lacoste et
al.,, 2019). However, longer running Transformer experiments were conducted on
local servers using 324 gCO2 per kWh
222https://www.seai.ie/publications/Energy-in-Ireland-2020.pdf(SEAI,, 2020).
The net result was just under 10 kgCO2 created for a full run of model
development. Models developed during this study, will be reused for ensemble
experiments in future work.

##  6 Discussion

Validation accuracy, and model perplexity, in developing the baseline and
optimal models for the DGT corpus are illustrated in Figure 4 and Figure 5.
Rapid convergence was observed while training the baseline model such that
little accuracy improvement occurs after 20k steps. Including a subword model
led to much slower convergence and there were only marginal gains after 60k
steps. Furthermore, it is observed that training the DGT model, with a 16k BPE
submodel, boosted validation accuracy by over 8% compared with its baseline.

With regard to the key metric of perplexity, it is shown to rise after
training for 15k steps in the baseline models. PPL was observed to rise at
later stages, typically after 40k steps in models developed using subword
models. Perplexity (PPL), shows how many different, equally probable words can
be produced during translation. As a metric for translation performance, it is
important to keep low scores so the number of alternative translations is
reduced. Therefore, for future model development it may be worthwhile to set
PPL as an early stopping parameter.

On examining the PPL graphs of Figure 4 and Figure 5, it is clear that a lower
global minimum is achieved when the Transformer approach is used with a 16k
BPE submodel. The PPL global minimum (2.7) is over 50% lower than the
corresponding PPL for the Transformer base model (5.5). Such a finding
illustrates that choosing an optimal submodel delivers significant performance
gains.

Translation engine performance was bench-marked against Google Translate’s
333https://translate.google.com/ English to Irish translation service which is
freely available on the internet. Four random samples were selected from the
English source test file and are presented in Table 6. Translation of these
samples was carried out on the optimal DGT Transformer model and using Google
Translate. Case insensitive, sentence level BLEU scores were recorded and are
presented in Table 7. The results are encouraging and indicate well-performing
translation models on the DGT dataset.

The optimal parameters selected in this discovery process are identified in
bold in Table 2. A higher initial learning rate of 2 coupled with an average
decay of 0.0001 led to longer training times but more accurate models. Despite
setting an early stopping parameter, many of the Transformer builds continued
for the full cycle of 200k steps over periods of 20+ hours.

Training transformer models with a reduced number of attention heads led to a
marginal improvement in translation accuracy with a smaller corpus. Our best
performing model on a 55k DGT corpus, with 2 heads and a 16k BPE submodel,
achieved a BLEU score of 60.5 and a TER score of 0.33. By comparison, using 8
heads with the same architecture and dataset yielded 60.3 for the BLEU and
0.34 for the TER. In the case of a larger 88k PA corpus, all transformer
models using 8 heads performed better than equivalent models using just 2
heads.

Source Language (English) | Reference Human Translation (Irish)
---|---
| A clear harmonised procedure, including the
---
necessary criteria for disease–free status,
should be established for that purpose.
| Ba cheart nós imeachta comhchuibhithe soiléir,
---
lena n-áirítear na critéir is gá do stádas saor
ó ghalar, a bhunú chun na críche sin.
| the mark is applied anew, as appropriate.
---
| déanfar an mharcáil arís, mar is iomchuí.
---
| If the court decides that a review is
---
justified on any of the grounds set out in
paragraph 1, the judgment given in the
European Small Claims Procedure shall
be null and void.
| Má chinneann an chúirt go bhfuil bonn cirt
---
le hathbhreithniú de bharr aon cheann de na
forais a leagtar amach i mír 1, beidh an
breithiúnas a tugadh sa Nós Imeachta Eorpach
um Éilimh Bheaga ar neamhní go hiomlán.
households where pet animals are kept; | teaghlaigh ina gcoimeádtar peataí;
Table 6: Samples of human reference translations Transformer (16 kBPE) |  BLEU $\uparrow$ | Google Translate |  BLEU $\uparrow$
---|---|---|---
| Ba cheart nós imeachta soiléir
---
comhchuibhithe, lena n-áirítear
na critéir is gá maidir le
stádas saor ó ghalair, a bhunú
chun na críche sin.
61.6 |  | Ba cheart nós imeachta
---
comhchuibhithe soiléir, lena
n-áirítear na critéir riachtanacha
maidir le stádas saor ó ghalair,
a bhunú chun na críche sin.
70.2
| go gcuirtear an marc i bhfeidhme,
---
de réir mar is iomchuí.
21.4 |  | cuirtear an marc i bhfeidhm as
---
an nua, de réir mar is cuí.
6.6
| Má chinneann an chúirt go bhfuil
---
bonn cirt le hathbhreithniú ar aon
cheann de na forais a leagtar amach
i mír 1, beidh an breithiúnas a
thugtar sa Nós Imeachta Eorpach
um Éilimh Bheaga ar neamhní.
77.3 |  | Má chinneann an chúirt go bhfuil
---
údar le hathbhreithniú ar aon
cheann de na forais atá leagtha
amach i mír 1, beidh an
breithiúnas a thugtar sa
Nós Imeachta Eorpach um
Éilimh Bheaga ar neamhní
59.1
teaghlaigh ina gcoimeádtar peataí; | 100 | teaghlaigh ina gcoinnítear peataí; | 30.2
Table 7: Transformer model compared with Google Translate using random samples
from the DGT corpus. Full evaluation of Google Translate on the DGT test set,
with 1.3k lines, generated a BLEU score of 46.3 and a TER score of 0.44.
Comparative scores on the test set using our Transformer model, with 2
attention heads and 16k BPE submodel realised 60.5 for BLEU and 0.33 for TER.

Standard Transformer parameters for batch size (2048) and the number of
encoder / decoder layers (6) were all observed to perform well on the DGT and
PA corpora. Reducing hidden neurons to 256 and increasing regularization
dropout to 0.3 improved translation performance and were chosen when building
all Transformer models.

##  7 Conclusion

In our paper, we demonstrated that a random search approach to hyperparameter
optimization leads to the development of high-performing translation models.

We have shown that choosing subword models, in our low-resource scenarios, is
an important driver for the performance of MT engines. Moreover, the choice of
vocabulary size leads to varying degrees of performance. Within the context of
low-resource English to Irish translation, we achieved optimal performance, on
a 55k generic corpus and an 88k in-domain corpus, when a Transformer
architecture with a 16k BPE submodel was used. The importance of selecting
hyperparameters in training low-resource Transformer models was also
demonstrated. By reducing the number of hidden layer neurons and increasing
dropout, our models performed significantly better than baseline models and
Google Translate.

Performance improvement of our optimized Transformer models, with subword
segmentation, was observed across all key indicators namely a higher
validation accuracy, a PPL achieved at a lower global minimum, a lower post
editing effort and a higher translation accuracy.

## Acknowledgements

This work was supported by ADAPT, which is funded under the SFI Research
Centres Programme (Grant 13/RC/2016) and is co-funded by the European Regional
Development Fund. This research was also funded by the Munster Technological
University.

## References

  * Alam and Imran, (2015) Alam, K. and Imran, S. (2015).  The digital divide and social inclusion among refugee migrants.  Information Technology & People.
  * Araabi and Monz, (2020) Araabi, A. and Monz, C. (2020).  Optimizing transformer for low-resource neural machine translation.  arXiv preprint arXiv:2011.02266.
  * Bahdanau et al., (2014) Bahdanau, D., Cho, K., and Bengio, Y. (2014).  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473.
  * Bender et al., (2021) Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021).  On the dangers of stochastic parrots: Can language models be too big?  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623.
  * Bergstra and Bengio, (2012) Bergstra, J. and Bengio, Y. (2012).  Random search for hyper-parameter optimization.  Journal of machine learning research, 13(2).
  * Bisong, (2019) Bisong, E. (2019).  Google colaboratory.  In Building Machine Learning and Deep Learning Models on Google Cloud Platform, pages 59–64. Springer.
  * Bojar et al., (2017) Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huang, S., Huck, M., Koehn, P., Liu, Q., Logacheva, V., Monz, C., Negri, M., Post, M., Rubino, R., Specia, L., and Turchi, M. (2017).  Findings of the 2017 conference on machine translation (WMT17).  In Proceedings of the Second Conference on Machine Translation, pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics.
  * Bojar et al., (2018) Bojar, O., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Koehn, P., and Monz, C. (2018).  Findings of the 2018 conference on machine translation (WMT18).  In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. Association for Computational Linguistics.
  * Cho et al., (2014) Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014).  On the properties of neural machine translation: Encoder-decoder approaches.  arXiv preprint arXiv:1409.1259.
  * Ding et al., (2019) Ding, S., Renduchintala, A., and Duh, K. (2019).  A call for prudent choice of subword merge operations in neural machine translation.  arXiv preprint arXiv:1905.10453.
  * Dowling et al., (2018) Dowling, M., Lynn, T., Poncelas, A., and Way, A. (2018).  Smt versus nmt: Preliminary comparisons for irish.
  * Gage, (1994) Gage, P. (1994).  A new algorithm for data compression.  C Users Journal, 12(2):23–38.
  * Ghifary et al., (2016) Ghifary, M., Kleijn, W. B., Zhang, M., Balduzzi, D., and Li, W. (2016).  Deep reconstruction-classification networks for unsupervised domain adaptation.  In European Conference on Computer Vision, pages 597–613. Springer.
  * Gowda and May, (2020) Gowda, T. and May, J. (2020).  Finding the optimal vocabulary size for neural machine translation.  arXiv preprint arXiv:2004.02334.
  * Klein et al., (2017) Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, A. M. (2017).  Opennmt: Open-source toolkit for neural machine translation.  arXiv preprint arXiv:1701.02810.
  * Kudo, (2018) Kudo, T. (2018).  Subword regularization: Improving neural network translation models with multiple subword candidates.  arXiv preprint arXiv:1804.10959.
  * Kudo and Richardson, (2018) Kudo, T. and Richardson, J. (2018).  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  arXiv preprint arXiv:1808.06226.
  * Lacoste et al., (2019) Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019).  Quantifying the carbon emissions of machine learning.  arXiv preprint arXiv:1910.09700.
  * Liu et al., (2018) Liu, C.-H., Silva, C. C., Wang, L., and Way, A. (2018).  Pivot machine translation using chinese as pivot language.  In China Workshop on Machine Translation, pages 74–85. Springer.
  * MacFarlane et al., (2008) MacFarlane, A., Glynn, L. G., Mosinkie, P. I., and Murphy, A. W. (2008).  Responses to language barriers in consultations with refugees and asylum seekers: a telephone survey of irish general practitioners.  BMC Family Practice, 9(1):1–6.
  * Montgomery, (2017) Montgomery, D. C. (2017).  Design and analysis of experiments.  John wiley & sons.
  * Papineni et al., (2002) Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002).  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  * Popović, (2015) Popović, M. (2015).  chrf: character n-gram f-score for automatic mt evaluation.  In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395.
  * Sanders and Giraud-Carrier, (2017) Sanders, S. and Giraud-Carrier, C. (2017).  Informing the use of hyperparameter optimization through metalearning.  In 2017 IEEE International Conference on Data Mining (ICDM), pages 1051–1056. IEEE.
  * SEAI, (2020) SEAI (2020).  Sustainable Energy in Ireland.
  * Sennrich et al., (2015) Sennrich, R., Haddow, B., and Birch, A. (2015).  Neural machine translation of rare words with subword units.  arXiv preprint arXiv:1508.07909.
  * Snover et al., (2006) Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006).  A study of translation edit rate with targeted human annotation.  In Proceedings of association for machine translation in the Americas, volume 200. Citeseer.
  * Stanojević et al., (2015) Stanojević, M., Kamran, A., Koehn, P., and Bojar, O. (2015).  Results of the wmt15 metrics shared task.  In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273.
  * Steinberger et al., (2013) Steinberger, R., Eisele, A., Klocek, S., Pilos, S., and Schlüter, P. (2013).  Dgt-tm: A freely available translation memory in 22 languages.  arXiv preprint arXiv:1309.5226.
  * Van Biljon et al., (2020) Van Biljon, E., Pretorius, A., and Kreutzer, J. (2020).  On optimal transformer depth for low-resource language translation.  arXiv preprint arXiv:2004.04418.
  * Vaswani et al., (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017).  Attention is all you need.  arXiv preprint arXiv:1706.03762.
  * Way, (2020) Way, A. (2020).  MT Developments in the EU: Keynote AMTA 2020.


# From masses and radii of neutron stars to EOS of nuclear matter through
neural network

Zehan Wu  School of Physics and Optoelectronics, South China University of
Technology, Guangzhou 510641, P.R. China  Dehua Wen111Corresponding author.
<EMAIL_ADDRESS>School of Physics and Optoelectronics, South China
University of Technology, Guangzhou 510641, P.R. China

###### Abstract

The equation of state (EOS) of dense nuclear matter is a key factor to
determine the internal structure and properties of neutron stars. However, the
EOS of high-density nuclear matter has great uncertainty mainly because the
terrestrial nuclear experiments cannot reproduce matter as dense as that in
the inner core of a neutron star. Fortunately, continuous improvements in
astronomical observations of neutron stars provide the opportunity to
inversely constrain the EOS of high-density nuclear matter. A number of
methods have been proposed to implement this inverse constraint, such as the
Bayesian analysis algorithm, the Lindblom’s approach, and so on. Neural
network algorithm is an effective new method developed in recent years. By
employing a set of isospin-dependent parametric EOSs as the training sample of
neural network algorithm, we set up an effective way to reconstruct the EOS
with relative accuracy through a few mass-radius data. Based on the obtained
neural network algorithms and according to the NICER observations on masses
and radii of neutron stars with assumed precision, we get the inversely
constrained EOS and further calculate the corresponding macroscopic properties
of the neutron star. The results are basically consistent with the constraint
on EOS from the Huth $et~{}al.$ based on Bayesian analysis. Moreover, the
results show that even though the neural network algorithm was obtained by
using the finite parameterized EOS as the training set, it is valid for any
rational parameter combination of the parameterized EOS model.

###### pacs:

97.60.Jd; 04.40.Dg; 04.30.-w; 95.30.Sf

##  I Introduction

Understanding the nature of neutron stars and determining the equation of
state (EOS) of high-density asymmetric nuclear matter are both long-standing
problems in nuclear physics and astrophysics Chen1 ; Li25 ; Li26 ; Li2019EPJA
. The EOS of asymmetric nuclear matter is a key input for probing the
structure and properties of neutron stars. However, the theoretical
predictions on the EOS at supra-saturation densities diverge broadly. Except
possible phase transitions, density dependence of the symmetry energy is the
most uncertain part of the EOS, especially at supra-saturation densities
Dieperink03PRC ; Li26 ; Li2019EPJA . Although significant progresses have been
made in probing the symmetry energy in terrestrial nuclear laboratories
Russotto2 ; LeFevre3 ; Adhikari4 , the inability to reproduce the authentic
high-density neutron-rich nuclear matter restricts the full and accurate
understanding of symmetry energy and EOS at supra-saturation densities.

Fortunately, many groundbreaking observational discoveries from scientific
facilities in recent years have led to a new upsurge in understanding the
neutron stars and EOS of high-density asymmetric nuclear matter. For example,
the mass-radius measurement on neutron stars from the Neutron Star Interior
Composition Explorer (NICER) and the gravitational radiation detection of
binary neutron star mergers from LIGO and Virgo have taken the understanding
of neutron stars and EOS of the asymmetric nuclear matter to a new level
Miller5 ; Riley6 ; Pang7 ; Abbott8 . In the near future, the Large Observatory
for X-ray Timing (LOFT), the Advanced Telescope for High Energy Astrophysics
(ATHENA), the Square Kilometre Array (SKA) and Einstein Telescope (ET) will
also bring more observations of neutron stars when they become operational
Feroci40 ; Nandra41 ; Punturo42 ; Dewdney43 ; Rezzolla44 . It is believed that
they will provide powerful insights in understanding the neutron stars and the
EOS.

If we regard obtaining the properties of neutron stars from the EOS of nuclear
matter as forward method, which normally can be numerically calculated
according to the EOS and TOV equations for static spherically symmetrical
stars. In combination with the inverse derivation from the mass-radius to EOS
given in Ref. Lindblom9  using enthalpy, there is a one-to-one mapping between
EOS and the mass-radius relation of neutron stars. That is, if a set of
accurate mass-radius observations of neutron stars is obtained, the EOS of
nuclear matter can be reversely achieved. But how can we reversely obtain the
EOS from the properties of neutron stars? At present, there is no reliable and
widely accepted method. The purpose of this work is to try to find an
effective way reversely mapping the EOS of nuclear matter through the mass-
radius relations of neutron stars.

Lindblom had pioneered the inverse TOV mapping to obtain the pressure-density
sequence P($\rho$) of the EOS from the accurate mass-radius sequence of a
neutron star Lindblom9 . The reality is that the accurate observation data of
the mass-radius of the neutron star is still very rare. As more advanced
algorithms are introduced, a small amount of observation data can provide
effective constraint on the EOS. For example, Ref. XuJPRC21  implemented
constraints on EOS-related parameters within the framework of Bayesian
analysis. Bayesian analysis is an algorithm for updating information by
correcting existing information based on observed events. The use of this
algorithm therefore requires ensuring both the correctness of the prior
probabilities and the reasonableness of the observed events. This year, Huth
$et~{}al.$ also used this algorithm to achieve constraints on the EOS by
combining data from heavy-ion collisions (HIC) with results from multi-
messenger astronomical observations Huth11 . Furthermore, a substantial amount
of Bayesian analysis work has been contributed to neutron star research YX12 ;
Silva13 ; Xie14 .

Neural networks (NN) is a branch of machine learning (ML) and a computing
system inspired by biological neural networks Amari17 . It has the advantages
of Bayesian analysis algorithm and polynomial fitting algorithm. Only a few
less precise observations are needed to obtain the effective constraint
through NN, which can also be used to calculate other properties of neutron
stars. In the 1990s, NN was pioneered by Clark and collaborators for research
related to nuclear physics Gazula92 ; Gazula93 ; Gazula95 ; Gazula99 . This
algorithm is now widely used in theoretical prediction work to improve the
accuracy of nuclear mass and nuclear charge radii measurements ZhaoNPA22 ;
Niu18 ; Utama16PRC ; Utama16JPG . In the field of nuclear astrophysics, this
algorithms have also been introduced into neutron star research. For example,
the inverse TOV mapping of NN was constructed by using the EOS with sound
velocity segmentation in Ref. Fujimoto16 , the reconstructed EOS was
implemented with NN at 1 to 7 times the saturated nuclear density
$\rho_{0}\approx 0.16\ \mathrm{fm}^{-3}$ in Ref. Soma22AX , Krastev
constructed NNs from mass-radius and mass-tidal deformations to symmetry
energy Krastev15 , and NN from nuclear parameters to neutron star properties
was constructed in Ref. Thete22AX . Additionally, the mapping from X-ray
spectra to EOS parameters was achieved by using NN in Ref. Farrell22AX .

Although much of the NN work has given exciting results, many questions remain
unsolved. For example, the plausibility of the symmetry energy parameters of
the EOS after noise addition, the implementation of the NN prediction function
and its general applicability, and the validation of the effectiveness. In
this work, we will construct a new NN based on the parametric EOS, which can
be efficiently implemented for any combination of parameters satisfying
terrestrial experiments and multi-messenger astronomical observations with a
small number of mass-radius relations to inverse TOV mapping.

This paper is organized as follows. In Sec. II we briefly review the isospin-
dependent parametric EOS and the basic formula for calculating the properties
of neutron stars. The NN implementation of the inverse TOV mapping is
presented in Sec. III. NN combined with NICER observational constraints are
presented in Sec. IV. The summary and outlook are in Sec. V.

Unless otherwise stated, in the formulae we use the gravitational units
($G=c=1$).

##  II Isospin-dependent parametric EOS and the fundamental formula of neutron
stars

###  II.1 Isospin-dependent parametric EOS

Implementing a NN for inverse TOV mapping requires a large number of the EOS
as training samples. Moreover, these EOS also need to meet the range of
nuclear parameters as much as possible, such as $L$, $K_{\mathrm{sym}}$, etc.
The isospin-dependent parametric EOS is able to satisfy both the need to
generate a large number of EOS and the rationalization of nuclear parameters.

For asymmetric nuclear matter, the energy per nucleon can be expanded by
isospin asymmetry Li25 ; Li26

$E\left(\rho,\delta\right)\approx
E_{0}\left(\rho\right)+E_{\mathrm{sym}}\left(\rho\right)\delta^{2},$ (1)

where
$\delta=\left(\rho_{\mathrm{n}}-\rho_{\mathrm{p}}\right)/\left(\rho_{\mathrm{n}}+\rho_{\mathrm{p}}\right)$
is the isospin asymmetry. The first term on the right side
$E_{0}\left(\rho\right)$ is usually referred to as the energy of symmetric
nuclear matter, and the second term $E_{\mathrm{sym}}\left(\rho\right)$ is
referred to as the symmetry energy, which has the physical meaning of the
energy difference per nucleon between asymmetric nuclear matter and pure
neutron matter.

Usually, the two terms on the right side in Eq. (1) can be expanded at the
saturation density $\rho_{0}$ Zhang18APJ ; Xie14

$\displaystyle
E_{0}\left(\rho\right)=E_{0}\left(\rho_{0}\right)+\frac{K_{0}}{2}\left(\frac{\rho-\rho_{0}}{3\rho_{0}}\right)^{2}+\frac{J_{0}}{6}\left(\frac{\rho-\rho_{0}}{3\rho_{0}}\right)^{3},$
(2.2) $\displaystyle
E_{\mathrm{sym}}\left(\rho\right)=E_{\mathrm{sym}}\left(\rho_{0}\right)+L\left(\frac{\rho-\rho_{0}}{3\rho_{0}}\right)+\frac{K_{\mathrm{sym}}}{2}\left(\frac{\rho-\rho_{0}}{3\rho_{0}}\right)^{2}+\frac{J_{\mathrm{sym}}}{6}\left(\frac{\rho-\rho_{0}}{3\rho_{0}}\right)^{3}.$
(2.3)

Constrained by terrestrial nuclear experiments, the most probable values of
the parameters in the above equations are as follows: $L=58.7\pm 28.1\
\mathrm{MeV}$, $-400\leqslant K_{\mathrm{sym}}\leqslant 100\ \mathrm{MeV}$,
$-200\leqslant J_{\mathrm{sym}}\leqslant 800\ \mathrm{MeV}$, $-300\leqslant
J_{0}\leqslant 400\ \mathrm{MeV}$, $K_{0}=240\pm 20\ \mathrm{MeV}$,
$E_{\mathrm{sym}}\left(\rho_{0}\right)=31.7\pm 3.2\ \mathrm{MeV}$ Shlomo33 ;
Piekarewicz34 ; Zhang35 ; Oertel36 ; Li37 . It is worth pointing out that in
recent years, PREX-II has given higher $L$ values ($L=106\pm 37\ \mathrm{MeV}$
Adhikari4 ). This result is also considered later in this work.

The pressure of the system can be calculated numerically from

$P\left(\rho,\delta\right)=\rho^{2}\frac{d\left(E/\rho\right)}{d\rho}.$ (2.4)

To construct the EOS in the whole density range, the NV EOS model Negele38
and BPS EOS model Baym39  are used for the inner crust and the outer crust,
respectively.

###  II.2 Fundamental formula of statically spherically symmetric neutron
stars

Neutron stars in this work are considered to be isolated, non-rotating, and
statically spherically symmetric. The Tolman-Oppenheimer-Volkoff (TOV)
equations consist of the equation of hydrostatic equilibrium Oppenheimer19 ;
Tolman20

$\frac{dP}{dr}=-\frac{\left[m\left(r\right)+4\pi
r^{3}P\left(r\right)\right]\left[\rho\left(r\right)+P\left(r\right)\right]}{r\left[r-2m\left(r\right)\right]},$
(2.5)

and

$\frac{dm\left(r\right)}{dr}=4\pi r^{2}\rho\left(r\right).$ (2.6)

The external boundary condition of the star is as follows: $\rho=p=0$. Given
the center density of the neutron star, it can be solved layer by layer from
the interior to the exterior of the star using the method of solving high-
precision initial value problems.

Tidal deformation of neutron star is an EOS-dependent property which can be
measured through the gravitational wave events such as GW170817 Abbott18PRL .
To linear order, the tidal deformation $\varLambda$ is defined as Flanagan22

$Q_{ij}=-\varLambda\varepsilon_{ij},$ (2.7)

where $Q_{\mathrm{ij}}$ and $\varepsilon_{\mathrm{ij}}$ represent the induced
quadrupole moment and the static tidal field, respectively. The second tidal
Love number $k_{2}$ is defined as

$k_{2}=\frac{3}{2}\varLambda R^{-5},$ (2.8)

which can be solved together with the TOV equation Chamel23 ; Chamel24 .

##  III Neural network implementation of inverse TOV mapping

To clearly demonstrate the process of implementing an NN for inverse TOV
mapping, we draw the flow chart as shown in Fig. 1. The first step requires
the provision of a relatively complete training set. After determining the
range of symmetry energy parameters, the massive EOS (output of NN) is
generated by the method in Section II.1, while the corresponding mass-radius
points (input of NN) are calculated by the TOV equation. The training set of
NN is filtered by astronomical observations. In the second step, the training
set is substituted into the initialized NN training to examine the loss.
Adjust the key parameters of NN until the loss converges. The final step is to
find the minimum number of neurons at the input, which means comparing the
relative errors under different mass-radius point conditions. The different
mass-radius points come from the non-training sample, which is the EOS in the
non-training set and the corresponding mass-radius (consistent with the
parameter range and filtering in the first step).

ptStartSelect the range of parameters (such as
$L-K_{sym}-K_{0}-J_{0}$)Calculation of EOS with parametric EOSCalculation of
M-R using TOV equationFiltering by astronomical observationsInitialized NNLoss
convergence?Adjusting the parameters of NNReduction of neurons at the
inputComparative Relative ErrorEndNon-training set samplesNY Figure 1:
Flowchart of inverse TOV mapping neural network implementation

The most significant prerequisite for the implementation of this algorithm is
the training sample processing. Four groups of variable parameters are used to
generate training samples to cover as large EOS range as possible. The
variable parameter space is as follows: $30\leqslant L\leqslant 143\
\mathrm{MeV}$, $-400\leqslant K_{\mathrm{sym}}\leqslant 100\ \mathrm{MeV}$,
$-200\leqslant J_{\mathrm{sym}}\leqslant 800\ \mathrm{MeV}$, $-300\leqslant
J_{0}\leqslant 400\ \mathrm{MeV}$, $K_{0}=240\ \mathrm{MeV}$,
$E_{\mathrm{sym}}\left(\rho_{0}\right)=31.7\ \mathrm{MeV}$ Shlomo33 ;
Piekarewicz34 ; Zhang35 ; Oertel36 ; Li37 . It is especially worth noting that
the slope $L$ range was developed by combining terrestrial Adhikari4  and
astrophysical data. To obtain preliminary training samples, we generate EOS
(Section II.1) by taking points at equal intervals in the above parameter
interval, then solve the corresponding M-R with the TOV equation. Moreover, in
order to make the output more reasonable, we used astronomical observation
results during the sample generation stage: $M_{\max}\geqslant 2.14\
M_{\odot}$ Cromartie20NatAs , $\varLambda_{1.4}=190_{-120}^{+390}$ Abbott18PRL
, $1.44\pm 0.15\ M_{\odot}$ Miller5  corresponding to $\left[11.2,\
13.3\right]\ \mathrm{km}$, $2.08\pm 0.07\ M_{\odot}$ Riley6  corresponding to
$\left[12.2,\ 16.3\right]\ \mathrm{km}$. Similarly to Ref. Fujimoto_JHEP2021
considering the causality condition ($c_{s}<c$) and the stability condition
($dp/d\rho>0$), if the maximum mass filter is then added, the sample size is
comparable to that after the screening of astronomical observations as
described above, with a limited improvement in the generalization ability. In
the sample processing stage, due to the randomness of observations, we
randomly sampled the mass-radius sequences over the whole mass range. Note
that because it is more difficult to form low-mass neutron stars, as in Ref.
Fujimoto_JHEP2021  the sampling point is set to be greater than 1 $M_{\odot}$.
But, considering the theoretical lower mass limit of neutron stars that can
reach 0.1 $M_{\odot}$ Haensel_AA2002 , as well as recent observations of the
HESS J1731-347 Doroshenko_NA2022 , a wider mass sampling interval was adopted.
The final sample size involved in the NN is approximately 880,000, where M-R
points are used as input and P-$\rho$ points are used as output.

Table 1: Architecture of neural network Layer |  Number of neurons |  Activation function
---|---|---
0(Input) |  80 |  N/A
1 |  100 |  ReLU
2 |  300 |  ReLU
3 |  400 |  ReLU
4 |  400 |  ReLU
5 |  400 |  ReLU
6 |  300 |  ReLU
7(Output) |  182 |  tanh

The NN was initialized with reference to the framework of Ref. Fujimoto16  and
Krastev15  and according to the background of our training set. By adjusting
the relevant parameters, such as the number of layers, the optimization
function, the epoch size, etc., the loss convergence of the NN is achieved. We
finalized the NN architecture as shown in Table 1. The 80 neurons in the input
layer means that the NN can provide accurate predictions under the condition
of 40 pairs of (M,R) points. The other key parameters for this NN are shown in
Appendix A and the predicted results for the different input points are shown
in Appendix B.

To verify the accuracy, we used several mass-radius sequences of the EOS to
substitute into this NN for prediction. In particular, we use APR3 and APR4 to
test the effectiveness of this NN for non-parameter EOS.

Fig. 2 shows the NN predictions for three adopted EOS. It is shown that our NN
method achieves a relatively accurate prediction function with approximately
40 mass-radius points as input. The solid black line for the (a) panel
equation of state is randomly generated and substituted into Eq.(2.5) to give
the solid black line for the mass-radius sequence. Additionally, We also test
some EOS other than the parameterized EOS and find that some of them can also
be predicted well, as shown in (b) and (c) in Fig. 2 for APR3 and APR4 EOS.
This result means that our NN method will no longer be limited to parametric
EOS even though we use only parametric EOS as training samples.

Figure 2: The EOS predictions of NN from 40 sampled points for the
$L-J_{0}-J_{\mathrm{sym}}-K_{\mathrm{sym}}$ panel, where the solid black line
represents the original EOS, and the blue dots denote the NN predictions. The
three sample EOSs are, (a) parametric EOS with randomly generated parameters
as $L=40.5\ \mathrm{MeV}$, $K_{\mathrm{sym}}=-200.5\ \mathrm{MeV}$,
$J_{\mathrm{sym}}=320\ \mathrm{MeV}$, $J_{0}=110\ \mathrm{MeV}$, $K_{0}=240\
\mathrm{MeV}$, $E_{\mathrm{sym}}\left(\rho_{0}\right)=31.7\ \mathrm{MeV}$, (b)
APR3 EOS and (c) APR4 EOS. The inset shows the NN sampling from the mass-
radius sequence. The lower panels are the relative error of the fitting.

For comparison with the results in Ref. Krastev15 , we also probe the output
of NN by setting the parameters of EOS in the $L-K_{\mathrm{sym}}$ panel,
where $J_{0}=J_{\mathrm{sym}}=0$, and $L$ and $K_{\mathrm{sym}}$ take a range
of $\left[30,\ 90\right]\ \mathrm{MeV}$ and $\left[-400,\ 100\right]\
\mathrm{MeV}$, respectively. Using the same sample generation process as for
the $L-J_{0}-J_{\mathrm{sym}}-K_{\mathrm{sym}}$ panel, the sample size at this
point is approximately 8,000. Fig. 3 shows the NN predictions for the randomly
selected test EOS with the parameters limited in the previously mentioned
range. We try to use as few input samples as possible to achieve the
prediction function. Here the prediction function for the full density segment
is implemented in the NN with 4 rows of mass-radius sequences as input (the
data of the input is shown in Table 2). Since the low-dimensional parameter
space is simpler compared to the high-dimensional, e.g., the same (M, R) point
corresponds to fewer EOS, only four output points and some parameters of the
NN in the simplified Table 1 are needed to implement the function. The above
results show that our NN design is more suitable for inverse TOV mapping under
isospin-dependent parametric EOS conditions.

Figure 3: The EOS predictions of NN from 4 sampled points for the $L-K_{\mathrm{sym}}$ panel (The EOS parameters are as follows: $L=40.5\ \mathrm{MeV}$, $K_{\mathrm{sym}}=80.5\ \mathrm{MeV}$, $J_{\mathrm{sym}}=0\ \mathrm{MeV}$, $J_{0}=0\ \mathrm{MeV}$, $K_{0}=240\ \mathrm{MeV}$, $E_{\mathrm{sym}}\left(\rho_{0}\right)=31.7\ \mathrm{MeV}$). The solid black line is the original EOS from which the sample points shown as the red dots were generated. The blue dots are the EOS data output by NN. The inset shows the NN sampling from the mass-radius sequence (Greater than 1 $M_{\odot}$). Table 2: Mass-radius sampling points for Fig. 3 Mass ($M_{\odot}$) |  Radius (km)
---|---
1.650030 |  11.8838
1.916370 |  11.6289
2.046519 |  11.3741
2.201511 |  10.3368

Compared to Fig. 2, Fig. 3 demonstrates a greater accuracy of the neural
network predictions after reducing the two parameter dimensions. The fact that
other different kinds of the EOS can be valid suggests that this NN prediction
has certain universality.

##  IV Combining NICER observations to verify the validity of NN

Figure 4: NN predictions combining the most probable points of the two NICER
observations. (a) The red curve is NN’s predicted EOS result, and the blue
area presents the constraint of EOS from Ref. Huth11  in 95% (light) and 68%
(dark) credible intervals. (b) The red curve represents the dimensionless
tidal deformation and stellar mass calculated using the EOS from (a). The
constraint on the $\Lambda$ of canonical neutron star from GW170817 Abbott8
is also presented.

To validate the predictions of the inverse TOV mapping-type NNs, prediction
EOS from NICER observations are shown and discussed in this section within the
parametric EOS framework as well as adopting the results of Refs. Abbott8  and
Huth11 . Furthermore, due to the insufficient number of observations at the
neutron star mass-radius, the following approach was adopted to satisfy the NN
input points: 1. The EOS is generated in batch with parametric EOS within the
range of nuclear parameters ($L$, $K_{s}$, $J_{s}$, $J_{0}$) mentioned in Sec.
III; 2. Take one EOS that fits near the most probable point of the two NICER
observations ($R_{1.44M\odot}=13.0\ km$ Miller21APJ , $R_{2.08M\odot}=13.7\
km$ Riley19APJ ); 3. Forty points were randomly taken on the M-R curve
corresponding to this EOS. However, for the real situation, 40 precise (M,R)
observations from different neutron stars are necessary for NN input. It is
expected that such accurate and sufficient observations will be available in
the near future.

The NN-based predicted EOS together with further calculated dimensionless
tidal deformations as a function of stellar mass are displayed in Fig. 4(a).
To demonstrate the effect of NN, we also plot the constraint on EOS from Huth
$et~{}al.$ Huth11  and $\Lambda$ of canonical neutron star ($M=1.4M_{\odot}$)
from GW170817 Abbott8 . It is shown that the predictions obtained under the
assumption of observational accuracy prefer relatively stiff EOS at high
density, which is basically in agreement with the results of Ref. Huth11 .
Further calculation of the predicted EOS shows that the canonical neutron star
has a dimensionless tidal deformation of $\varLambda_{1.4}\approx 507$,
indicating a high tidal deformability. From Fig. 4(b), we can see that when
the stellar mass is less than 1.6 $M_{\odot}$, the tidal deformability
decreases relatively quickly as the mass increases, when the mass is higher
than 1.6 $M_{\odot}$, the tidal deformability decreases relatively slowly with
the increase of mass. For massive neutron stars with $M=2.0M_{\odot}$, its
tidal deformation is $\varLambda_{2.0}\approx 51$, which is far smaller than
that of canonical neutron stars.

##  V Summary and Outlook

The NN algorithms to implement the inverse mapping TOV equation is constructed
by employing the parametric EOS as the training sample. Through the practical
application of the obtained neural network algorithm, we get the following
results.

  * (i)

We implemented the inverse constraint EOS of NN in 4-dimensional parameter
space ($L-J_{0}-J_{\mathrm{sym}}-K_{\mathrm{sym}}$) and 2-dimensional
parameter space ($L-K_{\mathrm{sym}}$), respectively. About 40 mass-radius
points are required to output higher precision EOS under the condition that
$L-J_{0}-J_{\mathrm{sym}}-K_{\mathrm{sym}}$ is a variable. Four mass-radius
points are required to output a higher precision EOS with $L-K_{\mathrm{sym}}$
as the variable.

  * (ii)

The EOS based on the obtained NN algorithm and the two sets of NICER
observational values with assumed precision is predicted. Similar to the
results from other method, the EOS constrained by the NICER observation and NN
is relatively stiff, and the corresponding tidal deformability of canonical
neutron star is relatively high. Our NN predictions are basically consistent
with the results of Huth $et~{}al.$ Huth11 .

Despite the generalisation ability of the algorithm we provide, the output of
the neural network is difficult to ensure credible results when the EOS is
other than the isospin-dependent parametric EOS. To fill this gap, more types
of EOSs need to be added to the training samples in the future. We believe
that by combining precise observations from multiple sources, NN will be a
promising tool for achieving more precise constraints on the EOS of nuclear
matter.

###### Acknowledgements.

This work is supported by NSFC (Grants No. 12375144 and 11975101), Guangdong
Natural Science Foundation (Grants No. 2022A1515011552 and 2020A151501820).

##  Appendix A Neural network implementation

NN, one of the common algorithms in ML, is now used in almost every aspects of
scientific research and engineering Wright28 . Based on the universal
approximation theorem, NN can implement complex nonlinear mappings Hornik29 ;
Leshno30 . Its construction is mainly divided into an input layer, a hidden
layer and an output layer. Each neuron can be treated as a container of
numbers. Similar to the transmission of electrical signals between synapses,
neurons are using numbers as signals between them. Starting from the input
layer, each layer of neurons goes through the following process between each
other

$z^{\mathrm{l}}=w_{1}a_{2}^{\mathrm{l-1}}+\cdots+w_{\mathrm{n}}a_{\mathrm{n}}^{\mathrm{l-1}}+b.$
(2)

where $w$ represents the weight, $l$ represents the serial number of layers,
and $b$ represents the bias value Goodfellow . The calculated $z$ is then
substituted into the activation function to obtain the value of the output of
the neuron to the next layer:

$a_{\mathrm{j}}^{\mathrm{l}}=f\left(z^{\mathrm{l}}\right),$ (3)

where $f(x)$ is called the activation function and is usually used as Sigmoid
or ReLU, the latter being adopted in this work. When summarizing the
operations of the whole layer of neurons, we can obtain the following
expression

$f\left(\left[\begin{matrix}w_{0,0}&w_{0,1}&\cdots&w_{0,\mathrm{n}}\\\
w_{1,0}&w_{1,1}&\cdots&w_{1,\mathrm{n}}\\\ \vdots&\vdots&\ddots&\vdots\\\
w_{\mathrm{k},0}&w_{\mathrm{k},1}&\cdots&w_{\mathrm{k},\mathrm{n}}\\\
\end{matrix}\right]\left[\begin{array}[]{c}a_{0}^{0}\\\ a_{1}^{0}\\\ \vdots\\\
a_{\mathrm{n}}^{0}\\\ \end{array}\right]+\left[\begin{array}[]{c}b_{0}\\\
b_{1}\\\ \vdots\\\ b_{\mathrm{n}}\\\ \end{array}\right]\right)=output.$ (4)

At this point, the NN has completed one forward propagation. We also need a
metric to evaluate how well the output compares to the true value. This type
of evaluation metric is called the loss function, in which we usually use the
mean squared error
($MSE=\frac{1}{\mathrm{n}}\sum_{\mathrm{i}=1}^{\mathrm{n}}{\left(Y_{\mathrm{i}}-\widehat{Y_{\mathrm{i}}}\right)^{2}}$)
method . The MSE of each batch is equivalent to the construction drawing,
guiding the optimizer to continuously adjust a huge number of weights ($w$)
and biases ($b$) in the NN. The available optimizers are Stochastic Gradient
Descent (SGD) and Adam Adam , the latter being adopted in this paper.

The platform for our implementation of NN is Keras, with Tensorflow as its
backend Keras ; Tensorflow . It integrates the excellent features of Compute
Unified Device Architecture (CUDA) for parallel computing on GPU with
Tensorflow as the backend thus providing a rich interface. The training of the
NN was performed on NVIDIA GeForce GTX 1650. It can significantly save the
time cost of computing while implementing complex NN. To prevent overfitting,
Dropout is set to 0.3 between layers. Furthermore, we chose an initial
learning rate of 0.0003, a batch size of 500, and an epoch of 800. The
proportion of the training dataset used for the validation dataset was 0.1.
The hidden layer is used as a fully connected layer. Based on the above
conditions, we have designed six layers of NN to achieve inverse TOV mapping.
All the data in the NN need to be normalized, which in this paper is used as
(M$/$3, R$/$35) and (($\log_{10}p$)$/$40, $(\log_{10}\rho)/{20}$). The EOS is
taken logarithmic in order to avoid too large a gap in pressure magnitude
between different density intervals affecting the prediction accuracy of the
NN. For ease of calculation both P and $\rho$ are taken as logarithmic results
in $MeV\cdot fm^{-3}$ and $kg\cdot m^{-3}$ respectively.

##  Appendix B Results for different input points

We gradually reduce the input points after achieving loss convergence, while
improving the relevant parameters of NN. Within the conventional NN framework,
most of the improvements to achieve convergence of the NN can only affect the
convergence rate of the loss cannot affect the minimum number of input points,
such as increasing the number of hidden layers. As the number of neurons in
the input layer continues to decrease, a significant error occurs in the
$L-J_{0}-J_{\mathrm{sym}}-K_{\mathrm{sym}}$ panel for 35 pairs of (M,R) (see
Fig. 5), while a significant error occurs in the $L-K_{\mathrm{sym}}$ panel
for 3 pairs of (M,R) (see Fig. 6). It is important to note that over-
concentration of (M,R) sampling points affects the NN predictions. Such
sampling is a small probability event in astronomical observations.

Figure 5: Similar to Fig 2 (a), but for 35 sampled points. Figure 6: Similar
to Fig. 3, but for 3 sampled points.

Data Availability Statement: The main code used in this work can be found in
the Github repository https://github.com/zhscut/NN-for-Neutron-Star.

## References

  * (1) L. W. Chen, C. M. Ko and B. A. Li, Phys. Rev. Lett. 94, 032701. (2005).
  * (2) B. A. Li, X. Han, Phys. Lett. B 727, 276-281 (2013).
  * (3) B. A. Li, L. W. Chen and C. M. Ko, Phys. Rep. 464, 113-281 (2008).
  * (4) B. A. Li, P. G. Krastev, D. H. Wen, and N. B. Zhang, (2019), arXiv:1905.13175.
  * (5) A.E.L. Dieperink, Y. Dewulf, D. Van Neck, M. Waroquier, V. Rodin, Phys. Rev. C 68 064307\. (2003).
  * (6) P. Russotto, S. Gannon, and S. Kupny, $et$ $al$., Phys. Rev. C 94, 034608 (2016).
  * (7) A. Le Fevre, Y. Leifels, $et$ $al$., Nucl. Phys. A 945, 112-133 (2016).
  * (8) D. Adhikari, H. Albataineh, $et$ $al$., Phys. Rev. Lett. 126, 172502. (2021).
  * (9) M. C. Miller $et$ $al$., Astrophys J. Lett. 918, L28 (2021).
  * (10) T. E. Riley $et$ $al$., Astrophysical J. Lett. 918, L27 (2021).
  * (11) P. T. Pang, I. Tews, and M. W. Coughlin $et$ $al$., Astrophysical. J. 922, 14 (2021).
  * (12) B. P. Abbott $et$ $al$., Phys. Rev. lett. 121, 161101 (2018).
  * (13) M. Feroci, L. Stella, $et$ $al$., Experimental Astronomy 34, 415-444 (2012).
  * (14) http://www. the-athena-x-ray-observatory. eu.
  * (15) M. Punturo, M. Abernathy, $et$ $al$., Classical and Quantum Gravity 27, 194002 (2010).
  * (16) P. E. Dewdney, P. J. Hall, $et$ $al$., Proceedings of the IEEE 97, 1482-1496 (2009).
  * (17) L. Rezzolla, P. Pizzochero, $et$ $al$., Springer International Publishing.(2018).
  * (18) L. Lindblom, Astrophysical. J. 398, 569-573 (1992).
  * (19) Y. Fujimoto, K. Fukushima, and K. Murase, Phys. Rev. D 98, 023019 (2018).
  * (20) J. Xu, Z. Zhang, B. A. Li, Phys. Rev. C 104(5), 054324 (2021).
  * (21) S .Huth, P. T. Pang, $et$ $al$., Nature 606, 276-280 (2022).
  * (22) Y. Li, H. Chen, D. Wen, $et$ $al$., Eur. Phys. J. A 57 1-10 (2021).
  * (23) H. O. Silva, A. M. Holgado, A. Cardenas-Avendano and N. Yunes, Phys. Rev. Lett. 126, 181101 (2021).
  * (24) W. J. Xie, B. A. Li, Astrophysical J. 883, 174 (2019).
  * (25) S. I. Amari, M. A. Arbib, Systems neuroscience, 119-165 (1977).
  * (26) S. Gazula, J. Clark, and H. Bohr, Nucl. Phys. A 540, 1 (1992).
  * (27) K. Gernoth, J. Clark, J. Prater, and H. Bohr, Phys. Lett. B 300, 1 (1993).
  * (28) K. Gernoth and J. Clark, Neural Networks 8, 291 (1995).
  * (29) J. W. Clark, T. Lindenau, and M. Ristig, Springer Lecture Notes in Physics, Vol. 522 (Springer-Verlag, Berlin, 1999).
  * (30) T. Zhao, H. Zhang, Nucl. Phys. A 1021, 122420 (2022).
  * (31) Z.M. Niu, H.Z. Liang, Phys. Lett. B 778 48\. (2018).
  * (32) R. Utama, J. Piekarewicz, H.B. Prosper, Phys. Rev. C 93 014311 (2016).
  * (33) R. Utama, W.C. Chen, J. Piekarewicz, J. Phys. G, Nucl. Part. Phys. 43 114002 (2016).
  * (34) S. Soma, L. Wang, S. Shi, $et$ $al$., arXiv preprint arXiv:2201.01756. (2022).
  * (35) P. G. Krastev, Galaxies 10, 16 (2022).
  * (36) A. Thete, K. Banerjee, T. Malik, arXiv:2208.13163. (2022).
  * (37) D. Farrell, P. Baldi, J. Ott, A. Ghosh, $et$ $al$., arXiv:2209.02817. (2022).
  * (38) N. B. Zhang, B. A. Li, J. Xu, Astrophysical. J. 859(2), 90. (2018).
  * (39) N. B. Zhang, B. J. Cai, B. A. Li, W. G. Newton, and J. Xu, Nucl. Sci. Technol. 28, 181 (2017).
  * (40) S. Shlomo, V. M. Kolomietz, G. Colo, Eur. Phys. J. A, 30(1), 23-30 (2006).
  * (41) J. Piekarewicz. Journal of Physics G Nuclear Physics. 37, 064038 (2010).
  * (42) M. Oertel, M. Hempel, T. Klähn, and S. Typel, Rev. Mod. Phys. 89, 015007 (2017).
  * (43) B. A. Li, Nuclear Physics News, 27, 7 (2017).
  * (44) J. W. Negele, D. Vautherin, Nucl. Phys. A 207, 298-320 (1973).
  * (45) G. Baym, C. Pethick, P. Sutherland, Astrophysical. J. 170, 299. (1971).
  * (46) J. R. Oppenheimer, G. M. Volkoff, Phys. Rev. 55, 374 (1939).
  * (47) R. C. Tolman, Phys. Rev. 55, 364 (1939).
  * (48) B. P. Abbott, R. Abbott, T. D. Abbott, $et$ $al$., Phys. Rev. Lett. 121, 161101. (2018).
  * (49) E. E. Flanagan, T. Hinderer, Phys. Rev. D 77, 021502(R) (2008).
  * (50) T. Hinderer, Astrophysical. J. 677, 1216.(2008).
  * (51) T. Hinderer, B. D. Lackey, R. N. Lang, J. S. Read, Phys. Rev. D 81, 123016 (2010).
  * (52) H. T. Cromartie, E. Fonseca, S.M Ransom, $et$ $al$., Nat. Astron. 4: 72-76. (2020).
  * (53) Y. Fujimoto, M. Nitta, Journal of High Energy Physics, 2021(9), 1-36. (2021).
  * (54) P. Haensel, J. L. Zdunik, F. Douchin, Astron. Astrophys. 385, 301-307. (2002).
  * (55) V. Doroshenko, $et$ $al$., Nat. Astron., 6(12), 1444-1451. (2022).
  * (56) M. C. Miller, F. K. Lamb, A. J. Dittmann, $et$ $al$., Astrophys. J. Lett. 887, L24 (2019).
  * (57) T. E. Riley, A. L. Watts, S. Bogdanov, $et$ $al$., Astrophys. J. Lett. 887, L21 (2019).
  * (58) L. G. Wright, T. Onodera, and M. M. Stein, $et$ $al$., Nature, 601, 549-555 (2022).
  * (59) K. Hornik, M. Stinchcombe, and H. White, Neural networks, 3, 551-560. 171 (1990).
  * (60) M. Leshno, V. Y. Lin, $et$ $al$., Neural Networks, 6, 861-867.(1993).
  * (61) I. Goodfellow, Y. Bengio, A. Courville Deep learning. (MIT press), (2016).
  * (62) D. P. Kingma,J. Ba, Adam: A method for stochastic optimization. arXiv:1412.6980, (2014).
  * (63) F. Chollet, "Keras: Deep learning library for theano and tensorflow," https://github.com/fchollet/keras (2015).
  * (64) M. Abadi, A. Agarwal, P. Barham, $et$ $al$., Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv:1603.04467, (2016).