:

# Scope and Arbitration in Machine Learning Clinical EEG Classification

Yixuan Zhu<EMAIL_ADDRESS>
University of the West of England  UK  Luke J. W. Canham
<EMAIL_ADDRESS>
North Bristol NHS Trust  UK  David Western<EMAIL_ADDRESS>
University of the West of England  UK

###### Abstract

A key task in clinical EEG interpretation is to classify a recording or
session as normal or abnormal. In machine learning approaches to this task,
recordings are typically divided into shorter windows for practical reasons,
and these windows inherit the label of their parent recording. We hypothesised
that window labels derived in this manner can be misleading – for example,
windows without evident abnormalities can be labelled ‘abnormal’ – disrupting
the learning process and degrading performance. We explored two separable
approaches to mitigate this problem: increasing the window length and
introducing a second-stage model to arbitrate between the window-specific
predictions within a recording. Evaluating these methods on the Temple
University Hospital Abnormal EEG Corpus, we significantly improved state-of-
the-art average accuracy from 89.8 percent to 93.3 percent. This result defies
previous estimates of the upper limit for performance on this dataset and
represents a major step towards clinical translation of machine learning
approaches to this problem.

##### Data and Code Availability

Our study includes electroencephalography (EEG) datasets collected from
https://isip.piconepress.com/projects/tuh_eeg/. Our code is shared on
https://github.com/zhuyixuan1997/EEGScopeAndArbitration.

##  1 Introduction

###  1.1 Background

Electroencephalography (EEG) recordings are used for the diagnosis and
monitoring of a wide range of neurological conditions. Classification of EEG
recordings as normal or abnormal is an essential task in their clinical
interpretation. Substantial research has been conducted on the application of
machine learning to this task (Schirrmeister et al., 2017; Amin et al., 2019;
Banville et al., 2021, 2022; Gemein et al., 2020; Muhammad et al., 2020; Wagh
and Varatharajah, 2020; Alhussein et al., 2019; Roy et al., 2019).

Recent work in this field largely makes use of the Temple University Hospital
Abnormal EEG Corpus (TUAB) (López et al., 2017) for training and evaluation.
TUAB is a labelled subset of the Temple University Hospital EEG Corpus (TUEG)
(Obeid and Picone, 2016).

Since the presentation of the Deep4 convolutional neural network in 2017
(Schirrmeister et al., 2017) there have been only modest improvements in the
accuracy of machine learning approaches to this problem, as measured on TUAB:
from 85.4 percent (Deep4) up to 89.8 percent (Muhammad et al., 2020) – see
tab:models for further detail. Gemein et al. (2020) proposed that there may be
an upper limit of around 90 percent accuracy in this task, based on known
values of inter-rater agreement between human experts in conventional clinical
practice.

tab:models

Table 1: Summary of state-of-the-art performance metrics for different models
applied to abnormal EEG classification

Model | Accuracy | Sensitivity | Specificity
---|---|---|---
1D-CNN (T5-O1 channel)(Yıldırım et al., 2020) | 79.3 % | 71.4 % | 86.0 %
1D-CNN (F4-C4 channel)(Yıldırım et al., 2020) | 74.4 % | 55.6 % | 90.7 %
Deep4 (Schirrmeister et al., 2017) | 85.4 % | 75.1 % | 94.1 %
TCN (Gemein et al., 2020) | 86.2 % |  |
ChronoNet (Roy et al., 2019) | 86.6 % |  |
Alexnet(Amin et al., 2019) | 87.3 % | 78.6 % | 94.7 %
VGG-16 (Amin et al., 2019) | 86.6 % | 77.8 % | 94.0 %
Fusion Alexnet(Alhussein et al., 2019) | 89.1 % | 80.2 % | 96.7 %
(Muhammad et al., 2020) | 89.8 % | 81.3 % | 96.9 %
Proposed | 93.3 % | 92.0 % | 92.9 %

A notable but little-discussed difference between conventional clinical
practice and virtually all deep learning approaches is that in clinical
practice, the label of normal/abnormal is applied to a full EEG session (i.e.
a single clinical visit). In clinical practice, experts judge whether the
patient exhibits abnormal brain activity based on all the recordings in the
session, effectively resulting in a single label for that session. In most
recent machine learning approaches, a typical full recording cannot be
directly input into the model due to computational constraints – a large input
vector length necessitates a large number of parameters in the model. Instead,
the recording is divided into smaller windows, with the added advantage of
increasing the total number of examples available for training. For training
purposes, each window inherits the label of its recording, while evaluation is
typically performed on a per-recording basis by aggregating per-window outputs
from the classifier. We refer to this downstream aggregation as ‘arbitration’.

Western et al. (2021) noted that this inheritance of window/recording labels
from broader session labels was potentially confounding to the machine
learning process. For example, a session may be labelled as ‘abnormal’ based
on several temporally isolated abnormal graphoelements. Many windows in this
session may be completely free of abnormal activity, yet they will carry
‘abnormal’ labels in the training process. These labels are arguably false,
depending on whether they are considered to apply to the signal within the
window or to the wider session from which it is taken.

###  1.2 Proposal

####  1.2.1 Overview

An intuitive approach to address this problem of unrepresentative window
labels is to expand the model to accept full (and perhaps multiple) recordings
from a session, instead of individual windows, as the input. However,
implementation of this approach is inhibited by notable practical challenges,
including the reduced number of available samples (reducing the learning
opportunity) and the increase in model complexity and associated computational
expense. In the present study, we propose and evaluate two approaches to the
problem: increasing the window length or introducing a second-stage machine
learning model for arbitration.

####  1.2.2 Increasing the Window Length

Increasing the window length can be considered to increase the accuracy of the
labels applied to those windows. For example, it increases the probability of
abnormal features being included within any given window from an abnormal
session. It also increases the scope of the deep learning model, enabling it
to use more information to make decisions. Hence we established the following
hypothesis. However, it could also have negative effects. When the windows
become longer, the number of windows available as training examples will
decrease and the number of parameters in the model will increase, making the
model more difficult to train. Nonetheless, given the potential significance
of temporally isolated abnormalities for the problem of clinical EEG
classification, we hypothesised that the net effect of increased window length
would be positive, as follows.

######  Conjecture 1.1 (Increased Window Length).

Increasing the window length of a deep learning clinical EEG classifier will
increase its accuracy and, more specifically, its sensitivity to abnormal
cases.

####  1.2.3 Second-Stage Model for Arbitration

All recent state-of-the-art applications of deep learning to this problem make
use of some form of windowing. Hence their final stage must be some form of
arbitration to combine the outputs from each window into a single label for
that recording, as depicted in fig:EEG_classification. One notable variation
is the approach of Alhussein et al. (2019), in which higher-dimensional
features are fused across multiple windows in a multi-layer perceptron
producing a per-recording classification, rather than applying arbitration to
downstream class probabilities. They demonstrate notable benefits from this
approach, but it can be assumed that the required increase in model parameters
introduces new challenges to the training process. Otherwise, the arbitration
stage receives little attention in prior literature in this area, implying
that it has not been considered to be important to performance. Schirrmeister
et al. (2017) explored the effects of total recording duration, but maintained
a fixed window length of 60 s. The arbitration function is not mentioned in
that paper, but can be seen in the source code to be the mean of the per-
window outputs (SoftMax-estimated probabilities of ‘normal’ and ‘abnormal’).
The per-recording class is then determined based on whichever mean probability
is greater, ‘normal’ or ‘abnormal’ (argmax). This use of the ‘Mean’ to
arbitrate between multiple windows differs substantially from conventional
clinical practice, in which the presence of any temporally isolated
abnormality may be sufficient to warrant labelling the recording as abnormal
if the rest of the recording appears normal.

Of course the confidence of a human expert or artificial intelligence in
whether any given window contains an abnormality should never be absolute. It
can instead be expressed as some fractional probability, $0<p<1$. Hence, if
class probabilities have been determined on a per-window basis then an
arbitration algorithm must be used. Two categories of clear-cut cases may
exist:

clearly abnormal

one or more windows has a very high abnormality score
($p_{abnormal}>t_{upper}$)

clearly normal

all windows have very low abnormality scores ($p_{abnormal}<t_{lower}$)

Even in these cases, suitable decision thresholds $t_{lower}$ and $t_{upper}$
must be identified. In all other cases, more complex decision-making is
required. However, given the dimensionality reduction achieved by the first
stage model, we hypothesised that this arbitration model would not need to be
as complex as the multilayer perceptron used by Alhussein et al. (2019) for
fusion of higher-dimensional features across windows. Furthermore, it could be
trained separately to minimise the computational and practical expense of
iterative development, while avoiding the need to reduce the number of
training samples available to the first-stage deep learning model. Thus
thorough optimisation of the full model would be achieved more easily.

######  Conjecture 1.2 (Arbitration).

For a deep learning clinical EEG classifier with windowing, the use of a
shallow neural network as the arbitration stage will improve accuracy compared
with conventional ‘Mean’ arbitration or ‘No arbitration’.

fig:EEG_classification

Figure 1: Generic diagram of a typical deep learning approach to clinical EEG
classification, as used e.g. by Schirrmeister et al. (2017)

##  2 Method

###  2.1 Data

TUEG (Obeid and Picone, 2016) is a rich archive of over 30,000 clinical EEG
recordings collected at Temple University Hospital (TUH) from 2002 – present,
using the standard 10-20 system of electrode placement. TUAB (López et al.,
2017) is a subset of 2993 recordings from TUEG that have been labelled as
normal/abnormal and divided into training and evaluation sets. The training
set contains 1371 normal sessions and 1346 abnormal sessions. The test set
contains 150 normal sessions and 126 abnormal sessions. Also, only one file
from each session was included in this corpus. All our current results are
trained and tested on TUAB. Due to computational cost, all experiments related
to window length are carried out on TUAB, including one-stage models, two-
stage models and window length. TUAB has already been marked, so we use the
original label and its original test-training split. To compare the results
with other studies, we followed the pre-processing method in the Deep4
article, which TCN (Gemein et al., 2020) and Fusion Alexnet (Alhussein et al.,
2019) also used.

###  2.2 First-Stage Model

A recent study by Gemein et al. (2020) demonstrated that the Temporal
Convolutional Network (TCN) and Deep4 architectures offer near-state-of-the-
art performance on TUAB, so we experimented with both of these as the first-
stage model. Both are composed of blocks with convolutional and pooling
layers. However, TCN replaces common convolution with dilated convolution and
introduces a residual structure in the temporal block (Bai et al., 2018). To
achieve baseline performance consistent with past studies, we use the
hyperparameters from Schirrmeister et al. (2017) for Deep4 and from Gemein et
al. (2020) for TCN.

To explore whether our proposals are applicable to first-stage architectures
other than convolutional networks, we also implemented a Vision Transformer
(ViT) (Dosovitskiy et al., 2020). For simplicity, the majority of our
experiments focussed on Deep4 only. We use Deep4 with the 60s, 180s, 300s,
400s, and 600s windows to perform reproducibility experiments on TUAB. For
each Window length, five experiments were performed to avoid the influence of
chance factors.

###  2.3 Second-Stage Models for Arbitration

As stated in sec: pro_arb, the purpose of the arbitration stage is to combine
the per-window class probabilities into a single classification of the EEG
session. Previous work does not discuss arbitration, although some form of
arbitration is inevitable where models are for trained on windows and
evaluated on a per-recording basis (e.g. (Schirrmeister et al., 2017; Gemein
et al., 2020)). When looking through the code of Deep4 (Schirrmeister et al.,
2017), we found that they used the ‘Mean’ method to integrate the results of
windows. In some studies based on time-frequency images, they use a method,
such as the Fourier transform(Alhussein et al., 2019), to freely choose the
size of the image, thus eliminating the need for windowing.

Hence we employ ‘Mean’ as a baseline arbitration model. As alternatives, we
explore several implementations of multi-layer perceptrons as the arbitration
model. These are distinguished from each other by the pre-processing of the
input data (per-window scores) and the specific architecture used. The input
pre-processing methods considered are as follows:

‘Raw’

As shown in fig:Raw and Histogram, this approach inputs all the results in
each recording directly(‘Raw’). Since the number of windows in each recording
is different, padding 0 at the end is required for less than 20 windows data.
The value of 20 is chosen here to reflect the approach of Schirrmeister et al.
(2017); Gemein et al. (2020), which used 1-minute windows and a maximum of 20
minutes per recording.

‘Histogram’

Being intended as a flexible approach to handling variations in recording
length, a histogram (fig:Raw and Histogram) was calculated from the per-window
scores. The range of potential per-window abnormality scores (0-1) was divided
into ten equal bins. The input to the model was a vector of length ten
containing, for each bin, the proportion of windows with scores in the
corresponding range, as shown in Figure 6. Bin counts were divided by the
total number of windows as a form of normalisation.

‘Hybrid’

Additionally, we considered a hybrid of the ‘Raw’ and ‘Histogram’ methods. As
shown in fig:Hybrid, in this approach, the ‘Raw’ and ‘Histogram’ input forms
are concatenated.

The architecture of the arbitration models we propose is a fully-connected
layer followed by a softmax layer. We experimented with multi-layer
perceptrons of different depths (from 1 layer to 4 layers), the hidden layers
of different lengths (from 5 to 20), convolutional layers instead of fully
connected layers, activation functions (RELU, ELU, GELU), but these parameters
were found not to significantly influence performance.

For each arbitration model architecture and hyperparameter setting, we conduct
five experiments on the results of each first-stage model experiment. So, when
we consider the two-stage model as a whole, we run $5\times 5=25$ experiments
for each architecture and hyperparameter setting.

fig:Raw and Histogram

Figure 2: ‘Raw’ and ‘Histogram’ pre-processing for the arbitration model. Each
small square in ‘Raw’ is the output of the first-stage model (probability of
‘abnormal’) for one window. In this example there are 16 windows in the
recording. In the general case, since we use the data between 1 and 21 minutes
in a recording at most, a recording contains at most 20 windows with a length
of 1 minute. When there are fewer than 20, we pad zero at the end. Then we
count the ‘Raw’ into a histogram of ten equal bins across the range 0-10.

fig:Hybrid

Figure 3: ‘Hybrid’ pre-processing for the arbitration model.

##  3 Results

###  3.1 Performance of Our Two-Stage Model

As shown in fig:deep4 60s and 600s,fig:window length, all our proposed machine
learning arbitration methods outperform the baseline methods (‘No arbitration’
and ‘Mean’), regardless of window length. The highest average accuracy (25
experiments) for the whole two-stage model achieved by any approach was 93.3,
while the highest average accuracy for a single instance of the first-stage
model (5 experiments) was 96.2, both achieved by the ‘Hybrid’ approach with a
window length of 600 s. In addition, from fig:window length, we can find that
the arbitration stage and increased window length both greatly improve the
sensitivity of the model without substantially compromising specificity.

fig:deep4 60s and 600s [60 s window length] [600 s window length]

Figure 4: Performance of different arbitration models using window lengths of
(a) 60 s and (b) 600 s. Points with the same marker shape come from the same
instance of the first-stage model. The dashed lines represent the mean for
each arbitration method.

###  3.2 Effect of Window Length

As shown in fig:window length, both the performance of the one-stage model
(‘No arbitration’) and the two-stage approaches gets better as the window
length increases, although the performance is not strictly monotonically
increasing; all models have the worst performance at 60 s and the best
performance at 600 s. For the effect on sensitivity and specificity, we can
see similar effects to the two-stage model.

fig:window length [accuracy] [sensitivity] [specificity]

Figure 5: Effect of window length on (a) accuracy, (b) sensitivity, and (c)
specificity. Note that the accuracy of the ‘no_arbitration’ approach is
calculated across all windows ($$4340$\leq N\leq$57482$$, depending on window
length), whereas the accuracy of the arbitration models is calculated across
all recordings ($N=$2993$$).

###  3.3 The Search for the Arbitration Model Architecture

As shown in tab:hidden layer depth and length, we examined the effect of
hidden layer depth and length on the performance of the arbitration model. The
results show that they have no significant impact on the model performance,
although when the model depth is greater than or equal to three, the model is
hard to train. (When the model parameters are initialised in a high loss
position, the model will be difficult to train, that is, maintaining a high
loss, although when it is initially in a relatively low loss position, the
model can reach the same performance as the shallower architectures.). We also
experimented with varying the activation function (RELU (Glorot et al., 2011),
ELU(Clevert et al., 2015), GELU(Hendrycks and Gimpel, 2016)), but the results
show that they did not affect the model performance substantially. Therefore,
we finally chose to use a simple fully-connected layer and a SoftMax layer to
form our model to pursue the optimisation of computational efficiency.

tab:hidden layer depth and length

Table 2: The effect of hidden layer depth and length on the performance of the
arbitration model

2.2ex Hidden layer depth |  2.2exHidden layer length
---|---
5 | 10 | 15 | 20
2.2ex0 | 0.9330 | 0.9330 | 0.9330 | 0.9330
1 | 0.9333 | 0.9342 | 0.9334 | 0.9339
2 | 0.9176 | 0.9334 | 0.9328 | 0.9325
0.2ex 3 | 0.6686 | 0.6732 | 0.6102 | 0.7045

##  4 Discussion

###  4.1 Window Length (Scope)

As noted in Section 1.1, it can be argued that EEG window labels inherited
from the full recording/session label are often misleading. The smaller the
window, the less it represents the wider recording. In particular, a single
transient abnormal event may be sufficient for a recording to be labelled as
‘abnormal’ even when the majority of the windows contain no discernible
abnormality. Hence we hypothesised that increasing the window length would
improve training (of the first stage model) by making the window labels more
accurate (con:WL).

fig:window length shows that the performance of the one-stage model
(‘no_arbitration’) roughly increases with increasing window length, which
confirms that hypothesis. For a window length of 600 s, the accuracy increases
by about 5 percentage points compared with the 60 s window. The performance of
the two-stage models show similar trends.

###  4.2 The Second-Stage Model (Arbitration)

As seen in fig:deep4 60s and 600s,fig:window length, all our proposed variants
of the arbitration model improve accuracy substantially compared with the one-
stage model (‘No arbitration’) and baseline arbitration model (‘Mean’). ‘Mean’
offers minor improvement over ‘No arbitration’: less than two percentage
points. As a simple non-parametric algorithm, ‘Mean’ can mitigate occasional
anomalous outputs but cannot learn more complex or finely tuned decisions. All
neural network arbitration models outperformed the baseline methods,
confirming con:arbitration.

Comparing panels in fig:window length indicates that the improvements in
accuracy (both for increased window length and the proposed arbitration
models) are underpinned by improved sensitivity with relatively little
compromise, if any, in specificity. This supports our supposition that, when
many small windows inherit an abnormal label from their parent recording, many
of the resulting labels are misleading; the window many contain no evident
abnormalities, leading to increased ‘false negative’ results.

As discussed in the Appendix, we experimented with applying the arbitration
models to alternative first case architectures. Similar benefits were observed
when using a Vision Transformer (ViT), but not when using a Temporal
Convolutional Network (TCN).

fig:window lengthfig:accuracy shows that machine learning arbitration models
outperform the baseline methods across all window lengths in our experiments,
although the effect is less pronounced at a window length of 60 s.

Although evidence of arbitration stages can be found in the codebase of
previous studies such as that of Schirrmeister et al. (2017), the concept and
the selection of the model are not discussed, suggesting they were not
considered to be important. We have proved that using a machine learning
arbitration method can substantially exceed the baseline performance of the
same model.

In some other approaches, such as that of Alhussein et al. (2019), arbitration
of classifier outputs is not applicable because a single model fuses features
across all windows to achieve a single classifier output for the recording.
This approach has sound justification, but the increased complexity of the
first stage model poses a challenge for optimisation. The results we present
using relatively simple architectures demonstrate substantially greater
accuracy. This may simply reflect the ease of achieving relatively thorough
optimisation for our approach. Alternatively, the arbitration approach may
present some distinct advantage in terms of robustness to transient non-
clinical anomalies, which might dominate the decision in an architecture with
upstream fusion of features across windows. Confirmation of an explanation for
the performance differences between these methods would require a more
extensive case-by-case comparison.

###  4.3 Label Quality and Performance Ceilings of Machine-Learning-Based
Models on EEG binary Classification Problems

Gemein et al. (2020) suggested that EEG pathology decoding accuracies observed
Van Leeuwen et al. (2019); Schirrmeister et al. (2017); Roy et al. (2019) at
approximately 86 percent were approaching the theoretical optimum imposed by
label noise. This suggestion was based on the observation that inter-rater
agreement in the binary classification of EEGs into pathological and non-
pathological has been reported as 86–88% (Houfek and Ellingson, 1959; Rose et
al., 1973), although these scores were based on EEG ratings of only two
neurologists. In a more recent, broader study, Beuchat et al. (2021) found
interrater agreement to be even lower, $82-86$ percent. However, our study
demonstrated that the performance of machine learning-based models in EEG
binary classification could be much greater than 86%. Although different
raters may give different labels to the same EEG signal, a machine learning
model can learn to replicate the judgement of one rater (or team of raters, as
used in the curation of TUAB (López et al., 2017)). Now that machine learning
approaches can, in some sense, match human expert performance in this task,
future work should include the curation of datasets that combine a diverse
range of human expert judgements and/or data on clinical outcomes to optimise
label accuracy.

###  4.4 Future Work

In this study, we explored a limited range of arbitration model architectures
to demonstrate the importance of arbitration in windowed EEG classifiers. In
immediate future work, we will explore a wider range of arbitration models,
such as random forests. Arguably, the inputs to the arbitration model can be
thought of as tabular data. Random forests are frequently found to outperform
neural networks on tabular data.

It is likely that our pre-processing of the first-stage outputs can also be
optimised further. We will explore the use of overlapping windows to increase
the resolution of information available to the arbitration model. Further
enhancement may be achieved by optimising the binning of the ‘Histogram’ pre-
processing. Rather than using a simple linear spacing of windows, it may be
more effective to use narrower bins in ranges with a higher density of samples
and wider bins (coarser resolution) elsewhere.

We will also extend the application of arbitration to cases in which the
classification task spans multiple recordings from a single clinical visit,
using the wider TUEG dataset in combination with automated labelling based on
the text reports (Western et al., 2021).

In addition to efforts to improve the arbitration stage, we will continue to
explore alternative first-stage architectures. Comparing fig:deep4 60s and
600sfig:60 s window length and fig:60s windows on TCN,fig:60s windows on ViT
indicates the degree of improvement achieved by arbitration varies
significantly between different first-stage architectures. It is possible that
the best first-stage architecture for use with arbitration is not the same as
the best single-stage architecture (for per-window classification, i.e. ‘No
arbitration’). Furthermore, as we move on from TUAB to the larger TUEG
dataset, we may find that data-hungry architectures such as transformers may
outperform those that have achieved previous state-of-the-art results on TUAB.

The arbitration principle is likely to be transferable to other time-series
applications where a holistic classification is to be applied to a windowed
signal. For example, in ECG arrhythmia detection, end-to-end training of
architectures with densely connected output layers is common (Ebrahimi et al.,
2020), but we are not aware of other cases where this final classification
layer is trained separately. Our results suggest that this approach is an
effective way to increase the input scope of the system with minimal added
computational expense. For cardiac electrophysiology, enabling the application
of machine learning classifiers to holistic analysis of long-term Holter
recordings could be important for the detection of subtle abnormalities that
cannot be discerned from shorter signals.

##  5 Conclusion

Our proposed approach, combining increased window length and a machine
learning arbitration stage, substantially improved upon previous state-of-the-
art performance in clinical EEG classification. The results support our
premise that the inheritance of window labels from recording labels
compromised the sensitivity of previous state-of-the-art solutions. Given the
importance of sensitivity for promising applications such as routine screening
or accelerating the workflow of human EEG interpreters, this improvement
presented here is an important step towards the broader translation of machine
learning EEG classifiers into clinical practice. The principles may also be
transferable to other time-series classification problems.

This work was supported by a PhD studentship funded by Southmead Hospital
Charity and the University of the West of England.

## References

  * Alhussein et al. (2019) Musaed Alhussein, Ghulam Muhammad, and M Shamim Hossain.  Eeg pathology detection based on deep learning.  _IEEE Access_ , 7:27781–27788, 2019.
  * Amin et al. (2019) Syed Umar Amin, M Shamim Hossain, Ghulam Muhammad, Musaed Alhussein, and Md Abdur Rahman.  Cognitive smart healthcare for pathology detection and monitoring.  _IEEE Access_ , 7:10745–10753, 2019.
  * Bai et al. (2018) Shaojie Bai, J. Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  2018\.  10.48550/ARXIV.1803.01271.  URL https://arxiv.org/abs/1803.01271.
  * Banville et al. (2021) Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort.  Uncovering the structure of clinical eeg signals with self-supervised learning.  _Journal of Neural Engineering_ , 18(4):046020, 2021.
  * Banville et al. (2022) Hubert Banville, Sean UN Wood, Chris Aimone, Denis-Alexander Engemann, and Alexandre Gramfort.  Robust learning from corrupted eeg with dynamic spatial filtering.  _NeuroImage_ , 251:118994, 2022.
  * Beuchat et al. (2021) Isabelle Beuchat, Senubia Alloussi, Philipp S Reif, Nora Sterlepper, Felix Rosenow, and Adam Strzelczyk.  Prospective evaluation of interrater agreement between eeg technologists and neurophysiologists.  _Scientific Reports_ , 11(1):13406, 2021.
  * Clevert et al. (2015) Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  _arXiv preprint arXiv:1511.07289_ , 2015.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Ebrahimi et al. (2020) Zahra Ebrahimi, Mohammad Loni, Masoud Daneshtalab, and Arash Gharehbaghi.  A review on deep learning methods for ECG arrhythmia classification.  _Expert Systems with Applications: X_ , 7:100033, September 2020.  ISSN 2590-1885.  10.1016/j.eswax.2020.100033.  URL https://www.sciencedirect.com/science/article/pii/S2590188520300123.
  * Gemein et al. (2020) Lukas AW Gemein, Robin T Schirrmeister, Patryk Chrabąszcz, Daniel Wilson, Joschka Boedecker, Andreas Schulze-Bonhage, Frank Hutter, and Tonio Ball.  Machine-learning-based diagnostics of eeg pathology.  _NeuroImage_ , 220:117021, 2020.
  * Glorot et al. (2011) Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 315–323. JMLR Workshop and Conference Proceedings, 2011.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Houfek and Ellingson (1959) Edward E Houfek and Robert J Ellingson.  On the reliability of clinical eeg interpretation.  _The Journal of nervous and mental disease_ , 128(5):425–437, 1959.
  * López et al. (2017) Silvia López, I Obeid, and J Picone.  _Automated interpretation of abnormal adult electroencephalograms_.  PhD thesis, 2017.
  * Muhammad et al. (2020) Ghulam Muhammad, M Shamim Hossain, and Neeraj Kumar.  Eeg-based pathology detection for home health monitoring.  _IEEE Journal on Selected Areas in Communications_ , 39(2):603–610, 2020.
  * Obeid and Picone (2016) Iyad Obeid and Joseph Picone.  The temple university hospital eeg data corpus.  _Frontiers in neuroscience_ , 10:196, 2016.
  * Rose et al. (1973) Stephen W Rose, J Kiffin Penry, Billy G White, and Susumu Sato.  Reliability and validity of visual eeg assessment in third grade children.  _Clinical Electroencephalography_ , 4(4):197–205, 1973.
  * Roy et al. (2019) Subhrajit Roy, Isabell Kiral-Kornek, and Stefan Harrer.  Chrononet: a deep recurrent neural network for abnormal eeg identification.  In _Artificial Intelligence in Medicine: 17th Conference on Artificial Intelligence in Medicine, AIME 2019, Poznan, Poland, June 26–29, 2019, Proceedings 17_ , pages 47–56. Springer, 2019.
  * Schirrmeister et al. (2017) Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball.  Deep learning with convolutional neural networks for eeg decoding and visualization.  _Human brain mapping_ , 38(11):5391–5420, 2017\.
  * Van Leeuwen et al. (2019) KG Van Leeuwen, H Sun, M Tabaeizadeh, AF Struck, MJAM Van Putten, and MB Westover.  Detecting abnormal electroencephalograms using deep convolutional networks.  _Clinical neurophysiology_ , 130(1):77–84, 2019\.
  * Wagh and Varatharajah (2020) Neeraj Wagh and Yogatheesan Varatharajah.  Eeg-gcnn: Augmenting electroencephalogram-based neurological disease diagnosis using a domain-guided graph convolutional neural network.  In _Machine Learning for Health_ , pages 367–378. PMLR, 2020.
  * Western et al. (2021) D Western, T Weber, R Kandasamy, F May, S Taylor, Y Zhu, and L Canham.  Automatic report-based labelling of clinical eegs for classifier training.  In _2021 IEEE Signal Processing in Medicine and Biology Symposium (SPMB)_ , pages 1–6. IEEE, 2021.
  * Yıldırım et al. (2020) Özal Yıldırım, Ulas Baran Baloglu, and U Rajendra Acharya.  A deep convolutional neural network model for automated identification of abnormal eeg signals.  _Neural Computing and Applications_ , 32:15857–15868, 2020\.

##  Appendix A Results on Alternative First-Stage Architectures

As shown in fig:TCN and ViT, we also explored the effect of the arbitration
models on two alternative first-stage architectures: a temporal convolutional
networks (TCN) (Bai et al., 2018; Gemein et al., 2020) and vision transformer
(ViT) (Dosovitskiy et al., 2020). Full details of the implementation and
hyperparameter tuning are beyond the scope of this paper, but the
implementations are available in our code repository. We present the results
briefly here to demonstrate the extent to which our method is transferrable to
other first-stage models.

For TCN, the performance of the proposed arbitration models is not
substantially different from the baseline (‘Mean’). For ViT, our proposed
arbitration models can provide about two percentage points of performance
improvement. Based on the present evidence, the proposed methods appear to
offer a safe improvement in the sense that no cases were observed in which
accuracy was substantially worsened. We will test the effect of the
arbitration models on a wider selection of first-stage models as well as
longer window lengths in future work.

fig:TCN and ViT [TCN] [ViT]

Figure 6: Performance of different arbitration methods using (a) TCN and (b)
ViT as the first-stage architecture with a window length of 60 s.


# Better Neural Machine Translation by
Extracting Linguistic Information from BERT

Hassan S. Shavarani Anoop Sarkar
School of Computing Science
Simon Fraser University
BC, Canada
<EMAIL_ADDRESS>

###### Abstract

Adding linguistic information (syntax or semantics) to neural machine
translation (NMT) has mostly focused on using point estimates from pre-trained
models. Directly using the capacity of massive pre-trained contextual word
embedding models such as BERT (Devlin et al., 2019) has been marginally useful
in NMT because effective fine-tuning is difficult to obtain for NMT without
making training brittle and unreliable. We augment NMT by extracting dense
fine-tuned vector-based linguistic information from BERT instead of using
point estimates. Experimental results show that our method of incorporating
linguistic information helps NMT to generalize better in a variety of training
contexts and is no more difficult to train than conventional Transformer-based
NMT.

##  1 Introduction

Probing studies into large contextual word embeddings such as BERT (Devlin et
al., 2019) have shown that these deep multi-layer models essentially
reconstruct the traditional NLP pipeline capturing syntax and semantics
(Jawahar et al., 2019); information such as part-of-speech tags, constituents,
dependencies, semantic roles, co-reference resolution information (Tenney et
al., 2019a, b) and subject-verb agreement information can be reconstructed
from BERT embeddings (Goldberg, 2019). In this work, we wish to extract the
relevant pieces of linguistic information related to various levels of syntax
from BERT in the form of dense vectors and then use these vectors as
linguistic “experts” that neural machine translation (NMT) models can consult
during translation.

But can syntax help improving NMT? Linzen et al. (2016); Kuncoro et al.
(2018); Sundararaman et al. (2019) have reported that learning grammatical
structure of sentences can lead to higher levels of performance in NLP models.
In particular, Sennrich and Haddow (2016) show that augmenting NMT models with
explicit linguistic annotations improves translation quality.

BERT embeddings have been previously considered for improving NMT models.
Clinchant et al. (2019) replace the encoder token embedding layer in a
Transformer NMT model with BERT contextual embeddings. They also experiment
with initializing all the encoder layers of the translation model with BERT
parameters, in which case they report results on both freezing and fine-tuning
the encoder parameters during training. In their experiments BERT embeddings
can help with noisy inputs to the NMT model, but otherwise do not help
improving NMT performance.

Imamura and Sumita (2019) suggest that replacing the encoder layer with BERT
embeddings and fine-tuning BERT while training the decoder leads to a
catastrophic forgetting phenomenon where useful information in BERT is lost
due to the magnitude and number of updates necessary for training the
translation decoder and fine-tuning BERT. They present a two-step optimization
regime in which the first step freezes the BERT parameters and trains only the
decoder while the next step fine-tunes the encoder (BERT) and the decoder at
the same time. Yang et al. (2020) also try to address the catastrophic
forgetting phenomenon by thinking of BERT as a teacher for the encoder of the
neural translation model (student network) Hinton et al. (2015). They propose
a dynamic switching gate implemented as a linear combination of the encoded
embeddings from BERT and the encoder of NMT. However these papers do not
really focus on the linguistic information in BERT, but rather try to combine
pre-trained BERT and NMT encoder representations.

Sundararaman et al. (2019) identify part-of-speech, case, and sub-word
position as essential linguistic information to improve the quality of both
BERT and the neural translation model. They extract each linguistic feature
using the Viterbi output of separate models, embed the extracted linguistic
information (similar to trained word embeddings) and append these vectors to
the token embeddings. However, their model uses point estimates of the
syntactic models and they do not use the linguistic information in BERT
embeddings.

Weng et al. (2019) use multiple multi-layer perceptron (MLP) modules to
combine the information from different layers of BERT into the translation
model. To make the most out of the fused information, they also alter the
translation model training objective to contain auxiliary knowledge
distillation (Hinton et al., 2015) parts concerned with the information coming
from the pre-trained language model. Zhu et al. (2020) also inject BERT into
all layers of the translation model rather than only input embeddings. Their
model uses an attention module to dynamically control how each layer interacts
with the representations. In both of these works, the training of the
Transformer for NMT becomes quite brittle and is prone to diverge to local
optima.

In this paper, we propose using pre-trained BERT as a source of linguistic
information rather than a source of frozen pre-trained contextual embedding.
We identify components of the BERT embeddings that correspond to different
types of linguistic information such as part-of-speech, etc. and fine-tune
dense vector embeddings for these linguistic aspects of the input and use them
within an NMT model. Our approach does not radically complicate the
Transformer NMT model training process both in terms of time and hardware
requirements and also in terms of training difficulty (avoids bad local
optima).

Our contributions are as follows: (1) A method of linguistic information
extraction from BERT which needs supervision while training but works without
supervision afterwards. (2) An easily trainable procedure for integrating the
extracted information into the translation model. (3) Evaluation of the
proposed model on small, medium and large translation datasets.

The source code and trained aspect extractors are available at
https://github.com/sfu-natlang/SFUTranslate and our experiments can be
replicated using scripts under resources/ exp-scripts/aspect_exps.

##  2 NMT and BERT

Machine translation is the problem of transforming an input utterance sequence
X in source language $l_{f}$ into another utterance sequence Y (possibly with
varying length) in target language $l_{e}$. Machine translation models search
among all possible sequences in target language to find the most probable
sequence based on the probability distribution of Equation 1.

$P(y|X,y\in l_{e})=\prod_{i=0}^{|max\ len|}p(y_{i}|X,y_{0},...,y_{i-1})$ (1)

Neural machine translation (NMT) tries to model the probability distribution
$p(y|X)$ using neural networks by taking advantage of deep learning
techniques. Transformers (Vaswani et al., 2017) are one type of encoder-
decoder neural networks used for translation tasks. In Transformers, the input
(in one-hot format) is passed through $N$ layers of encoder and $N$ layers of
decoder. In each layer, the layer input passes through multiple attention
heads ($h$ heads; each considered a specialist in a different sentence-level
linguistic attribute) and then gets transformed to the input for the next
layer using a two layer feed-forward perceptron module with input size of
$d_{\textit{model}}$ and hidden layer size of $d_{\textit{ff}}$. The final
probability distribution $p(y|X)$ is generated using an affine transformation
applied to the output of the last feed-forward module in the $N^{th}$ decoder
layer. Please see (Vaswani et al., 2017) for further details.

BERT (Devlin et al., 2019) adopts the encoder part of the transformer model
and requires training it on large amounts of text data using a masked language
model objective over sub-words
$p(y_{i}|X,y_{0},...,y_{i-1},y_{i+1},...,y_{max\ len})$ instead of guessing
the next sub-word $p(y_{i}|X,y_{0},...,y_{i-1})$. This bidirectional context
turns BERT into a provider of strong contextual sub-word embeddings in many
languages. These massively over-parameterized neural networks have
revolutionized many different NLP tasks. Effective application of BERT in NMT
has been studied in a number of contemporary research projects; Language
Modeling, Named Entity Recognition, Question Answering, Natural Language
Inference, Text Classification Devlin et al. (2019), and Question Generation
Chan and Fan (2019). We approach this problem from the novel perspective of
extracting linguistic information encoded in BERT and applying such
information in NMT.

##  3 Linguistic Aspect Extraction from BERT

Since BERT contextual embeddings contain a variety of information (linguistic
and non-linguistic), extraction of relevant information plays an important
role in further improvement of the downstream tasks. In the rest of this
section, we define aspect vectors as single-purpose dense vectors of extracted
linguistic information from BERT, discuss how aspect vectors can be extracted,
and explain how to integrate aspect vectors into NMT.

###  3.1 Aspect Vectors

To start the information extraction process, we initially need to choose a
limited (desired) set of linguistic attributes to look for in BERT embeddings.
This attribute set can contain a number of linguistic aspects (e.g. part-of-
speech). Each linguistic aspect itself will be defined over a possible aspect
tag set (e.g. the set of {NOUN, ADJ, …} in part-of-speech). In this paper, we
show a linguistic attribute set with $\mathbb{A}$, show a generic aspect with
$a$ and point to its relative tag set with $t_{a}$.

Given the definition of a linguistic aspect and inspired by the information
bottleneck idea (Tishby and Zaslavsky, 2015), we define an aspect vector as a
single-purpose dense vector extracted from BERT and containing information
about a certain linguistic aspect of a particular (sub-word) token in the
input sequence. Aspect vectors can be interpreted as feature values equivalent
to a specific key (aspect).

###  3.2 Aspect Vector Extraction

For each embedding vector $\mathbf{E}$ and linguistic aspect $a$, we define
$M_{a}$ as an aspect-extraction function where
$\mathbf{e}_{a}=M_{a}(\mathbf{E})$ is a single-purpose dense vector containing
maximum aspect information and minimum irrelevant other information.

We ensure the aspect encoding power of $\mathbf{e}_{a}$ by retrieving its
equivalent tag in $t_{a}$ using a classifier. The aspect prediction loss for a
linguistic attribute set $\mathbb{A}$ of size $n$ can be calculated as the
average cross entropy loss ($\mathcal{L}_{CE}$) between the classifier
prediction and the expected aspect tags for each aspect (Equation 2).

$\mathcal{L}_{a}=\frac{1}{n}\sum_{i=0}^{|n|}\mathcal{L}^{i}_{CE}$ (2)

We also ensure information integrity111We don’t expect $M_{a}$ to change the
information inside $\mathbf{E}$ but rather to extract the relevant
information. of $\mathbf{e}_{a}$ by concatenating all the aspects (in addition
to a “left-over” aspect equivalent to all the other non-interesting
information) and reconstructing the original embedding vector $\mathbf{E}$
from them222This idea is analogous to stack-propagation (Zhang and Weiss,
2016) in which propagating the information loss for two tasks helps improving
the quality of the encoded representations. in reconstruction vector
$\mathbf{R}$. The reconstruction loss ($\mathcal{L}_{r}$) for the extracted
aspect vectors can be calculated as the euclidean distance of the
reconstruction vector $\mathbf{R}$ and the original embedding vector
$\mathbf{E}$ (Equation 3).

Linguistic
Aspect
Extractors Token Bert Embedding Reconstructed Bert Embedding POS
Shape…LeftOver POS Classifier Prediction

Figure 1: Schematic Aspect Extraction from BERT

Positional Encodig Positional Encodig Input Embedding Output Embedding
OutputsInputsAdd & NormAdd & NormAdd & NormAdd & NormAdd & Norm Feed Forward
MultiHead Attention MultiHead Attention Feed Forward Masked MultiHead
Attention SoftmaxLinear Output Probabilities N$\times$N$\times$ Integration
Module BERT Aspect Extractors …Aspect 1Aspect Ncat…TokenEmbsLingEmbsDimension
Transformation Linguistic Embedding Linguistic
Embedding Linguistic Aspect Extractors Token Bert Embedding Reconstructed Bert
Embedding POS Shape…LeftOver POS Classifier Prediction

Figure 2: Integration of Extracted Aspect Vectors into NMT. The right hand
side part of this figure is taken from Vaswani et al. (2017).

$\mathcal{L}_{r}=||\mathbf{R}-\mathbf{E}||^{2}$ (3)

In addition, since our aspect extractor is similar in architecture to a multi-
head attention module (with a difference in the fact that we know what exactly
each head will be responsible for), to prevent learning redundant
representations (Michel et al., 2019), we add the average euclidean similarity
($\mathcal{L}_{s}$) of each pair of aspect vectors to the training loss
function (Equation 4).

$\mathcal{L}_{s}=1-\left(\frac{1}{n(n-1)}\sum_{i=0}^{|n|}{\sum_{j\neq
i=0}^{|n|}{||{e}_{i}-{e}_{j}||^{2}}}\right)$ (4)

The aspect extractor will be trained over the accumulation of the three
mentioned loss components (Equation 5). Figure 1 demonstrates different parts
of the aspect extractor and their connections.

$\mathcal{L}_{fe}=\mathcal{L}_{a}+\mathcal{L}_{r}+\mathcal{L}_{s}$ (5)

As another important point, a pre-trained BERT model has multiple encoder
layers as well as an embedding layer. Choosing the proper layer which contains
all of our desired aspects is not simply possible since different layers
specialize in different linguistic aspects (Jawahar et al., 2019; Tenney et
al., 2019a).

Therefore, as Peters et al. (2018) suggest, we define BERT embedding vector
$\mathbf{E}$ as a weighted sum of all BERT layers (of size $\ell$) using
Equation 6 where $\alpha$ weights are learnable parameters and will be trained
along with the other aspect extractor parameters.

$\mathbf{E}=\sum_{j=0}^{\ell}\alpha_{j}\mathbf{E}^{BERT}_{j}$ (6)

###  3.3 Integrating Aspect Vectors into NMT

Once the aspect vectors are created, we throw away the classifiers and the
reconstruction layers and place the encoder part of our trained aspect
extractor (the mapping from BERT contextual embeddings to aspect vectors) in
an input integration module designed to augment the neural translation model
input with aspect vectors333We use the same sub-word model in pre-trained BERT
to provide sub-word tokens to our NMT model..

The integration module (constructed using a two layer perceptron network)
receives the concatenated aspect vectors (we call this concatenated vector a
linguistic embedding444This embedding vector can be similar to what a factor
token contains in Factored-NMT García-Martínez et al. (2016) with a difference
that it is generated in the space of linguistic aspects and does not need an
embedding layer.) and the token embedding (inherited from the Transformer
model), and maps the linguistic embedding into a vector of the same size as
the token embedding. Then, it projects the concatenation of both embeddings to
a vector with the same size as the token embedding of the original Transformer
model555This step is necessary to prevent any change in other parts of the
model which would make comparison of the results unfair due to effects on the
number of parameters and the learning capability of the model.. Figure 2
demonstrates this process.

##  4 Experiments

In this section, we initially examine our designed aspect extractor and report
its classification accuracy scores. Next, we integrate the extracted aspect
vectors into the neural machine translation framework as explained in Section
3.3 and study the effects of integrated vectors on the performance of the
models.

###  4.1 Data

We choose three German (which has explicit and nuanced linguistic features) to
English datasets in different data sizes to examine our proposed framework.

We use Multi30k (M30k)666AKA Flickr30K provided in task 1 of WMT17 multimodal
machine translation, http://www.statmt.org/wmt17/ multimodal-task.html as our
small dataset. This dataset contains a multilingual set of image descriptions
in German, English and French. Due to this reason, we also consider
experimenting on German to French as our second small dataset. The M30k data
contains 29K training sentences, 1014 validation sentences ($val$) and 1000
test sentences ($test2016$).

We take IWSLT (Cettolo et al., 2012)7772017 was the last year that the data
for this task got updated; https://wit3.fbk.eu/mt.php?release=2017-01-mted-
test as our medium sized dataset. The sentences in this dataset are quite
different from M30k since they are composed from the transcriptions of TED
talks as well as dialogues and lectures888While the talks are quite polished,
they still contain many verbal structures and sometimes even sounds (e.g.
“Imagine an engine going clack, clack, clack, clack, clack, clack, clack.”)..
The IWSLT data contains 208K training sentences, 888 validation sentences
($dev2010$) and multiple test sets ($tst2010$ to $tst2015$ with 1568, 1433,
1700, 993, 1305, and 1080 sentences, respectively).

For the large data size, we consider WMT999Europarl+CommonCrawl+NewsCommentary
https://www.statmt.org/wmt14/translation-task.html, please note that in the
later years this training set remained the same, but ParaCrawl data was added
to it. We do not use ParaCrawl data since it is quite noisy and we aim to
limit the effects of uncontrolled variables in our training data. However, we
report our results on all the test tests after 2014., a large (4.5M training
sentences) set of parallel sentences from the proceedings of the European
Parliament as well as web crawled news articles. We remove 0.05% of the
training data (2290 sentences; lines with numbers divisible by 2000) and use
it as the validation set (we call it $wmt\\_val$) and take $newstest$ data
from 2014 to 2019 as our test sets (with 3003, 2169, 2999, 3004, 2998 and 1997
sentences, respectively).

We remove train data sentences longer than 100 words and uncase and normalize
both side sentences using
MosesPunctNormalizer101010https://github.com/alvations/sacremoses/ before
tokenization. The reference side of the test data remains untouched in all the
steps of our experiments.

###  4.2 Linguistic Aspect Vector Extraction

In this section, we study our linguistic aspect extractor training procedure
and analyze the quality of the extracted aspect vectors.

We choose our linguistic attribute set ($\mathbb{A}$) as Sundararaman et al.
(2019) suggest, however, we replace ‘case’ with ‘word-shape’111111Representing
capitalization (changing alphabet to x or X), punctuation, and digits
(changing digits to d). As an example for word-shape, the sub-word ##arxiv. in
the token ‘myarxiv.org’ will turn to ##xxxxx.. since we believe the complete
shape of the word is much more informative specially in sub-word settings. In
addition, we consider a two-level hierarchy in part-of-speech tags to benefit
from both higher accuracy in exploring the syntactic search space and lower
model confusion in cases where the fine-grained tags are not helpful.
Therefore, we consider coarse-grained and fine-grained part-of-speech (CPOS
and FPOS), word-shape (WSH), and sub-word position121212Encoding the word with
one of the three labels “Begin”, “Inside”, or “Single”. (SWP) to form our
experimental linguistic attribute set ($\mathbb{A}$). Other linguistic
attributes such as dependency parses or sentiment could be considered as
aspects in our model but we leave that for future work.

| Sub-word Level | Word Level
---|---|---
CPOS | FPOS | WSH | SWP | #tokens | CPOS | FPOS | WSH | #tokens
M30k | 96.88 | 96.18 | 99.79 | 99.93 | 16096 | 97.95 | 97.34 | 99.74 | 12823
IWSLT | 92.69 | 90.48 | 99.73 | 97.14 | 22687 | 94.84 | 93.07 | 99.69 | 19039
WMT | 92.64 | 91.60 | 97.74 | 98.94 | 70139 | 94.86 | 94.01 | 97.38 | 55135
Table 1: F-1 scores acquired after training the aspect extractor on German side of parallel data and passing the validation sets of each data set through trained aspect extractors. The #tokens column shows the number of tokens in the validation set. | Aspect Extractor
---
Training Data
FPOS | SWP
M30k | 79.39 | 90.63
IWSLT | 77.80 | 88.34
WMT | 82.13 | 91.42
TIGER | 84.64 | 92.64
Table 2: F-1 scores of fine-grained part-of-speech prediction of TIGER corpus
test data (BERT encoded) fed to each of the trained aspect classifiers. The
scores are calculated over a total of 7516 sub-word tokens in 358 test
sentences of TIGER. Extractors trained on M30k, IWSLT, and WMT have not been
provided with any part of TIGER before evaluation.

We use the spaCy German tagger131313https://spacy.io/models/de model to
acquire our intended linguistic aspect labels. Since spaCy is trained on word-
level while BERT is trained on sub-word level, we had to align the sequences
using a monotonic alignment algorithm (see Appendix A.1.1). The fine-grained
part-of-speech tagger in spaCy141414SpaCy reports 96.52% accuracy for this
model. is pre-trained on TIGER Corpus151515https://www.ims.uni-stuttgart.de/
(Smith et al., 2003) and inherits its 55 fine-grained tags from TIGER
treebank. The coarse-grained spaCy part-of-speech tagger has been trained by
defining a direct mapping from 55 tags of the TIGER treebank to the 16 tags in
the Universal Dependencies v2 POS tag
set161616https://universaldependencies.org/v2/postags.html.

We use a 12-layer171717Hidden state size of 768 with 12 heads; written in
PyTorch and distributed by Wolf et al. (2019). You can find model
configurations in https://github.com/dbmdz/berts. German pre-trained BERT
model for encoding the source sentences in aspect extractors. We use an
uncased model as our translation model performs on lowercased data and the
results are recased using the moses recaser so that the results are cased BLEU
scores comparable to other systems181818We recommend using a cased BERT model
for translation systems that handle casing differently.. We pass the BERT-
encoded source sentences through a single perceptron middle layer of size
1000. We divide the output of this layer to ‘number of aspects + 1’ splits to
form our desired aspect vectors (of size 200). Please see Appendix A.1.1 for
more implementation details.

We train three different aspect extractors, one for each dataset and feed in
the source sentences of the dataset to our model in batches of size 32 for 3
epochs191919Since the number of WMT sentences are much bigger, we stop
training WMT aspect extractors when there is no improvement in aspect
classification result (rounded to have 3 decimal places) of any label for at
least 40 batches.. Table 1 shows F-1 scores of classifying the validation set
data using different aspect vectors after training the aspect extractors on
the train set sentences. Please note that for calculating the word-level
scores, in cases of disagreement between different sub-word tokens, the sub-
word prediction of the first sub-word token has been counted as the prediction
for the word label.

We also validate our trained (on M30k, IWSLT, and WMT) aspect extractors
against the manual annotations of TIGER treebank with which the spaCy fine-
grained part-of-speech tagger has been trained. We train an extra aspect
extractor using the train set of TIGER corpus and test all four trained aspect
extractors against TIGER data test set202020We use german_tiger_test_gs.conll
in the version of TIGER released in 2006 CoNLL Shared Task - Ten Languages.
Both train and test data are accessible through
https://catalog.ldc.upenn.edu/LDC2015T11.. This experiment evaluates the
absolute power of our simple feed-forward aspect extractors in performing the
aspect classification task. Please note that our goal in this experiment is
not to achieve the state-of-the-art fine-grained part-of-speech tagging
results as our aspect extractors receive their input from BERT and do not
directly access the tagged input sentences. Table 2 contains the results of
comparison between predictions of different aspect extractor classifiers and
TIGER gold labels.

###  4.3 Uniqueness of Information in Linguistic Aspect Vectors

Considering the high F-scores for each aspect category in each dataset (Table
1), we can conclude that our aspect extractor maximizes the relevant
information extraction from BERT embeddings. The loss in Equation 4 maximizes
the distance between aspect vectors. To test whether this leads to a diverse
set of aspect vectors, each specialized to their own linguistic attributes, we
consider each aspect category $a$, after training the aspect extractors. We
take each of the other extracted aspect vectors $a^{\prime}$ (except the
“left-over” vector) and use each of them to train a new classifier212121We
thank the anonymous reviewers for their valuable feedback on this procedure.
that predicts the right class for category $a$ based on aspect vector
$a^{\prime}$. This will test the correlation between the information in aspect
vectors $a^{\prime}$ and the tags in category $a$. If the classification
scores for this counterfactual test are high then our model has failed in
fine-tuning each aspect vector to predict a particular linguistic aspect. We
compare the classification scores to a trivial baseline: predict the most
frequent class always. Table 3 shows the results of this counterfactual test
on the aspect extractor trained on TIGER data. We can see that the average F-1
scores are very low when we use counterfactual aspect vectors to predict a
linguistic aspect on which it was not fine-tuned (e.g. use aspect vector
trained on part-of-speech to predict word shape). This shows that our training
method fine-tunes each aspect vector to its linguistic task.

TIGER test | Sub-word Level
---|---
CPOS | FPOS | WSH | SWP
| most frequent
---
class
NOUN | NN | xxxx | single
| percentage
---
in total
27.12 | 27.07 | 39.07 | 59.92
| average
---
classification F-1
1.89 | 0.23 | 12.20 | 42.97
#tokens | 7516 $\times$ 3 = 22548
Table 3: Classification scores of each aspect classifier when fed with other
extracted aspect vectors. We expect the F-1 scores to be low so we can
conclude that our aspect extractor truly excludes irrelevant information from
each aspect.

To validate the loss in Equation 3, we calculate the average euclidean
distance of the aspect extractor reconstructed vectors and the original BERT
embedding vectors222222Average results of Equation 3 for all the tokens in the
train set. for M30k German to English dataset. We unit normalize each of the
vectors for a score in $[0,1]$. The average euclidean distance value of 0.1863
tells us that the reconstruction component of the aspect extractor is capable
of reconstructing vectors that are close to the original embedding vectors.

###  4.4 Linguistic Aspect Integrated Machine Translation

a) M30k$\dagger$ | German to English |  | German to French
---|---|---|---
val | test2016 | #param | runtime$*$ |  | val | test2016 | #param | runtime$*$
| Vaswani et al. 2017
---
39.63 | 38.35 |  9.5 M |  84 min |  | 31.07 | 30.29 |  9.4 M |  93 min
| Sundararaman et al. 2019
---
40.03 | 38.32 | 13.9 M | 514 min |  | 32.55 | 32.71 | 13.6 M | 504 min
| Clinchant et al. 2019
---
(bert freeze)
40.07 | 39.73 |  9.1 M |  99 min |  | 33.83 | 33.15 |  9.0 M | 104 min
| Aspect Augmented
---
+M30k asp. vectors
40.47 | 40.19 | 10.1 M | 104 min |  | 34.45 | 34.42 |  9.9 M | 108 min
| Aspect Augmented
---
+WMT asp. vectors
38.72 | 41.53 | 10.1 M | 102 min |  | 34.73 | 34.28 |  9.9 M | 118 min
b) IWSLT$\dagger$ | dev2010 | tst2010 | tst2011 | tst2012 | tst2013 | tst2014 | tst2015 | #param | runtime$*$
| Vaswani et al. 2017
---
27.69 | 27.93 | 31.88 | 28.15 | 29.59 | 25.66 | 26.76 | 18.4 M |  172 min
| Sundararaman et al. 2019
---
29.53 | 29.67 | 33.11 | 29.42 | 30.89 | 27.09 | 27.78 | 28.9 M | 1418 min
| Clinchant et al. 2019
---
(bert freeze)
30.31 | 30.00 | 34.20 | 30.04 | 31.26 | 27.50 | 27.88 | 18.0 M |  212 min
| Aspect Augmented
---
+IWSLT asp. vectors
29.03 | 29.17 | 33.42 | 29.58 | 30.63 | 26.86 | 27.83 | 18.9 M |  214 min
| Aspect Augmented
---
+WMT asp. vectors
31.22 | 30.82 | 34.79 | 30.29 | 32.34 | 27.71 | 28.40 | 18.9 M |  211 min
c) WMT$\dagger$ | wmt_val | nt2014 | nt2015 | nt2016 | nt2017 | nt2018 | nt2019 | #param | runtime$*$
| Vaswani et al. 2017
---
28.96 | 26.91 | 26.93 | 31.42 | 28.07 | 33.56 | 29.77 | 68.7 M |  35 h
| Sundararaman et al. 2019
---
28.56 | 27.80 | 26.93 | 30.44 | 28.63 | 33.87 | 30.48 | 93.8 M | 258 h
| Clinchant et al. 2019
---
(bert freeze)
28.63 | 27.54 | 27.15 | 31.69 | 28.30 | 33.89 | 31.48 | 69.1 M |  33 h
| Aspect Augmented
---
+WMT asp. vectors
28.98 | 28.05 | 27.58 | 32.29 | 29.07 | 34.74 | 31.48 | 70.3 M |  46 h
Table 4: Evaluated cased Bleu score (calculated using mteval-v14.pl script)
results on M30k, IWSLT, and WMT datasets. #param represents the number of
trainable parameters (size of BERT model parameters [110.5M] has not been
added to the model size for the aspect augmented and bert-freeze models since
BERT is not trained in these settings). runtime is the total time the training
script has ran and includes time taken for reading the data and training the
model from scratch (iterating over the instances for all the epochs).
All the baseline results are achieved using our re-implementation of the
mentioned papers.
$*$ We have used a single GeForce GTX 1080 GPU for M30k experiments and a
single Titan RTX GPU for IWSLT and WMT experiments.
$\dagger$ Each experiment was repeated three times, and we report the average
in this table.

After confirming the adequacy and uniqueness of linguistic information in
aspect vectors, we integrate the encoder part of aspect extractors into the
translation model and perform translation experiments on M30k, IWSLT, and WMT
datasets. In our experiments, we compare our model to three baselines : (1)
the vanilla transformer model (Vaswani et al., 2017) which does not use any
external source of information, (2) the syntax-infused transformer model
(Sundararaman et al., 2019) which explicitly embeds linguistic aspect labels
and concatenates their embedding to the token embedding, (3) the transformer
model with bert-freeze input setting (Clinchant et al., 2019) which replaces
the input embedding layer of the encoder module in transformer with a fully
pre-trained BERT model. Appendix A.1.2 provides the configurations and
sufficient details for replication of our experiments in this section.

During each training trial, we perform 9 validation set evaluation steps (one
after visiting each 10% of the data). In each step, the validation set is
translated with the current state of the model (at the time of evaluation) and
the generated sentences are detokenized and compared to the validation set
reference data to produce sentence-level Bleu (Lin and Och, 2004) scores. The
best scoring model throughout training is selected as the model with which the
test set(s) are translated.

For M30k and IWSLT data sets, we train two separate models, one using the
aspect vectors trained on the source side of its own training data (in-domain)
and the other using the aspect vectors trained on the source side of WMT data
(out-of-domain). We use cased Bleu (evaluated with the standard mteval-v14.pl
script) and METEOR Denkowski and Lavie (2014) to compare different models.
Tables 4 and 7 show the results of evaluating the models trained with
different mentioned settings.

The evaluation results show that taking advantage of aspect vectors improves
the accuracy of translating German to both English and French in M30k as well
as German to English in IWSLT and WMT. Also, in majority of the cases WMT-
trained aspect vectors have pushed the model to produce more accurate results
since they contain more generalized information. Based on these results, we
conjecture that aspect vectors trained on large out-of-domain data can be
helpful in low-resource settings but we leave the examination of this idea for
future work.

Source | Ihm werde weiterhin vorgeworfen, unerlaubt geheime Informationen weitergegeben zu haben.
---|---
Reference | He is still accused of passing on secret information without authorisation.
Vaswani et al. 2017 | He has also been accused of having illegally passed on secret information.
Clinchant et al. 2019 | He continues to be accused of fraudulently passing on secret information.
Sundararaman et al. 2019 | He is also accused of having pass unauthorised secret information on.
Aspect Augmented NMT | He is still accused of passing on illegal secret information.
Source | Auto und Traktor krachen zusammen: Frau stirbt bei schrecklichem Unfall
Reference | Car and tractor crash together: woman dies in terrible accident
Vaswani et al. 2017 | Car and traktor cranes together: women die in the event of a terrible accident.
Clinchant et al. 2019 | Cars and tractors are killing women in the event of a terrible accident.
Sundararaman et al. 2019 | Auto and tractor are blowing together: woman dies when the terrible accident occurs.
Aspect Augmented NMT | Car and tractor crash together: woman dies in terrible accidents.
Table 5: Examples of improved translation quality of WMT data where part-of-speech aspect vectors have helped the model choose better words both syntactically and semantically. Source | Bucht die besten Hostels in Ouarzazate über Hostelsclub.
---|---
Reference | Book the best hostels in Ouarzazate with Hostelsclub.
Vaswani et al. 2017 | Book the best hostels in ouarzazazate with Hostelsclub.
Clinchant et al. 2019 | Book the best hostels in Ouarzate with Hostelsclub.
Sundararaman et al. 2019 | Book the best hostels in ouarzazazate with Hostelsclub.
Aspect Augmented NMT | Book the best hostels in Ouarzazate with Hostelsclub.
Source | Die Deutsche Bahn will im kommenden Jahr die Kinzigtal-Bahnstrecke verbessern.
Reference | The Deutsche Bahn hopes to improve the Kinzigtal railway line in the coming year.
Vaswani et al. 2017 |  The German Railway wants to improve the Kinzig valley railway line next year.
Clinchant et al. 2019 |  Christian Deutsche Bahn intends to improve the Kinzig valley railway next year.
Sundararaman et al. 2019 |  The German Railway wants to improve the kinziggia railway line next year.
Aspect Augmented NMT |  Deutsche Bahn wants to improve the Kinzig valley railway in the coming year.
Table 6: Examples of improved translation quality of WMT data where word-shape and sub-word position aspect vectors have helped the model choose a better sequence of sub-words when it faces out-of-vocabulary tokens. a) M30k$\dagger$ | German to English |  |  |  | German to French
---|---|---|---|---|---
val | test2016 |  |  |  | val | test2016
| Vaswani et al. 2017
---
37.20 | 36.56 |  |  |  | 53.22 | 52.58
| Sundararaman et al. 2019
---
38.14 | 37.13 |  |  |  | 54.18 | 54.37
| Clinchant et al. 2019
---
(bert freeze)
38.44 | 37.42 |  |  |  | 55.10 | 54.50
| Aspect Augmented
---
+M30k asp. vectors
39.22 | 38.17 |  |  |  | 56.21 | 56.40
| Aspect Augmented
---
+WMT asp. vectors
38.90 | 38.57 |  |  |  | 56.12 | 55.98
b) IWSLT$\dagger$ | dev2010 | tst2010 | tst2011 | tst2012 | tst2013 | tst2014 | tst2015
| Vaswani et al. 2017
---
31.82 | 31.99 | 34.57 | 32.65 | 32.49 | 30.65 | 31.13
| Sundararaman et al. 2019
---
32.91 | 32.95 | 35.35 | 33.10 | 33.17 | 31.32 | 31.90
| Clinchant et al. 2019
---
(bert freeze)
33.34 | 32.78 | 35.42 | 33.12 | 33.20 | 31.22 | 31.45
| Aspect Augmented
---
+IWSLT asp. vectors
32.86 | 32.86 | 35.38 | 33.43 | 33.23 | 31.37 | 31.87
| Aspect Augmented
---
+WMT asp. vectors
33.78 | 33.56 | 36.14 | 33.51 | 33.98 | 31.86 | 32.37
c) WMT$\dagger$ | wmt_val | nt2014 | nt2015 | nt2016 | nt2017 | nt2018 | nt2019
| Vaswani et al. 2017
---
30.65 | 33.80 | 33.70 | 37.10 | 34.44 | 37.81 | 36.05
| Sundararaman et al. 2019
---
29.23 | 31.57 | 31.61 | 34.05 | 31.87 | 35.18 | 33.60
| Clinchant et al. 2019
---
(bert freeze)
30.39 | 33.46 | 33.20 | 36.13 | 33.73 | 37.24 | 35.68
| Aspect Augmented
---
+WMT asp. vectors
30.61 | 33.97 | 33.99 | 37.01 | 34.71 | 38.17 | 36.48
Table 7: Evaluated METEOR score (calculated using the tool provided by Alon
Lavie (https://www.cs.cmu.edu/ $\sim$alavie/METEOR/; version 1.5)) results on
M30k, IWSLT, and WMT datasets.
$\dagger$ Each experiment was repeated three times, and we report the average
in this table.

Aside from performance, our model is approximately 5 times faster than syntax-
infused translation model (Sundararaman et al., 2019) while demanding less
number of trainable parameters. Although it is not as fast as bert-freeze
model (Clinchant et al., 2019) in large settings (because of the size of
computations required for calculating the linguistic embedding), it is
comparable in speed to bert-freeze in medium and small scale settings.
Appendix A.2 contains some additional insights regarding how aspect vectors
can help translation systems trained on different dataset sizes.

Tables 5 and 6 demonstrate some examples of cases where aspect vectors has
been useful in improving the translation quality.

##  5 Conclusion and Future Work

In this paper, we proposed a simple method of extracting linguistic
information from BERT contextual embeddings and integrating them into neural
machine translation framework. We showed that the linguistic aspect vectors
provide the translation models with out-of-domain knowledge which not only
improves the translation quality but also helps the model to better deal with
out-of-vocabulary words. In the future, we would like to reconsider the
integration module as a multi-head attention module, except that it will
attend to different linguistic aspects of the current sub-word or sub-word
tokens of a single word. Increasing the number of linguistic aspects
(especially the use of syntactic dependencies and morphology) and studying the
effects of the aspect vector size on the quality of generated translations are
other directions of future research. We would also like to examine the
effectiveness of aspect vectors trained on large out-of-domain data in low-
resource settings and explore the effects of using linguistic aspect vectors
in tasks other than machine translation.

## Acknowledgments

We would like to thank the anonymous reviewers for their helpful comments. The
research was partially supported by the Natural Sciences and Engineering
Research Council of Canada grants NSERC RGPIN-2018-06437 and RGPAS-2018-522574
and a Department of National Defence (DND) and NSERC grant DGDND-2018-00025.

## References

  * Cettolo et al. (2012) Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012.  Wit3: Web inventory of transcribed and translated talks.  In _Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT)_, pages 261–268, Trento, Italy.
  * Chan and Fan (2019) Ying-Hong Chan and Yao-Chung Fan. 2019.  BERT for question generation.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 173–177, Tokyo, Japan. Association for Computational Linguistics.
  * Clinchant et al. (2019) Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019.  On the use of BERT for neural machine translation.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 108–117, Hong Kong. Association for Computational Linguistics.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the EACL 2014 Workshop on Statistical Machine Translation_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gale and Church (1993) William A. Gale and Kenneth W. Church. 1993.  A program for aligning sentences in bilingual corpora.  _Computational Linguistics_ , 19(1):75–102.
  * García-Martínez et al. (2016) Mercedes García-Martínez, Loïc Barrault, and Fethi Bougares. 2016\.  Factored neural machine translation architectures.  In _HAL archives ouvertes_.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing bert’s syntactic abilities.  _Computation and Language Research Repository_ , arXiv:1901.05287.  Version 1.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  In _NIPS Deep Learning and Representation Learning Workshop_.
  * Imamura and Sumita (2019) Kenji Imamura and Eiichiro Sumita. 2019.  Recycling a pre-trained BERT encoder for neural machine translation.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 23–31, Hong Kong. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Kuncoro et al. (2018) Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018.  LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1426–1436, Melbourne, Australia. Association for Computational Linguistics.
  * Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004.  ORANGE: a method for evaluating automatic evaluation metrics for machine translation.  In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_ , pages 501–507, Geneva, Switzerland. COLING.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one?  In _Advances in Neural Information Processing Systems_ , pages 14014–14024.
  * Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018.  Scaling neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 1–9, Brussels, Belgium. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Press and Wolf (2017) Ofir Press and Lior Wolf. 2017.  Using the output embedding to improve language models.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 157–163, Valencia, Spain. Association for Computational Linguistics.
  * Rush (2018) Alexander Rush. 2018.  The annotated transformer.  In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_ , pages 52–60, Melbourne, Australia. Association for Computational Linguistics.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5149–5152. IEEE.
  * Sennrich and Haddow (2016) Rico Sennrich and Barry Haddow. 2016.  Linguistic input features improve neural machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers_ , pages 83–91, Berlin, Germany. Association for Computational Linguistics.
  * Smith et al. (2003) George Smith et al. 2003.  A brief introduction to the tiger treebank, version 1.  _Potsdam Universität_.
  * Sundararaman et al. (2019) Dhanasekar Sundararaman, Vivek Subramanian, Guoyin Wang, Shijing Si, Dinghan Shen, Dong Wang, and Lawrence Carin. 2019.  Syntax-infused transformer and bert models for machine translation and natural language understanding.  _Computation and Language Research Repository_ , arXiv:1911.06156.  Version 1.
  * Tenney et al. (2019a) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Tenney et al. (2019b) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b.  What do you learn from context? probing for sentence structure in contextualized word representations.  In _International Conference on Learning Representations_.
  * Tishby and Zaslavsky (2015) Naftali Tishby and Noga Zaslavsky. 2015.  Deep learning and the information bottleneck principle.  In _2015 IEEE Information Theory Workshop (ITW)_ , pages 1–5. IEEE.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Weng et al. (2019) Rongxiang Weng, Heng Yu, Shujian Huang, Shanbo Cheng, and Weihua Luo. 2019.  Acquiring knowledge from pre-trained model to neural machine translation.  _AAAI_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _Computation and Language Research Repository_ , arXiv:1910.03771.  Version 1.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _Computation and Language Research Repository_ , arXiv:1609.08144.  Version 2.
  * Yang et al. (2020) Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Weinan Zhang, Yong Yu, and Lei Li. 2020.  Towards making the most of bert in neural machine translation.  _AAAI_.
  * Zhang and Weiss (2016) Yuan Zhang and David Weiss. 2016.  Stack-propagation: Improved representation learning for syntax.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1557–1566, Berlin, Germany. Association for Computational Linguistics.
  * Zhu et al. (2020) Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020.  Incorporating bert into neural machine translation.  _ICLR 2020_.

##  Appendix A Appendices

###  A.1 Implementation Details

In this section, we provide implementation details that could not be placed in
the main write-up due to space limitations, but we believe are quite helpful
for replication of our work. We divide this section into two parts, one
focused on linguistic aspect vector extraction (Section 4.2) and the other on
linguistic aspect integrated machine translation (Section 4.4).

####  A.1.1 Linguistic Aspect Vector Extraction Implementation Details

The pre-trained spaCy tagger that we used in our experiments is trained on the
word-level while the pre-trained BERT operates on sub-word level232323The
alignment is non-trivial e.g. “hadn’t” is tokenized to “hadn” and “’t” by
spaCy and to “had” and “n’t” by BERT, causing many-to-many alignments.. The
two sequences need to be aligned, so we can assign aspect attributes to BERT
sub-word tokens. Inspired by Gale and Church (1993), we align the two
sequences using a heuristic divide-and-conquer monotonic alignment technique
which finds the parts of the two sequences that are certainly equal and aligns
the parts in between using recursive calls to
itself242424https://github.com/sfu-natlang/SFUTranslate/translate/.

Next, we explain how we implement the aspect extractors. We implement our
aspect extractors using PyTorch framework and initialize them using Xavier
initialization (Glorot and Bengio, 2010). We perform backpropagation using SGD
(initial learning rate of 0.05, momentum value of 0.9, gradient clip norm of
5.0). To cope with inequality in the frequency of the different tags in each
aspect tag set ($t_{a}$, see §3.1), we practice weighted backpropagation with
weights proportional to the inverse frequency of each tag. We decay learning
rate with a factor of 0.9 when the loss value stops improving.

####  A.1.2 Linguistic Aspect Integrated Machine Translation Implementation
Details

We implement our baseline transformer model using the guidelines suggested by
Rush (2018) in our translation toolkit SFUTranslate

Dataset | WMT | IWSLT | M30k
---|---|---|---
N | 6 | 6 | 4
$d_{model}$ | 512 | 256 | 256
$d_{ff}$ | 2048 | 512 | 512
h | 8 | 4 | 4
opt factor | 1 | 2 | 1
opt warmup | 4000 | 8000 | 2000
grad accumulation | 8 | 2 | 1
batch size$\ast$ | 4096 | 4096 | 2560
epochs | 7 | 20 | 20
Table 8: The transformer model settings for each dataset given the training
data size. “N” is the number of layers in both encoder and decoder. Please see
§2 for more information about model parameters.
$\ast$The maximum number of sub-word tokens per batch.

and extend it for implementing the aspect-augmented model as well as the
syntax-infused transformer and transformer with bert-freeze input setting.
Table 8 provides the configuration settings for each of the models used in our
experiments.

We use the pre-trained
WordPiece252525https://github.com/huggingface/tokenizers (Schuster and
Nakajima, 2012) tokenizer packaged and shipped with BERT (containing 31,102
sub-word tokens for German language) to tokenize the source side data, and
tokenize the target side data with
MosesTokenizer262626https://github.com/alvations/sacremoses followed by the
same WordPiece tokenizer model, trained on target data, to split the target
tokens into sub-tokens. We set the target side WordPiece vocabulary size to
30,000 sub-words for English and French. Our models share the vocabulary and
embedding modules of both source and target (Press and Wolf, 2017) since both
source and target are trained in sub-word space. The shared vocabulary sizes
of M30k (German to English), M30k (German to French), IWSLT, and WMT are
16645, 16074, 40807, 47940, respectively.

We generate target sentences using beam search with beam size 4 and length
normalization factor (Wu et al., 2016) of 0.6. We merge the WordPiece tokens
in the generated sentences (a post-processing step to create words) and use
MosesDetokenizer272727https://github.com/alvations/sacremoses to detokenize
the generated outputs. We use Moses recaser282828https://github.com/moses-
smt/mosesdecoder to produce cased translation outputs. We use mteval-v14.pl
script for cased Bleu evaluation.

For all models, we set positional encoding max length to 4096, dropout to 0.1,
loss prediction smoothing to 0.1, and initialize the models using Xavier
initialization (Glorot and Bengio, 2010). We train all models using NoamOpt
optimizer (Rush, 2018) and perform the gradient accumulation trick (Ott et
al., 2018) with one update per a number of batches (Table 8; grad accumulation
) to simulate larger batch sizes on a single GPU.

###  A.2 Additional Analysis of Linguistic Aspect Integrated Machine
Translation Results

In this section, we analyze the results of our aspect integrated translation
experiments. We provide our analysis in two parts, one for small and medium
sized datasets and the other for large ones.

For smaller datasets (containing a few hundred thousand sentence pairs or
less), the broader perspective of BERT knowledge is helpful in limiting the
search space for the model. So using our technique, the translation model
receives more information regarding the general use cases of (locally) rare
words. Linguistic aspect vectors also help the model better understand less
familiar (in comparison to what is frequent in its limited size training data)
syntactic structures in input sentences. This is why we believe aspect vectors
can be helpful in low-resource settings.

Improving models with large amounts of data (with several million sentence
pairs) is a challenging task. The best practice in training neural translation
models is to initialize the embedding module with small random values and let
the model search through the parameter space to find the optimal parameter
settings. Extracted aspect vectors, as an external source of monolingual
knowledge on the source side, are a more reasonable starting point for large
models than random initialization. Integrating aspect vectors thus helps these
models find a better path towards the optimal point(s) and increases the
chances of the model ending up in a more desirable point in search space.