# Slovene SuperGLUE Benchmark: Translation and Evaluation

###### Abstract

We present a Slovene combined machine-human translated SuperGLUE benchmark. We
describe the translation process and problems arising due to differences in
morphology and grammar. We evaluate the translated datasets in several modes:
monolingual, cross-lingual, and multilingual, taking into account differences
between machine and human translated training sets. The results show that the
monolingual Slovene SloBERTa model is superior to massively multilingual and
trilingual BERT models, but these also show a good cross-lingual performance
on certain tasks. The performance of Slovene models still lags behind the best
English models.

Keywords: benchmarks, evaluation, transformer models, natural language
understanding, cross-lingual models, multilingual models

languageresourceLanguage Resources

Slovene SuperGLUE Benchmark: Translation and Evaluation

Aleš Žagar, Marko Robnik-Šikonja
---
University of Ljubljana, Faculty of Computer and Information Science, Večna
pot 113, 1000 Ljubljana,
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

To measure the progress across the fast evolving area of natural language
processing (NLP), several prominent benchmarking suites have been proposed,
such as SentEval [Conneau et al., 2018], GLUE [Wang et al., 2019b], and
SuperGLUE [Wang et al., 2019a]. SentEval aims to evaluate sentence encoders,
while GLUE and its more challenging successor, SuperGLUE, evaluate general
language understanding. These two benchmark suites contain public training
sets and private test sets that can be assessed through an evaluation server
giving separate scores for each of the tasks and an overall single-number
score. The tasks in SuperGLUE are diverse and comprised of question answering
(QA), natural language inference (NLI), coreference resolution, and word sense
disambiguation (WSD). Non-expert humans evaluated all the tasks to provide a
human performance baseline.

Despite recent criticism of the way how the evaluation scores in these
benchmarks are computed (e.g., the arithmetic mean of separate metrics is used
for all tasks of different complexity and different sizes of testing sets)
[Shavrina and Malykh, 2021, Kasai et al., 2021], there is little doubt that
the datasets contained in the benchmarks significantly contribute towards the
progress of the NLP area. Unfortunately, most of the research is on English,
which often limits the generality of the approaches and does not address the
full scale of language complexity. Still, we can observe significant efforts
to make relevant datasets cross-lingual or adapted to more languages. Broad
multilingual benchmarking suites, such as XTREME [Hu et al., 2020] and XGLUE
[Liang et al., 2020], provide machine-translated datasets for several relevant
tasks and languages. Complementary to that, we present an in the direction of
less-resourced languages, namely a combined machine-human translation of
SuperGLUE benchmarking suite to Slovene. We describe the difficulties arising
in the translation and provide the first evaluation of the datasets using
large pretrained BERT-like language models. Our evaluation encompasses
monolingual, cross-lingual and multilingual approaches, comparison of human
and machine-translated parts, and comparison of two state-of-the-art machine
translation systems.

The paper is structured into further five sections. In Section 2, we present
related work on non-English benchmarking suites. In Section 3, we describe the
translation process and the final SuperGLUE datasets in Slovene. We describe
three evaluation settings (monolingual, cross-lingual, and multilingual) in
Section 4, and present the results in Section 5. Section 6 presents
conclusions, limitations of our work, and ideas for further improvements. To
make the paper self-contained and ease understanding of the different datasets
in the SuperGLUE benchmark, we follow ?) and present an example for each of
the tasks in Appendix.

## 2\. Related work

In the area of NLP, there are several prominent benchmarking suites, the most
prominent being GLUE [Wang et al., 2019b] and SuperGLUE [Wang et al., 2019a].
While most benchmarking datasets are available in English, there are a few
notable attempts to extend them to other languages.

XTREME [Hu et al., 2020] is a multi-task benchmark for evaluating the cross-
lingual representations across 40 languages and 9 tasks. It focuses on the
zero-shot cross-lingual transfer and provides large English training sets,
much smaller testing sets in target languages (from 5 to 40 human translated
or checked, the rest are machine translated without human interventions). The
suite covers tasks such as natural language inference (NLI), paraphrasing,
part-of-speech (POS) tagging, named entity recognition (NER), question
answering, and parallel sentence extraction and detection.

XGLUE [Liang et al., 2020] is intended to evaluate cross-lingual pretrained
models on 11 tasks: NER, POS-tagging, question answering, paraphrasing, ad
relevance, web page relevance, question-answer matching, question generation,
and news title generation. Similarly to XTREME, the suite provides training
data in English and tests the cross-lingual models on testing data on several
target languages (from 3 to 18).

SuperGLUE is likely the most prominent benchmarking suite for English. It
follows the design of the GLUE suite, consisting of a public leaderboard for
eight tasks, which are evaluated individually and jointly. The suite provides
public training and development datasets, while testing data is hidden and
only used to evaluate predictions submitted to the leaderboard. The benchmark
includes QA, NLI, coreference resolution, and WSD tasks, for which non-expert
human baselines are provided.

Following the English example, Russian SuperGLUE [Shavrina et al., 2020] was
developed independently or manually translated from English SuperGLUE. It
contains nine tasks: linguistic diagnostic, WSD, QA, NLI, and coreference
resolution.

Slovene language possesses significantly fewer resources than English or
Russian and is not included in the XTREME and XGLUE massively cross-lingual
benchmarks. We aimed to provide a modern natural language understanding suite
for this less-resourced language using modest resources at our disposal. For
that purpose, we followed the English SuperGLUE design and translated the
entire datasets (except WiC) to Slovene, mainly using machine translation and
in small part human translation (120,000 words). This allows testing of
monolingual, cross-lingual and multilingual approaches. Keeping the format of
the original SuperGLUE, our datasets can be evaluated with the original
SuperGLUE leaderboard and can be compared to English baselines and state-of-
the-art.

## 3\. Slovene SuperGLUE translation

To evaluate cross-lingual transfer and test specifics of morphologically rich
languages, we translated the SuperGLUE datasets to Slovene. Due to limited
funds, we partially used human translation (HT) and partially machine
translation (MT). Altogether, approximately 120,000 words were human
translated. Some datasets are too large (BoolQ, MultiRC, ReCoRD, RTE) to be
fully human translated. We thus provide ratios between the human translated
and the original English sizes in Table 1. For MT from English to Slovene, we
used the GoogleMT Cloud service. In our evaluation, we use six of the original
eight tasks. As explained below, we excluded ReCoRD and WiC.

Table 1: Number of instances in the original English and translated Slovene
SuperGLUE tasks. HT stands for human translation and MT for machine
translation. The “ratio“ indicates the ratio between the number of human
translated instances and all instances.

Dataset | split | English | HT | ratio | MT
---|---|---|---|---|---
BoolQ | train | 9427 | 92 | 0.0098 | yes
| val | 3270 | 18 | 0.0055 | yes
| test | 3245 | 30 | 0.0092 | yes
CB | train | 250 | 250 | 1.0000 | yes
| val | 56 | 56 | 1.0000 | yes
| test | 250 | 250 | 1.0000 | yes
COPA | train | 400 | 400 | 1.0000 | yes
| val | 100 | 100 | 1.0000 | yes
| test | 500 | 500 | 1.0000 | yes
MultiRC | train | 5100 | 15 | 0.0029 | yes
| val | 953 | 3 | 0.0031 | yes
| test | 1800 | 30 | 0.0167 | yes
ReCoRD | train | 101000 | 60 | 0.0006 | /
| val | 10000 | 6 | 0.0006 | /
| test | 10000 | 30 | 0.0030 | /
RTE | train | 2500 | 232 | 0.0928 | yes
| val | 278 | 29 | 0.1043 | yes
| test | 300 | 29 | 0.0967 | yes
WiC | train | 6000 | / | / | /
| val | 638 | / | / | /
| test | 1400 | / | / | /
WSC | train | 554 | 554 | 1.0000 | /
| val | 104 | 104 | 1.0000 | /
| test | 146 | 146 | 1.0000 | /

We decided to use only the HT translated part of our test sets in our
evaluations to avoid noise due to translations. This makes some test sets much
smaller compared to English test sets. We did not include ReCoRD in the
Slovene benchmark due to the low quality of our test set, consisting of
confusing and ambiguous examples. Further, there are differences between
English and Slovene ReCoRD tasks due to the morphological richness of Slovene.
Namely, in Slovene, the correct declension of a query is often not present in
the text, making it impossible to provide the right answer. Finally, similarly
to WSC (discussed below), ReCoRD is also affected by the problem of
translating HTML tags with GoogleMT.

The WiC task cannot be translated and would have to be conceived anew because
it is impossible to transfer the same set of meanings of a given word from
English to a target language. Taking the WiC example in Table 7, we note that
the word _board_ in two different contexts translates to two completely
different words in Slovene (Context 1: Bivanje in _hrana_. Context 2: Čez okna
je nabil _deske_.).

The WSC dataset cannot be machine-translated because it requires human
assistance and verification. First, GoogleMT translations cannot handle the
correct placement of HTML tags indicating coreferences. The second reason is
that in Slovene coreferences can also be expressed with verbs, while
coreferences in English are mainly nouns, proper names and pronouns. This
makes the task different in Slovene compared to English. On the one hand, the
task is more difficult in Slovene because solutions cover more types of words;
on the other hand, the Slovene verbs might reveal the coreference information
for some instances.

## 4\. Evaluation settings

SuperGLUE benchmark is extensively used to compare large pretrained models in
English111https://super.gluebenchmark.com/leaderboard. In contrast to that, we
concentrate on the Slovene translation of the SuperGLUE tasks and new
opportunities which arise from that.

We compare four BERT models available for Slovene: monolingual Slovene
SloBERTa [Ulčar and Robnik-Šikonja, 2021], trilingual (Croatian-Slovene-
English) CroSloEngual BERT [Ulčar and Robnik-Šikonja, 2020], massively
multilingual mBERT [Devlin et al., 2019] (bert-base-multilingual-
cased222https://huggingface.co/bert-base-multilingual-cased), and XLM-R
[Conneau et al., 2019] (xlm-roberta-base333https://huggingface.co/xlm-roberta-
base).

The single-number overall average score (i.e. Avg in the second column of,
e.g., Table 2) comprises equally weighted tasks. For English, there are eight
tasks; for Slovene, there are all six translated tasks: BoolQ, CB, COPA,
MultiRC, RTE, and WSC. In tasks with multiple metrics, we averaged those
metrics to get a single task score. For the details on how the score is
calculated for each task, see [Wang et al., 2019a].

We test models in three settings. The monolingual setting uses the same
language (Slovene or English) for training and testing. In the cross-lingual
setting, we tested the cross-lingual models (CroSloEngual, mBERT, and XLM-R)
and transfer between English and Slovene datasets in both directions. In the
multilingual setting, the models were trained on the combined full size
English and Slovene data. The results for the three settings as well as a
comparison of two MT systems (GoogleMT and DeepL) are reported in Section 5.

We fine-tuned the available Slovene BERT models on SuperGLUE tasks using the
Jiant tool [Phang et al., 2020]. We used a single-task learning setting for
each task and fine-tuned models for 100 epochs, with the initial learning rate
of $10^{-5}$. Each model was fine-tuned using the dataset corresponding to one
of the three settings.

## 5\. Results

We first report the results for each of the three settings (monolingual,
cross-lingual, and multilingual) separately. We end the section with the
comparison between human and machine-translated data and the comparison of
machine translation systems.

Table 2: The SuperGLUE benchmarks in English (upper part) and Slovene (lower
part). All English results are taken from [Wang et al., 2019a]. The best score
for each task and language is in bold. The best average scores (Avg) for each
language are underlined. *MultiRC has multiple test sets released on a
staggered schedule, and these results evaluate on an installation of the test
set that is a subset of ours.

Task | Avg | BoolQ | CB | COPA | MultiRC | ReCoRD | RTE | WiC | WSC
---|---|---|---|---|---|---|---|---|---
Models/Metrics |  | Acc. | F1/Acc. | Acc. | F1a/EM | F1/EM | Acc. | Acc. | Acc.
Most Frequent | 45.7 | 62.3 | 21.7/48.4 | 50.0 | 61.1/0.3 | 33.4/32.5 | 50.3 | 50.0 | 65.1
CBoW | 44.7 | 62.1 | 49.0/71.2 | 51.6 | 0.0/0.4 | 14.0/13.6 | 49.7 | 53.0 | 65.1
BERT | 69.3 | 77.4 | 75.7/83.6 | 70.6 | 70.0/24.0 | 72.0/71.3 | 71.6 | 69.5 | 64.3
BERT++ | 73.3 | 79.0 | 84.7/90.4 | 73.8 | 70.0/24.1 | 72.0/71.3 | 79.0 | 69.5 | 64.3
Human (est.) | 89.8 | 89.0 | 95.8/98.9 | 100.0 | 81.8*/51.9* | 91.7/91.3 | 93.6 | 80.0 | 100.0
Most Frequent | 49.1 | 63.3 | 21.7/48.4 | 50.0 | 76.4/0.6 | - | 58.6 | - | 65.8
SloBERTa | 63.9 | 66.6 | 74.0/76.8 | 61.8 | 62.7/21.9 | - | 62.1 | - | 73.3
CroSloEngual | 57.8 | 66.6 | 62.1/72.4 | 58.2 | 56.7/15.6 | - | 62.1 | - | 56.2
mBERT | 59.1 | 70.0 | 66.6/73.6 | 54.2 | 57.4/16.3 | - | 62.1 | - | 61.6
XML-R | 58.7 | 76.7 | 66.2/73.2 | 50.0 | 55.3/13.9 | - | 55.2 | - | 65.8

### 5.1. Monolingual results

Our monolingual analysis compares different Slovene prediction models on the
complete datasets, composed of existing human translated instances while the
remaining instances are machine-translated. Table 2 shows the results together
with several baselines trained on the original English datasets. Some
comparisons to English baselines are questionable as the Slovene models are
trained on only a small fraction of better quality HT data (BoolQ, MultiRC)
and tested on smaller set of HT test data. In the case of the BERT++ model,
the English model was additionally pretrained with transfer learning tasks
similar to a target one (CB, RTE, BoolQ, COPA). In terms of datasets, the only
fair comparison is possible with the CB, COPA, and WSC.

Considering the Avg scores in Table 2, the monolingual SloBERTa is the best
performing Slovene model. On average, all Slovene BERT models perform better
than the Most Frequent baseline. Concerning individual tasks, none of the
Slovene models exceeds the Most Frequent baseline in the MultiRC task.
SloBERTa was significantly better than the rest of the models in CB, COPA, and
WSC, while XML-R was the best on BoolQ.

Compared to English models, the best Slovene model (SloBERTa) achieved better
results on WSC. It seems that none of the English models learned anything from
WSC (they are below the Most Frequent baseline), but the SloBERTa model
achieved the score of 73.3 (the Most Frequent baseline gives 65.8). The
success of SloBERTa on WSC might stem from the morphology of Slovene verbs,
which include the information on the gender; this information is beneficial in
coreference resolution and makes some instances easier in Slovene than in
English. Nevertheless, there is still a large gap compared to human
performance. All Slovene models showed good performance on CB and fell between
English CBoW and BERT.

### 5.2. Cross-lingual results

In the cross-lingual scenario, we tested the three multilingual BERT models
(mBERT, CroSloEngual, XLM-R) and the transfer between English and Slovene
datasets (both directions). For Slovene as the source language, we used the
available human translated examples. To make the comparison balanced, we only
used the same examples from the English datasets. We tested both zero-shot
transfer (no training data in the target language) and few-shot transfer. In
the few-shot training, we used 10 additional examples from the target language
for each task. We randomly sampled these 10 examples for 5 times and reported
averages to achieve more statistically valid results. The fine-tuning
hyperparameters are the same as in the monolingual setup.

Table 3: Cross-lingual results on human translated SuperGLUE test sets. The
best results for zero-shot and few-shot scenarios are in bold.

Evaluation | Model | source | target | Avg | BoolQ | CB | COPA | MultiRC | RTE | WSC
---|---|---|---|---|---|---|---|---|---|---
| acc. | F1/acc. | Acc. | F1a/EM | Acc. | Acc.
Zero-shot | CroSloEngual | english | slovene | 49.8 | 56.7 | 43.7/60.0 | 54.6 | 48.0/6.6 | 58.6 | 50.7
slovene | english | 52.6 | 60.0 | 53.8/70 | 59.6 | 56.7/9.6 | 48.3 | 58.2
mBERT | english | slovene | 47.4 | 56.7 | 36.2/57.2 | 50.2 | 47.3/8.7 | 55.2 | 64.4
slovene | english | 48.3 | 60.0 | 44.6/50.4 | 49.8 | 56.2/8.7 | 51.7 | 57.5
XLM-R | english | slovene | 53.8 | 63.3 | 62.9/68.4 | 53.6 | 48.5/0.3 | 62.1 | 56.2
slovene | english | 51.7 | 63.3 | 59.1/67.2 | 47.2 | 52.9/12.9 | 51.7 | 65.8
Few-shot | CroSloEngual | english | slovene | 54.4 | 60.0 | 52.4/68.6 | 55.0 | 52.8/9.72 | 65.5 | 54.1
slovene | english | 53.0 | 60.0 | 53.8/70.0 | 59.5 | 56.0/12.1 | 49.7 | 58.2
mBERT | english | slovene | 50.9 | 60.1 | 53.1/66.2 | 50.4 | 50.8/9.8 | 53.8 | 64.4
slovene | english | 51.3 | 60.7 | 51.8/58.2 | 50.3 | 57.2/11.1 | 56.5 | 56.8
XLM-R | english | slovene | 57.0 | 63.3 | 65.8/69.8 | 53.3 | 76.4/0.6 | 62.1 | 57.4
slovene | english | 53.0 | 63.3 | 63.0/69.6 | 48.3 | 51.4/10.6 | 55.8 | 65.8
| Most frequent |  |  | 52.4 | 63.3 | 23.0/52.7 | 50.0 | 77.3/0.3 | 58.6 | 65.8

Table 4: Multilingual results on English and Slovene (human translated) test sets. Models were trained on combined full size English and Slovene data. Slovene data is comprised of human translated and machine translated data. The best results for each task is in bold.  Evaluation on | Model | Avg | BoolQ | CB | COPA | MultiRC | RTE | WSC
---|---|---|---|---|---|---|---|---
|  |  | Acc. | F1/acc. | Acc | F1a/EM | Acc. | Acc.
Slovene | CroSloEngual | 59.8 | 70.0 | 67.7/74.7 | 59.4 | 58.4/15.6 | 51.7 | 58.2
mBERT | 60.2 | 73.0 | 66.5/71.9 | 51.6 | 57.5/17.0 | 62.1 | 58.9
XML-R | 59.9 | 63.3 | 69.9/74.7 | 52.8 | 58.8/18.8 | 58.6 | 61.0
English | CroSloEngual | 59.9 | 63.3 | 67.3/75.5 | 62.4 | 59.5/16.7 | 55.2 | 57.5
mBERT | 64.2 | 76.7 | 69.9/74.7 | 58.6 | 60.4/21.5 | 65.5 | 63.0
XML-R | 61.4 | 70.0 | 74.1/79.9 | 51.8 | 60.1/19.4 | 48.3 | 65.8
| Most frequent | 52.4 | 63.3 | 23.0/52.7 | 50 | 77.3/0.3 | 58.6 | 65.8
Table 5: Comparing human translation (HT) with machine translation (MT) on the Slovene SuperGLUE benchmark. Note that for this experiment the average score Avg is computed as the average of the five listed tasks and not six as in Table 2. The best score for each task is in bold. Task | Avg | BoolQ | CB | COPA | MultiRC | RTE
---|---|---|---|---|---|---
Models/Metrics |  | Acc. | F1/Acc. | Acc. | F1a/EM | Acc.
Most Frequent | 49.1 | 63.3 | 21.7/48.4 | 50.0 | 76.4/0.6 | 58.6
HT-mBERT | 54.3 | 63.3 | 66.6/73.6 | 54.2 | 45.1/8.1 | 57.2
MT-mBERT | 55.2 | 63.3 | 65.1/68.8 | 54.4 | 55.4/11.7 | 57.9
HT-CroSloEngual | 55.6 | 63.3 | 62.1/72.4 | 58.2 | 53.0/8.4 | 58.6
MT-CroSloEngual | 53.4 | 63.3 | 59.8/68.4 | 55.0 | 51.2/10.5 | 53.8
HT-SloBERTa | 57.2 | 63.3 | 74.0/76.8 | 61.8 | 53.0/10.8 | 53.8
MT-SloBERTa | 55.8 | 63.3 | 68.6/74.8 | 58.2 | 57.1/12.0 | 49.6
HT-XLM-R | 53.5 | 63.3 | 66.2/73.2 | 50.0 | 53.3/0.9 | 57.2
MT-XLM-R | 50.1 | 63.3 | 62.0/68.4 | 51.4 | 55.3/0.6 | 42.8
HT-Avg | 55.1 | 63.3 | 70.6 | 56.0 | 29.1 | 56.7
MT-Avg | 53.6 | 63.3 | 67.0 | 54.8 | 31.7 | 51.0

The results are presented in Table 3. Averaged over all tasks, some models
improved the Most frequent baseline. In general, they were quite unsuccessful
on BoolQ, MultiRC, and WSC but showed some promising results on COPA, RTE, and
especially CB. Additional training examples in the few-shot scenario brought
some visible improvements. It seems that models perform better in the English-
Slovene direction than vice versa. The best performing model is XLM-R,
followed by CroSloEngual BERT and mBERT.

The low overall performance can be explained by a low number of training
examples in the source language. If we take a closer look at specific models,
we can observe that XLM-R shows very good results on CB in both directions.
CroSloEngual BERT achieved a similar outstanding result on COPA. It is the
only model that learned something on this dataset as well.

We can conclude that for the difficult SuperGLUE benchmark, the cross-lingual
transfer is challenging but not impossible. In the future, we plan to expand
the current set of experiments in several directions. First, we will train
English models on the complete SuperGLUE datasets and transfer them to Slovene
human and machine-translated datasets. Second, we will train Slovene models on
the combined machine and human translated datasets and transfer them to
complete English datasets. Finally, we will also combine training for several
tasks and test transfer learning scenarios.

### 5.3. Multilingual results

In the multilingual setting, the three multilingual BERT models (CroSloEngual
BERT, mBERT, and XLM-R) were trained on the combined full-size English and
Slovene data. Slovene data is comprised of human-translated and machine-
translated data. For Slovene, the models are tested on only HT data. The
results are reported in Table 4.

Interestingly, all the best scores for all tasks were achieved when tested on
English testing sets (the training sets were identical and comprised of both
languages). This might be due to the fact that Slovene BERT models were
pretrained on lower amounts of data and the quality of the Slovene
translation.

The overall best model with the highest Avg score is mBERT (best in BoolQ,
MultiRC and RTE), followed by XLM-R (best in CB and WSC) and CroSloEngual BERT
(best in COPA). In the MultiRC and WSC all models lag behind the Most Frequent
baseline.

### 5.4. Comparing human and machine translation

To test the difference in human and machine translation, we repeated the
monolingual experiments separately on human translated (HT) data and the same
machine translated (MT) instances (the sizes of these HT datasets are reported
in Table 1). Each model was fine-tuned using either MT or HT datasets of the
same size. Only the translated content varies between both translation types;
otherwise, they contain exactly the same examples. The splits of instances
into train, validation and test sets is the same as in the English variant
(but mostly considerably smaller, see Table 1). WSC is excluded from this
evaluation as it can only be human translated, so the average score (Avg) is
computed from the five remaining tasks. Table 5 shows the results. The
comparison with results in Table 2 is not fair because the models there used
significantly more training examples.

Considering the Avg scores in Table 5, the monolingual SloBERTa is again the
best performing Slovene model and all BERT models, regardless of translation
type, perform better than the Most Frequent baselines. From the translation
type perspective, the models trained on HT datasets perform better than those
trained on MT datasets by 1.5 points. The only task where MT is better than HT
is MultiRC, but looking at single scores, we can observe that none of the
models learned anything in this task as there is a large gap between the Most
Frequent baseline and the rest of the models.

Analysis of specific tasks shows that For the BoolQ dataset, all models
predict the most frequent class (the testing set might be too small for
reliable conclusions in BoolQ). We can safely assume that training sample
sizes are too small in BoolQ and MultiRC (none of the models learned anything)
and must be increased (we have only 92 HT examples in BoolQ and 15 HT examples
in MultiRC). The same is also true for RTE.

### 5.5. Comparing machine translation systems

To check the effect of MT systems on the performance of models, we perform a
separate study in the monolingual setting using two currently best MT systems
for Slovene: GoogleMT444https://translate.google.com/ and
DeepL555https://www.deepl.com/translator.

Table 6: Comparison of GoogleMT and DeepL MT systems on the CB and COPA
datasets. The best score for each task is in bold.

| GoogleMT | DeepL
---|---|---
Model | CB | COPA | CB | COPA
SloBERTa | 68.6/74.8 | 58.2 | 73.6/79.1 | 54.2
CroSloEngual | 59.8/68.4 | 55.0 | 66.1/75.1 | 53.8
mBERT | 65.1/68.8 | 54.4 | 60.4/70.7 | 50.8
XML-R | 62.0/68.4 | 51.4 | 44.3/63.5 | 53.6

We compare all four Slovene BERT models on two datasets where all models
performed better than the Most Frequent baseline, CB and COPA (WSC needs
manual translation as discussed in Section 3). Results in Table 6 show
surprisingly large differences between the two MT systems. DeepL gives
favourable results on the CB dataset, while the inverse is true on the CB
dataset. We cannot explain the differences and leave a more detailed analysis
of the issue for further work.

## 6\. Conclusions

We prepared the Slovene translation of natural language understanding
benchmark suite SuperGLUE and released it under an open-source licence
SloSuperGLUE666http://hdl.handle.net/11356/1380. We described the translation
process and obstacles in the transfer to a morphologically rich language. The
partially machine and partially human translated datasets were used in the
assessment of four BERT-based models available for Slovene. The results show
that the monolingual SloBERTa model is currently the best performing Slovene
pretrained model. However, the performance of Slovene models is still
significantly worse compared to the state-of-the-art English models, showing
considerable potential for improvement of NLP approaches for less-resourced
languages.

Our analyses show that the models’ performance improved with human translated
datasets, and in future, we intend to increase the share of human translated
data. For two of the English SuperGLUE datasets, the MT was not possible. We
intend to create a Slovene version of the WiC task from scratch using word
sense disambiguation tasks and manually adapt the ReCoRD task to get the full
SuperGLUE suite. In the WSC tasks, Slovene verb morphology leaks some
coreference information. We intend to analyze this issue and form a more
challenging Slovene WSC. Next, both Slovene and Russian are Slavic languages,
and therefore more similar to each other than English. It would be interesting
to combine equivalent Russian and Slovene tasks from the translated SuperGLUE
tasks in further multilingual and cross-lingual experiments. Finally, as we
kept the format of the original SuperGLUE benchmarks, Slovene datasets can be
evaluated with the original SuperGLUE leaderboard. While this allows
comparison to English models, the obtained results are not publicly available.
We intend to prepare a separate Slovene leaderboard and encourage the NLP
community to pay attention to less-resourced languages.

## Acknowledgements

The work was partially supported by the Slovenian Research Agency (ARRS) core
research programmes P6-0411, as well as the Ministry of Culture of Republic of
Slovenia through project Development of Slovene in Digital Environment (RSDO).
This paper is supported by European Union’s Horizon 2020 research and
innovation programme under grant agreement No 825153, project EMBEDDIA (Cross-
Lingual Embeddings for Less-Represented Languages in European News Media). We
thank the authors of SuperGLUE benchmark for sharing test set answers for some
of the tasks.

## Appendix

Table 7: Examples from the development set of SuperGLUE tasks. Bold texts represent parts of examples’ format. Texts in italics are part of models’ input. Underlined texts are specially marked in inputs. Texts in the monospaced font represent the expected models’ outputs. The WiC task cannot be translated into other languages as different meanings of a single word would translate into different words. The ReCoRD cannot be handled with MT systems as they do not preserve tagged words. Further, in morphologically rich languages such as Slovene, the correct answer would not necessarily use the same word form as the one tagged. BoolQ |  Passage:Barq’s – Barq’s is an American soft drink. Its brand of root beer is notable for having caffeine. Barq’s, created by Edward Barq and bottled since the turn of the 20th century, is owned by the Barq family but bottled by the Coca-Cola Company. It was known as Barq’s Famous Olde Tyme Root Beer until 2012.
---|---
Question:is barq’s root beer a pepsi product
Answer:No
CB |  Text:B: And yet, uh, I we-, I hope to see employer based, you know, helping out. You know, child, uh, care centers at the place of employment and things like that, that will help out. A: Uh-huh. B: What do you think, do you think we are, setting a trend?
Hypothesis:they are setting a trend
Entailment:Unknown
COPA |  Premise:My body cast a shadow over the grass.
Question:What’s the CAUSE for this
Alternative 1:The sun was rising.
Alternative 2:The grass was cut.
Correct Alternative:1
MultiRC |  Paragraph:Susan wanted to have a birthday party. She called all of her friends. She has five friends. Her mom said that Susan can invite them all to the party. Her first friend could not go to the party because she was sick. Her second friend was going out of town. Her third friend was not so sure if her parents would let her. The fourth friend said maybe. The fifth friend could go to the party for sure. Susan was a little sad. On the day of the party, all five friends showed up. Each friend had a present for Susan. Susan was happy and sent each friend a thank you card the next week.
Question:Did Susan’s sick friend recover?
Candidate answers: Yes, she recoverd, No (F), Yes (T), No, she didn’t recover
(F), Yes, she was at Susan’s party (T)
ReCoRD |  Paragraph:(CNN) Puerto Rico on Sunday overwhelmingly voted for statehood. But Congress, the only body that can approve new states, will ultimately decide whether the status of the US commonwealth changes. Ninety-seven percent of the votes in the nonbinding referendum favored statehood, an increase over the results of a 2012 referendum, official results from the State Electoral Commission show. It was the fifth such vote on statehood. Today, we the people of Puerto Rico are sending a strong and clear message to the US Congress … and to the world … claiming our equal rights as American citizens, Puerto Rico Gov. Ricardo Rossello said in a news release.@highlight Puerto Rico voted Sunday in favor of US statehood
Query:For one, they can truthfully say, ”Don’t blame me, I didn’t vote for
them,” when discussing the $\langle$placeholder$\rangle$ presidency.
Correct Entities: US
RTE |  Text:Dana Reeve, the widow of the actor Christopher Reeve, has died of lung cancer at age 44, according to the Christopher Reeve Foundation.
Hypothesis:Christopher Reeve had an accident.
Entailment: False
WiC |  Context 1:Room and board.
Context 2:He nailed boards across the windows.
Sense match: False
WSC |  Text:Mark told Pete many lies about himself, which Pete included in his book. He should have been more truthful.
Coreference: False

## 7\. Bibliographical References

## References

  * Conneau et al., 2018 Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and Jégou, H.  (2018).  Word translation without parallel data.  In Proceedings of International Conference on Learning Representation ICLR.
  * Conneau et al., 2019 Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V.  (2019).  Unsupervised cross-lingual representation learning at scale.  arXiv preprint arXiv:1911.02116.
  * Devlin et al., 2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
  * Hu et al., 2020 Hu, J., Ruder, S., Siddhant, A., Neubig, G., Firat, O., and Johnson, M.  (2020).  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In International Conference on Machine Learning, pages 4411–4421. PMLR.
  * Kasai et al., 2021 Kasai, J., Sakaguchi, K., Bras, R. L., Dunagan, L., Morrison, J., Fabbri, A. R., Choi, Y., and Smith, N. A.  (2021).  Bidimensional leaderboards: Generate and evaluate language hand in hand.  ArXiv preprint 2112.04139.
  * Liang et al., 2020 Liang, Y., Duan, N., Gong, Y., Wu, N., Guo, F., Qi, W., Gong, M., Shou, L., Jiang, D., Cao, G., et al.  (2020).  Xglue: A new benchmark datasetfor cross-lingual pre-training, understanding and generation.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008–6018.
  * Phang et al., 2020 Phang, J., Yeres, P., Swanson, J., Liu, H., Tenney, I. F., Htut, P. M., Vania, C., Wang, A., and Bowman, S. R.  (2020).  jiant 2.0: A software toolkit for research on general-purpose text understanding models.  http://jiant.info/.
  * Shavrina and Malykh, 2021 Shavrina, T. and Malykh, V.  (2021).  How not to lie with a benchmark: Rearranging NLP leaderboards.  ArXiv preprint 2112.01342.
  * Shavrina et al., 2020 Shavrina, T., Fenogenova, A., Anton, E., Shevelev, D., Artemova, E., Malykh, V., Mikhailov, V., Tikhonova, M., Chertok, A., and Evlampiev, A.  (2020).  RussianSuperGLUE: A Russian language understanding evaluation benchmark.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4717–4726.
  * Ulčar and Robnik-Šikonja, 2021 Ulčar, M. and Robnik-Šikonja, M.  (2021).  SloBERTa: Slovene monolingual large pretrained masked language model.  In Proceedings of Data Mining and Data Warehousing, SiKDD.
  * Ulčar and Robnik-Šikonja, 2020 Ulčar, M. and Robnik-Šikonja, M.  (2020).  FinEst BERT and CroSloEngual BERT: less is more in multilingual models.  In Proceedings of Text, Speech, and Dialogue, TSD 2020, pages 104–111.
  * Wang et al., 2019a Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  (2019a).  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  Advances in Neural Information Processing Systems, 32.
  * Wang et al., 2019b Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  (2019b).  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In International Conference on Learning Representations.

## 8\. Language Resource References

lrec2022-bib languageresource


# GMP$\scriptstyle\bigstar$ : Well-Tuned Gradual Magnitude Pruning Can
Outperform Most BERT-Pruning Methods

Eldar Kurtic  Corresponding author<EMAIL_ADDRESS>Institute of
Science and Technology Austria  Dan Alistarh  Institute of Science and
Technology Austria  Neural Magic Inc.

###### Abstract

We revisit the performance of the classic gradual magnitude pruning (GMP)
baseline for large language models, focusing on the classic BERT benchmark on
various popular tasks. Despite existing evidence in the literature that GMP
performs poorly, we show that a simple and general variant, which we call
GMP$\scriptstyle\bigstar$ , can match and sometimes outperform more complex
state-of-the-art methods. Our results provide a simple yet strong baseline for
future work, highlight the importance of parameter tuning for baselines, and
even improve the performance of the state-of-the-art second-order pruning
method in this setting.

##  1 Introduction

The massive recent growth of the computational cost of accurate deep learning
models, in particular large language models (LLMs), has motivated the
development of several advanced model compression techniques (Hoefler et al.,
2021; Gholami et al., 2021), encompassing unstructured and structured pruning,
quantization, and knowledge distillation. In this paper, we focus on the
unstructured pruning, for which we follow the standard pipeline. Such models
are first _pre-trained_ on a large _upstream_ corpus of unlabelled text. Then,
they are _fine-tuned_ in a supervised manner on a smaller _downstream_ task,
such as question-answering or text classification. In the context of
compression, this pipeline led to two paradigms: 1) _upstream pruning_ ,
followed by fine-tuning of the remaining weights on a downstream task, and 2)
_downstream pruning_ , pruning and fine-tuning directly on the downstream
task.

A tempting baseline approach in most settings is _gradual magnitude pruning
(GMP)_ (Hagiwara, 1994; Zhu and Gupta, 2017), that is, periodically removing
the smallest fraction of weights during training, possibly interspersed with
fine-tuning steps designed to recover accuracy. GMP has been shown to be an
extremely strong baseline in the context of computer vision (Gale et al.,
2019; Hoefler et al., 2021). However, the literature on pruning LLMs, and in
particular BERT models Sanh et al. (2020); Chen et al. (2020); Zafrir et al.
(2021), clearly suggests that GMP _does not_ perform well.

Figure 1: Performance of state-of-the-art unstructured pruning methods
relative to the dense $\textrm{BERT}_{{\textrm{BASE}}}\,$model at high
sparsities and two tasks, SQuADv1.1 and MNLI.

##### Contribution.

In this paper, we re-examine this conclusion and investigate whether GMP can
be a competitive baseline, once carefully tuned. Specifically, we show that a
well tuned variant which we call GMP$\scriptstyle\bigstar$ , can produce
highly accurate and sparse language models in both upstream and downstream
pruning regimes, matching or even outperforming more complex methods. We
explore effects of the crucial parameters for gradual pruning, and provide
simple and intuitive guidelines on how to integrate them in a principled
manner.

Our results are summarized in Figure 1, which presents performance of state-
of-the-art unstructured pruning techniques on two benchmarks. Specifically, we
compare GMP$\scriptstyle\bigstar$ with the Lottery Ticket approach (Chen et
al., 2020), Movement Pruning (MvP) (Sanh et al., 2020) (as well as its GMP
baseline $\textnormal{GMP}_{\textnormal{MvP}}$), upstream Prune OFA (Zafrir et
al., 2021), as well as the recently-proposed second-order pruning oBERT
(Kurtic et al., 2022). We observe that: 1) for both benchmarks,
GMP$\scriptstyle\bigstar$ is only second to the more complex oBERT method; 2)
GMP$\scriptstyle\bigstar$ in fact outperforms the highly competitive Prune OFA
and MvP methods; and 3) GMP$\scriptstyle\bigstar$ outperforms both Lottery
Tickets and $\textnormal{GMP}_{\textnormal{MvP}}$ by extremely wide margins.

##### Prior Work.

Following the vast BERT-pruning literature, we focus on the unstructured
pruning of the $\textrm{BERT}_{{\textrm{BASE}}}\,$model (Devlin et al., 2019).
As previously noted, upstream and downstream pruning paradigms exist, and
methods are usually developed and specialized for only one of the two. For
example, Movement Pruning (MvP) (Sanh et al., 2020; Lagunas et al., 2021) for
downstream pruning and Prune Once for All (Prune OFA) (Zafrir et al., 2021)
for upstream pruning. Simplicity and generality of the GMP makes it suitable
for both paradigms, without any regime-specific modifications. New and more
advanced pruning techniques, which are, contrary to GMP, able to leverage
gradients (Sanh et al., 2020; Lagunas et al., 2021), loss curvature (Kurtic et
al., 2022), compute-intensive pre-training setup (Zafrir et al., 2021) are
built on the premise that the simple magnitude-based GMP method falters when
applied to BERT-pruning. In this work, contrary to what is currently available
in the literature, we present empirical evidence that GMP, when tuned
carefully, can produce very accurate sparse models which are competitive or
even better than most state-of-the-art pruning techniques across both regimes
(upstream and downstream). As can be seen from Figure 1 and our later results,
we massively improve upon existing GMP-based pruning baselines, in some cases
by even more than 20 accuracy points.

##  2 Competitive Gradual Magnitude Pruning (GMP$\scriptstyle\bigstar$ )

##### Experimental setup.

We focus our attention on the standard
$\textrm{BERT}_{{\textrm{BASE}}}\,$model, composed of embedding and encoder
layers, which has approximately 110M parameters. All methods focus on pruning
among approximately 85M weights of encoder layers and report sparsities with
respect to that number. We evaluate models on the validation split of the
respective dataset, and to improve confidence in the obtained results we
perform multiple runs with different seeds and report mean performance.

###  2.1 Downstream pruning

Following the literature, we consider three popular tasks: question-answering
SQuADv1.1 (Rajpurkar et al., 2016), recognition of textual entailment MNLI
(Williams et al., 2017), and duplicate question detection QQP (Iyer et al.,
2017). Now, we reflect upon the most important constituents of the gradual
pruning framework that enabled us to attain massive improvements.

##### Sparsity schedule.

In all of our gradual runs, there is no pruning during the first two and the
last two epochs. The former fine-tunes the pre-trained model, and the latter
fine-tunes the sparse model with the fixed mask. In between the two,
GMP$\scriptstyle\bigstar$ follows the cubic sparsity scheduler (Zhu and Gupta,
2017) and prunes weights with the frequency of ten times per epoch. Motivated
by the fact that $\textrm{BERT}_{{\textrm{BASE}}}\,$is heavily
overparametrized for downstream tasks, we deviate from the standard cubic
schedule by introducing a large first pruning step. This showed to be of a
crucial importance when pruning the model to high target sparsities (e.g. 97%)
as it leaves more time to recover from later pruning steps which are much more
difficult. In Table 8 we report results from an ablation study with respect to
the size of the initial step. For convenience, we visualize the sparsity
scheduler in Figure 2. Our preliminary experiments showed similar performance
between uniform and global sparsity distributions, so we use the former.

Figure 2: Learning rate and sparsity schedules for the proposed gradual
pruning framework.

##### Learning rate schedule.

Our goal is to provide a simple baseline setup that works well across wide
range of datasets without any additional task-dependent tuning. Currently,
papers either report best results following an extensive hyperparameter search
for each task, e.g. Zafrir et al. (2021), or they make use of carefully
crafted schedulers for each setup independently which may include warm-up
phases with and without rewinds (Sanh et al., 2020; Kurtic et al., 2022). This
may lead to high specialization to the target task/model, which is undesirable
in practice and makes it hard to distinguish benefits from the pruning
technique itself. We propose to simply replicate the standard 2-epoch fine-
tuning schedule (Devlin et al., 2019) by a certain factor and intertwine it
with pruning steps. For a fair comparison with Sanh et al. (2020) we replicate
it by a factor of 5, reproducing their 10-epoch setup. And for a fair
comparison with Chen et al. (2020) we replicate it by a factor of 15,
reproducing their 30-epoch setup. For convenience, we visualize the learning
rate schedule in Figure 2. In appendix F, we describe results with other
schedulers that didn’t work.

##### Knowledge Distillation (KD) Hardness.

We leverage KD Hinton et al. (2015) of outputs from a fine-tuned dense
teacher. KD is a standard practice when pruning, e.g. Sanh et al. (2020);
Zafrir et al. (2021); Xu et al. (2021). The loss function is formulated as a
linear combination of the standard loss associated with the specific task
(e.g. cross-entropy for classification $\mathcal{L}_{CE}$) and the Kullback-
Leibler divergence ($\mathcal{L}_{KL}$) between output distributions of the
dense (teacher) model and the sparse (student) model in the form:
$\mathcal{L}=(1-h)\mathcal{L}_{CE}+h\mathcal{L}_{KL}$. The ratio between the
two is controlled with the hardness hyperparameter $h$. To determine its
optimal value at high sparsities we run an ablation study reported in Table
10, and adopt the hardness $h=1$.

Figure 3: Teacher’s output distribution at commonly used temperatures
${T\in\\{1.0,2.0\\}}$ and the proposed $T=5.5$.

##### Knowledge Distillation Temperature.

The temperature T is an additional KD-hyperparameter that requires proper
tuning, as it controls the “softness” of the output distribution. In the
pruning literature, it is standard to use the “stronger” $T=1$ or $T=2$ values
(Xu et al., 2021; Zafrir et al., 2021; Sanh et al., 2020; Lagunas et al.,
2021; Kurtic et al., 2022); we revisit this by visualizing teacher’s output
distributions to get an insight into what the sparse student is learning. In
Figure 3, we visualize generated distributions for randomly picked samples
from the SQuADv1.1 task softened with three values of the temperature. As can
be seen, teacher’s high confidence in predicting the correct class at the
commonly used temperatures $T\in\\{1.0,2.0\\}$ makes the knowledge
distillation almost obsolete. Motivated by this observation, we run an
ablation study for many higher temperatures and report a fraction of results
in Table 11. Given the results, we adopt the temperature $T=5.5$.

####  2.1.1 GMP$\scriptstyle\bigstar$ vs. other GMP-based baselines

Due to space constraints, we aggregate all the previously analyzed
improvements in a downstream pruning recipe and present it in detail in
Appendix B. We compare our optimized GMP$\scriptstyle\bigstar$ with other GMP
results reported in the pruning literature. For a fair comparison, we consider
both setups, 10 and 30-epoch. In the 10-epoch setup, we compare against the
GMP baselines reported in Sanh et al. (2020) and refer to them as
GMP${}_{{\textrm{MvP}}}$ . In the 30-epoch setup, we compare against the best
reported results in Chen et al. (2020), obtained either via GMP or via Lottery
Ticket (LTH) approach, and refer to them as GMP${}_{{\textrm{LTH}}}$ . As can
be seen from the Table 1, our GMP$\scriptstyle\bigstar$ remarkably outperforms
all other results; in some cases the improvements are more than 20 points!

Table 1: Downstream pruning comparison of GMP$\scriptstyle\bigstar$ with other GMP-based baselines. Method | Spars. | Ep. | SQuAD | MNLI | QQP
---|---|---|---|---|---
F1 | m-acc | acc
$\textrm{BERT}_{{\textrm{BASE}}}\,$ | 0% |  | 88.5 | 84.5 | 91.1
GMP${}_{{\textrm{MvP}}}$ | 90% | 10 | 80.1 | 78.3 | 79.8
GMP$\scriptstyle\bigstar$ | 86.7 | 81.9 | 90.6
GMP${}_{{\textrm{MvP}}}$ | 97% | 10 | 59.6 | 69.4 | 72.4
GMP$\scriptstyle\bigstar$ | 81.3 | 79.1 | 89.7
GMP${}_{{\textrm{LTH}}}$ | 90% | 30 | 68.0 | 75.0 | 90.0
GMP$\scriptstyle\bigstar$ | 87.9 | 82.7 | 90.8
GMP$\scriptstyle\bigstar$ | 97% | 30 | 85.4 | 80.9 | 90.6

####  2.1.2 GMP$\scriptstyle\bigstar$ vs. advanced pruning techniques

Now, we wish to compare our GMP$\scriptstyle\bigstar$ with methods that rely
on higher-order information to make pruning decisions, like gradients in MvP
Sanh et al. (2020) and the loss curvature in oBERT Kurtic et al. (2022). Both
of these impose higher computational overhead compared to the magnitude-based
pruning, but we still put our results in the context with respect to theirs to
fully grasp the scope of improvements introduced by careful optimizations of
GMP. As can be seen from results in Table 2, GMP$\scriptstyle\bigstar$ is able
to improve upon the performance of Movement Pruning in 4 out of 6 analyzed
configurations, but unfortunately can’t match the performance of the oBERT
method. In addition to these comparisons, we make use of the open-source
implementation of oBERT, current state-of-the-art BERT-pruning method, and run
it with optimized hyperparameters from GMP$\scriptstyle\bigstar$ on the
SQuADv1.1 task. We refer to these results as oBERT$\scriptstyle\bigstar$ . As
can be seen from the Table 2, even the very competitive oBERT results benefit
from the GMP$\scriptstyle\bigstar$ setup. For all GMP$\scriptstyle\bigstar$
runs, we report mean performance across three runs with different seeds, and
additional metrics in Tables 5 and 6.

Table 2: Downstream pruning comparison of GMP$\scriptstyle\bigstar$ with advanced pruning techniques. Method | Spars. | Ep. | SQuAD | MNLI | QQP
---|---|---|---|---|---
F1 | m-acc | acc
$\textrm{BERT}_{{\textrm{BASE}}}\,$ | 0% |  | 88.5 | 84.5 | 91.1
GMP$\scriptstyle\bigstar$ | 90% | 10 | 86.7 | 81.9 | 90.6
MvP | 84.9 | 81.2 | 90.2
GMP$\scriptstyle\bigstar$ | 97% | 10 | 81.3 | 79.1 | 89.7
MvP | 82.3 | 79.5 | 89.1
GMP$\scriptstyle\bigstar$ | 90% | 30 | 87.9 | 82.7 | 90.8
oBERT | 88.3 | 83.8 | 91.4
oBERT$\scriptstyle\bigstar$ | 88.6 |  |
GMP$\scriptstyle\bigstar$ | 97% | 30 | 85.4 | 80.9 | 90.6
oBERT | 86.0 | 81.8 | 90.9
oBERT$\scriptstyle\bigstar$ | 86.6 |  |

###  2.2 Upstream pruning

To validate the optimized GMP$\scriptstyle\bigstar$ setup introduced in the
previous section, we apply it now to the pre-training phase of LLMs. This is a
two-stage process. In the first stage, the
$\textrm{BERT}_{{\textrm{BASE}}}\,$model is pruned during pre-training and
then, in the second stage, the remaining weights are fine-tuned with the fixed
mask on a specific downstream task to evaluate performance. Given the high
costs of experimenting in the pre-training phase, we use the dense teacher
open-sourced by Kurtic et al. (2022). Due to the space constraints, we
summarize all hyperparameters in an upstream pruning recipe and present it in
detail in Appendix C. In Table 3 we present results obtained in this setup and
compare against other methods that are utilizing the same approach. More
specifically, we compare against the Lottery Ticket (Chen et al., 2020), Prune
OFA (Zafrir et al., 2021), and The Optimal BERT Surgeon (oBERT) (Kurtic et
al., 2022). In addition to this, we report the GMP baselines obtained in the
Prune OFA work and refer to them as GMP${}_{{\textrm{Prune OFA}}}$ . As can be
seen from the Table 3, the GMP$\scriptstyle\bigstar$ significantly outperforms
GMP${}_{{\textrm{Prune OFA}}}$ , Lottery Tickets and even the Prune OFA, and
comes really close to the performance of oBERT. For all
GMP$\scriptstyle\bigstar$ runs, we report mean performance across four runs
with different seeds. These results confirm findings from the previous section
and establish the GMP$\scriptstyle\bigstar$ as an extremely competitive
baseline in all regimes.

Table 3: Upstream pruning comparison of GMP$\scriptstyle\bigstar$ with other GMP-based baselines and more advanced pruning techniques. Method | Sparsity | SQuAD | MNLI | QQP
---|---|---|---|---
F1 | m-acc | acc
$\textrm{BERT}_{{\textrm{BASE}}}\,$ | 0% | 88.5 | 84.5 | 91.1
GMP${}_{{\textrm{Prune OFA}}}$ | 85% | 86.2 | 82.5 | 90.9
Lottery Ticket | 90% | 68.0 | 75.0 | 90.0
Prune OFA | 87.3 | 81.5 | 90.9
GMP$\scriptstyle\bigstar$ | 88.2 | 83.2 | 90.8
oBERT | 88.5 | 83.4 | 91.0
GMP$\scriptstyle\bigstar$ | 97% | 84.7 | 80.3 | 89.8
oBERT | 84.9 | 80.9 | 90.3

##  3 Conclusion

In this work, we presented a set of updates to the standard gradual pruning
setup for BERT models which enabled us to achieve very competitive results
with the simple magnitude pruner. These results outperformed, by significant
margins, all magnitude-based results currently available in the pruning
literature which have been used as baselines for development and benchmarking
of the new and more advanced pruning techniques. We hope that these new
baselines will help the community to start off from a competitive set of
results when compressing large language models. Moreover, our
GMP$\scriptstyle\bigstar$ has even outperformed some results obtained with
more advanced and computationally heavier pruning techniques. At this point,
we would like to strongly emphasize that these results should not be
interpreted as evidence that magnitude pruning is better than other more
advanced methods. Rather, they should be interpreted as evidence that their
current results could significantly benefit from updates of the gradual setup
presented on the GMP$\scriptstyle\bigstar$ use-case. To support this claim, we
ran the state-of-the-art oBERT pruner with the GMP$\scriptstyle\bigstar$ setup
and managed to improve its results by non-trivial margins.

##  4 Limitations

As any academic study, our work is not without its limitations. Following the
literature, our extensive empirical studies were conducted only on the
standard $\textrm{BERT}_{{\textrm{BASE}}}\,$model, giving us opportunity to
compare against a vast amount of different pruning techniques. Throughout the
literature, this model emerged as a consistent benchmark for unstructured
pruning methods. However, the current results don’t directly imply that our
findings will be generally applicable to other language models as well. To
partially fill in this uncertainty gap, we conduct a few experiments on the
three times larger $\textrm{BERT}_{{\textrm{LARGE}}}\,$model and report
results in the Appendix A. Another limitation which we aim to remove in future
work is the focus on fine-grained unstructured sparsity type, and explore
other variants such as semi-structured and structured pruning.

## References

  * Chen et al. (2020) Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020.  The lottery ticket hypothesis for pre-trained bert networks.  _Advances in neural information processing systems_ , 33:15834–15846.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Gale et al. (2019) Trevor Gale, Erich Elsen, and Sara Hooker. 2019.  The state of sparsity in deep neural networks.  In _International Conference on Machine Learning (ICML)_.
  * Gholami et al. (2021) Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021.  A survey of quantization methods for efficient neural network inference.  _arXiv preprint arXiv:2103.13630_.
  * Hagiwara (1994) Masafumi Hagiwara. 1994.  A simple and effective method for removal of hidden units and weights.  _Neurocomputing_ , 6(2):207 – 218.  Backpropagation, Part IV.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7).
  * Hoefler et al. (2021) Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021\.  Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks.  _arXiv preprint arXiv:2102.00554_.
  * Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, Kornél Csernai, et al. 2017.  First quora dataset release: Question pairs.  _data. quora. com_.
  * Kurtic et al. (2022) Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. 2022.  The optimal bert surgeon: Scalable and accurate second-order pruning for large language models.  _arXiv preprint arXiv:2203.07259_.
  * Kurtz et al. (2020) Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. 2020.  Inducing and exploiting activation sparsity for fast inference on deep neural networks.  In _International Conference on Machine Learning (ICML)_.
  * Lagunas et al. (2021) François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. 2021.  Block pruning for faster transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10619–10629. Association for Computational Linguistics.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Sanh et al. (2020) Victor Sanh, Thomas Wolf, and Alexander Rush. 2020.  Movement pruning: Adaptive sparsity by fine-tuning.  _Advances in Neural Information Processing Systems_ , 33:20378–20389.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2021) Dongkuan Xu, Ian EH Yen, Jinxi Zhao, and Zhibin Xiao. 2021.  Rethinking network pruning–under the pre-train and fine-tune paradigm.  _arXiv preprint arXiv:2104.08682_.
  * Zafrir et al. (2021) Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. 2021\.  Prune once for all: Sparse pre-trained language models.  _arXiv preprint arXiv:2111.05754_.
  * Zhu and Gupta (2017) Michael Zhu and Suyog Gupta. 2017.  To prune, or not to prune: exploring the efficacy of pruning for model compression.  _arXiv preprint arXiv:1710.01878_.

##  Appendix A Additional models

All of our experiments in the paper focus on the
$\textrm{BERT}_{{\textrm{BASE}}}\,$model as it is the standard benchmark used
in the pruning literature and we are able to compare results with a vast
amount of other techniques. To verify that our proposed
GMP$\scriptstyle\bigstar$ setup doesn’t pertain only to the
$\textrm{BERT}_{{\textrm{BASE}}}\,$model, in the Table 4 we present results on
the three times larger $\textrm{BERT}_{{\textrm{LARGE}}}\,$ model. As a proof
of concept, we run our downstream gradual pruning setup crafted for the
$\textrm{BERT}_{{\textrm{BASE}}}\,$model without any hyper-parameter tuning.

Table 4: Downstream pruning results when pruning the $\textrm{BERT}_{{\textrm{LARGE}}}\,$ model with the GMP$\scriptstyle\bigstar$ and gradual setup crafted for the $\textrm{BERT}_{{\textrm{BASE}}}\,$model. Method | Sparsity | SQuAD
---|---|---
F1 | EM
$\textrm{BERT}_{{\textrm{LARGE}}}\,$ | 0% | 91.22 | 84.45
GMP$\scriptstyle\bigstar$ | 90% | 90.12 | 83.21
GMP$\scriptstyle\bigstar$ | 97% | 87.93 | 80.50

##  Appendix B Downstream pruning recipe

All of our implementations are built on top of HuggingFace’s Transformers
111https://github.com/huggingface/transformers Wolf et al. (2020) and Datasets
222https://github.com/huggingface/datasets Lhoest et al. (2021) libraries, and
NeuralMagic’s SparseML 333https://github.com/neuralmagic/sparseml Kurtz et al.
(2020) library for model compression, and will be open-sourced to community
along with our sparse models.

As our goal is to provide a simple and unique gradual pruning setup, all of
our downstream runs (for all datasets) are using the same set of
hyperparameters. The ones used to obtain results reported in Tables 1, 2, 5,
and 6 are as follows:

  * •

learning-rate: recurring 2-epoch scheduler (visualized in Figure 2) with the
initial value of 1e-4, and the final value of 1e-6,

  * •

number-of-epochs: 10 or 30 epochs, depending on the methods we compare
against,

  * •

sparsity: cubic scheduler with the initial pruning step of 70% sparsity
(visualized in Figure 2),

  * •

pruning: prune frequency of ten times per epoch, except during the first and
last 2-epochs when only fine-tuning happens and masks are fixed,

  * •

student-initialization: standard $\textrm{BERT}_{{\textrm{BASE}}}\,$(bert-
base-uncased444https://huggingface.co/bert-base-uncased),

  * •

knowledge-distillation (KD): (hardness, temperature) = (1.0, 5.5),

  * •

KD-teachers: standard $\textrm{BERT}_{{\textrm{BASE}}}\,$fine-tuned on the
corresponding task,

  * •

weight-decay: 0.0,

  * •

all other hyper-parameters are set to the standard default values, e.g. Sanh
et al. (2020):

    * –

SQuADv1.1: batch-size=16, max-sequence-length=384, doc-stride=128,

    * –

MNLI and QQP: batch-size=32, max-sequence-length=128.

##  Appendix C Upstream pruning recipe

For a fair comparison with Zafrir et al. (2021) and Kurtic et al. (2022), we
adopt the same gradual setup for pruning and fine-tuning, but apply our
specific GMP$\scriptstyle\bigstar$ updates. The entire process is carried out
in two stages. The first stage prunes the
$\textrm{BERT}_{{\textrm{BASE}}}\,$model at upstream datasets, BookCorpus and
English Wikipedia. Both datasets are available via Lhoest et al. (2021). The
upstream pruning recipe can be summarized as follows:

  * •

learning-rate: recurring 0.5-epoch scheduler with the initial learning rate
value of 5e-4, and the final value of 5e-6,

  * •

number-of-epochs: 3,

  * •

sparsity: cubic scheduler with the initial pruning step of 70% sparsity,

  * •

pruning: prune frequency of hundred times per epoch, except during the last
epoch when only fine-tuning happens and masks are fixed,

  * •

knowledge-distillation (KD): (hardness, temperature) = (1.0, 5.5),

  * •

KD teacher and student initialization:
$\textrm{BERT}_{{\textrm{BASE}}}\,$prepared and open-sourced by Kurtic et al.
(2022),

  * •

weight-decay: 0.01,

  * •

batch-size: 256,

  * •

max-sequence-length: 512.

The second stage makes use of this upstream pruned model and fine-tunes it on
a specific downstream task (SQuADv1.1, MNLI, QQP) for 8-epochs with fixed
masks. All task-specific hyperparameters (batch-size, max-sequence-length,
doc-stride, weight-decay) are the same as in Appendix B, and the remaining
ones are as follows:

  * •

learning-rate: linear decay with initial value of 1.5e-5,

  * •

knowledge-distillation: (hardness, temperature) = (1.0, 5.5),

  * •

KD-teachers: standard $\textrm{BERT}_{{\textrm{BASE}}}\,$fine-tuned on the
corresponding task.

##  Appendix D Additional metrics

Due to space constraints, for corresponding runs in Tables 1, 2, and 3, we
present additional performance metrics in Tables 5, 6, and 7.

Table 5: Downstream pruning comparison of GMP$\scriptstyle\bigstar$ with other GMP-based baselines. We report complementary evaluation metrics for results in Table 1. Method | Spars. | Ep. | SQuAD | MNLI | QQP
---|---|---|---|---|---
EM | mm-acc | F1
$\textrm{BERT}_{{\textrm{BASE}}}\,$ | 0% |  | 81.4 | 85.0 | 88.0
GMP${}_{{\textrm{MvP}}}$ | 90% | 10 | 70.2 | 79.3 | 65.0
GMP$\scriptstyle\bigstar$ | 78.7 | 82.1 | 87.4
GMP${}_{{\textrm{MvP}}}$ | 97% | 10 | 45.5 | 70.6 | 57.8
GMP$\scriptstyle\bigstar$ | 71.3 | 79.6 | 86.1
GMP${}_{{\textrm{LTH}}}$ | 90% | 30 | - | - | -
GMP$\scriptstyle\bigstar$ | 80.4 | 83.2 | 87.7
GMP$\scriptstyle\bigstar$ | 97% | 30 | 77.1 | 81.2 | 87.3
Table 6: Downstream pruning comparison of GMP$\scriptstyle\bigstar$ with advanced pruning techniques. We report complementary evaluation metrics for results in Table 2. Method | Spars. | Ep. | SQuAD | MNLI | QQP
---|---|---|---|---|---
EM | mm-acc | F1
$\textrm{BERT}_{{\textrm{BASE}}}\,$ | 0% |  | 81.4 | 85.0 | 88.0
GMP$\scriptstyle\bigstar$ | 90% | 10 | 78.7 | 82.1 | 87.4
MvP | 76.6 | 81.8 | 86.8
GMP$\scriptstyle\bigstar$ | 97% | 10 | 71.3 | 79.6 | 86.1
MvP | 72.7 | 80.1 | 85.5
GMP$\scriptstyle\bigstar$ | 90% | 30 | 80.4 | 83.2 | 87.7
oBERT | 81.1 | 84.4 | 88.3
oBERT$\scriptstyle\bigstar$ | 88.6 |  |
GMP$\scriptstyle\bigstar$ | 97% | 30 | 77.1 | 81.2 | 87.3
oBERT | 78.1 | 82.0 | 87.7
oBERT$\scriptstyle\bigstar$ | 78.8 |  |
Table 7: Upstream pruning comparison of GMP$\scriptstyle\bigstar$ with other GMP-based baselines and advanced pruning techniques. We report complementary evaluation metrics for results in Table 3. Method | Sparsity | SQuAD | MNLI | QQP
---|---|---|---|---
EM | mm-acc | F1
$\textrm{BERT}_{{\textrm{BASE}}}\,$ | 0% | 81.4 | 85.0 | 88.0
GMP${}_{{\textrm{Prune OFA}}}$ | 85% | 78.0 | 83.1 | 87.7
Lottery Ticket | 90% | - | - | -
Prune OFA | 79.8 | 82.4 | 87.7
GMP$\scriptstyle\bigstar$ | 81.1 | 83.8 | 87.6
oBERT | 81.4 | 83.8 | 87.8
GMP$\scriptstyle\bigstar$ | 97% | 76.3 | 81.0 | 86.5
oBERT | 76.9 | 81.1 | 87.0

##  Appendix E Ablation studies

In Tables 8, 9, 10, 11 we present a subset of results from ablation studies
conducted to find the optimal values of hyperparameters for the
GMP$\scriptstyle\bigstar$ gradual pruning setup. These results illustrate the
general trend of effects caused by varying one hyperparameter. Therefore, they
don’t cover all possible scenarios (i.e. higher-order effects when multiple
hyperparameters are updated together), but such studies are computationally
too expensive and we don’t conduct them.

Table 8: Initial sparsity step ablation study on the SQuADv1.1 dataset. Initial sparsity (%) | F1 score at
---|---
90% | 97%
00 | 85.2 | 77.2
30 | 85.5 | 77.8
50 | 85.8 | 78.5
70 | 85.8 | 79.1
Table 9: Initial learning rate ablation study on the MNLI dataset. Initial learning rate | Accuracy at
---|---
90% | 97%
3e-5 | 80.8 | 76.3
5e-5 | 81.4 | 77.8
8e-5 | 81.9 | 78.6
1e-4 | 81.6 | 79.3
Table 10: Knowledge Distillation (KD) hardness ablation study on the SQuADv1.1 dataset. Knowledge distillation hardness | F1 score at
---|---
90% | 97%
0.6 | 84.6 | 78.4
0.8 | 85.9 | 80.1
0.9 | 86.2 | 80.7
1.0 | 86.7 | 81.0
Table 11: Knowledge Distillation (KD) temperature ablation study on the SQuADv1.1 dataset. Knowledge distillation temperature | F1 score at
---|---
90% | 97%
1.0 | 84.7 | 77.3
2.0 | 85.8 | 79.0
5.5 | 86.7 | 81.0
8.5 | 86.4 | 80.9

##  Appendix F Learning rate schedulers we tried, but didn’t work

The schedulers we tried but didn’t work: 1) linearly decaying learning rate,
2) the default fine-tuning learning rates (3e-5 for SQuADv1.1 and 2e-5 for
MNLI and QQP), 3) learning rates with the warm-up phase. In the preliminary
experiments, we have noticed that 1) and 2) have problems in recovering from
the pruning steps at higher sparsities. The former one has extremely small
learning rate values during the last few epochs when the model is pruned to
high sparsities. The latter one continuously fails to recover properly even at
moderate sparsity targets, which is why we run a sweep over a range of initial
learning rate values. Given the results in Table 9, we decided to proceed with
the 1e-4 as it helped to recover significantly at high sparsities. We haven’t
observed any benefits from the warmup phase, which is why we have decided not
to use it as it adds an additional hyperparameter to tune.


# Effect of Rotation Angle in Self-Supervised Pre-training is Dataset-
Dependent

Amy Saranchuk  Division of Engineering Science
University of Toronto
Toronto, Ontario, Canada
<EMAIL_ADDRESS>Michael Guerzhoy  Division of Engineering
Science and
Dept. of Mechanical & Industrial Engineering
University of Toronto
Toronto, Ontario, Canada
<EMAIL_ADDRESS>
###### Abstract

Self-supervised learning for pre-training (SSP) can help the network learn
better low-level features, especially when the size of the training set is
small. In contrastive pre-training, the network is pre-trained to distinguish
between different versions of the input. For example, the network learns to
distinguish pairs (original, rotated) of images where the rotated image was
rotated by angle $\theta$ vs. other pairs of images. In this work, we show
that, when training using contrastive pre-training in this way, the angle
$\theta$ and the dataset interact in interesting ways. We hypothesize, and
give some evidence, that, for some datasets, the network can take “shortcuts”
for particular rotation angles $\theta$ based on the distribution of the
gradient directions in the input, possibly avoiding learning features other
than edges, but our experiments do not seem to support that hypothesis. We
demonstrate experiments on three radiology datasets. We compute the saliency
map indicating which pixels were important in the SSP process, and compare the
saliency map to the ground truth foreground/background segmentation. Our
visualizations indicate that the effects of rotation angles in SSP are
dataset-dependent. We believe the distribution of gradient orientations may
play a role in this, but our experiments so far are inconclusive.

###### Index Terms:

Self-supervised pre-training; pre-training; saliency map; segmentation

##  I Introduction

Machine learning has demonstrated remarkable success in various computer
vision-for-medical imaging tasks, such as classification and segmentation.
However, a major challenge in using these models is the requirement for large
labelled datasets, which are costly to create, especially in a field where
expert annotation is required. With an abundance of unlabelled data available,
self-supervised pre-training is a promising approach learning low-level
features without the need for large labelled datasets.

In this paper, we investigate a common self-supervised learning for pre-
training method: augmenting the dataset via rotation, and pre-training the
network to distinguish between pairs of images where one is rotated by
$\theta$ vs other pairs. We observe that the network behaves differently
depending on the rotation angle $\theta$, and that this behaviour is dataset-
dependent.

We refer to semi-supervised learning for pretaining using a pretext task as
“semi-supervised pretraining” (SSP).

We explore the behaviour of the network by inspecting the correspondence
between saliency maps produced by the SSP-trained network using rotation by
$\theta$ and the ground-truth segmentation image.

We hypothesize that this has to do with the distribution of the directions of
the gradients in the image, and provide some preliminary (but inconclusive)
experiments that address this hypothesis.

Work going back to [1] and [2] indicates that hisotgrams of oriented gradients
(HoGs) often characterize the appearance of objects very well. If the network
can rely on HoG-like features to perform the classification task of rotated
vs. non-rotated, it would not learn any other features. We hypothesize that
this is a reason that some rotation angles seem to produce “worse” features.
We explore this intuition by correlating the performance of SVM-classified HoG
features for the unrotated vs. rotated by $\theta$ task, and the
correspondence between saliency maps produced by the SSP-trained network using
rotation by $\theta$ and the ground-truth segmentation image. We present an
experiment to attempt to confirm the hypothesis, but the experiment does not
seem to support the hypothesis.

##  II Background

###  II-A Self-Supervised Pre-training (SSP)

In SSP, the data “supervises itself” during training instead of using labels
as an indication of its performance by creating artificial supervisory signals
from unlabelled data. It allows the network to leverage unlabelled data by
learning meaningful representations without manual annotations [3] [4].

Self-supervised pertaining is used when there are two stages: the pretext task
and the downstream task. The overall idea behind self-supervised learning is
to first pre-train the model using the pretext task and then further fine-tune
it for a specific task. The pretext task is used for pre-training and guides
the model toward learning intermediate representations of the data. The
downstream task is the task which we wish to solve, and to which we wish to
transfer knowledge from the pretext task [5].

Pretext tasks are unrelated to the downstream task of interest and are used to
generate pseudo-labels from data without human annotation. They force the
network to learn information about the data such as where edges, colours, and
shapes appear in an image. Some common pretext tasks include recolouring an
image, predicting the rotation angle, and unscrambling a Jigsaw puzzle [6].
The downstream task is the primary task of interest (for example, image
classification or segmentation).

###  II-B Contrastive Learning

A popular discriminative pretext approach is contrastive learning. For
example, Mikolov et al. [7] proposed such a method to learn word embeddings.
Since then, it has been adapted to other domains and according to [8], has
“revolutionized” the field of computer vision through learning meaningful
representations that can be applied to a variety of vision tasks from
unlabelled data.

The general idea behind contrastive learning is that various transformations
of the same image still hold the same semantic information [6] [9]. The goal
is to bring data which is semantically-similar closer together in an embedding
space, while pushing dissimilar data apart [3] [7] [9].

The following outlines the contrastive learning algorithm. First, each sample
image from the training set is augmented. The image along with its
augmentation are considered a positive pair, and the image along with every
other image in the dataset are considered negative pairs. The model is then
trained so that it can learn to distinguish between positive and negative
pairs by minimizing the distance between positive pairs, and maximizing the
distance between negative pairs. By doing so, the model learns quality
representations of each image, the knowledge of which is subsequently
transferred to the downstream task [7].

Some common data augmentation approaches for creating positive pairs include
cropping, flipping, rotating, and adding noise to the image. However, like
pretext tasks, not all data augmentations are equally effective. It’s crucial
to choose augmentations that preserve the semantic information of the data.

There have been numerous frameworks proposed to improve vanilla contrastive
learning, with one of the most popular being Momentum Contrast (MoCo) [10].
Proposed in 2020, this method introduces a momentum-encoded queue to retain
negative samples. First an image from the training set is augmented into two
versions (for example, a flipped version and a blurred version). Each version
is passed through a separate encoder: the query encoder and key encoder. Both
output vector representations of their respective images. The output of the
key encoder is added to a dynamically-sized dictionary which stores negative
samples with respect to the output of the query encoder. The contrastive loss
is then calculated between the query encoder’s output and the vectors in the
dictionary. Over time, this process allows the model to learn effective
representations of the images in the dataset.

###  II-C SSP in Medical Imaging

SSP has achieved success in numerous areas, including medical imaging [3].
Assembling large-scale medical imaging datasets requires domain knowledge, is
costly, and is time-consuming, limiting the efficacy of medical imaging models
[9][6].

##  III Approach

Our goal is to explore the effect of changing the rotation angle $\theta$ on
the learned features when using SSP with the rotation augmentation.

For each rotation angle $\theta$ used in generating the image pairs for
contrastive pre-training, we visualize the salient pixels in the image using
SmoothGrad [11], and compare the salient pixel mask to the ground truth
segmentation mask for that image using the Dice coefficient.

We acknowledge that this is an indirect method of observing the effect of
setting the rotation angle $\theta$. In future work, more direct methods, such
as measuring the performance on a downstream task or visualizing the learned
features, will be used to observe the effect of changing the rotation angle
$\theta$.

##  IV Datasets

Three publicly available datasets were used in this study:

  * •

BraTS: Mutlimodal MRI scans with annotated brain tumour segments (see Fig. 1)
[12].

  * •

Lung Mask Image Dataset: Segmented lung X-ray images (see Fig. 2) [13].

  * •

Kvasir-SEG: Gastrointestinal polyp images with corresponding segmentation
masks (see Fig. 3) [14].

Figure 1: Sample image and segmentation mask from the BraTS dataset Figure 2:
Sample image and segmentation mask from the Lung Mask Image dataset Figure 3:
Sample image and segmentation mask from the Kvasir-SEG dataset

###  IV-A Self-Supervised Model and Fine-Tuning

In this study, an ImageNet-pre-trained MoCo v2 model with a ResNet-50 backbone
was used [15]. This model was fine-tuned on the medical images of each dataset
separately (the segmentation masks were not used at all during training).
Fine-tuning was performed for 100 epochs with a batch size of 16. The
optimizer used was stochastic gradient descent with a learning rate of 0.01,
momentum of 0.9, and weight decay of $1\times 10^{-4}$.

Prior to training, the images were normalized and subjected randomly to
various augmentations to generate the positive pairs, including resizing,
Gaussian blurring, horizontal and vertical flips, rotation, and affine
translation.

We crop the outer border of the image in order to avoid the network’s being
able to rely on boundary artifacts to detect rotations by $\theta$.

###  IV-B Saliency Map Generation and Evaluation

Following training, to evaluate the regions of the image that were focused on
by the model, saliency maps were generated using the SmoothGrad [11] method,
which adds noise to the input images and averages the resulting gradients. See
examples in Figs. 4 5 6. The implementation was adapted from a publicly
available GitHub repository111https://github.com/fawazsammani/explain-cl.

75 images were selected from each dataset and rotated through a full
360-degree range, with saliency maps generated at every single degree
increment. The saliency maps were then evaluated based on their similarity to
the corresponding ground-truth segmentation masks. The similarity was
calculated using the Dice score.

Figure 4: Comparison of original images from Lung Mask Image dataset, segmentation masks, and saliency maps. Top row: Original orientation. Bottom row: Rotated by 95 $\degree$. Figure 5: Comparison of original images from Kvasir-SEG dataset, segmentation masks, and saliency maps. Top row: Original orientation. Bottom row: Rotated by 95 $\degree$. Figure 6: Comparison of original images from BraTS dataset, segmentation masks, and saliency maps. Top row: Original orientation. Bottom row: Rotated by 95 $\degree$. TABLE I: Average correspondence (using Dice score) between ground truth segmentation and saliency map obtained using trained features vs. rotation angle (left) and accuracy of an SVM classifier classifying original images versus rotated image vs. rotation angle (right). .  |
---|---
|
|

##  V Results: Saliency Maps and Segmentation Ground Truth

For each dataset, the Dice score between the 75 saliency maps and their
corresponding segmentation masks were averaged for each rotation angle
$\theta$. The Dice scores were then plotted against the rotation in degrees to
reveal distinctive patterns across each dataset.

The results are in Table I.

The Dice score versus rotation angle for the BraTS (brain imaging) dataset in
exhibits periodic behaviour. Dice scores oscillated between approximately
$0.627$ and $0.658$ with peaks around every $90\degree$ of rotation. The
maxima occur at $90\degree$, $180\degree$, and $270\degree$. The minima occur
at $45\degree$, $135\degree$, $225\degree$, and $315\degree$.

The plot for the Lung Mask Image dataset reveals another periodic shape with a
different amplitude range, namely between approximately $0.470$ and $0.476$.
For this dataset the maxima occur at $66\degree$, $156\degree$, $246\degree$,
and $336\degree$ degrees. The minima occur at $90\degree$, $180\degree$, and
$270\degree$.

Another periodic shape is presented in the plot for the Kvasir-SEG (GI tract)
dataset, this time oscillating between Dice scores of approximately $0.346$
and $0.360$. The maxima of this plot occur at $45\degree$, $135\degree$,
$225\degree$, and $315\degree$. Similarly to the chest dataset, the minima
occur at $90\degree$, $180\degree$, and $270\degree$.

##  VI Classifying HoG Descriptors of Images

We observe that angles of $45\degree,90\degree,135\degree,...$ seem to often
occur at minima and maxima of the correspondence between the saliency maps and
the segmentation ground truth. We hypothesize that, when good features are
learned, the saliency maps correspond to the segmentation ground truth better.
We hypothesize that worse features are learned when the network can take
“shortcuts” in figuring out the angle $\theta$. For example, the network could
rely on Histogram-of-Gradients (HoG) [1]-like features. (Although those
features are famously good, they are not specific to our dataset; we do not
expect that learning HoG features would be a part of a successful fine-tuning
of a network that was already pretrained on ImageNet.)

To explore this hypothesis, we train SVMs to classify the HoG features of
images rotated by $\theta$ vs unrotated images. We compute and concatenate HoG
descriptors for every $64\times 64$ cell. We use cross-validation to select
the best parameters for a Guassian-kernel SVM.

The results are in Table I. We observe that the HoG classification accuracy is
low for $\theta$ close to $0\degree$ and (and $360\degree$), reflecting the
increased difficulty of the task.

For the Kvasir-SEG dataset, we observe minima in the accuracy of the HoG
classifier that correspond to minima in the correspondence between the
saliency map and the ground truth segmentation. This seems to be evidence
against our theory: when it is more difficult to classify based on HoGs and
there are no shortcuts (at least via HoGs), it seems that the correspondence
between the saliency map and the ground truth segmentation is lower.

##  VII Conclusions and Future Work

We observe an intriguing property of the rotation pretext text in self-
supervised pre-training: it seems that the relationship between the rotation
angle and the learned features is periodic and non-monotonic in some sense.
The relationship between the rotation angle $\theta$ and the Dice score seems
dataset-dependent.

We hypothesize that this is related to the ease with which one can use edges
to determine what rotation is performed, but our experiments do not seem to
confirm that hypothesis.

In the future, we will measure the learned features more directly, both by
visualizing them and by measuring the performance using the features on a
downstream task.

(a) Sample HoG features for BraTS dataset

(b) Sample HoG features for Lung Mask Imaging dataset

(c) Sample HoG features for the Kvasir-SEG

Figure 7:

## References

  * [1] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Proceedings of the seventh IEEE international conference on computer vision_ , vol. 2. Ieee, 1999, pp. 1150–1157.
  * [2] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , vol. 1. Ieee, 2005, pp. 886–893.
  * [3] V. Rani, S. T. Nabi, M. Kumar, A. Mittal, and K. Kumar, “Self-supervised learning: A succinct review,” _Archives of Computational Methods in Engineering_ , vol. 30, p. 2761–2775, 2023.
  * [4] A. Chowdhury, J. Rosenthal, J. Waring, and R. Umeton, “Applying self-supervised learning to medicine: Review of the state of the art and medical implementations,” _Informatics_ , 2021.
  * [5] D. Shah and A. Jha. Self-supervised learning and its applications. [Online]. Available: https://neptune.ai/blog/self-supervised-learning
  * [6] J. Xu, “A review of self-supervised learning methods in the field of medical image analysis,” _International Journal of Image, Graphics and Signal Processing_ , vol. 4, pp. 33–46, 2021.
  * [7] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon, “A survey on contrastive self-supervised learning,” _Technologies_ , 2021.
  * [8] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” _Proceedings of the 37th International Conference on Machine Learning_ , 2020.
  * [9] S.-C. Huang, M. J. Anuj Pareek, M. P. Lungren, S. Yeung, and A. S. Chaudhari, “Self-supervised learning for medical image classification: a systematic review and implementation guidelines,” _npj Digital Medicine_ , vol. 6, no. 74, 2023.
  * [10] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” 2020.
  * [11] F. Sammani, B. Joukovsky, and N. Deligiannis, “Visualizing and understanding contrastive learning,” _IEEE Transactions on Image Processing_ , 2023.
  * [12] Multimodal brain tumor segmentation challenge 2020: Data. [Online]. Available: https://www.med.upenn.edu/cbica/brats2020/data.html
  * [13] N. Newra, “Lung mask image dataset,” 2022. [Online]. Available: https://www.kaggle.com/dsv/3559777
  * [14] D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, and H. D. Johansen, “Kvasir-seg: A segmented polyp dataset,” in _International Conference on Multimedia Modeling_. Springer, 2020, pp. 451–462.
  * [15] Meta Research. Moco: Momentum contrast for unsupervised visual representation learning. [Online]. Available: https://github.com/facebookresearch/moco