# SimCSum: Joint Learning of Simplification and Cross-lingual Summarization
for Cross-lingual Science Journalism

Mehwish Fatima†, Tim Kolber†, Katja Markert‡ Michael Strube†
†Heidelberg Institute for Theoretical Studies, Heidelberg, Germany
‡Department of Computational Linguistics, Heidelberg University
Heidelberg, Germany
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Cross-lingual science journalism generates popular science stories of
scientific articles different from the source language for a non-expert
audience. Hence, a cross-lingual popular summary must contain the salient
content of the input document, and the content should be coherent,
comprehensible, and in a local language for the targeted audience. We improve
these aspects of cross-lingual summary generation by joint training of two
high-level nlp tasks, simplification and cross-lingual summarization. The
former task reduces linguistic complexity, and the latter focuses on cross-
lingual abstractive summarization. We propose a novel multi-task architecture
- simcsum consisting of one shared encoder and two parallel decoders jointly
learning simplification and cross-lingual summarization. We empirically
investigate the performance of simcsum by comparing it with several strong
baselines over several evaluation metrics and by human evaluation. Overall,
simcsum demonstrates statistically significant improvements over the state-of-
the-art on two non-synthetic cross-lingual scientific datasets. Furthermore,
we conduct an in-depth investigation into the linguistic properties of
generated summaries and an error analysis.

##  1 Introduction

A real-world example of cross-lingual science journalism is Spektrum der
Wissenschaft111https://www.spektrum.de/magazin. It is the German version of
Scientific American and an acclaimed bridge between local readers and the
latest scientific research in Germany. Spektrum’s journalists read English
scientific articles and summarize them into popular science stories in German
that are comprehensible by local non-expert readers. Spektrum der Wissenschaft
approached us to automate the process of their journalist’s work. We define
cross-lingual science journalism as the fusion of two high-level nlp tasks:
text simplification and cross-lingual scientific summarization.

Cross-lingual science journalism aims to generate science summaries in a
target language from scientific documents in a source language while
emphasizing simplification. The readers of science magazines, usually non-
experts in scientific fields, can grasp the complex scientific concepts
expressed in easy-to-understand language. Moreover, this task is extendable
for different readability levels according to age and education: adults, teens
and kids, and in various local languages. However, we limit our study to
investigate the task for English to German, targeting adult non-expert
readers, to automate the process for Spektrum der Wissenschaft.

As no prior work exists to the best of our knowledge, we consider the two
closest tasks for finding the recent advancements: monolingual science
journalism and cross-lingual summarization222Cross-lingual scientific
summarization is an under-studied area, so we focus on cross-lingual
summarization.. In monolingual science journalism, we discover the trend of
taking it a downstream task of abstractive summarization (Dangovski et al.,
2021; Zaman et al., 2020) with customized datasets (Zaman et al., 2020;
Goldsack et al., 2022). However, these datasets are not suitable for cross-
lingual science journalism. Cross-lingual summarization studies can be divided
as pipeline (Ouyang et al., 2019; Zhu et al., 2019, 2020) and Multi-Task
Learning (mtl) (Cao et al., 2020; Bai et al., 2021, 2022) models with
synthetic datasets, and direct cross-lingual summarization with non-synthetic
datasets (Ladhak et al., 2020; Fatima and Strube, 2021). We find that Fatima
and Strube (2021) have collected their datasets for cross-lingual scientific
summarization, so we use them to explore the task.

To investigate cross-lingual science journalism, we propose an mtl-based model
- simcsum that jointly trains for simplification and cross-lingual
summarization to improve the quality of cross-lingual popular science
summaries. simcsum consists of one shared encoder and two independent decoders
for each task based on a transformer architecture, where we consider cross-
lingual summarization as our main task and simplification as our auxiliary
task.

#### Contributions

We summarize the contributions as follows:

  1. 1.

We introduce simcsum that jointly learns simplification and cross-lingual
summarization to improve the quality of cross-lingual science summaries for
non-expert readers. We also introduce a strong baseline - Simplify-Then-
Summarize to compare the performance of our proposed model.

  2. 2.

We empirically evaluate the performance of simcsum against several existing
cross-lingual summarization models on two cross-lingual scientific datasets.
We also conduct a human evaluation to find the linguistic qualities of
generated summaries.

  3. 3.

We further analyze the outputs for various lexical, readability and syntactic-
based linguistic features. We also perform error analysis to assess the
quality of outputs.

##  2 Related Work

###  2.1 Scientific Summarization

This section focuses on the datasets for scientific summarization. Most
science summarization datasets are collected from English scientific papers
paired with abstracts: arxiv (Kim et al., 2016; Cohan et al., 2018), pubmed
(Cohan et al., 2018; Nikolov et al., 2018), medline (Nikolov et al., 2018) and
science blogs (Vadapalli et al., 2018b, a). Some work has been conducted for
extreme summarization with monolingual dataset (Cachola et al., 2020),
extended for cross-lingual extreme summarization (Takeshita et al., 2022). The
extreme summarization task generates a one/two-line summary from a scientific
abstract/paper, which makes it different from science journalism.

Cross-lingual scientific summarization is an understudied area due to its
challenging nature. We find two studies: a synthetic dataset from English to
Somali, Swahili, and Tagalog with round trip translation (Ouyang et al.,
2019), two real cross-lingual datasets from Wikipedia Science Portal and
Spektrum der Wissenschaft for English-German (Fatima and Strube, 2021).

###  2.2 Cross-lingual Summarization

This section focuses on mtl-based cross-lingual summarization. Zhu et al.
(2019) develop an mtl model for English-Chinese cross-lingual summarization.
They develop two variations of the transformer model (Vaswani et al., 2017),
where the encoder is shared, and two decoders are independent. Cao et al.
(2020) present a mtl model for cross-lingual summarization by joint learning
of alignment and summarization. Their model consists of two encoders and two
decoders, each dedicated to one task while sharing contextual representations.
The authors evaluate their model on synthetic cross-lingual datasets for the
English-Chinese language pairs. Takase and Okazaki (2020) introduce an mtl
framework for cross-lingual abstractive summarization by augmenting
(monolingual) training data with translations for three pairs: Chinese-
English, Arabic-English, and English-Japanese. The model consists of a
transformer encoder-decoder model with prompt-based learning in which each
training instance is affixed with a special prompt to signal example type. Bai
et al. (2021) develop a variation of multi-lingual bert for English-Chinese
cross-lingual abstractive summarization. The model is trained with a few shots
of monolingual and cross-lingual examples. Bai et al. (2022) extend their work
by introducing a mtl model to improve cross-lingual summaries by combining
cross-lingual summarization and translation rates. They add a compression
scoring method at the encoder and decoder of their model. They augment their
datasets for different compression levels of summaries. One variation consists
of cross-lingual and monolingual summarization decoders, while the other
consists of cross-lingual and translation decoders.

Most of these studies focus on English-Chinese synthetic datasets emphasizing
summarization and translation. By architecture, simcsum is similar to Zhu et
al. (2019) model as it also consists of one shared encoder and two task-
specific decoders.

###  2.3 Monolingual Science Journalism

This section focuses on science journalism models. Zaman et al. (2020) develop
an extension of pgn (See et al., 2017) by modifying the loss function, so the
model is trained for joint simplification and summarization. It is not a mtl
model but a summarization model with an added loss for simplification.
Moreover, the model is trained on a customized dataset that contains
simplified summaries from the Eureka Alert science news website.

Dangovski et al. (2021) introduce science journalism as a downstream task of
abstractive summarization and story generation. They apply BERT-based models
with a prompting method for data augmentation on a monolingual dataset
collected from Science daily press releases and scientific papers. They use
three existing models for their work: sci-bert, cnn-based sequence-to-sequence
model and story generation model.

We find no similarity between these studies and our work except for the
overlap of abstractive summarization, however, we focus on cross-lingual
summarization.

##  3 Proposed Model

Our model jointly trains for Simplification and Cross-lingual Summarization
(simcsum). We first define mtl and our tasks, and then discuss the
architecture of our proposed model.

###  3.1 Multi-Task Learning

mtl is an approach in deep learning which improves generalization by learning
different noise patterns from data related to different tasks. We define our
mtl-based model trained on two tasks: simplification and cross-lingual
summarization. We adopt hard parameter sharing as it improves the positive
transfer and reduces the risk of overfitting (Ruder, 2017).

###  3.2 Summarization

We define single-document abstractive summarization as follows. Given a text
$X\\!=\\!\left\\{x_{1},\cdots,x_{m}\right\\}$ with $m$ number of sentences
comprising of a set of words (vocabulary)
$W_{X}\\!=\\!\left\\{w_{1},\cdots,w_{X}\right\\}$, an (encoder-decoder-based)
abstractive summarizer generates a summary
$Y\\!=\\!\left\\{y_{1},\cdots,y_{n}\right\\}$ with $n$ sentences that contain
salient information of $X$, where $m\\!\gg\\!n$ and $Y$consisting of a set of
words $W_{Y}\\!=\\!\left\\{w_{1},\cdots,w_{Y}|\,\exists\,w_{i}\notin
W_{X}\right\\}$. The decoder learns the conditional probability distribution
over the given input and all previously generated words, where $t$ denotes the
time step.

$P_{\theta}(Y|X)=\log P(y_{t}|\;y_{<t},X)$ (1)

Cross-lingual summarization adds another dimension of language for
simultaneous translation and summarization. Given a text
$X^{l}\\!=\\!\left\\{x_{1}^{l},\cdots,x_{m}^{l}\right\\}$ in a language $l$
with $m$ sentences comprising of a vocabulary
$W_{X}^{l}\\!=\\!\left\\{w_{1}^{l},\cdots,w_{X}^{l}\right\\}$, a cross-lingual
summarizer generates a summary
$Y^{k}\\!=\\!\left\\{y_{1}^{k},\cdots,y_{n}^{k}\right\\}$ in a language $k$
that contains salient information in $X$, where $m\\!\gg\\!n$ and $Y$
consisting of a vocabulary
$W_{Y}^{k}\\!=\\!\left\\{w_{1}^{k},\cdots,w_{Y}^{k}|\,\exists\,w_{i}\,\notin
W_{X}^{l}\right\\}$. The conditional probability is the same as in Eq.1, the
only difference being that the language on the decoder side is different from
the encoder side.

Figure 1: simcsum consists of one shared encoder with two decoding sides for
Simplification and cross-lingual Summarization.

###  3.3 Simplification

We define the document-level (lexical and syntactic) simplification task as
follows. Given a text $X\\!=\\!\left\\{x_{1},\cdots,x_{m}\right\\}$ with $m$
sentences comprising of a vocabulary
$W_{X}\\!=\\!\left\\{w_{1},\cdots,w_{X}\right\\}$, a simplification model
generates the output text $Y\\!=\\!\left\\{y_{1},\cdots,y_{n}\right\\}$ that
retains the primary meaning of $X$, yet more comprehensible as compared to
$X$, where $m\\!\approx\\!n$ and $Y$ consisting of a vocabulary
$W_{Y}\\!=\\!\left\\{w_{1},\cdots,w_{Y}|\,\exists\,w_{i}\notin
W_{X}\right\\}$. The conditional probability is also the same as in Eq.1.

###  3.4 SimCSum

We illustrate the framework of simcsum333The code of this work will be public
on GitHub upon acceptance. in Figure 1. simcsum jointly trains on
simplification and cross-lingual summarization. simcsum adopts hard parameter
sharing where the encoder is shared between the tasks while having two task-
specific decoders. The decoders only share the cross-attention layer, and the
loss is combined to update the parameters ($\theta$). We opt for two decoders
because each task’s output language and length differ. The training method is
described in Algorithm 1. Here we discuss the further details of simcsum. For
all mathematical definitions, $\mathcal{T}\in\left\\{sim,sum\right\\}$ denotes
a task.

Algorithm 1 Training of simcsum for Simplification and Cross-lingual
Summarization

Input:

for each $d\in trainset$ do

$\triangleright$ Process each instance $d$ of dataset $D$ for tuples $I$ of
input $x$ and targets for each task $\mathcal{T}$

Create $I\langle x,y_{\mathcal{T}}\rangle$

end for

Initialize model parameters $\theta$

Set maximum Epoch $Ep$

for epoch 1 to $Ep$ do

for $b\in trainset$ do

$\triangleright$ $b$ is a mini-batch containing I from $trainset$

$\triangleright$ simcsum consists of Encoder $E$, two Decoders
$D_{\mathcal{T}}$

Feed $x$ to $E$ and get the cross-attention

Feed $y_{\mathcal{T}}$ to $D_{\mathcal{T}}$

Feed the cross-attention to $D_{\mathcal{T}}$ [eq. 2]

$t\leftarrow 0$

while $\theta_{t}$ is not converged do

$t\leftarrow t+1$

Compute $\mathcal{L}(\theta)$ [eq. 3]

Compute gradient $\nabla(\theta_{t})$

Update $\theta_{t}\leftarrow\theta_{t-1}-\eta\nabla(\theta)$

end while

end for

end for

####  3.4.1 Architecture

Considering the excellent text generation performance of multi-lingual Bart
(mbart) (Liu et al., 2020), we implement the simcsum model based on it and
modify it for two decoding sides for each task. Each encoder and decoder stack
consists of 12 layers.

Self-Attention. Each layer of encoder/decoder has its self-attention,
consisting of keys, values, and queries generated from the same sequence.

$\displaystyle A(Q,K,V)=Softmax(\frac{Q\cdot K^{T}}{\sqrt{d_{k}}})\cdot V$

where $Q$ is a query, $K^{T}$ is transposed $K$ (key) and $V$ is the value.
All parallel attentions are concatenated to generate multi-head attention
scaled with a weight matrix $W$.

$\displaystyle MH(Q,K,V)=Concat(A_{1},\cdots,A_{h})\cdot W^{O}$

Cross-attention. The cross-attention connects the encoder and decoder and
provides the decoder with a weight distribution at each step, indicating the
importance of each input token in the current context. We concatenate the
cross-attention of both decoders.

$\displaystyle A(E,D_{\mathcal{T}})=Concat(Softmax(\frac{D_{\mathcal{T}}\cdot
E^{T}}{\sqrt{d_{k}}})\cdot E)$ (2)

where $E$ is the encoder representation, $D_{\mathcal{T}}$ is the task-
specific decoder contextual representation, and $d_{k}$ is the model size.

####  3.4.2 Training Objective

We train our model end-to-end to maximize the conditional probability of the
target sequence given a source sequence. We define the task-specific loss as
follows.

$\displaystyle\mathcal{L}_{\mathcal{T}}(\theta)=\sum_{n=1}^{N}\log
P(y_{\mathcal{T}_{t}}|y_{\mathcal{T}_{<t}},x;\theta)$

where $x$ represents the input, $y$ is the target, $N$ is the mini-batch size,
$t$ is the time step and $\theta$ denotes to learnable parameters. We define
the total loss of our model by task-specific losses where
$\lambda_{\mathcal{T}}$ is an assigned weight to each task.

$\displaystyle\mathcal{L}(\theta)=\sum\lambda_{\mathcal{T}}\cdot\mathcal{L}_{\mathcal{T}}(\theta)$
(3)

##  4 Experiments

###  4.1 Datasets

We use two non-synthetic cross-lingual scientific summarization datasets.

####  4.1.1 Summarization

wikipedia. It is harvested from Wikipedia Science Portal for English and
German (Fatima and Strube, 2021). Wikipedia Science Portal contains articles
in various science fields. The wikipedia dataset consists of two versions:
monolingual and cross-lingual. We use only the cross-lingual part of this
dataset. It consists of 50,132 English articles (avg. 1572 words) and German
summaries (avg. 100 words).

spektrum. It is collected from Spektrum der Wissenschaft (Fatima and Strube,
2021). Spektrum is a famous science magazine (Scientific American) in Germany.
It covers various topics in diversified science fields: astronomy, biology,
chemistry, archaeology, mathematics, physics, _etc_. The spektrum dataset
contains 1510 English articles (avg. 2337 words) and German summaries (avg.
361 words).

####  4.1.2 Simplification

We construct a synthetic wikipedia dataset for the simplification task by
applying Keep-It-Simple (kis) (Laban et al., 2021). To create the simplified
wikipedia, we fine-tune kis on wikipedia English articles as kis is an
unsupervised model and does not require parallel data. The simplified
wikipedia consists of the original English articles paired with simplified
English articles.

We perform English text simplification because most of the simplification work
has been done in the English language (Al-Thanyyan and Azmi, 2021), and very
few studies cover the German language (Aumiller and Gertz, 2022; Weiss and
Meurers, 2018; Hancke et al., 2012) for children and dyslexic persons (not
suitable for scientific simplification). Moreover, most of the work focuses on
lexical or sentence level (Sun et al., 2021). To the best of our knowledge,
kis is the only sota paragraph-level unsupervised simplification model.

###  4.2 Split and Usage

We use wikipedia for training, validation and testing (80/10/10), while we use
spektrum for zero-shot adaptability as a case study.

All plm baselines are trained on wikipedia where each instance $I$ in the
training set consists of $<\\!x,y\\!>$ where $x$ is the input English text and
$y$ is the target German summary.

simcsum is trained on wikipedia where each instance $I$ in the training set
contains $<\\!x,y_{sim},y_{sum}\\!>$ where $x$ denotes the input English
article and $y_{sim}$ refers to the simplified English article and $y_{sum}$
is the target German summary.

###  4.3 Models

Baselines. Almost all cross-lingual mtl models in §2 are based on translation
and summarization, and none of them applies simplification. So we select
several state-of-the-art (sota) plms that accept long input texts as
baselines.

We fine-tune the following baselines: (1) mt5 (Xue et al., 2021), (2) mbart
(Liu et al., 2020), (3) pegasus (Zhang et al., 2020a), (4) LongFormer-Encoder-
Decorder (long-ed) (Beltagy et al., 2020), and (5) xlsum (Hasan et al., 2021)
and (6) bigbird (Zaheer et al., 2020).

In addition, we define a baseline, Simplify-Then-Summarize, based on kis and
mbart models as a pipeline. We report it as kis-mbart in our experiments.

SimCSum. We set $\lambda_{Sum}\\!=\\!0.75$ for simcsum based on the best
results on the wikipedia validation set.

###  4.4 Training and Inference

The libraries, hardware and training time details are presented in Appendix A.
Here, we discuss hyper-parameters.

Baselines. We fine-tune all models for a maximum of 25 epochs and average the
results of 5 runs for each model. We use a batch size of 4-16, depending on
the model size. We use a learning rate (lr) of $5e^{-5}$ and 100 warm-up steps
to avoid over-fitting of the fine-tuned models. We use the Adam optimizer with
a lr linearly decayed lr scheduler. The encoder language is set to English,
and the decoder language is German.

SimCSum. We adopt similar settings as used for baselines, except for the batch
size fixed to 4. We only generate tokens from the Summarization decoder side
in the inference period. We use beam search of size 5 and a tri-gram block
during the decoding stage to avoid repetition.

###  4.5 Evaluation

Automatic. We evaluate all models with three metrics. rouge (Lin, 2004) is a
standard metric for summarization. bert-score (Zhang et al., 2020b) (bs) is a
recent metric for summarization and simplification as an alternative metric to
n-gram-based metrics and applies contextual embeddings. For English text
simplification, sari and Flesch Kincaid Grade Level (fkgl) are the mostly used
metrics (Kariuk and Karamshuk, 2020; Omelianchuk et al., 2021; Laban et al.,
2021). As our output language is German, we decide to use a variation of
Flesch Kincaid score for the German language, _i.e_., Flesch Kincaid Reading
Ease (fre) (Kincaid et al., 1975) (Appendix B §B.2).

Human. We conduct a human evaluation to compare the outputs of simcsum with
mbart (baseline) for the same linguistic properties. Our annotators are two
university students from the Computational Linguistics department with fluent
German and English skills. It is worth mentioning that human evaluation of
long cross-lingual scientific text is challenging and costly because it
requires bi-lingual annotators with a scientific background.

##  5 Results

###  5.1 wikipedia

We report f-score444Precision and Recall of rouge and bert-score can be found
in Table E.4 in Appendix E. of rouge and bert-score and fre of all models in
Table 1. The first block includes the fine-tuned plms models, the second block
presents the pipeline baseline, and the last block includes simcsum. From
Table 1, we find that simcsum outperforms all baselines for every metric. We
compute the statistical significance of the results with the Mann-Whitney two-
tailed test for a p-value ($p\\!<\\!.001$). Interestingly, wikipedia summaries
are not simplified compared to spektrum summaries; still, simcsum performs
better on wikipedia than the baselines. We interpret that the simplification
auxiliary task helps the simcsum to learn better contextual representation and
produce more relevant German words. We infer from the results that joint
learning of simplification and cross-lingual summarization improves the
quality of summaries.

models r1 r2 rl bs fre gold - - - - 36.93 mt5 26.79 12.65 23.40 69.12 45.42
mbart 31.43 13.20 25.12 70.52 44.67 pegasus 29.30 13.93 24.62 69.83 43.39
xlsum 31.91 13.30 24.14 70.04 37.83 bigbird 29.23 13.72 24.60 69.19 41.42
long-ed 15.11 06.82 13.67 63.94 24.48 kis-mbart 32.02 12.39 24.72 70.52 45.76
simcsum 34.50† 14.36† 25.85† 71.60† 46.86†

Table 1: The wikipedia results for all baselines and simcsum. gold denotes the
reference summaries. Underline refers to the best baseline results, and
bold$\dagger$ denotes the best overall results with significant improvements
($p\\!<\\!.001$).

Among the baselines, almost all models demonstrate comparable performance
except long-ed. For r1, kis-mbart perform better than other models, however,
mbart and xlsum performance are also similar. pegasus takes the lead for r2,
and mbart shows higher performance for rl. kis-mbart and mbart take the lead
for bs among the baselines. For fre, a score between $30-50$ presents the
readability level best understood by college graduates. The wikipedia
summaries fall in this range. For fre, kis-mbart performs better than the
other baselines. Interestingly, almost all baselines except bigbird and xlsum
demonstrate good performance. Comparing kis-mbart and mbart, kis-mbart
performs slightly better than mbart for r1 and fre, equal for bs and slightly
lower for r2 and rl. We infer that it is due to the impact of the
simplification module in kis-mbart.

###  5.2 Case Study: spektrum

Table 2 presents the results of all models on spektrum. We find a similar
pattern that simcsum outperforms all baselines. We also compute the
statistical significance of these results with the same procedure. The
spektrum results are on the lower side compared to the wikipedia results due
to zero-shot adaptability, especially for r2\. We infer that it is due to the
impact of the computation method of rouge score as it is an n-gram-based
metric (Ng and Abrecht, 2015). The spektrum summaries have higher fre scores
compared to wikipedia. Interestingly, we find that all baselines perform lower
than the gold summaries. However, the simcsum score is similar to the gold
summaries. Comparing the performance of mbart and kis-mbart, kis-mbart
performs slightly lower than mbart for all scores except r1because only
wikipedia is used for fine-tuning of both models in kis-mbart.

models r1 r2 rl bs fre gold - - - - 40.76 mt5 09.21 00.75 06.50 58.52 38.18
mbart 16.16 01.47 13.89 62.11 39.17 pegasus 11.49 00.95 08.01 60.56 37.93
xlsum 17.10 01.63 09.79 62.25 33.83 bigbird 12.28 01.04 08.65 59.97 36.24
long-ed 01.32 00.11 01.18 51.85 30.16 kis-mbart 17.33 01.61 12.97 61.83 39.21
simcsum 18.88† 01.82† 14.16† 63.47† 40.03†

Table 2: The spektrum results for all baselines and simcsum. gold denotes the
reference summaries. Underline refers to the best baseline results, and bold
with $\dagger$ denotes the best overall results with significant improvements
($p\\!<\\!.001$).

Human Evaluation. We compare the simcsum and mbart outputs for analyzing
linguistic qualities because simcsum’s architecture is based on mbart. We
provide $30\times 2$ (for each model) random summaries with their original
texts. We ask two annotators to evaluate each document for three linguistic
properties on a Likert scale from $1-5$. The first five samples are used to
calibrate the annotations of annotators, and then each annotator provides
independent judgments on the rest of the samples.

Table 3 shows the human evaluation results. The samples used for calibration
are not used for computing the scores (guidelines in Appendix C). We compute
the inter-rater reliability by using Krippendorff’s
$\alpha$555https://github.com/LightTag/simpledorff. We find that simcsum
improves the fluency, relevance and readability of outputs. We present a few
comparative examples of simcsum and mbart in Appendix E.

models fluency relevance simplicity mbart 2.28 (0.64) 1.64 (0.70 ) 1.86 (0.56)
simcsum 2.62 (0.87) 2.76 (0.78) 2.88 (0.81)

Table 3: The spektrum human evaluation for mbart and simcsum. The average
scores (Krippendorff’s $\alpha$) for each linguistic feature are presented
here.

##  6 Analysis: spektrum

We explore three further dimensions along with extended readability for in-
depth analysis: lexical diversity, syntactic and error types to determine the
quality of generated summaries. These types of analysis are well-known in nlp
for textual analysis (Aluisio et al., 2010; Hancke et al., 2012; Vajjala and
Lučić, 2018; Mosquera, 2022; Weiss and Meurers, 2022). The lexical diversity
and readability scores are computed over all spektrum’s reference summaries
(Gold) and outputs of mbart and simcsum. The gold summaries’ score is a
guideline for how similar the models’ outputs are to gold summaries.

###  6.1 Lexical Diversity

Lexical diversity estimates how language is distributed overall and how much
cohesion is present in the text as synonyms. It is a good indicator of the
readability of a text. We calculate Shannon Entropy Estimation (see) (Shannon,
1948) and Measure of Textual Lexical Diversity (mtld) (McCarthy, 2005) to find
lexical diversity (see Appendix B §B.1 for formula).

see presents a text’s “informational value” and language diversity. It is a
language-dependent feature, and its value varies for different languages.
Higher see scores suggest higher lexical diversity. We aim to get similar see
as Gold summaries. Table 4 shows see scores of mbart and simcsum that are
similar to Gold summaries suggesting the similar informational value of all
summaries.

mtld is considered a robust version of the type-token ratio (ttr) and
calculates lexical diversity with no impact of text length. Higher mtld
represents the greater vocabulary richness. Table 4 presents mtld scores of
mbart and simcsum. The gold summaries have the highest scores, while simcsum
is the second highest and mbart has the lowest score. These scores suggest
that the lexical richness of all three groups is not similar, in contrast to
see results. However, the simcsum outputs are more lexically diverse than the
mbart outputs. We infer from the improved simcsum scores that joint learning
of simplification and cross-lingual summarization impacts word generation.
These results also suggest that mtld provides a better estimation of lexical
diversity for our summaries.

features gold mbart simcsum Lexical Diversity see $\downarrow$ 4.25 (0.04)
4.26 (0.1) 4.25 (0.1) mtld $\uparrow$ 201 (41.4) 65.13 (33.3) 91.75 (33.1)
Readability scores cli $\downarrow$ 18.45 (1.7) 21.64 (4.7) 20.96 (4.8) ari
$\downarrow$ 18.99 (2.4) 21.07 (5.5) 20.26 (5.2)

Table 4: Lexical diversity and readability features’ average scores (standard
deviation).

###  6.2 Readability Scores

Readability scores measure comprehension levels of the text. One of the
syllables-based readability scores is already presented in §5. Coleman and
Liau (1975) suggests that word length in letters is a better predictor of
readability than syllables. We calculate Coleman Liau Index (cli) (Coleman and
Liau, 1975) and Automated Readability Index (ari) (Senter and Smith, 1967) as
these do not rely on syllables (see Appendix B §B.1 for formula).

cli computes scores on word lengths. ari computes scores on characters, words
and sentences. For both cli and ari, the lower score is better as it shows the
ease of reading and understanding. We interpret from Table 4 that Gold
summaries have the lowest score, simcsum has the second-lowest score, and
mbart has the highest score. We infer from the improved simcsum scores that
joint learning of simplification and cross-lingual summarization impacts both
word and sentence level because cli only focuses on words, while ari includes
sentences also.

features$\downarrow$ gold mbart simcsum ASL 24.09 (4.2) 24.15 (7.2) 20.97
(6.5) ADD 3.60 (0.3) 4.16 (1.1) 3.91 (1.1) ADW 0.93 (0.04) 0.95 (0.02) 0.94
(0.04) ATH 8.32 (0.7) 8.72 (1.5) 8.57 (1.5)

Table 5: Syntactic features’ average scores (standard deviation).

###  6.3 Syntactic Analysis

Syntactic analysis elaborates on how words and phrases are related in a
sentence structure. We perform it with constituency trees on $25\times 2$ (for
each model) random summaries from mbart, simcsumand the gold summaries. The
total number of sentences for mbart is 70, for simcsum is 80 and for gold is
131. Table 5 presents four syntactic features (see Appendix B §B.2 for
definitions).

We infer from the average sentence length (asl) that simcsum produces shorter
sentences than mbart and gold summaries, which exhibits syntactic simplicity.
A small average dependency distance (add) shows that words with a dependency
relation are close together, making the text easier to understand. Table 5
shows that simcsum summaries have a smaller average dependency than mbart,
much closer to gold summaries. Fewer dependents per word (adw) make a text
less ambiguous and thus easier to follow. Table 5 shows the simcsum outputs
have fewer dependents than the mbart outputs and are similar to gold
summaries. The average tree height (ath) explains the syntactic structural
complexity of a sentence. Table 5 shows that simcsum outputs are less
structurally complex than mbart outputs, however, gold summaries have the
least average tree height. We infer from the syntactic analysis that joint
learning of simplification and cross-lingual summarization positively impacts
the syntactic properties of summaries.

###  6.4 Error Analysis

To further explore the challenges of improving cross-lingual science
summaries, we randomly select $25\times 2$ (for each model) summaries from the
simcsum and mbart outputs. We find three main categories of errors in the
manual inspection. Table 6 presents the occurrences of these errors in each
model. Appendix D presents some examples from error analysis and its
guidelines.

Non-German Words. It is the error type where the models either produce non-
existent German words or partially English-German or another language words.
We find that mbart is more prone to produce such errors. We infer that it is
due to the imbalance between the pre-trained and fine-tuned dataset sizes.
These models are pre-trained on many languages and usually fine-tuned on
comparatively smaller data. simcsum tends to produce fewer errors due to data
augmentation (simplification data) during the training.

Wrong Name Entities. It is the error type where the models produce wrong name
entities, such as cities or country names and persons’ first and last names.
We find that both models tend to produce such errors, however, the percentage
of such errors is quite low. We infer that the models overestimate or
underestimate the probability of word sequences present in data.

Unfaithful Information. It is the error type where we find some (new)
information in generated summaries that is not faithful to the source
documents. We infer that this error is caused by long inputs where the model
tends to hallucinate and generates some content that cannot be verified from
the source. We find that simcsum makes similar errors as mbart for this error
type.

error types mbart simcsum Non-German words 83 35 Wrong name entities 1 2
Unfaithful information 3 3

Table 6: Error occurrences for mbart and simcsum summaries which may contain
multiple errors.

##  7 Conclusions

In this paper, we explore the task of cross-lingual science journalism. We
propose a novel multi-task model, simcsum, that combines two high-level nlp
tasks, simplification and cross-lingual summarization. simcsum jointly trains
for reducing linguistic complexity and cross-lingual abstractive
summarization. We also introduce a pipeline-based strong baseline for cross-
lingual science journalism. Our empirical investigation shows the
significantly superior performance of simcsum over the sota baselines on two
non-synthetic cross-lingual scientific datasets, also indicated by human
evaluation. Furthermore, our in-depth linguistics analysis shows how multi-
task learning in simcsum has lexical and syntactic impacts on the generated
summaries. In the last, we perform error analysis to find what kind of errors
has been produced by the model. In the future, we plan to add modules for
linguistically informed simplification.

##  8 Limitations

We proposed simcsum for the Cross-lingual Science Journalism task and verified
its performance for wikipedia and spektrum datasets for the English-German
language pair. We believe that simcsum is adaptable for other domains and
languages. However, we have not verified it experimentally and limited our
experiments to English-German scientific texts.

Our model jointly trains on two high-level nlp tasks, which takes slightly
more time than its base model - mbart, as it has more parameters to learn
during the training. Moreover, our model is trained on synthetic
simplification data, which may create a dependency on the simplification model
- kis. Therefore, we plan to add linguistically informed simplification
modules in our model in our future work. We also find during error analysis
that both mbart and simcsum have problems (repetition or unfaithful
information) with long inputs, which need further investigation.

##  9 Ethical Consideration

Reproducibility.  We discussed all relevant parameters, training details, and
hardware information in §4.4 and Appendix A.

Legal Consent. We obtained legal consent from Spektrum der Wissenschaft to use
their dataset. We adopted the public implementations with mostly recommended
settings, wherever applicable.

## References

  * Al-Thanyyan and Azmi (2021) Suha S Al-Thanyyan and Aqil M Azmi. 2021.  Automated text simplification: a survey.  _ACM Computing Surveys (CSUR)_ , 54(2):1–36.
  * Aluisio et al. (2010) Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. 2010.  Readability assessment for text simplification.  In _Proceedings of the NAACL HLT 2010 fifth workshop on innovative use of NLP for building educational applications_ , pages 1–9.
  * Aumiller and Gertz (2022) Dennis Aumiller and Michael Gertz. 2022.  Klexikon: A german dataset for joint summarization and simplification.  _arXiv preprint arXiv:2201.07198_.
  * Bai et al. (2021) Yu Bai, Yang Gao, and He-Yan Huang. 2021.  Cross-lingual abstractive summarization with limited parallel resources.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6910–6924.
  * Bai et al. (2022) Yu Bai, Heyan Huang, Kai Fan, Yang Gao, Yiming Zhu, Jiaao Zhan, Zewen Chi, and Boxing Chen. 2022.  Unifying cross-lingual summarization and machine translation with compression rate.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1087–1097.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Cachola et al. (2020) Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld. 2020.  TLDR: Extreme summarization of scientific documents.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4766–4777, Online. Association for Computational Linguistics.
  * Cao et al. (2020) Yue Cao, Hui Liu, and Xiaojun Wan. 2020.  Jointly learning to align and summarize for neural cross-lingual summarization.  In _Proceedings of the 58th annual meeting of the association for computational linguistics_ , pages 6220–6231.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.
  * Coleman and Liau (1975) Meri Coleman and Ta Lin Liau. 1975.  A computer readability formula designed for machine Scoring.  _Journal of Applied Psychology_ , 60(2):283.
  * Dangovski et al. (2021) Rumen Dangovski, Michelle Shen, Dawson Byrd, Li Jing, Desislava Tsvetkova, Preslav Nakova, and Marin Soljacic. 2021.  We Can Explain Your Research in Layman’s Terms: Towards Automating Science Journalism at Scale.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 12728–12737, Online.
  * Fatima and Strube (2021) Mehwish Fatima and Michael Strube. 2021.  A novel Wikipedia based dataset for monolingual and cross-lingual summarization.  In _Proceedings of the Third Workshop on New Frontiers in Summarization_ , pages 39–50, Online and in Dominican Republic. Association for Computational Linguistics.
  * Goldsack et al. (2022) Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022.  Making science simple: Corpora for the lay summarisation of scientific literature.  _arXiv preprint arXiv:2210.09932_.
  * Hancke et al. (2012) Julia Hancke, Sowmya Vajjala, and Detmar Meurers. 2012.  Readability classification for german using lexical, syntactic, and morphological features.  In _Proceedings of COLING 2012_ , pages 1063–1080.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021.  Xl-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703.
  * Kariuk and Karamshuk (2020) Oleg Kariuk and Dima Karamshuk. 2020.  Cut: Controllable unsupervised text simplification.  _arXiv preprint arXiv:2012.01936_.
  * Kim et al. (2016) Minsoo Kim, Dennis Singh Moirangthem, and Minho Lee. 2016.  Towards abstraction from extraction: Multiple timescale gated recurrent unit for summarization.  In _Proceedings of the 1st Workshop on Representation Learning for NLP_ , pages 70–77, Berlin, Germany. Association for Computational Linguistics.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975\.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.  Technical report, Naval Technical Training Command Millington TN Research Branch.
  * Laban et al. (2021) Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021.  Keep it simple: Unsupervised simplification of multi-paragraph text.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6365–6378, Online. Association for Computational Linguistics.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * McCarthy (2005) Philip M McCarthy. 2005.  _An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD)_.  Ph.D. thesis, The University of Memphis.
  * Mosquera (2022) Alejandro Mosquera. 2022.  Tackling data drift with adversarial validation: An application for german text complexity estimation.  In _Proceedings of the GermEval 2022 Workshop on Text Complexity Assessment of German Text_ , pages 39–44.
  * Ng and Abrecht (2015) Jun-Ping Ng and Viktoria Abrecht. 2015.  Better summarization evaluation with word embeddings for rouge.  _arXiv preprint arXiv:1508.06034_.
  * Nikolov et al. (2018) Nikola I Nikolov, Michael Pfeiffer, and Richard HR Hahnloser. 2018.  Data-driven Summarization of Scientific Articles.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Omelianchuk et al. (2021) Kostiantyn Omelianchuk, Vipul Raheja, and Oleksandr Skurzhanskyi. 2021.  Text simplification by tagging.  _arXiv preprint arXiv:2103.05070_.
  * Ouyang et al. (2019) Jessica Ouyang, Boya Song, and Kathy McKeown. 2019.  A robust abstractive system for cross-lingual summarization.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2025–2031, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ruder (2017) Sebastian Ruder. 2017.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get To The Point: Summarization with Pointer-Generator Networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Senter and Smith (1967) RJ Senter and Edgar A Smith. 1967.  Automated readability index.  Technical report, Cincinnati Univ OH.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423.
  * Sun et al. (2021) Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021.  Document-level text simplification: Dataset, criteria and baseline.  _arXiv preprint arXiv:2110.05071_.
  * Takase and Okazaki (2020) Sho Takase and Naoaki Okazaki. 2020.  Multi-task learning for cross-lingual abstractive summarization.  _arXiv preprint arXiv:2010.07503_.
  * Takeshita et al. (2022) Sotaro Takeshita, Tommaso Green, Niklas Friedrich, Kai Eckert, and Simone Paolo Ponzetto. 2022.  X-SCITLDR: cross-lingual extreme summarization of scholarly documents.  _ArXiv preprint_ , abs/2205.15051.
  * Vadapalli et al. (2018a) Raghuram Vadapalli, Bakhtiyar Syed, Nishant Prabhu, Balaji Vasan Srinivasan, and Vasudeva Varma. 2018a.  Sci-blogger: A step towards automated science journalism.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018_ , pages 1787–1790. ACM.
  * Vadapalli et al. (2018b) Raghuram Vadapalli, Bakhtiyar Syed, Nishant Prabhu, Balaji Vasan Srinivasan, and Vasudeva Varma. 2018b.  When science journalism meets artificial intelligence : An interactive demonstration.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 163–168, Brussels, Belgium. Association for Computational Linguistics.
  * Vajjala and Lučić (2018) Sowmya Vajjala and Ivana Lučić. 2018.  Onestopenglish corpus: A new corpus for automatic readability assessment and text simplification.  In _Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications_ , pages 297–304.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Weiss and Meurers (2018) Zarah Weiss and Detmar Meurers. 2018.  Modeling the readability of german targeting adults and children: An empirically broad analysis and its cross-corpus validation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 303–317.
  * Weiss and Meurers (2022) Zarah Weiss and Detmar Meurers. 2022.  Assessing sentence readability for german language learners with broad linguistic modeling or readability formulas: When do linguistic insights make a difference?  In _Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)_ , pages 141–153.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ , 33:17283–17297.
  * Zaman et al. (2020) Farooq Zaman, Matthew Shardlow, Saeed-Ul Hassan, Naif Radi Aljohani, and Raheel Nawaz. 2020.  Htss: A novel hybrid text summarisation and simplification architecture.  _Information Processing & Management_, 57(6):102351.
  * Zhang et al. (2020a) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a.  PEGASUS: pre-training with extracted gap-sentences for abstractive summarization.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 11328–11339. PMLR.
  * Zhang et al. (2020b) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhu et al. (2019) Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019.  NCLS: Neural cross-lingual summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3054–3064, Hong Kong, China. Association for Computational Linguistics.
  * Zhu et al. (2020) Junnan Zhu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2020.  Attend, translate and summarize: An efficient method for neural cross-lingual summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1309–1321, Online. Association for Computational Linguistics.

##  Appendix A Training and Inference

Libraries. We train all models with Pytorch666https://pytorch.org/, Hugging
Face777https://huggingface.co/ integrated with
DeepSpeed888https://www.microsoft.com/en-us/research/project/deepspeed/ for
parallel model training with ZeRO-2. We apply ZeRO-2999Initially, we used
ZeRO-3 offload with fp16 evaluation, and the training became quite slow as it
consumes a lot of time for offloading during evaluation. to enable model
parallelism. ZeRO-2 reduces the memory footprints for gradients and optimizer
because it shards the optimizer states and gradients across gpus.

Hardware. For all models, we complete training and inference on 4 Tesla p40
gpus each with 24gb memory.

Training Time. mbart takes 1 day and 17 hours, mt5 takes 10 hours, pegasus
takes 1 day and 8 hours, xlsum takes 1 day and 3 hours, long-ed takes almost 4
days, and bigbird takes 2 days to complete 25 epochs. simcsum takes 2 days to
complete 25 epochs.

##  Appendix B Analysis: spektrum

###  B.1 Lexical Diversity

see is calculated with a frequency table as follows.

$\displaystyle H(x)=\sum_{i=1}^{n}p(x_{i})log_{2}\frac{1}{p(x_{i})}$

where $H(x)$ is the total amount of information in an entire probability
distribution. $P(x_{i})$ refers to the frequency of a token appearing in the
text, and $1/p(x)$ denotes the information of each case.

mtld divides the texts into sequences having the same ttr and then calculates
the mean length of the sequences.

###  B.2 Readability

fre is calculated as follows:

$\displaystyle FRE=180-ASL-(58.5\times ASW)$

where average sentence length (asl) is the number of words divided by the
number of sentences in the text. The average number of syllables per word
(asw) is the number of syllables divided by the number of words in the text.
The numeric values are language-dependent constants.

cli is calculated as follows:

$\displaystyle CLI=5.88\times\frac{L}{W}-29.6\times\frac{S}{W}-15.8$

where $L$ is the total number of characters (including numbers and
punctuation), $W$ is the total number of words, and $S$ is the total number of
sentences in a given text.

ari is computed as follows:

$\displaystyle ARI=4.71\times\frac{L}{W}+0.5\times\frac{W}{S}-21.43$

where $L$ is the total number of characters (including numbers and
punctuation), $W$ is the total number of words, and $S$ is the total number of
sentences in a given text.

###  B.3 Syntactic Analysis

We use Stanza101010https://stanfordnlp.github.io/stanza/constituency.html to
extract dependency relations and Stanford Parser
111111https://nlp.stanford.edu/software/lex-parser.shtml to extract
constituency trees for each summary. Before tree generation, we replace all
German umlauts (ä, ö, ü and ß) in the summaries with their replacements (ae,
oe, ue and ss) due to encoding issues of the Stanford Parser.

Average Sentence Length. It is the number of tokens in the sentences averaged
over the number of sentences in a summary.

Average Dependency Distance. It is the averaged dependency distance over the
sentences, which means the distance between the dependency heads and their
dependents.

Average Dependents per Word. It computes the average number of dependents for
each word.

Average Tree Height. For computing the average tree height of a summary, we
calculate the height of every tree and average it over the sentences.

##  Appendix C Human Evaluation

###  C.1 Task

We provided annotators with 30 examples of documents paired with a reference
summary and two system-generated summaries. The models’ identities had not
been revealed. The annotators had to rate each model summary for the following
linguistic properties after reading the English document and the German
summaries. We asked annotators to use the first 5 examples to resolve the
annotator’s conflict and to find a common consensus for rating the linguistic
aspects. However, the rest of the examples were annotated independently.

###  C.2 Linguistic Properties

We asked annotators to annotate each summary for the following linguistic
properties.

Relevance. A summary delivers adequate information about the original text.
Relevance determines the content relevancy of the summary.

Fluency. The words and phrases fit together within a sentence, and so do the
sentences. Fluency determines the structural and grammatical properties of a
summary.

Simplicity. Lexical (word) and syntactic (syntax) simplicity of sentences. A
simple summary should have minimal use of complex words/phrases and sentence
structure.

###  C.3 Scale

We use a Likert scale from 1 to 5 to score each property (1:worst | 2:bad | 3:ok | 4:good | 5:best). These scores should be assigned by comparing the outputs of both models.

##  Appendix D Error Analysis

###  D.1 Guidelines

We define our informal guidelines for the error analysis as follows. To find
the errors in the mbart and simcsum outputs, we compare them to each other, to
the spektrum German gold summary and the original English text.

Non-German Words. To find them, it is sufficient to read through our model
outputs and look up any unknown words. If one of the unknown words turns out
to be a non-German word, we mark them in red.

Wrong name entities. We find wrong-name entities by comparing the names in
both system outputs to the reference summary. If the names differ, we verify
with the original text that they refer to the same person and thus represent a
mistake by the model, and we mark them in blue.

Unfaithful information. We find new/unfaithful information by looking up every
piece of information in the model outputs in the reference summary. We search
for this information in the original text, and if it is not present there, it
is clear that the model produced new information that is not faithful to the
source text. We mark this information in orange.

###  D.2 Examples

Target: im freigehege aufzuwachsen und sich dort im schlamm zu suhlen, stärkt
offenbar das immunsystem von schweinen : verglichen mit artgenossen, die in
einem stall gehalten wurden, hatten freilandschweine eine höhere anzahl von
gesundheitsfördernden darmbakterien. zu diesem ergebnis kommen jetzt
wissenschaftler um denise kelly von der university of aberdeen. auch gene, die
auf entzündungsreaktionen hindeuten, waren bei tieren, die im außenbereich
lebten, wesentlich weniger aktiv. […] Translation: Growing up in an outdoor
enclosure and wallowing in the mud there apparently strengthens the immune
system of pigs : compared to conspecifics that were kept in a pen, free-range
pigs had a higher number of health-promoting intestinal bacteria. scientists
led by denise kelly from the university of aberdeen are now coming to this
conclusion. genes that indicate inflammatory responses were also significantly
less active in animals that lived outdoors. […] mbart: als fruehmikrobielle
kolonisation wird dieabdeckung der brustdruese von schweinen bei der fruehen
entwicklung des erwachsenen menschen bezeichnet. sie wird in der medizin, der
mikrobiologie und der pathologie erforscht. early microbial colonization of
the gut reduced the incidence of infectious, entzuendlichen und
autoimmunischen diseasen reduced the incidence of infectious, inflammatory and
autoimmune diseases. Translation: the covering of the mammary gland of pigs
during the early development of the adult human is referred to as early
microbial colonization. it is researched in medicine, microbiology and
pathology. early microbial colonization of the gut reduced the incidence of
infectious, inflammatory and autoimmune diseases reduced the incidence of
infectious, inflammatory and autoimmune diseases. simcsum :  die
fruehzeitshygiene ist ein begriff aus der entwicklungsbiologie und bezeichnet
das phaenomen, dass die fruehzeitliche besiedlung des darmes durch
krankheitserreger verhindert wird. die fruehzeitshygiene unterscheidet sich
von anderen entwicklungsbiologischen forschungsgebieten wie der
entwicklungsphysiologie, der haematologie und der palaeontologie dadurch, dass
in ihrer gesamtheit zur fruehen entwicklungsphase die mikrobielle vielfalt des
darmes zaehlt. Translation: early hygiene is a term from developmental biology
and describes the phenomenon that prevents early colonization of the
intestines by pathogens. early hygiene differs from other developmental
biological research areas such as developmental physiology, haematology and
palaeontology in that the microbial diversity of the intestine counts in its
entirety for the early development phase.

Table D.1: An example of spektrum output, where mbart produces non-German
words (marked as red) and simcsum generates the summary with wrong attention
on hygiene. The summaries are translated via Google translate.

Target: nicht nur stress und genetische faktoren tragen zu essstörungen bei,
sondern auch vorübergehende hungerphasen. nun stießen forscher vom
europäischen zentrum für hirnforschung in rom auf einen möglichen grund dafür
- und zwar in den gehirnen von labormäuse, die sich auch durch schmerzen nicht
vom naschen abbringen ließen. die psychologin rossella ventura und ihre
kollegen trainierten mäuse darauf, in einer von zwei kammern nach einer
belohnung in form von schokolade zu suchen. dann versetzten sie den tieren
ausgerechnet in dem raum, wo die leckerei lockte, schmerzhafte elektroschocks.
die mäuse mieden fortan die schokolade und hielten sich vorwiegend in der
anderen kammer auf - außer jene, die zuvor einige zeit lang kaum futter
bekommen hatten. […] Translation: Not only stress and genetic factors
contribute to eating disorders, but also periods of starvation. now
researchers from the european center for brain research in rome have
discovered a possible reason for this - namely in the brains of laboratory
mice, which were not deterred from snacking even by pain. psychologist
rossella ventura and her colleagues trained mice to look for a chocolate
reward in one of two chambers. then they gave the animals painful electric
shocks in the room where the treat lured them. From then on, the mice avoided
the chocolate and mainly stayed in the other chamber - except for those that
had hardly received any food for a while. […] mbart: eating disorders (engl.
“ess-stoerungen”) nennt man eine vielzahl von ursachen und mechanismen, die
zur auspraegung von essstoerungen fuehren koennen. Translation: eating
disorders are a number of causes and mechanisms that can lead to the
development of eating disorders. simcsum :  als essverhalten oder magersucht
werden die verhaltensweisen von menschen und tieren bezeichnet, die excessive
(vergoren) oder chronisch (z.b. nach unfaellen oder erkrankungen ) zu es
verlangen oder dieses verhalten unterdruecken. essverhalten kann zu
verhaltensstoerungen fuehren, die ursache fuer die entstehung von
essstoerungen sind innere oder aeussere ursachen des essverhaltens.
Translation: eating habits or anorexia are the behaviors of people and animals
that demand excessive (fermented) or chronic (e.g. after accidents or
illnesses) or suppress this behavior. eating behavior can lead to behavioral
disorders, the cause of eating disorders are internal or external causes of
eating behavior.

Table D.2: An example of spektrum output, where mbart produces non-German
words (marked as red) and quite short summary. While simcsum generates also
generates a non-German word. The summaries are translated via Google
translate.

Target: der tod kam auf dem höhepunkt der macht : cangrandei. della scala, der
berühmteste spross der scaliger aus verona, hatte in zahlreichen kriegerischen
auseinandersetzungen die herrschaft über ganz venetien an sich gerissen. doch
kurz nach seinem triumphalen einzug in die norditalienische stadt treviso am
18. juli 1329 begannen mit durchfall und erbrechen die ersten symptome einer
tödlichen erkrankung. […] Translation: death came at the height of power :
cangrandei. della scala, the most famous offspring of the scaliger family from
verona, seized control of all of veneto in numerous military conflicts. but
shortly after his triumphant entry into the northern italian city of treviso
on july 18, 1329, the first symptoms of a fatal illness began with diarrhea
and vomiting. […] mbart: alberto cangrande ( * 1291 oder 1329 in verona ; †
29. oktober 1329 ebenda ) war ein italienischer dichter, schriftsteller und
politiker der flämischen republik. in der zeit der italienischen republik
bestieg er von seinen brudern die dynastie der scaligeri. bei der eroberung
der elfenbeinküste 1311 wurde er zum maßgebenden gründer der faction in rom.
ab 1314 war er herrscher im königreich venedig. cangrande gilt als vater der
italienischen Translation: alberto cangrande ( b. 1291 or 1329 in verona ; †
october 29, 1329 ibid ) was an italian poet, writer and politician of the
flemish republic. in the time of the italian republic he ascended from his
brothers the dynasty of the scaligeri. when he conquered the ivory coast in
1311, he became the main founder of the faction in rome. from 1314 he was
ruler of the kingdom of venice. cangrande is considered the father of the
italian simcsum :  cangrande della scala oder cangrande cangrande ( * 9. märz
1291 in verona ; † 22. juli 1329 ebenda ) war ein begründer der von den
scaligeri beeinflussten dynastie, der unter den herrschern von verona von 1308
bis 1387 regierte. der sohn des veronaer stadtbegründers und stadtmeisters
alberto della scala ( genannt „ albert “ oder „ adelsmann “ ), der nur als
unterstellt war, wurde nach seinem tod 1301 zum bruder von alberto verbannte.
Translation: cangrande della scala or cangrande cangrande ( born march 9, 1291
in verona ; † july 22, 1329 ibid ) was a founder of the dynasty influenced by
the scaligeri, who reigned under the rulers of verona from 1308 to 1387. the
son of verona’s founder and mayor alberto della scala ( called " albert " or "
nobleman " ), who was only subordinate, was banished to alberto’s brother
after his death in 1301.

Table D.3: An example of spektrum output, where mbart generates a wrong named
entity. spektrum, on the other hand, gets it right but generates a wrong alias
for this person. The summaries are translated via Google translate.

Target: inoffizieller weltrekordhalter im nicht - frieren trotz eisiger kälte
ist der niederländer wimhof : er lief bereits ziemlich nackt halbmarathons
nördlich des polarkreises und nahm das längste eisbad seit beginn der
aufzeichnungen - ohne sich ernsthaft zu beklagen. wissenschaftliche
untersuchungen des eismannes hatten bisher keine körperlichen auffälligkeiten
gefunden, die erklären könnten, warum hof mit kälte offensichtlich besser
umgeht als der durchschnittsmensch. […] Translation: the unofficial world
record holder for not freezing despite the icy cold is the dutchman wimhof :
he ran half marathons north of the arctic circle quite naked and took the
longest ice bath since records began - without seriously complaining.
scientific examinations of the ice cream man had not found any physical
abnormalities that could explain why hof obviously deals with the cold better
than the average person. […] mbart: wim hof ( * 20. april 1959 ) ist ein
niederländischer sportler, der sich unter anderem mit eisschwimmern
beschäftigt. Translation: wim hof ( born april 20, 1959 ) is a dutch athlete
who is involved with ice swimmers, among other things. simcsum :  wim hof ( *
20. april 1959 ), auch als eismann bekannt, ist ein niederländischer swimmer
und ein ehemaliger olympiateilnehmer. er hält seit dem 28\. februar 2015 den
bislang längsten direkten kontakt menschlichen körpers mit eis. Translation:
Wim Hof ( born April 20, 1959 ), also known as Eismann, is a Dutch swimmer and
a former Olympic competitor. since february 28, 2015, he has been in the
longest direct contact between the human body and ice.

Table D.4: An example of spektrum output, where both mbart and simcsum produce
unfaithful information. Marked in orange is unfaithful information to the
original text. The summaries are translated via Google translate.

##  Appendix E simcsum Examples and Results

Here we present some examples showing the difference between simcsum and
mbart.

Target: für menschen ist der einbeinige stand immer eine wackelpartie , vor
allem bei geschlossenen augen . um nicht umzukippen , müssen die muskeln
permanent die leichten schwankungen ausgleichen . nicht so bei flamingos : sie
kostet es weniger kraft , auf einem bein zu stehen als auf zweien . deswegen
können sie auch beruhigt auf einem bein ein schläfchen machen , ohne dabei
umzukippen . […] Translation: standing on one leg is always a shaky game for
humans, especially with closed eyes. in order not to tip over , the muscles
have to constantly compensate for the slight fluctuations . Not so with
flamingos: it takes less strength for them to stand on one leg than on two.
that’s why you can take a nap on one leg without tipping over. […] mbart: die
biogerontologie ( von griech. bíos „ leben “ und lat. protes „ befestigt “ )
beschäftigt sich mit dem verhältnis von körpergewicht und körperhaltung. so
untersucht die biogerontologie das verhältnis von körpergewicht und
körperhaltung : welche gewichtszustände und welche muskeln notwendig sind, um
einen pilz, der einer krankheit ausgesetzt ist, zu tragen? wie sehr wichtig es
ist, den jeweiligen körpergewichtszustand zu messen. dies wird in der
Translation: biogerontology (from greek bíos “life” and lat. protes “fixed”)
deals with the relationship between body weight and posture. this is how
biogerontology examines the relationship between body weight and posture:
which weight conditions and which muscles are necessary to carry a fungus that
is exposed to a disease? how very important it is to measure the respective
body weight condition. this will be in the simcsum :  flamingos (
phoenicopteridae ) oft sitzen auf einem bein, um eine muskelkontraktion zu
erzeugen. sie haben die fähigkeit, das körpergewicht aufzunehmen und zu
tragen, ohne dabei eine muskelaktive wirkung auszuüben. Translation: Flamingos
(phoenicopteridae) often perch on one leg to produce muscle contraction. they
have the ability to absorb and carry body weight without exerting a muscle-
active effect.

Table E.1: An example of spektrum output, where simcsum generates a better
summary than mbart . In this case, the mbart summary misses the article’s
point by focusing on biogerontology which has no relevancy in this case, and
simcsum generates a summary that better captures the idea and main focus of
the original text. Marked in orange is unfaithful information to the original
text. The summaries are translated via Google translate.

Target: eine besondere strategie, seinen fressfeinden zu entkommen, hat der
bombardierkäfer pheropsophus jessoensis entwickelt. er wehrt sich sogar noch,
nachdem er eigentlich schon verloren hat, und das sogar recht erfolgreich, wie
jetzt shinji sugiura und takuya sato von der universität kobe in den biology
letters berichten. gerade verschluckt, schießt der käfer im magen der kröte
einen strahl heißes, ätzendes sekret aus der afterdrüse - eine strategie, die
den bombardierkäfern ihren namen eingebracht hat. normalerweise allerdings
dient die waffe zur abschreckung. in diesem fall jedoch ätzen sich die käfer
mit ihrer hilfe quasi zurück in die freiheit : das sekret lässt die kröten
erbrechen ; und der eigentlich schon verspeiste käfer entkommt. […]
Translation: the bombardier beetle pheropsophus jessoensis has developed a
special strategy for escaping its predators. he even defends himself after he
has actually already lost, and even quite successfully, as shinji sugiura and
takuya sato from the university of kobe now report in the biology letters.
just swallowed, the beetle in the toad’s stomach shoots a jet of hot, caustic
secretion out of the anal gland - a strategy that has given the bombardier
beetle its name. normally, however, the weapon serves as a deterrent. in this
case, however, the beetles etch their way back to freedom with your help: the
toads vomit the secretion; and the actually already eaten beetle escapes. […]
mbart: unter zoologischer kompatibilitaet versteht man in der biologie die
faehigkeit von beutetieremischlingen, nach einem laubabschweif von artgenossen
in den verdauungstrakt von artgenossen zu entweichen. die oekologische
kompatibilitaet ist ein wesentlicher bestandteil der erfolgreichen entstehung
von beutegreifern.beim laubabschweifexperiment mit hilfe des
insektenpheropsophus jessoensis aus zwei kaeferarten, dem bufo japonicus und
dem b. torrenticola, konnte nachgewiesen werden, dass der kaefer pheropsophus
jessoensis unter laboratorischen bedingungen eine in den magensaftgastropfen
der jagdsekten einschleudern kann. Translation: in biology, zoological
compatibility means the ability of hybrid prey animals to escape into the
digestive tract of conspecifics after the leaves have drifted away from
conspecifics. ecological compatibility is an essential part of the successful
emergence of predators.in the leaf-dropping experiment with the help of the
insect pheropsophus jessoensis from two beetle species, the bufo japonicus and
the b. torrenticola, it could be proven that the beetle pheropsophus
jessoensis can inject one into the gastric juice droplets of the hunting sects
under laboratory conditions. simcsum :  als wehrverhalten
oderverteidigungsverhalten bezeichnet man in der verhaltensbiologie
verschiedene verhaltensweisen von tieren, die wehren gegen die fressfeinde
oder andere fresstiere bilden. nicht zu verwechseln ist das wehrverhalten mit
der phylogenetik, die sich aus der evolutionsgeschichte entwickelt hat.
Translation: in behavioral biology, defensive behavior is the term used to
describe various behaviors of animals that defend themselves against predators
or other predators. the defensive behavior should not be confused with the
phylogenetics that have developed from the history of evolution.

Table E.2: An example of spektrum output, both simcsum and mbart produce
reasonable summaries. The simcsum summary is preferred because it explains the
basic idea of the beetle defending itself against predators. The mbart summary
is filled with complex words that do not make sense or are unnecessary for
this context. These words are marked in orange. The summaries are translated
via Google translate.

Target: manche farben bleiben besser im gedächtnis haften als andere, doch auf
die erinnerung an den entsprechend gefärbten gegenstand hat das keinen
einfluss. das zeigt nun eine studie von psychologen um christof kuhbandner von
der universität regensburg. die ergebnisse machen deutlich, dass der
gegenstand selbst und dessen eigenschaften vom gehirn nicht immer als einheit
betrachtet werden. die wissenschaftler testeten für ihre studie das gedächtnis
ihrer versuchspersonen. dabei variierten sie sowohl, was zu merken war, als
auch, wie es dargeboten wurde. insbesondere präsentierten sie wörter und
gegenstände in unterschiedlichen farben. […] Translation: some colors stick in
the memory better than others, but this has no influence on the memory of the
correspondingly colored object. this is now shown by a study by psychologists
led by christof kuhbandner from the university of regensburg. the results make
it clear that the object itself and its properties are not always viewed as a
unit by the brain. for their study, the scientists tested the memory of their
subjects. they varied both what was to be remembered and how it was presented.
in particular, they presented words and objects in different colors. […]
mbart: die farbpsychologie ist ein teilgebiet der psychologie und beschäftigt
sich mit der wirkung von farben auf die wahrnehmung und das gedächtnis.
Translation: color psychology is a branch of psychology and deals with the
effect of colors on perception and memory. simcsum :  die farbe ist ein
farbmuster, das uns wahrnehmungen und die sinneswahrnehmung durch das sehen
vermittelt. die farbwahrnehmung gehört zu den kognitionswissenschaften. wir
sehen sie nicht nur als eindimensionales spektrum möglicher farben, sondern
auch als eindimensionales, oftmals auch als „ visuelles system “, das die
sinneswahrnehmung mitanpasst.  […] Translation: color is a color pattern that
conveys perceptions and sensory perception to us through seeing. Color
perception is one of the cognitive sciences. we see them not only as a one-
dimensional spectrum of possible colors, but also as a one-dimensional, often
also as a " visual system " that also adapts the sensory perception. […]

Table E.3: An example of spektrum output, where mbart performs better than
simcsum . mbart generates a summary that is too short but which better
recapitulates the main idea. The orange marked words in the simcsum summary
are incoherent and are not faithful to the original text. The summaries are
translated via Google translate.

models r1 r2 rl bs fre F P R F P R F P R F P R wikipedia mt5 26.79 38.64 20.50
12.65 17.23 9.99 23.4 33.65 17.94 69.12 72.32 66.20 45.42 mbart 31.43 36.40
27.66 13.20 15.22 11.65 25.12 28.08 22.73 70.52 71.22 69.83 44.67 pegasus
29.30 43.01 22.22 13.93 19.81 10.74 24.62 35.82 18.76 69.83 73.32 66.66 43.39
xlsum 31.91 36.93 28.09 13.30 16.83 11.00 24.14 29.58 20.39 70.04 71.45 68.68
37.83 bigbird 29.23 40.92 22.74 13.72 19.13 10.69 24.6 34.28 19.18 69.19 72.26
66.37 41.42 long-ed 15.11 46.43 9.02 6.82 19.57 4.13 13.67 41.33 8.19 63.94
70.87 58.25 24.48 kis-mbart 32.02 33.66 30.54 12.39 13.22 11.65 24.72 27.08
22.73 70.52 71.22 69.83 45.76 simcsum 34.50† 36.17 32.97 14.36† 15.13 13.66
25.85† 27.07 24.73 71.60† 72.35 70.87 46.86† spektrum mt5 9.21 22.15 5.81 0.75
1.89 0.47 6.5 15.84 4.09 58.52 61.84 55.53 39.18 mbart 16.16 28.23 11.32 1.47
2.66 1.02 13.89 20.06 10.62 62.11 64.66 59.75 39.17 pegasus 11.49 25.77 7.39
0.95 2.14 0.61 8.01 18.27 5.13 60.56 63.7 57.72 37.93 xlsum 17.10 26.98 12.52
1.63 2.68 1.17 9.79 16.92 6.89 62.25 64.52 60.13 33.83 bigbird 12.28 22.49
8.45 1.04 2.07 0.69 8.65 15.98 5.93 59.97 62.46 57.68 36.24 long-ed 1.32 1.18
1.49 0.11 0.09 0.13 1.18 1.07 1.32 51.85 50.49 53.29 30.16 kis-mbart 17.33
38.55 11.18 1.61 4.29 0.99 12.97 22.22 9.16 61.83 65.41 58.63 39.21 simcsum
18.88† 26.03 14.81 1.82† 2.56 1.41 14.16† 16.91 12.18 63.47† 65.22 61.81
40.03†

Table E.4: The wikipedia and spektrum results for all baselines and simcsum.
gold denotes the reference summaries. Underline refers to the best baseline
results, and bold$\dagger$ denotes the best overall results with significant
improvements ($p\\!<\\!.001$).