# Towards Human-Free Automatic Quality Evaluation of German Summarization

Neslihan Iskender
Technische Universität Berlin
Quality and Usability Lab
neslihan.iskender
@tu-berlin.de
Oleg Vasilyev
Primer Technologies Inc.
San Francisco, California
<EMAIL_ADDRESS>
Tim Polzehl
Technische Universität Berlin
Quality and Usability Lab
tim.polzehl1
@tu-berlin.de
John Bohannon
Primer Technologies Inc.
San Francisco, California
<EMAIL_ADDRESS>
Sebastian Möller
Technische Universität Berlin
Quality and Usability Lab
<EMAIL_ADDRESS>
###### Abstract

Evaluating large summarization corpora using humans has proven to be expensive
from both the organizational and the financial perspective. Therefore, many
automatic evaluation metrics have been developed to measure the summarization
quality in a fast and reproducible way. However, most of the metrics still
rely on humans and need gold standard summaries generated by linguistic
experts. Since BLANC does not require golden summaries and supposedly can use
any underlying language model, we consider its application to the evaluation
of summarization in German. This work demonstrates how to adjust the BLANC
metric to a language other than English. We compare BLANC scores with the
crowd and expert ratings, as well as with commonly used automatic metrics on a
German summarization data set. Our results show that BLANC in German is
especially good in evaluating informativeness.

##  1 Introduction

In text summarization, the human evaluation conducted by linguistic experts is
the gold standard, but it is expensive and slow Bhandari et al. (2020). The
wide range of quality evaluation criteria and non-standard annotation set-ups
used by researchers lead to conflicting and mostly non-reproducible results
Howcroft et al. (2020). Therefore, researchers have focused on developing
automatic summarization evaluation metrics relying on gold standard summaries
written by humans, such as BLEU Papineni et al. (2002), ROUGE Lin (2004) and
BERTScore Zhang et al. (2020), or human-free quality metrics such as Jensen-
Shannon (JS) Louis and Nenkova (2009), SUM-QE Xenouleas et al. (2019) and
BLANC Vasilyev et al. (2020a).

Currently, ROUGE is still the most popular automatic summarization quality
evaluation metric in ACL conferences Gao et al. (2019). Its requirement of
gold standard summaries is limiting researchers to use only the available data
sets with gold standard summaries such as the TAC and CNN/Daily Mail data sets
derived from high-quality English texts in the news domain Dang and Owczarzak
(2009); Nallapati et al. (2016). To overcome this limitation, we focus in this
paper on the human-free summarization quality evaluation metric BLANC, which
so far has been applied only to a limited number of English data sets. We
demonstrate how to adjust BLANC to a language other than English on the
example of the German language, and we compare its performance to the commonly
used automatic metrics and human annotations.

##  2 Related Work

The summarization quality evaluation is crucial for the development of
automatic summarization systems to determine whether the system has properly
outperformed the existing tools in terms of quality and speed or whether the
designed properties work as intended van der Lee et al. (2018). Traditional
summarization evaluation methods can be categorized into intrinsic evaluation
and extrinsic evaluation Belz and Reiter (2006).

In intrinsic evaluation, the summarization quality evaluation is directly
based on the summary itself without considering the source text, typically
carried out as pair comparison (compared to expert summaries) or using
absolute scales without having a reference Jones and Galliers (1996). However,
the extrinsic evaluation is based on measuring the summary’s impact on the
completion of some task using the source document Mani (2001). Both evaluation
methods can be performed by humans, usually linguistic experts, as well as by
automatic metrics.

Despite the high cost, subjectivity and lack of standards, the human
evaluation is still considered to be the gold standard in summarization
evaluation Howcroft et al. (2020); Bhandari et al. (2020). The lack of
standards results in usage of different quality criteria, experimental designs
and reporting schema across the research papers published in ACL and INLG
conferences van der Lee et al. (2019). Therefore, automatic evaluation metrics
are being introduced, such as BLEU Papineni et al. (2002), METEOR Denkowski
and Lavie (2014), ROUGE Lin (2004), ROUGE-WE Ng and Abrecht (2015), SUM-QE
Xenouleas et al. (2019), APES Eyal et al. (2019), Summa-QA Scialom et al.
(2019) and FEQA Durmus et al. (2020), Peyrard et al. (2017), BERTScore Zhang
et al. (2020), MoverScore Zhao et al. (2019) and SUPERT Gao et al. (2020) and
BLANC Vasilyev et al. (2020a), while ROUGE is still being by far the most
popular metric Gao et al. (2019).

The automated metrics have been criticized for inconsistent correlations with
human evaluations, and for the limited research on few existing English-
language data sets Reiter and Belz (2009); Graham (2015); Novikova et al.
(2017); Peyrard and Eckle-Kohler (2017); Gabriel et al. (2020). In this paper,
we explore moving BLANC measure from English to German, as a measure that
requires replacement of the underlying model. We compare it with human
annotations and with several lexical overlap based automated metrics, which
are easy to move to another language.

##  3 Methodology

###  3.1 Human Evaluation

For our data analysis, we used a German summarization data set with 50
summaries, crowd and expert evaluations for these 50 summaries.

The corpus contains queries with an average word count of 7.78, the shortest
one with four words, and the longest with 17 words; posts from a customer
forum of Deutsche Telekom with an average word count of 555, the shortest one
with 155 words, and the longest with 1005 words; and corresponding query-based
extractive summaries with an average word count of 63.32, the shortest one
with 24 words, and the longest one with 147 words.

The human annotations of the data set contain annotations both from two
linguistic experts and 24 different crowd workers per summary for the
following factors on a 5-point Mean Opinion Score (MOS) scale: overall
quality, intrinsic quality factors (grammaticality, non-redundancy,
referential clarity, focus, structure & coherence), and extrinsic quality
factors (summary usefulness, post usefulness, and summary informativeness).
Human annotators evaluated the first six quality factors intrinsically,
meaning that they did not read the source document and only see the summary
itself. During the evaluation of extrinsic quality factors, human annotators
read the query, the source, and the summary. In this paper, we only focus on
the extrinsic quality factors since they reflect the system’s performance on
the task, so they represent the content quality of a summary and are more
suitable for comparing human evaluation with content-based automatic quality
metrics.

In this data set, the first extrinsic metric summary usefulness, also called
content responsiveness, is defined as the version introduced in DUC 2003 NIST
(2003) which investigates the summary’s usefulness concerning how useful the
extracted summary is to satisfy the given goal Shapira et al. (2019).
Following, the second extrinsic metric post usefulness, also called relevance
assessment, examines if the source document contains relevant information
about the given goal Mani (2001); Conroy and Dang (2008). Lastly, the third
extrinsic metric summary informativeness answers the question of how much
information from the source document is preserved in the extracted summary
Mani (2001).

###  3.2 Automatic Metrics

We calculated the BLEU and ROUGE scores using the sumeval
library111https://github.com/chakki-works/sumeval for German and BERTScore
using the bert-score library222https://github.com/Tiiiger/bert_score using
bert-base-german-cased as the language model Zhang et al. (2020). All of these
three metrics are reference-based, so two linguistic experts created the gold
standard summaries, which have an average word count of 58.18, the shortest
one with 14 words, and the longest with 112 words. Additionally, we calculated
the reference-free Jensen-Shannon (JS) similarity as described in Louis and
Nenkova (2009).

###  3.3 BLANC for German

Moving BLANC from one language to another is simple: all that has to be done
is a replacement of the underlying language model, trained for generic Cloze
task. We explore using three German-language trained models: bert-base-german-
cased333https://huggingface.co/bert-base-german-cased, bert-base-german-dbmdz-
cased and bert-base-german-dbmdz-uncased444https://github.com/dbmdz/berts. The
algorithm555https://github.com/PrimerAI/blanc does not need to be changed.
BLANC does depend weakly Vasilyev et al. (2020a) on several parameters
defining the frequency of masking for the Cloze task.

The BLANC family is parametrized by a step between masked tokens, $gap$, and
by a minimal length $L$ at which a token is allowed to be masked. For example,
$gap=3$ would mean that each third token would be masked. The minimal length
may differ for three different kinds of tokens, accordingly to the
tokenization by the underlying language model: it is $L_{normal}$ for a whole-
word token; it is $L_{lead}$ for the first token when a word is split into two
or more tokens; and it is $L_{follow}$ for the follow-up tokens of the split
word.

If correlations with human scores can be trusted, one can specify the
parameters that provide higher correlations. The value $gap=6$ is special
because it approximately corresponds to the frequency of masking in standard
BERT training. The value $gap=2$ was found to give the best correlations with
human scores in English Vasilyev et al. (2020b). We certainly can take the
same parameters as a good configuration and be done. However, German words are
on average longer than English words, and we can anticipate a possibility of
slightly more beneficial thresholds on the length of the tokens to be masked.
In order to have firm conclusions, we have explored around 100, and narrowed
down to 72 promising configurations defined by the choice from the 3 models x
24 choices of parameters: $gap=2,6$; $L_{normal}=4,5,6$; $L_{lead}=1,2$;
$L_{follow}=1,100$. The latter $L_{follow}$ choices mean either allowing
masking all tokens or not masking any.

##  4 Results

Results are presented for the MOS values of the three extrinsic quality scores
(summary usefulness (SU), post usefulness (PU), and summary informativeness
(SI)) assessed by crowd workers and experts, as well as for automatic metrics
BLEU, ROUGE-1, ROUGE-2, ROUGE-L, BERTScore (we use F-scores), Jensen-Shannon
similarity, and different configurations of BLANC score.

###  4.1 Comparing Human Evaluation with BLANC

To test the normality of the human annotations and automatic metrics, we
carried out Anderson Darling tests showing that the summary usefulness
measured by the crowd, summary informativeness measured by experts and BLANC
were not normally distributed (p $<$ 0.05). Therefore, we apply Spearman rank-
order correlation coefficients to compare human ratings with automatic
metrics.

For our analysis, we correlated the summary usefulness, post usefulness and
summary informativeness measured by the crowd and the experts with different
BLANC configurations. For the correlations between crowd and BLANC
configurations, we observed that BLANC configuration with language model $=$
bert-base-german-dbmdz-cased, $gap=2$, $L_{normal}=4$, $L_{lead}=2$,
$L_{follow}=1$ has the highest significant correlation with summary usefulness
($\rho$ $=$ 0.350, p $<$ 0.05), post usefulness ($\rho$ $=$ 0.378, p $<$
0.05), and summary informativeness ($\rho$ $=$ 0.413, p $<$ 0.05).

When we look at the correlation betwen expert and BLANC configurations, we see
that the same BLANC configuration has the highest significant correlation with
summary usefulness ($\rho$ $=$ 0.349, p $<$ 0.05) and the second highest
correlation with summary informativeness ($\rho$ $=$ 0.460, p $<$ 0.05).
Blocking masking of follow-up tokens $L_{follow}=100$ increases the
correlation with summary informativeness ($\rho$ $=$ 0.485, p $<$ 0.05). There
is no significant correlation between post usefulness measured by the experts
and BLANC configurations. These results show that the single BLANC
configuration is robust across different correlations with human annotations
in German.

We also found that the top 5 configurations best correlated with summary
informativeness with experts and the top 5 with the crowd are all employing
the model bert-base-german-dbmdz-cased with $gap=2$, and differ only by the
$L$ thresholds (which have low influence). We recommend to use the language
model bert-base-german-dbmdz-cased, $gap=2$, $L_{normal}=4$, $L_{lead}=2$,
$L_{follow}=1$ when calculating BLANC for German language. Also, these results
are perfectly in line with the findings of Vasilyev et al. (2020b) for the
correlations of BLANC ($gap=2$) with summary informativeness on the English
data set. BLANC correlates significantly with summary informativeness for both
languages indicating that BLANC reflect the summary informativeness in a
language agnostic way.

Further, to analyze the effect of the source document’s length, summary
length, and compression level (summary length divided by source length), we
divide our data into two groups for each criterion by its mean. We only
observe significant correlations by grouping mean summary length (M $=$ 63.32)
for the correlation of summary informativeness measured by the expert with
BLANC. In the short summary group (N $=$ 30), the correlation ($\rho$ $=$
0.476, p $<$ 0.05) was lower than in the long summary group ($\rho$ $=$ 0.516,
p $<$ 0.05) showing that BLANC performs better when the summary is longer.

###  4.2 Comparing BLANC with Other Automatic Metrics

For comparing BLANC with commonly used automatic metrics, we calculated the
Spearman rank-order correlation coefficients of the summary informativeness
measured by the expert with BLEU, ROUGE-1, ROUGE-2, ROUGE-L, BERTScore-F, JS
similarity and top three configurations of BLANC selected based on the
correlation with summary informativeness. As noted in the previous section,
all BLANC top models turned out to use bert-base-german-dbmdz-cased with
$gap=2$, hence our notations for BLANC $B\\_$ in the figures include only
$L$-thresholds, for example the name ’$B\\_L4\\_Ll2\\_Lf1$’ means
$L_{normal}=4$, $L_{lead}=2$ and $L_{follow}=1$.

* not significant correlation

Figure 1: Spearman Rank-order Correlation between Summary Informativeness
measured by Experts and Automatic Metrics

Figure 1 shows the correlations of summary informativeness measured by the
expert with automatic metrics. We observe that ROUGE-2, BLEU, and BERTScore do
not correlate significantly with summary informativeness and the BLANC
configuration with language model $=$ bert-base-german-dbmdz-cased, $gap=2$,
$L_{normal}=4$, $L_{lead}=2$, $L_{follow}=100$ outperforms all the automatic
metrics. However, Figure 2 shows that Rouge-1 correlates better than BLANC
with summary informativeness measured by the crowd.

* not significant correlation

Figure 2: Spearman Rank-order Correlation between Summary Informativeness
measured by Crowd and Automatic Metrics

Why the BLANC versions are much better with the experts, while ROUGE-1 is
better with the crowd workers? We speculate that ROUGE-1 is a lexical overlap
measure, and as such it picks up more obvious commonality between a summary
and a reference summary. BLANC is more ’semantic’ and is closer to the
experts. The same probably explains that BERTScore is on the same level as
BLANC with the crowd sources, but is much worse with the experts.

##  5 Conclusion

We illustrated how to effortlessly adapt the BLANC metric to a language other
than English. We compared the BLANC scores to the human ratings and found the
configuration giving the best correlations on the considered German dataset.
We have shown that BLANC can perform on German as good as the most used
automatic metric ROUGE when evaluating informativeness, and it performs
comparable to the experts. However, our study has limitations because we
conducted our analysis using only one monolingual human-annotated dataset. As
part of future work, we will consider applying BLANC variations to more
datasets, with a broader range of human annotations, and comparing them to
more automated evaluation measures.

## References

  * Belz and Reiter (2006) Anja Belz and Ehud Reiter. 2006.  Comparing automatic and human evaluation of NLG systems.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , Trento, Italy. Association for Computational Linguistics.
  * Bhandari et al. (2020) Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020.  Re-evaluating evaluation in text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9347–9359, Online. Association for Computational Linguistics.
  * Conroy and Dang (2008) John M. Conroy and Hoa Trang Dang. 2008.  Mind the gap: Dangers of divorcing evaluations of summary content from linguistic quality.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 145–152, Manchester, UK. Coling 2008 Organizing Committee.
  * Dang and Owczarzak (2009) Hoa Trang Dang and Karolina Owczarzak. 2009.  Overview of the tac 2009 summarization track.  In _Proceedings of the Text Analysis Conference_.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 376–380, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070. Association for Computational Linguistics.
  * Eyal et al. (2019) Matan Eyal, Tal Baumel, and Michael Elhadad. 2019.  Question answering as an automatic evaluation metric for news article summarization. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1_ , pages 3938–3948. Association for Computational Linguistics.
  * Gabriel et al. (2020) Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2020\.  Go figure! a meta evaluation of factuality in summarization.
  * Gao et al. (2020) Yang Gao, Wei Zhao, and Steffen Eger. 2020.  Supert: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1347–1354. Association for Computational Linguistics.
  * Gao et al. (2019) Yanjun Gao, Chen Sun, and Rebecca J. Passonneau. 2019.  Automated pyramid summarization evaluation.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 404–418, Hong Kong, China. Association for Computational Linguistics.
  * Graham (2015) Yvette Graham. 2015.  Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 128–137, Lisbon, Portugal. Association for Computational Linguistics.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Jones and Galliers (1996) Karen Sparck Jones and Julia R. Galliers. 1996.  _Evaluating Natural Language Processing Systems: An Analysis and Review_.  Springer-Verlag, Berlin, Heidelberg.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Louis and Nenkova (2009) Annie Louis and Ani Nenkova. 2009.  Automatically evaluating content selection in summarization without human models.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , pages 306–314, Singapore. Association for Computational Linguistics.
  * Mani (2001) Inderjeet Mani. 2001.  Summarization evaluation: An overview.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar GuÌ‡lçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Ng and Abrecht (2015) Jun-Ping Ng and Viktoria Abrecht. 2015.  Better summarization evaluation with word embeddings for ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.
  * NIST (2003) NIST. 2003.  Duc 2003: Documents, tasks, and measures.  https://duc.nist.gov/duc2003/tasks.html.  Accessed: 2019-11-20.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017.  Why we need new evaluation metrics for NLG.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Peyrard et al. (2017) Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017.  Learning to score system summaries for better content selection evaluation. In _Proceedings of the Workshop on New Frontiers in Summarization_ , pages 74–84, Copenhagen, Denmark. Association for Computational Linguistics.
  * Peyrard and Eckle-Kohler (2017) Maxime Peyrard and Judith Eckle-Kohler. 2017.  Supervised learning of automatic pyramid for optimization-based multi-document summarization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1084–1094, Vancouver, Canada. Association for Computational Linguistics.
  * Reiter and Belz (2009) Ehud Reiter and Anja Belz. 2009.  An investigation into the validity of some metrics for automatically evaluating natural language generation systems.  _Computational Linguistics_ , 35(4):529–558.
  * Scialom et al. (2019) Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019\.  Answers unite! unsupervised metrics for reinforced summarization models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ , pages 3246–3256. Association for Computational Linguistics.
  * Shapira et al. (2019) Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019.  Crowdsourcing lightweight pyramids for manual summary evaluation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 682–687, Minneapolis, Minnesota. Association for Computational Linguistics.
  * van der Lee et al. (2019) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019.  Best practices for the human evaluation of automatically generated text.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 355–368, Tokyo, Japan. Association for Computational Linguistics.
  * van der Lee et al. (2018) Chris van der Lee, Bart Verduijn, Emiel Krahmer, and Sander Wubben. 2018.  Evaluating the text quality, human likeness and tailoring component of PASS: A Dutch data-to-text system for soccer.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 962–972, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Vasilyev et al. (2020a) Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020a.  Fill in the BLANC: Human-free quality estimation of document summaries. In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pages 11–20. Association for Computational Linguistics.
  * Vasilyev et al. (2020b) Oleg Vasilyev, Vedant Dharnidharka, Nicholas Egan, Charlene Chambliss, and John Bohannon. 2020b.  Sensitivity of blanc to human-scored qualities of text summaries. _arXiv_ , arXiv:2010.06716.
  * Xenouleas et al. (2019) Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019.  SUM-QE: a BERT-based summary quality estimation model.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6005–6011, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.