# The unified method for the three-wave equation on the half-line

Jian Xu  School of Mathematical Sciences
Fudan University
Shanghai 200433
People’s Republic of China<EMAIL_ADDRESS>and  Engui Fan  School of
Mathematical Sciences, Institute of Mathematics and Key Laboratory of
Mathematics for Nonlinear Science
Fudan University
Shanghai 200433
People’s Republic of China  correspondence<EMAIL_ADDRESS>
###### Abstract.

We present a Riemann-Hilbert problem formalism for the initial-boundary value
problem of the three-wave equation:

$p_{ij,t}-\frac{b_{i}-b_{j}}{a_{i}-a_{j}}p_{ij,x}+\sum_{k}(\frac{b_{k}-b_{j}}{a_{k}-a_{j}}-\frac{b_{i}-b_{k}}{a_{i}-a_{k}})p_{ik}p_{kj}=0,\quad
i,j,k=1,2,3,$

on the half-line.

###### Key words and phrases:

Riemann-Hilbert problem, Three-wave equation, Initial-boundary value problem

##  1\. Introduction

The 3-wave resonant interaction model described by the equations

$\begin{array}[]{ll}p_{ij,t}-\frac{b_{i}-b_{j}}{a_{i}-a_{j}}p_{ij,x}+\sum_{k}(\frac{b_{k}-b_{j}}{a_{k}-a_{j}}-\frac{b_{i}-b_{k}}{a_{i}-a_{k}})p_{ik}p_{kj}=0,\\\\[4.0pt]
i,j,k=1,2,3;\ a_{i}\neq a_{j},b_{i}\neq b_{j},\ {\rm for}\ i\neq
j,\end{array}$ (1.1)

is one of the important nonlinear models with numerous applications to physics
[14]. The $3$\- and $N$-wave interaction models describe a special class of
wave-wave interactions that are not sensitive on the physical nature of the
waves and bear an universal character. This explains why they find numerous
applications in physics and attract the attention of the scientific community
over the last few decades [23, 24, 25, 20, 19, 22, 21, 9] and the references
therin.

The 3-wave equations can be solved through the inverse scattering method due
to the fact that equation (1.1) admits a Lax representation [24, 25]. But
until the 1990s, the inverse scattering method was pursued almost entirely for
pure initial value problems. In 1997, Fokas announced a new unified approach
for the analysis of inital-boundary value problems for linear and nonlinear
integrable PDEs [1, 2, 3]. The Fokas method provides a generalization of the
inverse scattering formalism from initial value to IBV problems, and over the
last fifteen years, this method has been used to analyze boundary value
problems for several of the most important integrable equations with $2\times
2$ Lax pairs, such as KdV, Schrödinger, sine-Gordon, and stationary
axisymmetric Einstein equations, see e.g. [4, 6]. Just like the IST on the
line, the unified method yields an expression for the solution of an initial-
boundary value problem with that of a Riemann-Hilbert problem. In particular,
the asymptotic behavior of the solution can be analyzed in an effective way by
employing the Riemann-Hilbert problem and the steepest descent method
introduced by Deift and Zhou [10]. Recently, Lenells develop a methodology for
analyzing initial-boundary value problems for integrable evolution equations
with Lax pairs involving $3\times 3$ matrices [7]. He also used this method to
analyze the Degasperis-Procesi equation in [8].

Pelloni and Pinotsis also studied the boundary value problem of the $N-$wave
equation by using the unified method [11]. Recently, Gerdjikov and Grahovski
considered Cauchy problem of the 3-wave equation with with non-vanishing
initial values [12]. In this paper we analyze the initial-boundary value
problem of the three-wave equation (1.1) on the half-line. Compared with these
two papers, there are two differences in our paper. The first difference is
that we get the residue conditions of matrix function $M$ in the Riemann-
Hilbert problem (see (2.27) in the next section 2). The second difference is
that we the jump matrix $J$ is explicitly constructed ( see the equations
(2.14) and (2.22) in the next section 2). Of course, the initial-boundary
value problem for the $3-$wave equation does not need to analysis the global
relation, because the initial data and the boundary data are all known.

The organization of the paper is as follows. In the following section 2, we
perform the spectral analysis of the associated Lax pair. And we formulate the
main Riemann-Hilbert problem in section 3.

##  2\. Spectral Analysis

Our goal in this section is to define analytic eigenfunctions of the Lax pair
(2.1) which are suitable foe the formulation of a Riemann-Hilbert problem.

###  2.1. Lax pair

We first consider the three-wave equations (1.1), with $(x,t)\in\Omega$, and
$\Omega$ denoting the half-line domain

$\Omega=\\{0<x<+\infty,0<t<T\\}$

and $T>0$ being a fixed final time. We denote the initial and boundary values
by $p_{ij,0}(x)$ and $q_{ij,0}(t)$, respectively

$p_{ij,0}(x)=p_{ij}(x,0),\quad q_{ij,0}(t)=p_{ij}(0,t)$

with $p_{ij,0}(x)$ and $q_{ij,0}(t)$ are rapidly decaying. Equation (1.1)
admits the following Lax representation [14]

$\left\\{\begin{array}[]{l}\phi_{x}=M\phi,\\\
\phi_{t}=N\phi,\end{array}\right.$ (2.1)

where $M=i\lambda A+P$ and $N=i\lambda B+Q$, with

$\begin{array}[]{ll}A=\left(\begin{array}[]{ccc}a_{1}&0&0\\\ 0&a_{2}&0\\\
0&0&a_{3}\end{array}\right)&P=\left(\begin{array}[]{ccc}0&p_{12}&p_{13}\\\
p_{21}&0&p_{23}\\\ p_{31}&p_{32}&0\end{array}\right)\\\
B=\left(\begin{array}[]{ccc}b_{1}&0&0\\\ 0&b_{2}&0\\\
0&0&b_{3}\end{array}\right)&Q=\left(\begin{array}[]{ccc}0&n_{12}p_{12}&n_{13}p_{13}\\\
n_{21}p_{21}&0&n_{23}p_{23}\\\
n_{31}p_{31}&n_{32}p_{32}&0\end{array}\right).\end{array}$ (2.2)

Obviously, $trace(A)=trace(B)=0$. We also assume that $a_{1}>a_{2}>a_{3}$ and
$b_{1}<b_{2}<b_{3}$. By using transformation

$\mu=\phi e^{-i\lambda Ax-i\lambda Bt},$

we change Lax pair (2.1) in the form

$\left\\{\begin{array}[]{l}\mu_{x}-i\lambda[A,\mu]=P\mu,\\\
\mu_{t}-i\lambda[B,\mu]=Q\mu,\end{array}\right.$ (2.3)

which can be further written in differential form as

$d(e^{-i\lambda\hat{A}x-i\lambda\hat{B}t}\mu)=W(x,t,\lambda),$ (2.4)

where

$W(x,t,\lambda)=e^{-i\lambda\hat{A}x-i\lambda\hat{B}t}(Pdx+Qdt)\mu$ (2.5)

and $e^{\hat{A}}X=e^{A}Xe^{-A}$.

###  2.2.  Spectral functions

We define three eigenfunctions $\\{\mu_{j}\\}_{1}^{3}$ of (2.3) by the
Volterra integral equations

$\mu_{j}(x,t,\lambda)=\mathbb{I}+\int_{\gamma_{j}}e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}W_{j}(x^{\prime},t^{\prime},\lambda).\qquad
j=1,2,3.$ (2.6)

where $W_{j}$ is given by (2.5) with $\mu$ replaced with $\mu_{j}$, and the
contours $\\{\gamma_{j}\\}_{1}^{3}$ are showed in Figure 1.

Figure 1. The three contours $\gamma_{1},\gamma_{2}$ and $\gamma_{3}$ in the
$(x,t)-$domain.

And we have the following inequalities on the contours:

$\begin{array}[]{ll}\gamma_{1}:&x-x^{\prime}\geq 0,t-t^{\prime}\leq 0,\\\
\gamma_{2}:&x-x^{\prime}\geq 0,t-t^{\prime}\geq 0,\\\
\gamma_{3}:&x-x^{\prime}\leq 0.\end{array}$ (2.7)

So, these inequalities imply that the functions $\\{\mu_{j}\\}_{1}^{3}$ are
bounded and analytic for $\lambda\in{\mathbb{C}}$ such that $\lambda$ belongs
to

$\begin{array}[]{ll}\mu_{1}:&(D_{2},\emptyset,D_{1}),\\\
\mu_{2}:&\emptyset,\\\ \mu_{3}:&(D_{1},\emptyset,D_{2}),\end{array}$ (2.8)

where $\\{D_{n}\\}_{1}^{2}$ denote two open, pairwisely disjoint subsets of
the complex $\lambda$ plane showed in Figure 2.

Figure 2. The sets $D_{n}$, $n=1,2$, which decompose the complex $k-$plane.

And the sets $\\{D_{n}\\}_{1}^{2}$ has the following properties:

$\begin{array}[]{l}D_{1}=\\{\lambda\in{\mathbb{C}}|\mathrm{Re}{l_{1}}<\mathrm{Re}{l_{2}}<\mathrm{Re}{l_{3}},\mathrm{Re}{z_{1}}>\mathrm{Re}{z_{2}}>\mathrm{Re}{z_{3}}\\},\\\
D_{2}=\\{\lambda\in{\mathbb{C}}|\mathrm{Re}{l_{1}}>\mathrm{Re}{l_{2}}>\mathrm{Re}{l_{3}},\mathrm{Re}{z_{1}}<\mathrm{Re}{z_{2}}<\mathrm{Re}{z_{3}}\\},\\\
\end{array}$

where $l_{i}(\lambda)$ and $z_{i}(\lambda)$ are the diagonal entries of
matrices $i\lambda A$ and $i\lambda B$, respectively.

###  2.3. Matrix valued FUNCTIONS $M_{n}$’s

For each $n=1,2$, define a solution $M_{n}(x,t,\lambda)$ of (2.3) by the
following system of integral equations:

$(M_{n})_{ij}(x,t,\lambda)=\delta_{ij}+\int_{\gamma_{ij}^{n}}(e^{(-i\lambda\hat{A}x-i\lambda\hat{B}t)}W_{n}(x^{\prime},t^{\prime},\lambda))_{ij},\quad\lambda\in
D_{n},\quad i,j=1,2,3.$ (2.9)

where $W_{n}$ an $M_{n}$ are given by (2.5), and the contours
$\gamma_{ij}^{n}$, $n=1,2$, $i,j=1,2,3$ are defined by

$\gamma_{ij}^{n}=\left\\{\begin{array}[]{lclcl}\gamma_{1}&if&\mathrm{Re}l_{i}(\lambda)<\mathrm{Re}l_{j}(\lambda)&and&\mathrm{Re}z_{i}(\lambda)\geq\mathrm{Re}z_{j}(\lambda),\\\
\gamma_{2}&if&\mathrm{Re}l_{i}(\lambda)<\mathrm{Re}l_{j}(\lambda)&and&\mathrm{Re}z_{i}(\lambda)<\mathrm{Re}z_{j}(\lambda),\\\
\gamma_{3}&if&\mathrm{Re}l_{i}(\lambda)\geq\mathrm{Re}l_{j}(\lambda)&&.\\\
\end{array}\right.\quad\mbox{for }\quad\lambda\in D_{n}.$ (2.10)

The following proposition ascertains that the $M_{n}$’s defined in this way
have the properties required for the formulation of a Riemann-Hilbert problem.

######  Proposition 2.1.

For each $n=1,2$, the function $M_{n}(x,t,\lambda)$ is well-defined by
equation (2.9) for $\lambda\in\bar{D}_{n}$ and $(x,t)\in\Omega$. For any fixed
point $(x,t)$, $M_{n}$ is bounded and analytic as a function of $\lambda\in
D_{n}$ away from a possible discrete set of singularities $\\{\lambda_{j}\\}$
at which the Fredholm determinant vanishes. Moreover, $M_{n}$ admits a bounded
and continuous extension to $\bar{D}_{n}$ and

$M_{n}(x,t,\lambda)=\mathbb{I}+O(\frac{1}{\lambda}),\qquad\lambda\rightarrow\infty,\quad\lambda\in
D_{n}.$ (2.11)

###### Proof.

The bounedness and analyticity properties are established in appendix B in
[7]. And substituting the expansion

$M=M_{0}+\frac{M^{(1)}}{\lambda}+\frac{M^{(2)}}{\lambda^{2}}+\cdots,\qquad\lambda\rightarrow\infty.$

into the Lax pair (2.3) and comparing the terms of the same order of $\lambda$
yield the equation (2.11). ∎

###  2.4. The jump matrices

We define spectral functions $S_{n}(\lambda)$, $n=1,2$, and

$S_{n}(\lambda)=M_{n}(0,0,\lambda),\qquad\lambda\in D_{n},\quad n=1,2.$ (2.12)

Let $M$ denote the sectionally analytic function on the complex
$\lambda-$plane which equals $M_{n}$ for $\lambda\in D_{n}$. Then $M_{n}$
satisfies the jump conditions

$M_{1}=M_{2}J,\qquad\lambda\in{\mathbb{R}},$ (2.13)

where the jump matrices $J(x,t,\lambda)$ are defined by

$J=e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}(S_{2}^{-1}S_{1}).$ (2.14)

According to the definition of the $\gamma^{n}$, we find that

$\begin{array}[]{ll}\gamma^{1}=\left(\begin{array}[]{lll}\gamma_{3}&\gamma_{1}&\gamma_{1}\\\
\gamma_{3}&\gamma_{3}&\gamma_{1}\\\
\gamma_{3}&\gamma_{3}&\gamma_{3}\end{array}\right)&\gamma^{2}=\left(\begin{array}[]{lll}\gamma_{3}&\gamma_{3}&\gamma_{1}\\\
\gamma_{3}&\gamma_{3}&\gamma_{3}\\\
\gamma_{1}&\gamma_{1}&\gamma_{3}\end{array}\right)\\\ \end{array}$ (2.15)

###  2.5. The adjugated eigenfunctions

We will also need the analyticity and boundedness properties of the minors of
the matrices $\\{\mu_{j}(x,t,\lambda)\\}_{1}^{3}$. We recall that the adjugate
matrix $X^{A}$ of a $3\times 3$ matrix $X$ is defined by

$X^{A}=\left(\begin{array}[]{ccc}m_{11}(X)&-m_{12}(X)&m_{13}(X)\\\
-m_{21}(X)&m_{22}(X)&-m_{23}(X)\\\
m_{31}(X)&-m_{32}(X)&m_{33}(X)\end{array}\right),$

where $m_{ij}(X)$ denote the $(ij)$th minor of $X$.

It follows from (2.3) that the adjugated eigenfunction $\mu^{A}$ satisfies the
Lax pair

$\left\\{\begin{array}[]{l}\mu_{x}^{A}+[i\lambda A,\mu^{A}]=-P^{T}\mu^{A},\\\
\mu_{t}^{A}+[i\lambda B,\mu^{A}]=-Q^{T}\mu^{A}.\end{array}\right.$ (2.16)

where $V^{T}$ denote the transform of a matrix $V$. Thus, the eigenfunctions
$\\{\mu_{j}^{A}\\}_{1}^{3}$ are solutions of the integral equations

$\mu_{j}^{A}(x,t,\lambda)=\mathbb{I}-\int_{\gamma_{j}}e^{-i\lambda\hat{A}(x-x^{\prime})-i\lambda
B(t-t^{\prime})}(P^{T}dx+Q^{T})\mu^{A},\quad j=1,2,3.$ (2.17)

Then we can get the following analyticity and boundedness properties:

$\begin{array}[]{ll}\mu_{1}^{A}:&(D_{1},\emptyset,D_{2}),\\\
\mu_{2}^{A}:&\emptyset,\\\ \mu_{3}^{A}:&(D_{2},\emptyset,D_{1}).\end{array}$
(2.18)

###  2.6. The computation of jump matrices

Let us define the $3\times 3-$matrix value spectral functions $s(\lambda)$ and
$S(\lambda)$ by

$\mu_{3}(x,t,\lambda)=\mu_{2}(x,t,\lambda)e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}s(\lambda),$
(2.19a)
$\mu_{1}(x,t,\lambda)=\mu_{2}(x,t,\lambda)e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}S(\lambda).$
(2.19b)

Thus,

$s(\lambda)=\mu_{3}(0,0,\lambda),\qquad S(\lambda)=\mu_{1}(0,0,\lambda).$
(2.20)

And we deduce from the properties of $\mu_{j}$ and $\mu_{j}^{A}$ that
$s(\lambda)$ and $S(\lambda)$ have the following boundedness properties:

$\begin{array}[]{ll}s(\lambda):&(D_{1},\emptyset,D_{2}),\\\
S(\lambda):&(D_{2},\emptyset,D_{1}),\\\
s^{A}(\lambda):&(D_{2},\emptyset,D_{1}),\\\
S^{A}(\lambda):&(D_{1},\emptyset,D_{2}).\end{array}$

Moreover,

$M_{n}(x,t,\lambda)=\mu_{2}(x,t,\lambda)e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}S_{n}(\lambda),\quad\lambda\in
D_{n}.$ (2.21)

######  Proposition 2.2.

The $S_{n}$ can be expressed in terms of the entries of $s(\lambda)$ and
$S(\lambda)$ as follows:

$\begin{array}[]{l}S_{1}=\left(\begin{array}[]{ccc}s_{11}&\frac{m_{33}(s)M_{21}(S)-m_{23}(s)M_{31}(S)}{(s^{T}S^{A})_{11}}&\frac{S_{13}}{(S^{T}s^{A})_{33}}\\\
s_{21}&\frac{m_{33}(s)M_{11}(S)-m_{13}(s)M_{31}(S)}{(s^{T}S^{A})_{11}}&\frac{S_{23}}{(S^{T}s^{A})_{33}}\\\
s_{31}&\frac{m_{23}(s)M_{11}(S)-m_{13}(s)M_{21}(S)}{(s^{T}S^{A})_{11}}&\frac{S_{33}}{(S^{T}s^{A})_{33}}\end{array}\right),\\\
S_{2}=\left(\begin{array}[]{ccc}\frac{S_{11}}{(S^{T}s^{A})_{11}}&\frac{m_{21}(s)M_{33}(S)-m_{31}(s)M_{23}(S)}{(s^{T}S^{A})_{33}}&s_{13}\\\
\frac{S_{21}}{(S^{T}s^{A})_{11}}&\frac{m_{11}(s)M_{33}(S)-m_{31}(s)M_{13}(S)}{(s^{T}S^{A})_{33}}&s_{23}\\\
\frac{S_{31}}{(S^{T}s^{A})_{11}}&\frac{m_{11}(s)M_{23}(S)-m_{21}(s)M_{13}(S)}{(s^{T}S^{A})_{33}}&s_{33}\end{array}\right),\end{array}$
(2.22)

where $m_{ij}$ and $M_{ij}$ denote that the $(i,j)-$th minor of $s$ and $S$,
respectively.

###### Proof.

Let $\gamma_{3}^{X_{0}}$ denote the contour $(X_{0},0)\rightarrow(x,t)$ in the
$(x,t)-$plane, here $X_{0}>0$ is a constant. We introduce
$\mu_{3}(x,t,k;X_{0})$ as the solution of (2.6) with $j=3$ and with the
contour $\gamma_{3}$ replaced by $\gamma_{3}^{X_{0}}$. Similarly, we define
$M_{n}(x,t,\lambda;X_{0})$ as the solution of (2.9) with $\gamma_{3}$ replaced
by $\gamma_{3}^{X_{0}}$. We will first derive expression for
$S_{n}(\lambda;X_{0})=M_{n}(0,0,\lambda;X_{0})$ in terms of $S(\lambda)$ and
$s(\lambda;X_{0})=\mu_{3}(0,0,\lambda;X_{0})$. Then (2.22) will follow by
taking the limit $X_{0}\rightarrow\infty$.

First, We have the following relations:

$\left\\{\begin{array}[]{l}M_{n}(x,t,\lambda;X_{0})=\mu_{1}(x,t,\lambda)e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}R_{n}(\lambda;X_{0}),\\\
M_{n}(x,t,\lambda;X_{0})=\mu_{2}(x,t,\lambda)e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}S_{n}(\lambda;X_{0}),\\\
M_{n}(x,t,\lambda;X_{0})=\mu_{3}(x,t,\lambda)e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}T_{n}(\lambda;X_{0}).\end{array}\right.$
(2.23)

Then we get $R_{n}(\lambda;X_{0})$ and $T_{n}(\lambda;X_{0})$ are defined as
follows:

$R_{n}(\lambda;X_{0})=e^{-i\lambda\hat{B}T}M_{n}(0,T,\lambda;X_{0}),$ (2.24a)
$T_{n}(\lambda;X_{0})=e^{-i\lambda\hat{A}X_{0}}M_{n}(X_{0},0,\lambda;X_{0}).$
(2.24b)

The relations (2.23) imply that

$s(\lambda;X_{0})=S_{n}(\lambda;X_{0})T^{-1}_{n}(\lambda;X_{0}),\qquad
S(\lambda)=S_{n}(\lambda;X_{0})R^{-1}_{n}(\lambda;X_{0}).$ (2.25)

These equations constitute a matrix factorization problem which, given
$\\{s(\lambda),S(\lambda)\\}$ can be solved for the $\\{R_{n},S_{n},T_{n}\\}$.
Indeed, the integral equations (2.9) together with the definitions of
$\\{R_{n},S_{n},T_{n}\\}$ imply that

$\left\\{\begin{array}[]{lll}(R_{n}(\lambda;X_{0}))_{ij}=0&if&\gamma_{ij}^{n}=\gamma_{1},\\\
(S_{n}(\lambda;X_{0}))_{ij}=0&if&\gamma_{ij}^{n}=\gamma_{2},\\\
(T_{n}(\lambda;X_{0}))_{ij}=0&if&\gamma_{ij}^{n}=\gamma_{3}.\end{array}\right.$
(2.26)

It follows that (2.25) are 18 scalar equations for 18 unknowns. By computing
the explicit solution of this algebraic system, we find that
$\\{S_{n}(\lambda;X_{0})\\}_{1}^{2}$ are given by the equation obtained from
(2.22) by replacing $\\{S_{n}(\lambda),s(\lambda)\\}$ with
$\\{S_{n}(\lambda;X_{0}),s(\lambda;X_{0})\\}$. taking $X_{0}\rightarrow\infty$
in this equation, we arrive at (2.22). ∎

###  2.7. The residue conditions

Since $\mu_{2}$ is an entire function, it follows from (2.21) that M can only
have singularities at the points where the $S_{n}^{\prime}s$ have
singularities. We infer from the explicit formulas (2.22) that the possible
singularities of $M$ are as follows:

  * •

$[M]_{1}$ could have poles in $D_{2}$ at the zeros of
$(S^{T}s^{A})_{11}(\lambda)$;

  * •

$[M]_{2}$ could have poles in $D_{1}$ at the zeros of
$(s^{T}S^{A})_{11}(\lambda)$;

  * •

$[M]_{2}$ could have poles in $D_{2}$ at the zeros of
$(s^{T}S^{A})_{33}(\lambda)$;

  * •

$[M]_{3}$ could have poles in $D_{1}$ at the zeros of
$(S^{T}s^{A})_{33}(\lambda)$.

We denote the above possible zeros by $\\{\lambda_{j}\\}_{1}^{N}$ and assume
they satisfy the following assumption.

######  Assumption 2.3.

We assume that

  * •

$(S^{T}s^{A})_{11}(\lambda)$ has $n_{0}$ possible simple zeros in $D_{2}$
denoted by $\\{\lambda_{j}\\}_{1}^{n_{0}}$;

  * •

$(s^{T}S^{A})_{11}(\lambda)$ has $n_{1}-n_{0}$ possible simple zeros in
$D_{1}$ denoted by $\\{\lambda_{j}\\}_{n_{0}+1}^{n_{1}}$;

  * •

$(s^{T}S^{A})_{33}(\lambda)$ has $n_{2}-n_{1}$ possible simple zeros in
$D_{2}$ denoted by $\\{\lambda_{j}\\}_{n_{1}+1}^{n_{2}}$;

  * •

$(S^{T}s^{A})_{33}(\lambda)$ has $n_{3}-n_{2}$ possible simple zeros in
$D_{1}$ denoted by $\\{\lambda_{j}\\}_{n_{2}+1}^{N}$;

and that none of these zeros coincide. Moreover, we assume that none of these
functions have zeros on the boundaries of the $D_{n}$’s.

We determine the residue conditions at these zeros in the following:

######  Proposition 2.4.

Let $\\{M_{n}\\}_{1}^{2}$ be the eigenfunctions defined by (2.9) and assume
that the set $\\{\lambda_{j}\\}_{1}^{N}$ of singularities are as the above
assumption. Then the following residue conditions hold:

$\displaystyle\begin{array}[]{l}{Res}_{\lambda=\lambda_{j}}[M]_{1}=\frac{1}{\dot{(S^{T}s^{A})_{11}(\lambda_{j})}}\frac{(S_{11}s_{23}-S_{21}s_{13})(\lambda_{j})}{m_{31}(\lambda_{j})}e^{\theta_{21}(\lambda_{j})}[M(\lambda_{j})]_{2},\\\
\quad 1\leq j\leq n_{0},\lambda_{j}\in D_{2}\end{array},$ (2.27c)
$\displaystyle\begin{array}[]{l}{Res}_{\lambda=\lambda_{j}}[M]_{2}=-\frac{1}{\dot{(s^{T}S^{A})_{11}(\lambda_{j})}}\frac{M_{21}(S^{T}s^{A})_{33}(\lambda_{j})}{(S_{13}(\lambda_{j})s_{31}(\lambda_{j})-S_{33}(\lambda_{j})s_{11}(\lambda_{j}))}e^{\theta_{12}(\lambda_{j})}[M(\lambda_{j})]_{1},\\\
\quad n_{0}<j\leq n_{1},\lambda_{j}\in D_{1}\end{array},$ (2.27f)
$\displaystyle\begin{array}[]{l}{Res}_{\lambda=\lambda_{j}}[M]_{2}=-\frac{1}{\dot{(s^{T}S^{A})_{33}(\lambda_{j})}}\frac{M_{23}(S^{T}s^{A})_{11}(\lambda_{j})}{(s_{13}(\lambda_{j})S_{31}(\lambda_{j})-s_{33}(\lambda_{j})S_{11}(\lambda_{j}))}e^{\theta_{32}(\lambda_{j})}[M(\lambda_{j})]_{3},\\\
\quad n_{1}<j\leq n_{2},\lambda_{j}\in D_{2}\end{array},$ (2.27i)
$\displaystyle\begin{array}[]{l}{Res}_{\lambda=\lambda_{j}}[M]_{3}=\frac{1}{\dot{(S^{T}s^{A})_{33}(\lambda_{j})}}\frac{(S_{13}s_{21}-S_{23}s_{11})(\lambda_{j})}{m_{33}(\lambda_{j})}e^{\theta_{23}(\lambda_{j})}[M(\lambda_{j})]_{2},\\\
\quad n_{2}<j\leq N,\lambda_{j}\in D_{1}\end{array},$ (2.27l)

where $\dot{f}=\frac{df}{d\lambda}$, and $\theta_{ij}$ is defined by

$\theta_{ij}(x,t,\lambda)=(l_{i}-l_{j})x+(z_{i}-z_{j})t,\quad i,j=1,2,3.$
(2.28)

###### Proof.

We will prove (2.27c), (2.27f), the other conditions follow by similar
arguments. Equation (2.21) implies the relation

$M_{1}=\mu_{2}e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}S_{1},$ (2.29a)
$M_{2}=\mu_{2}e^{(i\lambda\hat{A}x+i\lambda\hat{B}t)}S_{2}.$ (2.29b)

In view of the expressions for $S_{1}$ and $S_{2}$ given in (2.22), the three
columns of (2.29a) read:

$\displaystyle[M_{1}]_{1}=[\mu_{2}]_{1}s_{11}(\lambda)+[\mu_{2}]_{2}e^{\theta_{21}}s_{21}(\lambda)+[\mu_{2}]_{3}e^{\theta_{31}}s_{31}(\lambda),$
(2.30a)
$\displaystyle\begin{array}[]{ll}[M_{1}]_{2}=&[\mu_{2}]_{1}e^{\theta_{12}}\frac{m_{33}M_{21}-m_{23}M_{31}}{(s^{T}S^{A})_{11}}(\lambda)+[\mu_{2}]_{2}\frac{m_{33}M_{11}-m_{13}M_{31}}{(s^{T}S^{A})_{11}}(\lambda)\\\
&+[\mu_{2}]_{3}e^{\theta_{32}}\frac{m_{23}M_{11}-m_{13}M_{21}}{(s^{T}S^{A})_{11}}(\lambda)\end{array},$
(2.30d)
$\displaystyle[M_{1}]_{3}=[\mu_{2}]_{1}e^{\theta_{13}}\frac{S_{13}}{(S^{T}s^{A})_{33}}(\lambda)+[\mu_{2}]_{2}e^{\theta_{23}}\frac{S_{23}}{(S^{T}s^{A})_{33}}(\lambda)+[\mu_{2}]_{3}\frac{S_{33}}{(S^{T}s^{A})_{33}}(\lambda).$
(2.30e)

while the three columns of (2.29b) read:

$\displaystyle[M_{2}]_{1}=[\mu_{2}]_{1}\frac{S_{11}}{(S^{T}s^{A})_{11}}(\lambda)+[\mu_{2}]_{2}e^{\theta_{21}}\frac{S_{21}}{(S^{T}s^{A})_{11}}(\lambda)+[\mu_{2}]_{3}e^{\theta_{31}}\frac{S_{31}}{(S^{T}s^{A})_{11}}(\lambda),$
(2.31a)
$\displaystyle\begin{array}[]{ll}[M_{2}]_{2}=&[\mu_{2}]_{1}e^{\theta_{12}}\frac{m_{21}M_{33}-m_{31}M_{23}}{(s^{T}S^{A})_{33}}(\lambda)+[\mu_{2}]_{2}\frac{m_{11}M_{33}-m_{31}M_{13}}{(s^{T}S^{A})_{33}}(\lambda)\\\
&+[\mu_{2}]_{3}e^{\theta_{32}}\frac{m_{11}M_{23}-m_{21}M_{13}}{(s^{T}S^{A})_{33}}(\lambda)\end{array},$
(2.31d)
$\displaystyle[M_{2}]_{3}=[\mu_{2}]_{1}s_{13}e^{\theta_{13}}+[\mu_{2}]_{2}s_{23}e^{\theta_{23}}+[\mu_{2}]_{3}s_{33}.$
(2.31e)

We first suppose that $\lambda_{j}\in D_{2}$ is a simple zero of
$(S^{T}s^{A})_{11}(\lambda)$. Solving (2.31d) and (2.31e) for $[\mu_{2}]_{1}$
and $[\mu_{2}]_{2}$ and substituting the result in to (2.31a), we find

$\begin{array}[]{rl}[M_{1}]_{1}=&\frac{S_{11}s_{23}-S_{21}s_{13}}{(S^{T}s^{A})_{11}m_{31}}e^{\theta_{21}}[M_{2}]_{2}+\frac{M_{33}}{m_{31}}e^{\theta_{31}}[M_{2}]_{3}\\\
&+\frac{1}{m_{31}}e^{\theta_{31}}[\mu_{2}]_{3}\end{array}.$

Taking the residue of this equation at $\lambda_{j}$, we find the condition
(2.27c) in the case when $\lambda_{j}\in D_{2}$. Similarly, we can get the
equation (2.27l).

Then let us consider that $\lambda_{j}\in D_{1}$ is a simple zero of
$(s^{T}S^{A})_{11}(\lambda)$. Solving (2.30a) and (2.30e) for $[\mu_{2}]_{1}$
and $[\mu_{2}]_{3}$ and substituting the result in to (2.30d), we find

$\begin{array}[]{rl}[M_{1}]_{2}=&-\frac{M_{21}(S^{T}s^{A})_{33}}{(s^{T}S^{A})_{11}(S_{13}s_{31}-S_{33}s_{11})}e^{\theta_{12}}[M_{1}]_{1}-\frac{(S^{T}s^{A})_{33}}{S_{13}s_{31}-S_{33}s_{11}}[\mu_{2}]_{2}\\\
&-\frac{m_{23}(S^{T}s^{A})_{33}}{S_{13}s_{31}-S_{33}s_{11}}e^{\theta_{32}}[M_{1}]_{3}\end{array}.$

Taking the residue of this equation at $\lambda_{j}$, we find the condition
(2.27f) in the case when $\lambda_{j}\in D_{1}$. Similarly, we can get the
equation (2.27i). ∎

##  3\. The Riemann-Hilbert problem

The sectionally analytic function $M(x,t,\lambda)$ defined in section 2
satisfies a Riemann-Hilbert problem which can be formulated in terms of the
initial and boundary values of $p_{ij}(x,t)$. By solving this Riemann-Hilbert
problem, the solution of (1.1) can be recovered for all values of $x,t$.

######  Theorem 3.1.

Suppose that $p_{ij}(x,t)$ are a solution of (1.1) in the half-line domain
$\Omega$ with sufficient smoothness and decays as $x\rightarrow\infty$. Then
$p_{ij}(x,t)$ can be reconstructed from the initial value
$\\{p_{ij,0}(x)\\}_{i,j=1}^{3}$ and boundary values
$\\{q_{ij,0}(t)\\}_{i,j=1}^{3}$ defined as follows,

$p_{ij,0}(x)=p_{ij}(x,0),\quad q_{ij,0}(t)=p_{ij}(0,t).$ (3.1)

Use the initial and boundary data to define the jump matrices $J(x,t,\lambda)$
as well as the spectral $s(\lambda)$ and $S(\lambda)$ by equation (2.19).
Assume that the possible zeros $\\{\lambda_{j}\\}_{1}^{N}$ of the functions
$(S^{T}s^{A})_{33}(\lambda),(s^{T}S^{A})_{11}(\lambda)$,
$(s^{T}S^{A})_{33}(\lambda),(S^{T}s^{A})_{33}(\lambda)$ are as in assumption
2.3.

Then the solution $\\{p_{ij}(x,t)\\}_{i,j=1}^{3}$ is given by

$p_{ij}(x,t)=-i(a_{i}-a_{j})\lim_{\lambda\rightarrow\infty}(\lambda
M(x,t,\lambda))_{ij}.$ (3.2)

where $M(x,t,\lambda)$ satisfies the following $3\times 3$ matrix Riemann-
Hilbert problem:

  * •

$M$ is sectionally meromorphic on the complex $\lambda-$plane with jumps
across the contour ${\mathbb{R}}$, see Figure 2.

  * •

Across the contour ${\mathbb{R}}$, $M$ satisfies the jump condition

$M_{1}(x,t,\lambda)=M_{2}(x,t,\lambda)J(x,t,\lambda),\quad\lambda\in{\mathbb{R}}.$
(3.3)

where the jump $J$ is defined by the equation (2.14).

  * •

$M(x,t,\lambda)=\mathbb{I}+O(\frac{1}{\lambda}),\qquad\lambda\rightarrow\infty$.

  * •

The residue condition of $M$ is showed in Proposition 2.4.

###### Proof.

It only remains to prove (3.2) and this equation follows from the large
$\lambda$ asymptotics of the eigenfunctions.

We write the large $\lambda$ asymptotics of $M$ as follows:

$M(x,t,\lambda)=M_{0}(x,t)+\frac{M_{1}(x,t)}{\lambda}+\cdots.\qquad\lambda\rightarrow+\infty.$
(3.4)

And insert this equation into the equation (2.3) and compare the coeffients of
the same order $\lambda$, for the $O(\lambda)$, we have $M_{0}$ is a diagonal
matrix ; for the $O(1)$, we get $M_{0}=\mathbb{I}$ by comparing the diagonal
elements, and we can have the following equation by comparing the other
elements

$-i[A,M_{1}]=P,$ (3.5)

this equation reads the required result of $p_{ij}(x,t)$

$p_{ij}(x,t)=-i(a_{i}-a_{j})M_{1,ij}(x,t).$ (3.6)

∎

Acknowledgements The work of Xu was partially supported by Excellent Doctor
Research Funding Project of Fudan University. The work described in this paper
was supported by grants from the National Science Foundation of China (Project
No.10971031;11271079), Doctoral Programs Foundation of the Ministry of
Education of China, and the Shanghai Shuguang Tracking Project (project
08GG01).

## References

  * [1] A. S. Fokas, A unified transform method for solving linear and certain nonlinear PDEs, Proc. R. Soc. Lond. A 453(1997), 1411-1443.
  * [2] A. S. Fokas, Integrable nonlinear evolution equations on the half-line, Commun. Math. Phys. 230(2002), 1-39.
  * [3] A.S. Fokas, A unified approach to boundary value problems, in: CBMS-NSF Regional Conference Series in Applied Mathematics, SIAM, 2008.
  * [4] A. Boutet De Monvel, A.S. Fokas, D. Shepelsky, Integrable nonlinear evolution equations on a finite interval, Comm. Math. Phys. 263 (2006) 133-172.
  * [5] J. Lenells, Boundary value problems for the stationary axisymmetric Einstein equations: a disk rotating around a black hole, Comm. Math. Phys. 304 (2011) 585-635.
  * [6] J. Lenells, A.S. Fokas, Boundary-value problems for the stationary axisymmetric Einstein equations: a rotating disc, Nonlinearity 24 (2011) 177-206.
  * [7] J. Lenells, Initial-boundary value problems for integrable evolution equations with $3\times 3$ Lax pairs, Phys D 241(2012) 857-875.
  * [8] J. Lenells, The Degasperis-Procesi equation on the half-line, Nonlinear Analysis  76(2013) 122-139.
  * [9] R. Beals, R. R. Coifman, Scattering and inverse scattering for first order systems, Comm. Pure Appl. Math. 37 (1984), 39-90.
  * [10] P. Deift and X. Zhou, A steepest descent method for oscillatory Riemann–Hilbert problems, Ann. of Math. (2) 137(1993), 295-368.
  * [11] B. Pelloni and D. Pinotsis, Boundary value problems for the N-wave interaction equation, Phys. Lett. A 373 (2009), 1940-1950.
  * [12] V. S. Gerdjikov and G. G. Grahovski, On the 3-wave equations with constant boundary conditions, arXiv:1204.5346.
  * [13] V. S. Gerdjikov, G. G. Grahovski and N. A. Kostov, On N-wave type systems and their gauge equivalent, Euro. Phys. J. B 29 (2002) 243 248\.
  * [14] Y. S. Li, Soliton and Integrable System, Advanced Series in Nonlinear Science,(1999) (in Chinese).
  * [15] S. C. Chiu, On the self-induced transparency effect of the three-wave resonant process, J. Math. Phys. 19 (1978), 168-176.
  * [16] F. Calogero, A. Degasperis, Novel solution of the system describing the resonant interaction of three waves, Phys. D 200 (2005), 242-256.
  * [17] F. Calogero, Universality and integrability of the nonlinear evolution PDE s describing N-wave integrations, J. Math. Phys., 30 (1989), 28-40.
  * [18] F. Calogero, Solutions of certain integrable nonlinear PDE s describing nonresonant N-wave integrations, J. Math. Phys., 30 (1989), 639-654.
  * [19] L. D. Faddeev , L. A Takhtadjan., Hamiltonian approach in the theory of solitons, Springer Verlag, Berlin (1987).
  * [20] D. J Kaup., The three-wave interaction-a nondispersive phenomenon, Stud. Appl. Math. 55 (1976), 9-44.
  * [21] D. J Kaup., A. Reiman , A. Bers , Space-time evolution of nonlinear three-wave interactions. I. Interactions in an homogeneous medium, Rev. Mod. Phys. 51 (1979), 275-310.
  * [22] S. V. Manakov, An example of a completely integrable nonlinear wave field with non-trivial dynamics (Lee model), Teor. Mat. Phys. 28 (1976), 172-179.
  * [23] V. E. Zakharov, S. V Manakov., Exact theory of resonant interaction of wave packets in nonlinear media, INF preprint 74-41, Novosibirsk (1975) (In Russian).
  * [24] V. E. Zakharov, S. V Manakov., On the theory of resonant interaction of wave packets in nonlinear media, Zh. Exp. Teor. Fiz., 69 (1975), 1654-1673 (In Russian).
  * [25] V. E. Zakharov, S. V. Manakov, S. P. Novikov, L. I. Pitaevskii, Theory of solitons: the inverse scattering method, Consultant Bureau, Plenum Press, N.Y. (1984).


# Perturbation CheckLists for Evaluating NLG Evaluation Metrics

Ananya B. Sai1,2 Tanay Dixit1 Dev Yashpal Sheth1,2
Sreyas Mohan3 Mitesh M. Khapra1,2,4
1 Indian Institute of Technology, Madras
2 Robert Bosch Centre for Data Science and Artificial Intelligence, IIT Madras
3 Center for Data Science, New York University 4 AI4Bharat
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Natural Language Generation (NLG) evaluation is a multifaceted task requiring
assessment of multiple desirable criteria, e.g., fluency, coherency, coverage,
relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG
tasks, we observe that the human evaluation scores on these multiple criteria
are often not correlated. For example, there is a very low correlation between
human scores on fluency and data coverage for the task of structured data to
text generation. This suggests that the current recipe of proposing new
automatic evaluation metrics for NLG by showing that they correlate well with
scores assigned by humans for a single criteria (overall quality) alone is
inadequate. Indeed, our extensive study involving 25 automatic evaluation
metrics across 6 different tasks and 18 different evaluation criteria shows
that there is no single metric which correlates well with human scores on all
desirable criteria, for most NLG tasks. Given this situation, we propose
CheckLists for better design and evaluation of automatic metrics. We design
templates which target a specific criteria (e.g., coverage) and perturb the
output such that the quality gets affected only along this specific criteria
(e.g., the coverage drops). We show that existing evaluation metrics are not
robust against even such simple perturbations and disagree with scores
assigned by humans to the perturbed output. The proposed templates thus allow
for a fine-grained assessment of automatic evaluation metrics exposing their
limitations and will facilitate better design, analysis and evaluation of such
metrics.111Our templates and code are available at
https://iitmnlp.github.io/EvalEval/

##  1 Introduction

As the number of tasks and benchmarks for NLG have increased Gehrmann et al.
(2021), the challenges in evaluating NLG systems have also continued to grow
Liu et al. (2016); Nema and Khapra (2018); Sai et al. (2019). One reliable way
of evaluating NLG systems is to collect human judgements. However, this is a
time consuming and expensive process (Freitag et al., 2021; Deriu et al.,
2019; Howcroft et al., 2020). Hence, automatic evaluation metrics such as BLEU
Papineni et al. (2002) which are quicker to compute have become popular,
despite being less reliable Callison-Burch et al. (2006); Reiter (2018).

The survey by Sai et al. (2020b) shows that more than 35 automatic evaluation
metrics have been proposed for NLG since 2014, however, there is no careful
evaluation of the ability of such metrics to assess the quality of the output
of an NLG system on multiple desired criteria. For example, consider the task
of dialog evaluation, where humans are asked to score the output on multiple
criteria such as fluency, adequacy, coherence, informativeness, engagingness,
consistency, etc. Contrast this with automatic evaluation metrics such as
BLEU, BLEURT Sellam et al. (2020), DEB Sai et al. (2020a), ADEM Lowe et al.
(2017), etc., which assign a single score to the output. What does this score
indicate? More specifically, does a low DEB score indicate that the output is
not fluent or does it indicate that the output is fluent but not coherent or
neither fluent nor coherent? Hence, a single overall score assigned by
automatic evaluation metrics is not very informative in deciding which aspects
of model improvement should one focus on.

The question then is why do current automatic evaluation metrics produce only
a single overall score? This is simply because of a conscious choice made
while designing automatic evaluation metrics. In particular, current works
only focus on evaluating whether the scores assigned by the proposed metric
correlate well with the overall quality scores assigned by humans as opposed
to all relevant criteria. In this work, we make a case for shifting the focus
to all relevant criteria while evaluating such metrics. To this end, we first
do a systematic study involving 6 NLG tasks, 18 different human evaluation
criteria (fluency, coverage,coherence, consistency, etc) and 25 automatic
evaluation metrics. We take existing English datasets containing human
judgements for various tasks and criteria and make two important observations.
First, for a given task, human scores for different criteria often have a low
correlation, thereby suggesting that these criteria cannot be clubbed together
and evaluated using a single score assigned by an automatic evaluation metric.
Second, none of the automatic evaluation metrics have a high correlation with
human scores for any of the desired criteria for a given task.

The above results highlight a lacunae in the evaluation of automatic
evaluation metrics wherein their ability to assess the output on multiple
criteria is not evaluated. In this work, we propose a flexible framework which
allows a systematic evaluation of the capabilities of an automatic evaluation
metric. In particular, we propose CheckList-style templates Ribeiro et al.
(2020) which evaluate the robustness of the metrics to certain perturbations
targeting specific criteria. We illustrate this idea with an example in Table
1. In row 2 of Table 1 a gold standard output is perturbed by changing named
entities, thereby affecting its factual correctness which is important for
data-to-text generation. If an automatic evaluation metric indeed evaluates
factual correctness then its score should drop when presented with such a
perturbed output.

Original sentence: |  Cameron is the director of Titanic
---|---
Change names: |  Kate is the director of Titanic (incorrect)

Table 1: Example of a perturbed output.

For the 6 NLG tasks mentioned earlier, we create 34 such perturbation
templates covering 18 different evaluation criteria.We then instantiate these
templates to create large-scale test cases. For every perturbation, we also
collect human judgements to understand how much would a human change his/her
score when shown such a perturbed output. We find that for several
perturbations, the scores assigned by automatic evaluation metrics do not
agree with the scores assigned by humans, thereby indicating that current
automatic evaluation metrics are not robust to such perturbations (i.e., they
do not really evaluate the desired criteria). Overall, we believe that the
proposed templates provide a better framework for a more fine-grained
evaluation of automatic evaluation metrics which goes much beyond computing
correlations with human scores.

Task |  Criteria
---|---
Machine Translation |  Adequacy: The generated translation should adequately represent all the information present in the reference.
Question Generation |  Relevance: Is the question related to the source material they are based upon.
|  Answerability: Is the generated question answerable given the context.
|  Informativeness: The summary should convey the key points of the text.
|  Non-redundancy: The summary should not repeat any points, and ideally have
maximal information coverage within the limited text length.
Abstractive Summarization |  Referential clarity: Any intra-sentence or cross-sentence references in the summary should be unambiguous and within the scope of the summary.
|  Focus: The summary needs to have a focus and all the sentences need to
contain information related to this focal point.
|  Structure and Coherence: The summary should be a well-organized and
coherent body of information
Dialogue Generation |  Making sense: Does the bot say things that don’t make sense?
|  Engagingness: Is the dialogue agent enjoyable to talk to?
|  Interestingness: Did you find the bot interesting to talk to?
|  Inquisitivenes: Does the bot ask a good amount of questions?
|  Listening: Does the bot pay attention to what you say?
|  Avoiding Repetition: Does the bot repeat itself? (either within or across
utterances)
|  Humanness: Is the conversation with a person or a bot?
Image Captioning |  Relevance: The caption should be specific and related to the image.
|  Thoroughness: The caption should adequately describe the image.
|  Data Coverage: Does the text include descriptions of all predicates
presented in the data?
|  Relevance: Does the text describe only such predicates which are found in
the data?
Data to Text Generation |  Correctness: When describing predicates which are found in the data, does the text mention correct the objects and adequately introduces the subject for this specific predicate?
|  Text Structure: Is the text grammatical, well-structured, written in
acceptable English?
All above tasks |  Fluency: How fluent is the generated text?

Table 2: Criteria used for human judgements to evaluate NLG systems for each
of the 6 tasks

##  2 Criteria used in Human Evaluations

The goal of this work is to carefully evaluate automatic evaluation metrics
with a focus on their ability to capture the diverse set of criteria used by
humans while assessing NLG systems. To begin with, we describe the criteria
that the output of an NLG system must satisfy for the 6 NLG tasks that we
consider in this work, viz., machine translation (MT), dialog generation (DG),
automatic summarisation (AS), question generation(QG), data-to-text generation
(D2T) and image captioning (IC). Over the years, different works have proposed
different criteria for evaluating NLG systems. In this work, we consider a
popular set of criteria for each task as summarised in Sai et al. (2020b) and
presented in Table 2. Given the wide variety of criteria used for each task,
one obvious question to ask is whether we really need so many criteria or is a
single overall score enough.One could argue that it is obvious from the
definitions of the criteria that each of them is unique and a good score on
one (say, fluency) may not necessarily imply a good score on another (say,
coverage). However, we provide a quantitative argument for this by computing
the correlations between human scores for different criteria as described
below.

###  2.1 Correlations between different criteria

We use existing publicly available datasets containing human judgement scores
on multiple criteria for each of the 6 tasks described earlier. For example,
(Castro Ferreira et al., 2020) contains 3025 samples of outputs generated by
data-to-text generation systems that participated in the WebNLG 2020
challenge. For each of these samples, the organisers asked humans to rate the
output based on 5 criteria, viz., fluency, data coverage, relevance,
correctness and text structure. We use these scores to compute the
correlations between the scores of all the $5\choose 2$ pairs of criteria. We
repeat this for the other tasks using the datasets described in Table 3222For
AS, we could not find a dataset containing human judgements for the set of
criteria in Sai et al. (2020b). Hence, we use the dataset provided by Fabbri
et al. (2020). 333Note that all of the datasets mentioned in Table 3 were
collected using well established methods to ensure that the annotations were
of high quality. Some of these datasets do not explicitly report the Inter
Annotator Agreement (IAA) scores whereas others (Fabbri et al., 2020; Castro
Ferreira et al., 2020; Nema and Khapra, 2018) report a good IAA score ranging
from 0.63-0.71. Using these annotations, we compute the pairwise Kendall tau
correlations between all criteria for each task as seen in Figure 1. (Refer to
appendix A for pearson correlations of the criteria.)

Task | Data collected/ released by | # Samples (annotators)
---|---|---
QG | Nema and Khapra (2018) | 1000 (in-house)
AS | Fabbri et al. (2020) | 1600 (expert, crowdsource)
D2T | Castro Ferreira et al. (2020) | 3025 (crowdsourced)
DG | See et al. (2019) | 3316 (crowdsourced)
MT | Callison-Burch et al. (2007) | 10,754 (crowdsourced)
IC |  Aditya et al. (2015) (Coco subset) | 2007 (crowdsourced)

Table 3: Datasets with human scores on many criteria. Figure 1: Correlations
between criteria for different tasks. The darker the shade of the cell, the
lower is the correlation.

We see that, across tasks, for most pairs of criteria, the correlation is
moderate (between 0.3 and 0.5) to low ($<0.3$). The highest correlation of
0.76 is observed between interestingness and enjoyability for dialogue
generation. However other criteria such as avoiding repetition,
inquisitiveness, and making sense have low correlations with most of the other
criteria. We make similar observations for the correlations between the
criteria for other tasks. Even for IC the correlation between the 2 criteria
of thoroughness and correctness is 0.41. For MT, the commonly used criteria of
fluency and adequacy were found to be highly correlated with Pearson
correlation co-efficient of 0.69 (Banchs et al., 2015). This justifies why WMT
evaluations now ask humans to give to only a single score indicating overall
quality. However, given the low to moderate correlations between criteria for
other tasks, a similar strategy is not prudent for these tasks.
Takeaway: For tasks whose linguistic criteria show a low or at-best moderate
correlation with each other, a single score assigned by a automatic metric is
inadequate for a comprehensive assessment.

##  3 Perturbation Checklists

Task |  Criteria |  Perturbation |  Unmodified sentence |  Perturbed sentence
---|---|---|---|---
|  |  Jumbling word order |  We play badminton every evening. |  We badminton every evening play.
|  |  Subject-verb disagreement |  He doesn’t know how to bake. |  He don’t know how to bake.
|  Fluency |  Spelling errors |  Make the most of every opportunity presented to you. |  Make the most of evry opportunity presented to you.
|  |  Replace with synonyms |  The mangoes are delicious. |  The mangoes are tasty.
|  |  Contractions |  We are going to embark on an adventure. |  We’re going to embark on an adventure.
All tasks |  Invariance |  Numerals to words |  The flight will be delayed by 2 hours. |  The flight will be delayed by two hours.
|  |  Dropping out words or phrases |  I was being followed. |  I followed.
|  |  Add extra text |  This book is so inspiring. |  This book is so inspiring,I forgot.
MT |  Adequacy |  Negation/antonyms |  It will rain on Monday. |  It will not rain on Monday.
|  Informativeness |  Use hyponyms to create misinformation |  The girl my brother Andy met through MySpace turned out to be completely made up . |  The girl my friend Andy met through MySpace turned out to be completely made up.
|  Flow / coherence |  Reorder sentences |  The pandemic was spreading uncontrollably. Vaccines are being developed and tested rapidly. |  Vaccines are being developed and tested rapidly. The pandemic was spreading uncontrollably.
AS |  Referential clarity |  Replace nouns by pronouns |  The pandemic was spreading uncontrollably. Vaccines are being developed rapidly. |  It was spreading uncontrollably. They are being developed rapidly.
|  |  Drop question word |  When was he born ? |  Was he born?
|  Answerability |  Change question to assertive statement |  Who is the director of Titanic? |  The director of Titanic is James Cameron.
|  |  Mask a few words and predict |  How could Tesla run the experiments? |  How could Tesla run to the beach?
QG |  Relevance |  Perturb nouns |  Why did Mary go to the school? |  Why did Raj go to the market?
|  |  Negate a previous statement by same speaker |  Bot: I enjoy having your daughter in my class. User: I’m glad to hear that. |  I don’t enjoy having your daughter in my class.
|  Making sense |  Add extra non-sensible text |  Do you know where Dr. XYZ lives? |  Yes, my father is my grandmother’s father
|  |  Repeat previous utterances |  Do you know Dr.XYZ? |  Do you know where Dr. XYZ lives?
|  Avoid repetition |  Repeat phrases |  I like ice creams |  I like ice creams, ice creams
|  Listening |  Replace with "Can you repeat?" |  I need to book a taxi |  I’m sorry, can you repeat?
DG |  Relevance |  Map random responses |  I am new to coding. |  I am scared of snakes.
|  |  Change gender |  Two girls are playing with a doll |  Two boys are playing with a doll
|  Correctness |  Change attributes |  A small boy playing with a red ball |  A tall boy playing with a green ball
|  |  Drop objects/noun |  A small boy playing with a red ball |  A small playing with a red
IC |  Thoroughness |  Repeat (append) object |  A lady riding a horse. |  A lady riding a horse and a lady.
|  |  Use hyponyms |  Beethoven was a German musician |  Beethoven was a German architect
|  Correctness |  Change numbers |  The cricketer was born in 1990. |  The cricketer was born in 1950.
|  |  Drop phrases |  A small boy playing with a red ball |  A small boy playing with a
|  Data Coverage |  Repeat phrases |  Beethoven was a German musician |  Beethoven was a German musician and German musician.
D2T |  Relevance |  Perturb names |  Phillips was a child prodigy. |  James was a child prodigy.

Table 4: Perturbation templates targeting various criteria with examples. The
blue highlights indicate the portions of the original sentence affected by the
perturbation template. The red highlights indicate the changes in the modified
sentence.

So far we have established that if automatic evaluation metrics are to be used
as a substitute for human evaluations as a whole, then they should be capable
of evaluating the output on multiple desired criteria. However, the current
recipe of proposing and evaluating evaluation metrics does not take this into
account. To enable such a systematic evaluation of automatic evaluation
metrics, we propose perturbation checklists. Similar to the original Checklist
paper Ribeiro et al. (2020), the idea is to evaluate the performance of the
evaluation metric in detecting criteria-specific changes in the output.

We design such perturbation templates for each relevant criteria for each of
the 6 tasks as shown in Table 4. For example, consider the criteria fluency
which is relevant for all the tasks. Now consider a perturbation template for
this criteria which simply drops the stop words in the output. Such a
perturbation would definitely affect the fluency of the output. If an
automatic evaluation metric is capable of assessing fluency, then this drop in
the fluency of the output should get reflected in the score assigned by the
metric. More formally, let $p$ be the original output and $\hat{p}^{t}_{c}$ be
the output obtained by applying the perturbation template $t$ for the criteria
$c$. Further, let $f_{e}(p)$ be the score assigned by a given evaluation
metric $e$ to the output $p$, normalised to be in the range $[0,1]$. If the
metric $e$ is capable of assessing fluency then we would expect
$f_{e}(\hat{p}^{t}_{c})$ to be lower than $f_{e}(p)$. Now, further let $h(p)$
and $h(\hat{p}^{t}_{c})$ be the scores (also normalised to have range $[0,1]$)
assigned to the original and perturbed outputs by human annotators. We then
define a metric $s^{t}_{c}(e)$ which captures the ability of the metric $e$ to
detect the perturbation $t$ for the desired criteria $c$.

$s^{t}_{c}(e)=(h(\hat{p}^{t}_{c})-h(p))-(f_{e}(\hat{p}^{t}_{c})-f_{e}(p))$ (1)

The score $s^{t}_{c}(e)$ as defined above thus captures the deviation between
a human’s perception about the drop in the quality and the metric $e$’s
perception about the drop in the quality.

Automatic Evaluation Metrics Task-agnostic (Context Free metrics) Task-
specific (Context Dependent metrics) Word Based BLEU METEOR Banerjee and Lavie
(2005) ROUGE Lin (2004) TER Snover et al. (2006) CIDEr Vedantam et al. (2015)
Character Based chrF++ Popovic (2017) Pretrained (Static) Embeddings Greedy
Matching Rus and Lintean (2012) Embedding Average Landauer and Dumais (1997)
Vector Extrema Forgues and Pineau (2014) SMS Clark et al. (2019) WMDo Chow et
al. (2019) BERT Embeddings MoverScore Zhao et al. (2019) GRUEN Zhu and Bhat
(2020) BERTScore Zhang et al. (2020) End-to-End Trained BLEURT Sellam et al.
(2020) AS SUPERT Gao et al. (2020) BLANC Vasilyev et al. (2020) IC SPICE
Anderson et al. (2016) TIGEr Jiang et al. (2019) VilBERTScore Lee et al.
(2020) QG Q-metrics Nema and Khapra (2018) DG ADEM Lowe et al. (2017) RUBER
Tao et al. (2018) DEB Sai et al. (2020a) D2T PARENT Dhingra et al. (2019)

Figure 2: Metrics analysed in this study

We design a total of 34 such perturbation templates across all the criteria
and all the tasks. Each template is manually created by us and targets a
specific criteria. We also present invariant templates that do not affect any
criteria although they modify the sentences. For perturbations resulting from
such invariant templates the score of the metric should not drop. The
invariant and fluency-based templates are common for all the tasks considered
in this work. Table 4 shows sample perturbations generated by each of the
templates. (Please refer to appendix C for a more comprehensive list of the
proposed perturbations with examples for each task). These perturbed sentences
are generated automatically using the checklist framework Ribeiro et al.
(2020). This framework contains modules for performing simple string
manipulations such as dropping stop words, replacing/dropping named entities,
masking words or replacing them by other words/phrases. We also extend the
framework with additional modules for jumbling words, changing numbers to
words, subject-verb disagreement, changing gender, reordering sentences,
adding spurious text, and adding redundancy at the word/ phrase /sentence-
level.

##  4 Experimental setup

We first do a coarse grained evaluation of several metrics by computing their
correlations with the scores assigned by humans for multiple criteria. Note
that unlike existing studies which study such correlations for a small number
of metrics (typically, $n$-gram based metrics) for a specific task (say, MT)
and a single criteria (typically, overall quality), we do a more comprehensive
study involving a combination of 6 tasks, 25 metrics and multiple criteria.
Apart from this coarse grained evaluation which simply looks at correlations,
we also do a more fine-grained evaluation of the robustness of these metrics
to different criteria-specific perturbations as summarised in Table 4. This
fine-grained evaluation augments the coarse-grained evaluation and helps us
understand the evaluation capabilities of these metrics. Below, we describe
the datasets and evaluation metrics used in our work.

(a) Question Generation

(b) Abstractive Summarization

(c) Image Captioning

(d) Data-to-Text Generation

(e) Dialogue Generation

Figure 3: Correlations of metrics with different criteria (Note: For MT, we
refer to the results of the WMT 2020 shared metrics task Mathur et al.
(2020b). For DG, we only present the reference-less metrics since the dataset
does not contain references to compute the task-agnostic metric scores.)
(Refer appendix D for full plots)

Datasets. For the coarse-grained evaluation, we use the datasets containing
human judgements as described earlier in Table 3 in Section 2. For the fine-
grained evaluation, we use datasets containing multiple ground truth
references which can then be perturbed using our templates. For MT, we use the
expanded version of newstest2017 Chinese to English dataset Hassan et al.
(2018) which contains two references for each sentence. For QG, we use the
SQuAD dataset Rajpurkar et al. (2016) which contains multiple questions for
each passage. For AS, we use the curated personal narrative corpus Ouyang et
al. (2017). For DG, we use DailyDialog++ Sai et al. (2020a) which contains
two-speaker conversations on generic topics. For IC, we use the COCO component
of the Composite dataset containing 5 reference captions for each image Aditya
et al. (2015). Lastly, for D2T, we use the Triples-to-Text data of the WebNLG
2020 challenge dataset Castro Ferreira et al. (2020).

Applying perturbations. We take the reference sentences from the above task-
specific datasets and apply perturbations using the Checklist framework
described earlier in section 3. We first preprocess the sentences by
performing tokenization, part-of-speech tagging, named entity recognition,
etc. The targeted part of the sentence is then modified either by leveraging
simple string manipulation functions or by masking and generating the words/
phrases using the predictions by RoBERTa Liu et al. (2019). We provide more
implementation details in appendix B.

Automatic Evaluation Metrics. We study a total of 25 evaluation metrics
belonging to different classes as shown in Figure 2. For BLEU Papineni et al.
(2002), METEOR Banerjee and Lavie (2005), ROUGE-L Lin (2004), CIDEr Vedantam
et al. (2015), Embedding Averaging (EAvg) Landauer and Dumais (1997), Greedy
Matching (GM) Rus and Lintean (2012), and Vector Extrema (VE) Forgues and
Pineau (2014), we use the implementation provided by Sharma et al. (2017). For
chrF++ Popovic (2017), TER Snover et al. (2006), BERTScore Zhang et al.
(2020), and BLEURT Sellam et al. (2020) we use the repository of Castro
Ferreira et al. (2020). For SMS Clark et al. (2019), WMDo Chow et al. (2019),
and MoverScore Zhao et al. (2019), we use the implementation provided by
Fabbri et al. (2020). For all the task-specific metrics in Figure 2, we use
the official codes from the respective papers.

Collecting human judgements. For eq 1, we need human judgement scores. We
collect these with the help of 15 annotators who were computer science
graduates with a background in the field of Natural Language Processing (NLP)
and are also proficient in English language. For each task and criteria, the
annotators were provided with the corresponding perturbation templates and
asked to provide a penalty score indicating by how much a perturbation would
alter the meaning/essence of a sentence on a scale of 0-10. A score of 0
indicates that there is no difference between the original and modified
sentences upon application of the perturbation whereas a score of 10 indicates
that the perturbation drastically alters sentences. These scores are
normalised to be in the range $[0,1]$. The term $h(\hat{p}^{t}_{c})$ in eq 1
for perturbation $t$ is computed by subtracting the mean of all normalised
human penalty scores from $1$.

The standard deviation of the normalised annotator scores for each
perturbation lies between 0.03 to 0.2. We also measure the inter annotator
agreement by splitting annotators randomly into 2 groups and computing the
kendall tau correlation score between the average scores of the 2 groups
following Liu et al. (2016). This process was run 5 times with different seeds
and the final inter-annotator correlation score was found to be 0.79.

(a) Machine Translation

(b) Abstractive Summarization

(c) Question Generation

(d) Dialogue Generation

(e) Data-to-Text Generation

(f) Image Captioning

Figure 4: Heatmap of deviation of metric scores from human averages. The
darker the cell, the more the absolute deviation from human scores.

##  5 Results and Discussion

We now discuss the results of our experiments.

###  5.1 Insights from coarse-grained evaluation

Figure 3 shows the correlations of different automatic evaluation metrics with
multiple evaluation criteria, across 6 different NLG tasks. Our main
observations from the figure are:

Mostly poor correlations of metrics across criteria. We observed that across
all tasks and all criteria, most of the metrics have poor correlations. In
particular, out of the 271 correlation values reported in Figure 3, 228 are
poor ($<$0.3), 35 are moderate (between 0.3 and 0.5) and only 2 are high
($>$0.5). Surprisingly even some very recently proposed metrics such as DEB
Sai et al. (2020a), BLANC Vasilyev et al. (2020) and MaUde Sinha et al. (2020)
do not correlate well with human judgements on other criteria. This is despite
the fact that these are task-specific metrics which use the modern machinery
of pre-trained BERT-based models and are fine-tuned on human judgements for
overall quality. This vindicates our stand that simply tuning for overall
quality does not lead to good correlations with other criteria. We do observe
a few decently correlated metrics for some of the tasks along a few criteria.
Specifically, the moderate correlations are found (i) in D2T for BLEURT along
the dimensions of fluency, correctness and text structure, (ii) in IC for all
of the task-specific metrics, majority of the embedding-based metrics and for
METEOR, (iii) in QG for most metrics (except SMS) along fluency, for
BERTScore, Moverscore, SMS and BLEURT along answerability, as well as along
completeness, on which we find some of the highest correlations.

Pre-training and/or training often helps. While almost all the metrics have
poor correlations with different criteria across tasks, we observe that the
ones which use a pre-trained component such as static or contextualised
word/sentence embeddings and/or use task-specific training data perform
better. For example, BLEURT which uses a pre-trained BERT and is fine-tuned
using human judgements for MT, is among the top performing metrics across all
the tasks. These findings are also consistent with those reported in the WMT20
shared task on evaluation metrics for MT Mathur et al. (2020b).

Task-agnostic metrics versus task-specific metrics. For the tasks of QG, AS,
IC and D2T we find that task-agnostic metrics such as BERTScore, Moverscore
and BLEURT are consistently among the top performing metrics (i.e., their
correlation scores are either the best or close to the best scores for a given
task and criteria) along with the task-specific metrics. This is interesting
as these task-agnostic metrics were not fine-tuned on any task specific human
judgements and were originally proposed for a different task (MT). Among the
task-specific metrics, SUPERT for AS has a relatively better correlation with
consistency than all other metrics. Overall, there seems to be scope for more
work/improvements on task-specific metrics to capture the criteria peculiar to
each task.

###  5.2 Insights from fine-grained evaluation

We now complement the above analysis with a more fine-grained analysis using
perturbation templates. To do so, we use the perturbation templates described
in Section 3 and plot the deviation between metric scores and human scores
using the formula in Equation 1. These results are presented in Figure 4 and
summarised below.

Correlations do not reveal everything. In the previous section we observed
that BERT-based metrics such as BERTScore, BLEURT and MoverScore are among the
top performing metrics across tasks and criteria. However, our anslysis with
perturbation templates reveals that even these metrics are not robust to very
simple perturbations. For example, for the task of MT consider the
perturbations of adding negations, changing names, changing numeric values or
replacing by antonyms in the output which can significantly alter the meaning
of the sentence and thereby affect adequacy. However, BERTScore, BLEURT and
MoverScore are not able to perceive this drop in quality and have a
substantial deviation from human scores. We make similar observations across
tasks that such metrics are not able to detect these simple perturbations.

Task-specific nuances are not captured. Our analysis also shows that existing
metrics are not capable of addressing well known task specific grievances. For
example, for the task of DG, it is known that many NLG systems generate
generic responses which leads to poor engagement with the users. However, none
of the metrics are sensitive to perturbations producing ‘generic responses’
such as ok, thanks or back-off responses such as I‘m sorry, can you
repeat?444A model that frequently produces generic responses is undesirable.
Hence, the expert human annotators assigned a high penalty to this
perturbation. Most of the task-specific metrics do use the context as an input
whereas, the task-agnostic metrics do not. Similarly, for the task of QG, Nema
and Khapra (2018) show that the answerability of a question is affected if we
drop/replace question words or change named entities in the question. However,
we find that most metrics (including the task specific QBLEU4/QROUGE) are not
sensitive to such perturbations with very high deviation from human scores. On
similar lines, ViLBERTScore which is a state of the art evaluation metric for
IC is not sensitive to perturbations in gender, order of objects or attributes
used for describing objects. This is of concern as many IC systems are known
to produce generic captions containing genders, attributes and objects which
are most prevalent in the training data. Similarly, for the task of D2T, where
coverage and factual correctness are important we observe that most metrics
are unable to detect perturbations which add extra/random text to the output
or drop named entities (which often contain the most important information).
Lastly, for AS it is important that an evaluation metric should penalise
summaries which are not coherent or contain redundant sentences or do not have
referential clarity. However, we observe that most metrics are not sensitive
to perturbations which reorder the sentences or repeat sentences/phrases or
replace nouns with pronouns (affecting referential clarity).

Different metrics have different skills. While no single metric is capable of
detecting all types of perturbations, we observe that some metrics are more
robust to certain perturbations. BLEURT and Moverscore are robust to jumbling
of words, but BERTScore is not, revealing their differences in detecting
fluency. Moverscore, BERTScore and the embedding-based metrics like Greedy
Matching and Embedding Average are quite robust to simple transformations of
converting numbers to corresponding words, which is an important criteria for
the task of D2T, while BLEURT is relatively less robust to it. Similarly,
while BERTScore performs poorly for many perturbations, it is able to respect
alternative references, i.e., similar to humans, it does not drop its score
when presented with alternative correct references from the dataset (last row
in Figure 4(a) to 4(f)). An interesting observation from the IC task is that
SoTA metrics like SPICE and ViLBERTScore show a complementary behaviour on our
set of perturbation criteria (third to last and last column in Figure 4(f)).
This opens up interesting avenues for future research where different
automatic metrics could be combined to take advantage of their relative
strengths.

##  6 Related Work

Some of the related work, particularly the relevant datasets, human evaluation
criteria, and automatic metrics were already discussed earlier and hence not
covered again here. We refer the readers to two recent surveys Sai et al.
(2020b); Çelikyilmaz et al. (2020) for a detailed overview of automatic
evaluation metrics as well as related work on criticising the use of automatic
evaluation metrics. We mention a few such important works here. BLEU is one of
the most widely analysed metric with several studies showing that it does not
correlate well with human judgements for machine translation Callison-Burch et
al. (2006). This issue of poor correlations of metrics with human judgements
has been reported on not just BLEU, but also on various other metrics, across
several NLG tasks including Question Generation Nema and Khapra (2018), Data-
to-Text generation Dhingra et al. (2019), Dialogue generation Liu et al.
(2016), and Summarisation Kryscinski et al. (2019). Apart from poor
correlations, Kryscinski et al. (2019) criticize the automatic metrics for
abstractive summarization since they don’t check for factual inconsistencies
in the summaries. Similarly Wiseman et al. (2017) discuss the lack of a
reliable measurement of faithfulness in the context of Data-to-Text
Generation. In case of dialogue, several n-gram-based and embedding-based
metrics have been shown to fall short in capturing the diversity of the valid
responses Liu et al. (2016); Sai et al. (2020a). The alternative of trained
metrics, such as ADEM have been shown to be susceptible to adversarial attacks
Sai et al. (2019).

Similar to the main message of our work, some recent works have also called
for a more robust evaluation of automatic evaluation metrics Choshen and Abend
(2018); Mathur et al. (2020a). Ethayarajh and Jurafsky (2020) also critically
examine the current approaches towards NLP leaderboards and point towards
having multiple metrics along different dimensions such as fairness,
efficiency, robustness, etc.

##  7 Conclusion

We conduct a large-scale study involving 6 tasks, 25 automatic evaluation
metrics and 18 human evaluation criteria and observe that (i) different
criteria such as fluency, coverage, etc are often not correlated and (ii)
existing metrics have a low correlation with most criteria across different
tasks. Based on these observations, we suggest an alternative framework for
evaluating evaluation metrics which goes beyond computing correlations with
the human scores for overall quality. More specifically, we propose
perturbation templates which allow a more fine-grained evaluation of such
metrics and help in understanding their strengths and more importantly their
limitations. We hope that future work on designing evaluation metrics will use
our perturbation checklist for evaluating the effectiveness of the proposed
metric in assessing different relevant criteria.

## Acknowledgements

We thank the annotators for helping us evaluate and annotate the
perturbations. We thank the anonymous EMNLP-21 reviewers whose comments and
feedback helped enhance the paper. We thank the Google India Ph.D. Fellowship
Program and the Prime Minister’s Fellowship Scheme for Doctoral Research for
supporting Ananya Sai and Samsung IITM-Pravartak Undergraduate Fellowship for
supporting Dev Yashpal Sheth. Finally, we thank the Robert Bosch Centre for
Data Science and AI for supporting this work.

## References

  * Aditya et al. (2015) Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, and Yiannis Aloimonos. 2015.  From images to sentences through scene description graphs using commonsense reasoning and knowledge.  _arXiv preprint arXiv:1511.03292_.
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  SPICE: semantic propositional image caption evaluation.  In _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V_ , volume 9909 of _Lecture Notes in Computer Science_ , pages 382–398. Springer.
  * Banchs et al. (2015) Rafael E. Banchs, Luis F. D’Haro, and Haizhou Li. 2015.  Adequacy-fluency metrics: Evaluating MT in the continuous space model framework.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 23(3):472–482.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Callison-Burch et al. (2007) Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007.  (meta-) evaluation of machine translation.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 136–158, Prague, Czech Republic. Association for Computational Linguistics.
  * Callison-Burch et al. (2006) Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006.  Re-evaluating the role of Bleu in machine translation research.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , Trento, Italy. Association for Computational Linguistics.
  * Castro Ferreira et al. (2020) Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020.  The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020).  In _Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)_ , pages 55–76, Dublin, Ireland (Virtual). Association for Computational Linguistics.
  * Çelikyilmaz et al. (2020) Asli Çelikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020.  Evaluation of text generation: A survey.  _CoRR_ , abs/2006.14799.
  * Choshen and Abend (2018) Leshem Choshen and Omri Abend. 2018.  Automatic metric validation for grammatical error correction.  _CoRR_ , abs/1804.11225.
  * Chow et al. (2019) Julian Chow, Lucia Specia, and Pranava Madhyastha. 2019.  WMDO: Fluency-based word mover’s distance for machine translation evaluation.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 494–500, Florence, Italy. Association for Computational Linguistics.
  * Clark et al. (2019) Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019.  Sentence mover’s similarity: Automatic evaluation for multi-sentence texts.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 2748–2760. Association for Computational Linguistics.
  * Deriu et al. (2019) Jan Deriu, Álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2019.  Survey on evaluation methods for dialogue systems.  _CoRR_ , abs/1905.04071.
  * Dhingra et al. (2019) Bhuwan Dhingra, Manaal Faruqui, Ankur P. Parikh, Ming-Wei Chang, Dipanjan Das, and William W. Cohen. 2019.  Handling divergent reference texts when evaluating table-to-text generation.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 4884–4895. Association for Computational Linguistics.
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the eye of the user: A critique of NLP leaderboards.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 4846–4853. Association for Computational Linguistics.
  * Fabbri et al. (2020) Alexander R. Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2020.  Summeval: Re-evaluating summarization evaluation.  _CoRR_ , abs/2007.12626.
  * Forgues and Pineau (2014) Gabriel Forgues and Joelle Pineau. 2014.  Bootstrapping dialog systems with word embeddings.  In _NeurIPS, modern machine learning and natural language processing workshop_.
  * Freitag et al. (2021) Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _CoRR_ , abs/2104.14478.
  * Gao et al. (2020) Yang Gao, Wei Zhao, and Steffen Eger. 2020.  SUPERT: towards new frontiers in unsupervised evaluation metrics for multi-document summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 1347–1354. Association for Computational Linguistics.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021.  The GEM benchmark: Natural language generation, its evaluation and metrics.  _CoRR_ , abs/2102.01672.
  * Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018.  Achieving human parity on automatic chinese to english news translation.  _CoRR_ , abs/1803.05567.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Jiang et al. (2019) Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. 2019.  TIGEr: Text-to-image grounding for image caption evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2141–2152. Association for Computational Linguistics.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Landauer and Dumais (1997) Thomas K Landauer and Susan T Dumais. 1997.  A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.  _Psychological review_ , 104(2):211.
  * Lee et al. (2020) Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020.  ViLBERTScore: Evaluating image caption using vision-and-language BERT.  In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pages 34–39, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016.  How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2122–2132. The Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017.  Towards an automatic turing test: Learning to evaluate dialogue responses.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , pages 1116–1126. Association for Computational Linguistics.
  * Mathur et al. (2020a) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a.  Tangled up in BLEU: reevaluating the evaluation of automatic machine translation evaluation metrics.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4984–4997. Association for Computational Linguistics.
  * Mathur et al. (2020b) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. 2020b.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation, WMT@EMNLP 2020, Online, November 19-20, 2020_ , pages 688–725. Association for Computational Linguistics.
  * Nema and Khapra (2018) Preksha Nema and Mitesh M. Khapra. 2018.  Towards a better metric for evaluating question generation systems.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 3950–3959. Association for Computational Linguistics.
  * Ouyang et al. (2017) Jessica Ouyang, Serina Chang, and Kathy McKeown. 2017.  Crowd-sourced iterative annotation for narrative summarization corpora.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers_ , pages 46–51. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Popovic (2017) Maja Popovic. 2017.  chrf++: words helping character n-grams.  In _Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017_ , pages 612–618. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100, 000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2383–2392. The Association for Computational Linguistics.
  * Reiter (2018) Ehud Reiter. 2018.  A structured review of the validity of BLEU.  _Comput. Linguistics_ , 44(3).
  * Ribeiro et al. (2020) Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020\.  Beyond accuracy: Behavioral testing of NLP models with checklist.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4902–4912. Association for Computational Linguistics.
  * Rus and Lintean (2012) Vasile Rus and Mihai C. Lintean. 2012.  A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics.  In _BEA@NAACL-HLT_.
  * Sai et al. (2019) Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019\.  Re-evaluating ADEM: A deeper look at scoring dialogue responses.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 6220–6227. AAAI Press.
  * Sai et al. (2020a) Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020a.  Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining.  _Trans. Assoc. Comput. Linguistics_ , 8:810–827.
  * Sai et al. (2020b) Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2020b.  A survey of evaluation metrics used for NLG systems.  _CoRR_ , abs/2008.12009.
  * See et al. (2019) Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019.  What makes a good conversation? how controllable attributes affect human judgments.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 1702–1723. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020.  BLEURT: learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7881–7892. Association for Computational Linguistics.
  * Sharma et al. (2017) Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017.  Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation.  _CoRR_ , abs/1706.09799.
  * Sinha et al. (2020) Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, and Joelle Pineau. 2020.  Learning an unreferenced metric for online dialogue evaluation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 2430–2441. Association for Computational Linguistics.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A study of translation edit rate with targeted human annotation.  In _In Proceedings of Association for Machine Translation in the Americas_ , pages 223–231.
  * Tao et al. (2018) Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018.  RUBER: an unsupervised method for automatic evaluation of open-domain dialog systems.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 722–729. AAAI Press.
  * Vasilyev et al. (2020) Oleg V. Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020.  Fill in the BLANC: human-free quality estimation of document summaries.  _CoRR_ , abs/2002.09836.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_ , pages 4566–4575. IEEE Computer Society.
  * Wiseman et al. (2017) Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , pages 2253–2263. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 563–578. Association for Computational Linguistics.
  * Zhu and Bhat (2020) Wanzheng Zhu and Suma Bhat. 2020.  GRUEN for evaluating linguistic quality of generated text.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_ , pages 94–108. Association for Computational Linguistics.

##  Appendix A Criteria correlations

The pearson correlations among the criteria are presented in Figure 6. Most of
the correlation ranges are similar for pearson correlation and kendall tau
correlation, except for D2T task. We refer the studies on such correlations
Mathur et al. (2020a), discussing various points such as the influence of
outliers and noisy points on the correlations. Additionally, we observe that
the expertise of the annotators also influences the criteria-criteria
correlations. In particular, we were able to study this in case of AS using
the data released by Fabbri et al. (2020) containing both expert and
crowdsourced annotations. From Figure 5, we observe that the scores by expert
annotators have far lesser correlations amidst various criteria than the
crowdsourced annotations.

(a) Expert annotations

(b) Crowdsourced annotations

Figure 5: Expert v/s Crowsourced annotations

(a) Data to Text Generation criteria

(b) Machine Translation criteria

(c) Question Generation criteria

(d) Dialogue Generation criteria

(e) Abstractive Summarisation criteria

(f) Image Captioning criteria

Figure 6: Correlations between criteria for 6 different tasks (MT, DG, AS, QG,
D2T and IC)

##  Appendix B Detailed examples to illustrate the implementation of the
perturbation templates

Our perturbation templates mainly draw from the official github
repository555https://github.com/marcotcr/checklist of the checklist paper and
are also publicly available666https://github.com/iitmnlp/EvalEval. The
implementation involves preprocessing with the help of tokenization, POS
tagging, NER recognition, etc. Synonyms, antonyms, etc., are obtained with the
help of WordNet framework. Additionally, the masked language model of RoBERTa
is used to mask and predict replacements for the targeted words. For example,
the application of the template for ‘dropping stop words’ involves the
tokenization of the sentence using the NLTK word tokenizer as the first step.
The list of tokens is compared with the set of stopwords provided by NLTK to
filter out the stop words from the list of tokens. The modified sentence is
then reconstructed using the string join function by iterating over the tokens
in the modified list. Similarly, for the template of ‘changing the attributes’
in case of image captioning, the sentence is first tokenized, then the
adjectives are identified using part-of-speech tagging (again a functionality
provided by NLTK). The list of ‘related words’ (i.e., hyponyms of hypernyms or
‘sibling words’) are obtained using WordNet framework. Unless the list returns
empty, one of the entries in the list is used to replace the original
adjective. In order to ‘change question to an assertive statement’, the
question words (such as who, what, why, when, etc) are replaced with a ‘mask’
token and the ‘?’ character at the end is replaced with ‘.’ using string
replace function. This modified sentence is then fed to RoBERTa model which
generates different predictions to be used in place of the ‘mask’ token. One
of the suggested words is used to form the modified assertive sentence. In
case of perturbations involving dropping words, we additionally decide if
we’re dropping stop words, adjectives, question words, etc in the particular
perturbation and estimate the extent of effect it’ll have on each criteria.
The perturbations of adding text, appends random words / phrases / sentences
to a given text to account for not just the cases where there is missing
information, but also cases where there is spurious wrong information, even if
it accompanies / follows the correct version. The complete implementations /
details of our perturbation templates are hosted
publicly777https://github.com/iitmnlp/EvalEval.

Note that some of the perturbations cannot be applied to every sentence in the
dataset. For example, the template of “changing names” cannot be applied if
there are no named entities in a particular sentence. We hence shortlist only
the successfully modified samples from the dataset for analysing the metrics’
performance on each perturbation.

##  Appendix C Perturbation Templates for various criteria

Table 5 contains the comprehensive list of perturbation templates used in our
work.

Task |  Criteria |  Perturbation |  Unmodified sentence |  Perturbed sentence
---|---|---|---|---
|  |  Misplaced/missing punctuation |  Could you let me know if I can meet him now or later? |  Could you let me know, if I can meet him now or later.
|  |  Jumbling word order |  We play badminton every evening. |  We badminton every evening play.
|  |  Subject-verb disagreement |  He doesn’t know how to bake. |  He don’t know how to bake.
|  |  Dropping words (such as prepositions/articles, etc) |  The bank is willing to approve the loan. |  Bank willing to approve the loan.
All tasks |  Fluency |  Spelling errors |  Make the most of every opportunity presented to you. |  Make the most of evry opportunity presented to you.
|  |  Dropping out words or phrases |  I was being followed. |  I followed.
|  |  Adding extra wrong information |  This book is so inspiring. |  This book is so inspiring,I forgot.
|  |  Negation / antonyms |  It will rain on Monday. |  It will not rain on Monday.
MT |  Adequacy |  Repeat phrases |  My relatives are in town. |  My relatives are in town, my relatives.
|  |  Dropping words |  Here is the no parking sign. |  Here is the sign.
|  |  Negation and antonyms |  This book is so inspiring. |  This book is so uninspiring.
|  Informa-tiveness |  Use hyponyms to create misinformation |  The girl my brother Andy met through MySpace turned out to be completely made up . |  The girl my friend Andy met through MySpace turned out to be completely made up.
|  Flow / coherence |  Reorder sentences |  The pandemic was spreading uncontrollably. Vaccines are being developed and tested rapidly. |  Vaccines are being developed and tested rapidly. The pandemic was spreading uncontrollably.
|  Non-Redundancy |  Repeat sentences |  My relatives are in town. |  My relatives are in town. My relatives in town.
AS |  Referential clarity |  Replace nouns by pronouns |  The pandemic was spreading uncontrollably. Vaccines are being developed rapidly. |  It was spreading uncontrollably. They are being developed rapidly.
|  |  Change question word |  When was he born? |  What was he born?
|  |  Drop question word |  When was he born ? |  Was he born?
|  |  Change question to assertive statement |  Who is the director of Titanic? |  The director of Titanic is James Cameron.
|  Answera-bility |  Drop NLTK stop words |  Who was not a world leader |  Who world leader?
|  |  Mask a few words and predict |  How could Tesla run the experiments? |  How could Tesla run to the beach?
QG |  Relevance |  Perturb named entities, nouns, verbs |  Why did Mary go to the school? |  Why did Raj go to the market?
|  |  Negate a previous statement by same speaker |  Bot: I enjoy having your daughter in my class. User: I’m glad to hear that. |  I don’t enjoy having your daughter in my class.
|  Making sense |  Add extra non-sensible text |  Do you know where Dr. XYZ lives? |  Yes, my father is my grandmother’s father
|  |  Repeat previous utterances |  Do you know Dr.XYZ? |  Do you know where Dr. XYZ lives?
|  Avoid repetition |  Repeat phrases |  I like ice creams |  I like ice creams, ice creams
|  Listening |  Replace with "Can you repeat?" |  I need to book a taxi |  I’m sorry, can you repeat?
DG |  Relevance |  Map random responses |  I am new to coding. |  I am scared of snakes.
|  |  Change the order of objects |  The man is standing in front of a tree |  The tree is standing in front of man
|  |  Change gender |  Two girls are playing with a doll |  Two boys are playing with a doll
|  Correctness |  Change attributes |  A small boy playing with a red ball |  A tall boy playing with a green ball
|  |  Drop objects/noun |  A small boy playing with a red ball |  A small playing with a red
IC |  Thoroughness |  Repeat (append) object |  A lady riding a horse. |  A lady riding a horse and a lady.
|  |  Use hyponyms |  Beethoven was a German musician |  Beethoven was a German architect
|  |  Change numbers |  The cricketer was born in 1990. |  The cricketer was born in 1950.
|  Correctness |  Negation / antonyms |  The author of Harry Potter is J.K Rowling |  The author of Harry Potter is not J.K Rowling.
|  |  Drop phrases |  A small boy playing with a red ball |  A small boy playing with a
|  Data Coverage |  Repeat phrases |  Beethoven was a German musician |  Beethoven was a German musician and German musician.
|  |  Random text |  Beethoven was a German musician |  The cricketer was born in 1990.
D2T |  Relevance |  Perturb names |  Phillips was a child prodigy. |  James was a child prodigy.
|  |  Replace with synonyms |  The mangoes are delicious. |  The mangoes are tasty.
|  |  Contractions |  We are going to embark on an adventure. |  We’re going to embark on an adventure.
|  |  Expansions |  There weren’t any clear winners of the contest |  There were not any clear winners of the contest.
All tasks |  Invariance |  Numerals to words |  Aron Ralston who was trapped for 127 hours. |  Aron Ralston who was trapped for one hundred twenty seven hours.

Table 5: Perturbation templates targeting various criteria with examples. The
blue highlights indicate the portions of the original sentence affected by the
perturbation template. The red highlights indicate the changes in the modified
sentence.

##  Appendix D Correlations of Metrics with various Criteria (complete plots)

(a) Question Generation (QG)

(b) Abstractive Summarization (AS)

(c) Data to Text Generation (D2T)

(d) Image Captioning

(e) Dialogue Generation (DG) Figure 7: Correlations of metrics with different
criteria

Figure 7 is a more comprehensive version of Figure 3. It shows the
correlations of the complete set of metrics considered in this study with
various criteria across different tasks.