11institutetext: Evgenii Sovetkin and Bart E. Pieters 22institutetext:
IEK5-Photovoltaik, Forschungszentrum Jülich, 52425 Germany
22email<EMAIL_ADDRESS>
# Application of the Singular Spectrum Analysis on electroluminescence images
of thin-film photovoltaic modules

Evgenii Sovetkin and Bart E. Pieters

###### Abstract

This paper discusses an application of the singular spectrum analysis method
(SSA) in the context of electroluminescence (EL) images of thin-film
photovoltaic (PV) modules. We propose an EL image decomposition as a sum of
three components: global intensity, cell, and aperiodic components. A
parametric model of the extracted signal is used to perform several image
processing tasks. The cell component is used to identify interconnection lines
between PV cells at a sub-pixel accuracy, as well as to correct incorrect
stitching of EL images. Furthermore, an explicit expression of the cell
component signal is used to estimate the inverse characteristic length, a
physical parameter related to the resistances in a PV module.

##  1 Introduction

There has been an increasing interest in automated image analysis of spatially
resolved characterisation methods for photovoltaic (PV) modules such as
electroluminescence (EL) Demant et al. (2018); Deitsch et al. (2018, 2019); de
Oliveira et al. (2019); Karimi et al. (2019); Sovetkin and Steland (2019);
Sovetkin et al. (2021, 2020). Such automated image analysis aims at quality
control of modules and is thus of great interest for manufacturers, PV system
owners, and insurance companies, as it allows for a systematic inspection of a
large number of modules, both prior and after installation.

Electroluminescence is a commonly used imaging technique for PV modules. It
relies on the reciprocal operation of the photovoltaic module as a light-
emitting diode, so instead of generating an electric current from light, an
electric current is driven through the solar cell, which then emits light. As
generating electricity and emitting light are reciprocal processes, one
process reveals much about the other. Electroluminescence (EL) images provide
spatially resolved information on the solar module and is commonly used to
locate and identify defects in the device or extract other (local) solar cell
properties.

In this paper our focus lies on EL images of thin-film PV modules. For thin-
film technology, unlike for a more common crystalline silicon, little is known
about shapes and appearances of defects in EL images. For the crystalline
silicon PV modules there exists a well-established catalogue of defects
visible in EL images (see Köntges et al. (2015)), whereas such a catalogue
does not exist for thin-film modules.

To study defects in EL images, it is important to find a compact way to
represent EL image data. This paper proposes such an approach and considers
several image processing algorithms for EL images of thin-film PV modules that
are based on the singular spectrum analysis (SSA). Our contributions here are
manifold.

Firstly, a specific grouping in the SSA algorithm decomposes an EL image into
several components: global intensity variation component, local periodic
intensity component (or cell component), and a residual image that contains
various local aperiodic features. We argue that each of these components has a
different physical origin.

Secondly, the extracted components of an EL image can be approximated by a
parametric model, that represents an EL image as a small dimensional vector,
and hence our methods can also be considered as a dimensionality reduction
technique. Furthermore, the parametric model of the cell components is used to
estimate the position of the interconnection line between individual PV cells.
Our algorithm features symbolic differentiation and estimates positions of the
interconnection lines at a sub-pixel accuracy. A similar technique is used in
the estimation of the inverse characteristic length, a physical characteristic
of a PV module that equals the square of the ratio between different
resistances in a module.

Lastly, the cell component signal is used to estimate a non-linear
transformation of an image to adjust an incorrectly stitched image.

The rest of the paper is organised as follows. Section 2 reviews the methods
and the corresponding literature used in this paper. Section 3 describes the
data used in the project. The main contribution of this paper is given in
Section 4, which focuses on various applications of SSA to the EL images of
thin-film modules. Lastly, the paper is concluded in Section 5.

##  2 Methods overview

This section describes the methods used in this paper and overviews related
literature. Our main tool is the singular spectrum analysis method (SSA).

The history of SSA can be traced to the works of Broomhead and King (1986),
where an SSA-like method was established and applied in the context of
nonlinear dynamics for the purpose of reconstructing the attractordynamical
system analysis and SSA of a system from measured time series. Further, in the
context nonlinear dynamical system, SSA can be also used for phase space
reconstruction algorithm, Elsner and Tsonis (1996).

The so-called “Caterpillar” methodology is a parallel development of SSA that
originated in the former Soviet Union, especially in Saint Petersburg,
independently of the mainstream SSA work in the West, Danilov and Zhigljavsky
(1997). This methodology became known to the rest of the world more recently.
“Caterpillar-SSA” Golyandina et al. (2001), emphasises the concept of
separability, a concept that leads, for example, to specific recommendations
concerning the choice of the SSA parameters.

Originally, the SSA method was applied to the one-dimensional time-series
data. In fact, by now, it is not easy to find an applied area related to the
analysis of temporal data, where one-dimensional SSA is not being applied. To
name a few applications, the method found its way to the analysis of climate
and atmospheric data, Vautard and Ghil (1989); Fraedrich (1986), to
meteorological data, Weare and Nasstrom (1982), as well as to the marine
science, Colebrook (1982). Lima et al. (2016) used SSA for gap filling in
precipitation data. This method has been also applied in the financial sector
to discover hidden economic cycles, Sella et al. (2016). Groth and Ghil (2015)
used a multivariate extension of SSA and defined a Procrustes test to the
analysis of interannual variability in the North Atlantic sea surface
temperature. For further references to various applications of SSA in time-
series analysis see Zhigljavsky (2010); Golyandina et al. (2018).

More recently, SSA was also used to analyse digital images and other objects
that are not necessarily of planar or rectangular form. This particular
development is utilised in this paper. Rodriguez-Aragon and Zhigljavsky
(2010), used SSA to define a distance between images with a possible
application in face verification. In Zabalza et al. (2014, 2015); Qiao et al.
(2016) the 1D-/2D-SSA variants were used in the context of hyperspectral
images for the purpose of denoising, feature extraction, and classification
tasks. In Mamou and Feleppa (2007) SSA was applied in the context of
ultrasonic imaging for improving the imaging of brachytherapy seed. In
application related to geoscientific data, 2D-SSA was utilised for gap-
filling, Zscheischler et al. (2014). 2D-SSA was also applied in texture
classification Monadjemi (2004), seismology Trickett (2008), gene expression
Holloway et al. (2011), and medical imaging Shin et al. (2014).

In our application of the 2D-SSA, we utilise the ability of the method to
separate signal into trend and periodic components. Further, we use the
parametric form of the extracted signal to perform various image processing
tasks.

In order to explain our algorithms, we give a review of necessary theory in
the following subsections. The SSA algorithm is reviewed in Section 2.1. SSA
itself is non-parametric, however, a parametric model can be given to describe
the extracted signal. There are two sets of parameters to be estimated in the
model. The ESPRIT method (Section 2.2) estimates frequencies and damping
factors of a signal, where the least squares provides an estimation of the
amplitude and phases (Section 2.3). Lastly, remarks on implementation and
comparison to similar methods are discussed in Section 2.4.

###  2.1 SSA

Singular Spectrum Analysis (SSA) is a model-free time series analysis method
that belongs to the so-called subspace methods, Van Der Veen et al. (1993). In
subspace methods, a signal estimation is performed by taking a certain linear
subspace. SSA can also be considered as a low-rank approximation method,
Markovsky (2018).

The general theory of SSA for one-dimensional time series is elaborated in
Golyandina et al. (2001). Golyandina et al. (2018) provide more updated
information on extensions with a strong focus on applications.

The output of SSA-like methods is a decomposition of an observed signal $x$
(e.g. a time series, a multivariate time series or an image) into a sum of
identifiable components:

$x=x_{1}+\ldots+x_{n}.$ (1)

Among all SSA-like methods, the following four common algorithm steps can be
isolated.

  1. 1.

Embedding. The original signal $x$ (e.g. time series or an image) is mapped
into a matrix $\mathbf{X}$, that is called a trajectory matrix.

The embedding is parametrised by a single parameter denoted by $L$ (a number
or a vector).

  2. 2.

Decomposition. The second step consists of the decomposition of the trajectory
matrix $\mathbf{X}$ into a sum of matrices of rank 1.

Often the singular value decomposition (SVD) is used for this purpose, which
is an optimal rank-one matrix decomposition in the Frobenius norm sense.

  3. 3.

Grouping. The third step is a grouping of the decomposition components. At the
grouping step, the elementary rank-one matrices are grouped and summed within
groups.

The grouping algorithm step is often semi-automatic and depends on the type of
data and application.

  4. 4.

Reconstruction. The grouped components are not necessarily valid trajectory
matrices. Hence a projector operator to trajectory matrix space is applied on
the grouped components, resulting in the final signal decomposition (1).

In this paper, we utilise the 2D-SSA variant of the algorithm, Golyandina and
Usevich (2009, 2010); Golyandina et al. (2015). For this variant, the
trajectory matrix is a Hankel-block-Hankel matrix and the corresponding
embedding is parametrised by a two-dimensional parameter
$L=(L_{1},L_{2})\in\mathbb{N}^{2}$. The decomposition is performed with SVD,
and the reconstruction projection operator is a 2-step diagonal averaging
procedure. The precise forms of the embedding, SVD, and reconstruction
projection operator are provided in the Appendix.

The grouping step of the SSA algorithm determines the form of the final signal
decomposition. A typical SSA decomposition of a signal is the decomposition
into a slowly-varying trend, regular oscillations, and noise. An important
notion in the SSA theory is the notion of the signal of finite rank. This
notion allows us to classify and group together rank-one matrices into a
trend, oscillations, and noise components.

Informally, finite rank signals are those that have a trajectory matrix of a
finite rank (see a formal definition in the Appendix). 2D-SSA produces a class
of signals of specific objects of finite rank. Those objects have the form
described in the following theorem, Golyandina and Usevich (2009); Golyandina
et al. (2015).

###### Theorem 2.1

And infinite 2D-array $\\{x(n,m)\\}_{n,m\in\mathbb{N}}$ is of finite rank if
and only if

$x(n,m)=\sum_{k=1}^{K}p_{k}(n,m)\rho_{1k}^{m}\rho_{2k}^{n}\cos\big(2\pi(\omega_{1k}n+\omega_{2k}m)+\phi_{k}\big{missing}),\quad
n,m\in\mathbb{N},$ (2)

where $p_{k}(n,m)$ are polynomials in $n$ and $m$ variables, $\rho_{\cdot k}$
are the damping factors, $\omega_{\cdot k}$ the frequency parameters and
$\phi_{k}$ are the phase parameters.

###  2.2 ESPRIT

Estimation of Signal Parameters via Rotational Invariance Techniques (ESPRIT)
is a method to estimate parameters of a mixture of one-dimensional, Roy and
Kailath (1989), and two-dimensional amplitude-modulated sinusoids, Rouquette
and Najim (2001); Wang et al. (2005), in background noise.

For the 2D-ESPRIT method, Rouquette and Najim (2001), the observed data $y$ is
generated by the following additive model:

$y(m,n)=x(m,n)+\varepsilon(m,n),$ (3)

where $0\leq m\leq N_{x}-1$ and $0\leq n\leq N_{y}-1$, $\varepsilon$ is the
zero-mean Gaussian noise with variance $\sigma^{2}$. The model for the signal
$x$ is given by the sum of amplitude modulated two-dimensional sinusoid

$x(m,n)=\sum_{k=1}^{K}s_{k}\rho_{1k}^{m}\rho_{2k}^{n}\cos\big(2\pi(\omega_{1k}m+\omega_{2k}n)+\phi_{k}\big{missing}),$
(4)

where $\omega_{1k},\omega_{2k}$ are the normalised frequencies in different
directions, $\alpha_{1k},\alpha_{2k}$ are the damping factors, $s_{k}$
amplitudes, and $\phi_{k}$ phases.

By Theorem 2.1 the signal (4) is of finite rank. The ESPRIT methods utilises
the fact of the rank-deficiency of the trajectory matrix of the observed
signal $y$, and a certain transformation matrix between sub-trajectory
matrices is obtained. The frequency and damping factor parameters are computed
from the argument and absolute values of the complex-valued eigenvalues of the
obtained transformation matrix. See more details in Roy and Kailath (1989);
Rouquette and Najim (2001).

###  2.3 Amplitude, phase estimation

The ESPRIT estimates the frequencies $\omega_{\cdot k}$ and the damping
factors $\rho_{\cdot k}$ of the signal model (4). Here only the amplitudes
$s_{k}$ and phases $\phi_{k}$ remain to be estimated. This problem can be
reformulated as a linear regression model using the formula of the cosine of
sums. With this formula, (4) can be rewritten as

$\displaystyle
x(m,n)=\sum_{k=1}^{K}A_{k}\rho_{1k}^{m}\rho_{2k}^{n}\cos(2\pi(\omega_{1k}n+\omega_{2k}m))$
$\displaystyle-
B_{k}\rho_{1k}^{m}\rho_{2k}^{n}\sin(2\pi(\omega_{1k}n+\omega_{2k}m)),$ (5)

where $\rho_{\cdot k},\omega_{\cdot k}$ are the parameters estimated from the
ESPRIT, and $A_{k},B_{k}$ are the parameters of the linear model to be
estimated.

Note that the dependent variable in the linear regression model are the values
of $x(n,m)$, the finite-rank signal extracted from $y$. In terms of the SSA
algorithm this corresponds to the series reconstructed from the selected
components.

The amplitude and phase are given by:

$s_{k}=\sqrt{A_{k}^{2}+B_{k}^{2}},\quad\phi_{k}=\atan(B_{k}/A_{k}).$ (6)

###  2.4 Implementation. Comparison to alternatives methods

For our application, we use an R-package “Rssa”, Golyandina et al. (2018);
Korobeynikov (2010); Golyandina and Korobeynikov (2014); Golyandina et al.
(2015), where all the required functionality including the ESPRIT method is
implemented. It should be noted that the trajectory matrix for 2D-SSA is large
and has $O(N^{2})$ number of elements, where $N$ is a number of pixels in the
signal image. However, the structure of the Hankel-block-Hankel matrix allows
implementing of SVD with Lanczos algorithm efficiently in time and memory by
computing product of a matrix and a vector with Fast Fourier Transform,
Korobeynikov (2010); Lanczos (1950).

For computing amplitude and phase parameters, we utilise a standard R-function
“lm”, Chambers (1992).

The form of the parametric model of the SSA signal suggests that Fourier
analysis can be used to obtain similar results. However, in order to obtain a
compact representation of a signal a small Fourier coefficient should be
discarded. For that purpose a sparse Fourier analysis approach can be used,
Hassanieh et al. (2012).

However, the parametric model of the SSA signal is more flexible, as every
periodic has an amplitude modulation. Furthermore, the Fourier analysis is a
low-resolution type method, as in the context of time series, frequency can be
estimated only up to $1/N$, where $N$ is the length of the time series.
Whereas ESPRIT is regarded as a high-resolution method.

##  3 Data

In this section we discuss the data that is used in this paper. The data was
acquired within the framework of the PEARL-TF project. The PEARL-TF (2020)
website contains detailed information about the project and the involved
partners. In this project, the data from several solar parks with thin-film
modules were collected. In addition to EL images, also performance
characteristics of the modules were measured.

The EL images are taken at predefined conditions (selected fixed applied
current and/or fixed applied voltage). A silicon CCD sensor camera is used to
measure subsequently several parts of the module, with the images being
stitched afterwards. The applied voltage and the applied current together with
the temperature of the module are being recorded. The I/V characteristics are
also measured and the solar cell performance parameters are determined.

The database contains over 9500 EL images of thin-film PV modules. The bulk of
these EL images (about 6000) are from co-evaporated Copper Indium Gallium di-
Selenide (CIGS) modules with a chemical bath Cadmium Sulfide buffer. All EL
images shown in this work are from such CIGS modules. Every image is supplied
with measured performance data. A typical EL image of a thin-film module from
our database is depicted in Figure 1. The module consists of 150 connected
cells in series (in Figure 1 the cells are recognised as horizontal stripes).
The cells are separated by interconnection lines (horizontal dark lines in
Figure 1). In addition, the module is separated into 5 parallel sub-modules by
vertical isolation lines (dark vertical lines).

Figure 1: Thin-film module EL image. A module consists of 150 cells
(positioned horizontally) connected in series. The cells are separated by
interconnection lines (horizontal dark lines). The module consists of several
submodules separated by vertical isolation lines, which appear dark in the EL
image. The EL image is stitched (there are 1 horizontal and 3 vertical stitch
lines); overall intensities of different patches of images are different.
These intensity differences are attributed to metastable changes during the
measurement.

Every EL image consists of several stitched images. Different stitched parts
of the image have different overall intensities (see Figure 1). This is
attributed to the metastable behaviour of thin-film solar cells, where the
electrical properties of the cell can change during the measurement.

##  4 Results

In this section we discuss a selection of image processing methods that we
build on top of the SSA framework.

Firstly, Section 4.1 describes a decomposition of EL images into several
components: global intensity, cell, and aperiodic components. This is a direct
result of the SSA decomposition, where grouping is performed using prior
knowledge of the image size and number of cells in a module.

Secondly, a parametric model of the cell component signal is used to achieve
several goals. The symbolic differentiation allows a global search for the
minimum points in the cell components, that can be used to identify the
interconnection lines (Section 4.2). Furthermore, we demonstrate the
estimation of the inverse characteristic length, a physical parameter that
depends on several resistances in a PV module (Section 4.3).

Lastly, MSSA and its parametric model is used to obtain a non-linear
transformation needed to correct the stitch line in EL images.

###  4.1 Image decomposition

The grouping step of the SSA algorithm allows combining of decomposed rank-one
components into groups. We define these groups that result in EL image
decomposition onto 3 components: global intensity variation, cell, and
aperiodic components.

Figure 2 shows a close up image of the module, that clearly identifies a set
of 10 parallel cells separated by interconnection lines. The EL intensity
varies systematically over the cell width. This is the result of the series
resistance of the electrodes. As the transparent zinc-oxide front electrode
exhibits a much larger sheet resistance than the molybdenum back contact, the
voltage over the diode junction drops from one side to the other Helbig et al.
(2010); Pieters and Rau (2015). As the EL intensity depends primarily on the
cell voltage this leads to a clear intensity gradient over eachphysical reason
for periodic cell.

Figure 2: An enlargement of a 10-cells EL image region. Each cell exhibit
periodic behaviour.

To achieve good separability of the periodic components, the parameter $L$ of
the SSA embedding is selected to equal approximately half of the dimensions of
the input image, Golyandina et al. (2001, 2018). The ESPRIT estimates
frequencies and damping factors of the components, which are used in the
decision of the grouping step.

A thin-film module consists of a fixed number of interconnected cells,
therefore, the cell components can be identified as periodic components that
have a period smaller than $\frac{\text{image width}}{150}$. The other
components are grouped together as the global variation and the residual image
captures information of non-low rank signals, such as aperiodic features of an
EL image.

The image decomposition algorithm steps are described in the Algorithm 1.

Input: EL image $X$ with dimensions $N_{x}\times N_{y}$.

Output: Three images with the same dimensions as $X$: global intensity
component $G$, cell component $S$ and aperiodic component $R$.

  1. 1.

Perform embedding and decomposition steps of the 2D-SSA algorithm for $X$.
Compute first $50$ elements in the rank-one decomposition of SVD, denote the
sum of this components as $\tilde{\mathbf{X}}$.

  2. 2.

Apply ESPRIT on the low-rank signal $\tilde{\mathbf{X}}$.

  3. 3.

     * •

Set the cell components $S$ as in (4), choosing only periodics with frequency
$\omega_{1k}>\frac{150}{N_{x}}$.

     * •

The global intensity component $G$ is composed from components of non-
periodics and periodics with frequency $\omega_{1k}<\frac{150}{N_{x}}$.

     * •

The $R\coloneqq X-S-G$.

Algorithm 1 EL image decomposition

The choice of the 50 computed components is arbitrary, as 10 components
incorporate $99.9\%$ of the Frobenius norm of the trajectory matrix
$\mathbf{X}$. The RMSE of the linear regression model in the ESPRIT indicates
the accuracy of the model (4). For a set of 50 EL images, the mean RMSE equals
0.033.

Figure 3 depicts the non-cell components, the cell components, and the
residual image. It can be argued that different components have a different
physical origin. By its nature, the global variation component describes
changes in a material which results in large losses that spread over large
portions of a module. The cell component variation is influenced by the EL
measurement conditions. Lastly, the aperiodic component captures effects
caused by non-regular changes in material like shunts, or droplets (see
Sovetkin et al. (2021)).physical interpretation of components

Shunts are characterized by a more conductive connection between the front and
back electrodes than the normal solar cell structure (i.e. the solar cell
structure is damaged or missing). There are many causes for shunts. Commonly
shunts originate from debris of the copper evaporation source or pinholes in
the CIGS absorber Misic et al. (2015); Misic (2017). Shunts are generally
relevant to the solar module performance, in particular under low light
conditions Weber et al. (2011).

In addition to shunts we noticed the CIGS modules often exhibit “droplets” in
the EL images. The appearance of droplets resembles water stains and thus we
speculate these structures originate from the chemical bath deposition. At
this point it is unknown what the impact of droplets is on the module
performance, however, the bright appearance imply a local change in quantum
efficiency according to the reciprocity relations between luminescence and
quantum efficiency Rau (2007).

(a) Original image

(b) Cell component

(c) Global variations component

(d) Aperiodic component

Figure 3: Decomposition of an EL image onto 3 components

We remark that decomposition can be applied to an image (additive model
assumptionadditive and multiplicative models) as well as to the logarithm of
an image (multiplicative model assumption). The logarithm of an image
corresponds to the internal voltage (see (8), Section 4.3).

Furthermore, we remark that it is important to correct any perspective
distortion present in images, as a periodic image distorted in such a way is
no longer a signal with a finite rank in the settings of the 2D-SSA
methodology.

###  4.2 Interconnection line detection

In order to identify interconnection lines, it is sufficient to locate the
global minima for every level in the normal direction of the interconnection
lines.

The ESPRIT model satisfies equation (4), which is a sum of amplitude-modulated
cosine functions. A derivative of such function is again a sum of polynomial
amplitude-modulated cosine functions, similar to the general form of the
signal of finite rank (2). Such derivatives can be computed symbolically.
Hence all the global minima in the normal direction of the interconnection
lines can be identified by evaluating precise values of the derivatives, and
filtering out points that satisfy a minima extreme point requirement.

Algorithm 2 describes the steps of the interconnection line identification.
Symbolic derivatives are computed using the “Deriv” R-package, Clausen and
Sokol (2019).

Input:

  * •

EL image with dimensions $N_{x}\times N_{y}$.

  * •

Regular mesh of points $P=\\{n,m\in\mathbb{R},1\leq n\leq N_{x},1\leq m\leq
N_{y}\\}$.

Output: Array of coordinates $O\subset P$, corresponding to the estimated
locations of interconnection lines.

  1. 1.

Compute EL image decomposition with Algorithm 1.

  2. 2.

Use ESPRIT and linear regression to estimate parameters of the model (4), see
Sections 2.2 and 2.3. Denote the resulting signal as $S:P\to\mathbb{R}$.

  3. 3.

Compute symbolically expression for derivative $dS\coloneqq\frac{\partial
S}{\partial m}:P\to\mathbb{R}$.

  4. 4.

The output set $O\coloneqq\\{p\in P:dS(p-)<0\text{ and }dS(p+)>0\\}$, where
$p-,p+$ are the neighbours in $P$ of the point $p$ in the $m$-variable
direction.

Algorithm 2 Interconnection line detection

We remark that the resulting expression for the cell component signal, as well
as its derivatives, can be computed on a finer grid $P$ than the original
pixel coordinates. Hence the interconnection line identification is performed
with sub-pixel accuracy.

Figure 4 visually demonstrates the steps of the method. Figure 4a displays the
estimated cell component. Figure 4b shows a slice of a pixel value intensities
in Figure 4a, where computed local minima identified with red dots. Lastly,
Figure 4c shows the module with its interconnection lines identified by red
lines.

(a) Isolated cell components

(b) Pixel value intensity in the normal direction to the interconnection lines

(c) A cell with indicated interconnection lines (red)

Figure 4: Detection of the interconnection lines

###  4.3 Inverse characteristic length estimation

Another application of the obtained parametric expression for a module is an
estimation of the inverse characteristic length $\lambda$ Helbig et al.
(2010); Pieters and Rau (2015).

In a defect-free PV module, there are no currents in the vertical direction
(along the interconnection line direction), and thus current satisfies the
following linearised 1D Poisson equation, Helbig et al. (2010); Augarten et
al. (2016).

$\frac{\partial^{2}V}{\partial x^{2}}=\lambda^{2}V,$ (7)

where $\lambda=\sqrt{\frac{R_{\mathrm{sheet}}}{r_{\textrm{j}}}}$ is the
inverse characteristic length, where $R_{\mathrm{sheet}}$ is the sheet
resistance and $r_{\textrm{j}}$ is the local differential junction resistance.
Note, that (7) is an equation between two fields, where $V(x,y)$ and
$\lambda(x,y)$ depend on the position $(x,y)$ in a PV module.

The luminescence intensity relatedintensity and voltage to the internal
voltage via the following relation

$I=c\exp(Vc_{0})\implies V=\frac{1}{c_{0}}\ln(I/c),$ (8)

where the constant $c_{0}=\frac{q}{kT}$ is the thermal voltage (with the
elementary charge, $q$, Boltzmann’s constant, $k$, and the temperature, $T$),
$c$ is a parameter that describes the optical system from the photon
generation in the solar cell absorber material to the photon detection within
the camera. As such the parameter $c$ depends on the quantum efficiency of
both the solar cell and the used camera including all optical components, and
the spectral photon density of a black body Rau (2007). Generally, the
constant $c$ may vary over the solar cell area due to the camera optics and
variations in the solar cell properties. However, in general, the variations
in $c$ are small compared to the exponential voltage-dependency Abou-Ras et
al. (2016). In our analysis, we assume that $c$ is constant over the whole
module area, and is accessible to a researcher.

The decomposition of a signal onto trend and cell components provides us an EL
image signal without small aperiodic defects like shunts (small dark areas
within cells boundaries). The global intensity and cell components without
small aperiodic defects can be considered as a module without defects. Hence,
we consider a module without defects to be an image $G+S$, of the output of
Algorithm 1.

Hence combining (7) and (8), allows us to express inverse characteristic
length as a function of intensity and its derivative:

$\frac{\partial^{2}V}{\partial x^{2}}=\frac{\frac{\partial^{2}I}{\partial
x^{2}}I-\left(\frac{\partial I}{\partial x}\right)^{2}}{c_{0}I^{2},}$ (9)

and therefore,

$\lambda^{2}=\frac{\frac{\partial^{2}I}{\partial x^{2}}I-\left(\frac{\partial
I}{\partial x}\right)^{2}}{I^{2}\ln(I/c)}.$ (10)

All derivatives are computed symbolically from the estimated cell component
signal.

###  4.4 Stitched image correction

In order to achieve higher image resolution, several EL images can be stitched
together. However, the image aligning can be imperfect, as shown in Figure 5a.
These misaligned image patches and the resulting stitch line can be attributed
to incorrect perspective as well as radial distortions. The latter distortion
leads to a non-linear transformation that is needed to be applied to an image
for the stitch line correction.

Algorithm 3 proposes an approach to correct such distortion. The basic idea of
the algorithm is the estimation of phase shifts in neighbouring locations,
where each shift is estimated by application of MSSA, Golyandina et al.
(2018), in a direction perpendicular to the interconnection lines.

Input: EL image $X$ with dimensions $N_{x}\times N_{y}$.

Output: A displacement map $M$, a vector of dimension $N_{x}-1$ indicating
size of horizontal shifts in image $X$ (except for the first row).

  1. 1.

For each image row $i$ starting from the second row:

    1. (a)

Compute MSSA of the $i$ and $i-1$ row of the image $I$.

    2. (b)

Estimate the ESPRIT parameters.

    3. (c)

Filter out periods not corresponding to cell components. Let cell component
indices be $I$.

    4. (d)

Compute shift for cell each component
$s_{k}\coloneqq\frac{\phi_{k}}{2\pi\omega_{k}},k\in I$.

  2. 2.

Set $M_{i}\coloneqq\max_{k\in I}s_{k}$.

Algorithm 3 Stitched image correction

The result of the algorithm is a displacement map that defines shifts in the
horizontal direction. Note that shifts have sub-pixel accuracy.

Figure 5 shows the result of the algorithm. Figure 5a shows a patch of the
original EL image with a stitched part. Figure 5b depicts the shifted image,
resulting from applying the displacement map. As each displacement is
estimated locally between neighbours, the resulting transformation is a non-
linear transformation.

(a) Original stitched image

(b) Corrected image

Figure 5: An example of the stitched image correction

To evaluate the accuracy of the phase shift estimation we model the following
two time series

$\displaystyle s_{1}(x)=\cos(2\pi x/50)$ $\displaystyle+\cos(2\pi
x/20)+\cos(2\pi x/30)$ $\displaystyle+\varepsilon_{1}(x),$ (11) $\displaystyle
s_{2}(x)=2\cos(2\pi x/70)$
$\displaystyle+\underbrace{\cos(2\pi(x+7)/20)+\cos(2\pi(x+7)/30)}_{\text{signal}}$
$\displaystyle+\varepsilon_{2}(x),$ (12)

where $x\in\\{1,2,\ldots,1000\\}$, periodics with period $20$ and $30$
correspond to the signal (models the cell component), periodics with period
$50$ and $70$ slowly varying trend, and $\varepsilon_{1},\varepsilon_{2}$ are
two independent Gaussian iid processes with zero-mean and unit variance.

The signal part of the series $s_{1}$ is shifted by $7$ units relative to the
signal of the series $s_{2}$. Figure 6 shows a part of those series on the
interval $[1,100]$.

Figure 6: $s_{1}(x)$ and $s_{2}(x)$ time series, for $x\in[1,100]$. The signal
(periodics with periods 20 and 30) are “hidden” with a trend and a noise
components

Table 1 shows the accuracy of the shift estimation of the selected signal
components. That simulation was performed using 100 repetitions.

Table 1: Accuracy of the shift estimation between signals in time series $s_{1}$ and $s_{2}$ RMSE | Estimate mean | 25% quantile | 75% quantile
---|---|---|---
0.248 | 7.003 | 6.836 | 7.160

##  5 Conclusions

In this paper we demonstrated an application of SSA on EL images of thin-film
PV modules. This low-rank approach allows capturing several important aspects
of those images, namely, global and local repetitive variations (or cell
components) in an EL image.

Several image processing algorithms based on parametric models of SSA are
proposed. The first method identified the interconnection lines between the
individual cells at sub-pixel accuracy, and the second method corrects the
incorrectly stitched images.

Furthermore, we propose an approach based on symbolic differentiation of the
SSA signal to estimate the so-called inverse characteristic length, a physical
parameter of a module.

We note that in the settings of 2D-SSA it is important to correct perspective
distortion in such EL images. The two-dimensional cosine array transformed
with perspective distortion is no longer a finite-rank signal.

It should be noted that the information captured by SSA is not complete, as
local aperiodic features, such as shunts and droplets are not signals of
finite rank. Therefore, the full analysis of such PV modules requires other
methods. For example, we complement the signal captured by SSA using an
encoder-decoder segmentation approach of individual defects (see Sovetkin et
al. (2021, 2020)).

Lastly, we remark that source code and a sample of EL images data are
available upon request.

###### Acknowledgements.

This work is supported by the Solar-era.net framework in the project “PEARL
TF-PV” (Förderkennzeichen: 0324193A) and partly funded by the HGF project
“Living Lab Energy Campus (LLEC)”.

## Appendix

### 2D-SSA embedding and projection

Following notations of Golyandina and Usevich (2010), let $x$ be an 2D-array
with dimensions $(N_{x},N_{y})\in\mathbb{N}^{2}$:

$\footnotesize
x=\left(\begin{array}[]{cccc}x(0,0)&x(0,1)&\ldots&x(0,N_{y-1})\\\
x(1,0)&x(1,1)&\ldots&x(1,N_{y-1})\\\ \vdots&\vdots&\ddots&\vdots\\\
x(N_{x-1},0)&x(N_{x-1},1)&\ldots&x(N_{x-1},N_{y-1})\\\ \end{array}\right).$
(14)

The 2D-SSA embedding is defined by the window size vector $(L_{x},L_{y})$,
which is restricted by $1\leq L_{x}\leq N_{x},1\leq L_{y}\leq N_{y}$, and
$1<L_{x}L_{y}<N_{x}N_{y}$. Let $K_{x}\coloneqq N_{x}-L_{x}+1$ and
$K_{y}\coloneqq N_{y}-L_{y}+1$, then the trajectory matrix of 2D-SSA is given
by the following Hankel-block-Hankel matrix:

$\footnotesize\mathbf{X}\coloneqq\left(\begin{array}[]{ccccc}\mathbf{X}_{0}&\mathbf{X}_{1}&\mathbf{X}_{2}&\ldots&\mathbf{X}_{K_{y}-1}\\\
\mathbf{X}_{1}&\mathbf{X}_{2}&\mathbf{X}_{3}&\ldots&\mathbf{X}_{K_{y}}\\\
\mathbf{X}_{2}&\mathbf{X}_{3}&\ddots&\ddots&\vdots\\\
\vdots&\vdots&\ddots&\ddots&\vdots\\\
\mathbf{X}_{L_{y}-1}&\mathbf{X}_{L_{y}}&\ldots&\ldots&\mathbf{X}_{N_{y}-1}\\\
\end{array}\right),$ (15)

where each block

$\footnotesize\mathbf{X}_{i}\coloneqq\left(\begin{array}[]{cccc}x(0,i)&x(1,i)&\ldots&x(K_{x}-1,i)\\\
x(1,i)&x(2,i)&\ldots&x(K_{x},i)\\\ \vdots&\vdots&\ddots&\vdots\\\
x(L_{x}-1,i)&x(L_{x},i)&\ldots&x(N_{x}-1,i)\\\ \end{array}\right).$ (16)

By construction there is a one-to-one correspondence between 2D-arrays of size
$N_{x}\times N_{y}$ and the Hankel-block-Hankel matrices.

Let $\mathbf{Z}$ be an arbitrary matrix with a block-structure, where each
block $\mathbf{Z}_{i}$ has the same dimension as the matrix $\mathbf{X}_{i}$

$\footnotesize\mathbf{Z}=\left(\begin{array}[]{ccccc}\mathbf{Z}_{0}&\mathbf{Z}_{1}&\mathbf{Z}_{2}&\ldots&\mathbf{Z}_{K_{y}-1}\\\
\mathbf{Z}_{1}&\mathbf{Z}_{2}&\mathbf{Z}_{3}&\ldots&\mathbf{Z}_{K_{y}}\\\
\mathbf{Z}_{2}&\mathbf{Z}_{3}&\ddots&\ddots&\vdots\\\
\vdots&\vdots&\ddots&\ddots&\vdots\\\
\mathbf{Z}_{L_{y}-1}&\mathbf{Z}_{L_{y}}&\ldots&\ldots&\mathbf{Z}_{N_{y}-1}\\\
\end{array}\right).$ (17)

Then a projection of $\mathbf{Z}$ to Hankel-block-Hankel matrix can be
computed in two steps. Firstly, diagonal averaging is performed within blocks
$\mathbf{Z}_{i},i\in 0,\ldots,N_{y}-1$. Secondly, the blocks of the matrix $Z$
are averaged between themselves. The projection can be applied in the reverse
order as well.

### SVD

Let $\mathbf{S}=\mathbf{X}\mathbf{X}^{T}$,
$\lambda_{1}\geq\ldots\geq\lambda_{d}>0$ be non-zero eigenvalues of the matrix
$\mathbf{S}$, $U_{1},\ldots,U_{d}$ be the corresponding eigenvectors, and
$V_{i}\coloneqq\mathbf{X}^{T}U_{i}/\sqrt{\lambda_{i}},i=1,\ldots,d$.

Then SVD of the matrix $\mathbf{X}$ can be written as

$\mathbf{X}=\mathbf{X}_{1}+\ldots+\mathbf{X}_{d}.$ (18)

The values $\sqrt{\lambda_{i}}$ are the singular values of $\mathbf{X}$.

### Finite-rank signal

An image $x$ has rank $r$ if the rank of trajectory matrix $\mathbf{X}$ equals
$r<\min(L_{x},L_{y},K_{y},K_{y})$. In other words, the trajectory matrix is
rank-deficient. If rank $r$ does not depend on the choice of $L$ for any
sufficiently large dimensions of $x$, then $x$ is called to have a finite
rank.

Objects of finite rank are closely related to the linear recurrent sequences,
Kurakin et al. (1995). Linear recurrent formulae are used to build forecast
based on the SSA signal.

### Computational complexity

computational complexity

From the computational point of view, the hardest steps of the proposed
algorithms are the singular value decomposition (SVD), ESPRIT (Section 2.2),
and the phase and amplitude least-squares fit (Section 2.3).

In the context of the SSA application, the computational complexity of the SVD
of a Hankel-block-Hankel matrix is $O(kN\log N+k^{2}N)$, where $N$ is the
number of pixels in an image and $k$ is the number of computed eigentriples,
Korobeynikov (2010). The linear equations solved for ESPRIT and the phase and
amplitude parameters estimations are solved using QR-decomposition, and
require $O(N^{3})$ operations. Our numerical experiments show that the SVD
decomposition dominates the computational time for the proposed algorithms for
image sizes with width and height less than 4000 pixels.

To measure the required time and memory of the proposed algorithms we utilise
a machine with Intel(R) Xeon(R) CPU E5-1620 3.5GHz processor and 31GB of RAM.
The time measurements are performed on a single CPU.

Figure 7 shows the time required to perform steps of Algorithm 1 for square
images (width equals height) for different image widths. The red points
correspond to the measured time in seconds, and the black line is a parabola
fitted to the points. Table 2 shows the amount of memory required for the
steps of Algorithm 1.

Figure 7: Computation time required for Algorithm 1 for different input image sizes. Table 2: Maximum memory required for Algorithm 1 for different input image sizes. Image width and height | 500 | 1000 | 1500 | 2000
---|---|---|---|---
Maximum RAM in GB | 0.5 | 1.6 | 3.2 | 5.5

Algorithm 2 utilises results of the Algorithm 1 and requires additional
symbolical differentiation. Such computation is also required for the inverse
characteristic length (Section 4.3). The running time of the symbolical
differentiation depends on the number of terms of the signal (4). Table 3
demonstrates the relationship between the number of terms in the signal and
the running time required for the symbolic differentiation. Algorithm 2 and
the inverse characteristic length computation works with the cell component
that typically consists of 5–7 terms.

Table 3: Running time of the symbolical differentiation routine used in Algorithm 2 and the inverse characteristic length estimation in Section 4.3 Number of components | 5 | 9 | 13 | 17
---|---|---|---|---
Execution time in seconds | 0.15 | 0.46 | 1.27 | 1.79

Algorithm 3 utilises a different version of SSA, that requires less time and
memory, however, the algorithm computes MSSA multiple times for several pairs
of neighbour rows. A single iteration of the algorithm loop requires 0.7
seconds and 0.78 of RAM for an image with width of 4000 pixels. Different
iterations of the loop can be run in parallel. Note that most of the used
memory is occupied by an image itself, and hence the memory can be shared by
multiple processes. Algorithm 3 usually requires running about 100–200
iterations, as an approximate location of the stitching lines is known. Hence,
the total running time an 8-core processor can be as little as 20 seconds.

Lastly, we remark the proposed algorithms run in a deterministic amount of
time. Hence, the methods can be run in real-time applications.real time remark

## References

  * Abou-Ras et al. (2016) Abou-Ras D, Kirchartz T, Rau U (2016) Advanced characterization techniques for thin film solar cells. John Wiley & Sons
  * Augarten et al. (2016) Augarten Y, Wrigley A, Rau U, Pieters B (2016) Calculation of the TCO sheet resistance in thin film modules using electroluminescence imaging. In: 2016 IEEE 43rd Photovoltaic Specialists Conference (PVSC), IEEE, pp 1527–1531
  * Broomhead and King (1986) Broomhead D, King G (1986) Extracting qualitative dynamics from experimental data. Physica D: Nonlinear Phenomena 20(2–3):217–236
  * Chambers (1992) Chambers JM (1992) Linear models. Statistical models in S
  * Clausen and Sokol (2019) Clausen A, Sokol S (2019) Deriv: R-based Symbolic Differentiation. URL https://CRAN.R-project.org/package=Deriv, deriv package version 4.0
  * Colebrook (1982) Colebrook J (1982) Continuous plankton records-phytoplankton, zooplankton and environment, northeast atlantic and north-sea, 1958-1980. Oceanologica Acta 5(4):473–480
  * Danilov and Zhigljavsky (1997) Danilov D, Zhigljavsky A (1997) Principal components of time series: the “Caterpillar” method. St Petersburg: University of St Petersburg pp 1–307
  * Deitsch et al. (2018) Deitsch S, Buerhop-Lutz C, Maier A, Gallwitz F, Riess C (2018) Segmentation of photovoltaic module cells in electroluminescence images. arXiv preprint arXiv:180606530
  * Deitsch et al. (2019) Deitsch S, Christlein V, Berger S, Buerhop-Lutz C, Maier A, Gallwitz F, Riess C (2019) Automatic classification of defective photovoltaic module cells in electroluminescence images. Solar Energy 185:455–468
  * Demant et al. (2018) Demant M, Virtue P, Kovvali AS, Yu SX, Rein S (2018) Deep learning approach to inline quality rating and mapping of multi-crystalline si-wafers. In: 35th European Photovoltaic Solar Energy Conference and Exhibition, pp 814–818
  * Elsner and Tsonis (1996) Elsner JB, Tsonis AA (1996) Phase Space Reconstruction, Springer US, Boston, MA, pp 143–155. DOI 10.1007/978-1-4757-2514-8˙10, URL https://doi.org/10.1007/978-1-4757-2514-8_10
  * Fraedrich (1986) Fraedrich K (1986) Estimating the dimension of weather and climate attraction. Journal of Atmosphere Science 43:419–432
  * Golyandina and Korobeynikov (2014) Golyandina N, Korobeynikov A (2014) Basic singular spectrum analysis and forecasting with R. Computational Statistics and Data Analysis 71:934–954, r package version 1.0
  * Golyandina and Usevich (2009) Golyandina N, Usevich K (2009) An algebraic view on finite rank in 2d-ssa. In: Proceedings of the 6th St.Petersburg Workshop on Simulation, pp 308–313
  * Golyandina and Usevich (2010) Golyandina N, Usevich K (2010) 2D-extension of singular spectrum analysis: algorithm and elements of theory. In: Matrix Methods: Theory, Algorithms and Applications: Dedicated to the Memory of Gene Golub, World Scientific, pp 449–473
  * Golyandina et al. (2001) Golyandina N, Nekrutkin V, Zhigljavsky A (2001) Analysis of time series structure: SSA and related techniques. Chapman and Hall/CRC
  * Golyandina et al. (2015) Golyandina N, Korobeynikov A, Shlemov A, Usevich K (2015) Multivariate and 2D extensions of singular spectrum analysis with the Rssa package. Journal of Statistical Software 67(2):1–78, DOI 10.18637/jss.v067.i02
  * Golyandina et al. (2018) Golyandina N, Korobeynikov A, Zhigljavsky A (2018) Singular Spectrum Analysis with R. Springer
  * Groth and Ghil (2015) Groth A, Ghil M (2015) Monte carlo singular spectrum analysis (SSA) revisited: Detecting oscillator clusters in multivariate datasets. Journal of Climate 28(19):7873–7893
  * Hassanieh et al. (2012) Hassanieh H, Indyk P, Katabi D, Price E (2012) Simple and practical algorithm for sparse fourier transform. In: Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms, Society for Industrial and Applied Mathematics, pp 1183–1194
  * Helbig et al. (2010) Helbig A, Kirchartz T, Schaeffler R, Werner JH, Rau U (2010) Quantitative electroluminescence analysis of resistive losses in Cu (In, Ga) Se2 thin-film modules. Solar Energy Materials and Solar Cells 94(6):979–984
  * Holloway et al. (2011) Holloway DM, Lopes FJ, da Fontoura Costa L, Travençolo BA, Golyandina N, Usevich K, Spirov AV (2011) Gene expression noise in spatial patterning: hunchback promoter structure affects noise amplitude and distribution in drosophila segmentation. PLoS Comput Biol 7(2):e1001069
  * Karimi et al. (2019) Karimi AM, Fada JS, Hossain MA, Yang S, Peshek TJ, Braid JL, French RH (2019) Automated pipeline for photovoltaic module electroluminescence image processing and degradation feature classification. IEEE Journal of Photovoltaics 9(5):1324–1335
  * Köntges et al. (2015) Köntges M, Kurtz S, Packard C, Jahn U, Berger K, Kato K, Friesen T, Liu H, Van Iseghem M (2015) Iea-pvps t13-01 2014 review of failures of photovoltaic modules final. Tech. rep., Technical Report IEA-PVPS T13-01: 2014, IEA-PVPS Task 13
  * Korobeynikov (2010) Korobeynikov A (2010) Computation- and space-efficient implementation of SSA. Statistics and Its Interface 3(3):357–368, r package version 1.0
  * Kurakin et al. (1995) Kurakin V, Kuzmin A, Mikhalev A, Nechaev A (1995) Linear recurring sequences over rings and modules. Journal of Mathematical Sciences 76(6):2793–2915
  * Lanczos (1950) Lanczos C (1950) An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. United States Governm. Press Office Los Angeles, CA
  * Lima et al. (2016) Lima G, et al. (2016) Gap filling of precipitation data by SSA-singular spectrum analysis. In: Journal of Physics: Conference Series, IOP Publishing, vol 759, p 012085
  * Mamou and Feleppa (2007) Mamou J, Feleppa EJ (2007) Singular spectrum analysis applied to ultrasonic detection and imaging of brachytherapy seeds. The Journal of the Acoustical Society of America 121(3):1790–1801
  * Markovsky (2018) Markovsky I (2018) Low-Rank Approximation, vol 139. Springer
  * Misic (2017) Misic B (2017) Analysis and Simulation of Macroscopic Defects in Cu(In,Ga)Se2 Photovoltaic thin film modules, Schriften des Forschungszentrums Jülich. Reihe Energie und Umwelt / Energy and Environment, vol 372, Forschungszentrum Jülich GmbH, pp 17–36
  * Misic et al. (2015) Misic B, Pieters BE, Schweitzer U, Gerber A, Rau U (2015) Defect diagnostics of scribing failures and cu-rich debris in cu(in,ga)se2 thin-film solar modules with electroluminescence and thermography. IEEE Journal of Photovoltaics 5(4):1179–1187, DOI 10.1109/JPHOTOV.2015.2422143
  * Monadjemi (2004) Monadjemi A (2004) Towards efficient texture classification and abnormality detection. PhD thesis, University of Bristol
  * de Oliveira et al. (2019) de Oliveira AKV, Aghaei M, Rüther R (2019) Automatic fault detection of photovoltaic array by convolutional neural networks during aerial infrared thermography. In: 36th European Photovoltaic Solar Energy Conference and Exhibition
  * PEARL-TF (2020) PEARL-TF (2020) PEARL-TF project website. URL https://pearltf.eu/, accessed: 2020-09-01
  * Pieters and Rau (2015) Pieters BE, Rau U (2015) A new 2d model for the electrical potential in a cell stripe in thin-film solar modules including local defects. Progress in Photovoltaics: Research and Applications 23(3):331–339, DOI 10.1002/pip.2436, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/pip.2436, https://onlinelibrary.wiley.com/doi/pdf/10.1002/pip.2436
  * Qiao et al. (2016) Qiao T, Ren J, Wang Z, Zabalza J, Sun M, Zhao H, Li S, Benediktsson JA, Dai Q, Marshall S (2016) Effective denoising and classification of hyperspectral images using curvelet transform and singular spectrum analysis. IEEE transactions on geoscience and remote sensing 55(1):119–133
  * Rau (2007) Rau U (2007) Reciprocity relation between photovoltaic quantum efficiency and electroluminescent emission of solar cells. Phys Rev B 76:085303, DOI 10.1103/PhysRevB.76.085303, URL https://link.aps.org/doi/10.1103/PhysRevB.76.085303
  * Rodriguez-Aragon and Zhigljavsky (2010) Rodriguez-Aragon LJ, Zhigljavsky A (2010) Singular spectrum analysis for image processing. Statistics and Its Interface 3(3):419–426
  * Rouquette and Najim (2001) Rouquette S, Najim M (2001) Estimation of frequencies and damping factors by two-dimensional ESPRIT type methods. IEEE Transactions on signal processing 49(1):237–245
  * Roy and Kailath (1989) Roy R, Kailath T (1989) ESPRIT-estimation of signal parameters via rotational invariance techniques. IEEE Transactions on acoustics, speech, and signal processing 37(7):984–995
  * Sella et al. (2016) Sella L, Vivaldo G, Groth A, Ghil M (2016) Economic cycles and their synchronization: A comparison of cyclic modes in three european countries. Journal of Business Cycle Research 12(1):25–48
  * Shin et al. (2014) Shin PJ, Larson PE, Ohliger MA, Elad M, Pauly JM, Vigneron DB, Lustig M (2014) Calibrationless parallel imaging reconstruction based on structured low-rank matrix completion. Magnetic resonance in medicine 72(4):959–970
  * Sovetkin and Steland (2019) Sovetkin E, Steland A (2019) Automatic processing and solar cell detection in photovoltaic electroluminescence images. Integrated Computer-Aided Engineering (Preprint):1–15
  * Sovetkin et al. (2020) Sovetkin E, Pieters BE, Weber T, Achterberg EJ, Weeber A, Rau B, Rennhofer M, Theelen M (2020) PV-AIDED: Photovoltaic artificial intelligence defect identification. multichannel encoder-decoder ensemble models for electroluminescence images of thin-film photovoltaic modules, PEARL TF-PV. In: 37th EU PVSEC
  * Sovetkin et al. (2021) Sovetkin E, Weber T, Achterberg EJ, Pieters BE (2021) Encoder–decoder semantic segmentation models for electroluminescence images of thin-film photovoltaic modules. IEEE Journal of Photovoltaics 11(2):444–452, DOI 10.1109/JPHOTOV.2020.3041240
  * Trickett (2008) Trickett S (2008) F-xy cadzow noise suppression. In: SEG Technical Program Expanded Abstracts 2008, Society of Exploration Geophysicists, pp 2586–2590
  * Van Der Veen et al. (1993) Van Der Veen AJ, Deprettere EF, Swindlehurst AL (1993) Subspace-based signal analysis using singular value decomposition. Proceedings of the IEEE 81(9):1277–1308
  * Vautard and Ghil (1989) Vautard R, Ghil M (1989) Singular spectrum analysis in nonlinear dynamics, with applications to paleoclimatic time series. Physica D: Nonlinear Phenomena 35(3):395–424
  * Wang et al. (2005) Wang Y, Chen JW, Liu Z (2005) Comments on” estimation of frequencies and damping factors by two-dimensional ESPRIT type methods”. IEEE transactions on signal processing 53(8):3348–3349
  * Weare and Nasstrom (1982) Weare BC, Nasstrom JS (1982) Examples of extended empirical orthogonal function analyses. Monthly Weather Review 110(6):481–485
  * Weber et al. (2011) Weber T, Albert A, Ferretti N, Roericht M, Krauter S, Grunow P (2011) Electroluminescence investigation on thin film modules. In: 26th European Photovoltaic Solar Energy Conf.(26th EU PVSEC), pp 2584–2588
  * Zabalza et al. (2014) Zabalza J, Ren J, Wang Z, Marshall S, Wang J (2014) Singular spectrum analysis for effective feature extraction in hyperspectral imaging. IEEE Geoscience and Remote Sensing Letters 11(11):1886–1890
  * Zabalza et al. (2015) Zabalza J, Ren J, Zheng J, Han J, Zhao H, Li S, Marshall S (2015) Novel two-dimensional singular spectrum analysis for effective feature extraction and data classification in hyperspectral imaging. IEEE transactions on geoscience and remote sensing 53(8):4418–4433
  * Zhigljavsky (2010) Zhigljavsky A (2010) Singular spectrum analysis for time series: Introduction to this special issue. Statistics and its Interface 3(3):255–258
  * Zscheischler et al. (2014) Zscheischler J, Mahecha M, et al. (2014) An extended approach for spatiotemporal gapfilling: Dealing with large and systematic gaps in geoscientific datasets. Nonlinear Processes in Geophysics 21(1):203–215


#  Atomic: An Atlas of Machine Commonsense for _If-Then_ Reasoning

Maarten Sap†⋆ Ronan Le Bras† Emily Allaway⋆ Chandra Bhagavatula† Nicholas
Lourie†
Hannah Rashkin⋆ Brendan Roof† Noah A. Smith†⋆ Yejin Choi†⋆
⋆Paul G. Allen School of Computer Science & Engineering, University of
Washington, Seattle, USA
†Allen Institute for Artificial Intelligence, Seattle, USA
<EMAIL_ADDRESS>
###### Abstract

We present Atomic, an atlas of everyday commonsense reasoning, organized
through 877k textual descriptions of inferential knowledge. Compared to
existing resources that center around taxonomic knowledge, Atomic focuses on
inferential knowledge organized as typed if-then relations with variables
(e.g., “if X pays Y a compliment, then Y will likely return the compliment”).
We propose nine if-then relation types to distinguish causes vs. effects,
agents vs. themes, voluntary vs. involuntary events, and actions vs. mental
states. By generatively training on the rich inferential knowledge described
in Atomic, we show that neural models can acquire simple commonsense
capabilities and reason about previously unseen events. Experimental results
demonstrate that multitask models that incorporate the hierarchical structure
of if-then relation types lead to more accurate inference compared to models
trained in isolation, as measured by both automatic and human evaluation.

## Introduction

Given a snapshot observation of an event, people can easily anticipate and
reason about unobserved causes and effects in relation to the observed event:
what might have happened just before, what might happen next as a result, and
how different events are chained through causes and effects. For instance, if
we observe an event “X repels Y’s attack” (Figure 1), we can immediately infer
various plausible facts surrounding that event. In terms of the _plausible
motivations_ behind the event, X probably wants to protect herself. As for the
_plausible pre-conditions_ prior to the event, X may have been trained in
self-defense to successfully fend off Y’s attack. We can also infer the
_plausible characteristics_ of X; she might be strong, skilled, and brave. As
a result of the event, X probably feels angry and might want to file a police
report. Y, on the other hand, might feel scared of getting caught and want to
run away.

Figure 1: A tiny subset of Atomic, an atlas of machine commonsense for
everyday events, causes, and effects.  Figure 2:  The taxonomy of if-then
reasoning types. We consider nine _if-then_ relations that have overlapping
hierarchical structures as visualized above. One way to categorize the types
is based on the type of content being predicted: (1) If-Event-Then-Mental-
State, (2) If-Event-Then-Event, and (3) If-Event-Then-Persona. Another way is
to categorize the types based on their causal relations: (1) “causes”, (2)
“effects”, and (3) “stative”. Some of these categories can further divide
depending on whether the reasoning focuses on the “agent” (X) or the “theme”
(Other) of the event.

The examples above illustrate how day-to-day commonsense reasoning can be
operationalized through a densely connected collection of inferential
knowledge. It is through this knowledge that we can watch a two-hour movie and
understand a story that spans over several months, as we can reason about a
great number of events, causes, and effects, while observing only on a small
fraction of them. It also enables us to develop Theories of Mind about others
(?). However, this ability, while common and trivial for humans, is lacking in
today’s AI systems. This is in part because the vast majority of AI systems
are trained for task-specific datasets and objectives, which lead to models
that are effective at finding task-specific correlations but lack simple and
explainable commonsense reasoning (?; ?; ?).

In this paper, we introduce Atomic,111An ATlas Of MachIne Commonsense,
available to download or browse at
https://homes.cs.washington.edu/~msap/atomic/. an atlas of machine
commonsense, as a step toward addressing the rich spectrum of inferential
knowledge that is crucial for automated commonsense reasoning. In contrast
with previous efforts (?; ?) that predominantly contain taxonomic or
encyclopedic knowledge (?), Atomic focuses on inferential _if-then_ knowledge.
The goal of our study is to create a knowledge repository that meets three
requirements: scale, coverage, and quality. Therefore, we focus on
crowdsourcing experiments instead of extracting commonsense from corpora,
because the latter is subject to the significant reporting bias in language
that can challenge both the coverage and quality of the extracted knowledge
(?).

We propose a new taxonomy of _if-then_ reasoning types as shown in Figure 2.
One way to categorize the types is based on the content being predicted: (1)
If-Event-Then-Mental-State, (2) If-Event-Then-Event, and (3) If-Event-Then-
Persona. Another way to categorize is based on their causal relations: (1)
“causes”, (2) “effects”, and (3) “stative”. Using this taxonomy, we gather
over 877K instances of inferential knowledge.

We then investigate neural network models that can acquire simple commonsense
capabilities and reason about previously unseen events by embedding the rich
inferential knowledge described in Atomic. Experimental results demonstrate
that neural networks can abstract away commonsense inferential knowledge from
Atomic such that given a previously unseen event, they can anticipate the
likely causes and effects in rich natural language descriptions. In addition,
we find that multitask models that can incorporate the hierarchical structure
of if-then relation types lead to more accurate inference compared to models
trained in isolation.

##  If-Then Relation Types

Event | Type of relations | Inference examples | Inference dim.
---|---|---|---
“PersonX pays PersonY a compliment” | If-Event-Then-Mental-State |  | PersonX wanted to be nice
---
PersonX will feel good
PersonY will feel flattered
| xIntent
---
xReact
oReact
If-Event-Then-Event |  | PersonX will want to chat with PersonY
---
PersonY will smile
PersonY will compliment PersonX back
| xWant
---
oEffect
oWant
If-Event-Then-Persona |  | PersonX is flattering
---
PersonX is caring
| xAttr
---
xAttr
“PersonX makes PersonY’s coffee” | If-Event-Then-Mental-State |  | PersonX wanted to be helpful
---
PersonY will be appreciative
PersonY will be grateful
| xIntent
---
oReact
oReact
If-Event-Then-Event |  | PersonX needs to put the coffee in the filter
---
PersonX gets thanked
PersonX adds cream and sugar
| xNeed
---
xEffect
xWant
If-Event-Then-Persona |  | PersonX is helpful
---
PersonX is deferential
| xAttr
---
xAttr
“PersonX calls the police” | If-Event-Then-Mental-State |  | PersonX wants to report a crime
---
Others feel worried
| xIntent
---
oReact
If-Event-Then-Event |  | PersonX needs to dial 911
---
PersonX wants to explain everything to the police
PersonX starts to panic
Others want to dispatch some officers
| xNeed
---
xWant
xEffect
oWant
If-Event-Then-Persona |  | PersonX is lawful
---
PersonX is responsible
| xAttr
---
xAttr
Table 1: Examples of If-Event-Then-X commonsense knowledge present in Atomic.
For inference dimensions, “x” and “o” pertain to PersonX and others,
respectively (e.g., “xAttr”: attribute of PersonX, “oEffect”: effect on
others).

To enable better reasoning about events, we improve upon existing resources of
commonsense knowledge by adding nine new causal and inferential dimensions.
Shown in Figure 2, we define dimensions as denoting a particular type of If-
Then knowledge, answers to questions about an event, collected through
crowdsourcing. Contrary to most previous work, Atomic also characterizes
knowledge of events and their implied participants (e.g., “Alex calls for
help” implies someone will answer the call), in addition to explicitly
mentioned participants (e.g., “Alex calls Taylor for help”).

Illustrated in Table 1, our nine dimensions span three types of If-Then
relations, outlined below.

#### If-Event-Then-Mental-State

We define three relations relating to the mental pre- and post-conditions of
an event. Given an event (e.g., “X compliments Y”), we reason about (i) likely
intents of the event (e.g., “X wants to be nice”), (ii) likely (emotional)
reactions of the event’s subject (“X feels good”), and (iii) likely
(emotional) reactions of others (“Y feels flattered”).

#### If-Event-Then-Event

We also define five relations relating to events that constitute probable pre-
and post-conditions of a given event. Those relations describe events likely
required to precede an event, as well as those likely to follow. For instance,
people know that “X needs to put coffee in the filter” before “X makes Y’s
coffee”. For post-conditions, we focus on both voluntary (“X adds cream and
sugar”) and involuntary (“X gets thanked by Y”) possible next events. We also
define voluntary and involuntary possible next events for (implied)
participants.

#### If-Event-Then-Persona

In addition to pre- and post-conditions, we also define a stative relation
that describes how the subject of an event is described or perceived. For
instance, when “X calls the police”, X is seen as “lawful” or “responsible”.

#### An Alternative Hierarchy

The above relation types can be categorized via a different _hierarchical
structure_ as shown in Figure 2. In particular, they can be categorized based
on their causal relations: (1) “causes”, (2) “effects”, and (3) “stative”.
Each of these categories can be further divided depending on whether the
reasoning focuses on the “agent” or the “theme” of the event. We omit cases
where the combination is unlikely to lead to commonsense anticipation. For
example, it is usually only the “agent” who causes the event, rather than the
“theme”, thus we do not consider that branching. We later exploit this
hierarchical structure of inferential relations for designing effective neural
network architectures that can learn to reason about a given event.

## Data

Figure 3: Template of the crowdsourcing task for gathering commonsense
knowledge around events. Specific setups vary depending on the dimension
annotated.

To build Atomic, we create a crowdsourcing framework that allows for scalable,
broad collection of If-Then knowledge for given events.

### Compiling Base Events

As _base events_ for our annotations, we extract 24K common event phrases from
a variety of corpora. To ensure broad and diverse coverage, we compile common
phrases from stories, books, Google Ngrams, and Wiktionary idioms (?; ?; ?).
Following ? (?), we define events as verb phrases with a verb predicate and
its arguments (“drinks dark roast in the morning”). If a verb and its
arguments do not co-occur frequently enough,222We use frequency thresholds of
5 and 100 for stories and blogs, respectively, and limit ourselves to the top
10,000 events in Google Ngrams. we replace the arguments with a blank
placeholder (“drinks ___ in the morning”). In order to learn more general
representations of events, we replace tokens referring to people with a Person
variable (e.g. “PersonX buys PersonY coffee”). In future work, other types of
variables could be added for other entity references (e.g. “PersonX moves to
CityX”).

For events with multiple people explicitly involved, we run a short annotation
task to help resolve coreference chains within phrases. Disambiguating the
participants is important, since it can drastically change the meaning of the
event (e.g., “PersonX breaks PersonX’s arm” vs. “PersonX breaks PersonY’s arm”
have very different implications). Three workers selected whether each
“Person” mention in an event refers to PersonX, PersonY, or PersonZ, and we
keep base events with combinations that at least two workers selected as valid
(ppa=77%).

| Count | #words
---|---|---
# triples: If-Event-Then-* | 877,108 | -
\- Mental-State | 212,598 | -
\- Event | 521,334 | -
\- Persona | 143,176 | -
# nodes: If-Event-Then-* | 309,515 | 2.7
\- Mental-State | 51,928 | 2.1
\- Event | 245,905 | 3.3
\- Persona | 11,495 | 1.0
Base events | 24,313 | 4.6
# nodes appearing $>1$ | 47,356 | –
Table 2: Statistics of Atomic. Triples represent distinct $<$event, relation,
event$>$. #words represents the average number of words per node.

### Crowdsourcing Framework

To ensure scalability, we implement a free-form text annotation setup which
asks workers to write answers to questions about a specific event. We chose
free-text over structured or categorical annotation for two reasons. First,
categorical annotations with a large labeling space have a substantial
learning curve, which limits the annotation speed and thereby the coverage of
our knowledge graph. Second, the categorical labels are likely to limit the
ability to encode the vast space of commonsense knowledge and reasoning as
depicted in Figure 1 and Table 1.

We create four tasks on Amazon Mechanical Turk (MTurk) (sample task in Figure
3) for gathering commonsense annotations.333The tasks were used to collect the
following four sets of dimensions: (1) intent and reaction, (2) need and want,
(3) effects, and (4) attributes., 444Our payment rate was above $12/hour,
going well beyond the federal minimum rate of $8/hour. For each dimension, up
to three workers are asked to provide as many as four likely annotations for
an event, covering multiple possible situations (e.g., if “PersonX drinks
coffee”, then “PersonX needed to brew coffee” or “PersonX needed to buy
coffee”; both are distinct but likely). Note that some events are not caused
by PersonX, and some do not affect other people, making annotations for
certain dimensions not necessary (specifically, for xIntent, xNeed, oReact,
oEffect, and oWant) for all events. For those dimensions, we first ask workers
whether this specific inference dimension is relevant given an event.

Dataset | Model | xIntent | xNeed | xAttr | xEffect | xReact | xWant | oEffect | oReact | oWant
---|---|---|---|---|---|---|---|---|---|---
Dev | 9enc9dec | 8.35 | 17.68 | 5.18 | 10.64 | 5.38 | 13.24 | 6.49 | 5.17 | 12.08
NearestNeighbor | 6.14 | 11.36 | 3.57 | 5.81 | 4.37 | 7.73 | 8.02 | 6.38 | 8.94
Event2(In)voluntary | 7.51 | 17.80 | 5.18 | 10.51 | 4.78 | 12.76 | 7.04 | 4.84 | 12.48
Event2PersonX/Y | 7.31 | 17.08 | 5.26 | 9.78 | 4.83 | 12.14 | 6.38 | 4.84 | 11.45
Event2Pre/Post | 7.58 | 17.17 | – | 10.50 | 4.73 | 11.78 | 6.71 | 4.87 | 11.52
Test | 9enc9dec | 8.68 | 18.15 | 5.18 | 10.34 | 5.43 | 14.50 | 6.61 | 5.08 | 12.73
NearestNeighbor | 6.64 | 11.35 | 3.37 | 5.52 | 4.59 | 8.17 | 7.58 | 5.88 | 9.18
Event2(In)voluntary | 7.94 | 18.22 | 5.02 | 9.78 | 4.78 | 13.67 | 7.16 | 4.71 | 13.23
Event2PersonX/Y | 7.67 | 17.33 | 5.09 | 9.45 | 4.82 | 13.19 | 6.59 | 4.68 | 11.70
Event2Pre/Post | 7.96 | 17.42 | – | 9.79 | 4.75 | 12.85 | 6.90 | 4.76 | 11.97
Table 3: Average BLEU score (reported as percentages) for the top 10
generations for each inference dimension: comparison of multitask models to
single-task model. Note that BLEU scores are known to be brittle to
generations worded differently from the references (?). We embolden the best
performing model for each dimension.

###  Atomic Statistics

Table 2 lists descriptive statistics of our knowledge graph. Our resulting
knowledge graph contains over 300K nodes, collected using 24K base events.
Nodes in the graph are short phrases (2.7 tokens on average), ranging from 1
token for stative events (attributes) to 3.3 and 4.6 tokens on average for
more active events. Unlike denotational tasks where experts would only
consider one label as correct, our annotations correspond to a distribution
over likely inferences (?). To measure the degree of agreement, we run a small
task asking turkers to determine whether an individual annotation provided by
a different turker is valid. Table 4 shows that annotations are deemed valid
on average 86.2% of the time for a random subset of events. For quality
control, we manually and semi-automatically detected and filtered out
unreliable workers.

## Methods

Our goal is to investigate whether models can learn to perform If-Then
commonsense inference given a previously unseen event. To this extent, we
frame the problem as a conditional sequence generation problem: given an event
phrase $\mathbf{e}$ and an inference dimension $c$, the model generates the
target $\mathbf{t}=f_{\theta}(\mathbf{e},c)$. Specifically, we explore various
multitask encoder-decoder setups.

#### Encoder

We represent the event phrase as a sequence of $n$ word vectors
$\mathbf{e}=\\{e_{0},e_{1},\ldots,e_{n-1}\\}\in\mathbb{R}^{n\times
i_{\mathit{enc}}}$ where each word is an $i_{\mathit{enc}}$-dimensional
vector. The event sequence is compressed into a hidden representation
$\mathbf{h}$ through an encoding function
$f_{\mathit{enc}}:\mathbb{R}^{i\times
h_{\mathit{enc}}}\rightarrow\mathbb{R}^{h}$.

In this work, we use 300-dimensional static GloVe pre-trained embeddings (?)
as our base word vectors. We augment these embeddings with 1024-dimensional
ELMo pre-trained embeddings (?). ELMo provides deep contextualized
representation of words using character-based representations, which allows
robust representations of previously unseen events. The encoding function is a
bidirectional GRU (?) of hidden size $h_{\mathit{enc}}$.

Model | xNeed | xIntent | xAttr | xEffect | xReact | xWant | oEffect | oReact | oWant | average
---|---|---|---|---|---|---|---|---|---|---
9enc9dec | 48.74 | 51.70 | 52.20 | 47.52 | 63.57 | 51.56 | 22.92 | 32.92 | 35.50 | 45.32
Event2(In)voluntary | 49.82 | 61.32 | 52.58 | 46.76 | 71.22 | 52.44 | 26.46 | 36.04 | 34.70 | 47.93
Event2PersonX/Y | 54.04 | 53.93 | 52.98 | 48.86 | 66.42 | 54.04 | 24.72 | 33.80 | 35.08 | 46.41
Event2Pre/Post | 47.94 | 57.77 | 52.20 | 46.78 | 72.22 | 47.94 | 26.26 | 34.48 | 35.78 | 46.76
gold Atomic annotations | 81.98 | 91.37 | 78.44 | 83.92 | 95.18 | 90.90 | 84.62 | 86.13 | 83.12 | 86.18
Table 4: Precision at 10 (%) of generated inferences as selected by human
judges for four models, averaged and broken down by dimension. We embolden the
best performing model for each dimension. Event2(In)voluntary outperforms all
other models significantly ($p<$ 0.05). For comparison, we show precision of
gold Atomic annotations. Note that there is a varying number of gold
annotations per event/dimension, while all models were constrained to make 10
predictions.

#### Decoder

Each decoder is a unidirectional GRU of hidden size $h_{\mathit{dec}}$, with a
hidden state initialized to $\mathbf{h}_{\mathit{dec}}^{(0)}=\mathbf{h}$. The
target is represented by a sequence of vectors
$\mathbf{t}=\\{t_{0},t_{1},\ldots\\}$, where each
$t_{i}\in\mathbb{R}^{h}_{\mathit{dec}}$ is based on a learned embedding. The
decoder then maximizes
$p(t_{i+1}\mid\mathbf{h}_{\mathit{dec}}^{(i)},t_{0},\ldots,t_{i})=\text{softmax}(W_{o}\times\text{GRU}(\mathbf{h}_{dec}^{(i)},t_{i})+b_{o})$.

#### Single vs. Multitask Learning

We experiment with various ways to combine the commonsense dimensions with
multitask modeling. We design models that exploit the hierarchical structure
of the commonsense dimensions (depicted in Figure 2), sharing encoders for
dimensions that are related. Specifically, we explore the following models:

  * •

Event2(In)voluntary: We explore grouping dimensions together depending on
whether they denote voluntary (e.g., xIntent, oWant) or involuntary (e.g.,
xReact, oEffect) events. This model has one encoder for four “voluntary”
decoders, as well as another encoder for five “involuntary” decoders.

  * •

Event2PersonX/Y: We dissociate dimensions relating to the event’s agent
(PersonX) from those relating to the event’s theme (others, or PersonY). This
model has one encoder for six “agent” decoders as well as another encoder for
three “theme” decoders.

  * •

Event2Pre/Post: We split our dimensions based on whether they are related to
causes (xNeed, xIntent) or effects (e.g., xWant, oEffect, xReact). In this
model, there are two encoders and eight decoders.555We omit xAttr in this
model, as it is trivially covered in the single task baseline.

As a single task baseline, we train nine separate encoder-decoders, one for
each dimension (9enc9dec).

#### Training Details

To test our models,

we split seed events into training, validation, and test sets (80%/10%/10%),
ensuring that events that share the same first two content words are in the
same set.

As is common in generation tasks, we minimize the cross entropy of the
distribution over predicted targets compared to the gold distribution in our
data.666All our experiments were run using AllenNLP (?). During multitask
training, we average the cross entropy of each task. Since multiple
crowdworkers annotated each event, we define our training instances to be the
combination of one worker’s annotations. During experiments, we use the
300-dimensional GloVe embeddings, yielding an encoder input size of
$i_{\mathit{enc}}$ = 1324 once concatenated with the 1,024-dimensional ELMo
embeddings. In the encoder, ELMo’s character-level modeling allows for an
unlimited vocabulary. We set the encoder and decoder hidden sizes to
$h_{\mathit{enc}}=100$ and $h_{\mathit{dec}}=100$.

## Results

We evaluate models on their ability to reason about previously unseen events.
Given an unseen event, models generate natural language expressions for each
of the nine dimension of if-then inferences. We report performance using
automatic scores and a human evaluation of the generated inferences.

### Automatic Scores

We automatically evaluate the sequence generation for each model and each
inference dimension using BLEU scores. Specifically, we compute the average
BLEU score ($n=2$, Smoothing1; ?, ?) between each sequence in the top 10
predictions and the corresponding set of MTurk annotations. As an event may
not involve all nine inference dimensions (e.g., “PersonX sees PersonX’s
house” has no implications for anybody other than “PersonX”), annotators may
decide to leave an inference dimension _empty_. When computing BLEU scores, we
omit instances with one-third or more _empty_ annotations. Table 3 presents
the results on both Dev and Test datasets. The experiments show that models
that exploit the hierarchical structure of the commonsense relations perform
better than the model that uses separate parameters (9enc9dec). Importantly,
BLEU is a crude measure of performance as it is based on the exact match of
$n$-grams and fails to capture semantically relevant generations that are
worded differently (?). As shown in Figure 4, the generated samples depict
varying word and phrase choices, thus we also perform human evaluation to
complement automatic evaluations.

Figure 4: Examples of machine () generated inferences for three events from
the development set, ordered from most likely (top) to least likely (bottom)
according to the Event2(In)voluntary model. Human () generated inferences are
also shown for comparison.

### Human Evaluation

Since automatic evaluation of generated language is an open research question
(?), we also assess our models’ performance through human evaluation. We
randomly select 100 events from the test set and use beam search to generate
the 10 most likely inferences per dimension. We present five crowdworkers with
the 10 generated inferences, and ask them to select all inferences they think
are valid. Table 4 shows each model’s precision at 10, computed as the average
number of correct generations per dimension. Following the same crowdsourcing
setup, we also assess the quality of the gold Atomic annotations for the same
set of test events. Human evaluation (last line of Table 4) indicates that
86.2% of the descriptions are valid, showcasing the quality of commonsense
knowledge contained in Atomic.

Human evaluation supports our conclusion from automatic evaluation – that
models that leverage the if-then hierarchy perform better than models that
don’t. Specifically, explicitly modeling whether inference dimensions describe
voluntary actions (e.g., what X wants to do next) or involuntary effects
(e.g., X or Y’s reactions) yields more sensible generations, as evidenced by
the performance of Event2(In)voluntary.

### Qualitative Results

We present sample commonsense predictions in Figure 4. Given an event “PersonX
bakes bread”, our model can correctly infer that X probably _needs_ to “go to
the store” or “mix ingredients” or “turn on the oven”. Our model also
correctly predicts that the likely effect of this event would be that X will
“get dirty” or “eat food”.

### Comparison with ConceptNet

ConceptNet (?) represents commonsense knowledge as a graph of concepts
connected by relations. Concepts consist of words or phrases, while relations
come from a fixed set of edge types.

While ConceptNet captures general commonsense knowledge—much of which is
taxonomic in nature777While ConceptNet includes various inferential relations
(e.g., “entails”, “causes”, “motivated by”), their instances amount to only
about 1% of ConceptNet.—Atomic focuses on sequences of events and the social
commonsense relating to them. This focus means that while events and
dimensions in Atomic loosely correspond to concepts and relations from
ConceptNet, individual dimensions, such as intents, can’t be mapped cleanly
onto any combination of ConceptNet’s relations. The correspondence is neither
one-to-one nor one-to-many. Still, in order to empirically investigate the
differences between ConceptNet and Atomic, we used the following best-effort
mappings between the dimensions and relations:

  * •

Wants: MotivatedByGoal, HasSubevent, HasFirstSubevent, CausesDesire

  * •

Effects: Causes, HasSubevent, HasFirstSubevent, HasLastSubevent

  * •

Needs: MotivatedByGoal, Entails, HasPrerequisite

  * •

Intents: MotivatedByGoal, CausesDesire, HasSubevent, HasFirstSubevent

  * •

Reactions: Causes, HasLastSubevent, HasSubevent

  * •

Attributes: HasProperty

We then computed the overlap of <event1, dimension, event2> triples in Atomic
with the <concept1, relation, concept2> triples in ConceptNet. We found the
overlap to only be as high as 7% for wants, 6% for effects, 6% for needs, 5%
for intents, 2% for reactions, and 0% for attributes. Moreover, only 25% of
the events in Atomic are found in ConceptNet. Thus, Atomic offers a
substantial amount of new inferential knowledge that has not been captured by
existing resources.

## Related Work

#### Descriptive Knowledge from Crowdsourcing

Knowledge acquisition and representation have been extensively studied in
prior research (?; ?; ?). However, most prior efforts focused on taxonomic or
encyclopedic knowledge (?), which, in terms of epistemology, corresponds to
_knowledge of “what”_. Relatively less progress has been made on _knowledge of
“how”_ and _“why”_. For example, OpenCyc 4.0 is a large commonsense knowledge
base consisting of 239,000 concepts and 2,039,000 facts in LISP-style logic
(?), known to be mostly taxonomic (?). In fact, only 0.42% of Atomic events
appear in OpenCyc, which we found contains 99.8% relations that are either
taxonomic (`isA`), string formatting relations, or various definitional
relations. A typical example is shown below:

    (genls (LeftObjectOfPairFn
       SuperiorLobeOfLung) LeftObject)
    (isa (WordNetSynsetReifiedFn
       460174) WordNetSynset)
    (genls (AssociatesDegreeInFn
       EngineeringField) AssociatesDegree)

Importantly, these LISP-based representations of OpenCyc are non-trivial to
integrate into modern neural network based models, as it is not
straightforward to compute their embedding representations. In contrast, the
natural language representations in Atomic can be readily used to obtain their
neural embeddings, which can also be mixed with pretrained embeddings of words
or language models.

Similarly, ConceptNet (?) represents commonsense knowledge as a graph that
connects words and phrases (concepts) with labeled edges (relations). While
ConceptNet provides relatively more inferential relations (e.g., “entails”,
“causes”, “motivated by”), they still amount to only about 1% of all triples
in the graph. In contrast, Atomic is centered around events represented with
natural language descriptions. While events and dimensions in Atomic loosely
correspond to concepts and relations in ConceptNet, the two represent very
different information and ultimately have relatively small overlap as
discussed in the Results section.

Recent work by ? (?) compiles a list of nearly 1,400 commonsense axioms in
formal logic, which connect abstract concepts to each other. For example, they
define an `event` as being made up of `subevents`, expressed by:

    (forall (e)
     (iff (event e)
      (or (exists (e1 e2)
       (and (nequal e1 e2)(change’ e e1 e2)))
        (exists (e1)
         (subevent e1 e)))))

These axioms are abstract in that they are not grounded with respect to
specific objects, events, or actions. In contrast, our work presents 880K
triples of commonsense knowledge expressed in natural language and fully
grounded with concrete events, actions, mental states.

The recent work of ? (?) introduced a commonsense inference task about events
and mental states: given an event described in natural language, the task is
to generate the reaction and intent of actors involved in the event. Atomic is
inspired by this work, but substantially scales up (i) the crowdsourcing
procedure to nine dimensions per event, and (ii) the size of the knowledge
graph—from 77K events in Event2Mind to 300K events in Atomic. Moreover, while
the primary focus of (?) was inferential knowledge, its scope was limited to
mental states.

#### Acquired Knowledge from Extraction and Induction

More generally, the goal of moving beyond static commonsense knowledge to
enable automated commonsense reasoning has inspired much research. Several
projects have sought to extract commonsense inferential rules from naturally
occurring resources such as large corpora (?), movie scripts (?), and web how-
tos (?). Such systems must inevitably deal with reporting bias (?), or the
fact that the frequency and selection of phenomena represented in natural
language systematically differ from what occurs in the real world. Other
approaches have sought to induce commonsense rules from large knowledge bases
(?; ?). While these approaches have also had success, the choice of schema and
information represented in current knowledge bases limits the scope of
propositions such systems can learn.

#### Scripts and Narrative Reasoning

Other work has focused more specifically on representing and reasoning about
sequences of events, similarly to Atomic. Early work on event sequences
studied scripts, a kind of structured representation for prototypical
sequences of events (?). More recently, narrative event chains have been
proposed as a similar formalism for prototypical sequences of events that may
be learned from raw text (?). This work additionally proposed the Narrative
Cloze Test as a benchmark for story understanding. In contrast to narrative
event chains, the ROC Stories Corpus crowdsources event sequences represented
as natural language stories rather than using a specific formalism (?).
Additionally, the Story Cloze Test adapts these stories into a new benchmark
by requiring systems to choose between the true and a false ending to the
story. Our work interpolates between these two approaches by representing
events in natural language while structuring the relationships between events
into the edges of a graph. The Choice of Plausible Alternatives (COPA) task
offers a similar benchmark for commonsense understanding of events and their
relationships (?). In COPA, a system is presented a premise and two
alternatives that might have a causal relationship with the premise. While
COPA, like Atomic, represents events as free-form text with structured
relationships, it covers only a limited number of relations (cause and effect)
and is smaller in scale (contains only 1,000 instances).

## Conclusion

We present Atomic, an atlas of everyday commonsense inferential knowledge
about events described in natural language and associated with typed if-then
relations. Atomic consists of over 300k events associated with 877k
inferential relations, making it the largest knowledge graph of its kind. Our
crowdsourcing framework gathers annotations in the form of free-form textual
responses to simple questions which enables large-scale high quality
collection of commonsense about events. We also present neural network models
that can learn to reason about previously unseen events to generate their
likely causes and effects in natural language.

## Acknowledgments

We thank the anonymous reviewers for their many insightful comments. We also
thank Peter Clark, Dan Weld, Keisuke Sakaguchi, Vidur Joshi, Mark Neumann,
xlab, Mosaic and AllenNLP team members, for their helpful comments and
suggestions. Experiments were conducted on the AllenAI Beaker platform. This
work was supported in part by NSF GRFP DGE-1256082, NSF IIS-1714566,
IIS-1524371, IIS-1703166, Samsung AI Grant, DARPA CwC program through ARO
(W911NF-15-1-0543), and the IARPA DIVA program through D17PC00343.

## References

  * [Chambers and Jurafsky 2008] Chambers, N., and Jurafsky, D.  2008\.  Unsupervised learning of narrative event chains.  In ACL.
  * [Chen and Cherry 2014] Chen, B., and Cherry, C.  2014\.  A systematic comparison of smoothing techniques for sentence-level bleu.  In Proceedings of the Ninth Workshop on Statistical Machine Translation, 362–367.
  * [Cho et al. 2014] Cho, K.; van Merrienboer, B.; Bahdanau, D.; and Bengio, Y.  2014\.  On the properties of neural machine translation: Encoder-decoder approaches.  In SSST@EMNLP.
  * [Chu, Tandon, and Weikum 2017] Chu, C. X.; Tandon, N.; and Weikum, G.  2017\.  Distilling task knowledge from how-to communities.  In WWW.
  * [Davis and Marcus 2015] Davis, E., and Marcus, G.  2015\.  Commonsense reasoning and commonsense knowledge in artificial intelligence.  Commun. ACM 58:92–103.
  * [de Marneffe, Manning, and Potts 2012] de Marneffe, M.-C.; Manning, C. D.; and Potts, C.  2012\.  Did it happen? the pragmatic complexity of veridicality assessment.  Comput. Linguist. 38(2):301–333.
  * [Espinosa and Lieberman 2005] Espinosa, J. H., and Lieberman, H.  2005\.  Eventnet: Inferring temporal relations between commonsense events.  In MICAI.
  * [Galárraga et al. 2013] Galárraga, L.; Teflioudi, C.; Hose, K.; and Suchanek, F. M.  2013\.  Amie: association rule mining under incomplete evidence in ontological knowledge bases.  In WWW.
  * [Gardner et al. 2017] Gardner, M.; Grus, J.; Neumann, M.; Tafjord, O.; Dasigi, P.; Liu, N. F.; Peters, M.; Schmitz, M.; and Zettlemoyer, L. S.  2017\.  Allennlp: A deep semantic natural language processing platform.
  * [Goldberg and Orwant 2013] Goldberg, Y., and Orwant, J.  2013\.  A dataset of syntactic-ngrams over time from a very large corpus of english books.  In SEM2013.
  * [Gordon and Hobbs 2017] Gordon, A. S., and Hobbs, J. R.  2017\.  A Formal Theory of Commonsense Psychology: How People Think People Think.  Cambridge University Press.
  * [Gordon and Swanson 2008] Gordon, A. S., and Swanson, R.  2008\.  StoryUpgrade: finding stories in internet weblogs.  In ICWSM.
  * [Gordon and Van Durme 2013] Gordon, J., and Van Durme, B.  2013\.  Reporting bias and knowledge acquisition.  In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC ’13, 25–30.  New York, NY, USA: ACM.
  * [Lake et al. 2017] Lake, B. M.; Ullman, T. D.; Tenenbaum, J. B.; and Gershman, S. J.  2017\.  Building machines that learn and think like people.  The Behavioral and brain sciences 40:e253.
  * [Lenat 1995] Lenat, D. B.  1995\.  Cyc: A large-scale investment in knowledge infrastructure.  Communications of the ACM 38(11):33–38.
  * [Liu et al. 2016] Liu, C.-W.; Lowe, R.; Serban, I. V.; Noseworthy, M.; Charlin, L.; and Pineau, J.  2016\.  How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In EMNLP.
  * [Marcus 2018] Marcus, G.  2018\.  Deep learning: A critical appraisal.  CoRR abs/1801.00631.
  * [Moore 2013] Moore, C.  2013\.  The development of commonsense psychology.  Psychology Press.
  * [Mostafazadeh et al. 2016] Mostafazadeh, N.; Chambers, N.; He, X.; Parikh, D.; Batra, D.; Vanderwende, L.; Kohli, P.; and Allen, J.  2016\.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In NAACL.
  * [Pennington, Socher, and Manning 2014] Pennington, J.; Socher, R.; and Manning, C. D.  2014\.  Glove: Global vectors for word representation.  In EMNLP.
  * [Peters et al. 2018] Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L.  2018\.  Deep contextualized word representations.  In Proc. of NAACL.
  * [Rashkin et al. 2018] Rashkin, H.; Sap, M.; Allaway, E.; Smith, N. A.; and Choi, Y.  2018\.  Event2mind: Commonsense inference on events, intents, and reactions.  In ACL.
  * [Roemmele, Bejan, and Gordon 2011] Roemmele, M.; Bejan, C. A.; and Gordon, A. S.  2011\.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.
  * [Schank and Abelson 1977] Schank, R., and Abelson, R.  1977\.  Scripts, Plans, Goals, and Understanding: An Inquiry Into Human Knowledge Structures.  The Artificial Intelligence Series. Lawrence Erlbaum Associates.
  * [Schubert 2002] Schubert, L.  2002\.  Can we derive general world knowledge from texts?  In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, 94–97.  San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.
  * [Speer and Havasi 2012] Speer, R., and Havasi, C.  2012\.  Representing general relational knowledge in conceptnet 5.  In LREC.
  * [Speer, Chin, and Havasi 2017] Speer, R.; Chin, J.; and Havasi, C.  2017\.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In AAAI, 4444–4451.
  * [Tandon, de Melo, and Weikum 2017] Tandon, N.; de Melo, G.; and Weikum, G.  2017\.  Webchild 2.0 : Fine-grained commonsense knowledge distillation.  In ACL.
  * [Yang et al. 2015] Yang, B.; Yih, S. W.-t.; He, X.; Gao, J.; and Deng, L.  2015\.  Embedding entities and relations for learning and inference in knowledge bases.  In ICLR.