# Half-plane diffraction problems on a triangular lattice

D. Kapanadze<EMAIL_ADDRESS>E. Pesetskaya
<EMAIL_ADDRESS>A. Razmadze Mathematical Institute, TSU, Merab
Aleksidze II Lane 2, Tbilisi 0193, Georgia  Free University of Tbilisi,
Tbilisi 0159, Georgia

###### Abstract

We investigate thin-slit diffraction problems for two-dimensional lattice
waves. The peculiar structure allows us to consider the problems on the semi-
infinite triangular lattice, consequently, we study Dirichlet problems for the
two-dimen-sional discrete Helmholtz equation in a half-plane. In view of the
existence and uniqueness of the solution, we provide new results for the real
wave number $k\in(0,3)\backslash\\{2\sqrt{2}\\}$ without passing to the
complex wave number and derive an exact representation formula for the
solution. For this purpose, we use the notion of the radiating solution.
Finally, we propose a method for numerical calculation. The efficiency of our
approach is demonstrated in an example related to the propagation of wave
fronts in metamaterials through two small openings.

###### keywords:

discrete Helmholtz equation, Dirichlet boundary value problem, half-plane
diffraction, metamaterials, triangular lattice model

††journal: arXiv.org

##  1 Introduction

Wave propagation through discrete structures remains an active area of
research today. The triangular lattice is one of the five two-dimensional
Bravais lattice types and appears naturally in applications. For example,
close-packed planes occur frequently in some kinds of crystals in the form of
triangular lattice [2, 3]. Besides, one can consider two-dimensional passive
propagation media, a host microstrip line network periodically loaded with
series capacitors and shunt inductors for signal processing and filtering (as
shown in Figure 1). This type of inductor-capacitor lattice is referred to a
negative-refractive-index transmission-line (NRI-TL) metamaterial [6] or,
simply, left-handed 2D metamaterial.

Motivated by applications of recent interest related to analog circuits,
crystalline materials, and metamaterials, we investigate thin-slit diffraction
problems for triangular lattices. The peculiar structure allows us to consider
the problems on the semi-infinite triangular lattice. For this lattice, we
study Dirichlet problems for the two-dimensional discrete Helmholtz equation
in a half-plane (mathematically formulated in the next section). When one is
interested in the analysis of regular processes in which waves corresponding
to the microstructural scales can be neglected, then the continuum limit of
corresponding equations can be investigated. In this case, we arrive at the
famous problem in applied mathematics – the half-plane diffraction problem. It
is well known that the classical continuum model of wave diffraction can be
considered only as the slowly-varying approximation of a discrete or
structured material. Nowadays the understanding of nanostructure and
microstructure phenomena in modern materials and composites in critical
conditions is of crucial importance. It turns out that a discrete structure
provides an effective way to describe microstructural processes and influences
within, cf., e.g., [5, 6, 13, 17]. Therefore, there is an obvious necessity to
avoid continuum limits and instead directly analyze the discrete Helmholtz
diffraction problems. Although the similar problem for square lattice has been
studied in [1], see also [15], its extension to a triangular lattice model is
not direct.

Our main interest lies in the investigation of the problems with wave number
$k$ within the pass-band which is an arbitrary non-zero complex number in
general. In this paper, we use the well approved approach to provide new
results for a real wave number $k\in(0,3)\backslash\\{2\sqrt{2}\\}$ without
passing to a complex wave number. In order to have the unique solvability
results and an exact representation formula for the solution, we use the
notion of the radiating solution [14] and the method of images to construct
the Dirichlet Green’s function for the half-plane. Notice that the cases with
the wave numbers within the stop-band is mathematically more simple, they do
not need the radiation conditions and is not considered here. Finally, we
propose a method for numerical calculation. The efficiency of our approach is
demonstrated in an example related to the propagation of wave fronts in
metamaterials through two small openings.

##  2 Basic notations and formulation of the problem

Following the customary notation in mathematics, let $\mathbb{Z}$,
$\mathbb{Z}^{+}$, $\mathbb{Z}^{-}$, $\mathbb{N}$, $\mathbb{R}$, and
$\mathbb{C}$ denote the set of integers, positive integers, negative integers,
non-negative integers, real numbers and complex numbers, respectively. We
denote by $e_{1}=(1,0)$, $e_{2}=(0,1)$ the standard base of $\mathbb{Z}^{2}$
($=\mathbb{Z}\times\mathbb{Z}$).

Consider a two-dimensional infinite triangular lattice $\mathfrak{T}$ defined
as a periodic simple graph $\\{\mathcal{V},\mathcal{E}\\}$, where

$\mathcal{V}=\\{(x_{1}+x_{2}/2,\sqrt{3}x_{2}/2)\subset\mathbb{R}^{2}:(x_{1},x_{2})\in\mathbb{Z}^{2}\\}$

is a vertex set, and $\mathcal{E}$ is an edge set, whose endpoints
$(a,b)\in\mathcal{V}\times\mathcal{V}$ are adjacent points, i.e., $\mid
a-b\mid=1$, cf. Figure 1. The time-harmonic discrete waves in $\mathfrak{T}$
can be described by solutions of the discrete Helmholtz equation

Figure 1: Triangular lattice. Connection between
$x=(x_{1},x_{2})\in\mathbb{Z}^{2}$ and the Euclidean coordinates of the
vertexes (black dots) is established via
$(x_{1},x_{2})\to(x_{1}+x_{2}/2,\sqrt{3}x_{2}/2)$. The nearest neighbor
interactions based on the triangular lattice are shown with thick blue lines.
Triangular lattice can be viewed as a left-handed 2D inductor-capacitor
metamaterial. A host transmission-line is loaded periodically with series
capacitors and shunt inductors.

$(\Delta_{d}+k^{2})u(x)=g(x),\quad x=(x_{1},x_{2})\in\mathbb{Z}^{2}.$ (1)

Here, $\Delta_{d}$ denotes the discrete (a 7-point) Laplacian defined as
follows

$\displaystyle\Delta_{d}u(x)=$ $\displaystyle
u(x+e_{1})+u(x-e_{1})+u(x+e_{2})+u(x-e_{2})$
$\displaystyle+u(x+e_{1}-e_{2})+u(x-e_{1}+e_{2})-6u(x),$

where $g$ is a function with bounded support, i.e., $g\in
C_{0}(\mathbb{Z}^{2})$.

Let $\Omega=\mathring{\Omega}\cup\Gamma$ be a discrete half-plane in
$\mathbb{Z}^{2}$, where
$\mathring{\Omega}:=\mathbb{Z}\times\mathbb{\mathbb{Z}^{+}}$ and
$\Gamma:=\partial\Omega=\\{(x_{1},0):x_{1}\in\mathbb{Z}\\}$. We consider the
Dirichlet poblem for the discrete Helmholtz equation on the semi-infinite
lattice:

$\displaystyle(\Delta_{d}+k^{2})u(x)$ $\displaystyle=0,\quad\quad\ \
\textup{in}\ \mathring{\Omega},$ (2a) $\displaystyle u(y)$
$\displaystyle=f(y),\quad\textup{on}\ \Gamma.$ (2b)

Here, $f$ is a given function supported on a finite subset of $\Gamma$.

Thus, we are interested in studying the problem of the existence and
uniqueness of a function $u:\Omega\to\mathbb{C}$ such that $u$ satisfies the
discrete Helmoltz equation (2a) with $k\in(0,3)\backslash\\{2\sqrt{2}\\}$ and
the boundary condition (2b). From now on we will refer to this problem as
Problem $\mathcal{P}_{\mathrm{D}}$.

##  3 Lattice Green’s function

Denote by $\mathcal{G}(x;y)$ the Green’s function for the discrete Helmholtz
equation (1) centered at $y$ and evaluated at $x$. Then the function
$\mathcal{G}(x;y)$ satisfies

$(\Delta_{d}+k^{2})\mathcal{G}(x;y)=\delta_{x,y},$ (3)

where $\delta_{x,y}$ is the Kronecker delta. For brevity, we use the notation
$\mathcal{G}(x)$ for $\mathcal{G}(x;0)$. Notice that
$\mathcal{G}(x;y)=\mathcal{G}(x-y)$.

Using the discrete Fourier transform and the inverse Fourier transform we get

$\mathcal{G}(x)=\frac{1}{4\pi^{2}}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi}\frac{e^{\iota(x\cdot\xi)}}{\sigma(\xi;k)}d\xi,\quad\xi=(\xi_{1},\xi_{2}),$
(4)

where

$\displaystyle\sigma(\xi;k^{2})$
$\displaystyle=e^{\iota\xi_{1}}+e^{-\iota\xi_{1}}+e^{\iota\xi_{2}}+e^{-\iota\xi_{2}}+e^{\iota\xi_{1}}e^{-\iota\xi_{2}}+e^{-\iota\xi_{1}}e^{\iota\xi_{2}}-6+k^{2}$
(5) $\displaystyle=k^{2}-6+2\cos\xi_{1}+2\cos\xi_{2}+2\cos(\xi_{1}-\xi_{2}).$

The lattice Green’s function $\mathcal{G}$ is quite well known when
$k^{2}\in\mathbb{C}\backslash[0,9]$ (cf., e.g., [16]). Notice that if
$k^{2}\in\mathbb{C}\backslash[0,9]$ then $\sigma\neq 0$ and, consequently,
$\mathcal{G}$ in (4) is well defined. In this case $\mathcal{G}(x)$ decays
exponentially when $\mid x\mid\to\infty$. Moreover, we have

$\mathcal{G}(x_{1},x_{2})=\mathcal{G}(x_{2},x_{1})=\mathcal{G}(-x_{1},-x_{2})=\mathcal{G}(x_{1}+x_{2},-x_{2})$
(6)

for all $x=(x_{1},x_{2})\in\mathbb{Z}^{2}$.

Let us show $\mathcal{G}(x_{1},x_{2})=\mathcal{G}(x_{1}+x_{2},-x_{2})$ while
other identities are trivial. According to (4), we have

$\displaystyle\mathcal{G}(x_{1}+x_{2},-x_{2})$ $\displaystyle=$
$\displaystyle\frac{1}{4\pi^{2}}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi}\frac{e^{\iota
x_{1}\xi_{1}+\iota
x_{2}(\xi_{1}-\xi_{2})}}{\sigma(\xi_{1},\xi_{2};k)}d\xi_{1}d\xi_{2}$
$\displaystyle=$
$\displaystyle\frac{1}{4\pi^{2}}\int_{-\pi}^{\pi}\int_{\eta_{1}-\pi}^{\eta_{1}+\pi}\frac{e^{\iota
x_{1}\eta_{1}+\iota
x_{2}\eta_{2}}}{\sigma(\eta_{1},\eta_{2};k)}d\eta_{2}d\eta_{1}$

with $\eta_{1}=\xi_{1},\,\eta_{2}=\xi_{1}-\xi_{2}$. The following equality can
be easily obtained by changing the variable $\eta_{2}$ to $\eta_{2}-2\pi$

$\int_{0}^{\pi}\int_{\pi}^{\eta_{1}+\pi}\frac{e^{\iota x_{1}\eta_{1}+\iota
x_{2}\eta_{2}}}{\sigma(\eta_{1},\eta_{2};k)}d\eta_{2}d\eta_{1}=\int_{0}^{\pi}\int_{-\pi}^{\eta_{1}-\pi}\frac{e^{\iota
x_{1}\eta_{1}+\iota
x_{2}\eta_{2}}}{\sigma(\eta_{1},\eta_{2};k)}d\eta_{2}d\eta_{1}.$

The factor $e^{\iota x_{2}2\pi}$ is equal to $1$ since $x_{2}\in\mathbb{Z}$.
Similar arguments give us

$\int_{-\pi}^{0}\int_{\eta_{1}-\pi}^{-\pi}\frac{e^{\iota x_{1}\eta_{1}+\iota
x_{2}\eta_{2}}}{\sigma(\eta_{1},\eta_{2};k)}d\eta_{2}d\eta_{1}=\int_{-\pi}^{0}\int_{\eta_{1}+\pi}^{\pi}\frac{e^{\iota
x_{1}\eta_{1}+\iota
x_{2}\eta_{2}}}{\sigma(\eta_{1},\eta_{2};k)}d\eta_{2}d\eta_{1}.$

Taking into account the last two equalities, we get

$\mathcal{G}(x_{1}+x_{2},-x_{2})=\frac{1}{4\pi^{2}}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi}\frac{e^{\iota
x_{1}\eta_{1}+\iota
x_{2}\eta_{2}}}{\sigma(\eta_{1},\eta_{2};k)}d\eta_{2}d\eta_{1}=\mathcal{G}(x_{1},x_{2}).$

In the case $k^{2}\in(0,9)\backslash\\{8\\}$ the expression (4) is understood
as follows: we replace $k^{2}$ by $k^{2}+\iota\varepsilon$ with
$0<\varepsilon\ll 1$ and let $\varepsilon\to 0$ at the end of the calculation,
cf. [14]. Thus, we define the lattice Green’s function for
$k\in(0,3)\backslash\\{2\sqrt{2}\\}$ as a pointwise limit of

$(R_{\lambda+\iota\varepsilon}\delta_{x,0})(x):=\frac{1}{4\pi^{2}}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi}\frac{e^{\iota
x\cdot\xi}d\xi_{1}d\xi_{2}}{\sigma(\xi;k^{2}+\iota\varepsilon)}$

as $k^{2}+\iota\varepsilon\to k^{2}+\iota 0$ and denote it again by
$\mathcal{G}(x)$, i.e., $\mathcal{G}(x)=(R_{\lambda+\iota 0}\delta_{x,0})(x)$.
Clearly, $\mathcal{G}(x)$ is a solution to equation (3) and satisfies
equalities (6).

##  4 Unique solvability result

In order to simplify further arguments let us introduce following vectors:

$\displaystyle e_{1}=(1,0),\quad e_{2}=(0,1),\quad e_{3}=e_{1}-e_{2},\quad
e_{4}=-e_{1},\quad e_{5}=-e_{2},\quad e_{6}=-e_{3}.$

For any point $x\in\mathbb{Z}^{2}$ we define the 6-neighbourhood $F^{0}_{x}$
as the set of points $\\{x+e_{j}:j=1,...,6\\}$ and the neighbourhood $F_{x}$
as $F^{0}_{x}\bigcup\\{x\\}$. We say that $R\subset\mathbb{Z}^{2}$ is a region
if there exist disjoint nonempty subsets $\mathring{R}$ and $\partial R$ of
$R$ such that

  * (a)

$R=\mathring{R}\cup\partial R$,

  * (b)

if $x\in\mathring{R}$ then $F_{x}\subset R$,

  * (c)

if $x\in\partial R$ then there is at least one point $y\in F^{0}_{x}$ such
that $y\in\mathring{R}$.

Clearly, the subsets $\mathring{R}$ and $\partial R$ are not defined uniquely
by $R$, but henceforth, it will be always assumed that $\mathring{R}$ and
$\partial R$ are also given and fixed for a given region $R$ in
$\mathbb{Z}^{2}$. We also say that $x$ is an interior (boundary) point of $R$
if $x\in\mathring{R}$ ($x\in\partial R$).

Denote by $(\partial R)_{j}$, $j=1,...,6$, a set of all boundary points
$y\in\partial R$ such that $y-e_{j}\in\mathring{R}$ and call it the sides of
the boundary $R$. Note that a boundary point $x$ can simultaneously belong to
all six sides of $R$. Thus, these sides may overlap each other. Clearly,
$\partial R$ is the union of its six sides, $\partial
R=\cup_{j=1}^{6}(\partial R)_{j}$.

As it was already mentioned above, $y\in\partial R$ may be a point of
intersection of several sides of $\partial R$. However, in our arguments
presented below it will be always clear which side is needed to be considered.
Under this condition, we define the discrete derivative in the outward normal
direction $e_{j}$, $j=1,\dots,6$,

$Tu(y)=u(y)-u(y-e_{j}),\quad y\in(\partial R)_{j}.$

Let us introduce the following set $H_{0}=\\{(0,0)\\}$ and then define
$H_{N}$, $N\in\mathbb{N}$, with the help of recurrence formula

$H_{N}:=\bigcup_{x\in H_{N-1}}F_{x}$

with $\mathring{H}_{N}:=H_{N-1}$ and $(\partial
H)_{N}:=H_{N}\backslash\mathring{H}_{N}$.

Let $R$ be a finite region. Representing
$R=\bigcup_{x\in\mathring{R}}{F_{x}}$, we can easily derive a discrete
analogue of Green’s second identity

$\sum_{x\in\mathring{R}}(u(x)\Delta_{d}v(x)-v(x)\Delta_{d}u(x))=\sum_{y\in\partial
R}(u(y)Tv(y)-v(y)Tu(y)).$ (7)

Now let us give a definition of a radiating solution on the discrete half-
plane. First, we consider the case $k^{2}\in(0,8)$. We say that
$u:\Omega\to\mathbb{C}$ satisfies the radiation condition at infinity if

$\left\\{\begin{aligned} u(x)&=O(\mid x\mid^{-\frac{1}{2}}),\\\
u(x+e_{j})&=e^{\iota\xi^{*}_{j}(\alpha,k)}u(x)+O(\mid
x\mid^{-\frac{3}{2}}),\quad j=1,2,\end{aligned}\right.$ (8)

with the remaining term decaying uniformly in all directions $x/\mid x\mid$,
where $x$ is characterized as $x_{1}=\mid x\mid\cos\alpha$, $x_{2}=\mid
x\mid\sin\alpha$, $0\leq\alpha\leq\pi$. Here, $\xi^{*}_{j}(\alpha,k)$ is the
$j$th coordinate of the point $\xi^{*}(\alpha,k)$.

For $k^{2}\in(8,9)$ we say that $u:\Omega\to\mathbb{C}$ satisfies the
radiation condition at infinity if it can be represented as a sum of functions
$u_{1}$ and $u_{2}$ such that each of them satisfies

$\left\\{\begin{aligned} u_{i}(x)&=O(\mid x\mid^{-\frac{1}{2}}),\\\
u_{i}(x+e_{j})&=e^{\iota\xi^{*,i}_{j}(\alpha,k)}u_{i}(x)+O(\mid
x\mid^{-\frac{3}{2}}),\quad i,j=1,2,\end{aligned}\right.$ (9)

with the remaining term decaying uniformly in all directions $x/\mid x\mid$,
where $x$ is characterized as $x_{1}=\mid x\mid\cos\alpha$, $x_{2}=\mid
x\mid\sin\alpha$, $0\leq\alpha\leq\pi$. Here, $\xi^{*,i}_{j}(\alpha,k)$ is the
$j$th coordinate of the point $\xi^{*,i}(\alpha,k)$, $i=1,2$.

It is shown in [14] that $\mathcal{G}(x)$ satisfies the radiation conditions
introduced above. Now we are ready to introduce a notion of a radiating
solution to the discrete Helmholtz equation (1).

###### Definition 4.1

Let $k^{2}\in(0,8)$. A solution $u$ to the discrete Helmholtz equation (2a) is
called radiating if it satisfies the radiation condition (8).

Let $k^{2}\in(8,9)$. A solution $u$ to the discrete Helmholtz equation (2a) is
called radiating if it satisfies the radiation condition (9).

We represent the Dirichlet Green’s function for the half-plane as follows:

$\mathcal{G}^{+}(x;y)=\mathcal{G}(x;y)-\mathcal{G}(\hat{x};y),\quad
y\in\mathbb{Z}^{2},$

with $x=(x_{1},x_{2})\in\Omega$ and $\hat{x}=(x_{1}+x_{2},-x_{2})$. Notice
that $\mathcal{G}^{+}(x;y)$ can be represented equivalently as

$\mathcal{G}^{+}(x;y)=\mathcal{G}(x;y)-\mathcal{G}(x;\hat{y}).$

Indeed, using the property (6), we have

$\mathcal{G}(\hat{x};y)=\mathcal{G}(x_{1}+x_{2}-y_{1},-x_{2}-y_{2})=\mathcal{G}(x_{1}-y_{1}-y_{2},x_{2}+y_{2})=\mathcal{G}(x;\hat{y}).$
(10)

Since $\widehat{(y_{1},0)}=(y_{1},0)$, we immediately get
$\mathcal{G}^{+}(x;y)=0$ for $y\in\Gamma$. Moreover,

$(\Delta_{d}+k^{2})\mathcal{G}^{+}(x;y)=(\Delta_{d}+k^{2})\mathcal{G}(x;y)-(\Delta_{d}+k^{2})\mathcal{G}(\hat{x};y)=\delta_{x,y}-\delta_{\hat{x},y}=\delta_{x,y}$

for all $x\in\mathring{\Omega}$ and $y\in\Omega$. From (10) we see that
$\mathcal{G}^{+}(x;y)$ satisfies the radiation condition. Indeed, for any
fixed $y\in\Gamma$ the angle $\alpha$ from the radiation conditions (8) and
(9) is the same for $\mathcal{G}(x;y)=\mathcal{G}(x-y)$ and
$\mathcal{G}(x;\hat{y})=\mathcal{G}(x-\hat{y})$ when $\mid x\mid\to\infty$.

###### Theorem 4.2

Let $k\in(0,3)\backslash\\{2\sqrt{2}\\}$, and $u$ be a given function
$\Omega\to\mathbb{C}$ that satisfies the radiation condition. The function
$u\mid_{\Gamma}$ has a finite support. Then, for any point
$x\in\mathring{\Omega}$, we have the following representation formula

$u(x)=\sum_{y\in\Gamma}\big{(}u(y)T\mathcal{G}^{+}(x;y)-\mathcal{G}^{+}(x;y)Tu(y)\big{)}+\sum_{y\in\mathring{\Omega}}\mathcal{G}^{+}(x;y)(\Delta_{d}+k^{2})u(y).$

In particular, if $u$ is a solution to the discrete Helmholtz equation

$(\Delta_{d}+k^{2})u(x)=0\quad\textup{in}\ \mathring{\Omega},$

then

$u(x)=\sum_{y\in\Gamma}u(y)T\mathcal{G}^{+}(x;y).$ (11)

  * Proof.  Denote by $\tilde{u}$ the extension of $u$ to $\mathbb{Z}^{2}$ by zeros, i.e, $\tilde{u}(x)=u(x)$ if $x\in\Omega$ and $\tilde{u}(x)=0$ if $x\in\mathbb{Z}\times\mathbb{Z}^{-}$. Then, for any finite region $H_{N}$, $N\in\mathbb{N}$, we apply Green’s second identity (7) where we take $v(y)=\mathcal{G}^{+}(x;y)$. Here, note that $\mathcal{G}^{+}$ is a fundamental solution for $y\in\mathring{\Omega}$, and $\tilde{u}(y)=0$ for $y\in\mathbb{Z}\times\mathbb{Z}^{-}$. Thus, $u(y)(\Delta_{d}+k^{2})v(y)$ disappears in the following identity

$u(y)\Delta_{d}v(y)-v(y)\Delta_{d}u(y)=u(y)(\Delta_{d}+k^{2})v(y)-v(y)(\Delta_{d}+k^{2})u(y)$

as far as $y\in\mathbb{Z}\times\mathbb{Z}^{-}$. For $x\in\mathring{H}_{N}$, we
get

$\displaystyle\tilde{u}(x)$
$\displaystyle=\sum_{y\in\mathring{H}_{N}}\tilde{u}(y)\delta_{x,y}$
$\displaystyle=\sum_{y\in\partial
H_{N}}\big{(}\tilde{u}(y)T\mathcal{G}^{+}(x;y)-\mathcal{G}^{+}(x;y)T\tilde{u}(y)\big{)}+\sum_{y\in\mathring{H}_{N}}(\mathcal{G}^{+}(x;y)(\Delta_{d}+k^{2})\tilde{u}(y)).$

Passing to the limit $N\to\infty$, we use exactly the same arguments as in the
proof of Theorem 5.2 from [14] and, consequently, we get

$\tilde{u}(x)=\sum_{y\in\mathbb{Z}^{2}}\mathcal{G}^{+}(x;y)(\Delta_{d}+k^{2})\tilde{u}(y).$

For the function $u$, we obtain

$u(x)=\sum_{y\in\Gamma}\big{(}u(y)T\mathcal{G}^{+}(x;y)-\mathcal{G}^{+}(x;y)Tu(y)\big{)}+\sum_{y\in\mathring{\Omega}}\mathcal{G}^{+}(x;y)(\Delta_{d}+k^{2})u(y).$

Taking into the account that $\mathcal{G}^{+}(x;y)=0$ when $y\in\Gamma$, for a
solution $u$ to the discrete Helmholtz equation we get the following quality

$u(x)=\sum_{y\in\Gamma}u(y)T\mathcal{G}^{+}(x;y),\quad x\in\mathring{\Omega}.$

$\Box$

Now we are ready to prove the unique solvability result for the discrete
Helmholtz equation on the semi-infinite triangular lattice.

###### Theorem 4.3

Let $k\in(0,3)\backslash\\{2\sqrt{2}\\}$ then the Problem
$\mathcal{P}_{\mathrm{D}}$ has a unique radiating solution $u$ which can be
represented as

$u(x)=\sum_{y\in\Gamma}(\delta_{x,y}-\mathcal{G}^{+}(x;y+e_{2})-\mathcal{G}^{+}(x,y+e_{2}-e_{1}))f(y).$
(12)

  * Proof.  To prove the uniqueness result, it is sufficient to show that the corresponding homogeneous problem has only the trivial solution. Let $u$ be a radiating solution to the homogeneous problem $\mathcal{P}_{D}$. Then Theorem 4.2 immediately implies $u(x)=0$ for all $x\in\mathring{\Omega}$.

To show the existence results, let us first check that $u(x)$ satisfies the
boundary condition (2b). Since $\mathcal{G}^{+}(x;y)=0$ for all $x\in\Gamma$
and any $y$ we get

$u(x_{1},0)=\sum_{y_{1}\in\mathbb{R}}\delta_{x_{1},y_{1}}\delta_{0,0}f(y)=f(x_{1}).$

Now let us check that $u(x)$ satisfies the Helmholtz equation (2a). For
$x_{2}>1$ all terms $(\Delta_{d}+k^{2})\delta_{x,y}$,
$(\Delta_{d}+k^{2})\mathcal{G}^{+}(x;y+e_{2})=\delta_{x,y+e_{2}}$ and
$(\Delta_{d}+k^{2})\mathcal{G}^{+}(x;y+e_{2}-e_{1})=\delta_{x,y+e_{2}-e_{1}}$
are equal to zero. Consequently, $(\Delta_{d}+k^{2})u(x)=0$ for points
$x=(x_{1},x_{2})$ with $x_{2}>1$. It remains to consider the case $x_{2}=1$.
By the direct calculation we have

$\displaystyle(\Delta_{d}+k^{2})\delta_{x,y}f(y)$ $\displaystyle=$
$\displaystyle f(x_{1})+f(x_{1}+1),$
$\displaystyle(\Delta_{d}+k^{2})\mathcal{G}^{+}(x;y+e_{2})f(y)$
$\displaystyle=$ $\displaystyle f(x_{1}),$
$\displaystyle(\Delta_{d}+k^{2})\mathcal{G}^{+}(x;y+e_{2}-e_{1})f(y)$
$\displaystyle=$ $\displaystyle f(x_{1}+1),$

and as a result $(\Delta_{d}+k^{2})u(x)=0$. Thus, $u(x)$ is a solution to
(2a). Since $\mathcal{G}^{+}$ satisfies the radiation condition and $f$ is
supported on the finite subset of $\Gamma$ it follows that $u(x)$ is the
unique radiating solution to the problem (2a), (2b). $\Box$

##  5 Numerical results

The main issue for numerical evaluation of the solution (12) is to compute the
lattice Green’s function. For this purpose we apply the method developed in
[4]. Using 8-fold symmetry, we need only to compute the lattice Green’s
function $\mathcal{G}(i,j)$ with $i\geq j\geq 0$. Following to [4], let us
introduce the vectors
$\mathcal{V}_{2p}=(\mathcal{G}(2p,0),\mathcal{G}(2p-1,1),\dots,\mathcal{G}(p,p))^{\top}$
and
$\mathcal{V}_{2p+1}=(\mathcal{G}(2p+1,0),\mathcal{G}(2p,1),\dots,\mathcal{G}(p+1,p))^{\top}$
that collect all distinct Green’s functions $\mathcal{G}(i,j)$ with “Manhattan
distances” $\mid i\mid+\mid j\mid$ of $2p$ and $2p+1$, respectively. For any
Manhattan distance larger than 1, equation

$(\Delta_{d}+k^{2})\mathcal{G}(x)=\delta_{x,0}$

can be written in the matrix form
$\gamma_{n}(k)\mathcal{V}_{n}=\alpha_{n}(k)\mathcal{V}_{n-1}+\beta_{n}(k)\mathcal{V}_{n+1}$
where $\alpha_{n}(k)$, $\beta_{n}(k)$ and $\gamma_{n}(k)$ are sparse matrices
(cf., Appendix A). Notice that only the dimensions of these matrices depend on
$n$. It is shown in [4] that, for any $n\geq 1$, we have

$\mathcal{V}_{n}=A_{n}(k)\mathcal{V}_{n-1},$

where the matrices $A_{n}(k)$ are defined by the following recurrence formula

$A_{n}(k)=[\gamma_{n}(k)-\beta_{n}(k)A_{n+1}]^{-1}\alpha_{n}(k).$

They can be computed starting from a sufficiently large $N$ with
$A_{N+1}(k)=0$. Here, it is worth mentioning, that for $k=2$ we need to choose
a better “initial guess” than $A_{N+1}(k)=0$, since in this case
$\mathrm{det}\gamma_{n}(k)=0$ and the matrix
$\gamma_{n}(k)-\beta_{n}(k)A_{n+1}$ is not invertible.

Once $A_{n}(k)$ are known, we have $\mathcal{V}_{n}=A_{n}(k)\dots
A_{1}(k)\mathcal{V}_{0}$, where $\mathcal{V}_{0}=\mathcal{G}(0,0)$. In
particular, $\mathcal{V}_{1}=\mathcal{G}(1,0)=A_{1}(k)\mathcal{G}(0,0)$ which,
together with $6\mathcal{G}(1,0)-(6-k^{2})\mathcal{G}(0,0)=1$, gives
$\mathcal{G}(0,0)=1/[6A_{1}(k)-6+k^{2}]$. This completes the calculation of
the Green’s function using elementary operations and no integrals. Notice also
one more important advantage of this method. The $A_{n}(k)$ matrices are
calculated coming down from asymptotically large Manhattan distances. As they
are propagated towards smaller Manhattan distances, it definitely gives us the
physical solution.

Finally, we demonstrate our theoretical and numerical approaches for the
diffraction problem on triangular lattice. For this purpose we take
$k=\sqrt{2}$ and consider the wave $U(x_{1},x_{2},t)=\exp(\iota
x_{2}\pi/3-\iota\omega t)$ on the semi-infinite lattice
$\mathbb{Z}\times\mathbb{Z}^{-}$ which encounters an obstacle at $x_{2}=0$
with two small openings formed by four nodes
$\\{(-11,0),(-10,0),(10,0),(11,0)\\}$. It is easy to check that $\exp(\iota
x_{2}\pi/3)$ satisfies the discrete Helmholtz equation. Thus, our goal is to
evaluate numerically a radiating solution of Problem
$\mathcal{P}_{\mathrm{D}}$ where, for the given data, we take $f(\pm 10)=f(\pm
11)=1$ and $f(m)=0$, $m\in\mathbb{Z}^{+}\backslash\\{-11,-10,10,11\\}$.

(a) The density plot of $\mathrm{Re}\,\mathcal{G}$.

(b) The density plot of $\mathrm{Re}\,u$.

(c) The graph of $\mathrm{Re}\,u(\cdot,25\sqrt{3})$.

Figure 2: Plots are represented in original coordinates of the triangular
lattice $\mathfrak{T}$. $k=\sqrt{2}$.

The results of numerical evaluations are plotted in Figure 2 in original
coordinates of $\mathfrak{T}$, where (a) shows the density plots of the
Green’s function $\mathrm{\Re e}\,\mathcal{G}$, (b) shows the density plots of
$\cos(\iota x_{2}\pi/3)$ on the lower half-plane and $\mathrm{\Re e}\,u$ for
the case $k=\sqrt{2}$ on the upper half-plane, and (c) presents the graph of
$\mathrm{Re}\,u(\cdot,25\sqrt{3})$. Some key features of the numerical
solution can be immediately observed, namely, as expected, the symmetry of
$\mathrm{\Re e}\,u$ and contributions from the wavefront that create a
variable intensity.

##  6 Discussion

In this paper, we have constructed the discrete scattering theory for the two-
dimensional discrete Helmholtz equation with a real wave number
$k\in(0,3)\backslash\\{2\sqrt{2}\\}$ for the semi-infinite triangular
lattices. The main objective was to prove the unique solvability result and
derive an exact formula for the solution to a Dirichlet problem. For
simplicity we restricted ourselves to compact boundary data. For non-compact
boundary data we should require some extra conditions at infinity, cf. [15].

Similarly to the continuum theory, we used the notion of the radiating
solution for the continuous Helmholtz equation

$\Delta u(x)+\kappa^{2}u(x)=0,\quad\kappa\in\mathbb{R}\backslash\\{0\\},\
x\in\mathbb{R}^{2}.$ (13)

Recall that a solution $u$ to the equation (13) is called radiating if it
satisfies the Sommerfeld radiation condition

$\frac{\partial}{\partial\mid x\mid}u(x)-\iota\mid\kappa\mid u(x)=o(\mid
x\mid^{-\frac{1}{2}})$ (14)

for $\mid x\mid\to\infty$ uniformly in all directions $x/\mid x\mid$. There
are evident similarities and differences between conditions (8), (9) and (14).
For example, we see the same decay at infinity, but, in contrast to (14), the
second condition of (8) or (9) is anisotropic: the factor
$e^{\iota\xi^{*}_{j}(\alpha,k)}$ is not a constant. Moreover, it is well known
that the radiating solutions $u$ to the continuous Helmholtz equation
automatically satisfy the Sommerfeld’s finiteness condition, however this fact
is not shown for the discrete case so far. Therefore, the first condition of
(8) or (9) is included in our definition.

In the present paper, the problems under consideration have an infinite
boundary. Within the continuum framework it is well-known that, in general,
when the surface is unbounded, we cannot neglect the contribution of that
surface waves at infinity. In this case, the Sommerfeld radiation condition is
no longer appropriate and a proper modification is needed. To the best of the
authors’ knowledge, different radiation conditions are provided only for the
half-plane (and locally perturbed half-plane) problems, cf. [7, 8, 9, 10, 11,
12], and finding radiation conditions for arbitrary wedge-shaped regions
remains open. In case of the square lattice we have proposed sufficient
conditions for the given boundary data at infinity, which ensures to have an
unique radiation solution to the corresponding problem, cf. [15]. Here, in
order to avoid further technical difficulties, we restricted ourselves with
compactly supported data on the boundary.

Finally, let us note that comparing the results of problems in the continuous
framework and results obtained in the continuum limit of discrete problems
deserves the high interest of scientists, however, it is beyond the scope of
this paper.

## Acknowledgments

This work was supported by Shota Rustaveli National Science Foundation of
Georgia (SRNSFG) [FR-21-301]

## Appendix A Sparse matrices

The sparse matrices $\alpha_{n}(k)$, $\beta_{n}(k)$ and $\gamma_{n}(k)$ are
defined as follows: if $n=2p$ then $\alpha_{2p}(k)$ is a $(p+1)\times p$
matrix such that $\alpha_{2p}(k)\mid_{i,i}=1$, $i=\overline{1,p}$,
$\alpha_{2p}(k)\mid_{i,i-1}=1$, $i=\overline{2,p}$, while
$\alpha_{2p}(k)\mid_{p+1,p}=2$, and all other matrix elements are zero. The
$\beta_{2p}(k)$ is a $(p+1)\times(p+1)$ matrix such that
$\beta_{2p}(k)\mid_{i,i}=1$, $i=\overline{1,p}$,
$\beta_{2p}(k)\mid_{i,i+1}=1$, $i=\overline{2,p}$, while
$\beta_{2p}(k)\mid_{p+1,p+1}=\beta_{2p}(k)\mid_{1,2}=2$, and all other matrix
elements are zero. The $\gamma_{2p}(k)$ is a $(p+1)\times(p+1)$ matrix such
that $\gamma_{2p}(k)\mid_{i,i}=6-k^{2}$, $i=\overline{1,p+1}$,
$\gamma_{2p}(k)\mid_{i,i+1}=\gamma_{2p}(k)\mid_{i,i-1}=-1$,
$i=\overline{2,p}$, and
$\gamma_{2p}(k)\mid_{1,2}=\gamma_{2p}(k)\mid_{p+1,p}=-2$.

If $n=2p+1$ then $\alpha_{2p+1}(k)$ is a $(p+1)\times(p+1)$ matrix such that
$\alpha_{2p+1}(k)\mid_{i,i}=1$, $i=\overline{1,p+1}$,
$\alpha_{2p+1}(k)\mid_{i,i-1}=1$, $i=\overline{2,p+1}$, and all other matrix
elements are zero. The $\beta_{2p+1}(k)$ is a $(p+1)\times(p+2)$ matrix such
that $\beta_{2p+1}(k)\mid_{i,i}=1$, $i=\overline{1,p+1}$,
$\beta_{2p+1}(k)\mid_{i,i+1}=1$, $i=\overline{2,p+1}$, while
$\beta_{2p+1}(k)\mid_{1,2}=2$, and all other matrix elements are zero. The
$\gamma_{2p}(k)$ is a $(p+1)\times(p+1)$ matrix such that
$\gamma_{2p+1}(k)\mid_{i,i}=6-k^{2}$, $i=\overline{1,p}$,
$\gamma_{2p+1}(k)\mid_{p+1,p+1}=5-k^{2}$, while
$\gamma_{2p+1}(k)\mid_{i,i+1}=-1$, $i=\overline{2,p}$,
$\gamma_{2p+1}(k)\mid_{i,i-1}=-1$, $i=\overline{2,p+1}$, and
$\gamma_{2p+1}(k)\mid_{1,2}=\gamma_{2p+1}(k)\mid_{p+1,p}=-2$. Finally,
$\gamma_{1}(k)$ is a $1\times 1$ matrixs with an element $4-k^{2}$.

## References

  * [1] Bhat H.S, Osting B. Diffraction on the Two-Dimensional Square Lattice, SIAM J. Appl. Math. 70 (2009), 1389-1406. https://doi.org/10.1137/080735345.
  * [2] Born M, Huang K. _Dynamical Theory of Crystal Lattices_. Clarendon Press, Oxford; 1954.
  * [3] Burke J.G. _Origins of the Science of Crystals_. University of California Press, Berklay and Los Angeles; 1966.
  * [4] Berciu M, Cook A.M. Efficient computation of lattice Green’s functions for models with nearest-neighbour hopping, Europhys. Lett. 2010; 92: 40003. DOI: 10.1209/0295-5075/92/40003.
  * [5] Brillouin L. _Wave Propagation in Periodic Structures. Electric Filters and Crystal Lattices_. International Series in Pure and Applied Physics: McGraw-Hill; 1946.
  * [6] Caloz C, Itoh T. _Electromagnetic Metamaterials: Transmission Line theory and microwave applications: the engineering approach_. John Wiley & Sons, Inc. Hoboken, New Jersey; 2006.
  * [7] Chandler-Wilde S.N., Boundary value problems for the Helmholtz equation in a half-plane, in: G. Cohen, (ed.), Proc. 3rd Int. Conf. on Mathematical and Numerical Aspects of Wave Propagation, SIAM, Philadelphia, 1995; 188-197.
  * [8] Chandler-Wilde S.N., The impedance boundary value problem for the Helmholtz equation in a half-plane, Math. Methods Appl. Sci. 20 (1997) 813-840. https://doi.org/10.1002/(SICI)1099-1476(19970710)20:10$<$813::AID-MMA883$>$3.0.CO;2-R.
  * [9] Chandler-Wilde S.N., Zhang B., A uniqueness result for scattering by infinite rough surfaces, SIAM J. Appl. Math. 58 (1998) 1774-1790. https://doi.org/10.1137/S0036139996309722.
  * [10] Chandler-Wilde S.N., Zhang B., Electromagnetic scattering by an inhomogeneous conducting or dielectric layer on a perfectly conducting plate, Proc. R. Soc. Lond. A. 454 (1998) 519-542. https://doi.org/10.1098/rspa.1998.0173.
  * [11] Durán M. , I. Muga, Nédélec J.C., The Helmholtz equation in a locally perturbed half-plane with passive boundary, IMA J. Appl. Math. 71 (2006) 853-876. https://doi.org/10.1093/imamat/hxl023.
  * [12] Durán M. , I. Muga, Nédélec J.C., The Helmholtz equation in a locally perturbed half-space with non-absorbing boundary, Arch. Ration. Mech. Anal. 191(2009) 143-172. https://doi.org/10.1007/s00205-008-0135-3.
  * [13] Dove MT. _Structure and Dynamics: An Atomic View of Materials_. Oxford University Press; 2002.
  * [14] Kapanadze D. The far-field behaviour of Green’s function for a triangular lattice and radiation conditions, _Math Meth Appl Sci._ 2021; 44(17):12746-12759. DOI: 10.1002/mma.7575
  * [15] Kapanadze D, Pesetskaya E. Diffraction problems for two-dimensional lattice waves in a quadrant, _Wave Motion_ 2021; 100:102671. DOI: 10.1016/j.wavemoti.2020.102671
  * [16] Horiguchi T. Lattice Green’s Functions for the Triangular and Honeycomb Lattices, _J. Math. Phys._ 1972; 13(9):1411-1419. DOI: 10.1063/1.1666155
  * [17] Slepyan LI. _Models and Phenomena in Fracture Mechanics_. Springer, New York; 2002.


# Word2rate: Training and evaluating multiple word embeddings as statistical
transitions

Dario Poletti
Engineering Product Development
Singapore University of Technology and Design
<EMAIL_ADDRESS>
&Lin Shaowei
Engineering Systems and Design
Singapore University of Technology and Design
<EMAIL_ADDRESS>
&Gary Phua Hwee Leng
Information Systems Technology and Design
Singapore University of Technology and Design
<EMAIL_ADDRESS>

###### Abstract

Using pretrained word embeddings has been shown to be a very effective way in
improving the performance of natural language processing tasks. In fact almost
any natural language tasks that can be thought of has been improved by these
pretrained embeddings. These tasks range from sentiment analysis, translation,
sequence prediction amongst many others. One of the most successful word
embeddings is the Word2vec CBOW model proposed by Mikolov trained by the
negative sampling technique. Mai et al. modifies this objective to train CMOW
embeddings that are sensitive to word order. We used a modified version of the
negative sampling objective for our context words, modelling the context
embeddings as a Taylor series of rate matrices. We show that different modes
of the Taylor series produce different types of embeddings. We compare these
embeddings to their similar counterparts like CBOW and CMOW and show that they
achieve comparable performance. We also introduce a novel left-right context
split objective that improves performance for tasks sensitive to word order.
Our Word2rate model is grounded in a statistical foundation using rate
matrices while being competitive in variety of language tasks.

##  1 Introduction

Word embeddings are mathematical representations of words in vector space.
They first took the form of one hot vectors in a local representation, where
the size of the vector is equal to the vocabulary size of the language model.
All elements take in the value of zero, apart from the index of the vector
that corresponds to a word in the vocabulary, which takes the value of one.
These representations however, have achieved limited success. It is with the
introduction of pretrained word embeddings like Mikolov’s Word2vec[1] that
arguably kickstarted the level of improvements we see in natural language
processing today. These word embeddings have a distributed representation
where they are encoded with semantic meaning in a geometrical space. This
means that words that have similar meaning are close together in vector space.
They have also been used to improve the performance of many downstream tasks
such as analogy([1]), translation([2], [3]), sentiment analysis([4],[5]),
named entity recognition([6]) among many others. While these embeddings have
been effective, mathematically they only represent the hidden layer of an
autoencoder and do not have much statistical significance. For this paper, we
will be modelling embeddings as a series of statistical transitions along a
Markov chain. We will be using a modified negative sampling objective of
Mikolov, that differs in the composition function in forming the context
embedding $v_{c}$. Others such as Mai et al. [7] have attempted to modify the
objective before, using matrix multiplication instead of addition as the
composition for the context.

Our model uses a Taylor series of rate matrices acting on a initial
distribution. for the composition of the context embedding. Depending on the
mode of the Taylor series, the context embedding could be composed of
addition, product or a mixture of both. We call this novel word embedding
Word2rate, and compare the different modes to their existing counterparts. We
evaluate these word embeddings on a series of 10 linguistic probing and 11
downstream tasks from Senteval[8],[9], following Mai. Senteval covers a
widerange of tasks, including semantic and syntactic information of a sentence
embedding and sentiment classification.

We introduce 3 different modes of the Taylor series from our Word2rate model,
namely First Order Series (FOS), First Order Product(FOP) as well as Second
Order Series (SOS). We also evaluate 2 Hybrid models formed from these
embeddings, namely Word2rate FOS-FOP and Word2rate FOS-SOS, which were formed
using the same methods introduced by Mai. We also introduce a novel left-right
context split to form context embeddings. We find these embeddings to improve
performance on tasks sensitive to word order. We show overall that the
different modes of our Word2rate embedding are competitive with their
counterparts. At the same time, our model grounds word embedding formation
with a statistical foundation by modelling context formation as a series of
transitions.

##  2 Related Work

In this paper, we will be training semantic unigram and bigram word embeddings
using a model similar to Word2vec. These bigrams will be formed by some
composition rules using algebraic operators. Mitchell and Lapata[10] first
gave a mathematical basis to build upon when studying language composition.
They first defined a basic composition $p$ as a function two elements u and v:

$p=f(u;v)$

These elements $u$ and $v$ can represent words and $p$ can represent a phrase.
They then go on to define a more complete composition that will include the
syntactic relation R of u and v as well as the existing knowledge used during
composition, K:

$p^{\prime}=f(u;v;R;K)$

They studied the performance of various vector composition models on a
similarity rating task for phrase pairs in the english language. The aim being
able to represent higher level structures of language in a semantic space.
Their experiment does not use $K$ as it does not use any outside knowledge
during composition. The relationship $R$ that they use is the relationship
between the word pairs. They use both semantic and probabilistic models for
their word vectors and apply the same algebraic operators to them. These
algebraic operators include addition, multiplication, tensor product and
dilation among others. For our model, the algebraic operators we use will be
mainly addition and matrix multiplication, which involves acting rate matrices
on an initial distribution as well combining these operators together. The
relationships of the words will be the context-target pair that is the same as
Word2vec.

Socher[11] also builds upon the work of Mitchell and Lapata but applies a
different approach to language composition. He assigns a matrix $A$ and a
vector $a$ to each word. By doing so, 2 words can be described as a cross
matrix vector pair:

$j=g(W[Ba\;Ab])$

Where $a$ and $b$ are $n$ dimensional vectors that represent words, with their
counterparts $A$ and $B$ being their corresponding matrix.$[Ba\;Ab]$ is the
concatenation between the 2 resultant vectors. $g$ is a non-linear function,
$W$ is a $2n\times n$ composite matrix that maps the concatenation back to $n$
dimensions, and $j$ is the composition vector of the 2 words. These
composition vectors are trained recursively in a tree structure on some
supervised sentiment tasks. This tree structure allows the model to capture
the syntactic composition of the words.

Mikolov[1] produced the seminal paper on word embeddings, training word
embeddings on a large unsupervised corpus by splitting words in a set of
context-target pairs. There are two different methods to do so, namely
skipgram and CBOW. We will focus on CBOW. In CBOW, a collection of context
words are summed to form a context embedding $v_{c}$:

$v_{c}=\sum_{i=-m}^{m}W_{c}c_{i}$

These context words $w_{c}$ are within $m$ words of a target word $w_{t}$.
$c_{i}$ can be thought of as a one-hot vector acting on the list of context
embeddings $W_{c}$. For a corpus with vocabulary size $n$ and embedding
dimension $d$, $W_{c}$ is a $n\times d$ matrix. Forming this context embedding
is essentially what differentiates our model from Word2vec. For training does
not only involve training individual words, their composition rules or
relationships between the words are also incorporated into the embeddings
during training. After forming the context embedding, it is then used to
predict the target word from a softmax function:

$p(w_{t}|w_{c})=\frac{v_{c}^{T}v_{t}}{\sum_{i=1}^{W}\exp(v_{i}^{T}v_{t})}$

Where $v_{t}$ is the associated target embedding. Unlike the context
embedding, it is simply equal to one of the rows of the target embedding
matrix $W_{t}$ instead of a sum. Given a large vocabulary size, computing the
softmax directly is computationally expensive. Hence, Mikolov approximates
this using a negative sampling objective function:

$F=\log\sigma(v_{t}^{T}v_{c})+\sum_{i=1}^{k}\log\sigma(-v_{nsi}^{T}v_{c})$

This negative sampling objective learns to discriminate between good embedding
pairs $v_{c}$, $v_{t}$ with bad embedding pairs drawn from a unigram
distribution, $v_{ns}$ and $v_{c}$. This will be the objective function that
we use for training our embeddings.

Mai et al.[7] on the other hand, trains word embedding matrices on an
unsupervised task using a modified version of the context embedding. They
essentially replaced the sum operation of the context embedding vector $v_{c}$
with matrix multiplication. Instead of a vector, they form their context
embedding using matrices:

$M_{c}=\prod_{i=-m}^{m}W_{ci}$

Where in this case, $M_{c}$ is now the context embedding and $W_{c}$ is now a
$n\times d\times d$ context embedding matrix. They show that this is useful as
the non-commutating properties of matrix multiplication allow word order to be
learnt by the model. This is compared to the usual summing of word vectors,
where word order is invariant under summation and will not give the model that
information. They also propose using an assymmetric window for the context
words by randomly selecting the target word, as well as initialising the
embedding matrices with an identity matrix to prevent vanishing gradient
problems from multiplication.

Our model shares similar representation to that of Socher, while having a more
similar objective function to Mikolov and Mai. We will also be training our
model on a modified version of the context embedding. However, by representing
our embeddings as statistical operators, we hope to ground our model with a
theoretical statistical foundation. In the next section we will introduce the
different modes of our Word2rate embeddings as well as our novel left-right
context split technique.

##  3 Word2rate

In Word2vec, Mikolov has shown that one can learn word embeddings through
negative sampling by maximising the correlation between a correct context-
target pair $v_{c}$ and $v_{t}$ and penalising the correlation between a wrong
context-target pair $v_{c}$ and $v_{n}s$. The context word embedding $v_{c}$
was formed by simply summing the neighboring words embeddings of the target.
This summation can be treated as a composition function of the neighboring
words. In our model of Word2rate, while using the same negative sampling
objective, we employ a different composition function to obtain $v_{c}$ that
represents a Markov chain of transitions from an initial probability
distribution. The target embeddings $v_{t}$ are also unchanged in our model.
We decide to use probability distributions as we want to view word embeddings
as a distributed representation where each element of the vector can be
thought of as some abstract concept state. A word is then uniquely represented
by a configuration of these concept states, while a sequence of words can be
represented as a transition across multiple states. We first define the
initial distribution $p_{u}$ as a uniform distribution where:

$p_{u}\in\mathbb{R}$ $(p_{u})_{i}=\frac{1}{d}$ $\sum_{i=1}^{d}(p_{u})_{i}=1$

Where $d$ is the dimensions of the vector and $(p_{u})_{i}$ is the
$i^{\textnormal{th}}$ element of $p_{u}$. We can then form a word embedding as
a transition from the initial distribution. This can be then achieved by
applying a stochastic matrix $S_{1}$ to $p_{u}$:

$p_{1}=S_{1}p_{0}$ Where the elements of a stochastic matrix $S$, $s_{ij}$ are
conditioned with the following properties:

$s_{ij}\leq 0$ $\sum_{i=1}s_{ij}=1$

Equation 3 is conditioned due to the elements of the stochastic matrices being
probabilities and hence have to be non-negative. Equation 3 is conditioned for
each column to sum to 1 to conserve probability as a left stochastic matrix
which acts on the left of the probability distribution. Having introduced the
properties of stochastic matrices, we can then define our word embedding as
the stochastic matrix of a particular word acting on a distribution. To form a
word embedding like ’cat’, we can apply the corresponding stochastic matrix
$S_{cat}$ to the initial distribution $p_{u}$

$v_{cat}=S_{cat}p_{u}$

We can also defined a sequence of words as series of stochastic matrices
acting on an initial distribution. We will demonstrate this with an example of
a sentence ’We love cats’. We can let ’cats’ be the target word while the
phrase ’We love’ is the context. We can then represent the context embedding
as:

$v_{c}=S_{love}S_{We}p_{u}$

In order to gain a richer representation of the composition function, we can
approximate these stochastic matrices using a Taylor series. Firstly, a
stochastic matrix $S$ can be expressed as an exponential function of a rate
matrix $Q$:

$S=\exp(\epsilon Q)$

Where $\epsilon\ll 1$. Since $Q$ is a matrix, expressing it as an exponential
is non-trivial. We can typically do so by approximating the exponential
function as a Taylor expansion:

$S=I+\epsilon Q+\frac{\epsilon^{2}Q^{2}}{2!}+...$

We can then apply Equation 3 above to Equation 3:

$v_{c}=(I+\epsilon
Q_{love}+\frac{\epsilon^{2}Q_{love}^{2}}{2!}+...)(I+\epsilon
Q_{We}+\frac{\epsilon^{2}Q_{We}^{2}}{2!}+...)p_{u}$

Equation 3 is the crux of our Word2rate model. We can see a variety of ways
that we can choose to approximate this infinite expansion. Depending on how we
choose to approximate, we will get different composition rules which will
provide different qualities for embeddings trained under these rules.

###  3.1 Word2rate First Order Series(FOS)

$v^{FOS}_{c}=(I+\epsilon Q_{love}+\epsilon Q_{We})p_{u}$

We first approximate the Taylor series using only up to the first order of
$\epsilon$. We illustrate this in equation 3.1 by continuing our ’We love’
example earlier on. We can see from the equation that our FOS embeddings are
formed using only addition between rate matrices. This form of additive
composition is the same as that used by Mikolov for Word2vec. Hence we can say
that

###  3.2 Word2rate First Order Product(FOP)

$v^{FOP}_{c}=(I+\epsilon Q_{love})(I+\epsilon Q_{We})p_{u}$

Instead of approximating the entire Taylor series up to the first order, we
can approximate the individual product terms to the first order instead. This
would form our Word2rate FOP context embedding in this manner. We can first
observe that this form of approximation would allow the context embedding to
contain many more terms in the expansion than our FOS embeddings implicitly.
For our FOS embeddings or CBOW, the number of terms of the embedding scale
linearly with the number of context words. For our 2 word example, we have 2
terms in the expansion comprising of the rate matrix $Q_{love}$, $Q_{We}$ as
well as the identity matrix. For our FOP embeddings, we will have 2 term, the
product contains 4 terms, including the bilinear term $\epsilon
Q_{love}Q_{We}$. This will allow us to train a relationship between the words
‘We’ and ‘love’ that incorporates the syntactic properties of the sequence.
This respects the order that ‘love’ comes after ‘We’ in this case, since our
rate matrices like CMOW are sensitive to word order. However, unlike CBOW our
CMOW, our expansion implicitly contains $2^{2m}$ terms with $m$ being the
window size after expansion. This is due to the addition of the identity
matrix, which allows for cross product between words. While CMOW introduces
the identity matrix as an engineering technique to avoid the vanishing
gradient problem, our identity matrix is a mathematical feature arising from
statistical theory, and it is added independently of the trained rate
matrices. We will later show in our experiments the role of $\epsilon$ in
avoided vanishing gradients instead. Furthermore, unlike CMOW, our matrix
embeddings do not unroll to form word embeddings. Instead, they act on an
initial uniform distribution and transit that initial state as it progresses
through the sequence of context to form our final context embedding.

###  3.3 Left-Right Context Split

CBOW and CMOW use a combination of a sequence of words on both the left and
right side of the target word to form their context embeddings. For example,
for a sentence like ‘The quick fox jumps’, if the target word is ‘fox’, the
the context embedding can be formed as:

$v_{c}=f(w_{the},w_{quick},w_{jumps})$

Where $f$ could be the sum operation of CBOW, product operation of CMOW or one
of our Word2rate series operations. In all cases, the context embedding is
formed as ‘The quick jumps’. Recall earlier that we mentioned that the context
embedding learns the relationships between words in a sequence as well. This
is why sentence embeddings for downstream tasks are formed by adding CBOW word
embeddings, while being formed by multiplying CMOW embeddings instead.
However, a problem arises as ‘The quick jumps’ is not syntactically correct in
the english language. Hence, this will cause inaccuracies especially when
using reprensentations like matrices which are sensitive to word order. To
solve this issue, we propose a novel left-right context split that separates
the words appearing on the left and right of the target word as 2 separate
embeddings. For this example, the left and right context embeddings would take
the form $v_{cl}=f(^{\prime}w_{the},w_{quick})$ and $v_{cr}=f(w_{jumps})$
respectively. After separation, our negative sampling objective will now be a
modified version of equation 2:

$\displaystyle F$
$\displaystyle=\log\sigma(v_{t}^{T}v_{cl})+\sum_{i=1}^{k}\log\sigma(-v_{nsi}^{T}v_{cl})+\log\sigma(v_{t}^{T}v_{cr})+\sum_{i=1}^{k}\log\sigma(-v_{nsi}^{T}v_{cr})$
(3.8)

We will be using this new objective function in 3.8 to train our left-right
context embeddings. While we change the training objective, we still use draw
the word embeddings from the same context embedding matrix $W_{c}$.

###  3.4 Hybrid Embeddings

Hybrid embeddings was a technique first proposed by Mai. These embeddings
involve training both their CBOW and CMOW embeddings concatenated together as
a context embedding. In their paper they argue that training the embeddings
together helps to remove redundancies which would occur if they were trained
separately and then concatenated afterward. The representation of their hybrid
context embedding $v_{ch}$is written as:

$v_{ch}=[v_{cmow};v_{cbow}]$

Where $v_{cmow}$ and $v_{cbow}$ are the context embeddings formed by their
respective operations. We will be using the same form of hybrid embeddings as
equation $3.9$, but replacing $v_{cbow}$ and $v_{cmow}$ with our own Word2rate
embeddings.

###  3.5 Word2rate Second Order Series

$\displaystyle v^{SOS}_{c}$ $\displaystyle=(I+\epsilon Q_{love}+\epsilon
Q_{We}+\epsilon^{2}Q_{love}Q_{We}+\frac{\epsilon^{2}Q_{love}^{2}}{2!}+\frac{\epsilon^{2}Q_{We}^{2}}{2!})p_{u}$
(3.10)

We can take all terms up to the second order from the Taylor series expansion
in equation 3 to form our Word2rate Second Order Series(SOS) represented in
equation 3.10. From the equation we can see that compared to our FOS
embeddings, there are exists the bilinear term $\epsilon^{2}Q_{love}Q_{We}$.
Also unlike our FOP embeddings this term is now explicit in nature. As we have
discussed earlier, having these bilinear interactions in matrix form allows us
to learn relationships between words. At the same time, our SOS context
embedding also contains the original terms from our FOS embeddings. Hence we
can say that it learns relationships between word embeddings using properties
from both CBOW and CMOW.

$\epsilon$ Length WC Depth TopConst Bshift Tense SubjNum ObjNum SOMO Coordinv
CBOW 25 - 24.8 23.0 21.8 20.5 50.1 71.8 74.0 75.1 50.0 51.5 CMOW 25 - 47.5
13.7 22.6 20.3 50.7 67.4 67.2 66.6 50.9 51.8 Word2rate FOS 25 0.01 59.0 18.5
24.2 23.8 50.5 75.2 74.7 76.0 49.6 51.8 Word2rate FOP 0.001 29.4 13.5 22.2
18.4 53.6 68.7 66.8 68.3 50.6 53.3 Word2rate FOP 25 lr 0.001 24.8 10.1 20.8
18.2 58.2 64.0 62.5 63.3 50.9 54.8 Word2rate SOS 25 0.001 66.7 8.9 23.9 18.7
50.7 68.9 66.2 68.1 49.2 55.1 CBOW 50 - 25.3 42.3 23.1 24.0 49.8 76.3 75.1
78.5 49.9 52.1 Hybrid FOS-FOP 50 0.0001 44.1 29.8 23.9 25.1 50.9 76.0 72.9
75.5 50.4 55.1

Table 1: Results of linguistic probing tasks from Senteval

$\epsilon$ MR CR MPQA SUBJ SST2 SST5 TREC MRPC SICK-E SICK-R STS-b CBOW 25 -
64.3 69.7 79.4 84.9 64.7 33.2 57.0 70.4 65.8 62.9 48.6 CMOW 25 - 59.0 64.3
77.0 74.2 59.5 30.2 56.4 67.1 62.7 54.8 24.6 Word2rate FOS 25 0.01 65.1 70.3
77.4 84.2 67.1 33.9 57.2 71.5 69.7 62.3 45.2 Word2rate FOP 25 0.001 60.0 63.8
69.2 77.7 60.6 30.4 54.0 66.5 58.0 52.3 45.5 Word2rate FOP 25 lr 0.001 57.2
63.8 70.9 73.6 55.1 26.2 48.8 66.5 58.3 50.5 38.3 Word2rate SOS 25 0.001 61.2
65.6 68.8 78.9 61.9 32.0 50.2 68.6 57.0 50.2 36.3 CBOW 50 - 68.1 72.4 82.6
85.9 70.5 35.6 59.2 71.4 70.6 65.1 53.4 Hybrid FOS-FOP 50 0.0001 63.9 71.0
79.3 83.3 65.7 34.1 59.4 70.5 72.7 63.9 53.1

Table 2: Results of downstream tasks from Senteval

##  4 Experiments

For our experiments, we will be training different modes of the Word2rate
Taylor series that we just described while comparing them to CBOW and CMOW. We
will also conduct various ablation studies on these embeddings. These
embeddings will be trained on 50 percent of the billion words dataset with
vocabulary consisting of words that occur at least 100 times and sentence
length between 10-20 words. This is equal to a corpus with 5982756 sentences
with vocabulary size 57025. We will be using the Adam([12]) optimiser with
learning rate 0.001, batch size 1000 and number of epochs equal to 10. We will
train embeddings of dimension 25 for CBOW, CMOW, FOS, FOP and SOS while
training 50 dimensional embeddings for our hybrid models. The window size for
the context words will be set to 4 while the size of the negative samples per
context-target pair will be set to 5. While the elements of the rate matrices
$Q$ will be altered during gradient descent, we enforce the rate matrix
conditions manually after each training step to make sure that they stay
continue to fulfil the conditions for a rate matrix. We will evaluate these
embeddings on a number of tasks, from Senteval. The Senteval framework that we
will be using consists of 10 linguistic probing tasks as well as 11 supervised
downstream tasks. The 10 linguistic probing tasks can be divided into 3
categories, namely surface information, syntactic information as well as
semantic information of a sentence. The 11 downstream tasks mostly involve
some kind of classification tasks, with datasets ranging from entity
recognition(TREC) to sentiment classification(CR, MR etc.). As we have
mentioned earlier, we will form the sentence embeddings the same way we train
our context embeddings. For tasks that are sensitive to word order and require
product, we use the assymmetric context window proposed by Mai, and also use
their initialisation for evaluating the CMOW embeddings. While CMOW randomly
selects 30 words as targets from each sentence, since we are using a
relatively smaller dataset we randomly select $n$ words for a sentence of
length $n$ as the target instead. We present our results for the linguistic
probing and downstream tasks in Tables 1 and 2 respectively.

###  4.1 Word2rate FOS

We first start by comparing our Word2rate FOS embeddings with CBOW since they
both use the same add operation in composing the context embedding. For the
linguistic probing tasks, we can see that our FOS embeddings outperforms CBOW
in eight out of 10 of the tasks, losing slightly in SOMO and noticeably in WC
by 4.5 percent. Where WC is a word memorisation tasks. FOS performs
significantly better in Length by about 30 percent, which involves predicting
the number of words a sentence possessed based on its sentence vector. For the
downstream tasks, FOS performs better in 7 out of the 11 tasks, while being
within 1 percent accuracy for the SUBJ and SICK-R tasks. One likely
explanation for the improve in performance could be due to the richer
representation of our FOS embedding as each $n$ dimensional embedding is
formed by a $n\times n$ rate matrix multiplied by the uniform distribution
$q_{u}$. Training this higher dimensional representation and then compressing
it to a lower dimensional embedding could explain the better performance for
the Length task.

On the flipside, one possible limitation of our FOS embeddings is in the
relatively higher training complexity compared to the regular CBOW embedding.
To produce a $n$ dimensional word embedding $q_{w}$ we also have to train its
corresponding rate matrix $Q_{w}$ which is $n\times n$ dimensions. Even though
training time is longer, the trained word2rate embeddings have the same
dimensionality as that of word2vec. As such there will not be an increase in
complexity when using them on downstream tasks. It is also important to note
that our FOS embeddings and CBOW use the same number of trainable parameters
for downstream tasks. The only difference is in the way we form the sentence
vector.

We have shown experimentally that our Word2rate FOS embeddings is competitive
and even outperforms CBOW in most linguistic probing and downstream tasks,
likely due to our compressed representation of rate matrix-vector pairs.

###  4.2 Word2rate FOP

Figure 1: Mean of absolute values of matrix elements

In the previous section, the value of the hyperparameter $\epsilon$ was kept
constant at 0.01. Since addition is linear, varying the $\epsilon$ does not
make a difference in performance which we show in the Appendix. However,
$\epsilon$ will have an impact in performance for FOP due to the nature of the
matrix product. In the case of Mai, there arises vanishing gradient problem
due to elements of the matrix initialising around a mean of 0. This is not
applicable to us as we add the identity to the rate matrix for each product
term. However, we still see an impact in $\epsilon$ as we can see in Figure 1.
This Figure is a plot of the mean absolute value of a rate matrix product
after being multiplied by number of times equal to sequence length. This is
the same experiment conducted by Mai, but using our FOP composition. From the
experiment we can see a clear trend that the magnitude of fluctuations
decreases with smaller values of $\epsilon$, being relatively stable at
$\epsilon=0.001$. Stable values is good for training as it provides more
stable gradient updates. Hence we will be using $\epsilon$ values of 0.001 or
lower for tasks involving matrix product.

####  4.2.1 Comparison with CMOW

We can next refer to Tables 1 and 2 and compare our FOP embeddings with CMOW
as both involve matrix product. For linguistic probing tasks, FOP outperforms
CMOW in 4 out of the 10 tasks, while bieng within 1 percent accuracy in 4
other tasks, WC, Depth, SUBJ and SOMO. However, it performs noticeably worse
in Length. This is in contrast to our FOS embeddings which outperforms both
CBOW and CMOW in the Length tasks. This seemingly contradicts our hypothesis
that a compressed representation would allow for better performing embeddings.
However, one possible reason for the drop in performance could be that our FOP
embeddings implicitly contains a multitude of terms, up to $2^{2m}$ with $m$
being the window size. This large number of terms could possibly be
interpreted as noise during training which could explain the dip in
performance. For the downstream tasks, FOP in 5 out of the 11 tasks while
being within 1 percent accuracy for 2 of the tasks. The largest drop in
performance is in MPQA with CMOW outperforming FOP by close to 9 percent,
where MPQA is an opinion polarity classification task. On the other hand, FOP
outperforms CMOW by more than 20 percent on STS-b, a semantic similarity
tasks. While we cannot pinpoint the exact reason for the variance in these 2
tasks, it could be due to the nature of the dataset, or the noise and
compressed representation factors we mentioned earlier. While there are some
differences in performance, most of the tasks are relatively even with CMOW,
and we have shown that our rate matrix-vector combination is comparable to the
matrix unrolling representation of CMOW.

####  4.2.2 Comparison with CBOW

In linguistic probing, FOP outperforms CBOW in 5 out of the 10 tasks. Notably,
FOP outperforms CBOW in Bshift and Coordinv by 2 and 3 percent respectively.
These 2 tasks are sensitive to word order, more specifically, Bshift involves
detecting when there is a swapping of words while Coordinv involves swapping
of phrases. Usual additive composition employed by CBOW is unable to detect
such changes as addition is invariant to order. This is one of the benefits of
matrix multiplication, which is sensitive to order and non-commutable. These
properties are also present in CMOW which also outperforms CBOW in these
tasks. FOP fairs much more poorly in downstream tasks, losing to CBOW across
all tasks. However, this is not surprising and a similar behavior can also be
observed with CMOW embeddings in Table 2. Furthermore, this is behavior is
also somewhat observed in the original CMOW paper by Mai, where his embeddings
lose to CBOW on 9 out of the 11 downstream tasks. This seems to be a property
of using multiplication to compose context embeddings, where addition seems to
have an advantage in downstream tasks. This could be due to the geometric
properties of addition, where the summed context embedding is similar to its
constituent words. This is opposed to matrix product which is more of a
transformation of the embedding from one space to another. To alleviate this
issue, Mai proposes using a hybrid training method of both CBOW and CMOW. We
will train a Hybrid FOS-FOP embedding with the same method and evaluate its
performance later on. For now, we will introduce our novel left-right context
split which we used to improve performance for our FOP embeddings in some
tasks.

####  4.2.3 Left-Right Context Split

We introduced our novel left-right context (lrcs) split previously in Section
3.3. We can see the performance under ‘Word2rate FOP 25 lr’ in Tables 1 and 2.
We will be comparing its performance with the usual FOP embeddings in the
previous section. For the linguistic probing tasks, we can see that our lrcs
embeddings outperforms FOP in 3 of the 11 tasks, namely Bshift, SOMO and
Coordinv. Bshift and Coordinv notably are sensitive to word order, and our
lrcs embeddings outperforms FOP by 4.6 and 1.5 percent on these two tasks
respectively. This shows that adding the left-right context split does improve
performance of word order due to preserving syntactically correct sequences.
However, our lrcs embeddings does overall perform worse in both linguistic
probing and especially downstream tasks, performing only slightly better or
tieing in 4 out of the 11 tasks.

We think there could be 2 possible reasons for the drop in performance
compared to FOP. Firstly, a left-right split means the embeddings learn a
shorter sequence length in some cases. If the target word is at the edge of
the window, there is no difference. However, if there target word is in the
middle then the context embedding formed without the split would be longer
compared to a left-right split. Training a longer context embedding instead of
two shorter ones may result in better performance in downstream tasks.

Secondly, while forming a sentence embedding for downstream tasks, there are
out of vocabulary words present depending on the tasks and respective dataset.
These out of vocabulary words are typically padded with an identity matrix.
However, a sentence formed with out of vocabulary words could also be
syntactically inaccurate. Consider the previous example of ‘the quick fox
jumps’. We have illustrated earlier that without the left-right context split
the model would learn the context embedding ‘the quick brown jumps’. However,
if the word ‘fox’ were to be replaced by an out of vocabulary word like
‘florpus’ for the downstream task, then the resultant sentence vector would
also be ’the quick jumps’. This negates some of the potential impact of
performing the left-right split.

###  4.3 Hybrid FOS-FOP

We evaluated our Hybrid FOS-FOP against CBOW, CMOW and FOP in 50 dimensions to
see if they would overcome the limitations of our matrix product model for
FOP, especially for the downstream tasks. Firstly we can see for our
linguistic probing tasks that our Hybrid embeddings outperform CBOW in 5 out
of 10 tasks. As usual our Hybrid model is better than CBOW in Bshift and
Coordinv. However, it is worth noting that the accuracy for Bshift is lower
than our FOP embeddings by 2.7 percent. Training a mixture of FOS and FOP
could possibly negate part of the benefits of either model.

We also compared our Hybrid embedding with CMOW and FOP at 50 dimensions. We
can see that having a Hybrid model does not degrate its performance in
linguistic probing task. From Table 1 we can see that our Hybrid model
outperforms FOP in 8 out of 10 tasks and CMOW in 7 out of the 10 tasks.
Compared to CMOW, the tasks Hybrid loses on differ in accuracy by close to 1
percent. For CMOW, Hybrid fairs worse notably in Bshift losing by 2.7 percent.
This is likely due to the mixture of linear and product operations reducing
the signal of word sensitivity from the matrix products.

For the downstream tasks, we can see that our Hybrid embeddings doing better
than FOP and CMOW across all tasks, while beating CBOW in 2 tasks and being
within 1 percent accuracy in 2 other tasks. This shows that training a Hybrid
model does help strengthen the performance in tasks where each individual
embedding is lacking. However, it could also deter its performance in tasks
where they are strong as well. The expanded version of FOP contains all terms
from FOS implicitly. This could result in a repetition during training. To
investigate this we trained a Hybrid FOS-FOP minus embedding where we
subtracted the FOS terms from FOP. We will discuss this further in the
Appendix.

###  4.4 Word2rate Second Order Series

Our Word2rate SOS embeddings contain both addition and product in the
formation of the context embedding while containing bilinear interactions
between rate matrices. Even though there are implicit bilinear interactions in
FOP, there are also many additional terms higher than the second order.
Limiting ourselves to only second order terms allows us to investigate the
impact of the high order terms as well. Since SOS contains both addition and
product, we will be comparing it with the respective CBOW and CMOW embeddings.

####  4.4.1 Comparison with CBOW

For the linguistic probing tasks, we can see that SOS outperforms CBOW in 4
out of 10 tasks. SOS still retains the properties of our usual word2rate
embeddings, outperforming CBOW in Length by a large margin of more than 40
percent. It also fairs better in both Bshift and Coordinv. This is likely due
to the bilinear interactions of the second order terms of the rate matrix.
However, similar to our hybrid embeddings it fairs better for Coordinv but
worse for Bshift compared to our FOP embeddings. The drop in performance for
Bshift is likely due to the presence of linear terms which are invariant to
word order. However, we are unsure of why these embeddings do perform better
for Coordinv. One possible reason could be that Coordinv measure phrase rather
than word order and having addition as part of phrase formation has a positive
impact. For the downstream tasks we observe that CBOW outperforms SOS in all
tasks. The bilinear product terms are now the ones causing a dip as we observe
addition only for CBOW to perform better overall for the downstream tasks.

####  4.4.2 Comparison with CMOW

Our SOS embeddings are roughly equal to CMOW for the linguistic probing tasks,
with SOS outperforming SOS in 5 out of the 10 tasks while drawing in 1 of them
with most accuracies being fairly close. Our SOS embeddings continue to have
good performance in Length, while losing out to both CBOW and CMOW in WC which
is a memorisation tasks. This drop in performance in WC is not due to using
bilinear terms as our FOP embeddings have roughly the same result as CMOW. It
could be likely that having a mixture of linear and bilinear terms is not that
good for memorisation.

We see a marked increase in performance of SOS against CMOW for the downstream
tasks. Our previous FOP embeddings are only better in 5 out of 11 tasks
compared to CMOW. However SOS outperforms CMOW in 7 out of the 11 tasks. This
increase is likely due to the presence of the addition operation. We have also
previously discussed that the drop in performance of Length for our FOP
embeddings is due to the multitude of high order terms. We can see that this
may be the case as our SOS embeddings with only bilinear interactions actually
is the best performing in the Length tasks, beating both FOS and FOP. This is
interesting as we have earlier discussed along with our Hybrid embedding that
addition and product together helps each others weakness while likely
decreasing their strength in certain tasks. Length is an example where having
both addition and product increases overall performance.

##  5 Conclusion

In conclusion, we have show that using our novel Word2rate FOS, FOP, SOS as
well as Hybrid embeddings are effective feature representations and that are
competitive with existing embeddings on a number of tasks. In other words, it
is feasible to model word embeddings as a series of transitions from an
initial distribution. We have shown how these word embeddings can be formed by
taking the first order, second order series or product approximation of our
Taylor expansion of rate matrices. The FOS embeddings have shown comparable
performance to the existing word2vec model on a number of linguistic probing
as well asdownstream tasks. We have also shown that FOP embeddings, while
having some limitations, has shown promising properties that make it a better
candidate than the word2vec addiitive composition model when forming phrases.
Their sensitivity to word order is also useful in linguistic probing tasks
such as Bshift and Coordinv where they outperform CBOW.

We also introduce a novel left-right context split of the context embedding
that further improves the performance of such tasks. While our model does
share some similarities with CMOW, our model uses rate matrices rather than
random matrix which are constrained in their diagonal elements. This allows us
to ground our embeddings in a statistical foundation. Our embeddings are also
formed from a combination of rate matrix-vector pair instead of the unrolling
operation employed by CMOW.

At the same time we have shown that combining our Word2rate FOS and FOP
embeddings as a hybrid model allows us to show comparable performance to CBOW
embeddings with the same dimensions on the downstream tasks. On the other
hand, the second order series Taylor expansion seems to be a mixture the first
order series and the first order product, sharing both sum and product matrix
operations. While it has shown to have outperform both CBOW and CMOW in some
tasks, it does not perform as well as training the sum and product terms
individually like in our FOS and FOP embeddings.

In essence, we have proposed a novel way of representing a language model and
in a deeper sense a model of how we form concepts in our mind. This is
modelled as a series of statistical transitions from an initial distribution
can be interpreted as the evolution of our thoughts.

## References

  * [1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [2] Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning.  Bilingual word embeddings for phrase-based machine translation.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1393–1398, 2013.
  * [3] Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio.  Context-dependent word representation for neural machine translation.  Computer Speech & Language, 45:149–160, 2017.
  * [4] Liang-Chih Yu, Jin Wang, K Robert Lai, and Xuejie Zhang.  Refining word embeddings for sentiment analysis.  In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 534–539, 2017.
  * [5] Maria Giatsoglou, Manolis G Vozalis, Konstantinos Diamantaras, Athena Vakali, George Sarigiannidis, and Konstantinos Ch Chatzisavvas.  Sentiment analysis leveraging emotions and word embeddings.  Expert Systems with Applications, 69:214–224, 2017.
  * [6] Hakan Demir and Arzucan Özgür.  Improving named entity recognition for morphologically rich languages using word embeddings.  In 2014 13th International Conference on Machine Learning and Applications, pages 117–122. IEEE, 2014.
  * [7] Florian Mai, Lukas Galke, and Ansgar Scherp.  Cbow is not all you need: Combining cbow with the compositional matrix space model.  arXiv preprint arXiv:1902.06423, 2019.
  * [8] Alexis Conneau and Douwe Kiela.  Senteval: An evaluation toolkit for universal sentence representations.  arXiv preprint arXiv:1803.05449, 2018.
  * [9] Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni.  What you can cram into a single vector: Probing sentence embeddings for linguistic properties.  arXiv preprint arXiv:1805.01070, 2018.
  * [10] Jeff Mitchell and Mirella Lapata.  Composition in distributional models of semantics.  Cognitive science, 34(8):1388–1429, 2010.
  * [11] Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng.  Semantic compositionality through recursive matrix-vector spaces.  In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pages 1201–1211. Association for Computational Linguistics, 2012.
  * [12] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.