# Texture Based Classification of High Resolution Remotely Sensed Imagery
using Weber Local Descriptor

Decky Aspandi Latif
Department of Computer Engineering
King Mongkut’s University of Technology Thailand
&Sally Goldin
Department of Computer Engineering
King Mongkut’s University of Technology Thailand
&Preesan Rakwatin
Research and Development Group
Geo-informatics and Space Technology Development Agency (GISTDA)
&Kurt Rudahl
Department of Computer Engineering
KMUTT Geospatial Engineering and InnOvation Centre (kGeo), Geoinformation
King Mongkut’s University of Technology Thailand
corresponding author<EMAIL_ADDRESS>
###### Abstract

Traditional image classification techniques often produce unsatisfactory
results when applied to high spatial resolution data because classes in high
resolution images are not spectrally homogeneous. Texture offers an
alternative source of information for classifying these images. This paper
evaluates a recently developed, computationally simple texture metric called
Weber Local Descriptor (WLD) for use in classifying high resolution QuickBird
panchromatic data. We compared WLD with state-of-the art texture descriptors
(TD) including Local Binary Pattern (LBP) and its rotation-invariant version
LBPRIU. We also investigated whether incorporating VAR, a TD that captures
brightness variation, would improve the accuracy of LBPRIU and WLD. We found
that WLD generally produces more accurate classification results than the
other TD we examined, and is also more robust to varying parameters. We have
implemented an optimised algorithm for calculating WLD which makes the
technique practical in terms of computation time. Overall, our results
indicate that WLD is a promising approach for classifying high resolution
remote sensing data.

_K_ eywords Texture Classification $\cdot$ Remote Sensing $\cdot$ Image
Processing

##  1 Introduction

As high resolution satellite imagery becomes more common, remote sensing
applications can benefit from previously unavailable levels of detail.
However, many traditional digital image analysis techniques produce
unsatisfactory results when applied to high spatial resolution data [1]. In
particular, classification algorithms such as maximum likelihood which are
based on per-pixel spectral information often perform poorly [2, 3]. At high
spatial resolution, areas belonging to a particular class do not necessarily
have homogeneous spectral characteristics. That is, adjacent pixels frequently
have very different colour or brightness.

Texture provides an alternative information source for classifying high
resolution imagery. High resolution images contain rich geometric structure
with repeating variations in brightness, which may be regular within regions
[4]. Texture based classification incorporates information about the
distribution of brightness or colour in the neighbourhood of the pixel of
interest into the classification decision process. Many approaches for
classifying images based on texture have been investigated, including Grey
Level Co-occurrence Matrix (GLCM)[5], Wavelet-based Transform [6, 7], Gabor
Filter [8, 9, 10] and Scale Invariant Feature Transform (SIFT) [11, 12]. These
techniques have shown promise but are rarely practical in operational contexts
because all are very computationally intensive.

Recently, several less computationally complex texture descriptors (TD) have
been proposed, all of which compute a measure of variation in the immediate
neighbourhood of the focus pixel, then construct a histogram of these measures
within a somewhat larger window. The Local Binary Pattern (LBP) texture
descriptor introduced by [13] describes the local characteristics of the
texture by comparing the centre pixel with its neighbours, then coding each
comparison as either one (centre pixel is brighter than its neighbour) or zero
(centre pixel is darker). This produces a small, fixed number of possible
patterns of zeros and ones; the histogram of these patterns within a larger
neighbourhood becomes the texture value for the focus pixel. This method has
gained considerable support and has been successfully applied in texture
analysis [14], in face recognition [15], and in image segmentation [16].
However, LBP is limited due to the small size of the moving window. It is also
not rotation-invariant and it cannot discriminate between textures defined by
varying contrast. The first two problems can be addressed by combining several
neighbourhood sizes to create a multi-resolution descriptor, and by adding
rotation invariance to create the LBPRIU descriptor [17]. Musci et al. [18]
created a contrast sensitive LBPRIU by concatenating its histogram with the
histogram from the Variance Texture Descriptor (VAR), creating the LBPRIU_VAR
descriptor. Although these efforts have produced encouraging results, the
enhancements add to the computational costs, and thus reduce the simplicity
that makes LBP so attractive.

A new texture descriptor called Weber Local Descriptor (WLD) was presented by
in [19]. This TD uses ratios rather than simple comparisons and captures two
aspects of texture, gradient orientation and central pixel value
differentiation. WLD has been shown to rival some other classification methods
based on texture such as SIFT and Gabor Filter [19]. To the best of our
knowledge, few remote sensing studies have utilised WLD and none has focused
on texture classification in high resolution imagery. In this research, we
apply the WLD method to remotely sensed images, employing a supervised pixel-
wise classification approach based on a minimum distance metric. We compare
the classification accuracy of WLD with LBP and several variants and examine
the effects of neighbourhood size. As baselines, we also perform
classifications on our test data using a classical GLCM-based texture approach
and a traditional, non-texture-based maximum likelihood classification (MLC).
Additional experimental conditions test the effectiveness of the Variance TD
texture descriptor (VAR), both independently and combined with WLD, in images
with significant illumination change within classes. Finally, we create
optimised versions of the LBP and WLD algorithms which compute classified
results sufficiently quickly to make these techniques practical for
operational remote sensing applications.

##  2 Methodology

###  2.1 Image Data

In this study, we used a panchromatic QuickBird image with 0.6 m spatial
resolution acquired on 3 January 2007 and provided by the Geo-Informatics and
Space Technology Development Agency (GISTDA). The image covers part of
Pathumthani province in Thailand located at (14.083646 N,100.650593 E), WGS
1984 datum. Two 1024 x1024 pixel regions were extracted as test areas. Both
regions include crop areas, naturally vegetated areas, barren areas and water
bodies. Region 1 includes residential areas which represent artificial
textures. Region 2 has texture areas affected by illumination changes that can
be utilised to evaluate the potential benefits of introducing the VAR TD. We
deliberately selected test areas with different characteristics to explore how
this would affect accuracy. The test areas are shown in Figure 2.

Figure 1: Study areas, (a) Region 1 and (b) Region 2.

###  2.2 Reference Data

Given the time lapse since image acquisition, we did not expect ground surveys
to yield accurate reference data. Instead, in order to create reference data
for evaluation of classification accuracy, we asked an independent expert
analyst with three decades of experience to locate and label major classes in
each region using visual interpretation of both panchromatic and multispectral
(2.4 m per pixel) QuickBird images. The expert identified ten classes based on
various aspects of the image appearance, including colour, texture and spatial
context, by outlining polygons surrounding each area representing a class. In
total, there were 44 and 47 reference polygons identified for Region 1 and
Region 2 respectively.

Some of the expert’s ten classes exhibited very inconsistent textures while
others were too small to provide both training and test pixels. For our
experiments, we selected four texture classes per region for a total of five
texture classes. Table 1 shows the selected texture classes for each region,
with their pixel counts and other statistics.

No. | Texture Class | Region 1 | Region 2
---|---|---|---
Pixel Count | Mean | SD | Pixel Count | Mean | SD
1. | Residential | 115142 | 49.33 | 14.81 | 0 | - | -
2. | Orchard | 176459 | 39.75 | 9.24 | 74720 | 39.55 | 10.35
3. | Water | 67534 | 27.42 | 9.28 | 147822 | 17.73 | 4.71
4. | Barren | 100061 | 47.63 | 8.59 | 121952 | 39.48 | 7.64
5. | Crop | 0 | - | - | 145214 | 33.96 | 5.78
Table 1: Statistics of Texture Classes.

###  2.3 Training Samples

We extracted training samples for each class by defining square areas within
texturally homogeneous regions of the study images. Within each training area
we calculated the pixel-wise TD for each pixel, then assembled the histogram
of TD values as the standard model for that class. We considered three sample
sizes: 30 x 30, 40 x 40 and 50 x 50.

To determine the best training area size, we conducted a preliminary
classifications using each size using the state-of-the-art LBPRIU TD [17] and
selected the sample size with the highest accuracy. Training areas of 50 x 50
(2500 pixels) provided the best accuracy for region 1; 30 x 30 (900 pixel)
training areas were best for region 2. The difference reflects the fact that
texture units in region 1 tend to be larger than those in region 2. We chose
four training areas of the appropriate size for each class, resulting in 10
000 training pixels per class for Region 1 and 3 600 training pixels per class
for Region 2. TD values for all training pixels for a texture class were
combined into a single histogram.

###  2.4 Texture Descriptors

All the primary TD features tested in this study have a similar structure.
They require two main parameters: the radius of the neighbourhood around the
focus pixel (R) and the window size for aggregating individual pixel TD values
into a histogram (termed ’ring size’ in the literature). An R value of 1 means
that the neighbourhood will involve the pixel’s eight immediate neighbours,
hence this situation is referred to as ’P:8,R:1’. A ring size of 40 means that
each histogram will contain 40 x 40 or 1600 pixels.

The TD value for a particular pixel is calculated by comparing its brightness
value with its neighbours to calculate one or two (in the case of WLD)
numerical values. The texture within the window is characterised by creating a
histogram that records the frequencies of each TD value, for all pixels in the
window. The different TD have different methods for calculating the value(s)
for an individual pixel.

####  2.4.1 Local Binary Pattern (LBP)

The LBP value for the centre pixel in a neighbourhood is calculated using the
following equation [13]:

$LBP_{P,R}=\sum_{i=0}^{p-1}u(t_{i}-t_{c})x2^{i}$ (1)

where P is the total number of neighbouring pixels and R is the radius used to
form the circularly symmetric set of neighbours. When we are examining a
pixel’s immediate neighbours, P = 8, R = 1, ti is the brightness of a
neighbour pixel while tc is the centre pixel brightness. The function
$u(t_{i}-t_{c})$ is a step function, where $u(x)=1$ when $x\geq
0;else,u(x)=0$. The formula produces an integer between 0 and 2P-1. If the
values of R and P increase, changing the effective spatial resolution of the
descriptor, the maximum possible value of the TD and hence the number of bins
in the window histogram also increases. The power of 2 term in equation (1)
means this increase is exponential.

####  2.4.2 Local Binary Pattern Rotation Invariant (LBPRIU)

The binary pattern produced by LBP depends on the pixel values in each spatial
neighbourhood. When the image is rotated, the pixel neighbourhood is also
rotated. This will cause the binary pattern to move along the perimeter of the
circle around the centre pixel. Hence, the LBP histogram for a texture will
change if that texture is rotated, which may not be desirable.

LBPRIU [17] eliminates this rotation effect by grouping LBP values according
to the pattern of transitions between 0 and 1 in the binary representation of
the value. For example, 000001002 and 000000102 will be mapped to the same
LBPRIU group because there are two transitions, from 0 to 1 and 1 to 0, for
both of these patterns. Only the position of the transitions differs, a
variation that would result from rotating a texture. The mapping procedure
uses the equation below:

$LBP_{P,R}^{riu2}=\binom{\sum_{p=0}^{p-1}s(t_{p}-t_{c})\;if\;U(LBP_{P,r})\leq
2)}{P+1\;otherwise,}$ (2)

P and R represent the number of pixels and radius of the neighbourhood as
before. The function s is the signum fuction, $s(x)=1ifx>=0,else\;s(x)=-1$.
The uniformness function U is defined by equation (3).

$U(LBP_{P,R})=(s(t_{p-1}-t_{c})-s(t_{0}-t)c))+sum_{p=1}^{p-1}(s(t_{p}-t_{c})-s(t_{p-1}-t_{c}))$
(3)

The value generated by U depends on the number and pattern of binary digit
transitions. The mapping procedure implemented by LBPRIU collects LBP values
with U <= 2 into P distinct classes. LBP values with U > 2 are grouped into
one miscellaneous class labelled as (P+1). The histogram for the window
reflects the counts of pixels in each class. The number of bins in the LPBRIU
histogram increases with the neighbourhood size, but not exponentially as in
LBP.

####  2.4.3 VAR and LBPRIU_VAR (Variance)

Both the LBP and the LBPRIU descriptors are invariant to monotonic grey scale
changes and consequently do not capture either absolute brightness or contrast
information. Ojala et al. [17] proposed a local contrast descriptor, denoted
as $VAR_{P,R}$, which is also rotation invariant, defined as:

$VAR_{P,R}(w)=\frac{1}{P}\sum_{p=0}^{p-1}(t_{p}-u)^{2}$ (4)

where,

$u=\frac{1}{P}\sum_{p=0}^{p-1}t_{p}$ (5)

$VAR_{P,R}$ is an approximation of local variance, which can be computed
efficiently if performed concurrently with the computation of $LBPRIU_{P,R}$ .
LBPRIU and VAR can be combined to produce the LBPRIU_VAR texture descriptor,
an extension of LBPRIU that also captures the contrast feature of texture.
This can be achieved by concatenating the VAR histogram with the LBPRIU
histogram [18].

####  2.4.4 Weber Local Descriptor (WLD

Weber Local Descriptor (WLD) was introduced b [19]. This descriptor is
inspired by Weber’s Law in physiological psychology, which states that the
minimum distinguishable change of a perceptual stimulus is a constant fraction
of the original stimulus. When the change is smaller than this constant, a
human being would consider the stimulus as background noise rather than a
valid signal. WLD adopts this idea by using a ratio to represent differences
in the values of neighbouring pixels.

WLD consists of two components: differential excitation $(\xi)$ and
orientation $(\theta)$. $\xi$ is a function of the Weber fraction (i.e., the
intensity differences of a pixel’s neighbours relative to its own intensity).
$\theta$ is a gradient orientation of the current pixel. The excitation is
defined by:

$\xi(x_{c})=arctan\begin{bmatrix}\sum_{i=0}^{n-1}\frac{I_{i}-I_{c}}{I_{c}}\end{bmatrix}$
(6)

where $n$ is the number of neighbouring pixels and $I_{i}$ is pixel value at
position i. Pixels are numbered clockwise starting with 0 at the upper left.
The gradient orientation is defined by:

$\theta(x_{c})=arctan\begin{bmatrix}\sum_{i=0}^{n-1}\frac{I_{7}-I_{3}}{I_{5}-I_{1}}\end{bmatrix}$
(7)

Based on these two terms, a two dimensional joint histogram can be
constructed, followed by conversion to a one dimensional histogram, which is
the WLD descriptor for a window. For details of the conversion process see
[19] and [20].

####  2.4.5 Texture based pixel-wise supervised classification

We used texture based pixel-wise supervised classification to classify each
pixel into a particular group by considering the texture in its neighbourhood.
The histograms of the training areas described in section 2.3 constitute the a
priori definitions of our classes. Each pixel in the image is classified by
calculating the histogram of TD values within a window centred on that pixel,
calculating a similarity measure against the training histogram for each class
and placing the pixel into the most similar class [21]. Based on the
literature plus initial experiments, we chose the Bhatacharya distance as our
similarity metric. Bhatacharya distance measures the distance or difference
between two histograms. In remote sensing this metric is often used to assess
spectral separability of classes [22], but it can also measure similarity
between the TD histogram calculated for a pixel and the TD histogram for a
training sample [23]. Bhatacharya distance is calculated as follows:

$BD(H^{1},H^{2})=-ln\left(\sum_{x=x}\sqrt{H^{1,i}\;\times\;H^{2,i}}\right)$
(8)

$H^{1,i}$ and $H^{2,i}$ are the counts for bin i in the first and second
histogram respectively. This distance metric has moderate computational
complexity and produces bounded values between 0 to 1, where smaller values
imply greater similarity between two histograms.

##  3 Experiments

###  3.1 Experimental Settings

Our main experiments varied the following conditions:

  1. 1.

Texture descriptor: LPBP, LBPRIU, WLD, VAR, LBPRIU_VAR and WLD_VAR. The first
three TD do not capture contrast differences between texture classes. The
second three attempt to involve contrast features by incorporating variance.
WLD_VAR is a new TD first investigated by Latif [20] that combines WLD and VAR
into a single histogram, analogous to LBPRIU_VAR.

  2. 2.

Scale: We calculated each TD at several scales (P:8,R:1; P:16,R:2; P:24,R:3)
to examine whether larger scales might generate higher classification accuracy
as has been reported by some authors. We also examined multi-resolution
classification, by concatenating the histograms of all scales in a single
classification.

We used confusion matrices to calculate the overall accuracy and Kappa
coefficient [24] for each classification condition. After removing pixels used
for training, all pixels labelled by the expert image analyst in each of our
selected classes were used as test pixels. Finally, we also recorded the
elapsed time for each experimental condition.

###  3.2 Tools

The texture-extraction and classification software used in this study was
written by the first author in GNU C++ using the Dragon Programmer’s Toolkit™.
Experiments were executed on an Intel Centrino dual core processor 2.1 Ghz,
with 4 GB of RAM, running Ubuntu Linux 12.04. Dragon/ips® [25] was used to
perform standard remote sensing procedures, such as defining test areas,
clipping, calculation of accuracy, image display, etc.

###  3.3 Experimental Results

####  3.3.1 LBP and LBPRIU versus WLD: general accuracy comparison.

Tables 2 and 3 present the overall classification accuracy using LBP, LBPRIU,
and WLD texture descriptors on regions 1 and 2 respectively, for various
scales. The Kappa coefficients show a similar pattern, although their values
as expected are lower than overall accuracy.

P,R | LBP (%) | LBPRIU (%) | WLD (%)
---|---|---|---
8,1 | 73.94 | 62 | 84.55
16,2 | 83.59 | 65.01 | 82.05
24,3 | 70.48 | 60.1 | 73.29
Multi-scale | 84.49 | 66.52 | 81.76
Mean | 78.13 | 63.41 | 80.41
Table 2: Overall percent correct classification from LBP, LBPRIU and WLD on region 1. P,R | LBP (%) | LBPRIU (%) | WLD (%)
---|---|---|---
8,1 | 66.91 | 64.33 | 67.63
16,2 | 52.99 | 60.84 | 68.49
24,3 | 35\. 99 | 51.01 | 62.73
Multi-scale | 62.13 | 60.03 | 71.56
Mean | 54.51 | 59.05 | 67.60
Table 3: Overall percent correct classification from LBP, LBPRIU and WLD on
region 2.

These tables show that WLD consistently outperforms both generic LBP and
LBPRIU with the exception that in region 1 basic LBP with the spatial
resolution of 2 and multi-scale are slightly more accurate than WLD. In these
resolutions, however, LBP is extremely slow.

Note that the accuracies of all TD are better for region 1 than region 2. We
believe this is due to the fact that the illumination in the second region is
less consistent. This makes the classification problem harder than in the
first region where textures are relatively homogeneous in terms of brightness.
Another notable result is that adding rotation invariance to LBP to create
LBPRIU does not provide any consistent improvements on accuracy, especially in
region 1, where basic LBP produces better accuracy than LBPRIU overall. In
region 2, LBPRIU sometimes produced more accurate results than LBP. This may
be due to the fact that most of the textured areas have similar orientations.
In region 2, LBPRIU sometimes produced more accurate results than LBP, but was
not consistently superior.

####  3.3.2 LBP and LBPRIU versus WLD: multi-resolution analysis

Tables 2 and 3 also allow us to examine the accuracy of LBP, LBPRIU and WLD
when applied at different scales. Although accuracy does vary with scale,
there are few clearfew consistent patterns within a TD. Generally, the spatial
resolution of 1 for each TD produces more accurate results than other scales.
Moreover, in contrast to the reported results in [19] we found no improvement
with the multi-scale version of WLD compared to the single scale version.
Instead the multi-scale results are close to the mean results across scales
for each TD.

We can make other observations. First, WLD provides more consistent results
across different resolutions. In other words, WLD is less sensitive to
variation of parameters. This robustness is a desirable feature. Second, the
accuracy of different resolutions very likely depends on the characteristics
of textures in the specific images. Thus it is possible that scales greater
than 1 would provide improved accuracy for images with larger texture units.

###  3.4 The effect of variance on WLD and LBPRIU: general accuracy
comparison.

Tables 4 and 5 compare the overall accuracy for the single TD metrics LBPRIU,
WLD and VAR, plus the compound metrics LBPRIU_VAR and WLD_VAR. The VAR TD by
itself is capable of classifying images quite accurately, with a mean of about
67% for both regions. However in region 1, at least, WLD continues to provide
higher accuracy in most cases. Furthermore, combining LBPRIU with VAR
noticeably increases accuracy compared to plain LBPRIU, by more than 10% in
some cases. This result agrees with [18]. However, in general VAR makes no
contribution to WLD, at least in region 1. In region 2, where the image data
exhibit illumination changes within texture regions, VAR slightly enhances the
overall WLD results in all scale, with the maximum increase in accuracy of
about 7

P,R | LBPRIU (%) | WLD(%) | VAR(%) | LBPRIUVAR(%) | WLDVAR(%)
---|---|---|---|---|---
8,1 | 62 | 84.55 | 69.68 | 69.35 | 79.13
16,2 | 65.01 | 82.05 | 66.87 | 69.42 | 80.53
24,3 | 60.1 | 73.29 | 66.63 | 71.22 | 79.85
Multi-scale | 66.52 | 81.76 | 68.21 | 70,36 | 80.59
Mean | 63.41 | 80.41 | 67.85 | 70.09 | 80.04
Table 4:  Percent correct classification of region 1 using several TD combined with VAR. P,R | LBPRIU(%) | WLD(%) | VAR(%) | LBPRIUVAR(%) | WLDVAR(%)
---|---|---|---|---|---
8,1 | 64.33 | 67.63 | 68.25 | 68.37 | 75.34
16,2 | 60.84 | 68.49 | 66.23 | 66.41 | 70.69
24,3 | 51.01 | 62.73 | 62.59 | 62.92 | 65.18
Multi-scale | 60.03 | 71.56 | 66.05 | 66.17 | 71.60
Mean | 59.05 | 67.60 | 65.78 | 65.97 | 70.70
Table 5: Percent correct classification of region 2 using several TD combined
with VAR.

####  3.4.1 The effect of variance on WLD and LBPRIU: multi-resolution
analysis.

We can also consider the accuracy of WLD, LBPRIU, VAR, LBPRIU_VAR and WLD_VAR
when applied at different scales. Tables 4 and 5 show patterns similar to the
previous multi-resolution analysis. Using different spatial scales affects
accuracy, but not in any consistent way. The multi-scale version is not more
accurate than the single scale classifications, but generates the results
close to the mean of all resolutions.

###  3.5 Comparison with GLCM and maximum likelihood classification

To further examine the performance of WLD and the other histogram-based TDs,
we classified our test data using two additional methods. Specifically, we
conducted a texture-based classification using the classical GLCM approach. We
also performed a traditional, non-texture-oriented Maximum Likelihood
classification.

Software for the GLCM classification was implemented by the first author using
the tools described in Section 3.2. We selected GLCM parameters values based
on the most commonly reported values in the remote sensing literature [26, 27,
28, 26]. We used four orientation angles (0, 45, 90, 135), seven GLCM
statistics ( Entropy, Energy, Homogeneity, Dissimilarity, Variance, Shade ,
Correlation and Contrast), a window size of 7x7, GLCM quantization levels of
32x32 and 64x64, and distance values 1, 2 and 3. MLC was performed using
Dragon/ips®. Since this method requires multiple bands, we used the four
multispectral bands of QuickBird data that correspond to the panchromatic
image in our primary experiment. Training signature statistics (mean, variance
and covariance) were calculated from the same training pixels used in
experiments on the primary TDs.

Table 6 shows the results of these traditional classification approaches,
compared to the average results from the primary TDs, for regions 1 and 2. The
accuracies for both GLCM and MLC Maximum Likelihood are quite high in Region 1
but lower in Region 2. This is consistent with the results from earlier
experiments. However, as expected, the traditional methods produce less
accurate results than the primary TDs of interest in every case, with a
maximum difference of about 10% for MLC and 13% for GLCM compared with the
best accuracy results of each TD. Full details of all results reported in this
article (corresponding confusion matrix, omission errors, etc.) can be found
at https://sites.google.com/a/mail.kmutt.ac.th/upload/.

It is important to note that the spatial resolution of the data used in the
maximum likelihood classification (2.4 m multispectral Quickbird) is lower
than that used for the texture-based classifications (0.6 m panchromatic
Quickbird). Each pixel in the multi-spectral bands corresponds to nine pixels
in the panchromatic image. This should favour maximum likelihood, since the
influence of within-class spectral diversity will be reduced to some extent
compared to the higher resolution. Nevertheless, our results suggest that we
might improve overall accuracy by incorporating spectral information into our
texture descriptors.

Overall Accuracy | Region 1 % correct (Kappa) | Region 2 % correct (Kappa)
---|---|---
GLCM | 71.53 (0.60) | 61 (0.48)
Maximum Likelihood | 74.48 (0.65) | 62.29 (0.49)
LBP | 78.13 (0.71) | 54.51 (0.37)
LBPRIU | 63.41 (0.50) | 59.05 (0.46)
WLD | 80.41 (0.73) | 67.60 (0.57)
VAR | 67.85 (0.56) | 65.78 (0.55)
LBPRIU_VAR | 70.09 (0.59) | 65.97 (0.55)
WLD_VAR | 80.04 (0.72) | 70.70 (0.61)
Table 6:  Result for GLCM and maximum likelihood in region 1 and region 2
compared with average result of earlier TDs classification.

###  3.6 Computation time

Figure 2: Comparison of Pixelwise classification running time between original
and optimised texture descriptors.

Our initial implementation of TD algorithms, reported in Latif [20], was a
naive translation from equations logical structure to code. We found that the
computation time for the different TD varied widely. In particular, LBP was
the fastest TD at scale 1, but became impractically slow at higher
resolutions. We attributed this to the exponentially increasing number of bins
in the LBP histogram at higher scales, all of which must be scanned during
similarity computations. The computation time for LBPRIU did not grow in the
same manner, partly because the LBPRIU remapping produces a small number of
histogram bins regardless of the scale. WLD was somewhat slower than LBP or
LBPRIU at scale 1, but provided more consistent run times across scales and
regions.

Leaving out the multi-resolution case, the average run time for WLD was 4 004
s for region 1 and 2 865 s for region 2. If we extrapolate from our 1024 x
1024 subsets to a more typical image size of 6000 x 6000 pixels, the
processing time for a texture based classification using WLD would be more
than 100 000 seconds (about 30 hours), which is not really practical for an
operational environment.

However, when we reviewed our code, we noted that our original implementation
repeated many calculations.. We redesigned our software to cache various
intermediate values. Our optimisations produced a dramatic speed-up, as shown
in Figure 2. For WLD, for instance, the classification time dropped to about
52 s for a 1024 x 1024 region. This translates to about half an hour for a
full scene. Figure 2 shows only the P:8,R1 condition, but results were
comparable for other resolutions.

This calculation time is much faster than GLCM-based classification which
required about 750 s and 2 400 s for region 1 and region 2 respectively. Note
that our implementation of GLCM was optimised using similar caching strategies
similar to those employed for other algorithms. Hence, our optimised
implementation of WLD provides both high accuracy and relatively good
computational performance. Further performance gains can probably be achieved
by exploiting the inherent parallelism in the pixel-wise computations.

##  4 Discussion

To our knowledge, this research is the first to apply WLD for texture based
classification of high resolution remotely sensed images. The results are
encouraging. In almost every case, WLD produced more accurate results than
LBP, the state-of-the-art LBPRIU or traditional GLCM-based methods.
Concatenating the VAR TD with the WLD TD improved accuracy further in an image
with obvious within-class illumination variation, but had no effect in an
image with more homogeneous illumination. Unlike LBP, WLD values do vary
somewhat depending on pixel brightness. The VAR metric, by comparison, is
sensitive to the range of brightness values around a pixel but not their
absolute levels. We believe that VAR improves WLD performance in Region 2
because it ignores illumination variability within classes while incorporating
the additional feature of contrast.

Our work also suggests that varying the scale of these TDs will not
necessarily improve classification accuracy, and that using multiple scales
concurrently provides no benefit. This conclusion is tentative since scale
effects probably depend strongly on the images being classified. After
demonstrating the superior accuracy of WLD, we optimised our software to
improve computational performance by almost two orders of magnitude. Thus our
implementation of WLD is practical for operational remote sensing
applications.

This research is of course preliminary. We have not attempted statistical
analyses of significance on due to limitations on available training data.
Comparable experiments should be performed using other sensors and geographic
areas, with ground truth based on actual surveys, rather than independent
visual interpretation.

We see many interesting directions for future research. WLD and other binary
TD approaches are currently limited to a single band of data. We are working
on schemes to incorporate spectral information into a WLD-like TD. We are also
investigating alternative methods for incorporating variance information into
WLD, since simple histogram concatenation has a negative impact on
computational performance. Finally, we hope to explore the use of WLD for
segmentation and unsupervised classification of high resolution remotely
sensed images.

## References

  * [1] Minho Kim, Bo Xu, Marguerite Madden, and ISPRS.  Object-based vegetation type mapping from an orthorectified multispectral ikonos image using ancillary information.  Proceedings of the GEOBIA, 2008.
  * [2] AP Carleer, Olivier Debeir, and Eléonore Wolff.  Assessment of very high spatial resolution satellite image segmentations.  Photogrammetric Engineering & Remote Sensing, 71(11):1285–1294, 2005.
  * [3] Soe W Myint, Patricia Gober, Anthony Brazel, Susanne Grossman-Clarke, and Qihao Weng.  Per-pixel vs. object-based classification of urban land cover extraction using high spatial resolution imagery.  Remote sensing of environment, 115(5):1145–1161, 2011.
  * [4] Yanmei Zhang, Jixian Zhang, Guoman Huang, and Zheng Zhao.  Object-oriented classification of polarimetric sar imagery based on texture features.  In 2011 International Symposium on Image and Data Fusion, pages 1–4. IEEE, 2011.
  * [5] Robert M Haralick, Karthikeyan Shanmugam, and Its’ Hak Dinstein.  Textural features for image classification.  IEEE Transactions on systems, man, and cybernetics, (6):610–621, 1973.
  * [6] Thierry Ranchin and Lucien Wald.  The wavelet transform for the analysis of remotely sensed images.  International Journal of Remote Sensing, 14(3):615–619, 1993.
  * [7] Cle Pohl and John L Van Genderen.  Review article multisensor image fusion in remote sensing: concepts, methods and applications.  International journal of remote sensing, 19(5):823–854, 1998.
  * [8] Tianhorng Chang and C-CJ Kuo.  Texture analysis and classification with tree-structured wavelet transform.  IEEE Transactions on image processing, 2(4):429–441, 1993.
  * [9] Vladimir Risojević, Snježana Momić, and Zdenka Babić.  Gabor descriptors for aerial image classification.  In International Conference on Adaptive and Natural Computing Algorithms, pages 51–60. Springer, 2011.
  * [10] David A Clausi and M Ed Jernigan.  Designing gabor filters for optimal texture separability.  Pattern Recognition, 33(11):1835–1849, 2000.
  * [11] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [12] Yong Xu, Sibin Huang, Hui Ji, and Cornelia Fermüller.  Scale-space texture description on sift-like textons.  Computer Vision and Image Understanding, 116(9):999–1013, 2012\.
  * [13] Timo Ojala, Matti Pietikäinen, and David Harwood.  A comparative study of texture measures with classification based on featured distributions.  Pattern recognition, 29(1):51–59, 1996.
  * [14] Timo Ojala, Kimmo Valkealahti, Erkki Oja, and Matti Pietikäinen.  Texture discrimination with multidimensional distributions of signed gray-level differences.  Pattern Recognition, 34(3):727–739, 2001.
  * [15] Timo Ahonen, Abdenour Hadid, and Matti Pietikainen.  Face description with local binary patterns: Application to face recognition.  IEEE transactions on pattern analysis and machine intelligence, 28(12):2037–2041, 2006.
  * [16] Ma Li and Richard C Staunton.  Optimum gabor filter design and local binary patterns for texture segmentation.  Pattern Recognition Letters, 29(5):664–672, 2008.
  * [17] Timo Ojala, Matti Pietikainen, and Topi Maenpaa.  Multiresolution gray-scale and rotation invariant texture classification with local binary patterns.  IEEE Transactions on pattern analysis and machine intelligence, 24(7):971–987, 2002.
  * [18] Marcelo Musci, Raul Queiroz Feitosa, Gilson AOP Costa, and Maria Luiza Fernandes Velloso.  Assessment of binary coding techniques for texture characterization in remote sensing imagery.  IEEE Geoscience and Remote Sensing Letters, 10(6):1607–1611, 2013\.
  * [19] Jie Chen, Shiguang Shan, Chu He, Guoying Zhao, Matti Pietikäinen, Xilin Chen, and Wen Gao.  Wld: A robust local image descriptor.  IEEE transactions on pattern analysis and machine intelligence, 32(9):1705–1720, 2009.
  * [20] Decky Aspandi-Latif, Sally Goldin, and Sumaryono.  Texture based classification of high resolution remotely sensed imagery using weber local descriptor.  In Asia GIS 2014 International Conference, volume 10, 2014.
  * [21] Mäenpää Topi, Pietikäinen Matti, and Ojala Timo.  Texture classification by multi-predicate local binary pattern operators.  In Proceedings 15th International Conference on Pattern Recognition. ICPR-2000, volume 3, pages 939–942. IEEE, 2000.
  * [22] A Richards John and Jia Xiuping.  Remote sensing digital image analysis: an introduction, 1999.
  * [23] Oncel Tuzel, Fatih Porikli, and Peter Meer.  Region covariance: A fast descriptor for detection and classification.  In European conference on computer vision, pages 589–600. Springer, 2006.
  * [24] Jacob Cohen.  A coefficient of agreement for nominal scales.  Educational and psychological measurement, 20(1):37–46, 1960.
  * [25] Kurt Rudahl and Sally E Goldin.  Opendragon: software and a programmer’s toolkit for teaching remote sensing and geoinformatics.  Open Geospatial Data, Software and Standards, 2(1):1–7, 2017.
  * [26] David A Clausi and Yongping Zhao.  Rapid extraction of image texture by co-occurrence using a hybrid data structure.  Computers & Geosciences, 28(6):763–774, 2002.
  * [27] L-K Soh and Costas Tsatsoulis.  Texture analysis of sar sea ice imagery using gray level co-occurrence matrices.  IEEE Transactions on geoscience and remote sensing, 37(2):780–795, 1999.
  * [28] Maria MP Petrou and Sei-ichiro Kamata.  Image processing: dealing with texture.  John Wiley & Sons, 2021.


# Word2Vec vs DBnary: Augmenting METEOR using Vector Representations or
Lexical Resources?

1LIG – Univ. Grenoble Alpes
Domaine Universitaire
38401 St Martin d’Hères, France
firstname.lastname
@imag.fr
Christophe Servan1,2, Alexandre Bérard1, Zied Elloumi1,3,
Hervé Blanchon1 & Laurent Besacier1

2SYSTRAN
5 Rue Feydeau
75002 Paris, France
firstname.lastname
@systran.fr

3LNE
29 Avenue Roger Hennequin
78190 Trappes, France
firstname.lastname
@lne.fr

###### Abstract

This paper presents an approach combining lexico-semantic resources and
distributed representations of words applied to the evaluation in machine
translation (MT). This study is made through the enrichment of a well-known MT
evaluation metric: METEOR. This metric enables an approximate match (synonymy
or morphological similarity) between an automatic and a reference translation.
Our experiments are made in the framework of the Metrics task of WMT 2014. We
show that distributed representations are a good alternative to lexico-
semantic resources for MT evaluation and they can even bring interesting
additional information. The augmented versions of METEOR, using vector
representations, are made available on our Github page.

##  1 Introduction

00footnotetext: This work is licensed under a Creative Commons Attribution 4.0
International Licence. Licence details:
http://creativecommons.org/licenses/by/4.0/

Learning vector representations of words using neural networks has generated a
strong enthusiasm in the NLP research community. In particular, many
contributions were proposed after the work of [Mikolov et al., 2013a, Mikolov
et al., 2013b, Mikolov et al., 2013c] on training word embeddings. The main
reasons for this strong interest are: the proposal of a simple and efficient
neural architecture to learn word vector representations, the availability of
an open source tool Word2Vec111http://word2vec.googlecode.com/svn/trunk/ and
the rapid structuring of a user
community222https://groups.google.com/d/forum/word2vec-toolkit. Later on,
several contributions have extended the work of Mikolov on word vectors to
phrases (sequences of words) [Mikolov et al., 2013b, Le and Mikolov, 2014a]
and to bilingual representations [Luong et al., 2015]. All these vector
representations capture similarities between words, phrases or sentences at
different levels (morphological, semantic).

However, although these representations can be semantically informative, they
do not exactly replace fine-grained information available in lexical-semantic
resources such as WordNet [Fellbaum, 1998], BabelNet [Navigli and Ponzetto,
2010], or DBnary [Sérasset, 2012]. Such lexical resources are also more easily
interpretable by humans as shown in [Panchenko, 2016], but their construction
is costly while word embeddings can be trained ad infinitum on any monolingual
or bilingual corpora.

In short, both approaches (lexical resources and word embeddings) have their
pros and cons. However, few studies have attempted to compare and combine
them. Pioneering work of ?) proposed to refine representations learning using
lexical resources. The idea is to force words connected in the lexical
network, to have a close representation (for example through a synonymy link).
The technique proposed is evaluated on several benchmarks (word similarity,
sentiment analysis, finding of synonyms). More recently, ?) and ?) extended
word embeddings to sense embeddings and tried to compare them to lexical
synsets.

Contributions: this article attempts to review the contribution of vector
representations to measure sentence similarity. We compare them with
similarity measures based on lexical resources such as WordNet or DBnary.
Machine Translation (MT) evaluation was identified as a particularly
interesting application to investigate, since MT evaluation is still an open
problem nowadays. More precisely, we propose to augment a well known MT
evaluation metric (METEOR [Banerjee and Lavie, 2005]) which allows an
approximate matching (through synonymy or morphological similarity) between MT
hypothesis and reference. The augmented versions of METEOR proposed (using
word embeddings, lexical resources or both) allow us to objectively compare
the contribution of each approach to measure sentence similarity. For this,
correlations between METEOR and human judgements (of MT outputs) are measured
within the framework of WMT 2014 Metrics task. The code of the augmented
versions of METEOR is also provided on our Github
page333https://github.com/cservan/METEOR-E.

Outline: in section 2 (Related Work), we quickly present METEOR, lexical
resources and word embeddings. Section 3 presents our propositions to augment
METEOR in order to conduct a fair comparison between lexical resources and
vector representations respectively. Section 4 presents our experiments made
within the framework of WMT 2014, as well as quantitative and qualitative
analyses. Finally, section 5 concludes this work and gives some perpectives.

##  2 Related Work

###  2.1 An automatic metric for MT evaluation: METEOR

####  2.1.1 The origins

METEOR was proposed to compensate BLEU’s and NIST’s weaknesses [Papineni et
al., 2002, Doddington, 2002]. In short, METEOR was created to better correlate
with human judgements by using more than word-to-word alignments between a
hypothesis and some references. The alignment is made according to three
modules: the first stage uses exact match between word surface forms (Exact
module), the second one compares word stems (Stems module) and the third one
uses synonyms (Synonym module) from a lexical resource such as WordNet
(available for English only in METEOR).

One contribution of this paper is to propose an alternative to Stems and
Synonym modules: our proposed add-on will be called Vectors module later on.

####  2.1.2 Recent extensions of METEOR

METEOR-NEXT [Denkowski and Lavie, 2010a] was proposed to better correlate with
HTER (Human-targeted Translation Edit Rate – HTER [Snover et al., 2006]). HTER
is a semi-automatic post-editing based metric, which measures the edit
distance between a hypothesis and a reference. METEOR-NEXT proposes to go
further than just word-to-word alignment by using phrase-to-phrase alignments.
For this, phrase databases were created for several languages like English
[Snover et al., 2009], German, French or Czech [Denkowski and Lavie, 2010b].
More recently, another version called METEOR Universal used bitexts to extract
paraphrases [Denkowski and Lavie, 2014].

METEOR was also extended by using Word Sense Disambiguation (WSD) techniques
[Apidianaki and Marie, 2015]. The authors used Babelfly [Moro et al., 2014]
for several langage pairs (translation from French, Hindi, German, Czech and
Russian to English). A better correlation with human judgement at segment
level was observed using WSD in METEOR.

Finally, to extend the use of Synonym module to target languages others than
English, ?) proposed to replace WordNet by DBnary [Sérasset, 2012]. The new
target languages equipped with a Synonym module were French, German, Spanish,
Russian and English.

###  2.2 Lexical resources

####  2.2.1 WordNet

WordNet is a well known lexical resource for English. Created at the
University of Princeton [Fellbaum, 1998], it is used in several NLP tasks such
as Machine Translation, Word Sense Disambiguation, Cross-lingual Information
Retrieval, etc. WordNet links nouns, verbs, adjectives and adverbs to a set of
synonyms called “synsets”. Each synset represents a specific concept.

Synsets are linked to each other according to semantic, conceptual and lexical
relations. Words with multiple meanings correspond to multiple synsets and
meanings are sorted according to their frequency. WordNet is available in
several languages (Arabic, French, etc.) but these versions are not freely
available. In METEOR, only English WordNet is used to match hypothesis and
reference words according to their meanings. It contains more than 117,000
synsets.

To extract lemmatized forms, METEOR uses a function called Morphy-7WN1 which
firstly checks special cases in an exception list and secondly uses rules to
lemmatize words according to their syntactic class.

####  2.2.2 DBnary

DBnary is a multilingual lexical resource in RDF format [Klyne and Carroll,
2004]. This resource has been collected by ?). Lexical data are represented
using the LEMON vocabulary [McCrae et al., 2011]. Most Part-of-Speech tags are
linked with Olia standards or Lexinfo vocabularies [Chiarcos and Sukhareva,
2015, Cimiano et al., 2011] which makes them reusable in many contexts.

DBnary is downloadable or available online through a SPARQL access point.
Lexical data are automatically extracted from Wiktionary, Wikipedia’s
dictionary for 21 languages444Bulgarian, Dutch, English, Finnish, French,
German, (Modern) Greek, Indonesian, Italian, Japanese, Latin, Lithuanian,
Malagasy, Norwegian, Polish, Portuguese, Russian, Serbo-Croat, Spanish,
Swedish and Turkish.

| English | French | Russian | German
---|---|---|---|---
Number of entries | 620 K | 322 K | 185 K | 104 K
Number of meanings | 498 K | 416 K | 176 K | 116 K
Number of synsets | 35 K | 36 K | 31 K | 33 K
Table 1: Detail of the data used from DBnary for the languages targeted in
this paper.

Among available lexical data, one may find 2.9M lexical entries (with parts-
of-speech, canonical form for all of them, along with pronunciations when
available and inflected forms for some languages). Lexical entries are
subdivided into 2.5M lexical senses (with their definitions and some usage
example).

DBnary also contains more than 4.6M translations going from the 21 extracted
sources languages to more than 1500 different target languages. Additionally,
DBnary contains lexicosemantic relations (syno/anto-nyms, hypo/hypero-nyms,
etc.). Table 1 shows the size of the data for languages involved in the
experiments later reported in this paper. Additional figures are available on
the DBnary public web site555http://kaiko.getalp.org/about-dbnary/.

Lemmatized forms for DBnary are based on the TreeTagger module [Schmid, 1995],
which enables us to find the corresponding synsets.

###  2.3 Monolingual and bilingual embeddings

####  2.3.1 Overview

Learning word embeddings is an active research area [Bengio et al., 2003,
Turian et al., 2010, Collobert et al., 2011, Huang et al., 2012]. The main
idea is to learn a word representation according to its context: the
surrounding words [Baroni and Zamparelli, 2010]. The words are projected on a
continuous space and those with similar context should be close in this multi-
dimensional space. When word vectors are available, a similarity between two
words can be measured by a metric such as a cosine similarity.

Using word-embeddings for machine translation evaluation is appealing since
they can be used to compute similarity between words or phrases in the same
language (monolingual embeddings capture intrinsically synonymy or
morphological closeness) or in two different languages (bilingual embeddings
allow to directly compute a distance between two sentences in different
languages). We use the MultiVec [Bérard et al., 2016] toolkit for computing
and managing the continuous representations of texts. It includes word2vec
[Mikolov et al., 2013a], paragraph vector [Le and Mikolov, 2014b] and
bilingual distributed representations [Luong et al., 2015] features.

####  2.3.2 Use of vector representations in NLP evaluation

?) proposed to use bilingual word embeddings to detect similarities for word
alignment. This information is used as an additional parameter in a phrase-
based machine translation system. [Banchs et al., 2015] proposed to explore a
metric funded on latent semantic analysis [Salton et al., 1975] to extract
semantic embeddings and measure the similarity between two sentences. Finally,
these word embeddings were used to enrich ROUGE, a metric for evaluating
automatic summarization [Ng and Abrecht, 2015].

As far as MT evaluation is concerned, ?) proposed a metric based on neural
network language models jointly with dependency trees to link an hypothesis to
a reference. Meanwhile, ?) proposed an approach to model document embeddings
to predict translation adequacy.

These works are close to ours but they propose metrics which need to be
learned and optimized to a specific task or domain. In our work, we use word
embeddings trained once and for all on a (large) general corpus. Our detailed
methodology to augment METEOR metric is presented in the next section.

##  3 Augmented METEOR

###  3.1 Data and protocol

We evaluate our augmented METEOR through WMT14 framework (metrics task
[Machacek and Bojar, 2014]). This framework enables us to estimate the
correlation of proposed evaluation metric with human judgements for several
machine translation outputs and several language pairs (English-French,
English-German, English-Russian, and vice versa). In our experiments, we use
segment level Kendall’s $\tau$ correlation coefficient, as proposed in WMT14
(based on systems ranking at sentence level by humans, compared to automatic
metric ranking).

We augment METEOR in two ways: firstly, we replace the use of lexical
resources by the use of word embeddings. In other words, we replace Stem and
Synonym modules by our new Vector module. Secondly, we combine lexical
resources and word embeddings by using jointly Stem, Synonym and our Vector
module.

To summarize, the following variants of METEOR are evaluated:

  * •

METEOR Baseline: the METEOR score is estimated using Exact, Stem, Synonym and
Paraphrase modules for English as a target language and Exact, Stem and
Paraphrase modules for other target languages,

  * •

METEOR DBnary: similar to METEOR Baseline but Synonym module is available for
any target language since it uses DBnary resource instead of Wordnet,

  * •

METEOR Vector: the Stem and Synonym modules are replaced by the Vector module
;

  * •

METEOR Baseline $+$ Vector: the METEOR Baseline configuration is augmented
with the Vector module ;

  * •

METEOR DBnary $+$ Vector: the METEOR DBnary configuration is augmented with
the Vector module.

###  3.2 METEOR DBnary

As mentioned in section 2.1, the Synonym module of METEOR uses WordNet’s
synsets (117K entries for English). As an alternative, we use another lexical
resource: DBnary [Sérasset, 2012], as proposed recently by ?). This allows us
to use Synonym module for any target language: French, German, Spanish,
Russian and English.

More precisely, synonym relations are extracted from DBnary using SPARQL
request on the DBnary server666http://kaiko.getalp.org/about-dbnary/online-
access/. We extract data for English, French, Russian and German languages.
The extraction process outputs relations in the following format: $lemma$
$\rightarrow$ $Synonym$. Then, these data are projected to the WordNet format
used in METEOR code. This process gives an identifier (ID) for each lemma and
builds a list of synonym IDs for each lemma such as:
$lemma\leavevmode\nobreak\ \rightarrow\leavevmode\nobreak\
ID\\_Syn_{1},\leavevmode\nobreak\ ID\\_Syn_{2},\leavevmode\nobreak\
ID\\_Syn_{3}$.

The first two lines of Table 3 compare METEOR DBnary and METEOR Baseline for
several French-English MT systems submitted to WMT14 [Bojar et al., 2014].

METEOR DBnary improved the score by 0.7 points from METEOR Baseline. In other
words, DBnary seems to match more synonyms than WordNet, despite the fact that
WordNet is 3.3 time bigger than DBnary in English. This could be due to the
fact that WordNet has only 4 morpho-syntactic categories (Noun, Verbs,
Adjectives and Adverbs) while DBnary has more morpho-syntactic categories.

###  3.3 METEOR Vector

As mentioned in section 2.3.2, we propose to replace lexical resources by word
embeddings. Word embeddings capture the context of the words. Consequently,
similar word vectors may correspond to synonyms or morphological variants (see
section 2.3).

Language | Corpora | # of lines | # of source | # of target
---|---|---|---|---
words | words
French–English | Europarl V7 + news commentary V10 | 2.2 M | 67.2 M | 60.7 M
German–English | Europarl V7 + news commentary V10 | 2.1 M | 57.2 M | 59.7 M
Russian–English | Common Crawl + news commentary V10 + Yandex | 2.0 M | 47.2 M | 50.3 M
Table 2: Bilingual corpora used to train the word embeddings for each language
pair.

In our Vector module, the matching between two words is done using a
similarity score derived from the cosine similarity. If the similarity score
is higher than a threshold, the words are considered as matched (potential
synonyms or morphological proximity). In our experiments, we evaluate using:
(a) a default threshold fixed to 0.80 (b) an oracle threshold obtained
empirically on the WMT14 data set [Machacek and Bojar, 2014].

Table 2 summarizes data used to train monolingual word embeddings and
bilingual word embeddings. These word embeddings were trained with a CBOW
model, a vector size of 50 and a windows size $\pm$5 words, thanks to the
MultiVec toolkit [Bérard et al., 2016].

Metrics | Systems:
---|---
online A | online B | online C | rbmt 1 | rbmt 4
METEOR Baseline | 36.33 | 36.71 | 31.19 | 33.00 | 31.65
METEOR DBnary | 36.93 | 37.33 | 32.01 | 33.69 | 32.42
METEOR Vector | 37.00 | 37.34 | 31.87 | 33.67 | 32.34
METEOR Baseline $+$ Vector | 37.08 | 37.40 | 31.96 | 33.75 | 32.45
METEOR DBnary $+$ Vector | 37.53 | 37.88 | 32.60 | 34.32 | 33.05
Table 3: METEOR scores (all configurations) on the newstest corpus of the
WMT14 translation evaluation task from French to English.

The results presented in table 3 show that word embeddings (Vector module) can
efficiently replace lexical resources (Synonym and Stem modules) to match
words in the translation hypothesis with those in the reference. In addition,
their combination shows a good potential to match even more words between
hypothesis and reference. In the next section, we evaluate if the proposed
versions of augmented METEOR better correlate with human judgements.

##  4 Correlations of Augmented METEOR with Human Judgements

###  4.1 Results of different METEOR configurations

In these experiments, we present results obtained with the Vector module based
on two threshold values: a default one (0.80) and an oracle one which
maximizes the correlation with human judgement.

Table 4 presents the correlation scores obtained within the framework of WMT14
metrics task [Machacek and Bojar, 2014]777For better readability, we do not
add standard deviations in the tables. These numbers will be, however,
provided in supplementary material put on the paper web page
(https://github.com/cservan/METEOR-E/paper).. The evaluation is done according
to several translation tasks: from English to French (en–fr), German (en–de)
and Russian (en–ru), and vice versa. French, German and Russian as target
languages represent a growing difficulty due to their morphology. English as
target language allows to compare the lexical databases (Wordnet vs DBnary).

To English. Firstly, when the translation direction is to English, we can
observe that METEOR Baseline and METEOR Vector get equivalent results in
average. METEOR DBnary also obtains similar results to METEOR Baseline. When
we combine WordNet lexical resource and word embeddings (METEOR Baseline $+$
Vector), the reference score is increased by $0,005$ points. If the
combination is done with DBnary’s lexical data (METEOR DBnary $+$ Vector), the
improvement is similar.

For Vector module, optimization of the threshold slightly improves the average
correlation. Combination of METEOR Baseline $+$ Vector or METEOR DBnary $+$
Vector improves by $0,002$ points when the threshold is optimized.

From English. Secondly, when the translation direction is from English, we can
observe an improvement of the correlation score obtained with METEOR DBnary,
compared with METEOR Baseline. This is due to the fact that for French, German
and Russian as target languages, METEOR Baseline does not use any Synonym
module. Our METEOR Vector with the default threshold also gets better
correlation scores compared to METEOR DBnary ($+0.003$ points in average). The
combinations METEOR Baseline $+$ Vector and METEOR DBnary $+$ Vector further
improve correlations with human judgements ($+0.001$ points in average).
Finally, when we use an oracle threshold for Vector module, improvements are
bigger and can reach $0.013$ points in average, compared to METEOR Baseline.

Language pairs | fr–en | de–en | ru–en | Average
---|---|---|---|---
Metric | Threshold | $\tau$ | Threshold | $\tau$ | Threshold | $\tau$ | Threshold | $\tau$
METEOR Baseline | – | 0.406 | – | 0.334 | – | 0.329 | – | 0.356
METEOR DBnary | – | 0.408 | – | 0.334 | – | 0.328 | – | 0.357
METEOR Vector | 0.80 | 0.407 | 0.80 | 0.332 | 0.80 | 0.328 | 0.80 | 0.356
METEOR Baseline $+$ Vector | 0.80 | 0.407 | 0.80 | 0.343 | 0.80 | 0.332 | 0.80 | 0.361
METEOR DBnary $+$ Vector | 0.80 | 0.407 | 0.80 | 0.337 | 0.80 | 0.338 | 0.80 | 0.361
METEOR Vector | 0.89 | 0.411 | 0.78 | 0.333 | 0.80 | 0.328 | 0.82 | 0.357
METEOR Baseline $+$ Vector | 0.73 | 0.412 | 0.80 | 0.343 | 0.88 | 0.333 | 0.80 | 0.363
METEOR DBnary $+$ Vector | 0.73 | 0.413 | 0.79 | 0.338 | 0.80 | 0.338 | 0.77 | 0.363

Language pairs | en-fr | en-de | en-ru | Average
---|---|---|---|---
Metric | Threshold | $\tau$ | Threshold | $\tau$ | Threshold | $\tau$ | Threshold | $\tau$
METEOR Baseline | – | 0.280 | – | 0.238 | – | 0.427 | – | 0.315
METEOR DBnary | – | 0.284 | – | 0.239 | – | 0.435 | – | 0.319
METEOR Vector | 0.80 | 0.290 | 0.80 | 0.241 | 0.80 | 0.436 | 0.80 | 0.322
METEOR Baseline $+$ Vector | 0.80 | 0.288 | 0.80 | 0.241 | 0.80 | 0.440 | 0.80 | 0.323
METEOR DBnary $+$ Vector | 0.80 | 0.289 | 0.80 | 0.242 | 0.80 | 0.439 | 0.80 | 0.323
METEOR Vector | 0.72 | 0.295 | 0.79 | 0.241 | 0.72 | 0.439 | 0.74 | 0.325
METEOR Baseline $+$ Vector | 0.86 | 0.296 | 0.79 | 0.242 | 0.79 | 0.445 | 0.81 | 0.328
METEOR DBnary $+$ Vector | 0.88 | 0.294 | 0.75 | 0.245 | 0.79 | 0.443 | 0.81 | 0.327

Table 4: Correlation score at segment level between several METEOR
configurations and human judgements (WMT14 framework). Scores obtained with
the Vector module are presented firstly with the default threshold (0.80) and
secondly with the oracle threshold (under the dashed line).

###  4.2 Investigating more embeddings configurations

In the previous section, METEOR Vector used a simple and monolingual word
embedding configuration. This section investigates more configurations
(monolingual and bilingual) to improve METEOR.

In this experiment, we focus only on METEOR Vector. Indeed, the monolingual
(baseline) shown in table 6 corresponds to the line METEOR Vector in Table 4.
Firstly, we propose to train our embeddings on bitexts (Table 2) using bivec
approach [Luong et al., 2015]. We also try to train pseudo-bilingual
embeddings on a pseudo bitext with target language text and POS tags (see an
example in Table 5).

madam president , on a point of order . | $\Leftrightarrow$ | NOUN NOUN PUNCT ADP DET NOUN ADP NOUN PUNCT
---|---|---
Table 5: Example of bitext where the target side is replaced by POS.

The main idea is to strongly link words with their syntactic class when
learning word embeddings. We call this kind of model pseudo-bilingual with
POS. In the same way, we train bilingual models called pseudo-bilingual with
lemmas, where the POS tags are replaced by lemmas. In addition, we also learn
word embeddings with lemmas only and bilingual models with lemmas only.

Models: | monolingual | bilingual | pseudo-bilingual | pseudo-bilingual | monolingual | bilingual
---|---|---|---|---|---|---
(baseline) | with POS | with lemmas | (lemmas) | (lemmas)
To English | 0.356 | 0.354 | 0.355 | 0.354 | 0.357 | 0.357
From English | 0.322 | 0.322 | 0.320 | 0.325 | 0.324 | 0.318
Table 6: Average correlation score at segment level for METEOR Vector with
several training configurations of word embeddings with the default threshold
(0.80).

In the Table 6, we compare several training configuration of the word
embeddings through the same protocol as previous section (only average
correlations are reported while the detailed results will be provided as
supplementary material on the paper web page). When we observe the average
results, the bilingual embeddings seem not to be as efficient as the
monolingual baseline. The pseudo-bilingual approaches with POS and Lemmas
obtained slightly the same results as the monolingual baseline regarding all
the configurations we have. Finally, the monolingual model learned on lemmas
(instead of words) tends to be slightly better when the translation direction
is to English. However, this trend should be confirmed in a future
investigation.

###  4.3 Discussion

The correlation scores obtained with the enriched metric tend to suggest that
distributed representations are as powerful as lexico-semantic resources for
automatic MT evaluation. Furthermore, vector representations can bring
additional information, and they are definitely useful when no lexical
resource is available in the target language.

Considering the average correlation scores obtained, the configurations METEOR
Vector and METEOR DBnary are comparable, except on German language, for which
METEOR Vector obtained a better correlation score. On the other hand, when we
combine lexical data with Vector module (METEOR DBnary $+$ Vector), we observe
a small increase of the correlation score, in particular when threshold is
tuned, which suggests a tunable version of METEOR.

Finally, several embeddings variants were trained but it seems that
monolingual models are efficient enough for the specific task (MT evaluation)
considered here.

####  4.3.1 Examples

To illustrate the word matching obtained by our versions of METEOR, we analyze
two examples from the evaluation data set. In these examples, we present the
alignments obtained with METEOR DBnary $+$ Vector.

hypothesis: |  je pense qu’ il est concevable que ces données soient employées pour le bénéfice mutuel .
---|---
Reference: |  j’ estime qu’ il est concevable que ces données soient utilisées dans leur intérêt mutuel .
Table 7: First example from the system rbmt 1 evaluated with the combination
METEOR DBnary $+$ Vector. The relations detected with the lexical resource
DBnary are framed in continuous line while those obtained thanks to the
distributed representations are framed in dotted line.

The example presented in table 7 shows rbmt 1 system output submitted during
the WMT14 translation task. METEOR baseline found only alignments for words
with the same surface forms (“qu’ ”, “il”, “est”, etc. – these forms are found
identical thanks to the Exact module and are not highlighted here). The
Synonym module based on DBnary makes it possible to find a correspondence
between words “employées” – “utilisées” and “pour” – “dans”. Lastly, Vector
module indicates that words “pense” and “estime” are contextually closed, just
as the words “je” and “j”’. When the example is only evaluated with METEOR
Vector, words “employées” and “utilisées” are also paired with the default
threshold (0.80). On the other hand, the words “bénéfice” and “intérêt” are
paired by the module Vector only if the decision threshold is lowered to 0.75.

hypothesis: | le créateur de SAS disait il faisait un genre du feuilleton géopolitique .
---|---
Reference: | le père de SAS disait faire un genre de feuilleton géopolitique .
Table 8: Another example scored with the combination METEOR DBnary $+$ Vector.

In the second example presented in table 8, the hypothesis is provided by rbmt
4 system. As in the previous example, the correspondences found with Synonym
module based on DBnary (framed by one continuous line) are supplemented by
those found by Vector module (dotted line): Synonym module found “créateur” –
“père” and “faisait” – “faire”; while “du” and “de” are aligned thanks to
Vector module.

These examples illustrate the complementarity between lexical resources and
word embeddings for sentence similarity detection. Word vectors can enable to
match important words (like “pense” and “estime” in our first example), but
also empty words (like “du” et “de” in our second example).

####  4.3.2 Limitations of Word Embeddings

So far, we did not deal with Out-Of-Vocabulary (OOV) words in METEOR Vector.
By OOV we mean words that do not have a vector representation because they
were not found in the training corpus for word embeddings. In that case, no
matching can occur between the word in the hypothesis and words in reference.
Consequently, it might be interesting to carefully select the training corpus
for word vectors so that it will be close enough to the machine translation
outputs to evaluate. This could be considered in future works.

##  5 Conclusion and Perspectives

In this paper, we proposed to compare text similarity measures based on vector
representations with similarity measures based on lexico-semantic resources.
Our work was applied to machine translation evaluation and we extended an
existing evaluation metric called METEOR. Our experiments have shown that word
vector representations can be useful when no lexical resource is available in
the target language. Moreover, it seems that these representations can bring
complementary information in addition to lexical resources (experiments done
for French, English, German and Russian as target languages).

Our future works on this topic will focus on the use of phrase embeddings to
complement the Paraphrase module of METEOR. We also plan to introduce a syntax
flavor in our Vector module by weighting the cosine distances differently
according to the morpho-syntactic category of the words. Finally, we will
study the adaptation of our approach to other metrics such as TER-Plus, for
instance.

The tool, the data and the models presented in this paper will be put
online888https://github.com/cservan/METEOR-E to facilitate reproducibility of
the experiments we carried out.

## Acknowledgements

This work was supported by the KEHATH project funded by the French National
Agency for Research (ANR) under the grant number ANR-14-CE24-0016-03.

## References

  * [Apidianaki and Marie, 2015]  Marianna Apidianaki and Benjamin Marie.  2015\.  METEOR-WSD: Improved Sense Matching in MT Evaluation.  In the Proceedings of the 9th Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST’9).
  * [Banchs et al., 2015]  Raphael E. Banchs, Luis F. D’Haro, and Haizhou Li.  2015\.  Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):472–482, March.
  * [Banerjee and Lavie, 2005]  Satanjeev Banerjee and Alon Lavie.  2005\.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.  In Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43th Annual Meeting of the Association of Computational Linguistics.
  * [Baroni and Zamparelli, 2010]  Marco Baroni and Roberto Zamparelli.  2010\.  Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space.  In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.
  * [Bengio et al., 2003]  Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin.  2003\.  A neural probabilistic language model.  The Journal of Machine Learning Research, 3:1137–1155.
  * [Bérard et al., 2016]  Alexandre Bérard, Christophe Servan, Olivier Pietquin, and Laurent Besacier.  2016\.  MultiVec: a Multilingual and Multilevel Representation Learning Toolkit for NLP.  In The 10th edition of the Language Resources and Evaluation Conference (LREC 2016).
  * [Bojar et al., 2014]  Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna.  2014\.  Findings of the 2014 Workshop on Statistical Machine Translation.  In Proceedings of the Ninth Workshop on Statistical Machine Translation.
  * [Chiarcos and Sukhareva, 2015]  Christian Chiarcos and Maria Sukhareva.  2015\.  Olia – ontologies of linguistic annotation.  Web Semantics: Science, Services and Agents on the World Wide Web, 6(4):379–386.
  * [Cimiano et al., 2011]  P. Cimiano, P. Buitelaar, J. McCrae, and M. Sintek.  2011\.  Lexinfo: A declarative model for the lexicon-ontology interface.  Web Semantics: Science, Services and Agents on the World Wide Web, 9(1):29 – 51.
  * [Collobert et al., 2011]  Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  2011\.  Natural language processing (almost) from scratch.  The Journal of Machine Learning Research, 12:2493–2537.
  * [Denkowski and Lavie, 2010a]  Michael Denkowski and Alon Lavie.  2010a.  Extending the meteor machine translation evaluation metric to the phrase level.  In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics.
  * [Denkowski and Lavie, 2010b]  Michael Denkowski and Alon Lavie.  2010b.  METEOR-NEXT and the METEOR paraphrase tables: Improved evaluation support for five target languages.  In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR.
  * [Denkowski and Lavie, 2014]  Michael Denkowski and Alon Lavie.  2014\.  METEOR Universal: Language Specific Translation Evaluation for Any Target Language.  In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
  * [Doddington, 2002]  George Doddington.  2002\.  Automatic evaluation of machine translation quality using n-gram co-occurrence statistics.  In Proceedings of the second international conference on Human Language Technology Research.
  * [Elloumi et al., 2015]  Zied Elloumi, Hervé Blanchon, Gilles Serasset, and Laurent Besacier.  2015\.  METEOR for Multiple Target Languages using DBnary.  In Proceedings of MT Summit 2015.
  * [Faruqui et al., 2014]  Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard H. Hovy, and Noah A. Smith.  2014\.  Retrofitting word vectors to semantic lexicons.  CoRR.
  * [Fellbaum, 1998]  Christiane Fellbaum.  1998\.  WordNet: An Electronic Lexical Database.  MA: MIT Press.
  * [Gupta et al., 2015]  Rohit Gupta, Constantin Orasan, and Josef Van Genabith.  2015\.  Machine Translation Evaluation using Recurrent Neural Networks.  In Proceedings Workshop on Machine Translation (WMT), Metrics Shared Task.
  * [Huang et al., 2012]  Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng.  2012\.  Improving word representations via global context and multiple word prototypes.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1.
  * [Klyne and Carroll, 2004]  Graham Klyne and Jeremy J. Carroll.  2004\.  Resource Description Framework (RDF): Concepts and Abstract Syntax.  Technical report.
  * [Le and Mikolov, 2014a]  Quoc V. Le and Tomas Mikolov.  2014a.  Distributed Representations of Sentences and Documents.  In Proceedings of The 31st International Conference on Machine Learning.
  * [Le and Mikolov, 2014b]  Quoc V. Le and Tomas Mikolov.  2014b.  Distributed Representations of Sentences and Documents.  In Proceedings of the 31th International Conference on Machine Learning (ICML’14).
  * [Luong et al., 2015]  Thang Luong, Hieu Pham, and Christopher D. Manning.  2015\.  Bilingual word representations with monolingual quality in mind.  In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing.
  * [Machacek and Bojar, 2014]  Matous Machacek and Ondrej Bojar.  2014\.  Results of the WMT14 Metrics Shared Task.  In Proceedings of the Ninth Workshop on Statistical Machine Translation.
  * [McCrae et al., 2011]  John McCrae, Dennis Spohr, and Philipp Cimiano, 2011.  Linking Lexical Resources and Ontologies on the Semantic Web with Lemon, pages 245–259.  Springer Berlin Heidelberg, Berlin, Heidelberg.
  * [Mikolov et al., 2013a]  Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  2013a.  Efficient Estimation of Word Representations in Vector Space.  In The Workshop Proceedings of the International Conference on Learning Representations (ICLR) 2013.
  * [Mikolov et al., 2013b]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013b.  Distributed Representations of Words and Phrases and their Compositionality.  In Advances in Neural Information Processing Systems 26.
  * [Mikolov et al., 2013c]  Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig.  2013c.  Linguistic Regularities in Continuous Space Word Representations.  In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  * [Moro et al., 2014]  Andrea Moro, Alessandro Raganato, and Roberto Navigli.  2014\.  Entity linking meets word sense disambiguation: a unified approach.  Transactions of the Association for Computational Linguistics, 2:231–244.
  * [Navigli and Ponzetto, 2010]  Roberto Navigli and Simone Paolo Ponzetto.  2010\.  Babelnet: Building a very large multilingual semantic network.  In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010).
  * [Ng and Abrecht, 2015]  Jun-Ping Ng and Viktoria Abrecht.  2015\.  Better Summarization Evaluation with Word Embeddings for ROUGE.  In In The Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * [Panchenko, 2016]  Alexander Panchenko.  2016\.  Best of both worlds: Making word sense embeddings interpretable.  In the 10th edition of the Language Resources and Evaluation Conference (LREC 2016).
  * [Papineni et al., 2002]  Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  2002\.  BLEU: a Method for Automatic Evaluation of Machine Translation.  In ACL.
  * [Rothe and Schütze, 2015]  Sascha Rothe and Hinrich Schütze.  2015\.  Autoextend: Extending word embeddings to embeddings for synsets and lexemes.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.
  * [Salton et al., 1975]  G. Salton, A. Wong, and C. S. Yang.  1975\.  A Vector Space Model for Automatic Indexing.  ACM, 18(11):613–620, November.
  * [Schmid, 1995]  Helmut Schmid.  1995\.  Treetagger| a language independent part-of-speech tagger.  Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart, 43:28.
  * [Sérasset, 2012]  Gilles Sérasset.  2012\.  Dbnary: Wiktionary as a lemon-based multilingual lexical resource in rdf.  Semantic Web Journal-Special issue on Multilingual Linked Open Data, 6(4):355–361.
  * [Snover et al., 2006]  Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul.  2006\.  A study of translation edit rate with targeted human annotation.  In Proceedings of association for machine translation in the Americas.
  * [Snover et al., 2009]  Matthew Snover, Nitin Madnani, Bonnie J Dorr, and Richard Schwartz.  2009\.  Fluency, adequacy, or HTER?: exploring different human judgments with a tunable MT metric.  In Proceedings of the Fourth Workshop on Statistical Machine Translation.
  * [Turian et al., 2010]  Joseph Turian, Lev Ratinov, and Yoshua Bengio.  2010\.  Word representations: a simple and general method for semi-supervised learning.  In Proceedings of the 48th annual meeting of the association for computational linguistics.
  * [Vela and Tan, 2015]  Mihaela Vela and Liling Tan.  2015\.  Predicting Machine Translation Adequacy with Document Embeddings.  In Proceedings Workshop on Machine Translation (WMT), Metrics Shared Task.
  * [Zou et al., 2013]  Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning.  2013\.  Bilingual Word Embeddings for Phrase-Based Machine Translation.  In EMNLP.