# GREN: Graph-Regularized Embedding Network for Weakly-Supervised Disease
Localization in X-ray Images

Baolian Qi  Gangming Zhao  Xin Wei  Changde Du  Chengwei Pan  Yizhou Yu
Fellow, IEEE  Jinpeng Li  Member, IEEE This work was supported in part by
National Natural Science Foundation of China (62106248), Zhejiang Provincial
Natural Science Foundation of China (LQ20F030013), Ningbo Public Service
Technology Foundation, China (202002N3181) and Medical Scientific Research
Foundation of Zhejiang Province, China (2021431314). B. Qi, X. Wei, and J. Li
are with Center for Pattern Recognition and Intelligent Medicine (PRIM),
HwaMei Hospital, University of Chinese Academy of Sciences, NO.41 Northwest
Street, Haishu District, Ningbo, Zhejiang, 315010, China. They are also with
Ningbo Institute of Life and Health Industry, University of Chinese Academy of
Sciences, Ningbo, Zhejiang, China.G. Zhao is with Department of Computer
Science, the University of Hong Kong, Hong Kong, China.C. Du is with National
Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of
Sciences, Beijing, China.C. Pan is with Institute of Artificial Intelligence,
Beihang University, Beijing, China.Y. Yu is with Department of Computer
Science, the University of Hong Kong, Hong Kong, China.B. Qi and G. Zhao
contribute equally to this work. Corresponding author: Jinpeng Li
<EMAIL_ADDRESS>
###### Abstract

Locating diseases in chest X-ray images with few careful annotations saves
large human effort. Recent works approached this task with innovative weakly-
supervised algorithms such as multi-instance learning (MIL) and class
activation maps (CAM), however, these methods often yield inaccurate or
incomplete regions. One of the reasons is the neglection of the pathological
implications hidden in the relationship across anatomical regions within each
image and the relationship across images. In this paper, we argue that the
cross-region and cross-image relationship, as contextual and compensating
information, is vital to obtain more consistent and integral regions. To model
the relationship, we propose the Graph Regularized Embedding Network (GREN),
which leverages the _intra-image_ and _inter-image_ information to locate
diseases on chest X-ray images. GREN uses a pre-trained U-Net to segment the
lung lobes, and then models the intra-image relationship between the lung
lobes using an intra-image graph to compare different regions. Meanwhile, the
relationship between in-batch images is modeled by an inter-image graph to
compare multiple images. This process mimics the training and decision-making
process of a radiologist: comparing multiple regions and images for diagnosis.
In order for the deep embedding layers of the neural network to retain
structural information (important in the localization task), we use the Hash
coding and Hamming distance to compute the graphs, which are used as
regularizers to facilitate training. By means of this, our approach achieves
the state-of-the-art result on NIH chest X-ray dataset for weakly-supervised
disease localization. Our codes111https://github.com/qibaolian/GREN are
accessible online.

Computer-assisted diagnosis, Chest X-ray, Weakly supervised localization,
Domain knowledge.

##  1 Introduction

The automatic disease localization on chest X-ray images has become an
increasingly important technique to support clinical diagnosis and treatment.
The past decade has witnessed the application of convolutional neural networks
(CNNs) in medical image analysis, including disease classification [33], [11],
[2], [19], segmentation [6], [8], [28], [29], detection [39], [4], and report
generation [12], [14], [31]. CNNs require a large amount of finely annotated
training data to locate diseases. However, the annotation is tedious and
laborious for radiologists. As a result, the X-ray datasets often only have
the _image-level labels_ and few _location labels_ , which leads to inferior
localization results. Developing localization models with coarse-grained
(image-level) labels and few location annotations is an urgent problem, which
arouses a weakly-supervised localization task [38].

Existing weakly-supervised localization algorithms are mainly based on multi-
instance learning (MIL) [3], [16] and class activation maps (CAM) [23], [30],
[26]. However, these methods often yield inaccurate or incomplete regions. For
example, the CAM-based method only activates the most discriminative region
and generates incomplete targets. To overcome these problems, beyond mere
algorithmic innovations [22], [4], [10], [24], we argue that considering the
complementary information between anatomical regions and multiple images can
lead to a better localization result. This insight is inspired by the domain
knowledge in the medical field: radiologists are trained to read a large
number of X-ray images and analyse them by recognizing and comparing the
differences (e.g., shapes, textures, contrast, etc.). Radiologists make
decisions based on what they have learned and observed though comparison.

Recently, several attention-based works [18], [37] conducted disease
localization by comparing abnormal and normal instances. However, they
involved no direct supervision (e.g., the position label) to conduct error
feedback, and it was difficult to interpret what the models have learned. To
overcome these insufficiencies, we proposed the cross chest graph (CCG) [34]
to regularize the model instead of the attention operations [18], [37]. CCG
considered the inter-image relationship and neglected the intra-image
relationship. More importantly, CCG obtained regions via clustering, and the
self-organized regions lack pathological explanations. Based on but quite
different from CCG, the proposed Graph Regularized Embedding Network (GREN)
(Figure 1) considers both inter-image and intra-image relationships, and
explicitly models the anatomical structures. We calculate the relationship
between the two lung lobes because most chest diseases such as pneumonia,
infiltration and consolidation, rarely appear symmetrically in both sides
[37], and cardiomegaly usually makes left lung region smaller on X-ray image.
Thus, the features of these abnormal regions can be identified via comparing
the lung lobes. Our main contributions are:

Figure 1: The general design of GREN. The inter-image and intra-image
relations modeled by structural graphs are used to regularize CNNs to obtain
accurate and complete regions. GREN mimics a radiologist by leveraging cross-
image and cross-region information during training and decision-making.

  * •

We propose GREN, the first method exploring relational graphs across images
and regions to regularize training in weakly-supervised localization. Compared
with the attention-based methods, GREN avoids the normal control sampling
problem. Compared with Graph-Rise [13], GREN requires no additional manual
annotations. As the core elements, we present the intra-image and inter-image
knowledge learning modules to model the complementary information.

  * •

We achieve the state-of-the-art result for disease localization on NIH chest
X-ray dataset. Ablation experiments demonstrate the effectiveness of each
component. We make the code publicly available for subsequent studies to
reproduce our results.

##  2 Related Work

Deep learning has been widely applied in X-ray images analysis, including
disease classification [30], [25], [10], [36] and detection [16], [18], [24].
However, locating diseases in chest X-ray images with few careful annotations
remains a challenging problem. Wang et al. [30] contributed a chest X-ray
dataset and localized the disease with a multi-label disease classification
model according to the heatmaps. Li et al. [16] proposed to jointly model the
disease identification and localization with multi-instance loss and binary
cross-entropy loss. Tang et al. [25] proposed an iterative attention-guided
refinement framework to improve the classification and weakly-supervised
localization performance via CAM [23]. Sedai et al. [22] proposed a weakly
supervised method based on class-aware multiscale convolutional feature to
localize chest pathologies. Cai et al. [4] proposed an attention mining
strategy to improve the sensitivity and saliency of model to disease patterns.
All these methods overlooked the image-to-image and region-to-region
relationship during modelling.

For the last three years, some works have attempted to embed image relations
into deep learning. Zhao et al. [35] modeled the contralateral context
information using a spatial transformer network to enhance disease
representations. Lian et al. [17] leveraged the constant structure and disease
relations using a structure-aware relation extraction network. These methods
utilized the region-to-region relationship in individual image and overlooked
the image-to-image relationship in modelling. There are also some works using
inter-image relationship for weakly-supervised medical image analysis. Liu et
al. [18] utilized the contrast-induced attention acquired on paired images
between healthy and unhealthy samples to provide more information for
localization. Zhou et al. [37] exploited two contrastive abnormal attention
models to improve the performance of thoracic multi-disease recognition. They
designed a left-right lung contrastive network to learn intra-attentive
abnormal features, and an inter-contrastive abnormal attention model to
compare healthy samples with multiple unhealthy samples to compute the
abnormal attention map. However, the uncertainty of the normal control may
lead to the uncertainty of performance. Regularizing the models, as another
strategy, involves no such sampling problem. Juan et al. [13] proposed a
neural graph learning framework leveraging graph structures to regularize
training, where the similarity between images is used as edges. Graph-Rise
[13] used the distance graph between images for regularization. CCG [34]
constructed graph nodes using clustered regions and lacked interpretability.
The proposed GREN, in contrast, is free from additional manual annotation and
the regions has explicit anatomical definitions.

Figure 2: Flowchart of the proposed GREN. The main part conducts weakly-
supervised localization using a CNN-based feature extractor and a task head
optimized via MIL. The graph-induced regularization module (in blue
background) models the inter-image and intra-image similarities, which are
used to regularize the representations of the main task for better structural
and contextual awareness. The CNNs (FCs) refer to the same model.

##  3 Method

The architecture of GREN is presented at Figure 2. The input images
$X=\\{x_{1},x_{2},...,x_{n}\\}$ are chest X-ray images in a training batch.
They go through a parenchymal segmentation network [1], and the left lung
region and right lung region of each image are thus obtained, which is
$X^{{}^{\prime}}=\\{x_{1}^{l},x_{1}^{r},x_{2}^{l},x_{2}^{r},...,x_{n}^{l},x_{n}^{r}\\}$.
Then, the input images $X$ and the images of lung regions $X^{{}^{\prime}}$
are fed to the feature extractor to yield feature maps
$F_{Inter}=\\{f_{1},f_{2},...,f_{n}\\}$ and
$F_{Intra}=\\{f_{1}^{l},f_{1}^{r},f_{2}^{l},f_{2}^{r},...,f_{n}^{l},f_{n}^{r}\\}$.
The distance of the feature map $F_{Intra}$ and the feature map $F_{Inter}$ is
computed to obtain the intra-image distance and inter-image distance,
respectively. At the same time, the Hash coding with $M$ digits is used to
represent each patch. We calculate the Hamming distances between paired
patches, resulting into the intra-image and inter-image similarity. Finally,
the intra-image similarity and inter-image similarity are used to minimize the
intra-image distance and the inter-image distance, respectively. A task head
is added upon the embedding space to complete the weakly-supervised
localization task. Please note that the feature extractor and the task head
share weights. GREN aims to utilize the structural relationship of images and
formulate a more informative embedding space to improve the localization
performance.

###  3.1 Backbone

The ResNet-50 pre-trained on the ImageNet dataset is used as the feature
extractor of GREN. The input images are
$X=\\{x_{1},x_{2},...,x_{n}\\},x_{i}\in\mathbb{R}^{512\times 512}$. After
removing the final classification and global pooling layer of ResNet-50, the
feature map $F=\\{f_{1},f_{2},...,f_{n}\\},f_{i}\in\mathbb{R}^{2048\times
16\times 16}$ is extracted, which is 32 times down-sampled compared with the
input. Meanwhile, the input image is divided into $P\times P$ patch grids, and
for each grid, the existent probability of disease is predicted by the
network. Note that $P$ is an adjustable hyper-parameter ($P$=16 in our
experiments). Then, we pass $F$ through two $1\times 1$ convolutional layers
and a sigmoid layer to obtain the final predictions of $P\times P$ with $K$
channels, where $K$ is the number of possible disease types. Finally, we
compute losses and make predictions in each channel for the corresponding
class, following the paradigm used in [18] and [16].

For images with _box-level_ annotations, if the grid in the feature map is
covered partially by the ground truth box, we assign label 1 to the grid.
Otherwise, we assign 0 to it. We use the binary cross-entropy (BCE) loss for
each grid:

$L^{k}_{i}(\theta)_{1}=\sum_{j}-y_{ij}^{k}\log(p_{ij}^{k})-\sum_{j}(1-y_{ij}^{k})\log(1-p_{ij}^{k}),$
(1)

where $k$, $i$, and $j$ are the indexes of class, sample and grid,
respectively. $y^{k}_{ij}$ denotes the target label of the $j$th grid in the
$i$th image for the class $k$, and $p^{k}_{ij}$ denotes the predicted
probability of the $j$th grid in the $i$th image for the class $k$.

For images with only _disease-level_ annotations, we use the MIL loss:

$\begin{split}L^{k}_{i}(\theta)_{2}=-&y^{k}_{i}\log(1-\prod_{j}(1-p^{k}_{ij}))\\\
-&(1-y^{k}_{i})\log(\prod_{j}(1-p^{k}_{ij})),\end{split}$ (2)

where $y^{k}_{i}$ denotes the target label of the image. For the backbone
network, the loss $L(\theta)$ is formulated as

$\begin{split}L(\theta)=\sum_{i}\sum_{k}\beta\lambda^{k}_{i}L^{k}_{i}(\theta)_{1}+(1-\lambda^{k}_{i})L^{k}_{i}(\theta)_{2},\end{split}$
(3)

where $\theta$ denotes the network parameters, $\lambda^{k}_{i}\in{0,1}$
denotes if the $k_{th}$ class in the $i_{th}$ sample has the box annotation,
and $\beta$ is the balance weight of the two losses and is set to 4.

###  3.2 Graph Regularization

Although existing works indeed paved the way to locate and identify diseases
in chest X-ray images with limited box-level labels, there are still abundant
information about the inter-image and intra-image structural relationship
(similarity) that can be exploited to compensate for the lack of labels. These
structural similarity can be naturally represented as a graph, where the graph
nodes denote the images and the edges denote their relationships. To model the
structural similarities, we train the network using the graph regularization.
Namely, _the more similar the images are, the closer their representations are
in the embedding space._ To achieve this goal, we exploit two graph
regularization terms to encourage similar images (within batches) to move
closer in the embedding space.

In an X-ray image, the areas containing the common lesions are the left and
right lung regions, and the clavicle area and the blank area outside the body
contain rare information. Therefore, the left and right lung regions of all
samples are extracted by the segmentation algorithm [1]. The similarity is
then calculated to measure the structural relationships of lung regions. We
opt to computational efficiency and choose the Hamming distance and cosine
distance for CNNs during computation.

Hash coding [27] is an algorithm that produces a snippet or fingerprint of
images, which analyses the image structure and shows whether two images look
nearly identical. We construct the relation graph based on Hash coding for two
reasons. First, Hash coding has a higher computation and storage efficiency,
which can fast calculate the Hamming distance to obtain the similarity of two
images without additional manual annotation [15]. Second, Hash coding
preserves structural information of X-ray images, which is ideal for the
localization task. Since it generates a low frequency signature in the
spectrum space, the low frequency signature of the monochromatic X-ray image
can be viewed as structural information. The insight has been verified in [16]
and [18]. Thus, we use the Hash coding and Hamming distance to measure the
similarity of X-ray images. Specifically, the perceptual image Hash is
designed not to change much when an image undergoes minor modifications such
as compression, color-correction, and brightness change [21].

The computation has three steps, including image pre-processing, perceptual
feature extraction, and quantization or compression to generate the Hash
string. First, an image is pre-processed by resizing, color transformation and
normalization. This step reduces the size of the data, and thus reducing the
processing time. Second, the discrete cosine transform (DCT) is used to make
perception feature extraction of an image invariable to content preserving
manipulation. Third, numerical values that represent features of an image are
quantized, generating a fixed sized Hash which is a compact and somewhat
unique representation of an image [21]. Finally, the Hamming distance is used
to quantify the similarity of two Hash codings. Additionally, the cosine
similarity is a measurement that quantifies the similarity between two or more
vectors, which is the cosine of the angle between vectors, and it is most
commonly used in high-dimensional features of CNNs [32]. We use the cosine
distance as another similarity algorithm to measure the similarity of the CNN
features.

####  3.2.1 Intra-image Knowledge Learning

The left and right lung regions in chest X-ray image show some symmetric
structures, and the lesions of most diseases rarely appear symmetrically in
both sides, such as pneumonia, infiltration and consolidation [37]. It is very
useful to compare and distinguish the differences (e.g., textures and shadows)
between the two lung regions for radiologists.

The structural relationship of X-ray regions is modeled as a graph
$G_{Intra}=(N,E)$, where the nodes $N$ denote the two regions of left lung and
right lung. The edges $e_{i}^{lr}\in E$ denote the similarity of the graph
nodes $N$,

$e_{i}^{lr}=\frac{1}{D_{e}}H(h_{i}^{l},h_{i}^{r}),$ (4)

where $h_{i}^{l}$ and $h_{i}^{r}$ are the representations of the Hash coding
of the left lung regions $x_{i}^{l}$ and the right lung regions $x_{i}^{r}$ in
$i_{th}$ image, respectively. It is obtained by using perceptual hashing
algorithm. $H(h_{i}^{l},h_{i}^{r})$ calculates the Hamming distance of
$h_{i}^{l}$ and $h_{i}^{r}$. $D_{e}$ is a normalized parameter, which makes
$e_{i}^{lr}\in(0,1)$. The value of it is the product of the length of the Hash
coding ($D_{e}$ = 64 in our experiments). The graph regularization term
$D(\theta)_{Intra}$ is obtained,

$D(\theta)_{Intra}=-\sum_{i}e_{i}^{lr}d(f_{i}^{l},f_{i}^{r}),$ (5)

where $e_{i}^{lr}$ denotes the similarity of the left lung region $x_{i}^{l}$
and the right lung region $x_{i}^{r}$, $d(*)$ is the distance metric function,
which is the Euclidean distance, $f_{i}^{l}$ and $f_{i}^{r}$ mean the feature
map of the left lung region $x_{i}^{l}$ and the right lung $x_{i}^{r}$,
respectively. Please note that the $f_{i}^{l}$ and $f_{i}^{r}$ are obtained by
taking the masks of the left lung region $x_{i}^{l}$ and the right lung region
$x_{i}^{r}$ in the feature map $f_{i}$.

####  3.2.2 Inter-image Knowledge Learning

There are phenomena of “different X-ray images of the similar diseases” and
“different diseases of the similar X-ray images”, so it is difficult for the
deep learning models to locate and classify diseases with limited box-level
labels. Therefore, we propose a inter-image knowledge learning module to dig
out the structural differences between different X-ray images, which provides
a large difference for the similar diseases and similar X-ray images.

A pair of images $x_{u}$ and $x_{v}$ are randomly selected from the input
images $X$. The structural relationship information of different X-ray images
is modeled to get the graph $G_{Inter}=(N,E)$, where the graph nodes $N$
denote the images in a mini-batch of the network ($N=4$ in our experiments),
the edges $e_{uv}\in E$ denote the similarity of graph nodes $N$. Since there
are left and right lung regions in an image, the Hamming distance of two lung
regions in all images should be calculated,

$\begin{split}e_{uv}=\frac{1}{2D_{e}}(H(h_{u}^{l},h_{v}^{l})+H(h_{u}^{r},h_{v}^{r})),\end{split}$
(6)

where $h_{u}^{l}$, $h_{u}^{r}$, $h_{v}^{l}$ and $h_{v}^{r}$ are the Hash
codings of left and right lung regions of the image $x_{u}$ and the image
$x_{v}$, respectively. $H(*)$ is the Hamming distance calculation. Note that
the Hash coding generation is training-free. $D_{e}$ is a normalization
parameter, which denotes the product of the length of the Hash coding ($D_{e}$
= 64 in our experiments). The graph regularization term $D(\theta)_{Inter}$ is
written as

$D(\theta)_{Inter}=-\sum_{u,v}e_{uv}d(f_{u},f_{v}),$ (7)

where $e_{uv}$ denotes the similarity of $x_{u}$ and $x_{v}$, and $d(*)$ is
the Euclidean distance, and $f_{u}$ and $f_{v}$ mean the feature maps of image
$x_{u}$ and image $x_{v}$, respectively.

The final objective $Q(\theta)$ is the summation of the loss $L(\theta)$ of
the localization network and the graph regularization terms
$D(\theta)_{Intra}$ and $D(\theta)_{Inter}$,

$Q(\theta)=L(\theta)+\lambda_{1}D(\theta)_{Intra}+\lambda_{2}D(\theta)_{Inter},$
(8)

where the hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ control the balance
between the three losses and are set to 0.11 and 0.15 by searching from 0 to 1
with a step of 0.05 and 0.01 for coarse-to-fine searching. For example, it is
assumed that the hyper-parameter in coarse searching is 0.10 (with the step of
0.05), and the interval in fine searching is [0.96, 0.99] and [0.11, 0.14]
(with the step of 0.01). When $\lambda_{1}$ = 0 and $\lambda_{2}$ = 0, the
network reduces to a localization model without the graph regularization.

Table 1: Performance comparison of disease localization using 50% unannotated
images and 80% annotated images. Note that the data partition settings are
inherited from existing studies for fair comparisons. For each column, the
bold or red values denote the best results.

T (IoU) | Models | Atelectasis | Cardiomegaly | Effusion | Infiltration | Mass | Nodule | Pneumonia | Pneumothorax | Mean
---|---|---|---|---|---|---|---|---|---|---
0.5 | Wang et al. [30] | 0.05 | 0.18 | 0.11 | 0.07 | 0.01 | 0.01 | 0.03 | 0.03 | 0.06
Li et al [16] | 0.14 | 0.84 | 0.22 | 0.30 | 0.22 | 0.07 | 0.17 | 0.19 | 0.27
Liu et al. [18] | 0.32 | 0.78 | 0.40 | 0.61 | 0.33 | 0.05 | 0.37 | 0.23 | 0.39
Zhou et al. [37] | 0.39 | 0.86 | 0.46 | 0.65 | 0.39 | 0.13 | 0.43 | 0.27 | 0.45
Zhao et al. [34] | 0.27 | 0.86 | 0.48 | 0.72 | 0.53 | 0.14 | 0.58 | 0.35 | 0.49
Baseline (B) | 0.12 | 0.79 | 0.32 | 0.36 | 0.26 | 0.07 | 0.13 | 0.20 | 0.28
B+Intra | 0.37 | 0.86 | 0.48 | 0.80 | 0.37 | 0.14 | 0.58 | 0.30 | 0.49
B+Inter | 0.32 | 0.89 | 0.52 | 0.80 | 0.42 | 0.21 | 0.50 | 0.35 | 0.50
| GREN (B+Intra+Inter) | 0.37 | 0.86 | 0.52 | 0.84 | 0.42 | 0.29 | 0.54 | 0.45 | 0.54
0.7 | Wang et al. [30] | 0.01 | 0.03 | 0.02 | 0.00 | 0.00 | 0.00 | 0.01 | 0.02 | 0.01
Li et al. [16] | 0.04 | 0.52 | 0.07 | 0.09 | 0.11 | 0.01 | 0.05 | 0.05 | 0.12
Liu et al.[18] | 0.18 | 0.70 | 0.28 | 0.41 | 0.27 | 0.04 | 0.25 | 0.18 | 0.29
Zhou et al. [37] | 0.24 | 0.75 | 0.33 | 0.45 | 0.33 | 0.09 | 0.35 | 0.23 | 0.35
Zhao et al. [34] | 0.20 | 0.86 | 0.48 | 0.68 | 0.32 | 0.14 | 0.54 | 0.30 | 0.44
Baseline (B) | 0.05 | 0.71 | 0.16 | 0.04 | 0.16 | 0.00 | 0.08 | 0.15 | 0.17
B+Intra | 0.27 | 0.86 | 0.44 | 0.72 | 0.32 | 0.14 | 0.58 | 0.20 | 0.44
B+Inter | 0.29 | 0.86 | 0.48 | 0.68 | 0.37 | 0.21 | 0.46 | 0.35 | 0.46
| GREN (B+Intra+Inter) | 0.34 | 0.86 | 0.48 | 0.60 | 0.37 | 0.21 | 0.46 | 0.35 | 0.46

####  3.2.3 CNN Features Instead of Hash Coding

To calculate the similarity between regions, it is most intuitive to use CNN
features. To investigate on this, we replace the similarity computing with CNN
features to compare with the Hash coding with Hamming distance. The feature
map
$F_{Intra}=\\{f_{1}^{l},f_{1}^{r},f_{2}^{l},f_{2}^{r},...,f_{n}^{l},f_{n}^{r}\\}$
is extracted using the same method as described previously. The similarities
between the feature maps are calculated according to

$s_{uv}=\frac{1}{2}(d_{cos}(f_{u}^{l},f_{v}^{l})+d_{cos}(f_{u}^{r},f_{v}^{r})),s^{lr}_{i}=d_{cos}(f^{l}_{i},f^{r}_{i}),$
(9)

where $f_{u}^{l(r)}$ and $f_{v}^{l(r)}$ are the feature maps of the left
(right) regions of the image $x_{u}$ and image $x_{v}$. $s_{uv}$ is the
similarity of feature maps $f_{u}$ and $f_{v}$. $s^{lr}_{i}$ is the similarity
of the right regions and left regions in $i_{th}$ feature maps. $d_{cos}(*)$
is the cosine distance of the feature maps. The graph regularization term
$D(\theta)_{Inter}^{s}$ and $D(\theta)_{Intra}^{s}$ are written as

$\displaystyle D(\theta)^{s}_{Inter}=-\sum_{u,v}s_{uv}d(f_{u},f_{v}),$ (10)
$\displaystyle
D(\theta)^{s}_{Intra}=-\sum_{i}s^{lr}_{i}d(f_{i}^{l},f_{i}^{r}),$

where $d(*)$ is the Euclidean distance metric. $f_{u}$ and $f_{v}$ mean the
feature maps of image $x_{u}$ and image $x_{v}$, respectively. The final
objective $Q(\theta)^{s}$ is the summation of the localization loss
$L(\theta)$ and the graph regularization loss $D(\theta)_{Inter}^{s}$ and
$D(\theta)_{Intra}^{s}$,

$Q(\theta)^{s}=L(\theta)+\lambda_{3}D(\theta)^{s}_{Inter}+\lambda_{4}D(\theta)^{s}_{Intra},$
(11)

where the hyper-parameter $\lambda_{3}$ and $\lambda_{4}$ control the balance
between the three losses and are set to 0.15 with the same method as described
previously.

##  4 Experiments Results

###  4.1 Datasets and Evaluation Metrics

The NIH chest X-ray dataset [30] consists of 112,120 frontal-view X-ray images
with 14 classes of diseases. Furthermore, the dataset contains 880 images with
984 labeled bounding boxes, and the provided bounding boxes only have 8 type
of disease instances. Since we pay more attention to the task of locating
diseases, we follow the terms in [16] and [18] to call the 880 images with
labeled bounding boxes as ‘annotated’ and the remaining 111,240 images as
‘unannotated’ (Figure 3). For fast processing, we resize the original
3-channel images from resolution of $1024\times 1024$ to $512\times 512$
without any data augmentation techniques.

We follow the metrics used in [16]. For localization, the intersection over
union (IoU) between predictions and ground truths is used to evaluate the
performance of the models. The localization results are regarded as correct
when $IoU>T(IoU)$, where $T(*)$ is the threshold. In practice, we usually pay
most attention to the accuracy of high thresholds of T(IoU), so two thresholds
of 0.5 and 0.7 are set in our experiments. For more accurate localization with
limited data with location annotations, the localization predictions are
discrete small rectangles. The performance of the eight diseases with ground
truth boxes is reported, which is inherited from existing researches [16] and
[18].

Figure 3: Three participation methods.

###  4.2 Experimental Settings

All the models are trained on the NIH chest X-ray dataset using the stochastic
gradient descent (SGD) algorithm with the Nesterov momentum. The learning rate
starts from 0.001 and decreases by 10 times after every 4 epochs with a total
of 9 epochs. The weight decay is 0.0001 and the momentum is 0.9. All the
weights are initialized with pre-trained ResNet-50 [9] on ImageNet [7]. The
mini-batch size is set to 4 with the NVIDIA 2080Ti GPU. All the algorithms are
implemented with PyTorch [20]. The threshold of 0.5 is used to distinguish
positive grids from negative grids in the class-wise feature map, which has
been adopted in previous studies [16] and [18]. Note that the up-sampling
operation [18] of feature maps is used to gain a more accurate localization
for lesion location before two last fully convolutional layers. The paired
images are randomly sampled in the train phase, and the test phase does not
need the paired images.

###  4.3 Comparison with State-of-the-Art

In order to effectively evaluate GREN in weakly supervised disease
localization, we strictly follow [16], [34], [18] and use three data
participation methods (Figure 3) and compute the average values with a _five-
fold cross validation_ in each participation. In the first participation, we
use 50% unannotated images and 80% annotated images for training and the
remaining 20% annotated images for testing (0.5_0.8). In the second
participation, we use the 100% unannotated images without annotated images for
training and all annotated images for testing (1.0_0.0). In the third
participation, we use the 100% unannotated images and 40% annotated images for
training and the remaining 60% annotated images for testing (1.0_0.4).

Our experiment results are compared with six state-of-the-art methods. The
first method is Wang et al. [30], which uses a multi-label disease
classification model with disease heatmaps for location disease. The second
method is Li et al. [16], which integrates the multi-instance level loss and
binary cross-entropy loss into one framework for disease localization. The
third method is Liu et al. [18], which utilizes the contrast-induced attention
acquired on paired images between healthy and unhealthy samples to provide
information for localization. The fourth method is Zhou et al. [37], which
exploits two contrastive abnormal attention models and a dual-weighting graph
convolution to improve the performance of thoracic multi-disease recognition.
The fifth method is Zhao et al. [35], which proposes the contralateral context
information for enhancing feature representations of disease proposals by
using a spatial transformer network. The sixth method is Zhao et al. [34],
which proposed the CCG to regularize the model using the similarities computed
via of clustered regions.

Table 2: Performance comparison of disease localization using unannotated
images only.

T(IoU) | Models | Atelectasis | Cardiomegaly | Effusion | Infiltration | Mass | Nodule | Pneumonia | Pneumothorax | Mean
---|---|---|---|---|---|---|---|---|---|---
0.5 | Wang et al.∗ [30] | 0.00 | 0.25 | 0.01 | 0.06 | 0.09 | 0.05 | 0.06 | 0.00 | 0.07
Li et al.∗ [16] | 0.18 | 0.10 | 0.27 | 0.46 | 0.18 | 0.03 | 0.00 | 0.11 | 0.17
Liu et al. [18] | 0.19 | 0.53 | 0.19 | 0.47 | 0.33 | 0.03 | 0.08 | 0.11 | 0.24
Zhao et al. [35] | 0.32 | 0.68 | 0.39 | 0.61 | 0.49 | 0.07 | 0.15 | 0.21 | 0.36
Zhao et al. [34] | 0.31 | 0.79 | 0.37 | 0.75 | 0.40 | 0.06 | 0.24 | 0.27 | 0.40
Baseline (B) | 0.18 | 0.51 | 0.14 | 0.47 | 0.27 | 0.03 | 0.01 | 0.12 | 0.22
B+Intra | 0.30 | 0.79 | 0.38 | 0.72 | 0.42 | 0.05 | 0.25 | 0.27 | 0.40
B+Inter | 0.31 | 0.79 | 0.37 | 0.70 | 0.41 | 0.08 | 0.08 | 0.24 | 0.37
| GREN (B+Intra+Inter) | 0.36 | 0.86 | 0.41 | 0.70 | 0.49 | 0.10 | 0.11 | 0.22 | 0.41
0.7 | Wang et al.∗ [30] | 0.00 | 0.01 | 0.00 | 0.03 | 0.04 | 0.04 | 0.01 | 0.00 | 0.01
Li et al.∗ [16] | 0.09 | 0.01 | 0.07 | 0.28 | 0.08 | 0.01 | 0.00 | 0.05 | 0.07
Liu et al. [18] | 0.08 | 0.30 | 0.09 | 0.25 | 0.19 | 0.01 | 0.04 | 0.07 | 0.13
Zhao et al. [35] | 0.13 | 0.54 | 0.19 | 0.27 | 0.27 | 0.04 | 0.05 | 0.17 | 0.21
Zhao et al. [34] | 0.06 | 0.64 | 0.08 | 0.38 | 0.19 | 0.01 | 0.08 | 0.09 | 0.19
Baseline (B) | 0.11 | 0.34 | 0.06 | 0.32 | 0.20 | 0.01 | 0.00 | 0.06 | 0.14
B+Intra | 0.21 | 0.74 | 0.22 | 0.62 | 0.35 | 0.03 | 0.18 | 0.18 | 0.32
B+Inter | 0.20 | 0.75 | 0.25 | 0.55 | 0.38 | 0.04 | 0.06 | 0.21 | 0.30
| GREN (B+Intra+Inter) | 0.28 | 0.77 | 0.22 | 0.61 | 0.44 | 0.06 | 0.09 | 0.16 | 0.33

Here, a model needs to predict the location of eight diseases, so we pay more
attention to the overall performance of the model rather than to a particular
disease. Therefore, in the following discussion, we are most concerned with
the mean accuracy. In the first participation, we compare the localization
results of our model with [30], [16], [18], [37] and [34]. We can observe that
GREN outperforms the existing methods in most cases, as shown in Table 1.
Particularly, with the increase of T(IoU), GREN has greater advantages over
the reference models. For examples, when T(IoU) = 0.5, the mean accuracy is
0.54, and outperforms [30], [16], [18], [37] and [34] by 0.48, 0.27, 0.15,
0.09 and 0.05. However, when T(IoU) = 0.7, the mean accuracy of GREN is 0.46,
and outperforms [30], [16], [18], [37] and [34] by 0.45, 0.34, 0.17, 0.11 and
0.02. In the second participation, we train our model without any annotated
images. Since [16] only provides the results at T(IoU) = 0.1, in order to
better show the performance of our model, we reproduce the results of T(IoU) =
0.5 and 0.7. Note that the sysbol $*$ denotes the reproduced results. For each
disease, the bold and red values denote the best results of the eight disease
types and mean accuracy, respectively. We compare the localization results of
our model with [30], [16], [18], [35] and [34]. It can be seen that GREN
outperforms existing methods in most cases, as shown in Table 2. In the third
participation, we use more annotated images comparing the second
participation. We compare the localization results of GREN with [16], [18] and
[34] in exact data setting. It can be seen that GREN still has the advantages
over the existing methods, as shown in Table 3.

Overall, the experiment results demonstrate that GREN achieves state-of-the-
art results for disease localization. Even when there is no annotated data for
training, GREN still achieves decent localization results. Specifically,
compared with the methods without image relations, the methods of modeling the
relations in X-ray image have better performance. For example, the mean
accuracies of [18], [34], [35], [37] and GREN outperform [30], [16]. The
comparative results show that image relations are useful in weakly-supervised
medical image analysis. Compared with the methods that using only inter-image
or intra-image relationship [18], [34], [35], [37], GREN that integrates the
intra-image and inter-image relationship is obviously superior to these
methods. It can be concluded that GREN is effectively applied to the weakly
supervised disease localization tasks.

Table 3: Performance comparison of disease localization using 100% unannotated
images and 40% annotated images. For each column, the bold or red values
denote the best results. Note that the sysbol $*$ denotes the reproduced
results.

T(IoU) | Models | Atelectasis | Cardiomegaly | Effusion | Infiltration | Mass | Nodule | Pneumonia | Pneumothorax | Mean
---|---|---|---|---|---|---|---|---|---|---
0.5 | Li et al.∗ [16] | 0.23 | 0.72 | 0.30 | 0.60 | 0.22 | 0.02 | 0.32 | 0.20 | 0.32
Liu et al. [18] | 0.36 | 0.57 | 0.37 | 0.62 | 0.34 | 0.13 | 0.23 | 0.17 | 0.35
Zhao et al. [34] | 0.26 | 0.80 | 0.41 | 0.67 | 0.15 | 0.06 | 0.42 | 0.18 | 0.37
Baseline (B) | 0.27 | 0.76 | 0.39 | 0.58 | 0.24 | 0.02 | 0.39 | 0.21 | 0.36
B+Intra | 0.44 | 0.91 | 0.60 | 0.82 | 0.52 | 0.13 | 0.58 | 0.33 | 0.54
B+Inter | 0.45 | 0.74 | 0.62 | 0.85 | 0.43 | 0.19 | 0.70 | 0.33 | 0.54
| GREN (B+Intra+Inter) | 0.42 | 0.83 | 0.60 | 0.86 | 0.54 | 0.10 | 0.71 | 0.36 | 0.55
0.7 | Li et al.∗ [16] | 0.07 | 0.64 | 0.17 | 0.38 | 0.17 | 0.00 | 0.20 | 0.17 | 0.21
Liu et al. [18] | 0.19 | 0.47 | 0.20 | 0.41 | 0.22 | 0.06 | 0.12 | 0.11 | 0.22
Zhao et al. [34] | 0.18 | 0.71 | 0.20 | 0.50 | 0.20 | 0.02 | 0.29 | 0.06 | 0.27
Baseline (B) | 0.14 | 0.62 | 0.20 | 0.42 | 0.07 | 0.00 | 0.23 | 0.08 | 0.22
B+Intra | 0.35 | 0.89 | 0.49 | 0.76 | 0.37 | 0.06 | 0.51 | 0.26 | 0.46
B+Inter | 0.38 | 0.71 | 0.49 | 0.72 | 0.43 | 0.13 | 0.58 | 0.24 | 0.46
| GREN (B+Intra+Inter) | 0.32 | 0.78 | 0.52 | 0.74 | 0.43 | 0.06 | 0.65 | 0.26 | 0.47

###  4.4 Ablation Studies

####  4.4.1 Ablation of the Inter and Intra Construction

We compare the localization results of the baseline model (B) with three
models under different configurations, including the model with the intra-
image knowledge learning module (B+Intra), the model with the inter-image
knowledge learning module (B+Inter), and the model with both knowledge
learning modules (B+Intra+Inter). The experiment results in Table 1, Table 2
and Table 3 show that the model (B+Intra) and the model (B+Inter) demonstrate
more advantages over the baseline method, and the model (B+Intra+Inter)
achieves the best localization result. For example, in Table 1, when T(IoU) =
0.5, the mean accuracy of the model (B+Intra) and the model (B+Inter) are 0.49
and 0.50, and outperform the baseline (B) by 0.21 and 0.22, respectively. The
model (B+Intra+Inter) achieves the best localization results, which is 0.54,
and outperforms the baseline (B) by 0.26. Additionally, the localization
results in Table 1 and Table 2 demonstrate that when there is more annotated
data in the training process, the model using the inter-image knowledge
learning module has more advantages; when there is no annotated data in the
training process, the model using the intra-image knowledge learning module
has more advantages. For example, the model (B+Inter) performs better in most
cases compared with the model (B+Intra) in Table 1, and the model (B+Intra)
performs better in most cases compared with the model (B+Inter) in Table 2.
Overall, the experiment results demonstrate that using intra-image and inter-
image structural relationship information can improve the performance of
models for disease localization. Moreover, the experiment results show that
the combined effect of the inter and intra construction can further boost the
performance of weakly supervised disease localization.

Figure 4: (a) The performance comparison of the baseline model, the CNN model
(B+Cosine) and GREN in three data settings, each of which includes the mean
accuracies with T(IOU)=0.5 and 0.7. (b) The performance comparison of
different batch sizes. The batch size is set to 2, 4, 8, and 16. The mean
accuracies improve with the increase of batch size, but the trend is not an
unbounded growth. Note that the data partition settings are inherited from
existing studies for fair comparisons. Table 4: Performance comparison of
disease localization using 50% unannotated images and 80% annotated images.

T (IoU) | Models | Atelectasis | Cardiomegaly | Effusion | Infiltration | Mass | Nodule | Pneumonia | Pneumothorax | Mean
---|---|---|---|---|---|---|---|---|---|---
0.5 | Baseline (B) | 0.12 | 0.79 | 0.32 | 0.36 | 0.26 | 0.07 | 0.13 | 0.20 | 0.28
B+Attention | 0.34 | 0.89 | 0.52 | 0.72 | 0.37 | 0.21 | 0.71 | 0.15 | 0.49
GREN | 0.37 | 0.86 | 0.52 | 0.84 | 0.42 | 0.29 | 0.54 | 0.45 | 0.54
0.7 | Baseline (B) | 0.05 | 0.71 | 0.16 | 0.04 | 0.16 | 0.00 | 0.08 | 0.15 | 0.17
B+Attention | 0.29 | 0.89 | 0.44 | 0.64 | 0.32 | 0.14 | 0.58 | 0.15 | 0.43
GREN | 0.34 | 0.86 | 0.48 | 0.60 | 0.37 | 0.21 | 0.46 | 0.35 | 0.46

####  4.4.2 Ablation of Similarity Computing Methods

To investigate the similarity computing method, we compare the Hash coding
using Hamming distance with CNN features using cosine similarity (B+Cosine),
which is shown in Figure 4(a). The B+Cosine model is competitive to the
baseline model but is inferior to GREN for all the three data settings. There
are two possible reasons. First, it is not easy to obtain a CNN-based model
that needs to not only perform as a feature extractor but also learn class-
aware similarity. Hash coding solved this problem via projecting features to a
low frequency signature. Second, as a comparison, Hash coding is data-
independent and training-free. Overall, the structural relationship
(similarity) can be used to regularize the training of deep neural networks,
so as to achieve more accurate and integral localization results with few
careful annotations. Moreover, using Hash coding and Hamming distance instead
of the cosine distance of CNN features can achieve better performance, which
demonstrates that the image features extracted directly using Hash coding can
to some extent compensate for the information lost in CNNs, making the
proposed method reasonable.

####  4.4.3 The Number of Graph Nodes

We explore the influence of different number of graph nodes $N$ on the
results, which is shown in Figure 4(b). The value is equal to the batch size
of the model, including 2, 4, 8, and 16. The data of 100% unannotated images
and 40% annotated images are used to evaluate the performance. The mean
accuracies improve with the increase of batch size, whereas the trend is not
an unbounded growth. For example, when T(IoU) = 0.7, the mean accuracy of the
model with batch size of 16 is 0.47, outperforming the model with batch size
of 2 by 0.03. However, when T(IoU) = 0.5, the mean accuracy of the model with
batch size of 16 is 0.54, which is the same as the model with the batch size
of 4. Using more images in each graph is good to effective regularization, but
the growth is bounded (slow after the batch size exceeds 4). Therefore, the
batch size is set to 4 in our experiment by considering the trade-off between
memory cost burden and performance.

####  4.4.4 Ablation of Inter-image and Intra-image Similarity

Graph is a concise and effective way to characterize inter- and intra-image
relations. There are several works to investigate the relationship and achieve
a superior performance in multi-label natural image recognition, such as [5].
In the X-ray image analysis, the inter- and intra-image relations can be used
to simulate the physician’s practice of comparing multiple images and
symmetric regions of the same image for diagnosis respectively. Existing
literatures [35], [34] imply that the relationship between the left and right
lung lobes is useful in automatic diagnosis. However, these methods do not
explore the underlying reasons why the inter- and intra- relationship modeling
is effective. Here, we construct relation graphs using the left and right lung
lobes as the nodes of graphs, and then incorporate the graph as a regularizer
to facilitate the representation learning. Specially, we try to explore the
underlying reasons of this.

We explore the influence of the inter-image and intra-image similarity as
shown in Figure 5. We randomly choose 10 images from the dataset. In Figure
5(a), the horizontal and longitudinal axis are the same lung region images,
and the two adjacent images have the same classes. We calculate the similarity
of lung regions of these images using the Hash coding and Hamming distance.
The two adjacent images have higher similarity compared with other images,
where the darker color denotes higher similarity. For example, the diagonal
rectangles are the similarities of the same images, so the values are 1.
Additionally, we calculate the similarity of the left lung and right lung
regions in an image, which is diagonal rectangle, and other rectangles are
filled with the same color. In Figure 5(b), the horizontal and longitudinal
axis are the corresponding left lung and right lung regions in an image. The
darker-colored rectangle denotes higher similarity of the left lung and right
lung in an image. Particularly, the sixth and ninth darker color rectangles
(from upper left to low right) imply that the sixth and ninth images have
highly similar left and right lung, and their categories are actually “No
Finding” (The categories of the second, sixth, and ninth images are “No
Finding”). The first, fourth, and eighth light color rectangles imply that the
left and right lung regions of the first, fourth, and eighth images have low
similarity, and their categories are “Cardiomegaly”, “Cardiomegaly”, and
“Pneumonia”, respectively. To some degree, the inter-image similarity can
indicate the categories of images without annotation data, and the intra-image
similarity can indicate the information of differences of left and right lung
regions, which is helpful to distinguish the unilateral disease of X-ray
images.

Figure 5: The inter-image and intra-image similarities. Each color denotes a
value of similarity, where the darker color denotes higher similarity. (a).
The inter-image similarity. The horizontal and longitudinal axis are the same
lung region images. The two adjacent images have the same classes and higher
similarity compared with other images. (b). The intra-image similarity. The
horizontal and longitudinal axis are the left lung region and right lung
region in an image. Please note that the same color boxes denote the same
classes of the images and the grey boxes denote the image only has one class.
Table 5: The AUC (%) scores of the baseline model and GREN. The bold or red
values denote the best results.

Models | Atelectasis | Cardiomegaly | Consolidation | Edema | Effusion | Emphysema | Fibrosis |
---|---|---|---|---|---|---|---|---
Baseline (B) | 76.77 | 87.09 | 77.73 | 87.90 | 84.65 | 90.04 | 77.87 |
B+Intra | 77.39 | 87.78 | 76.16 | 87.13 | 85.24 | 90.98 | 77.83 |
B+Inter | 76.61 | 86.17 | 77.02 | 89.00 | 84.16 | 90.22 | 79.43 |
GREN | 77.57 | 88.19 | 79.01 | 89.25 | 85.11 | 91.98 | 79.50 |
Models | Hernia | Infiltration | Mass | Nodule | Pleural Thickening | Pneumonia | Pneumothorax | Mean
Baseline (B) | 88.71 | 64.53 | 80.57 | 68.37 | 77.68 | 67.62 | 85.62 | 79.65
B+Intra | 91.84 | 64.84 | 80.59 | 73.01 | 77.62 | 67.06 | 85.99 | 80.25
B+Inter | 81.15 | 64.08 | 81.23 | 70.63 | 78.53 | 72.36 | 86.40 | 80.32
GREN | 86.65 | 66.04 | 81.34 | 72.07 | 78.96 | 67.47 | 87.25 | 80.74

Figure 6: Visualizations of the predicted results on both the baseline model
and our method. The first column shows the original images, the second and
third columns show the predicted results of the Baseline and GREN. The green
bounding box and red area mean the ground truth and prediction, respectively.

####  4.4.5 Ablation of Attention-based Method

We compare the localization results of our model with the attention-based
method, which integrates the attention mechanism into the baseline to learn
the difference of lung regions. In Table 4, the B+Attention model outperforms
the baseline by 0.21 and 0.26 at T(IoU) = 0.5 and 0.7, however, GREN still
holds advantages. It ourperforms the B+Attention model by 0.26 and 0.29 under
the two thresholds. Overall, the attention-based method can indeed improve the
accuracy, but GREN using the intra-image and inter-image structural
relationship has larger advantages to enhance the weakly-supervised disease
localization.

Figure 7: Visualizations of the predicted results on Baseline (B), B+Inter,
B+Intra and GREN. From left to right, it shows the localization results of the
baseline model, B+Intra, B+Inter and GREN. The green bounding box and red area
mean the ground truth and prediction, respectively.

Figure 8: visualizations of the predicted results on both [30] and GREN. The
first column shows the original images, the second and third columns show the
predicted results of the [30] and GREN. The green bounding box and red area
mean the ground truth and prediction, respectively. [30] only highlights the
most discriminative areas using CAM, and GREN makes more integral predictions.

####  4.4.6 Disease Identification Results

Although we focus on the weakly-supervised localization task, we additionally
evaluate the results of disease identification. Table 5 presents the AUC
scores (the area under the receiver operating characteristic curve) for all
the classes. Following [16] and [18], we use 70% images for training and 20%
images for testing. We compare the performance of disease identification of
the Baseline (B) with the B+Intra, B+Inter and GREN. The mean AUC scores of
the B+Intra, B+Inter and GREN are 80.25%, 80.32% and 80.74%, outperforming the
Baseline (79.65%) by 0.60%, 0.67% and 1.09%, respectively. GREN is more
effective in disease localization than disease identification because the task
of disease identification has more sufficient supervision. Overall, GREN can
maintain performance of disease identification, but enjoys better ability for
weakly-supervised disease localization. Particularly, GREN has better
application value in the task with limited annotation data.

####  4.4.7 Visualizations

To better demonstrate the final effect of our method on disease localization,
we visualize some of typical predictions of both the baseline model and GREN,
as shown in Figure 6. The first column shows the original images, the second
and the third column show the localization results of the baseline model and
GREN. The green bounding box and red area mean the ground truth and
prediction, respectively. It can be seen that GREN can predict more accurate
regions compared with the baseline model. For example, for the class of
“Nodule”, the localization results of the baseline model are completely
inconsistent with the ground truth, but the localization results of GREN are
consistent with the ground truth.

In order to demonstrate the difference of GREN (B+Intra+Inter) with B+Intra
and B+Inter, we visualize some of typical predictions of them, as shown in
Figure 7. From left to right, it shows the localization results of the
Baseline (B), B+Intra, B+Inter and GREN. GREN plays an important role in
integrating the advantages of B+Inter and B+Intra to make the predictions
better. For a sample (Figure 7), some of B+Inter results are better than
B+Intra results, such as (a), (b), (d), some of B+Intra results are better
than B+Inter results, such as (e), (f). It can be seen that GREN can modify
and complement the predicted results of B+Inter and B+Intra to make better
predictions. The underlying reason may be that the functions of Inter-image
and Intra-image knowledge learning module are not only complementary but also
share common mechanisms. For example, the lungs are highly structured. For an
X-ray image, there are not only similarities but also differences between left
and right lungs. For different X-ray images, there are not only differences
but also similarities in ipsilateral lung. Therefore, the functions of Inter-
image and Intra-image knowledge learning module cannot be completely
separated, and some overlap and crossover inevitably exist between them.

We also visualize some of typical predictions of both the CAM-based method
[30] and GREN, as shown in Figure 8. The first column shows the original
images, the second and the third column show the localization results of [30]
and GREN. It can be seen that GREN has greater advantages over [30]. Moreover,
compared with [30], GREN uses the regularization of the intra-image and inter-
image structural information to pay more attention to the lung regions than
other regions (e.g., clavicle and arm regions). It further demonstrates that
GREN can effectively improve the performance of weakly supervised disease
localization.

##  5 Conclusion

We have proposed and evaluated GREN, which leverages the intra-image and
inter-image information to regularize CNNs to preserve the structural
similarities between lung regions and image pairs in the embedding space. The
experiment results on the NIH Chest-14 dataset demonstrate that GREN achieves
the state-of-the-art performance in various settings. We demonstrate that the
relationship of intra-image and inter-image can be used to model the graphs to
facilitate training in weakly supervised disease localization. The proposed
method has a very practical use in the weakly-supervised localization task.
Since the public dataset available for X-ray lung segmentation only allowed us
to segment the left and right lungs, we build graphs based on two lungs.
However, many areas of the lung have explicit anatomical meanings, such as
right upper lobe, right middle lobe, right lower lobe, left upper lobe, and
left lower lobe. In theory, the more comprehensive lung regions are
considered, the more relationship information GREN can model. For future work,
we will exploit the relationship information of five lung lobes from CT images
to construct graphs to compensate for the lack of labels in weakly supervised
disease localization. We will also investigate algorithms applicable to the
localization of small targets (especially pulmonary nodules).

## References

  * [1] “https://pypi.org/project/lungs-segmentation/.”
  * [2] A. I. Aviles-Rivero, N. Papadakis, R. Li, P. Sellars, Q. Fan, R. T. Tan, and C.-B. Schönlieb, “Graphx${}^{NET}-$ chest x-ray classification under extreme minimal supervision,” 2020.
  * [3] B. Babenko, “Multiple instance learning: algorithms and applications,” _View Article PubMed/NCBI Google Scholar_ , pp. 1–19, 2008.
  * [4] J. Cai, L. Lu, A. P. Harrison, X. Shi, P. Chen, and L. Yang, “Iterative attention mining for weakly supervised thoracic disease pattern localization in chest x-rays,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 589–598.
  * [5] B. Chen, Z. Zhang, Y. Lu, F. Chen, G. Lu, and D. Zhang, “Semantic-interactive graph convolutional network for multilabel image recognition,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , pp. 1–13, 2021.
  * [6] C. Chen, Q. Dou, H. Chen, and P.-A. Heng, “Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation,” in _International workshop on machine learning in medical imaging_. Springer, 2018, pp. 143–151.
  * [7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [8] G. Gaál, B. Maga, and A. Lukács, “Attention u-net based adversarial architectures for chest x-ray lung segmentation,” _arXiv preprint arXiv:2003.10304_ , 2020.
  * [9] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [10] R. Hermoza, G. Maicas, J. C. Nascimento, and G. Carneiro, “Region proposals for saliency map refinement for weakly-supervised disease localisation and classification,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 539–549.
  * [11] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya _et al._ , “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 590–597.
  * [12] B. Jing, P. Xie, and E. Xing, “On the automatic generation of medical imaging reports,” _arXiv preprint arXiv:1711.08195_ , 2017.
  * [13] D.-C. Juan, C.-T. Lu, Z. Li, F. Peng, A. Timofeev, Y.-T. Chen, Y. Gao, T. Duerig, A. Tomkins, and S. Ravi, “Graph-rise: Graph-regularized image semantic embedding,” 2019.
  * [14] C. Y. Li, X. Liang, Z. Hu, and E. P. Xing, “Hybrid retrieval-generation reinforced agent for medical image report generation,” _arXiv preprint arXiv:1805.08298_ , 2018.
  * [15] Q. Li, Z. Sun, R. He, and T. Tan, “Deep supervised discrete hashing,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 2479–2488.
  * [16] Z. Li, C. Wang, M. Han, Y. Xue, W. Wei, L.-J. Li, and L. Fei-Fei, “Thoracic disease identification and localization with limited supervision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8290–8299.
  * [17] J. Lian, J. Liu, S. Zhang, K. Gao, X. Liu, D. Zhang, and Y. Yu, “A structure-aware relation network for thoracic diseases detection and segmentation,” _arXiv preprint arXiv:2104.10326_ , 2021.
  * [18] J. Liu, G. Zhao, Y. Fei, M. Zhang, Y. Wang, and Y. Yu, “Align, attend and locate: Chest x-ray diagnosis via contrast induced attention network with limited supervision,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 10 631–10 640.
  * [19] G. Nousias, E.-K. Panagiotopoulou, K. Delibasis, A.-M. Chaliasou, A.-M. Tzounakou, and G. Labiris, “Video-based eye blink identification and classification,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 26, no. 7, pp. 3284–3293, 2022.
  * [20] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [21] P. Samanta and S. Jain, “Analysis of perceptual hashing algorithms in image manipulation detection,” _Procedia Computer Science_ , vol. 185, pp. 203–212, 2021.
  * [22] S. Sedai, D. Mahapatra, Z. Ge, R. Chakravorty, and R. Garnavi, “Deep multiscale convolutional feature learning for weakly supervised localization of chest pathologies in x-ray images,” in _International Workshop on Machine Learning in Medical Imaging_. Springer, 2018, pp. 267–275.
  * [23] R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, and D. Batra, “Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization,” _arXiv:1611.01646_ , 2016.
  * [24] L. K. Tam, X. Wang, E. Turkbey, K. Lu, Y. Wen, and D. Xu, “Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 45–55.
  * [25] Y. Tang, “Large scale semi-supervised object detection using visual and semantic knowledge transfer,” _In Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [26] Y. Tang, X. Wang, A. P. Harrison, L. Lu, J. Xiao, and R. M. Summers, “Attention-guided curriculum learning for weakly supervised classification and localization of thoracic diseases on chest radiographs,” in _International Workshop on Machine Learning in Medical Imaging_. Springer, 2018, pp. 249–258.
  * [27] R. Venkatesan, S. M. Koon, M. H. Jakubowski, and P. Moulin, “Robust image hashing,” in _International Conference on Image Processing_ , 2000.
  * [28] O. Viniavskyi, M. Dobko, and O. Dobosevych, “Weakly-supervised segmentation for disease localization in chest x-ray images,” in _International Conference on Artificial Intelligence in Medicine_. Springer, 2020, pp. 249–259.
  * [29] D. Wang, S. Chen, X. Sun, Q. Chen, Y. Cao, B. Liu, and X. Liu, “Afp-mask: Anchor-free polyp instance segmentation in colonoscopy,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 26, no. 7, pp. 2995–3006, 2022.
  * [30] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2097–2106.
  * [31] X. Wang, Y. Peng, L. Lu, Z. Lu, and R. M. Summers, “Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9049–9058.
  * [32] Wikipedia contributors, “Cosine similarity — Wikipedia, the free encyclopedia,” https://en.wikipedia.org/w/index.php?title=Cosine_similarity&oldid=1050346098, 2021, [Online; accessed 19-October-2021].
  * [33] C. Yan, J. Yao, R. Li, Z. Xus, and J. Huang, “Weakly supervised deep learning for thoracic disease classification and localization on chest x-rays,” in _Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics_. ACM, 2018, pp. 103–110.
  * [34] G. Zhao, “Cross chest graph for disease diagnosis with structural relational reasoning,” in _MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021_. ACM, 2021, pp. 612–620. [Online]. Available: https://doi.org/10.1145/3474085.3475221
  * [35] G. Zhao, C. Fang, G. Li, Z. Zhou, J. Li, L. Jiao, and Y. Yu, “Contralaterally enhanced networks for thoracic disease detection,” _IEEE Transactions on Medical Imaging_ , 2021.
  * [36] G. Zhao, Q. Feng, C. Chen, Z. Zhou, and Y. Yu, “Diagnose like a radiologist: Hybrid neuro-probabilistic reasoning for attribute-based medical image diagnosis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [37] Y. Zhou, T. Zhou, T. Zhou, H. Fu, J. Liu, and L. Shao, “Contrast-attentive thoracic disease recognition with dual-weighting graph reasoning,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 4, pp. 1196–1206, 2021\.
  * [38] Z.-H. Zhou, “A brief introduction to weakly supervised learning,” _National Science Review_ , vol. 5, no. 1, pp. 44–53, 08 2017. [Online]. Available: https://doi.org/10.1093/nsr/nwx106
  * [39] W. Zhu, Y. S. Vang, Y. Huang, and X. Xie, “Deepem: Deep 3d convnets with em for weakly supervised pulmonary nodule detection,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 812–820.


# Attentive fine-tuning of Transformers for Translation of low-resourced
languages @LoResMT 2021

Karthik Puranik1 Adeep Hande1 Ruba Priyadharshini2 Thenmozhi Durairaj3
Anbukkarasi Sampath4 Kingston Pal Thamburaj5 Bharathi Raja Chakravarthi6
1Department of Computer Science, Indian Institute of Information Technology
Tiruchirappalli
2ULTRA Arts and Science College, Madurai, Tamil Nadu, India
3Sri Sivasubramaniya Nadar College of Engineering, Tamil Nadu, India
4Kongu Engineering College, Erode, Tamil Nadu, India
5Sultan Idris Education University, Tanjong Malim, Perak, Malaysia
6Insight SFI Research Centre for Data Analytics, National University of
Ireland Galway
{{karthikp<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper reports the Machine Translation (MT) systems submitted by the IIITT
team for the English$\rightarrow$Marathi and English$\Leftrightarrow$Irish
language pairs LoResMT 2021 shared task. The task focuses on getting
exceptional translations for rather low-resourced languages like Irish and
Marathi. We fine-tune IndicTrans, a pretrained multilingual NMT model for
English$\rightarrow$Marathi, using external parallel corpus as input for
additional training. We have used a pretrained Helsinki-NLP Opus MT
English$\Leftrightarrow$Irish model for the latter language pair. Our
approaches yield relatively promising results on the BLEU metrics. Under the
team name IIITT, our systems ranked 1, 1, and 2 in
English$\rightarrow$Marathi, Irish$\rightarrow$English, and
English$\rightarrow$Irish respectively. The codes for our systems are
published111https://github.com/karthikpuranik11/LoResMT.

##  1 Introduction

Today, a large number of text and written materials are present in English.
However, with roughly around 6,500 languages in the
world222https://blog.busuu.com/most-spoken-languages-in-the-world/
(Chakravarthi,, 2020; Hande et al., 2021a, ; Sarveswaran et al.,, 2021), every
native monoglot should not be deprived of this knowledge and information. The
manual translation is a tedious job involving much time and human resources,
giving rise to Machine Translation (MT). Machine Translation involves the
automated translation of text from one language to another by using various
algorithms and resources to produce quality translation predictions (Pathak
and Pakray,, 2018; Krishnamurthy,, 2015, 2019). Neural Machine Translation
(NMT) brought about a great improvement in the field of MT by overcoming flaws
of rule-based and statistical machine translation (SMT) (Revanuru et al.,,
2017; Achchuthan and Sarveswaran,, 2015; Parameswari et al.,, 2012; Thenmozhi
et al.,, 2018; Kumar et al., 2020b, ). NMT incorporates the training of neural
networks on parallel corpora to predict the likeliness of a sequence of words.
sequence-to-sequence neural models (seq2seq) (Sutskever et al.,, 2014;
Kalchbrenner and Blunsom,, 2013) are the widely adopted as the standard
approach by both industrial and research communities (Jadhav, 2020a, ; Bojar
et al.,, 2016; Cheng et al.,, 2016).

Even though NMT performs exceptionally well for all the languages, it requires
a tremendous amount of parallel corpus to produce meaningful and successful
translations (Kumar et al., 2020a, ). With little research on low resourced
languages, finding a quality parallel corpus to train the models can be
arduous. The two low-resourced languages worked on in this paper are Marathi
(mr) and Irish (ga). With about 120 million Marathi speakers in Maharashtra
and other states of India, Marathi is recognized as one of the 22 scheduled
languages of India333https://en.wikipedia.org/wiki/Marathi_language. The
structural dissimilarity which occurs while translating from English (Subject-
Verb-Object) to Marathi (Subject-Object-Verb) or vice versa adds up to issues
faced while translation (Garje,, 2014). The Irish language was recognized as
the first official language of Ireland and also by the EU(Dowling et al.,
2020a, ; Scannell,, 2007). Belonging to the Goidelic language family and the
Celtic family(Scannell,, 2007; Lynn et al.,, 2015), Irish is also claimed as
one of the low resourced languages due to its limited resources by the META-
NET report (Dhonnchadha et al.,, 2012; Scannell,, 2006).

Our paper represents the work conducted for the LoResMT @ MT Summit
2021444https://sites.google.com/view/loresmt/ shared task to build MT systems
for the low-resourced Marathi and Irish languages on COVID-19 related parallel
corpus. We implement Transformer-based (Vaswani et al.,, 2017) NMT models to
procure BLEU scores(Papineni et al.,, 2002) of 24.2, 25.8, and 34.6 in
English$\rightarrow$Marathi, Irish$\rightarrow$English, and
English$\rightarrow$Irish respectively.

##  2 Related works

Neural Machine Translation has been exhaustively studied over the years
(Kalchbrenner and Blunsom,, 2013), with several intuitive approaches involving
collective learning to align and translate (Bahdanau et al.,, 2016), and a
language-independent attention bridge for multilingual translational systems
(Vázquez et al.,, 2019). There have been several approaches to NMT, with zero-
shot translational systems, between language pairs that have not seen the
parallel training data during training (Johnson et al.,, 2017). The
introduction of artificial tokens has reduced the architectural changes in the
decoder (Ha et al.,, 2016). There have been some explorations towards neural
machine translation in low resource languages, with the development of a
multi-source translational system that targets the English string for any
source language (Zoph and Knight,, 2016).

There has been subsequent research undertaken by researchers for machine
translation in low-resource Indian languages. Chakravarthi et al., surveyed
orthographic information in machine translation, examining the orthography’s
influence on machine translation and extended it to under-resourced Dravidian
languages (Chakravarthi et al., 2019a, ). Another approach of leveraging the
information contained in rule-based machine translation systems to improve
machine translation of low-resourced languages was employed (Torregrosa et
al.,, 2019). Several approaches involving the improvement of WordNet for low-
resourced languages have been explored (Chakravarthi et al.,, 2018;
Chakravarthi et al., 2019b, ). Chakravarthi et al., 2019c constructed MMDravi,
a multilingual multimodal machine translation dataset for low-resourced
Dravidian languages, extending it from the Flickr30K dataset, and generating
translations for the captions using phonetic transcriptions (U Hegde et al.,,
2021).

There have been relatively fewer approaches experimented with and benchmarked
when it comes to translating from Marathi to English and vice versa. (Aharoni
et al.,, 2019) tried to build multilingual NMT systems, comprising 103
distinct languages and 204 translational directions simultaneously. (Jadhav,
2020b, ; Puranik et al.,, 2021) developed a machine translation system for
Marathi to English using transformers on a parallel corpus. Other works on
improving machine translation include (Adi Narayana Reddy et al.,, 2021)
proposing an English-Marathi NMT using local attention. The same can be stated
to Irish, as it is a poorly resourced language, as the quality of the MT
outputs have struggled to achieve the same level as well-supported languages
(Dowling et al.,, 2016; Rehm and Uszkoreit,, 2012). In recent years, several
researchers tried to overcome the resource barrier by creating artificial
parallel data through back-translation (Poncelas et al.,, 2018), exploiting
out-of-domain data (Imankulova et al.,, 2019), and leveraging other better-
resourced languages as a pivot (Dowling et al., 2020b, ; Wu and Wang,, 2007).

##  3 Dataset

We use the dataset provided by the organizers of LoResMT @ MT Summit 2021. The
datasets can be found here555https://github.com/loresmt/loresmt-2021. It is a
parallel corpus for English and the low resourced language, i.e., Irish and
Marathi, mostly containing text related to COVID-19 (Ojha et al.,, 2021).

Language pair | English$\Leftrightarrow$Irish | English$\Leftrightarrow$Marathi
---|---|---
Train | 8,112 | 20,933
Dev | 502 | 500
Test | 1,000 | 1,000
Total | 9,614 | 22,433
Table 1:  Number of sentences distribution

We have used bible-uedin666https://opus.nlpl.eu/JRC-Acquis.php
(Christodoulopoulos and Steedman,, 2015) an external dataset for Marathi. It
is a multilingual parallel corpus dataset containing translations of the Bible
in 102 languages(Christodoulopoulos and Steedman,, 2014) and shows the
possibility of using the Bible for research and machine translation. English-
Marathi corpus contains 60,495 sentences. CVIT
PIB777http://preon.iiit.ac.in/~jerin/bhasha/ (Philip et al.,, 2020) has also
been used for the purpose of this research. It contains 1,14,220 parallel
corpora for English-Marathi.

##  4 Methodology

Figure 1: Our approach for the English$\rightarrow$Marathi language pair.

###  4.1 IndicTrans

Fairseq PyTorch888https://github.com/pytorch/fairseq (Ott et al.,, 2019) is an
open-source machine learning library supported as a sequence modeling toolkit.
Custom models can be trained for various tasks, including summarization,
language, translation, and other generation tasks. Training on fairseq enables
competent batching, mixed-precision training, multi-GPU and multi-machine
training. IndicTrans (Ramesh et al.,, 2021), a Transformer-4x multilingual NMT
model by AI4Bharat, is trained on the Samanantar dataset. The architecture of
our approach is displayed in Fig.1.
Samanantar999https://indicnlp.ai4bharat.org/samanantar/ is the most extensive
collection of parallel corpora for Indic languages available for public use.
It includes 46.9 million sentence pairs between English and 11 Indian
languages. IndicTrans is claimed to successfully outperform the existing best
performing models on a wide variety of benchmarks. Even commercial translation
systems and existing publicly available systems were surpassed for the
majority of the languages. IndicTrans is based on fairseq, and it was fine-
tuned on the Marathi training dataset provided by the organizers and the
external datasets. The model was fine-tuned with the cross-entropy criterion
to compute the loss function, Adam optimizer (Zhang,, 2018), dropout of 0.2,
fp16 (Micikevicius et al.,, 2017), maximum tokens of 256 for better learning,
and a learning rate if 3e-5 in GPU. The process was conducted for a maximum of
3 epochs.

###  4.2 Helsinki-NLP Opus-MT

OPUS-MT (Tiedemann and Thottingal,, 2020) supports both bilingual and
multilingual models. It is a project that focuses on the development of free
resources and tools for machine translation. The current status is a
repository of over 1,000 pretrained neural MT models. We fine-tune a
transformer-align model that was fine-tuned for the Tatoeba-
Challenge101010https://github.com/Helsinki-NLP/Tatoeba-Challenge. Helsinki-
NLP/opus-mt-en-ga model from the HuggingFace Transformers (Wolf et al.,, 2020)
for English$\rightarrow$ Irish and Helsinki-NLP/opus-mt-ga-en for
Irish$\rightarrow$ English were used.

Language pair | Method | BLEU
---|---|---
English$\rightarrow$Marathi | IndicTrans baseline | 14.0
English$\rightarrow$Marathi | IndicTrans TRA | 17.8
English$\rightarrow$Marathi | IndicTrans CVIT-PIB | 23.4
English$\rightarrow$Marathi | IndicTrans bible-uedin | 27.7
English$\rightarrow$Irish | Opus MT | 30.4
English$\rightarrow$Irish | M2M100 | 25.6
Irish$\rightarrow$English | Opus MT | 37.2
Irish$\rightarrow$English | M2M100 | 30.4
Table 2: BLEU scores obtained for the various models for the development set
Figure 2: The graph depicting the increase in val lossafter the third epoch

##  5 Results and Analysis

For Marathi, it is distinctly visible that our system model, i.e., IndicTrans
fine-tuned on the training data provided by the organizers or TRA and the
bible-uedin dataset, gave the best BLEU scores. It was surprising how the
model fine-tuned on a parallel corpus of 60,495 sentences of bible-uedin
surpassed the model fine-tuned on 1,14,220 sentences from the CVIT PIB
dataset. The possible explanation is the higher correlation between the
sentences of the bible-uedin dataset with the test dataset than the CVIT PIB
dataset. Another reason could be the presence of excessive noise in the CVIT
PIB dataset. The other reason for this could be noise and a lower quality of
translations in the CVIT PIB dataset compared to bible-uedin.

To infer the same, 1000 random pairs of sentences were picked from the
datasets, and the average LaBSE or language-agnostic BERT Sentence Embedding
(Feng et al.,, 2020) scores were found out. LaBSE gives a score between 0 and
1, depending on the quality of the translation. It was seen that the average
was 0.768 for the bible-uedin dataset, while it was 0.58 for the CVIT PIB
dataset. This might have been one of the reasons for the better BLEU scores.
The model also showed constant overfitting after the second and third epoch,
as the BLEU scores reduced considerably as they reached the 6th epoch. The
BLEU scores decreased by a difference of 6.The validation loss starts
increasing after the third epoch, thus, showing the overfitting occurring in
training. So, the model was fine-tuned for three epochs while maintaining a
low learning rate of around 3e-5 to get a BLEU score of 24.2.

Language pair | BLEU | CHRF | TER | Rank
---|---|---|---|---
English$\rightarrow$Marathi | 24.2 | 0.59 | 0.597 | 1
Irish$\rightarrow$English | 34.6 | 0.61 | 0.711 | 1
English$\rightarrow$Irish | 25.8 | 0.53 | 0.629 | 2
Table 3: Result and ranks obtained for the test dataset(Popović,, 2015; Snover
et al.,, 2006)

Training a model to predict for a low-resourced language was highly
challenging due to the absence of prominent pretrained models (Kalyan et al.,,
2021; Yasaswini et al.,, 2021; Hande et al., 2021b, ). However, as an
experiment, two models from HuggingFace
Transformers111111https://huggingface.co/transformers/, M2M100 (Fan et al.,,
2020) and Opus-MT from Helsinki NLP (Tiedemann,, 2020) were compared. For the
dev data, Opus MT produced a BLEU score of 30.4 while M2M100 gave 25.62 for
translations from English to Irish and 37.2 and 30.37 respectively for Irish
to English translations. Probably, the individual models pretrained on
numerous datasets gave Opus MT an edge over M2M100. This led us to submit the
Opus MT model for the LoResMT Shared task 2021. The model gave exceptional
BLEU scores of 25.8 for English to Irish, which ranked second in the shared
task, while 34.6 for Irish to English stood first.

##  6 Conclusion

It is arduous and unyielding to get accurate translations for low-resourced
languages due to limited datasets and pretrained models. However, our paper
puts forward a few methods to better the already existing accuracies. Ranked
1, 1, and 2 in English$\rightarrow$Marathi, Irish$\rightarrow$English, and
English$\rightarrow$Irish respectively in the LoResMT 2021 shared task,
IndicTrans fine-tuned on the bible-uedin, and the dataset provided by the
organizers manages to surpass the other models due to its high correlation
with the test set and minimal noise for the Marathi language. The Irish
language task was dominated by the Opus MT model by Helsinki-NLP,
outperforming other Transformer models, M2M100.

## References

  * Achchuthan and Sarveswaran, (2015) Achchuthan, Y. and Sarveswaran, K. (2015).  Language localisation of Tamil using statistical machine translation.  In 2015 Fifteenth International Conference on Advances in ICT for Emerging Regions (ICTer), pages 125–129. IEEE.
  * Adi Narayana Reddy et al., (2021) Adi Narayana Reddy, K., Shyam Chandra Prasad, G., Rajashekar Reddy, A., Naveen Kumar, L., and Kannaiah (2021).  English-marathi neural machine translation using local attention.  In Garg, D., Wong, K., Sarangapani, J., and Gupta, S. K., editors, Advanced Computing, pages 280–287, Singapore. Springer Singapore.
  * Aharoni et al., (2019) Aharoni, R., Johnson, M., and Firat, O. (2019).  Massively multilingual neural machine translation.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Bahdanau et al., (2016) Bahdanau, D., Cho, K., and Bengio, Y. (2016).  Neural machine translation by jointly learning to align and translate.
  * Bojar et al., (2016) Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., Negri, M., Névéol, A., Neves, M., Popel, M., Post, M., Rubino, R., Scarton, C., Specia, L., Turchi, M., Verspoor, K., and Zampieri, M. (2016).  Findings of the 2016 conference on machine translation.  In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics.
  * Chakravarthi, (2020) Chakravarthi, B. R. (2020).  Leveraging orthographic information to improve machine translation of under-resourced languages.  PhD thesis, NUI Galway.
  * Chakravarthi et al., (2018) Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2018).  Improving wordnets for under-resourced languages using machine translation.  In Proceedings of the 9th Global Wordnet Conference, pages 77–86.
  * (8) Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2019a).  Comparison of different orthographies for machine translation of under-resourced dravidian languages.  In 2nd Conference on Language, Data and Knowledge (LDK 2019). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
  * (9) Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2019b).  Wordnet gloss translation for under-resourced languages using multilingual neural machine translation.  In Proceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation, pages 1–7.
  * (10) Chakravarthi, B. R., Priyadharshini, R., Stearns, B., Jayapal, A. K., Sridevy, S., Arcan, M., Zarrouk, M., and McCrae, J. P. (2019c).  Multilingual multimodal machine translation for dravidian languages utilizing phonetic transcription.  In Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages, pages 56–63.
  * Chakravarthi et al., (2021) Chakravarthi, B. R., Rani, P., Arcan, M., and McCrae, J. P. (2021).  A survey of orthographic information in machine translation.  SN Computer Science, 2(4):1–19.
  * Cheng et al., (2016) Cheng, Y., Xu, W., He, Z., He, W., Wu, H., Sun, M., and Liu, Y. (2016).  Semi-supervised learning for neural machine translation.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1965–1974, Berlin, Germany. Association for Computational Linguistics.
  * Christodoulopoulos and Steedman, (2014) Christodoulopoulos, C. and Steedman, M. (2014).  A massively parallel corpus: the bible in 100 languages.  Language Resources and Evaluation, 49:1–21.
  * Christodoulopoulos and Steedman, (2015) Christodoulopoulos, C. and Steedman, M. (2015).  A massively parallel corpus: the bible in 100 languages.  Language Resources and Evaluation, 49:375 – 395.
  * Dhonnchadha et al., (2012) Dhonnchadha, E., Judge, J., Chasaide, A., Dhubhda, R., and Scannell, K. (2012).  The irish language in the digital age: An ghaeilge sa ré dhigiteach.
  * (16) Dowling, M., Castilho, S., Moorkens, J., Lynn, T., and Way, A. (2020a).  A human evaluation of english-irish statistical and neural machine translation.
  * (17) Dowling, M., Castilho, S., Moorkens, J., Lynn, T., and Way, A. (2020b).  A human evaluation of English-Irish statistical and neural machine translation.  In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 431–440, Lisboa, Portugal. European Association for Machine Translation.
  * Dowling et al., (2016) Dowling, M., Judge, J., Lynn, T., and Graham, Y. (2016).  English to irish machine translation with automatic post-editing.
  * Fan et al., (2020) Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Celebi, O., Wenzek, G., Chaudhary, V., Goyal, N., Birch, T., Liptchinsky, V., Edunov, S., Grave, E., Auli, M., and Joulin, A. (2020).  Beyond english-centric multilingual machine translation.
  * Feng et al., (2020) Feng, F., Yang, Y., Cer, D., Arivazhagan, N., and Wang, W. (2020).  Language-agnostic bert sentence embedding.
  * Garje, (2014) Garje, G. (2014).  Marathi to english machine translation for simple sentences.  International Journal of Science and Research (IJSR), 3.
  * Ha et al., (2016) Ha, T.-L., Niehues, J., and Waibel, A. (2016).  Toward multilingual neural machine translation with universal encoder and decoder.  ArXiv, abs/1611.04798.
  * (23) Hande, A., Puranik, K., Priyadharshini, R., and Chakravarthi, B. R. (2021a).  Domain identification of scientific articles using transfer learning and ensembles.  In Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2021 Workshops, WSPA, MLMEIN, SDPRA, DARAI, and AI4EPT, Delhi, India, May 11, 2021 Proceedings 25, pages 88–97. Springer International Publishing.
  * (24) Hande, A., Puranik, K., Priyadharshini, R., Thavareesan, S., and Chakravarthi, B. R. (2021b).  Evaluating pretrained transformer-based models for covid-19 fake news detection.  In 2021 5th International Conference on Computing Methodologies and Communication (ICCMC), pages 766–772.
  * Imankulova et al., (2019) Imankulova, A., Dabre, R., Fujita, A., and Imamura, K. (2019).  Exploiting out-of-domain parallel data through multilingual transfer learning for low-resource neural machine translation.  In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 128–139, Dublin, Ireland. European Association for Machine Translation.
  * (26) Jadhav, S. (2020a).  Marathi to english neural machine translation with near perfect corpus and transformers.  ArXiv, abs/2002.11643.
  * (27) Jadhav, S. A. (2020b).  Marathi to english neural machine translation with near perfect corpus and transformers.
  * Johnson et al., (2017) Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., Hughes, M., and Dean, J. (2017).  Google’s multilingual neural machine translation system: Enabling zero-shot translation.
  * Kalchbrenner and Blunsom, (2013) Kalchbrenner, N. and Blunsom, P. (2013).  Recurrent continuous translation models.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709, Seattle, Washington, USA. Association for Computational Linguistics.
  * Kalyan et al., (2021) Kalyan, P., Reddy, D., Hande, A., Priyadharshini, R., Sakuntharaj, R., and Chakravarthi, B. R. (2021).  IIITT at CASE 2021 task 1: Leveraging pretrained language models for multilingual protest detection.  In Challenges and Applications of Automated Extraction of Socio-political Events from Text, pages 98–104, Online. Association for Computational Linguistics.
  * Krishnamurthy, (2015) Krishnamurthy, P. (2015).  Development of Telugu-Tamil transfer-based machine translation system: With special reference to divergence index.  In Proceedings of the 1st Deep Machine Translation Workshop, pages 48–54, Praha, Czechia. ÚFAL MFF UK.
  * Krishnamurthy, (2019) Krishnamurthy, P. (2019).  Development of Telugu-Tamil transfer-based machine translation system: An improvization using divergence index.  Journal of Intelligent Systems, 28(3):493–504.
  * (33) Kumar, A., Mundotiya, R. K., and Singh, A. K. (2020a).  Unsupervised approach for zero-shot experiments: Bhojpuri–Hindi and Magahi–Hindi@LoResMT 2020.  In Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages, pages 43–46, Suzhou, China. Association for Computational Linguistics.
  * (34) Kumar, B. S., Thenmozhi, D., and Kayalvizhi, S. (2020b).  Tamil paraphrase detection using encoder-decoder neural networks.  In International Conference on Computational Intelligence in Data Science, pages 30–42. Springer.
  * Lynn et al., (2015) Lynn, T., Scannell, K., and Maguire, E. (2015).  Minority language twitter: Part-of-speech tagging and analysis of irish tweets.
  * Micikevicius et al., (2017) Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017).  Mixed precision training.  arXiv preprint arXiv:1710.03740.
  * Ojha et al., (2021) Ojha, A. K., Liu, C.-H., Kann, K., Ortega, J., Satam, S., and Fransen, T. (2021).  Findings of the LoResMT 2021 Shared Task on COVID and Sign Language for Low-Resource Languages.  In Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages.
  * Ott et al., (2019) Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019).  fairseq: A fast, extensible toolkit for sequence modeling.  In Proceedings of NAACL-HLT 2019: Demonstrations.
  * Papineni et al., (2002) Papineni, K., Roukos, S., Ward, T., and Zhu, W. J. (2002).  Bleu: a method for automatic evaluation of machine translation.
  * Parameswari et al., (2012) Parameswari, K., Sreenivasulu, N., Uma Maheshwar Rao, G., and Christopher, M. (2012).  Development of Telugu-Tamil bidirectional machine translation system: A special focus on case divergence.  In proceedings of 11th International Tamil Internet conference, pages 180–191.
  * Pathak and Pakray, (2018) Pathak, A. and Pakray, D. P. (2018).  Neural machine translation for indian languages.  Journal of Intelligent Systems, 28.
  * Philip et al., (2020) Philip, J., Siripragada, S., Namboodiri, V. P., and Jawahar, C. V. (2020).  Revisiting low resource status of indian languages in machine translation.  8th ACM IKDD CODS and 26th COMAD.
  * Poncelas et al., (2018) Poncelas, A., Shterionov, D., Way, A., Wenniger, G. M. D. B., and Passban, P. (2018).  Investigating backtranslation in neural machine translation.  ArXiv, abs/1804.06189.
  * Popović, (2015) Popović, M. (2015).  chrF: character n-gram F-score for automatic MT evaluation.  In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Puranik et al., (2021) Puranik, K., Hande, A., Priyadharshini, R., Thavareesan, S., and Chakravarthi, B. R. (2021).  IIITT@LT-EDI-EACL2021-hope speech detection: There is always hope in transformers.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 98–106, Kyiv. Association for Computational Linguistics.
  * Ramesh et al., (2021) Ramesh, G., Doddapaneni, S., Bheemaraj, A., Jobanputra, M., AK, R., Sharma, A., Sahoo, S., Diddee, H., J, M., Kakwani, D., Kumar, N., Pradeep, A., Deepak, K., Raghavan, V., Kunchukuttan, A., Kumar, P., and Khapra, M. S. (2021).  Samanantar: The largest publicly available parallel corpora collection for 11 indic languages.
  * Rehm and Uszkoreit, (2012) Rehm, G. and Uszkoreit, H. (2012).  The irish language in the digital age.
  * Revanuru et al., (2017) Revanuru, K., Turlapaty, K., and Rao, S. (2017).  Neural machine translation of indian languages.  In Proceedings of the 10th Annual ACM India Compute Conference, Compute ’17, page 11–20, New York, NY, USA. Association for Computing Machinery.
  * Sarveswaran et al., (2021) Sarveswaran, K., Dias, G., and Butt, M. (2021).  Thamizhimorph: A morphological parser for the Tamil language.  Machine Translation, 35(1):37–70.
  * Scannell, (2006) Scannell, K. P. (2006).  Machine translation for closely related language pairs.  In Proceedings of the Workshop Strategies for developing machine translation for minority languages, pages 103–109. Citeseer.
  * Scannell, (2007) Scannell, K. P. (2007).  The crúbadán project: Corpus building for under-resourced languages.  In Building and Exploring Web Corpora: Proceedings of the 3rd Web as Corpus Workshop, volume 4, pages 5–15.
  * Snover et al., (2006) Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006).  A study of translation edit rate with targeted human annotation.
  * Sutskever et al., (2014) Sutskever, I., Vinyals, O., and Le, Q. V. (2014).  Sequence to sequence learning with neural networks.
  * Thenmozhi et al., (2018) Thenmozhi, D., Kumar, B. S., and Aravindan, C. (2018).  Deep learning approach to English-Tamil and Hindi-Tamil verb phrase translations.  In FIRE (Working Notes), pages 323–331.
  * Tiedemann, (2020) Tiedemann, J. (2020).  The tatoeba translation challenge – realistic data sets for low resource and multilingual mt.
  * Tiedemann and Thottingal, (2020) Tiedemann, J. and Thottingal, S. (2020).  OPUS-MT – building open translation services for the world.  In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
  * Torregrosa et al., (2019) Torregrosa, D., Pasricha, N., Masoud, M., Chakravarthi, B. R., Alonso, J., Casas, N., and Arcan, M. (2019).  Leveraging rule-based machine translation knowledge for under-resourced neural machine translation models.  In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 125–133, Dublin, Ireland. European Association for Machine Translation.
  * U Hegde et al., (2021) U Hegde, S., Hande, A., Priyadharshini, R., Thavareesan, S., and Chakravarthi, B. R. (2021).  UVCE-IIITT@DravidianLangTech-EACL2021: Tamil troll meme classification: You need to pay more attention.  In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pages 180–186, Kyiv. Association for Computational Linguistics.
  * Vaswani et al., (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017).  Attention is all you need.
  * Vázquez et al., (2019) Vázquez, R., Raganato, A., Tiedemann, J., and Creutz, M. (2019).  Multilingual NMT with a language-independent attention bridge.  In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 33–39, Florence, Italy. Association for Computational Linguistics.
  * Wolf et al., (2020) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. (2020).  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  * Wu and Wang, (2007) Wu, H. and Wang, H. (2007).  Pivot language approach for phrase-based statistical machine translation.  In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 856–863, Prague, Czech Republic. Association for Computational Linguistics.
  * Yasaswini et al., (2021) Yasaswini, K., Puranik, K., Hande, A., Priyadharshini, R., Thavareesan, S., and Chakravarthi, B. R. (2021).  IIITT@DravidianLangTech-EACL2021: Transfer learning for offensive language detection in Dravidian languages.  In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pages 187–194, Kyiv. Association for Computational Linguistics.
  * Zhang, (2018) Zhang, Z. (2018).  Improved adam optimizer for deep neural networks.  In 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pages 1–2. IEEE.
  * Zoph and Knight, (2016) Zoph, B. and Knight, K. (2016).  Multi-source neural translation.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 30–34, San Diego, California. Association for Computational Linguistics.