# Multi-scale Adaptive Task Attention Network for Few-Shot Learning

Haoxing Chen  Huaxiong Li  Yaohui Li  Chunlin Chen
Nanjing University
$\\{$haoxingchen<EMAIL_ADDRESS>$\\{$huaxiongli,
<EMAIL_ADDRESS>
###### Abstract

The goal of few-shot learning is to classify unseen categories with few
labeled samples. Recently, the low-level information metric-learning based
methods have achieved satisfying performance, since local representations
(LRs) are more consistent between seen and unseen classes. However, most of
these methods deal with each category in the support set independently, which
is not sufficient to measure the relation between features, especially in a
certain task. Moreover, the low-level information-based metric learning method
suffers when dominant objects of different scales exist in a complex
background. To address these issues, this paper proposes a novel Multi-scale
Adaptive Task Attention Network (MATANet) for few-shot learning. Specifically,
we first use a multi-scale feature generator to generate multiple features at
different scales. Then, an adaptive task attention module is proposed to
select the most important LRs among the entire task. Afterwards, a similarity-
to-class module and a fusion layer are utilized to calculate a joint multi-
scale similarity between the query image and the support set. Extensive
experiments on popular benchmarks clearly show the effectiveness of the
proposed MATANet compared with state-of-the-art methods.

##  1 Introduction

Figure 1: The two main problems of the previous local representation based
methods. (a) In different tasks, the most discriminative features are
different. In task 1, the beak is the key distinguishing feature, while the
most critical feature is the wing in task 2. (b) The scales of the dominant
objects varies from image to image.

Humans can learn new concepts and objects with only one or a few samples
easily. In order to imitate this ability of humans, many few-shot learning
methods [30, 6, 27, 29, 28, 18, 19] have been proposed. However, most of these
methods [30, 6, 27, 29, 28] adopt image-level features for classification. Due
to the scarcity of samples in few-shot image recognition tasks, classifying at
such a level may not be effective enough. Instead, many methods [18, 19] based
on low-level information were proposed, i.e., local representations (LRs) of
feature embeddings. These methods use low-level information to measure the
distance between query images and support images, and they can achieve better
recognition results. However, these methods do not measure the similarity
between query images and support images in the context of the whole task,
which does not make full use of the representation ability of local feature
descriptors. When humans classify an image into one of several unseen classes,
it is natural to look for semantic features that are shared only between
certain classes and the query image. In other words, humans do not pay much
attention to the features shared between classes when recognizing a category
they have not seen before. For example, consider two 5-way 1-shot tasks in
Figure 1(a). In task1, we need to recognize a ‘Black-footed Albatross’ among
‘Laysan Albatross’, ‘Heermann Gull’, ‘California Gull’ and ‘Rhinoceros
Auklet’. While in task 2, we need to recognize a ‘Black-footed Albatross’
among ‘European Goldfinch’, ‘Blue Grosbeak’, ‘Pine Grosbeak’ and ‘Yellow
Warbler’. For task 1, the beak is a very distinguishing feature, but for task
2 it is not the most critical feature. Similarly, the wing is more important
for task 2 than task 1. In summary, the importance of each LRs varies from
task to task.

As previously mentioned, although these existing methods can extract the
relation between the query image and each support set independently, they do
not consider the importance of each LRs under the whole task, and all the LRs
are weighted equally, rather than the task-relevant LRs enjoy the higher
weights. Moreover, these methods can only calculate their similarity at a
single scale. As shown in Figure 1(b), due to the scales of dominant objects
in different images are dissimilar, we argue that it is more reasonable to
calculate the similarity between query image and support set at multiple
scales simultaneously.

To this end, we propose a novel _Multi-scale Adaptive Task Attention Network_
for metric-learning based few-shot learning, which can be trained in an end-
to-end manner. First, we represent all images as a collection of LRs at
different scales by a multi-scale feature generator, rather than a global
feature representation at the image level. Second, we measure the semantic
similarity between the query image and the support set by calculating the
semantic relation matrix. Afterwards, we employ an adaptive task attention
module to select the most distinguishing feature in the current task. Third,
to further make full use of LRs, we employ a similarity-to-class mechanism to
determine which support class the query image belongs to at each scale.
Finally, we adaptively fuse the similarities calculated from the features of
different scales together.

To sum up, the main contributions are summarized as follows:

  * •

To generate different scales features, we propose a multi-scale feature
generator in few-shot learning tasks, which can provide multi-scale
information for more comprehensive measurements.

  * •

We further propose a novel adaptive task attention mechanism by finding and
weighing most discriminative local representations in the entire task, aiming
to learn task-relevant feature representations for few-shot learning.

  * •

We conduct sufficient experiments on four benchmark datasets to verify the
advancement of our model, and the performance of our model achieves the state-
of-the-art.

##  2 Related Work

In the recent few-shot learning literature, there are roughly two types of
methods: meta-learning based method and metric-learning based method.

Meta-learning based methods. The main idea of meta-learning based methods [23,
6, 28, 13] is how to use existing experience or knowledge reasonably to
achieve fast learning when faced with a new task, rather than starting from
scratch. Santoro et al. [25] adopted an LSTM to control the interaction
between the network and the external memory module. Santoro et al. [23]
describe a new meta-learning method by interpreting SGD update rules as a
recursive gated model with trainable parameters. The purpose of MAML [6] is to
learn a good parameter initialization so that the model can quickly adapt to
new tasks. Sun et al. [28] proposed meta-transfer learning, which learns the
scaling and shifting functions of DNN weight for each task. Jamel et al. [13]
proposed a task unbiased method, by introducing regularized loss terms to
constrain the model to have as little preference for all tasks as possible
when parameters are updated. Although these meta-learning based methods have
achieved outstanding results on few-shot learning tasks, their complex memory
addressing architecture is difficult to train. Compared with these methods,
our proposed MATANet can be trained in an end-to-end manner.

Figure 2: The framework of MATANet under the 5-way 1-shot image classification
setting. The model mainly consists of four modules: the feature extractor
$\mathcal{F}_{\theta}$ to learn local representations, the multi-scale feature
generator $g_{\phi}$ to generate multiple features at different scales, the
adaptive task attention module $\mathcal{F}_{\varphi}$ to generate adaptive
task attention mask for selecting more important elements of semantic relation
matrix, and the similarity-to-class module $\mathcal{F}_{\omega}$ to get a
similarity score to determine which support class the query image belongs to.
The black square indicates the predicted label. (Best view in color.)

Metric-learning based methods.  The metric-learning based methods aim to learn
an informative distance metric, as demonstrated by [30, 27, 29, 1, 8, 19, 18,
26]. Koch et al. [16] used a Siamese Neural Network to tackle the one-shot
learning problem. Snell et al. [27] proposed Prototypical Networks, which
first assumes that each type can be represented by a prototype, and the
prototype can be obtained by calculating the mean value of the embedding
representation of each class and then using a distance function to classify.
In fact, we don’t know which distance function is the best. Therefore, Sung et
al. [29] proposed a Relation Network to obtain the most suitable distance
metric function through learning. The above methods are all based on the
feature representation at the image level. Due to the scarcity of the number
of samples, we can not well represent the distribution of each category on the
image level features. In contrast, some recent work, such as DN4 [18] and
CovaMNet [19] shows that the rich low-level features (i.e., LRs) have better
representation capabilities. However, these methods measure the similarity
between the query image and each support class independently, without
considering the entire task together. Moreover, the methods based on low-level
features only measure the similarity between query image and support set at a
single scale, which may lead to a lower classification accuracy when the
scales of dominant objects are different.

Unlike the above method, our MATANet calculates the similarity between the
query image and the support set at multi-scale. And then we can obtain the
final result through integrate multiple similarities from different scales. In
addition, our MATANet can adaptively select task-relevant local features with
discriminative semantics, as the process of human recognition.

##  3 The proposed Method

###  3.1 Description on Few-shot Learning

In few-shot learning, there are usually three sets of data: a query set
$\mathcal{Q}$, a support set $\mathcal{S}$, and an auxiliary set
$\mathcal{A}$. Note that $\mathcal{Q}$ and $\mathcal{S}$ share the same label
space, while they have no intersection with the label space of $\mathcal{A}$.

In this paper, we follow the definition of the common few-shot learning task.
Given a support set that contains N previously unseen classes, with K samples
for each class. We need to determine which class the query sample belongs to,
which is called N-way K-shot tasks (e.g., 5-way 1-shot or 5-way 5-shot). To
achieve this goal, we use an auxiliary set to train a model to learn
transferable knowledge. The model is trained by the episodic training
mechanism [30]. In each episode, a new task is constructed randomly in
$\mathcal{A}$, and each task consists of two subsets: auxiliary support set
$\mathcal{A_{S}}$ and auxiliary query set $\mathcal{A_{Q}}$. Generally, in the
training stage, hundreds of tasks are adopted to train the model.

As shown in Figure 2, our MATANet is mainly composed of four modules: a
feature extractor $\mathcal{F}_{\theta}$, a multi-scale feature generator
$g_{\phi}$, an adaptive task attention module $\mathcal{F}_{\varphi}$, and a
similarity-to-class module $\mathcal{F}_{\omega}$. All image samples are first
fed into the $\mathcal{F}_{\theta}$ to get feature embeddings and rich LRs. In
practice the feature extractor module could be 4-layer CNN, ResNet12 [11] or
WRN [32]. Then the multi-scale feature generator generates multiple features
at different scales. Afterwards, semantic relation matrixes are calculated to
measure the semantic relevance between query image and support set at each
scale. The adaptive task attention module learns a task attention mask which
can adaptively calculate the importance of each LR in the current task. And we
use task attention masks to weighting semantic relation matrix to prominently
display task-relevant elements. After that, the weighted semantic relation
matrix is processed by similarity-to-class module $\mathcal{F}_{\omega}$ to
determine which support class the query image belongs to. Finally, we
adaptively fuse the similarities calculated from the features of different
scales together by a learned vector. All the modules can be trained jointly in
an end-to-end manner.

###  3.2 Multi-scale Feature Generator

As some recent studies [18, 19] on few-shot learning have proved, LRs show
richer representation ability and can alleviate the problem of sample scarcity
in few-shot learning. Therefore we use LRs to represent the features of each
image. Given a query image $\mathcal{A}^{q}_{\mathcal{Q}}$ through the feature
extractor we can get a three-dimensional (3D) vector
$\mathcal{F}_{\theta}(\mathcal{A}^{q}_{\mathcal{Q}})\in\mathbb{R}^{C\times
H\times W}$. Under the N-way K-shot few-shot learning setting, there are K
images for each support class in a certain task. Through feature extractor we
can get a four-dimensional (4D) vector of support set $\mathcal{S}$, which can
be denoted as $\mathcal{F}_{\theta}(\mathcal{S})\in\mathbb{R}^{NK\times
C\times H\times W}$.

Figure 3: The architecture of the multi-scale feature generator $g_{\phi}$.

The multi-scale feature generator aims to generate multiple features at
different scales. As illustrated in Figure 3, the multi-scale feature
generator consists of five components: the first part has no processing unit,
the input and output are the same; the second part has a $2\times 2$ max-
pooling layer; the third part has a $3\times 3$ convolutional layer; the
fourth part has a $5\times 5$ convolutional layer; the fifth part has a
$1\times 7$ convolutional layer and a $7\times 1$ convolutional layer.

Through the multi-scale feature generator, we can get the 3D features
$\mathcal{L}^{qz}\in\mathbb{R}^{C_{z}\times H_{z}\times
W_{z}},z\in\left\\{1,2,3,4,5\right\\}$, which can be regarded as a set of
$H_{z}\times W_{z}$ $C_{z}$-dimensional LRs

$\mathcal{L}^{qz}=[x_{1},...,x_{H_{z}W_{z}}]\in\mathbb{R}^{C_{z}\times
H_{z}W_{z}}$ (1)

where $x_{i}$ is the $i$-th LRs. Through multi-scale feature generator we can
also get the LRs of support set $\mathcal{S}$

$\mathcal{L}^{\mathcal{S}z}=[x_{1},...,x_{NKH_{z}W_{z}}]\in\mathbb{R}^{C_{z}\times
NKH_{z}W_{z}}$ (2)

###  3.3 Adaptive Task Attention Module

Under the N-way K-shot few-shot learning setting, we calculate the semantic
relation matrix $\mathcal{R}^{z}$ between a query image
$\mathcal{A}^{q}_{\mathcal{Q}}$ and support set $\mathcal{S}$ to measure
semantic relevance by LRs at the $z$-th scale. Then the $\mathcal{R}^{z}$ can
be calculated as below

$\displaystyle\mathcal{R}_{i,j}^{z}=cos(\mathcal{L}^{qz}_{i},\mathcal{L}^{\mathcal{S}z}_{j})$
(3) $\displaystyle
cos(\mathcal{L}^{qz}_{i},\mathcal{L}^{\mathcal{S}z}_{j})=\frac{(\mathcal{L}^{qz}_{i})^{T}\mathcal{L}^{\mathcal{S}z}_{j}}{\left\|\mathcal{L}^{qz}_{i}\right\|\cdot\left\|\mathcal{L}^{\mathcal{S}z}_{j}\right\|}$
(4)

where
$i\in\left\\{1,...,H_{z}W_{z}\right\\},j\in\left\\{1,...,NKH_{z}W_{z}\right\\},z\in\left\\{1,2,3,4,5\right\\}$,
$\mathcal{R}^{z}_{i,j}$ is $(i,j)$ element of $\mathcal{R}^{z}$ reflecting the
distance between the $i$-th LR of the query image and the $j$-th LR of support
set at the $z$-th scale and $cos(\cdot,\cdot)$ is Cosine distence function.

Each row in $\mathcal{R}^{z}$ represents the semantic relation of each LR in
the query image to all LRs of all images in the support set, i.e., semantic
relation vector $\mathcal{R}^{z}_{i}$ represent the relation between $i$-th LR
of query image $\mathcal{A}^{q}_{\mathcal{Q}}$ to all $NKH_{z}W_{z}$ LRs of
support set at $z$-th scale. $\mathcal{R}^{z}$ can be decomposed into N
submatrices $\mathcal{R}^{zn}$, $n\in\left\\{1,...,N\right\\}$ according to
columns, representing the semantic relation between the query image and each
support class.

Then we can calculate the task attention score of each element of
$\mathcal{R}^{z}$ for the current task by

$\alpha^{z}_{i}=\frac{\sum_{j=1}^{NKH_{z}W_{z}}\mathcal{R}^{z}_{i,j}}{\sqrt{\sum_{i=1}^{H_{z}W_{z}}\sum_{j=1}^{NKH_{z}W_{z}}\mathcal{R}^{z}_{i,j}}}$
(5)

The task attention mask $\alpha^{z}$ is consist of all task attention scores
$\alpha^{z}_{i}$, $i\in\left\\{1,...,H_{z}W_{z}\right\\}$. Afterwards, we use
dot-product to weight $\mathcal{R}^{z}_{i}$ by $\alpha^{z}_{i}$

$\mathcal{M}^{z}_{i}=\alpha^{z}_{i}\cdot\mathcal{R}^{z}_{i}$ (6)

where $\mathcal{M}^{z}_{i}$ is the $i$-th row of weighted semantic relation
matrix. By this way, we can get the weighted semantic relation matrix
$\mathcal{M}^{z}$, which can be decomposed into N submatrices
$\mathcal{M}^{zn}$, $n\in\left\\{1,...,N\right\\}$ according to columns. While
the semantic relations of task-irrelevant regions are suppressed; meanwhile,
the semantic relations of task-relevant regions are enhanced.

###  3.4 Similarity-to-Class Module

Similarity-to-Class Module aims to determine which support class the query
image belongs to. In this module, for each LR of the query image, we find the
$k$ most similar LR of all support LRs for class $n$. Then, we sum
$kH_{z}W_{z}$ selected LRs as the similarity score between the query image and
the $n$-th support class at the $z$-th scale

$\mathcal{P}^{zn}=\sum_{i=1}^{kH_{z}W_{z}}Topk(\mathcal{M}^{zn}_{i})$ (7)

where $\mathcal{P}^{zn}$ means the similarity between the query image and
support class $n$ at the $z$-th scale and $Topk(\cdot)$ means selecting the
$k$ lagest elements in each row of the weighted semantic relation matrix
$\mathcal{M}^{zn}$. Typically, we set $k$ to 3 on the miniImagenet dataset.
Since the small diversity within the different classes, we set $k$ to 1 on
three fine-grained datasets for our MATANet to capture the most discriminative
features and avoid introducing noise. Under the N-way K-shot few-shot learning
setting, we can get semantic similarity vectors
$\mathcal{P}^{z}\in\mathbb{R}^{N}$.

###  3.5 Classification with an adaptive fusion strategy

Since the five relation scores have been calculated, we need to design a
fusion module to integrate them. In order to solve this problem, we adopt a
learnable five-dimensional $w=[w_{1},w_{2},w_{3},w_{4},w_{5}]$ vector to
adaptively integrate these five parts. Specifically, the final fusion
similarity between a query $\mathcal{A}^{q}_{\mathcal{Q}}$ and support set
$\mathcal{S}$ can be defined as follows

$\mathcal{P}^{q}=w_{1}\cdot\mathcal{P}^{1}\
+w_{2}\cdot\mathcal{P}^{2}+w_{3}\cdot\mathcal{P}^{3}\ \\\
+w_{4}\cdot\mathcal{P}^{4}+w_{5}\cdot\mathcal{P}^{5}$ (8)

Under the 5-way 1-shot few-shot learning setting, input a query image, we will
get five similarity vectors $\mathcal{P}^{z},z\in\left\\{1,2,3,4,5\right\\}$.
We first balance the size of these five vectors by a Batch Normalization
layer. Then, we concatenate these five vectors and use a 1D convolution layer
with the kernel size of $1\times 1$ and the dilation value of 5. Then we can
get a weighted 5-dimensional similarity vector $\mathcal{P}^{q}$, we use it
for final classification.

##  4 Experiments

In this section, we perform extensive experiments to verify the advance and
effectiveness of MATANet.

###  4.1 Datasets

_mini_ ImageNet. As a small subset of ImageNet [4], The dataset consists of
100 categories, each containing 600 images. We use common splits as in [6],
which devides the dataset into training, validation and test dataset with
64/16/20 classes respectively.

We also conduct experiments on three fine-grained image recognition datasets.

CUB Birds [31] is composed of 11, 788 images of 200 birds species. Stanford
Dogs [14] is contains 120 categories of dogs and 20, 480 images. Stanford Cars
[17] consists of 196 categories of cars with 16, 185 images.

For fair comparisons, we strictly follow the splits used in [18, 19] on
Stanford Dogs and Stanford Cars, and follow the splits used in [3] on CUB
Birds as Table 1 shows.

Dataset | _Stanford Dogs_ | _Stanford Cars_ | _CUB Birds_
---|---|---|---
$N_{all}$ | 120 | 196 | 200
$N_{train}$ | 70 | 130 | 100
$N_{val}$ | 20 | 17 | 50
$N_{test}$ | 30 | 49 | 50
Table 1: The splits of three fine-grained datasets. $N_{all}$ is the number of all classes. $N_{train}$, $N_{val}$ and $N_{test}$ indicate the number of classes in training set, validation set and test set. Model | Venue | Backbone | Type | 5-way 1-shot | 5-way 5-shot
---|---|---|---|---|---
Meta LSTM∗ [23] | ICLR’17 | Conv-32F | Meta | 43.44$\pm$0.77 | 60.60$\pm$0.71
MAML∗ [6] | ICML’17 | Conv-32F | Meta | 48.70$\pm$1.84 | 63.11$\pm$0.92
TAML-Entropy∗ [13] | CVPR’19 | Conv-32F | Meta | 49.33$\pm$1.80 | 66.05$\pm$0.85
MAML+L2F∗ [2] | CVPR’20 | Conv-32F | Meta | 52.10$\pm$0.49 | 69.38$\pm$0.46
WarpGrad∗ [7] | ICLR’20 | Conv-32F | Meta | 52.30$\pm$0.80 | 68.40$\pm$0.60
Matching Nets∗ [30] | NeurIPS’16 | Conv-64F | Metric | 43.56$\pm$0.84 | 55.31$\pm$0.73
Prototypical Nets∗ [27] | NeurIPS’17 | Conv-64F | Metric | 49.42$\pm$0.78 | 68.20$\pm$0.66
Relation Nets∗ [29] | CVPR’18 | Conv-64F | Metric | 50.44$\pm$0.82 | 65.32$\pm$0.70
GNN∗ [8] | CVPR’18 | Conv-256F | Metric | 50.33$\pm$0.36 | 66.41$\pm$0.63
IMP∗ [1] | ICML’19 | Conv-64F | Metric | 49.60$\pm$0.80 | 68.10$\pm$0.80
CovaMNet∗ [19] | AAAI’19 | Conv-64F | Metric | 51.19$\pm$0.76 | 67.65$\pm$0.63
DN4∗ [18] | CVPR’19 | Conv-64F | Metric | 51.24$\pm$0.74 | 71.02$\pm$0.64
SAML∗[10] | ICCV’19 | Conv-64F | Metric | 52.22$\pm$0.00 | 66.49$\pm$0.00
DSN∗[26] | CVPR’20 | Conv-64F | Metric | 51.78$\pm$0.96 | 68.99$\pm$0.69
MATANet | Ours | Conv-64F | Metric | 53.63$\pm$0.83 | 72.67$\pm$0.76
Table 2: Comparison with other state-of-the-art methods with $95\%$ confidence
intervals on mini-ImageNet. The third column shows which kind of embedding is
employed. The fourth column shows which type of the method belongs to, i.e,
meta-learning based, metric-learning based, and other kinds of methods. ∗
Results reported by the original work. (Top two performances are in bold)

###  4.2 Network architecture

It is a well-known fact that using deeper networks to extract features or
using pre-trained models can achieve higher accuracy. We follow the basic
feature extractor module which is adopted in previous works, to make a fair
comparison with other works. The feature extractor module
$\mathcal{F}_{\theta}$ consists of 4 convolutional blocks. Specifically, each
convolutional block consists of a convolutional layer (with $3\times 3$
convolution and 64 filters), a batch normalization layer, and a leaky ReLU
non-linearity. Besides, we add a $2\times 2$ max-pooling layer to the first
two convolution blocks. The reason for using only two max-pooling layers is we
can get more LRs to capture the semantic relation between them. For example,
in a 5-way 1-shot few-shot learning task, if we use four maxpooling layers, we
can only get 25 LRs for an $84\times 84$ input image. In contrast, if we only
use two max-pooling layers, we will get 441 LRs, which will be helpful for us
to find local semantic relations.

###  4.3 Implementation Details

Our experiments are conducted under the N-way K-shot setting on four
benchmarks. All the images in four benchmarks are resized to $84\times 84$.
During the training stage, we randomly construct 250, 000 episodes to train
our MATANet for the _mini_ ImageNet and Stanford Cars, and 300,000 for the
other datasets by episodic training mechanism. In each episode, we select 15
or 10 query images from each class for the 1-shot or 5-shot setting,
respectively, i.e., in a 5-way 1-shot task, we have 75 query images and 5
support images. We adopt the Adam algorithm [15] with a cross-entropy (CE)
loss to train the network. Also, the initial learning rate is set to 0.001 and
reduce it by half of every 50,000 episodes. During the test stage, 600
episodes are constructed from the test set, and this test process will be
repeated five times. Then the mean accuracy and $95\%$ confidence intervals
will be reported simultaneously.

###  4.4 Baselines

To evaluate the effectiveness of our MATANet on the _mini_ ImageNet dataset,
we make comparisons with state-of-the art methods. Since our method is a
metric-learning based method, we mainly compare our MATANet with methods in
this branch, including Matching Nets [30], Prototypical Nets [27], Relation
Nets [29], GNN [8], IMP [1], CovaMNet [19], DN4 [18], SAML[10] and DSN [26] .
We also pick five meta-learning models Meta LSTM [23], MAML [6], TAML-Entropy
[13], MAML+L2F [2] and WarpGrad [7] for reference.

We compare seven few-shot leraning methods on fine-grained datasets, Matching
Nets [30], Prototypical Nets [27], MAML [6], Relation Nets [29], CovaMNet
[19], DN4 [18], and $\rm{PABN_{+cpt}}$/$\rm{LRPABN_{+cpt}}$ [12].

Model | 5-Way Accuracy($\%$)
---|---
_Stanford Dogs_ | _Stanford Cars_ | _CUB Birds_
1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5shot
Matching Nets† [30] |  35.80$\pm$0.99 |  47.50$\pm$1.03 |  34.80$\pm$0.98 |  44.70$\pm$1.03 |  61.16$\pm$0.89 |  72.86$\pm$0.70
Prototypical Nets† [27] |  37.59$\pm$1.00 |  48.19$\pm$1.03 |  40.90$\pm$1.01 |  52.93$\pm$1.03 |  51.31$\pm$0.91 |  70.77$\pm$0.69
MAML‡ [6] |  44.81$\pm$0.34 |  58.68$\pm$0.31 |  47.22$\pm$0.39 |  61.21$\pm$0.28 |  55.92$\pm$0.95 |  72.09$\pm$0.76
Relation Nets‡ [29] |  43.33$\pm$0.42 |  55.23$\pm$0.41 |  47.67$\pm$0.47 |  60.59$\pm$0.40 |  62.45$\pm$0.98 |  76.11$\pm$0.69
CovaMNet∗ [19] |  49.10$\pm$0.76 |  63.04$\pm$0.65 |  56.65$\pm$0.86 |  71.33$\pm$0.62 |  60.58$\pm$0.69 |  74.24$\pm$0.68
DN4∗ [18] |  45.41$\pm$0.76 |  63.51$\pm$0.62 |  59.84$\pm$0.80 |  88.65$\pm$0.44 |  52.79$\pm$0.86 |  81.45$\pm$0.70
$\rm{PABN_{+cpt}}^{*}$ [12] |  45.65$\pm$0.71 |  61.24$\pm$0.62 |  54.44$\pm$0.71 |  67.36$\pm$0.61 |  63.56$\pm$0.79 |  75.35$\pm$0.58
$\rm{LRPABN_{+cpt}}^{*}$ [12] |  45.72$\pm$0.75 |  60.94$\pm$0.66 |  60.28$\pm$0.76 |  73.29$\pm$0.58 |  63.63$\pm$0.77 |  76.06$\pm$0.58
MATANet(Ours) |  55.63$\pm$0.88 |  70.29$\pm$0.62 |  73.15$\pm$0.88 |  91.89$\pm$0.45 |  67.33$\pm$0.84 |  83.92 $\pm$0.63
Table 3: Experimental results compared with other methods on three fine-grained datasets. For Stanford Dog and Stanford Car, ∗ results reported by the original work, † results reported by [18], ‡ results re-implemented in the same setting for a fair comparison. For CUB Birds, we adopt the results for first four methods from [3], re-implement CovaMNet [19] and DN4 [18], and adopt the results reported by the original work for $\rm{PABN_{+cpt}}$/$\rm{LRPABN_{+cpt}}$ [12]. (Top two performances are in bold) Model | Venue | Backbone | Type | 5-Way Accuracy($\%$)
---|---|---|---|---
1-shot | 5-shot
Dynamic-Net∗ [9] | CVPR’18 | ResNet12 | Meta | 55.45$\pm$0.89 | 70.13$\pm$0.68
SNAIL∗ [21] | ICLR’18 | ResNet12 | Meta | 55.71$\pm$0.99 | 68.88$\pm$0.92
TPN∗ [20] | ICLR‘19 | ResNet12 | Others | 59.46$\pm$0.00 | 75.64$\pm$0.00
MAML+L2F∗ [2] | CVPR‘20 | ResNet12 | Meta | 57.48$\pm$0.49 | 74.68$\pm$0.43
MATANet | Ours | ResNet12 | Metric | 60.13$\pm$0.81 | 75.42$\pm$0.72
Qiao∗ [22] | CVPR’18 | WRN-28-10 | Meta | 59.60$\pm$0.41 | 73.74$\pm$0.19
LEO-trainal∗ [24] | ICLR’19 | WRN-28-10 | Meta | 61.76$\pm$0.08 | 77.59$\pm$0.12
Fine-tuning∗ [5] | ICLR’20 | WRN-28-10 | Others | 57.73$\pm$0.62 | 78.17$\pm$0.49
LEO+L2F∗ [2] | CVPR‘20 | WRN-28-10 | Meta | 62.12$\pm$0.13 | 78.13$\pm$0.15
MATANet | Ours | WRN-28-10 | Metric | 62.43$\pm$0.78 | 79.02$\pm$0.72
Table 4:  Comparison with other state-of-the-art methods that use deeper
backbones with $95\%$ confidence intervals on mini-ImageNet. The third column
shows which kind of embedding is employed. The fourth column shows which type
of method belongs to. ∗ Results reported by the original work.

###  4.5 Comparisons with the SOTA Methods

Our method is compared with several state-of-the-art methods under 5-way
1-shot and 5-way 5-shot few-shot learning settings.

Results on _mini_ ImageNet. The experimental results on _mini_ ImageNet are
reported in Table 2. It can be observed that our method significantly
outperforms other methods under both 5-way 1-shot and 5-shot settings.
Especially, we are 2.5% better than the second best method [7] under the 5-way
1-shot setting, with an accuracy rate of 53.63%. Similarly, we achieve 72.67%
under the 5-way 5-shot setting, with an improvement of 2.3% from the second
best method [18]. Note that, our model gains 4.7% and 2.3% improvements over
the most relevant work [18] on 1-shot and 5-shot, respectively, which proposes
an image-to-class mechanism to find the relation at class-level. This
improvement verifies the effectiveness of our model, which can adaptively
select the most discriminative local features at multiple scales in a certain
task.

Results on fine-grained datasets. From Table 3, it can be observed that the
proposed MATANet outperforms all other state-of-the-art methods under both
5-way 1-shot and 5-way 5-shot few-shot learning settings. Especially for the
5-way 1-shot task, our method achieves 13.3%, 21.6%, and 5.8% gains over the
second best on Stanford Dogs, Stanford Cars, and CUB Birds, respectively. For
the 5-way 5-shot task, our method achieves 10.7%, 3.7%, and 3.0% gains over
the second best on three datasets. The reason why we can achieve these state-
of-the-art performances is that MATANet can adaptively select the task-
relevant LRs at multiple scales for classification.

Figure 4: Experimental results of MATANet using different superparameter $k$
on _mini_ ImageNet.  Figure 5: Visualization of the selected LRs. In a 5-way
1-shot task, for each LR of the query image, our method can find the $k$
(i.e., $k$=3) most similar LRs of all support LRs for a certain class, and
weight them according to the importance of the query LR in the current task.

###  4.6 Discussion

Influence of superparameter $\bm{k}$. The results of the empirical study about
using different $k$ in MATANet. In the similarity-to-class module, for each LR
of the query image, we need to search out the $k$ most similar LRs of all
support LRs in each class. Next, we adaptively integrate five relation scores
that be calculated at different scales for final prediction. How to choose a
suitable $k$ is particularly significant. For this purpose, we conduct a
contrast experiment on _mini_ ImageNet dataset under both 5-way 1-shot and
5-way 5-shot settings by varying the value of
$k\in\left\\{1,3,5,7,9\right\\}$. As shown in Figure 4, the value of $k$ has a
moderate influence on classification performance, so we should choose a
specific $k$ for each task.

Metric functions | 5-Way Accuracy($\%$)
---|---
1-shot | 5-shot
$e^{-d(a,b)}$ | 52.82$\pm$0.83 | 71.85$\pm$0.74
$\frac{1}{1+d(a,b)}$ | 52.11$\pm$0.85 | 71.25$\pm$0.76
Tanimito Index | 53.25$\pm$0.84 | 72.11$\pm$0.73
Cosine Similarity | 53.63$\pm$0.83 | 72.67$\pm$0.76
Table 5: Experimental results of MATANet using different metric functions on
_mini_ ImageNet, $d(a,b)$:the euclidean distance between vector $a$ and $b$.

Influence of backbone networks. Besides the simple Conv-64F, we also use other
deeper feature extractors to evaluate our model, i.e., ResNet12 and WRN-28-10.
We compared other state-of-the-art methods that using these deeper feature
extractors, including Dynamic-Net [9], SNAIL [21], TPN [20], MAML+L2F [2],
Qiao [22], LEO [24], Fine-tuning [5] and LEO+L2F [2]. When using deeper
feature extractors, the accacy of MATANet reaches 60.13% and 62.43% for the
5-way 1-shot task, 75.42% and 79.13% for the 5-shot task, by using ResNet12
and WRN-28-10, repectively. Moreover, when using same deeper feature exactor,
our MATANet outperforms all other methods under both 5-way 1-shot and 5-way
5-shot few-shot learning settings (see Table 4).

Influence of metric functions. The results on the different metric functions
using in Adaptive Task Attention Module are reported in Table 5, and the
Tanimito Index is represented as $T(a,b)=\frac{a\cdot
b}{\left\|a\right\|\cdot\left\|b\right\|-a\cdot b}$. To measure the semantic
relations between feature descriptors, a suitable metric functions is a key
factors. It can be seen in Table 5, the best metric function is Cosine
Similarity.

Ablation study. To further verify the effectiveness of the multi-scale feature
generator, adaptive task-attention module, and similarity-to-class module, we
perform an ablation study on _mini_ ImageNet. We remove $g_{\phi}$,
$\mathcal{F}_{\varphi}$ and $\mathcal{F}_{\omega}$ from the MATANet
respectively to confirm that each part of the model is indispensable. We
remove $g_{\phi}$, $\mathcal{F}_{\varphi}$ and $\mathcal{F}_{\omega}$
simultaneously as the baseline method. As seen in Table 6, the main
improvement comes from the adaptive task-attention module
$\mathcal{F}_{\varphi}$. If we remove $\mathcal{F}_{\varphi}$, the performance
will be reduced by $3.7\%$, $4.8\%$ on 1-shot, 5-shot tasks, respectively.
This empirical study proves that the discriminative $\mathcal{F}_{\varphi}$
gives a performance boost and results in more discriminative features for
classification. Similarly, if we remove $g_{\phi}$, the performance will be
reduced by $1.3\%$, $0.9\%$ on 1-shot, 5-shot tasks, respectively. Moreover,
if we remove $\mathcal{F}_{\omega}$, the performance will be reduced by
$1.1\%$, $0.8\%$ on 1-shot, 5-shot tasks, respectively.

Model | 5-Way Accuracy($\%$)
---|---
1-shot | 5-shot
baseline | 51.12$\pm$0.73 | 67.35$\pm$0.63
w/o $g_{\phi}$ | 52.95$\pm$0.77 | 72.02$\pm$0.69
w/o $\mathcal{F}_{\varphi}$ | 51.66$\pm$0.77 | 69.12$\pm$0.77
w/o $\mathcal{F}_{\omega}$ | 53.02$\pm$0.79 | 72.07$\pm$0.73
MATANet(ours) | 53.63$\pm$0.83 | 72.67$\pm$0.76
Table 6: The ablation study on _mini_ ImageNet for the proposed MATANet.

Visualization of the selected LRs. As shown in Figure 5, for the LRs in red,
yellow, green and orange boxes in the query image, we visualized the $k$
(i.e., $k$=3) most discriminative LRs selected by MATANet. In [18], they will
equally use the selected LRs for the final classification. However, in our
method, the LRs corresponding to these boxes are treated differently. In task
1, the task attention score corresponding to the red box and yellow box is
0.069 and 0.041, respectively, so the LRs corresponding to the red box will
play a more important role in the final classification. Similarly, in task 2,
the LRs corresponding to the orange box will play a more important role in the
final classification. This is because the beak is obviously more
discriminative than the wing in task 1. While in Task 2 the wing is
significantly more discriminative than the beak, which verifies that our
method can automatically select the most discriminative LRs in the current
task. Moreover, it can be seen from Figure 5 that the scales of the dominant
objects in different images are different, which may affect the performance of
the model. This once again verified the necessity of our multi-scale feature
generator.

##  5 Conclusion

In this paper, we revisit the local representation based metric-learning and
propose a novel Multi-scale Adaptive Task Attention Network (MATANet) for few-
shot learning, aiming to learn more discriminative task-relevant local
representations at different scales by generating multiple features at
different scale and looking at the context of the entire task. By taking a
view of the entire task, our method is able to adaptively select the most
discriminative local representations in the current task at different scales.
Extensive experiments on four benchmark datasets demonstrate the effectiveness
and advantages of the proposed MATANet.

## References

  * [1] Kelsey R Allen, Evan Shelhamer, Hanul Shin, and Joshua B Tenenbaum.  Infinite mixture prototypes for few-shot learning.  arXiv preprint arXiv:1902.04552, 2019.
  * [2] Sungyong Baik, Seokil Hong, and Kyoung Mu Lee.  Learning to forget for meta-learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2379–2387, 2020.
  * [3] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  arXiv preprint arXiv:1904.04232, 2019.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [5] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto.  A baseline for few-shot image classification.  arXiv preprint arXiv:1909.02729, 2019.
  * [6] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  arXiv preprint arXiv:1703.03400, 2017.
  * [7] Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell.  Meta-learning with warped gradient descent.  arXiv preprint arXiv:1909.00025, 2019.
  * [8] Victor Garcia and Joan Bruna.  Few-shot learning with graph neural networks.  arXiv preprint arXiv:1711.04043, 2017.
  * [9] Spyros Gidaris and Nikos Komodakis.  Dynamic few-shot visual learning without forgetting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4367–4375, 2018.
  * [10] Fusheng Hao, Fengxiang He, Jun Cheng, Lei Wang, Jianzhong Cao, and Dacheng Tao.  Collect and select: Semantic alignment metric learning for few-shot learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8460–8469, 2019.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] Huaxi Huang, Junjie Zhang, Jian Zhang, Jingsong Xu, and Qiang Wu.  Low-rank pairwise alignment bilinear network for few-shot fine-grained image classification.  IEEE Transactions on Multimedia, 2020.
  * [13] Muhammad Abdullah Jamal and Guo-Jun Qi.  Task agnostic meta-learning for few-shot learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11719–11727, 2019.
  * [14] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC), volume 2, 2011.
  * [15] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [16] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.  Siamese neural networks for one-shot image recognition.  In ICML deep learning workshop, volume 2. Lille, 2015.
  * [17] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [18] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo.  Revisiting local descriptor based image-to-class measure for few-shot learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7260–7268, 2019.
  * [19] Wenbin Li, Jinglin Xu, Jing Huo, Lei Wang, Yang Gao, and Jiebo Luo.  Distribution consistency based covariance metric networks for few-shot learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8642–8649, 2019.
  * [20] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang.  Learning to propagate labels: Transductive propagation network for few-shot learning.  arXiv preprint arXiv:1805.10002, 2018.
  * [21] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel.  A simple neural attentive meta-learner.  arXiv preprint arXiv:1707.03141, 2017.
  * [22] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille.  Few-shot image recognition by predicting parameters from activations.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7229–7238, 2018.
  * [23] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  2016\.
  * [24] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  arXiv preprint arXiv:1807.05960, 2018.
  * [25] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-learning with memory-augmented neural networks.  In International conference on machine learning, pages 1842–1850, 2016.
  * [26] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4136–4145, 2020.
  * [27] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In Advances in neural information processing systems, pages 4077–4087, 2017.
  * [28] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 403–412, 2019.
  * [29] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1199–1208, 2018.
  * [30] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In Advances in neural information processing systems, pages 3630–3638, 2016.
  * [31] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [32] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.

## Supplementary Material

More Visualization Results

In our article, we visualized the $k$ (i.e., $k$=3) most discriminative LRs
selected by MATANet. This is achieved by outputting their index and
corresponding task attention values.

We provide some classification examples in Figure 6, compare with the most
relevant work DN4. By sending a fixed testing batch through the model, which
consists of one support sample and five query samples for each of five
classes, the prediction of MATANet only contains 7 mislabels in the entire 25
queries, while the prediction of DN4 has 10 wrong labels. That validates the
effectiveness of the MATANet. Our model is able to adaptively select the most
discriminative local representations in the current task at different scales.
We also find that in some classes like Black-footed Akbatross and Rhinoceros
Auklet, the high intra-variance and low inter-variance confuse all the models.

(a) by DN4

(b) by MATANet

Figure 6: Some visual classification results of comparing methods over CUB
Birds dataset. Both DN4 and MATANet use the same data batch under the 5-way
1-shot setting, and for each class, we randomly select five query images as
the testing data. We adopt five colors to label the support classes
separately. As to the query images, we label the images with the color
corresponding to the class label predicted by different models.

The Number of Trainable Parameters

We compare the number of trainable parameters to verify the efficiency of the
proposed MATANet, as Table 7 shows. Since no other trainable parameters are
introduced except for the embedding module $\mathcal{F}_{\theta}$ Prototypical
Nets and DN4 become the most light-weight models. GNN adopts a larger
embedding module (i.e., the filter number is 256), which draws a great
contribution from the number of parameters. Relation Network and CovaMNet
adopt additional architectures to boost the result, which also introduces a
huge number of trainable parameters. However, the proposed MATANet only
introduces a small number of the trainable parameters, while achieves a better
result than the methods above.

Model | Params | Accuracy(%)
---|---|---
Prototypical Net | 0.113M | 49.42$\pm$0.78
Relation Net | 0.229M | 50.44$\pm$0.82
GNN | 1.619M | 50.33$\pm$0.36
DN4 | 0.113M | 51.24$\pm$0.74
MATANet(Ours) | 0.314M | 53.63$\pm$0.83
Table 7: The number of trainable parameters in different models and the
corresponding classification accuracies on _mini_ ImageNet under 5-way 1-shot
setting.

Tranining procedure

The training procedure of the proposed MATANet is shown in Algorithm 1.

Algorithm 1 Tranining procedure

Input: Eposidic task
$\mathcal{T}=\left\\{\mathcal{A_{S}},\mathcal{A_{Q}}\right\\}$, superparameter
$k$.

while no converge do

for $\mathcal{A}^{q}_{\mathcal{Q}}$ in $\mathcal{A}_{\mathcal{Q}}$ do

$\mathcal{L}^{qz}\leftarrow\mathcal{G}_{\phi}(\mathcal{F}_{\theta}(\mathcal{A_{Q}}))$

$\mathcal{L}^{\mathcal{S}z}\leftarrow\mathcal{G}_{\phi}(\mathcal{F}_{\theta}(\mathcal{S}))$

Obtain semantic relation matrix $\mathcal{R}^{z}$ by Eq. (3)

Calculate adaptive task score $\alpha^{z}$ by Eq. (5)

Reweighting $\mathcal{R}^{z}$ by Eq. (6)

Calculate $\mathcal{P}^{q}$ by Eq. (7) and Eq. (8)

end for

$L$ $\leftarrow$ $-\sum\mathcal{Y}log(\mathcal{P})$

mini-Batch Adam to minimize $L$, update $\theta$, $\phi$, $\varphi$ and
$\omega$

end while

Implementation of MATANet

We provide a PyTorch implementation of MATANet for few-shot learning. Our code
is avaliable at https://github.com/chenhaoxing/MATANet.


# Intelligent methods for business rule processing: State-of-the-art

Cristiano André da Costa, Uélison Jean Lopes dos Santos, Eduardo Souza dos
Reis,
Rodolfo Stoffel Antunes, Henrique Chaves Pacheco, Thaynã da Silva França
Rodrigo da Rosa Righi, Jorge Luis Victória Barbosa
Software Innovation Laboratory - SOFTWARELAB
Universidade do Vale do Rio dos Sinos - UNISINOS
{cac,uelisonj,esreis, rsantunes, hcpacheco, tfrancanan, rrrighi,
<EMAIL_ADDRESS>
Franklin Jebadoss, Jorge Montalvao, Rogerio Kunkel
Dell Digital Services
Dell EMC
{franklin_jebadoss, j_Montalvao<EMAIL_ADDRESS>

###### Abstract

In this article, we provide an overview of the latest intelligent techniques
used for processing business rules. We have conducted a comprehensive survey
of the relevant literature on robot process automation, with a specific focus
on machine learning and other intelligent approaches. Additionally, we have
examined the top vendors in the market and their leading solutions to tackle
this issue.

##  1 Introduction

Business automation processes have gained popularity in recent times. Robot
Process Automation (RPA) reached its peak in September 2018, according to
Google Trends data [1]. In this article, we provide an in-depth analysis of
selected papers that describe the current state-of-the-art on RPA and
Intelligent Process Automation (IPA).

The main objective of this article is to present the latest research and
understanding of intelligent methods for processing business rules, especially
related to service order handling. The methods discussed involve the use of
machine processing techniques and natural language processing.

The article is structured as follows: Section 2 describe the research
methodology. Section 3 focuses on Robot Process Automation (RPA). Section 4
discusses Intelligent Process Automation (IPA). Section 5 explains the machine
learning approaches to IPA. Section 6 presents the leading vendors of RPA and
IPA solutions. Finally, in Section 7, we draw conclusions based on our
research.

##  2 Method

To begin our research process, we needed to select specific keywords and
define the scope of our search. To accomplish this, we utilized keywords
associated with RPA, IPA, and CPA and searched through Google Scholar
111https://scholar.google.com and Scopus 222https://www.scopus.com to identify
relevant articles. After conducting additional filtering and exclusion
processes, we removed works that did not fully address our research questions
from the corpus. The initial search yielded 107 articles, but after filtering,
we thoroughly reviewed 34 articles.

In the following sections, we will outline the most significant findings from
recent studies on process automation, particularly RPAs, and discuss the
latest advancements in IPAs, which incorporate cutting-edge techniques such as
machine learning, deep learning, and generative modeling.

##  3 Robot Process Automation

Currently, the literature refers to RPAs as the set of tools used to develop
bots, which mimic human behavior on business tasks, typically process-aware
tasks [2, 1]. For instance, opening a spreadsheet, changing values, and
committing. Bots bring many advantages to human agents, among which we
highlight enabling humans to make only creative, social, and decision-making
tasks [2]; accuracy can be expected to reach 100%, and availability of 24/7
working schedule [1]; cost reduction of the process (a standard RPA costs a
third of the cost of a full-time employee [3]); RPAs are highly scalable to
meet a varying intensity of demands [4]; and bots allow for transparent and
detailed documentation, increasing compliance [4].

Another essential characteristic of an RPA is its non-intrusiveness, meaning
its adoption and integration with business tools should be as effortless as
possible [5]. In turn, social skills are more suited for human agents,
demanding empathy and social interactions in order to build trust and customer
relationships [4]. In other words, the critical idea of RPA is to allow the
workers to spend more time on decision-making tasks instead of repetitive
manual labor, which can be replaced by a bot [3]. Formally, the IRPA-AI
Institute 333https://irpaai.com/ available on April 2023 definition of RPA is:

  1. $\bullet$

“the application of technology allowing employees in a company to configure
computer software or a ‘robot’ to capture and interpret existing applications
for processing a transaction, manipulating data, triggering responses, and
communicating with other digital systems”.

Quantitatively, Syed et al[1] report that RPA technology has proven to cut the
cost of human resource-related spending by 20–50% and a significant reduction
(from 30% to 70%) in the process cycle time. [3] experiments show an
improvement of 21% on the number of cases agents can handle while supported by
an RPA. Consequently, in the last five years, RPA became mainstream,
consisting of a set of tools that can automate processes based on business
rules, being classified as a highly promising approach [4, 3].

Within a business RPA tool, there are many modules and multiple bots. Yet, an
overview was defined by [1], consisting of three main components: a graphical
modeling tool, an orchestrator, and the bot itself. Graphical modeling tools
allow configuration and customization of the bots’ features by the human
agent. In a complex process, more than a single bot (defined per task) is
needed, and a collection of dependent bots must be developed, thus the need
for an orchestrator. The orchestrator is the software responsible for the
monitoring and availability of the system as a whole, composed of a task
scheduler and an urgency metric, to enable prioritization of resources to a
given bot when the system is overloaded. Other standard components are
performance analytic tools, which enable the evaluation and comparison of the
bots’ work with human agents to contrast their efficiency.

Aguirre and Rodriguez [3] propose criteria to identify which tasks to
automate, choosing highly structured tasks, usually back office. Other
proposed criteria are:

  * •

Low cognitive requirements. A task that does not require subjective judgment,
creativity, or interpretation skills.

  * •

High frequency. A task that is repeated constantly during the human agent’s
work cycle.

  * •

Access to multiple systems. A process that requires access to multiple
applications and systems to perform the job.

  * •

Limited exception handling. Tasks that have limited or no exceptions to
handle.

  * •

Prone to human errors.

One of the considered works [6] includes IT ticket automation as a core task
for RPAs in the enterprise context. They are usually composed of unstructured
(client interaction logs) and structured (agent’s notes) text. Therefore,
natural language processing is essential in IT ticket processing, offering a
broad spectrum of project opportunities.

Figure 1: The IPA area is defined in the literature as the intersection
between RPA and Cognitive Process Automation [7]. Consulting firms like
Deloitte and Capgemini argue that the main areas where RPA can be applied are
accounts payable, accounts receivable, travel and expenses, fixed assets, and
human resource administration. With cognitive strategies, it is possible to
address intricate problems that deal with probabilities and pattern
recognition .

##  4 Intelligent Process Automation

The tasks performed by RPAs perform are typically rule-based, well-structured,
and repetitive [1]. Nevertheless, future RPAs or IPA, must include modules
capable of dealing with unstructured data [2]. In contrast, IPAs are more
expensive to build. One prevalent application of IPAs is the natural language
processing bots that, combined with machine learning, will replace human
agents in customer relations activities [1].

An IPA adds another layer to the standard RPA: Cognitive Process Automation
(CPA), as depicted in Figure 1. CPA focuses on knowledge work and utilizes
constructed AI instead of classical AI. Moreover, CPA can integrate with deep
learning to incorporate natural language generation, computer vision (AI-
screen recognition), and self-improvement [7]. Improving over time and
changing the decision-making process as the actual business process changes
are the main feature of current IPAs, yet remains largely an open issue.
Online learning is achieved through monitoring and retraining the models [8].
The objective is to enable IPAs to identify such changes, predict future
risks, and either alert or adapt when feasible. In this context, generative
models are suited for validating, monitoring, and adapting models to novel
situations.

##  5 Machine Learning approaches to IPA

When it comes to machine learning, IPAs (Intelligent Personal Assistants)
utilize online learning, which involves continuous changes in model weights,
to process complex inputs. These inputs typically consist of unconstrained
textual information mixed with categorical elements annotated by a human
agent, such as urgency, observations, or required parts.

Most tasks can be accomplished using a sequence-to-sequence modeling
framework, which is especially useful for natural language processing. This
framework involves the mapping of input tokens, such as words, to output
tokens by way of similarity. Essentially, this means mapping the probability
distribution of the next token in the output sequence to a variable-length
input sequence.

In practice, the mapping is not done directly, and the model also yields
intermediate representations that encode the context in which the input tokens
are presented. These representations are referred to as context vectors in
natural language processing, and the two-step procedure is known as encoder-
decoder methods [9].

Therefore, the set of input tokens is a vector of words encoded in a
continuous representation, such as word2vec [10].

Figure 2: Gartner’s magic quadrant for RPA shows UiPath, Automation Anywhere
and Blueprism as the leaders in the sector [1].

Initially, the community leaned towards recurrent models for both decoders and
encoders, mainly to the RNNs [11]. Nonetheless, RNN-based models have visible
shortcomings. For one, relying on a unidirectional pipeline, the network can
access the full context at the end of a sentence. Still, earlier iterations
need to have information on the incoming tokens. Instead of increasing the
length of the context vector, the Transformers [12] selectively look for the
most informative tokens at each timestep through self-attention. Self-
attention is an attention mechanism that correlates different positions of the
same sequence, given higher weights for correlations among tokens within the
same context, due to referring to the same object or being close together in
the sentence. Through self-attention, each cell of the context vector is
informed by all previous inputs, resulting in a sizeable receptive field over
the whole sentence [11].

In contrast, for categorical objects, classifiers such as Neural Networks or
Random Forests have been widely explored. Therefore, an ideal IPA must
implement different machine learning models to act over other domains on each
bot. As the number of models increases, the communication among them becomes a
bottleneck. Thus, models capable of projecting the heterogeneous inputs in a
single vectorized encoding are needed. In other words, the goal is to use the
heterogeneous input projection to a regularized subspace of simple
N-dimensional vectors as a preprocessing step and then feed this vector
representation to the actual classifiers or decision models.

##  6 Vendors

In recent years, RPA has emerged as the solution for process automation. The
RPA market is predicted to reach a market volume of $ 2.9 billion in 2021 [5].
As a result, a large number of vendors proposed their tools, creating a
challenging competition between players. Market leaders are Blue Prism,
arguably the pioneer RPA product, UIPath, and Automation Anywhere [1]. All
three leaders include modern techniques, such as OCR, for extracting data from
documents, and Computer Vision, allowing the bots to interact with objects on
HTML, PDF, or virtual desktop interfaces. Another key component is
understanding the unstructured text consumers use when interacting with the
bots. Therefore, natural language processing became central in the RPA market,
automating chatbots, form filling, and voice interactions. Other widely
mentioned products on our corpus include Workfusion, Kryon Systems,
Softomotive, Contextor, EdgeVerve, niCE, and Redwood Software. Pegasystems and
Cognizant provide RPA functionality embedded into more traditional CRM and BI
functionalities, providing less intrusiveness. Figure 2 classifies most of the
studied products by distinct features.

When RPA exploded in popularity, some important issues, such as scalability,
interoperability, and portability, emerged. Since a large portion of
enterprise applications are developed based on graphical user interfaces
(GUI), some vendors are developing specific APIs to facilitate portability
between client software and bots. Furthermore, according to Simek et al.[13],
multiple RPA solutions can be integrated via these APIs.

As for performance metrics, most businesses rely on quality metrics that are
manageable to generalize. Hence, valuable indicators such as resource
allocation and execution time are more straightforward metrics for IPAs [7].
Other options for the current project context include benchmarks, such as
management ticket texts provided in [6]. Based on the same RPA development
methodology that we adopted, Figure 3 compares the leading players by
efficiency on each step.

Figure 3: Comparison by development steps of the most complete vendor tools
[14].

##  7 Conclusion

This article provides a literature review of RPA and its related concepts such
as Cognitive Process Automation and IPA. The review helps to understand the
current issues and possible technological applications of RPA from both
scientific and business perspectives. We also compare the existing RPA
vendors.

The existing literature on RPA primarily focuses on definitions and case
studies, while studies on technical and implementation strategies are scarce.
To address this gap, future research could combine machine learning and
pattern recognition with RPA, which some researchers refer to as IPA. It is
crucial for RPA strategies to integrate with current systems, other tools, and
technologies to provide scalability and performance according to business
needs. Therefore, future studies must combine these technologies.

## References

  * [1] Rehan Syed, Suriadi Suriadi, Michael Adams, Wasana Bandara, Sander JJ Leemans, Chun Ouyang, Arthur HM ter Hofstede, Inge van de Weerd, Moe Thandar Wynn, and Hajo A Reijers.  Robotic process automation: contemporary themes and challenges.  Computers in Industry, 115:103162, 2020.
  * [2] Wil MP Van der Aalst, Martin Bichler, and Armin Heinzl.  Robotic process automation, 2018.
  * [3] Santiago Aguirre and Alejandro Rodriguez.  Automation of a business process using robotic process automation (rpa): A case study.  In Workshop on engineering applications, pages 65–71. Springer, 2017.
  * [4] Judith Wewerka and Manfred Reichert.  Robotic process automation–a systematic literature review and assessment framework.  arXiv preprint arXiv:2012.11951, 2020.
  * [5] Julia Hindel, Lena M Cabrera, and Matthias Stierle.  Robotic process automation: Hype or hope?  WI2020 Zentrale Tracks, pages 1750–1762, 2020.
  * [6] Nina Rizun, Vera Meister, and Aleksandra Revina.  Discovery of stylistic patterns in business process textual descriptions: It ticket case.  Innovation Management and Education Excellence through Vision, 2020.
  * [7] Sharon Richardson.  Cognitive automation: A new era of knowledge work?  Business Information Review, 37(4):182–189, 2020.
  * [8] Tathagata Chakraborti, Vatche Isahagian, Rania Khalaf, Yasaman Khazaeni, Vinod Muthusamy, Yara Rizk, and Merve Unuvar.  From robotic process automation to intelligent process automation.  In International Conference on Business Process Management, pages 215–228. Springer, 2020.
  * [9] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv preprint arXiv:1406.1078, 2014.
  * [10] Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  CoRR, abs/1301.3781, 2013.
  * [11] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser.  Universal transformers.  arXiv preprint arXiv:1807.03819, 2018.
  * [12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  CoRR, abs/1706.03762, 2017.
  * [13] Dalibor Šimek and Roman Šperka.  How robot/human orchestration can help in an hr department: a case study from a pilot implementation.  Organizacija, 52(3), 2019.
  * [14] José Gonzalez Enríquez, A Jiménez-Ramírez, FJ Domínguez-Mayo, and JA García-García.  Robotic process automation: a scientific and industrial systematic mapping study.  IEEE Access, 8:39113–39129, 2020.


[table]capposition=bottom capbtabboxtable[][]

# Contrastive Bootstrapping for Label Refinement

Shudi Hou† Yu Xia† Muhao Chen‡ Sujian Li†
†Key Laboratory of Computational Linguistics, MOE, Peking University
‡University of Southern California
{housd, yuxia<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Traditional text classification typically categorizes texts into pre-defined
coarse-grained classes, from which the produced models cannot handle the real-
world scenario where finer categories emerge periodically for accurate
services. In this work, we investigate the setting where fine-grained
classification is done only using the annotation of coarse-grained categories
and the coarse-to-fine mapping. We propose a lightweight contrastive
clustering-based bootstrapping method to iteratively refine the labels of
passages. During clustering, it pulls away negative passage-prototype pairs
under the guidance of the mapping from both global and local perspectives.
Experiments on NYT and 20News show that our method outperforms the state-of-
the-art methods by a large margin.111Code is available at
https://github.com/recorderhou/contrastive_bootstrapping_label_refinement

##  1 Introduction

Traditional text classification often categorize into a set of coarse-grained
classes, which falls short in real-world scenarios where finer categories
emerge. To this end, coarse-to-fine text classification is introduced (Mekala
et al., 2021), which performs fine-grained classification given only
annotation of coarse-grained categories and the coarse-to-fine mapping. Then,
it finetunes a pre-trained language model for each coarse prototype.222We use
prototype and category interchangeably. However, this two-step method could be
sub-optimal. For example, it is vulnerable to the noise which is propagated
and accumulated through the pipeline. Besides, it requires finetuning and
saving a pre-trained language model for each coarse prototype which is
heavyweight.

Figure 1: Passages with “Arts” coarse prototype on NYT dataset. Colors are
used to denote different fine prototypes. (a) warm-up. (b) bootstrapping w/o
selection strategy. (c) bootstrapping w/ selection strategy.

To this end, we propose a lightweight bootstrapping method based on
contrastive clustering to iteratively refine the labels of passages.333We
focus on passage-level classification as it is consistent with prior studies
Mekala et al. (2021). Though, without loss of generality, the studied problem
as well as the proposed method can be extended to classifying natural language
text in other granularities. To be more specific, the method starts with an
epoch of warm-up on the weakly-labeled dataset. During warm-up, it pulls away
negative passage-prototype pairs under the guidance of the mapping from both
global and local perspectives, i.e., coarse inter-cluster and fine inter-
cluster perspectives. After the warm-up, the distances between clusters are
not significant which causes misclassification. Instead of continuing training
on the weakly-labeled dataset which might greatly increase the noise (Figure
1(b)), we perform a bootstrapping process which finetunes the model on the
selected dataset and updates the selected dataset by the finetuned model
alternately. To mitigate the noise, we propose a selection strategy to
identify high-quality pairs in terms of similarity and distinction. To further
boost our method, we adopt a modified similarity metric from Lample et al.
(2018) and use the gloss knowledge to augment the prototype representation. As
shown in (Figure 1(c)), the resulting clusters are well separated with less
noise.

Our contributions are summarized as follows:

  * •

We propose a lightweight bootstrapping method based on contrastive clustering
to address the problem of coarse-to-fine text classification.

  * •

Our method outperforms the state-of-the-art methods on two widely-used
datasets. Further analysis verifies the effectiveness of our proposed
techniques.

##  2 Proposed Method

This section describes the technical details of the proposed method, starting
with the task description.

###  2.1 Task Description

We follow the task definition of coarse-to-fine text classification in
previous work Mekala et al. (2021). Given $n$ passages $\\{p_{1},...,p_{n}\\}$
with their corresponding coarse-grained labels $\\{c_{1},...,c_{n}\\}$, along
with the coarse-to-fine mapping $\mathcal{T}$, our goal is to assign a fine-
grained label to each passage. The key notations used in our paper are defined
as follows: (1)
$\mathcal{C}=\\{\mathcal{C}_{1},\mathcal{C}_{2},...,\mathcal{C}_{m}\\}$
denotes the coarse prototypes. (2)
$\mathcal{F}=\\{\mathcal{F}_{1},\mathcal{F}_{2},...,\mathcal{F}_{k}\\}$
denotes the fine prototypes. (3) $\mathcal{T}:\mathcal{C}\to\mathcal{F}$
denotes the coarse-to-fine mapping, a surjective mapping which separates
$\mathcal{F}$ into $|\mathcal{C}|$ non-overlapping partitions. (4)
$\mathcal{S}_{pf}=\mathcal{T}(c_{i})$ denotes the fine-grained candidate
prototype of $p_{i}$, which is also dubbed as $p$ for simplicity. (5)
$\mathcal{S}_{nf}={\mathcal{F}}/{\mathcal{S}_{pf}}$ denotes fine prototypes
not belonging to $\mathcal{T}(c_{i})$. (6)
$\mathcal{S}_{nc}={\mathcal{C}}/{c_{i}}$ denotes coarse prototypes in
$\mathcal{C}$ other than $c_{i}$.

###  2.2 Our Method

#### Training Process

As illustrated in Figure 2, we start with an epoch of warm-up, during which we
optimize two contrastive losses $\mathcal{L}_{global}$, $\mathcal{L}_{local}$
on the weakly-labeled dataset and only the $\mathcal{L}_{global}$ on the
unlabeled dataset. The two contrastive losses are detailed in the following
paragraphs. Then, we conduct several epochs of bootstrapping with the above
model. At each bootstrapping step, we first select a small set of passages on
which labels are predicted with high confidence by the model. Then, we
finetune the model on the selected dataset with the same losses as warm-up. We
repeat the finetuning and the selection alternately.

#### Initial Weak Supervision

Following previous work, we consider samples that exclusively contain the
label surface name as their respective weak supervision. More details can be
referred to the prior study.

Figure 2: Illustration of our training process.

#### Passage and Prototype Representation

We encode passages $\\{p_{1},...,p_{n}\\}$ and all prototypes
$\mathcal{C}\cup\mathcal{F}$ into the same embedding space with a pretrained
language model. The resulting passage representation and prototype
representation are denoted as $\bm{p}$ and $\bm{l}$ respectively. During the
training process, the prototype representations are dynamically updated to fit
the current passage representations. Specifically, we use the last hidden
representation of [CLS] as their representations.

#### Similarity Metric

Cosine similarity is often used to measure semantic similarity of embedding
representations. However, in high-dimensional spaces, some “hub” vectors may
be close to many other vectors while some other vectors are instead being
isolated. For example, a passage’s representation $\bm{p}$ may get high cosine
with a large number of labels in $S_{pf}$ due to such hubness issues. In this
case, a high similarity score does not necessarily lead to a high discrepancy
among labels. Selecting a highly-scored label from the hub as the seed is
potentially detrimental to our pairing-based method. Inspired by cross-domain
similarity local scaling (Lample et al., 2018), we adopt a modified similarity
metric $c(\bm{p},\bm{l})$ to prevent passage vectors from becoming hubs:

$\displaystyle c(\bm{p},\bm{l})=\cos(\bm{p},\bm{l})-KNN(\bm{p})$ (1)
$\displaystyle
KNN(\bm{p})=\frac{1}{K}\sum\max_{\bm{l}\in\mathcal{F}}K\\{\cos(\bm{p},\bm{l})\\}$
(2)

where $KNN(.)$ denotes $K$ nearest neighbors.

#### Warm-up

Viewing a passage as an anchor, we expect that its semantic similarity to the
correct fine-grained prototype should be closer than any other fine-grained
candidate prototypes. We regard the distance in the representation space as
the similarity. Specifically, we optimize the following margin ranking loss:

$\displaystyle\mathcal{L}_{global}=\frac{1}{|S_{pf}|}\sum_{\begin{subarray}{c}l\in
S_{pf}\\\ l^{\prime}\in
S_{nf}\end{subarray}}\max\\{c(\bm{p},\bm{l})-c(\bm{p},\bm{l^{\prime}})+\gamma,0\\}$
(3)

where $\gamma$ is a hyper-parameter denoting the margin. We use all fine
candidate prototypes in $S_{pf}$ as positive examples and randomly sample the
same number of prototypes from $S_{nf}$ as negative examples. We view this
loss as a global loss to cluster samples according to their coarse labels
(Figure 3).

For instances labeled in the initial weak supervision stage, we adopt another
margin ranking loss:

$\displaystyle\mathcal{L}_{local}=\max\\{sec\\_max-c(\bm{p},\bm{l})+\sigma,0\\}$
(4) $\displaystyle sec\\_max=\max_{l^{\prime}\in
S_{pf},l^{\prime}!=l}c(\bm{p},\bm{l^{\prime}})$ (5)

We regard this loss as a local loss to cluster samples according to their
fine-grained labels (Figure 1 (a)).

#### Bootstrapping

After the warm-up, representations show an inclination to form clusters. Yet,
the distances between them are not significant enough to separate the classes.
To further get compact clusters, we perform bootstrapping which finetunes the
model on the selected dataset and updates the selected dataset by the
finetuned model alternately. Instead of using the initial weak supervision
which might greatly increase the noise as observed, we propose a selection
strategy to select high-quality passage-prototype pairs. Specifically, we
assign a pseudo label to each passage by their similarity (Eq.(6)). Apart from
similarity, we assume high-quality pairs should also be discriminative
(Eq.(7)):

$\displaystyle l=arg\max_{l\in S_{pf}}c(\bm{p},\bm{l})$ (6) $\displaystyle
c(\bm{p},\bm{l})-\max_{l^{\prime}\in
S_{pf},l^{\prime}!=l}c(\bm{p},\bm{{l^{\prime}}})>\beta$ (7)

where $\beta$ is a threshold updated at each epoch. We construct a confident
set $\mathrm{CS}$ with top $r\%$ pairs satisfying these two conditions. We
update $\beta$ with the lowest similarity in $\mathrm{CS}$. Then, we optimize
Eq.(4) and Eq.(3) on $\mathrm{CS}$ and the rest passages accordingly.

#### Gloss Knowledge

Since the surface names alone can not well represent the semantics of labels,
we enrich them with external semantic knowledge. To be more specific, we
select the first two sentences in each surface name’s first Wikipedia webpage
to augment the original surface name with a predefined template (Table 3). We
adopt the format of “template, surface name, gloss” and use the last hidden
representation of [CLS] as their representation.

#### Prediction

It is worth noticing that applying our similarity metric $c(\bm{p},\bm{{l}})$
do not change the relative ranking among labels in $\mathcal{S}_{pf}$ compared
with the cosine similarity. For simplicity, we use cosine similarity for
prediction.

$\displaystyle l=arg\max_{l\in S_{pf}}\cos(\bm{p},\bm{l})$ (8)

##  3 Experiments

In this section, we describe the experimental evaluation for the proposed
method.

###  3.1 Datasets and Metrics

For a fair comparison with prior work, we use the same hierarchical datasets
used by We report both Macro-F1 and Micro-F1 for evaluation on the following
two datasets.

#### The 20 Newsgroups (20News)

The passages in 20News was organized into 5 coarse-grained newsgroups and 20
fine-grained newsgroups corresponding to different topics (Table 2). Passages
in 20News were partitioned evenly across the 20 different fine-grained
newsgroups.444http://qwone.com/~jason/20Newsgroups/ Following Mekala et al.
(2021), we omitted the 3 miscellaneous newsgroups (“misc.forsale,”
“talk.politics.misc” and “talk.religion.misc”) and expanded the abbreviation
to full words.

#### The New York Times (NYT)

This dataset contains 5 coarse-grained topics and 25 subtopics (Table 2). The
NYT dataset is highly skewed with the coarse-grained topic “sports” containing
more than 80% passages.

Figure 3: Passage representations after warm-up on 20News dataset. Colors are used to denote different coarse prototypes. | NYT | 20News
---|---|---
| Mi-F1(%) | Ma-F1(%) | Mi-F1(%) | Ma-F1(%)
LOT-Class | 79.26 | 63.16 | 56.38 | 54.80
X-Class | 58.15 | 60.50 | 52.95 | 53.47
C2F | 89.23 | 84.36 | 75.77 | 75.24
C2F w/ our select $\star$ | 89.64 | 82.72 | 77.20 | 76.41
Ours | 92.64 | 89.90 |  77.64 |  77.22
w/o fine | 91.15 ($\downarrow$ 1.49) | 84.90 ($\downarrow$ 5.00) | 74.34 ($\downarrow$ 3.30) | 73.78 ($\downarrow$ 3.44)
w/o bootstrap | 89.49 ($\downarrow$ 3.15) | 82.50 ($\downarrow$ 7.40) | 76.01 ($\downarrow$ 1.63) | 75.46 ($\downarrow$ 3.30)
w/o gloss | 89.91 ($\downarrow$ 2.73) | 80.48 ($\downarrow$ 9.42) | 72.68 ($\downarrow$ 4.86) | 70.31 ($\downarrow$ 6.91)
w/o select | 87.56 ($\downarrow$ 5.08) | 81.98 ($\downarrow$ 8.02) |  79.74 ($\uparrow$ 2.10) |  79.21 ($\uparrow$ 1.99)
w/o similarity | 89.25 ($\downarrow$ 3.39) | 82.44 ($\downarrow$ 7.46) | 61.21 ($\downarrow$ 16.43) | 54.76 ($\downarrow$ 22.46)
w/ Manhattan similarity $\dagger$ | 33.45 ($\downarrow$ 59.19) | 39.47 ($\downarrow$ 50.43) | 41.83 ($\downarrow$ 35.81) | 36.50 ($\downarrow$ 40.72)
w/ Euclidean similarity $\ddagger$ |  92.46 ($\downarrow$ 0.18) |  89.17 ($\downarrow$ 0.73) | 72.11 ($\downarrow$ 5.53) | 70.65 ($\downarrow$ 6.57)
Table 1: Results on NYT and 20News. “$\star$" equips C2F with our selection
strategy. “$\dagger$” replaces our similarity metric with Manhattan distance.
“$\ddagger$” replaces our similarity metric with Euclidean distance.

###  3.2 Main Results

We compare our model with the previous work (Mekala et al., 2021), as well as
several zero-shot weakly supervised text classification methods (Wang et al.,
2021b; Meng et al., 2020a) following previous works. We reproduce them using
their
implementation.555https://github.com/yumeng5/LOTClass666https://github.com/ZihanWangKi/XClass777https://github.com/dheeraj7596/C2F

As shown in Table 1, our method outperforms the baselines by 5.67% in Micro-F1
and 5.54% in Macro-F1 on the NYT dataset, as well as 3.97% in Micro-F1 and
3.04% in Macro-F1 on 20News dataset.

###  3.3 Analysis

To verify the effectiveness of different model components , we conduct
ablation studies to test each of those.

#### Effect of Bootstrapping

The “w/o bootstrap” results in Table 1 report the performance with warm-up
only. These results are consistently lower than those with bootstrapping.
Specifically, bootstrapping improves the warm-up by 3.15% Micro-F1, 7.40%
Macro-F1 and 1.63% Micro-F1, 3.30% Macro-F1 on NYT and 20News respectively.
Figure 1(a)(c) shows passage representations are more separated from each
other.

#### Effect of Selection Strategy

We replace the selection strategy in bootstrapping with the initial weakly-
labeled samples. From the “w/o bootstrap” results in Table 1, we can see that,
our selection strategy brings an improvement of 4.26% Micro-F1, 7.46% Macro-F1
on NYT. It is better to use the seed dataset on 20News. We hypothesize that
this observation is because the seed dataset has a more balanced label
distribution than our selected high-quality samples on 20News. We also
incorporate our selection strategy to the C2F baseline in the bootstrapping
stage. As shown in Table 1 row “C2F w/ our select,” this strategy improves the
performance of C2F by 1.43% Micro-F1, 1.17% Macro-F1 on 20News and 0.41%
Micro-F1 on NYT, exhibiting the effectiveness of our strategy.

#### Effect of Similarity Metric

We replace our similarity metric with the cosine similarity. From Table 1 “w/o
similarity” we can see that, our similarity metric brings along an improvement
of 3.39% in Micro-F1, 7.46% in Macro-F1 on NYT, and 16.43% in Micro-F1 and
22.46% in Macro-F1 on 20News. From Figure 4, we can see that 63% of samples
belonging to the “Law Enforcement” prototype are misclassified using the
cosine similarity. However, 18% are misclassified using our similarity metric,
verifying its effectiveness. Besides, results for “w/ Manhattan similarity”
and “w/ Euclidean similarity” show that alternating cosine similarity in
$c(\bm{p},\bm{l})$ causes performance drops of 35.81% (5.53%) in Micro-F1,
40.72% (6.57%) in Macro-F1 and 50.19% (0.18%) in Micro-F1, 50.43% (0.73%) in
Macro-F1 on 20News and NYT data, further proving the effectiveness of our
similarity metric.

Figure 4: Confusion matrix on “Politics” coarse prototype. Our similarity
metric (right) outperforms cosine similarity (left) by 12.25% Macro-F1 and
16.68% Micro-F1 under “Politics.”

#### Effect of Gloss Knowledge

We remove the gloss knowledge and use the label surface name only. Comparing
the “w/o gloss” results in Table 1 with the full-setting ones, we observe that
the gloss knowledge brings an improvement of 2.73% in Micro-F1, 9.42% in
Macro-F1 on NYT and 4.86% in Micro-F1, 6.91% in Macro-F1 on 20News. Figure 5
further shows the effect of gloss knowledge on different prototypes.

#### Extending to the setting without coarse-to-fine mapping

We extend our method to the setting without the coarse-to-fine mapping. In
other words, the only supervision is the gold coarse labels. We modify
$\mathcal{L}_{global}$ as follows:

$\displaystyle\mathcal{L}_{c\\_global}=\max\\{c(\bm{p},\bm{l_{c}})-c(\bm{p},\bm{l_{c}^{\prime}})+\gamma,0\\}$
(9)

where we use the golden coarse label $\bm{l_{c}}$ as the positive example and
randomly sample one coarse label $\bm{l_{c}^{\prime}}$ from $\mathcal{S}_{nc}$
as the negative example. The “w/o fine” results in Table 1 show that the
performance does not degrade much when the association between coarse and
fine-grained labels does not exist, showing the feasibility of our method in a
more general setting.

##  4 Related Work

Previous works in weakly supervised text classification have explored
different kinds of weak supervision. (1) a set of related keywords. Mekala and
Shang (2020) augment and disambiguate the initial seed words with
contextualized and highly label-indicative keywords. Meng et al. (2020b)
identify keywords for classes by querying replacements for class names using
BERT and pseudo-labels the documents by heuristics with the selected keywords.
(2) a few labeled documents. Tang et al. (2015) represent the labeled
documents and different levels of word co-occurrence information as a large-
scale text network. Meng et al. (2018) propose a pseudo-document generator
that leverages the seed labeld documents to generate pseudo-labeled documents
for model pre-training. (3) label surface names. Wang et al. (2021b) propose
an adaptive representation learning method to obtain label and document
embedding, and cluster them to pseudo-label the corpus. Our setting is
different from theirs in that we use coarse-grained annotation to improve the
fine-grained text classification.

Contrastive learning (He et al., 2020; Chen et al., 2020; Khosla et al., 2020)
aims at learning representations by contrasting the positive pairs and
negative pairs. In NLP, existing works can be primarily categorized into two
distinct streams. Unsupervised contrastive learning seeks to contrast grouped
or perturbed instances to generate more robust representation of unlabeled
textual data (Gao et al., 2021; Wei et al., 2021; Kim et al., 2021; Wang et
al., 2021a). On the contrary, supervised contrastive learning (Suresh and Ong,
2021; Zhou et al., 2021; Yu et al., 2021; Huang et al., 2022) is label-aware
and seeks to create representations for differently labeled data with more
discrepancy. Our work has shown that supervised contrastive learning
incorporating label names, with minimal external knowledge, improves the
model’s performance in label refinement.

##  5 Conclusion

In this paper, we study the task of coarse-to-fine text classification. We
propose a novel contrastive clustering-based bootstrapping method to refine
the label in an iterative manner. Experiments on two real-world datasets for
coarse-to-fine text classification verify the effectiveness of our method.
Future work could consider extending this method to other fine-grained
decision-making tasks that could potentially benefit from coarse-grained
labels, such as various kinds of lexical semantic typing tasks Huang et al.
(2022). Another meaningful direction is to consider incorporating other
partial-label learning techniques Zhang et al. (2016) that are relevant to
coarse-to-fine prediction tasks.

## Limitations

Our paper has the following limitations: (1) In real-world applications, the
label hierarchy may be more than two levels. It is worth extending our method
to such a setting and empirically verifying it. (2) Our selection strategy
simply takes top $r\%$ confident samples, which might result in class
imbalance problem. Alleviating the imbalance problem may further improve our
performance. We leave them as future work.

## Acknowledgement

We appreciate the reviewers for their insightful comments and suggestions. We
would like to express our gratitude to the authors of the C2F paper Mekala et
al. (2021) for their collective effort in open-sourcing the dataset and code.
Their released materials played a vital role in our research.

Shudi Hou, Yu Xia and Sujian Li were supported by National Key R&D Program of
China (No. 2020AAA0109703). Muhao Chen was supported by the National Science
Foundation of United States Grant IIS 2105329, a subaward of the INFER Program
through UMD ARLIS, an Amazon Research Award and a Cisco Research Award.

## References

  * Buitinck et al. (2013) Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. 2013.  API design for machine learning software: experiences from the scikit-learn project.  In _ECML PKDD Workshop: Languages for Data Mining and Machine Learning_ , pages 108–122.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1597–1607. PMLR.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Huang et al. (2022) James Y. Huang, Bangzheng Li, Jiashu Xu, and Muhao Chen. 2022.  Unified semantic typing with meaningful label inference.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2642–2654, Seattle, United States. Association for Computational Linguistics.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18661–18673. Curran Associates, Inc.
  * Kim et al. (2021) Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021.  Self-guided contrastive learning for BERT sentence representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2528–2540, Online. Association for Computational Linguistics.
  * Lample et al. (2018) Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018.  Word translation without parallel data.  In _International Conference on Learning Representations_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Fixing weight decay regularization in adam.  _ArXiv_ , abs/1711.05101.
  * Mekala et al. (2021) Dheeraj Mekala, Varun Gangal, and Jingbo Shang. 2021.  Coarse2Fine: Fine-grained text classification on coarsely-grained annotated data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 583–594, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Mekala and Shang (2020) Dheeraj Mekala and Jingbo Shang. 2020.  Contextualized weak supervision for text classification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 323–333.
  * Meng et al. (2018) Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018.  Weakly-supervised neural text classification.  In _proceedings of the 27th ACM International Conference on information and knowledge management_ , pages 983–992.
  * Meng et al. (2020a) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020a.  Text classification using label names only: A language model self-training approach.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9006–9017, Online. Association for Computational Linguistics.
  * Meng et al. (2020b) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020b.  Text classification using label names only: A language model self-training approach.  _arXiv preprint arXiv:2010.07245_.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Suresh and Ong (2021) Varsha Suresh and Desmond Ong. 2021.  Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4381–4394, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tang et al. (2015) Jian Tang, Meng Qu, and Qiaozhu Mei. 2015.  Pte: Predictive text embedding through large-scale heterogeneous text networks.  In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1165–1174.
  * Wang et al. (2021a) Dong Wang, Ning Ding, Piji Li, and Haitao Zheng. 2021a.  CLINE: Contrastive learning with semantic negative examples for natural language understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2332–2342, Online. Association for Computational Linguistics.
  * Wang et al. (2021b) Zihan Wang, Dheeraj Mekala, and Jingbo Shang. 2021b.  X-class: Text classification with extremely weak supervision.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3043–3053, Online. Association for Computational Linguistics.
  * Wei et al. (2021) Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021\.  On learning universal representations across languages.  In _International Conference on Learning Representations_.
  * Yu et al. (2021) Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. 2021.  Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training approach.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1063–1077, Online. Association for Computational Linguistics.
  * Zhang et al. (2016) Min-Ling Zhang, Bin-Bin Zhou, and Xu-Ying Liu. 2016.  Partial label learning via feature-aware disambiguation.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 1335–1344, New York, NY, USA. Association for Computing Machinery.
  * Zhou et al. (2021) Wenxuan Zhou, Fangyu Liu, and Muhao Chen. 2021.  Contrastive out-of-distribution detection for pretrained transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1100–1111, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

##  Appendix A Dataset Statistics

We list the statistics of the datasets in Table 2.

Dataset | Passage | $|\mathcal{C}|$ | $|\mathcal{F}|$ | Coarse Prototype | Fine Prototype
---|---|---|---|---|---
20News | 16468 | 5 | 17 | computer, politics, recreation, | graphics, windows, ibm, mac, x window, mideast, guns, autos, motorcycles,
|  |  |  | religion, science | baseball, hockey, christian, atheism, encryption, electronics, medicine, space
|  |  |  |  | dance, music, movies, television, economy, energy companies, international
NYT | 11744 | 5 | 26 | arts, business, politics, | business, stocks and bonds, abortion, federal budget, gay rights, gun control,
|  |  |  | science, sports | immigration, law enforcement, military, surveillance, the affordable care act,
|  |  |  |  | cosmos, environment, baseball, basketball, football, golf, hockey, soccer, tennis

Table 2: Dataset Statistics.

##  Appendix B Templates

We list the templates used in Table 3.

Dataset | Template
---|---
NYT | 1 : The news is about, 2 : The news is related to, 3 : The topic of this passage is
20News | 1 : The topic of this post is , 2 : They are discussing , 3 : This post mainly talks about

Table 3: Three variants of templates used to concatenate the gloss knowledge
and the surface name. The first template is best for NYT and the third
template is best for 20News.

##  Appendix C Effect of gloss knowledge on different prototypes

We show the confusion matrix over all fine prototypes in Figure 5.

##  Appendix D Implementation Details

We use RoBERETa-base Liu et al. (2019) as the encoder. The models are trained
on one GeForce RTX 3090 GPU. We set the batch size as 8. We do one epoch of
warmup and four epochs of bootstrapping. We use the predictions from the last
epoch as the final predictions. We use AdamW Loshchilov and Hutter (2017) as
the optimizer. $r$ is set as 15 for NYT and 1 for 20News. $\gamma$ and
$\sigma$ are set as 0.05 for both NYT and 20News. We run our model 3 times
using different random seeds. We used t-SNE Pedregosa et al. (2011); Buitinck
et al. (2013) for the visualization in this paper.

##  Appendix E Selection of $r$

We select the value of $r$ from set {1, 5, 10, 15, 20}. For each coarse
prototype $\mathcal{C}_{i}$, we calculate the ratio of initial weak
supervision $W_{\mathcal{C}_{i}}$ in category $\mathcal{C}_{i}$ to the total
number of instance $I_{\mathcal{C}_{i}}$ in $\mathcal{C}_{i}$, we denote the
ratio as $R_{\mathcal{C}_{i}}=W_{\mathcal{C}_{i}}/I_{\mathcal{C}_{i}}$. After
that, we select the $r$ closest to
$\underset{\mathcal{C}_{i}\in\mathcal{C}}{\min}\\{R_{\mathcal{C}_{i}}\\}$. As
shown in Table 4(a) and Table 4(b), the minimal $R_{\mathcal{C}_{i}}$ in NYT
dataset is 13.43%, closest to 15, while the minimal $R_{\mathcal{C}_{i}}$ in
20News dataset is 2.05%, closest to 1.

$\mathcal{C}_{i}$ | $W_{\mathcal{C}_{i}}$ | $I_{\mathcal{C}_{i}}$ |  $R_{\mathcal{C}_{i}}$ (%)
---|---|---|---
arts | 184 | 1043 | 17.64
business | 132 | 983 | 13.43
politics | 216 | 989 | 21.84
science | 42 | 90 | 46.67
sports | 1890 | 8639 | 21.88
(a) Ratio of the initial weak supervision in NYT

$\mathcal{C}_{i}$ | $W_{\mathcal{C}_{i}}$ | $I_{\mathcal{C}_{i}}$ |  $R_{\mathcal{C}_{i}}$ (%)
---|---|---|---
computer | 100 | 4880 | 2.05
politics | 56 | 1850 | 3.03
recreation | 924 | 3976 | 23.24
religion | 150 | 1976 | 8.35
science | 100 | 3951 | 2.53
(b) Ratio of the initial weak supervision in 20News

Table 4: Ratio of the initial weak supervision Figure 5: Confusion matrix over
all fine prototypes without (left) and with (right) the gloss knowledge.