11institutetext: Department of Computer Science, City University of Hong Kong,
Hong Kong 11email: tnguyenhu2-c<EMAIL_ADDRESS>
22institutetext: School of Computing and Information Systems, Singapore
Management University, Singapore 22email:
yu.cao.2022<EMAIL_ADDRESS>
# Incremental Learning
on Food Instance Segmentation

Huu-Thanh Nguyen 11 Yu Cao 22 Chong-Wah Ngo 22 Wing-Kwong Chan 11

###### Abstract

Food instance segmentation is essential to estimate the serving size of dishes
in a food image. The recent cutting-edge techniques for instance segmentation
are deep learning networks with impressive segmentation quality and fast
computation. Nonetheless, they are hungry for data and expensive for
annotation. This paper proposes an incremental learning framework to optimize
the model performance given a limited data labelling budget. The power of the
framework is a novel difficulty assessment model, which forecasts how
challenging an unlabelled sample is to the latest trained instance
segmentation model. The data collection procedure is divided into several
stages, each in which a new sample package is collected. The framework
allocates the labelling budget to the most difficult samples. The unlabelled
samples that meet a certain qualification from the assessment model are used
to generate pseudo-labels. Eventually, the manual labels and pseudo-labels are
sent to the training data to improve the instance segmentation model. On four
large-scale food datasets, our proposed framework outperforms current
incremental learning benchmarks and achieves competitive performance with the
model trained on fully annotated samples.

###### Keywords:

Food Computing Incremental learning Instance segmentation Semi-supervisor
learning Membership inference.

##  1 Introduction

Instance segmentation is a fundamental task in computer vision with several
applications, including food portion size estimation [1, 2, 3], text
localization [4, 5], vehicle surveillance [6, 7]. With the development of deep
learning, several neural networks like Mask R-CNN [8], CenterMask [9],
Watershed [6], Terrace [2], have been developed to address the problem.
Nonetheless, these techniques prioritize segmentation quality and computation
efficiency over the annotation effort. This paper aims to fill this gap with
the proposal of an incremental learning framework that maximizes the
competence of the labour force by automatically selecting the most difficult
samples for annotation and high-quality pseudo-labels from the remaining
samples for model improvement.

Figure 1: Inferences of the latest instance segmentation model are made on new
collected samples. For each sample, the model outputs a clustering and an
instance map. The PQ [10] score is calculated from the generated and the
actual instance maps.

Over the past few years, food computing has received great attention with
several research papers on food recognition [11], detection [12] and
segmentation [2]. Among these tasks, food instance segmentation is essential
for portion size estimation [13]. However, preparing ground-truth instance
segmentation masks for model training is time-consuming. To address the
annotation problem, most existing strategies primarily reduce the amount of
supervised information, ranging from image-level [14], instance-level [15],
bounding box [12], and polygon [2, 3], introducing a trade-off between
segmentation performance and annotation effort. Instead of an instance mask,
the image-level and instance-level may either generate a peak response map or
a blob that provides the instance locations in the input image. Bounding box
information is better for estimating instance size but fails to predict the
instance shape. Polygon supervision exhibits the most appropriate technique
that annotates critical corner points on the instances to secure segmentation
quality. However, for a complex appearance, such as in the food domain, the
large number of critical points leads to expensive polygon annotation. For
such a situation, semi-supervision [16] is a potential solution where some
random samples are annotated, and the remaining ones are prepared with model-
generated instance masks. A risky aspect of this approach is that the quality
of the generated masks is not guaranteed for model improvement.

This paper presents an incremental learning framework for instance
segmentation, with the novelty being the proposal of an assessment module for
automatically scoring the difficulty of newly collected samples. In practice,
data samples are collected gradually in package units for research and
commercial purposes. Our framework annotates and trains an initial instance
segmentation model on the first package. In parallel, we use this package to
train a difficulty assessment model that evaluates quantitatively how much
each new sample is challenging to the latest instance segmentation model.
Usually, a new package comprises easy, neutral, and hard samples, as
illustrated in Fig. 1. The ready-to-use assessment model is employed to
forecast their difficulty levels. With inspiration from hard-mining [17] and
self-paced learning [18], we then direct hard samples to the labour force for
polygon annotation and the easy samples to generate pseudo-labels. The neutral
and easy samples will then be preserved in pooling data for consideration in
the next stage. The manual labels and high-quality pseudo-labels are used to
fine-tune the instance segmentation model. We repeat this procedure when
either a new sample package is collected or the labour force is available for
the annotation work.

##  2 Related Work

Instance segmentation Current mainstream deep learning techniques for instance
segmentation can be classified into two major groups: proposal-based [8, 9]
and clustering-based [2, 6]. Proposal-based techniques, such as Mask R-CNN [8]
and Center Mask [9], generate a set of candidate bounding boxes for instance
detection, and following each box is a prediction for the corresponding
instance mask. Clustering-based generates one or many instance feature maps,
such as multiple terrace layers [2], watershed energy and centre-direction
maps [6], that can construct an instance segmentation map. This paper exploits
Terrace [2], a clustering-based instance segmentation technique which exhibits
impressive performance in the food domain, for our incremental learning
framework. The use of the Terrace technique is two-fold. First, given a new
sample, the output is a terrace probability distribution, representing how
certain the model is about its prediction. This paper investigates the
generated probability distribution to forecast the difficulty level of a new
image sample. Second, we employ Terrace [2] as the core instance segmentation
model to develop and evaluate the proposed incremental learning framework.

Semi-supervision instance segmentation To improve the instance segmentation
performance with a limited human resource budget for image annotation, [16]
proposes a semi-supervised learning mechanism. There are two instance
segmentation models proposed to work in sequence: an annotation model and a
production model. When a new package of samples is collected, a small number
of samples will be randomly selected and manually labelled. The annotation
model is trained on samples with manual labels and then used to generate
pseudo-labels for the remaining unlabelled samples. The production model is
trained on both manual and pseudo-labels. [16] argues that the extra knowledge
from pseudo-labels improves the segmentation performance at no additional
cost. However, the generated pseudo-labels are not properly qualified. While a
good pseudo-label may provide more context to support the model in
distinguishing instance versus background pixels to be aware of instance
boundaries, a bad pseudo-label may give the wrong information and confuse the
production model. In our framework, we propose a difficulty assessment
technique to forecast the quality of generated pseudo-labels. Then, only
samples with good pseudo-labels are used to improve the production model,
while the remaining samples can be either skipped or annotated depending on
labour force availability.

Membership inference attack Given a deep learning model with open access, [19]
proposes a technique to predict whether a query sample is inside or outside
the training set. An incremental learning framework may employ this technique
to filter out the used samples and spend the annotation budget on new samples.
This paper proposes a more advanced method, which may forecast the panoptic
quality (PQ) [10] score of every new sample (without ground-truth instance
segmentation mask), reflecting how much the sample is challenging to the
latest instance segmentation model.

##  3 Incremental Learning Framework

Fig. 2 illustrates the proposed incremental learning on instance segmentation
framework, including an incremental learning engine interconnecting the states
of collected samples, supervised labels, and instance segmentation models
between successive stages. In the scope of this paper, we employ the
clustering-based technique Terrace [2] for instance segmentation. At a
specific stage, our system stores unlabelled samples in a data pool, a set of
supervised samples, an annotation model, and a production model. The
incremental learning engine receives unlabelled samples, forecasts their
challenging scores, and navigates them to the manual labelling module, pseudo-
labelling module, or holds them in the data pool for future use.

Figure 2: Incremental learning framework continuously updates newly collected
data and improves the instance segmentation model.

###  3.1 Data Pool, Labels and Instance Segmentation Model

In both academia and industry, the data collection process is either automatic
by crawler tools or manual by humans. As the required data for machine
learning is usually large-scale, it takes a timely basis, such as daily,
weekly, or monthly, for data collection and annotation. This paper defines
such a timely basis as a “stage” and presents the incremental learning
framework between consecutive stages. At stage $k$, the following states are
recorded in the learning system::

  * Data pool is to store the set of samples where the annotation for instance segmentation masks has not been prepared yet. As the task is time-consuming, there is always a long queue of unlabelled samples. The pool holds the samples unlabelled in the previous stage for use in the current stage.

  * Supervised labels is to hold the set of pairs (sample, instance segmentation label). In the proposed framework, we annotate most of the samples collected in the first stage. In the subsequent stages, we eliminate the labelling cost by only selecting a small percentage of samples to annotate and navigating them to the supervised labels. In other words, the expense of the labour force is only expensive for the first stage and can be flexibly adjusted in the subsequent stages.

  * Annotation model, inspired from [16], is an instance segmentation model trained on supervised samples. At the beginning of each stage, the annotation model makes inferences on all pooling samples to generate clustering maps and pseudo-labels. At the end of the stage, the model is improved when the set of supervised samples is updated.

  * Production model shares the same architecture as the annotation model, but it is trained on supervised labels and pseudo-labels. As pseudo-labels are generated from a large-scale data pool, they contain a diverse food context. This diversity supports the production model to understand the data population better. Unlike [16], we do not include all the pseudo-labels to improve the production model. Instead, we pick high-quality pseudo-labels, which are automatically recommended by the difficulty assessment module.

When a new package $k+1$ is collected, we merge it with the current pooling
data $k$ and feed it into the incremental learning engine.

###  3.2 Incremental Learning Engine

The novelty of this paper is the proposal of an incremental learning engine
that forecasts the difficulty level of collected samples and distributes them
to manual annotation, pseudo-label generation, and data pool. In particular,
the samples collected in package $k$ and current samples in the data pool $k$
are merged. Here, each sample is subjected to the latest annotation model,
generating a set of clustering maps. Each clustering map compromises a pre-
defined number of terrace layers. The probability distribution on terrace
layers reveals how the annotation model is confident with its prediction of
the sample. Therefore, we explore the outputted terrace layers to train a
difficulty assessment model as follows:

Figure 3: Architecture of the difficulty assessment model. Figure 4: Pipeline
to generate training clustering maps and ground-truth PQ scores for the
difficulty assessment model.

  * Difficulty assessment model The proposed difficulty assessment model, as shown in Fig. 3, is a convolutional neural network composed of convolutional and fully connected layers. This paper employs ResNet-50 [20], pre-trained on ImageNet [21], in experiments. The model’s input is a stack of clustering layers representing the predicted terrace probability distributions by the latest annotation model. The model’s output is a regression value within the range $[0,1]$, describing how the model forecasts the PQ score of a new sample made on the annotation model. A low score means a difficult sample, and a high score means an easy sample. The loss function for difficulty assessment model training is formulated as follows:

$L=\frac{\sum^{N}_{n=1}(s_{n}-\hat{s}_{n})^{2}}{N}$ (1)

where $s_{n}$ and $\hat{s}_{n}$ are ground-truth and predicted PQ scores for
$n^{th}$ sample, and $N$ is the total number of training samples.

  * Generating training samples Inspired by the membership attack framework [19], we generate training samples for the difficulty assessment model via shadow models, as illustrated in Fig. 4. First, a sample package is collected and manually annotated. In transfer learning, an alternative solution is to use annotated samples from the ready-to-use datasets. Second, the labelled samples are distributed into shadow pairs of training and testing sets. Third, each shadow training set is used to train a shadow Terrace model for instance segmentation, which shares the same hyper-parameters (e.g., number of terrace layers) as the production model. It is noted that all these samples, both for training and testing, are prepared with ground-truth instance segmentation masks. Therefore, the generated instance maps from a shadow model can be measured with PQ scores. A predicted clustering map and the corresponding PQ score form a pair (clustering map, PQ score). We repeat the second and the third steps to build up large-scale (clustering map, PQ score) samples to train and test the difficulty assessment model. We iterated the process 80 times in the experiment, resulting in over 35,000 pairs of (clustering map, PQ score). We select roughly 9,100 pairs in a uniform distribution for the PQ score for the assessment model training and evaluation.

To this end, the proposed incremental learning framework automatically
forecasts the challenging level of a new sample by feeding it into the latest
annotation model and the difficulty assessment model. Depending on the budget
available, a percentage of the most challenging samples is selected for manual
labelling. The remaining samples are navigated into neutral and easy sets,
where the easy samples are required to meet a certain threshold of difficulty
score. The generated instance maps of easy samples on the latest annotation
model are used as pseudo-labels. For the next production model, low-weighted
pseudo-labels and high-weighted supervised labels are sent to the training
set. In particular, we weigh the samples in the first package, the hard and
the easy samples in a ratio of 2:4:1 to better mine the challenging cases. For
the annotation model, only supervised labels are used. Finally, neutral and
easy samples are put into the data pool for use in the next stage.

##  4 Experiments

###  4.1 Experimental Setup

####  4.1.1 Dataset

Four food datasets, including Dimsum, Sushi, Cookie, and UECFoodPixComp (UEC)
[2, 22], are employed to evaluate the incremental learning performance. While
Sushi is only used as a pre-trained dataset to evaluate various sampling
strategies for transfer learning on the Dimsum dataset, each of the remaining
datasets is split into six packages to evaluate the incremental learning
framework. It is noted that the number of food instances is different among
image samples. To be fair for the instance segmentation labelling budget,
which is instance-based, we organize the packages with a similar number of
food instances: 1,575. The number of samples for evaluation in Dimsum, Cookie,
and UEC is 768, 1152, and 1000, respectively.

####  4.1.2 Evaluation Metric

Regarding instance segmentation performance, we employ Panoptic Quality (PQ)
[10] to evaluate the performance of the production model. For the difficulty
assessment model, which outputs a PQ-based difficulty score, we measure the
absolute error between the predicted and ground-truth PQ scores. To justify
the efficiency of the proposed incremental framework, we evaluate the instance
segmentation performance across various settings of manual labelling efforts,
including 5%, 10%, 20%, 30% and 100% annotation time for each data package.

Table 1: The instance segmentation performance (% PQ) with 5% and 10% training samples labelled by various data sampling strategies on the Dimsum dataset. Transfer learning is made with a model pre-trained on Sushi dataset. Sampling strategy | Trivial learning | Transfer learning
---|---|---
| 5% | 10% | 0% | 5% | 10%
easy | 45.66 | 54.79 | 76.33 | 76.88 | 77.54
random | 58.77 | 62.75 | 76.33 | 79.52 | 80.74
hard | 62.99 | 65.96 | 76.33 | 80.61 | 82.84

###  4.2 Experimental Results

####  4.2.1 Data sampling for instance segmentation

First, we investigate easy, random, and hard sampling strategies for food
instance segmentation. Table 1 lists their performances on the Dimsum dataset.
The hard-mining strategy consistently outperforms the easy and random sampling
in both trivial and transfer learning with a gap of over 1-4% of PQ score when
training on only 5% training samples and 1-3% of PQ score on 10% training
samples. When the annotation budget is limited, it is better to prioritize the
labour resources for the most challenging samples.

Figure 5: The forecast of difficulty level on unlabelled samples. A lower
predicted PQ score means a more difficult sample.

####  4.2.2 Difficult assessment model

We evaluate the assessment model by the mean absolute error on the validation
set presented in Section 3.2. The recorded error is relatively low, at 7.8%.
Therefore, the model is promising in forecasting the difficulty score for an
unlabeled sample. Fig. 5 visualizes some food images and the predicted PQ
scores. On the first three samples with the challenges due to the obscure
boundary, stacking, and occlusion problems, the assessment model gives lower
PQ scores, meaning challenges to the current instance segmentation model.
Meanwhile, higher scores, considered easy samples, are given to the last three
samples, where the boundaries among instances are relatively clear.

Table 2: The trade-off between the quality and quantity of pseudo-labels to incremental learning . threshold for easy samples | 0 | 20 | 40 | 60 | 80
---|---|---|---|---|---
performance | 82.52% | 82.85% | 83.06% | 83.34% | 83.19%

####  4.2.3 Pseudo-labels

Next, we examine how pseudo-labels contribute to incremental learning. Table 2
lists the instance segmentation quality on the Dimsum dataset given different
thresholds to accept pseudo-labels. There is a trade-off between the pseudo-
label quality and the number of easy samples explored. On the one hand, a low-
quality threshold (e.g., at 0% or 20% PQ scores), proposed by [16], lets the
model explore the context from a large number of unlabelled samples. On the
other hand, a high-quality threshold (e.g., at 80% PQ score) guarantees the
model learns from more precise pseudo-labels. The medium PQ threshold, at 60%,
is recorded as a balancing point to retrieve pseudo-labels for incremental
learning.

####  4.2.4 Incremental Learning strategies

We compare the proposed incremental learning strategies, namely PQ-based∗,
against various benchmarks of random and hard sampling techniques on the
Dimsum dataset. Random sampling shuffles the newly collected samples and
navigates 10% of them to the human resource for annotation. The advanced
method, random∗ [16], employs the latest annotation model to generate pseudo-
labels for the remaining 90% samples. Hard sampling techniques forecast the
difficulty scores for every sample in the new package and give a higher
annotation priority to the more difficult samples. Membership-based method
[19] evaluates whether a new sample is similar to one of the existing members
in the current training data. While the output label of this method is either
yes or no to the question, the output confidence score can be used as an
indicator to assess how challenging the samples are. However, as membership
inference is a binary classification problem, the confidence score
distribution is skewed to 0% for non-membership and 100% for membership. Our
techniques, PQ-based and PQ-based ∗, generate difficulty scores in a more
balanced distribution. The PQ-based ∗ method adds pseudo-labels to training
samples. Different from random∗ [16], we can exclude low-quality pseudo
samples thanks to the proposed assessment model.

Table 3 lists the performance of these incremental learning strategies on the
Dimsum dataset, where the annotation budget is used for only 10% of the new
sample package. Overall, PQ-based∗ consistently outperform the remaining
strategies across six stages. The 10% package in hard sampling techniques,
membership-based and PQ-based, contributes to around 1.5% improvement in
panoptic quality compared to random sampling. PQ-based demonstrates a better
assessment approach than membership-based, especially at the early stages. PQ-
based∗, aggregating the advantages of hard sampling and good pseudo-labels,
make improvements of 1.5%-2% compared to the benchmark random∗ [16].

Table 3: The instance segmentation performances (% PQ) of various incremental learning techniques on the Dimsum dataset. For a new data package, only 10% of food instances are selected to prepare ground-truth instance segmentation masks. package | full annotation | random sampling | hard sampling
---|---|---|---
| random | random∗ [16] | membership [19] | PQ-based | PQ-based∗
1 | 74.60 |  |  |  |  |
2 |  | 76.23 | 76.73 | 76.57 | 77.08 | 78.32
3 |  | 77.30 | 78.81 | 78.42 | 79.49 | 80.98
4 |  | 79.39 | 79.87 | 80.57 | 81.24 | 81.99
5 |  | 80.27 | 80.47 | 80.97 | 81.90 | 82.74
6 |  | 80.75 | 81.95 | 82.11 | 82.39 | 83.34

(a) Dimsum

(b) Cookie

(c) UEC

Figure 6: Incremental learning: labelling effort and instance segmentation
performance.

####  4.2.5 Annotation effort and model performance

Last, we compare model performances given by the proposed method, PQ-based∗,
on a little annotation effort and the traditional approach training on manual
labels for all collected samples. Fig. 6a illustrates the panoptic quality
over annotation time on the Dimsum dataset. Given the pre-trained model PQ-
based∗ 10% on Dimsum, we perform transfer learning on Cookie and UEC datasets.
On the Dimsum dataset, at the last package, with only one-fourth of the
annotation effort, PQ-based∗ 10% is competitive with the fully annotated
approach. The approaches PQ-based∗ 20% and PQ-based∗ 30%, with one-third and
one-half of the annotation time, can achieve equivalent performance to the
model with 100% data annotated. Using the pre-trained model of PQ-based∗ 10%
on Dimsum, the transfer learning models of PQ-based∗ 30% on Cookie and UEC
show a minor gap of around 1% PQ less than the model with 100% sample
annotated.

##  5 Conclusion

We have presented an incremental learning framework for food instance
segmentation given a fixed budget for annotation. The framework’s power comes
from the proposed assessment model, which forecasts the difficulty scores for
unlabelled samples. The score enables the framework to select hard samples for
manual labelling and high-confidence samples for pseudo labels. The
experimental results justify the efficiency of the assessment model in scoring
food images regarding how challenging they are to instance segmentation. The
proposed framework outperforms current incremental learning benchmarks on the
Dimsum dataset with the same amount of annotation effort. The framework
exhibits competitive performance with fully annotated models on the Dimsum
dataset and the transfer learning on the Cookie and UEC datasets, with a much
shorter labelling time. The proposed incremental learning strategy is a
promising solution to train and deploy a food instance segmentation model in
practice.

## References

  * [1] Eduardo Aguilar, Beatriz Remeseiro, Marc Bolanos, and Petia Radeva.  Grab, pay, and eat: Semantic food detection for smart restaurants.  In IEEE Transactions on Multimedia, pages 3266–3275, 2018.
  * [2] Huu-Thanh Nguyen and Chong-Wah Ngo.  Terrace-based food counting and segmentation.  In The Association for the Advancement of Artificial Intelligence (AAAI), pages 2364–2372, May 2021.
  * [3] Huu-Thanh Nguyen, Chong-Wah Ngo, and Wing-Kwong Chan.  Sibnet: Food instance counting and segmentation.  In Pattern Recognition, volume 124, page 108470, 2022.
  * [4] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao.  Shape robust text detection with progressive scale expansion network.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9328–9337, Jun 2019.
  * [5] Yixing Zhu and Jun Du.  Textmountain: Accurate scene text detection via instance segmentation.  In Pattern Recognition, page 107336, 2021.
  * [6] Min Bai and Raquel Urtasun.  Deep watershed transform for instance segmentation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2858–2866, Jul 2017.
  * [7] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool.  Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019.
  * [8] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B. Girshick.  Mask R-CNN.  In IEEE International Conference on Computer Vision (ICCV), pages 2980–2988, 2017.
  * [9] Youngwan Lee and Jongyoul Park.  Centermask: Real-time anchor-free instance segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [10] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar.  Panoptic segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019.
  * [11] Jingjing Chen, Lei Pang, and Chong-Wah Ngo.  Cross-modal recipe retrieval: How to cook this dish?  In MultiMedia Modeling, pages 588–600, 2017.
  * [12] Lixi Deng, Jingjing Chen, Qianru Sun, Xiangnan He, Sheng Tang, Zhaoyan Ming, Yongdong Zhang, and Tat Seng Chua.  Mixed-dish recognition with contextual relation networks.  In Proceedings of the 27th ACM International Conference on Multimedia, page 112–120, 2019.
  * [13] Jiabao Lei, Jianing Qiu, Frank P.-W. Lo, and Benny Lo.  Assessing individual dietary intake in food sharing scenarios with food and human pose detection.  In Pattern Recognition. ICPR International Workshops and Challenges, pages 549–557, 2021.
  * [14] Yanzhao Zhou, Yi Zhu, Qixiang Ye, Qiang Qiu, and Jianbin Jiao.  Weakly supervised instance segmentation using class peak response.  In Conference on Computer Vision and Pattern Recognition, pages 3791–3800, 2018.
  * [15] Issam H. Laradji, Negar Rostamzadeh, Pedro O. Pinheiro, David Vazquez, and Mark Schmidt.  Where are the blobs: Counting by localization with point supervision.  In The European Conference on Computer Vision (ECCV), Sep 2018.
  * [16] Miriam Bellver, Amaia Salvador, Jordi Torres, and Xavier Giró i Nieto.  Budget-aware semi-supervised semantic and instance segmentation.  In CVPR Workshops, 2019.
  * [17] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining.  In Conference on Computer Vision and Pattern Recognition, 04 2016\.
  * [18] Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, and Yueting Zhuang.  Self-paced boost learning for classification.  In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’16, page 1932–1938. AAAI Press, 2016.
  * [19] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In IEEE Symposium on Security and Privacy, 05 2017.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: a large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 06 2009.
  * [22] Kaimu Okamoto and Keiji Yanai.  UEC-FoodPIX Complete: A large-scale food image segmentation dataset.  In Proc. of ICPR Workshop on Multimedia Assisted Dietary Management(MADiMa), 2021.


# Efficient Image-Text Retrieval via Keyword-Guided Pre-Screening

Min Cao, Yang Bai, Jingyao Wang, Ziqiang Cao∗, Liqiang Nie, and Min Zhang
Ziqiang Cao is the corresponding author (email: zqcao@suda.edu.cn). Min Cao,
Yang Bai, Jingyao Wang, Ziqiang Cao and Min Zhang are with School of Computer
Science and Technology, Soochow University, Suzhou 215006, China (email:
mcao@suda.edu.cn). Liqiang Nie and Min Zhang are Institute of Computing and
Intelligence, Harbin Institute of Technology (Shenzhen), China.

###### Abstract

Under the flourishing development in performance, current image-text retrieval
methods suffer from $N$-related time complexity, which hinders their
application in practice. Targeting at efficiency improvement, this paper
presents a simple and effective keyword-guided pre-screening framework for the
image-text retrieval. Specifically, we convert the image and text data into
the keywords and perform the keyword matching across modalities to exclude a
large number of irrelevant gallery samples prior to the retrieval network. For
the keyword prediction, we transfer it into a multi-label classification
problem and propose a multi-task learning scheme by appending the multi-label
classifiers to the image-text retrieval network to achieve a lightweight and
high-performance keyword prediction. For the keyword matching, we introduce
the inverted index in the search engine and create a win-win situation on both
time and space complexities for the pre-screening. Extensive experiments on
two widely-used datasets, _i.e._ , Flickr30K and MS-COCO, verify the
effectiveness of the proposed framework. The proposed framework equipped with
only two embedding layers achieves $O(1)$ querying time complexity, while
improving the retrieval efficiency and keeping its performance, when applied
prior to the common image-text retrieval methods. Our code will be released.

###### Index Terms:

Image-Text Retrieval, Efficient Search, Multi-Label Classification, Inverted
Index.

##  I Introduction

Recent years have witnessed that cross-modal image-text retrieval has been
gradually becoming one of the mainstream research topics in the fields of
multimedia computing and information retrieval [1, 2, 3, 4]. It aims to
retrieve the gallery samples in one modality from a large-scale repository
with a given query sample in another, whereby the cross-modal alignment is
well-established to estimate the pairwise similarity between the query and
each gallery sample. Specifically, taking a text as the query to retrieve its
corresponding images is called text-to-image retrieval, and vice versa. Image-
text retrieval is challenging due to the heterogeneity and semantic gap
between these two modalities.

Figure 1: Illustration of various image-text retrieval methods. Rank-1 denotes
the expectation of correct match at the 1-th in the ranking list; Time
represents the online running time (in minutes); Acc is the abbreviation for
acceleration; Para denotes the number of modal parameters (in millions). These
results are computed based on the early fusion network ViLT [5] and the late
fusion one ALBEF0 [2] on MS-COCO. In the proposed framework, we adopt the
classification by a multi-task learning on the ALBEF. It is worth noting that
the running time varies across platforms and here we lay stress on the
comparison among the running times.

Broadly speaking, the studies on image-text retrieval are in two variants:
late and early fusion. In particular, the former [6, 7, 8, 3, 9, 10]
emphasizes the image and text feature encodings separately, and then utilizes
a simple inner product between the image and text features to estimate the
similarities. Its antithesis is the early fusion methods [11, 5, 12, 13],
paying more attention to designing complex interaction modules for deeply
fusing the image and text features. Thanks to the deep fusion of cross-modal
information, the early fusion methods have been the leading paradigm in
boosting the search performance. It, nevertheless, suffers from a huge gap
between the laboratory experiments and the real-world applications since we
need to online compute the feature representations in the interaction module
quadratically by traversing through all image-text pairs, leading to the
prohibitive reference cost. Taking the text-to-image retrieval task as an
example, we show a simple flowchart of the early fusion method with its online
running time in minutes on the MS-COCO dataset [14] in Fig. 1 (a). To address
the low efficiency problem, some researchers [15, 16, 2, 17] propose to first
screen the gallery samples by a fast late fusion technique and then retrieve
the remaining samples by a slow early fusion technique (Late2Early for short),
thus yielding a win-win on accuracy and efficiency, as shown in Fig. 1 (b).
However, an online search operation with $O(N)$ time complexity for the
gallery screening is needed in the Late2Early method, still restricting its
application in reality where there are usually millions of galleries with a
tremendous number $N$.

Figure 2: An exposition of necessity of pre-screening prior to image-text
retrieval. As an example of the text-to-image retrieval task, the images with
the dotted arrow are irrelevant to the textual query at the semantic level and
are pre-screened out in advance.

In this paper, we present a novel keyword-guided pre-screening framework to
solve the low efficiency problem of image-text retrieval. We argue that a
large number of gallery samples semantically irrelevant to the query can be
screened out prior to the image-text retrieval algorithm, thus speeding up the
following retrieval computation. Taking the text-to-image retrieval as an
example, as shown in Fig. 2, given a textual query about the dog, it is
unnecessary to compute the similarities with images of the boy or men, and we
can pre-screen such image galleries. Towards this end, we predict the keywords
of the texts and images and compare them to enable a coarse-grained gallery
pre-screening. To accomplish this, we transfer the keyword prediction into a
multi-label classification problem and develop multi-label classifiers to
predict the keywords. Correspondingly, two critical issues should be
considered. Firstly, since the obtained keywords can be viewed as a semantic
expression of the sample on the discrete space and used for screening, the
quality of keywords largely impacts the screening accuracy and the subsequent
retrieval result. Therefore, the accuracy of keyword prediction is a critical
issue. Secondly, the purpose of the pre-screening process is to improve the
retrieval efficiency, yet the process itself can bring extra computational
overhead, offsetting the efficiency to some extent. Therefore, the efficiency
of the pre-screening process is another critical issue. Regarding to the first
issue, we aim to improve the classifiers’ quality by employing the booming
image-text retrieval technique. Specifically, we propose a multi-task learning
scheme by appending the multi-label classifiers to the image-text retrieval
network. As a result, only two embedding layers for the multi-label
classification are introduced in the proposed pre-screening framework.
Combined with the inverted index in the search engine [18], the lightweight
pre-screening is fulfilled only with $O(1)$ querying time complexity and hence
the second issue is solved accordingly. The proposed pre-screening framework
can be applied prior to the common image-text retrieval methods to improve
retrieval efficiency. Beyond that, the keyword prediction, which is considered
as a multi-label classification task, can be readily achieved by other state-
of-the-art classification techniques, thereby further improving the proposed
framework. Fig. 1 (c) shows the simple flowchart of the proposed framework
applied prior to the Late2Early method. We achieve efficiency improvement
($\sim 4.1\times$ speed up) with little extra cost ($\sim+1.8$ million modal
parameters, $\sim+2.5$e-6 minutes of online running), even while enhancing the
accuracy performance ($+0.1\%$ Rank-1).

The main contributions of this paper are four folds. (1) We focus on the
relatively undervalued low efficiency problem of image-text retrieval and
propose a simple yet effective keyword-guided pre-screening framework for
improving retrieval efficiency correspondingly. (2) We convert the keyword
prediction into a multi-label classification task and further propose a multi-
task learning scheme for lightweight and high-performance keyword prediction.
(3) We incorporate the inverted index in the search engine into the keyword
matching for improving pre-screening efficiency. (4) The proposed framework
has robust compatibility. The experimental results on two public benchmarks
show that it can be readily applied to almost all image-text retrieval methods
to boost efficiency with only a minimal cost.

##  II Related Work

We categorize the existing cross-modal image-text retrieval methods into three
lines: late fusion, early fusion and efficiency-focused methods, and we brief
them as follows.

Late fusion methods usually focus on developing various image and text
processing techniques to extract feature representations of each modality
separately, after which the two modalities interact with each other only with
a loss function for training [19, 20, 21, 22, 23, 3]. For instance, Zheng _et
al._ [19] constructed an end-to-end dual-path convolutional neural network to
learn the image and text global representations and proposed an instance loss
for better weight initialization; Park _et al._ [21] explored the important
components in the image and text independently by a multi-head self-attention
network for local correspondence; Radford _et al._ [3] recently proposed a
contrastive dual-path pre-training network trained on a large-scale dataset of
400 million image-text pairs and achieved satisfactory performance on various
downstream tasks, including the image-text retrieval task. The late fusion
methods yield the pairwise similarities by a simple inner product interaction
during inference, promising retrieval efficiency. Nevertheless, due to the
lack of information fusion across modalities when learning the feature
representations, the performance of this kind of method is generally limited.

Early fusion methods pay more attention to information fusion between the
visual and textual domains, and the image and text data are pre-processed
separately [24, 25, 11, 26, 27, 13, 12, 28, 29]. Researchers concentrated on
the local correspondence across modalities in the early fusion methods for
maximizing performance. For example, the cross-attention mechanism [30] is
used for the message interaction between the regions of the image and the
words of the text [24, 25]; the graph neural network [31] is adopted to
explore higher-order relational information between locals across modalities
[11, 26, 27]. In recent years, with the success of Transformer architecture
[30], the large-scale pre-training methods [12, 13] have gradually become a
central paradigm in vision-and-language (V+L) research and achieved many
state-of-the-art results on V+L tasks, including the image-text retrieval
task. The cross-modal information is usually fused early and freely by a
single-stream transformer-like network in most pre-training methods,
classified as the early fusion methods. The information fusion across
modalities benefits the semantic alignment between modalities, thus the early
fusion method has an advantage on performance compared with the late fusion
method. However, we need to quadratically compute the feature representation
online in the early fusion method. It is, therefore, not practical to apply it
in real-world scenarios.

Figure 3: A summary of the proposed framework with taking the text-to-image
retrieval as an example. The ‘Image-Text Retrieval Network’ module can be
replaced with the common image-text retrieval method.

Efficiency-focused methods improve the image-text retrieval efficiency mainly
from two perspectives. (1) Researchers optimized the model architecture
smaller and lighter. Yang _et al._ [32, 33, 34] proposed to learn the hash
codes of the feature representations with the help of the cross-modal hashing
technique for saving memory and improving efficiency; Gan _et al._ [35]
noticed that the over-parameterization issue results in the large memory and
computation in the cross-modal pre-training models for the image-text
retrieval, and proposed to search their sparse subnetwork with the aid of the
lottery ticket hypothesis [36]; similarly, Wang _et al._ [37] borrowed the
MiniLM [38] structure to reduce the computation cost of the transformer module
in the cross-modal pre-training models. And (2) some studies make efforts to
yield a powerful late fusion model for narrowing down to a list of relevant
galleries, within which the early fusion model is used to achieve the
retrieval. For instance, Miech _et al._ [15] distilled the knowledge of the
early fusion model into the late fusion model, and adopted the distilled late
fusion model for the preliminary screening of the gallery samples; Geigle _et
al._ [39] proposed to jointly train the early fusion model and the late fusion
model with shared parameters for obtaining the high-quality late fusion model;
Li _et al._ [2] introduced a contrastive loss on the feature representations
from the image and text processing branches, and the similarity between the
representations is used to select the relevant galleries for the following
interaction module. Unfortunately, the first group even with lightweight
architecture still needs a long reference time due to the quadratic
executions, and the second group has to perform an online search operation
with $O(N)$ time complexity. Towards this end, there is further room for
improving retrieval efficiency. Unlike the above two groups, we proposed a
general low-cost pre-screening framework, which can be employed prior to them
to further accelerate the retrieval.

##  III Methodology

For an image-text retrieval system, given a query sample in one modality, we
need to retrieve a rank list from $N$ gallery samples in another based on the
similarity with the query sample. The current state-of-the-art retrieval
methods suffer from $N$-related complexity and come at prohibitive
computational costs in reality. Regarding this problem, we propose a keyword-
guided pre-screening framework with minor computational overhead to narrow
down to a list of $N_{r}$ relevant galleries ($N_{r}\ll N$) prior to the
common image-text retrieval methods. The proposed framework involves two
modules: keyword prediction and pre-screening. At first, we present the multi-
label classification technique for keyword prediction in visual and textual
modalities, respectively, referring to Section III-A. Thereafter, we introduce
a keyword matching strategy based on the inverted index for the gallery
screening in Section III-B.

The block diagram of the proposed framework is illustrated in Fig. 3. The
proposed framework can be embedded into almost all common image-text retrieval
networks. Combining the proposed coarse-grained pre-screening and the state-
of-the-art fine-grained retrieval brings a superior tradeoff between accuracy
and efficiency.

###  III-A Keyword Prediction

Visual Modality. We train a multi-label image classifier for predicting the
image’s keywords. Any common multi-label classification techniques [40, 41]
can be adopted. However, we can not directly adopt the classifier trained on
the multi-label classification benchmarks, such as COCO-80 and VG-500 [41]. In
doing so, the image keywords obtained from the class annotations in these
benchmarks may be misaligned with the actual semantic content in the image due
to the data gap between the image-text retrieval and the multi-label
classification, thereby causing the imprecise keyword-guided screening. To
this end, we train the classifier based on the image-text retrieval
benchmarks. Specifically, the image’s ground-truth annotations are generated
from the paired texts. Referring to the setting in the classification task
where the class annotations are generally nouns, we use the nouns in the
paired texts as the image annotations. Alternatively, we also experimentally
use the nouns, verbs and adjectives from the paired texts as the ground-truth
annotations. Better performance results are obtained by only using the nouns
as the annotations.

Figure 4: Illustration of the proposed advanced classifiers appended in the
ALBEF [2] for keyword prediction.

Beyond adopting the existing common classification technique for image keyword
prediction, we further propose an advanced classification to enhance the
classifier’s performance and the follow-up screening accuracy and retrieval
result. The multi-label image classification task can essentially be viewed as
an image-to-label retrieval task, sharing certain characteristics with the
image-text retrieval task. We believe that the booming image-text retrieval
technique can positively contribute to the multi-label image classification on
performance. Thereby, we propose a multi-task learning scheme by appending the
multi-label classifier to the image-text retrieval network. Specifically,
after the image processing branch in the retrieval network, we add an extra
label embedding layer with a multi-label classification loss, thus enabling
the multi-task learning of the image-text retrieval and the multi-label
classification. As an accessory, the advanced classification introduces
minimal computational overhead into the proposed pre-screening framework
compared to the common classification.

For the classification loss, rather than adopting a frequently-used binary
cross-entropy loss [41], we adopt a state-of-the-art asymmetric loss (ASL)
[40]. Compared to the binary cross-entropy loss, the ASL loss operates
dynamically on positive and negative samples during training and considers the
positive-negative imbalance problem in the classification task. Specifically,

$L=\sum_{i=1}^{l}-y_{i}L_{+}-\left(1-y_{i}\right)L_{-},$ (1)

where $y_{i}=1$ represents that the $i$-th class annotation is the ground-
truth of the image $x$ and vice versa, and

$\begin{cases}L_{+}=\left(1-p_{i}\right)^{\alpha_{+}}\log\left(p_{i}\right)\\\
L_{-}=\left(\tilde{p}_{i}\right)^{\alpha_{-}}\log\left(1-\tilde{p}_{i}\right),\end{cases}$
(2)

where $\alpha_{+}$ and $\alpha_{-}$ are the positive and negative focusing
parameters, respectively. With a high $\alpha_{+}$ (resp. $\alpha_{-}$), the
contribution from easy positives with $p_{i}\gg 0.5$ (resp. easy negatives
with $p_{i}\ll 0.5$) to the loss is weakened, leading to more focus on more
challenging samples in training.
$\tilde{p}_{i}=max\left(p{{}_{i}}-\Delta,0\right)$ is a shifted label
probability, and the negative sample will be discarded when
$\tilde{p}_{i}<\Delta$.

Figure 5: Illustration of the pre-screening with the inverted index. We take
text-to-image retrieval as an example.

Textual Modality. Intuitively, we can extract the words directly from the text
by using the natural language toolkit [42] as its keywords. As a result, the
inference text’s keywords are derived from the inference data, and yet the
image’s keywords are still from the class annotations of the training data.
The gap between the training and inference data tends to bring a possible
situation with no overlap between the keywords of text and image in inference,
thus leading to the failure of pre-screening. Alternatively, we can introduce
a multi-label classification into the keyword prediction in textual modality,
as in viual one. The text’s ground-truth annotations for training are
equivalent to the ones of the paired images, ensuring the overlap of keywords
across modalities in inference and avoiding the pre-screening’s failure. Yet
at the same time, the classifier is likely to predict the wrong keywords for
the text. From the above analysis, merging the predicted labels and extracted
nouns to the keywords is a complementary solution. We experimentally study the
performance of the above models for generating text keywords. The multi-label
classification results in better retrieval performance and is used in the
proposed framework. Beyond using the common classification technique, we
append a multi-label classifier after the text processing branch in the
retrieval network for the advanced classifier.

We take an image-text retrieval method ALBEF [2] as an example for the two
advanced classifiers, as illustrated in Fig. 4. The label embedding layer
together with the ASL are appended to the end of the image encoder and text
one, respectively. In inference, the class annotations with top-$R_{I}$ and
top-$R_{T}$ highest probabilities are used as the image and text keywords,
respectively.

###  III-B Keyword Matching for Pre-screening

The sample information is abstracted as the keywords expressed in discrete
form and used as guidance for gallery pre-screening prior to the image-text
retrieval network. Intuitively, we can compare the keywords of the query and
each gallery sample and screen out the galleries without overlap with the
query regarding the keywords. However, it has a high computational complexity
in reality due to the traversal operator in the gallery set with a tremendous
number of $N$. For this, we achieve the high-efficient pre-screening with the
inverted index technique, inspired by the search engine [18]. In particular,
the pre-screening involves an index step that builds the mapping between the
keywords and the gallery samples, a search step that picks out the gallery
samples sharing the same keywords with the query sample and discards the rest
of the gallery samples.

Index. After the keyword prediction, we have obtained a mapping from the
gallery to the keyword. It implies that a naive forward database index has
been implicitly built where the gallery sample ID is specified as the key and
its keywords as the value, as shown in Fig. 5. A time-consuming pre-screening
is required based on the database. Consequently, we construct an inverted
database index in which the keyword is specified as the key and its paired
gallery samples as the value, resulting in a fast pre-screening as follows.

Search. As shown in Fig. 5, with the inverted index created, the pre-screening
can be resolved by two steps: a query step with quickly jumping to the keys
having the same as the query’s keywords, and a mergence step with merging the
associated values to as the retaining gallery samples.

##  IV Complexity Analysis and Discussion

While achieving efficiency improvement, the proposed framework itself can
bring extra resource overhead. We analyze its time and space complexities and
compare it with the Late2Early method, which is closely related to the
proposed framework.

The online computation is composed of query processing and gallery screening.
In the proposed framework, there is no extra time to spend on the query
processing (i.e., query keyword prediction) due to the proposed multi-task
learning scheme; during the gallery screening, with an $O(1)$ querying time
complexity, the next merging step with far less than $O(N)$ complexity
dominates the time complexity of the screening. In the Late2Early method,
given a real-time query, its feature is extracted with high complexity and
$k$-nearest neighbors from the gallery samples are selected with $O(N)$ time
complexity.

For the space complexity, we only need to add two label embedding layers and
store the inverted index only including ID numbers, giving a clear advantage
over the Late2Early method, usually with two heavy visual and textual encoding
networks and storing the gallery samples’ features for the gallery screening.

In addition, the proposed framework can be applied prior to almost all image-
text retrieval methods to improve retrieval efficiency. When applied prior to
the Late2Early method, the proposed pre-screening selects $N_{r}$ gallery
samples ($N_{r}\ll N$) to the follow-up Late2Early method, bringing $O(N_{r})$
time complexity instead of the original $O(N)$ in the Late2Early method. When
applied prior to the other method, the proposed pre-screening plays a similar
role as the late fusion network in the Late2Early method, yet offers a
significant advantage over the resource overhead compared with the Late2Early
methods.

##  V Experiments

###  V-A Experiment Settings

Datasets. We conduct experiments on two widely-used image-text retrieval
benchmark datasets: Flickr30K [43] and MS-COCO [14], containing $31,014$ and
$123,287$ images, respectively. Both datasets have five associated textual
descriptions per image. Following the common settings [8, 5], we split
Flickr30K into $29,000$ images for training, $1,014$ for validation and
$1,000$ for inference; we use $113,287$ images for training, $5,000$ for
validation and $5,000$ for inference in MS-COCO. We report the results for
both image-to-text retrieval (TR) and text-to-image retrieval (IR) in the
experiments.

TABLE I: The results of the proposed framework applied prior to various image-
text retrieval methods. The symbols ‘+’ and ‘++’ in the Time(m) and Para(M)
columns refer to the online running time and the number of modal parameters
resulted from the method in the corresponding row. The symbol ‘$\times$’ in
the Speedup column refers to the retrieval acceleration of the method in the
previous row and the ‘$\times\times$’ refers to the screening acceleration of
the method in the previous row. For clarity, we present the R@(sum) variation
of the proposed framework compared to the method in the top row above ‘+Ours’
or ‘++Ours’.

Genre | Method | Flickr30K | MS-COCO |
---|---|---|---|---
TR | IR | Time(m) | Speedup | TR | IR | Time(m) | Speedup | Para(M)
R@1 | R@5 | R@sum | R@1 | R@5 | R@sum | R@1 | R@5 | R@sum | R@1 | R@5 | R@sum
Early fusion | ALBEFall | 95.8 | 99.7 | 195.5 | 84.3 | 95.9 | 180.2 | 2.2e+3 |  - | 77.6 | 94.3 | 171.9 | 60.6 | 84.2 | 144.8 | 5.5e+4 |  - | 56.7
Late2Early | +ALBEF0 | 95.9 | 99.8 | 195.7 | 85.5 | 97.5 | 183.0 | +1.6e+1 | 23.4$\times$ | 77.6 | 94.3 | 171.9 | 60.7 | 84.3 | 145.0 | +7.8e+1 | 117.2$\times$ | +152.9
| ++Ours | 95.9 | 99.8 | 195.7 ($\rightarrow$) | 85.2 | 97.0 | 182.2 ($\downarrow 0.8$) | ++4.2e-3 | 2.1$\times\times$ | 77.6 | 94.1 | 171.7 ($\downarrow 0.2$) | 60.5 | 84.0 | 144.5 ($\downarrow 0.5$) | ++6.2e-2 | 4.1$\times\times$ | ++1.3
Early fusion | ViLT-B/32a | 83.7 | 96.8 | 180.5 | 64.3 | 88.8 | 153.1 | 5.9e+3 |  - | 61.6 | 86.4 | 148.0 | 42.8 | 72.9 | 115.7 | 1.5e+5 |  - | 111.6
Late2Early | +ALBEF0 | 85.6 | 97.2 | 182.8 | 66.7 | 90.1 | 156.8 | +1.6e+1 | 23.4$\times$ | 65.6 | 89.5 | 155.1 | 46.0 | 75.8 | 121.8 | +7.8e+1 | 117.2$\times$ | +152.9
| ++Ours | 86.4 | 98.4 | 184.8 ($\uparrow 2.0$) | 66.2 | 90.2 | 156.4 ($\downarrow 0.4$) | ++4.2e-3 | 2.1$\times\times$ | 65.6 | 89.4 | 155.0 ($\downarrow 0.1$) | 46.1 | 75.3 | 121.4 ($\downarrow 0.4$) | ++6.2e-2 | 4.1$\times\times$ | ++1.3
Early fusion | ViLT-B/32a | 83.7 | 96.8 | 180.5 | 64.3 | 88.8 | 153.1 | 5.9e+3 |  - | 61.6 | 86.4 | 148.0 | 42.8 | 72.9 | 115.7 | 1.5e+5 |  - | 111.6
Late2Early | +LightDOT | 85.5 | 97.8 | 183.3 | 66.2 | 90.2 | 156.4 | +1.4 | 23.4$\times$ | 65.3 | 88.7 | 154.0 | 45.9 | 75.6 | 121.5 | +6.9 | 117.2$\times$ | +222.9
| ++Ours | 85.7 | 97.9 | 183.6 ($\uparrow 0.3$) | 65.6 | 90.5 | 156.1 ($\downarrow 0.3$) | ++4.2e-3 | 2.1$\times\times$ | 65.3 | 88.5 | 153.8 ($\downarrow 0.2$) | 45.6 | 74.8 | 120.4 ($\downarrow 1.1$) | ++6.2e-2 | 4.1$\times\times$ | ++1.3
Late fusion | LightDOT | 83.9 | 97.2 | 181.1 | 69.9 | 91.1 | 161.0 | 1.4 |  - | 60.0 | 85.2 | 145.2 | 45.8 | 74.5 | 120.3 | 6.9 |  - | 222.9
| +Ours | 84.9 | 97.9 | 182.8 ($\uparrow 1.7$) | 69.3 | 90.6 | 159.9 ($\downarrow 1.1$) | +4.2e-3 | 2.1$\times$ | 60.4 | 85.3 | 145.7 ($\uparrow 0.5$) | 45.9 | 74.7 | 120.6 ($\uparrow 0.3$) | +6.2e-2 | 4.1$\times$ | +1.3
Late fusion | CLIP | 84.9 | 97.2 | 182.1 | 64.8 | 87.2 | 152.0 | 2.7 |  - | 56.1 | 79.0 | 135.1 | 36.1 | 60.9 | 97.0 | 13.2 |  - | 427.6
| +Ours | 84.9 | 97.3 | 182.2 ($\uparrow 0.1$) | 65.0 | 87.2 | 152.2 ($\uparrow 0.2$) | +4.2e-3 | 2.1$\times$ | 56.1 | 79.0 | 135.1 ($\rightarrow$) | 36.2 | 61.1 | 97.3 ($\uparrow 0.3$) | +6.2e-2 | 4.1$\times$ | +1.3
Early fusion | ViLT-B/32a | 83.7 | 96.8 | 180.5 | 64.3 | 88.8 | 153.1 | 5.9e+3 |  - | 61.6 | 86.4 | 148.0 | 42.8 | 72.9 | 115.7 | 1.5e+5 |  - | 111.6
| +Ours | 83.7 | 96.8 | 180.5 ($\rightarrow$) | 65.2 | 89.0 | 154.2 ($\uparrow 1.1$) | +4.2e-3 | 2.1$\times$ | 61.9 | 86.5 | 148.4 ($\uparrow 0.4$) | 44.8 | 73.7 | 118.5 ($\uparrow 2.8$) | +6.2e-2 | 4.1$\times$ | +1.3
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Evaluation metrics. We adopt the widely used metric Rank-k (R@k) for
evaluation. R@k represents the expectation of correct match at the $k$-th in
the ranking list and we report R@1, R@5 and R@sum (_i.e._ , the sum of R@1 and
R@5). In addition, in order to well reveal the method’s effectiveness, we
report the number of modal parameters (Para. for short, in millions), the
online running time111The running time is measured without acceleration
operation for a fair comparison. (in minutes) of the modal for all queries on
a dataset and the speedup ratio for retrieval. For the running time and the
speedup ratio, we report the mean average for TR and IR. It is worth noting
that the number of modal parameters in the proposed framework is different on
different datasets, since there are only two label embedding layers in the
proposed framework and its modal parameter depends on the number of
labels222There are $0.8$ and $1.8$ million parameters for the proposed
framework on Flickr30K and MS-COCO, respectively.. We report the average of
the values on Flickr30K and MS-COCO datasets.

Implementation details. The experiments are conducted on eight GeForce RTX
3090 GPUs with 24 GB of memory. For the advanced classification, we append the
classifiers to the image-text retrieval network ALBEF [2] in the experiments.
Based on the trained ALBEF on Flickr30K (resp. MS-COCO), we continue to train
the classifier for 10 (resp. 5) epochs with a batch size of 128. For training
the classifier better, we remove the class annotations paired with fewer than
$100$ images. As a result, there are $539$ and $1,122$ class annotations in
Flickr30K and MS-COCO, respectively. We set $\alpha_{+}=0$, $\alpha_{-}=3$ and
$\Delta=0.05$ for ASL loss referring to the settings in [40], and set
$R_{I}=15$ and $R_{T}=3$ in the image and text classifications, respectively.

Baselines. The proposed framework can be applied prior to the common image-
text retrieval method, _e.g._ , the early fusion, late fusion or the
Late2Early methods. Several image-text retrieval methods are involved as the
baselines in the experiments: the early fusion methods ALBEFall and ViLT-B/32a
[5]; the late fusion methods ALBEF0, LightDOT [16] and CLIP [3]; the
Late2Early methods from the free combination of the above late fusion and
early fusion baselines333Due to unpublished codes, the pure Late2Early methods
[15, 16, 17] can not be used in the experiments.. Referring to the setting in
[2], we select $128$ gallery samples by the late fusion network and fed them
into the early fusion network in the Late2Early. Specifically, ALBEFall and
ALBEF0 are evolved from the ALBEF [2]. In the ALBEF, given one query, all
gallery samples first get through the image/text processing branch to compute
the similarities with the query, and then the top-$128$ gallery samples
together with the query are sent to the following interaction module for
retrieval. ALBEFall refers to all galleries instead of top-$k$ ones being sent
to the interaction module and is essentially an early fusion method, and
ALBEF0 refers to that there are no galleries to the interaction module and is
essentially a late fusion method. We report the baselines’ results by running
the published code in the experiments.

###  V-B Results and Discussion

We report the results of the proposed pre-screening framework applied prior to
the Late2Early, late fusion or early fusion methods in Table I. No matter what
type of method is applied, the proposed framework can achieve acceleration,
while keeping the performance and sometimes offering a remarkable improvement.
For example, a $2.0\%$ increase at R@sum is achieved by the proposed framework
applied prior to the ALBEF0+ViLT-B/32a method on Flickr30K (TR), and a $2.8\%$
increase is achieved applied prior to the ViLT-B/32a method on MS-COCO (IR).
The price for this is only the $4.2$e-3 (resp. $6.2$e-2) minutes of online
running time on Flickr30K (resp. MS-COCO) and an average of $1.3$ million
modal parameters. It is worth noting that the proposed pre-screening framework
working on the early fusion method plays the same role as the late fusion
network in the Late2Early method, that is, both aim at improving the retrieval
efficiency of the early fusion method. Targeting the acceleration of the same
early fusion method (_i.e._ , ViLT-B/32a), the cost of the proposed framework
is far less than that of the late fusion network (_i.e._ , ALBEF0 and
LightDOT). Specifically, the proposed framework is an average of $2,072$
(resp. $685$) times faster than the late fusion network on Flickr30K (resp.
MS-COCO) for the running time and $145$ times less than the late fusion
network for the modal parameter.

TABLE II: Comparison with the ANN-based Late2Early methods. We use the ANN
algorithms: product quantization (PQ) and PQ with inverted file (IVFPQ). The
symbol ‘$\times$’ in the Speedup column refers to the screening acceleration
of the Late2Early method. For clarity, we present the R@(sum) variation of the
method compared to the Late2Early method.

Genre | Method | Flickr30K | MS-COCO
---|---|---|---
TR | IR | Speedup | TR | IR | Speedup
R@1 | R@5 | R@sum | R@1 | R@5 | R@sum | R@1 | R@5 | R@sum | R@1 | R@5 | R@sum
Late2Early | ALBEF0+ALBEFall | 95.9 | 99.8 | 195.7 | 85.5 | 97.5 | 183.0 |  - | 77.6 | 94.3 | 171.9 | 60.7 | 84.3 | 145.0 |  -
ANN Late2Early | +PQ | 95.7 | 99.7 | 195.4 ($\downarrow 0.3$) | 66.0 | 72.5 | 138.5 ($\downarrow 44.5$) | 1.3$\times$ | 75.9 | 92.0 | 167.9 ($\downarrow 4.0$) | 26.2 | 30.0 | 56.2 ($\downarrow 88.8$) | 2.2$\times$
ANN Late2Early | +IVFPQ | 76.5 | 80.8 | 157.3 ($\downarrow 38.4$) | 57.2 | 62.2 | 119.4 ($\uparrow 63.6$) | 3.2$\times$ | 56.5 | 71.1 | 127.6 ($\downarrow 44.3$) | 45.3 | 62.4 | 107.7 ($\downarrow 37.3$) | 3.6$\times$
| +Ours | 95.9 | 99.8 | 195.7 ($\rightarrow$) | 85.2 | 97.0 | 182.2 ($\downarrow 0.8$) | 2.1$\times$ | 77.6 | 94.1 | 171.7 ($\downarrow 0.2$) | 60.5 | 84.0 | 144.5 ($\downarrow 0.5$) | 4.1$\times$
Late2Early | ALBEF0+ViLT-B/32a | 85.6 | 97.2 | 182.8 | 66.7 | 90.1 | 156.8 |  - | 65.6 | 89.5 | 155.1 | 46.0 | 75.8 | 121.8 |  -
ANN Late2Early | +PQ | 86.0 | 97.9 | 183.9 ($\uparrow 1.1$) | 55.9 | 70.6 | 126.5 ($\downarrow 30.3$) | 1.3$\times$ | 64.9 | 87.5 | 152.4 ($\downarrow 2.7$) | 22.8 | 29.4 | 52.2 ($\downarrow 69.6$) | 2.2$\times$
ANN Late2Early | +IVFPQ | 74.0 | 83.8 | 157.8 ($\downarrow 25.0$) | 50.2 | 61.4 | 111.6 ($\downarrow 45.2$) | 3.2$\times$ | 53.5 | 74.0 | 127.5 ($\downarrow 27.6$) | 35.4 | 56.6 | 92.0 ($\downarrow 29.8$) | 3.6$\times$
| +Ours | 86.4 | 98.4 | 184.8 ($\uparrow 2.0$) | 66.2 | 90.2 | 156.4 ($\downarrow 0.4$) | 2.1$\times$ | 65.6 | 89.4 | 155.0 ($\downarrow 0.1$) | 46.1 | 75.3 | 121.4 ($\downarrow 0.4$) | 4.1$\times$
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |

###  V-C Further Analysis

Comparison with ANN-based Late2Early method. Though the Late2Early method
suffers from an $O(N)$ screening time complexity, the Approximate Nearest
Neighbor search (ANN) [44] can be applied to speed it up. Two commonly used
ANN algorithms from the Facebook AI Similarity Search (FAISS) library, _i.e._
, the Product Quantization (PQ) and the Product Quantization with InVerted
File (IVFPQ), are applied in the Late2Early for comparing with the proposed
framework. The comparison results are shown in Table II. The Late2Early method
is speeded up by using ANN during the screening process. The proposed
framework also achieves the screening acceleration over the Late2Early method,
which is superior to ANN in most cases. More importantly, the proposed
framework is nearly the same as the Late2Early method on performance, while
the ANN seriously hurts the performance of the Late2Early.

TABLE III: Comparison of the proposed framework with various settings. The
Adv. and Com. refer to the proposed advanced classification and the common
one, respectively. ASL and BCE refer to the asymmetric loss and the binary
cross-entropy loss, respectively. GT refers to the ground-truth annotations.
NVA refers to the nouns, verbs and adjectives in the text used as the ground-
truth annotations. For clarity, we present the R@(sum) variation of the
proposed framework with other settings compared to the proposed framework
(_i.e._ , Ours).

Method | Setting | Flickr30K | MS-COCO
---|---|---|---
Classifier | Loss | Image keyword | Text keyword | Keyword attribute | TR | IR | TR | IR
Adv. | Com. | ASL | BCE | Predict | GT | Predict | Extract | GT | Noun | NVA | R@1 | R@sum | R@1 | R@sum | R@1 | R@sum | R@1 | R@sum
Baseline#1: ALBEF0+ALBEFall (Late2Early method) | 95.9 | 195.7 | 85.5 | 183.0 | 77.6 | 171.9 | 60.7 | 145.0
+Ours | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 95.9 | 195.7 | 85.2 | 182.2 | 77.6 | 171.7 | 60.5 | 144.5
+Ours(Com.) | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 95.8 | 195.6 ($\downarrow 0.1$) | 84.6 | 181.0 ($\downarrow 1.2$) | 77.5 | 171.4 ($\downarrow 0.3$) | 59.7 | 142.8 ($\downarrow 1.7$)
+Ours(BCE) | ✓ | $\times$ | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 95.8 | 195.6 ($\downarrow 0.1$) | 85.3 | 182.4 ($\uparrow 0.2$) | 77.7 | 171.8 ($\uparrow 0.1$) | 60.5 | 144.5 ($\rightarrow$)
+Ours(Ext.) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | 95.6 | 195.3 ($\downarrow 0.4$) | 83.3 | 177.9 ($\downarrow 4.3$) | 77.7 | 171.8 ($\uparrow 0.1$) | 60.1 | 143.6 ($\downarrow 0.9$)
+Ours(Merge) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | 95.9 | 195.7 ($\rightarrow$) | 85.3 | 182.2 ($\rightarrow$) | 77.6 | 171.7 ($\rightarrow$) | 60.6 | 144.8 ($\uparrow 0.3$)
+Ours(NVA) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | ✓ | 95.8 | 195.5 ($\downarrow 0.2$) | 85.0 | 181.8 ($\downarrow 0.4$) | 77.7 | 171.8 ($\uparrow 0.1$) | 60.5 | 144.6 ($\uparrow 0.1$)
+Ours(GT) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | ✓ | $\times$ | 96.1 | 196.0 | 86.4 | 184.3 | 77.8 | 172.1 | 61.1 | 145.8
Baseline#2: ALBEF0+ViLT-B/32a (Late2Early method) | 85.6 | 182.8 | 66.7 | 156.8 | 65.6 | 155.1 | 46.0 | 121.8
+Ours | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 86.4 | 184.8 | 66.2 | 156.4 | 65.6 | 155.0 | 46.1 | 121.4
+Ours(Com.) | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 85.6 | 183.7 ($\downarrow 1.1$) | 66.5 | 156.4 ($\rightarrow$) | 65.3 | 154.2 ($\downarrow 0.8$) | 45.5 | 119.3 ($\downarrow 2.1$)
+Ours(BCE) | ✓ | $\times$ | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 86.0 | 184.2 ($\downarrow 0.6$) | 67.3 | 157.7 ($\uparrow 1.3$) | 65.5 | 154.8 ($\downarrow 0.2$) | 45.9 | 120.7 ($\downarrow 0.7$)
+Ours(Ext.) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | 85.3 | 183.0 ($\downarrow 1.8$) | 65.7 | 153.8 ($\downarrow 2.6$) | 65.7 | 155.3 ($\uparrow 0.3$) | 45.8 | 120.6 ($\downarrow 0.8$)
+Ours(Merge) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | 86.0 | 183.6 ($\downarrow 1.2$) | 66.9 | 157.2 ($\uparrow 0.8$) | 65.8 | 155.5 ($\downarrow 0.5$) | 45.8 | 120.9 ($\downarrow 0.5$)
+Ours(NVA) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | ✓ | 85.9 | 184.1 ($\downarrow 0.7$) | 67.1 | 157.2 ($\uparrow 0.8$) | 65.7 | 155.5 ($\uparrow 0.5$) | 45.8 | 121.2 ($\downarrow 0.2$)
+Ours(GT) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | ✓ | $\times$ | 87.0 | 185.1 | 69.7 | 161.8 | 65.7 | 154.5 | 46.9 | 122.9
Baseline#3: ViLT-B/32a (Early fusion method) | 83.7 | 180.5 | 64.3 | 153.1 | 61.6 | 148.0 | 42.8 | 115.7
+Ours | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 83.7 | 180.5 | 65.2 | 154.2 | 61.9 | 148.4 | 44.8 | 118.5
+Ours(Com.) | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 83.5 | 180.0 ($\downarrow 0.5$) | 65.6 | 154.2 ($\rightarrow$) | 61.7 | 148.1 ($\downarrow 0.3$) | 44.5 | 117.2 ($\downarrow 1.3$)
+Ours(BCE) | ✓ | $\times$ | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | 83.7 | 180.7 ($\uparrow 0.2$) | 66.2 | 154.6 ($\uparrow 0.4$) | 61.9 | 148.4 ($\rightarrow$) | 44.9 | 118.5 ($\rightarrow$)
+Ours(Ext.) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | 83.3 | 179.7 ($\downarrow 0.8$) | 64.7 | 151.5 ($\downarrow 2.7$) | 62.2 | 148.7 ($\uparrow 0.3$) | 44.8 | 117.8 ($\downarrow 0.7$)
+Ours(Merge) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | ✓ | $\times$ | ✓ | $\times$ | 83.8 | 180.6 ($\uparrow 0.1$) | 65.7 | 155.0 ($\uparrow 0.8$) | 61.9 | 148.4 ($\rightarrow$) | 44.8 | 118.4 ($\uparrow 0.1$)
+Ours(NVA) | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | ✓ | 83.5 | 180.0 ($\downarrow 0.5$) | 66.5 | 155.6 ($\uparrow 1.4$) | 61.9 | 148.4 ($\rightarrow$) | 44.7 | 118.4 ($\downarrow 0.1$)
+Ours(GT) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | ✓ | $\times$ | 85.6 | 183.1 | 68.1 | 159.7 | 62.6 | 149.8 | 46.1 | 120.9
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Comparison with common classification. The performance of the multi-label
classification for keyword prediction directly influences the result of the
proposed framework. Aiming at a high-performance classification, we propose
the advanced classification. Alternatively, the classification can be replaced
with any existing common one. For comparison, we use the ViT-B/16 [45] and
BERT [46] together with the ASL as the image and text classifiers,
respectively. In terms of performance, as shown in Table III, the proposed
framework with the advanced classification (Ours) has a consistent advantage
over that with the common classification (Ours(Com.)) based on different
baselines, which verifies the superiority of the proposed advanced
classification. In terms of the classification-related metrics, we show the
number of model parameters, the recall rate of ground-truth gallery samples
and the mAP values of image and text classifiers in Table IV. The advanced
classification model surpasses the common one at all metrics. Specifically,
the number of parameters for the proposed framework with the advanced
classification is far less than that with the common classification, which
shows the superiority of the proposed framework in resource consumption.

Analysis of the advanced classification. We analyze the proposed advanced
classification from three aspects. (1) We adopt the common binary cross-
entropy loss (BCE) to replace the ASL as the classification loss. The
comparison results (Ours vs. Ours(BCE)) in Table III show that the ‘Ours(BCE)’
has a slight drop at R@sum compared to the ‘Ours’ on Flickr30K (TR) when
applied in the Baseline#1 and Baseline#2 and on MS-COCO when applied in
Baseline#2, and has a slight increase or remains unchanged in other cases. In
general, the ‘Ours’ and ‘Ours(BCE)’ are much of a muchness on performance. By
adopting other state-of-the-art classification losses, the proposed framework
with high flexibility is expected to further improve. (2) We develop a multi-
label text classifier for text keyword prediction. Instead, we can also
extract the nouns directly from the text by using the natural language toolkit
(Ours(Ext.)), or use both the predicted labels and extracted nouns as the
keywords (Ours(Merge)). As shown in Table III, the ‘Ours’ performs better than
the ‘Ours(Ext.)’ in all cases except for MS-COCO (TR). In the ‘Ours(Ext.)’,
the text keywords are derived from itself. In contrast, the keywords are from
all ground-truth annotations in the ‘Ours’, enabling the exploration of
keyword synonyms and positively affecting performance. However, the classifier
is likely to predict the wrong keywords in the ‘Ours’, thus hurting retrieval
performance. Beyond that, as mentioned in Section III-A, there are pre-
screening failures in the ‘Ours(Ext.)’, yet not in the ‘Ours’. Specifically,
$37$ and $9$ texts can not incorporate into the pre-screening process by the
‘Ours(Ext.)’ in Flickr30K and MS-COCO, respectively. In view of the above,
merging the predicted labels and extracted nouns to the keywords is a
complementary solution. It can be seen from Table III that ‘Ours(Merge)’
surpasses the ‘Ours’ by a small margin in most cases. At the same time,
though, the ‘Ours(Merge)’ suffers from a lower screening rate due to the
increasing number of text keywords from the mergence operation. As a result,
the ‘Ours(Merge)’ is weaker than the ‘Ours’ on retrieval efficiency.
Specifically, the $2.0\times$ and $3.9\times$ speedups are reached by the
‘Ours(Merge)’ on Flickr30K and MS-COCO, respectively; by contrast, the ‘Ours’
introduces the $2.1\times$ and $4.1\times$ speedups. (3) We use the nouns in
the text as the image and text ground-truth annotations for training
classifiers. One alternative is to use the nouns, verbs and adjectives
(Ours(NVA)). The results in Table III show that the ‘Ours(NVA)’ is inferior to
the ‘Ours’ on performance in some cases and performs better than the ‘Ours’ in
others. In general, the ‘Ours’ and ‘Ours(NVA)’ are much of a muchness on
performance. However, it is remarkable that more keywords can be predicted in
the ‘Ours(NVA)’ due to more diverse ground-truth annotations, as a result of
which the screening rate is reduced and the retrieval efficiency is hurt.

The upper bound of the proposed framework. Excluding the harmful effects of
the wrong classification results on performance, we directly use the ground-
truth annotations as the keywords to explore the upper bound of the proposed
framework on performance (Ours(GT)). As shown in Table III, the ‘Ours(GT)’
performs better than the ‘Ours’ and also consistently achieves remarkable
improvement over the baseline on performance. Owing the high flexibility, the
proposed framework is hopeful to realize a powerful win-win situation for
accuracy and efficiency by adopting a better classification technique.

TABLE IV: Comparison of different classification techniques. The mAP(image)
and mAP(text) refer to the mAP value of the image and text classifiers,
respectively. The Adv. and Com. refer to the proposed advanced classification
and the common one, respectively.

Method | Flickr30K | MS-COCO | Para(M)
---|---|---|---
Recall | mAP(image) | mAP(text) | Recall | mAP(image) | mAP(text)
Adv. | 99.4 | 52.0 | 58.9 | 99.4 | 56.9 | 62.2 | 1.3
Com. | 98.7 | 48.4 | 58.9 | 98.1 | 50.5 | 61.9 | 196.4
|  |  |  |  |  |  |

Parameter analysis. There are two parameters in the proposed framework, _i.e._
, $R_{I}$ in image keyword prediction and $R_{T}$ in text keyword prediction.
We show the results of the proposed framework with different parameters based
on the ALBEF0+ALBEFall in Table V. With an increasing value of $R_{I}$ or
$R_{T}$, more ground-truth gallery samples are kept in the pre-screening stage
and enter into the follow-up retrieval network (_i.e._ , a higher Recall
value), resulting in better accuracy (_i.e._ , a higher R@1), yet at the same
time, causing a decline in efficiency (_i.e._ , a lower Speedup). For
achieving a trade-off between accuracy and efficiency, we set $R_{I}=15$ and
$R_{T}=3$ in the experiments.

TABLE V: Results of the proposed framework with different parameters. The
ALBEF0+ALBEFall is used as the baseline.

Para. | Flickr30K | MS-COCO
---|---|---
R@1(TR) | R@1(IR) | Recall | Speedup | R@1(TR) | R@1(IR) | Recall | Speedup
Varying $R_{I}$ and fixing $R_{T}=3$.
5 | 95.5 | 83.0 | 96.4 |  $\times$2.9 | 77.4 | 59.4 | 97.2 |  $\times$7.0
10 | 95.8 | 84.6 | 98.6 |  $\times$2.3 | 77.6 | 60.3 | 98.9 |  $\times$5.0
15 | 95.9 | 85.2 | 99.4 |  $\times$2.1 | 77.6 | 60.5 | 99.4 |  $\times$4.1
20 | 95.9 | 85.2 | 99.6 |  $\times$1.9 | 77.6 | 60.6 | 99.6 |  $\times$3.7
25 | 95.9 | 85.3 | 99.8 |  $\times$1.8 | 77.6 | 60.6 | 99.7 |  $\times$3.3
30 | 95.9 | 85.3 | 99.8 |  $\times$1.7 | 77.6 | 60.6 | 99.8 |  $\times$3.1
Varying $R_{T}$ and fixing $R_{I}=15$.
1 | 95.0 | 79.5 | 91.7 |  $\times$3.3 | 76.9 | 57.1 | 92.4 |  $\times$8.4
2 | 95.7 | 84.3 | 98.1 |  $\times$2.5 | 77.5 | 60.0 | 98.3 |  $\times$5.4
3 | 95.9 | 85.2 | 99.4 |  $\times$2.1 | 77.6 | 60.5 | 99.4 |  $\times$4.1
4 | 95.9 | 85.4 | 99.6 |  $\times$1.8 | 77.7 | 60.6 | 99.7 |  $\times$3.3
5 | 95.9 | 85.5 | 99.8 |  $\times$1.7 | 77.7 | 60.6 | 99.8 |  $\times$2.9
10 | 95.9 | 85.5 | 100.0 |  $\times$1.3 | 77.7 | 60.7 | 100.0 |  $\times$1.9
20 | 95.9 | 85.5 | 100.0 |  $\times$1.1 | 77.7 | 60.7 | 100.0 |  $\times$1.4
|  |  |  |  |  |  |  |

Efficiency comparison on larger-scale data. To verify the superiority of the
proposed framework in reality, we make an efficiency comparison with the
Late2Early method ALBEF0+ALBEFall on larger-scale data, which is constructed
by merging all data in Flickr30K and MS-COCO and results in $154,301$ images
and $771,837$ texts. We adopt the classifiers trained on MS-COCO in the
proposed framework in the experiments. Fig. 6 shows the running time444The
running time is composed of the query processing time and gallery screening
time and the change in the number of gallery samples only impacts the
screening time, and thus we only report the screening time in Fig. 6. The
query processing time is 3.0e-2 in TR and 3.1e-4 in IR for the Late2Early
method, and is 0 for the proposed framework thanks to the proposed multi-task
learning scheme. of one query on the new benchmark with different numbers of
gallery samples. The proposed framework shows significant superiority over the
Late2Early method on efficiency. Moreover, with an increasing number of the
gallery samples, the increasing trend of the proposed framework’s running time
is gradually slowed down, by contrast, the Late2Early method’s running time
increases linearly. These results indicate the potential application of the
proposed framework in reality.

Figure 6: Running time (in minutes) of one query on large gallery data (in
thousands). ‘$\sim$750’ and ‘$\sim$150’ specifically refer to 771,837 texts
and 154,301 images, respectively. TABLE VI: Performance of the proposed
framework on Flickr30K-all.

Genre | Method | TR | IR | Speedup
---|---|---|---|---
R@1 | R@5 | R@sum | R@1 | R@5 | R@sum
Late2Early | ALBEF0+ALBEFall | 68.8 | 87.1 | 155.9 | 48.1 | 70.6 | 118.7 |  -
| +Ours | 68.8 | 87.0 | 155.8 ($\downarrow 0.1$) | 47.8 | 70.2 | 118.0 ($\downarrow 0.7$) | 2.0$\times$
|  |  |  |  |  |  |  |

Simulation of practical application scenario. In practical application
scenario, the proposed framework usually deals with millions of wide-open
inference data with more diverse content. As a result, the keywords predefined
from training data may not have appropriate coverage of the semantic space of
the inference data, which tends to lead to false predictions for keywords and
affect the framework’s performance. To investigate the performance of the
proposed framework in such a scenario, we perform the proposed framework on
Flickr30K-all, which is constructed by merging the training, validataion and
inference data in Flickr30K and results in $31,014$ images and $155,070$ texts
and we adopt the proposed framework trained in MS-COCO. There is a relatively
big gap between the training and inference data. Table VI presents the results
on Flickr30K-all. It can be seen that a decent performance in accuracy and
speedup can still be obtained by the proposed framework. The proposed
framework would work as long as there is only one correctly predicted keyword
among the resulting keywords, so that the negative impact of the data gap on
performance is alleviated to some extent.

Visualization and limitation. We visualize the predicted keywords and the
screening results555For lack of space, we randomly select a few samples among
large number of gallery samples for visualization. from the proposed framework
in Fig. 7. It can be seen that the proposed framework yields the correct and
reasonable keywords at the semantic level and can bring decent screening
results, _i.e.,_ screening out the gallery samples semantically irrelevant to
the query. Notably, the results in Flickr30K-all are obtained by the framework
trained in MS-COCO and the gap between the training and inference data does
not have much effect on performance.

Taking a step further, we discuss the limitation of the proposed framework.
The proposed framework relies on the quality of the predicted keywords. With
that in mind, we propose a multi-task learning scheme for improving the
classification accuracy and prompting the keyword prediction, and present the
detailed results of the classifier with various settings in Table III and the
in-depth analysis in Section V-C. Nevertheless, there still leaves room for
further development of the proposed framework with a more powerful
classification technique. We could consider the labels from the object
detection as a powerful supplementary to the classification results, which
meanwhile could well apply to the case of the collections without extensive
textual annotation.

Figure 7: Visualizing the predicted keywords and the screening results from
the proposed framework taking TR example. The keywords enclosed by the curly
braces ‘$\\{\\}$’ are below the sample. We randomly select a few samples among
large number of gallery samples for visualization.

##  VI Conclusion

This paper focuses on the low efficiency problem in the image-text retrieval
task. Admittedly, the existing image-text retrieval methods suffer from at
least $O(N)$ time complexity and may not be economically practical in many
real cases. To this end, we present a simple and effective keyword-guided pre-
screening framework, in which the image and text samples are projected into
the keywords, and then a fast keyword matching across modalities is executed
to screen out the gallery samples irrelevant to the query sample. The
remaining gallery samples with an amount that is much less than the number of
the original gallery samples are fed into the common image-text retrieval
network, thus realizing the retrieval acceleration. The proposed framework is
characterized by low consumption and excellent compatibility. We
experimentally verify the effectiveness of the proposed framework.

## References

  * [1] M. Cao, S. Li, J. Li, L. Nie, and M. Zhang, “Image-text retrieval: A survey on recent research and development,” _arXiv preprint arXiv:2203.14713_ , 2022\.
  * [2] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [4] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in _Proceedings of the European Conference on Computer Vision_ , 2020.
  * [5] W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5583–5594.
  * [6] Y. Liu, Y. Guo, E. M. Bakker, and M. S. Lew, “Learning a recurrent residual fusion network for multimodal matching,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4107–4116.
  * [7] L. Wang, Y. Li, J. Huang, and S. Lazebnik, “Learning two-branch neural networks for image-text matching tasks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 2, pp. 394–407, 2018.
  * [8] J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang, “Learning the best pooling strategy for visual semantic embedding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 789–15 798.
  * [9] A. Andonian, S. Chen, and R. Hamid, “Robust cross-modal representation learning with progressive self-distillation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 430–16 441.
  * [10] H. Lu, N. Fei, Y. Huo, Y. Gao, Z. Lu, and J.-R. Wen, “Cots: Collaborative two-stream vision-language pre-training model for cross-modal retrieval,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 15 692–15 701.
  * [11] C. Liu, Z. Mao, T. Zhang, H. Xie, B. Wang, and Y. Zhang, “Graph structured network for image-text matching,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 921–10 930.
  * [12] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei _et al._ , “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in _Proceedings of the European Conference on Computer Vision_ , 2020, pp. 121–137.
  * [13] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, “Vinvl: Revisiting visual representations in vision-language models,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5579–5588.
  * [14] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Proceedings of the European Conference on Computer Vision_ , 2014, pp. 740–755.
  * [15] A. Miech, J.-B. Alayrac, I. Laptev, J. Sivic, and A. Zisserman, “Thinking fast and slow: Efficient text-to-visual retrieval with transformers,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9826–9836.
  * [16] S. Sun, Y.-C. Chen, L. Li, S. Wang, Y. Fang, and J. Liu, “Lightningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 982–997.
  * [17] H. Liu, T. Yu, and P. Li, “Inflate and shrink: Enriching and reducing interactions for fast text-image retrieval,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021, pp. 9796–9809.
  * [18] B. B. Cambazoglu and R. Baeza-Yates, “Scalability and efficiency challenges in large-scale web search engines,” in _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_ , 2016, pp. 1223–1226.
  * [19] Z. Zheng, L. Zheng, M. Garrett, Y. Yang, M. Xu, and Y.-D. Shen, “Dual-path convolutional image-text embeddings with instance loss,” _ACM Transactions on Multimedia Computing, Communications, and Applications_ , vol. 16, no. 2, pp. 1–23, 2020.
  * [20] N. Sarafianos, X. Xu, and I. A. Kakadiaris, “Adversarial representation learning for text-to-image matching,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5814–5824.
  * [21] G. Park, C. Han, W. Yoon, and D. Kim, “Mhsan: Multi-head self-attention network for visual semantic embedding,” in _Proceedings of the IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 1518–1526.
  * [22] Y. Huang, Q. Wu, C. Song, and L. Wang, “Learning semantic concepts and order for image and sentence matching,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6163–6171.
  * [23] A. Karpathy, A. Joulin, and L. F. Fei-Fei, “Deep fragment embeddings for bidirectional image sentence mapping,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [24] Z. Wang, X. Liu, H. Li, L. Sheng, J. Yan, X. Wang, and J. Shao, “Camp: Cross-modal adaptive message passing for text-image retrieval,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5764–5773.
  * [25] K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He, “Stacked cross attention for image-text matching,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 201–216.
  * [26] K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu, “Visual semantic reasoning for image-text matching,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 4654–4662.
  * [27] K.-H. Lee, H. Palangi, X. Chen, H. Hu, and J. Gao, “Learning visual relation priors for image-text matching and image captioning with neural scene graph generators,” _arXiv preprint arXiv:1909.09953_ , 2019.
  * [28] Y. Wang, H. Yang, X. Bai, X. Qian, L. Ma, J. Lu, B. Li, and X. Fan, “Pfan++: Bi-directional image-text retrieval with position focused attention network,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 3362–3376, 2020\.
  * [29] K. Zhang, Z. Mao, A. Liu, and Y. Zhang, “Unified adaptive relevance distinguishable attention network for image-text matching,” _IEEE Transactions on Multimedia_ , 2022.
  * [30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [31] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” _IEEE transactions on neural networks_ , vol. 20, no. 1, pp. 61–80, 2008.
  * [32] E. Yang, C. Deng, W. Liu, X. Liu, D. Tao, and X. Gao, “Pairwise relationship guided deep hashing for cross-modal retrieval,” in _proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [33] E. Yang, C. Deng, C. Li, W. Liu, J. Li, and D. Tao, “Shared predictive cross-modal deep quantization,” _IEEE transactions on neural networks and learning systems_ , vol. 29, no. 11, pp. 5292–5303, 2018.
  * [34] D. Zhang, X.-J. Wu, T. Xu, and J. Kittler, “Watch: Two-stage discrete cross-media hashing,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [35] Z. Gan, Y.-C. Chen, L. Li, T. Chen, Y. Cheng, S. Wang, and J. Liu, “Playing lottery tickets with vision and language,” _arXiv preprint arXiv:2104.11832_ , 2021.
  * [36] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” _arXiv preprint arXiv:1803.03635_ , 2018.
  * [37] J. Wang, X. Hu, P. Zhang, X. Li, L. Wang, L. Zhang, J. Gao, and Z. Liu, “Minivlm: A smaller and faster vision-language model,” _arXiv preprint arXiv:2012.06946_ , 2020.
  * [38] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 5776–5788, 2020.
  * [39] G. Geigle, J. Pfeiffer, N. Reimers, I. Vulić, and I. Gurevych, “Retrieve fast, rerank smart: Cooperative and joint approaches for improved cross-modal retrieval,” _arXiv preprint arXiv:2103.11920_ , 2021.
  * [40] T. Ridnik, E. Ben-Baruch, N. Zamir, A. Noy, I. Friedman, M. Protter, and L. Zelnik-Manor, “Asymmetric loss for multi-label classification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2021, pp. 82–91.
  * [41] J. Lanchantin, T. Wang, V. Ordonez, and Y. Qi, “General multi-label image classification with transformers,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 16 478–16 488.
  * [42] J. Perkins, _Python text processing with NLTK 2.0 cookbook_. PACKT publishing, 2010.
  * [43] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2641–2649.
  * [44] S. Arya, D. M. Mount, N. S. Netanyahu, R. Silverman, and A. Y. Wu, “An optimal algorithm for approximate nearest neighbor searching fixed dimensions,” _Journal of the ACM (JACM)_ , vol. 45, no. 6, pp. 891–923, 1998.
  * [45] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [46] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.