# Benchmarking Robustness of Text-Image Composed Retrieval

Shitong Sun1 Jindong Gu2 Shaogang Gong1
1Queen Mary University of London 2 University of Oxford
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Text-image composed retrieval aims to retrieve the target image through the
composed query, which is specified in the form of an image plus some text that
describes desired modifications to the input image. It has recently attracted
attention due to its ability to leverage both information-rich images and
concise language to precisely express the requirements for target images.
However, the robustness of these approaches against real-world corruptions or
further text understanding has never been studied. In this paper, we perform
the first robustness study and establish three new diversified benchmarks for
systematic analysis of text-image composed retrieval against natural
corruptions in both vision and text and further probe textural understanding.
For natural corruption analysis, we introduce two new large-scale benchmark
datasets, CIRR-C and FashionIQ-C for testing in open domain and fashion domain
respectively, both of which apply 15 visual corruptions and 7 textural
corruptions. For textural understanding analysis, we introduce a new
diagnostic dataset CIRR-D by expanding the original raw data with synthetic
data, which contains modified text to better probe textual understanding
ability including numerical variation, attribute variation, object removal,
background variation, and fine-grained evaluation. The code and benchmark
datasets are available at https://github.com/SunTongtongtong/Benchmark-
Robustness-Text-Image-Compose-Retrieval.

##  1 Introduction

Text-image composed retrieval, known as composed image retrieval or text-
guided image retrieval, attempts to retrieve an image of interest from gallery
images through a composed query of a reference image and its corresponding
modified text. As a single word (‘dog’) can correspond to thousands of images
(dogs in different breeds, poses, and scenarios), language is considered to be
discrete and sparse, while images are regarded as dense and continuous. Using
both images and text as queries enables the effective utilization of the
continuous and dense nature of images to accurately express the requirements
while leveraging discrete and sparse text to bridge semantic gaps beyond what
the images alone can capture. It becomes fascinating for its potential in a
wide range of real-world applications including fashion domain e-commerce [13,
14, 10, 12, 5] and open domain internet search [25, 2, 11, 35]. However,
existing text-image composed retrieval methods mostly are trained and tested
on clean data, while the models in the real world may naturally encounter
distribution shift [43], such as text typos and image corruptions owing to
weather change. Additionally, there is currently no analysis of whether the
model understands the meaning of the text rather than solely relying on
finding correspondences with the main objects as a shortcut to the text-image
composed retrieval task. For example, with a source image of a dog and
modified text ‘change to two dogs on the table’, the model may retrieve the
target image by only recognizing the word ‘dog’ and ‘table’ without the
ability of numerical counting. Whether text-image composed retrieval models
are robust in real-world applications, where natural corruption exists in both
images and text, remains unexplored. Also, whether these models are robust
across diverse textural understanding requirements, remains a domain blank.

In this work, we make the first attempt to evaluate the robustness of text-
image composed retrieval by building three new large-scale robustness
benchmarks on both fashion and open domains. We raise the following two
questions: Q1: _How robust are text-image composed retrieval models to natural
corruption including both visual and textual?_ Further to evaluate the text
understanding ability, we have the second question Q2:  _How robust are text-
image composed retrieval models to text understanding?_

To answer the first question, we present two benchmark datasets on text-image
composed retrieval task. Based on two widely used datasets FashionIQ [45] in
the fashion domain and CIRR [25] in the open domain, we propose our benchmark
datasets, namely FashionIQ-C and CIRR-C, both with 15 visual corruptions and 7
textual corruptions to evaluate model robustness against natural corruption in
both image and text. To answer the second question, we introduce a new
diagnostic dataset CIRR-D to probe text understanding abilities on five
elementary scenarios including numerical variation, attribute manipulation,
object removal, background variation, and fine-grained variation. In detail,
we construct the diagnostic dataset by synthetic triplets based on the CIRR
validation set and use existing triplets from the CIRR validation set with
both main captions and extended captions. Our experiments show that the new
benchmarks we introduced are suitable for robustness analysis against natural
corruption on both image and text and further probe text understanding
ability.

Our contributions are: (1) We make the first attempt to analyze the robustness
of text-image composed retrieval methods against natural corruption (including
visual and textual) and textual understanding (including five elementary
variations). (2) We introduce three new large-scale benchmarks including two
benchmark datasets (FashionIQ-C and CIRR-C) to evaluate robustness against
natural corruption in both image and text and one diagnostic benchmark CIRR-D
to probe text understanding robustness. (3) We present an empirical analysis
and conduct extended experiments.

##  2 Related Works

Quantifying robustness aims to evaluate the model stability to defend against
corruption including natural corruption [16, 4, 42], adversarial attacks [6,
41, 42], or to probe certain ability such as logical reasoning [36] and visual
content manipulation [22]. Traditional works about robustness analysis mainly
focus on single modality involving visual modality-based tasks like image
classification [16], face detection [9]; textual modality based task like text
classification [47] and audio modality based task like speech recognition
[28]. Recently, robustness analysis against multimodal tasks, which is closer
to real life and attempts to take a step towards a reliable system, has
appeared but is still in its infancy. For example, Li et al. [22] take the
first step to systematically analyze the robustness of a multimodal task,
Visual Question Answering (VQA), against 4 generic robustness including
linguistic variation and visual content manipulation. However, it is limited
to VQA tasks and doesn’t introduce benchmarks to pinpoint sophisticated
reasoning abilities. Schiappa et al. [4] introduce natural corrupted visual
and textual benchmarks on text-to-video retrieval. However, the robustness
analysis of the multimodal underlying hypothesis, which aims to generalize
textual semantic and reasoning ability to visual space, is not discussed. We
consider the analysis of both natural corruption in image and text and further
underlying text understanding and take the first step to conduct an extensive
analysis of the natural corruption and text understanding of the robustness of
deep neural networks in text-image composed retrieval. More related work about
text-image composed retrieval and diagnostic analysis can be found in
supplementary D.

Figure 1:  Sample visualization of proposed benchmarks under natural
corruption with both visual and textual. Top: CIRR-C with impluse noise image
corruption Middle: FashionIQ-C with zoom blur image corruption; Bottom: CIRR-C
under character-level (Swap and Qwerty) and word-level (Repetition and
Homophones) textual corruptions. Gallery images are shown without particular
order.  Figure 2:  Sample visualization of proposed benchmarks probing text
understanding. Top: CIRR-D with attribute variations; Middle: CIRR-D with
number variations; Bottom: CIRR-D with object removal. CIRR-D with background
variation and fine-grained variation can be found in the supplementary
materials.

##  3 Robustness Criteria for Text-Image Composed Retrieval

Foundation of text-image composed retrieval. Text-image composed retrieval
aims to retrieve the target visual content through dense continuous images
guided by sparse discrete text. We discuss ‘dense’ or ‘sparse’ in the semantic
space, where a single semantic word can correspond to thousands of images.
Therefore, text-image composed query can overcome the limitation of singular
modality image retrieval, where text-image retrieval suffers from the
unprecise description and unlimited correct targets, and image-image retrieval
suffers from expression limitation without the ability to generalise to
different visual content. In light of this, the foundation abilities of text-
image composed retrieval are threefold: (1) Image representation to supply a
precise anchor in the dense continuous visual space; (2) Text representation
to supply subtle or significant differences between various visual contents,
providing an unprecise target direction the model can generalize to; (3)
Generalize sparse modified text attributes to dense reference images to
precisely predict the target visual content by different variations of
modality fusion.

Definition of robustness in text-image composed retrieval. According to the
foundation of text-image composed retrieval above, a robust model should
perform stable image feature extraction, text feature extraction, and modality
fusion. In light of this, the robustness of text-image composed retrieval can
be defined in twofold: _robustness against natural corruption_ for both text
and image and _robustness against textual understanding_ for consistent
reasoning between textual and visual modality. Specifically, for robustness
against natural corruption, we evaluate text-image composed retrieval models
under ubiquitous corruption frequently encountered in real-life in both visual
and textual. We evaluate 15 standard image corruptions with 5 severity
categorized into noise, blur, weather, and digital following [16]. We also
evaluate 7 text corruptions categorized into character-level and word-level.
Further, for robustness against textual understanding, we evaluate common
linguistic reasoning by selecting modified text with specific keywords or
gallery set, categorized into numerical variation, attributes variation,
object removal, background variation, and fine-grained variation respectively.

Evaluation metrics. To evaluate the performance of models in text-image
composed retrieval, we adopt the standard evaluation metric in retrieval, i.e.
Recall@K donated as R@K for short. Further to measure robustness, we adopt
relative robustness metrics $\gamma=1-\left(R_{c}-R_{p}\right)/R_{c}$
following [4, 16], where $R_{c}$ and $R_{p}$ are the R@K under clean data and
data with corruption respectively. Additionally, in order to facilitate fair
comparison among different models, we have expanded [7] and established a
unified testing platform for the convenient integration of various models. In
detail, we set gallery the whole validation set as in [7, 2], which includes
more distractors and results to higher discriminative requirement, instead of
setting gallery the same as the query set as in [5, 20]. Specifically for
evaluating the fashionIQ dataset, we combine the two captions in a single
query as [2, 8] instead of combining the two modified captions in forward and
reverse direction as [20]. All the evaluated models are trained in three
categories jointly and tested individually for dress, shirt, and toptee
categories. The reported results for fashionIQ are average of the three
categories.

Evaluation datasets. We utilize three new benchmarks for our text-image
composed retrieval experiments, which are generated from two existing
datasets: FashionIQ [45] in the fashion domain and CIRR [25] in the open
domain. Both datasets include human-generated captions that distinguish image
pairs. FashionIQ is based on the fashion domain containing 77,684 garment
images, which can be divided into three categories: dress, shirt, and toptee.
Each image in FashionIQ contains a single subject positioned centrally with a
clean background. CIRR is composed of 21,552 real-life images extracted from
NLVR2 [38], which contains rich visual content in diverse backgrounds. As
shown in Figure. 1, we build our benchmark and evaluate text-image composed
retrieval models on text and image natural corruption robustness. Further as
shown in Figure. 2, we expand the CIRR dataset and evaluate text understanding
robustness.

_Natural vision and text corruption._ To evaluate the robustness of the text-
image composed retrieval model against natural corruption in both image and
text, we create our robustness benchmark CIRR-C and FashionIQ-C with 15 visual
corruptions and 7 textual corruptions. For vision corruption, we follow [16]
to implement 15 standard natural corruptions which fall into four categories:
noise, blur, weather, and digital, each having a severity from 1 to 5. For
text corruption, we follow [34] and implement the most related seven
corruptions including four character-level corruptions and three word-level
corruptions respectively.

_Diagnostic datasets._ Following the current methods [25, 2] reporting the
results on the validation set, we expand and build our probing datasets CIRR-D
based on the validation set of CIRR to pinpoint text understanding ability
including numerical variation, attributes composition, object removal,
background transformation, and fine-grained evaluation. We hypothesize that
the model’s corresponding capabilities can be evaluated when the modified text
involves descriptions such as numbers, attributes, objects removal, or
changing the background; and the ability to deal with fine-grained variations
can be evaluated when the gallery images are highly similar following [25]. In
light of this, we build the triplets (reference image, modified text, and
target image) of our probing dataset according to the appearances of specific
keywords in modified text: "zero" to "ten", "number" for numerical query;
color, shape and size for attribute query, "remove" for object removal query;
"background" for background variation query. As shown in Table 3, the
construction of CIRR-D dataset can fall into five probing categories from
three sources as follows: (1) The existing validation set of CIRR is composed
of 2297 images and 4181 triplets, which is currently widely used. Each image
has a subset which is composed of 6 highly similar images as a gallery to
better detect fine-grained discriminative ability. (2) The auxiliary captions
of the CIRR validation set, which is supplied but has not been used in the
conventional evaluation. These captions indicate the differences in removal
content or background changes between image pairs, but they may not provide
enough information to accurately locate the target image. Therefore, we
manually eliminated triplets that led to an excessive number of target images.
(3) Synthetic images we generate through Visual ChatGPT [44] to bridge the
language reasoning and visual recognition. Based on the image from CIRR
validation set, we generate its image caption by Visual ChatGPT and generate
ten variants of the captions by ChatGPT including four for number variants,
three for color variants, two for size variants, one for removing the objects
respectively. Afterward, based on the reference image and caption variants,
Visual ChatGPT utilizes groundingDINO [24] to do object detection, segment
anything [19] to generate mask and stable diffusion [33] to generate target
image. We manually eliminated the unplausible generated images. Our synthetic
image pairs preserve the original background and only modify the specific
areas mentioned in the text.

Evaluated models. We perform our experiments on six text-image composed
retrieval models. The modality fusion of these approaches is roughly
summarized in Supplementary Table 4 including [40, 25, 2, 7, 13, 8], which can
be divided into overlapped categories: (1) Large pretrained model: FashionViL,
CIRPLANT, CLIP4CIR, whose pretrained dataset size are 1.35 million, 6.5
million, and 400 million image-text pairs respectively; (2) Multi-task model:
FashionViL, which is pretrained with four tasks at the same time; (3) Light
attention-based methods: ARTEMIS; (4) Transformer-based models: MAAF,
CIRPLANT, CIRPLANT and FashionViL. (5) Lightweight models: MAAF, TIRG, and
ARTEMIS (all with ResNet50 image encoder and LSTM text encoder for fair
comparison). (6) Single-modality models: Image-only (RN50) and Image-
only(CLIP) are queried with images embedded by ResNet50 (same as evaluated
TIRG, MAAF, ARTEMIS) and CLIP image encoder RN50x4 (same as evaluated
CLIP4CIR) respectively. Text-only model is queried with text embedded using
the CLIP text encoder. These methods were chosen because the reproduced
results match with original reported results. We test FashionViL [13] in the
fashion domain, CIRPLANT in the open domain, and all the rest published models
in both the fashion domain and open domain.

Table 1: Relative robustness score for text-image composed retrieval under 15
natural image corruptions in CIRR-C Recall@10 and FashionIQ-C Recall@10.
Recall@10 performance under clean conditions on the left. Bold is the highest
relative robustness for the five composed retrieval methods.

|  | Noise | Blur | Weather | Digital
---|---|---|---|---|---
CIRR-C | Clean | Gauss. | Shot | Implu. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contr. | Elast. | Pixel | JPEG
Image-only(RN50) | $50.4$ | $0.57$ | $0.55$ | $0.58$ | $0.68$ | $0.28$ | $0.82$ | $0.45$ | $0.38$ | $0.34$ | $0.64$ | $0.86$ | $0.20$ | $0.48$ | $0.76$ | $0.88$
Image-only(CLIP) | $36.2$ | $0.56$ | $0.55$ | $0.58$ | $0.66$ | $0.32$ | $0.83$ | $0.49$ | $0.52$ | $0.45$ | $0.77$ | $0.91$ | $0.24$ | $0.41$ | $0.78$ | $0.91$
Text-only(CLIP) | $51.2$ | $0.79$ | $0.76$ | $0.81$ | $0.85$ | $0.29$ | $1.0$ | $0.55$ | $0.65$ | $0.70$ | $0.89$ | $1.0$ | $0.19$ | $0.40$ | $0.96$ | $1.0$
TIRG [40] | $55.1$ | $0.34$ | $0.36$ | $0.34$ | $0.48$ | $0.21$ | $0.70$ | $0.43$ | $0.31$ | $0.22$ | $0.40$ | $0.70$ | $0.12$ | $0.47$ | $0.74$ | $0.84$
MAAF [8] | $49.9$ | $0.50$ | $0.49$ | $0.50$ | $0.62$ | $0.26$ | $0.80$ | $0.41$ | $0.36$ | $0.31$ | $0.50$ | $0.74$ | $0.11$ | $0.48$ | $0.83$ | $0.87$
ARTEMIS [7] | $59.0$ | $0.39$ | $0.42$ | $0.38$ | $0.51$ | $0.25$ | $0.70$ | $0.44$ | $0.31$ | $0.26$ | $0.45$ | $0.71$ | $0.10$ | $0.47$ | $0.75$ | $0.86$
CIRPLANT [25] | $68.8$ | 0.70 | 0.69 | 0.71 | $0.77$ | $0.28$ | $0.89$ | $0.51$ | $0.44$ | $0.43$ | $0.66$ | $0.88$ | 0.17 | 0.56 | $0.85$ | $0.92$
CLIP4CIR [2] | 80.3 | $0.68$ | $0.68$ | $0.69$ | 0.77 | 0.28 | 0.90 | 0.52 | 0.55 | 0.60 | 0.80 | 0.91 | $0.16$ | $0.39$ | 0.91 | 0.92

|  | Noise | Blur | Weather | Digital
---|---|---|---|---|---
FashionIQ-C | Clean | Gauss. | Shot | Implu. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contr. | Elast. | Pixel | JPEG
TIRG [40] | 23.8 | $0.28$ | $0.26$ | $0.23$ | $0.34$ | $0.22$ | $0.61$ | $0.57$ | $0.32$ | $0.27$ | $0.37$ | $0.61$ | $0.12$ | $0.64$ | $0.85$ | $0.85$
MAAF [8] | 23.4 | $0.31$ | $0.27$ | $0.25$ | $0.44$ | $0.21$ | $0.67$ | $0.53$ | $0.29$ | $0.24$ | $0.31$ | $0.54$ | $0.13$ | $0.54$ | $0.83$ | $0.83$
ARTEMIS [7] | 24.9 | $0.24$ | $0.24$ | $0.20$ | $0.38$ | $0.26$ | $0.65$ | $0.60$ | $0.36$ | $0.25$ | $0.38$ | $0.55$ | $0.14$ | $0.63$ | $0.86$ | $0.87$
FashionViL [13] | 23.4 | $0.26$ | $0.28$ | $0.25$ | $0.40$ | 0.31 | 0.82 | 0.67 | $0.33$ | $0.31$ | $0.34$ | 0.70 | $0.15$ | 0.86 | 1.09 | 1.06
CLIP4CIR [2] | 35.9 | 0.44 | 0.42 | 0.44 | 0.54 | $0.21$ | $0.72$ | $0.50$ | 0.46 | 0.43 | 0.60 | 0.70 | 0.22 | $0.37$ | $0.74$ | $0.83$

##  4 Results and Analysis

###  4.1 Natural corruption analysis

Figure 3: Models average performance in CIRR under 15 vision corruptions.
Left: Recall vs. rank K. Middle: $\gamma$ vs. rank K. Right: $\gamma$ vs.
recall@10, circle size indicates number of model parameters.

In order to evaluate whether the text-image composed retrieval models are
robust under natural corruptions, we perform experiments under 15 visual
corruptions which can be further categorized into noise, blur, weather,
digital corruptions on both fashion domain and open domain. We report the
relative robustness $\gamma$ under the highest severity of each natural visual
corruption in Table 1, which shares the same trend as other corruption
severities. To evaluate the robustness against textual corruption, we conduct
experiments under 7 textual corruptions on text-image composed retrieval
models in character-level and word-level. Analysis about textual corruption is
shown in supplementary C.1.

Pretraining. Among the compared models, FashionViL CIRPLANT and CLIP4CIR are
pretrained on large datasets, with respective sizes of 1.35 million, 6.5
million, and 400 million image-text pairs, while another three compared models
are based on ImageNet pretrained ResNet50 as image encoder and random
initialized LSTM as text encoder. As shown in Figure 3, the models with large
pretrained datasets consistently show better robustness in both open domain
and fashion domain. _This implies that models with large pretrained datasets
may result in better robustness against visual corruptions, which is in
alignment with the statement from Paul et al.[31]._

Bottleneck of robustness. We visualize the recall performance with rank K
improvement on the left of Figure 3 and relative robustness on the right. With
ranking $K$ improved, both the recall and relative robustness improved for all
of the models. Additionally, we find out in Figure 3 that ResNet50-based
image-only search shows even better accuracy as well as robustness compared
with TIRG, ARTEMIS, and MAAF, which utilizes the same image encoder ResNet50.
According to the three foundations of text-image composed retrieval in Sec. 3,
both the image encoder and modality fusion module can be vulnerable to vision
corruption. We observe that the ResNet50 backbone shows relatively high
robustness while the modality fusion of the three models (TIRG, MAAF, and
ARTEMIS) exacerbates the instability of the model. However, this phenomenon is
not applicable to the CLIP feature, whose text and image embedding are aligned
in a unified space in the pretraining process. Comparing Image-only (CLIP) and
CLIP4CIR, which query with CLIP image embedding and CLIP text-image composed
embedding respectively, we can find out CLIP4CIR consistently performs better
recall performance as well as robustness in Figure 3. _Thus, we speculate that
text features from aligned space can help boost the robustness, while text
features from independent space will damage the model robustness._

###  4.2 Robustness against text understanding

In this section, we analyze model reasoning ability through variation of
modified text on numerical variation, attributes variation, object removal,
background variation, and fine-grained variation, which are supplied by our
proposed CIRR-D dataset. We take the 4181 queries from the origin CIRR dataset
and evaluate them in our expanded CIRR-D dataset as a baseline, which
incorporates diverse reasoning instruction and can represent the models’
average performance across various instructions. However, unlike other
function detection, the gallery set for fine-grained is a subset, which is
composed of six highly similar images following [25]. We first evaluate
performance on each query type as shown in Table 2. Detailed analyses are
discussed below.

Numerical variation. To probe the ability of numerical variation, the modified
text contains either a precise value of the number from zero to ten or an
estimated value by comparison like ‘ _reduce/increase the number_ ’. Comparing
numerical specific query with CIRR query as shown in Table 2, we can not
observe significant variation which may result from the long-tailed
distribution. Namely, the numerical set has a large number of samples in the
range of 1 to 3, while a small number of samples in the range of 4 to 10. More
analysis can be found in the supplementary E. For now, we speculate that
_numerical modification may not be the bottleneck of the current text-image
composed retrieval._

Table 2: Recall of CIRR-D dataset. The red and green arrows indicate the performance increase or decrease compared with CIRR queries. Bold and underline are the largest decrease and increase.  | R@5 | Rsub@1
---|---|---
| CIRR | Numerical | Attribute | Removal | Background | Fine grained
Image-only(RN50) | 31.55 |  31.47 $\downarrow$ (0.08) |  32.57 $\uparrow$ (1.02) |  35.99 $\uparrow$ (4.44) |  39.15 $\uparrow$ (7.60) | 20.25
Image-only(CLIP) | 22.51 |  24.80 $\uparrow$ (2.29) |  29.09 $\uparrow$ (6.58) |  27.90 $\uparrow$ (5.39) |  25.64 $\uparrow$ (3.13) | 20.02
Text-only | 39.02 |  42.84 $\uparrow$ (3.82) |  49.45 $\uparrow$ (10.43) |  11.62 $\downarrow$ (27.4) |  11.62 $\downarrow$ (27.4) | 53.73
TIRG [40] | 36.35 |  39.64 $\uparrow$ (3.29) |  37.77 $\uparrow$ (1.42) |  30.41 $\downarrow$ (5.94) |  32.82 $\downarrow$ (3.53) | 35.90
MAAF [8] | 32.19 |  32.53 $\uparrow$ (0.34) |  35.57 $\uparrow$ (3.38) |  31.09 $\downarrow$ (1.10) |  34.27 $\uparrow$ (2.08) | 28.63
ARTEMIS [40] | 40.05 |  39.56 $\downarrow$ (0.49) |  42.68 $\uparrow$ (2.63) |  33.26 $\downarrow$ (6.79) |  35.56 $\downarrow$ (4.49) | 40.80
CIRPLANT [25] | 48.82 |  45.07 $\downarrow$ (3.75) |  47.73 $\downarrow$ (1.09) |  41.12 $\downarrow$ (7.70) |  45.98 $\downarrow$ (2.84) | 38.19
CLIP4CIR [2] | 62.94 |  64.18 $\uparrow$ (1.24) |  69.15 $\uparrow$ (6.21) |  31.66 $\downarrow$ (31.28) |  41.88 $\downarrow$ (21.06) | 62.66

Attributes variation. To evaluate the model’s discriminative ability when
querying elementary attributes, the modified text includes variations of
color, shape and size. As observed in Table 2, all of the methods (except
CIRPLANT) achieve higher performance with attribute queries than with CIRR
queries. Additionally, the performance of CLIP based image-only model and
CLIP4CIR have an obvious increment of over 6% compared with their performance
with CIRR queries, which have a strong ability of attribute recognition
including color, shape, and size. This implies that _attribute is the one of
main focuses during training and models gain strong attribute discriminative
ability._

Object removal. Object removal is a convenient approach to describe the
differences between images but is universally overlooked by current methods in
text-image composed retrieval. To probe the ability of object removal through
CIRR-D, the modified text of the query explicitly contains the word
_’remove’_. As shown in Table 2, all of the five compared methods achieve
their lowest performance in object removal with an average decrement of 10.6%
compared with the CIRR query. In particular, CLIP4CIR has a drop of over 30%,
which may be a result of its static pretraining process by aligning only image
text pairs without comparison between images. Surprisingly, image-only methods
can have an increment over CIRR queries, which illustrates that visual
similarity can boost the robustness over object removal but the text condition
over guidance the model decision. This aligns with the foundation of the task:
images are dense and continuous while text is sparse and discrete. _In the
case of object removal, text guidance expands the possibility of the targets,
which distracts the model and results in lower performance._

Background variation. To probe the robustness against background
modifications, the modified text of the query explicitly includes the word
"background". We observe a similar phenomenon as in object removal, where the
performance of compared models (except MAAF) decreases but the performance of
image-only models increases compared to CIRR queries. As the CIRR-D sample
visualization in Figure 6, we can observe that the background modification
method is limited such as changing the background color or making the
background blur, which can lead to unrelated targets by relying solely on the
text itself. We further speculate that _a modified text leading to more
satisfactory candidates may result in impaired outcomes._

Fine-grained variation. To probe the fine-grained variation discriminative
ability, we utilize the subset in the CIRR dataset, where each image is
retrieved from its subset composed of another five highly similar images. As
the gallery is different from the above reasoning function, the recall cannot
be compared with CIRR query performance directly. We can observe from Table 2
that image-only models perform similarly to random guessing and text-only by
CLIP embedding can achieve an acceptable result. Among the five compared
methods, TIRG achieve the lowest performance which hypothesizes the slight
adjustment in visual space is sufficient rather than exploring text deeply and
establishing a fusion space. This phenomenon indicates that it is difficult to
distinguish between two states in continuous visual space. In contrast, text
can precisely define subtle differences due to its discrete nature. We also
speculate that _a modified text offers accurate information while minimizing
the number of feasible targets can enhance the model’s discriminative
ability._

##  5 Conclusion

In this work, we proposed three robustness benchmarks for text-image composed
retrieval including two for natural corruption in both image and text and one
for probing textual understanding. Concretely, we first introduced two
benchmark datasets, CIRR-C and FashionIQ-C with natural corruption (both image
and text) in the open domain and fashion domain respectively. Further, we
create benchmark CIRR-D to assess the text understanding including number,
attribute, object removal, background, and fine-grained variation. Based on
our observation, we provide the following suggestions to enhance model
robustness in text-image composed retrieval: 1) model pretrained on large
datasets with little distribution shift will lead to better robustness, 2)
text features from aligned space can help boost the robustness, while text
features from independent space will damage the model robustness, 3) a
modified text is more likely to enhance the model’s discriminative ability
when it minimizes the number of feasible targets and will distract the model
when it leads to more satisfactory candidates. These findings can potentially
boost the robustness of text-image composed retrieval in the future.

## References

  * [1] Vedika Agarwal, Rakshith Shetty, and Mario Fritz.  Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9690–9698, 2020.
  * [2] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo.  Effective conditioned and composed image retrieval combining clip-based features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21466–21474, 2022.
  * [3] Tamara L Berg, Alexander C Berg, and Jonathan Shih.  Automatic attribute discovery and characterization from noisy web data.  In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 11, pages 663–676. Springer, 2010.
  * [4] Madeline Chantry, Shruti Vyas, Hamid Palangi, Yogesh Rawat, and Vibhav Vineet.  Robustness analysis of video-language models against visual and language perturbations.  Advances in Neural Information Processing Systems, 35:34405–34420, 2022.
  * [5] Yanbei Chen, Shaogang Gong, and Loris Bazzani.  Image search with text feedback by visiolinguistic attention learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3001–3011, 2020.
  * [6] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein.  Robustbench: a standardized adversarial robustness benchmark.  arXiv preprint arXiv:2010.09670, 2020.
  * [7] Ginger Delmas, Rafael Sampaio de Rezende, Gabriela Csurka, and Diane Larlus.  Artemis: Attention-based retrieval with text-explicit matching and implicit similarity.  arXiv preprint arXiv:2203.08101, 2022.
  * [8] Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, and Kofi Boakye.  Modality-agnostic attention fusion for visual search with text feedback.  arXiv preprint arXiv:2007.00145, 2020.
  * [9] Samuel Dooley, George Z Wei, Tom Goldstein, and John Dickerson.  Robustness disparities in face detection.  Advances in Neural Information Processing Systems, 35:38245–38259, 2022.
  * [10] Sonam Goenka, Zhaoheng Zheng, Ayush Jaiswal, Rakesh Chada, Yue Wu, Varsha Hedau, and Pradeep Natarajan.  Fashionvlp: Vision language transformer for fashion retrieval with feedback.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14105–14115, 2022.
  * [11] Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun.  Compodiff: Versatile composed image retrieval with latent diffusion.  arXiv preprint arXiv:2303.11916, 2023.
  * [12] Xiao Han, Sen He, Li Zhang, Yi-Zhe Song, and Tao Xiang.  Uigr: unified interactive garment retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2220–2225, 2022.
  * [13] Xiao Han, Licheng Yu, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang.  Fashionvil: Fashion-focused vision-and-language representation learning.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 634–651. Springer, 2022.
  * [14] Xiao Han, Xiatian Zhu, Licheng Yu, Li Zhang, Yi-Zhe Song, and Tao Xiang.  Fame-vil: Multi-tasking vision-language model for heterogeneous fashion tasks.  arXiv preprint arXiv:2303.02483, 2023.
  * [15] Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis.  Automatic spatially-aware fashion concept discovery.  In Proceedings of the IEEE international conference on computer vision, pages 1463–1471, 2017.
  * [16] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  Proceedings of the International Conference on Learning Representations, 2019.
  * [17] Phillip Isola, Joseph J Lim, and Edward H Adelson.  Discovering states and transformations in image collections.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1383–1391, 2015.
  * [18] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  * [19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  arXiv:2304.02643, 2023.
  * [20] Seungmin Lee, Dongwan Kim, and Bohyung Han.  Cosmo: Content-style modulation for image retrieval with text feedback.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 802–812, 2021.
  * [21] Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski.  Data roaming and early fusion for composed image retrieval.  arXiv preprint arXiv:2303.09429, 2023.
  * [22] Linjie Li, Zhe Gan, and Jingjing Liu.  A closer look at the robustness of vision-and-language pre-trained models.  arXiv preprint arXiv:2012.08673, 2020.
  * [23] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [24] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [25] Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould.  Image retrieval on real-life images with pre-trained vision-and-language models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2125–2134, 2021.
  * [26] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna.  Crepe: Can vision-language foundation models reason compositionally?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910–10921, 2023.
  * [27] Long Mai, Hailin Jin, Zhe Lin, Chen Fang, Jonathan Brandt, and Feng Liu.  Spatial-semantic image search by visual feature synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4718–4727, 2017.
  * [28] Vikramjit Mitra, Horacio Franco, Richard M Stern, Julien Van Hout, Luciana Ferrer, Martin Graciarena, Wen Wang, Dimitra Vergyri, Abeer Alwan, and John HL Hansen.  Robust features in deep-learning-based speech recognition.  New Era for Robust Speech Recognition: Exploiting Deep Learning, pages 187–217, 2017.
  * [29] Andrei Neculai, Yanbei Chen, and Zeynep Akata.  Probabilistic compositional embeddings for multimodal image retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4547–4557, 2022.
  * [30] Devi Parikh and Kristen Grauman.  Relative attributes.  In 2011 International Conference on Computer Vision, pages 503–510. IEEE, 2011.
  * [31] Sayak Paul and Pin-Yu Chen.  Vision transformers are robust learners.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2071–2081, 2022.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [34] Barbara Rychalska, Dominika Basaj, Alicja Gosiewska, and Przemysław Biecek.  Models in the wild: On corruption robustness of neural nlp systems.  In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12–15, 2019, Proceedings, Part III 26, pages 235–247. Springer, 2019.
  * [35] Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister.  Pic2word: Mapping pictures to words for zero-shot composed image retrieval.  arXiv preprint arXiv:2302.03084, 2023.
  * [36] Soumya Sanyal, Zeyi Liao, and Xiang Ren.  Robustlr: A diagnostic benchmark for evaluating logical robustness of deductive reasoners.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9614–9631, 2022.
  * [37] Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi.  Foil it! find one mismatch between image and language caption.  arXiv preprint arXiv:1705.01359, 2017.
  * [38] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs.  arXiv preprint arXiv:1811.00491, 2018.
  * [39] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248, 2022.
  * [40] Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays.  Composing text and image for image retrieval-an empirical odyssey.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6439–6448, 2019.
  * [41] Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li.  Adversarial glue: A multi-task benchmark for robustness evaluation of language models.  arXiv preprint arXiv:2111.02840, 2021.
  * [42] Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, et al.  Textflint: Unified multilingual robustness evaluation toolkit for natural language processing.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 347–355, 2021.
  * [43] Xuezhi Wang, Haohan Wang, and Diyi Yang.  Measure and improve robustness in nlp models: A survey.  arXiv preprint arXiv:2112.08313, 2021.
  * [44] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  arXiv preprint arXiv:2303.04671, 2023.
  * [45] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris.  The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback.  CVPR, 2021.
  * [46] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In The Eleventh International Conference on Learning Representations, 2022.
  * [47] Jiehang Zeng, Xiaoqing Zheng, Jianhan Xu, Linyang Li, Liping Yuan, and Xuanjing Huang.  Certified robustness to text adversarial attacks by randomized [mask].  arXiv preprint arXiv:2105.03743, 2021.
  * [48] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre.  A benchmark for compositional visual reasoning.  arXiv preprint arXiv:2206.05379, 2022.

##  Appendix A Creating Benchmark Datasets

We build three benchmark datasets for this work evaluating both natural
corruption (CIRR-C and FashionIQ-C) and textual understanding (CIRR-D). To
evaluate natural corruption with both image and text, we introduce CIRR-C and
FashionIQ-C based on the existing dataset CIRR and FashionIQ. To evaluate
textual understanding including variations of numerical, attributes (colour,
shape and size), object removal, background and fine-grained details, we
introduce CIRR-D by categorizing and expanding CIRR with synthetic images. We
provide raw images and complete code for generating all types of natural
corruptions and the evaluation testbed in our code zip file. For shortcut, we
provide raw image link for CIRR, FashionIQ and CIRR-D. To implement CIRR-D
dataset, both raw images and queries in different categories (numerical,
attribute, object removal, background and fine-grained variations) are
provided directly. To implement CIRR-C and FashionIQ-C dataset, research can
recreate the same benchmark datasets with the following steps:

1\. Download CIRR and FashionIQ raw images with our provided link.

2\. Preprocess image or text with the provided code of image corruption and
text corruption.

3\. Apply the proposed corruptions with our testbed for downstream model
evaluation.

Evaluation settings. In order to ensure the fairness of the evaluation, we
establish a standardized testbed for variant models except FashionViL to unify
the evaluation process. Further to reproduce the original performance, we
implement the official pre-trained weight for FashionViL and CLIP4CIR. We
retrain the model and receive similar results as reported for models MAAF,
TIRG, ARTEMIS, and CIRPLANT. We implement zero-shot evaluation for BLIP2 and
instructBLIP. And train BLIP2-CIR following [2]. In detail, we extend the
existing ARTEMIS code framework to facilitate the convenient interface of
different trained models, where TIRG and ARTEMIS are already implemented. For
image input among these models, CIRPLANT is based on frozen ResNet152 pre-
trained features while other models take raw images as input. We implement a
frozen ResNet152 image encoder so that we can introduce corruption to the raw
image directly. We leverage OPT-2.7b captioner for BLIP2 and vicuna-7b for
instructBLIP. For all the 15 image corruptions, we perform the highest
severity of corruption for obvious performance. For text input among these
models, TIRG, MAAF, and ARTEMIS build the vocabulary based on appearance words
in target evaluation caption, while FashionViL, CIRPLANT, and CLIP4CIR
implement their vocabulary from large pretraining dataset. We implement
textual corruption and change the raw text directly. For a concise
explanation, we show the result of FashionIQ by showing the average of the
three categories: dress, shirt, and toptee.

Implementation details We supply 5 models (TIRG, MAAF, ARTEMIS, CIRPLANT and
CLIP4CIR) in the same testbed to have a fair comparison with different
benchmark datasets. The selection of models or datasets can be easily
accomplished through input parameters. Further models can be implemented in
our testbed by simply providing model structure files with the necessary
interface. In detail, the necessary interfaces include image feature
extraction, text feature extraction, feature composing process and distance
comparison. Our testbed is compatible with two environments and five models.
CIRPLANT is implemented with Python(3.1) and Pytorch(1.8.1). TIRG, MAAF,
CLIP4CIR and ARTEMIS were implemented with Python(3.8) and Pytorch(2.0). All
the experiments are conducted and trained on NVIDIA A100 GPUs. We will
maintain our code for benchmarking and testbed future in GitHub. Any questions
about benchmark creation or evaluation can be raised by an issue on the GitHub
page.

##  Appendix B Sample visualization

###  B.1 CIRR-C visualization

We show the visualization samples from CIRR-C in Figure. 7. Our CIRR-C is
based on the CIRR dataset and implemented with both image corruptions and text
corruptions. We apply 15 standard natural image corruptions, as depicted in
Figure 7 (a), and demonstrate 5 levels of severity using brightness corruption
as an example in Figure 7 (b). We further visualize 7 text corruptions in
Figure 7 (c). For both image and text corruption, humans can easily recognize
them.

###  B.2 FashionIQ-C visualization

FashionIQ-C follows the same natural corruption in both image and text as in
CIRR-C. We show the visualization samples from FashionIQ-C in Figure. 8.
FashionIQ-C is based on the FashionIQ dataset and implemented with both image
corruptions and text corruptions. We apply 15 standard natural image
corruptions, as depicted in Figure 8 (a), and demonstrate 5 levels of severity
using zoom blur corruption as an example in Figure 8 (b). We further visualize
7 text corruptions in Figure 8 (c).

###  B.3 Textual corruption definition

In this work, we implement 7 natural textual corruptions following [34]. The
definition of the textual corruptions are as follows:

  * •

Swap: Randomly shuffles two characters within a word.

  * •

Qwerty: Simulates errors made while writing on a QWERTY-type keyboard.
Characters are swapped for their neighbors on the keyboard

  * •

RemoveChar: Randomly removes characters from words.

  * •

RemoveSpace: Removes a space from text, merging two words.

  * •

Misspelling: Misspells words appearing in the Wikipedia list of commonly
misspelled English words.

  * •

Repetition: Randomly repeat words.

  * •

Homophone: Changes words into their homophones from the Wikipedia list of
common homophones. The list contains around 500 pairs or triples of homophonic
words.

Examples are shown in Figure. 7 for CIRR-C and Figure. 8 for FashionIQ-C
respectively.

###  B.4 CIRR-D visualization

To detect textual understanding ability, we build a CIRR-D dataset with five
different types of queries containing specific instructions to probe five
different abilities The source of the CIRR-D dataset is from the original
CIRR, CIRR extends caption and our generated synthetic images. The triplets
from the original CIRR dataset are normally with obvious variations while the
synthetic triplets are normally following the same structure and only local
variations. Some extended caption from the original CIRR dataset can only
supply partial difference and cannot locate the target images. Therefore, we
manually remove samples and retain only those triplets where the extended
caption can provide sufficient variations. In detail, we visualize numerical
samples in Figure 9, which is composed of triplets from the original CIRR
dataset in Figure 9 (a) and our generated synthetic triplets in Figure.9 (b).
Attribute variation visualization samples are shown in Figure 10, which is
composed of triplets from the original CIRR dataset in Figure 10 (a) and our
generated synthetic triplets in Figure. 10 (b). To evaluate object removal
ability, the triplet source consists of three aspects. We visualize object
removal triplets from the original CIRR dataset in Figure. 11 (a), extended
caption triplets in Figure. 11 (b) and our generated synthetic triplets in
Figure. 11 (c). we visualize background variations samples in Figure 12, which
is composed of triplets from original CIRR dataset in Figure 12 (a) and from
extended captions in Figure.12 (b). The evaluation of fine-grained variations
follows the original CIRR dataset, whose gallery set is composed of 5 highly
similar images as shown in Figure. 13.

Table 3: Details of CIRR-D dataset. The first column is the number of images. The rest columns contain the number of triplets for five probing abilities. | Images | Numerical | Attribute | Removal | Background | Fine-grained
---|---|---|---|---|---|---
Val. | $2297$ | $820$ | $1397$ | $233$ | $358$ | $4181$
Extend caption | - | - | - | $505$ | $812$ | -
Synthetic | $1245$ | $305$ | $700$ | $140$ | - | -
Total | $3542$ | $1125$ | $2097$ | $878$ | $1170$ | $4181$
Table 4: Details of modality fusion for the evaluated models in this study. $R_{i},M_{t},T_{i}$ represent reference image, modified text and target image feature respectively. $C$ donates the composed of the two features. <,> represents cosine similarity.  Model | $C_{RiMt}$ | $C_{RiTi}$ | $C_{MtTi}$ | distance
---|---|---|---|---
TIRG [40] | cat+residual | - | - |  <$C_{RiMt}$, $T_{i}$>
MAAF [8] | self attn+ cross attn | - | - |  <$C_{RiMt}$, $T_{i}$>
Artemis [7] | dot product | dot product | dot product |  <$C_{RiMt}$,$C_{TiMt}$> \+ <$C_{TiMt}$,$M_{t}$>
CIRPLANT [25] | transformer | - | - |  <$C_{RiMt}$, $T_{i}$>
CLIP4CIR [2] | cat + residual | - | - |  <$C_{RiMt}$, $T_{i}$>
FashionViL [13] | transformer | - | - |  <$C_{RiMt}$, $T_{i}$>
Table 5: Relative robustness score for text-image composed retrieval under 7
natural text corruptions in CIRR-C recall@10 and FashionIQ-C recall@10 on
average of three categories. Recall@10 performance under clean conditions on
the left. Bold are the highest relative robustness among the five compared
methods.

|  | Character | Word
---|---|---|---
CIRR-C | Clean | Swap | QWERTY | RemoveChar | RemoveSpace | Misspelling | Repetition | Homophone
Text-only | 51.2 | $0.75$ | $0.74$ | $0.78$ | $1.0$ | $0.99$ | $0.98$ | $0.92$
TIRG [40] | 55.1 | $0.77$ | $0.76$ | $0.80$ | $1.0$ | $0.98$ | $1.0$ | $0.89$
MAAF [8] | 49.9 | 0.95 | 0.97 | 0.96 | 1.0 | 1.0 | 1.0 | 0.97
ARTEMIS [7] | 59.0 | $0.61$ | $0.58$ | $0.65$ | 1.0 | $0.98$ | $0.98$ | $0.82$
CIRPLANT [25] | 68.8 | $0.92$ | $0.93$ | $0.93$ | 1.0 | 1.0 | 1.0 | 0.97
CLIP4CIR [2] | 80.3 | $0.89$ | $0.89$ | $0.90$ | 1.0 | 1.0 | $0.99$ | 0.97

|  | Character | Word
---|---|---|---
FashionIQ-C | Clean | Swap | QWERTY | RemoveChar | RemoveSpace | Misspelling | Repetition | Homophone
TIRG [40] | 23.8 | $0.26$ | $0.20$ | $0.29$ | $0.66$ | $0.63$ | $0.61$ | $0.52$
MAAF [8] | 23.4 | $0.40$ | $0.39$ | $0.39$ | $0.70$ | $0.68$ | $0.68$ | $0.62$
ARTEMIS [7] | 24.9 | $0.25$ | $0.20$ | $0.31$ | $0.70$ | $0.67$ | $0.67$ | $0.55$
FashionViL [13] | 23.4 | 0.55 | 0.59 | 0.60 | 0.86 | 0.84 | 0.85 | 0.76
CLIP4CIR [2] | 35.9 | $0.52$ | $0.51$ | $0.54$ | $0.71$ | $0.70$ | $0.69$ | $0.67$

##  Appendix C More experiments result

###  C.1 Textual robustness against natural corruption

Comparing the relative robustness against textual corruption in Table 5 and
visual corruption in Table 1, we can observe that the robustness is higher
against textual corruptions. Among the compared models, MAAF and FashionViL
show the highest robustness in open domain and fashion domain respectively.
This aligns with the findings we discovered regarding vision corruptions,
where large pretrained models (FashionViL with fashion-specific pretraining)
and models with sufficient modality fusion result to higher robustness.
Additionally, comparing CLIP4CIR and the Text-only retrieval method
implementing CLIP text embedding, we find out the robustness can be boosted
after fusion with vision modality. However, with images corrupted, CLIP4CIR
shows lower robustness than text-only model, from which _we speculate that
aligned clean image feature can boost the robustness, while the corrupted
image feature will impair robustness._

###  C.2 Fine grained subset analysis of CIRR-C

In this section, we supplement some experimental results. As shown in Figure.
4, we visualize the recall performance on the CIRR-C subset. Comparing the
subset retrieval with the whole gallery in CIRR-C (shown in the main paper
Figure.3), we can observe that subset relative robustness (range from 0.6 to
0.9) overall is higher than the whole set (range from 0.4 to 0.8). This result
suggests that a smaller gallery can lead to more stable retrieval. In essence,
the overall trend aligns with retrieval on all images: CLIP4CIR consistently
performs the best, while IMAGE-ONLY with CLIP embedding consistently exhibits
the worst retrieval performance.

Figure 4: Models average performance in CIRR subset under 15 vision
corruptions. Left: Recall vs. rank K. Right: Relative robusentess vs. rank K.

###  C.3 Subcategories analysis of FashionIQ-C

For detailed results in the FashionIQ-C dataset, we report the results on the
three categories, namely dress, shirt and toptee respectively, as shown in
Table. 6. Overall, a similar trend is observed across the three categories,
with FashionViL and CLIP4CIR consistently exhibiting the highest relative
robustness. In the shirt category, overall robustness tends to be slightly
higher than in the dress and toptee categories. We further report the
recall@10 performance on FashionIQ-C dataset as shown in Table. 7. By
comparing the relative robustness in Table. 6 and corresponding recall
performance in Table. 7, we can find out higher robustness doesn’t mean higher
recall performance. As according to the definition of relative robustness:
$\gamma=1-\left(R_{c}-R_{p}\right)/R_{c}$ following [16], lower recall
performance under clean condition $R_{c}$ will lead to higher relative
robustness $\gamma$.

Table 6: Relative robustness score for text-image composed retrieval under 15
natural image corruptions in FashionIQ-C Recall@10 for dress, shirt and toptee
respectively. Bold is the highest relative robustness for the five composed
retrieval methods.

| Noise | Blur | Weather | Digital
---|---|---|---|---
FashionIQ-C Dress | Gauss. | Shot | Impluse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contrast | Elastic | Pixel | JPEG
TIRG [40] | 0.21 | 0.18 | 0.17 | 0.37 | 0.22 | 0.64 | 0.58 | 0.33 | 0.25 | 0.35 | 0.63 | 0.12 | 0.62 | 0.82 | 0.85
MAAF [8] | 0.30 | 0.24 | 0.22 | 0.42 | 0.19 | 0.65 | 0.56 | 0.28 | 0.21 | 0.32 | 0.58 | 0.10 | 0.54 | 0.78 | 0.81
ARTEMIS [7] | 0.23 | 0.22 | 0.18 | 0.38 | 0.24 | 0.66 | 0.62 | 0.39 | 0.26 | 0.37 | 0.59 | 0.14 | 0.67 | 0.85 | 0.9
FashionViL [13] | 0.21 | 0.22 | 0.23 | 0.38 | 0.34 | 0.84 | 0.72 | 0.29 | 0.29 | 0.3 | 0.79 | 0.13 | 0.88 | 1.1 | 1.1
CLIP4CIR [2] | 0.44 | 0.38 | 0.44 | 0.54 | 0.24 | 0.74 | 0.52 | 0.41 | 0.36 | 0.55 | 0.68 | 0.16 | 0.42 | 0.75 | 0.82

| Noise | Blur | Weather | Digital
---|---|---|---|---
FashionIQ-C Shirt | Gauss. | Shot | Impluse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contrast | Elastic | Pixel | JPEG
TIRG [40] | 0.33 | 0.32 | 0.27 | 0.28 | 0.20 | 0.57 | 0.54 | 0.32 | 0.28 | 0.37 | 0.51 | 0.15 | 0.60 | 0.86 | 0.81
MAAF [8] | 0.33 | 0.30 | 0.27 | 0.46 | 0.20 | 0.67 | 0.50 | 0.30 | 0.27 | 0.34 | 0.47 | 0.16 | 0.57 | 0.84 | 0.79
ARTEMIS [7] | 0.27 | 0.28 | 0.25 | 0.39 | 0.26 | 0.62 | 0.61 | 0.36 | 0.24 | 0.38 | 0.54 | 0.16 | 0.61 | 0.84 | 0.88
FashionViL [13] | 0.29 | 0.34 | 0.26 | 0.38 | 0.26 | 0.77 | 0.6 | 0.33 | 0.32 | 0.37 | 0.63 | 0.17 | 0.83 | 1.09 | 1.02
CLIP4CIR [2] | 0.47 | 0.48 | 0.45 | 0.50 | 0.18 | 0.65 | 0.48 | 0.51 | 0.50 | 0.65 | 0.71 | 0.27 | 0.31 | 0.69 | 0.82

| Noise | Blur | Weather | Digital
---|---|---|---|---
FashionIQ-C Toptee | Gauss. | Shot | Impluse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contrast | Elastic | Pixel | JPEG
TIRG [40] | 0.30 | 0.28 | 0.25 | 0.36 | 0.24 | 0.63 | 0.58 | 0.32 | 0.27 | 0.39 | 0.58 | 0.10 | 0.69 | 0.88 | 0.88
MAAF [8] | 0.30 | 0.28 | 0.27 | 0.45 | 0.24 | 0.71 | 0.52 | 0.28 | 0.23 | 0.28 | 0.56 | 0.14 | 0.52 | 0.88 | 0.88
ARTEMIS [7] | 0.21 | 0.23 | 0.18 | 0.37 | 0.28 | 0.68 | 0.57 | 0.33 | 0.25 | 0.38 | 0.53 | 0.13 | 0.66 | 0.88 | 0.82
FashionViL [13] | 0.28 | 0.28 | 0.27 | 0.44 | 0.32 | 0.85 | 0.69 | 0.38 | 0.33 | 0.36 | 0.69 | 0.15 | 0.88 | 1.09 | 1.06
CLIP4CIR [2] | 0.42 | 0.4 | 0.42 | 0.58 | 0.21 | 0.76 | 0.49 | 0.46 | 0.44 | 0.60 | 0.71 | 0.24 | 0.39 | 0.78 | 0.84

Table 7: Recall@10 for text-image composed retrieval under 15 natural image corruptions in FashionIQ-C. | Noise | Blur | Weather | Digital
---|---|---|---|---
FashionIQ-C | Clean | Gauss. | Shot | Impluse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contrast | Elastic | Pixel | JPEG
TIRG [40] | 23.8 | 6.6 | 6.1 | 5.4 | 8.1 | 5.3 | 14.6 | 13.5 | 7.7 | 6.3 | 8.8 | 13.8 | 3.0 | 15.2 | 20.3 | 20.2
MAAF [8] | 23.4 | 7.2 | 6.4 | 5.9 | 10.4 | 5.0 | 15.8 | 12.3 | 6.6 | 5.5 | 7.3 | 12.7 | 3.1 | 12.7 | 19.4 | 19.4
ARTEMIS [7] | 24.9 | 5.8 | 6.0 | 4.9 | 9.4 | 6.5 | 16.4 | 14.9 | 9.0 | 6.2 | 9.4 | 13.9 | 3.5 | 16.2 | 21.4 | 21.5
FashionViL [13] | 23.4 | 6.1 | 6.5 | 5.9 | 9.5 | 7.2 | 19.3 | 15.8 | 7.8 | 7.3 | 8.0 | 16.5 | 3.5 | 20.3 | 25.7 | 24.9
CLIP4CIR [2] | 35.9 | 15.9 | 15.2 | 15.6 | 19.4 | 7.5 | 25.7 | 17.8 | 16.5 | 15.6 | 21.5 | 25.2 | 8.2 | 13.3 | 26.5 | 29.7

###  C.4 Analysis of COCO with image corruptions

To evaluate the compared models on more general domain, we implement our image
corruptions on the validation set of COCO [23], represented by CIRR-C. We set
masked bounding box as the reference image, the raw image as the target image,
and the labels of objects as modified text the following [29, 35]. The three
compared models are trained on the CIRR dataset and evaluated on the
validation set of COCO with 5000 images. The results show that large
pretrained model CLIP4CIR have higher robustness than smaller models TIRG and
ARTEMIS, which follow the same conclusion in paper Section 4.1.

Table 8: Relative robustness score for text-image composed retrieval under 15 natural image corruptions in COCO-C Recall@10. | Noise | Blur | Weather | Digital
---|---|---|---|---
COCO-C | Gauss. | Shot | Impluse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contrast | Elastic | Pixel | JPEG
TIRG [40] | 0.19 | 0.21 | 0.14 | 0.42 | 0.25 | 0.62 | 0.58 | 0.35 | 0.21 | 0.51 | 0.89 | 0.05 | 0.40 | 0.40 | 0.72
ARTEMIS [7] | 0.14 | 0.16 | 0.08 | 0.43 | 0.22 | 0.72 | 0.52 | 0.41 | 0.32 | 0.45 | 1.06 | 0.05 | 0.40 | 0.48 | 0.70
CLIP4CIR [2] | 0.52 | 0.58 | 0.52 | 0.65 | 0.12 | 0.85 | 0.36 | 0.51 | 0.49 | 0.71 | 0.90 | 0.10 | 0.24 | 0.77 | 0.77

##  Appendix D More Related Works

Diagnostic analysis. Recently, a range of benchmarks for visual understanding
have been proposed, including datasets for image captioning [37], visual
question answering [18], visual reasoning [48] and visio-linguistic
compositional reasoning [39, 46, 26].  For text-image composed retrieval, the
benchmarks can be categorized into sythetic-based datasets by cubes [40] or
natural scenes [11], fashion-based datasets [15, 3, 45], object-state dataset
[17] and open domain dataset [25]. Among them, the majority of the textual
descriptions are limited by predefined attributes [15, 40, 17]. To overcome
this limitation, FashionIQ [45] and CIRR [25] leverage the flexibility of
natural language and becomes the most widely used benchmarks in fashion domain
and open domain respectively. We expand and categorize the test set of CIRR
benchmark from its main and unused extended annotation to probe specific text
understanding in numerical variation, attribute variation, object removal,
background variation, and fine-grained variation. Similar to ours, many
diagnostic datasets are also synthetic ones. CLVER [18] is a synthetic dataset
to probe elementary vision reasoning including color, shape, and spatial
relationships. CVR [48] generates irregular shape, location, color, etc, and
designed for detecting the outlier from a small set of generated images.
However, these are all simulated images and are not generated by imitating
natural scenes. CasualVQA [1] and CompoDiff [11] generate images imitating
natural scenes. However, CasualVQA is designed for visual question answering
tasks and the generated images include noticeable artifacts. While ComoDiff is
designed for text-image composed retrieval, but they generate by replacing the
objects (noun) only instead of attributes (numeral, adjective, manipulation
instruction) like ours, so cannot pinpoint the target reasoning abilities.  To
detect the compositional abilities, visio-linguistic compositional reasoning
diagnostic benchmarks supply variation of objects [26, 46], attributes [26,
46], or text order [46, 39]. However, these benchmarks only supply image-text
pairs for single-modality queries instead of image-text-image triplets for
multi-modality queries. Also their text composition [26, 46] may not have
corresponding image like ‘grass eat horse’. In comparision, our CIRR-D
supplies the first visio-linguistic composition reasoning benchmark for text-
image composed retrivel task in natural scenes.

Text-image composed retrieval. Composed image retrieval aims to retrieve the
target image, where the input query is specified in the form of an image plus
other interactions, such as relative attribute[30], natural language[5, 40],
spatial layout[27], to describe the desired modifications. Among them, natural
language as the most pervasive interaction between humans and computers to
convey intricate specification has attracted increasing attention, which has
often led ‘composed image retrieval’ to become interchangeable with ‘text-
guided image retrieval’ in the literature. We term the task as text-image
composed retrieval to clarify the composition of the query. Traditional text-
image composed retrieval models implement separate independent image and text
encoders, whose features are combined with late fusion. For example, TIRG [40]
and Artemis [7] implement separate pre-trained ResNet as image encoder and
LSTM as text encoder. Until recently, with the power of unified multimodal
space CLIP [32], current text-image composed retrieval models achieved a
noticeable improvement. For example, CLIP4CIR [2] implements a light adapter
as image-text late fusion and further tune it in target domains. Further based
on CLIP, FAME [14] and CASE [21] separately implement early cross attention
between text and image, which shows obvious improvement.

##  Appendix E Limitation

Figure 5: CIRR-D distribution for attribute variants and numerical variations.

We discuss the limitation of the proposed benchmarks in this section. For
benchmarking natural corruption in CIRR-C and FashionIQ-C, the method of
simulating real-world corruption with the noise still has limitations. For
benchmarking textual understanding in CIRR-D, it has long-tail distribution.
As shown in Figure. 5, both numerical and attribute evaluation set follows the
long-tail distribution. The numerical set has a large number of samples in the
range of 1 to 3, while each category from 4 to 10 has only a small number of
samples. The attribute evaluation set has a large number of samples with
colour variations and a small number of samples with size variations. The
imbalanced distribution can lead to bias towards the categories with more
data. Further, we visualise the performance of the query with number one to
three, four to ten respectively shown in Figure 6 left. The average recall@5
of five evaluated methods are 43.06% on number one to three, 42.36% on numbers
four to ten and 44.2% on number one to ten respectively. (A sentence with
multiple numbers will be categorized to multiple categories, thus number one
to three and number four to ten can overlap.) Based on this subtle accuracy
change, we speculate that the model also possesses a similar capability for
recognizing the less frequent samples (number four to ten) in the long-tail
distribution as it does for the more frequent samples (number one to three).

Figure 6: Left: Recall@5 on CIRR-D numerical queries. Right: Samples
visualization of CIRR-D. The left and right images are the reference and
target images. Except the upper left triplet, rest target images are
synthetic.

##  Appendix F License

All the models in this study are available to the public. The model code for
TIRG [40] and MAAF [8] have the Apache License Version 2.0, ARTEMIS [7] has CC
BY-NC-SA 4.0 License, CIRPLANT [25] has MIT license and FashionViL [13] has
BSD License. We will provide CIRR-C, FashionIQ-C and CIRR-D publicly. These
datasets are based on existing CIRR [25] and FashionIQ [45]. For CIRR-C and
FashionIQ-C, we didn’t add any new images or text sources. For CIRR-D, we
further generate synthetic images and text to expand the original CIRR
dataset. All of these datasets are available to the public and we apply
similar licenses to our testbed code and our proposed benchmarks.

(a) Sample visualization with 15 standard image corruptions.

(b) Sample visualization of brightness corruption with 5 severities.

(c) Sample visualization of 7 text corruptions.

Figure 7: CIRR-C sample visualization: (a) 15 standard image corruptions, (b)
5 severities of brightness corruption and (c) 7 text corruption.

(a) Sample visualization with 15 standard image corruptions.

(b) Sample visualization of zoom blur corruption with 5 severities.

(c) Sample visualization of 7 text corruptions.

Figure 8: fashionIQ-C sample visualization: (a) 15 standard image corruptions,
(b) 5 severities of zoom blur corruption and (c) 7 text corruptions.

(a) Numerical queries from original CIRR dataset, four triplets are included.

(b) Our generated numerical queries, eight triplets are included.

Figure 9: CIRR-D sample visualization for numerical queries.

(a) Attribute queries from original CIRR dataset, four triplets are included.

(b) Our generated attribute queries, 10 triplets are included.

Figure 10: CIRR-D sample visualization for attribute queries including color,
shape and size.

(a) Object removal queries from original CIRR dataset, four triplets are
included.

(b) Object removal queries from extend captions of original CIRR dataset, four
triplets are included.

(c) Our generated object removal queries, four triplets are included.

Figure 11: CIRR-D sample visualization for object removal queries.

(a) Background queries from original CIRR dataset, four triplets are included.

(b) Background variation queries from extend captions of original CIRR
dataset, four triplets are included.

Figure 12: CIRR-D sample visualization for background variations. Figure 13:
CIRR-D sample visualization for fine-grained variation queries, 2 triplets are
included. The images with a green border are the correct targets, while the
other images are highly similar composing the gallery set.