(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Laboratoire d’imagerie, de vision et d’intelligence
artificielle (LIVIA)
Dept. of Systems Engineering, ETS Montreal, Canada
11email<EMAIL_ADDRESS>
{rafael.menelau-cruz<EMAIL_ADDRESS>
# Bidirectional Multi-Step Domain Generalization
for Visible-Infrared Person Re-Identification

Mahdi Alehdaghi  0000-0001-6258-8362 Pourya Shamsolmoali  0000-0002-0263-1661
Rafael M. O. Cruz  0000-0001-9446-1040 Eric Granger  0000-0001-6116-7945

###### Abstract

A key challenge in visible-infrared person re-identification (V-I ReID) is
training a backbone model capable of effectively addressing the significant
discrepancies across modalities. State-of-the-art methods that generate a
single intermediate bridging domain are often less effective, as this
generated domain may not adequately capture sufficient common discriminant
information. This paper introduces the Bidirectional Multi-step Domain
Generalization (BMDG), a novel approach for unifying feature representations
across diverse modalities. BMDG creates multiple virtual intermediate domains
by finding and aligning body part features extracted from both I and V
modalities. Indeed, BMDG aims to reduce the modality gaps in two steps. First,
it aligns modalities in feature space by learning shared and modality-
invariant body part prototypes from V and I images. Then, it generalizes the
feature representation by applying bidirectional multi-step learning, which
progressively refines feature representations in each step and incorporates
more prototypes from both modalities. In particular, our method minimizes the
cross-modal gap by identifying and aligning shared prototypes that capture key
discriminative features across modalities, then uses multiple bridging steps
based on this information to enhance the feature representation.
Experiments111Our code is provided in the supplementary materials and will be
made public. conducted on challenging V-I ReID datasets indicate that our BMDG
approach outperforms state-of-the-art part-based models or methods that
generate an intermediate domain from V-I person ReID.

.

###### Keywords:

Visible-Infrared Person Re-Identification Gradual Domain Generalization Cross-
Modal Image Retrieval

##  1 Introduction

Visible-infrared ReID (V-I ReID) is a variant of person ReID that involves
matching individuals across RGB and IR cameras. V-I ReID is challenging since
it requires matching individuals with significant differences in appearance
between V and I modality images. In this context, V-I ReID systems must train
a discriminant feature extraction backbone to encode consistent and
identifiable attributes in RGB and IR images. Most state-of-the-art methods
seek to learn global representations from the whole image by alignment at the
image-level [38, 21] or feature-level [53, 49, 62, 33]. Fig. 1(a) illustrates
the global representation approaches. Some other methods extract global
modality-invariant features by disentangling them from modality-specific
information [8, 48]. Global feature-based approaches cannot compare local
details, resulting in the loss of important discriminant cues about the
person. To address this issue and focus on the unique information in different
body regions, part-based approaches (see Fig. 1(b)) extract local part-based
feature representations through horizontal stripes, clustering, or attention
mechanisms of spatially extracted features [52, 27, 18, 46]. However, given
the significant domain discrepancies between V and I modalities and the lack
of attention to these gaps at the part-level features, these methods tend to
learn modality-specific attributes for each part, often leading to less
discriminative feature representations.

(a) Global representation.

(b) Part-based representation.

(c) Intermediate modality.

(d) Our bidirectional multi-step domain generalization.

Figure 1: A comparison of training architectures for V-I ReID. Approaches
based on (a) a global features representation, and (b) a local part-based
representation to preserve locality and global features. (c) Approaches based
on an intermediate modality generate an intermediate bridging domain. (d) Our
BMGD extracts and combines prototypes from modalities in each step to
gradually create multiple intermediate bridging domains.

Some recent methods leverage an intermediate modality to guide the training
process such that the cross-modal domain gap is reduced [22, 54, 2, 43, 58] as
shown in Fig. 1(c). These intermediate modalities may be static (e.g., gray-
scale) images [54, 2], or synthesized images generated from the V or I
modality [22, 43]. Although the resulting intermediate representations can
leverage shared relations between I and V images and improve matching
accuracy, the created images tend to contain information that is common in the
background and lose discriminative ID information in the foreground. Moreover,
these one-step training strategies only create one intermediate space between
V and I, preventing them from controlling differences w.r.t. the primary
modalities to reduce the modality gap. To effectively bridge the large cross-
modal domain gap, we introduce a gradual, multi-step transformation that can
preserve more ID-related information by controlling the amount of modality-
specific information. Moreover, generating consecutive images results in
degradation due to the loss of discriminative details accumulated over
successive generations, as the images cannot provide useful information in the
next steps. Therefore, our proposed method creates multiple intermediate
domains in feature space by leveraging and aligning ID-informative subset of
features from both modalities and then combining them.

This paper introduces a novel Bidirectional Multi-step Domain Generalization
(BMDG) training strategy for person V-I ReID (see Fig. 1(d)). At first, it
learns to align V and I modalities and then creates multiple virtual
intermediate domains between them to gradually reduce the domain gap for a
backbone model in the training process. Our training strategy comprises two
parts. The aligning learning module based on part-prototype222We define a part
prototype as a feature representation corresponding to a specific body part in
a cropped person image. discovery, which extracts discriminative sub-features
linked to body parts in V and I modality images. Then, the bidirectional
multi-step learning progressively generates auxiliary intermediate feature
representations by combining prototypes from both modalities at each step. For
ID-aware and robust alignment between modalities, the extracted prototypes
must meet three conditions: (1) they should be complementary with respect to
other parts, (2) they should be interchangeable represent consistent parts,
and (3) they should be discriminant for person matching. So, hierarchical
contrastive learning is proposed to train the alignment module to extract
these properties and without affecting the ReID accuracy. To leverage common
semantic information between modalities and discriminant information among
people, our method creates intermediate domains by mixing a growing number of
body part-prototypes extracted from I and V modalities. The conditions
guarantee that mixing these part-prototypes does not duplicate or degrade ID-
related information of the original images.

Bidirectional multi-step learning relies on auxiliary intermediate feature
spaces that are created at each step, by progressively mixing semantically
consistent prototypes from each modality. While prototype-based approaches
such as [52, 26, 12] concentrate solely on enhancing discriminatory
representation using local descriptors, BMDG aligns and mixes local part
information in a gradual training process to mitigate domain discrepancies
within the representation space. Unlike the Part-Mix method[20], which
utilizes a fixed proportion of mixed parts for auxiliary features, our
approach employs bidirectional multi-step generation for intermediate steps to
bolster the robustness of V-I ReID training for large discrepancy samples. Our
strategy enables effective bridging of cross-modality gaps, and allows gradual
training by adjusting the proportion of exchanged information. Discriminative
and contrastive regularization are used to guide the model training to detect
these parts without supervision. Parts of the same individual are exchanged
across modalities. Contrastive objectives are used to learn for both original
and virtual features in a bidirectional way.

Our main contributions are summarized as follows.
(1) A BMDG method is proposed to train a backbone V-I ReID model that learns a
common feature embedding by gradually reducing the gap between I and V
modalities through multiple auxiliary intermediate steps.
(2) A novel hierarchical prototype learning module is introduced to align
discriminative, complementary, and interchangeable part-prototypes between
modalities using two-level contrastive learning. These prototypes should
maintain semantic consistency to facilitate the exchange of V and I
information, thus enabling the generation of intermediate bridging domains.
(3) A bidirectional multi-step learning model is proposed to progressively
combine prototypes from both modalities and create intermediate feature spaces
at each step. It integrates attentive prototype embedding (APE), which refines
these prototypes to extract attributes that are shared between modalities, and
thereby improve ReID accuracy. Indeed, BMDG can find better modality-shared
attributes by optimizing the model on two levels: prototype alignment and
bidirectional multi-step embedding modules.
(4) An extensive set of experiments on the challenging SYSU-MM01 [42] and
RegDB [32] datasets indicate that our proposed BMDG strategy can outperform
SOTA methods for V-I person ReID. They also show that our framework can be
integrated into any part-based V-I ReID method and can improve the performance
of cross-modal retrieval applications.

##  2 Related Work

In V-I ReID, deep backbone models are trained with labeled V and I images
captured using RGB and IR cameras. During inference, cropped images of people
are matched across camera modalities. That is, query I images are matched
against V gallery images, or vice versa [7, 10, 14, 15, 33, 47, 50, 62].
State-of-the-art methods can be classified into approaches for global or part-
based representation, or that leverage an intermediate modality.

(a) Global and Part-Based Representation Methods: Representation approaches
focus on training a backbone model to learn a discriminant representation that
captures the essential features of each modality while exploiting their shared
information [50, 10, 49, 44, 25, 53]. In [53, 54, 51, 19, 33, 6], authors only
use global features combined through multimodal fusion. The absence of local
information limits their accuracy in more challenging cases. To improve
robustness, recent methods combine global information with local part-base
features [52, 27, 18, 46, 12, 20]. For example, [52, 27] divides the spatial
feature map into fixed horizontal sections and applies a weighted-part
aggregation. [12, 46, 45] dynamically detects regions in the spatial feature
map to alleviate the misalignment of fixed horizontally divided body parts.
However, these part detectors might overfit to specific parts, varying based
on the modality or person. [20] address this issue using Part-Mix data
augmentation to regularize the model part discovery. The bipartite graph-based
correlation model between part regions has been proposed to maximize
similarities [45]. While part-based methods enhance representation ability,
they often overlook the shared information between similar semantic parts
across different individuals or modalities. This leads to inconsistencies in
the extracted part features, hindering effective comparison or exchange.
Similar to the human perception system, which distinguishes individuals by
comparing attributes of similar body parts (e.g., head, torso, leg, etc.)
learned from a diverse set of individuals, our approach seeks to capture and
leverage this shared information for accurate matching. To achieve this, our
proposed BMDG method discovers different body parts as prototypes,
disregarding the person’s identity, and then leverages their ID-discriminative
attributes.

(b) Methods Based on an Intermediate Modality: In recent years, V-I ReID
methods have relied on an intermediate modality to address the significant gap
between V and I modalities [11, 54, 2, 22, 43, 58, 13, 3]. Using fixed
intermediate modality like grayscale [11, 54] or random channels [51, 2] can
improve accuracy. [22] transform V images into a new modality to reduce the
gap with I images. [43] conducted a pixel-to-pixel feature fusion operation on
V and I images to build the synthetic images, and [58] used a shallow auto-
encoder to generate intermediate images from both modalities. While these
generated images bridge the cross-modal gap, details of the original image are
lost. Accuracy is degraded in the context of multi-step learning. To address
this issue, we instead extract prototypes that represent body parts and
gradually use them to define multiple intermediate feature spaces.

(c) Data Augmentation for Domain Generalization: Data augmentation is an
effective strategy to learn domain-invariant features and improve model
generalization [59]. DG methods focus on manipulating the inputs to assist in
learning general representations. [60] relies on domain information to create
additive noise that increases the diversity of training data while preserving
its semantic information. [56, 61] uses Mixup to increase data diversity by
blending examples from the different training distributions. The
Representation Self-Challenging method [17] iteratively discards the dominant
features from training data, aiming to improve generalization. This approach
is inspired by the style transfer literature, where the feature statistics
encode domain-related information. Similarly, MixStyle [61] synthesizes novel
domains by mixing the feature statistics of two instances. Feature
augmentation networks can transform domain styles by generating augmented
examples at the feature level [53, 15, 61, 28]. [31] proposes disentangling
style encodings from class categories to prevent style-biased predictions and
focus more on the contents. The performance of these methods depends on
whether the augmentation can help the model learn data invariance. Finally,
_et al_. [4] relies on uniform sampling and a rejection reward, inspired by
TrivialAugment [30] and TeachAugment [36], respectively.

Some domain adaptation (DA) methods allow for bridging significant shifts
between source and target data distributions. Gradual and multi-step (or
transitive) methods [40] define the number and location of intermediate
domains. [1] proposes a variant of gradual self-training that can generate
pseudo-data, interpolated from the source and target domains, to define
intermediate domains. Intermediate Domain Labeler [5] also creates multiple
intermediate domains, where a coarse domain discriminator sorts them such that
the cycle consistency of self-training is minimized. [57] proposes an
auxiliary model self-training method (AuxSelfTrain) that learns models
intermediate domains and gradually overcomes the shift by combining a
decreasing proportion of source data while increasing the proportion of target
data. In this paper, we iteratively generate features to define multiple
intermediate virtual domains by gradually mixing various body part prototypes
extracted from I and V modalities. This DG approach gradually reduces the
cross-modality gap over multiple intermediate training steps.

##  3 Proposed Method

Our BMDG approach relies on multiple virtual intermediate domains created from
V and I images to learn a common representation space for V-I person ReID.
Fig. 2(a) shows the overall BMDG training architecture comprising two modules.
(1) Our part prototype alignment learning module extracts semantically aligned
and discriminative part prototypes from I and V modalities through
hierarchical contrastive learning. Each prototype represents a specific part
feature in a cropped person image. Exchanging aligned part prototypes allows
for the gradual creation of ID-informative intermediate spaces. This module
discovers and aligns part features that are distinct and informative about the
person to transform the representation feature of one person between
modalities robustly. (2) Our bidirectional multi-step learning module creates
the auxiliary intermediate feature spaces at each step by progressively mixing
more learned part prototypes from each modality. This gradually reduces the
modality-specific information in the final feature representation. In other
words, two intermediate feature spaces are generated, at each step, by
combining proportions of aligned sub-features from each modality. The model
gradually learns to reduce modality-specific information and to leverage more
common information from such intermediate domains by increasing the portions
and making more complex training cases. BMDG trains the feature backbone by
learning from easy samples with a lower modality gap to more complex ones with
a higher gap.

Figure 2: (a) Overall BMDG training architecture comprises two parts. The
prototype learning module (left) extracts body part prototype representations
from V and I images. The bidirectional multi-step learning module (right)
extracts discriminant features using multiple intermediate domains created by
mixing prototype information. (b) Prototype discovery (PD) architecture mines
prototypes from spatial features, and Hierarchical contrastive learning (HCL)
encourages the prototypes to focus on similar semantics for all individuals
without losing ID-discriminative information.

Preliminaries: For the training process, let us assume a multimodal dataset
for V-I ReID, represented by
$\mathcal{V}=\\{x^{j}_{v},y_{v}^{j}\\}_{j=1}^{N_{v}}$ and
$\mathcal{I}=\\{x^{j}_{i},y_{i}^{j}\\}_{j=1}^{N_{i}}$ which contains sets of V
and I images from $C_{y}$ distinct individuals, with their ID labels. V-I ReID
systems seek to match cropped person images captured from V and I cameras by
training a deep backbone model to encode modality-invariant person embedding,
denoted by $\mathbf{f}_{v}$ and $\mathbf{f}_{i}$. Given images of the same
person, the objective is to maximize similarity:

$\max S(\mathbf{f}_{v}^{n},\mathbf{f}_{i}^{l})\;\forall
n\in\\{1,\dots,N_{v}\\}\text{ and }l\in\\{1,\dots,N_{i}\\}\text{ with
}y_{v}^{n}=y_{i}^{l},\vspace{-0.3cm}$ (1)

and for different people, we seek to minimize similarity:

$\min S(\mathbf{f}_{v}^{n},\mathbf{f}_{i}^{l})\;\forall
n\in\\{1,\dots,N_{v}\\}\text{ and }l\in\\{1,\dots,N_{i}\\}\text{ with
}y_{v}^{n}\neq y_{i}^{l}\vspace{-0.1cm}$ (2)

where $\mathbf{f}_{v}^{n},\mathbf{f}_{i}^{l}\in\mathbb{R}^{d}$, $d$ is the
dimension of the representation space, and $S(.,.)$ is similarity matching
function.

###  3.1 Prototype Learning Module

To align pedestrians’ sub-features between different images in I and V
modalities, this module is proposed to discovery different body-part
attributes from diverse regions and then learns to align these prototypes by
making them complementary, interchangeable, and discriminative, as discussed
in Section 1.

(a) Prototype Discovery (PD): This module is proposed to extract the
prototypes and discover several discriminative regions on the backbone feature
maps. At first, the PD predicts a pixel-wise masking score (probability)
$M\in\mathbb{R}^{H\cdot W\times K}$ for each prototype from the features map
$\mathbf{F}_{m}\in\mathbb{R}^{H\cdot W\times d}$ where $H,W,d$ are the feature
sizes, and $K$ is the total number of prototypes. $m$ is modality index which
indicates $v$ as visible or $i$ as infrared domain. Then, it computes each
prototype feature $\mathbf{p}_{m}^{k}$ by weighted aggregating of features in
all pixels as:

$\mathbf{p}_{m}^{k}=\frac{M_{m}^{k}}{|M_{m}^{k}|}\sum_{u\in
U}[\mathbf{F}_{m}]_{u}\ ,\;\;\;|M_{m}^{k}|=\sum_{u\in
U}[M_{m}^{k}]_{u}\vspace{-0.1cm}$ (3)

where for each pixel $u\in U=\\{1,\dots,H\cdot W\\}$, we have
$\sum_{k=1}^{K}[M_{m}^{k}]_{u}=1$. This module (illustrated in Fig. 2(b) uses
a shallow U-Net[34] architecture to create a region mask for each prototype.
In the down-sample block, a convolution receptive field is applied on masks to
make them lose locality by leveraging their neighbor pixels. At the same time,
the up-sample propagates these high-level, coarsely localized masks into each
original size. Skip-connection is used to re-inject the local details lost in
the down-sample phase.

(b) Hierarchical Contrastive Learning: To ensure the part prototypes represent
the diverse attributes of each person, they must be non-redundant, and each
part must be independent of the others. This independence can be formulated
that the mutual information ($MI$) between the distribution of random
variables describing parts as $P^{k}$ should be zero as:

$MI(P^{k},P^{q})=0\;\forall k,q\in\\{1,\dots,K\\},\;k\neq q,$ (4)

where $P^{k}$ is the distribution of the random variable of part $k$. Since
the estimation of mutual information is complex and time-consuming,
$MI(P^{k},P^{q})$ is minimized by directly reducing the cosine similarity
between the extracted features of different parts as:

$\min S(\mathbf{p}^{k},\mathbf{p}^{q}))\;\forall k,q\in\\{1,\dots,K\\},\;k\neq
q,$ (5)

where $\mathbf{p}^{k}$ and $\mathbf{p}^{q}$ are features for part $k$ and $q$,
respectively. To assure that the prototype features are semantically
consistent around body parts can be aligned in different samples, we maximize
the similarity of $\mathbf{p}^{k}$ and all other prototypes with the same
index in the training batch as $\mathbf{\hat{p}}^{k}$.

To select samples in each batch with a positive prototype index, there are two
options: the same parts belonging to the same or different persons (inter-
person) as $\Psi^{k}$, and the same parts belonging to the same person with
identity $y$ (intra-person) as $\Gamma^{k}_{y}$ ( see Fig. 2.b). In a deep
ReID model, the goal is to extract dissimilar features in the representation
space for two different persons, so pulling apart the same prototypes from
these two persons may disrupt this objective. It is preferable to let them
encode ID-related attributes. In contrast, the early backbone layers focus on
semantically similar prototype attributes, disregarding a person’s identity.
Therefore, two contrastive losses as $\mathcal{L}_{\text{lfc}}$ and
$\mathcal{L}_{\text{hfc}}$ are employed to provide inter-peeson information at
a low-level and intra-person at a high-level:

$\mathcal{L}_{\text{lfc}}=-\sum_{k=1}^{K}\sum_{\mathbf{\hat{p}^{\prime}}\in\Psi^{k}}\log\frac{e^{\mathbf{p}^{\prime
k}\cdot\mathbf{\hat{p}^{\prime}}/\tau}}{e^{\mathbf{p}^{\prime
k}\cdot\mathbf{\hat{p}^{\prime}}/\tau}+\sum_{q\neq
k}e^{\mathbf{p^{\prime}}^{k}\cdot\mathbf{p^{\prime}}^{q}/\tau}},$ (6)
$\mathcal{L}_{\text{hfc}}=-\sum_{y=1}^{C_{y}}\sum_{k=1}^{K}\sum_{\mathbf{\hat{p}}\in\Gamma^{k}_{y}}\log\frac{e^{\mathbf{p}^{k}\cdot\mathbf{\hat{p}}/\tau}}{e^{\mathbf{p}^{k}\cdot\mathbf{\hat{p}}/\tau}+\sum_{q\neq
k}e^{\mathbf{p}^{k}\cdot\mathbf{p}^{q}/\tau}},$ (7)

where $\mathbf{p}_{m}^{\prime k}$ are low-level part prototype features
computed same as Eq. (3) but extracted from lower-level layers of the feature
backbone. Unlike contrastive learning introduced in [24, 9] that are used only
for final features, the HCL optimizes in two levels as low-level and high-
level features to force the model to cluster inputs based on part and then
cluster inside identities, as illustrated in Fig. 2(b). For example,
$\mathcal{L}_{\text{lfc}}$ encourages the low-level prototype of legs for two
persons be similar and differ from other prototypes such as torso. While
$\mathcal{L}_{\text{hfc}}$ encourages the high-level legs prototypes only
differs from other prototypes in that person. Also, minimizing the distance
between features of pixels belonging to the same part occurrence encourages
the model to extract more similar features for each prototype:

$\mathcal{L}_{\text{part}}=\sum_{k=1}^{K}\sum_{u\in
U}[M_{m}^{k}\|\mathbf{p}_{m}^{k}-\mathbf{F}_{m}\|]_{u}.\vspace{-0.15cm}$ (8)

Describing part occurrence to a single feature vector enables us to exchange
these features for the corresponding prototype in different modalities and
creates the intermediate step. Apart from the semantic feature contrastive of
prototypes, for pushing more on prototype to point different semantic body-
parts in different locations of images, the visual contrastive is used as:

$\mathcal{L}_{\text{vc}}=\sum_{k=1}^{K}\sum_{q=k+1}^{K}\sum_{u\in
U}[\|M^{k}_{m}-M^{q}_{m}\|]_{u},\vspace{-0.15cm}$ (9)

which makes the model extract them from diverse regions from the backbone
features with the minimum intersection.

(c) Part Discrimination: The information encoded by prototypes must describe
the object in the foreground. That is, the mutual information between the
joint probability $\\{P^{k}\\}_{k=1}^{K}$ and identity $Y$ should be
maximized:

$MI(P^{1},\dots,P^{K};Y).\vspace{-.2cm}$ (10)

In the supplementary material, we proofed that for maximizing Eq. (10), we can
minimize cross-entropy loss between each part-prototypes features and
identities. In fact, by doing this, the model makes the part prototype regions
lie on the object and have enough information about the foreground. In other
words, each part prototype should be able to recover the identity of the
person in the images. So part ID-discriminative loss is defined as:

$\mathcal{L}_{\text{pid}}=\frac{-1}{K}\sum_{k=1}^{K}\sum_{c=1}^{C_{y}}y_{c}\log(W^{k}_{c}(\mathbf{p}_{m}^{k})),\vspace{-0.25cm}$
(11)

where $W^{k}$ is linear layer predicting the probability of identity $y_{c}$
from $\mathbf{p}_{m}^{k}$. To avoid the prototype diversity, we do not share
parameters of $W^{k}$ and $W^{q}$.

###  3.2 Bidirectional Multi-step Learning

The ultimate goal of V-I person ReID is to match individuals by extracting the
features that not only discriminate the person attributes but are also
independent of V or I modality. To extract discriminative features, the
Attentive Prototype Embedding (APE) module uses the extracted part prototypes
to represent the final features by weighting the significance of the parts.

(a) Attentive Prototype Embedding Instead of directly concatenating learned
prototypes as the final feature, we process all other prototypes in input
images. The motivation of this design is that each prototype may discriminate
the part-attributes shared for all individuals rather than ID attributes. The
Attentive Prototype Embedding module (which is detailed in Fig. 3),
$\mathcal{F}$, leverages relevant person information between prototypes by
applying an attention mechanism to achieve person-aware final features. To
aggregate the ID-discriminative features of prototypes, the APE module first
uses a fully connected layer to score and emphasize important channels in
their feature map. It weights each prototype feature by computing the
similarities between them.

$\begin{gathered}\mathcal{F}(\mathbf{A}_{m})=\mathbf{W}_{\text{mlp}}(\mathbf{C}_{m}),\text{
where }\\\
\mathbf{C}_{m}=\mathbf{W}_{v}(\mathbf{A}_{m})\otimes\mathbf{B}_{m}\text{,
}\mathbf{B}_{m}=\sigma(\mathbf{W}_{q}(\mathbf{A}_{m})\otimes\mathbf{W}_{k}(\mathbf{A}_{m})),\vspace{-0.25cm}\end{gathered}$
(12)

and

$\mathbf{A}_{m}=[\mathbf{p}_{m}^{1},\mathbf{p}_{m}^{2},\dots,\mathbf{p}_{m}^{K}],\vspace{-0.2cm}$
(13)

Figure 3: Attentive prototype embedding (APE) architecture.

where $\mathbf{A}_{m}\in\mathbb{R}^{K\times d}$, $\otimes$ is matrix
multiplication and $\sigma$ is the Sigmund function.
$\mathbf{W}_{\text{mlp}}$, $\mathbf{W}_{v}$, $\mathbf{W}_{q}$ and
$\mathbf{W}_{k}$ are linear layer. Also, to increase the discriminative
ability of the final feature embedding, the global features, $\mathbf{g}_{m}$,
extracted by the backbone head are concatenated to the output of APE:

$\mathbf{f}_{m}=[\mathcal{F}(\mathbf{A}_{m});\mathbf{g}_{m}],\vspace{-0.1cm}$
(14)

where $\mathbf{g}_{m}$ is denoted by $\frac{1}{H\cdot W}\sum_{u\in
U}[\mathbf{F}_{m}]_{u}$.

Our APE has two advantages: (1) it allows adjusting modality-agnostic
attention between prototypes regardless of the modality by applying cross-
modality prototype features, and (2) it improves the representation ability of
final embedding by emphasizing most discriminative prototypes.

(b) Bidirectional Multi-Step Learning To deal with the significant shift
between modalities in the feature space, the model is trained via multiple
intermediate steps that gradually bridge this domain gap. Using intermediates
with less domain shift, the model gradually learns to leverage cross-modality
discriminating clues at each step[39, 5, 1, 57]. In the beginning, the
discrepancies between modalities are small letting the model learn from the
easier samples first, then converge later on more complex cases with larger
shifts. One solution to achieve these intermediates is to start from one
modality as the source to transform gradually to the other as the target.
However, by doing this, the model may forget the learned knowledge of the
source and be biased on the target.

To address this issue, modalities are transformed bidirectionally to
intermediate domain be created at each step. By gradually transforming domains
to each other from V and from I, the domain gap vanishes over multiple steps.
Each intermediate auxiliary step provides discriminative information about the
person across both modalities, enabling the training process to transform
inputs from one to the other. By creating these intermediate domains, our
model aims to neither lose nor duplicate information from the main domains.
For gradual transformation in each step, we exchange the features of the same
prototypes from both modalities to create a mixed and virtual representation
space with less domain discrepancies. These prototype-mixed intermediate
feature spaces are used alongside V and I prototypes to provide APE along with
the ability to extract common features from the modalities. For step $t$,
which $t\in\\{1,\dots,T\\}$ and $T$ is the number of intermediate steps, the
intermediate features for the same person are generated using a mixing
function $\mathcal{G}(.,.,.)$ as:

$\mathbf{A}^{(t)}_{m}=\mathcal{G}(\mathbf{A}_{m},\mathbf{A}_{\tilde{m}},t),$
(15)

where $\tilde{m}\neq m$ which mixes the prototypes from two modalities. For
example, we use a simple but yet effective random prototype mixing strategy
for $\mathcal{G}(.,.,.)$:

$\mathcal{G}(\mathbf{A}_{m},\mathbf{A}_{\tilde{m}},t)=[\mathbf{p}_{r(m,\tilde{m},t)}^{1},\dots,\mathbf{p}_{r(m,\tilde{m},t)}^{K}],\vspace{-0.1cm}$
(16)

where $r(.,.,.)$ is a weighted random selector:

$r(m,\tilde{m},t)=\begin{cases}m&t/T\leq\mathcal{U}(0,1)\\\ \tilde{m}&\text{
else}\end{cases}\vspace{-0.2cm}$ (17)

where $\mathcal{U}(0,1)$ is a uniform random generator. For intermediate step
($t<T$) we have
$\mathbf{f}^{(t)}_{m}=[\mathcal{F}(\mathbf{A}^{(t)}_{m});\mathbf{g}_{m}]$ and
in the last step
$\mathbf{f}^{(T)}_{m}=[\mathcal{F}(\mathbf{A}^{(T)}_{m});\mathbf{g}_{\tilde{m}}]$
is used. The module $\mathcal{F}$ learns to gradually reduce the discrepancies
in representation space by applying metric learning objectives bidirectionally
between $\mathbf{f}_{i}$ and $\mathbf{f}^{(t)}_{i}$ and between
$\mathbf{f}_{v}$ and $\mathbf{f}^{(t)}_{v}$ as follows:

$\displaystyle\begin{split}\mathcal{L}_{\text{bcc}}&=\mathcal{L}_{\text{cc}}(\mathbf{f}_{v},\mathbf{f}^{(t)}_{v})+\mathcal{L}_{\text{cc}}(\mathbf{f}_{i},\mathbf{f}^{(t)}_{i})\\\
\mathcal{L}_{\text{bce}}&=\mathcal{L}_{\text{ce}}(\mathbf{f}_{v})+\mathcal{L}_{\text{ce}}(\mathbf{f}^{(t)}_{v})+\mathcal{L}_{\text{ce}}(\mathbf{f}_{i})+\mathcal{L}_{\text{ce}}(\mathbf{f}^{(t)}_{i})\\\
\mathcal{L}_{\text{reid}}&=\mathcal{L}_{\text{bce}}+\mathcal{L}_{\text{bcc}},\end{split}\vspace{-0.2cm}$
(18)

where $\mathcal{L}_{\text{cc}}$, $\mathcal{L}_{\text{ce}}$ are center cluster
loss and cross-entropy loss [46].

Since $\mathbf{f}^{(t)}_{i}$ begins from $\mathbf{f}_{i}$ at first steps and
approaches to $\mathbf{f}_{v}$ at ending steps, the model is gradually trained
to extract more robust modality invariant features from images by seeing
samples with low modality gap to the harder ones with higher gap[39].

###  3.3 Training and Inference

The deep backbone model is trained over multiple steps, where in each step,
the model tries to represent the same feature space for two input domains, V/I
and the intermediate ones, using the overall loss:

$\mathcal{L}=\mathcal{L}_{\text{reid}}+\lambda_{\text{f}}(\mathcal{L}_{\text{lfc}}+\mathcal{L}_{\text{hfc}})+\lambda_{\text{v}}\mathcal{L}_{\text{vc}}+\lambda_{\text{p}}\mathcal{L}_{\text{part}}+\lambda_{\text{i}}\mathcal{L}_{\text{pid}}\vspace{-0.1cm}$
(19)

After training, the tail and head backbones are combined for the inference.
Then, it extracts the prototypes and global features as $\mathbf{A}_{i}$,
$\mathbf{A}_{v}$, $\mathbf{g}_{i}$ and $\mathbf{g}_{v}$ from given input image
$x_{i}$ and $x_{v}$. Using Eq. (14), the $\mathbf{f}_{i}$ and $\mathbf{f}_{v}$
are computed by the APE. The matching score for a query input image is
computed by cosine similarly.

##  4 Results and Discussion

In this section, we compare the proposed BDMG with SOTA person ReID methods,
including global, part-based, and intermediate approaches. We also conduct
extensive ablation studies to demonstrate the effectiveness of the proposed
bi-directional learning as well as the impact of the number of part-prototypes
and intermediate steps during learning. The supplementary materials detail our
experimental methodology protocol and implementation of baselines, and
performance metrics, and SYSU-MM01 [42] and RegDB [32] datasets.

Table 1: Accuracy of the proposed BMDG and state-of-the-art methods on the
SYSU-MM01 (single-shot setting) and RegDB datasets. All numbers are percent.
Results for methods were obtained from the original papers. AIM [12] means re-
ranking method.

Family | SYSU-MM01 | RegDB
---|---|---
All Search | Indoor Search | Visible $\rightarrow$ Infrared | Infrared $\rightarrow$ Visible
Method | Venue | R1 | R10 | mAP | R1 | R10 | mAP | R1 | R10 | mAP | R1 | R10 | mAP
Global | AGW [53] | TPAMI’20 | 47.50 | - | 47.65 | 54.17 | - | 62.97 | 70.05 | - | 50.19 | 70.49 | 87.21 | 65.90
CAJ [51] | ICCV’21 | 69.88 | - | 66.89 | 76.26 | 97.88 | 80.37 | 85.03 | 95.49 | 79.14 | 84.75 | 95.33 | 77.82
RAPV-T [55] | ESA’23 | 63.97 | 95.30 | 62.33 | 69.00 | 97.39 | 75.41 | 86.81 | 95.81 | 81.02 | 86.60 | 96.14 | 80.52
G2DA [37] | PR’23 | 63.94 | 93.34 | 60.73 | 71.06 | 97.31 | 76.01 | - | - | - | - | - | -
Part-based | DDAG [52] | ECCV’20 | 54.74 | 90.39 | 53.02 | 61.02 | 94.06 | 67.98 | 69.34 | 86.19 | 63.46 | 68.06 | 85.15 | 90.31
SSFT [27] | CVPR’20 | 63.4 | 91.2 | 62.0 | 70.50 | 94.90 | 72.60 | 71.0 | - | 71.7 | - | - | -
MPANet [46] | CVPR’22 | 70.58 | 96.21 | 68.24 | 76.74 | 98.21 | 80.95 | 83.70 | - | 80.9 | 82.8 | - | 80.7
SAAI [12]a | ICCV’23 | 75.03 | - | 71.69 | - |  | - | - | - | - | - | - | -
SAAI [12] b | ICCV’23 | 75.90 | - | 77.03 | 82.20 | - | 80.01 | 91.07 | - | 91.45 | 92.09 | - | 92.01
CAL [45] | ICCV’23 | 74.66 | 96.47 | 71.73 | 79.69 | 98.93 | 83.68 | 94.51 | 99.70 | 88.67 | 93.64 | 99.46 | 87.61
PartMix [20] | CVPR’23 | 77.78 | - | 74.62 | 81.52 | - | 84.83 | 84.93 | - | 82.52 | 85.66 | - | 82.27
Intermediate | SMCL[41] | ICCV’21 | 67.39 | 92.84 | 61.78 | 68.84 | 96.55 | 75.56 | 83.93 | - | 79.83 | 83.05 | - | 78.57
MMN [58] | ICM’21 | 70.60 | 96.20 | 66.90 | 76.20 | 99.30 | 79.60 | 91.60 | 97.70 | 84.10 | 87.50 | 96.00 | 80.50
RPIG [2] | ECCVw’22 | 71.08 | 96.42 | 67.56 | 82.35 | 98.30 | 82.73 | 87.95 | 98.3 | 82.73 | 86.80 | 96.02 | 81.26
FTMI [35] | MVA’23 | 60.5 | 90.5 | 57.3 | - | - | - | 79.00 | 91.10 | 73.60 | 78.8 | 91.3 | 73.7
G2DA [37] | PR’23 | 63.94 | 93.34 | 60.73 | 71.06 | 97.31 | 76.01 | - | - | - | - | - | -
SEFL [13] | CVPR’23 | 75.18 | 96.87 | 70.12 | 78.40 | 97.46 | 81.20 | 91.07 | - | 85.23 | 92.18 | - | 86.59
| BMDG (ours)a | - | 75.43 | 97.42 | 72.86 | 83.53 | 98.02 | 83.77 | 93.92 | 98.11 | 89.18 | 94.08 | 97.0 | 88.67
| BMDG (ours)b | - | 76.39 | 97.90 | 78.22 | 83.59 | 98.96 | 85.87 | 94.76 | 98.91 | 92.21 | 94.56 | 98.31 | 93.07

  * a

without AIM

  * b

with AIM

###  4.1 Comparison with State-of-Art Methods:

Table 1 compares the accuracy of BMDG with state-of-the-art V-I ReID
approaches. Our experiments on SYSU-MM01 and RegDB datasets show that BMDG
outperforms these methods in various situations. This is due to BMDG ability
to capture more ID-related knowledge across modalities by learning
discriminative part-prototypes and gradually reducing the modality-specific
information in extracted final features. The prototype alignment module also
creates gradual and bidirectional intermediate spaces without sacrificing
discriminative ability, allowing to outperform the intermediate family of
methods. In addition, our approach has the advantage that it can be easily
integrated into different part-based V-I ReID models to improve the generality
by training gradually. However, in comparison to [20] on SYSU-MM01 dataset, we
have lower R1 accuracy (-2%) in the All Search settings, in Indoor settings
(+2%) and RegDB (+10%) we have higher performance.

###  4.2 Ablation Study:

(a) Step size and number of part prototypes. We evaluate the best option for
the number of parts and steps. Table 2(a) shows that using our BMDG approach
can increase the R1 accuracy when growing the number of steps $T$ for a
specific number of prototypes $K$. This table shows performance increases for
multi-step for specific $K$, which lets the model learn from the lower gap
sample and converge on harder cases.

Table 2: Accuracy (R1%) of BMDG using (a) our prototype mixing and (b) Mixup
[56] setting for different numbers of part prototypes ($K$) and intermediate
steps ($T$). The mAP is reported in the supplementary materials.

(a) Prototype exchanging

$T$ | Number of part prototypes ($K$)
---|---
3 | 4 | 5 | 6 | 7 | 10
0 | 68.25 | 69.24 | 69.40 | 70.27 | 70.03 | 68.12
1 | 69.97 | 70.81 | 72.07 | 71.97 | 71.31 | 69.28
2 | 71.20 | 72.35 | 73.98 | 73.61 | 72.45 | 71.25
3 | 73.32 | 73.94 | 74.11 | 74.98 | 73.22 | 71.67
4 | - | 74.08 | 74.15 | 75.43 | 73.51 | 71.99
6 | - | - | - | 75.37 | 73.52 | 72.15
10 | - | - | - | - | - | 72.07

(b) Mixup [56]

$T$ | Number of part prototypes ($K$)
---|---
3 | 4 | 5 | 6 | 7 | 10
0 | 68.25 | 69.24 | 69.4 | 70.27 | 70.03 | 68.12
1 | 68.53 | 69.19 | 69.56 | 70.53 | 70.3 | 68.41
4 | 68.70 | 70.03 | 69.75 | 70.84 | 70.77 | 68.53
6 | 69.13 | 70.81 | 70.1 | 71.16 | 70.29 | 68.90
10 | 69.21 | 70.15 | 69.52 | 70.95 | 70.90 | 68.13

Another option for each step, instead of exchanging some parts, is to use the
Mixup [56] strategy for all parts from two modalities with normalized weight:
$\mathbf{A}^{(t)}_{m}=\alpha^{(t)}\mathbf{A}_{m}+(1-\alpha^{(t)})\mathbf{A}_{m^{\prime}}$,
where $\alpha$ starts from 0 to 1 w.r.t. to step $t$ and number of steps.
Results in Table 2(b) show that using multi-step increases performance
marginally since it does not utilize the ability of prototypes to create
meaningful intermediate steps. This is explained by the fact that prototypes
contain some modality-specific information, which is crucial for transforming
domains in intermediate steps and may be lost in the averaging process.

(b) Part-based baselines. The effectiveness of BMDG is shown when integrated
into state-of-the-art part-based or prototype-based baseline models [52, 46,
12]. We executed the original code published by authors for each method in our
servers, using the hyper-parameters and other configurations they provided
with and without BMDG. Table 3 shows that BMDG improves performance for mAP
and rank-1 accuracy for all methods by average of 2.02% and 1.54 %
respectively when integrated into these baselines.

Table 3: Accuracy of part-based ReID methods with BMDG on the SYSU-MM01, under
single-shot setting. Results were obtained by executing the author’s code on
our servers.

Method | R1 (%) | mAP (%)
---|---|---
DDAG [52] | 53.62 | 52.71
DDAG with BMDG |  55.36 (+1.74$\uparrow$) |  54.05 (+1.34$\uparrow$)
MPANet [46] | 66.24 | 62.89
MPANet with BMDG |  68.74 (+2.50$\uparrow$) |  64.25 (+1.36$\uparrow$)
SAAI [12] | 71.87 | 68.16
SAAI with BMDG |  73.69 (+1.82$\uparrow$) |  70.08 (+1.92$\uparrow$)

(c) Loss functions. Contrastive objectives between parts
($\mathcal{L}_{\text{lfc}}$ and $\mathcal{L}_{\text{hfc}}$) are essential for
the model to detect similar semantics between objects, thereby creating an
intermediate step by swiping corresponding parts. Table 4 shows this necessity
when used in conjunction with $\mathcal{L}_{\text{part}}$ – this loss provides
appropriate supervision to create steps and increase performance at test time.
As can be seen, using such regularization in the multi-step approach increases
mAP by 3% and R1 by 4%. In Table 4, the effect of part separation loss
$\mathcal{L}_{\text{part}}$ is shown. Results indicate that using this loss
pushes the parts to be scattered on the body regions and gives the part
features a more robust representation. Also, it enables the intermediate steps
to generalize the feature space better by increasing the performance over the
single-step with multi-step setting. The part identity loss
$\mathcal{L}_{\text{pid}}$ allows the model to extract ID-related prototypes
features that contain discriminative information about the object and generate
an id-informative intermediate step. Without this objective, parts features
are potent for focusing on background information. As shown in Table 4
(compare the two last rows), performance increases when using this objective.

Table 4: Impact on the accuracy of using different BMGD losses, using SYSU-
MM01 data under single-shot for single-step ($T=0$) and multi-step ($T=K$)
settings. The baseline $\mathcal{L}_{\text{reid}}$ loss is used in all cases.

Losses | R1 (%) | mAP (%)
---|---|---
$\mathcal{L}_{\text{pid}}$ | $\mathcal{L}_{\text{lfc}}$ | $\mathcal{L}_{\text{hfc}}$ | $\mathcal{L}_{\text{vc}}$ | $\mathcal{L}_{\text{part}}$ | $T=0$ | $T=K$ | $T=0$ | $T=K$
baseline | 51.09 | - | 49.68 | -
| ✓ | ✓ |  |  | 62.19 | 65.28 | 58.91 | 60.73
| ✓ | ✓ | ✓ |  | 66.35 | 69.50 | 63.01 | 66.48
|  |  | ✓ | ✓ | 66.67 | 67.31 | 61.04 | 61.68
✓ |  |  |  | ✓ | 59.78 | 60.66 | 55.41 | 56.05
✓ | ✓ | ✓ | ✓ |  | 68.30 | 69.04 | 66.87 | 66.99
| ✓ | ✓ | ✓ | ✓ | 69.68 | 71.20 | 67.08 | 68.13
✓ | ✓ | ✓ | ✓ | ✓ | 70.27 | 75.37 | 68.45 | 72.71

(d) Bidirectional domain generalization. To show the impact of the BMGD
approach, we conduct experiments under the following settings, with (1)
single-step, (2) one-directional multi-step (from V $\rightarrow$ I and I
$\rightarrow$ V), and (3) bidirectional for creating the intermediate series.
The results are shown in Table 4.2. Although using multiple intermediate steps
from one modality to another improves the model’s accuracy due to letting the
model leverage common discriminative by gradually learning, this transition
may forget about the source modality. So, the bi-directional approach deals
with forgetting challenges and has the best performance since it is not biased
on a specific modality.

Table 5: Impact on the rank-1 and mAP accuracy using different settings for
multi-step domain generalization. All numbers are percent. $V\rightarrow I$
means that the intermediates are created only from V, by replacing prototypes
from I features.

|c||cc|cc|cc|cc| Settings Number of part prototypes ($K$)
4  5  6  7
R1 mAP R1 mAP R1 mAP R1 mAP
Single step 69.2 65.9 69.4 66.1 70.2 66.3 70.4 66.5
One (V $\rightarrow$ I) 71.0 66.5 71.5 67.1 72.3 67.5 71.6 66.8
One (I $\rightarrow$ V) 70.8 66.3 71.2 67.0 72.6 67.6 71.0 66.7
Bidirectional 74.0 71.8 74.1 71.9 75.4 72.8 73.5 70.2

##  5 Conclusion

This paper introduces the Bidirectional Multi-step Domain Generalization
(BMDG) framework for V-I ReID. It uses a novel prototype learning approach
that learns the most discriminative and complementary set of prototypes from
both modalities. This set of prototypes generates multiple intermediate
domains between modalities by progressively mixing prototypes from each
modality, reducing the domain gap. The effectiveness of BMDG is shown
experimentally on the SYSU-MM01 and RegDB datasets, where it outperforms
state-of-the-art methods for V-I person ReID. Results also highlight the
impact of considering multiple intermediate steps and bidirectional training
to improve the model’s accuracy significantly. Moreover, BMDG can be applied
to other state-of-the-art part-based V-I ReID methods, significantly improving
their ReID accuracy.

## References

  * [1] Abnar, S., Berg, R.v.d., Ghiasi, G., Dehghani, M., Kalchbrenner, N., Sedghi, H.: Gradual domain adaptation in the wild: When intermediate distributions are absent. arXiv preprint arXiv:2106.06080 (2021)
  * [2] Alehdaghi, M., Josi, A., Cruz, R.M., Granger, E.: Visible-infrared person re-identification using privileged intermediate information. In: ECCVws. pp. 720–737. Springer (2022)
  * [3] Alehdaghi, M., Josi, A., Shamsolmoali, P., Cruz, R.M., Granger, E.: Adaptive generation of privileged intermediate information for visible-infrared person re-identification. arXiv preprint arXiv:2307.03240 (2023)
  * [4] Aminbeidokhti, M., Peña, F.A.G., Medeiros, H.R., Dubail, T., Granger, E., Pedersoli, M.: Domain generalization by rejecting extreme augmentations. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2024)
  * [5] Chen, H.Y., Chao, W.L.: Gradual domain adaptation without indexed intermediate domains. Advances in Neural Information Processing Systems 34, 8201–8214 (2021)
  * [6] Chen, K., Pan, Z., Wang, J., Jiao, S., Zeng, Z., Miao, Z.: Joint feature learning network for visible-infrared person re-identification. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV). pp. 652–663. Springer (2020)
  * [7] Chen, Y., Wan, L., Li, Z., Jing, Q., Sun, Z.: Neural feature search for rgb-infrared person re-identification. In: CVPR. pp. 587–597 (2021)
  * [8] Choi, S., Lee, S., Kim, Y., Kim, T., Kim, C.: Hi-cmd: hierarchical cross-modality disentanglement for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10257–10266 (2020)
  * [9] Choudhury, S., Laina, I., Rupprecht, C., Vedaldi, A.: Unsupervised part discovery from contrastive reconstruction. Advances in Neural Information Processing Systems 34, 28104–28118 (2021)
  * [10] Dai, P., Ji, R., Wang, H., Wu, Q., Huang, Y.: Cross-modality person re-identification with generative adversarial training. In: IJCAI. vol. 1, p. 2 (2018)
  * [11] Fan, X., Luo, H., Zhang, C., Jiang, W.: Cross-spectrum dual-subspace pairing for rgb-infrared cross-modality person re-identification. ArXiv abs/2003.00213 (2020)
  * [12] Fang, X., Yang, Y., Fu, Y.: Visible-infrared person re-identification via semantic alignment and affinity inference. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11270–11279 (2023)
  * [13] Feng, J., Wu, A., Zheng, W.S.: Shape-erased feature learning for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22752–22761 (2023)
  * [14] Fu, C., Hu, Y., Wu, X., Shi, H., Mei, T., He, R.: Cm-nas: Cross-modality neural architecture search for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11823–11832 (2021)
  * [15] Hao, Y., Li, J., Wang, N., Gao, X.: Modality adversarial neural network for visible-thermal person re-identification. Pattern Recognition 107, 107533 (2020)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] Huang, Z., Wang, H., Xing, E.P., Huang, D.: Self-challenging improves cross-domain generalization. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 124–140. Springer (2020)
  * [18] Huang, Z., Liu, J., Li, L., Zheng, K., Zha, Z.J.: Modality-adaptive mixup and invariant decomposition for rgb-infrared person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1034–1042 (2022)
  * [19] Jiang, J., Jin, K., Qi, M., Wang, Q., Wu, J., Chen, C.: A cross-modal multi-granularity attention network for rgb-ir person re-identification. Neurocomputing (2020)
  * [20] Kim, M., Kim, S., Park, J., Park, S., Sohn, K.: Partmix: Regularization strategy to learn part discovery for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18621–18632 (2023)
  * [21] Kniaz, V.V., Knyaz, V.A., Hladuvka, J., Kropatsch, W.G., Mizginov, V.: Thermalgan: Multimodal color-to-thermal image translation for person re-identification in multispectral dataset. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 0–0 (2018)
  * [22] Li, D., Wei, X., Hong, X., Gong, Y.: Infrared-visible cross-modal person re-identification with an x modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 4610–4617 (2020)
  * [23] Li, P., Xu, Y., Wei, Y., Yang, Y.: Self-correction for human parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020). https://doi.org/10.1109/TPAMI.2020.3048039
  * [24] Li, Y., He, J., Zhang, T., Liu, X., Zhang, Y., Wu, F.: Diverse part discovery: Occluded person re-identification with part-aware transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2898–2907 (2021)
  * [25] Liu, H., Cheng, J.: Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. CoRR abs/1907.09659 (2019), http://arxiv.org/abs/1907.09659
  * [26] Liu, H., Ma, S., Xia, D., Li, S.: Sfanet: A spectrum-aware feature augmentation network for visible-infrared person re-identification. arXiv preprint arXiv:2102.12137 (2021)
  * [27] Lu, Y., Wu, Y., Liu, B., Zhang, T., Li, B., Chu, Q., Yu, N.: Cross-modality person re-identification with shared-specific feature transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13379–13389 (2020)
  * [28] Mancini, M., Akata, Z., Ricci, E., Caputo, B.: Towards recognizing unseen categories in unseen domains. In: European Conference on Computer Vision. pp. 466–483. Springer (2020)
  * [29] McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
  * [30] Müller, S.G., Hutter, F.: Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 774–782 (2021)
  * [31] Nam, H., Lee, H., Park, J., Yoon, W., Yoo, D.: Reducing domain gap by reducing style bias. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8690–8699 (2021)
  * [32] Nguyen, D.T., Hong, H.G., Kim, K.W., Park, K.R.: Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17(3), 605 (2017)
  * [33] Park, H., Lee, S., Lee, J., Ham, B.: Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12046–12055 (2021)
  * [34] Ronneberger, O., Fischer, P., Brox, T.: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015 Conference Proceedings (2022)
  * [35] Sun, J., Li, Y., Chen, H., Peng, Y., Zhu, J.: Visible-infrared person re-identification model based on feature consistency and modal indistinguishability. Machine Vision and Applications 34(1), 14 (2023)
  * [36] Suzuki, T.: Teachaugment: Data augmentation optimization using teacher knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10904–10914 (2022)
  * [37] Wan, L., Sun, Z., Jing, Q., Chen, Y., Lu, L., Li, Z.: G2da: Geometry-guided dual-alignment learning for rgb-infrared person re-identification. Pattern Recognition 135, 109150 (2023). https://doi.org/https://doi.org/10.1016/j.patcog.2022.109150, https://www.sciencedirect.com/science/article/pii/S003132032200629X
  * [38] Wang, G., Zhang, T., Cheng, J., Liu, S., Yang, Y., Hou, Z.: Rgb-infrared cross-modality person re-identification via joint pixel and feature alignment. In: The IEEE International Conference on Computer Vision (ICCV) (October 2019)
  * [39] Wang, H., Li, B., Zhao, H.: Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In: International Conference on Machine Learning. pp. 22784–22801. PMLR (2022)
  * [40] Wang, M., Deng, W.: Deep visual domain adaptation: A survey. Neurocomputing 312, 135–153 (2018)
  * [41] Wei, Z., Yang, X., Wang, N., Gao, X.: Syncretic modality collaborative learning for visible infrared person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 225–234 (2021)
  * [42] Wu, A., Zheng, W.S., Gong, S., Lai, J.: Rgb-ir person re-identification by cross-modality similarity preservation. International journal of computer vision 128(6), 1765–1785 (2020)
  * [43] Wu, A., Zheng, W.S., Yu, H.X., Gong, S., Lai, J.: Rgb-infrared cross-modality person re-identification. In: Proceedings of the IEEE international conference on computer vision. pp. 5380–5389 (2017)
  * [44] Wu, A., Zheng, W.S., Yu, H.X., Gong, S., Lai, J.: Rgb-infrared cross-modality person re-identification. In: Proceedings of the IEEE international conference on computer vision. pp. 5380–5389 (2017)
  * [45] Wu, J., Liu, H., Su, Y., Shi, W., Tang, H.: Learning concordant attention via target-aware alignment for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11122–11131 (2023)
  * [46] Wu, Q., Dai, P., Chen, J., Lin, C.W., Wu, Y., Huang, F., Zhong, B., Ji, R.: Discover cross-modality nuances for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4330–4339 (2021)
  * [47] Xu, X., Wu, S., Liu, S., Xiao, G.: Cross-modal based person re-identification via channel exchange and adversarial learning. In: International Conference on Neural Information Processing. pp. 500–511. Springer (2021)
  * [48] Yang, Y., Zhang, T., Cheng, J., Hou, Z., Tiwari, P., Pandey, H.M., et al.: Cross-modality paired-images generation and augmentation for rgb-infrared person re-identification. Neural Networks 128, 294–304 (2020)
  * [49] Ye, M., Lan, X., Wang, Z., Yuen, P.C.: Bi-directional center-constrained top-ranking for visible thermal person re-identification. IEEE Transactions on Information Forensics and Security 15, 407–419 (2020)
  * [50] Ye, M., Lan, X., Li, J., Yuen, P.: Hierarchical discriminative learning for visible thermal person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
  * [51] Ye, M., Ruan, W., Du, B., Shou, M.Z.: Channel augmented joint learning for visible-infrared recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13567–13576 (2021)
  * [52] Ye, M., Shen, J., J. Crandall, D., Shao, L., Luo, J.: Dynamic dual-attentive aggregation learning for visible-infrared person re-identification. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision – ECCV 2020. pp. 229–247. Springer International Publishing, Cham (2020)
  * [53] Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., Hoi, S.C.H.: Deep learning for person re-identification: A survey and outlook. arXiv preprint arXiv:2001.04193 (2020)
  * [54] Ye, M., Shen, J., Shao, L.: Visible-infrared person re-identification via homogeneous augmented tri-modal learning. IEEE Transactions on Information Forensics and Security 16, 728–739 (2020)
  * [55] Zeng, X., Long, J., Tian, S., Xiao, G.: Random area pixel variation and random area transform for visible-infrared cross-modal pedestrian re-identification. Expert Systems with Applications 215, 119307 (2023)
  * [56] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (2018)
  * [57] Zhang, Y., Deng, B., Jia, K., Zhang, L.: Gradual domain adaptation via self-training of auxiliary models. arXiv preprint arXiv:2106.09890 (2021)
  * [58] Zhang, Y., Yan, Y., Lu, Y., Wang, H.: Towards a unified middle modality learning for visible-infrared person re-identification. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 788–796 (2021)
  * [59] Zhou, K., Liu, Z., Qiao, Y., Xiang, T., Loy, C.C.: Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [60] Zhou, K., Yang, Y., Hospedales, T., Xiang, T.: Deep domain-adversarial image generation for domain generalization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 13025–13032 (2020)
  * [61] Zhou, K., Yang, Y., Qiao, Y., Xiang, T.: Domain generalization with mixstyle. arXiv preprint arXiv:2104.02008 (2021)
  * [62] Zhu, Y., Yang, Z., Wang, L., Zhao, S., Hu, X., Tao, D.: Hetero-center loss for cross-modality person re-identification. Neurocomputing 386, 97–109 (2020)

Bidirectional Multi-Step Domain Generalization
for Visible-Infrared Person Re-Identification: Supplementary

##  Appendix 0.A Proofs

In Section 3 of the manuscript, our model seeks to represent input images
using discriminative prototypes that are complementary. To ensure their
complementarity, we minimize the mutual information (MI) between the data
distributions of each pair of prototypes using a contrastive loss between
feature representations of the different prototypes. To improve the
discrimination, the prototype representations encode ID-related information by
maximizing the MI between the joint distribution among all prototypes in the
label distribution space. In this section, we prove that by minimizing the
cross-entropy loss between features of each prototype class and the person ID
in images, we can learn discriminative prototype representations.

###  0.A.1 Maximizing $\text{MI}(P^{1},\dots,P^{K};Y)$

This section provides a proof that $\text{MI}(P^{1},\dots,P^{K};Y)$ (Eq. 10)
can be lower-bounded by :

$\text{MI}(P^{1};Y)+\dots+\text{MI}(P^{K};Y),$ (0.A.1)

following the properties of mutual information.

P.1 (Nonnegativity) For every pair of random variables $X$ and $Y$:

$\text{MI}(X;Y)\geq 0$ (0.A.2)

P.2  For every pair random variables $X$, $Y$ that are independent:

$\text{MI}(X;Y)=0.$ (0.A.3)

P.3 (Monotonicity) For every three random variables $X$, $Y$ and $Z$:

$\text{MI}(X;Y;Z)\leq\text{MI}(X;Y)$ (0.A.4)

P.4  For every three random variables $X$, $Y$ and $Z$, the mutual information
of joint distortions $X$ and $Z$ to $Y$ is:

$\text{MI}(X,Z;Y)=\text{MI}(X;Y)+\text{MI}(Z;Y)-\text{MI}(X;Z;Y)$ (0.A.5)

$MI(P^{1},\dots,P^{K};Y).\vspace{-.2cm}$ (0.A.6)

Figure 0.A.1: Venn diagram of theoretic measures for three variables $X$, $Y$,
and $Z$, represented by the lower left, upper, and lower right circles,
respectively.

###### Theorem 0.A.1

Let $P^{1},\dots,P^{K}$ and $Y$ be random variables with domains
$\mathcal{P}^{1},\dots,\mathcal{P}^{K}$ and $\mathcal{Y}$, respectively. Let
every pair $P^{k}$ and $P^{q}$ ($k\neq q$) be independent. Then, maximizing
$\text{MI}(P^{1},\dots,P^{K};Y)$ can be approximated by maximizing the sum of
MI between each of $P^{k}$ to $Y$, $\sum_{k=1}^{K}{\text{MI}(P^{k};Y)}$.

###### Proof

First, we define $\tilde{P}^{k}$ as the joint distribution of
$P^{k+1},\dots,P^{K}$. Using the P.4 we have:

$\text{MI}(P^{1},\tilde{P}^{1};Y)=\underbrace{\text{MI}(P^{1};Y)}_{\alpha}+\underbrace{\text{MI}(\tilde{P}^{1};Y)}_{\beta}-\underbrace{\text{MI}(P^{1};\tilde{P}^{1};Y)}_{\gamma},$
(0.A.7)

To maximize this, $\alpha$ and $\beta$ should be maximized and $\gamma$
minimized. Given P.3, $\min\text{MI}(P^{1};\tilde{P}^{1};Y)$ can be upper-
bounded by $\min\text{MI}(P^{1};\tilde{P}^{1})$:

$\text{MI}(P^{1};\tilde{P}^{1};Y)\leq\text{MI}(P^{1};\tilde{P}^{1}),$ (0.A.8)

So by minimizing the right term of Eq. 0.A.8, $\gamma$ is also minimized. We
show that $\text{MI}(P^{1};\tilde{P}^{1})=0$ by expanding $\tilde{P}^{1}$ to
$(P^{2},\tilde{P}^{2})$:

$\text{MI}(P^{1};P^{2},\tilde{P}^{2})=\text{MI}(P^{1};P^{2})+\text{MI}(P^{1};\tilde{P}^{2})-\text{MI}(P^{1};P^{2};\tilde{P}^{2}).$
(0.A.9)

Given P.1 and Eq. 4, $\text{MI}(P^{1};P^{2})=0$ and
$\text{MI}(P^{1};P^{2};\tilde{P}^{2})\leq\text{MI}(P^{1};P^{2})=0$ so that:

$\text{MI}(P^{1};P^{2},\tilde{P}^{2})=\text{MI}(P^{1};\tilde{P}^{2}).$
(0.A.10)

After recursively expanding $\tilde{P}^{2}$:

$\text{MI}(P^{1};P^{2},\tilde{P}^{2})=0.$ (0.A.11)

To maximize $\beta$, we can rewrite and expand recursively in Eq. 0.A.7 :

$\text{MI}(\tilde{P}^{1};Y)=\text{MI}(P^{2},\tilde{P}^{2};Y)=\underbrace{\text{MI}(P^{2};Y)}_{\text{maximizing}}+\underbrace{\text{MI}(\tilde{P}^{2};Y)}_{\text{expanding}}-\underbrace{\text{MI}(P^{2};\tilde{P}^{2};Y)}_{0}.$
(0.A.12)

Therefore, it can be shown that:

$\text{MI}(\tilde{P}^{k};Y)=\text{MI}(P^{k+1},\tilde{P}^{k+1};Y)=\text{MI}(P^{k};Y)+\dots+\text{MI}(P^{K};Y)\;\;\;\forall
k\in\\{0,\dots,K-1\\}.$ (0.A.13)

and for $k=0$, we have:

$\text{MI}(P^{1},\dots,P^{K};Y)=\sum_{k=1}^{K}{\text{MI}(P^{k};Y)},$ (0.A.14)

Finally, for maximizing $\text{MI}(P^{1},\dots,P^{K};Y)$, we need to maximize
each $\text{MI}(P^{k};Y)$ so that each prototype feature contains Id-related
information and is complemented. In other words, each prototype seeks to
describe the input images from different aspects.

###  0.A.2 Maximizing $\text{MI}(P^{k};Y)$

In Section 3, the MI between the representation of each prototype and the
label of persons are maximized by minimizing cross-entropy loss (see Eq: 11 of
the manuscript). This approximation is formulated as Proposition 1.

###### Proposition 1

Let $P^{k}$ and $Y$ be random variables with domains $\mathcal{P}^{k}$ and
$\mathcal{Y}$, respectively. Minimizing the conditional cross-entropy loss of
predicted label $\hat{Y}$, denoted by $\mathcal{H}(Y;\hat{Y}|P^{k})$, is
equivalent to maximizing the $\text{MI}(P^{k};Y)$

###### Proof

Let us define the MI as entropy,

$\text{MI}(P^{k};Y)=\underbrace{\mathcal{H}(Y)}_{\delta}-\underbrace{\mathcal{H}(Y|P^{k})}_{\xi}$
(0.A.15)

Since the domain $\mathcal{Y}$ does not change, the entropy of the identity
$\delta$ term is a constant and can therefore be ignored. Maximizing
$\text{MI}(P^{k},Y)$ can only be achieved by minimizing the $\xi$ term. We
show that $\mathcal{H}(Y|P^{k})$ is upper-bounded by our cross-entropy loss
(Eq. 11), and minimizing such loss results in minimizing the $\xi$ term. By
expanding its relation to the cross-entropy [boudiaf2020unifying]:

$\mathcal{H}(Y;\hat{Y}|P^{k})=\mathcal{H}(Y|P^{k})+\underbrace{\mathcal{D}_{\text{KL}}(Y||\hat{Y}|P^{k})}_{\geq
0},$ (0.A.16)

where:

$\mathcal{H}(Y|P^{k})\leq\mathcal{H}(Y;\hat{Y}|P^{k}).$ (0.A.17)

Through the minimization of Eq. 11, training can naturally be decoupled in 2
steps. First, weights of the prototype module are fixed, and only the
classifier parameters (i.e., weight $W^{k}$ of the fully connected layer) are
minimized w.r.t. Eq. 0.A.16. Through this step,
$\mathcal{D}_{\text{KL}}(Y||\hat{Y}|P^{k})$ is minimized by adjusting
$\hat{Y}$ while the $\mathcal{H}(Y|P^{k})$ does not change. In the second
step, the prototype module’s weights are minimized w.r.t.
$\mathcal{H}(Y|P^{k})$, while the classifier parameters $W^{k}$ are fixed.

##  Appendix 0.B Training Algorithm

To train the model, BMDG uses a batch of data containing $N_{b}$ person with
$N_{p}$ positive images from the V and I modalities. Algorithm 1 shows the
details of the BMGD training strategy for optimizing the feature backbone by
gradually increasing mixing prototypes.

At first, the prototype mining module extracts $K$ prototypes from infrared
and visible images in lines 4 and 5 . Then, at lines 6 and 7, $\mathcal{G}$
function mixes these prototypes from each modality to create two intermediate
features. It is noted that the ratio of mixing gradually increases w.r.t the
step number $t$ to create more complex samples. To refine the final feature
descriptor for input images, the attentive embedding module, $\mathcal{F}$, is
applied to prototypes to leverage the attention between them. At the end of
each iteration, the model’s parameters will be optimized by minimizing the
cross-modality ReID objectives between each modality features vector and its
gradually created intermediate.

Algorithm 1 BMDG Training Strategy.

1:$\mathcal{S}=\\{\mathcal{V},\mathcal{I}\\}$ as training data and $T,K$ as
hyper-parameters

2:for $t=1,\dots,T$ do $\triangleright$ over $T$ steps

3: while all batches are not selected do

4: $x_{v}^{j},x_{i}^{j}\leftarrow\textbf{batchSampler}(N_{b},N_{p})$

5: extract prototypes $\mathbf{A}_{v}^{j}$ and global features
$\mathbf{g}_{v}^{j}$ from visible images $v^{j}$ $\triangleright$ left-side of
Fig. 2(a)

6: extract prototypes $\mathbf{A}_{i}^{j}$ and global features
$\mathbf{g}_{i}^{j}$ from infrared images $i^{j}$ $\triangleright$ left-side
of Fig. 2(a)

7:
$\mathbf{A}_{v}^{(t)}\leftarrow\mathcal{G}({\mathbf{A}_{v}^{j}},{\mathbf{A}_{i}^{j}},t)$
$\triangleright$ V intermediate by gradually increasing the mixing ratio w.r.t
$t$ from I prototypes

8:
$\mathbf{A}_{i}^{(t)}\leftarrow\mathcal{G}({\mathbf{A}_{i}^{j}},{\mathbf{A}_{v}^{j}},t)$
$\triangleright$ I intermediate by gradually increasing the mixing ratio w.r.t
$t$ from V prototypes

9: if $t\leq T$ then:

10:
$\mathbf{f}_{v}^{(t)}\leftarrow[\mathcal{F}(\mathbf{A}^{(t)}_{v});\mathbf{g}_{v}]$
$\triangleright$ embedding intermediate visible features

11:
$\mathbf{f}_{i}^{(t)}\leftarrow[\mathcal{F}(\mathbf{A}^{(t)}_{i});\mathbf{g}_{i}]$
$\triangleright$ embedding intermediate infrared features

12: else:

13:
$\mathbf{f}_{v}^{(t)}\leftarrow[\mathcal{F}(\mathbf{A}^{j(t)}_{v});\mathbf{g}_{i}]$

14:
$\mathbf{f}_{i}^{(t)}\leftarrow[\mathcal{F}(\mathbf{A}^{(t)}_{i});\mathbf{g}_{v}]$

15: end if

16: update model’s parameters by optimizing Eq. 19

17: end while

18:end for

##  Appendix 0.C Additional Details on the Experimental Methodology

###  0.C.1 Datasets:

Research on cross-modal V-I ReID has extensively used the SYSU-MM01 [SYSU] and
RegDB [regDB] datasets. SYSU-MM01 is a large dataset containing more than 22K
RGB and 11K IR images of 491 individuals captured with 4 RGB and 2 near-IR
cameras, respectively. Of the 491 identities, 395 were dedicated to training,
and 96 were dedicated to testing. Depending on the number of images in the
gallery, the dataset has two evaluation modes: single-shot and multi-shot.
RegDB contains 4,120 co-located V-I images of 412 individuals. Ten trial
configurations randomly divide the dataset into two sets of 206 identities for
training and testing. The tests are conducted in two ways – comparing I to V
(query) and vice versa.

###  0.C.2 Experimental protocol:

We used a pre-trained ResNet50 [resnet] as the deep backbone model. Each batch
contains 8 RGB and 8 IR images from 10 randomly selected identities. Each
image input is resized to 288 by 144, then cropped and erased randomly, and
filled with zero padding or mean pixels. ADAM optimizer with a linear warm-up
strategy was used for the optimization process. We trained the model by 180
epochs, in which the initial learning rate is set to 0.0004 and is decreased
by factors of 0.1 and 0.01 at 80 and 120 epochs, respectively. $K=6$, $T=4$,
$\lambda_{\textit{f}}=0.1$, $\lambda_{\textit{v}}=0.05$,
$\lambda_{\textit{p}}=0.2$ and $\lambda_{\textit{i}}=0.4$ are set based on the
analyses shown in the ablation study in the main paper and in Section 0.D.1.

###  0.C.3 Performance measures:

We use Cumulative Matching Characteristics (CMC) and Mean Average Precision
(mAP) as assessment metrics in our study. In CMC, rank-k accuracy is measured
to determine how likely it is that a precise cross-modality image of the
person will be present in the top-k retrieved results. As an alternative, mAP
can be used as a measure of image retrieval performance when multiple matching
images are found in a gallery.

##  Appendix 0.D Additional Quantitative Results

###  0.D.1 Hyperparameter values:

This subsection analyzes the impact of $\lambda_{\textit{f}}$,
$\lambda_{\textit{v}}$, $\lambda_{\textit{p}}$, and $\lambda_{\textit{i}}$ on
V-I ReID accuracy. We initially set $\lambda_{\textit{v}}=0.01$,
$\lambda_{\textit{p}}=0.05$, and $\lambda_{\textit{i}}=0.8$, experimenting
with various values for $\lambda_{\textit{f}}$. As shown in Fig. 0.D.1,
accuracy improves with increasing $\lambda_{\textit{f}}$ until it reaches 0.1.
Elevated $\lambda_{\textit{f}}$ enhances prototype diversity, boosting the
discriminative ability of final features in the diverse space. However,
excessively high values disperse prototype features in the feature space,
diminishing discriminability and hindering accurate identification.

Similar trends are observed when $\lambda_{\textit{p}}=0.05$ and
$\lambda_{\textit{i}}=0.8$, varying $\lambda_{\textit{v}}$ from 0.01 to 0.05,
resulting in improved performance. Higher $\lambda_{\textit{v}}$ compresses
prototype regions excessively, lacking sufficient ID-related information.
Conversely, $\lambda_{\textit{i}}$ enhances the discriminative capabilities of
prototypes in images. Balancing these factors, we find optimal values of 0.05
and 0.4 for $\lambda_{\textit{v}}$ and $\lambda_{\textit{i}}$, respectively.
Additionally, based on experimentation, we set $\lambda_{\textit{p}}=0.2$ at
the end of our analysis.

(a) $\lambda_{\textit{f}}$.

(b) $\lambda_{\textit{v}}$.

(c) $\lambda_{\textit{i}}$.

(d) $\lambda_{\textit{p}}$.

Figure 0.D.1: Accuracy of the proposed BMDG over $\lambda_{\textit{f}}$,
$\lambda_{\textit{v}}$, $\lambda_{\textit{i}}$, and $\lambda_{\textit{p}}$
values on SYSU-MM01 dataset in all-search and single-shot mode.

###  0.D.2 Step size and number of part prototypes:

In section 4.2, we discussed the step size and number of part prototypes based
on Rank-1 accuracy. Here we report the mAP measurement for Table 2(a) in paper
in Table 1(a) and Table 2(b) in paper in Table 1(b), respectively:

Table 0.D.1: mAP accuracy of BMDG using (a) our prototype mixing and (b) Mixup
[mixup] setting for different numbers of part prototypes ($K$) and
intermediate steps ($T$).

(a) Prototype exchanging

$T$ | Number of part prototypes ($K$)
---|---
3 | 4 | 5 | 6 | 7 | 10
0 | 65.98 | 67.03 | 67.23 | 68.11 | 67.55 | 65.66
1 | 67.42 | 68.72 | 69.28 | 69.46 | 68.32 | 69.28
2 | 69.51 | 70.08 | 70.72 | 71.14 | 69.97 | 71.25
3 | 71.44 | 71.69 | 71.82 | 72.02 | 71.06 | 71.67
4 | - | 71.98 | 72.15 | 72.86 | 71.19 | 69.00
6 | - | - | - | 72.40 | 71.17 | 69.54
10 | - | - | - | - | - | 69.46

(b) Mixup [mixup]

$T$ | Number of part prototypes ($K$)
---|---
3 | 4 | 5 | 6 | 7 | 10
0 | 65.98 | 67.03 | 67.23 | 68.11 | 67.55 | 65.66
1 | 66.31 | 67.22 | 67.31 | 68.28 | 68.5 | 65.8
4 | 66.50 | 67.68 | 67.79 | 68.52 | 68.49 | 65.88
6 | 66.98 | 67.77 | 68.05 | 68.73 | 68.56 | 66.02
10 | 67.04 | 67.60 | 67.93 | 68.65 | 68.54 | 65.57

###  0.D.3 Model efficiency:

Our BMDG proposed a method to extract alignable part-prototypes in feature
extraction and then compute an attention embedding for the final
representation features. In Table 0.D.2, we showed each component size and
time complexity in the inference time compared to the baseline we used.

Table 0.D.2: Number parameters and floating-point operations at inference time for BMDG and all its sub-modules.  Model | # of Para. (M) | Flops (G)
---|---|---
Feature Backbone | 24.8 | 5.2
Prototype Mining | 3.1 | 0.2
Attentive Prototype Embedding | 1.8 | 0.3
baseline [all-survey] | 24.9 | 5.2
BMDG | 29.7 | 5.7

##  Appendix 0.E Visual Results

###  0.E.1 UMAP projections:

To show the effectiveness of BMDG, we randomly select 7 identities from the
SYSU-MM01 dataset and project their feature representations using the UMAP
method [mcinnes2018umap] for (a) Baseline, (b) one-step (prototypes without
gradual training), and (c) BMDG. Visualization results (Figure 0.E.1) show
that compared with the baseline and one-step approach, the feature
representations learned with our BMDG method are well clustered according to
their respective identity, showing a strong capacity to discriminate. BMDG is
effective for learning robust and identity-aware features. Our BMDG method
reduces this distance across modalities for each person and provides more
separation among samples from different people.

(a) Baseline.

(b) One Step.

(c) BMDG.

Figure 0.E.1: Distributions of learned V and infrared features of 7 identities
from SYSU-MM01 dataset for (a) the baseline, (b) one-step using part-
prototypes, and (c) our BMDG method by UMAP [mcinnes2018umap]. Each color
shows the identity.

Also, to show how the intermediate features gradually mix the modalities, we
draw intermediate features for 6 steps in Figure 0.E.2. At the beginning of
training, the features are based on modality while at step 6, the features are
concentrated on each identity.

(a) Step 1.

(b) Step 2.

(c) Step 3.

(d) Step 4.

(e) Step 5.

(f) Step 6.

Figure 0.E.2: Distributions of learned V and infrared features of 7 identities
from SYSU-MM01 dataset in training for 6 steps at epochs of 10,30,50,70,90 and
160 respectively by UMAP [mcinnes2018umap]. Each color shows the identity. The
intermediate features are drawn with lower opacity.

###  0.E.2 Domain shift:

To estimate the level of domain shift over data from V and I modalities, we
measured the MMD distance for each training epoch. To this end, for each
epoch, we selected 10 random images from 50 random identities and extracted
the prototype and global features, then measured the MMD distance between the
centers of those features for each modality as shown in Fig. 0.E.3(b). We
report the normalized MMD distances between I and V features for our BMDG
approach when compared with the baseline. Our method reduces this distance
more than the y baseline. Thus, the results show that the intermediate domains
improve the model robustness to a large multi-modal domain gap by gradually
increasing the mix in prototypes over multiple steps.

(a) Learning Curve.

(b) BMDG.

Figure 0.E.3:  The mAP and domain shift (MMD distance) between I and V
modalities over training epochs. (a) The learning curve of BMDG vs
Baseline[all-survey]. (b) MMD distance over the center of multiple person’s
infrared features to visible modality.

###  0.E.3 Part-prototype masking:

To show spatial information related to prototype features, we visualize the
score map in the PM module (see Fig. 0.E.4). Our approach encodes prototype
regions linked to similar body parts without considering person identity. Our
model tries to find similar regions for each class of prototypes and then
extracts ID-related information for that region. Therefore, BMDG is more
robust for matching the same part features.

(a) Infrared.

(b) Visible.

Figure 0.E.4:  Prototypes regions extracted by the PRM module for (a) infrared
and (b) visible images. Note that the mask size is 18$\times$9, which is then
resized to fit the original input image. As shown, the mask of prototypes
focuses on similar body parts without accounting for identity.

###  0.E.4 Semi-supervised body part detection:

An additional benefit of our HCL module lies in its ability to detect
meaningful parts in a semi-supervised manner. By forcing the model to identify
semantic regions that are both informative about foreground objects and
contrastive to each other, our hierarchical contrastive learning provides
robust part detection, even in the absence of part labels. To assess HCL, we
fine-tuned our ReID model as a student using a pre-trained part detector
[part_detec] on the PASCAL-Part Dataset [chen2014detect] as the teacher. In
Fig. 0.E.5, the results show our model’s strong capacity for detecting body
parts compared to its teacher.

(a) Infrared.

(b) Visible.

Figure 0.E.5: Semi-supervised part discovery on the SYSU-MM01 dataset. The
first columns are the (a) infrared and (b) visible images. The second column
images are the result from [part_detec], and the last column are results with
our fine-tuned model in BMDG.

## References

  * [1] Abnar, S., Berg, R.v.d., Ghiasi, G., Dehghani, M., Kalchbrenner, N., Sedghi, H.: Gradual domain adaptation in the wild: When intermediate distributions are absent. arXiv preprint arXiv:2106.06080 (2021)

  * [2] Alehdaghi, M., Josi, A., Cruz, R.M., Granger, E.: Visible-infrared person re-identification using privileged intermediate information. In: ECCVws. pp. 720–737. Springer (2022)
  * [3] Alehdaghi, M., Josi, A., Shamsolmoali, P., Cruz, R.M., Granger, E.: Adaptive generation of privileged intermediate information for visible-infrared person re-identification. arXiv preprint arXiv:2307.03240 (2023)
  * [4] Aminbeidokhti, M., Peña, F.A.G., Medeiros, H.R., Dubail, T., Granger, E., Pedersoli, M.: Domain generalization by rejecting extreme augmentations. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2024)
  * [5] Chen, H.Y., Chao, W.L.: Gradual domain adaptation without indexed intermediate domains. Advances in Neural Information Processing Systems 34, 8201–8214 (2021)
  * [6] Chen, K., Pan, Z., Wang, J., Jiao, S., Zeng, Z., Miao, Z.: Joint feature learning network for visible-infrared person re-identification. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV). pp. 652–663. Springer (2020)
  * [7] Chen, Y., Wan, L., Li, Z., Jing, Q., Sun, Z.: Neural feature search for rgb-infrared person re-identification. In: CVPR. pp. 587–597 (2021)
  * [8] Choi, S., Lee, S., Kim, Y., Kim, T., Kim, C.: Hi-cmd: hierarchical cross-modality disentanglement for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10257–10266 (2020)
  * [9] Choudhury, S., Laina, I., Rupprecht, C., Vedaldi, A.: Unsupervised part discovery from contrastive reconstruction. Advances in Neural Information Processing Systems 34, 28104–28118 (2021)
  * [10] Dai, P., Ji, R., Wang, H., Wu, Q., Huang, Y.: Cross-modality person re-identification with generative adversarial training. In: IJCAI. vol. 1, p. 2 (2018)
  * [11] Fan, X., Luo, H., Zhang, C., Jiang, W.: Cross-spectrum dual-subspace pairing for rgb-infrared cross-modality person re-identification. ArXiv abs/2003.00213 (2020)
  * [12] Fang, X., Yang, Y., Fu, Y.: Visible-infrared person re-identification via semantic alignment and affinity inference. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11270–11279 (2023)
  * [13] Feng, J., Wu, A., Zheng, W.S.: Shape-erased feature learning for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22752–22761 (2023)
  * [14] Fu, C., Hu, Y., Wu, X., Shi, H., Mei, T., He, R.: Cm-nas: Cross-modality neural architecture search for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11823–11832 (2021)
  * [15] Hao, Y., Li, J., Wang, N., Gao, X.: Modality adversarial neural network for visible-thermal person re-identification. Pattern Recognition 107, 107533 (2020)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] Huang, Z., Wang, H., Xing, E.P., Huang, D.: Self-challenging improves cross-domain generalization. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 124–140. Springer (2020)
  * [18] Huang, Z., Liu, J., Li, L., Zheng, K., Zha, Z.J.: Modality-adaptive mixup and invariant decomposition for rgb-infrared person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1034–1042 (2022)
  * [19] Jiang, J., Jin, K., Qi, M., Wang, Q., Wu, J., Chen, C.: A cross-modal multi-granularity attention network for rgb-ir person re-identification. Neurocomputing (2020)
  * [20] Kim, M., Kim, S., Park, J., Park, S., Sohn, K.: Partmix: Regularization strategy to learn part discovery for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18621–18632 (2023)
  * [21] Kniaz, V.V., Knyaz, V.A., Hladuvka, J., Kropatsch, W.G., Mizginov, V.: Thermalgan: Multimodal color-to-thermal image translation for person re-identification in multispectral dataset. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 0–0 (2018)
  * [22] Li, D., Wei, X., Hong, X., Gong, Y.: Infrared-visible cross-modal person re-identification with an x modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 4610–4617 (2020)
  * [23] Li, P., Xu, Y., Wei, Y., Yang, Y.: Self-correction for human parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020). https://doi.org/10.1109/TPAMI.2020.3048039
  * [24] Li, Y., He, J., Zhang, T., Liu, X., Zhang, Y., Wu, F.: Diverse part discovery: Occluded person re-identification with part-aware transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2898–2907 (2021)
  * [25] Liu, H., Cheng, J.: Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. CoRR abs/1907.09659 (2019), http://arxiv.org/abs/1907.09659
  * [26] Liu, H., Ma, S., Xia, D., Li, S.: Sfanet: A spectrum-aware feature augmentation network for visible-infrared person re-identification. arXiv preprint arXiv:2102.12137 (2021)
  * [27] Lu, Y., Wu, Y., Liu, B., Zhang, T., Li, B., Chu, Q., Yu, N.: Cross-modality person re-identification with shared-specific feature transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13379–13389 (2020)
  * [28] Mancini, M., Akata, Z., Ricci, E., Caputo, B.: Towards recognizing unseen categories in unseen domains. In: European Conference on Computer Vision. pp. 466–483. Springer (2020)
  * [29] McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
  * [30] Müller, S.G., Hutter, F.: Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 774–782 (2021)
  * [31] Nam, H., Lee, H., Park, J., Yoon, W., Yoo, D.: Reducing domain gap by reducing style bias. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8690–8699 (2021)
  * [32] Nguyen, D.T., Hong, H.G., Kim, K.W., Park, K.R.: Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17(3), 605 (2017)
  * [33] Park, H., Lee, S., Lee, J., Ham, B.: Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12046–12055 (2021)
  * [34] Ronneberger, O., Fischer, P., Brox, T.: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015 Conference Proceedings (2022)
  * [35] Sun, J., Li, Y., Chen, H., Peng, Y., Zhu, J.: Visible-infrared person re-identification model based on feature consistency and modal indistinguishability. Machine Vision and Applications 34(1), 14 (2023)
  * [36] Suzuki, T.: Teachaugment: Data augmentation optimization using teacher knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10904–10914 (2022)
  * [37] Wan, L., Sun, Z., Jing, Q., Chen, Y., Lu, L., Li, Z.: G2da: Geometry-guided dual-alignment learning for rgb-infrared person re-identification. Pattern Recognition 135, 109150 (2023). https://doi.org/https://doi.org/10.1016/j.patcog.2022.109150, https://www.sciencedirect.com/science/article/pii/S003132032200629X
  * [38] Wang, G., Zhang, T., Cheng, J., Liu, S., Yang, Y., Hou, Z.: Rgb-infrared cross-modality person re-identification via joint pixel and feature alignment. In: The IEEE International Conference on Computer Vision (ICCV) (October 2019)
  * [39] Wang, H., Li, B., Zhao, H.: Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In: International Conference on Machine Learning. pp. 22784–22801. PMLR (2022)
  * [40] Wang, M., Deng, W.: Deep visual domain adaptation: A survey. Neurocomputing 312, 135–153 (2018)
  * [41] Wei, Z., Yang, X., Wang, N., Gao, X.: Syncretic modality collaborative learning for visible infrared person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 225–234 (2021)
  * [42] Wu, A., Zheng, W.S., Gong, S., Lai, J.: Rgb-ir person re-identification by cross-modality similarity preservation. International journal of computer vision 128(6), 1765–1785 (2020)
  * [43] Wu, A., Zheng, W.S., Yu, H.X., Gong, S., Lai, J.: Rgb-infrared cross-modality person re-identification. In: Proceedings of the IEEE international conference on computer vision. pp. 5380–5389 (2017)
  * [44] Wu, A., Zheng, W.S., Yu, H.X., Gong, S., Lai, J.: Rgb-infrared cross-modality person re-identification. In: Proceedings of the IEEE international conference on computer vision. pp. 5380–5389 (2017)
  * [45] Wu, J., Liu, H., Su, Y., Shi, W., Tang, H.: Learning concordant attention via target-aware alignment for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11122–11131 (2023)
  * [46] Wu, Q., Dai, P., Chen, J., Lin, C.W., Wu, Y., Huang, F., Zhong, B., Ji, R.: Discover cross-modality nuances for visible-infrared person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4330–4339 (2021)
  * [47] Xu, X., Wu, S., Liu, S., Xiao, G.: Cross-modal based person re-identification via channel exchange and adversarial learning. In: International Conference on Neural Information Processing. pp. 500–511. Springer (2021)
  * [48] Yang, Y., Zhang, T., Cheng, J., Hou, Z., Tiwari, P., Pandey, H.M., et al.: Cross-modality paired-images generation and augmentation for rgb-infrared person re-identification. Neural Networks 128, 294–304 (2020)
  * [49] Ye, M., Lan, X., Wang, Z., Yuen, P.C.: Bi-directional center-constrained top-ranking for visible thermal person re-identification. IEEE Transactions on Information Forensics and Security 15, 407–419 (2020)
  * [50] Ye, M., Lan, X., Li, J., Yuen, P.: Hierarchical discriminative learning for visible thermal person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
  * [51] Ye, M., Ruan, W., Du, B., Shou, M.Z.: Channel augmented joint learning for visible-infrared recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13567–13576 (2021)
  * [52] Ye, M., Shen, J., J. Crandall, D., Shao, L., Luo, J.: Dynamic dual-attentive aggregation learning for visible-infrared person re-identification. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision – ECCV 2020. pp. 229–247. Springer International Publishing, Cham (2020)
  * [53] Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., Hoi, S.C.H.: Deep learning for person re-identification: A survey and outlook. arXiv preprint arXiv:2001.04193 (2020)
  * [54] Ye, M., Shen, J., Shao, L.: Visible-infrared person re-identification via homogeneous augmented tri-modal learning. IEEE Transactions on Information Forensics and Security 16, 728–739 (2020)
  * [55] Zeng, X., Long, J., Tian, S., Xiao, G.: Random area pixel variation and random area transform for visible-infrared cross-modal pedestrian re-identification. Expert Systems with Applications 215, 119307 (2023)
  * [56] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (2018)
  * [57] Zhang, Y., Deng, B., Jia, K., Zhang, L.: Gradual domain adaptation via self-training of auxiliary models. arXiv preprint arXiv:2106.09890 (2021)
  * [58] Zhang, Y., Yan, Y., Lu, Y., Wang, H.: Towards a unified middle modality learning for visible-infrared person re-identification. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 788–796 (2021)
  * [59] Zhou, K., Liu, Z., Qiao, Y., Xiang, T., Loy, C.C.: Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [60] Zhou, K., Yang, Y., Hospedales, T., Xiang, T.: Deep domain-adversarial image generation for domain generalization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 13025–13032 (2020)
  * [61] Zhou, K., Yang, Y., Qiao, Y., Xiang, T.: Domain generalization with mixstyle. arXiv preprint arXiv:2104.02008 (2021)
  * [62] Zhu, Y., Yang, Z., Wang, L., Zhao, S., Hu, X., Tao, D.: Hetero-center loss for cross-modality person re-identification. Neurocomputing 386, 97–109 (2020)