# Multi-Attention-Based Soft Partition Network for Vehicle Re-Identification

Sangrok Lee
MODULABS
<EMAIL_ADDRESS>Taekang Woo
NAVER Corporation
<EMAIL_ADDRESS>Sang Hun Lee
Kookmin University
<EMAIL_ADDRESS>Corresponding author

###### Abstract

Vehicle re-identification (Re-ID) distinguishes between the same vehicle and
other vehicles in images. It is challenging due to significant intra-instance
differences between identical vehicles from different views and subtle inter-
instance differences of similar vehicles. Researchers have tried to address
this problem by extracting features robust to variations of viewpoints and
environments. More recently, they tried to improve performance by using
additional metadata such as key points, orientation, and temporal information.
Although these attempts have been relatively successful, they all require
expensive annotations. Therefore, this paper proposes a novel deep neural
network called a multi-attention-based soft partition (MUSP) network to solve
this problem. This network does not use metadata and only uses multiple soft
attentions to identify a specific vehicle area. This function was performed by
metadata in previous studies. Experiments verified that MUSP achieved state-
of-the-art (SOTA) performance for the VehicleID dataset without any additional
annotations and was comparable to VeRi-776 and VERI-Wild.

##  1 Introduction

Vehicle re-identification (Re-ID) identifies the same vehicle from a large
number of images. It finds the same car in gallery images as depicted in a
given query image. This task received considerable attention recently because
Re-ID technology could be used to analyze traffic flow to build smart cities
and is an essential technology for surveillance systems. Vehicle Re-ID is
particularly challenging because vehicle exteriors can be captured in a wide
variety of environments, and different lights and viewpoints can cause
significant intra-instance differences. Other vehicles can also look similar
due to matching colors and general vehicle types.

Figure 1: Activation heatmaps (attention weights) for the baseline and
proposed models: (a) baseline model using average pooling, (b) proposed model
with double attentions, and (c) proposed model with five attentions. Grad-CAM
[27] was used to visualize (a), (b), and (c). Right-most heatmaps in (b) and
(c) are background weights. Red regions correspond to high activation, blue
regions to low activation. The proposed model captures the entire vehicle
shape accurately while capturing finer details as more attention threads are
added compared with the baseline.

Recent studies [34, 29, 37, 10, 20, 8, 13] have used convolutional neural
networks (CNNs) and metric learning methods. In metric learning, vehicle
images are encoded to a representative vector in embedding space, and
distances between the vectors are compared. Thus, it is critical to select
robust features to accommodate variations in environments, light conditions,
and viewpoints.

Previous studies [30, 29, 27, 8] adopted metadata attributes (e.g.,
orientation, color, type, key point(s), viewpoint, and spatio-temporal
information) to identify the same vehicles. More recent studies semantically
divided a vehicle into parts to extract features. He _et al_. [3] proposed a
part detection model and extracted features from the part area. Chen _et al_.
[2] leveraged vehicle orientation and mask, using a model to predict the
vehicle mask, with each vehicle part segmented differently depending on
orientation. Meng _et al_. [24] used part segmentation, separating the vehicle
into four parts and extracting view-aware features from segmentation regions.

These methods can compare not only global appearance but also vehicle parts so
that they can embed and compare subtle vehicle parts. However, they have one
major drawback that they require extensive image annotation. In particular,
labeling vehicle parts, including segmentation and bounding box creation,
requires much more time than labeling images. According to a report [15],
segmenting takes 15 times longer than spotting object locations and 60 times
longer than image labeling. segmenting takes 15 times longer than spotting
object locations and 60 times longer than image labeling.

Therefore, we propose a multi-attention-based soft partition (MUSP) network to
identify vehicles efficiently without additional annotation work. As
illustrated in Figure 1, we introduced multiple attentions to obtain weighted
feature maps focusing on different vehicle regions. Each weighted feature map
is abstracted to a feature vector using average pooling. We also introduced
soft partitioning of vehicle images based on the soft attention method. In
contrast to hard attention approaches where the region mask is Boolean, our
model provides continuous values [0,1] of the region mask, allowing softer
partitioning. Thus, the activated region of a feature can include any area
without restrictions. MUSP operates by taking as input the feature map
extracted from a backbone network such as ResNet. Therefore, it can be applied
to all types of backbone networks, and performance can be significantly
improved by merely attaching MUSP at the end of a backbone. Our study has
three main contributions:

  * •

We propose a multi-attention-based soft attention network called MUSP to
provide part-aware attention weights and extract more representative and
robust features for vehicle Re-ID.

  * •

In contrast to previous approaches, our method does not require any additional
annotation for vehicle parts. Ours is the first study that exploits part-aware
features without additional annotation or metadata attributes to the best of
our knowledge.

  * •

Our approach achieved state-of-the-art (SOTA) performance for the VehicleID
dataset and comparable performance for VeRi-776 and VERI-Wild datasets,
compared with other methods using additional annotation.

##  2 Related Works

Vehicle Re-ID technology has advanced enormously, strongly driven by access to
several large datasets [21, 9, 18], enabling models to be trained and tested
on more closely real-world environments. Deep learning and metric learning
have been used for the vehicle Re-ID task. Additional representative features
must be extracted when embedding vehicle images in the feature space to
increase metric learning performance. Consequently, many attempts have been
introduced that use the metadata of vehicles, such as orientation, color,
type, key points, viewpoint, and spatio-temporal data.

Temporal data have been adopted by several studies [27, 18, 8]. Shen _et al_.
[27] use temporal information to track gradual vehicle changes from different
cameras, enabling them to recognize the same vehicle that looks different and
overcome the method’s limitation using only spatial information. However,
there is a disadvantage—a continuous stream of images is required. Liu _et
al_. [18] perform re-ranking using temporal information after vehicle
detection from images. This approach requires the temporal information of each
vehicle even in the inference stage. Jiang _et al_. [8] also use temporal
information together with spatial information for re-ranking.

Vehicle key points are used by several previous studies [30, 10]. Wang _et
al_. [30] estimated orientation using key points and extracted orientation-
invariant features to improve performance. They also used temporal
information. Khorramshahi _et al_. [10] used key points to exploit local
features. The key-point-based method has a disadvantage: it is difficult to
cope with various types of vehicles that do not exist in the training data,
and additional key point labels are required.

Recent studies[20, 2, 24] introduced a method of segmenting and comparing
vehicle parts using metadata. This method is similar to the way humans
identify objects by segmenting the parts of a vehicle and comparing each part
separately. Liu _et al_. [20] used a detection model to segment the vehicle
parts. Chen _et al_. [2] proposed a model that segments the parts of a vehicle
in a weakly-supervised method using the vehicle’s orientation to improve
performance. Meng _et al_. [24] uses a supervised segmentation model to divide
vehicles. These methods have achieved performance improvements but have the
drawback of requiring additional annotations or models. Detection and
segmentation require many resources for data, and the model is heavy.

Finally, various methods of using generative adversarial networks (GANs) have
also been proposed [36, 37]. However, there exists a large gap between the
generated features and reality because of the limitations of the generation
ability of existing GANs and the lack of adversarial samples.

Our approach follows the latest part recognition methodology [28, 24, 2],
except for soft partitioning. We introduced multiple soft attentions for soft
partitioning and recognition to obtain weighted feature maps focused on
various vehicle regions. Because no annotation is required, it is cost-
effective while improving performance through part recognition.

##  3 Proposed Method

Figure 2: Overall architecture of MUSP. First, the image is fed into the
backbone network, ResNet50. The features extracted from the backbone network
go through the attention-based network composed of spatial attention module
and channel-based attention modules sequentially. The $n-1$ vectors obtained
from the attention-based network and one global vector obtained using average
pooling are used to train the model using triplet, spatial diversity and
cross-entropy losses. In the inference stage, a total of $n$ vectors are
computed with the co-occurrence attentive module.

The proposed multi-attention network comprises a backbone network to encode a
convolutional feature map for a given image and an attention-based network to
extract a set of weighted feature vectors, each of which focuses on a specific
vehicle region. The attention-based network consists of two modules: the
spatial attention module for soft partitioning of vehicle regions and the
channel-wise attention module based on the squeeze and excitation (SE) method.
The weighted feature vectors compare the distance between images for metric
learning and are fed to a classifier to predict vehicle ID. The classifier
includes batch normalization (BN) [7] and linear layers [23]. $n-1$
classifiers are applied to $n-1$ weighted feature vectors, excluding the
background vector, respectively. The overall architecture of MUSP is dipicted
in Figure 2 and its components are described in the following subsections in
more detail.

###  3.1 Feature extractor

We selected ResNet-50 [4] as a backbone for feature extraction, removed the
last fully connected (FC) layer, and used the last convolution layer’s output.
Thus, the feature extraction process is

$M=CNN_{B}(I),M\in R^{h\times w\times d},$ (1)

where $CNN_{B}$ is the base network, $M$ is a feature map extracted from $B$,
and $h,w,d$ are dimensions of $M$, which depend on the feature extractor and
input image $I$.

###  3.2 Spatial attention module

We use vehicle partitioning to extract subtle vehicle parts for vehicle Re-ID
[28, 24, 2], with an attention method to refine the embedded features.
Khorramshahi _et al_. [11] proposed a method of detecting and re-cropping a
vehicle during preprocessing to reduce background regions. They used a
detection model and bounding box annotation to depress noisy background. We
assume that the same function can be processed within the deep learning model
without additional model or artificial intervention. Figure 1(b) illustrates
that the vehicle area was accurately recognized without additional annotation
or detection. Meng _et al_. [24] found that subtle vehicle components
significantly impact part division. However, they cannot be captured
accurately with single attention because attention focuses on easily compared
features, such as headlights and bumpers. Therefore, we use multiple
attentions that are spatially separated and focus on different vehicle areas.
This distributed attention can consider different parts, so the model can see
and compare more vehicle details. Consequently, we designed a spatial multiple
attention mechanism.

We apply convolution layers to feature $M$ encoded by the backbone network to
extract two feature maps for attention weights $A$ and values $V$. An
attention feature map has $n$ channels with size $h\times w$. Each channel
corresponds to each vehicle part. A value feature map has $c$ channels with
size $h\times w$. Attention weights that passed softmax are multiplied by the
corresponding value to obtain $n$ weighted values to which average pooling is
applied to extract final weighted feature vectors $\\{f_{i}\\}_{i=0,n}$.

We compute softmax along with the last dimension $n$, rather than spatial
dimension $hw$. The attention weight passing softmax has exclusive activation
at each spatial point of the value map. We discard the final weighted feature
vector $f_{n}$. Due to the properties of softmax, activation is also given to
the background. However, if the final vector is discarded, the model is
trained to assign the background region to the discarded vector, which is
noise, as depicted in Figure 3. The entire process can be summarized as

$V=CNN_{VE}(M),V\in R^{h\times w\times c},$ (2) $A=CNN_{AE}(M),A\in R^{h\times
w\times n},$ (3) $F=(V\otimes\sigma(A))/|hw|,F\in R^{c\times n},$ (4)
$F_{d}=\\{f_{1},f_{2},...,f_{n-1}\\},f\in R^{c},$ (5)

where $CNN_{VE}$ and $CNN_{AE}$ are value and attention extractors, i.e., a
simple single convolution layer with a $3\times 3$ kernel with $1\times 1$
padding. $F$ is the set of extracted feature vectors, and $f$ is a single
vector in feature set $F$, $F_{d}$ is the final feature set with the last
background feature discarded, $|\cdot|$ is the matrix size, and $\sigma$ is
softmax operation.

Figure 3: Proposed spatial attention implementation, where $\bigotimes$
denotes matrix multiplication.

###  3.3 Channel-wise attention module

A set of the weighted feature vectors $F_{d}$ with spatial attention are
recalibrated by capturing and applying channel-wise attention, as depicted in
the SE block that modulates channel activation [6]. Channel-wise attention
adjusts the activation intensity according to each channel’s importance. This
attention reduces unnecessary feature element intensities, hence reducing
their influence on distance calculations. Because each feature vector relates
to a feature map highlighted on a specific vehicle area, channel-wise
attention should be applied to all $n-1$ feature vectors, in contrast to the
original SE that controls one feature with FC layers.

We propose a channel-wise attention network based on an extended SE (ESE)
algorithm. We reshape a set of the weighted feature vectors $F_{d}$ into one
vector and then feed that vector to the SE block to modulate channel
activation. The ESE module comprises two linear layers, where the first layer
is followed by a rectified linear unit, and the second by a sigmoid operation.
The ESE input dimension is 2,048, and its output dimensions are 128 and 2048
in sequence. The result from ESE is channel-wise attention, which is then
multiplied by the original $F_{d}$. Thus, ESE can be summarized as

$f_{d}=reshape(F_{d}),f_{d}\in R^{r},$ (6)
$f_{e}=f_{d}\times\rho(MLP(f_{d})),f_{e}\in R^{r},$ (7)
$F_{e}=reshape(f_{e}),F_{e}\in R^{c\times(n-1)},$ (8)

where $r$ is $|c\times(n-1)|$, $MLP$ is a multi-layer perceptron as described
above, $\rho$ is the sigmoid operation, and $F_{e}$ is a set of $n-1$ final
recalibrated features. $f_{d}$ and $f_{d}$ are 1-D vectors, while $F_{d}$ and
$F_{d}$ are 2-D matrices.

###  3.4 Distance computation

The feature vector set $F_{e}$ extracted from the attention-based network and
one feature vector $f_{g}$ obtained by global average pooling of $M$ are used
to calculate losses. We apply triplet loss to each feature vector separately
to train the model and adopt a multi-feature re-weighting function called a
co-occurrence attentive Module (CAM) [2] with some modifications to calculate
the distance by integrating these features for inference.

The distance weights between two vehicles are calculated as

$w_{(a,b),i}=\frac{AR_{a,i}\times AR_{b,i}}{\sum_{i=0}^{n-1}{AR_{a,i}\times
AR_{b,i}}},$ (9)

where $AR_{a,i}$ is an area ratio with the $i^{th}$ attention weight for the
$a^{th}$ image, and $AR$ is calculated by averaging the attention weights.

The original paper used weight = 1 for the global feature, whereas we use
$\frac{1}{n-1}$ for the global feature weight $w_{(a,b),g}$. Hence, the
distance between two vehicles is

$\begin{split}D_{(a,b)}=\sum_{i=0}^{n-1}{w_{(a,b),i}\times\|f_{a,i}-f_{b,i}\|_{2}}+\\\
w_{(a,b),g}\times\|f_{a,g}-f_{b,g}\|_{2},\end{split}$ (10)

where $f_{a,i}$ is the $i^{th}$ feature, $f_{a,g}$ is a global feature for the
$a^{th}$ image, and $\|\cdot\|_{2}$ is the Euclidean distance.

##  4 Loss Function

We use three loss functions to train the model: cross-entropy loss for vehicle
id prediction ($\mathcal{L}_{id}$), triplet loss for distance learning
($\mathcal{L}_{tri}$), and production loss to separate each attention feature
($\mathcal{L}_{div}$). The overall loss function is

$\mathcal{L}=\mathcal{L}_{id}+\mathcal{L}_{tri}+\mathcal{L}_{div}$ (11)

###  4.1 Cross-entropy loss

We apply cross-entropy loss following the vehicle ID prediction layer:

$\mathcal{L}_{id}=-\frac{1}{K}\sum_{l=1}^{n}\sum_{i=1}^{K}\sum_{j=1}^{C}y_{ijl}\log(\hat{y}_{ijl}),$
(12)

where $n$ is the number of features, $K$ is the number of images in a mini-
batch, $C$ is the number of classes, $y_{ijl}$ is the $j^{\mathit{th}}$
element for the one-shot encoded vector describing ground-truth for the
$i^{\mathit{th}}$ sample in a mini-batch and $l^{\mathit{th}}$ feature vector,
and $\hat{y}_{ijl}$ is the $j^{\mathit{th}}$ element of the output vector of
the softmax FC layer for the $i^{\mathit{th}}$ image and $l^{\mathit{th}}$
feature vector.

###  4.2 Triplet loss

The proposed network is optimized with triplet loss for metric learning, which
trains the network to minimize the distance between features from the same
image classes and simultaneously maximize the distance between features from
different image classes. In a mini-batch that contains $P$ identities and $Q$
images for each identity, each image (anchor) has $Q-1$ images of the same
identity (positives) and $(P-1)\times Q$ images of different identities
(negatives). Triplet loss is defined as [5]:

$\begin{split}\mathcal{L}_{tri}=\sum_{e=1}^{n}\sum_{i=1}^{P}\sum_{a=1}^{Q}\Bigg{[}m&+\operatorname*{max}_{\begin{subarray}{c}p=1\dots
Q\\\ p\neq a\end{subarray}}D(v_{a,i},v_{p,i})\\\
&-\operatorname*{min}_{\begin{subarray}{c}j=1\dots P\\\ n=1\dots Q\\\ j\neq
i\end{subarray}}D(v_{a,i},v_{n,j})\Bigg{]}_{+}\end{split}$ (13)

where $v_{a,i}$ is the prediction vector for the $a^{th}$ image of the
$i^{th}$ identity group, and $m$ is the margin to control the difference
between positive and negative pair distances, which helps cluster the
distribution more densely.

###  4.3 Spatial diversity loss

We adopt spatial diversity loss [2] to restrict overlapped areas and hence
ensure each attention weight acts on a different position:

$\mathcal{L}_{div}=\sum_{i=1}^{K}{(a_{1}^{i}\cdot a_{2}^{i}\cdot\dots
a_{n-1}^{i})},$ (14)

where $a_{n}^{i}$ is $n^{th}$ attention weight for the $i^{th}$ image in the
mini-batch. Spatial diversity loss is the summation of space-wise production
of attention weights.

##  5 Experiments

###  5.1 Dataset

We test the proposed approach using three Re-ID datasets - VeRi-776,
VehicleID, and VERI-Wild - as follows:

  * •

The VeRi-776 [18] dataset contains approximately 50,000 images, comprising 776
vehicle ID images captured by 20 cameras. We used 576 IDs (37,778 images) for
training and the remaining 200 IDs (11,579 images) for testing. VeRi-776 data
includes vehicle color and type labels.

  * •

The VehicleID [16] dataset contains 221,763 images from 26,267 vehicle IDs
arranged in three test sets (small, medium, and large) according to the number
of query IDs (800, 1,600, and 2,400).

  * •

VERI-Wild [21] is a recently created dataset from a more challenging
environment than the other two. It contains 416,314 images for 40,671 vehicle
IDs collected by 174 cameras over one month under different weather and time
conditions.

###  5.2 Implementation details

Preprocessing resizes all images to 256×256 pixels and applies random erasing
and translation effects. We use the Adam [12] optimizer with weight decay of
5e-4 and a momentum of 0.9. The proposed model was trained with a batch size
of 64, 16 unique vehicle IDs, a training epoch of 90, and an initial learning
rate of 0.00035, divided by 10 at 30 and 60 epochs, where we used the warmup
method with initial 10 epochs of 0.000035 to 0.00035. Label smoothing was also
applied to avoid overfitting. Training required 6 and 2 h on the VehicleID and
VeRi-776 datasets, respectively, using an NVIDIA Quadro RTX 6000 GPU system.
The training code was written in PyTorch [25].

The training phase used weighted feature vectors from the spatial attention
module and vehicle ID prediction vector as described in Section 4 in the loss
function. The inference phase only used $F_{e}$ and $f_{g}$ with a re-
weighting method to compute distances between vehicles.

###  5.3 Baseline

We set our baseline as the strong baseline with bag-of-tricks proposed by Luo
_et al_. [23]. Our proposed model uses the attention modules to extract $n$
weighted feature vectors. In contrast, the baseline replaces the attention
modules with an average pooling layer, with the remaining preprocessing,
learning process, and architecture being the same.

###  5.4 Experiments on VehicleID dataset

Table 1 compares VehicleID dataset outcomes for MUSP, baseline, and various
relevant previous models using CMC@1 and CMC@5 metrics. PRN [3], PVEN [24],
and MUSP used region-based methods and achieved higher performance than other
methods. However, in contrast to PRN and PVEN, MUSP does not require
additional metadata. Thus, MUSP outperforms the other methods significantly
even without additional information. MUSP achieved performance improvements
for all metrics compared with the baseline, with 2.5% and 1.1%, 2.9% and 1.7%,
and 3% and 2.6% improvement on CMC@1 and CMC@5 for the small, medium, and
large test sets, respectively. MUSP achieved smaller but still significant
improvements compared with PVEN: comparable and 0.6%, 2.1% and 0.6%, and 3.6%
and 1.2% improvements on CMC@1 and CMC@5 for small, medium, and large test
sets, respectively. Thus, MUSP achieves greater performance improvement for
the large dataset than the small.

Table 1: Model performance (CMC@1 and CMC@5) on VehicleID. A denotes using
additional metadata

Method | A | small | medium | large
---|---|---|---|---
@1 | @5 | @1 | @5 | @1 | @5
MD+CCL[16] | ✓ | 0.490 | 0.735 | 0.428 | 0.668 | 0.382 | 0.616
OIFE[30] | ✓ | - | - | - | - | 0.670 | 0.829
VAMI[37] | ✓ | 0.631 | 0.833 | 0.529 | 0.751 | 0.473 | 0.703
RAM[20] |  | 0.752 | 0.915 | 0.723 | 0.870 | 0.677 | 0.845
EALN[22] | ✓ | 0.751 | 0.881 | 0.718 | 0.839 | 0.693 | 0.814
AAVER[10] | ✓ | 0.747 | 0.938 | 0.686 | 0.900 | 0.635 | 0.856
PRN[3] | ✓ | 0.784 | 0.923 | 0.750 | 0.883 | 0.742 | 0.864
PVEN[24] | ✓ | 0.847 | 0.970 | 0.806 | 0.945 | 0.778 | 0.920
Baseline |  | 0.82 | 0.965 | 0.794 | 0.934 | 0.776 | 0.905
MUSP(ours) |  | 0.845 | 0.976 | 0.823 | 0.951 | 0.806 | 0.931

###  5.5 Experiments on VeRi-776 dataset

Table 2: Model performance (mAP, CMC@1 and CMC@5) on VeRi-776. A denotes using
additional metadata.

Method | A | mAP | CMC@1 | CMC@5
---|---|---|---|---
BOW-CN[35] |  | 0.122 | 0.339 | 0.537
LOMO[14] |  | 0.096 | 0.253 | 0.465
GoogLeNet[32] |  | 0.170 | 0.498 | 0.712
FACT[17] |  | 0.185 | 0.510 | 0.735
FACT+Plate+STR[19] |  | 0.278 | 0.614 | 0.788
Siamese+Path[27] |  | 0.583 | 0.835 | 0.900
OIFE[30] | ✓ | 0.480 | 0.894 | -
VAMI[37] | ✓ | 0.501 | - | -
RAM[20] |  | 0.615 | 0.886 | 0.940
EALN[22] | ✓ | 0.574 | 0.844 | 0.941
AAVER[10] | ✓ | 0.612 | 0.890 | 0.947
PRN[3] | ✓ | 0.743 | 0.943 | 0.989
PVEN[24] | ✓ | 0.795 | 0.956 | 0.984
Baseline |  | 0.768 | 0.952 | 0.976
MUSP(ours) |  | 0.78 | 0.956 | 0.979

Table 2 compares VeRi-776 dataset outcomes for MUSP, baseline, and various
relevant previous models using mAP, CMC@1, and CMC@5 metrics. MUSP achieves
1.2% and 0.4% improvement for mAP and CMC@1, respectively, compared with the
baseline, with comparable performance compared with SOTA models using
additional metadata, 3.7% and 1.3%, and slightly lower and equivalent on mAP
and CMC@1 for PRN, respectively. While most high-performing models (e.g.,
AAVER, PRN, and PVEN) use metadata, our MUSP does not require metadata and
achieves a reasonable performance improvement.

###  5.6 Experiments on VERI-Wild dataset

Table 3: Model performance (mAP) on VERI-Wild.

Method | A | small | medium | large
---|---|---|---|---
GoogLeNet[32] |  | 0.243 | 0.242 | 0.215
Triplet[26] |  | 0.157 | 0.133 | 0.099
Softmax[19] |  | 0.264 | 0.227 | 0.176
CCL[16] |  | 0.225 | 0.193 | 0.148
HDC[33] |  | 0.291 | 0.248 | 0.183
GSTE[1] |  | 0.314 | 0.262 | 0.195
Unlable-GAN[38] |  | 0.299 | 0.247 | 0.182
FDA-Net[21] |  | 0.351 | 0.298 | 0.228
PVEN [24] | ✓ | 0.825 | 0.770 | 0.697
Baseline |  | 0.798 | 0.74 | 0.66
MUSP(ours) |  | 0.846 | 0.796 | 0.726

Table 4: Model performance (CMC@1 and CMC@5) on VERI-Wild.

Method | A | small | medium | large
---|---|---|---|---
@1 | @5 | @1 | @5 | @1 | @5
GoogLeNet[32] |  | 0.572 | 0.751 | 0.532 | 0.711 | 0.446 | 0.636
Triplet[26] |  | 0.447 | 0.633 | 0.403 | 0.590 | 0.335 | 0.514
Softmax[16] |  | 0.534 | 0.750 | 0.462 | 0.699 | 0.379 | 0.599
CCL[16] |  | 0.570 | 0.750 | 0.519 | 0.710 | 0.446 | 0.610
HDC[33] |  | 0.571 | 0.789 | 0.496 | 0.723 | 0.440 | 0.649
GSTE[1] |  | 0.605 | 0.801 | 0.521 | 0.749 | 0.454 | 0.665
Unlabled Gan[38] |  | 0.581 | 0.796 | 0.516 | 0.744 | 0.436 | 0.655
FDA-Net[21] |  | 0.640 | 0.828 | 0.578 | 0.783 | 0.494 | 0.705
PVEN[24] | ✓ | 0.967 | 0.992 | 0.954 | 0.988 | 0.934 | 0.978
Baseline |  | 0.952 | 0.987 | 0.935 | 0.982 | 0.909 | 0.969
MUSP(ours) |  | 0.961 | 0.989 | 0.947 | 0.987 | 0.927 | 0.977

The VERI-Wild dataset is the largest vehicle Re-ID dataset and includes
various weather environments, in contrast to the previous two datasets. Like
VeRi-776, VERI-Wild also defines small, medium, and large test datasets with
3,000, 5,000, and 10,000 vehicle IDs, respectively. Table 3 compares the
performance of MUSP, baseline, and various relevant previous models using mAP.
MUSP exhibits a remarkable performance improvement compared with the baseline,
achieving 4.8%, 5.6%, and 6.6% improvement for the small, medium, and large
datasets, respectively. Like the VehicleID dataset, the performance
improvement is particularly noticeable for complex test sets with many IDs.
SOTA performance was achieved with 2.1%, 2.6%, and 2.9% improvement over PVEN,
the current SOTA method, for the small, medium, and large datasets. Compared
with the baseline and PVEN, the MUSP improvement increases with increasing
test dataset size.

Table 4 compares the performance of MUSP, the baseline, and various relevant
previous models using the CMC metric. MUSP achieves 0.9%, 1.2%, and 1.8%
improvement on CMC@1 for the small, medium, and large datasets, respectively,
compared with the baseline, and comparable performance to PVEN [24], the
current SOTA. Thus, MUSP consistently improves performance across all test
sets and confirms that increasing model representation capability is
sufficient to achieve comparable or superior performance.

###  5.7 Ablation study

####  5.7.1 Number of attentions

Table 5: MUSP performance on VeRi-776 by number of attentions Number of attentions | mAP | CMC@1 | CMC@5
---|---|---|---
6 | 0.771 | 0.951 | 0.976
5 | 0.78 | 0.956 | 0.979
4 | 0.777 | 0.956 | 0.978
3 | 0.763 | 0.953 | 0.977
2 | 0.769 | 0.951 | 0.972

Experiments were conducted to determine how the number of attentions affects
performance. Table 5 presents optimal performance using five attentions. Thus,
increasing the number of attentions increases the number of areas vehicles
that can be segmented, which can improve model performance. Attentions of less
than three produce a significantly lower performance than four or more
attentions. It is difficult to segment a vehicle semantically with a small
number of attentions. The experiments illustrate that the desired part
recognition and comparison can only be performed if four or more attentions
are used.

####  5.7.2 Effects of channel-wise attention module

Figure 4: ESE impact on MUSP performance by training epoch on the small
VehicleID dataset. MUSP achieves consistently higher performance with faster
convergence when including the ESE module.

Table 6 compares MUSP performance with and without the channel-wise attention
module on the VehicleID dataset. ESE is the central element of the channel-
wise attention module. ESE improved 0.015 for CMC@1 and CMC@5, verifying that
it successfully recalibrates channel information and is helpful for distance
computation and embedding. ESE receives n-1 features representing each vehicle
part region and adjusts the channel activation by considering the overall
system performance. If some channel information is unnecessary for vehicle
comparison, the channel activation is reduced by ESE to suppress its effect.
Figure 4 compares the ESE impact by epoch. ESE improved performance for all
epochs and accelerated convergence.

Table 6: The effects of the channel-wise attention module on the VehicleID dataset(small test). ESE is the main part of the module. Method | CMC@1 | CMC@5
---|---|---
MUSP w ESE | 0.845 | 0.976
MUSP w/o ESE | 0.83 | 0.961
Table 7: Sigmoid- and softmax-based attention modules for MUSP on VeRi-776. Method | mAP | CMC@1 | CMC@5
---|---|---|---
MUSP w softmax | 0.78 | 0.956 | 0.979
MUSP w sigmoid | 0.772 | 0.954 | 0.977

####  5.7.3 Activation functions of spatial attention module

CBAM [31] and SENet [6] used a sigmoid-based attention module, whereas the
proposed spatial attention module is based on softmax. Softmax satisfies our
spatial partition purposes more closely because it has a normalization effect
that sets the sum of the dimension elements equal to 1. Combining softmax and
spatial diversity loss produces exclusively spatial activation. Gradient
vanishing can occur for the sigmoid approach as training progresses, degrading
performance. We compared the softmax and sigmoid-based attention modules to
verify that sofmax is the more suitable activation function. The spatial
attention module discards the last attention weight, so we retained four
attentions for the sigmoid-based and five for the softmax-based module. Table
7 illustrates that the softmax-based attention achieves by 0.8%, 0.3%, and
0.2% improvement compared with sigmoid-based attention for mAP, CMC@1, and
CMC@5 metrics, respectively. Thus, overall performance improvement from
softmax-based attention is superior to sigmoid-based attention.

###  5.8 Cross-domain experiment

Table 8: The mAP, CMC@1 and CMC@5 on cross-domain settings.

Method | train | test | CMC@1 | CMC@5
---|---|---|---|---
RAM[20] | VehicleID | VehicleID | 0.752 | 0.915
EALN[22] | VehicleID | VehicleID | 0.751 | 0.881
PVEN[24] | VERI-Wild | VehicleID | 0.772 | 0.944
Baseline | VERI-Wild | VehicleID | 0.737 | 0.931
MUSP(ours) | VERI-Wild | VehicleID | 0.797 | 0.951

Figure 5: Results of applying MUSP on three primary vehicle Re-ID datasets.
This model is trained on the VehicleID dataset and tested on three different
datasets. Each column corresponds to each attention map. Regardless of
orientation and vehicle type, each attention has a high activation on one or
more specific regions. For example, the first attention reacts to the bottom
of the headlights. The second reacts to the headlights. The third reacts to
the front or rear window. The fourth has a high activation on the lower parts
of the window and vehicle. The fifth is for the background. Even for the
unseen domains which were not used in training the model (e.g., the VERI-Wild
and VeRi-776 datasets), attentions are invariant to the domain and orientation
attributes.

These experiments confirm that the proposed MUSP outperforms particularly well
for larger test datasets close to real-world environments. Another problem
that arises in real-world environments is to recognize previously unseen
vehicles. Therefore, we conducted a cross-domain experiment comparing RAM [20]
and EALN[22], trained and tested on VehicleID, with PVEN[24] and MUSP trained
on VERI-Wild and tested on VehicleID.

Table 8 presents overwhelming MUSP performance for cross-domain tests: the
attention partition operates effectively even for vehicles not previously
learned. MUSP exceeded models trained on the same dataset, and also achieved
6% and 2.5% improvement compared with the baseline and PVEN, respectively.
These results are consistent with Figure 5, where vehicle parts on the
VeRi-776 and VERI-Wild datasets were equally identified even though MUSP was
trained on the VehicleID dataset.

##  6 Conclusions

In this paper, we propose MUSP, a network that divides vehicle areas and
extracts features without metadata using attention. The visualization in
Figure 5 illustrates that the attention parts selected by the the spatial
attention module also operate effectively on unseen data and perform invariant
orientations. The spatial and channel-wise attention modules are vital MUSP
components and were verified experimentally on three datasets. The experiments
demonstrated that the proposed method was comparable to or superior to current
SOTA methods.

For future research, we will consider applying MUSP to feature maps extracted
from various levels of the layers of a backbone network such as SENet[6]. MUSP
is currently applied only to the resulting feature map of the last layer of
the backbone. However, MUSP can also be applied to feature maps extracted from
the middle layers of the backbone. We expect this approach can improve
performance significantly.

## Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MEST) (No.2020R1A2C1102767).

## References

  * [1] Yan Bai, Yihang Lou, Feng Gao, Shiqi Wang, Yuwei Wu, and Ling-Yu Duan.  Group-sensitive triplet embedding for vehicle reidentification.  IEEE Transactions on Multimedia, 20(9):2385–2399, 2018.
  * [2] Tsai-Shien Chen, Chih-Ting Liu, Chih-Wei Wu, and Shao-Yi Chien.  Orientation-aware vehicle re-identification with semantics-guided part attention network.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 330–346, 2020.
  * [3] Bing He, Jia Li, Yifan Zhao, and Yonghong Tian.  Part-regularized near-duplicate vehicle re-identification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3997–4005, 2019.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [5] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  ArXiv, abs/1703.07737, 2017.
  * [6] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu.  Squeeze-and-excitation networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:2011–2023, 2020.
  * [7] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proceedings of the 32nd International Conference on Machine Learning, PMLR, volume 37, pages 448–456, 2015.
  * [8] Na Jiang, Yue Xu, Zhong Zhou, and Wei Wu.  Multi-attribute driven vehicle re-identification with spatial-temporal re-ranking.  In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP), pages 858–862, 2018.
  * [9] Aytaç Kanaci, Xiatian Zhu, and Shaogang Gong.  Vehicle re-identification in context.  In Proceedings of the German Conference on Pattern Recognition (GCPR), pages 377–390, 2018.
  * [10] Pirazh Khorramshahi, Amit Kumar, Neehar Peri, Sai Saketh Rambhatla, Jun-Cheng Chen, and Rama Chellappa.  A dual-path model with adaptive attention for vehicle re-identification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6131–6140, 2019.
  * [11] Pirazh Khorramshahi, Neehar Peri, Amit Kumar, Anshul Shah, and Rama Chellappa.  Attention driven vehicle re-identification and unsupervised anomaly detection for traffic understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 239–246, 2019.
  * [12] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv, abs/1412.6980, 2014.
  * [13] Sangrok Lee, Eunsoo Park, Hongsuk Yi, and Sang Hun Lee.  Strdan: Synthetic-to-real domain adaptation network for vehicle re-identification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 608–609, 2020.
  * [14] Shengcai Liao, Yang Hu, Xiangyu Zhu, and Stan Z. Li.  Person re-identification by local maximal occurrence representation and metric learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2197–2206, 2015.
  * [15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrece Zitnick.  Microsoft coco: Common objects in context.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 740–755, 2014.
  * [16] Hongye Liu, Yonghong Tian, Yaowei Wang, Lu Pang, and Tiejun Huang.  Deep relative distance learning: Tell the difference between similar vehicles.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2167–2175, 2016.
  * [17] Xinchen Liu, Wu Liu, Huadong Ma, and Huiyuan Fu.  Large-scale vehicle re-identification in urban surveillance videos.  In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2016.
  * [18] Xinchen Liu, Wu Liu, Tao Mei, and Huadong Ma.  A deep learning-based approach to progressive vehicle re-identification for urban surveillance.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 869–884, 2016.
  * [19] Xinchen Liu, Wu Liu, Tao Mei, and Huadong Ma.  Provid: Progressive and multimodal vehicle reidentification for large-scale urban surveillance.  IEEE Transactions on Multimedia, 20(3):645–658, 2017.
  * [20] Xiaobin Liu, Shiliang Zhang, Qingming Huang, and Wen Gao.  Ram: A region-aware deep model for vehicle re-identification.  In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2018.
  * [21] Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, and Lingyu Duan.  Veri-wild: A large dataset and a new method for vehicle re-identification in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3235–3243, 2019.
  * [22] Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, and Ling-Yu Duan.  Embedding adversarial learning for vehicle re-identification.  IEEE Transactions on Image Processing, 28(8):3794–3807, 2019.
  * [23] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang.  Bag of tricks and a strong baseline for deep person re-identification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4321–4329, 2019.
  * [24] Dechao Meng, Liandeng Li, Xuejing Liu, Yongli Li, Shijie Yang, Zheng-Jun Zha, Xingyu Gao, Shuhui Wang, and Qingming Huang.  Parsing-based view-aware embedding network for vehicle re-identification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7101–7110, 2020.
  * [25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [26] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823, 2015.
  * [27] Yantao Shen, Tong Xiao, Hongsheng Li, Shuai Yi, and Xiaogang Wang.  Learning deep neural networks for vehicle re-id with visual-spatio-temporal path proposals.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1900–1909, 2017.
  * [28] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline).  In Proceedings of the European Conference on Computer Vision (ECCV), pages 480–496, 2018.
  * [29] Yi Tang, Di Wu, Zhi Jin, Wenbin Zou, and Xia Li.  Multi-modal metric learning for vehicle re-identification in traffic surveillance environment.  In Proceedings of the IEEE International Conference on Image Processing (ICIP), pages 2254–2258, 2017.
  * [30] Zhongdao Wang, Luming Tang, Xihui Liu, Zhuliang Yao, Shuai Yi, Jing Shao, Junjie Yan, Shengjin Wang, Hongsheng Li, and Xiaogang Wang.  Orientation invariant feature embedding and spatial temporal regularization for vehicle re-identification.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 379–387, 2017.
  * [31] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018.
  * [32] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  A large-scale car dataset for fine-grained categorization and verification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3973–3981, 2015.
  * [33] Yuhui Yuan, Kuiyuan Yang, and Chao Zhang.  Hard-aware deeply cascaded embedding.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 814–823, 2017.
  * [34] Yiheng Zhang, Dong Liu, and Zheng-Jun Zha.  Improving triplet-wise training of convolutional neural network for vehicle re-identification.  In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), pages 1386–1391, 2017.
  * [35] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1116–1124, 2015.
  * [36] Yi Zhou and Ling Shao.  Cross-view gan based vehicle generation for re-identification.  In Proceedings of the British Machine Vision Conference (BMVC), pages 1–12, 2017.
  * [37] Yi Zhou and Ling Shao.  Viewpoint-aware attentive multi-view inference for vehicle re-identification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6489–6498, 2018.
  * [38] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2223–2232, 2017.