# ShareCMP: Polarization-Aware RGB-P Semantic Segmentation

Zhuoyan Liu Bo Wang Lizhi Wang Chenyu Mao Ye Li
Harbin Engineering University
<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Multimodal semantic segmentation is developing rapidly, but the modality of
RGB-Polarization remains underexplored. To delve into this problem, we
construct a UPLight RGB-P segmentation benchmark with 12 typical underwater
semantic classes. In this work, we design the ShareCMP, an RGB-P semantic
segmentation framework with a shared dual-branch architecture, which reduces
the number of parameters by about 26-33% compared to previous dual-branch
models. It encompasses a Polarization Generate Attention (PGA) module designed
to generate polarization modal images with richer polarization properties for
the encoder. In addition, we introduce the Class Polarization-Aware Loss
(CPALoss) to improve the learning and understanding of the encoder for
polarization modal information and to optimize the PGA module. With extensive
experiments on a total of three RGB-P benchmarks, our ShareCMP achieves state-
of-the-art performance in mIoU with fewer parameters on the UPLight
(92.45(+0.32)%), ZJU (92.7(+0.1)%), and MCubeS (50.99(+1.51)%) datasets
compared to the previous best methods. The code is available at
https://github.com/LEFTeyex/ShareCMP.

##  1 Introduction

Figure 1: Analysis of our ShareCMP, CMX [45] and CMNeXt [46]. Our ShareCMP
outperforms CMX and CMNeXt on the UPLight and MCubeS [25] RGB-P datasets and
reduces the number of parameters by about 26-33%.

Multimodal semantic segmentation provides the autonomous vehicle with scene
understanding capability [13, 54] using perceptual modalities such as RGB-
Depth, -Thermal, -Event, and -LiDAR, and is also developing rapidly in recent
years. In underwater scenes, Autonomous Underwater Vehicles (AUVs) also need
this kind of scene understanding capability to perceive comprehensive
environmental information and make correct decisions. Inspired by the use of
polarized light by the mantis shrimp to perceive the underwater environment
[30, 9], we apply the modality of RGB-Polarization to the perception system of
AUVs to improve the perception capability of AUVs in complex underwater
environments.

Most of the current multimodal semantic segmentation methods [34, 52, 47, 24]
are designed for RGB-D and RGB-T, and the model designed based on RGB-P also
refers to the mainstream methods using the Degree of Linear Polarization
(DoLP) or the Angle of Linear Polarization (AoLP) calculated from polarized
images as the polarization modal input [25, 21, 37, 40, 1]. However, the
polarization properties of polarized images are not further explored. We
believe that this fixed paradigm of DoLP and AoLP cannot fully represent the
polarization image properties and has certain limitations. Meanwhile, the
current mainstream multimodal semantic segmentation methods [45, 46, 24]
(RGB-X) adopt a dual-branch architecture to extract features for RGB and X
modalities separately and perform attention interaction and fusion for these
two modalities. Although these mainstream methods perform well on many
multimodal benchmarks, they have two backbone branches and a large number of
parameters that are not convenient for model deployment.

To delve into RGB-P multimodal semantic segmentation, we create an underwater
RGB-P semantic segmentation benchmark that includes 0, 45, 90, and 135 degree
polarized images with a total of 12 classes, which we call UPLight. In this
paper, we investigate the performance of different polarization modalities in
multimodal semantic segmentation and design a non-fixed paradigm Polarization
Generate Attention (PGA) module using four-angle polarized images to generate
better polarization modal images for the RGB-P multimodal semantic
segmentation model. While based on the principle that the polarization angles
of light reflected by different categories of materials are different, we
propose a Class Polarization-Aware Loss (CPALoss) to assist the model in
learning the polarization properties of different classes and improve the
multimodal feature understanding and interaction capabilities of the RGB-P
model. To reduce the number of parameters of the dual-branch model and reduce
the resource consumption of model deployment, we design a shared dual-branch
multimodal semantic segmentation architecture. Our method is called ShareCMP,
which is inspired by CMX [45].

With extensive experiments on UPLight and two additional public RGB-P
datasets, we obtain performance evaluations of the ShareCMP model. ShareCMP
achieves top mIoU of 92.45(+0.32)% on UPLight, 92.4(+0.2)% (MiT-B2
[39])/92.7(+0.1)% (MiT-B4) on ZJU [37], and 50.34(+1.92)%
(RGB-A)/50.55(+16.45)% (RGB-D)/50.99(+1.51)% (RGB-A-D) on MCubeS [25] datasets
compared to the previous best methods. Our ShareCMP outperforms all previous
RGB-P methods on these three datasets, achieving state-of-the-art performance
with fewer parameters.

In conclusion, we deliver the following contributions:

  * •

We construct a new benchmark UPLight for underwater RGB-P semantic
segmentation, which provides data support for AUVs to perform special
perception tasks.

  * •

We consider the shortcomings of the current dual-branch multimodal model and
present a ShareCMP architecture with a shared dual-branch architecture to
reduce the number of model parameters.

  * •

We explore and compare different polarization modal representations and
propose the PGA module to generate the representation with richer polarization
properties.

  * •

CPALoss is proposed, with Class Polarization-Aware Auxiliary Head (CPAAHead)
to improve the learning and understanding of the encoder for polarization
properties.

##  2 Related Work

Semantic segmentation. Semantic segmentation can be seen as an extension of
image classification from image level to pixel level. Since fully
convolutional networks [28] introducing the end-to-end per-pixel
classification paradigm, semantic segmentation has advanced significantly. The
methods include capturing multi-scale features [4, 5, 16, 49], appending
channel and self-attention blocks [7, 12, 18, 42], refining context priors
[19, 26, 41, 43], and leveraging edge cues [2, 11, 23, 36]. More recent
methods prove the effectiveness of transformer-based architectures for
semantic segmentation [50, 38, 35, 39]. While these works achieve high
performance, they still suffer under real-world conditions where RGB images do
not provide sufficient textures such as low illumination and high dynamic
areas.

Multimodal semantic segmentation. At present, multimodal semantic segmentation
has been widely studied, such as RGB-Depth [51, 34, 52], RGB-Thermal [13, 47,
24], RGB-Event [44], RGB-LiDAR [55], RGB-Polarization [37, 25, 21, 40, 1, 31],
_etc_. These methods adopt dual-branch or multi-branch architectures and use
different attention modules between different branches for multi-scale cross-
modal information interaction. Recently, unified multimodal segmentation CMX
[45] and arbitrary-modal semantic segmentation CMNeXt [46] have achieved
better performance. However, the multiple branches of these methods have a
large number of parameters, which makes model deployment difficult. Although
the single-branch method based on data modal fusion [27] is convenient for
model development, its performance is far inferior to the dual-branch model.
Therefore, there is an urgent need to develop a multimodal semantic
segmentation method that is easy to use and has excellent performance. Among
the many data modalities, the polarization modality is quite different from
other modalities. Its raw data are RGB images with different polarization
angles, which is also a part of the RGB image because the RGB image consists
of all polarization angle lights. We believe that the DoLP and AoLP [1, 22]
calculated from the raw data cannot fully represent the polarization image
properties, which also limits the performance of the RGB-P model. To this end,
we propose ShareCMP, a shared dual-branch RGB-P multimodal semantic
segmentation framework.

##  3 Proposed Framework: ShareCMP

###  3.1 ShareCMP Architecture

Figure 2: ShareCMP framework. The ShareCMP encoder consists of modules with
modal exclusive Overlap Patch Embeddings (OPEmbed) and other shared parameters
in each stage. The Polarization Generate Attention (PGA) module generates a
polarization modal image input with rich polarization properties based on
channel attention and large receptive fields. The Class Polarization-Aware
Auxiliary Head (CPAAHead) uses the $\bm{f}_{3}$ and $\bm{f}_{4}$ fused
features in the ShareCMP encoder to construct Class Polarization-Aware Loss
(CPALoss) $Loss_{CPA}$ to improve the capability of the encoder to perceive
optical polarization properties of different classes and optimize the PGA
module.

ShareCMP consists of a shared dual-branch and four-stage encoder, and its
architecture is designed based on CMX [45] and Segfromer [39]. Both RGB and
polarization modalities are encoded by a shared dual-branch encoder, and the
four-stage architecture provides pyramidal fused features to the decoder.
Meta-Transformer [48] shows that different modalities can be uniformly encoded
using different data-to-sequence tokenization and a shared encoder. Inspired
by this, we design a unified encoder with shared parameters for RGB and
polarization modalities, as shown in Fig. 2. Each stage of the shared encoder
consists of three modules: Overlap Patch Embeddings (OPEmbed), Efficient Self-
Attention (ESAttn), and Mix-FFN (MFFN) [39]. OPEmbed generates feature patches
with local continuous information. The feature patches are forwarded to the
following ESAttn and MFFN modules. At each stage, the modal exclusive OPEmbed
(ME OPEmbed) is constructed for the input $\bm{x}_{RGB}$ and $\bm{x}_{P}$,
respectively, and using the shared-parameters ESAttn and MFFN introduces self-
attention and leak location information [39] to obtain $\bm{y}_{RGB}$ and
$\bm{y}_{P}$.

$\displaystyle\bm{y}_{RGB}^{patch}$
$\displaystyle=\text{OPEmbed}_{RGB}(\bm{x}_{RGB})$ (1)
$\displaystyle\bm{y}_{P}^{patch}$
$\displaystyle=\text{OPEmbed}_{P}(\bm{x}_{P})$ (2) $\displaystyle\bm{y}_{RGB}$
$\displaystyle=\text{MFFN}(\text{ESAttn}(\bm{y}_{RGB}^{patch}))$ (3)
$\displaystyle\bm{y}_{P}$
$\displaystyle=\text{MFFN}(\text{ESAttn}(\bm{y}_{P}^{patch}))$ (4)

where $\bm{y}_{X}^{patch}$ is feature patches of $\bm{x}_{X}$. After that, the
Feature Rectification Module (FRM) and the Feature Fusion Module (FFM) [45]
introduce multimodal cross-attention for $\bm{y}_{RGB}$ and $\bm{y}_{P}$ and
fuse their feature information. After the encoder, the four-stage fused
features $\bm{f}_{i}\in\\{\bm{f}_{1},\bm{f}_{2},\bm{f}_{3},\bm{f}_{4}\\}$ are
forwarded to the MLP decoder [39] for the segmentation prediction. We also
refer to the shared dual-branch encoder (ShareCMP encoder) as the pseudo-
single-branch encoder. Although it still retains the dual-branch inference
architecture, it reduces the number of multimodal encoder parameters by about
30%.

###  3.2 Polarization Generate Attention Module

To better represent the properties of polarization modal data, we develop the
Polarization Generate Attention (PGA) module to generate polarization modal
information with more polarization properties to better serve downstream tasks
as semantic segmentation. As shown in Fig. 2, PGA uses Convolution (Conv) with
$3\times 3$ kernel size for four input images with 0, 45, 90, and 135
polarization angles $\\{\bm{I}_{a}|a\in\\{0,45,90,135\\},\bm{I}_{a}\in H\times
W\\}$ to extract features and increases the receptive field of features
without reducing the resolution of the image features. Finally, the
concatenated feature $\bm{f}_{P}\in H\times W$ is obtained, as in Eq. 5.

$\begin{split}\bm{f}_{P}=\text{Concat}(&\text{Conv}_{3\times
3}(\bm{I}_{0}),\,\text{Conv}_{3\times 3}(\bm{I}_{45}),\\\
&\text{Conv}_{3\times 3}(\bm{I}_{90}),\,\text{Conv}_{3\times
3}(\bm{I}_{135}))\end{split}$ (5)

Previous multimodal fusion methods indicate channel information is crucial [6,
17]. Inspired by this, after obtaining the concatenated feature $\bm{f}_{P}$,
using Depth-Wise Convolution (DWConv), which consists of convolution with
$1\times 1$ kernel size and with $3\times 3$ kernel size composed of grouped
convolution with 2 dilation [27], provides the following channel attention
module with advanced features with a larger receptive field to enhance the
generated channel attention features $\bm{Attn}_{P}$. The processing can be
formalized as:

$\bm{Attn}_{P}=\text{ChannelAttn}(\text{DWConv}_{dila=2}(\bm{f}_{P}))$ (6)

Finally, the concatenated feature $\bm{f}_{P}$ is computed by attention,
residual connection, feed-forward network composed of DWConv and PReLU [15]
activation, then the three-channel polarization modal image $\bm{I}_{P}\in
H\times W$ is generated. It can be formalized as:

$\bm{I}_{P}=\text{PReLU}(\text{DWConv}(\bm{f}_{P}+\bm{Attn}_{P}*\bm{f}_{P}))$
(7)

The activation function is PReLU in PGA to improve performance [27]. And PGA
is a lightweight module with a simple architecture.

###  3.3 Class Polarization-Aware Loss

To improve the learning and understanding of the encoder for polarization
modal information, we design the Class Polarization-Aware Loss (CPALoss) and
build the corresponding Class Polarization-Aware Auxiliary Head (CPAAHead) for
CPALoss, as illustrated in Fig. 2. Based on the principle that the reflected
light from different categories of materials produces different degrees of
polarization and angles of polarization, CPALoss uses the estimates of AoLP
and DoLP corresponding to different classes generated by CPAAHead and the
ground truth AoLP and DoLP calculated from polarized images at four angles of
0, 45, 90, and 135 to calculate the loss $Loss_{CPA}$. The loss is optimized
to improve the capability of the encoder for perceiving the optical
polarization properties of different classes while improving the quality of
the polarization modal image generated by the PGA module. CPAAHead, which has
a similar architecture to the MLP decoder [39], uses the multi-scale
multimodal fusion features generated in the encoder to estimate the AoLP and
DoLP of different classes in the image, respectively. CPAAHead constructs two
convolution layers with $1\times 1$ kernel size for each scale feature.
Upsampling operations are performed after the first convolution layer and the
second convolution layer to unify the different scales of the features and to
align the estimation and ground truth size of the AoLP or DoLP. The CPALoss
can be written as:

$\displaystyle\hat{\bm{A}}_{i}/\hat{\bm{D}}_{i}$
$\displaystyle=\text{Conv}^{up}_{1\times 1}(\text{Conv}^{up}_{1\times
1}(\bm{f}_{i}))$ (8) $\displaystyle Loss_{CPA}$
$\displaystyle=\lambda\sum^{Sta}_{i}\sum^{Cls}_{c}((\bm{A}^{c}_{i}-\hat{\bm{A}}^{c}_{i})^{2}+(\bm{D}^{c}_{i}-\hat{\bm{D}}^{c}_{i})^{2})$
(9)

where $i\in Sta\\{1,2,3,4\\}$ represents the index of the $i$ stage in the
encoder, $c\in Cls$ represents the class in the dataset.
$\text{Conv}^{up}_{1\times 1}$ is a convolution with $1\times 1$ followed by
upsampling. $\hat{\bm{A}}/\hat{\bm{D}}$, $\bm{A}/\bm{D}$ are the estimation
and ground truth of AoLP/DoLP, respectively. $\lambda$ is the loss weight of
CPALoss, which we set to 0.01 after experiments. We set CPALoss at 3 and 4
stages, more detailed analysis in Sec. 4.3.

##  4 Experiments

###  4.1 Polarization Modal Representations

Polarization is a basic property of light, expressing the direction of light
vibration. The vibration of polarized light has a fixed direction or a regular
change of direction and is classified as linearly polarized light, circularly
polarized light, and elliptically polarized light. The polarization sensor is
usually set four polarization plates with different angles
($0^{\circ}$,$45^{\circ}$,$90^{\circ}$,$135^{\circ}$) on every four pixels,
which collects linearly polarized images
{$\bm{I}_{0}$,$\bm{I}_{45}$,$\bm{I}_{90}$,$\bm{I}_{135}$} with pixels aligned
at these four angles at the same time. We investigate four polarization modal
representations [37, 1], the Angle of Linear Polarization (AoLP), the Degree
of Linear Polarization (DoLP), the Sin Angle of Linear Polarization (SAoLP),
and the Cos Angle of Linear Polarization (CAoLP), which the SAoLP and the
CAoLP are designed based on the principles of trigonometric functions. These
polarization modal representations are derived from stokes vectors
$\bm{S}=\\{S_{0},S_{1},S_{2},S_{3}\\}$ that describe the polarization state of
the light. Precisely, $S_{0}$ represents the total light intensity, $S_{1}$
and $S_{2}$ denote the ratio of $0^{\circ}$ and $45^{\circ}$ linear
polarization over its perpendicular polarized portion, and $S_{3}$ stands for
the circular polarization power which is not involved in our work. The stokes
vectors $\bm{S}$ can be calculated by the following Eq. 10.

$\bm{S}=\left\\{\begin{split}S_{0}&=\bm{I}_{0}+\bm{I}_{90}=\bm{I}_{45}+\bm{I}_{135}\\\
S_{1}&=\bm{I}_{0}-\bm{I}_{90}\\\
S_{2}&=\bm{I}_{45}-\bm{I}_{135}\end{split}\right.$ (10)

In the work, to represent the total light intensity more accurately, we use
Eq. 11 to calculate $S_{0}$ [22, 1], and make $S_{0}$ be the RGB input image
of the RGB-P multimodal semantic segmentation model.

$S_{0}=(\bm{I}_{0}+\bm{I}_{90}+\bm{I}_{45}+\bm{I}_{135})/2$ (11)

Then, AoLP, DoLP, SAoLP, and CAoLP are formally computed as follows:

AoLP $\displaystyle=\frac{1}{2}\arctan(\frac{S_{2}}{S_{1}})$ (12) DoLP
$\displaystyle=\frac{\sqrt{S^{2}_{1}+S^{2}_{2}}}{S_{0}}$ (13) SAoLP
$\displaystyle=\frac{1}{2}\arcsin(\frac{S_{2}}{S_{0}})$ (14) CAoLP
$\displaystyle=\frac{1}{2}\arccos(\frac{S_{1}}{S_{0}})$ (15)

In our experiments, we further studied the impact of AoLP, DoLP, SAoLP, and
CAoLP on the performance of RGB-P multimodal semantic segmentation. We use
AoLP, DoLP, SAoLP, and CAoLP as inputs to the polarization modality of the
RGB-P model, respectively, to analyze the differences between them.

###  4.2 Datasets and Implementation Details

(a) Collecting device and images of the UPLight.

(b) Distribution of object quantity in 12 semantic classes.

Figure 3: UPLight RGB-P multimodal dataset.

UPLight dataset is an underwater RGB-P semantic segmentation dataset that we
create for AUV perception, which has 12 classes. We use a LUCID_TRI050S-QC
polarization sensor with a C-C-39N0-160-R12 liquid lens and customize a
waterproof housing for underwater use, as the green box in Fig. 3(a) (the
sonar on the left side of the polarization sensor and another camera below are
used for other tasks, which are not related to this work). We mainly collect
color polarized images of typical underwater objects arranged in the pool to
provide data support for AUV to perform special perception tasks. Detailed
dataset information is as shown in Fig. 3. The dataset is split into 1441/350
image pairs for training/validation at the size of $1224\times 1024$. Each
image pair includes four polarized images with different polarization angles
($0^{\circ}$,$45^{\circ}$,$90^{\circ}$,$135^{\circ}$). The input image is
resized to $612\times 512$.

ShareCMP | Stage | mIoU$\uparrow$
---|---|---
1 | 2 | 3 | 4
w/ ME OPEmbed | ✓ |  |  |  | 91.91
w/ ME OPEmbed | ✓ | ✓ |  |  | 92.36
w/ ME OPEmbed | ✓ | ✓ | ✓ |  | 92.01
w/ ME OPEmbed | ✓ | ✓ | ✓ | ✓ | 92.45
Table 1: Ablation study of modal exclusive OPEmbed (ME OPEmbed) on different
encoder stages.

ZJU dataset [37] is an RGB-P multimodal dataset for automated driving on
complex campus street scenes, which has 8 classes. It has 344/50 image pairs
for training/validation at the size of $1224\times 1024$. Each image pair
includes four polarized images with different polarization angles
($0^{\circ}$,$45^{\circ}$,$90^{\circ}$,$135^{\circ}$). The input image is
resized to $612\times 512$.

MCubeS dataset [25] is an RGB-P-NIR multimodal dataset for material
segmentation on different street scenes, which has 20 classes. It has
302/96/102 image pairs for training/validation/test at the size of $1224\times
1024$. Each image pair includes RGB, AoLP, DoLP, and Near-Infrared (NIR)
images. The input image is resized to $612\times 512$.

Implementation details. Our implementations are based on the MMSegmentation
toolbox [8]. Unless otherwise specified, we train our models on 2 A6000 GPUs
with an initial learning rate (LR) of $6e^{-5}$, which is scheduled by the
poly strategy with power 1.0 over 200 epochs. The first 5 epochs are to warm
up models with $1e^{-6}$$\times$ the initial LR. The optimizer is AdamW [29]
with a weight decay of 0.01, and the batch size is 4 per GPU. We use cross-
entropy as the semantic segmentation loss function. The images are augmented
by random resize with a ratio of 0.5-2.0, random horizontal flipping, random
color jitter, and random cropping to $612\times 512$ on all datasets. To
conduct comparisons, the ImageNet-1K [10] pre-trained weight is used for
ShareCMP on all datasets, but not for OPEmbed [39] modules in the polarization
modal branch. We use mean Intersection over Union (mIoU) averaged across
semantic classes as the primary evaluation metric to measure the segmentation
performance.

###  4.3 Ablation Study

Ablation of ShareCMP encoder. The ablation of setting non-shared parameters
modal exclusive OPEmbed [39] on different stages of ShareCMP encoder is
conducted, as shown in Tab. 1. When the ShareCMP encoder sets modal exclusive
OPEmbed in stage 1, its encoder architecture is similar to that of the Meta-
Transformer unified encoder [48], but it has poor performance. We believe that
ShareCMP has multimodal attention interaction operations (by FRM and FFM [45])
after each stage. Simply setting a modal exclusive OPEmbed in stage 1 cannot
meet the generalization performance requirements of the ShareCMP encoder.
ShareCMP achieves the best performance when modal exclusive OPEmbed is set in
all encoder stages.

ShareCMP | Stage | mIoU$\uparrow$
---|---|---
1 | 2 | 3 | 4
w/ CPALoss | ✓ | ✓ | ✓ | ✓ | 91.79
w/ CPALoss |  | ✓ | ✓ | ✓ | 91.63
w/ CPALoss |  |  | ✓ | ✓ | 92.45
w/ CPALoss |  |  |  | ✓ | 91.72
Table 2: Ablation study of the CPALoss.

Ablation of CPALoss. As shown in Tab. 2, we ablate the combination of CPALoss
used in different ShareCMP encoder stages and require that deep and high-level
features must be optimized for CPALoss. When the CPALoss is optimized in
stages 3 and 4, ShareCMP achieves the best performance.

Method | Modal |  Unlabeled |  Mooringmine |  UUV |  Float |  Reflector |  Person |  Bottommine |  Cup |  Ironball |  Conch |  Fish |  Shell |  Starfish | mIoU$\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SegFormer-B2 [39] | RGB | 99.49 | 92.77 | 94.47 | 89.12 | 97.96 | 95.00 | 97.44 | 89.66 | 82.92 | 84.12 | 89.31 | 83.25 | 69.25 | 89.60
MCubeSNet [25] | RGB-AoLP | 89.72 | 83.32 | 84.78 | 85.20 | 88.79 | 85.63 | 88.94 | 81.76 | 75.47 | 83.37 | 80.77 | 81.40 | 65.18 | 82.64
EAFNet [37] | RGB-AoLP | 94.61 | 88.21 | 89.67 | 90.09 | 93.68 | 90.52 | 93.83 | 86.65 | 80.36 | 88.26 | 85.66 | 86.29 | 70.07 | 87.53
CMX (MiT-B2) [45] | RGB-AoLP | 99.53 | 93.08 | 94.90 | 92.27 | 98.07 | 95.27 | 97.88 | 88.67 | 84.83 | 93.60 | 90.40 | 92.62 | 76.53 | 92.13
MCubeSNet | RGB-DoLP | 87.88 | 81.48 | 82.94 | 83.36 | 86.95 | 83.79 | 87.10 | 79.92 | 73.63 | 81.53 | 78.93 | 79.56 | 63.34 | 80.80
EAFNet | RGB-DoLP | 94.42 | 88.02 | 89.48 | 89.90 | 93.49 | 90.33 | 93.64 | 86.46 | 80.17 | 88.07 | 85.47 | 86.10 | 69.88 | 87.34
CMX (MiT-B2) | RGB-DoLP | 99.53 | 92.88 | 94.79 | 90.48 | 98.19 | 95.22 | 97.51 | 92.00 | 85.48 | 91.26 | 90.95 | 92.51 | 76.17 | 92.07
ShareCMP (MiT-B2) | RGB-{$\bm{I}_{0}$,$\bm{I}_{45}$,$\bm{I}_{90}$,$\bm{I}_{135}$} | 99.53 | 93.13 | 94.59 | 95.01 | 98.60 | 95.44 | 98.75 | 91.57 | 85.28 | 93.18 | 90.58 | 91.21 | 74.99 | 92.45

Table 3: Results on UPLight. Method | Modal | mIoU$\uparrow$
---|---|---
ShareCMP | RGB-{$\bm{I}_{0}$,$\bm{I}_{45}$,$\bm{I}_{90}$,$\bm{I}_{135}$} | 92.45
w/o CPALoss | RGB-{$\bm{I}_{0}$,$\bm{I}_{45}$,$\bm{I}_{90}$,$\bm{I}_{135}$} | 91.09(-1.36)
w/o PGA | RGB-AoLP | 91.96(-0.49)
w/o PGA | RGB-DoLP | 92.03(-0.42)
w/o PGA | RGB-SAoLP | 90.63(-1.82)
w/o PGA | RGB-CAoLP | 91.15(-1.30)
Table 4: Ablation study of the ShareCMP framework.

Ablation of ShareCMP. In Tab. 4, we show the semantic segmentation performance
of ShareCMP without CPALoss or PGA and investigate the effect of the four
polarization modalities AoLP, DoLP, SAoLP, and CAoLP on the RGB-P semantic
segmentation performance. When removing the CPALoss for polarization-aware
learning, the performance decreases significantly by 1.36%. Our designed SAoLP
and CAoLP, which are new representation methods for the polarization modality,
result in poor performance decreases of 1.82% and 1.30%, respectively. Since
SAoLP and CAoLP are part of AoLP, their polarization modal representation is
not as complete as AoLP, resulting in reduced performance. While AoLP and DoLP
result in a similar slight decrease in performance, it indicates that AoLP and
DoLP have similar effects on the ShareCMP and are also relatively effective
polarization modal representations. Regardless of the fixed paradigm of the
polarization modal representation, its performance is not as good as the
representation generated by our PGA. The results confirm that our CPALoss and
PGA are more advantageous for RGB-P semantic segmentation.

###  4.4 Comparison against the State of the Art

To verify the efficacy of our proposed ShareCMP framework, we perform
extensive experiments on three RGB-P segmentation datasets. The results and
comparisons against the state-of-the-art are shown in Tabs. 3, 5 and 6.

Results on UPLight. As shown in Tab. 3, we train models [37, 25, 45] that can
be used for RGB-P semantic segmentation and compare them with ShareCMP on the
UPLight dataset we built. Our ShareCMP achieves the best performance of
92.45%. Compared to the RGB-only baseline with SegFormer-B2 [39], the IoU
improvements on classes with polarization properties are clear, such as Float
(+5.9%), Bottom mine (+1.3%), Ironball (+2.4%), Conch (+9.1%), Shell (+8.0%),
and Starfish (+5.7%). This also shows that our RGB-P multimodal semantic
segmentation model ShareCMP has important application significance in
underwater aquaculture, fishing, ocean exploration, _etc_. ShareCMP has only
about 65% of the number of parameters of CMX [45] (detailed parameter number
analysis in Sec. 4.5), and has a 0.32% performance improvement compared to
CMX, which provides more advantages in the deployment of ShareCMP and its
performance improvement on AUVs.

Method | Modal | Building | Glass | Car | Road | Tree | Sky | Pedestrian | Bicycle | mIoU$\uparrow$
---|---|---|---|---|---|---|---|---|---|---
SwiftNet [32] | RGB | 83.0 | 73.4 | 91.6 | 96.7 | 94.5 | 84.7 | 36.1 | 82.5 | 80.3
SegFormer-B2 [39] | RGB | 90.6 | 79.0 | 92.8 | 96.6 | 96.2 | 89.6 | 82.9 | 89.3 | 89.6
NLFNet [40] | RGB-AoLP | 85.4 | 77.1 | 93.5 | 97.7 | 93.2 | 85.9 | 56.9 | 85.5 | 84.4
EAFNet [37] | RGB-AoLP | 87.0 | 79.3 | 93.6 | 97.4 | 95.3 | 87.1 | 60.4 | 85.6 | 85.7
EAFNet | RGB-DoLP | 86.4 | 76.9 | 93.0 | 97.1 | 95.5 | 86.1 | 62.1 | 86.0 | 85.4
CMX (MiT-B2) [45] | RGB-AoLP | 91.5 | 87.3 | 95.8 | 98.2 | 96.6 | 89.3 | 85.6 | 91.9 | 92.0
CMX (MiT-B2) | RGB-DoLP | 91.8 | 87.8 | 96.1 | 98.2 | 96.7 | 89.4 | 86.1 | 91.8 | 92.2
ShareCMP (MiT-B2) | RGB-{$\bm{I}_{0}$,$\bm{I}_{45}$,$\bm{I}_{90}$,$\bm{I}_{135}$} | 91.4 | 88.4 | 96.2 | 98.2 | 96.7 | 90.2 | 85.7 | 92.2 | 92.4
CMX (MiT-B4) | RGB-AoLP | 91.6 | 88.8 | 96.3 | 98.3 | 96.8 | 89.7 | 86.2 | 92.8 | 92.6
CMX (MiT-B4) | RGB-DoLP | 91.6 | 88.6 | 96.3 | 98.3 | 96.7 | 89.5 | 86.4 | 92.2 | 92.5
ShareCMP (MiT-B4) | RGB-{$\bm{I}_{0}$,$\bm{I}_{45}$,$\bm{I}_{90}$,$\bm{I}_{135}$} | 92.0 | 88.6 | 96.3 | 98.3 | 96.8 | 90.0 | 87.2 | 92.5 | 92.7

Table 5: Results on ZJU [37]. ShareCMP is trained for 500 epochs.

Results on ZJU. In Tab. 5, we perform experiments to compare the ShareCMP
method with the previous state-of-the-art methods [32, 40, 37, 45] on the ZJU
[37] dataset. Our ShareCMP outperforms the previous best RGB-P method by 0.2%
and 0.1% based on MiT-B2 [39] and MiT-B4, respectively. Compared to dual-
branch multimodal semantic segmentation models [40, 37, 45], our ShareCMP
beats them in segmentation performance with only about 65% of their
parameters.

Method | Modal |  Asphalt |  Concrete |  Metal |  Road Ma |  Fabric |  Glass |  Plaster |  Plastic |  Rubber |  Sand |  Gravel |  Ceramic |  Cobbles |  Brick |  Grass |  Wood |  Leaf |  Water |  Sky | mIoU$\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
DRConv [3] | RGB-A-D-N | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 34.63
DDF [53] | RGB-A-D-N | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 36.16
TransFuser [33] | RGB-A-D-N | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 37.66
MMTM [20] | RGB-A-D-N | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 39.71
FuseNet [14] | RGB-A-D-N | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 40.58
MCubeSNet [25] | RGB-A-D-N | 85.7 | 42.6 | 47.0 | 59.2 | 12.5 | 44.3 | 3.0 | 10.6 | 12.7 | 66.8 | 67.1 | 27.8 | 65.8 | 36.8 | 54.8 | 39.4 | 73.0 | 13.3 | 94.8 | 42.90
MCubeSNet | RGB-A | 83.3 | 42.3 | 43.0 | 58.4 | 8.8 | 27.3 | 0.6 | 9.8 | 12.0 | 55.5 | 57.7 | 18.1 | 64.6 | 36.6 | 56.5 | 34.8 | 71.8 | 6.8 | 95.0 | 39.10
CMNeXt (MiT-B2) [46] | RGB-A | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 48.42
ShareCMP (MiT-B2) | RGB-A | 88.2 | 49.3 | 51.0 | 66.1 | 20.7 | 50.1 | 1.2 | 30.9 | 18.2 | 64.2 | 75.2 | 28.5 | 74.9 | 45.7 | 59.4 | 43.9 | 74.1 | 69.9 | 95.6 | 50.34
MCubeSNet | RGB-D | 75.2 | 40.2 | 37.8 | 53.9 | 4.2 | 32.3 | 1.9 | 14.3 | 11.3 | 59.7 | 21.8 | 11.6 | 28.9 | 29.1 | 54.6 | 29.4 | 71.4 | 9.6 | 94.3 | 34.10
ShareCMP (MiT-B2) | RGB-D | 88.5 | 50.4 | 52.6 | 65.9 | 23.2 | 49.8 | 0.0 | 36.7 | 18.8 | 64.6 | 75.4 | 30.4 | 72.2 | 45.3 | 57.3 | 43.9 | 74.5 | 66.1 | 95.7 | 50.55
MCubeSNet | RGB-A-D | 83.0 | 42.6 | 45.5 | 59.8 | 17.0 | 44.2 | 1.2 | 18.6 | 4.8 | 54.8 | 51.5 | 26.4 | 67.6 | 41.9 | 57.0 | 39.4 | 74.0 | 15.5 | 95.3 | 42.00
CMNeXt (MiT-B2) | RGB-A-D | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 49.48
ShareCMP (MiT-B2) | RGB-A-D | 88.8 | 49.7 | 52.5 | 66.4 | 27.6 | 51.0 | 0.2 | 31.5 | 18.0 | 69.9 | 79.0 | 30.9 | 71.4 | 42.8 | 58.3 | 44.1 | 74.5 | 67.6 | 95.6 | 50.99

Table 6: Results on MCubeS [25]. The human body class is omitted as its result
is 0%.

Results on MCubeS. In Tab. 6, we benchmark 7 semantic segmentation methods on
the MCubeS [25] dataset, which contains AoLP, DoLP, and NIR modalities. Since
the MCubeS dataset does not provide corresponding four-angle polarized images,
our ShareCMP uses AoLP and DoLP as polarization modal inputs. On RGB-A and
RGB-D modal inputs, our ShareCMP outperforms the previous best method by 1.92%
on CMNeXt [46] and 16.45% on MCubeSNet [25]. ShareCMP achieves a state-of-the-
art performance of 50.99% when concatenating AoLP and DoLP modalities. In
addition, our ShareCMP (RGB-A-D) even outperforms previous methods [3, 53, 33,
20, 14, 25] using RGB-A-D-N modalities, by 8.09% compared to the best. The
results indicate that our ShareCMP has great potential in RGB-P multimodal
semantic segmentation.

###  4.5 Quantitative and Visual Analysis

Quantitative analysis of ShareCMP. In Tab. 7, our ShareCMP reduces the number
of parameters by 22.12 M compared to CMX [45], while the number of parameters
of the ShareCMP encoder is only 65.7% of the CMX encoder. Compared to CMNeXt
[46], our ShareCMP also significantly reduces the number of parameters while
increasing the computational complexity by only 0.77 G FLOPs. Compared to
these state-of-the-art methods, our ShareCMP reduces a large number of
parameters while still improving the segmentation performance, which also
proves the efficacy of our proposed ShareCMP framework.

Structure | #Params(M)$\downarrow$ | FLOPs(G)$\downarrow$
---|---|---
CMX [45] | 65.52 | 52.03
\- Encoder | 64.98 | 47.07
ShareCMP |  43.40(-22.12) | 66.19(+14.16)
\- Encoder |  42.72(-22.26) | 46.71
\- PGA | 0.16 | 14.53
CMNeXt [46] | 58.73 | 65.42
Table 7: Comparison of the number of parameters (#Params) and FLOPs which are
counted in $512\times 512$. Figure 4: Visualization of different polarization
modal representations. Figure 5: Visualization of segmentation results. RGB-FP
represents RGB and four polarized images with different polarization angles
(RGB-{$\bm{I}_{0}$,$\bm{I}_{45}$,$\bm{I}_{90}$,$\bm{I}_{135}$}).

Visualization of polarization modal representations. We show five polarization
modal representations, as in Fig. 4. Compared with SAoLP and CAoLP, AoLP and
DoLP cover the polarization information in SAoLP and CAoLP and show richer
polarization features, which also verifies that the segmentation results of
RGB-AoLP and RGB-DoLP are better than RGB-SAoLP and RGB-CAoLP in Tab. 4. While
$\bm{I}_{P}$ has rich polarization properties, it also provides different
color features for objects with different polarization properties, such as the
glass reflection and the reflector of the motorcycle, the car shell at
different angles in the second group of images, the glass and the sticker on
the car glass on the right side, and different colors of classes in the third
and fourth groups of images. The AoLP and SAoLP in the third group of images
only have very few polarization features, while our $\bm{I}_{P}$ can still
provide rich polarization features, indicating that our PGA module has good
robustness in generating polarization features. In addition, compared to
$\bm{I}_{P}$, the other four polarization modal representations are very
coarse in some areas and have a lot of noise points, while $\bm{I}_{P}$ has
smoother polarization features that eliminate a large number of noise points,
especially for the underwater polarization modal representations in the
UPLight dataset.
Visualization of segmentation results. In Fig. 5, we show the semantic
segmentation results of our ShareCMP against the CMX [45] with RGB-A and RGB-D
and the CMNeXt [46] with RGB-A and RGB-A-D. It can be seen that our ShareCMP
achieves a more complete segmentation for glass windows on MCubeS [25] and
persons on ZJU [37] images, all in the case of insufficient illumination. For
blurred underwater images in UPLight, CMX does not segment the starfish in the
image or segments the starfish and shells incompletely. Our ShareCMP is more
robust in underwater scenes and accurately segments all objects.

##  5 Conclusion

In this work, to revitalize multimodal pixel-wise semantic scene understanding
for Autonomous Underwater Vehicles (AUVs), we investigate RGB-P semantic
segmentation and propose ShareCMP, a shared dual-branch architecture, which
reduces a large number of model parameters. We put forward the UPLight RGB-P
multimodal dataset with 12 typical underwater semantic classes. We propose the
Polarization Generate Attention (PGA) module, which can stably generate
polarization modal images with rich polarization properties. The designed
Class Polarization-Aware Loss (CPALoss) improves the learning and
understanding of the encoder for polarization modal information. Our ShareCMP
achieves state-of-the-art performance with fewer parameters on three RGB-P
datasets.
Acknowledgment. This research is funded by the National Natural Science
Foundation of China, grant number 52371350, by the Natural Science Foundation
of Hainan Province, grant number 2021JJLH0002, by the Foundation of National
Key Laboratory, grant number JCKYS2022SXJQR-06, and 2021JCJQ-SYSJJ-LB06912.

## References

  * Baltaxe et al. [2023] Michael Baltaxe, Tomer Pe’er, and Dan Levi.  Polarimetric imaging for perception.  _arXiv preprint arXiv:2305.14787_ , 2023.
  * Borse et al. [2021] Shubhankar Borse, Ying Wang, Yizhe Zhang, and Fatih Porikli.  Inverseform: A loss function for structured boundary-aware segmentation.  In _CVPR_ , pages 5901–5911, 2021.
  * Chen et al. [2021] Jin Chen, Xijun Wang, Zichao Guo, Xiangyu Zhang, and Jian Sun.  Dynamic region-aware convolution.  In _CVPR_ , pages 8064–8073, 2021.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE TPAMI_ , 40(4):834–848, 2017.
  * Chen et al. [2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _ECCV_ , pages 801–818, 2018.
  * Chen et al. [2020] Xiaokang Chen, Kwan-Yee Lin, Jingbo Wang, Wayne Wu, Chen Qian, Hongsheng Li, and Gang Zeng.  Bi-directional cross-modality feature propagation with separation-and-aggregation gate for rgb-d semantic segmentation.  In _ECCV_ , pages 561–577, 2020.
  * Choi et al. [2020] Sungha Choi, Joanne T Kim, and Jaegul Choo.  Cars can’t fly up in the sky: Improving urban-scene segmentation via height-driven attention networks.  In _CVPR_ , pages 9373–9383, 2020.
  * Contributors [2020] MMSegmentation Contributors.  Mmsegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.  https://github.com/open-mmlab/mmsegmentation.
  * Daly et al. [2016] Ilse M Daly, Martin J How, Julian C Partridge, Shelby E Temple, N Justin Marshall, Thomas W Cronin, and Nicholas W Roberts.  Dynamic polarization vision in mantis shrimps.  _Nature Communications_ , 7(1):12140, 2016.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , pages 248–255, 2009.
  * Ding et al. [2019] Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang.  Boundary-aware feature propagation for scene segmentation.  In _ICCV_ , pages 6819–6829, 2019.
  * Fu et al. [2019] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In _CVPR_ , pages 3146–3154, 2019.
  * Ha et al. [2017] Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada.  Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes.  In _IEEE IROS_ , pages 5108–5115, 2017.
  * Hazirbas et al. [2017] Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers.  Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture.  In _ACCV_ , pages 213–228, 2017.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _ICCV_ , pages 1026–1034, 2015.
  * Hou et al. [2020] Qibin Hou, Li Zhang, Ming-Ming Cheng, and Jiashi Feng.  Strip pooling: Rethinking spatial pooling for scene parsing.  In _CVPR_ , pages 4003–4012, 2020.
  * Hu et al. [2019] Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang.  Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation.  In _ICIP_ , pages 1440–1444, 2019.
  * Huang et al. [2019] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In _ICCV_ , pages 603–612, 2019.
  * Jin et al. [2021] Zhenchao Jin, Tao Gong, Dongdong Yu, Qi Chu, Jian Wang, Changhu Wang, and Jie Shao.  Mining contextual information beyond image for semantic segmentation.  In _ICCV_ , pages 7231–7241, 2021.
  * Joze et al. [2020] Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuzzolino, and Kazuhito Koishida.  Mmtm: Multimodal transfer module for cnn fusion.  In _CVPR_ , pages 13289–13299, 2020.
  * Kalra et al. [2020] Agastya Kalra, Vage Taamazyan, Supreeth Krishna Rao, Kartik Venkataraman, Ramesh Raskar, and Achuta Kadambi.  Deep polarization cues for transparent object segmentation.  In _CVPR_ , pages 8602–8611, 2020.
  * Lei et al. [2022] Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, and Qifeng Chen.  Shape from polarization for complex scenes in the wild.  In _CVPR_ , pages 12632–12641, 2022.
  * Li et al. [2020] Xiangtai Li, Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, and Yunhai Tong.  Improving semantic segmentation via decoupled body and edge supervision.  In _ECCV_ , pages 435–452, 2020.
  * Liang et al. [2023] Mingjian Liang, Junjie Hu, Chenyu Bao, Hua Feng, Fuqin Deng, and Tin Lun Lam.  Explicit attention-enhanced fusion for rgb-thermal perception tasks.  _IEEE RAL_ , 2023.
  * Liang et al. [2022] Yupeng Liang, Ryosuke Wakaki, Shohei Nobuhara, and Ko Nishino.  Multimodal material segmentation.  In _CVPR_ , pages 19800–19808, 2022.
  * Lin et al. [2017] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid.  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation.  In _CVPR_ , pages 1925–1934, 2017.
  * Liu et al. [2023] Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, and Xin Fan.  Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation.  In _ICCV_ , pages 8115–8124, 2023.
  * Long et al. [2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _CVPR_ , pages 3431–3440, 2015.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _ICLR_ , 2019.
  * Marshall [1988] N Justin Marshall.  A unique colour and polarization vision system in mantis shrimps.  _Nature_ , 333(6173):557–560, 1988.
  * Mei et al. [2022] Haiyang Mei, Bo Dong, Wen Dong, Jiaxi Yang, Seung-Hwan Baek, Felix Heide, Pieter Peers, Xiaopeng Wei, and Xin Yang.  Glass segmentation using intensity and spectral polarization cues.  In _CVPR_ , pages 12622–12631, 2022.
  * Orsic et al. [2019] Marin Orsic, Ivan Kreso, Petra Bevandic, and Sinisa Segvic.  In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images.  In _CVPR_ , pages 12607–12616, 2019.
  * Prakash et al. [2021] Aditya Prakash, Kashyap Chitta, and Andreas Geiger.  Multi-modal fusion transformer for end-to-end autonomous driving.  In _CVPR_ , pages 7077–7087, 2021.
  * Qian et al. [2021] Yeqiang Qian, Liuyuan Deng, Tianyi Li, Chunxiang Wang, and Ming Yang.  Gated-residual block for semantic segmentation using rgb-d data.  _IEEE TITS_ , 23(8):11836–11844, 2021.
  * Strudel et al. [2021] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In _ICCV_ , pages 7262–7272, 2021.
  * Takikawa et al. [2019] Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler.  Gated-scnn: Gated shape cnns for semantic segmentation.  In _ICCV_ , pages 5229–5238, 2019.
  * Xiang et al. [2021] Kaite Xiang, Kailun Yang, and Kaiwei Wang.  Polarization-driven semantic segmentation via efficient attention-bridged fusion.  _Optics Express_ , 29(4):4802–4820, 2021.
  * Xie et al. [2021a] Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu, Ding Liang, and Ping Luo.  Segmenting transparent object in the wild with transformer.  _arXiv preprint arXiv:2101.08461_ , 2021a.
  * Xie et al. [2021b] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  _NeurIPS_ , 34:12077–12090, 2021b.
  * Yan et al. [2021] Ran Yan, Kailun Yang, and Kaiwei Wang.  Nlfnet: non-local fusion towards generalized multimodal semantic segmentation across rgb-depth, polarization, and thermal images.  In _IEEE ROBIO_ , pages 1129–1135, 2021.
  * Yu et al. [2020] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang.  Context prior for scene segmentation.  In _CVPR_ , pages 12416–12425, 2020.
  * Yuan et al. [2021] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang.  Ocnet: Object context for semantic segmentation.  _IJCV_ , 129(8):2375–2398, 2021.
  * Zhang et al. [2018] Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal.  Context encoding for semantic segmentation.  In _CVPR_ , pages 7151–7160, 2018.
  * Zhang et al. [2021a] Jiaming Zhang, Kailun Yang, and Rainer Stiefelhagen.  Issafe: Improving semantic segmentation in accidents by fusing event-based data.  In _IEEE IROS_ , pages 1132–1139, 2021a.
  * Zhang et al. [2023a] Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen.  Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers.  _IEEE TITS_ , 2023a.
  * Zhang et al. [2023b] Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen.  Delivering arbitrary-modal semantic segmentation.  In _CVPR_ , pages 1136–1147, 2023b.
  * Zhang et al. [2021b] Qiang Zhang, Shenlu Zhao, Yongjiang Luo, Dingwen Zhang, Nianchang Huang, and Jungong Han.  Abmdrnet: Adaptive-weighted bi-directional modality difference reduction network for rgb-t semantic segmentation.  In _CVPR_ , pages 2633–2642, 2021b.
  * Zhang et al. [2023c] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue.  Meta-transformer: A unified framework for multimodal learning.  _arXiv preprint arXiv:2307.10802_ , 2023c.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In _CVPR_ , pages 2881–2890, 2017.
  * Zheng et al. [2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In _CVPR_ , pages 6881–6890, 2021.
  * Zhou et al. [2020] Hao Zhou, Lu Qi, Zhaoliang Wan, Hai Huang, and Xu Yang.  Rgb-d co-attention network for semantic segmentation.  In _ACCV_ , 2020.
  * Zhou et al. [2022] Hao Zhou, Lu Qi, Hai Huang, Xu Yang, Zhaoliang Wan, and Xianglong Wen.  Canet: Co-attention network for rgb-d semantic segmentation.  _PR_ , 124:108468, 2022.
  * Zhou et al. [2021] Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Ming-Hsuan Yang.  Decoupled dynamic filter networks.  In _CVPR_ , pages 6647–6656, 2021.
  * Zhou et al. [2019] Wei Zhou, Julie Stephany Berrio, Stewart Worrall, and Eduardo Nebot.  Automated evaluation of semantic segmentation robustness for autonomous driving.  _IEEE TITS_ , 21(5):1951–1963, 2019.
  * Zhuang et al. [2021] Zhuangwei Zhuang, Rong Li, Kui Jia, Qicheng Wang, Yuanqing Li, and Mingkui Tan.  Perception-aware multi-sensor fusion for 3d lidar semantic segmentation.  In _ICCV_ , pages 16280–16290, 2021.