# PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation
using Contextual Regularization and Cross Knowledge Distillation

Shuwei Zhai  Guotai Wang  Xiangde Luo  Qiang Yue  Kang Li and Shaoting Zhang
This work was supported by the National Natural Science Foundation of China
(61901084, 62271115), National Key Research and Development Program of China
(2020YFB1711500) and the 1·3·5 project for disciplines of excellence, West
China Hospital, Sichuan University (ZYYC21004). (Corresponding author: Guotai
Wang) S. Zhai, G. Wang, X. Luo and S. Zhang are with the School of Mechanical
and Electrical Engineering, University of Electronic Science and Technology of
China, Chengdu, 611731, China. G. Wang, X. Luo and S. Zhang are also with
Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China (e-mail:
guotai.wang@uestc.edu.cn).Q. Yue is with Department of Radiology, West China
Hospital, Sichuan University, Chengdu, 610041, ChinaK. Li is with West China
Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu
610041, ChinaThis work has been submitted to IEEE TMI for possible
publication. Copyright may be transferred without notice, after which this
version may no longer be accessible.

###### Abstract

The success of Convolutional Neural Networks (CNNs) in 3D medical image
segmentation relies on massive fully annotated 3D volumes for training that
are time-consuming and labor-intensive to acquire. In this paper, we propose
to annotate a segmentation target with only seven points in 3D medical images,
and design a two-stage weakly supervised learning framework PA-Seg. In the
first stage, we employ geodesic distance transform to expand the seed points
to provide more supervision signal. To further deal with unannotated image
regions during training, we propose two contextual regularization strategies,
i.e., multi-view Conditional Random Field (mCRF) loss and Variance
Minimization (VM) loss, where the first one encourages pixels with similar
features to have consistent labels, and the second one minimizes the intensity
variance for the segmented foreground and background, respectively. In the
second stage, we use predictions obtained by the model pre-trained in the
first stage as pseudo labels. To overcome noises in the pseudo labels, we
introduce a Self and Cross Monitoring (SCM) strategy, which combines self-
training with Cross Knowledge Distillation (CKD) between a primary model and
an auxiliary model that learn from soft labels generated by each other.
Experiments on public datasets for Vestibular Schwannoma (VS) segmentation and
Brain Tumor Segmentation (BraTS) demonstrated that our model trained in the
first stage outperformed existing state-of-the-art weakly supervised
approaches by a large margin, and after using SCM for additional training, the
model’s performance was close to its fully supervised counterpart on the BraTS
dataset.

Medical image segmentation, weakly supervised learning, noisy label, knowledge
distillation.

Figure 1: Illustration of our point annotation-based segmentation. Note that
our method is implemented in 3D and the figure shows a 2D view. The annotation
for each object consists of six points in different axis directions in the
background (four of them are shown in the 2D slice) and one in the foreground.
The sparse annotations are expanded based on geodesic distance transform of
the foreground seed and bounding box derived from the background seeds. Pseudo
labels are generated by a model trained with the expanded annotations.

##  1 Introduction

Medical image segmentation plays a vital role in computer-assisted diagnosis
and treatment planning of many diseases. In recent years, deep Convolutional
Neural Networks (CNNs) have achieved breakthrough performance for segmentation
when fully annotated datasets are available [1, 2]. However, their success
relies on large-scale training images with dense annotations that are time-
consuming to obtain and rely on experts with domain knowledge [3].

To reduce the annotation cost for medical image segmentation, increasing
efforts have been made to exploit weak or sparse annotations that are much
more efficient and easier to obtain than dense pixel-level annotations, e.g.,
image-level annotations [4, 5], scribbles [6, 7], bounding boxes [8, 9] and
point annotations [10, 11]. Despite the efficiency of image-level annotations,
their resulted segmentation performance is usually very limited due to the
extremely weak supervision. Scribbles are widely used for efficient annotation
of 2D images [6, 7], and they provide labels for a sparse set of pixels of
each class for training. For 3D volumes, scribble annotations need to be
provided for many slices, which is still time-consuming. In contrast, bounding
box annotations [8, 9] are cheaper than scribbles and can provide coarse
location of the target for training a better segmentation model than image-
level annotations. However, such annotations are faced with insufficient
labeled foreground pixels for training, and cannot accurately approximate the
target boundary for complex shapes, which will misguide the segmentation
model. Annotation with few points has a much lower cost than scribbles, and
can better deal with complex shapes compared with bounding box annotations,
thus has a potential for reducing the annotation cost while achieving good
performance. For example, some researchers [10, 11] used extreme points for
annotating 3D objects, where some paths between extreme points are used as
foreground annotations for training a segmentation model. However, the exact
extreme points in 3D space are hard to find, especially for 3D medical images
with a low contrast and tumors with irregular shapes, which increases the time
cost of annotators and may result in inaccurate extreme points that limit the
training process.

Despite its efficiency for annotation, there are two main challenges for
learning from weak annotations for segmentation of 3D objects. First, due to
the large number of unannotated pixels, the segmentation model tends to over-
fit the small number of annotated pixels if any. To tackle this problem, some
works utilize prior information for regularization, such as size constraint
[12] and tightness prior [9]. Such regularization strategies help to improve
the model’s performance, but their applicability is still subject to certain
assumptions, e.g., the segmentation target has a regular shape. Second, pseudo
dense annotations may be generated from the weak annotations, but they usually
contain errors (noises) that will cause severe performance degradation [13,
14]. Several approaches have been proposed to deal with this problem[15, 16],
and most of them assume samples with high loss values are likely to be noisy
and select those with low loss values for robust learning. However, it is
difficult to select a reliable loss threshold, and these methods can hardly
distinguish noisy samples from hard samples, which limits the model’s
performance.

In this work, we propose a novel point annotation-based weakly supervised
method named as PA-Seg for 3D medical image segmentation. It only requires
seven points (seeds) to annotate a 3D segmentation target, i.e., six points on
the background in different axis directions (left, right, anterior, posterior,
inferior and superior) and one point on the foreground. An example of the
axial view of our annotation is shown in the first column of Fig. 1. Compared
with bounding box annotation, our annotation provides extra foreground
information at the cost of only one additional mouse click. In addition,
providing the background points is more friendly and easier than finding the
extreme points for annotators.

The training process of PA-Seg consists of two stages: the first stage uses
the sparse point annotations to train an initial segmentation model that
generates dense pseudo labels for training images, and the second stage trains
an enhanced model based on the pseudo labels and noise-robust learning. A
challenge in the first stage is that the seven points cannot be directly used
for training due to the insufficient supervision signal. To deal with this
problem, we utilize geodesic distance transform [17, 18] to expand the raw
annotation seeds to derive the annotations of more pixels, as shown in the
third column of Fig. 1. The expanded annotations with fewer unannotated pixels
are used to train the initial segmentation model, where we introduce
contextual regularization on the pixels with unknown labels based on a multi-
view Conditional Random Field (mCRF) loss and a Variance Minimization (VM)
loss. Training with our seed expansion strategy and regularization methods can
obtain a high-performance initial segmentation model that is superior to
several state-of-the-art weakly supervised methods. The initial model
generates high-quality pseudo labels for training images, as shown in the last
column of Fig. 1.

In the second stage, to further learn from the noisy pseudo labels, we
introduce an auxiliary network to assist training of the primary network, and
both of them are pre-trained by the method in the first stage. They are then
trained by our proposed Self and Cross Monitoring (SCM) that combines self-
training and Cross Knowledge Distillation (CKD). The CKD is inspired by
Knowledge Distillation (KD) [19], where soft labels predicted by a teacher
model contain more information than the hard labels that may contain noises.
Differently from typical KD methods [19, 20] that focus on image
classification problems and only distill the knowledge from the teacher to the
student, our CKD lets two networks distill knowledge from each other for image
segmentation, where the different decision boundaries of two models can
regularize each other and reduce the risk of over-fitting in a single model.

###  1.1 Contributions

The contributions of this paper are three-fold. Firstly, we propose a novel
framework PA-Seg for weakly supervised 3D medical image segmentation, where
only seven points are needed to annotate a 3D object, and a two-stage method
based on pseudo label generation and learning from noisy pseudo labels is
proposed for training a segmentation model with high performance and very low
annotation cost. Secondly, to deal with the extremely sparse annotation seeds,
we propose a geodesic distance-based annotation expansion strategy, and
introduce contextual regularization based on multi-view conditional random
field and variance minimization to train a good initial segmentation model.
Thirdly, we propose self and cross monitoring that combines self-training and
cross knowledge distillation for learning from noisy pseudo labels in the
scenario of weakly supervised segmentation. Extensive experiments on two
public datasets for Vestibular Schwannoma (VS) segmentation and Brain Tumor
Segmentation (BraTS) demonstrated that our method outperformed several state-
of-the-art weakly supervised techniques, and it was comparable to fully
supervised setting on the BraTS dataset.

##  2 Related Works

###  2.1 Weakly Supervised Segmentation Using CNNs

Existing weakly supervised CNNs for segmentation can be roughly divided into
two categories. The first category of methods use traditional interactive
segmentation techniques (e.g, Grabcut [21] and random walker [22]) to produce
“fake” dense masks (i.e., pseudo labels) from the given weak annotations like
bounding boxes and scribbles. These pseudo labels are then used to train CNNs
in a fully-supervised manner [8, 6]. For example, ScribbleSup [6]
alternatively propagates scribble information to unlabeled pixels and
optimizes the network parameters. However, the pseudo labels are usually not
correct and contain a lot of noises, and standard segmentation loss functions
such as cross-entropy and Dice losses may over-fit the noises, leading to
limited performance [16, 15].

The second category of methods combine partial cross entropy loss for
annotated pixels with regularization for unannotated pixels. Kervadec [12]
used size loss to constrain volume of the predicted mask, and employed deep
tightness prior [9] for learning from bounding box annotations. Tang et al.
[13, 14] proposed normalized cut loss and Conditional Random Field (CRF) loss
to regularize unannotated pixels. However, CRF loss involves pairwise energy
for any pair of pixels in the image, which is not suitable for 3D images due
to the huge memory and computational cost.

###  2.2 Co-learning Framework

Co-learning is a kind of framework where two networks or two branches of a
single network are used for training with imperfect annotations, and it has
been recently used for semi-supervised [23, 24], weakly-supervised [25], and
noise-robust learning [26]. For semi-supervised learning, the Mean Teacher
(MT) framework [27, 23] utilized an exponential moving average of a student as
the teacher to provide supervision signal on unannotated images. Wang et al.
[28] let two or more models focus on easier-to-segment regions first, and then
gradually consider harder ones, while enforcing predictions from different
models to be consistent. Zhao et al. [29] applied MT to unsupervised domain
adaptation, where the student learns from two teacher models each from a
different domain in a semi-supervised manner. The Cross Pseudo Supervision
(CPS) [24] lets two networks generate pseudo labels for each other in semi-
supervised learning.

For weakly-supervised learning, Liu et al. [30] used a strongly supervised
branch to teach a less supervised branch. Luo et al. [25] employed a dynamic
mixture of predictions from two branches as the supervision. Patel et al. [31]
proposed cross-modality equivariant constraints for weakly supervised
segmentation. For noise-robust learning, Co-teaching [26] uses two networks
each selects samples with small loss values to supervise the other. Xue et al.
[32] extended co-teaching [26] by using three networks to select and correct
noisy samples for robust learning. Wei et al. [33] proposed to use symmetric
Kullback-Leibler divergence in addition to cross-entropy loss to select small-
loss instances. However, most existing works use two or more networks with the
same architecture for co-learning. The asymmetry design is important for
improving the performance of co-learning. Existing works have designed
asymmetry based on different modalities [31], processing inputs with different
noises or augmentation [27, 23], using different objectives [34] or prediction
representations [35]. In this work, we introduce architecture-level asymmetry,
i.e, use two networks with different architectures to avoid feature bias of a
single network, and combine them with cross knowledge distillation to better
deal with noisy pseudo labels.

###  2.3 Learning from Noisy Labels and Knowledge Distillation

Noisy labels exist in either inaccurate manual annotations or pseudo masks
generated by a trained model. Due to the strong fitting ability of CNNs,
training with such labels using standard fully supervised learning will over-
fit the noises and limit the performance of the segmentation model [36].
Existing noise-robust learning methods can be roughly grouped into two
categories. The first is label filtering and sample re-weighting by using two
or multiple branches as mentioned above [26, 32, 33]. The second is to design
noise-robust loss functions, such as Mean Absolute Error (MAE) loss [37],
Generalized Cross-Entropy (GCE) loss [15] and noise-robust Dice loss [16].

Knowledge Distillation (KD) [19] was originally proposed for transferring
knowledge from a complex teacher model to a lightweight student. In medical
imaging, KD has been applied to different tasks where the knowledge is
distilled from one model/branch to improve the performance of another. Wang et
al. [38] used KD to boost a lightweight student network for computationally
efficient segmentation of 3D volumes. Patel et al. [31] used cross-modality KD
to improve the accuracy of Class Activation Maps (CAMs) for learning from
image-level labels. Dou et al. [39] employed KD to explicitly constrain the
KL-divergence of prediction distributions between two different modalities.

KD [19] typically uses soft labels based on temperature scaling instead of
hard one-hot ones from the teacher to supervise the student. As soft labels
contain more information about the relationship between different classes,
they can lead to improved performance of the student compared with only
learning from standard one-hot hard labels that may contain noises. A similar
conclusion was found in [40] where label smoothing helps to improve the
model’s performance when the labels are inaccurate. Kats et al. [41] found
that soft labels constructed by morphological dilation of binary annotation
masks improved the performance of brain lesion segmentation. Sarfraz et al.
[20] showed the effectiveness of KD in learning efficiently under various
degrees of label noise and class imbalance. Li et al. [42] leveraged the
knowledge learned from a small clean dataset to guide the learning from a
noisy dataset. However, such a clean dataset is not available in weakly
supervised setting. In addition, most existing methods using KD for noise-
robust learning are proposed for image classification tasks, and their
applicability to 3D medical image segmentation has not been well investigated.

Figure 2: An overview of the proposed PA-Seg for weakly supervised 3D
segmentation based on point annotations. In the first stage (a), we expand the
annotation seeds based on geodesic distance transform, and train an initial
model using the expanded seeds, with the unlabeled pixels regularized by
multi-view CRF loss and Variance Minimization (VM) loss. Pseudo labels are
then obtained by using the initial model for inference. In the second stage
(b), to deal with noises in the pseudo labels, we propose Self and Cross
Monitoring (SCM), where a primary model and an auxiliary model supervise each
other via Cross Knowledge Distillation (CKD) based on soft labels, in addition
to self-training of each model.

##  3 Method

Fig. 2 shows an overview of our proposed PA-Seg for weakly supervised 3D
segmentation using seven points for annotation (six on the background and one
on the foreground). In the first stage, we utilize geodesic distance transform
to expand the extremely sparse seed points, and combine multi-view Conditional
Random Field (mCRF) loss with Variance Minimization (VM) loss to train an
initial segmentation model that can produce high-quality pseudo labels for
training images. In the second stage, the segmentation model is further
boosted by learning from the pseudo labels. We design a novel procedure named
Self and Cross Monitoring (SCM) that combines self-training and Cross
Knowledge Distillation (CKD) to alleviate the effect of noises in the pseudo
labels and obtain better segmentation results.

###  3.1 Learning from Point Annotation

####  3.1.1 Annotation Strategy

The first column of Fig. 1 shows our proposed annotation strategy based on
points. Note that the figure shows a 2D view and our method works in 3D. For a
3D object to annotate, the annotator just needs to provide seven points, i.e.,
one inside the target object as foreground seed and six outside the target as
background seeds. The foreground seed is provided near the center of the
object, but the annotator does not need to find the exact centroid for
convenience. The six background seeds are provided in different axis
directions, i.e., the left, right, anterior, posterior, inferior and superior
side out of the object. Note that each background seed should be away from the
corresponding extreme point, e.g., the x-axis coordinate of the left/right
seed should be smaller/larger than that of the most left/right point in the
object. The background seeds are so provided that a bounding box determined by
them will cover the entire target object. This requirement can be easily
satisfied when the annotator estimates the extreme points roughly and expands
them to some extent for clicking.

Compared with providing extreme points [10, 11], our annotation strategy
allows more freedom for the annotator who does not need to find the exact 3D
extreme points, which leads to higher efficiency and convenience for
annotating. In 3D medical images, it is very difficult to find the accurate
positions of extreme points in 3D space, especially for tumors with irregular
shapes. As shown in Fig. 3 (a), the extreme points identified in one slice by
the annotator may not be those for the 3D object, which leads to
underestimated bounding boxes that only cover a sub-region of the 3D object.
In contrast, as shown in Fig. 3 (b), our method does not require the user to
provide exact extreme points, but just some relaxed points outside the
visually estimated bounding box. A bounding boxed derived from these points
will be a superset of the real bounding box, which can tolerate inaccurate
points and ease the annotation.

Figure 3: Comparison between extreme points and background points for
annotation. The two rows are for VS and glioma, respectively. The green points
are annotations, and the dotted lines are the corresponding bounding boxes
overlaid on nearby slices.

####  3.1.2 Seed Expansion

As only seven pixels are labeled in a 3D image, they cannot be directly used
to train a segmentation model due to the insufficient supervision signal
compared with millions of parameters in the CNN. Therefore, we expand the
seeds to reduce the number of unlabeled pixels automatically before training.

Firstly, based on our annotation strategy, pixels outside the bounding box
determined by the background seeds can be safely considered as the background
region. Let $\Omega$ denote the entire image space, $\Omega_{in}$ and
$\Omega_{out}$ denote the regions inside and the outside the bounding box,
respectively. Note that in some segmentation tasks, the input image may
contain some air background or be prepossessed by removing some undesired
regions like the skull in the BraTS dataset, leading to zero values around the
valid image region (e.g., the brain), as shown in the second row of Fig. 1.
Let $\Omega_{M}$ denote the masked valid region that excludes the undesired
background. We therefore take a union of $\Omega_{out}$ and
$\Omega-\Omega_{M}$ as the set of background pixels:
$\Omega_{B}=\Omega_{out}\cup(\Omega-\Omega_{M})$.

Secondly, the foreground seed is expanded to some degree to obtain more
foreground pixels with the assumption that pixels close to the foreground seed
with a similar intensity are likely to be foreground as well. As Euclidean
distance does not contain contextual information and cannot deal with objects
with non-ellipsoid shapes, geodesic distance [17, 18] is used to deal with
this problem since it can distinguish adjacent pixels with different
appearances and improves label consistency in homogeneous regions. Let $X$ and
$i_{f}$ denote a training image and the foreground seed, respectively. For a
pixel in $i\in\Omega-\Omega_{B}$, the geodesic distance between $i$ and
$i_{f}$ in $X$ is:

$\mathcal{D}(i,i_{f})=\min_{p\in\mathcal{P}_{i,i_{f}}}{\int_{0}^{1}||\nabla{X}(p(n))\cdot\mathbf{v}(n)||dn}$
(1)

where $\mathcal{P}_{i,i_{f}}$ is the set of all possible paths between $i$ and
$i_{f}$. $p$ is one feasible path and parameterized by $n\in[0,1]$.
$\mathbf{v}(n)$ is a unit vector tangent to the direction of the path at
$p(n)$. The maximal geodesic distance value in $\Omega-\Omega_{B}$ is denoted
as $\mathcal{D}_{max}$. We use $\Omega_{F}$ to denote the set of expanded
foreground seeds, and it is obtained by:

$\Omega_{F}=\\{i|i\in\Omega-\Omega_{B}~{}\text{and}~{}\mathcal{D}(i,i_{f})<\epsilon\mathcal{D}_{max}\\}$
(2)

where $\epsilon\in[0,1]$ is a hyper-parameter to control the degree of seed
expansion. $\epsilon=0$ means no expansion, and $\epsilon=1$ means expansion
to the entire region of $\Omega-\Omega_{B}$. For simplicity, we denote the set
of labeled pixels as $\Omega_{L}=\Omega_{B}\cup\Omega_{F}$, and denote the set
of pixels with unknown labels as $\Omega_{U}=\Omega-\Omega_{L}$. An example of
the seed expansion result is shown in the third column of Fig. 1, where the
green and red colors show $\Omega_{B}$ and $\Omega_{F}$, respectively, and the
remaining pixels belong to $\Omega_{U}$.

####  3.1.3 Partially Supervised Loss

After seed expansion, we obtain a partial label map $Q$ where values for
pixels in $\Omega_{F}$ and $\Omega_{B}$ are 1 and 0, respectively. Then $Q$
can be used to train a segmentation network via a partially supervised loss:

$\mathcal{L}_{psup}(P,Q)=\big{(}\mathcal{L}_{pce}(P,Q)+\mathcal{L}_{pdc}(P,Q)\big{)}/2$
(3)

Where $P$ is the foreground probability map predicted by the segmentation
model. $\mathcal{L}_{pce}$ and $\mathcal{L}_{pdc}$ are partial cross-entropy
and partial Dice losses, respectively:

$\mathcal{L}_{pce}(P,Q)=\frac{1}{|\Omega_{L}|}\sum_{i\in\Omega_{L}}-\big{(}q_{i}\log(p_{i})+(1-q_{i})\log(1-p_{i})\big{)}$
(4)
$\mathcal{L}_{pdc}(P,Q)=1-\frac{2\sum_{i\in\Omega_{L}}{p_{i}q_{i}}}{\sum_{i\in\Omega_{L}}({p_{i}}^{2}+{q_{i}}^{2})}$
(5)

where $p_{i}$ is the foreground probability of pixel $i$ in $P$, and $q_{i}$
is the corresponding value in $Q$. Here we follow the common practice of
combing the cross entropy loss with Dice loss for labeled pixels [29], where
the former has a large gradient for each pixel, and the latter can better deal
with the class imbalance problem for segmentation tasks. Note that in Eq. 4
and Eq. 5, the loss values are only calculated in $\Omega_{L}$, and pixels
with unknown labels are ignored.

####  3.1.4 Multi-view CRF Loss

Dense CRF has been widely used for post-processing in segmentation due to its
ability to capture long-range dependency. In dense CRF, a pairwise potential
is defined between two different pixels in a 2D image to encourage consistent
labels between pixels with similar features:

$\psi(y_{i},y_{j})=\mu(y_{i},y_{j})K_{ij},~{}K_{ij}=\sum_{n=1}^{N}w_{n}k_{n}(\mathbf{f}^{(n)}_{i},\mathbf{f}^{(n)}_{j})$
(6)

where $y_{i}$ is the label of pixel $i$. $\mu$ is a class compatibility matrix
based on the Potts model, i.e., $\mu(y_{i},y_{j})=1$ if $y_{i}\neq y_{j}$ and
0 otherwise. $K_{ij}$ is the discontinuity cost based on a mixture of $N$
kernels. $w_{n}$ is the weight of the $n$-th kernel $k_{n}$ that is commonly
implemented by a Gaussian kernel [43], and the feature vector
$\mathbf{f}^{(n)}_{i}$ is specific to $k_{n}$, such as pixel intensity and
spatial coordinate used in the bilateral filter. Following [14], the Potts
model can be relaxed from hard $y_{i}$ to soft $p_{i}$. A quadratic relaxation
leads to the 2D dense CRF loss:

$\mathcal{L}_{crf}(P^{(s)})=\sum_{i,j\in\Omega^{(s)}}p_{i}(1-p_{j})K_{ij}$ (7)

where $P^{(s)}$ is the $s$-th slice of a 3D prediction map $P$, and
$\Omega^{(s)}$ denotes the set of pixels in that slice.

The CRF loss can be directly used to regularize the network’s prediction
during training, which is more efficient than using dense CRF for post-
processing after each round of training [8].

However, as the CRF loss involves $K_{ij}$ for any pair of pixels in the
image, it leads to huge computational and memory cost, which is not suitable
for 3D images. Assume the 3D image has a shape of $D\times H\times W$, then
the shape of the affinity matrix $K=[K_{ij}]$ will be $(DHW)\times(DHW)$,
which is inefficient and even impossible to compute for large 3D volumes. To
enable 3D contextual regularization and avoid the computational problem, we
introduce multi-view CRF loss that applies the CRF loss in axial, sagittal and
coronal views, respectively. The axial CRF loss for a 3D volume is defined as
the average CRF loss for the $D$ slices:
$\mathcal{L}^{A}_{crf}=\big{(}\sum_{d}\mathcal{L}_{crf}(P^{(d)})\big{)}/D$.
Similarly, the sagittal and coronal CRF losses are denoted as
$\mathcal{L}^{S}_{crf}$ and $\mathcal{L}^{C}_{crf}$, respectively, and they
are the average 2D CRF loss across the $W$ sagittal and $H$ coronal slices,
respectively. The proposed multi-view CRF loss is:

$\mathcal{L}_{mcrf}(P)=\frac{1}{3}\big{(}\mathcal{L}^{A}_{crf}(P)+\mathcal{L}^{S}_{crf}(P)+\mathcal{L}^{C}_{crf}(P)\big{)}$
(8)

####  3.1.5 Variance Minimization

Motivated by the Chan-Vese level set model [44] for unsupervised segmentation,
we use a similar formulation to regularize the segmentation by minimizing the
intensity variance in the segmented foreground and background regions,
respectively. Here we denote the average intensities for the segmented
foreground and background as $u_{f}$ and $u_{b}$, respectively. They are
obtained by:

$u_{f}=\frac{\sum_{i\in\Omega_{M}}{x_{i}p_{i}}}{\sum_{i\in\Omega_{M}}{p_{i}}},~{}~{}u_{b}=\frac{\sum_{i\in\Omega_{M}}{x_{i}(1-p_{i})}}{\sum_{i\in\Omega_{M}}{(1-p_{i})}}$
(9)

where $x_{i}$ is the intensity of pixel $i$ in the input image. Note that we
only calculate the average across the valid image region $\Omega_{M}$ to
exclude influence of the undesired image region (e.g., air background). The
Variance Minimization (VM) loss is:

$\displaystyle\mathcal{L}_{vm}(X,P)$
$\displaystyle=\frac{\sum_{i\in\Omega_{M}}{(x_{i}-u_{f})^{2}p_{i}}}{\sum_{i\in\Omega_{M}}{p_{i}}}$
(10)
$\displaystyle+\frac{\sum_{i\in\Omega_{M}}{(x_{i}-u_{b})^{2}(1-p_{i})}}{\sum_{i\in\Omega_{M}}{(1-p_{i})}}$

####  3.1.6 Overall Loss

The overall loss for training the intial segmentation model in the first stage
is summarized as:

$\displaystyle\mathcal{L}_{seg}$
$\displaystyle=\mathcal{L}_{psup}(P,Q)+\alpha\mathcal{L}_{mcrf}(P)+\beta\mathcal{L}_{vm}(X,P)$
(11)

where $\alpha$ and $\beta$ are hyper-parameters that control the weights of
the corresponding losses.

###  3.2 Noise-robust Learning Using Self and Cross Monitoring

The initial model trained in the first stage can generate pseudo labels for
training images to train an enhanced model. To deal with noises in the pseudo
labels, we design a novel Self and Cross Monitoring (SCM) method to improve
the model’s performance, as illustrated in Fig. 2(b). The idea is inspired by
Knowledge Distillation [19], where soft labels obtained by a teacher can
enhance the student’s performance. Differently from typical KD methods that
fix the teacher and only update the student, we introduce Cross Knowledge
Distillation (CKD) so that two models can teach each other with soft labels.
The motivation is that two different networks have different decision
boundaries, and an interaction between them can regularize each other and
reduce the risk of over-fitting caused by the bias of a single network.

Let $\theta^{a}$ denote a primary network and $\theta^{b}$ denote an auxiliary
network, and they have been pre-trained independently in the first stage. Let
$Z^{a}$ and $Z^{b}$ denote the logit map of the penultimate layer of the two
networks, respectively. Passing $Z^{a}$ or $Z^{b}$ to a standard Softmax layer
leads to a probability prediction, $P^{a}$ or $P^{b}$. For simplicity, we use
$P^{a}$ and $P^{b}$ to denote the foreground probability maps from the two
networks, respectively. The corresponding hard labels based on $argmax(Z)$ for
the two pre-trained networks are denoted as $\hat{Y}^{a}$ and $\hat{Y}^{b}$,
respectively, i.e, pseudo labels. At the same time, a soft prediction is
obtained by sending the logit map to a temperature-calibrated Softmax layer
for KD [19]:

$\tilde{p}_{i}=\frac{e^{z_{i,f}/T}}{e^{z_{i,f}/T}+e^{z_{i,b}/T}}$ (12)

where $z_{i,f}$ and $z_{i,b}$ are the logit scores for the foreground and
background at pixel $i$, respectively. $\tilde{p}_{i}$ is the soft foreground
probability at pixel $i$, and $T$ is the temperature parameter. Note that a
standard Softmax corresponds to $T=1$, and a higher $T$ produces a softer
probability output. As shown in previous works [45, 46, 47], temperature
scaling with $T>1$ plays an important role in reducing the effect of noise, as
it avoids over-confidence of CNNs. In our task, the pseudo labels from the
teacher may contain inaccurate parts that will misguide the student, softening
the pseudo labels can avoid over-confidence of the teacher for noise-robust
learning.

Let $\tilde{P}^{a}$ and $\tilde{P}^{b}$ denote the “soft” foreground
probability maps of the two networks, respectively. When taking $\theta^{a}$
and $\theta^{b}$ as the student and the teacher respectively, the KD loss
based on Kullback Leibler (KL) divergence is:

$\mathcal{L}_{KD}(\tilde{P}^{a},\tilde{P}^{b})=\frac{1}{|\Omega|}\sum_{i\in\Omega}{\tilde{p}_{i}^{b}}\log(\frac{\tilde{p}_{i}^{b}}{\tilde{p}_{i}^{a}})$
(13)

Note that the above loss only back-propagates to $\theta^{a}$ and the gradient
is stopped for $\theta^{b}$ (the teacher). Symmetrically, we can switch the
teacher and student, and the KD loss for distilling knowledge from
$\theta^{a}$ to $\theta^{b}$ is denoted as
$\mathcal{L}_{KD}(\tilde{P}^{b},\tilde{P}^{a})$. The overall losses for
training $\theta^{a}$ and $\theta^{b}$ are:

$\mathcal{L}_{\theta^{a}}=\lambda\mathcal{L}_{KD}(\tilde{P}^{a},\tilde{P}^{b})+(1-\lambda)\mathcal{L}_{self}({{P}^{a}},{\hat{Y}^{a}})$
(14)
$\mathcal{L}_{\theta^{b}}=\lambda\mathcal{L}_{KD}(\tilde{P}^{b},\tilde{P}^{a})+(1-\lambda)\mathcal{L}_{self}({P}^{b},\hat{Y}^{b})$
(15)

where ${L}_{self}$ is a self-training term that uses pseudo labels
$\hat{Y}^{a}$ and $\hat{Y}^{b}$ to supervise the two networks, respectively.
It is implemented by a combination of cross-entropy loss and Dice loss for
standard fully supervised learning. $\lambda$ is a hyper-parameter to balance
the contribution of self-training and KD.

##  4 Experiments and results

###  4.1 Datasets

We validated our proposed PA-Seg with two applications: 1) Segmentation of
vestibular schwannoma from Magnetic Resonance Imaging (MRI) scans, and 2)
Segmentation of glioma from MRI scans. Note that for both datasets, we
abandoned the pixel-level annotations for training images and instead provided
point annotations for them. For quantitative analysis, we adopted Dice,
Average Symmetric Surface Distance (ASSD) between segmentation results and the
ground truth to evaluate the segmentation performance.

####  4.1.1 Vestibular Schwannoma (VS) Dataset

A public VS dataset [48, 49] was used for the VS segmentation, and it consists
of contrast-enhanced T1-weighted (ceT1) and high-resolution T2-weighted (hrT2)
MRI images collected from 242 patients with VS undergoing Gamma Knife
stereotactic radiosurgery. In this work we used the hrT2 images for
segmentation, with in-plane resolution of approximately $0.5mm\times 0.5mm$,
in-plane matrix of $384\times 384$ or $448\times 448$, slice thickness of
$1.0$–$1.5mm$ and 40-80 slices in a scan. As the segmentation target is
relatively small, we cropped and padded if necessary the images to a uniform
size of $128\times 128\times 48$. We randomly split the dataset into 176, 20
and 46 cases for training, validation and testing, respectively.

####  4.1.2 Brain Tumor Segmentation (BraTS) Dataset

For glioma segmentation, we used the BraTS 2019 [50, 51] dataset that was
collected from 335 glioma patients across 19 institutions. Each patient was
scanned with T1, ceT1, T2 and Fluid Attenuated Inversion Recovery (FLAIR) MRI
that were co-registered to the same anatomical template. The images had been
skull-stripped and re-sampled to the resolution of $1mm\times 1mm\times 1mm$
with size of $240\times 240\times 155$. In this work, we used the FLAIR images
for whole glioma segmentation, and they were randomly split into 244 for
training, 27 for validation and 64 for testing, respectively.

###  4.2 Implementation Details

Our framework was implemented in PyTorch and PyMIC111https://github.com/HiLab-
git/PyMIC [52] on a Ubuntu desktop with an NVIDIA GeForce RTX 3090 GPU, and
the code is available online222https://github.com/HiLab-git/PA-Seg. We used
the 2.5D U-Net [49] as the primary network for the VS dataset due to its
anisotropic resolution, and used the 3D U-Net [53] as the primary network for
the BraTS dataset. For each network, we set the channel number in the first
block as 16, and it was doubled after each down-sampling layer in the encoder,
increasing to 256 at the bottleneck. The patch size was $128\times 128\times
48$ and $128\times 128\times 128$ for the VS and BraTS datasets, respectively.
The batch size was 1 for both datasets. Random cropping and flipping along
each of the three axes were used for data augmentation. Stochastic Gradient
Descent (SGD) was used for training, with weight decay $3\times 10^{-5}$ and
momentum 0.99.

For the first stage of PA-Seg, the maximal training epoch $e_{max}$ was set to
300 and 500 for the two datasets, respectively. The learning rate was
initialized as $\eta_{0}=10^{-2}$, and changed based on a polynomial decay
schedule: $\eta=\eta_{0}\times(1-\frac{e}{e_{max}})^{0.9}$, where $e$ is the
current epoch. For the CRF loss, $K_{ij}$ was implemented by a single Gaussian
kernel with bilateral feature vectors, and the spatial coordinate bandwidth
and image intensity bandwidth were 6 and 0.1, respectively. The hyper-
parameters $\alpha$ and $\beta$ were set to 1.0 and 0.1, respectively.

In the second stage using SCM, attention U-Net [54] was used as the auxiliary
network for both datasets, and it was pre-trained in the same way as the
primary network in the first stage. For CKD, $T$ was set to 4 according to
grid search, and it was chosen by the best final segmentation performance on
the validation set. For both datasets, the maximal training epoch was $100$,
and the pseudo labels and “soft” probability maps given by the teacher in
$\mathcal{L}_{KD}$ were updated per $20$ epochs for stable training, and the
other experimental settings were the same as those in the first stage. We
found that after training using SCM, the primary and auxiliary networks had a
very close performance, so we only used the primary network for inference on
testing images. In addition, the standard Softmax prediction branch was used
for inference at testing time.

Figure 4: Examples of expanded annotations based on different $\epsilon$
values. The two rows are for VS and glioma, respectively. The yellow curves
show the boundaries of full dense annotations. Red and green colors show
expanded foreground and background pixels, respectively, and the remaining
pixels are unlabeled. Figure 5: Mean precision and recall between expanded
foreground seeds and ground truth masks with different $\epsilon$ values on
validation datasets.

###  4.3 Results of The First Stage

In this section, we report the results of the initial segmentation model
trained in the first stage, i.e., only learning from the expanded seeds with
contextual regularization, and the results based on SCM in the second stage
will be shown in 4.4.

####  4.3.1 Threshold for Seed Expansion

The hyper-parameter $\epsilon$ controls the degree of expansion of the
foreground seed. As shown in Fig. 4, when $\epsilon$ is small (e.g., 0.1), the
region of expanded foreground annotation is small, with a high precision and
low recall. The foreground seed region grows as $\epsilon$ increases, and it
have a lot of false positives when $\epsilon$ is large (e.g., 0.3 and 0.4),
leading to high recall but low precision. To encourage a large number of
foreground seeds to provide strong supervision, a high recall is expected. At
the same time, to avoid incorrect annotations in the expanded region, a high
precision is desired. Therefore, a proper $\epsilon$ should be selected to
balance the precision and recall.

We argue that ensuring a high precision is more important than a high recall,
as the expanded seeds should not contain many wrong labels that will corrupt
the model. The evolution curves of precision and recall of expanded foreground
seeds on the validation datasets are shown in Fig. 5. It can be observed that
when $\epsilon=0.2$, the precision is higher than 0.9 and the recall is 0.33
and 0.25 for VS and glioma, respectively. Using a larger $\epsilon$ leads to a
rapid decrease of precision. Therefore, we set $\epsilon=0.2$ on both
datasets. This can also be confirmed from Fig. 4 where $\epsilon=0.2$ leads to
a large region of expanded foreground seeds with few false positives.

Figure 6: Effect of hyper-parameters $\alpha$ and $\beta$ on validation datasets. Figure 7: Visual comparison between different losses for learning from expanded partial annotations in the first stage. The yellow curves show ground truth segmentation. Table 1: Quantitative evaluation of different loss combinations in the first stage of PA-Seg.  Method | VS dataset | BraTS dataset
---|---|---
Dice | ASSD (mm) | Dice | ASSD (mm)
full supervision |  0.893$\pm$0.052 |  0.381$\pm$0.219 |  0.877$\pm$0.094 |  2.282$\pm$3.282
baseline |  0.697$\pm$0.080 |  1.289$\pm$0.537 |  0.718$\pm$0.120 |  4.904$\pm$3.460
+$\mathcal{L}^{A}_{crf}$ |  0.812$\pm$0.090 |  0.740$\pm$0.317 |  0.816$\pm$0.152 |  2.894$\pm$2.858
+$\mathcal{L}_{mcrf}$ |  0.823$\pm$0.095 |  0.673$\pm$0.437 |  0.828$\pm$0.113 |  2.574$\pm$2.698
+$\mathcal{L}_{vm}$ |  0.798$\pm$0.069 |  0.756$\pm$0.376 |  0.802$\pm$0.129 |  2.713$\pm$2.595
+$\mathcal{L}_{mcrf}$+$\mathcal{L}_{vm}$ | 0.836$\pm$0.099 | 0.622$\pm$0.453 | 0.842$\pm$0.120 | 2.386$\pm$2.690

####  4.3.2 The Contribution of Each Loss Term

To validate effectiveness of the multi-view CRF loss and VM loss terms in Eq.
11, we first investigated the influence of their weights $\alpha$ and $\beta$
on segmentation performance. We fixed one weight to zero and varied the other,
and calculated mean Dice on the validation datasets. The results are shown in
Fig. 6. It can be observed that setting $\alpha>0$ and $\beta>0$ leads to a
large improvement from using the partially supervised loss
$\mathcal{L}_{psup}$ alone, respectively. Fig. 6 shows that the optimal values
of $\alpha$ and $\beta$ are $1.0$ and $0.1$, respectively on both datasets,
and the performance does not change much when the weights are around the
optimal values.

Then, we conducted ablation study on the loss terms in Eq. 11. The results are
shown in Table 1, where the baseline means learning from the expanded
annotations only using the partially supervised loss $\mathcal{L}_{psup}$.
“Full supervision” means learning with full annotations based on cross-entropy
loss and Dice loss, which serves as the upper bound. Compared with the
baseline, using 2D CRF loss ($\mathcal{L}^{A}_{crf}$) and VM loss improved the
average Dice from 0.697 to 0.812 and 0.798 on the VS dataset, respectively,
and they improved the average Dice from 0.718 to 0.816 and 0.802 on the BraTS
dataset, respectively. The multi-view CRF loss outperformed the 2D CRF loss on
both datasets. The proposed loss combining $\mathcal{L}_{mcrf}$ and
$\mathcal{L}_{vm}$ obtained an average Dice of 0.836 and 0.842 on the two
datasets, respectively, which outperformed the other variants. A visual
comparison of these variants is shown in Fig. 7, which demonstrates the
effectiveness of our method in learning from partial annotations for 3D
segmentation.

Table 2: Quantitative comparison between our method in the first stage with state-of-the-art weakly supervised methods. Ours (extreme) denotes replacing six background points with extreme points for annotation. Method | VS dataset | BraTS dataset
---|---|---
Dice | ASSD (mm) | Dice | ASSD (mm)
full supervision |  0.893$\pm$0.052 |  0.381$\pm$0.219 |  0.877$\pm$0.094 |  2.282$\pm$3.282
baseline |  0.697$\pm$0.080 |  1.289$\pm$0.537 |  0.718$\pm$0.120 |  4.904$\pm$3.460
size loss [12] |  0.768$\pm$0.087 |  1.007$\pm$0.585 |  0.784$\pm$0.121 |  3.805$\pm$3.555
DTP [9] |  0.780$\pm$0.101 |  0.884$\pm$0.419 |  0.794$\pm$0.138 |  3.696$\pm$4.836
Deepcut [8] |  0.743$\pm$0.115 |  1.009$\pm$0.525 |  0.755$\pm$0.173 |  7.700$\pm$7.328
Mumford-Shah [55] |  0.731$\pm$0.084 |  1.128$\pm$0.484 |  0.736$\pm$0.110 |  4.623$\pm$3.438
EM [56] |  0.722$\pm$0.077 |  1.183$\pm$0.423 |  0.741$\pm$0.141 |  4.984$\pm$4.742
TV [57] |  0.711$\pm$0.080 |  1.302$\pm$0.490 |  0.721$\pm$0.098 |  4.776$\pm$2.986
ours | 0.836$\pm$0.099 | 0.622$\pm$0.453 | 0.842$\pm$0.120 | 2.386$\pm$2.690
ours (extreme) |  0.801$\pm$0.068 |  0.723$\pm$0.268 |  0.812$\pm$0.152 |  2.894$\pm$2.858
Figure 8: Visual comparison between our method in the first stage and existing
weakly supervised segmentation methods. The blue and yellow numbers represent
Dice and ASSD (mm) values, respectively.

####  4.3.3 Comparison with State-of-the-art Weakly Supervised Learning
Methods

The model trained in the first stage of our method was compared with six
state-of-the-art weakly supervised methods: 1) size loss [12] that employs a
global inequality constraint on the volume of the target area, 2) Deep
Tightness Prior (DTP) [9] that combines topological and emptiness priors with
size loss [12], 3) Deepcut [8] that iterates model training and dense CRF
post-processing, 4) Mumford-Shah loss [55] that adapts Mumford-Shah energy
function in level-set to constrain the segmented region, 5) Entropy
Minimization (EM) [56] that encourages the model to produce high-confidence
predictions for unannotated pixels, and 6) Total Variation (TV) loss [57] that
penalizes the L1-norm of the gradient of the predicted probability map. The
lower and upper bounds used in size loss [12] and DTP [9] were 0.9 and 1.1
times of the volume of the target, respectively. For DTP [9] and Deepcut [8]
that learn from bounding box annotations, we obtained the bounding boxes based
on the same set of six background points as our method with the foreground
seed ignored. Note that except for Deepcut [8] and DTP [9], the other compared
methods used the same expanded annotations as our method for training.

Figure 9: Performance of the primary and auxiliary networks on validation
datasets with different $\lambda$ values for SCM in the second stage of PA-
Seg.

Quantitative evaluation results of these methods are shown in Table 2. EM [56]
and TV [57] performed slightly better than the baseline. Mumford-Shah loss
[55] and Deepcut [8] outperformed these two methods, and they achieved an
average Dice of 0.731 and 0.743 on the VS dataset, respectively, and 0.736 and
0.755 on the BraTS dataset, respectively. DTP [9] achieved the top performance
among the existing methods, with Dice of 0.780 and ASSD of 0.884 $mm$ for VS,
and Dice of 0.794 and ASSD of 3.696 $mm$ for BraTS. Notably, our method
largely outperformed the existing weakly supervised methods, achieving Dice
and ASSD of 0.836 and 0.622 $mm$ for VS, respectively, and Dice and ASSD of
0.842 and 2.386 $mm$ for the BraTS dataset, respectively.

Fig. 8 shows some qualitative evaluation results. Although size loss [12] and
DTP [9] outperformed the other existing methods, they have some visible errors
nearby the target boundaries, because they only impose some constrains on the
volume and topology of the target, without considering the contrast between
foreground and background. The results of our method have the highest overlap
ratio with the ground truth without much false positives and false negatives,
indicating the effectiveness and robustness of our method.

####  4.3.4 Comparison with Annotation with Extreme Points

As shown in Fig. 3, conventional extreme points is less suitable for 3D
medical images, as the bounding boxes derived from extreme points in a user-
selected slice may not include the entire foreground region in the other
slices, which may mislead the model during training. Our proposed point
annotations are relaxed to the background in some distance from the exact 3D
extreme points, therefore expanding the bounding boxes and reducing the number
of false negative pixels. For comparison, we replaced the six background
points with extreme points for annotation while reserving one foreground
point, which is denoted as ours (extreme). The corresponding results are shown
in the last row of Table 2. It can be observed that the average Dice dropped
from 0.836 to 0.801 on the VS dataset, and 0.842 to 0.812 on the BraTS
dataset, respectively.

###  4.4 Results of SCM for Noise-robust Learning

####  4.4.1 Contribution of Self-training and CKD

Figure 10: Visual comparison between different noise-robust learning methods
on some hard cases in the second stage of our method. The blue and yellow
numbers represent Dice and ASSD (mm) values, respectively.

We then used the pre-trained primary and auxiliary models for noise-robust
learning based on SCM in the second stage. To investigate the influence of the
hyper-parameter $\lambda$ in Eq. 14 and Eq. 15, we varied its value from $0.0$
to $1.0$ and calculated the Dice scores on validation datasets, as shown in
Fig. 9. Note that $\lambda=0.0$ and $\lambda=1.0$ means only using the self-
training loss and the CKD loss, respectively. Fig. 9 shows that the optimal
value was $0.5$, indicating a combination of the two kinds of losses performed
better than using only one of them. The last section in Table 3 shows a
quantitative comparison between $\mathcal{L}_{self}$ ($\lambda=0.0$),
$\mathcal{L}_{KD}$ ($\lambda=1.0$) and $\lambda=0.5$ (their combination) on
testing datasets. Compared with the first-stage model, using
$\mathcal{L}_{self}$ and $\mathcal{L}_{KD}$ improved the average Dice from
0.836 to 0.838 and 0.846, respectively on the VS dataset, and from 0.842 to
0.844 and 0.850 on the BraTS dataset, respectively, showing the CKD loss is
superior to the self-training loss. SCM ($\lambda=0.5$) further improved the
average Dice to 0.852 on the VS dataset and 0.856 on the BraTS dataset,
respectively.

####  4.4.2 Effectiveness of Asymmetric Architectures for SCM

To investigate the effectiveness of using asymmetric architectures for primary
and auxiliary networks, we compared it with using the same architecture for
the two networks for SCM. The results are shown in Table 4, where Net-A means
2.5D U-Net [49] and 3D U-Net [53] for VS and BraTS datasets, respectively, and
Net-B means attention U-Net [54]. It can be observed that when Net-A was used
for both of the primary and auxiliary networks, the Dice improvement from the
first stage was 0.006 and 0.004 on the two datasets, respectively. When Net-B
was used for both of them, the Dice improvement from the first stage was 0.005
and 0.006 on the two datasets, respectively. In contrast, when Net-A and Net-B
were used for the primary and auxiliary networks respectively, the Dice
improvement from the first stage was 0.016 (0.836 to 0.852) and 0.014 (0.842
to 0.856) on the two datasets, respectively, which shows the advantages of
using asymmetric network structures for SCM.

####  4.4.3 Performance of Primary Model Compared with Ensemble for Inference

Fig. 9 shows the performance of the primary and auxiliary models on the
validation sets during training. We observed that after SCM, the two different
networks had a very slight difference, i.e. up to about 0.003 and 0.002 in
terms of Dice for VS and BraTS datasets, respectively.

Besides, we compared inference by the primary model with inference by an
ensemble of the primary and auxiliary models, which is denoted as SCM-E
($\lambda=0.5$). The results in Table 3 show that the ensemble led to a quite
slight improvement from the primary model, i.e., 0.004 and 0.002 in terms of
Dice for VS and BraTS, respectively. Therefore, we only used the primary
network for inference at test time due to its efficiency and high accuracy.

Table 3: Quantitative evaluation of different noise-robust learning methods in the second stage of PA-Seg. * denotes significant improvement (p-value $<$ 0.05) achieved by our method compared with the first-stage. SCM-E denotes an ensemble of the primary and auxiliary networks. Method | VS dataset | BraTS dataset
---|---|---
Dice | ASSD (mm) | Dice | ASSD (mm)
full supervision |  0.893$\pm$0.052 |  0.381$\pm$0.219 |  0.877$\pm$0.094 |  2.282$\pm$3.282
first-stage |  0.836$\pm$0.099 |  0.622$\pm$0.453 |  0.842$\pm$0.120 |  2.386$\pm$2.690
CPS[24] |  0.841$\pm$0.089 |  0.595$\pm$0.389 |  0.847$\pm$0.115 |  2.698$\pm$3.138
co-teaching[26] |  0.840$\pm$0.088 |  0.621$\pm$0.397 |  0.842$\pm$0.124 |  2.844$\pm$3.697
Wang et al.[16] |  0.844$\pm$0.094 |  0.588$\pm$0.437 |  0.845$\pm$0.102 |  2.888$\pm$3.322
Xue et al.[32] |  0.843$\pm$0.098 |  0.598$\pm$0.449 |  0.845$\pm$0.107 |  2.483$\pm$3.131
DML[58] |  0.838$\pm$0.091 |  0.613$\pm$0.402 |  0.843$\pm$0.128 |  2.719$\pm$3.278
SCM($\lambda=0$) |  0.838$\pm$0.090 |  0.613$\pm$0.407 |  0.844$\pm$0.114 |  2.872$\pm$3.615
SCM($\lambda=1$) |  0.846$\pm$0.082 |  0.577$\pm$0.386 |  0.850$\pm$0.114 |  2.241$\pm$2.421
SCM($\lambda=0.5$) |  0.852$\pm$0.082* |  0.557$\pm$0.399* |  0.856$\pm$0.103* | 2.204$\pm$2.169
SCM-E($\lambda=0.5$) |  0.856$\pm$0.079 |  0.543$\pm$0.384 |  0.858$\pm$0.102 |  1.910$\pm$1.511
Table 4: Comparison of different network settings for SCM in the second stage of our method. The first section shows results for the first stage. Net-A: 2.5D U-Net [49] and 3D U-Net [53] for VS and BraTS datasets, respectively; Net-B: Attention U-Net [54]. Primary | Auxiliary | VS dataset | BraTS dataset
---|---|---|---
Dice | ASSD (mm) | Dice | ASSD (mm)
Net-A | None |  0.836$\pm$0.099 |  0.622$\pm$0.453 |  0.842$\pm$0.120 |  2.386$\pm$2.690
Net-B | None |  0.831$\pm$0.092 |  0.652$\pm$0.423 |  0.841$\pm$0.107 |  2.381$\pm$2.303
Net-A | Net-A |  0.842$\pm$0.088 |  0.598$\pm$0.416 |  0.846$\pm$0.114 |  2.315$\pm$2.507
Net-B | Net-B |  0.836$\pm$0.091 |  0.628$\pm$0.404 |  0.847$\pm$0.118 |  2.320$\pm$2.153
Net-A | Net-B | 0.852$\pm$0.082 | 0.557$\pm$0.399 | 0.856$\pm$0.103 | 2.204$\pm$2.169

####  4.4.4 Comparison with State-of-the-art Methods for Learning from Noisy
Annotations

Our SCM was compared with five state-of-the-art methods: 1) CPS [24] using
masks predicted by two networks to supervise each other, 2) co-teaching [26]
with two networks as well where each network selects samples with low loss
values to supervise the other, 3) Wang et al. [16] combining a noise-robust
Dice loss with an adaptive self-ensembling framework, 4) Xue et al. [32]
utilizing three networks to select clean samples and correct each other by a
joint optimization framework, and 5) Deep Mutual Learning (DML) [58]
minimizing Kullback-Leibler divergence between predictions from two networks
and cross-entropy loss simultaneously. For Wang et al. [16] and Xue et al.
[32], we used the primary networks with different parameter initialization.
For CPS [24], co-teaching [26] and DML [58], we used the same two networks as
our SCM. Note that DML [58] used the same label to supervise the two networks,
and we implemented it by taking an ensemble of pre-trained primary and
auxiliary networks to generate the label. In addition, DML does not use
temperature scaling to soften the probability maps.

Table 3 shows the quantitative evaluation of these methods. Compared with CPS
[24], SCM ($\lambda=1$) replaces the hard labels with the soft ones predicted
by another network, and obtained better performance, which demonstrates that
the KD-based soft labels are helpful for learning from noisy labels. These
existing methods only achieved marginal improvement over the first-stage
model, i.e., 0.008 and 0.005 at most in terms of Dice for VS and BraTS
datasets, respectively. There was almost no improvement for DML [58], i.e.
0.002 and 0.001 in terms of Dice on VS and BraTS datasets, because the two
networks learned from the same set of labels in DML, which is less effective
for avoiding the bias of pseudo labels. Our SCM ($\lambda=0.5$) was
significantly better than the compared methods, improving the Dice by 0.016
and 0.014 for the VS and BraTS datasets, respectively. What’s more, on the
BraTS dataset, our method achieved an average Dice of 0.856 that was just a
bit lower than the full supervision method, and it was slightly better than
the full supervision method in terms of ASSD (2.204 $mm$ vs 2.282 $mm$).

Fig. 10 shows a comparison of these methods on some hard cases that were
inaccurately segmented in the first stage. Due to the low contrast and
ambiguous boundary of hrT2 images of VS, the first-stage model obtained many
false negatives. The existing methods corrected these errors to some extent,
but they still resulted in some mis-segmented regions. For the glioma cases
with low contrast and abnormal anatomical structures, both the first-stage
model and the existing methods had an incomplete segmentation with some holes.
Our SCM successfully corrected the mis-segmentation on both datasets, showing
its effectiveness on alleviating the impact of noise in the pseudo labels
predicted by the first-stage model.

##  5 Discussion and conclusion

Our method decomposes the challenging problem of learning from extremely
sparse point annotations into two steps: a first step using contextual
regularization of unlabeled pixels to train an initial model, and a second
step using SCM to learn from noisy pseudo labels. The contextual
regularization may also be applicable for other scenarios where incomplete
annotation exists, such as semi-supervised segmentation [23, 35] and learning
from scribbles [25]. The SCM is a general framework for noisy-label learning,
and it may also be applied to learning from non-expert annotations [16, 26].

Our SCM shares some spirit with Deep Mutual Learning (DML) [58], but they have
substantial differences. First, DML aims to avoid a pre-trained teacher to
improve the performance of a CNN for image classification tasks learning from
clean labels. In contrast, our SCM is proposed to deal with noisy labels for
3D medical image segmentation. Second, the same set of fixed labels are used
to supervise the two CNNs in DML, but the two CNNs in SCM learn from different
sets of labels (each network’s prediction for self-training) that are updated
after some epochs. Thirdly, DML does not use temperature scaling to soften the
probability maps during knowledge distillation, while our method uses soft
predictions during distillation to alleviate the effect of noisy labels. Table
3 shows that our method largely outperformed DML in learning from noisy labels
for segmentation.

We noticed that Patel and Dolz [31] also applied KD to weakly supervised
medical image segmentation. That work is different from ours in several
aspects: First, Patel and Dolz [31] learned from slice-level annotations for
segmentation, while our method learns from point annotations. Second, Patel
and Dolz [31] used networks based on the same architecture to deal with
different input modalities with consistency regularization on the class
activation maps, while our method uses asymmetric architectures for the two
networks to learn from noisy pseudo labels. Thirdly, Patel and Dolz [31]
applied KD to the classification outputs without temperature scaling, while
our method applies KD to segmentation predictions that are softened by
temperature scaling.

In conclusion, we propose a two-stage weakly supervised 3D segmentation
framework PA-Seg using very sparse point annotations, i.e., six and one points
for the background and foreground, respectively. In the first stage, the
single foreground seed is expanded based on geodesic distance transform, and a
model is trained with contextual regularization of unannotated pixels based on
multi-view CRF and variance minimization. In the second stage, a new self and
cross monitoring method based on cross knowledge distillation is proposed to
learn from noisy pseudo labels generated by the initial model. Experiments on
two datasets demonstrated that our model in the first stage already
outperformed several existing methods for weakly supervised segmentation. In
the second stage, our SCM can further improve the model’s performance by
robustly learning from noisy labels, which also outperformed existing noise-
robust learning methods. Our framework substantially reduces the annotation
cost for training high-performance 3D medical image segmentation models.

## References

  * [1] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical Image Analysis_ , vol. 42, pp. 60–88, 2017.
  * [2] M. H. Hesamian, W. Jia, X. He, and P. Kennedy, “Deep learning techniques for medical image segmentation: achievements and challenges,” _Journal of Digital Imaging_ , vol. 32, no. 4, pp. 582–596, 2019.
  * [3] J. Weese and C. Lorenz, “Four challenges in medical image analysis from an industrial perspective,” _Medical Image Analysis_ , vol. 33, pp. 44–49, 2016\.
  * [4] Y. Wang, J. Zhang, M. Kan, S. Shan, and X. Chen, “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation,” in _CVPR_ , 2020, pp. 12 275–12 284.
  * [5] K. Wu, B. Du, M. Luo, H. Wen, Y. Shen, and J. Feng, “Weakly supervised brain lesion segmentation via attentional representation learning,” in _MICCAI_. Springer, 2019, pp. 211–219.
  * [6] D. Lin, J. Dai, J. Jia, K. He, and J. Sun, “Scribblesup: Scribble-supervised convolutional networks for semantic segmentation,” in _CVPR_ , 2016, pp. 3159–3167.
  * [7] H. Lee and W.-K. Jeong, “Scribble2label: Scribble-supervised cell segmentation via self-generating pseudo-labels with consistency,” in _MICCAI_. Springer, 2020, pp. 14–23.
  * [8] M. Rajchl, M. C. Lee, O. Oktay, K. Kamnitsas, J. Passerat-Palmbach, W. Bai, M. Damodaram, M. A. Rutherford, J. V. Hajnal, B. Kainz _et al._ , “Deepcut: Object segmentation from bounding box annotations using convolutional neural networks,” _IEEE Transactions on Medical Imaging_ , vol. 36, no. 2, pp. 674–683, 2016.
  * [9] H. Kervadec, J. Dolz, S. Wang, E. Granger, and I. B. Ayed, “Bounding boxes for weakly supervised segmentation: Global constraints get close to full supervision,” in _MIDL_. PMLR, 2020, pp. 365–381.
  * [10] H. R. Roth, D. Yang, Z. Xu, X. Wang, and D. Xu, “Going to extremes: weakly supervised medical image segmentation,” _Machine Intelligence & Knowledge Extraction_, vol. 3, no. 2, pp. 507–524, 2021.
  * [11] R. Dorent, S. Joutard, J. Shapey, A. Kujawa, M. Modat, S. Ourselin, and T. Vercauteren, “Inter extreme points geodesics for end-to-end weakly supervised image segmentation,” in _MICCAI_. Springer, 2021, pp. 615–624.
  * [12] H. Kervadec, J. Dolz, M. Tang, E. Granger, Y. Boykov, and I. B. Ayed, “Constrained-CNN losses for weakly supervised segmentation,” _Medical Image Analysis_ , vol. 54, pp. 88–99, 2019.
  * [13] M. Tang, A. Djelouah, F. Perazzi, Y. Boykov, and C. Schroers, “Normalized cut loss for weakly-supervised CNN segmentation,” in _CVPR_ , 2018, pp. 1818–1827.
  * [14] M. Tang, F. Perazzi, A. Djelouah, I. Ben Ayed, C. Schroers, and Y. Boykov, “On regularized losses for weakly-supervised CNN segmentation,” in _ECCV_ , 2018, pp. 507–522.
  * [15] Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” _NeurIPS_ , vol. 31, 2018.
  * [16] G. Wang, X. Liu, C. Li, Z. Xu, J. Ruan, H. Zhu, T. Meng, K. Li, N. Huang, and S. Zhang, “A noise-robust framework for automatic segmentation of COVID-19 pneumonia lesions from CT images,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 8, pp. 2653–2663, 2020.
  * [17] G. Wang, M. A. Zuluaga, W. Li, R. Pratt, P. A. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest, S. Ourselin _et al._ , “DeepIGeoS: A deep interactive geodesic framework for medical image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 7, pp. 1559–1572, 2018.
  * [18] X. Luo, G. Wang, T. Song, J. Zhang, M. Aertsen, J. Deprest, S. Ourselin, T. Vercauteren, and S. Zhang, “MIDeepSeg: Minimally interactive segmentation of unseen objects from medical images using deep learning,” _Medical Image Analysis_ , vol. 72, p. 102102, 2021.
  * [19] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _NeurIPS_ , 2015, pp. 1–9.
  * [20] F. Sarfraz, E. Arani, and B. Zonooz, “Knowledge distillation beyond model compression,” in _ICPR_. IEEE, 2021, pp. 6136–6143.
  * [21] C. Rother, V. Kolmogorov, and A. Blake, “GrabCut: interactive foreground extraction using iterated graph cuts,” _ACM Trans. on Graphics_ , vol. 23, no. 3, pp. 309–314, 2004.
  * [22] L. Grady, “Random walks for image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 28, no. 11, pp. 1768–1783, 2006\.
  * [23] L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng, “Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation,” in _MICCAI_. Springer, 2019, pp. 605–613.
  * [24] X. Chen, Y. Yuan, G. Zeng, and J. Wang, “Semi-supervised semantic segmentation with cross pseudo supervision,” in _CVPR_ , 2021, pp. 2613–2622.
  * [25] X. Luo, M. Hu, W. Liao, S. Zhai, T. Song, G. Wang, and S. Zhang, “Scribble-supervised medical image segmentation via dual-branch network and dynamically mixed pseudo labels supervision,” in _MICCAI_. Springer, 2022, pp. 528–538.
  * [26] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in _NeurIPS_ , 2018, pp. 8527–8537.
  * [27] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in _NeurIPS_ , 2017, pp. 8527–8537.
  * [28] P. Wang, J. Peng, M. Pedersoli, Y. Zhou, C. Zhang, and C. Desrosiers, “Self-paced and self-consistent co-training for semi-supervised image segmentation,” _Medical Image Analysis_ , vol. 73, p. 102146, 2021.
  * [29] Z. Zhao, K. Xu, S. Li, Z. Zeng, and C. Guan, “Mt-uda: Towards unsupervised cross-modality medical image segmentation with limited source labels,” in _MICCAI_. Springer, 2021, pp. 293–303.
  * [30] B. Liu, C. Desrosiers, I. Ben Ayed, and J. Dolz, “Segmentation with mixed supervision: Confidence maximization helps knowledge distillation,” _Medical Image Analysis_ , vol. 83, p. 102670, 2023.
  * [31] G. Patel and J. Dolz, “Weakly supervised segmentation with cross-modality equivariant constraints,” _Medical Image Analysis_ , vol. 77, p. 102374, 2022\.
  * [32] C. Xue, Q. Deng, X. Li, Q. Dou, and P.-A. Heng, “Cascaded robust learning at imperfect labels for chest X-ray segmentation,” in _MICCAI_. Springer, 2020, pp. 579–588.
  * [33] H. Wei, L. Feng, X. Chen, and B. An, “Combating noisy labels by agreement: A joint training method with co-regularization,” in _CVPR_ , 2020, pp. 13 726–13 735.
  * [34] Y. Shi, J. Zhang, T. Ling, J. Lu, Y. Zheng, Q. Yu, L. Qi, and Y. Gao, “Inconsistency-aware uncertainty estimation for semi-supervised medical image segmentation,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 3, pp. 608–620, 2022.
  * [35] X. Luo, J. Chen, T. Song, and G. Wang, “Semi-supervised medical image segmentation through dual-task consistency,” in _AAAI_ , 2021, pp. 8801–8809.
  * [36] D. Karimi, H. Dou, S. K. Warfield, and A. Gholipour, “Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis,” _Medical Image Analysis_ , vol. 65, p. 101759, 2020.
  * [37] A. Ghosh, H. Kumar, and P. Sastry, “Robust loss functions under label noise for deep neural networks,” in _AAAI_ , 2017, pp. 1919–1925.
  * [38] H. Wang, D. Zhang, Y. Song, S. Liu, Y. Wang, D. Feng, H. Peng, and W. Cai, “Segmenting neuronal structure in 3d optical microscope images via knowledge distillation with teacher-student network,” in _ISBI_. IEEE, 2019, pp. 228–231.
  * [39] Q. Dou, Q. Liu, P. A. Heng, and B. Glocker, “Unpaired multi-modal segmentation via knowledge distillation,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 7, pp. 2415–2425, 2020.
  * [40] M. Lukasik, S. Bhojanapalli, A. Menon, and S. Kumar, “Does label smoothing mitigate label noise?” in _ICML_. PMLR, 2020, pp. 6448–6458.
  * [41] E. Kats, J. Goldberger, and H. Greenspan, “Soft labeling by distilling anatomical knowledge for improved MS lesion segmentation,” in _ISBI_. IEEE, 2019, pp. 1563–1566.
  * [42] Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li, “Learning from noisy labels with distillation,” in _ICCV_ , 2017, pp. 1910–1918.
  * [43] P. Krähenbühl and V. Koltun, “Efficient inference in fully connected CRFs with gaussian edge potentials,” in _NeurIPS_ , 2011, pp. 109–117.
  * [44] T. F. Chan and L. A. Vese, “Active contours without edges,” _IEEE Transactions on Image Processing_ , vol. 10, no. 2, pp. 266–277, 2001.
  * [45] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _ICML_. PMLR, 2017, pp. 1321–1330.
  * [46] R. Müller, S. Kornblith, and G. E. Hinton, “When does label smoothing help?” in _NeurIPS_ , 2019, pp. 1–10.
  * [47] K. Xu, L. Rui, Y. Li, and L. Gu, “Feature normalized knowledge distillation for image classification,” in _ECCV_ , 2020, pp. 664–680.
  * [48] J. Shapey, A. Kujawa, R. Dorent, G. Wang, S. Bisdas, A. Dimitriadis, D. Grishchuck, I. Paddick, N. Kitchen, R. Bradford _et al._ , “Segmentation of vestibular schwannoma from magnetic resonance imaging: An open annotated dataset and baseline algorithm,” _The Cancer Imaging Archive_ , 2021.
  * [49] G. Wang, J. Shapey, W. Li, R. Dorent, A. Dimitriadis, S. Bisdas, I. Paddick, R. Bradford, S. Zhang, S. Ourselin _et al._ , “Automatic segmentation of vestibular schwannoma from T2-weighted MRI by deep spatial attention with hardness-weighted loss,” in _MICCAI_. Springer, 2019, pp. 264–272.
  * [50] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest _et al._ , “The multimodal brain tumor image segmentation benchmark (BraTS),” _IEEE Transactions on Medical Imaging_ , vol. 34, no. 10, pp. 1993–2024, 2014.
  * [51] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features,” _Scientific data_ , vol. 4, no. 1, pp. 1–13, 2017.
  * [52] G. Wang, X. Luo, R. Gu, S. Yang, Y. Qu, S. Zhai, Q. Zhao, K. Li, and S. Zhang, “PyMIC: A deep learning toolkit for annotation-efficient medical image segmentation,” _Computer Methods and Programs in Biomedicine_ , vol. 231, p. 107398, 2023.
  * [53] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _MICCAI_. Springer, 2015, pp. 234–241.
  * [54] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, and D. Rueckert, “Attention gated networks: Learning to leverage salient regions in medical images,” _Medical Image Analysis_ , vol. 53, pp. 197–207, 2019\.
  * [55] B. Kim and J. C. Ye, “Mumford–Shah loss functional for image segmentation with deep learning,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 1856–1866, 2019.
  * [56] Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy minimization,” in _NeurIPS_ , 2005, pp. 1–8.
  * [57] M. Javanmardi, M. Sajjadi, T. Liu, and T. Tasdizen, “Unsupervised total variation loss for semi-supervised deep learning of semantic segmentation,” _arXiv preprint arXiv:1605.01368_ , 2016.
  * [58] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, “Deep mutual learning,” in _CVPR_ , 2018, pp. 4320–4328.


Lakshmanan Nataraj, Chandrakanth Gudavalli, Tajuddin Manhar Mohammed,
Shivkumar Chandrasekaran, and B.S. Manjunath 11institutetext: Mayachitra Inc.,
Santa Barbara, CA, USA,
https://mayachitra.com/

# Seam Carving Detection and Localization using Two-Stage Deep Neural Networks

Lakshmanan Nataraj  Chandrakanth Gudavalli  Tajuddin Manhar Mohammed
Shivkumar Chandrasekaran  B.S. Manjunath

###### Abstract

Seam carving is a method to resize an image in a content aware fashion.
However, this method can also be used to carve out objects from images. In
this paper, we propose a two-step method to detect and localize seam carved
images. First, we build a detector to detect small patches in an image that
has been seam carved. Next, we compute a heatmap on an image based on the
patch detector’s output. Using these heatmaps, we build another detector to
detect if a whole image is seam carved or not. Our experimental results show
that our approach is effective in detecting and localizing seam carved images.

###### keywords:

image forensics, seam carving detection, fake images, object removal

##  1 Introduction

Figure 1: Illustration of seam carving detection and localization: (a)
Original image, (b) Object marked in red to be removed and Object marked in
green to be preserved, (c) Seam carved image with object removed, (d) Seam
carving detection heatmap using proposed approach (red pixels are areas where
seams were likely removed).

With new cameras, mobile phones and digital tablets, the amount of digital
images has had an exponential increase. Social media platforms have also
contributed to their increased distribution. At the same time, software for
manipulating these digital images have also significantly evolved. These
software tools make it trivial for people to manipulate these digital images.
The objective of Media Forensics is to identify these manipulations and detect
these doctored images. Over the years, many techniques have been proposed to
identify image manipulations. These include digital artificats based on camera
forensics, resampling characteristics, compression, and others. A common
operation in image tampering is removing certain image regions in a “content-
aware” way. In this regard, seam carving is a popular technique for “content-
aware” image resizing [1, 39] In seam carving, the “important content” in an
image is left unaffected when the image is resized and it is generally assumed
that the “important content” is not characterized by the low energy pixels.
Since seam carving based object removal involves non-traditional ways of
removing objects, it is a challenge to detect doctored images that have been
seam carved. In this paper, we propose a novel method to detect and localize
seam carved images using two stages of convolutional neural networks (CNNs):
one for detection and one for localization. First, we train a CNN to identify
patches that have been seam carved. For every pixel in an image, we then
compute the detection score which results in a heatmap for the whole image,
that can be used for localization. Finally, we train another CNN on the
heatmaps which gives a score at the image level to determine if an image has
been seam carved or not. Fig. 1 illustrates the proposed approach.

The rest of the paper is organized as follows. Sec. 2 presents the related
work in this area and Sec. 3 introduces seam carving and seam insertion on
images. The methodology to detect seam carving is presented in Sec. 4 while
the experiments are detailed in Sec. 5. Finally, the conclusion is presented
in Sec. 6.

##  2 Related Work

There have been several works proposed to detect digital image manipulations.
These include detection of splicing, morphing, resampling artifacts, copy-
move, seam carving, computer-generated (CG) images, JPEG artifacts,
inpainting, compression artifacts, to name a few. Many methods have been
proposed to detect copy-move [21, 11], resampling [34, 29, 20, 33, 36, 15, 6,
8], splicing [18, 2, 37], and inpainting based object removal [44, 23]. Other
approaches exploit JPEG compression artifacts [13, 24, 28, 7] or artifacts
arising from artificial intelligence (AI) generated images [30, 48, 16, 3]. In
recent years, deep learning based methods have shown better performance in
detecting image manipulations [5, 4, 35, 8].

Several methods have been proposed over the past decade to detect seam carving
based manipulations [38, 27, 26, 9, 43, 25, 42, 14, 40, 46, 41, 19, 17, 47,
22]. These include methods using steganalysis [38], hashing [27, 14], local
binary pattern [46, 47], and deep learning based methods [45, 10, 31, 32]. In
this paper, our approach to detect seam carving based manipulations is also
based on deep learning.

Figure 2: Example of Seam Carving when a $4\times 5$ matrix ${\bf a}$ is seam
carved and a $4\times 4$ matrix ${\bf b}$ results due to the removal of a
single seam. Figure 3: Example of Seam Insertion: (i) ${\bf a}$ and (ii) ${\bf
b}$ are the $4\times 5$ and the $4\times 6$ image matrices before and after
seam insertion, respectively. For points along the seam, the values are
modified as shown for the first row:
$b_{1,1}\\!=\\!a_{1,1},~{}b_{1,2}\\!=\\!a_{1,2},~{}b_{1,3}\\!=\\!\mbox{round}(\frac{a_{1,2}+a_{1,3}}{2}),~{}b_{1,4}\\!=\\!\mbox{round}(\frac{a_{1,3}+a_{1,4}}{2}),~{}b_{1,5}\\!=\\!a_{1,4},~{}b_{1,6}\\!=\\!a_{1,5}$.

##  3 Seam Carving and Seam Insertion

A seam is defined as an optimal 8-connected path of pixels on an image either
from top-to-bottom or left-to-right. In seam carving, the seams are removed
and the image dimension is reduced by a column or a row. In seam insertion, a
seam is first removed and two pixels are inserted at the position where the
seam was removed. Fig. 2 and 3 illustrates the processes of seam carving and
seam insertion. An energy function computed for all points along a seam is
considered for the optimality criterion for seam selection. This choice of
seams helps in maintaining the image quality during the resizing process. We
consider the seam carved/inserted image as a tampered image because the image
dimensions and it’s content are altered.. Hence, the problem of detecting seam
carving/insertion is important from an image forensics perspective.
Interpolation kernel based methods for re-sampling detection will fail when
the resizing in the doctored image is done using seam carving/insertion.
Though it was initially proposed for automatic image resizing while
maintaining a good perceptual quality of the resized image, seam carving has
also been used for removal of certain image regions. It is to be noted that
seam carving can discard and retain certain regions, depending on the weight
we assign to certain regions. E.g. for an object removal problem, we may need
to ensure that certain image regions are left unaffected as distorting them
may cause significant perceptual distortion. We first explain how seam carving
is used for object removal and then discuss the interesting problems involved.

(a) Stage 1

(b) Stage 2

Figure 4: Two-stage approach

##  4 Detection of Seam Carving

Figure 5: Screenshot of: (a) Non-seam carved patches, (b) Seam carved patches

In order to detect and localize seam carving in images, we propose a two-stage
detection approach: one for detection of seam carved patches and the other for
localizing seam carving in an image by generating a heatmap. First, we train a
deep neural network to identify whether patches in an image have been seam
carved or not. We then divide an image into patches and for every patch, we
compute the detection score which results in a heatmap for the whole image.
This heatmap can be used for localization of seam carving. Finally, we train
another deep neural network with the heatmaps as input which gives a score at
the image level to determine whether an image has been seam carved or not. The
entire block schematic is shown in Fig. 4

##  5 Experiments

###  5.1 Experimental Setup

We first extract $64\times 128$ patches from images belonging to RAISE dataset
[12]. From these patches, we form two classes of image patches: first class
where the patches are further cropped to $64\times 64$, and the second class
where the patches are seam carved horizontally by 50% to obtain $64\times 64$
seam carved patches. In this way we obtained 16,000 patches from the RAISE
dataset (8000 in each class) and 40,000 patches from the Dresden dataset
(20,000 in each class). These were further randomly divided into 80% training,
10% testing and 10% validation.

###  5.2 Learning

The patches are trained using a multi layer deep convolutional neural network
which consists of convolution layer with 32 3x3 convs, followed by ReLu layer,
convolution layer with 32 5x5 convs followed by max pooling layer, convolution
layer with 64 3x3 convs followed by ReLu layer, convolution layer with 64 5x5
convs followed by max pooling layer, convolution layer with 128 3x3 convs
followed by ReLu layer, convolution layer with 128 5x5 convs followed by max
pooling layer, and finally a 256 dense layer followed by a 256 dense layer and
a sigmoid layer. We train this model till a high training accuracy and
validation accuracy are obtained.

###  5.3 Detection Heatmaps

Using the trained model on the patches, the probability of a pixel being seam
carved or not is computed on overlapping patches in an image. Fig. 6 show the
heatmaps on non-seam carved and seam carved images. As we can see, the
heatmaps on the seam carved images have more red regions than the images on
non-seam carved images. Even for an image that has the blue sky, the heatmaps
can be clearly identified for seam carved image and the non-seam carved image.
This motivated us to train the heatmaps with another CNN which takes the
heatmaps as input (Fig. 4(b)) and outputs the probability whether an image has
been seam carved or not. As we can see from Fig. 7, we obtained high accuracy
when trained on the heatmaps.

Figure 6: Detection heatmaps on images that have (a) not been seam carved, and (b) seam carved Figure 7: ROC curve of seam carving detection on the model trained on the heatmaps Table 1: Robustness to Percentage of Seams Removed Percentage | Area under the curve (AUC)
---|---
1 | 0.6464
2 | 0.7838
5 | 0.9274
8 | 0.9540
10 | 0.9724
20 | 0.9866
30 | 0.9919
40 | 0.9937
50 | 0.9916
60 | 0.9502
70 | 0.9150
80 | 0.8670
90 | 0.8223

###  5.4 Robustness to Percentage of seams removed

In this experiment, we varied the percentage of seams removed in the testing
set and evaluated the model which was trained with 50% seams removed, in order
to check the robustness of the model for different amounts of seams removed.
The Area Under the Curve (AUC) is the evaluation metric for varying percentage
of seams removed. The results are tabulated in Tab. 1. We observe that the AUC
is very high for percentages around 50% and decreases for lower percentages of
seams removed. This shows that the model is generalizable for most percentages
of seams removed. In future, we will train another model for lower
percentages.

Figure 8: Detection heatmaps on images where objects have been removed using
seam carving: (a) original image, (b) heatmap computed on original image, (c)
object marked for removal in red, (d) image with object removed using seam
carving, (e) heatmap computed on object removed image showing the possible
seam paths Figure 9: Explainability in the heatmaps: (a) original image, (b)
heatmap computed on original image, (c) object marked for removal in red and
area preserved in green (bottom row) (d) image with object removed using seam
carving (in the top row - person’s leg is removed while preserved in the
bottom row), (e) heatmap computed on object removed image showing the possible
seam paths with explainability. The seam paths change on the top row and
bottom row near the person’s leg.

###  5.5 Robustness to JPEG compression

In this experiment, we evaluated the robustness of our proposed against JPEG
compression. We varied the JPEG Quality Factors (QFs) of test images from 100
to 50. The model was trained on seam carved and non-seam carved patches and
images, which were also JPEG compressed between the Quality Factors of 70-100.
The Area Under the Curve (AUC) is chosen as the evaluation metric. The results
are tabulated in Tab. 2. We observe that the AUC is high when the QF is high
(compression is low) and the AUC reduces as the QF decreases (compression
increases). However, even at a QF of 50, the AUC is still reasonably high.

Table 2: Robustness to JPEG Compression JPEG Quality Factor (QF) | Area under the curve (AUC)
---|---
100 | 0.9376
90 | 0.9160
80 | 0.8578
70 | 0.8027
60 | 0.7658
50 | 0.7332

###  5.6 Explainability on Object Removed Images

Here, we evaluate our approach in a practical scenario where objects are
removed in images using seam carving. We chose an object or a region in an
image that has to be removed. The weights of this region are set to a low
value such that the seam carving algorithm is forced to pass through this
region, thus removing the object from the image. When our approach was
evaluated on these images, we observe that our model is able to localize the
region that was removed as well as the paths taken by the seam carving
algorithm as shown in Fig. 8.

The detection heatmaps also exhibit explainability as shown in Fig. 9 where an
object is marked for removal in red. While this object is removed
successfully, a person’s leg in the foreground also gets removed (top row). To
prevent this, another area is marked in green (bottom row) by giving high
weights so that the person’s legs are not removed. As we can see from the
heatmaps computed on the seam carved images (top and bottom row), the path
showing the possible seams also changes near the person’s legs, thus
exhibiting explainability.

###  5.7 Extension to Seam Insertion Detection

Finally, we also extend the seam carving detection approach to detecting seam
insertion. We first extract $64\times 64$ patches from images belonging to
RAISE dataset [12]. From these patches, we form two classes of image patches:
first class where the patches are further cropped to $64\times 64$, and the
second class where the patches are seam inserted from $64\times 32$ dimensions
to $64\times 64$ seam dimensions. In this way we obtained 16,000 patches from
the RAISE dataset (8000 in each class). These were further randomly divided
into 80% training, 10% testing and 10% validation. The patches are trained
using a multi layer convolutional neural network as explained in Sec. 5.2. We
train this model till a high training accuracy and validation accuracy are
obtained. Using the trained model on the patches, the probability of a pixel
being seam inserted or not is computed on overlapping patches in an image to
produce a heatmap. Another model is trained on the heatmaps to determine if an
image has seam insertions or not. As we can see from Fig. 10, we obtained high
accuracy when trained on the heatmaps.

Figure 10: ROC curves at image level to detect seam inserted images

##  6 Conclusion and Future Work

In this paper, we presented an approach to detect seam carved images. Using
two stages of CNNs, we detect and localize areas in an image that have been
seam carved. In future, we will focus on making our detections more robust,
combining seam carving and insertions, and also extend to other object removal
methods such as inpainting.

##  7 Acknowledgements

This research was developed with funding from the Defense Advanced Research
Projects Agency (DARPA). The views, opinions and/or findings expressed are
those of the author and should not be interpreted as representing the official
views or policies of the Department of Defense or the U.S. Government. The
paper is approved for public release, distribution unlimited.

## References

  * [1] Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM Trans. Graph. 26(3), 10 (2007)
  * [2] Bappy, J.H., Simons, C., Nataraj, L., Manjunath, B., Roy-Chowdhury, A.K.: Hybrid lstm and encoder–decoder architecture for detection of image forgeries. IEEE Transactions on Image Processing 28(7), 3286–3300 (2019)
  * [3] Barni, M., Kallas, K., Nowroozi, E., Tondi, B.: Cnn detection of gan-generated face images based on cross-band co-occurrences analysis. arXiv preprint arXiv:2007.12909 (2020)
  * [4] Bayar, B., Stamm, M.C.: Design principles of convolutional neural networks for multimedia forensics. In: The 2017 IS&T International Symposium on Electronic Imaging: Media Watermarking, Security, and Forensics. IS&T Electronic Imaging
  * [5] Bayar, B., Stamm, M.C.: A deep learning approach to universal image manipulation detection using a new convolutional layer. In: Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security. pp. 5–10 (2016)
  * [6] Bayar, B., Stamm, M.C.: On the robustness of constrained convolutional neural networks to jpeg post-compression for image resampling detection. In: The 42nd IEEE International Conference on Acoustics, Speech and Signal Processing (2017)
  * [7] Bianchi, T., De Rosa, A., Piva, A.: Improved dct coefficient analysis for forgery localization in jpeg images. In: Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. pp. 2444–2447. IEEE (2011)
  * [8] Bunk, J., Bappy, J.H., Mohammed, T.M., Nataraj, L., Flenner, A., Manjunath, B., Chandrasekaran, S., Roy-Chowdhury, A.K., Peterson, L.: Detection and localization of image forgeries using resampling features and deep learning. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. pp. 1881–1889 (2017)
  * [9] Chang, W.L., Shih, T.K., Hsu, H.H.: Detection of seam carving in jpeg images. In: 2013 International Joint Conference on Awareness Science and Technology & Ubi-Media Computing (iCAST 2013 & UMEDIA 2013). pp. 632–638. IEEE (2013)
  * [10] Cieslak, L.F.S., Da Costa, K.A., PauloPapa, J.: Seam carving detection using convolutional neural networks. In: 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI). pp. 000195–000200. Ieee (2018)
  * [11] Cozzolino, D., Poggi, G., Verdoliva, L.: Efficient dense-field copy–move forgery detection. IEEE Transactions on Information Forensics and Security 10(11), 2284–2297 (2015)
  * [12] Dang-Nguyen, D.T., Pasquini, C., Conotter, V., Boato, G.: Raise: a raw images dataset for digital image forensics. In: Proceedings of the 6th ACM Multimedia Systems Conference. pp. 219–224. ACM (2015)
  * [13] Farid, H.: Exposing digital forgeries from jpeg ghosts. IEEE transactions on information forensics and security 4(1), 154–160 (2009)
  * [14] Fei, W., Gaobo, Y., Leida, L., Ming, X., Dengyong, Z.: Detection of seam carving-based video retargeting using forensics hash. Security and Communication Networks 8(12), 2102–2113 (2015)
  * [15] Feng, X., Cox, I.J., Doerr, G.: Normalized energy density-based forensic detection of resampled images. IEEE Transactions on Multimedia 14(3), 536–545 (2012)
  * [16] Goebel, M., Nataraj, L., Nanjundaswamy, T., Mohammed, T.M., Chandrasekaran, S., Manjunath, B.: Detection, attribution and localization of gan generated images. arXiv preprint arXiv:2007.10466 (2020)
  * [17] Gong, Q., Shan, Q., Ke, Y., Guo, J.: Detecting the location of seam and recovering image for seam inserted image. Journal of Computational Methods in Sciences and Engineering 18(2), 499–509 (2018)
  * [18] Guillemot, C., Le Meur, O.: Image inpainting: Overview and recent advances. Signal processing magazine 31(1), 127–144 (2014)
  * [19] Han, R., Ke, Y., Du, L., Qin, F., Guo, J.: Exploring the location of object deleted by seam-carving. Expert Systems with Applications 95, 162–171 (2018)
  * [20] Kirchner, M.: On the detectability of local resampling in digital images. Security, Forensics, Steganography, and Watermarking of Multimedia Contents X 6819(1), 68190F (2008), http://link.aip.org/link/?PSI/6819/68190F/1
  * [21] Li, J., Li, X., Yang, B., Sun, X.: Segmentation-based image copy-move forgery detection scheme. IEEE Transactions on Information Forensics and Security 10(3), 507–518 (2015)
  * [22] Li, Y., Xia, M., Liu, X., Yang, G.: Identification of various image retargeting techniques using hybrid features. Journal of Information Security and Applications 51, 102459 (2020)
  * [23] Liang, Z., Yang, G., Ding, X., Li, L.: An efficient forgery detection algorithm for object removal by exemplar-based image inpainting. Journal of Visual Communication and Image Representation 30, 75–85 (2015)
  * [24] Lin, Z., He, J., Tang, X., Tang, C.K.: Fast, automatic and fine-grained tampered jpeg image detection via dct coefficient analysis. Pattern Recognition 42(11), 2492–2501 (2009)
  * [25] Liu, Q., Chen, Z.: Improved approaches with calibrated neighboring joint density to steganalysis and seam-carved forgery detection in jpeg images. ACM Transactions on Intelligent Systems and Technology (TIST) 5(4), 1–30 (2014)
  * [26] Liu, Q., Cooper, P.A., Zhou, B.: An improved approach to detecting content-aware scaling-based tampering in jpeg images. In: 2013 IEEE China Summit and International Conference on Signal and Information Processing. pp. 432–436. IEEE (2013)
  * [27] Lu, W., Wu, M.: Seam carving estimation using forensic hash. In: Proceedings of the thirteenth ACM multimedia workshop on Multimedia and security. pp. 9–14 (2011)
  * [28] Luo, W., Huang, J., Qiu, G.: Jpeg error analysis and its applications to digital image forensics. IEEE Transactions on Information Forensics and Security 5(3), 480–491 (2010)
  * [29] Mahdian, B., Saic, S.: Blind authentication using periodic properties of interpolation. Information Forensics and Security, IEEE Transactions on 3(3), 529–538 (Sept 2008)
  * [30] Marra, F., Gragnaniello, D., Cozzolino, D., Verdoliva, L.: Detection of gan-generated fake images over social networks. In: 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). pp. 384–389. IEEE (2018)
  * [31] Nam, S.H., Ahn, W., Mun, S.M., Park, J., Kim, D., Yu, I.J., Lee, H.K.: Content-aware image resizing detection using deep neural network. In: 2019 IEEE International Conference on Image Processing (ICIP). pp. 106–110. IEEE (2019)
  * [32] Nam, S.H., Ahn, W., Yu, I.J., Kwon, M.J., Son, M., Lee, H.K.: Deep convolutional neural network for identifying seam-carving forgery. arXiv preprint arXiv:2007.02393 (2020)
  * [33] Nataraj, L., Sarkar, A., Manjunath, B.S.: Improving re-sampling detection by adding noise. In: SPIE, Media Forensics and Security. vol. 7541 (2010), http://vision.ece.ucsb.edu/publications/lakshman˙spie˙2010.pdf
  * [34] Popescu, A.C., Farid, H.: Exposing digital forgeries by detecting traces of resampling. IEEE Transactions on signal processing 53(2), 758–767 (2005)
  * [35] Rao, Y., Ni, J.: A deep learning approach to detection of splicing and copy-move forgeries in images. In: Information Forensics and Security (WIFS), 2016 IEEE International Workshop on. pp. 1–6. IEEE (2016)
  * [36] Ryu, S.J., Lee, H.K.: Estimation of linear transformation by analyzing the periodicity of interpolation. Pattern Recognition Letters 36, 89–99 (2014)
  * [37] Salloum, R., Ren, Y., Kuo, C.C.J.: Image splicing localization using a multi-task fully convolutional network (mfcn). Journal of Visual Communication and Image Representation 51, 201–209 (2018)
  * [38] Sarkar, A., Nataraj, L., Manjunath, B.S.: Detection of seam carving and localization of seam insertions in digital images. In: Proceedings of the 11th ACM workshop on Multimedia and security. pp. 107–116. ACM (2009)
  * [39] Shamir, A., Avidan, S.: Seam carving for media retargeting. Communications of the ACM 52(1), 77–85 (2009)
  * [40] Sheng, G., Gao, T.: Detection of seam-carving image based on benford’s law for forensic applications. International Journal of Digital Crime and Forensics (IJDCF) 8(1), 51–61 (2016)
  * [41] Sheng, G., Li, T., Su, Q., Chen, B., Tang, Y.: Detection of content-aware image resizing based on benford’s law. Soft Computing 21(19), 5693–5701 (2017)
  * [42] Wattanachote, K., Shih, T.K., Chang, W.L., Chang, H.H.: Tamper detection of jpeg image due to seam modifications. IEEE transactions on information forensics and security 10(12), 2477–2491 (2015)
  * [43] Wei, J.D., Lin, Y.J., Wu, Y.J., Kang, L.W.: A patch analysis approach for seam-carved image detection. In: ACM SIGGRAPH 2013 Posters, pp. 1–1 (2013)
  * [44] Wu, Q., Sun, S.J., Zhu, W., Li, G.H., Tu, D.: Detection of digital doctoring in exemplar-based inpainted images. In: Machine Learning and Cybernetics, 2008 International Conference on. vol. 3, pp. 1222–1226. IEEE (2008)
  * [45] Ye, J., Shi, Y., Xu, G., Shi, Y.Q.: A convolutional neural network based seam carving detection scheme for uncompressed digital images. In: International Workshop on Digital Watermarking. pp. 3–13. Springer (2018)
  * [46] Zhang, D., Li, Q., Yang, G., Li, L., Sun, X.: Detection of image seam carving by using weber local descriptor and local binary patterns. Journal of information security and applications 36, 135–144 (2017)
  * [47] Zhang, D., Yang, G., Li, F., Wang, J., Sangaiah, A.K.: Detecting seam carved images using uniform local binary patterns. Multimedia Tools and Applications 79(13), 8415–8430 (2020)
  * [48] Zhang, X., Karaman, S., Chang, S.F.: Detecting and simulating artifacts in gan fake images. In: 2019 IEEE International Workshop on Information Forensics and Security (WIFS). pp. 1–6. IEEE (2019)