# A Dynamically Weighted Loss Function for Unsupervised Image Segmentation

1st Boujemaa Guermazi  Electrical, Computer, and Biomedical Engineering
Toronto Metropolitan University
Toronto, Ontario
<EMAIL_ADDRESS>2nd Riadh Ksantini  Computer Science
University of Bahrain
Zallaq, Bahrain
<EMAIL_ADDRESS>3rd Naimul Khan  Electrical, Computer, and Biomedical
Engineering
Toronto Metropolitan University
Toronto, Ontario
<EMAIL_ADDRESS>This work was funded by the Natural Sciences and
Engineering Research Council of Canada.Code for this project is available at:
https://github.com/bijou-bijou/DynamicSeg

###### Abstract

Image segmentation is the foundation of several computer vision tasks, where
pixel-wise knowledge is a prerequisite for achieving the desired target. Deep
learning has shown promising performance in supervised image segmentation.
However, supervised segmentation algorithms require a massive amount of data
annotated at a pixel level, thus limiting their applicability and scalability.
Therefore, there is a need to invest in unsupervised learning for
segmentation. This work presents an improved version of an unsupervised
Convolutional Neural Network (CNN) based algorithm that uses a constant weight
factor to balance between the segmentation criteria of feature similarity and
spatial continuity, and it requires continuous manual adjustment of parameters
depending on the degree of detail in the image and the dataset. In contrast,
we propose a novel dynamic weighting scheme that leads to a flexible update of
the parameters and an automatic tuning of the balancing weight between the two
criteria above to bring out the details in the images in a genuinely
unsupervised manner. We present quantitative and qualitative results on four
datasets, which show that the proposed scheme outperforms the current
unsupervised segmentation approaches without requiring manual adjustment.

###### Index Terms:

Image segmentation, unsupervised learning

##  I Introduction

Image segmentation is fundamental to many application domains such as medical
imaging, surveillance, self-driving cars, and sports. Image classification
allocates a category label to the entire image. In contrast, image
segmentation generates a category label for each input image pixel, dividing a
whole picture into subgroups known as image segments. Although we have made
notable progress, the segmentation process is still challenging due to various
factors such as illumination variation, occlusion, and background clutters.

Classical pioneering segmentation techniques such as active contour models
(ACM) [1], k-means [2], and graph-based segmentation method (GS) [3] impose
global and local data and geometry constraints on the masks. As a result,
these techniques become sensitive to initialization and require heuristics
such as point resampling, making them unsuitable for modern applications.

The current state-of-the-art models are deep learning methods based on
convolutional neural networks (CNNs), especially the Mask R-CNN framework [4],
which has been successfully put into practice for supervised semantic and
instance image segmentation. Even so, such methods require a considerable
amount of hand-labeled data, limiting their applicability in many areas. The
problem becomes much more acute when it comes to pixel-wise classification,
where the annotation cost per image is expensive. A possible solution to this
problem is unsupervised image segmentation, where the image is automatically
segmented into semantically similar regions. The task has been studied as a
clustering problem in recent literature, reaching promising results. The
Differentiable Feature Clustering [5] is a state-of-the-art CNN-based
clustering algorithm that simultaneously optimizes the pixel labels and
feature representations [5]. It applies a combination of feature similarity
and spatial continuity constraints to backpropagate the model’s parameters.
Feature similarity corresponds to the constraint that pixels in the same
cluster should be similar to each other. Spatial continuity refers to the
constraint that pixels in the same cluster should be next to each other
(continuous). However, to reach the desired segmentation result, [5] applies a
manual parameter tuning to find the optimal balancing weight $\mu$, which
fails to achieve a good balance between the two aforementioned constraints
depending on the degree of details in the image and dataset. This work
introduces a novel dynamic weighting scheme that leads to a flexible update of
the parameters and an automatic tuning of the balancing weight $\mu$. We
achieve this by conditioning the value of $\mu$ to the number of predicted
clusters and iteration number. We dynamically prioritize one of the
constraints at each iteration to achieve a good balance. Experimental results
on four benchmark datasets show that our method achieves better quantitative
metrics and qualitative segmentation results by striking a better balance
between feature similarity and spatial continuity.

##  II Related Works

Figure 1: The CNN Framework: a forward-backward process which is iterated $T$
times to obtain the final prediction of the cluster labels $c_{n}$.

Machine learning methods directly address the image segmentation problem by
considering various features found in the image, such as colour or pixel
information. K-means clustering [2], for instance, is a region-based method
that divides an image into K groups based on the discontinuity properties of
the extracted features. K-means is widely used for unsupervised segmentation.
However, the hard clustering that assumes sharp boundaries between clusters in
K-means does not guarantee continuous areas. The graph-based segmentation (GS)
[3] is another region-based method that performs segmentation based on pixel
similarity, distance, or colour weights. As a result, GS preserves details in
low variability image regions while ignoring details in high variability
regions, plus GS has a complex computation. On the other hand, The Invariant
Information Clustering [6] is an edge-based method and is easy to implement
and rigorously grounded in information theory. The deep net IIC directly
trains a randomly initialized neural network into semantic clusters without
the need for postprocessing to cluster the high-dimensional representations.

Majority of the recent unsupervised segmentation models utilize deep learning.
X. Xia et al. [7] tackles the problem of unsupervised image segmentation
inspired by the concept of U-net architecture [8]. They join two U-net
structures into an auto-encoder followed by a post-processing phase to refine
its prime segmentation. The model merges segments using a Hierarchical
Segmentation method to form the final image segments. It is a computationally
expensive process and requires extensive hyperparameter tuning. L. Zhou et al.
proposed another neural network-based algorithm; the Deep Image Clustering
(DIC) [9]. DIC divided the image segmentation problem into two steps; A
feature transformation sub-network (FTS) to extract the features first, then a
trainable deep clustering sub-network (DCS) that groups the pixels to split
the image into non-overlapping regions. The DIC model has proven to be less
sensitive to varying segmentation parameters and has lower computation costs.
However, it uses superpixels to optimize the model parameters, which results
in pre-determined fixed boundaries for segmentation regions.

In contrast to the approaches mentioned above that use intermediate
representations followed by post-processing, W. Kim et al. propose utilizing a
clustering algorithm that jointly optimizes the pixel labels and feature
representations and updates their parameters using backpropagation [5].
Furthermore, in contrast to the superpixel-based refinement process [10], the
authors present a novel spatial continuity loss function to achieve dynamic
segmentation boundaries as opposed to utilizing superpixels. It is a simple
process that uses the backpropagation of the feature similarity loss and a new
spatial continuity loss that resolves the problem caused by superpixels in
[10]. However, the weight coefficient for the loss function used must be
adjusted each time according to the datasets and the degree of detail of the
image. Instead, we propose a dynamic method for adjusting the weighting factor
automatically, as described in the following sections.

##  III Proposed Method

###  III-A The CNN framework

For a fair comparison, we use the same network architecture and training
framework used in [5]. The model is shown in Figure (1). $M$ convolutional
components are used to produce a $p$-dimensional feature map $r$. The CNN
subnetwork consists of $2D$ Conv layers, $Relu$ functions, batch
normalizations, and a final linear layer classifier that classifies the
features of each pixel into $q^{\prime}$ classes. A batch normalization
function is applied to the response map $r$ to get a normalized map
$r^{\prime}$. Lastly, the $argmax$ function is used to select the dimension
that has the maximum value in $r^{\prime}_{n}$. Each pixel is assigned the
corresponding cluster label $c_{n}$, which is identical to allocating each
pixel to the closest point among the $q^{\prime}$ representative points.
During the backward propagation, first, the loss $L$ (defined in the next
section) is calculated. Then, the convolutional filters’ parameters and the
classifier’s parameters are updated with stochastic gradient descent. This
forward-backward process is iterated $T$ times to obtain the final prediction
of the cluster labels $c_{n}$. The segmentation problem is handled in an
unsupervised manner without knowing the exact number of clusters. The latter
must be flexible according to the content of the image. Therefore, we must
allocate a large number q to the initial cluster labels q’. After that,
similar or spatially related pixels are iteratively integrated to update the
number of clusters $q^{\prime}$.

###  III-B Proposed dynamic weighting scheme

The loss function in [5] is designed to strike a balance between feature
similarity and spatial continuity. The loss function L from [5] is shown in
Equation (1).

$L=L_{sim}(\\{r^{\prime}_{n},c_{n}\\})+\mu L_{con}(\\{r^{\prime}_{n}\\}),$ (1)

where, $\mu$: Weight for balancing; $L_{sim}$ : feature similarity; $L_{con}$:
spatial continuity; $c_{n}$: cluster labels ; $r^{\prime}_{n}$: normalized
response.

The loss function in equation (1) consists of two parts. The first part is the
feature similarity loss $L_{sim}$ which is the cross-entropy loss between the
normalized response map $r^{\prime}_{n}$ and the cluster labels ${c_{n}}$.
Minimizing this loss would reveal network weights that ease the extraction of
more accurate attributes for segmentation. Thus, $L_{sim}$ ensures that pixels
of similar features should be assigned the same label. The second part is the
Manhattan Distance $L1$ Norm of horizontal and vertical differences of the
response map $r^{\prime}_{n}$ as a spatial continuity loss which redresses the
deficiency caused by superpixels [10]. This additional loss component
$L_{con}$ has proven to be efficient in removing an excessive number of labels
due to their complex patterns or textures and ensuring that continuous pixels
are assigned the same label.

While the loss L as mentioned above can result in reasonably accurate
unsupervised segmentation results as reported in [5], the segmentation results
are susceptible to the balancing parameter $\mu$. Figure (2) show examples of
the sensitivity to this parameter on an example image from the BSD500 dataset.
As can be seen, for $\mu=50$ and $\mu=100$, the segmentation is coarse,
resulting in sky, buildings, and coastal regions. However, the image is
further segmented with $\mu=1$ and $\mu=5$, where buildings are further
segmented into glass buildings, concrete buildings, and different floors.
Although the authors argue that the value of $\mu$ is proportional to the
coarseness of segmentation, We see that the results are not consistent, e.g.
the segmentation for $\mu=50$ appears coarser than $\mu=100$.

Figure 2: Results for different $\mu$ values on a sample image from the BSD500
dataset using the approach in [5].

This poses a problem in practice. Such high sensitivity to the parameter means
that for each dataset, this parameter has to be tuned extensively to obtain a
result that is semantically more meaningful.

TABLE I: Comparison of $mIOU$ for unsupervised segmentation on BSD500 and
PASCAL VOC2012. Best scores are in bold.

| dataset
---|---
Method | BSD500 All | BSD500 Fine | BSD500 Coarse | BSD500 Mean | PASCAL VOC2012
IIC [6] | 0.172 | 0.151 | 0.207 | 0.177 | 0.201
k-means clustering | 0.240 | 0.221 | 0.265 | 0.242 | 0.238
Graph-based Segmentation [3] | 0.313 | 0.295 | 0.325 | 0.311 | 0.286
CNN-based + superpixels [10] | 0.226 | 0.169 | 0.324 | 0.240 | ***
CNN-based + weighted loss, $\mu=5$ [5] | 0.305 | 0.259 | 0.374 | 0.313 | 0.288*
Spatial Continuity Focus $\mu^{\prime}=100/q^{\prime}$ | 0.329 | 0.288 | 0.406 | 0.341 | 0.289
Spatial Continuity Focus $\mu^{\prime}=50/q^{\prime}$ | 0.330 | 0.290 | 0.407 | 0.342 | 0.290
Feature Similarity Focus $\mu^{\prime}=q^{\prime}/10$ | 0.330 | 0.297 | 0.390 | 0.339 | 0.280
Feature Similarity Focus $\mu^{\prime}=q^{\prime}/15$ | 0.349 | 0.307 | 0.420 | 0.359 | 0.275

  * *

All the results are copied from[5], except for the results on Pascal VOC 2012,
as there is no indication in [5] which images from the dataset were used for
evaluation. Therefore, we chose 150 random images and re-produced the results
for this dataset only.

In this work, we propose changing the weighting parameter’s value during
training dynamically. Our observation is that we can prioritize feature
similarity during training at the earlier iterations and gradually shift focus
to spatial continuity (or vice versa). We suggest a new dynamic loss function
that includes a continuous variable $\mu$. The weight $\mu$ depends directly
on the number of predicted clusters and iterations. We examine two versions of
the proposed weighting scheme:

  * •

Start by gathering continuous regions and shifting focus to feature similarity
later. We call this approach Feature Similarity Focus (FSF). In this case, the
performed trials lead to a linear function of the number of clusters (q’) for
the new dynamic balancing weight $\mu^{\prime}=(q^{\prime}/\mu)$ as shown in
equation(2). We tried other versions that vary exponentially with the value of
$q^{\prime}$; However, such functions resulted in a rapid change in the value
of $\mu^{\prime}$, which was not conducive to the balance we sought to achieve
between the two constraints.

$L_{FSF}=L_{sim}(\\{r^{\prime}_{n},c_{n}\\})+(q^{\prime}/\mu)L_{con}(\\{r^{\prime}_{n}\\})$
(2)

  * •

Start by prioritizing feature similarity criteria early in training and end
with a higher spatial continuity weight. We call this approach Spatial
Continuity Focus (SCF). In this case, the proposed dynamic weight would be the
multiplicative inverse of the number of clusters
$\mu^{\prime}=(\mu/q^{\prime})$ as shown in equation (3). Similar to FSF, we
tried an exponential form, but the decay was too fast for it to be effective.

$L_{SCF}=L_{sim}(\\{r^{\prime}_{n},c_{n}\\})+(\mu/q^{\prime})L_{con}(\\{r^{\prime}_{n}\\})$
(3)

##  IV Experimental Results

Figure 3: Qualitative Results on select BSD500 and PASCAL VOC2012 images. Same
color corresponds to the pixels being assigned the same clustering label by
the algorithm. Please read Section IV-C for discussion on these results.

###  IV-A Experiment Setup

The objective of the experiments is to show that our proposed dynamic
weighting approach provides us with a more semantically meaningful
segmentation. We replicate the experiments performed in [5]. For all the
tests, we fixed the number of components in the feature extraction phase $M$
to $3$. In addition, we set the dimension of the features space $p$ equal to
the dimension of cluster space $q$ equal to $100$. Finally, we report the mean
Intersection Over Union $mIOU$ overall images for the benchmark datasets.
Ground truth is only used during the assessment phase and has no bearing on
the training process.

Berkley Segmentation Dataset BSD500 [11] and PASCAL Visual Object Classes 2012
[12] are used to evaluate the segmentation results quantitatively and
qualitatively. BSD500 consists of 500 color and grayscale natural images.
Following the experimental setup in [5], we used the 200 color images of the
BSD500 test-set to evaluate all the models. Since the BSD500 dataset contains
multiple types of ground truth, we set three types of mIOU counting to assess
the given results; ”BSD500 All” takes into account all the ground truth files,
”BSD500 Fine” considers the only ground truth file per image that has the most
significant number of segments, and ”BSD500 Coarse” takes only the ground
truth file that contains the smallest number of segments. We defined ”BSD500
Mean” as the average value of the above three measurements. For PASCAL
VOC2012, we considered each segment an individual entity ignoring the object
classification. VOC2012 is a large dataset containing 17,124 images with 2,913
images with semantic segmentation annotations. We randomly chose 150 of the
semantic segmentation images to evaluate our method.

For the Icoseg [13] and Pixabay [14] datasets, select images are used to
demonstrate additional qualitative results only. All of the aforementioned
experimental settings are identical to the settings from [5] for a fair
comparison.

###  IV-B Quantitative results

Figure 4: Qualitative Results on select Icoseg and Pixabay images. Same color
corresponds to the pixels being assigned the same clustering label by the
algorithm. Please read Section IV-C for discussion on these results.

To demonstrate the effectiveness of the dynamic loss function, we compare the
results with The Invariant Information Clustering (IIC) [6], the k-means
clustering [2] and the graph-based segmentation (GS) [3] in addition to the
two CNN-based methods in [10] and [5]. One important note is that we report
the same results as in [5], except for the results of the method [5] on Pascal
VOC 2012, as there is no clue about how the images were selected. In addition,
the method is sensitive to the random initialization of the neural network
weights, resulting in different mIOU scores every time we re-run the
experiment. Therefore, we could not exactly reproduce the results reported in
the paper using their provided code base. For a fair comparison, we fixed the
initialization to a single set of values for all methods, including ours.

The Differentiable Feature Clustering [5] uses a fixed value of $\mu$. Yet
five is the ideal value of $\mu$ for BSD500 [11], and PASCAL VOC 2012 datasets
[12], as that achieved the best results as reported in [5]. In contrast, for
our proposed SCF and FSF methods, the old $\mu$ is still the only hyper-
parameter that needs to be tuned, but it is integrated into a dynamic loss
function, as shown in equations (3) and (2). The best $\mu$ for SCF and FSF
clustering were experimentally determined from {25, 45, 50, 55, 60, 75 100,
200} and {2, 10, 15, 25,50, 100}, respectively. We report results for $\mu$ =
10 and $\mu$ = 15 for FSF clustering ; $\mu$ = 100 and $\mu$ = 50 for SCF
clustering.

As illustrated in Table I, the graph-based segmentation (GS) method
outperformed the Differentiable Feature Clustering [5] on “BSD500 all” and
“BSD500 fine”. However, our proposed method outperforms [10], IIC, k-means,
and, in particular, [5] and GS for both datasets obtaining the best mIOU
scores. While FSF shows the best performance for the BSD500 dataset, SCF
achieves the best scores in the Pascal VOC2012 dataset and the second-best
score in the BSD500 dataset.

###  IV-C Qualitative results

We also provide qualitative results on a few images as done in [5]. As shown
in Figure (3), our model is more effective in bringing out segmentation
regions that are semantically related. For example, For the ”Show Jumping”
image (column 5), the horse and the obstacle are classified as the same class
by [5] (both yellow). However, for both FSF and SCF, the horse and the
obstacle are appropriately distinguished. For the ”ship” image (column 2), [5]
fails to differentiate between the sky and the body of the ship (both red),
but both proposed SCF and FSF can do it successfully.

Further qualitative results are shown for select Icoseg [13] and Pixabay [14]
datasets that were also used in [5]. These results can be seen in Figure (4).
The qualitative results on these datasets are presented to further demonstrate
that our proposed approaches do not require as much parameter tuning as [5]
does. Figure (4) highlights that the baseline Differentiable Feature
Clustering [5] is quite parameter sensitive. For each dataset, the weighting
balance $\mu$ must be tuned extensively to obtain a more semantically
meaningful result. For instance, PASCAL VOC 2012 and BSD500 datasets require a
small balancing value $\mu=5$. While, Icoseg [13] and Pixabay [14] datasets
need a much larger balance value $\mu=50$ and $\mu=100$, respectively. On the
other hand, As illustrated in Figures (3) and (4), our proposed method has
proven effective in dealing with different datasets using the same weight for
both FSF and SCF. The proposed methods also bring out details in an
unsupervised manner that is semantically more meaningful. For example, using
the Feature Similarity Focus method (FSF), the red car from the iCoseg dataset
(column 2 row 4 in Figure (4)) displays more detail on the tires and more
precise building outlines than the details extracted by [5] (column 2 row 2),
where the car is partially blended into the road. Similarly, for the peppers
image (column 4 in Figure (4)), [5] was unable to identify the shapes of the
individual peppers accurately. Both of our proposed method do a much better
job, even with the same value of $\mu$ as the other images.

Comparing the proposed SCF and FSF, SCF shows potency in class segmentation,
while FSF performs better in object segmentation tasks. For instance, in the
third row in Figure (3), the SCF method segmented the image into the sky,
dust, and monuments. However, the FSF segmented the image into the sky, dust,
pyramid, and Sphinx. Depending on the application context, one of these
approaches could be more favorable.

##  V Conclusion

In this paper, we propose a dynamic weighting scheme for CNN-based
unsupervised segmentation. Our proposed FSF and SCF approaches can strike a
balance between feature similarity and spatial continuity for better
segmentation. Qualitative and quantitative results on four datasets show that
our proposed approach outperforms classical methods for unsupervised image
segmentation such as k-means clustering and a graph-based segmentation method,
and outperforms the state-of-the-art deep learning model, the Differentiable
Feature Clustering, [5] while producing semantically more meaningful results.

## References

  * [1] Michael Kass, Andrew Witkin, and Demetri Terzopoulos,  “Snakes: Active contour models,”  International Journal of Computer Vision, vol. 1, no. 4, pp. 321–331, jan 1988.
  * [2] J.MacQueen et all,  “Some methods for classification and analysis of multivariate observations,”  Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, vol. 1, no. 14, pp. 281–297, 1967.
  * [3] Pedro F. Felzenszwalb and Daniel P. Huttenlocher,  “Efficient Graph-Based Image Segmentation,”  International Journal of Computer Vision, vol. 59, no. 2, pp. 167–181, sep 2004.
  * [4] Piotr Doll, Ross Girshick, and Facebook Ai,  “Mask R-CNN,”  CVPR, 2017.
  * [5] Wonjik Kim, Asako Kanezaki, and Masayuki Tanaka,  “Unsupervised Learning of Image Segmentation Based on Differentiable Feature Clustering,”  IEEE Transactions on Image Processing, vol. 29, pp. 8055–8068, 2020\.
  * [6] J.Henriques X.Ji and A.Vedaldi,  “Invariant Information Clustering for Unsupervised Image Classification and Segmentation,”  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [7] Xide Xia and Brian Kulis,  “W-Net: A Deep Model for Fully Unsupervised Image Segmentation,”  arXiv, nov 2017.
  * [8] Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi,  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, vol. 9351 of Lecture Notes in Computer Science,  Springer International Publishing, Cham, 2015.
  * [9] Lei Zhou and Weiyufeng Wei,  “DIC: Deep Image Clustering for Unsupervised Image Segmentation,”  IEEE Access, vol. 8, pp. 34481–34491, 2020.
  * [10] Asako Kanezaki,  “Unsupervised Image Segmentation by Backpropagation,”  Icassp, pp. 2–4, 2018.
  * [11] Pablo Arbeláez, Michael Maire, Charless Fowlkes, and Jitendra Malik,  “Contour detection and hierarchical image segmentation,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898–916, 2011.
  * [12] M Everingham, M Everingham, A Zisserman, A Zisserman, C Williams, and C Williams,  “The PASCAL Visual Object Classes Challenge 2006 (VOC2006) results,”  International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010.
  * [13] Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jiebo Luo, and Tsuhan Chen,  “iCoseg: Interactive co-segmentation with intelligent scribble guidance,”  in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. jun 2010, pp. 3169–3176, IEEE.
  * [14] Asako Kanezaki,  “Unsupervised image segmentation by backpropagation,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 1543–1547.


* Hilditch  _et al._ [2016] D. Hilditch, A. Weyhausen, and B. Brügmann, Pseudospectral method for gravitational wave collapse, Phys. Rev. D93, 063006 (2016), arXiv:1504.04732 [gr-qc] .
  * Strand [1994] B. Strand, Summation by parts for finite differencing approximations for d/dx, J. Comput. Phys. 110, 47 (1994).
  * Bjørhus [1995] M. Bjørhus, The ODE Formulation of Hyperbolic PDEs Discretized by the Spectral Collocation Method, SIAM J. Sci. Comput. 16, 542 (1995).
  * Kopriva [2009] D. A. Kopriva,  _Implementing Spectral Methods for Partial Differential Equations_ (Springer, New York, 2009).
  * Pound [2017] A. Pound, Nonlinear gravitational self-force: second-order equation of motion, Phys. Rev. D 95, 104056 (2017), arXiv:1703.02836 [gr-qc] .