# Unsupervised Data-Driven Nuclei Segmentation for Histology Images

###### Abstract

An unsupervised data-driven nuclei segmentation method for histology images,
called CBM, is proposed in this work. CBM consists of three modules applied in
a block-wise manner: 1) data-driven color transform for energy compaction and
dimension reduction, 2) data-driven binarization, and 3) incorporation of
geometric priors with morphological processing. CBM comes from the first
letter of the three modules – “Color transform”, “Binarization” and
“Morphological processing”. Experiments on the MoNuSeg dataset validate the
effectiveness of the proposed CBM method. CBM outperforms all other
unsupervised methods and offers a competitive standing among supervised models
based on the Aggregated Jaccard Index (AJI) metric.

Index Terms—  Histology images, Nuclear segmentation, Color Transform,
Unsupervised, Morphological Processing

##  1 Introduction

Histology images provide strong cues to pathologists in cancer diagnosis and
treatment. Automated nuclei instance-level segmentation for histology images
provides not only the number, density and sizes of nuclei but also
morphological features, such as the magnitude and the cytoplasmic ratio. This
information facilitates cancer diagnosis and assessment of the tumor
aggressiveness rate. Nuclei segmentation tasks can be conducted in a
supervised or an unsupervised manner. For supervised methods, the annotation
of high resolution histology images in pixel-level accuracy is a time-
consuming job, being carried out by expertized physicians. Other identified
challenges include the variability of cell appearance from different organs,
unclear boundaries, as well as color and intensity variations in stained
images from different laboratories. Furthermore, it is a subjective task and
annotated labels tend to vary from one person to the other. All the above
factors challenge the practical generalizability of supervised segmentation
methods, as histology images become larger in their number and more
diversified in content.

Earlier methods on nuclei segmentation were mainly unsupervised. They were
based on thresholding [1, 2], mathematical morphology for robust feature
extraction [3], or statistical modeling for segmentation and likelihood
maximization for boundary refinement [4]. Another popular tool was the
watershed algorithm, which was combined with various ways for extracting
potential nuclei locations [5, 6]. Moreover, a work by Ali et al. [7] proposed
an adaptive active contour mechanism that takes boundary- and region-based
energy terms of the cell into account.

Recently, deep-learning-based (DL) methods [8, 9, 10, 11] have been applied to
this problem. A wide variety of DL models are trained by human labeled data.
In general, supervised methods provide significantly better performance than
the unsupervised ones. The motivation of our research stems from the fact that
nuclei instance labeling is a fairly laborious task, with reportedly highly
subjective annotations and miss-labeling rate [12] that challenges the
supervised solutions. Several studies have been made to mitigate the labeling
cost. For example, Qu et al. [13] proposed a two-stage learning framework
using coarse labels. Other researchers [14, 15] investigated the self-
supervised DL method to reduce the number of required labeled data by
exploiting the observation that nuclei size and texture can determine the
magnification scale. Besides, there is a domain shift problem [16] arising
from stain and nuclei variations in histology images of different organs,
patients and acquisition protocols.

Fig. 1: Illustration of nuclei cell appearance in histology images. Fig. 2: An
overview of the proposed CBM method.

In spite of the high performance, DL-based solutions have their own
shortcomings on top of the high labeling cost. First, they are perceived as a
“black-box” approach. Interpretable solutions are highly desired in medical
applications as they enable explainable decisions and the tools are more
trustworthy in clinical use. Second, they bear a high computational cost in
training and testing [17] because of a large number of parameters in order to
achieve a certain performance.

In this work, we propose an unsupervised data-driven method to solve the
nuclei segmentation problem. The solution consists of three modules applied to
each image block of size $50\times 50$: 1) data-driven color transform for
energy compaction and dimension reduction, 2) data-driven binarization, and 3)
incorporation of geometric priors with morphological image processing. It is
named “CBM” because of the first letter of the three modules – “Color
transform”, “Binarization” and “Morphological processing”.

We conduct experiments on the MoNuSeg dataset to demonstrate the effectiveness
of the proposed CBM method. It outperforms all other unsupervised approaches
and stands in a competitive position with supervised ones based on the
Aggregated Jaccard Index (AJI) performance metric. Thus, the proposed CBM
method is an attractive solution in practice nuclei segmentation, since it
yields comparable performance with state-of-the-art supervised methods while
requiring no training labels.

##  2 Proposed CBM Method

Each histology test image of the experimenting dataset has a spatial
resolution of $1000\times 1000$ pixels and each pixel has R, G and B three
color channels. We partition each image into nonoverlapping blocks of size
$50\times 50$ pixels and apply contrast enhancement to accentuate the
boundaries around nuclei as a pre-processing step. Then, CBM processes each
block independently with three modules as elaborated below.

###  2.1 Data-Driven Color Transform

Nuclei segmentation is a binary decision problem for each pixel; namely, it
belongs to either the nucleus or the background region. We need pixel
attributes to make decision. A pixel has R/G/B color values in raw histology
images. There exist strong correlations between RGB channels in histology
images as shown in Fig. 3(a). We can exploit this property for energy
compaction.

There are many well-known color spaces such as YUV, LAB, HVS, etc. However,
they are not data-dependent color transforms. To achieve optimal energy
compaction, we apply the principal component analysis (PCA) to the RGB 3D
color vector of pixels inside one block. That is, we can determine the
covariance matrix between the R/G/B color components based on pixels in the
region. Then, the three eigenvectors define the optimal transform and their
associated eigenvalues indicate the energy distributions among the three
channels. The transform output channels are named $P$, $Q$ and $R$ channels.
They are channels of the first, the second and the last components,
respectively.

The energy distributions of three color components of the RGB, LAB and PQR
color spaces for a representative histology image is shown in Table 1, where
the P/Q/R channel energy distribution is averaged over all blocks in one
image. As shown in the table, the first principal component, $P$, has an
extremely high energy percentage while the rest two components have very
limited energy. As a result, we can treat the latter two as background noise
and discard them. Instead of considering segmentation using three attributes,
we simplify it greatly using a single attribute. As the first principal
subspace, $P$ points at the direction where the variance is maximized. It
better captures the transition from background to nuclei cell areas, thus
leading to a more distinct local histogram.

We normalize raw $P$-channel values to the range of $[0,1]$ with linear
scaling. Note that, if ${\bf x}$ is an eigenvector of the covariance matrix of
RGB color channels, $-{\bf x}$ is also an eigenvector. We choose the one that
maps a higher $P$ value to background (i.e., brighter) and a lower $P$ value
to nuclei (i.e., darker). This can be easily achieved by imposing the $P$
value to be consistent with the luminance value in the LAB color space, since
the background luminance is higher than those of nuclei. The original color
and the normalized P-channel representations for a block are compared in Fig.
3. Performance using the $P$ channel against the $L$ channel (in the LAB color
space) will be compared in Sec. 3.

Table 1: Energy distribution of three channels of three color spaces in a representative histology image. RGB | LAB | PQR
---|---|---
38.1% (R) | 29.3% (L) | 97.7% (P)
27.2% (G) | 40.1% (A) | 1.9% (Q)
34.7% (B) | 30.6% (B) | 0.4% (R)

(a) (b)

(c)

Fig. 3: Comparison of two representations in a block image: (a) R/G/B color
and (b) $P$ value in gray and (c) its corresponding bi-modal histogram.

One can see that the energy compaction capability of P/Q/R depends on color
homogenuity in a region. In most images, as the block size grows larger, the
$P$-channel energy compaction property becomes lower. This is attributed to
the fact that the color distribution of a larger block is less homogeneous and
the correlation structure is more complicated.

###  2.2 Data-Driven Binarization

To conduct the binary classification of pixels, we study the histogram of the
$P$ value of each pixel in a block. A representative histogram is shown in
Fig. 3(c), which has a bi-modal distribution. The modality in the left is
contributed by the $P$ values of pixels in the nuclei region while that in the
right comes from the $P$ values of pixels in the background region. There
exists a valley between the two peaks, which are from pixels lying on
transitional boundaries between nuclei and background. Thus, binarization can
be achieved by identifying the intermediate point in between the two
modalities and use the associated $P$ value as the adaptive threshold, which
is denoted by $T$. A pixel of $P$ value is classified to the background region
if $P>T$ and the nuclei region if $P\leq T$. Since threshold $T$ is determined
by the histogram of the $P$ value of pixels in a block, the binarization
process is fundamentally an adaptive thresholding method. Whether it will
yield a successful outcome depends on the bi-modal histogram assumption.

The bi-modal histogram assumption holds under the following two conditions:

  1. 1.

if the block size is not too large,

  2. 2.

if the ratio of the nuclei pixel number and the background pixel number does
not deviate much from unity.

In case the first condition is not met, we may see $K$-modalities with $K>2$.
Then, there are multiple valley points, which makes the threshold selection
challenging. If the second condition is not met, it means that the majority of
pixels belong to one of two classes and we may see one dominant modality while
the second modality is weak. Because it is difficult to choose a robust
threshold under the first condition, we partition one block of size $50\times
50$ into four sub-blocks of size $25\times 25$ and conduct the data-driven
color transform and binarization in each of the four sub-blocks. On the other
hand, if the second condition happens, we merge 4 blocks into a super-block of
size $100\times 100$ and conduct the same processing in the super-block.

###  2.3 Morphological Processing

We have so far concentrated on pixel classification based on its color
attributes in the first two modules of the CBM method. Now, we would like to
take the neighborhood of a pixel into account. Typically, nuclei appear in
form of rounded blobs (i.e., convex objects). Yet, we observe the following
three common errors after the second module:

  * (a)

nuclei instances may be falsely connected,

  * (b)

false positives may appear because of the staining process,

  * (c)

holes exist inside the nuclei cell because of the inner intensity variations.

For (a), we can split the larger ones using the convex hull algorithm to find
high convexity areas that imply an underlying connectivity between two cells.
For (b), we can filter out abnormally small nuclei in the first place based on
the prior knowledge on the average area of a nucleus. For (c), the hole
filling filter is used to correct the false negatives inside the nuclei cell.
For some images with dense cell areas and blurred boundaries, the connectivity
may be more severe, having bundled together more than two cells. We found
that, after applying the previous procedure in an iterative manner, the
performance further increases. Thus, we repeatedly remove small cells and
split larger pieces –with relatively high convexity areas–, until no further
changes in the full image. The effect of this processing is illustrated in an
example as shown in Fig. 4.

Fig. 4: Visualization of the morphological processing effect: input block
(upper left), noisy binarized output (upper right), an improved result by
splitting two distinct nuclei (bottom left), and segmentation ground-truth
(bottom right).

##  3 Experimental Results

The data from the 2018 MICCAI MoNuSeg Challenge [9] are used to evaluate the
performance of our proposed CBM method. This is a popular dataset for existing
work, providing also different testing configurations. For the validation
purpose, we follow the data splitting as suggested in [9].

Table 2: Comparative results of different methods using the AJI on Test-$2$ set ($6$ histology images from bladder, colon, stomach) and the testing data ($14$ images) from the MoNuSeg Challenge, where the best performance is shown in boldface. Method |  Test-$2$ | Challenge Data
---|---|---
Unsupervised
Cell Profiler [18] | 0.0809 | 0.1232
Fiji [19] | 0.3030 | 0.2733
Self-Supervised [14] | - | 0.5354
CBM (Ours) | 0.5808 | 0.6142
Supervised
CNN2 [9] | - | 0.3482
CNN3 [9] | 0.4989 | 0.5083
DCAN [10] | 0.5449 | -
PA-Net [11] | 0.5608 | -
BES-Net [20] | 0.5823 | -
CIA-Net [8] | 0.6306 | 0.6907
SSL [15] | - | 0.7063

For performance benchmarking with other existing works, we report the results
on the 6 histology images set from bladder, colon, stomach, referred as
Test-2, as well as the test data from the challenge set (14 images from
various organs) in Table 2, where the commonly used AJI metric [9] is adopted.
The AJI metric is more accurate for instance-level segmentation tasks by
taking both the nuclei level detection and pixel-level error performance into
account.

As shown in the table, our method outperforms all other unsupervised ones by a
large margin in either testing scenarios. Notably, the gap is large even for
the DL-based unsupervised approach denoted by “self-supervised” [14] in the
table. As compared with supervised methods, CBM stands at a competitive
position. It achieves similar performance with sophisticated models such as
BES-Net [20]. It is also worthwhile to emphasize that most of the supervised
methods use expensive models as a backbone architecture (in terms of the
number of its parameters) such as Res-Net or Dense-Net. Instead, CBM is a
parameter-free method with simple components. Its computational complexity is
significantly lower.

It is also evident from Table 2 that supervision may help to reach higher
performance. Nevertheless, it is challenging for most DL supervised methods to
generalize well in this problem, especially when the training data are in
paucity. For instance, the current state-of-the-art SSL method achieves the
reported performance using the full training set. According to their weakly
supervision analysis [15], SSL achieves comparable performance with CBM when
trained with roughly 50% of the data.

To illustrate the energy compaction capability of the PQR channel
decomposition, we compare the AJI performance for the $P$ channel and the $L$
channel (in the LAB color space) in Table 3. The advantage of using the $P$
channel is clearly demonstrated in the table.

Table 3: AJI performance comparison between $P$ and $L$ channels. Colorspace |  Test-$2$ | Challenge Data
---|---|---
L | 0.5414 | 0.5856
P | 0.5808 | 0.6142

##  4 Conclusion and Future Work

Nuclei segmentation in histology images is a demanding and prone to errors
task for physicians, and its automation is of high importance for cancer
assessment. The proposed CBM method offers a promising lightweight
unsupervised direction for nuclei segmentation, requiring no labeled data and
no parameter cost. It addressed the problem based on data-driven color
conversion and binarization, as well as a morphological module that takes into
account priors about the nuclei shape and size. In the future, we would like
to boost the segmentation performance furthermore by exploring weakly or self
supervision.

## References

  * [1] S. Naik, S. Doyle, S. Agner, A. Madabhushi, M. Feldman, and J. Tomaszewski, “Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology,” in _2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro_. IEEE, 2008, pp. 284–287.
  * [2] K. Y. Win and S. Choomchuay, “Automated segmentation of cell nuclei in cytology pleural fluid images using otsu thresholding,” in _2017 International Conference on Digital Arts, Media and Technology (ICDAMT)_. IEEE, 2017, pp. 14–18.
  * [3] J.-P. Thiran and B. Macq, “Morphological feature extraction for the classification of digital images of cancerous tissues,” _IEEE Transactions on biomedical engineering_ , vol. 43, no. 10, pp. 1011–1020, 1996\.
  * [4] T. Mouroutis, S. J. Roberts, and A. A. Bharath, “Robust cell nuclei segmentation using statistical modelling,” _Bioimaging_ , vol. 6, no. 2, pp. 79–91, 1998.
  * [5] N. Malpica, C. O. De Solórzano, J. J. Vaquero, A. Santos, I. Vallcorba, J. M. García-Sagredo, and F. Del Pozo, “Applying watershed algorithms to the segmentation of clustered nuclei,” _Cytometry: The Journal of the International Society for Analytical Cytology_ , vol. 28, no. 4, pp. 289–297, 1997.
  * [6] M. Veta, A. Huisman, M. A. Viergever, P. J. van Diest, and J. P. Pluim, “Marker-controlled watershed segmentation of nuclei in h&e stained breast cancer biopsy images,” in _2011 IEEE international symposium on biomedical imaging: from nano to macro_. IEEE, 2011, pp. 618–621.
  * [7] S. Ali, R. Veltri, J. I. Epstein, C. Christudass, and A. Madabhushi, “Adaptive energy selective active contour with shape priors for nuclear segmentation and gleason grading of prostate cancer,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2011, pp. 661–669.
  * [8] Y. Zhou, O. F. Onder, Q. Dou, E. Tsougenis, H. Chen, and P.-A. Heng, “Cia-net: Robust nuclei instance segmentation with contour-aware information aggregation,” in _International Conference on Information Processing in Medical Imaging_. Springer, 2019, pp. 682–693.
  * [9] N. Kumar, R. Verma, S. Sharma, S. Bhargava, A. Vahadane, and A. Sethi, “A dataset and a technique for generalized nuclear segmentation for computational pathology,” _IEEE transactions on medical imaging_ , vol. 36, no. 7, pp. 1550–1560, 2017.
  * [10] H. Chen, X. Qi, L. Yu, Q. Dou, J. Qin, and P.-A. Heng, “Dcan: Deep contour-aware networks for object instance segmentation from histology images,” _Medical image analysis_ , vol. 36, pp. 135–146, 2017.
  * [11] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8759–8768.
  * [12] H. Irshad, L. Montaser-Kouhsari, G. Waltz, O. Bucur, J. Nowak, F. Dong, N. W. Knoblauch, and A. H. Beck, “Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: evaluating experts, automated methods, and the crowd,” in _Pacific symposium on biocomputing Co-chairs_. World Scientific, 2014, pp. 294–305.
  * [13] H. Qu, P. Wu, Q. Huang, J. Yi, Z. Yan, K. Li, G. M. Riedlinger, S. De, S. Zhang, and D. N. Metaxas, “Weakly supervised deep nuclei segmentation using partial points annotation in histopathology images,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 11, pp. 3655–3666, 2020.
  * [14] M. Sahasrabudhe, S. Christodoulidis, R. Salgado, S. Michiels, S. Loi, F. André, N. Paragios, and M. Vakalopoulou, “Self-supervised nuclei segmentation in histopathological images using attention,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 393–402.
  * [15] X. Xie, J. Chen, Y. Li, L. Shen, K. Ma, and Y. Zheng, “Instance-aware self-supervised learning for nuclei segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 341–350.
  * [16] W. Yan, Y. Wang, S. Gu, L. Huang, F. Yan, L. Xia, and Q. Tao, “The domain shift problem of medical image segmentation and vendor-adaptation by unet-gan,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 623–631.
  * [17] S. Lal, D. Das, K. Alabhya, A. Kanfade, A. Kumar, and J. Kini, “Nucleisegnet: Robust deep learning architecture for the nuclei segmentation of liver cancer histopathology images,” _Computers in Biology and Medicine_ , vol. 128, p. 104075, 2021.
  * [18] A. E. Carpenter, T. R. Jones, M. R. Lamprecht, C. Clarke, I. H. Kang, O. Friman, D. A. Guertin, J. H. Chang, R. A. Lindquist, J. Moffat _et al._ , “Cellprofiler: image analysis software for identifying and quantifying cell phenotypes,” _Genome biology_ , vol. 7, no. 10, pp. 1–11, 2006.
  * [19] J. Schindelin, I. Arganda-Carreras, E. Frise, V. Kaynig, M. Longair, T. Pietzsch, S. Preibisch, C. Rueden, S. Saalfeld, B. Schmid _et al._ , “Fiji: an open-source platform for biological-image analysis,” _Nature methods_ , vol. 9, no. 7, pp. 676–682, 2012.
  * [20] H. Oda, H. R. Roth, K. Chiba, J. Sokolić, T. Kitasaka, M. Oda, A. Hinoki, H. Uchida, J. A. Schnabel, and K. Mori, “Besnet: boundary-enhanced segmentation of cells in histopathological images,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 228–236.