# Meta Ordinal Regression Forest for Medical Image Classification with Ordinal
Labels

Yiming Lei, Haiping Zhu, Junping Zhang, Hongming Shan Y. Lei, H. Zhu, and J.
Zhang are with the Shanghai Key Laboratory of Intelligent Information
Processing, School of Computer Science, Fudan University, Shanghai 200433,
China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>jpzhang@fudan.edu.cn)H. Shan is with the Institute of Science and Technology
for Brain-inspired Intelligence and MOE Frontiers Center for Brain Science,
Fudan University, Shanghai 200433, China, and also with the Shanghai Center
for Brain Science and Brain-inspired Technology, Shanghai 200031, China
(e-mail: hmshan@fudan.edu.cn)_Corresponding author_ : Hongming Shan.

###### Abstract

The performance of medical image classification has been enhanced by deep
convolutional neural networks (CNNs), which are typically trained with cross-
entropy (CE) loss. However, when the label presents an intrinsic ordinal
property in nature, _e.g._ , the development from benign to malignant tumor,
CE loss cannot take into account such ordinal information to allow for better
generalization. To improve model generalization with ordinal information, we
propose a novel meta ordinal regression forest (MORF) method for medical image
classification with ordinal labels, which learns the ordinal relationship
through the combination of convolutional neural network and differential
forest in a meta-learning framework. The merits of the proposed MORF come from
the following two components: a tree-wise weighting net (TWW-Net) and a
grouped feature selection (GFS) module. First, the TWW-Net assigns each tree
in the forest with a specific weight that is mapped from the classification
loss of the corresponding tree. Hence, all the trees possess varying weights,
which is helpful for alleviating the tree-wise prediction variance. Second,
the GFS module enables a dynamic forest rather than a fixed one that was
previously used, allowing for random feature perturbation. During training, we
alternatively optimize the parameters of the CNN backbone and TWW-Net in the
meta-learning framework through calculating the Hessian matrix. Experimental
results on two medical image classification datasets with ordinal labels,
_i.e._ , LIDC-IDRI and Breast Ultrasound Dataset, demonstrate the superior
performances of our MORF method over existing state-of-the-art methods.

###### Index Terms:

Convolutional neural network, ordinal regression, medical image
classification, meta-learning, random forest.

##  I Introduction

Medical image classification has been assisted by the deep learning technique
[1, 2, 3, 4, 5, 6, 7, 8] and has achieved tremendous progress in the past
decade. Early detection and treatment of some diseases, such as cancers, are
critical for reducing mortality. Fortunately, it is implicit in medical images
that the image information across the different clinical stages exhibits an
ordinal relationship, which can be used to improve model generalization. For
example, computed tomographic (CT) images of lung nodule [9] are given with
the malignancy scores from 1 to 5, where 1 means highly unlikely to be
malignant, 3 is indeterminate, and 5 is highly likely to be malignant. The
majority of existing lung nodule classification methods conduct binary
classification while discarding indeterminate or unsure nodules [3, 8, 10, 11,
12, 13, 14]. In other words, the unsure nodules that are between benign and
malignant and cannot be classified by radiologists based on current scans
become useless [15]. As shown in Fig. 1, a large number of nodules are
indeterminate and then discarded in the binary classification problem. It is
evident that the images with ordinal labels represent the development of the
lesions, as do other diseases such as breast cancer. Considering that deep
learning methods are data-hungry and that these medical images differ from
natural images in fewer discriminative patterns, _leveraging the ordinal
relationship among limited medical data for training deep learning models_ is
becoming an important topic.

Figure 1: The histogram of averaged malignant scores of nodules in LIDC-IDRI
dataset. The red dashed lines split the nodules into three groups; _i.e._ ,
benign ($\leq 2.5$), unsure ($2.5\sim 3.5$), and malignant ($\geq 3.5$).

Generally, most medical image classification methods work by feeding medical
images into convolutional neural networks (CNNs) and updating the parameters
of the CNNs based on cross-entropy (CE) loss. However, CE loss is inferior for
fitting the ordinal distribution of labels. Therefore, ordinal regression-
based methods have been explored for medical image classification with ordinal
labels [15, 16, 17]. A simple solution is to construct a series of binary
classification problems and evaluate the cumulative probabilities of all
binary classifiers [17, 16]. However, those binary classifiers are trained
separately, which ignores the ordinal relationship. Recently, the unsure data
model (UDM) [15], neural-stick breaking (NSB) [18], unimodal method [19], and
soft ordinal label (SORD) [20] have been proposed to improve ordinal
regression performance based on rectified label space or the probability
calculation of the output of the CNNs.

Another method of ordinal regression without changing target distributions is
based on the combination of random forest and CNNs, which have been evaluated
to successfully estimate human age using facial images [4, 5, 21]. These
models regard the largest probability among all the dimensions of the learned
tree-wise distributions as the final prediction. To incorporate the global
ordinal relationship with forest-based methods, Zhu  _et al._ proposed the
convolutional ordinal regression forest (CORF) to allow the forest to predict
the ordinal distributions [22]. However, these forest-based methods suffer
from the following two drawbacks: 1) the compositions of all trees depend on
the random selection of split nodes from the feature vector of the fully-
connected (FC) layer, and the structure of the constructed forest is fixed at
the very beginning of training, leading to poor generalization due to the lack
of the random perturbation of features, as suggested in [23]; and 2) there
exists the tree-wise prediction variance (tree-variance) because the final
prediction of the forest is the average of the results obtained by all trees,
_i.e._ , all trees share the same weights and contribute equally to the final
prediction.

To address the aforementioned problems, we propose a meta ordinal regression
forest (MORF) for medical image classification with ordinal labels. Fig. 2
shows the overall framework of MORF including three parts: a CNN backbone
parameterized by $\bm{\theta}$ to extract feature representation from the
input medical image, a tree-wise weighting network (TWW-Net) parameterized by
$\bm{\phi}$ to learn the tree-wise weights for reduced tree-wise prediction
variance, and a grouped feature selection (GFS) module to construct the
dynamic forest that is equipped with random feature perturbation. We adopt the
meta-learning framework to optimize the parameters $\bm{\theta}$ and
$\bm{\phi}$ alternatively [24, 25]. The way we use meta-learning is similar to
those in [24, 25], which calculates the second derivatives, _i.e._ Hessian
matrix, via meta data or meta tasks. The main difference is that our meta data
are selected features via the GFS module rather than the original images; the
meta data in [24] are a subset of the validation set that contains equal
numbers of images of all classes, which may be infeasible in medical cases
because some classes have fewer samples. With meta-learning optimization, CNN
parameters can be updated with the guide of the dynamic forest, which can
achieve better generalizability.

The main contributions of this paper are summarized as follows.

  * •

We propose a meta ordinal regression forest (MORF) for medical image
classification with ordinal labels, which enables the forest on top of the
CNNs to maintain the random perturbation of features. The MORF comprises a CNN
backbone, a TWW-Net, and a GFS module.

  * •

TWW-Net assigns each tree in the forest with a specific weight and alleviates
the tree-wise variance that exists in the previous deep forest-related
methods. Furthermore, we provide a theoretical analysis of the weighting
scheme of TWW-Net and demonstrate how the meta data can guide the learning of
the backbone network.

  * •

The GFS module only works in the meta training stage for generating the
dynamic forest that is incorporated the feature random perturbation. Combined
with the TWW-Net, the final trained model can be further enhanced through the
randomness of the dynamic forest.

  * •

The experimental results of two medical image classification datasets with
ordinal labels demonstrate the superior performance of MORF over existing
methods including classification and ordinal regression. Furthermore, we also
verify that our MORF can enhance the benign-malignant classification when
leveraging the unsure data into the training set on the LIDC-IDRI dataset.

Figure 2: The proposed MORF framework. Left: the deep ordinal regression
forest with random construction of forest, which is followed by the TWW-Net.
Right: the deep ordinal regression forest followed by the TWW-Net, where the
forest is constructed from the GFS. During the meta train stage, the GFS
features are used to guide the update of $\bm{\theta}$. The MORF framework
involves three parts of parameters: $\bm{\theta}$ (CNN), $\bm{\pi}$ (leaf
nodes, _i.e._ , the ordinal distributions), and $\bm{\phi}$ (TWW-Net). Note
that the parameter $\bm{\widehat{\theta}}$ on the right side is the first-
order derivative calculated via (8). And $\bm{\widehat{\theta}}$ is used to
obtain the TWW-Net parameter $\bm{\phi}^{(u+1)}$ through (9).

We note that this work extends our previous conference paper [26] with the
following major improvements:

  1. 1.

We further explore the relationship between our re-constructed dynamic forest
and the random forest, which unveils that our MORF has stronger
generalizability than the previous deep forest methods due to its retained
random perturbation during training.

  2. 2.

We improve GFS module by using random selection without replacement, where the
resulting dimension of FC output should be equal to the number of split nodes
in a forest. This improvement can avoid selecting unused elements during meta-
training stage and achieve further performance gain.

  3. 3.

We provide the detailed training algorithm and the theoretical analysis on the
meta weighting scheme of TWW-Net.

  4. 4.

For the LIDC-IDRI dataset, we demonstrate that the binary classification on
the benign and malignant classes can be improved by MORF. Most importantly,
when we added the unsure data into the training set, the binary classification
results on the test set are improved further.

  5. 5.

We conduct extra experiments on a new BUSI dataset [27] to evaluate the
performance of the methods.

##  II Related Work

In this section, we review the related work on the following three aspects: 1)
CNN-based methods for medical image classification with emphasis on lung
nodule classification and breast cancer diagnosis, 2) ordinal regression
methods, and 3) meta-learning methods.

###  II-A CNN-based Medical Image Classification

Medical image classification, such as lung nodule and breast tumor
classification, has benefited from advanced CNN architectures and learning
strategies. In this paper, we focus on medical image classification with
ordinal labels.

Lung nodule classification. Liu  _et al._ combined statistical features and
artificial neural networks to detect lung nodules in full-size CT images [28].
Shen  _et al._ applied a single column network to classify lung nodule images
with different sizes [12]. Dou  _et al._ [13] explored an ensemble of
subnetworks, each of which has a specific convolutional kernel size. Cao  _et
al._ trained two 3-D networks on original data and augmented data, and
combined them for lung nodule detection [29]. However, 3-D networks are
difficult to train with limited medical data [30, 31]. Another kind of method
combines low- and high-level features that come from U-Net-like network
architectures [8, 32, 33, 34, 35, 36], which can avoid information loss
through the concatenation of different features.

Breast cancer diagnosis. Breast cancer diagnosis has also been enhanced by
deep learning [2, 18, 37, 38, 39, 40, 41, 42, 43, 44, 45]. In [41, 46], the
authors conducted the CNN-based image fusion, feature fusion, and classifier
fusion methods to classify breast tumors in ultrasound (US) images. Wu  _et
al._ explored the binary classification (benign and malignant) of breast
cancer based on the proposed view-wise, image-wise, breast-wise, and joint
learning approaches [43]. Ayelet  _et al._ combined CNN learned features with
the electronic health records to improve the early detection of breast cancer
using mammograms [44]. Dhungel  _et al._ applied a pretrained CNN model in the
classification of breast cancer and verified the effectiveness of deep
features against traditionally hand-crafted features [45]. Hagos  _et al._
incorporated symmetry patches with the symmetric locations of positive tumor
patches to help improve breast cancer classification performance [46].

All the methods we reviewed above did not take into account the progression of
the diseases that implies the intrinsic ordinal relationship among the
classes. In this paper, we use widely-used backbone networks such as VGG [47]
and ResNet [1] to verify the effectiveness of our method in exploring ordinal
relationship.

###  II-B Ordinal Regression

Ordinal regression is a classical problem that predicts ordinal labels [16,
17, 48], such as facial age [22], aesthetic image classification [49], and
medical image malignancy ranking [15, 18, 19]. Beckham  _et al._ enforced each
element of the FC output to obey unimodal distributions such as Poisson and
Binomial [19]. The unimodal method surpassed the normal cross-entropy
baseline. Neural stick-breaking (NSB) was proposed in [18], whose output is a
$(N-1)$-dimensional vector representing $N-1$ boundaries where $N$ is the
number of classes. NSB guaranteed that the cumulative probabilities will
monotonically decrease. The unsure data model (UDM) is a strategy-driven
method and focuses more on that the fact that normal samples and disease
samples should be classified with high precision and high recall, respectively
[15]. The UDM incorporates some additional parameters associated with
techniques like ordinal regression, class imbalance, and their proposed
strategies. Although the UDM outperforms the unimodal method and NSB, it
requires more effort to tune the model to obtain the optimal additional
parameters. Soft ordinal label (SORD) converted the ground truth labels of all
classes in a soft manner by penalizing the distance between the true rank and
the label value [20].

The methods discussed above contain a DNN backbone followed by a modified
classifier (FC layer) except for the SORD method. This pipeline cannot avoid
the redundant use of the FC feature and may lead to overfitting. Recently
appeared deep random forest-based methods targeted this problem. The deep
neural decision forest (DNDF) defines a differentiable partition function for
each split node [5]. Hence, the forest can be updated jointly with the deep
networks through backpropagation. The label distribution learning forest
(LDLF) extended DNDF to output a predicted distribution [21]. However, the
DNDF and LDLF have difficulty guaranteeing the global ordinal relationship of
the predictions of leaf nodes. Convolutional ordinal regression forest (CORF)
was proposed to incorporate the constraint of ordinal relation in the loss
function, which enabled the output of the forest to be globally ordinal
(monotonically decrease) [22].

In this paper, our MORF further improves the efficiency of the FC feature and
incorporates random perturbation of features for the forest. Moreover, our
MORF enables different trees to have specific weights through the guidance of
the meta data.

###  II-C Meta Learning

Meta-learning is tailored for learning the meta knowledge from the predefined
meta task set or meta dataset. It is widely used in few-shot learning. Model-
agnostic meta-learning (MAML) [50] learns the parameter initialization from
the few-shot tasks, and the new tasks only take a few training steps while
achieving better generalization than fine-tuning. Jamal  _et al._ proposed
extending the MAML to avoid overfitting on existing training tasks by
proposing a maximum-entropy prior that introduces some inequality measures in
the loss function [51]. Liu  _et al._ enhances the generalizability of the
main task from the predefined auxiliary task using meta-learning[25]. Meta-
weight-net is a novel weighting network to address class imbalance and noisy
label problems [24]. In summary, these meta-learning methods involve
optimizing two groups of parameters jointly. The meta training algorithm of
our MORF is similar to that of [24], but differs in its construction of the
meta data and weighting behavior over the decision trees. The construction of
the meta data in [24] is to select a subset of the validation set with an
equal number of samples for each class for meta training, whereas the meta
data in MORF is feature level.

##  III Methods

In this section, we first formulate the problem of ordinal regression forest
for medical images. Then we introduce the meta ordinal regression forest
(MORF) framework in order of the training objective, TWW-Net, and GFS module.
Finally, we present the meta-learning optimization algorithm and the
corresponding theoretical analysis.

###  III-A Problem Formulation

Ordinal regression solves the problem that the data belonging to different
classes have an ordered label, which implies that an intrinsic ordinal
relationship exists among the data. It learns a mapping function
$h:\mathcal{X}\to\mathcal{Y}$, where $\mathcal{X}$ represents the input space
and $\mathcal{Y}$ is the ordinal label space. Here,
$\mathcal{Y}=\\{y_{1},y_{2},\ldots,y_{C}\\}$ has the ordinal relationship
$y_{1}\preceq y_{2}\preceq\ldots\preceq y_{C}$, where $C$ is the number of
classes. In this study, $y_{c}\in\mathcal{Y}$ denotes the stage of the
progression of diseases; taking the lung nodule classification [9] as an
example, $C$ equals $3$ and $y_{1}$, $y_{2}$, and $y_{3}$ represent benign,
unsure and malignant, respectively.

To solve this kind of ordinal classification problem, the given label
$y\in\mathcal{Y}$ can be converted to an ordinal distribution label, _i.e._ ,
a one-dimensional vector
$\bm{d}=(d^{1},d^{2},\ldots,d^{C-1})^{\textrm{T}}\in\mathcal{D}$ [22], where
$d^{c}=1$ if $y>y_{c}$, otherwise $d^{c}=0$. Practically, we will obtain an
accurate $\bm{d}$ for a given image, and the $\bm{d}$ should maintain a
monotonically decreasing property across all the elements. Therefore, we
impose a constraint $d^{1}\geq d^{2}\geq\ldots\geq d^{C-1}$ on $\bm{d}$ during
training [22]. Under the framework of the ordinal regression forest (ORF) [4,
21, 22], the ordinal label $\bm{d}_{c}$ is given by the leaf nodes in the ORF,
and the probability of the given sample $\bm{x}$ falling into the $l$-th leaf
node is defined as:

$p(l|\bm{x};\bm{\theta})=\prod_{n\in\mathcal{N}}s_{n}(\bm{x};\bm{\theta})^{\bm{1}[l\in\mathcal{L}_{n}^{l}]}\left(1-s_{n}(\bm{x};\bm{\theta})^{\bm{1}[l\in\mathcal{L}_{n}^{r}]}\right),$
(1)

where $\mathcal{L}_{n}^{l}$ and $\mathcal{L}_{n}^{r}$ represent the subsets of
leaf nodes held by left and right subtrees of the split node $n$,
respectively, and $\mathcal{N}$ is the number of split nodes in one tree.
$s_{n}$ is the split function that determines which node (left or right) a
sample should be assigned to, _i.e._ , $s_{n}:\mathcal{X}\to\\{0,1\\}$.
Following [4, 5, 21, 22], we formulate $s_{n}$ as a probabilistic function:
$s_{n}(\bm{x};\bm{\theta})=\sigma(f_{\eta(n)}(\bm{x};\bm{\theta}))$, where
$\sigma(\cdot)$ is the sigmoid function, $f(\cdot;\bm{\theta})$ is the
backbone network (_e.g._ , a CNN with a FC layer on the top) and the output of
$f(\cdot;\bm{\theta})$ is a one-dimensional vector; $\eta(n)$ is an index
function that is used to assign elements selected from this one-dimensional
vector to all split nodes in the forest. Actually, $\eta(n)$ is implemented as
randomly selecting an $\eta(n)$-th element for each $n$-th split node.

When we obtain the probability $p(l|\bm{x};\bm{\theta})$, the output of one
tree can be defined as a mapping $\bm{g}:\mathcal{X}\to\mathcal{D}$:

$\bm{g}(\bm{x};\bm{\theta},\bm{\mathcal{T}})=\sum_{l\in\mathcal{L}}p(l|\bm{x};\bm{\theta})\bm{\pi}_{l},$
(2)

where $\bm{\mathcal{T}}$ denotes one decision tree, and $\mathcal{L}$ a set of
leaf nodes. $\bm{\pi}_{l}$ holds the ordinal distribution of the $l$-th leaf
node; _i.e._ ,
$\bm{\pi}_{l}=(\pi_{l}^{1},\pi_{l}^{2},\ldots,\pi_{l}^{C-1})^{\textrm{T}}$. In
this paper, the parameter $\bm{\pi}$ can be updated jointly with that of the
backbone network through back-propagation, which has been illustrated in [4,
21, 22]. Then, the final prediction of the forest is the average of outputs of
all trees:

$P(\bm{x})=\frac{1}{T}\sum_{t=1}^{T}\bm{g}(\bm{x};\bm{\theta},\bm{\mathcal{T}}_{t}),$
(3)

where $T$ is the total number of trees. Here, all the trees contribute equally
to the final prediction as well as that in previous deep forest methods such
as the CORF [22], whereas in our MORF model, we assign each tree a specific
learned weight.

Figure 3: The proposed grouped feature selection (GFS) assigning each tree
with features of different groups. Note that the number of groups (colored
boxes) in GFS equals to the number of split nodes in one decision tree.

###  III-B Meta Ordinal Regression Forests

The total framework of the MORF contains a CNN with an FC layer as backbone
network parameterized by $\bm{\theta}$, a TWW-Net parameterized by
$\bm{\phi}$, and the leaf nodes parameterized by $\bm{\pi}$. Note that the
$\bm{\pi}$ is updated according to [22].

####  III-B1 Objective function

As mentioned above, all trees in ORF and CORF are assigned the same weights,
which can increase the inevitable tree-wise prediction variance. To cope with
this drawback, we propose multiplying the tree-wise losses with specific
weights. Therefore, the gradients of $\bm{\theta}$, backpropagated from the
losses of all the trees, can be affected by the weights $\omega_{t}$.
Therefore, our training objective function is defined as:

$\displaystyle\bm{\theta}^{*}=\operatorname*{arg\,min}_{\bm{\theta}}L^{\mathrm{tr}}(\bm{\theta})=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{T}\sum_{t=1}^{T}\omega_{t}^{i}R_{t}^{i}(\bm{\theta}),$
(4)

where $N$ is the number of training images, $R_{t}^{i}$ denotes the
classification loss generated by (2), and $\omega_{t}$ represents the specific
weight for the $t$-th tree that is learned by the TWW-Net, which will be
subsequently introduced. Different from [24], equation (4) imposes the weights
on the different trees w.r.t. one training sample $i$, rather than on the
different samples.

####  III-B2 Tree-wise weighting network

Here, we introduce the TWW-Net that is used for learning the weights
$\omega_{t}$ in (4). Similar to the meta-weight-net [24], TWW-Net is
practically implemented as a group of multilayer perceptrons (MLPs), $V_{t}$,
and each MLP acts as a weight learning function for a specific tree because of
the universal approximation property of MLPs [52]. In Fig. 2, we can see that
the $t$-th tree generates a classification loss $R_{t}$, then $R_{t}$ will be
fed into the corresponding weighting net $V_{t}$, and finally $V_{t}$ outputs
the weight $w_{t}$ for the $t$-th tree. This process can be formulated as:

$\displaystyle w_{t}=V_{t}(R_{t}(\bm{\theta});\bm{\phi}_{t}).$ (5)

Therefore, TWW-Net is composed of $T$ weighting net $V_{t}$, where $T$ is the
number of trees. Here we use
$\bm{\phi}=\\{\bm{\phi}_{1},\bm{\phi}_{2},\ldots,\bm{\phi}_{T}\\}$ to
represent the set of parameters of TWW-Net, and they are updated together
which will be described in Section III-C. Through (5), a TWW-Net can assign
different weights to different trees.

Combined with (4), the training objective function can be modified as follows:

$\displaystyle\bm{\theta}^{*}(\bm{\phi})$
$\displaystyle=\operatorname*{arg\,min}_{\bm{\theta}}L^{\mathrm{tr}}(\bm{\theta};\bm{\phi})$
(6)
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{T}\sum_{t=1}^{T}V_{t}^{i}\left[R_{t}^{i}(\bm{\theta},\bm{\pi};S^{\mathrm{tr}});\bm{\phi}_{t}\right]\cdot
R_{t}^{i}(\bm{\theta},\bm{\pi};S^{\mathrm{tr}}),$

where $S^{\mathrm{tr}}$ denotes training set.

####  III-B3 Grouped feature selection (GFS)

Although we have incorporated the tree-wise weights in the training objective
function, the structure of the forest is still fixed. Therefore, we introduce
the GFS module to construct the dynamic forest with random feature
perturbation in this section. Then, we explore the relationship between the
GFS and random forest.

As shown in Fig. 3, the GFS first ranks all the activation values of the final
FC feature vector. Then it splits the ranked elements into $\mathcal{N}$
groups (denoted by different colors), where $\mathcal{N}$ equals the number of
split nodes in one decision tree. Both the elements inside and outside of one
group are in descending order, and each tree randomly selects its own nodes
across all the groups. Hence, one tree contains the features globally across
the FC feature, and it retains the local random perturbation of the feature
that is critical for the random forest. After repeating these procedures on
all the trees, the dynamic forest is constructed. Note that the final trained
model of MORF is also equipped with a forest that is fixed from the beginning
of training, and the GFS module only works in the training stage and has no
impact on the forest at the time of inference. Note that, the number of groups
corresponds to the number of split nodes in one tree, _i.e._ , the number of
groups increases along with an increase of tree depth.

Relationship between GFS and Random Forest. Random forest (RF) is a classical
ensemble learning method, which benefits from base learners that have feature
and data random perturbations. Specifically, feature perturbation means that
each node in the decision trees is the most discriminative attribute in a
subset of its whole attribute set [23]. Data perturbation is satisfied for all
the deep forest-based methods that trained over shuffled mini-batches,
however, feature perturbation occurs because of the fixed forest structure [4,
21, 22]. In Fig. 3 (b), we can see that each split node is randomly selected
from its own feature set. Although, in Fig. 3 (a), all the nodes in the forest
are also obtained through random selection within their subsets (indicated by
different colors) of the FC feature, it differs from the RF in that the nodes
in different trees share the same subsets, _i.e._ , the GFS-based forest also
maintains the node-wise feature random perturbation. Therefore, the MORF with
GFS possesses the merit of randomness with respect to all the split nodes, and
this advantage does not exist for previous methods [4, 5, 21, 22].

###  III-C Optimization via Meta-learning

When we obtain a dynamic forest, we expect it to guide the update of the CNN
with a fixed forest. Moreover, from (6) we observe that the objective function
involves two parts of parameters, $\bm{\theta}$ and $\bm{\phi}$, and
$\bm{\theta}$ is a function of $\bm{\phi}$, so we customize a meta-learning
framework enabling the meta data to guide the learning of the target model.
Here, the GFS selected features are regarded as the meta data, which is
different from those in [24].

To obtain the optimal $\bm{\theta}^{*}$, we need to obtain the optimal
$\bm{\phi}^{*}$. Therefore, we optimize $\bm{\phi}$ by minimizing the
following objective function:

$\displaystyle\bm{\phi}^{*}$
$\displaystyle=\operatorname*{arg\,min}_{\bm{\phi}}L^{\mathrm{meta}}(\bm{\theta}^{*}(\bm{\phi}))$
(7)
$\displaystyle=\frac{1}{M}\sum_{j=1}^{M}\frac{1}{T}\sum_{t=1}^{T}R_{t}^{j}(\bm{\theta}^{*}(\bm{\phi}_{t}),\bm{\pi};S^{\mathrm{meta}}),$

where $M$ is the number of meta data. This objective function indicates that
$\bm{\phi}$ is updated based on the optimal backbone parameter
$\bm{\theta}^{*}$.

First, we take the derivative of (6) with respect to $\bm{\theta}$:

$\displaystyle\bm{\widehat{\theta}}^{(u)}$ $\displaystyle=\bm{\theta}^{(u)}-$
(8) $\displaystyle\alpha\frac{1}{N\cdot
T}\sum_{i=1}^{N}\sum_{t=1}^{T}V_{t}^{i}\left[R_{t}^{i}(\bm{\theta}^{(u)});\bm{\phi}_{t}\right]\cdot\nabla_{\bm{\theta}}R_{t}^{i}(\bm{\theta})\Big{\rvert}_{\bm{\theta}^{(u)}},$

where $\alpha$ is the learning rate for $\bm{\theta}$. For simplicity, we omit
the parameters $\bm{\pi}$ and the datasets $S^{\mathrm{tr}}$ and
$S^{\mathrm{meta}}$ in the above equations. The superscript $(u)$ denotes the
$u$-th iteration. Therefore, $\widehat{\bm{\theta}}$ in (8) represents the
weights obtained through the first order derivative
$\nabla_{\bm{\theta}}R_{t}^{i}(\bm{\theta})$. Then we can use
$\widehat{\bm{\theta}}$ to update the parameters $\bm{\phi}$:

$\displaystyle\bm{\phi}^{(u+1)}=\bm{\phi}^{(u)}-\beta\frac{1}{M}\sum_{j=1}^{M}\nabla_{\bm{\phi}}\left[\frac{1}{T}\sum_{t=1}^{T}R_{t}^{j}(\widehat{\bm{\theta}}^{(u)})\right]\Big{|}_{\bm{\phi}^{(u)}}.$
(9)

To go a step further, we derive (9) and obtain the following equation:

$\displaystyle\bm{\phi}^{(u+1)}$ $\displaystyle=\bm{\phi}^{(u)}+$ (10)
$\displaystyle\frac{\alpha\beta}{N}\sum_{i=1}^{N}\left(\frac{1}{M}\sum_{j=1}^{M}G_{ij}\right)\frac{\sum_{t=1}^{T}\partial{V_{t}^{i}(R_{t}^{i}(\bm{\theta}^{(u)};\bm{\phi}_{t}))}}{\partial{\bm{\phi}}}\Big{|}_{\bm{\phi}^{(u)}},$

where

$\displaystyle
G_{ij}=\frac{1}{T}\sum_{t=1}^{T}\frac{\partial{R_{t}^{j}(\widehat{\bm{\theta}},\bm{\pi};S^{\mathrm{meta}})}}{\partial{\widehat{\bm{\theta}}}}|_{\widehat{\bm{\theta}}^{(u)}}\cdot\frac{1}{T}\sum_{t=1}^{T}\frac{\partial{R_{t}^{i}(\bm{\theta};S^{\mathrm{tr}})}}{\partial{\bm{\theta}}}|_{\bm{\theta}^{(u)}}$

stands for the similarity between two gradients—the gradient of the $i$-th
training data computed on training loss $R_{t}^{i}$ and the gradient of the
mean value of the mini-batch meta data calculated on meta loss $R_{t}^{j}$.
This enforces the gradient of the feature of training data to approach that of
meta data generated from GFS. Hence, the behavior of each tree is guided by
the meta gradient and is consistent with other trees. Consequently, the
predictions of different trees in our MORF are consistent, _i.e._ , have lower
variance, which guarantees a more stable prediction.

After we obtain the updated TWW-Net parameters $\bm{\phi}^{(u+1)}$, the update
rule of $\bm{\theta}$ can be defined as:

$\displaystyle\bm{\theta}^{(u+1)}$ $\displaystyle=\bm{\theta}^{(u)}-$ (11)
$\displaystyle\alpha\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}V_{t}^{i}\left[R_{t}^{i}(\bm{\theta}^{(u)});\bm{\phi}^{(u+1)}_{t}\right]\nabla_{\bm{\theta}}R_{t}^{i}(\bm{\theta}^{(u)})\Big{|}_{\bm{\theta}^{(u)}}.$

Input: Training data $S^{\mathrm{tr}}$, max iteration $U$.

Output: Ordinal regressor parameter $\bm{\theta}^{(U)}$.

1 Initialize $\bm{\theta}^{(0)}$ and $\bm{\phi}^{(0)}$.

2 for  _$u=0$ to $U-1$_ do

3 Sample a mini-batch of ($\bm{x}$, $\bm{d}$) from $S^{\mathrm{tr}}$.

4 _// Meta train (update $\bm{\phi}$):_

5 Forward: input ($\bm{x}$, $\bm{d}$) with
$\bm{\theta}^{(u)}(\bm{\phi}^{(u)})$.

6 Randomly construct the forest $F^{R}$ and compute tree-wise loss
$R_{t}(\bm{\theta}^{(u)}(\bm{\phi}^{(u)});S^{\mathrm{tr}})$.

7 Compute the first order derivative
$\widehat{\bm{\theta}}^{(u)}(\bm{\phi}^{(u)})$ by (8).

8 Forward: input ($\bm{x}$, $\bm{d}$) with
$\widehat{\bm{\theta}}^{(u)}(\bm{\phi}^{(u)})$.

9 Construct the forest via GFS and compute tree-wise loss
$R_{t}(\widehat{\bm{\theta}}^{(u)}(\bm{\phi}^{(u)});S^{\mathrm{meta}})$.

10 Update $\bm{\phi}^{(u+1)}$ by (9).

11 _// Model train (update $\bm{\theta}$):_

12 Forward: input ($\bm{x}$, $\bm{d}$) with
$\bm{\theta}^{(u)}(\bm{\phi}^{(u+1)})$.

13 Compute tree-wise loss
$R_{t}(\bm{\theta}^{(u)}(\bm{\phi}^{(u+1)});S^{\mathrm{tr}})$ based on the
constructed $F^{R}$ in Step 6.

14 Update $\bm{\theta}^{(u+1)}$ by (11).

15

16 end for

Algorithm 1 Training Algorithm of MORF.

In (9), the tree-wise loss
$R_{t}(\widehat{\bm{\theta}}^{(u)},\bm{\pi};S^{\mathrm{meta}})$ is calculated
via meta data, and the meta data are feature level. During the training
procedure, we first obtain the first-order derivative
$\widehat{\bm{\theta}}(\bm{\phi})$ in (8) via taking the image $\bm{x}$ as
input, and the forest is constructed based on this forward process as shown on
the left side of Fig. 2. Then we fix $\widehat{\bm{\theta}}(\bm{\phi})$ in (8)
and take the image $\bm{x}$ as input, and the forest here is reconstructed
through our GFS module as shown on the right side of Fig. 2. Once again, the
globally and locally selected features via GFS are the meta data in our
method. This retains the structural variability and the random feature
perturbation of the forest. Simultaneously, our training scheme can guide the
behavior of learning from training data to approach that of learning from the
GFS generated meta data. The details of the training procedure are shown in
Algorithm 1.

Note that the TWW-Net ($\bm{\phi}$) only works in the meta training stage, and
it does not affect the prediction in inference. In other words, the trained
model contains parameters $\bm{\theta}$ and $\bm{\pi}$, which are required
during inference.

TABLE I: Classification results on test set of Train(3)-Test(3) on LIDC-IDRI dataset. The values with underlines indicate the best results while less important in the clinical diagnosis [15]. MORFc is the conference version of the proposed MORF [26]. Backbone | Method | Accuracy | Benign | Malignant | Unsure
---|---|---|---|---|---
Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1
ResNet-18 | CE Loss |  0.542$\pm 0.006$ | 0.544 | 0.722 | 0.620 | 0.586 | 0.644 | 0.613 | 0.496 | 0.290 | 0.366
Poisson [19] |  0.527$\pm 0.007$ | 0.536 | 0.605 | 0.568 | 0.590 | 0.584 | 0.587 | 0.477 | 0.410 | 0.441
NSB [18] |  0.534$\pm 0.007$ | 0.517 | 0.807 | 0.630 | 0.607 | 0.673 | 0.638 | 0.500 | 0.160 | 0.242
UDM [15] |  0.546$\pm 0.004$ | 0.553 | 0.767 | 0.643 | 0.581 | 0.495 | 0.535 | 0.504 | 0.325 | 0.395
CORF [22] |  0.568$\pm 0.004$ | 0.569 | 0.713 | 0.633 | 0.632 | 0.613 | 0.623 | 0.523 | 0.385 | 0.443
MORFc [26] |  0.573$\pm 0.004$ | 0.695 | 0.623 | 0.657 | 0.627 | 0.683 | 0.654 | 0.479 | 0.685 | 0.564
| MORF (ours) | 0.634$\pm 0.003$ | 0.706 | 0.691 | 0.698 | 0.783 | 0.752 | 0.768 | 0.412 | 0.441 | 0.426
ResNet-34 | CE Loss |  0.534$\pm 0.007$ | 0.540 | 0.722 | 0.618 | 0.625 | 0.644 | 0.634 | 0.443 | 0.270 | 0.335
Poisson [19] |  0.540$\pm 0.006$ | 0.577 | 0.502 | 0.537 | 0.714 | 0.545 | 0.618 | 0.458 | 0.520 | 0.512
NSB [18] |  0.544$\pm 0.006$ | 0.580 | 0.619 | 0.599 | 0.608 | 0.614 | 0.611 | 0.462 | 0.425 | 0.443
UDM [15] |  0.536$\pm 0.004$ | 0.537 | 0.659 | 0.592 | 0.578 | 0.554 | 0.566 | 0.510 | 0.390 | 0.442
CORF [22] |  0.542$\pm 0.005$ | 0.581 | 0.605 | 0.593 | 0.616 | 0.524 | 0.566 | 0.466 | 0.480 | 0.473
MORFc [26] |  0.586$\pm 0.004$ | 0.619 | 0.619 | 0.619 | 0.504 | 0.700 | 0.565 | 0.509 | 0.560 | 0.533
MORF (ours) | 0.639$\pm 0.003$ | 0.751 | 0.650 | 0.697 | 0.773 | 0.742 | 0.757 | 0.422 | 0.535 | 0.472
VGG-16 | CE Loss |  0.517$\pm 0.008$ | 0.538 | 0.668 | 0.596 | 0.562 | 0.495 | 0.526 | 0.456 | 0.360 | 0.402
Poisson [19] |  0.542$\pm 0.005$ | 0.548 | 0.794 | 0.648 | 0.568 | 0.624 | 0.594 | 0.489 | 0.220 | 0.303
NSB [18] |  0.553$\pm 0.005$ | 0.565 | 0.641 | 0.601 | 0.566 | 0.594 | 0.580 | 0.527 | 0.435 | 0.476
UDM [15] |  0.548$\pm 0.002$ | 0.541 | 0.767 | 0.635 | 0.712 | 0.515 | 0.598 | 0.474 | 0.320 | 0.382
CORF [22] |  0.559$\pm 0.006$ | 0.590 | 0.627 | 0.608 | 0.704 | 0.495 | 0.581 | 0.476 | 0.515 | 0.495
MORFc [26] |  0.580$\pm 0.004$ | 0.660 | 0.479 | 0.556 | 0.719 | 0.596 | 0.652 | 0.492 | 0.680 | 0.571
MORF (ours) | 0.641$\pm 0.002$ | 0.754 | 0.632 | 0.688 | 0.798 | 0.743 | 0.769 | 0.429 | 0.574 | 0.492

##  IV Experimental Setup and Results

In this section, we present the experimental setup and evaluate the proposed
MORF method on the LIDC-IDRI [9] and BUSI [27] datasets. The experiments
reported the average results through five randomly independent split folds,
and for the accuracies, we reported the average values with standard
deviations.

Figure 4: Some examples in the BUSI [27] and LIDC-IDRI [9] datasets. For the
LIDC-IDRI dataset, we provide four examples each class.

###  IV-A Data Preparation

####  IV-A1 LIDC-IDRI dataset for lung nodule classification

LIDC-IDRI is a publicly available dataset for pulmonary nodule classification
or detection, which involves 1,010 patients. Some representative cases are
shown in Fig. 4. All the nodules were labeled by four radiologists, and each
nodule was rated with a score from 1 to 5, indicating malignant progression of
the nodule. In our experiments, we cropped the ROI with a square shape of a
doubled equivalent diameter at the corresponding center of a nodule. The
averaged score of a nodule was used as the ground-truth label during training.
Note that the averaged scores also range from 1 to 5, and we regard a nodule
with an average score between 2.5 and 3.5 as the unsure nodule, benign and
malignant nodules are those with scores that are lower than 2.5 and higher
than 3.5, respectively [15]. In each plane, all the CT volumes were
preprocessed to have $1mm$ spacing in each plane. Finally, we obtain the
training and testing data by cropping the $32\times 32\times 32$ volume ROIs
located at the annotated centers.

####  IV-A2 BUSI dataset for breast cancer classification

The BUSI dataset can be used for ultrasound image-based breast cancer
classification and segmentation, which contains 780 images of three classes:
133 normal, 487 benign, and 210 malignant images. Some representative cases
are shown in Fig. 4. We first resized the original 2-$D$ images into the same
sizes $128\times 128$, and then conducted the data augmentations, including
flipping and adding random Gaussian noise, for the training set of BUSI.
Finally, the training and test sets contain 1,872 images and 156 images for
each fold, respectively.

###  IV-B Implementation Details

####  IV-B1 Network architecture

We applied ResNet-18, ResNet-34, and VGG-16 [1, 47] as backbone networks to
compare our MORF with other methods. Because the scales of the two datasets
are relatively small, we use the 2-$D$ version of the backbone networks to
avoid the huge number of parameters in 3-$D$ networks. Therefore, the input of
the model, the $32\times 32\times 32$ volumes, can be treated as $32\times 32$
patches with $32$ channels each, and the corresponding number of channels of
the first layer is set as $32$. For our MORF, the output dimension of the
final FC layer equals to the number of split nodes in a forest due to the GFS
using random selection without replacement, and for CORF, it is set as $256$.

TABLE II: Classification results on test sets of Train(3)-Test(2) and Train(2)-Test(2) on LIDC-IDRI dataset. The values with underlines indicate the best results while less important in the clinical diagnosis [15]. In this Table, P., R., and F1 are abbreviations of Precision, Recall, and F1 score, respectively. B. is short for Backbone. MORFc is the conference version of the proposed MORF [26]. | Train(3)-Test(2) | Train(2)-Test(2)
---|---|---
B. | Method | Acc. | Benign | Malignant | Acc. | Benign | Malignant
P. | R. | F1 | P. | R. | F1 | P. | R. | F1 | P. | R. | F1
ResNet-18 | CE Loss |  0.833$\pm 0.006$ | 0.848 | 0.924 | 0.884 | 0.790 | 0.634 | 0.703 |  0.855$\pm 0.005$ | 0.879 | 0.915 | 0.897 | 0.793 | 0.723 | 0.756
Poisson [19] |  0.818$\pm 0.004$ | 0.820 | 0.942 | 0.877 | 0.809 | 0.545 | 0.651 |  0.840$\pm 0.003$ | 0.887 | 0.879 | 0.883 | 0.738 | 0.752 | 0.745
NSB [18] |  0.781$\pm 0.005$ | 0.814 | 0.906 | 0.858 | 0.761 | 0.505 | 0.607 |  0.849$\pm 0.007$ | 0.875 | 0.910 | 0.892 | 0.783 | 0.713 | 0.746
UDM [15] |  0.793$\pm 0.004$ | 0.855 | 0.870 | 0.862 | 0.741 | 0.624 | 0.677 |  0.846$\pm 0.005$ | 0.888 | 0.888 | 0.888 | 0.752 | 0.752 | 0.752
CORF [22] |  0.815$\pm 0.003$ | 0.811 | 0.959 | 0.878 | 0.862 | 0.495 | 0.628 |  0.830$\pm 0.002$ | 0.875 | 0.879 | 0.877 | 0.730 | 0.723 | 0.726
MORFc [26] |  0.839$\pm 0.004$ | 0.861 | 0.945 | 0.886 | 0.781 | 0.673 | 0.723 |  0.848$\pm 0.005$ | 0.881 | 0.901 | 0.891 | 0.771 | 0.733 | 0.751
| MORF (Ours) | 0.889$\pm 0.004$ | 0.923 | 0.915 | 0.919 | 0.816 | 0.832 | 0.824 | 0.886$\pm 0.003$ | 0.915 | 0.919 | 0.917 | 0.820 | 0.812 | 0.816
ResNet-34 | CE Loss |  0.824$\pm 0.005$ | 0.852 | 0.928 | 0.888 | 0.811 | 0.594 | 0.686 |  0.846$\pm 0.005$ | 0.862 | 0.923 | 0.891 | 0.800 | 0.673 | 0.731
Poisson [19] |  0.833$\pm 0.004$ | 0.839 | 0.937 | 0.886 | 0.836 | 0.604 | 0.701 |  0.840$\pm 0.006$ | 0.867 | 0.906 | 0.886 | 0.769 | 0.693 | 0.729
NSB [18] |  0.763$\pm 0.006$ | 0.883 | 0.812 | 0.846 | 0.815 | 0.653 | 0.725 |  0.846$\pm 0.005$ | 0.856 | 0.933 | 0.893 | 0.815 | 0.653 | 0.725
UDM [15] |  0.821$\pm 0.004$ | 0.860 | 0.906 | 0.882 | 0.790 | 0.634 | 0.703 |  0.870$\pm 0.003$ | 0.891 | 0.923 | 0.907 | 0.817 | 0.752 | 0.784
CORF [22] |  0.815$\pm 0.004$ | 0.867 | 0.883 | 0.875 | 0.752 | 0.663 | 0.705 |  0.851$\pm 0.005$ | 0.895 | 0.887 | 0.891 | 0.757 | 0.772 | 0.764
MORFc [26] |  0.830$\pm 0.003$ | 0.872 | 0.883 | 0.877 | 0.735 | 0.713 | 0.723 |  0.858$\pm 0.004$ | 0.886 | 0.910 | 0.898 | 0.789 | 0.743 | 0.765
| MORF (ours) | 0.907$\pm 0.003$ | 0.941 | 0.924 | 0.932 | 0.838 | 0.871 | 0.854 | 0.901$\pm 0.003$ | 0.906 | 0.955 | 0.930 | 0.887 | 0.782 | 0.831
VG-16 | CE Loss |  0.775$\pm 0.007$ | 0.848 | 0.848 | 0.848 | 0.816 | 0.614 | 0.701 |  0.855$\pm 0.005$ | 0.864 | 0.937 | 0.899 | 0.829 | 0.673 | 0.743
Poisson [19] |  0.784$\pm 0.006$ | 0.888 | 0.785 | 0.833 | 0.622 | 0.782 | 0.693 |  0.824$\pm 0.005$ | 0.858 | 0.897 | 0.877 | 0.770 | 0.663 | 0.713
NSB [18] |  0.772$\pm 0.004$ | 0.839 | 0.917 | 0.876 | 0.863 | 0.624 | 0.724 |  0.852$\pm 0.005$ | 0.869 | 0.924 | 0.896 | 0.805 | 0.693 | 0.745
UDM [15] |  0.796$\pm 0.003$ | 0.859 | 0.928 | 0.892 | 0.895 | 0.505 | 0.646 |  0.867$\pm 0.004$ | 0.898 | 0.910 | 0.904 | 0.796 | 0.772 | 0.784
CORF [22] |  0.796$\pm 0.005$ | 0.887 | 0.883 | 0.885 | 0.772 | 0.603 | 0.678 |  0.864$\pm 0.007$ | 0.918 | 0.879 | 0.898 | 0.786 | 0.802 | 0.794
MORFc [26] |  0.824$\pm 0.004$ | 0.856 | 0.892 | 0.875 | 0.739 | 0.673 | 0.705 |  0.870$\pm 0.003$ | 0.906 | 0.906 | 0.906 | 0.792 | 0.792 | 0.792
| MORF (ours) | 0.877$\pm 0.004$ | 0.889 | 0.937 | 0.912 | 0.842 | 0.742 | 0.789 | 0.877$\pm 0.004$ | 0.893 | 0.933 | 0.912 | 0.835 | 0.752 | 0.792

####  IV-B2 Hyperparameter setting

The learning rates for the LIDC-IDR and the BUSI datasets are $0.001$ and
$0.00005$, respectively, and are decayed by $0.1$ every $120$ epochs (150
epochs in total); the sizes of a mini-batch size are $16$, and the weight
decay values for the Adam optimizer are $0.0001$ and $0.00005$, respectively
[53]. The loss functions used in the MORF, CORF, and each tree-wise loss
during meta training are the standard CE loss [54, 55, 56].

The number of trees for the forest is $4$ and the tree depth is $3$. In
practice, the TWW-Net contains several MLPs, where the number of MLPs equals
the number of trees. For the LIDC-IDRI dataset, we evaluated whether or not
the unsure nodules were used for training or testing. Therefore, in Tables I
and II, we use the symbol Train($n_{1}$)-Test($n_{2}$) to represent that there
are $n_{1}$ classes of data for training, and $n_{2}$ classes for testing. The
value of $n_{1}$ and $n_{2}$ is $3$ (with unsure data) or $2$ (without unsure
data). All of our experiments are implemented with the PyTorch [57] framework
and trained with an NVIDIA GTX 2080 Ti GPU.

###  IV-C Training with Unsure Data for Lung Nodule Classification

In this section, we focus on the standard 3-class classification of lung
nodules. Following [15], we also care more about the recall of malignant
lesions and the precision of benign lesions, which fits more appropriately
with the clinical diagnosis.

In Table I, we illustrate the results of the CE loss-based methods, the
related ordinal regression methods, and our MORF as well as its conference
version denoted as MORFc. When using different backbones, the MORF achieves
the best accuracies and F1 scores of malignant and benign. For all the
backbones, the MORF also maintains the higher recall of malignant and
precision of benign. Under the meaning of the clinical diagnosis, the MORF is
better able to reduce the missing diagnosis rate, _i.e._ , there will be fewer
unsure and malignant nodules diagnosed as benign, and fewer malignant nodules
missing a diagnosis. Most importantly, MORF obtains the best precision for
malignancy, demonstrating a lower misdiagnosis rate.

Under the setting of Train(3)-Test(2) where training data includes all unsure
data, the feature space will be more complicated compared with the binary
classification setting. However, the left part of Table II shows that the MORF
significantly outperforms the other methods significantly on all measured
metrics. We emphasize that the ordinal relationship of the data is critical to
ordinal regression which can be regarded as a fine-grained classification, and
the accurate feature representation determines the ability of the final
classifier. Although both MORF and CORF [22] consider the global ordinal
relationship, the fixed forest of the CORF degrades its performance in that
the random feature perturbation is omitted. The MORF with reconstructed forest
via the GFS module enable the update of the parameter $\bm{\theta}$ to be
affected by the feature randomness, hence leading to a significant gain.
Therefore, this experiment verifies the robustness of MORF against the
influence of plugging unsure samples into the training data. Most importantly,
the results of MORF on the left side of Table II are slightly better than
those on the right side. This indicates that using the unsure class is helpful
for improving the classification of the other two classes, _i.e._ , the unsure
nodules act as a boundary between the malignant and benign. Especially for
ResNet-18/34, the recalls of malignant and the precisions of benign of
Train(3)-Test(2) are higher than those of Train(2)-Test(2). VGG-16 achieves
comparable results under these two settings, and we attribute this phenomenon
to the different feature spaces learned by ResNet and VGG-16.

When comparing the unsure class with existing methods, Table I shows both MORF
and MORFc achieve better results than other methods in terms of recall rather
than precision. Similar to the importance of recall of malignant [15], the
higher recall of the unsure class provides us with significant insights that
there will be fewer unsure nodules likely being classified as benign or
malignant. Although the unsure class contains mixed benign and malignant
samples, one should not miss any malignant samples in unsure class. That is,
the unsure class should be similar to malignant class that the recall is
relatively more important than the precision. Therefore, MORF and MORFc are
helpful for further diagnosis of nodules, such as biopsy. Consequently, MORF
is more suitable for real clinical circumstance while recommending more
accurate diagnosis of follow-ups. It is noticed that this superiority does not
hold in the UDM method [15]. On the other hand, for a certain backbone, both
the MORF and MORFc outperform the CE loss-based counterpart, which exhibits
their effectiveness on exploiting ordinal relationship.

TABLE III: Classification results on test set of BUSI dataset. The values with underlines indicate the best results while less important in the clinical diagnosis [15].MORFc is the conference version of the proposed MORF [26]. Backbone | Method | Accuracy | Normal | Benign | Malignant
---|---|---|---|---|---
Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1
ResNet-18 | CE Loss | 0.726$\pm 0.003$ | 0.750 | 0.556 | 0.638 | 0.779 | 0.761 | 0.770 | 0.627 | 0.762 | 0.688
Poisson [19] | 0.707$\pm 0.004$ | 0.692 | 0.667 | 0.679 | 0.817 | 0.659 | 0.730 | 0.583 | 0.833 | 0.686
NSB [18] | 0.751$\pm 0.005$ | 0.875 | 0.259 | 0.400 | 0.739 | 0.932 | 0.824 | 0.763 | 0.690 | 0.725
UDM [15] | 0.739$\pm 0.003$ | 1.000 | 0.259 | 0.412 | 0.762 | 0.875 | 0.814 | 0.653 | 0.761 | 0.703
CORF [22] | 0.758$\pm 0.006$ | 0.875 | 0.519 | 0.651 | 0.723 | 0.920 | 0.809 | 0.827 | 0.571 | 0.676
MORFc [26] | 0.764$\pm 0.003$ | 1.000 | 0.481 | 0.650 | 0.816 | 0.807 | 0.811 | 0.632 | 0.857 | 0.727
MORF (ours) | 0.809$\pm 0.003$ | 0.882 | 0.556 | 0.682 | 0.850 | 0.841 | 0.845 | 0.717 | 0.905 | 0.800
ResNet-34 | CE Loss | 0.751$\pm 0.007$ | 0.917 | 0.407 | 0.564 | 0.757 | 0.886 | 0.816 | 0.690 | 0.690 | 0.690
Poisson [19] | 0.701$\pm 0.005$ | 0.606 | 0.740 | 0.667 | 0.811 | 0.682 | 0.741 | 0.600 | 0.714 | 0.652
NSB [18] | 0.733$\pm 0.006$ | 0.933 | 0.519 | 0.667 | 0.784 | 0.784 | 0.784 | 0.592 | 0.761 | 0.667
UDM [15] | 0.707$\pm 0.003$ | 0.642 | 0.333 | 0.439 | 0.753 | 0.795 | 0.773 | 0.640 | 0.762 | 0.696
CORF [22] | 0.764$\pm 0.006$ | 0.923 | 0.444 | 0.600 | 0.726 | 0.931 | 0.815 | 0.838 | 0.619 | 0.712
MORFc [26] | 0.771$\pm 0.003$ | 1.000 | 0.518 | 0.683 | 0.839 | 0.772 | 0.804 | 0.629 | 0.928 | 0.750
MORF (ours) | 0.822$\pm 0.004$ | 0.724 | 0.778 | 0.750 | 0.880 | 0.830 | 0.854 | 0.778 | 0.833 | 0.805
VGG-16 | CE Loss | 0.688$\pm 0.006$ | 0.571 | 0.148 | 0.235 | 0.710 | 0.863 | 0.779 | 0.651 | 0.667 | 0.658
Poisson [19] | 0.713$\pm 0.005$ | 0.618 | 0.778 | 0.689 | 0.861 | 0.636 | 0.732 | 0.603 | 0.833 | 0.700
NSB [18] | 0.764$\pm 0.005$ | 1.000 | 0.370 | 0.540 | 0.781 | 0.852 | 0.815 | 0.686 | 0.833 | 0.752
UDM [15] | 0.777$\pm 0.004$ | 0.928 | 0.481 | 0.634 | 0.782 | 0.897 | 0.835 | 0.714 | 0.714 | 0.714
CORF [22] | 0.752$\pm 0.003$ | 0.916 | 0.407 | 0.564 | 0.707 | 0.989 | 0.824 | 0.909 | 0.476 | 0.624
MORFc [26] | 0.796$\pm 0.003$ | 1.000 | 0.593 | 0.744 | 0.777 | 0.909 | 0.837 | 0.763 | 0.690 | 0.725
MORF (ours) | 0.854$\pm 0.003$ | 0.807 | 0.778 | 0.792 | 0.905 | 0.864 | 0.884 | 0.787 | 0.881 | 0.831
Figure 5: Some prediction results of MORF and CORF under Train(3)-Test(2)
setting. The $y$ axis represents the prediction results: 1, 2 and 3 represent
benign, unsure and malignant. The GT denotes ground truth. The black vertical
line on each bar represents the variance over the predictions of all trees.
Green box contains some representative nodules, and red box include some
failure cases.

###  IV-D Training without Unsure Data for Lung Nodule Classification

To verify the effectiveness of MORF on binary classification, we compare the
results of all methods training without unsure data. The CE loss under
Train(2)-Test(2) in Table II (right) is the conventional binary classifier
whose output dimension is $2$, and this is different from that of
Train(3)-Test(2) whose output dimension is $3$. It is clear that the MORF also
achieves the best overall accuracy, precision of benign, and recall of
malignant using different backbones.

Through the comparison in Table II, we can see that the unsure data largely
affect the generalizability of the compared methods. There are no severe
fluctuations in the performance of MORF under the two settings, indicating
that MORF is able to distinguish the samples with ordinal labels regardless of
whether the ordinal margin is large (without unsure) or small (with unsure).

Here, we would like to clarify why the performance in Table I is much lower
than that seen in Table II. This is due to the imperfect performance of
recognizing the unsure samples, so that it becomes unavoidable to encounter
the classification errors of all classes.

###  IV-E Classification Results on BUSI Dataset

In Table III, we illustrate the results of all the methods on the BUSI
dataset. For the benign and malignant classes, we also focused more on the
precision of benign and recall of malignant. Different from the LIDC-IDRI
dataset, we can see from Fig. 4 that the first order in the BUSI dataset is
the normal class, which does not contain nodules. Therefore, the benign class
occupies different positions in the orders of the two datasets. Interestingly,
our MORF also retains the best precisions of benign and malignant recalls as
shown in Table III. This demonstrates the discriminative ability of MORF in
recognizing nodules of different orders without the influence of the normal
class that does not include nodules.

Clinically, false positives of the normal class indicate that benign or
malignant nodules are falsely classified as normal, which will result in an
increase of missing diagnosis; in contrast, false negatives of the normal
class will cause an increase in misdiagnosis. Since the precision and recall
correlate with false positives and the false negative, here we suggest that
the precision and recall of the normal class have equal importance weights.
The results of compared methods in Table III show that they are prone to
preferring precision or recall of the normal class. For example, the UDM
obtains a precision of $1.000$ when using ResNet-18 while the corresponding
recall is $0.259$; the NSB achieves similar results with a large margin
between precision and recall. However, the MORF has relatively balanced
precisions and recalls, and it also maintains the best F1 scores when applying
all of the backbones.

###  IV-F Comparisons between MORF and MORFc

From Tables I, II, and III, we can see that MORF consistently outperforms
MORFc in terms of overall accuracy, which is benefited from the improved GFS
using random selection without replacement. For both of the two datasets, MORF
is better at identifying benign and malignant classes compared with MORFc, and
this guarantees the improvements of overall performance. Recalling the
essential difference between MORF and MORFc that GFS in MORF uses random
selection without replacement, as a result, MORF makes more efficient use of
FC output vector while MORFc could be affected by selection of unused elements
in FC output vector. Therefore, the results suggest that GFS without
replacement will be more conducive to distinctive feature learning that
identifies benign and malignant.

###  IV-G Tree-wise Variance Reduction

As analyzed in Section III-C that all trees in our MORF are influenced by the
gradient of the meta loss through the TWW-Net weighting scheme. Here, we
provide experiments to verify the consistency of behaviors of different trees.
This experiment was conducted on LIDC-IDRI dataset. We estimated the variance
over predictions of all trees:
$var:=\frac{1}{T}\sum_{t=1}^{T}|p_{t}-\hat{p}|^{2}$, where $p_{t}$ and
$\hat{p}$ denote the prediction of the $t$-th tree and the final prediction,
respectively. In Fig. 5, we compare the predictions and the corresponding
$var$s on some special nodules such as cavities (b), ground-glass (c),
calcifications (d, f) and benign nodule with larger sizes (e). For the
prediction results, the MORF is more accurate than CORF on various kinds of
nodules using different backbones. Especially for the ground-glass nodules and
the large benign nodules, the CORF makes incorrect predictions, which will
cause severe diagnosis loss. For the large malignant nodule (a) and the
calcification case (d), the CORF tends to regard them as unsure while delaying
the treatment of the patient with the malignant nodule. This defect may be
attributed to the lower feature randomness in CORF. Moreover, the $var$s of
MORF are much lower than those of CORF in all cases. This indicates the
behavioral consistency of all of the trees in MORF, which benefits from the
guidance of meta GFS features and the meta training procedure.

In Fig. 5, we also provide some failure cases obtained by MORF and CORF,
_i.e._ , (g), (h), (i). (g) is malignant and its real malignant score is
$3.75$ which is referred to Fig. 1. However, both MORF and CORF make incorrect
predictions, and this is due to blur edges or shapes of the nodule. (h) is a
benign nodule in our study with malignant score $2.5$ which is an upper bound
of score range of benign class. We can see that MORF is prone to classifying
it as unsure, while CORF obtains predictions of benign. (i) is a malignant
nodule with malignant score $3.5$, _i.e._ , the lower bound of malignant, and
the predictions are incorrect obtained by the two methods. We conclude that
MORF and CORF can be confused by the malignant or benign nodules whose
malignant scores are close to unsure class, and MORF prefers predictions of
unsure for them. This phenomenon also reflects that MORF is more suitable for
real clinical circumstance that requires more nodules surrounding the unsure
class for further diagnosis.

###  IV-H Ablation Study

Here, we evaluated the effectiveness of the GFS module and TWW-Net based on
CORF. The experiments were conducted on the LIDC-IDRI dataset under the
Train(2)-Test(2) setting using the ResNet-18 backbone. Then, we evaluated the
effects of the number and depth of trees in MORF. Finally, we conducted a
significance analysis between MORF and other methods through predictions on
the LIDC-IDRI testing set (Train(3)-Test(3)).

TABLE IV: Effectiveness evaluation of GFS and TWW-Net on the LIDC-IDRI dataset, under the Train(2)-Test(2) setting using ResNet-18. Method | Accuracy | Benign | Malignant
---|---|---|---
Precision | F1 | Recall | F1
CORF | 0.830 | 0.893 | 0.874 | 0.772 | 0.739
+GFS | 0.843 | 0.881 | 0.886 | 0.733 | 0.744
+TWW-Net | 0.821 | 0.897 | 0.897 | 0.653 | 0.737
MORF | 0.886 | 0.915 | 0.917 | 0.812 | 0.816

####  IV-H1 GFS module

To verify the random feature perturbation enforced by the GFS, we added the
GFS to CORF termed CORF+GFS. The forest of the CORF is fixed during training
and inference. In contrast, the CORF+GFS enables the forest structure to be
dynamic during training only, and the training process does not include the
meta train stage. That is to say, the CORF is equipped with random feature
perturbations. From Table IV we can see that the CORF+GFS achieves better
performances than the vanilla CORF. This indicates that the training of the
CORF benefits from the GFS in that the GFS generated forest endows the target
model ($\bm{\theta}$) with the generalizability increased by the random
feature perturbation. However, the drawback of all trees sharing the same
weights is still not resolved. In addition, we observe that the precision of
benign and the recall of malignant of CORF+GFS are worse than CORF, which can
be explained as follows: GFS is specially designed to improve the meta-
training of the proposed MORF while CORF does not have meta training, as a
result, the GFS shall compromise the performance of CORF as expected.

####  IV-H2 TWW-Net

The CORF+TWW-Net in Table IV is tailored for evaluating the TWW-Net without
the GFS, _i.e._ , the structure of the forest is also fixed, and the training
process includes the meta train stage. Table IV shows that CORF+TWW-Net
performs worse than the CORF. This is due to that the training data and the
meta data are the same and consequently, the two terms of the multiplication
in $G_{ij}$ as shown in (10) are the same. Therefore, the $G_{ij}$ is almost
at the orientation of the largest gradient, and this phenomenon happens
equally to all training samples. So we argue that in (10), CORF+TWW-Net could
accelerate the update of $\bm{\phi}$, and hence, may trigger the overfitting
of TWW-Net. In other words, the update of $\bm{\theta}$ is not guided by the
meta data. Consequently, the meta weighting scheme of TWW-Net should be driven
by the model generalizability gain from GFS. For MORF (_i.e._ CORF + GFS +
TWW-Net), the update of parameters $\bm{\theta}$ can be guided by GFS
generated features, therefore, $G_{ij}$ involves gradients of GFS features,
then $G_{ij}$ will slow down the update of $\bm{\phi}$ according to (10).
Hence, the combination of GFS and meta training with TWW-Net achieves trade-
off between updating the parameters $\bm{\theta}$ and $\bm{\phi}$.

####  IV-H3 Number and depth of trees

Here, we further discuss the effects of the number and depth of trees. We fix
one of them and evaluate the settings with various values of another. The
backbone network is VGG-16. Fig. 6 shows the performances influenced by these
two factors on the LIDC-IDRI under the setting of Train(3)-Test(3) and BUSI
datasets.

In Fig. 6, we can see that the setting with the number of trees being 4 and
the depth of trees being 3 achieves the best performances for both of
datasets. Lower or higher values of number and depth will decrease the
performances. If the number of trees is fixed, there will be more nodes in a
forest which requires FC output vector to have higher dimension, _i.e._ , more
elements. Then the parameters of the framework begin to increase, and this
will affect the performance. If the depth of trees is fixed, we observe that
when the number is small, _i.e._ , $2$ or $3$, MORF achieves lower
performances on two datasets. This indicates that fewer ensembled trees can
affect the capability of the MORF, then hinder the performance improvement. In
contrast, too many trees ($>4$) also results in a decrease of performance
which is attributed to more MLPs for weights learning, _i.e._ , there are more
parameters to learn.

Figure 6: Classification accuracies with varying values of the depth of trees
and number of trees on (a) LIDC-IDRI and (b) BUSI datasets.

###  IV-I Significance Analysis

In order to show the significant differences between MORF and compared
methods, we compare the differences between MORF and other methods through
conducting the Wilcoxon signed-rank test [58] with respect to predicted
probabilities on test set. In Table V, we can see that all the $p$-values
obtained by MORF and baseline methods are less than the significance level
$0.05$. So we reject the null hypothesis that MORF possesses the same
prediction distribution as baseline methods. Finally, we conclude that MORF is
_significantly_ better than the baseline methods.

TABLE V: $p$-values between MORF and other methods using different backbones. Backbone | CE Loss |  Poisson [19] |  NSB [18] |  UDM [15] |  CORF [22]
---|---|---|---|---|---
ResNet-18 |  $<$0.01 |  $<$0.01 |  $<$0.01 |  $<$0.01 |  $<$0.01
ResNet-34 |  $<$0.01 |  $<$0.01 |  $<$0.01 |  $<$0.01 |  $<$0.01
VGG-16 |  $<$0.01 |  $<$0.01 |  $<$0.01 |  $<$0.01 |  $<$0.01

##  V Conclusion

In this paper, we propose a meta ordinal regression forest, termed MORF, for
improving the performances of the ordinal regression in medical imaging, such
as lung nodule classification and breast cancer classification. The MORF
contains a grouped feature selection module that is used to generate a dynamic
forest with feature random perturbation. Another critical component of the
MORF is the TWW-Net which assigns each tree with a learned weight, and this
enforces the predictions of different trees to have smaller variance while
maintaining stable performances. The parameters of the model are learned
through the meta-learning scheme which can solve the problem of integrating
two parts of parameters into one training loop, and it brings the gradients of
target data and meta data to be closer. Through the experiments, we have
verified that the MORF can help reduce the false positives and the false
negatives of the relevant classes, which is significant to the clinical
diagnosis. Moreover, we have also verified that the accurate recognition of
the intermediate order can improve the classification of the other classes on
both sides.

In the future, we will consider to explore attention mechanism for achieving
better generalizability of deep random forest such as in [59]. Also, we will
simplify the MORF framework such as using the design of light weight network
or efficient loss functions [60, 61].

## References

  * [1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [2] L. Shen, L. R. Margolies, J. H. Rothstein, E. Fluder, R. McBride, and W. Sieh, “Deep learning to improve breast cancer detection on screening mammography,” _Scientific Reports_ , vol. 9, no. 1, pp. 1–12, 2019.
  * [3] W. Zhu, C. Liu, W. Fan, and X. Xie, “Deeplung: 3D deep convolutional nets for automated pulmonary nodule detection and classification,” _arXiv preprint arXiv:1709.05538_ , 2017.
  * [4] W. Shen, Y. Guo, Y. Wang, K. Zhao, B. Wang, and A. L. Yuille, “Deep regression forests for age estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 2304–2313.
  * [5] P. Kontschieder, M. Fiterau, A. Criminisi, and S. Rota Bulo, “Deep neural decision forests,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 1467–1475.
  * [6] Z. Zhang and Y. Pang, “Cgnet: cross-guidance network for semantic segmentation,” _Science China Information Science_ , vol. 63, no. 2, pp. 1–16, 2020.
  * [7] Z. Wang, X. Liu, J. Lin, C. Yang, and H. Li, “Multi-attention based cross-domain beauty product image retrieval,” _Science China Information Science_ , vol. 63, no. 2, pp. 1–3, 2020.
  * [8] Y. Lei, Y. Tian, H. Shan, J. Zhang, G. Wang, and M. K. Kalra, “Shape and margin-aware lung nodule classification in low-dose CT images via soft activation mapping,” _Medical Image Analysis_ , vol. 60, p. 101628, 2020\.
  * [9] S. G. Armato III, G. McLennan, L. Bidaut _et al._ , “The lung image database consortium (LIDC) and image database resource initiative (IDRI): A completed reference database of lung nodules on CT scans,” _Medical Physics_ , vol. 38, no. 2, pp. 915–931, 2011.
  * [10] H. Shan, G. Wang, M. K. Kalra, R. de Souza, and J. Zhang, “Enhancing transferability of features from pretrained deep neural networks for lung nodule classification,” in _The Proceedings of the 2017 International Conference on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine (Fully3D)_ , 2017, pp. 65–68.
  * [11] S. Hussein, R. Gillies, K. Cao, Q. Song, and U. Bagci, “Tumornet: Lung nodule characterization using multi-view convolutional neural network with Gaussian process,” in _2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI)_ , 2017, pp. 1007–1010.
  * [12] W. Shen, M. Zhou, F. Yang, C. Yang, and J. Tian, “Multi-scale convolutional neural networks for lung nodule classification,” in _International Conference on Information Processing in Medical Imaging (IPMI)_ , 2015, pp. 588–599.
  * [13] Q. Dou, H. Chen, L. Yu, J. Qin, and P.-A. Heng, “Multilevel contextual 3D CNNs for false positive reduction in pulmonary nodule detection,” _IEEE Trans. Biomed. Eng._ , vol. 64, no. 7, pp. 1558–1567, 2017.
  * [14] A. A. A. Setio, F. Ciompi, G. Litjens _et al._ , “Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks,” _IEEE Trans. Med. Imag._ , vol. 35, no. 5, pp. 1160–1169, 2016.
  * [15] B. Wu, X. Sun, L. Hu, and Y. Wang, “Learning with unsure data for medical image diagnosis,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 10 590–10 599.
  * [16] P. A. Gutiérrez, M. Perez-Ortiz, J. Sanchez-Monedero, F. Fernandez-Navarro, and C. Hervas-Martinez, “Ordinal regression methods: survey and experimental study,” _IEEE Trans. Knowl. Data Eng._ , vol. 28, no. 1, pp. 127–146, 2015\.
  * [17] E. Frank and M. Hall, “A simple approach to ordinal classification,” in _European Conference on Machine Learning (ECML)_. Springer, 2001, pp. 145–156.
  * [18] X. Liu, Y. Zou, Y. Song, C. Yang, J. You, and B. K Vijaya Kumar, “Ordinal regression with neuron stick-breaking for medical diagnosis,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 335–344.
  * [19] C. Beckham and C. Pal, “Unimodal probability distributions for deep ordinal classification,” in _Proceedings of the International Conference on Machine Learning (ICML)_ , 2017, pp. 411–419.
  * [20] R. Diaz and A. Marathe, “Soft labels for ordinal regression,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 4738–4747.
  * [21] W. Shen, K. Zhao, Y. Guo, and A. L. Yuille, “Label distribution learning forests,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 834–843.
  * [22] H. Zhu, H. Shan, Y. Zhang, L. Che, X. Xu, J. Zhang, J. Shi, and F.-Y. Wang, “Convolutional ordinal regression forest for image ordinal estimation,” _IEEE Trans. Neural Netw. Learn. Syst._ , 2021.
  * [23] L. Breiman, “Random forests,” _Machine learning_ , vol. 45, no. 1, pp. 5–32, 2001.
  * [24] J. Shu, Q. Xie, L. Yi, Q. Zhao, S. Zhou, Z. Xu, and D. Meng, “Meta-weight-net: Learning an explicit mapping for sample weighting,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 1919–1930.
  * [25] S. Liu, A. Davison, and E. Johns, “Self-supervised generalisation with meta auxiliary learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019, pp. 1679–1689.
  * [26] Y. Lei, H. Zhu, J. Zhang, and H. Shan, “Meta ordinal regression forest for learning with unsure lung nodules,” in _IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2020, pp. 442–445.
  * [27] W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy, “Dataset of breast ultrasound images,” _Data in Brief_ , vol. 28, p. 104863, 2020.
  * [28] X. Liu, F. Hou, H. Qin, and A. Hao, “A CADe system for nodule detection in thoracic CT images based on artificial neural network,” _Science China Information Science_ , vol. 60, no. 7, p. 072106, 2017.
  * [29] G. Cao, T. Huang, K. Hou, W. Cao, P. Liu, and J. Zhang, “3D convolutional neural networks fusion model for lung nodule detection onclinical CT scans,” in _2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , 2018, pp. 973–978.
  * [30] L. Fang, Z. Wang, Z. Chen, F. Jian, S. Li, and H. He, “3D shape reconstruction of lumbar vertebra from two X-ray images and a CT model,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 7, no. 4, pp. 1124–1133, 2019\.
  * [31] A. Muzahid, W. Wan, F. Sohel, L. Wu, and L. Hou, “Curvenet: Curvature-based multitask learning deep networks for 3D object recognition,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 8, no. 6, pp. 1177–1187, 2020\.
  * [32] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , 2015, pp. 234–241.
  * [33] H. Shan, Y. Zhang, Q. Yang, U. Kruger, M. K. Kalra, L. Sun, W. Cong, and G. Wang, “3-d convolutional encoder-decoder network for low-dose CT via transfer learning from a 2-d trained network,” _IEEE Trans. Med. Imag._ , vol. 37, no. 6, pp. 1522–1534, 2018.
  * [34] N. Ibtehaz and M. S. Rahman, “MultiResUNet: Rethinking the u-net architecture for multimodal biomedical image segmentation,” _Neural Networks_ , vol. 121, pp. 74–87, 2020.
  * [35] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, and J. Wu, “Unet 3+: A full-scale connected unet for medical image segmentation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 1055–1059.
  * [36] W. Wu, G. Liu, K. Liang, and H. Zhou, “Pneumothorax segmentation in routine computed tomography based on deep neural networks,” in _2021 4th International Conference on Intelligent Autonomous Systems (ICoIAS)_. IEEE, 2021, pp. 78–83.
  * [37] A. Ciritsis, C. Rossi, M. Eberhard, M. Marcon, A. S. Becker, and A. Boss, “Automatic classification of ultrasound breast lesions using a deep convolutional neural network mimicking human decision-making,” _European Radiology_ , vol. 29, no. 10, pp. 5458–5468, 2019.
  * [38] C. D. L. Nascimento, S. D. d. S. Silva, T. A. d. Silva, W. C. d. A. Pereira, M. G. F. Costa, and C. F. F. Costa Filho, “Breast tumor classification in ultrasound images using support vector machines and neural networks,” _Research on Biomedical Engineering_ , vol. 32, no. 3, pp. 283–292, 2016\.
  * [39] Y. Wang, E. J. Choi, Y. Choi, H. Zhang, G. Y. Jin, and S.-B. Ko, “Breast cancer classification in automated breast ultrasound using multiview convolutional neural network with transfer learning,” _Ultrasound in Medicine & Biology_, 2020.
  * [40] Q. Sun, X. Lin, Y. Zhao, L. Li, K. Yan, D. Liang, D. Sun, and Z.-C. Li, “Deep learning vs. radiomics for predicting axillary lymph node metastasis of breast cancer using ultrasound images: Don’t forget the peritumoral region,” _Frontiers in Oncology_ , vol. 10, pp. 53–64, 2020.
  * [41] Q. Hu, H. M. Whitney, and M. L. Giger, “A deep learning methodology for improved breast cancer diagnosis using multiparametric MRI,” _Scientific Reports_ , vol. 10, no. 1, pp. 1–11, 2020.
  * [42] J.-Y. Chiao, K.-Y. Chen, K. Y.-K. Liao, P.-H. Hsieh, G. Zhang, and T.-C. Huang, “Detection and classification the breast tumors using mask r-cnn on sonograms,” _Medicine_ , vol. 98, no. 19, 2019.
  * [43] N. Wu, J. Phang, J. Park, Y. Shen, Z. Huang, M. Zorin, S. Jastrzkebski, T. Fevry, J. Katsnelson, E. Kim _et al._ , “Deep neural networks improve radiologists’ performance in breast cancer screening,” _IEEE Trans. Med. Imag._ , vol. 39, no. 4, pp. 1184–1194, 2019.
  * [44] A. Akselrod-Ballin, M. Chorev, Y. Shoshan, A. Spiro, A. Hazan, R. Melamed, E. Barkan, E. Herzel, S. Naor, E. Karavani _et al._ , “Predicting breast cancer by applying deep learning to linked health records and mammograms,” _Radiology_ , vol. 292, no. 2, pp. 331–342, 2019.
  * [45] N. Dhungel, G. Carneiro, and A. P. Bradley, “The automated learning of deep features for breast mass classification from mammograms,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. Springer, 2016, pp. 106–114.
  * [46] Y. B. Hagos, A. G. Mérida, and J. Teuwen, “Improving breast cancer detection using symmetry information with deep learning,” in _Image Analysis for Moving Organ, Breast, and Thoracic Images_. Springer, 2018, pp. 90–97.
  * [47] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations (ICLR)_ , 2015, pp. 1–14.
  * [48] H. Zhu, Y. Zhang, G. Li, J. Zhang, and H. Shan, “Ordinal distribution regression for gait-based age estimation,” _Science China Information Science_ , vol. 63, no. 2, pp. 1–14, 2020.
  * [49] G. Guo, H. Wang, Y. Yan, L. Zhang, and B. Li, “Large margin deep embedding for aesthetic image classification,” _Science China Information Science_ , vol. 63, no. 1, pp. 1–3, 2020.
  * [50] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1126–1135.
  * [51] M. A. Jamal and G.-J. Qi, “Task agnostic meta-learning for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 11 719–11 727.
  * [52] B. C. Csáji _et al._ , “Approximation with artificial neural networks,” _Faculty of Sciences, Etvs Lornd University, Hungary_ , vol. 24, no. 48, p. 7, 2001.
  * [53] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [54] K. Janocha and W. M. Czarnecki, “On loss functions for deep neural networks in classification,” _arXiv preprint arXiv:1702.05659_ , 2017.
  * [55] J. Zhang, Y. Xie, Q. Wu, and Y. Xia, “Medical image classification using synergic deep learning,” _Medical Image Analysis_ , vol. 54, pp. 10–19, 2019\.
  * [56] E. Pesce, S. J. Withey, P.-P. Ypsilantis, R. Bakewell, V. Goh, and G. Montana, “Learning to detect chest radiographs containing pulmonary lesions using visual attention networks,” _Medical Image Analysis_ , vol. 53, pp. 26–38, 2019.
  * [57] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [58] F. Wilcoxon, “Individual comparisons by ranking methods,” in _Breakthroughs in Statistics_. Springer, 1992, pp. 196–202.
  * [59] S. Xuan, G. Liu, and Z. Li, “Refined weighted random forest and its application to credit card fraud detection,” in _International Conference on Computational Social Networks_. Springer, 2018, pp. 343–355.
  * [60] Y. Liang, M. Li, C. Jiang, and G. Liu, “Cemodule: A computation efficient module for lightweight convolutional neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [61] Y. Qin, C. Yan, G. Liu, Z. Li, and C. Jiang, “Pairwise gaussian loss for convolutional neural networks,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 10, pp. 6324–6333, 2020.