# Cross-Cluster Weighted Forests

Maya Ramchandran<EMAIL_ADDRESS>
Department of Biostatistics
Harvard University
Boston, MA 02115, USA Rajarshi Mukherjee<EMAIL_ADDRESS>
Department of Biostatistics
Harvard University
Boston, MA 02115, USA Giovanni Parmigiani<EMAIL_ADDRESS>
Department of Data Science
Dana-Farber Cancer Institute
Boston, MA 02115, USA

###### Abstract

Adapting machine learning algorithms to better handle clustering or batch
effects within training data sets is important across a wide variety of
biological applications. This article considers the effect of ensembling
Random Forest learners trained on clusters within a single data set with
heterogeneity in the distribution of the features. We find that constructing
ensembles of forests trained on clusters determined by algorithms such as
k-means results in significant improvements in accuracy and generalizability
over the traditional Random Forest algorithm. We denote our novel approach as
the Cross-Cluster Weighted Forest, and examine its robustness to various data-
generating scenarios and outcome models. Furthermore, we explore the influence
of the data-partitioning and ensemble weighting strategies the benefits of our
method over the existing paradigm. Finally, we apply our approach to cancer
molecular profiling and gene expression data sets that are naturally divisible
into clusters and illustrate that our approach outperforms classic Random
Forest. Code and supplementary material are available at
https://github.com/m-ramchandran/cross-cluster.

Keywords: Ensemble learning, Random Forest, clustering, replicability, data
heterogeneity

##  1 Introduction

data sets containing natural clusters or batch effects are common across most
biological applications, motivating the need for prediction algorithms that
can adapt to the particular challenges of handling possible cluster-like
heterogeneity in the distribution of the features (Goh et al., 2017; Chauhan
et al., 2010). Numerous learning algorithms have been developed in such
settings where the covariate-outcome relationship varies across clusters (Luo
and Schumacher, 2010; Verbeke and Lesaffre, 1996; Dietterich, 2000;
Bouwmeester et al., 2013). Research has additionally suggested the possibility
of improving prediction accuracy in such scenarios by partitioning data
through suitable clustering algorithms before applying an ensembling framework
to the pre-clustered data. (Ramchandran et al., 2020; Trivedi et al., 2015;
Deodhar and Ghosh, 2007).

Learning algorithms that have been explored through the lens of data sets
having natural clustering structure include Neural Networks, Random Forest,
and (regularized) least squares linear regression (Patil and Parmigiani, 2018;
Sharkey, 1996; Liaw and Wiener, 2002). Random Forest, itself a highly popular
ensemble machine learning algorithm designed to reduce overfitting and handle
high-dimensional data, has shown widespread success in many scientific realms.
The method was introduced by Breiman (2001) and has since been adapted to
handle a wide variety of data types and applications, including missing data
imputation, survival analysis, and unsupervised clustering (Liaw and Wiener,
2002; Biau and Scornet, 2016). Ramchandran et al. (2020) previously showed
that in the presence of heterogeneity in both the outcome model and
distribution of the features across multiple training studies, building
ensembles of forests each trained on a single study produced significantly
more accurate predictions than training a single forest on all of the
available data when the resulting learners contained an equivalent number of
trees. Additionally, these results held when heterogeneity in the covariate-
outcome relationship was removed; when trained on randomly-chosen partitions
of a large single data set, forest-based ensembles considerably improved upon
a single forest trained on the entire data set. Moreover, the context in which
ensemble learners are built using pre-clustered data is analogous to the
multi-study framework 111Multi-study learning handles the availability of
multiple training studies that measure the same outcome and many of the same
covariates by building ensembles of learners each trained on a single study to
form the final predictor., as formalized by Patil and Parmigiani (2018), if
one thinks of separate clusters as individual studies. In view of these
results, it is therefore natural to hypothesize that for data sets containing
clusters, data can be even more optimally partitioned using clustering
algorithms in order to form ensembles of forests that outperform the classic
Random Forest algorithm. This manuscript is motivated by exploring the
aforementioned hypothesis in detail.

In this paper, we build upon the intuitions discussed above and additionally
consider the situation in which the true outcome model remains relatively
constant across clusters and the primary source of heterogeneity is in the
distribution of the covariates; this requires prediction algorithms to now be
able to disentangle the true covariate-outcome relationship from
distributional variation introduced by the presence of clusters in the
covariates. At first glance, this context is conceptually similar to covariate
shift, in which the training and test samples are drawn from different
distributions while the outcome model remains constant (Sugiyama et al.,
2007). However, whereas classic covariate shift applications typically only
consider dissimilarities between the training data as a whole and the test
data, we investigate the addition of internal heterogeneity in the marginal
distribution of the training covariates. We particularly focus on ensembles of
Random Forests, in which each forest is trained on a subsection of the total
available data and then combined using a weighting approach rewarding
generalizability in other clusters across the training set. We propose a
strategy of training ensembles of forests in which clusters of points that
produce more highly generalizable predictors are given greater influence.
Throughout, we will use the term ’estimated clusters’ to refer to clusters
obtained through clustering algorithms, and ’true clusters’ to refer to the
latent labels from the data-generation process.

Specifically, we introduce the Cross-Cluster Weighted Forest, a learning
approach with the following steps: first, the data set is separated into
estimated clusters using k-means or other clustering algorithms. Next, Random
Forests are trained on each cluster. Finally, stacked regression weights are
employed to construct the overall ensemble, in order to up-weight individual
predictors that show stability across estimated clusters. We illustrate that
our method greatly improves upon the existing paradigm across a variety of
realistic scenarios as well as real biological data naturally divisible into
clusters. Our overall aim is to improve upon the traditional Random Forest
algorithm in the presence of heterogeneity in the distribution of the
features, to form more replicable predictors.

We begin this article by describing the Cross-Cluster Weighted Forest approach
and the additional ensembling methods we will be comparing throughout our
analysis. We then present the results of our experimental explorations of its
performance on single data sets containing clusters, organizing our discussion
by isolating the beneficial effects of each step in our algorithm on the
behavior of the overall ensemble. We additionally determine the robustness of
our method to realistic variety in data generation, demonstrating that our
proposed approach has greater generalizability across scenarios than all other
methods considered. Next, we explore the multi-study setting, and illustrate
that our proposed addition of the clustering step improves upon the existing
paradigm of simply ensembling forests trained on each study. Finally, we
consider real gene expression and clinical data plausibly containing
heterogeneity in the distribution of the features, and demonstrate our
method’s superior performance over classic Random Forest.

The general strategy of utilizing clustering algorithms to partition data and
build ensembles of learners trained on each discovered cluster has been
previously explored in Deodhar and Ghosh (2007). They present a regime in
which co-clustering and model fitting are performed simultaneously, and
primarily consider prediction algorithms fitting into the generalized linear
model class as base learners. Their work highlights the efficacy of allowing
ensemble members to learn different aspects of the input space, and
particularly applies to the setting in which the independent variables contain
two or more known groups that are associated with their corresponding modes.
In our work, we remove such requirements and instead simply allow clustering
algorithms to find any natural substructure in the data prior to ensembling.
Additionally, by focusing on Random Forest as a base learner, we allow for
more complex interactions between covariates in the modeling process. Through
our particular choice of ensembling weights, we additionally explore the
relationship between the partitioning step and the generalizability of the
resulting models. In particular, the clustering step efficaciously removes
within-cluster feature distribution heterogeneity while maximizing cross-
cluster heterogeneity, and thus the strategy of ensembling learners trained on
these diverse partitions reflects the common ensemble learning standard of
combining dissimilar learners in order to improve prediction generalizability
(Schapire, 2003; Dietterich, 2000). In fact, this is the very paradigm that
motivates the classic Random Forest algorithm, in which the randomness of the
bootstrapping step and the subset of total variables available at each split
serve to create diversity in the trees that comprise the ensemble and thus
lower its variance (Breiman, 1996a; Liaw and Wiener, 2002).

##  2 Methods

The Cross-Cluster Weighted Forest (CCWF) approach is outlined in Algorithm 1.
We denote by $\hat{Y}_{j}(\mathbf{x}_{\star})$ the prediction of the forest
trained on cluster $j$ on new point $\mathbf{x}_{\star}$ for $j=1,\ldots,k$,
and by $\hat{Y}_{E}(\mathbf{x}_{\star})$ the prediction of the overall CCWF on
$\mathbf{x}_{\star}$.

Set the number of clusters $k\geq 2$; then,

  1. 1.

Run a clustering algorithm on the entire set of covariates

  2. 2.

For $j=1,...,k$

     * •

Train a Random Forest on the jth estimated cluster

     * •

Compute $\hat{Y}_{j}(\mathbf{x}_{\star})$

  3. 3.

Compute stacked regression weights $w_{1},...,w_{k}$ using the full data set
and all cluster-level forests as predictors.

  4. 4.

Construct the ensemble-level prediction:
$\hat{Y}_{E}(\mathbf{x}_{\star})=\sum_{j=1}^{k}w_{j}\hat{Y}_{j}(\mathbf{x}_{\star})$

Algorithm 1 Cross-Cluster Weighed Forest (CCWF)

Classically, the Random Forest algorithm trains trees on bootstrap samples of
the full data set and weights tree-level predictions equally in the final
ensemble. Utilizing bootstrap samples of the same size as the original data
set has never been proven to be optimal - it is simply an approach that works
well heuristically. The introduction of a non-random data partitioning step in
the training of tree-based ensembles has the potential to produce further
improvements. Throughout our simulations, we utilize k-means clustering as our
choice of clustering algorithm in step (2), both for its advantageous and
well-characterized properties such as its ability to scale, computational
speed, convergence, and generalization to clusters of different shapes and
sizes, as well as for its relative simplicity and ease of interpretation
(Bottou and Bengio, 1995; Hartigan and Wong, 1979; Coates and Ng, 2012).
However, any clustering method may be substituted into the general framework.

The majority of previous work connecting Random Forest with clustering
utilizes Random Forest itself as a clustering method (Yan et al., 2013; Shi
and Horvath, 2006; Bicego, 2019). To our knowledge, the only manuscript which
displays conceptual similarity in broadly using the estimated clusters
produced by clustering algorithms to train ensemble learners including Random
Forest is Trivedi et al. (2015). They present a slightly different paradigm,
in which the prediction for a given test point involves first running k-means
for several different values of k, approximating the cluster from each
iteration that most closely fits the test point, and then ensembling learners
trained solely on these successive clusters in order to form the final
prediction. This requires assigning the test point to a cluster for each value
of k and then removing all other data points from consideration. Their results
showed statistically significant improvements in certain scenarios of their
ensembling approach using Random Forest over the use of a single forest,
highlighting the potential efficacy of using clustering algorithms to build
ensembles of forests.

To construct the ensembling weights in step (3) of our algorithm, we adapted
the multi-study version of the stacked regression method, an approach which by
construction rewards empirical cross-cluster generalizability of cluster-
specific learners (Patil and Parmigiani, 2018; Ramchandran et al., 2020).
Stacked regression forms linear combinations of multiple predictors that have
been shown to improve upon the performance of any single component (Breiman,
1996b). The weights given to each predictor are determined by cross-validation
and least squares regression with $L_{2}$-norm regularization. Our extension
of stacking to the multi-cluster context proceeds as follows: once the
training set is split up into $k$ clusters and forests are trained on each,
all forest-level predictions are then ’stacked’ into matrix
$\mathbf{T}=[\hat{Y}_{1}^{{}^{\prime}},...,\hat{Y}_{k}^{{}^{\prime}}]^{{}^{\prime}}$,
where
$\hat{Y}_{j}=[\hat{Y}_{1j}^{{}^{\prime}}...\hat{Y}_{kj}^{{}^{\prime}}]^{\prime}$
for $j=1,\ldots,k$; $\hat{Y}_{ij}$ is the column vector of predictions of
forest $j$ on cluster $i$, and $\hat{Y}_{j}$ is the stacked vector of all
predictions made by forest $j$ on every cluster within the training set.
$\mathbf{T}$ is $n\times k$, with $n$ equaling the total number of
observations in the training set. The observed outcomes are similarly
aggregated across the entire training data set into the $n\times 1$ vector
$Y$. $Y$ is then regressed against $T$ with a ridge penalization and non-
negativity constraint. The learner-specific weights $\mathbf{w}$ are
determined by solving
$\min_{\mathbf{w}}||(Y-(\mathbf{T}\times\mathbf{w}))||^{2}$ such that
$\mathbf{w}>0$ and $||\mathbf{w}||^{2}\leq\lambda$, where $\lambda$ is
optimized using the cross-validation procedure in the glmnet package in R
(Friedman et al., 2010).

In the following section, we assess the utility of the CCWF algorithm across
various scenarios of naturally clustered data which will be described in more
detail in Section 3. Throughout, the abbreviation Cluster will refer to
training an ensemble through the CCWF approach. We additionally compare the
performance of CCWF with other forest-based ensembles representing variations
from Algorithm 1, in order to isolate the effect of each step on conferring
advantages to the method. The moniker Random will indicate the ensemble formed
by using $k$ equally sized random partitions of the data set instead of
clusters. Multi will refer to training a forest on each of the true clusters
(which are known from the simulation) and combining again using stacking
weights to form the final ensemble. The name derives from its similarity to
the multi-study paradigm, in which learners are trained on known studies prior
to ensembling. Finally, the term Merged will refer to a single forest with
$k\times 100$ trees trained on the entire data set. We kept the number of
trees in the overall ensemble consistent across methods, so that the
comparison consists solely in how the training data is differently used to
train the trees in each ensemble. We set the number of trees per forest at 100
(other than the Merged, as previously described) based on the observation that
higher number of trees did not significantly improve the prediction results
nor alter the relationship between approaches.

##  3 Simulation Experiments: Strategies and Results

In this section, we use simulations to evaluate the overall behavior of the
ensembling approaches described above on a variety of data generating
mechanisms and outcome-covariate models that we would expect to find in real
genomic data sets. We begin by delineating our choice of data generation and
simulation strategy. We then discuss the importance of partitioning the data
prior to the training of each component forest, where we particularly focus on
the benefits of using clustering algorithms as opposed to other methods. We
then examine the robustness of CCWF to realistic data situations, in which we
vary parameters such as the signal to noise ratio, the number of true clusters
within the data set, and the total sample size. Finally, we examine the effect
of ensemble weighting strategy on overall performance, specifically
highlighting the advantages of stacked regression weights over simple
averaging as well as how the differences in the distribution of weights across
the ensembling methods correlate with prediction accuracy. Throughout this
section we report averages over 250 simulations per scenario, with 95%
confidence bands computed as mean $\pm 1.96\times$ standard error.

###  3.1 Generating multivariate clustered data

We utilized two primary approaches to generate the necessary clustered data
sets for simulations. The first was to draw clusters from multivariate
gaussian mixture models, in order to easily simulate many data sets with
analytically tractable characteristics. We used the implementation within the
R package clusterGeneration, with specific parameters outlined in the
following section (Qiu and Joe, 2020). However, a major disadvantage is that
simulated clusters using this setup may lack important features of the real
biologic clusters for which we would want the methods to work well in
practice.

Thus, we additionally considered simulating clustered data sets that more
closely mimic what we find in relevant applications. Specifically, we utilized
the method proposed by Waller et. al. and implemented in R via the ’monte’
function within the fungible package (Waller, 2020). Their algorithm produces
artificial plasmodes with clusters that resemble authentic clusters with
respect to important features such as cluster size, shape, and orientation.
Throughout, we evaluated our ensembling approaches on clusters generated using
both methods for a variety of different scenarios, as described below.

###  3.2 Simulation setup

For N = 250 iterations at each set of generative parameters, we simulated a
training data set with ntrain number of clusters and ntest test data sets with
two clusters using either the mixture model or the plasmode framework, all
with ncoef number of covariates. At baseline, ntrain = ntest = 5 and ncoef =
20. Per iteration, we then randomly sampled 10 of these covariates to create a
linear data-generating model with gaussian noise, with coefficients drawn
uniformly from $[-5,-0.5]\cup[0.5,5]$ so that each has a non-trivial
contribution to the outcome. We additionally considered non-linear
relationships of the covariates to the outcome, adding quadratic terms,
between-covariate interaction terms, and binarization of the covariates before
application of the linear model in order to form a step function. Since the
focus of our investigation is to evaluate the effect of feature distribution
heterogeneity on ensemble performance, we kept the covariate-outcome
relationship relatively similar across clusters, by drawing the amount of
coefficient perturbation between clusters uniformly from the interval [0,
.25]. The size of each cluster was set at baseline to be 500, resulting in
training data sets with 2500 total samples. Using either the clusterGeneration
or fungible packages, the level of between-cluster separation was set at
median values, resulting in clusters with some overlap that would still
ostensibly be distinguishable using clustering strategies. All modifications
to this baseline strategy are outlined in the relevant results sections.

###  3.3 The importance of data partitioning on prediction accuracy

We commence with an exploration of how the choice of data partitioning method
affects the accuracy of the resulting ensemble. The Cluster and Random methods
train the component forests on the same number of partitions - however, the
composition of each partition is what constitutes primary difference between
the two. The Multi uses a fixed number of partitions corresponding to the true
clusters, while the Merged simply uses the bootstrap sampling inherent to the
Random Forest algorithm to build each tree. We denote by $k$ the number of
partitions used to construct the Cluster and Random, and investigate
throughout this section the effect of increasing $k$ on the results from both
approaches. We additionally examine whether changes in the distribution of the
training clusters influences the accuracy and relationship between all four
methods considered.

Figure 1: Percent change in average RMSE of ensembling approaches (color
labeled) compared to the Merged across different data-generating scenarios, as
a function of $k$. The first row depicts results using the non-gaussian
cluster simulation approach, while the second row uses a gaussian data
generating model. (A.1-A.2) A linear model was used to generate the outcome
from the covariates. (B.1-B.2) The binary outcome was created by using a
cutoff from the linear model to create a binary step function. (C.1 - C.2)
Quadratic terms for two of the variables were added to the linear outcome-
generating model.

Figure 1 displays the result of varying both $k$ and the data-generating
mechanism for a total of 6 different scenarios. Overall, we observe that the
Cluster and Random improve upon ensembling using the true clusters or the
entire data set for values of $k$ higher than the true number of clusters. The
level of improvement that the $k$-dependent approaches show over the Merged
(and Multi) is substantial, typically over 20-30% at optimal values of $k$. On
the other hand, the Multi is inconsistent in its relationship to the Merged
over the scenarios considered, showing that the perhaps more intuitive
approach of splitting the data into its true clusters does not necessarily
produce the best results. The Cluster and Random are robust to changes in the
outcome model and the distribution of the data, as evidenced by the similar
performance patterns seen over three separate covariate-outcome relationships
and two covariate-generating mechanisms. Finally, in the majority of cases,
the Cluster exhibits superior accuracy over the Random, indicating that using
k-means is generally preferable over randomly partitioning the data. As $k$
increases, the accuracy of both the Cluster and Random improves until finally
plateauing, typically at the point in which each forest is being trained on
35-40 observations out of 2500 total. The average depth of the trees within
each forest decreases correspondingly with the sample size per partition. Tree
depth is a measure of model complexity, indicating that for high values of
$k$, we are essentially ensembling locally weak learners in an approach akin
to local smoothing.

The differences between the Cluster and Random ensembles reveal the particular
advantages of using clustering as the partitioning strategy. The data
partitioning step reduces the range of the covariate space used to train each
tree, and we see throughout Figure 1 that in general, increasing the number of
partitions improves the performance of the ensemble. This additionally
supports the observation that for $k$ smaller than the number of true
clusters, the Multi approach typically outperforms the Random and Cluster. The
manner and efficacy to which this partitioning step is performed correlates
with the prediction performance. In fact, when examining the association of
prediction accuracy to the range reduction of the covariate space induced by
$k$, we find a clear relationship; at the optimal value of $k$, randomly
partitioning results in a slightly higher average covariate range than k-means
(4.22 CI:(4.13, 4.31) and 3.17 CI:(3.08, 3.26) respectively), while the true
clusters have a significantly larger average range (7.33 CI:(7.21, 7.45)).
Furthermore, using k-means (or other clustering methods) results in not only
greater range reduction but also greater between-cluster separation than by
random partitions of the data, which additionally contributes to the better
performance of the Cluster over the Random. We conjecture that clustering may
allow the learners to be uncorrelated and dissimilar, a property that has been
previously shown to be effective in other boosting and sequential ensembling
approaches (Friedman, 2002; Schapire, 2003).

It is additionally important to note that projecting the distinct partitions
produced by clustering onto the 1-dimensional space corresponding to a single
variable results in ranges that are highly overlapping between clusters. This
mitigates perhaps the greatest downside to the clustering strategy of
minimizing the euclidean distance, as Random Forest lacks the ability to
extrapolate beyond the data it is trained on. However, the marginal degree of
range restriction for a single variable is highly variable across clusters,
allowing in aggregate over all cluster-level forests for total coverage of not
only the complete range but also fluctuating relationships between covariates
in different regions of the joint distribution. While this may present a
slight trade-off by sacrificing accuracy in the areas of highest density for
better precision in the tails, the loss is not significant enough to
counteract the benefits.

We next examine the implications of these improvements on our understanding of
the classic Random Forest algorithm. Each tree within a Random Forest is
trained by formulating successive partitions of the bootstrapped covariate
space that maximize the amount of information gained by each split. A
prediction for a new point is then obtained by averaging the outcomes of the
training data in the same leaf node as the test point across all trees. In
this way, the Random Forest algorithm can itself be seen as an combination of
learners across different partitions of the data - however, since the
available training data for each learner is simply obtained by bootstrapping,
the partitions influencing final predictions are not achieved by optimization
of any criterion, simply randomness.

While this is often a very powerful paradigm, the improvement of the Cluster
over the Merged indicates that in the presence of heterogeneity in the
training data, more useful data partitions for the training of each component
tree in the overall ensemble can be obtained. Clustering the data prior to
training restricts each forest to a smaller subset of the covariate space. We
imagine that, as a result, these forests can more effectively form splits to
learn nuances in this restricted space that would not be possible were the
same number of total trees trained on the entire data set. Insight into this
phenomenon may be gleaned by considering how the bootstrap samples of the data
used to train each tree are differentially obtained between the two
approaches. A forest trained on a partition mostly comprising of observations
from one of the tails of the joint distribution of the covariates will consist
of trees trained on bootstrap samples of solely these observations, resulting
in a far greater representation of these observations within trees in the
overall ensemble than in trees trained on bootstrap samples of the merged data
set. Supplementary Figure S1 displays the differences in how the Cluster and
Merged learn the underlying covariate-outcome function when there is only one
covariate. The Cluster more effectively learns the function at the tails of
the covariate distribution whereas the Merged primarily learns the highest
density regions.

Interestingly, the estimated clusters produced by k-means do not correspond
with the true clusters for either simulation strategy, or when tested on real
genomic clustered data. This holds even when the value of $k$ is equal to the
number of true clusters. Furthermore, across all values of $k$ considered,
each estimated cluster typically ($>$ 90% of time) contains observations from
all of the true clusters - as $k$ increases, the total number of observations
from each true cluster simply decreases, but there is still full
representation. This further supports that the true clusters are not
necessarily the best partitions of the data for building ensembles of forests,
and that in fact the k-means algorithmic strategy of producing estimated
clusters with minimal variance within and maximal variance across is more
effective at capturing feature distribution heterogeneity regardless of
underlying data structure.

Silhouette analysis is often utilized to choose the optimal value of $k$ for
k-means; the $k$ that maximizes the silhouette score is the one for which the
ratio of between-cluster variance to within-cluster variance is maximized. We
found that the Cluster strategy was more effective than the Merged only if
silhouette analysis indicated that the optimal $k$ was greater than 1,
regardless of whether the data set was actually constructed to contain true
clusters or not. Furthermore, even when varying the separation of the
simulated true clusters from mostly overlapping to completely separated, we
found no exceptions to this observation. Of course, we do remark that in the
case of having completely separated true clusters, it is unlikely that
silhouette analysis will indicate the optimal number of partitions to be equal
to 1, but regardless, the algorithmic clusterability of the data set is the
criterion that matters in determining the efficacy of the Cluster approach,
not necessarily the presence of true clusters. We found similar results when
applying other clustering methods instead of k-means; in general, the
unsupervised clustering step seems to provide a good metric of whether the
supervised ensembling step will perform well.

###  3.4 Robustness of method to realistic data set variety

Our next simulations evaluate the robustness of these methods to a range of
data set characteristics likely to be encountered in real data, while holding
the value of $k$ constant at the optimal value chosen by Silhouette analysis.
We determine how prediction performance is affected by the number of true
clusters in training, the sample size of the training set, and the strength of
the relationship between the covariates and the outcome. For this set of
simulations, we are showing results using a quadratic outcome model and the
non-gaussian cluster generating setup, to present a fairly complex and
realistic data paradigm.

Figure 2: Percent change in average RMSE of ensembling approaches (color
labeled) compared to the Merged across different data-generating scenarios.
All simulations used a quadratic outcome and the non-gaussian cluster
generating algorithm. (a) Varying the magnitude of the coefficients in the
outcome-generating model to determine the effect of signal strength on
prediction accuracy gains. b Varying the number of true clusters within the
training set, while keeping the total sample size constant at 2500. (c)
Varying the sample size per cluster, while keeping the total number of
clusters per data set constant at 5.

In the simulation used to generate Figure 2a, the norm of the coefficients
relating the covariates to the outcome in the quadratic model was varied in
order to determine how well the approaches picked up signals of different
strength from the covariates. We see that as the signal from the covariates
increases, the magnitude of the change in performance from all ensembling
methods over the Merged expands significantly, eventually plateauing at rates
of improvement ranging from 20-60%. The Cluster is by far the most effective
learner, especially at higher coefficient norms, while the Random displays a
similar trajectory of improvement, albeit significantly lower than that of the
Cluster. Furthermore, the much smaller improvement of the Multi over the
Merged indicates that increasing the signal does not necessarily signify that
the true clusters will more effectively pick it up. This supports the
conjecture that reducing the amount of within-cluster variation (as most
effectively accomplished by clustering for optimally high values of $k$)
allows for more accurate discernment of the covariate-outcome relationship,
since it is no longer being obscured by heterogeneity in the distribution of
the covariates.

Next, we evaluated the effect of varying the number of true clusters within
each training data set while keeping the total sample size constant at 2500.
Figure 2b demonstrates that as the number of true clusters increases, the
Cluster method remains significantly better than all other approaches, while
the performance of the Multi catches up to that of the Random at higher
numbers of true clusters. This latter phenomenon makes sense in light of what
we have seen before, as the number of true clusters determines the number of
forests in the Multi. As illustrated in Figure 1, increasing the amount of
ensemble members through $k$ augments prediction accuracy, whereas the Multi
was set in those simulations to always include only 5 component forests.
However, when the number of true clusters is high, the difference between the
Cluster and the other two ensembles remains constant, highlighting that the
benefits of clustering persist no matter the true composition of the data.

In the simulation used to generate Figure 2c, we set the number of clusters
per training set constant at 5 while increasing the sample size of each data
set incrementally from 500-5000. We observe that as the sample size of each
cluster grows, the difference in prediction accuracy between the Cluster and
the other approaches similarly increases. On the other hand, the Multi becomes
less distinguishable from the Merged, primarily because the accuracy of the
Merged increases proportionally to the sample size. However, the Cluster and
Random methods continue to improve upon the Merged regardless of the sample
size, since the optimal number of partitions $k$ increases proportionally with
the sample size.

###  3.5 Importance of ensemble weighting strategy on performance

Value of k | 2 | 5 | 10 | 20 | 30 | 50 | 70 | 80
---|---|---|---|---|---|---|---|---
Simple Averaging | 11.93 | 29.97 | 43.60 | 59.63 | 68.17 | 79.09 | 86.65 | 89.48
Stacked Regression | -9.88 | -13.35 | -19.73 | -25.21 | -27.73 | -29.99 | -31.96 | -32.27
Table 1: Simple averaging vs. stacked regression weights. Percent change in
average RMSE compared to Merged for differently-weighted ensembles built on
estimated clusters determined by k-means, using the non-gaussian data set
simulation framework. The first row shows results for weighting each forest
equally and the second row depicts results from using stacked regression
weights.

Our next simulations explore the importance of the ensemble weighting strategy
on prediction performance. We specifically highlight the importance of using
stacked regression weights over equally weighting component learners or other
weighting approaches. The weighting step is critical in determining the
success of the ensemble; for instance, if we weight the cluster-trained
forests either equally or by the inverse of their sample sizes, the resulting
ensembles often perform similarly to the Merged, or worse. Table 1 displays
the percent change in average RMSE from the Merged achieved by ensembles of
forests trained on clusters determined by k-means and then combined using
either simple averaging or stacked regression weights. We observe that as $k$
increases to its optimally performing value of around 80, the stacking
approach continuously advances while equal weighting progressively declines.
In fact, the best number of partitions for equal weighting is 2, at which
point it is still significantly worse than merging.

This phenomenon indicates that not all estimated clusters from k-means produce
accurate predictors, but certain subsets in combination produce better
predictions than the Merged. Stacking, in the multi-study implementation we
use wherein studies are replaced by estimated clusters, up-weights the
cluster-specific learners that show the best cross-cluster prediction ability,
which likely translates into accuracy and robustness to test data. If a
cluster-level forest is performing well on other clusters, this indicates that
it has more effectively learned the true covariate-outcome rule and is
generalizable to training data dissimilar from its own, as clusters are
designed to have as much distributional separation as possible.

Figure 3 displays the distribution of stacking weights across ensembling
methods using the gaussian cluster-generating framework and a linear outcome
model, although results for the non-gaussian simulations were virtually
indistinguishable. For each simulated data set, we isolated the highest
weight, and show the distribution in green. The Cluster method most strongly
up-weights beneficial predictors out of all approaches considered,
demonstrating the presence of certain estimated clusters that produce highly
generalizable forests. Furthermore, we see significantly more upweighting in
the Cluster approach as compared to the Random, whose stacking weights all
occupy a more restricted, and lower, range. Forests within the Random approach
perform more similarly to each other, which stands to reason given that all
the partitions used to train each forest are randomly selected. The benefit of
clustering thus lies in producing more optimal partitions for training,
although we note through both the results from equally weighting all
predictors as well as the distributional range of stacking weights that only
some of these clusters produce predictors that are decidedly effective. To the
latter point, the Cluster produces a larger difference between the most up-
weighted forests and the weights given to the rest of the ensemble members
than the Random. The Multi displays the smallest difference of the three,
which also corresponds with its worse performance.

Figure 3: Distribution of the ensemble weights determined by stacked
regression for (a) the Cluster, (b) the Random, and ((c)) the Multi for $k$ =
20 and $k$ = 80 for the first two methods, and 5 true clusters for the latter.
We used the gaussian cluster-generation framework. The distribution of the
largest weight per ensemble is depicted in green, while the rest of the
weights are visualized in purple. Results are shown over 100 iterations at
each value of $k$.

Stacking can be considered as an indirect form of reweighting the influence of
each training data point on the overall predictor. The clustering step helps
the stacking algorithm to identify the most effective points to up-weight by
grouping together data that are likely to exert a similar influence. Points
belonging to estimated clusters that are not weighted as highly are still
necessary for the overall ensemble function, and should not be removed
entirely - when we regularized the stacked regression weights using Lasso
instead of Ridge, the effect was to zero out the contributions of several
clusters per ensemble. The resulting predictors were far more variable in
their accuracy across data-generating scenarios, likely because they adapted
too strongly to random noise in the training data. Thus, stacking with a Ridge
constraint gives more importance to data points in heavily up-weighted
estimated clusters while still recognizing the necessity of the the remaining
clusters to prevent overfitting and promote generalizability.

###  3.6 Extension to Multiple Studies

We next examine whether we can more optimally partition the total amount of
data when multiple studies are available for training that measure the same
covariates and outcome variable. The traditional multi-study ensembling
paradigm is to train a single learner on each study and combine using some
weighting strategy, such as stacking (Patil and Parmigiani, 2018). This is
analogous to the Multi method when we are able to separate data into its true
clusters. We now explore whether training k-means on the the merged data
(comprising covariate data from all of the training studies) produces
improvements comparatively to the single data set setting. We furthermore
evaluate the performance of these approaches on real gene expression data from
the CuratedOvarianData repository from Bioconductor in R (Ganzfried et al.,
2013). Finally, we explore whether the general strategy of ensembling learners
built on clusters also works for Neural Nets, and compare the results to those
when using Random Forest.

Figure 4: Average RMSE’s of ensembling approaches (color labeled) as a
function of $k$ in the multi-study analysis. 20 total covariates, 10
associated with the outcome; nonlinear outcome model of the form
$y=\beta^{T}\mathbf{X}+4.4x_{1}-1.8x_{2}+10\sin(10\pi x_{1})$.

The general framework for the simulation setup used to create Figure 4 was
drawn from Ramchandran et al. (2020) and Patil and Parmigiani (2018).
CuratedOvarianData provides data for gene expression meta-analysis of patients
with ovarian cancer. In this study, we used all 15 studies in
CuratedOvarianData that include survival information without any missing data
in the features. For N = 250 iterations per value of $k$, we randomly
separated the 15 data sets from CuratedOvarianData into 10 training and 5
validation sets. We then generated the outcome using the non-linear model
defined in the caption in order to test the ability of the candidate learning
methods to detect more difficult covariate-outcome relationships. We simulated
baseline levels of coefficient perturbation per study as described in the
Methods section. Using either Random Forest or Neural Nets as the base
learner, we then constructed ensembles using the four main approaches compared
throughout this paper: the Merged, Cluster, Random, and Multi. The Multi in
this case trained a learner on each study to form the final ensemble. All
ensembles were built using stacked regression weights with a ridge constraint.

We commence by discussing the performance of the Random Forest-based
ensembles. In the multi-study setting, ensembling based on k-means clustering
is significantly more accurate than ensembling based on study membership for
values of $k$ higher than the number of training studies. The relationship
between approaches follows exactly the pattern we observe in the single
clustered data set setting, with the Cluster performing the most effectively,
the Random performing on a similar trajectory but less accurately, the Multi
at a yet worse accuracy, and the Merged by far the worst of the four. The true
cluster or study-membership does not represent the most effective data
partitioning for Random Forest; again, it is more effective to partition the
data based on minimizing within-cluster heterogeneity and maximizing across.
Overall, these results suggest that for either single or multiple training
data sets, the Cluster approach should be used for Random Forest learners.

We next investigate the use of Neural Nets in this setting and compare the
results with Random forest. Figure 4 displays that the Cluster method with
Neural Net component learners improves upon training each learner on the true
studies or training a single Neural Net on the merged data. The level of
improvement of the Cluster over the Merged is less than that for Random
Forest, as the Neural Net Merged learner is significantly more accurate than
its Random Forest counterpart. Interestingly, the Cluster method produces
almost identically accurate predictors using either algorithm as its base
learner. While the traditional multi-study paradigm, or simply training a
single learner on the merged data ,favors Neural Nets, Random Forest catches
up when trained on estimated clusters. This displays the magnitude of
improvement capable by the cluster-based ensemble framework when applied to
Random Forest, and further highlights that even when the traditional Random
Forest algorithm falls short of the performance of a Neural Network, the
Cluster approach can elevate the two to the same level. In our view, the
Cluster strategy can itself be compared to a convolutional neural network, in
which the estimated clusters represent the convolutional neighborhoods and the
stacking weights delineate the relationship between the tree-learning and the
stacking layers. While this analogy does not numerically explain the similar
performance of the two algorithms, it provides intuitive insight into why we
may be seeing these results.

###  3.7 Examining the bias and variance

Figure 5: Squared bias, variance, and MSE of the Merged and Multi learners
(color coded) for data sets ranging from 500-5000 total samples (100-1000
samples per cluster). data sets were generated using the gaussian cluster
framework. Each panel shows the breakdown for a given sample size.

Our final set of simulations decomposes the MSE of the Merged and the Multi
into its bias and variance terms in order to individually characterize each.
As the sample size of the training data set increases, the MSE of both
ensembling approaches decreases as expected. However, there is a far more
minimal decrease from $n=2500$ to 5000 than from 500 to 2500, indicating that
there is an asymptotic limit to how well the ensembles can perform. Most
strikingly, across all sample sizes considered, the squared bias term
comprises almost the entirety of the MSE while the variance contribution is
minimal. All improvements in performance from the Multi over the Merged arise
through decreasing the bias of the ensemble - in fact, the variance slightly
increases between the two but is proportionally so small as to have almost no
effect. We limit our comparison to the Multi, since this ensemble construction
method can be analytically characterized much more easily than the Cluster or
Random. These results indicate that the squared bias defines the overall
performance of the Multi ensemble, and that any theoretical analysis of this
general approach can be largely restricted to the bias term.

##  4 Data Application

To explore the performance of these classifiers on real biological data with
natural feature-outcome relationships, we considered data sets measuring gene
expression and clinical data which plausibly contain clusters.

###  4.1 Low Grade Gliomas (LGG) data set

We applied our methods to the Low Grade Gliomas (LGG) data set within TCGA
Network (2015). We considered one binary and one continuous outcome in order
to assess the performance of the ensembles on both regression and
classification tasks. The binary variable measures the tumor grade, in which
the value $0$ indicates the presence of a low-grade tumor (WHO grades II and
III) and the value $1$ indicates high-grade glioblastoma (WHO grade IV). The
continuous outcome measures the number of mutations present in each sample.
These two outcomes were chosen for their importance in diagnosing and
informing the treatment of glioblastoma. We additionally experimented with two
different sets of covariate data. The first represents molecular profiling
data (for example, measures such as percent aneuploidy and TERT expression),
in which there are 513 total patients and 50 covariates. Twelve of the
covariates are continuous while the remainder are categorical; missing
covariate data was imputed using Random Forest through the mice package in R
prior to training. We additionally considered gene expression data for the
same set of patients; the original authors previously demonstrated that the
patient gene expression data is separable into clusters using unsupervised
approaches, so we included this information in our analysis to determine the
efficacy of the clusters they discovered within our ensembling paradigm
Network (2015).

We used a variable selection and clustering strategy based on mclust (as
implemented in the vscc package in R) for the molecular profiling covariate
data (Andrews and McNicholas, 2013; Scrucca et al., 2016). We chose this
method over k-means due to its simultaneous ability to perform variable
selection and cluster the data without prior specification of the number of
estimated clusters $k$, and to additionally display that the general cluster-
weighted ensemble framework is robust to the choice of clustering method.
mclust performs model-based clustering using Gaussian mixture models, in which
the variable selection step chooses the candidate variables that result in
estimated clusters with the highest likelihood. The algorithm can handle only
uniform data types, so we chose to solely restrict the clustering and variable
selection steps to the 12 continuous covariates. No further clustering was
necessary for the gene expression data beyond the analysis previously
conducted by the original authors. Per iteration, we randomly chose 100 sample
points for the test set and used the rest of the samples for training,
ultimately repeating this procedure 500 times to obtain the distributional and
median measures of prediction accuracy.

For this analysis, we consider modified definitions for the ensembling
strategies we have investigated thus far. The Merged will have the same
definition as before, representing a single forest with 500 trees trained on
the entire data set with all 50 covariates. We denote by Subset Merged a
single forest with 500 trees trained only on the 12 continuous candidate
variables used to cluster the molecular profiling data. Sample Weighted
indicates first clustering the training data using the vscc package, training
a forest with 100 trees on each estimated cluster, and ensembling using
weights proportional to the sample size of each cluster. While clusters are
determined by only considering the continuous covariates, all 50 covariates
are utilized during training. Finally, Stack Ridge represents the clustering
and ensembling method described above, instead employing stacked regression
weights with a ridge constraint to form the final predictor.

###  4.2 Results

Figure 6: Percent change in average RMSE of ensembling approaches (color
labeled) compared to the Merged across two sets of covariate data and two
different outcomes from the LGG study. The top row shows 500 iterations of
splitting the available data into training and test sets, while the second row
displays the median over all iterations with associated confidence intervals.
(a) Molecular profiling covariate data, mutation count outcome. (b.1) Gene
expression covariate data, mutation count outcome. (b.2) Gene expression
covariate data, tumor grade outcome.

Overall, Figure 6 illustrates that across combinations of both types of
covariate data and outcomes, the clustering methods demonstrate remarkable
improvement over the merging approaches. Figure 6A shows the results from the
molecular profiling data, in which we observe a median improvement of 20-30%
of the ensembles over the merged-based learners when predicting the number of
mutations. This suggests that the variable selection and model-based
clustering strategy in vscc is highly effective for construction of ensembles.
Figure 6b highlights the impressive improvement of the Sample Weighted and
Stack Ridge over the Merged when considering gene expression data and the
continuous mutation count outcome, with median percent changes around or above
75%. As described in the original paper, the clusters previously determined
through nonparametric methods were found to align with biologically relevant
characteristics. Interestingly, we observe across all scenarios that while the
Stack Ridge does improve upon the Sample Weighted, the difference between the
two is fairly marginal. This indicates that there are settings in which the
choice of weighting scheme does not drastically influence the prediction
ability of the ensemble; however, even in this case, the Stack Ridge proves
optimal, albeit by a lower margin than in many of our other demonstrative
examples. In 6c, we see a greater improvement from the Stack Ridge compared to
the Sample Weighted, with decreased but still significant improvements over
the Merged. Overall, these results display the remarkable robustness of the
CCWF method to different types of outcomes, covariate data, and cluster
construction, all in a real-life biological setting.

Finally, we explore the influence of variable importance measures on ensemble
performance. We posit that the CCWF approach is most effective when the
variables that determine the estimated clusters are not also the variables
most associated with the outcome. Otherwise, the true outcome model is
obscured within each cluster and the cluster-specific learners are unlikely to
generalize. We additionally experimented with simply training ensembles on the
variables chosen to cluster over within the molecular profiling data, and
obtained significantly worse overall performance. To check whether our
conjecture holds within this data set, we examined the variable importance
rankings from the Subset Merged of the variables chosen for clustering. The
Random Forest algorithm computes the importance of each variable in accurately
predicting the outcome in training; these metrics can be used to categorize
which variables are most essential and those that have less influence (Archer
and Kimes, 2008). We found that there was a median of 20% overlap between the
two sets (IQR 0 - 33%); this fairly low level supports our hypothesis that the
variables determining the clustering are not important in predicting the
outcome within this particular data set. These findings provide insight into
when ensembling over estimated clusters is a better strategy over merging.

##  5 Conclusions

We have demonstrated how to build CCFWs —ensembles of forests trained on
estimated clusters, that result in more generalizable and accurate predictors.
We improve upon the traditional Random Forest algorithm with equivalent total
numbers of trees when the data is naturally divisible by clustering methods.
Biological data sets (among several other applications) often contain
heterogeneity in the distribution of the features, so we posit that the CCWF
method may be potentially useful across a variety of settings. Based on the
experiments and data considered here, CCFW are robust to changes in the
covariate-outcome model as well as the structure and distribution of the
covariate data. We additionally illustrate the utility of this general
framework when dealing with multiple data sets, in which we improve upon the
existing multi-study ensemble paradigm.

The results of our simulations and data application provide insight into the
importance of data partitioning for the Random Forest algorithm. We have
demonstrated that it is not necessarily optimal to divide the data based on
true clusters (or studies, for the multi-study setting) even if those were
known. Interestingly, reducing within-cluster covariate heterogeneity appears
to encourage the forest to learn the covariate-outcome rule more efficiently
even though the sample size may be greatly reduced from that of the total data
set. Finally, we have established that a CCWF has the ability to equal the
performance of a complex Neural Net even when the traditional Random Forest
algorithm does not.

The utility of weighting each individual tree in the ensembles through
stacking weights instead of simply weighting the forests should be explored
further. Ramchandran et al. (2020) have previously demonstrated this paradigm
to be highly effective within the multi-study setting, and Supplementary
Figure S2 shows that similarly, weighting trees within ensembles built on
clusters produces improvements compared to the strategy of weighting forests.
We chose to limit the scope of the analysis within this paper to weighting
forests in order to characterize the general strategy more clearly and present
an analytically tractable and generalizable framework. However, future work
could more closely examine the effect of individually weighting the trees
within ensembles and determine whether the conclusions of this paper still
hold in that case.

In subsequent work, Ramchandran and Mukherjee (2021) have begun to investigate
the theoretical underpinnings of the success of this approach by analyzing
linear least squares and random forest regressions under a linear model. Their
primary findings demonstrate that the benefit of ensembling over merging is to
reduce the squared bias of the resulting predictor, confirming the empirical
result from Figure 5. Furthermore, for Random Forest regression under fixed
dimensional linear models, they provide upper bounds on the mean square error
that imply a strict improvement of ensembling over merging. Overall, the
theoretical insights from their work support the conclusions of this paper.

Random forests are among the most widely used methods in machine learning,
with thousands of published applications across a broad spectrum of data
types. We identified a simple method to provide substantial improvements. We
hope that in the immediate term, this will contribute to concrete advances in
predictive ability in many areas. We also believe that the architecture we
outlined, as a general concept, may provide further important insight into
combining simple and successful machine learning tools via multi-layer
approaches.

Acknowledgments

We thank Prasad Patil for his valuable insights and suggestions. Maya
Ramchandran was supported by NIH-NCI Training Grant T32CA009337, NSF-
DMS1810829, and NSF-DMS2113707. Rajarshi Mukherjee was partially supported by
NSF Grant EAGER-1941419. Giovanni Parmigiani was supported by NSF-DMS1810829,
and NSF-DMS2113707.

## References

  * Andrews and McNicholas (2013) J. L. Andrews and P. D. McNicholas.  Variable selection for clustering and classification.  _arXiv preprint_ , 2013.  arXiv:1303.5294.
  * Archer and Kimes (2008) K. J. Archer and R. V. Kimes.  Empirical characterization of random forest variable importance measures.  _Computational Statistics and Data Analysis_ , 52(4):2249–2260, 2008.  ISSN 0167-9473.  doi: https://doi.org/10.1016/j.csda.2007.08.015.  URL https://www.sciencedirect.com/science/article/pii/S0167947307003076.
  * Biau and Scornet (2016) G. Biau and E. Scornet.  A random forest guided tour.  _TEST_ , 25(2):197–227, Oct. 2016.  doi: 10.1007/s11749-016-0481-7.  URL https://doi.org/10.1007/s11749-016-0481-7.
  * Bicego (2019) M. Bicego.  K-random forests: a k-means style algorithm for random forest clustering.  In _2019 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8, 2019.  doi: 10.1109/IJCNN.2019.8851820.
  * Bottou and Bengio (1995) L. Bottou and Y. Bengio.  Convergence properties of the k-means algorithms.  In _Advances in Neural Information Processing Systems_ , pages 585–592, 1995.
  * Bouwmeester et al. (2013) W. Bouwmeester, J. W. R. Twisk, T. H. Kappen, W. A. van Klei, K. G. M. Moons, and Y. Vergouwe.  Prediction models for clustered data: comparison of a random intercept and standard regression model.  _BMC Medical Research Methodology_ , 13, 2013.  doi: 10.1186/1471-2288-13-19.
  * Breiman (1996a) L. Breiman.  Bagging predictors.  _Machine Learning_ , 24(2):123–140, Aug. 1996a.  ISSN 0885-6125.  doi: 10.1023/A:1018054314350.  URL http://dx.doi.org/10.1023/A:1018054314350.
  * Breiman (1996b) L. Breiman.  Stacked regressions.  _Machine Learning_ , 24(1):49–64, July 1996b.  ISSN 0885-6125.  doi: 10.1023/A:1018046112532.  URL http://dx.doi.org/10.1023/A:1018046112532.
  * Breiman (2001) L. Breiman.  Random forests.  _Machine Learning_ , 45(1):5–32, Oct. 2001.  ISSN 0885-6125.  doi: 10.1023/A:1010933404324.  URL https://doi.org/10.1023/A:1010933404324.
  * Chauhan et al. (2010) R. Chauhan, H. Kaur, and M. A. Alam.  Data clustering method for discovering clusters in spatial cancer databases.  _International Journal of Computer Applications_ , 10(6):9–14, 2010.
  * Coates and Ng (2012) A. Coates and A. Y. Ng.  Learning feature representations with k-means.  In _Neural Networks: Tricks of the Trade_ , pages 561–580. Springer, 2012.
  * Deodhar and Ghosh (2007) M. Deodhar and J. Ghosh.  A framework for simultaneous co-clustering and learning from complex data.  In _Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’07, page 250–259, New York, NY, USA, 2007. Association for Computing Machinery.  ISBN 9781595936097.  doi: 10.1145/1281192.1281222.  URL https://doi.org/10.1145/1281192.1281222.
  * Dietterich (2000) T. G. Dietterich.  Ensemble methods in machine learning.  In _Proceedings of the First International Workshop on Multiple Classifier Systems_ , MCS ’00, page 1–15, Berlin, Heidelberg, 2000. Springer-Verlag.  ISBN 3540677046.
  * Friedman et al. (2010) J. Friedman, T. Hastie, and R. Tibshirani.  Regularization paths for generalized linear models via coordinate descent.  _Journal of Statistical Software_ , 33(1):1–22, 2010.  URL http://www.jstatsoft.org/v33/i01/.
  * Friedman (2002) J. H. Friedman.  Stochastic gradient boosting.  _Computational Statistics and Data Analysis_ , 38(4):367–378, 2002.  ISSN 0167-9473.  doi: https://doi.org/10.1016/S0167-9473(01)00065-2.  URL https://www.sciencedirect.com/science/article/pii/S0167947301000652.  Nonlinear Methods and Data Mining.
  * Ganzfried et al. (2013) B. F. Ganzfried, M. Riester, B. Haibe-Kains, T. Risch, S. Tyekucheva, I. Jazic, X. V. Wang, M. Ahmadifar, M. J. Birrer, G. Parmigiani, C. Huttenhower, and L. Waldron.  curatedOvarianData: clinically annotated data for the ovarian cancer transcriptome.  _Database (Oxford)_ , 2013:bat013, 2013.  doi: 10.1093/database/bat013.  URL http://dx.doi.org/10.1093/database/bat013.  PMCID: PMC3625954.
  * Goh et al. (2017) W. W. B. Goh, W. Wang, and L. Wong.  Why batch effects matter in omics data, and how to avoid them.  _Trends in Biotechnology_ , 35(6):498–507, 2017\.
  * Hartigan and Wong (1979) J. A. Hartigan and M. A. Wong.  Algorithm as 136: A k-means clustering algorithm.  _Journal of the Royal Statistical Society, Series C (Applied Statistics)_ , 28(1):100–108, 1979.
  * Liaw and Wiener (2002) A. Liaw and M. Wiener.  Classification and regression by randomforest.  _R News_ , 2(3):18–22, 2002.  URL https://CRAN.R-project.org/doc/Rnews/.
  * Luo and Schumacher (2010) J. Luo and M. e. a. Schumacher.  A comparison of batch effect removal methods for enhancement of prediction performance using maqc-ii microarray gene expression data.  _The Pharmacogenomics Journal_ , 10(4):278–291, 2010.  URL http://jmlr.org/papers/v13/biau12a.html,doi={10.1038/tpj.2010.57}.
  * Network (2015) C. G. A. R. Network.  Comprehensive, integrative genomic analysis of diffuse lower-grade gliomas.  _The New England journal of medicine_ , 372(26):2481–2498, 2015.  doi: 10.1056/NEJMoa1402121.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4530011/.
  * Patil and Parmigiani (2018) P. Patil and G. Parmigiani.  Training replicable predictors in multiple studies.  _Proceedings of the National Academy of Sciences_ , 115(11):2578–2583, 2018.  ISSN 0027-8424.  doi: 10.1073/pnas.1708283115.  URL https://www.pnas.org/content/115/11/2578.
  * Qiu and Joe (2020) W. Qiu and H. Joe.  clustergeneration: Random cluster generation (with specified degree of separation).  _R_ , 2020.  URL https://CRAN.R-project.org/package=clusterGeneration.
  * Ramchandran and Mukherjee (2021) M. Ramchandran and R. Mukherjee.  On ensembling vs merging: Least squares and random forests under covariate shift, 2021.
  * Ramchandran et al. (2020) M. Ramchandran, P. Patil, and G. Parmigiani.  Tree-weighting for multi-study ensemble learners.  _Pacific Symposium on Biocomputing_ , 25:451–462, 2020\.  doi: https://doi.org/10.1101/698779.
  * Schapire (2003) R. Schapire.  The boosting approach to machine learning: An overview.  _Nonlinear Estimation and Classification. Lecture Notes in Statistics_ , 171:5–32, 2003.  doi: https://doi.org/10.1007/978-0-387-21579-2_9.
  * Scrucca et al. (2016) L. Scrucca, M. Fop, T. B. Murphy, and A. E. Raftery.  mclust 5: clustering, classification and density estimation using Gaussian finite mixture models.  _The R Journal_ , 8(1):289–317, 2016.  URL https://doi.org/10.32614/RJ-2016-021.
  * Sharkey (1996) A. J. C. Sharkey.  On combining artificial neural nets.  _Connection Science_ , 8(3-4):299–314, 1996.  doi: 10.1080/095400996116785.  URL https://doi.org/10.1080/095400996116785.
  * Shi and Horvath (2006) T. Shi and S. Horvath.  Unsupervised learning with random forest predictors.  _Journal of Computational and Graphical Statistics_ , 15(1):118–138, 2006.  doi: 10.1198/106186006X94072.  URL https://doi.org/10.1198/106186006X94072.
  * Sugiyama et al. (2007) M. Sugiyama, M. Krauledat, and K.-R. Müller.  Covariate shift adaptation by importance weighted cross validation.  _Journal of Machine Learning Research_ , 8(5), 2007.
  * Trivedi et al. (2015) S. Trivedi, Z. A. Pardos, and N. T. Heffernan.  The utility of clustering in prediction tasks.  _CoRR_ , abs/1509.06163, 2015.  URL http://arxiv.org/abs/1509.06163.
  * Verbeke and Lesaffre (1996) G. Verbeke and E. Lesaffre.  A linear mixed-effects model with heterogeneity in the random-effects population.  _Journal of the American Statistical Association_ , 91(433):217–221, 1996.
  * Waller (2020) N. G. Waller.  _fungible: Psychometric Functions from the Waller Lab._ , 2020.  version 1.95.4.8.
  * Yan et al. (2013) D. Yan, A. Chen, and M. I. Jordan.  Cluster forests.  _Computational Statistics and Data Analysis_ , 66:178–192, 2013.  ISSN 0167-9473.  doi: https://doi.org/10.1016/j.csda.2013.04.010.  URL https://www.sciencedirect.com/science/article/pii/S0167947313001400.