# LightAutoML: AutoML Solution for a Large Financial Services Ecosystem

Anton Vakhrushev  Sber AI Lab<EMAIL_ADDRESS>,  Alexander Ryzhkov
Sber AI Lab<EMAIL_ADDRESS>,  Maxim Savchenko  Sber AI Lab
<EMAIL_ADDRESS>,  Dmitry Simakov  Sber AI Lab<EMAIL_ADDRESS>,  Rinchin Damdinov  Sber AI Lab<EMAIL_ADDRESS>and  Alexander
Tuzhilin  Stern School of Business, NYU<EMAIL_ADDRESS>
###### Abstract.

We present an AutoML system called LightAutoML developed for a large European
financial services company and its ecosystem satisfying the set of
idiosyncratic requirements that this ecosystem has for AutoML solutions. Our
framework was piloted and deployed in numerous applications and performed at
the level of experienced data scientists while building high-quality ML models
significantly faster than these data scientists. We also compare the
performance of our system with various general-purpose open source AutoML
solutions and show that it performs better for most of the ecosystem and
OpenML problems. We also present the lessons we learned while developing the
AutoML system and moving it into production.

AutoML, financial modeling, banking applications, deployment

##  1\. Introduction

AutoML has attracted much attention over the last few years, both in the
industry and academia (Guyon et al., 2019). In particular, several companies,
such as H2O.ai111https://www.h2o.ai/, DataRobot 222https://www.datarobot.com/,
DarwinAI 333https://darwinai.ca/, and OneClick.ai444https://www.oneclick.ai/,
and existing AutoML libraries, such as AutoWeka (Thornton et al., 2013;
Kotthoff et al., 2017), MLBox, AutoKeras (Jin et al., 2019), Google’s Cloud
AutoML555https://cloud.google.com/automl/, Amazon’s AutoGluon (Erickson et
al., 2020), IBM Watson AutoAI 666https://www.ibm.com/cloud/watson-
studio/autoai, and Microsoft Azure AutoML777https://docs.microsoft.com/en-
us/azure/machine-learning/concept-automated-ml have provided industrial
solutions that automatically build ML-based models. Most of these libraries
propose general-purpose AutoML solutions that automatically develop ML-based
models across a broad class of applications in financial services, healthcare,
advertising, manufacturing, and other industries (Guyon et al., 2019).

The key assumption of this horizontal approach is that the process of
automated model development remains the same across all these applications. In
this paper, however, we focus on developing a _vertical_ AutoML solution
suitable for the needs of the ecosystem (Pidun et al., 2019) of a large
European financial services company comprising a wide range of banking and
other types of financial as well as non-financial services, including
telecommunications, transportation, and e-commerce for the B2B and B2C sectors
of the economy. We argue in the paper that such ecosystem has an idiosyncratic
set of requirements for building ML models and would be better served by a
domain-specific AutoML solution rather than using a generic horizontal AutoML
system. In particular, our ecosystem has the following set of requirements:

  * •

AutoML system should be able to work with different types of data collected
from hundreds of different information systems. Moreover, often such AutoML
system changes more rapidly than it can be fully documented and painstakingly
preprocessed by data scientists for the ML tasks using ETL tools.

  * •

Many of our models are typically built on large datasets, having thousands or
tens of thousands of features and millions of records. This makes it important
to develop fast AutoML methods that efficiently handle such datasets.

  * •

The number of production-level models across our complex ecosystem is very
large, measured in thousands, and continues to increase rapidly, forming a
Long Tail in terms of their popularity and economic efficiency. This makes it
necessary for the AutoML system to accurately build and maintain all these
models efficiently and cost-effectively. Furthermore, besides building
production models, it is necessary to build a large number of models to
validate numerous hypotheses tested across the entire ecosystem and do it
efficiently.

  * •

Many of our business processes are non-stationary and are rapidly changing
over time which complicates the process of validating and keeping up to date
the ML models that are included in these evolving processes. This results in
the need to satisfy specific model validation needs, including out-of-time
validation and validation of client behavioral models (models that take a
sequence of single object states as input).

In this paper, we introduce a vertical type of AutoML, called _LightAutoML_ ,
which focuses on the aforementioned needs of our complex ecosystem and that
has the following characteristics. First, it provides nearly optimal and fast
hyperparameter search, but it does not optimize them directly, nevertheless
ensuring satisfactory (Simon, 1956) results. Furthermore, we dynamically keep
the balance between hyperparameter optimization and speed, making sure that
our solutions are optimal on small problems and fast enough on large ones.
Second, we purposely limit the range of ML models to only two types, i.e.,
gradient boosted decision trees (GBMs) and linear models, instead of having
large ensembles of multiple algorithms. It is done to speed up LightAutoML
execution time without sacrificing its performance for our types of problems
and data. Third, we present a unique method of choosing preprocessing schemes
for different features used in our models based on certain types of meta-
statistics and selection rules.

We tested the proposed LightAutoML system on a wide range of open and
proprietary data sources across a wide range of applications and demonstrated
its superior performance in the experiments. Furthermore, we deployed
LightAutoML in our ecosystem in numerous applications across five different
platforms which enabled the company to save millions of dollars and present
our experiences with this deployment and business outcomes. In particular, the
initial economic effects of LightAutoML in these applications range from 3% to
5% of the total ML economic effects from deployed ML solutions in the company.
Moreover, LightAutoML provided certain novel capabilities that humans cannot
perform, such as generating massive amounts of ML models in record time in the
non-stop (24-7-365) working mode.

In this paper, we make the following contributions. First, we present the
LightAutoML system developed for the ecosystem of a large financial services
company comprising a wide range of banking and other types of financial and
non-financial services. Second, we compare LightAutoML with the leading
general-purpose AutoML solutions and demonstrate that LightAutoML outperforms
them across several ecosystem applications and on the open source AutoML
benchmark _OpenML_ (Gijsbers et al., 2019). Third, we compare the performance
of LightAutoML models with those manually tuned by the data scientists and
demonstrate that LightAutoML models usually outperform data scientists.
Finally, we describe our experiences with the deployment of LightAutoML in our
ecosystem.

##  2\. Related work

The early work on AutoML goes back to the mid-’90s when the first papers on
hyperparameter optimization were published (King et al., 1995; Kohavi and
John, 1995). Subsequently, the concepts of AutoML were expanded and the
interest in AutoML significantly increased after the publication of the Auto-
WEKA paper (Thornton et al., 2013) in 2013 and the organization of the AutoML
workshop at ICML in 2014.

One of the main areas of AutoML is the problem of hyperparameter search, where
the best performing hyperparameters for a particular ML model are determined
in a large hyperparameter space using various optimization methods (Bergstra
et al., 2011). Another approach is to estimate the probability that a
particular hyperparameter is the optimal one for a given model using Bayesian
methods that typically use historical data from other datasets and the
previously estimated models on these datasets (Swersky et al., 2013;
Springenberg et al., 2016; Perrone et al., 2017). Other methods, besides the
hyperparameter optimization, try to select the best models from the space of
several possible modeling alternatives (Yang et al., 2019; Lee et al., 2018;
Olson and Moore, 2019; Santos et al., 2019). For example, TPOT (Olson and
Moore, 2019) generates a set of best-performing models from Sklearn and
XGboost and automatically chooses the best subset of models. Further, other
papers focus on the problem of automated deep learning model selection and
optimization (Lee et al., 2018; Zimmer et al., 2020). Finally, several papers
propose various methods of automatic feature generation (Olson and Moore,
2019).

In addition to the specific approaches to AutoML described above, there has
been a discussion in the AutoML community on what AutoML is and how to
properly define it, with different authors expressing their points of view on
the subject. In particular, while some approaches focus only on the modeling
stage of the CRISP-DM model lifecycle, other approaches take a broader view of
the process and cover other lifecycle stages. For example, according to Shubha
Nabar from Salesforce, “most auto-ML solutions today are either focused very
narrowly on a small piece of the entire machine learning workflow, or are
built for unstructured, homogenous data for images, voice and language”
(Nabar, 2018). Then she argues that the real goal of the AutoML system is the
end-to-end approach across the whole CRISP-DM cycle that “transforms customer
data into meaningful actionable predictions” (Nabar, 2018). A similarly broad
view of AutoML is presented in (Guyon and Elisseeff, 2003) where it was argued
that AutoML focuses on “removing the need for human interaction in applying
machine learning (ML) to practical problems”. A similar argument for a broader
view of AutoML as an end-to-end process taking the input data and
automatically producing an optimized predictive model was presented in (MSV,
2018).

Furthermore, some successful examples of industry-specific AutoML solutions
for medical, financial, and advertisement domains are reviewed in (Guyon et
al., 2019). One particular application of AutoML in the financial sector is
presented in (Agrapetidou et al., 2020), where a simple general-purpose AutoML
system, utilizing models of random forest, support vector machines, k-nearest
neighbors with different kernels, and other ML methods, is created for the
task of detecting bank failures. This system is an experimental proof-of-
concept prototype focusing on a narrow task of bank failures but not an
industrial-level AutoML solution designed for a broad class of financial
applications.

In this paper, we focus on a broader approach to AutoML, which includes the
stages of data processing, model selection, and hyperparameter tuning. This is
in line with other popular approaches to AutoML incorporated into systems such
as AutoGluon (Erickson et al., 2020), H2O (LeDell and Poirier, 2020), AutoWeka
(Thornton et al., 2013), TPOT (Olson and Moore, 2019), Auto-keras (Jin et al.,
2019), AutoXGBoost (Thomas et al., 2018), Auto-sklearn (Feurer et al., 2019),
Amazon SageMaker (Das et al., 2020).

##  3\. Overview of LightAutoML

In this section, we describe implementation details of LightAutoML, an open
source modular AutoML framework that can be accessed at our GitHub
repository888https://github.com/sberbank-ai-lab/LightAutoML. LightAutoML
consists of modules that we call _Presets_. They are focused on the end-to-end
model development for typical ML tasks. Currently, LightAutoML supports the
following four Preset modules. First, _TabularAutoML_ Preset focuses on
classical ML problems defined on tabular datasets. Second, _WhiteBox_ Preset
solves binary classification tasks on tabular data using simple interpretable
algorithms, such as Logistic Regression over discretized features and _Weight
of Evidence (WoE)_ (Zeng, 2014) encoding. This is a commonly used approach to
model the probability of a client default in banking applications because of
interpretability constraints posed by regulators and high costs of loan
approval for bad customers. Third, _NLP_ Preset is the same as Tabular but is
also able to combine tabular pipelines with the NLP tools, such as specific
feature extractors or pre-trained deep learning models. The last _CV_ Preset
implements some basic tools to work with image data. In addition, it is also
possible to build custom modules and Presets using LightAutoML API. Some
examples are also available at our GitHub repository and on Kaggle
Kernels999https://www.kaggle.com/simakov/lama-custom-automl-pipeline-example.
Although LightAutoML supports all four Presets, only TabularAutoML is
currently used in our production-level system. Therefore, we will focus on it
in the rest of this paper.

A typical LightAutoML pipeline scheme is presented in Figure 1. Each pipeline
contains:

  * •

_Reader_ : an object that receives raw data and task type as input, calculates
some useful metadata, performs initial data cleaning, and decides what data
manipulations should be done before fitting different models.

  * •

_LightAutoML inner datasets_ which contain metadata and CV iterators that
implement validation schemes for the datasets.

  * •

_Multiple ML Pipelines_ that are stacked (Ting and Witten, 1997) and/or
blended (averaged) via _Blender_ to get a single prediction.

Figure 1. Main components of LightAutoML Pipeline

Smth2

An ML pipeline in LightAutoML is one or multiple ML models that share a single
data preprocessing and validation scheme. The preprocessing step may have up
to two feature selection steps, a feature engineering step or may be empty if
no preprocessing is needed. The ML pipelines can be computed independently on
the same datasets and then blended together using averaging (or weighted
averaging). Alternatively, a stacking ensemble scheme can be used to build
multilevel ensemble architectures.

In the next section we describe TabularAutoML Preset and compare it with other
popular open source AutoML frameworks.

###  3.1. LightAutoML’s Tabular Preset

TabularAutoML is the default LightAutoML pipeline that solves three types of
tasks on tabular data: _binary classification_ , _multiclass classification_ ,
and _regression_ for various types of loss functions and performance metrics.
The input data for TabularAutoML is a table having columns of the following
four types: numeric features, categorical features, timestamps, and a single
target column with continuous values or class labels.

The key features of our LightAutoML pipeline are:

  * •

_Strong baseline_ : works good on most datasets

  * •

_Fast_ : no metamodels or pipeline optimization

  * •

_Advanced data preprocessing_ compared to the other popular open source
solutions

One of our main goals in designing LightAutoML was to make a tool for fast
hypothesis testing. Therefore, we avoid using brute-force methods for optimal
pipeline search and focus only on the models and efficient techniques that
work across a wide range of datasets. In particular, we train _only_ two
classes of models represented by three types of algorithms in the following
order: _Linear Model_ with L2 penalty, the _LightGBM_ version of the GBM
method (Ke et al., 2017) and the _CatBoost_ version of GBM (Dorogush et al.,
2018).

The selected order matters here because it helps to manage time if the user
sets the time limit. The algorithm’s training order is ranked by the time they
usually spend training. Therefore, if the time limit is small but reasonable,
at least the fastest model will be computed. On the other hand, if the time
limit is large enough, fast evaluation of the previous models allows us to
allocate more time for training and hyperparameter tuning for the slower ones.

Traditional ML algorithms were selected for the LightAutoML system because,
despite the trend in the development of neural networks for different domains,
GBM-based methods show strong performance results on tabular data and
outperform other approaches in many benchmarks and competitions at the moment.
Furthermore, various GBM frameworks are commonly used in industry to develop
production models (Ke et al., 2017; Dorogush et al., 2018; Chen and Guestrin,
2016). In addition, linear models are fast, easy to tune, and they can boost
the performance of tree-based models in ensembles by adding a variety of
predictions (Breiman, 1996). In comparison, other popular open source AutoML
frameworks usually use significantly more classes of models and therefore
require more time to train. As we show in Section 4.1, the proposed approach
with a few additional features is able to outperform existing solutions on
internal datasets used in our company and on the OpenML benchmark (Gijsbers et
al., 2019).

###  3.2. Data Preprocessing and Auto-typing

As we initially started developing LightAutoML, we paid special attention to
the data preprocessing part. As already mentioned in Section 1, we need to be
ready to work with datasets in different formats, scales, containing
artifacts, NaNs, or unspecified user processing.

To handle different types of features in different ways, we need to know each
feature type. In the case of a single task with a small dataset, the user can
specify each feature type manually. However, it becomes more problematic in
the case of hundreds of tasks with datasets containing thousands of features.
This is very typical for bank applications, and it takes hours of work of data
scientists to perform this data analysis and labeling. So it is crucial for
the AutoML framework to solve the problem of automatic data type inference
_(auto-typing)_.

In case of TabularAutoML Preset we need to map features into three classes:
_numeric_ , _category_ , and _datetime_. One simple and obvious solution is to
use column array data types as actual feature types, that is, to map float/int
columns to numeric features, timestamp or string, that could be parsed as a
timestamp — to datetime, and others to category. However, this mapping is not
the best because of the frequent occurrence of numeric data types in category
columns. An ML-based decision for this problem is described in (Shah and
Kumar, 2019), where different models over meta statistics are used to predict
human feature type labeling. Deep learning is also used to solve a similar but
slightly different problem of semantic data type detection in (Hulsebos et
al., 2019).

We solve this problem in a slightly different way. Let us say, column has
category type if category encoding techniques, such as target encoder
_(OOFEnc)_ (Micci-Barreca, 2001; Dorogush et al., 2018) or frequency encoder
_(FreqEnc)_ , that encodes category by the number of occurrence in train
sample, perform better than numeric ones, such as raw or discretized by
quantiles _(QDiscr)_ values. Building a lot of models to verify the
performance of all encoding combinations becomes impractical, so we need some
proxy metric that is simple to calculate and agnostic for the type of
LightAutoML task, including given loss and metric functions. We choose
Normalized Gini Index (Chen et al., 1982) between target variable and encoded
feature as a measure of encoding quality because it estimates the quality of
sorting and could be computed for both classification and regression tasks.
The details of the auto-typing algorithm are presented in Appendix B (as
Algorithm 1). The final decision is made by ten expert rules over estimated
encoding qualities and some other meta statistics such as the number of unique
values. Exact list of rules is given at the LightAutoML
repository101010https://github.com/sberbank-ai-
lab/LightAutoML/blob/master/lightautoml/reader
/guess_roles.py.

Note that we do not use ML models for auto-typing because, as it was mentioned
before, our goal is not to predict human labeling but to guess what is
actually better for the final model performance. Sometimes LightAutoML _auto-
typing_ prediction may differ from human’s point of view, however it may lead
to a significant boost in the performance, see Section 4.2 and datasets
guilermo, amazon_employee, and robert in Table 7 that contain a lot of
categories from _auto-typing_ point of view.

After we infer the type of a feature, we can additionally guess the optimal
way to preprocess it, for example, to select the best category encoding or
decide if we should discretize numbers. A similar algorithm can be used for
this purpose with a small adaptation by using different rules and encoding
methods.

###  3.3. Validation Schemes

As was mentioned earlier, data in the industry may rapidly change over time in
some ecosystem processes, which made independent identically distributed
_(IID)_ assumption irrelevant in model development. There are cases when time
series-based, grouped, or even custom validation splits are required. This
becomes important because validation in AutoML is used not only for
performance estimation but also for hyperparameters search and out-of-fold
prediction generation. Out-of-fold prediction is used for blending and
stacking models on upper LightAutoML levels and also returned as the
prediction on train set for user analysis.

To the best of our knowledge, other popular AutoML frameworks use only
classical KFold or random holdout approaches, while advanced validation
schemes help to handle non _IID_ cases and make models more robust and stable
in time. This problem is out of the scope of OpenML benchmark tasks but
becomes important in production applications. Currently available validation
schemes in TabularAutoML are:

  * •

KFold cross-validation, which is used by default (including stratified KFold
for classification tasks or GroupKFold for behavioral models if group
parameter for folds splitting is specified)

  * •

Holdout validation if holdout set specified

  * •

Custom validation schemes (including time series split (Cerqueira et al.,
2020) cross-validation)

All models that are trained during the cross-validation loop on different
folds are then saved for the inference phase. Inference on new data is made by
averaging models from all train folds.

###  3.4. Feature Selection

Feature selection is a crucial part of industrial model development because it
allows one to reduce model implementation and inference costs. However,
existing open source AutoML solutions are not focused much on this problem. In
turn, TabularAutoML implements three strategies of feature selection: _No
selection_ , _Importance cut off selection_ (by default), _Importance based
forward selection_.

Feature importance could be estimated in two ways: split-based tree importance
(Lundberg and Lee, 2017) or permutation importance (Altmann et al., 2010) of
GBM model. Importance cutoff selection aims to reject only features that are
useless for the model (importance measure ¡= 0). This strategy helps to reduce
the number of features with no performance decrease, which may speed up model
training and inference.

However, in order to reduce inference costs, one may want to limit the number
of features in the model or to find a minimal possible model with a small
performance drop. For that purpose, we implement a variant of the classical
forward selection algorithm described in (Guyon and Elisseeff, 2003) with the
key difference of ranking the candidate features by the importance measure
mentioned above that helps to speed up the procedure significantly. The
specifics of the algorithm are provided in Appendix C (as Algorithm 2).

We show in Table 1 on internal datasets that it is possible to build much
faster and simpler models with slightly lower scores.

Table 1. Comparison of different selection strategies on binary bank datasets. Strategy | Avg ROC-AUC | Avg Inference Time (10k rows)
---|---|---
Cut off | 0.804 | 9.1078
Forward | 0.7978 | 5.6088

###  3.5. Hyperparameter Tuning

In TabularAutoML, we use different ways to tune hyperparameters depending on
what is tuned:

  * •

Early stopping to choose the number of iterations (trees in GBM or gradient
descent steps) for all models during the training phase

  * •

Expert system

  * •

Tree structured Parzen estimation _(TPE)_ for GBM models

  * •

Grid search

All hyperparameters are tuned by maximizing the metric function, which is the
default for the solved task or which is defined by the user.

####  3.5.1. Expert system

A simple way to quickly set hyperparameters for models in a satisfactory
(Simon, 1956) fashion is expert rules. TabularAutoML can initialize a
“reasonably good” set of GBM hyperparameters (learning rate, subsample,
columns sample, depth) depending on a task and a dataset size. Suboptimal
parameter choice is partially compensated by an adaptive selection of the
number of steps using early stopping. It prevents the final model from a high
decrease in score compared to hard-tuned models.

####  3.5.2. TPE and combined strategy

We introduce a mixed tuning strategy that works by default in TabularAutoML
(but the user might change it): for each GBM framework (LightGBM and CatBoost)
we train two types of models. The first one gets expert hyperparameters, and
the second one is fine-tuned while it fits into the time budget. TPE
algorithm, described in (Bergstra et al., 2011) is used for the model fine-
tuning. This algorithm is chosen because it shows state-of-the-art results in
tuning this class of models. We use realization of TPE by Optuna framework
(Akiba et al., 2019). In the final model, both models will be blended or
stacked. Also, one of the models (or even both) could be dropped from the
AutoML pipeline if it does not help to increase the final model performance.
We compare this combined strategy with the AutoML based on the expert system
only; results are given in Section 4.2.

####  3.5.3. Grid search

Grid search parameter tuning is used in TabularAutoML pipeline to fine-tune
regularization parameter of a linear model in combination with:

  * •

Early stopping. We assume the regularization parameter in the linear model has
a single optimal point, and after reaching it, we can finish the search.

  * •

Warm start. We use warm start strategy to initialize model weights between
regularization trials (Friedman et al., 2010). It helps to speed up model
training.

Both heuristics make the grid search efficient in fine-tuninig linear
estimators.

###  3.6. Model Ensembling in TabularAutoML

####  3.6.1. Multilevel stacking ensembles

As was mentioned before, LightAutoML allows users to build stacked ensembles
of unlimited depth. Similar strategy is common for AutoML systems and also
used in (LeDell and Poirier, 2020; Erickson et al., 2020). However, building
ensembles deeper than 3 levels shows no effect in practice.

TabularAutoML builds two-level stacking ensembles by default for multiclass
classification tasks only because it was the only case where we observed a
significant and stable boost in model performance; see Section 4.2. This
behavior is just the default setting and can be changed by the user to perform
stacking on any dataset.

####  3.6.2. Blending

The last level of LightAutoML ensemble, regardless of its depth, may contain
more than one model. To combine predictions of different models into a single
AutoML output, they are passed to the blending phase. Blending in terms of
LightAutoML differs from the full stacked model in the following ways. First,
it is a considerably simpler model. As a consequence, second, it does not
require any validation scheme to tune and control overfitting. Finally, it is
able to perform model selection to simplify the ensemble to speed up the
inference phase.

TabularAutoML uses weighted averaging as a blender model. Ensemble weights are
estimated using the coordinate descent algorithm to maximize the metric
function, which is the default for the solved task or which is defined by the
user. Models with weights close to $0$ are dropped from AutoML.

####  3.6.3. Ensemble of AutoMLs and time utilization

As we mentioned before, one of our goals was to limit the search space of ML
algorithms to speed up model training and inference, which is related to
production use cases of AutoML frameworks for medium and high dataset sizes.
However, this strategy will not perform well in cases such as ML competitions
when the user has a very high time budget and needs to utilize it all to get
the best performance regardless of its cost. Increasing the search space and
brute forcing will often take advantage here.

Typical examples here are small datasets from the OpenML benchmark that are
solved by TabularAutoML much faster than the given time limit of 1 hour (Table
2). In order to solve this problem and be competitive on benchmarks and ML
competitions in the case of small datasets, we implement the time utilization
strategy that blends multiple TabularAutoMLs with slightly different settings
and validation random seeds. A user may define multiple config settings and
the priority order or use defaults. AutoMLs with the same settings and
different seeds will be simply averaged together, and after that, different
settings ensembles will be weighted averaged. This strategy shows a
performance boost on OpenML tasks; see Section 4.2.

Table 2. Average training time in seconds for the smallest OpenML datasets for Utilized Tabular Preset version and Default (single run). Task type | Utilized | Single run
---|---|---
Binary (9 smallest) | 3268 | 360
Multi class (7 smallest) | 2984 | 1201

##  4\. Performance of LightAutoML

###  4.1. Comparison with Open Source AutoML

In this section, we compare the performance of LightAutoML TabularAutoML
Preset with the already existing open source solutions across various tasks
and show the superior performance of our method. First, to make this
comparison, we use datasets from the OpenML benchmark that is typically used
to evaluate the quality of AutoML systems. The benchmark is evaluated on $35$
datasets of binary and multiclass classification tasks. The complete
experimental description, including the framework versions, limitations, and
extended results, is presented in Appendix A. The summary of the results of
LightAutoML vis-a-vis five popular AutoML systems is presented in Table 3,
where all the AutoML systems are ranked by the total amount of wins in each
dataset.

Table 3. Aggregated framework comparison on OpenML. framework | Wins | Avg Rank | Avg Reciprocal Rank
---|---|---|---
autoweka | 0 | 5.7879 | 0.1747
autogluon | 0 | 4.2647 | 0.252
autosklearn | 3 | 2.6 | 0.4505
tpot | 6 | 3.8235 | 0.374
h2oautoml | 6 | 2.4857 | 0.4833
lightautoml | 20 | 1.9429 | 0.7233

However, the detailed comparison of frameworks in the context of dataset
groups provided in Table 4 shows that LightAutoML does not work equally well
on all the classes of tasks. For binary classification problems with a small
amount of data, LightAutoML shows average performance results and losses to
TPOT; moreover, it performs on par with H2O and Auto-Sklearn. The reason for
this is that the tasks with a small amount of data are not common in our
ecosystem and were not the main impetus behind the development of LightAutoML.

Table 4. Average reciprocal rank for frameworks comparison by OpenML datasets groups. | Small | Small | Medium | Medium
---|---|---|---|---
Framework | binary | multiclass | binary | multiclass
autoweka | 0.1741 | 0.1667 | 0.1783 | 0.1786
tpot | 0.6056 | 0.481 | 0.2061 | 0.2333
autogluon | 0.2796 | 0.2071 | 0.2606 | 0.2476
autosklearn | 0.4519 | 0.3571 | 0.4424 | 0.5417
h2oautoml | 0.4907 | 0.5595 | 0.4697 | 0.4271
lightautoml | 0.4481 | 0.6786 | 0.8939 | 0.8375

Another type of datasets for comparing different solutions is internal
datasets collected in the bank. In this study, we use 15 bank datasets for
various binary classification tasks performed in our company, such as credit
scoring (probability of defaults estimation), collection, and marketing
(response probability). As the primary goal of developing the LightAutoML
framework was to work with our internal applications, we expected better
performance of our system on the internal data. In Table 5 we show that the
performance gap between LightAutoML and other AutoML systems is significantly
higher on the bank datasets than on the OpenML data111111We cannot present
information about these internal datasets in this paper because they contain
proprietary and confidential information..

Table 5. Aggregated framework comparison on bank’s proprietary datasets. framework | Wins | Avg Rank | Avg Reciprocal Rank
---|---|---|---
autoweka | 0 | 5.7333 | 0.18
autogluon | 0 | 3.9333 | 0.2778
tpot | 1 | 3.9333 | 0.3056
autosklearn | 1 | 3.2667 | 0.3956
h2oautoml | 1 | 2.6667 | 0.4244
lightautoml | 12 | 1.4667 | 0.8667

###  4.2. Ablation Study

We perform the ablation study estimating average reciprocal rank change to
evaluate each TabularAutoML feature impact on the OpenML benchmark results. We
take the best existing AutoML configuration, including the time utilization
strategy, combined hyperparameter tuning, auto-typing, stacking for multiclass
tasks only (Utilized best) as baselines. First, we turn off time utilization
to estimate contribution of multi-start bagging (Default) described in Section
3.6.3. Second, we take Default and compare the two-level stacking for all the
tasks (Stacked all) and blending first level models only (Single level all) to
estimate the quality of alternative ensembling methods discussed in Section
3.6.1. Third, we exclude the advanced auto-typing module presented in Section
3.2 from Default (No auto-typing). Finally, we replace the combined tuning
strategy described in Section 3.5.2 from Default with the expert system
initialization only (No finetune). The ablation study results are presented in
Table 6. As Table 6 demonstrates, each feature removal decreases the
LightAutoML rank, which shows that all those features make our framework more
accurate than the others on OpenML datasets.

Table 6. Ablation study on OpenML. Configuration | Avg Reciprocal Rank | Avg Rank
---|---|---
No finetune | 0.6054 | 2.4118
Typing off | 0.6431 | 2.1765
Single level all | 0.65 | 2.2353
Stacked all | 0.6672 | 2.2059
Default (stacked multiclass) | 0.6907 | 2.0882
Utilized best | 0.7233 | 1.9429

###  4.3. LightAutoML vs. Building Models by Hand

We have also used our LightAutoML system as one of the “participants” in the
internal hackathon in our ecosystem, together with $433$ leading data
scientists in the company. The training dataset used in the competition had
$300$ features and $400\,000$ records, and the goal was to predict the churn
rates. The performance metric selected for this hackathon was ROC-AUC, and the
performance of the baseline used in this competition was ROC-AUC = $75.5\%$.

LightAutoML was presented in the hackathon by $4$ participants that used it in
different configurations. As Figure 2 demonstrates, LightAutoML outperformed
the baseline model. Although the average performance of LightAutoML (ROC-AUC =
76.54) was better than the average performance of the top-10% of hackathon
participants, i.e., belonging to the $90\%$ quantile (average ROC-AUC =
76.08), the performance improvements were not statistically significant. This
means that although LightAutoML significantly outperformed the average data
scientist in this hackathon (including the “bottom $90\%$”), its performance
was comparable to the top-$10\%$ of the best data scientists. Detailed results
are given in repository121212https://github.com/sberbank-ai-lab/Paper-
Appendix.

Figure 2. Performance Results of LightAutoML in the NextHack Competition vs.
433 Human Competitors.

##  5\. Deploying LightAutoML

In this section, we present our experiences with deploying, piloting, and
moving LightAutoML into production.

Deployment. Currently, the LightAutoML system is deployed in production by
five large ML platforms inside our financial services company and its
ecosystem, including cloud, B2B, and B2C platforms. Furthermore, seven more
divisions are currently piloting the latest version of the system. Moreover,
it is also used in several automated systems and various IT services across
the ecosystem. Altogether, more than $70$ teams involving several hundred data
scientists use LightAutoML to build ML models across the entire ecosystem. As
an example, just the B2C platform alone has more than $300$ business problems
solved using LightAutoML this year, resulting in the total P&L increase by
$3\%$. In what follows, we present some examples of successful deployments of
LightAutoML in the ecosystem and our experiences with these deployments.

_Operational audit._ LightAutoML has been applied to the problem of
operational audit of the bank branches with the goal to detect mistakes made
by bank’s employees across the organization and do it in the most effective
and efficient manner. These mistakes are of numerous types, depending on the
type of branch, its location, the type of employee who made a mistake, etc.
The goal of the operational audit is to detect and correct all these mistakes,
prevent their future occurrences, and minimize their consequences according to
the established practices of the bank.

Figure 3. Difference in revenues of the LightAutoML and the manually developed
impact estimation methods.

In this project, we focused on 60 major types of mistakes and developed one
predictive model per mistake and each of the $11$ divisions of the bank,
resulting in $660$ LightAutoML models in total. For comparison, prior mistake
detection methods were rule-based.

One of the reasons for the bank not developing prior ML-based mistake
detection models was the large number of such models ($660$ in our case)
requiring extensive resources of data scientists and a long time to produce
all of them (measured in person-years), making such project infeasible. The
second reason why such models have not been previously developed is that the
economic effect from each model is limited. For example, Figure 3 shows the
$660$ operational risk models sorted on the $y$-axis by the difference in the
economic impact between the LightAutoML models and the previously existing
rule-based methods, ranging from $\$60\,000$ highest positive impact on the
left to the $\$8\,000$ highest negative impact on the right of Figure 3. This
example demonstrates that detecting even the most important operational
mistakes resulted in limited savings, and the savings from the medium and
minor mistakes were considerably lower, making the development of such machine
learning models economically infeasible. Nevertheless, the _cumulative_
economic effect of building _all_ the $660$ models is significant, bringing
the bank millions of dollars in savings.

All the $660$ operational audit models were developed by our LightAutoML
system in $3$ days over the weekend, and the whole project took $10$ person-
days, taking into account data preparation costs. This contrasts sharply with
the cost of creating so many models manually, resulting in saving the bank
millions of dollars.

_Other Examples of LightAutoML Applications._ LightAutoML system has also
automatically built fraud detection models that were subsequently compared
against the manually developed classical ML-based fraud detection models
previously developed and deployed in the company. These models saved
development time by $40$ person-days and improving model performance by $6\%$
for the F1 performance metric, identifying several thousands of fraudulent
activities, and thus saving the bank millions of dollars.

Another example of successful LightAutoML deployment was a charitable
donations system involving $110$ different types of contributions focusing on
children’s welfare. Our LightAutoML system developed a model identifying the
donors over a period of two person-days. When it was moved into production,
only for the email channel it increased the number of donations by $18\%$ and
the total sum of donations by $40\%$.

Lessons Learned. We will start with the lessons learned when _developing and
piloting_ LightAutoML for our ecosystem. First, applications that are the most
amenable to the deployment of AutoML are those where the prediction problem is
formulated precisely and correctly and does not require human insights based
on deep prior knowledge of the application domain. Second, AutoML is well
suited for typical supervised learning tasks, where testing works well on
retrospective data. It is harder to effectively apply the current generation
of AutoML systems to non-standard problems. Third, AutoML solutions are
especially useful when an ML model should be built quickly with limited
resources. In those cases when there is plenty of time to build a model using
top data science talent working on complex problems requiring significant
insights and ingenuity, non-standard solutions, and careful fine-tuning of the
model parameters, humans can outperform AutoML solutions. As an example, when
Google conducted the IEEE competition (Rishi, 2019), its AutoML solution
outperformed 90% of the competing teams in the first two weeks. However,
humans eventually managed to catch up and outperform Google’s AutoML after the
first two weeks by putting extensive effort into the hackathon and constantly
improving their models. Finally, strong performance results of LightAutoML
were achieved not due to certain unique breakthrough features of our system
but because of numerous incremental improvements described in Section 3 that
were cohesively combined into a unified solution.

Furthermore, while moving LightAutoML into _production_ , we have learned a
different group of lessons. First, independent tests of our system by data
scientists in the company show that LightAutoML significantly outperformed
humans on only one-third of the ML tasks actually deployed in production,
which differs from the $90\%$ figure reported in Section 4.3 for the pilot
studies. This reduction of LightAutoML performance vs. humans in production
environments is due to the following reasons:

  * •

Presence of _data leaks_ , i.e., the situation when information about an
object target becomes obvious from features for the training and not for the
test data, and also of _data artifacts_ , such as special decimal separators
or character symbols (e.g., K standing for thousands) in numeric columns,
missing values for a specific feature block, etc., across hundreds of
automated data storage systems.

  * •

Existence of extremely small number of minority class events, compared with
the number of features.

  * •

In the same conditions, top data scientists build many more models than less
experienced ones at the same time.

The last point demonstrates that, instead of replacing data scientists with
AutoML systems, it is better to complement them with such systems. Now, in our
company, we focus on empowering our DSes with LightAutoML. In particular, we
are primarily using LightAutoML as a baseline generator and as a fast
hypothesis testing tool in the company now. This helps our data scientists to
focus on certain crucial parts of the model development process, such as
appropriate time period data selection, target variable formulation, selection
of suitable quality metrics, identification of business constraints, and so
on.

The second lesson is associated with the importance of integrating LightAutoML
with different production environments of our diverse ecosystem to implement
end-to-end solutions. Although we observed 4x to 10x model training time
reduction in comparison to the usual model creation process, overall time-to-
market managed to decrease by only $30\%$ on average for the _whole_ model
life-cycle process. Furthermore, we observed that this number can be improved
to almost $70\%$ for the cases when continuous integration with data sources
and inference environments was done, notably on our cloud platform.

In summary, we had encountered several issues when piloting our LightAutoML
system and moving it into production, most of these issues have to deal with
idiosyncratic requirements of the financial services industry and the diverse
ecosystem of our organization. We managed to resolve them successfully by
developing the LightAutoML system to suit the needs of our ecosystem. We have
also described the lessons learned while moving LightAutoML into production
across a diverse class of ML applications. All this makes a strong case for
developing vertical AutoML solutions for the financial services industry.

##  6\. Conclusion

In the paper, we present the LightAutoML system designed to satisfy the
specific needs of large financial services companies and their ecosystems. We
argue for the need to develop a special-purpose AutoML, as opposed to the
general-purpose system, such as H2O or AutoGluon, that would satisfy the
idiosyncratic needs of such organizations, including the ability to handle
large datasets having a broad range of data types, non-stationary data,
specific types of validations, including behavioral models and out-of-time
validations, and rapid development of a large number of models. The proposed
LightAutoML system has several incremental improvements: the “light-and-fast”
approach to AutoML development when only GBMs and linear models are used,
novel and fast combined hyperparameter tuning method that produces strong
tuning results, advanced data preprocessing including auto-typing that
collectively enhances functionality of LightAutoML and helps it to achieve
superior performance results.

Further, we show that our LightAutoML system outperforms some of the leading
general-purpose AutoML solutions in terms of the AUC-ROC and LogLoss metrics
on our proprietary applications and also on the OpenML benchmarks, as well as
the models manually developed by data scientists for the typical problems of
importance to large financial organizations.

Finally, the proposed LightAutoML system has been deployed in production in
numerous applications across the company and its ecosystem, which helped to
save the organization millions of dollars in development costs, while also
achieving certain capabilities that are impossible for the humans to realize,
such as the generation of massive amounts of ML models in record time. We have
also described several important lessons that we have learned while developing
and deploying the LightAutoML system at the company, including that (a) the
“light” approach to AutoML design in the LightAutoML system worked well in
practice, achieving superior performance results — mainly due to the careful
integration of various incremental improvements of different AutoML features
properly combined into the unified LightAutoML system; (b) realization that
LightAutoML outperformed data scientists on only one third of the deployed
models, as opposed to the expected $90\%$ of the cases — due to the
complexities and the “messiness” of the actually deployed vis-a-vis the pilot
cases; (c) realization that it is not always true that data scientists
outperform the machines when preparing the data to be used for building ML
models — LightAutoML outperformed data scientists in this data preparation
task in several use cases; (d) although LightAutoML significantly improved
model building productivity in our organization, the number of data scientists
actually increased significantly over the last year — mainly due to the fact
that we need many more ML models to better run our business. There is plenty
of work for both AutoML and data scientists to achieve our business goals.

As a part of the future work, we plan to develop functionality related to
model distillation and strengthen the work with NLP tasks. In particular, some
applications in our organization, including e-commerce, impose additional
constraints on the real-time performance of ML models, and we need the make
sure that the distillation component of LightAutoML satisfies these real-time
requirements. Furthermore, we plan further to enhance NLP functionality of
LightAutoML in its future releases.

## References

  * (1)
  * Agrapetidou et al. (2020) Anna Agrapetidou, Paulos Charonyktakis, Periklis Gogas, Theophilos Papadimitriou, and Ioannis Tsamardinos. 2020\.  An AutoML application to forecasting bank failures.  _Applied Economics Letters_ 28, 1 (2020), 1–5.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019.  Optuna: A next-generation hyperparameter optimization framework. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 2623–2631.
  * Altmann et al. (2010) André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. 2010.  Permutation importance: a corrected feature importance measure.  _Bioinformatics_ 26, 10 (2010), 1340–1347.
  * Bergstra et al. (2011) James Bergstra, Rémi Bardenet, Balázs Kégl, and Yoshua Bengio. 2011.  Implementations of algorithms for hyper-parameter optimization. In _NIPS Workshop on Bayesian optimization_. 29.
  * Breiman (1996) Leo Breiman. 1996\.  Bagging predictors.  _Machine learning_ 24, 2 (1996), 123–140.
  * Cerqueira et al. (2020) Vitor Cerqueira, Luis Torgo, and Igor Mozetič. 2020\.  Evaluating time series forecasting models: An empirical study on performance estimation methods.  _Machine Learning_ 109, 11 (2020), 1997–2028.
  * Chen et al. (1982) Chau-Nan Chen, Tien-Wang Tsaur, and Tong-Shieng Rhai. 1982\.  The Gini coefficient and negative income.  _Oxford Economic Papers_ 34, 3 (1982), 473–478.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_. 785–794.
  * Das et al. (2020) Piali Das, Nikita Ivkin, Tanya Bansal, Laurence Rouesnel, Philip Gautier, Zohar Karnin, Leo Dirac, Lakshmi Ramakrishnan, Andre Perunicic, Iaroslav Shcherbatyi, et al. 2020\.  Amazon SageMaker Autopilot: a white box AutoML solution at scale. In _Proceedings of the Fourth International Workshop on Data Management for End-to-End Machine Learning_. 1–7.
  * Dorogush et al. (2018) Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018\.  CatBoost: gradient boosting with categorical features support.  _arXiv preprint arXiv:1810.11363_ (2018).
  * Erickson et al. (2020) Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. 2020.  Autogluon-tabular: Robust and accurate automl for structured data.  _arXiv preprint arXiv:2003.06505_ (2020).
  * Feurer et al. (2019) Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. 2019.  Auto-sklearn: efficient and robust automated machine learning.  In _Automated Machine Learning_. Springer, Cham.
  * Friedman et al. (2010) Jerome Friedman, Trevor Hastie, and Rob Tibshirani. 2010\.  Regularization paths for generalized linear models via coordinate descent.  _Journal of statistical software_ 33, 1 (2010), 1\.
  * Gijsbers et al. (2019) Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, and Joaquin Vanschoren. 2019.  An open source automl benchmark.  _arXiv preprint arXiv:1907.00909_ (2019).  Accepted at AutoML Workshop at ICML 2019.
  * Guyon and Elisseeff (2003) Isabelle Guyon and André Elisseeff. 2003.  An introduction to variable and feature selection.  _Journal of machine learning research_ 3, Mar (2003), 1157–1182.
  * Guyon et al. (2019) Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag, et al. 2019\.  Analysis of the AutoML challenge series 2015–2018.  In _Automated Machine Learning_. Springer, 177–219.
  * Hulsebos et al. (2019) Madelon Hulsebos, Kevin Hu, Michiel Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Çagatay Demiralp, and César Hidalgo. 2019.  Sherlock: A deep learning approach to semantic data type detection. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1500–1508.
  * Jin et al. (2019) Haifeng Jin, Qingquan Song, and Xia Hu. 2019.  Auto-keras: An efficient neural architecture search system. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1946–1956.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017\.  Lightgbm: A highly efficient gradient boosting decision tree. In _Advances in neural information processing systems_. 3146–3154.
  * King et al. (1995) Ross D. King, Cao Feng, and Alistair Sutherland. 1995\.  Statlog: comparison of classification algorithms on large real-world problems.  _Applied Artificial Intelligence an International Journal_ 9, 3 (1995), 289–333.
  * Kohavi and John (1995) Ron Kohavi and George H John. 1995.  Automatic parameter selection by minimizing estimated error. In _Machine Learning Proceedings 1995_. Elsevier, 304–312.
  * Kotthoff et al. (2017) Lars Kotthoff, Chris Thornton, Holger H Hoos, Frank Hutter, and Kevin Leyton-Brown. 2017.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA.  _The Journal of Machine Learning Research_ 18, 1 (2017), 826–830.
  * LeDell and Poirier (2020) Erin LeDell and Sebastien Poirier. 2020.  H2o automl: Scalable automatic machine learning. In _7th ICML workshop on automated machine learning_.
  * Lee et al. (2018) Keon Myung Lee, Kyoung Soon Hwang, Kwang Il Kim, Sang Hyun Lee, and Ki Sun Park. 2018\.  A deep learning model generation method for code reuse and automatic machine learning. In _Proceedings of the 2018 Conference on Research in Adaptive and Convergent Systems_. 47–52.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Advances in neural information processing systems_. 4765–4774.
  * Micci-Barreca (2001) Daniele Micci-Barreca. 2001\.  A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems.  _ACM SIGKDD Explorations Newsletter_ 3, 1 (2001), 27–32.
  * MSV (2018) Janakiram MSV. 2018\.  _Why AutoML Is Set To Become The Future Of Artificial Intelligence_.  Retrieved 2018 from https://www.forbes.com/sites/janakirammsv/2018/04/15/why-automl-is-set-to-become-the-future-of-artificial-intelligence/#52d804b1780a
  * Nabar (2018) Shubha Nabar. 2018\.  _Open Sourcing TransmogrifAI: Automated Machine Learning for Structured Data_.  Retrieved 2018 from https://engineering.salesforce.com/open-sourcing-transmogrifai-4e5d0e098da2
  * Olson and Moore (2019) Randal S Olson and Jason H Moore. 2019.  TPOT: A Tree-Based Pipeline Optimization Tool for Automating.  _Automated Machine Learning: Methods, Systems, Challenges_ (2019), 151.
  * Perrone et al. (2017) Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cedric Archambeau. 2017.  Multiple adaptive Bayesian linear regression for scalable Bayesian optimization with warm start.  _arXiv preprint arXiv:1712.02902_ (2017).
  * Pidun et al. (2019) Ulrich Pidun, Martin Reeves, and Maximilian Schüssler. 2019\.  Do you need a business ecosystem?  _Boston Consulting Group_ 7 (2019).
  * Rishi (2019) Devvret Rishi. 2019\.  _Why we’re excited about AutoML_.  http://cloud.google.com/blog/products/ai-machine-learning/bringing-google-automl-to-3-million-data-scientists-on-kaggle
  * Santos et al. (2019) Aécio Santos, Sonia Castelo, Cristian Felix, Jorge Piazentin Ono, Bowen Yu, Sungsoo Ray Hong, Cláudio T Silva, Enrico Bertini, and Juliana Freire. 2019.  Visus: An interactive system for automatic machine learning model building and curation. In _Proceedings of the Workshop on Human-In-the-Loop Data Analytics_.
  * Shah and Kumar (2019) Vraj Shah and Arun Kumar. 2019.  The ML data prep zoo: Towards semi-automatic data preparation for ML. In _Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning_. 1–4.
  * Simon (1956) Herbert A Simon. 1956\.  Rational choice and the structure of the environment.  _Psychological review_ 63, 2 (1956), 129.
  * Springenberg et al. (2016) Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. 2016.  Bayesian optimization with robust Bayesian neural networks. In _Advances in neural information processing systems_. 4134–4142.
  * Swersky et al. (2013) Kevin Swersky, Jasper Snoek, and Ryan P Adams. 2013\.  Multi-task bayesian optimization. In _Advances in neural information processing systems_. 2004–2012.
  * Thomas et al. (2018) Janek Thomas, Stefan Coors, and Bernd Bischl. 2018\.  Automatic gradient boosting.  _arXiv preprint arXiv:1807.03873_ (2018).
  * Thornton et al. (2013) Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2013.  Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_. 847–855.
  * Ting and Witten (1997) Kai Ting and Ian Witten. 1997.  Stacking bagged and dagged models.  (1997).
  * Yang et al. (2019) Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell. 2019.  OBOE: Collaborative filtering for AutoML model selection. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_.
  * Zeng (2014) Guoping Zeng. 2014\.  A necessary condition for a good binning algorithm in credit scoring.  _Applied Mathematical Sciences_ 8, 65 (2014), 3229–3242.
  * Zimmer et al. (2020) Lucas Zimmer, Marius Lindauer, and Frank Hutter. 2020\.  Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl.  _arXiv preprint arXiv:2006.13799_ (2020).

##  Appendix A Experiment design and additional results

Datasets and experiment design were taken from the official OpenML benchmark
page. All the datasets were evaluated across 10 cross-validation folds made by
organizers, and the final score for each dataset was calculated by averaging
scores from all 10 folds. Models were scored by ROC-AUC metric for binary
classification tasks and LogLoss for multiclass classification tasks. In this
paper, we drop from evaluation 4 out of 39 datasets because most of the
frameworks failed on these datasets due to timeout with given limitations.

Models were evaluated with the following limitations: 1 hour runtime limit
(this limit was passed to the framework as the input parameter if the
framework supports time limitations, but actually process was killed after 2
hours), 8 CPU, 32 GB RAM per single cross-validation split. Each fold was
evaluated in a separated docker container on a cloud server under OS Ubuntu
18.04 with HDD and Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz.

Frameworks and versions that were compared in the benchmark are the following:
lightautoml==0.2.8, h2o_automl==3.32.0.1, autogluon==0.0.12,
autosklearn==0.11.1, autoweka==2.6, tpot==0.11.5. The code for benchmark
evaluation was taken from the OpenML
repository131313https://github.com/openml/automlbenchmark, were it was
published by frameworks developers. Finally, we have only 4 failure cases due
to timeout. Results across all 35 datasets are shown in Table 7.

The code to reproduce our experiment is available at the
repository141414https://github.com/sberbank-ai-
lab/automlbenchmark/tree/lightautoml. The code to reproduce LightAutoML
results is also published at the OpenML repository.

Note that, as it is mentioned in (Erickson et al., 2020), AutoGluon shows
state-of-the-art results on the OpenML benchmark, but we were not able to
reproduce the results using the code published at the OpenML repository. A
possible reason for this is non-default run settings or differences in
computational environments. Also, during the benchmarks process, we
encountered various errors in the tested frameworks; most of them are listed
in Appendix in (Erickson et al., 2020).

The comparison for the internal datasets was made on the same environment
except the time limit — we set the limit to two hours for inner data. Datasets
contain clients’ information that can not be published, so only aggregated
framework comparison is presented. Datasets were split independently into
train/test samples by data owners, depending on their business tasks. The
split may be random, out-of-time, or separated by group values (for example,
by client IDs for behavioral models). Split methods and test samples were
unknown to AutoMLs during the training phase. Models were trained on train
parts and scored by ROC-AUC metric values on test parts.

##  Appendix B Auto-typing algorithm

Input : Integer and float type train features $X_{train}^{if}$, train target
$Y_{train}$, set of expert rules $Rules$

1

Output : Boolean values for each feature in $X_{train}^{if}$ to be numeric
$IsNumber$

2

3

4for  _each feature $F$ in $X_{train}^{if}$_ do

5 $IsNumber[F]=False$

6 $notNaNSlice=isNotNaN(F)$

7 $F^{nn}=F[notNaNSlice]$

8 $Y_{train}^{nn}=Y_{train}[notNaNSlice]$

9 $NG_{noEnc}=NormGini(Y_{train}^{nn},F^{nn})$;

10

11 $NG_{OOFq}=NormGini(Y_{train}^{nn},OOFEnc(QDiscr(F^{nn})))$;

12

13 $NG_{FE}=NormGini(Y_{train}^{nn},FreqEnc(F^{nn}))$;

14

15 $NG_{OOF}=NormGini(Y_{train}^{nn},OOFEnc(F^{nn}))$;

16

17 for  _$R$ in $Rules$_ do

18 if  _$R(F^{nn},NG_{noEnc},NG_{OOFq},NG_{FE},NG_{OOF})$_ then

19 $IsNumber[F]=True$

20 Break;

21

22 end if

23

24 end for

25

26 end for

Algorithm 1 Splitting integer and float features into categorical and numeric
features.

##  Appendix C Permutation based forward selection algorithm

Input : Train features $X_{train}$, train target $Y_{train}$, valid features
$X_{valid}$, valid target $Y_{valid}$, MLAlgo $A$, features block size $N$,
metric $M$

1

Output : Selected features $OutFeats$

2

3

4$T_{perm}=A(X_{train},Y_{train})$;

5

6$PFimp=PermutationImp(X_{valid},T_{perm},Y_{valid},M)$;

7

8Sort $FeatureNames$ by descending $PFimp$;

9

10$OutFeats=[]$;

11

12$BaselineScore=-inf$;

13

14for  _$i$ from $0$ to $len(Features)$ with step $N$_ do

15 $CandidateFeats=FeatureNames[i:i+N]$;

16

17 $T_{i}=A(X_{train}[OutFeats+CandidateFeats])$;

18

19 $NewScore=M(Y_{valid},T_{i}(X_{valid}))$;

20

21 if  _$NewScore >BaselineScore$_ then

22 Append $CandidateFeats$ to $OutFeats$;

23 $BaselineScore=NewScore$;

24

25 end if

26

27 end for

Algorithm 2 Importance based forward selection. Table 7. Detailed results for OpenML datasets. dataset | metric | lightautoml | autogluon | h2oautoml | autosklearn | autoweka | tpot
---|---|---|---|---|---|---|---
australian | roc-auc | 0.9462 | 0.9393 | 0.934 | 0.9353 | 0.9337 | 0.9336
blood-transfusi… | roc-auc | 0.7497 | 0.719 | 0.758 | 0.75 | 0.7282 | 0.7479
credit-g | roc-auc | 0.7921 | 0.7766 | 0.7968 | 0.7756 | 0.7526 | 0.7824
kc1 | roc-auc | 0.8283 | 0.8168 | 0.8374 | 0.8404 | 0.8166 | 0.844
jasmine | roc-auc | 0.8806 | 0.8822 | 0.887 | 0.8826 | 0.8638 | 0.8897
kr-vs-kp | roc-auc | 0.9997 | 0.9994 | 0.9997 | 0.9999 | 0.981 | 0.9998
sylvine | roc-auc | 0.9882 | 0.9852 | 0.9882 | 0.9896 | 0.9729 | 0.9923
phoneme | roc-auc | 0.9655 | 0.9682 | 0.9668 | 0.9634 | 0.9552 | 0.9693
christine | roc-auc | 0.8307 | 0.8133 | 0.8247 | 0.8285 | 0.7905 | 0.8065
guillermo | roc-auc | 0.9322 | 0.9027 | 0.9078 | 0.9064 | 0.8901 | 0.8943
riccardo | roc-auc | 0.9997 | 0.9997 | 0.9997 | 0.9998 | 0.9981 | 0.9906
amazon_employee… | roc-auc | 0.9003 | 0.8758 | 0.8756 | 0.8524 | 0.8363 | 0.8674
nomao | roc-auc | 0.9976 | 0.9954 | 0.9959 | 0.9958 | 0.9826 | 0.9948
bank-marketing | roc-auc | 0.9401 | 0.9371 | 0.9373 | 0.938 | 0.8103 | 0.9314
adult | roc-auc | 0.9306 | 0.9286 | 0.9295 | 0.93 | 0.914 | 0.925
kddcup09_appete… | roc-auc | 0.8509 | 0.7932 | 0.8305 | 0.8383 | - | 0.8111
apsfailure | roc-auc | 0.9936 | 0.9915 | 0.9924 | 0.9921 | 0.9678 | 0.9904
numerai28.6 | roc-auc | 0.5306 | 0.5212 | 0.5311 | 0.5294 | 0.5249 | 0.5235
higgs | roc-auc | 0.8157 | 0.8055 | 0.815 | 0.8137 | 0.676 | 0.8024
miniboone | roc-auc | 0.9876 | 0.9842 | 0.9862 | 0.9865 | 0.9651 | 0.982
car | -logloss | -0.0038 | -0.1337 | -0.0033 | -0.0023 | -0.2216 | -0.0001
cnae-9 | -logloss | -0.1555 | -0.2917 | -0.2002 | -0.1784 | -0.8589 | -0.1448
connect-4 | -logloss | -0.3358 | -0.4956 | -0.3387 | -0.3481 | -3.1088 | -0.4411
dilbert | -logloss | -0.0327 | -0.1473 | -0.0411 | -0.0431 | -0.2455 | -0.0772
fabert | -logloss | -0.765 | -0.7715 | -0.764 | -0.7705 | -7.4649 | -0.8263
fashion-mnist | -logloss | -0.2519 | -0.3321 | -0.3025 | -0.2852 | -0.6423 | -0.3726
helena | -logloss | -2.5548 | - | -2.8187 | -2.6359 | -15.3287 | -
jannis | -logloss | -0.6653 | -0.7275 | -0.7232 | -0.6696 | -4.1177 | -0.7319
jungle_chess_2p… | -logloss | -0.1428 | -0.381 | -0.2366 | -0.1599 | -2.0565 | -0.2829
mfeat-factors | -logloss | -0.0823 | -0.1563 | -0.0941 | -0.093 | -0.542 | -0.1022
robert | -logloss | -1.3166 | -1.6828 | -1.6829 | -1.6571 | - | -1.9923
segment | -logloss | -0.0464 | -0.0854 | -0.0497 | -0.0615 | -0.4275 | -0.0522
shuttle | -logloss | -0.0008 | -0.0008 | -0.0005 | -0.0005 | -0.0059 | -0.0006
vehicle | -logloss | -0.3723 | -0.4812 | -0.3584 | -0.3816 | -2.5381 | -0.3745
volkert | -logloss | -0.8283 | -0.9197 | -0.8669 | -0.8054 | -1.7296 | -0.9945


providing details of a proof. A more detailed proof in a special case (based
on the approach in [9]) was given in [21].

######  Example 12.7.

Consider a stationary marked RCM with ${\mathbb{M}}$ as the space of all
locally finite simple counting measures on ${\mathbb{R}}^{d}$. Let
$\mathbb{Q}$ be a distribution of a simple stationary point process $\chi$
satisfying $\mathbb{Q}\\{0\\}=0$. For $x\in{\mathbb{R}}^{d}$ and
$p\in{\mathbb{M}}$ let $d(x,p)$ be the distance between $x$ and $p$. Similarly
as in Example 12.6 we consider a connection function of the form

$\displaystyle\varphi(x,p,q)=\rho(d(-x,p)^{-\alpha}d(x,q)^{-\alpha}\|x\|^{d})$

for a decreasing function $\rho\colon[0,\infty)\to[0,1]$ such that
$m_{\rho}:=\int\rho(\|x\|^{d})\,dx$ is positive and finite and where
$\alpha>0$ is a fixed parameter. By stationarity,

$\displaystyle\iint\varphi(x,p,q)\,\mathbb{Q}^{2}(d(p,q))\,dx$
$\displaystyle=\iint\rho(d(0,p)^{-\alpha}d(0,q)^{-\alpha}\|x\|^{d})\,dx\,\mathbb{Q}^{2}(d(p,q))$
$\displaystyle=m_{\rho}\int{\mathbf{1}}\\{d(0,p)<\infty,d(0,q)<\infty\\}d(0,p)^{\alpha}d(0,q)^{\alpha}\,\mathbb{Q}^{2}(d(p,q))$
$\displaystyle=m_{\rho}\bigg{(}\int
d(0,p)^{\alpha}\,\mathbb{Q}(dp)\bigg{)}^{2}.$

To ensure (4.2) we assume that $\int d(0,p)^{\alpha}\,\mathbb{Q}(dp)<\infty$,
which is a rather weak assumption. Since $m_{\rho}>0$ it is clear that
$d_{\varphi}(p)$ is positive whenever $p\neq 0$.

The function $\varphi(x,p,\cdot)$ is for all
$(x,p)\in{\mathbb{R}}^{d}\times{\mathbb{M}}$ non-decreasing with respect to
the natural partial ordering on ${\mathbb{M}}$. Therefore, if $\mathbb{Q}$ is
associated, then Theorem 5.13 implies that $\xi$ is irreducible. For instance
we might take $\mathbb{Q}$ as the distribution of a Poisson process; see e.g.
[26]. Hence Theorem 12.1 applies.

######  Remark 12.8.

Here we provide an example where Theorem 5.13 applies but not Theorem 5.9.
Consider Example 12.7 in the special case $d=1$. We assume that $\rho$ is
strictly decreasing. Take some $p_{0}\in{\mathbb{M}}$. We assert that
$\mathbb{P}(C)=0$, where

$\displaystyle C:=\\{\text{$\varphi(x,p,\chi)\geq\varphi(x,p,p_{0})$ for
$\lambda_{1}\otimes\mathbb{Q}$-a.e.\
$(x,p)\in{\mathbb{R}}\times{\mathbb{M}}$}\\}.$

Since $\rho$ is strictly decreasing and $\mathbb{P}(\chi=0)=0$ the equation
$\mathbb{P}(C)=0$ is equivalent with $\mathbb{P}(C^{\prime})=0$, where

$\displaystyle C^{\prime}:=\\{\text{$d(x,\chi)\geq d(x,p_{0})$ for
$\lambda_{1}$-a.e.\ $x\in{\mathbb{R}}$}\\}.$

Let us first assume that $p_{0}[a,\infty)=0$ for some $a\in{\mathbb{R}}$. Then
we have for each $c>0$ that

$\displaystyle C^{\prime}\subset\\{\text{$d(x,\chi)>c$ for $\lambda_{1}$-a.e.\
$x>a+c$}\\}=\\{\text{$\chi[x-c,x+c]=0$ for $\lambda_{1}$-a.e.\ $x>a+c$}\\}.$

This event is contained in $\\{\chi(a,a+2c)=0\\}$, whose probability does not
depend on $a$ and tends to $0$ as $c\to\infty$; see [22, Lemma 30.9 (i)].
Similarly we can treat the case $p_{0}(-\infty,a]=0$ for some
$a\in{\mathbb{R}}$.

Let us assume now that $p_{0}$ has atoms $\cdots<x_{-1}<x_{0}<x_{1}<\cdots$
with $\lim_{n\to-\infty}x_{n}=-\infty$ and $\lim_{n\to\infty}x_{n}=\infty$.
Take $n\in{\mathbb{Z}}$ and assume that $\chi(x_{n},x_{n+1}]\geq 1$. Then
$d(x,\chi)\geq d(x,p_{0})$ cannot hold for $\lambda_{1}$-a.e.
$x\in[x_{n},x_{n+1}]$. Hence
$C^{\prime}\subset\bigcap\limits_{n\in{\mathbb{Z}}}\\{\chi(x_{n},x_{n+1}]=0\\}=\\{\chi({\mathbb{R}})=0\\}$.
Therefore $\mathbb{P}(C^{\prime})=0$ so that the assumptions of Theorem 5.9
are not satisfied.

Acknowledgement: This research was supported by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) as part of the DFG
priority program ‘Random Geometric Systems’ SPP 2265) under grant LA 965/11-1.
The authors wish to thank Markus Heydenreich for several stimulating
discussions.

## References

  * [1] Aizenman, M., Kesten, H. and Newman, C.M. (1987). Uniqueness of the infinite cluster and continuity of connectivity functions for short and long range percolation. Commun. Math. Phys. 111, 505–531.
  * [2] Aldous, D. and Steele, M. (1992). Asymptotics for Euclidean minimal spanning trees on random points. Probab. Theory Relat. Fields 92, 247–258.
  * [3] Alexander, K.S. (1995). Percolation and minimal spanning forests in infinite graphs. Ann. Probab. 23, 87–104.
  * [4] Alexander, K.S. (1995). Simultaneous uniqueness of infinite clusters in stationary random labeled graphs. Commun. Math. Phys. 172, 221–221.
  * [5] Betsch, S. and Last, G. (2023). On the uniqueness of Gibbs distributions with a non-negative and subcritical pair potential. Ann. Inst. Henri Poincare (B) Probab. Stat. 59, 706–-725.
  * [6] Bezuidenhout, C.E., Grimmett, G. and Loffler, A. (1998). Percolation and minimal spanning trees. J. Statist. Phys. 92, 1–34.
  * [7] Bollobás, B., Janson, S. and Riordan, O. (2007). The phase transition in inhomogeneous random graphs. Random Struct. Algorithms., 31, 3–122.
  * [8] Bunde, A. and Havlin, S. (Eds.) (2012). Fractals and Disordered Systems. Springer Science & Business Media.
  * [9] Burton, R.M. and Keane, M. (1989). Density and uniqueness in percolation. Comm. Math. Phys. 121, 501–505.
  * [10] Caicedo, A. and Dickson, M. (2023). Critical exponents for marked random connection models. arXiv:2305.07398.
  * [11] Deprez, P. and Wüthrich, M.V. (2015). Poisson heterogeneous random-connection model. arXiv:1312.1948
  * [12] Dickson, M. and Heydenreich, M. (2022). The triangle condition for the marked random connection model. arXiv:2210.07727.
  * [13] Durrett, R. and Nguyen, B. (1985). Thermodynamic inequalities for percolation. Comm. Math. Phys. 99, 253–269.
  * [14] Folland, G.B. (1999). Real Analysis, Modern Techniques and Applications, 2nd Edition, John Wiley & Sons.
  * [15] Gandolfi, A., Keane, M.S. and Newman, C.M. (1992). Uniqueness of the infinite component in a random graph with applications to percolation and spin glasses. Probab. Theory Relat. Fields 92, 511–527.
  * [16] Gouéré, J.B. (2008). Subcritical regimes in some models of continuum percolation. Ann. Probab. 36, 1209–1220.
  * [17] Gracar, P., Lüchtrath, L. and Mörters, P. (2021). Percolation phase transition in weight-dependent random connection models. Adv. Appl. Probab. 53 (4), 1090–1114.
  * [18] Gracar, P., Heydenreich, M., Mönch, C. and Mörters, P. (2022). Recurrence versus transience for weight-dependent random connection models. Electron. J. Probab. 27, 1–31.
  * [19] Hall, P. (1985). On continuum percolation. Ann. Probab. 13, 1250–1266.
  * [20] Heydenreich, M., van der Hofstad, R., Last, G. and Matzke, K. (2024). Lace expansion and mean-field behavior for the random connection model. To appear in: Ann. Inst. H. Poincaré Probab. Statist.
  * [21] Jacob, E. and Mörters, P. (2017). Robustness of scale-free spatial networks. Ann. Appl. Probab. 45, 1680–1722.
  * [22] Kallenberg, O. (2021).  Foundations of Modern Probability.  3rd edn. Springer, Cham.
  * [23] Jiang, J., Zhang, S. and Guo, T. (2011). Russo’s formula, uniqueness of the infinite cluster, and continuous differentiability of free energy for continuum percolation. J. Appl. Probab., 48, 597–610.
  * [24] Last, G., Nestmann, F. and Schulte, M. (2021). The random connection model and functions of edge-marked Poisson processes: Second order properties and normal approximation. Ann. Appl. Probab. 31, 128–168.
  * [25] Last, G. and Penrose, M. (2018). Lectures on the Poisson Process. Cambridge University Press, Cambridge.
  * [26] Last, G., Szekli, R., Yogeshwaran, D. (2020). Some remarks on associated random fields, random measures and point processes. ALEA, Lat. Am. J. Probab. Math. Stat. 17, 355–374.
  * [27] Last, G. and Ziesche S. (2017). On the Ornstein–Zernike equation for stationary cluster processes and the random connection model. Adv. Appl. Probab. 49 (4), 1260-1287.
  * [28] Lindqvist, B.H. (1988). Association of probability measures on partially ordered spaces. J. Multivar. Anal. 26, 111–132.
  * [29] Meester, R., Penrose, M.D. and Sarkar, A. (1997). The random connection model in high dimensions. Stat. Probab. Lett. 35, 145–153.
  * [30] Meester, R. and Roy, R. (1994). Uniqueness of unbounded occupied and vacant components in Boolean models. Ann. Appl. Probab. 4, 933–951.
  * [31] Meester, R. and Roy, R. (1996). Continuum Percolation. Cambridge University Press, Cambridge.
  * [32] Nash-Williams, C. St. JA. (1971). Hamiltonian lines in infinite graphs with few vertices of small valency. Aequationes Math. 7, 59–81.
  * [33] Nguyen, X.X. and Zessin, H. (1976). Punktprozesse mit Wechselwirkung. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 37, 91–126.
  * [34] Penrose, M.D. (1991). On a continuum percolation model. Adv. Appl. Probab. 23 (3), 536–556.
  * [35] Penrose, M.D. (2016). Connectivity of soft random geometric graphs. Ann. Appl. Probab. 26 (2), 986–1028.
  * [36] Rockafeller, T.R. (1969). Convex Analysis. Princeton Univ. Press, Princeton.
  * [37] Shiryaev, A.N. (2016). Probability. Springer.
  * [38] Schneider, R. and Weil, W. (2008). Stochastic and Integral Geometry. Probability and Its Applications. Springer, Berlin.