MultiMCD     $\bm{0.353 \pm 0.005}$     $0.253 \pm 0.005$     $-0.253 \pm 0.005$     $\bm{0.571 \pm 0.000}$     $0.122 \pm 0.001$     $-0.122 \pm 0.001$

SWAG     $0.308 \pm 0.009$     $0.501 \pm 0.007$     $-0.500 \pm 0.007$     $0.520 \pm 0.003$     $0.327 \pm 0.003$     $-0.327 \pm 0.003$

MultiSWAG     $0.338 \pm 0.003$     $0.270 \pm 0.003$     $-0.270 \pm 0.003$     $0.570 \pm 0.001$     $0.124 \pm 0.001$     $-0.124 \pm 0.001$

LL SWAG     $0.307 \pm 0.006$     $0.514 \pm 0.001$     $-0.514 \pm 0.002$     $0.516 \pm 0.005$     $0.343 \pm 0.005$     $-0.343 \pm 0.005$

LL Laplace     $0.217 \pm 0.012$     $0.583 \pm 0.015$     $-0.583 \pm 0.015$     $0.374 \pm 0.013$     $0.445 \pm 0.012$     $-0.445 \pm 0.012$

LL MultiLaplace     $0.301 \pm 0.004$     $\bm{0.123 \pm 0.004}$     $\bm{-0.123 \pm 0.004}$     $0.517 \pm 0.002$     $\bm{0.059 \pm 0.002}$     $\bm{0.020 \pm 0.002}$

LL BBB     $0.306 \pm 0.008$     $0.448 \pm 0.010$     $-0.448 \pm 0.010$     $0.509 \pm 0.003$     $0.293 \pm 0.003$     $-0.293 \pm 0.003$

LL MultiBBB     $0.339 \pm 0.006$     $0.233 \pm 0.008$     $-0.233 \pm 0.008$     $0.561 \pm 0.001$     $0.102 \pm 0.001$     $-0.102 \pm 0.001$

Rank-1 VI     $0.296 \pm 0.007$     $0.497 \pm 0.004$     $-0.497 \pm 0.004$     $0.512 \pm 0.003$     $0.328 \pm 0.003$     $-0.328 \pm 0.003$

LL iVON     $0.300 \pm 0.009$     $0.514 \pm 0.009$     $-0.514 \pm 0.009$     $0.505 \pm 0.003$     $0.348 \pm 0.002$     $-0.348 \pm 0.002$

LL MultiiVON     $0.341 \pm 0.004$     $0.241 \pm 0.004$     $-0.241 \pm 0.004$     $0.560 \pm 0.001$     $0.112 \pm 0.002$     $-0.112 \pm 0.002$

SVGD     $0.310 \pm 0.007$     $0.526 \pm 0.009$     $-0.526 \pm 0.009$     $0.517 \pm 0.004$     $0.354 \pm 0.003$     $-0.354 \pm 0.003$

FMoW-wilds: Detailed results on the o.o.d. evaluation split on the worst region as measured by the accuracy on each region and across all regions. For the MultiX models, the entire model is ensembled, but the single-mode approximation is only applied to the classification head. WR = Worst Region, LL = Last-Layer.

§.§.§ RxRx1-wilds

Following <cit.>, we finetune a ResNet-50 <cit.>, pretrained on ImageNet <cit.>, for 90 epochs with the Adam optimizer <cit.>.
For each model, we replace the linear classification layer of the ResNet-50 by a randomly initialized one of the appropriate output dimension.
Following <cit.>, we use a learning rate of $10^{-4}$ and weight decay $10^{-5}$.
For MCD, we try dropout rates of $0.1$ and $0.2$ and select $0.1$ due to a slightly better accuracy on the evaluation split.
iVON uses a prior precision of $100$ as optimized by a grid search.
We use five seeds per model and build all ensembles by training six models independently and leaving out a different model for each of the five evaluation runs.

data/rxrx1.datood accuracyood accuracy_stdood seceood sece_std
legend columns = 2,
legend style =
at = (0.5, -0.2),
legend cell align=left
xticklabel style=
/pgf/number format/fixed,
/pgf/number format/precision=3

bbb_1BBBcolor=bbb, single
BBB ($\sigma=1.0, \lambda=1.0$);
bbb_prior0.1_1BBBcolor=swag, single
BBB ($\sigma=0.1, \lambda=1.0$);
bbb_prior0.5_1BBBcolor=laplace, single
BBB ($\sigma=0.5, \lambda=1.0$);
bbb_kl0.2_1BBBcolor=mcd, single
BBB ($\sigma=1.0, \lambda=0.2$);
rank1_1Rank-1 VIcolor=rank1, single
ll_ivon_1iVONcolor=ivon, single
Rank-1 VI;

RxRx1-wilds: BBB and Rank-1 VI under different prior variances $\sigma$ and posterior temperatures $\lambda$. We multiply $\lambda$ to the KL divergence in the ELBO during training to reduce the regularization strength. However, neither small prior variances nor colder posteriors make BBB competitive with the non-VI algorithms.

1lModel     1ci.d. Accuracy     1ci.d. ECE     1ci.d. sECE    1co.o.d. Accuracy     1co.o.d. ECE     1co.o.d. sECE

MAP     $0.105 \pm 0.002$     $0.232 \pm 0.015$     $-0.232 \pm 0.015$     $0.083 \pm 0.001$     $0.262 \pm 0.015$     $-0.262 \pm 0.015$

Deep Ensemble     $0.156 \pm 0.001$     $0.066 \pm 0.001$     $-0.026 \pm 0.002$     $0.122 \pm 0.000$     $0.071 \pm 0.001$     $-0.061 \pm 0.002$

MCD     $0.106 \pm 0.001$     $0.257 \pm 0.003$     $-0.257 \pm 0.003$     $0.083 \pm 0.001$     $0.288 \pm 0.004$     $-0.288 \pm 0.004$

MultiMCD     $0.158 \pm 0.001$     $0.069 \pm 0.001$     $-0.035 \pm 0.001$     $0.121 \pm 0.000$     $0.081 \pm 0.001$     $-0.073 \pm 0.000$

SWAG     $0.110 \pm 0.001$     $0.269 \pm 0.009$     $-0.269 \pm 0.009$     $0.086 \pm 0.001$     $0.301 \pm 0.010$     $-0.301 \pm 0.010$

MultiSWAG     $\bm{0.161 \pm 0.001}$     $0.075 \pm 0.002$     $-0.042 \pm 0.001$     $\bm{0.126 \pm 0.001}$     $0.085 \pm 0.001$     $-0.078 \pm 0.001$

LL Laplace     $0.077 \pm 0.000$     $0.034 \pm 0.004$     $-0.011 \pm 0.009$     $0.061 \pm 0.002$     $0.037 \pm 0.007$     $-0.028 \pm 0.007$

LL BBB ($\sigma=1.0, \lambda=1.0$)     $0.046 \pm 0.001$     $0.032 \pm 0.002$     $-0.032 \pm 0.002$     $0.038 \pm 0.000$     $0.045 \pm 0.002$     $-0.045 \pm 0.002$

LL BBB ($\sigma=0.5, \lambda=1.0$)     $0.040 \pm 0.001$     $\bm{0.007 \pm 0.002}$     $\bm{-0.006 \pm 0.003}$     $0.035 \pm 0.001$     $0.015 \pm 0.003$     $-0.015 \pm 0.003$

LL BBB ($\sigma=0.1, \lambda=1.0$)     $0.036 \pm 0.002$     $0.010 \pm 0.001$     $\bm{0.003 \pm 0.004}$     $0.031 \pm 0.001$     $\bm{0.008 \pm 0.002}$     $\bm{-0.006 \pm 0.004}$

LL BBB ($\sigma=1.0, \lambda=0.2$)     $0.054 \pm 0.001$     $0.102 \pm 0.002$     $-0.102 \pm 0.002$     $0.046 \pm 0.001$     $0.117 \pm 0.002$     $-0.117 \pm 0.002$

Rank-1 VI     $0.053 \pm 0.001$     $0.068 \pm 0.001$     $-0.068 \pm 0.001$     $0.043 \pm 0.001$     $0.078 \pm 0.000$     $-0.078 \pm 0.000$

LL iVON     $0.003 \pm 0.000$     $\bm{0.008 \pm 0.000}$     $-0.008 \pm 0.000$     $0.003 \pm 0.000$     $\bm{0.009 \pm 0.001}$     $\bm{-0.009 \pm 0.001}$

SVGD     $0.102 \pm 0.001$     $0.354 \pm 0.005$     $-0.354 \pm 0.005$     $0.081 \pm 0.002$     $0.382 \pm 0.005$     $-0.382 \pm 0.005$

RxRx1-wilds: Detailed results on the i.d. and the o.o.d. evaluation split. LL = Last-Layer. For the MultiX models, the entire model is ensembled, but the single-mode approximation is only applied to the classification head. We evaluate multiple hyperparameter combinations for LL BBB, as the standard parameters do not perform well. The failure of VI is equally present with LL BBB, LL Rank-1 VI, and LL iVON.

§.§.§ CivilComments-wilds

We use the pretrained DistilBERT <cit.> model from HuggingFace transformers <cit.> with a classification head consisting of two linear layers with a ReLU nonlinearity and a Dropout unit with a drop rate of $0.2$ between them.
Following <cit.>, we finetune the pretrained checkpoint with a learning rate of $1 \cdot 10^{-5}$ and, where applicable, a weight decay factor of $1 \cdot 10^{-2}$ for three epochs using the Adam optimizer <cit.>.
SWAG collects ten parameter samples during the last two epochs of training.
iVON uses a prior precision of $500$, as optimized by a grid search.
We use five seeds for all non-ensembled models.
The ensembles are build from four of the five single-model versions, leaving out a different member per model to create five different ensembled models of four members each.

We note in the main paper that MCD results in less accurate and more overconfident models.
We investigate this further by experimenting with different dropout rates in <Ref>.
While a dropout rate of $0.1$ had no impact, dropout rates of $0.05$ and $0.01$ lead to progressively better accuracy and calibration, coming close to MAP.
However, there is still no accuracy or calibration benefit to be gained from using MCD.

data/civil.datworst_acc accuracyworst_acc accuracy_stdworst_acc seceworst_acc sece_std
xlabel=Worst Group Accuracy,
legend columns=2,
legend style=
legend cell align=left

mcd$p=0.2$color=mcd, single
mcd-p0.05$p=0.05$color=laplace, single
mcd-p0.01$p=0.01$color=swag, single
mapMAPcolor=map, single

CivilComments-wilds: Accuracy and sECE for different MCD dropout rates $p$. While smaller dropout rates improve the accuracy, the models are still less accurate and more overconfident than MAP.

1lModel     1cWG Accuracy     1cWG ECE     1cWG sECE     1cAvg Accuracy     1cAvg ECE     1cAvg sECE

MAP     $0.420 \pm 0.021$     $0.353 \pm 0.025$     $-0.353 \pm 0.025$     $0.916 \pm 0.001$     $0.012 \pm 0.003$     $-0.012 \pm 0.003$

Deep Ensemble     $0.419 \pm 0.008$     $0.349 \pm 0.010$     $-0.349 \pm 0.010$     $0.916 \pm 0.000$     $0.010 \pm 0.001$     $-0.010 \pm 0.001$

MCD ($p=0.2$)     $0.326 \pm 0.023$     $0.417 \pm 0.030$     $-0.417 \pm 0.030$     $0.918 \pm 0.000$     $0.007 \pm 0.005$     $\bm{0.006 \pm 0.006}$

MCD ($p=0.1$)     $0.325 \pm 0.021$     $0.418 \pm 0.027$     $-0.418 \pm 0.027$     $0.918 \pm 0.000$     $\bm{0.007 \pm 0.005}$     $\bm{0.006 \pm 0.005}$

MCD ($p=0.05$)     $0.364 \pm 0.018$     $0.393 \pm 0.024$     $-0.393 \pm 0.024$     $0.918 \pm 0.000$     $\bm{0.005 \pm 0.002}$     $\bm{-0.003 \pm 0.004}$

MCD ($p=0.01$)     $0.396 \pm 0.015$     $0.374 \pm 0.023$     $-0.374 \pm 0.023$     $0.917 \pm 0.000$     $0.011 \pm 0.003$     $-0.011 \pm 0.003$

MultiMCD ($p=0.2$)     $0.326 \pm 0.005$     $0.412 \pm 0.007$     $-0.412 \pm 0.007$     $0.919 \pm 0.000$     $0.009 \pm 0.002$     $0.009 \pm 0.002$

SWAG     $0.448 \pm 0.021$     $\bm{0.197 \pm 0.041}$     $\bm{-0.184 \pm 0.027}$     $0.877 \pm 0.024$     $0.152 \pm 0.024$     $0.152 \pm 0.024$

MultiSWAG     $0.429 \pm 0.016$     $\bm{0.183 \pm 0.018}$     $\bm{-0.183 \pm 0.018}$     $0.901 \pm 0.002$     $0.184 \pm 0.002$     $0.184 \pm 0.002$

LL Laplace     $0.424 \pm 0.016$     $0.348 \pm 0.018$     $-0.347 \pm 0.018$     $0.916 \pm 0.001$     $0.011 \pm 0.002$     $-0.011 \pm 0.002$

LL MultiLaplace     $0.420 \pm 0.008$     $0.348 \pm 0.010$     $-0.348 \pm 0.010$     $0.916 \pm 0.000$     $0.010 \pm 0.001$     $-0.009 \pm 0.001$

LL BBB     $\bm{0.537 \pm 0.032}$     $0.362 \pm 0.032$     $-0.361 \pm 0.033$     $0.918 \pm 0.002$     $0.056 \pm 0.002$     $-0.056 \pm 0.002$

LL MultiBBB     $\bm{0.525 \pm 0.012}$     $0.338 \pm 0.012$     $-0.338 \pm 0.012$     $0.922 \pm 0.000$     $0.041 \pm 0.001$     $-0.041 \pm 0.001$

Rank-1 VI     $\bm{0.540 \pm 0.028}$     $0.373 \pm 0.030$     $-0.373 \pm 0.030$     $0.917 \pm 0.002$     $0.060 \pm 0.002$     $-0.060 \pm 0.002$

LL iVON     $0.480 \pm 0.045$     $0.421 \pm 0.048$     $-0.421 \pm 0.048$     $0.919 \pm 0.003$     $0.054 \pm 0.002$     $-0.054 \pm 0.002$

LL MultiiVON     $0.465 \pm 0.011$     $0.396 \pm 0.015$     $-0.396 \pm 0.015$     $\bm{0.924 \pm 0.001}$     $0.039 \pm 0.001$     $-0.039 \pm 0.001$

SVGD     $0.384 \pm 0.068$     $0.380 \pm 0.079$     $-0.379 \pm 0.079$     $0.915 \pm 0.003$     $0.011 \pm 0.005$     $-0.008 \pm 0.008$

CivilComments-wilds: Detailed results on the o.o.d. evaluation split for the worst group (WG, determined by the accuracy on each group) and averaged over all groups. LL = Last-Layer.

§.§.§ Amazon-wilds

We use the pretrained DistilBERT <cit.> model from HuggingFace transformers <cit.> with a classification head consisting of two linear layers with a ReLU nonlinearity and a Dropout unit with a drop rate of $0.2$ between them.
Following <cit.>, we finetune the pretrained checkpoint with a learning rate of $10^{-5}$ and, where applicable, a weight decay factor of $10^{-2}$ using the Adam optimizer <cit.>.
Contrary to <cit.>, we finetune for five epochs, as we find that the validation accuracy is still increasing after three epochs.
SWAG collects $30$ parameter samples during the last two epochs of training.
We also experiment with last-layer versions of SWAG and MCD, but find both to perform very similar to MAP (see <Ref>).
iVON uses a prior precision of $500$, as optimized by a grid search.
We use six seeds for all non-ensembled models.
The ensembles are build from five of the six single-model versions, leaving out a different member per model to create five different ensembled models of five members each.

1lModel     1co.o.d. $10\%$ Accuracy     1co.o.d. Avg Accuracy     1co.o.d. ECE     1co.o.d. sECE     1ci.d. $10\%$ Accuracy     1ci.d. Avg Accuracy     1ci.d. ECE     1ci.d. sECE

MAP     $0.453 \pm 0.010$     $0.655 \pm 0.003$     $0.067 \pm 0.006$     $-0.067 \pm 0.006$     $0.477 \pm 0.008$     $0.678 \pm 0.002$     $0.049 \pm 0.007$     $-0.049 \pm 0.007$

Deep Ensemble     $0.453 \pm 0.000$     $0.659 \pm 0.001$     $0.058 \pm 0.002$     $-0.058 \pm 0.002$     $0.480 \pm 0.000$     $0.682 \pm 0.000$     $0.040 \pm 0.002$     $-0.040 \pm 0.002$

MCD     $0.447 \pm 0.013$     $0.657 \pm 0.002$     $0.020 \pm 0.011$     $-0.019 \pm 0.012$     $0.472 \pm 0.012$     $0.678 \pm 0.001$     $0.015 \pm 0.006$     $\bm{-0.002 \pm 0.013}$

MultiMCD     $0.451 \pm 0.005$     $0.660 \pm 0.001$     $\bm{0.012 \pm 0.003}$     $\bm{-0.012 \pm 0.003}$     $0.475 \pm 0.007$     $0.682 \pm 0.000$     $\bm{0.007 \pm 0.002}$     $\bm{0.005 \pm 0.003}$

LL MCD     $0.451 \pm 0.008$     $0.656 \pm 0.003$     $0.069 \pm 0.008$     $-0.069 \pm 0.008$     $0.478 \pm 0.011$     $0.679 \pm 0.002$     $0.051 \pm 0.009$     $-0.051 \pm 0.009$

SWAG     $0.436 \pm 0.011$     $0.639 \pm 0.006$     $0.032 \pm 0.003$     $0.031 \pm 0.004$     $0.460 \pm 0.010$     $0.658 \pm 0.006$     $0.047 \pm 0.004$     $0.047 \pm 0.004$

MultiSWAG     $0.443 \pm 0.005$     $0.646 \pm 0.001$     $0.040 \pm 0.001$     $0.040 \pm 0.001$     $0.469 \pm 0.005$     $0.667 \pm 0.001$     $0.057 \pm 0.001$     $0.057 \pm 0.001$

LL SWAG     $0.452 \pm 0.012$     $0.656 \pm 0.003$     $0.048 \pm 0.009$     $-0.048 \pm 0.009$     $0.474 \pm 0.010$     $0.679 \pm 0.002$     $0.031 \pm 0.008$     $-0.030 \pm 0.009$

LL Laplace     $0.455 \pm 0.009$     $0.654 \pm 0.003$     $0.067 \pm 0.006$     $-0.067 \pm 0.006$     $0.482 \pm 0.009$     $0.678 \pm 0.002$     $0.048 \pm 0.007$     $-0.048 \pm 0.007$

LL MultiLaplace     $0.453 \pm 0.000$     $0.659 \pm 0.001$     $0.058 \pm 0.001$     $-0.058 \pm 0.001$     $0.480 \pm 0.000$     $0.682 \pm 0.000$     $0.040 \pm 0.002$     $-0.040 \pm 0.002$

LL BBB     $0.527 \pm 0.006$     $0.695 \pm 0.007$     $0.154 \pm 0.005$     $-0.154 \pm 0.005$     $0.560 \pm 0.000$     $0.730 \pm 0.006$     $0.128 \pm 0.004$     $-0.128 \pm 0.004$

LL MultiBBB     $\bm{0.533 \pm 0.000}$     $\bm{0.709 \pm 0.002}$     $0.105 \pm 0.001$     $-0.105 \pm 0.001$     $\bm{0.573 \pm 0.000}$     $\bm{0.746 \pm 0.001}$     $0.079 \pm 0.002$     $-0.079 \pm 0.002$

Rank-1 VI     $0.527 \pm 0.005$     $0.695 \pm 0.003$     $0.173 \pm 0.004$     $-0.173 \pm 0.004$     $0.558 \pm 0.004$     $0.729 \pm 0.003$     $0.147 \pm 0.004$     $-0.147 \pm 0.004$

LL iVON     $0.458 \pm 0.010$     $0.661 \pm 0.002$     $0.053 \pm 0.010$     $-0.053 \pm 0.010$     $0.484 \pm 0.009$     $0.684 \pm 0.002$     $0.037 \pm 0.010$     $-0.037 \pm 0.010$

LL MultiiVON     $0.459 \pm 0.007$     $0.665 \pm 0.001$     $0.045 \pm 0.002$     $-0.045 \pm 0.002$     $0.484 \pm 0.005$     $0.687 \pm 0.001$     $0.029 \pm 0.003$     $-0.029 \pm 0.002$

SVGD     $0.456 \pm 0.010$     $0.661 \pm 0.003$     $0.049 \pm 0.005$     $-0.049 \pm 0.005$     $0.477 \pm 0.010$     $0.682 \pm 0.002$     $0.035 \pm 0.005$     $-0.034 \pm 0.005$

Amazon-wilds: Detailed results on the i.d. and the o.o.d. evaluation split. LL = Last-Layer.


# Selected Machine Learning of HOMO-LUMO gaps with Improved Data-Efficiency

Bernard Mazouin  University of Vienna, Faculty of Physics, Kolingasse 14-16,
1090 Vienna, Austria  Alexandre Alain Schöpfer  Department of Chemistry,
University of Basel, Klingelbergstrasse 70, 4056 Basel, Switzerland  O.
Anatole von Lilienfeld<EMAIL_ADDRESS>University of
Vienna, Faculty of Physics, Kolingasse 14-16, 1090 Vienna, Austria  Department
of Chemistry, University of Basel, Klingelbergstrasse 70, 4056 Basel,
Switzerland

(August 27, 2024)

###### Abstract

Quantum Machine Learning (QML) models of molecular HOMO-LUMO-gaps often
struggle to achieve satisfying data-efficiency as measured by decreasing
prediction errors for increasing training set sizes. Partitioning training
sets of organic molecules (QM7 and QM9-data-sets) into three classes [systems
containing either aromatic rings and carbonyl groups, or single unsaturated
bonds, or saturated bonds] prior to training results in independently trained
QML models with improved learning rates. The selected QML models of band-gaps
(at GW, B3LYP, and ZINDO level of theory) reach mean absolute prediction
errors of $\sim$0.1 eV for up to an order of magnitude fewer training
molecules than for conventionally trained models. Direct comparison to
$\Delta$-QML models of band-gaps suggest that selected QML possesses
substantially more data-efficiency. The findings suggest that selected QML,
e.g. based on simple classifications prior to training, could help to
successfully tackle challenging quantum property screening tasks of large
libraries with high fidelity and low computational burden.

Quantum Machine Learning, HOMO-LUMO-gap

##  I Introduction

Machine Learning (ML) based surrogate models of quantum properties have gained
a lot of traction in recent years Ramakrishnan2017 ; Lilienfeld2018 ;
Lilienfeld2020 ; Lilienfeld2020a ; Huang2021 . This rise in interest is partly
driven by the computational efficiency of ML algorithms that typically outpace
the conventional quantum chemistry methods which attempt to numerically solve
sophisticated approximations to the electronic Schrödinger equation. The
application of these algorithms to Chemical Compound Space (CCS) is commonly
referred to as Quantum Machine Learning (QML). During training, QML models get
parameterized in terms of a heuristic functional form which encodes a
statistical relation between sample training molecules and their corresponding
labels (quantum property). The resulting QML model can subsequently be used to
make quantum property predictions throughout CCS, i.e. for unknown out-of-
sample molecules. Since its inception in 2012 Rupp2012 , QML has already been
applied to a variety of chemical classes including, among others, organic
molecules ramakrishnan2015many ; Faber2017 ; Stuke2019 , amino acids Stuke2019
, polymers Patra2020 , or solids faber2016machine ; Pilania2017 ; Isayev2017 ;
Li2018 ; Butler2018 . Within these applications, it has been used to predict
ab initio thermodynamic properties such as atomization energies Rupp2012 ;
Hansen2013 ; ramakrishnan2015many ; Faber2017 , energy above convex hull
Li2018 , or free energy of solvation Weinreich2021 , as well as electronic
properties such as HOMO and LUMO energies or dipole moments Montavon2013 ;
Ramakrishnan2015a ; ramakrishnan2015many ; Faber2017 ; Schuett2019 ;
christensen2019operators . Some state-of-the-art QML models can reach an
accuracy on par with quantum chemistry algorithms already for modest training
set sizes Faber2017 , and are thus well positioned for their direct
application in computational materials design efforts Pyzer-Knapp2015 ;
Shandiz2016 ; Gomez-Bombarelli2016 ; Sendek2019 ; Zunger2018 ; Jorgensen2018 .

Figure 1: Illustration of frontier molecular orbitals and eigenvalues being
dominated by simple features such as bond-saturation: Compositionally and
structurally similar molecules (cyclohexanol, cyclohex-2-enol and phenol)
exhibit vast differences in HOMOs, LUMOs, and eigenvalues. The orbitals are
visualized with Jmol Jmol  using results from B3LYP/6-31G(2df,p) calculations
performed with ORCA 4.0.1 Neese2012 .

Not surprisingly, the importance of rapid yet accurate QM property predictions
has inspired the development of specialized ML methods. For example, optimized
representations or Neural Network architectures have been designed just for
this purpose Behler2007 ; Montavon2013 ; Schuett2014 ; Faber2017 ; Pereira2017
; Stuke2019 ; Schuett2019 ; Unke2019 ; Westermayr2020a . In particular, one
can adjust the QML procedure to the property of interest by including more
information about the underlying physics in it. In order to obtain QML models
with higher data-efficiency for atomization energies, more descriptive
representations such as SLATM Huang2020  and FCHL faber2018alchemical ;
Christensen2020 , which include 3-body-terms and physically motivated power
laws, yield better results than the more heuristic CM Hansen2013  or BoB
Hansen2015  representations, which merely encode the nuclear repulsion terms.
The integration of gradients in KRR has led to reduced errors for response
properties such as the dipole moment or forces Chmiela2017 ; Chmiela2018 ;
christensen2019operators ; Christensen2020a . Furthermore, a biased selection
of training samples will also lead to QML models with improved accuracy
Browning2017 ; Huang2020 .

Figure 2: Left: Normalized histogram and kernel density estimate (solid line)
of HOMO-LUMO gaps in QM9 data-set. Gaps of 3 similar molecules (cyclohexanol,
cyclohex-2-enol and phenol) are indicated. Right: The HOMO and LUMO energies
plotted against the HOMO-LUMO gaps with the respective correlation
coefficients.

Among the various QM properties frequently evaluated, the eigenvalues of the
frontier orbitals, i.e. highest occupied and lowest unoccupied molecular
orbitals (HOMO and LUMO), are of special interest. These MO energies are
intimately related to chemical reactions, polarizability, the optical gap and
excitation energies. Their prediction often plays an important role for design
decisions in the development of technological applications such as synthesis
planning, electrochromic devices, light-emission diodes or photovoltaic solar
panels Kubatkin2003 ; Roncali2007 ; Jurow2010 ; Beaujuge2010 ; Tao2017 ;
Stoliaroff2020 . Interestingly, the generation of accurate QML models of
frontier orbital eigenvalues proves more difficult than for other quantum
properties—even when using molecular training sets of considerable size.
Consequently, significant research efforts are currently being made in order
to devise QML models of MO energies with improved data-efficiency.

We believe that this difficulty is partly, if not mostly, due to the intensive
nature of MO energies. Molecules with very similar stoichiometry and geometry
do not necessarily have similar HOMO-LUMO gap values (see e.g. the molecules
drawn in Fig. 1), whereas structurally dissimilar molecules can have very
close values. While the latter can be resolved easily by allowing for QML
models which are not monotonic in CCS, the former point represents the actual
challenge since all ML models are based on similarity arguments and smoothness
assumptions. The lack of smoothness (as on display in Fig. 1) indicates the
presence of additional dimensions, not properly being taken into account when
relying on the conventional QML representations Lilienfeld2020 . Inspection of
HOMO-LUMO gap distributions (see Fig. 2), however, indicates the superposition
of multiple groups, possibly accounting for the ‘hidden’ dimensions. In this
work, we have investigated this hypothesis in more depth.

We present the SML (Selected ML) method, a divide-and-conquer-like strategy
which improves the data-efficiency of QML models of MO energies. Prior to
training, we partition the training data into smaller classes, and we train
QML models separately for each class. The idea for such a classification is
based on the peculiar shape of the distribution of HOMO-LUMO-gaps obtained
from B3LYP in QM9 or ZINDO in QM7b: it is multi-modal and appears to be
composed of 3 sub-distributions, one per peak (see Fig. 2 and 3). According to
frequency analysis the molecules can be easily classified into three groups,
solely based on simple structural features. The three example molecules
indicated in Fig. 2, that are each located close to a different peak of the
distribution, indicate such features (saturated bonds, unsaturated bond,
aromatic ring) that turn out to encode important information about the gap.
Fig. 1 also illustrates their dramatic effect on the character of their
frontier orbitals, and thereby on eigenvalues and their gap. Based on the QM9
analysis, we have defined a set of simple rules for classification which (vide
infra) results in subsequently trained QML models, henceforth dubbed SML, with
much improved learning curves.

Figure 3: Histograms and KDEs of the HOMO-LUMO gaps of all molecules from QM7b
at ZINDO, PBE0 and GW levels of theory and of all molecules from QM9 at B3LYP
level of theory.

##  II Data and Methods

###  II.1 Data

The QM7b data-set Blum2009 ; Montavon2013  contains properties of 7211 organic
molecules with up to 7 heavy atoms (C, O, N, S and Cl). These molecules were
derived originally from the GDB-13 data-set. Thermodynamic and electronic
properties are available at different levels of theory, which makes the data-
set suitable for $\Delta$-ML applications Ramakrishnan2015 . For this work, we
use the HOMO and LUMO energies as obtained from ZINDO Ridley1973 ; Zerner1991
and GW Hedin1965 ; Aryasetiawan1998  calculation for direct and $\Delta$-ML.
The HOMO-LUMO gaps correspond to the differences of LUMO and HOMO energies.

The QM9 data-set, published in 2014 by Ramakrishnan at al. Ruddigkeit2012 ;
Ramakrishnan2014 , consists of more than 133k organic molecules with up to 9
heavy atoms (C, N, O and F) with corresponding geometries, thermodynamic and
electronic properties. These molecules were obtained from the GDB-17 data-set
which contains over 166 billion molecular graphs. The properties were computed
using DFT/B3LYP Hohenberg1964 ; Kohn1965 ; Becke1993a  with a 6-31G(2d,f)
basis set. Over the last years, it has become an increasingly popular data-set
in the QML community as it has been used as a staple to benchmark new QML
models Gilmer2017 ; faber2018alchemical ; Stuke2019 ; Schuett2018 ; Chen2019 ;
Anderson2019 ; Lu2019 ; Liu2020 .

###  II.2 Frequency Analysis and Classification

We perform a frequency analysis to identify functional groups that relate to
the HOMO-LUMO gap in our molecules (see Fig. 4, panel a)). By screening for a
set of structural features and functional groups such as double bonds,
aromatic rings or carbonyl groups e.g. using SMILES Weininger1988 ;
Weininger1989 ; Weininger1990  strings and substructure matching as
implemented in RDKit rdkit , we tag the molecules in the data-set. For each
tag, we compute the the Kernel Density Estimation (KDE) of the HOMO-LUMO-gaps
of the matching molecules, normalize it with respect to the total number of
molecules in the entire data-set, and draw it over its KDE. By visual
inspection of the resulting plots we have detected those functional groups
which govern the assignment to one of the classes in the HOMO-LUMO-gap
distribution.

In the next step, based on the frequency analysis, we define simple rules to
separate the molecules into disjoint classes. We make sure that the class
distributions have a unimodal shape and coincide with the peaks of the total
distribution (see Fig. 4, panel b)). For example, the distribution of
saturated molecules in QM9 fits closely underneath the right peak, so that we
can assign all saturated molecules to the class corresponding to that peak.
The distribution of all carbonyl compounds however has two peaks that coincide
with the left and middle peaks. Therefore we have further subdivided the group
of carbonyl compounds by distinguishing, for instance, between those with
aromatic rings from those without, until one ends up with unimodal
subdistributions.

Figure 4: a) General procedure of the frequency analysis. We use SMILES
strings obtained from QM9 molecules to screen for features such as double
bonds, aromatic rings, carbonyl groups and so on. The plot at the bottom shows
the results of this frequency analysis. Each quadrant shows the distribution
of HOMO-LUMO-gaps of a subgroup of QM9 matching a given criterion. The top row
shows the effect of saturation ($n_{u}$: number of unsaturated bonds, i.e. any
double, triple or aromatic bond), the middle row shows the effect of elemental
constitution (hc: hydrocarbons, w/ N: with nitrogen and so on) and the bottom
row shows various kinds of carbonyl groups (columns 1 - 3) and amino acids
(column 4). b) Flowchart detailing the sequence of decisions that results in
our final classification. The distributions of the classes are highlighted in
plot on the bottom left.

###  II.3 Kernel Ridge Regression

The main idea behind supervised learning is to establish and exploit
statistical relations between inputs $\mathbf{X}_{i}$ and corresponding target
property label outputs $y_{i}$. In our case, the inputs are molecular
representations which, in strict correspondence to Schrödinger’s equation,
encode stoichiometry and geometry. We have relied on the SLATM representation
Huang2020 , which describes a molecule as a spectrum of atomic, 2-body and
3-body terms. The target labels are the properties of interest, i.e. the HOMO-
LUMO gaps and the individual frontier orbital energies. A training set
$\\{\mathbf{X}_{i},y_{i}\\}_{i=1}^{N_{\mathrm{tr}}}$ is a sample for which
both the inputs and target values are known, whereas for the test set only the
inputs $\\{\mathbf{X}_{j}\\}_{j=1}^{N_{\mathrm{te}}}$ are known, but the
target values unknown. The ML model uses the training data to infer a
statistical model that relates the input $\mathbf{X}_{i}$ to the output
$y_{i}$. This statistical model can then be applied to the molecules in the
test set in order to produce a prediction error estimate for the corresponding
properties. As such, ML circumvents numerically solving the Schrödinger
equation, and provides instead statistical estimates which are computationally
more efficient than state-of-the-art quantum chemistry calculations.

We are dealing with a regresssion problem where the task is to predict
continuous target values. Our method of choice is KRR Muller2001 ;
Schoelkopf2002 ; hastie2009elements ; rasmussen2006gaussian  due to its ease
of implementation and interpretability. Moreover, it has worked successfully
in numerous applications Ramakrishnan2017 . We note, however, that the first
QML models of frontier orbital eigenvalues were presented using neural
networks Montavon2013 , and that we believe that the choice of the specific
regressor is rather secondary, i.e. our procedure could be used in combination
with any other regressors just as well. In the following, we briefly outline
the KRR methodology only for the sake of completeness.

Within KRR, the prediction of a given property $\hat{y}_{i}$ is given by a sum
over kernel matrix elements $k_{ij}=k(\mathbf{X}_{i},\mathbf{X}_{j})$
multiplied by regression coefficients $\alpha_{j}$:

$\hat{y}_{i}=\sum_{j=1}^{N_{\mathrm{tr}}}k(\mathbf{X}_{i},\mathbf{X}_{j})\alpha_{j}.$
(1)

The $\mathbf{\alpha}$-coefficients are obtained by solving the following
system of linear equations:

$\mathbf{y}^{\mathrm{tr}}=(\mathbf{K}+\lambda\mathbf{I})\boldsymbol{\alpha}.$
(2)

The parameter $\lambda$ is a regularization coefficient, a.k.a. noise-level,
that smooths out the noise. However, since we are dealing with computed target
values that are noiseless to machine precision, we can fix $\lambda$ to
correspond to a small value such as $10^{-12}$. $\mathbf{I}$ is the identity
matrix. The kernel matrix $\mathbf{K}$, for which we employ the Laplacian
kernel function ($k_{ij}=\exp(-|\mathbf{X}_{i}-\mathbf{X}_{j}|_{1}/\sigma)$),
quantifies the similarity between any two representations of the $i$-th and
$j$-th molecules. By virtue of this kernel matrix, each test molecule is
compared to all the training molecules in order to make a prediction. The
parameter $\sigma$ modulates the sensitivity of the kernel and is optimized
via grid search cross validation within each training set in this work. To
evaluate the performance of our method, we use the target values of the test
set to calculate the mean absolute error (MAE) between reference and predicted
values. The logarithm of the prediction error generally decreases linearly
with the logarithm of the training set size ($\log{(E)}\propto-\log{(N_{\rm
tr}})$) Cortes1994 ; Mueller1996 , which is shown numerically in terms of so-
called learning curves. We have employed the QML package ASChristensen2017  to
perform our calculations.

###  II.4 $\Delta$-Machine Learning

In $\Delta$-ML Ramakrishnan2015 , correlations between different levels of
theory are exploited to obtain better predictions of properties calculated at
higher levels of theory for fewer training molecules. We consider two levels
of theory, a lower baseline, at which we know the output, and a higher target
line, for which we want to obtain predictions. A machine is trained on the
differences between the two levels. In other words, a QML model of a
correction to the baseline model is being generated.

After that, these predictions are added to the baseline to generate estimates
of the property at the higher level of theory:

$\hat{y}_{i}^{\mathrm{target}}=y_{i}^{\mathrm{baseline}}+\sum_{j=1}^{N_{\mathrm{tr}}}k(\mathbf{X}_{i},\mathbf{X}_{j})\alpha_{j}$
(3)

The better the correlation between the levels of theory, the easier it is to
learn the difference between them. In a more generalized version of this
method called Multilevel-ML zaspel2018boosting , one can exploit the
correlations between more than 2 levels of theory and basis sets to improve
predictions. In this work, we combine the SML method with $\Delta$-ML using
data from the QM7b dataset, namely the ZINDO energies as baseline, and the GW
energies as target.

###  II.5 Selected Machine Learning

In order to compare SML to generic QML training set selection, we proceed as
visualized in Fig. 5. We train a model on all molecules drawn at random across
the data-set (generic QML), then 3 different machines, each only with
molecules from a single class (SML). Moreover, we generate 3 separate test
sets, one for each class, while making sure that there is no overlap between
any of the training and test sets. For each test set, we produce two
predictions: one obtained from generic QML - with training molecules from all
over the data-set - and a second one from SML - with training molecules from
the corresponding class only. We expect the prediction errors of SML to be
lower than those of generic QML for each class. By applying two different
machines on exactly the same test set, their performances can be properly
compared to one another.

Figure 5: Visual representation of the SML method. The data-set is first
classified into separate classes - in our case 3. For each class, a machine is
trained on its molecules (models SML I, II and III) and a disjoint test set is
put aside (TS I, II and III). In addition, one machine is trained on molecules
form all across the data-set (here dubbed QML). Eventually, for each test set,
two predictions are computed, one using generic QML ($\hat{y}_{\mathrm{I}}$,
$\hat{y}_{\mathrm{II}}$ and $\hat{y}_{\mathrm{III}}$) and the other using SML
($\hat{y}_{\mathrm{I}}^{\mathrm{SML}}$, $\hat{y}_{\mathrm{II}}^{\mathrm{SML}}$
and $\hat{y}_{\mathrm{III}}^{\mathrm{SML}}$). These two predictions are then
compared for each test set.

##  III Results and Discussion

###  III.1 Frequency Analysis and Classification

The graph at the bottom of panel a) in Fig. 4 shows the frequency analysis of
HOMO-LUMO gaps of QM9 molecules. The first row illustrates how different
degrees of saturation affect the gap. The more unsaturated a molecule, the
lower its HOMO-LUMO gap, with aromatic and fully saturated molecules having
the smallest and largest gaps, respectively. This observation was to be
expected since in unsaturated molecules, the frontier orbitals are often
$\pi$-orbitals, which are closer in energy. The second row compares molecules
with differing elemental composition. These distributions indicate that the
composition alone is a relatively poor predictor of the location of the gap,
and has thus been ignored for the classification. The third row illustrates
the impact of the presence of common functional group signatures including
carbonyl, ester, amide bonds, and amino acids. Carbonyl containing compounds
mostly have lower gaps, but their distribution is bimodal, with both peaks
coinciding with the left and mid peak of the reference distribution. Therefore
more specific distinctions between different types of carbonyl compounds are
required. The next two distributions suggest that amides and carboxylates
(with an N- and O-atom linked to the C-atom of the carbonyl group) can be
considered separately from the other carbonyl molecules, since their
distributions are slightly more localized. Albeit rather rare in the set
considered, amino acids appear to be located closer to the middle peak as
well. In conclusion, the most relevant features for the classification are
saturation vs. aromaticity, and the presence vs. absence of a carbonyl group,
since they lead to well localized sub-distributions. Note that HOMO-LUMO-gap
distributions from the QM7b data-set at ZINDO level of theory exhibit similar
structures across different groups of molecules (Fig. 3 of SI).

The graph at the bottom of panel b) of Fig. 4 showcases the resulting
classification rules of QM9 molecules used for this study. First, we separate
all saturated molecules from the rest and assign them to one class that we
call ’saturated’, which corresponds to the right peak. The remaining molecules
are then subdivided into carbonyl and non-carbonyl molecules, which we further
separate into aromatic and non-aromatic ones, amino acids and more specific
carbonyl compounds (amides and carboxylates). Finally, we end up with aromatic
and carbonyl molecules with more than one unsaturated bond in one class that
we name ’aromatic/carbonyl’, that overlaps with the left peak. We put the
remaining molecules together with amino acids and other carbonyl compounds
into the last class, which we call ’single unsaturated’, because most
molecules of that class have only one unsaturated bond. These rules lead to a
classification which results in three molecular classes exhibiting well-
behaved unimodal distributions for the data-sets considered here. To
facilitate comparison in the following, we have numbered the classes (from
left to right): class I - saturated, class II - single unsaturated, and class
III - aromatic/carbonyl. We note again that classification rules for the ZINDO
gaps of QM7b (Fig. 4 of SI) are similar. We have also performed two
consistency checks of the classification with a Linear Discriminant Analysis
(LDA) projection (Figs. 5-8 of SI) and a Decision Tree Classification (Fig. 10
of SI). As shown in the SI, these checks confirm the validity of our
classification scheme.

In essence, our analysis suggests that the implementation of simple chemical
rules enables the splitting of input molecules (solely based on structure) for
a molecular data-set (here shown for QM7b and for QM9) into classes such that
unimodal quantum property distributions can be obtained.

###  III.2 Learning Curves

The learning curves for the HOMO-LUMO gaps are presented in Fig. 6. They show
the prediction errors w.r.t. increasing training set size on a log-log scale.
In all cases, KRR with prior classificiation via SML (dotted lines) performs
better than without classification (solid lines). While the slopes of the
learning curves remain the same, there is a significant drop for the offsets.
The largest drop can be observed for the class of saturated molecules, which
is around 0.077 eV for 800 training molecules in QM7b (GW), 0.142 eV for
$\Delta$-ML in QM7b and 0.055 eV in QM9. Moreover, the prediction error for
the class of saturated molecules is the lowest of all 3 classes, followed by
the errors of the class of singly unsaturated molecules, and the highest
errors are for the class of aromatic and carbonyl molecules. This trend is
consistent for both data-sets. Only the learning curves of the saturated class
reaches an error lower than 0.1 eV. For QM9, SML reaches this error with 16k
training samples already, while more than 64k training samples would be
required without classification. Note that this prediction error is also on
par with neural network prediction errors by Faber et al. Faber2017  which had
required training on 110k training samples. It is still far from the errors
obtained by more recently developed NNs Schuett2018 ; Anderson2019 ; Liu2020 ,
such as, for example, a NN by Liu et al. Liu2021  reaches errors as low as
0.032 eV, however with training sets of $\sim$100k molecules. Linear
extrapolation of SML learning curves predicts an MAE of 0.065 eV for the class
of saturated molecules with 100k training molecules, close to the NN by Schütt
et al. (0.63 eV) Schuett2018 . All in all, a prior classification
systematically improves prediction errors of the HOMO-LUMO gaps.

The same model applied to the HOMO and LUMO energies results in the learning
curves shown in Fig. 7. We can see that the learning curves generally follow
the same trends as those for the HOMO-LUMO gaps: same slopes in both models,
lower offsets for SML, lowest errors for the class of saturated molecules and
highest errors for the class of aromatic and carbonyl molecules. A noteworthy
difference between the results for HOMO and LUMO is the extent of the
improvement in the prediction errors: the errors for the HOMO energies drop
less than those for the LUMO energies. The learning curves for the LUMO
energies of QM7b (GW) stand out since the error for the saturated molecules
(0.030 eV) is much lower than for the other classes. These results demonstrate
that our classification can also be transferred to other related properties,
such as individual HOMO and LUMO energies, even though it has been derived
from gaps only.

Figure 6: Learning curves for the HOMO-LUMO gap of QM7b (GW) (left),
$\Delta$-ML on QM7b with ZINDO as baseline and (GW) as targetline (middle) and
QM9 (B3LYP) (right). The points show the MAE averaged over 10 iterations with
a different selection of training set molecules each. The solid lines are the
learning curves obtained from training with molecules from all over the data-
sets, whereas the dotted lines are obtained from Selected ML. Reference
results from the literature Faber2017 ; Schuett2018 ; Collins2018 ;
Anderson2019 ; Nigam2021  are shown in black. The results for QM9 indicated
with horizontal lines were obtained with training set sizes of $\sim$110k
molecules.  Figure 7: Learning curves for the HOMO (first row) and LUMO
energies (second row). As in Fig. 6, results are shown for QM7b (GW),
$\Delta$-ML (ZINDO to GW) and QM9 (B3LYP) from left to right and references
are shown in black Faber2017 ; Schuett2018 ; Collins2018 ; Anderson2019 ;
Stuke2019 ; Montavon2013 .

###  III.3 Scatter Plots

In Fig. 8 scatter plots of prediction vs. reference values are shown for QM9
and in Figs. 11 and 12 of the SI those for QM7b. The scatter plots reveal that
the energies in single unsaturated and aromatic/carbonyl classes (I, and II)
span a much wider range of values than the saturated molecules (class III),
which explains the higher complexity and offsets of the learning curves. Some
striking outliers are labelled in the Figures. The most noticeable ones in the
QM9 data-set are small saturated ones such as C2H6 or CF4 (Fig. 8, right
column). The HOMO and LUMO energies as well as gaps of these molecules already
stand out compared to the rest, which is why ML predictions for these
molecules have a large MAE. Moreover, molecules with several rings and cage-
like geometries have large prediction errors as well. An explanation for these
outliers may be that similar molecules are scarce, such that they are not
necessarily well represented in the training set. This issue could be resolved
by always including such molecules in the training set. For the other outliers
highlighted we could not find a pattern that explains the large errors, we
nevertheless included them for the sake of completeness. It is worth noting
that the predictions with models based on SML are in most cases closer to the
exact reference values than without classification. An exception is for
instance C9H12 in the third column of Fig. 8, where the error from SML is
larger compared to generic QML. In Figs. 11 and 12 of the SI more outlier
examples are indicated. In general, the predictions within the classes from
SML are closer to the reference values then those from QML after random
training set selection.

Figure 8: Scatter plots of predicted ML vs. reference QM energies of QM9 for
the largest training set size possible for each class, indicated in brackets.
Some striking outliers are indicated in the figure, and if possible,
predictions from both models (with and without classification) are highlighted
for comparison.

###  III.4 Interpretation

The systematic improvements of the prediction errors across different data-
sets and properties may be explained by the reduction of effective
dimensionality achieved within each of the classes. Indeed, the lowest errors
are obtained for the least diverse: the class of saturated molecules. Many
functional groups such as carbonyl groups, aromatic rings or amides imply the
presence of unsaturated bonds, indicating more chemical diversity in the other
two classes. Because of the classification, the model parameters can more
easily adapt since training and test sets contain only molecules which have
increased similarity with each other. In other words, in order to predict the
gap for an unsaturated molecule, the model did not need to account for the
correlations of gaps with saturated molecules. In this sense, no fitting
coefficients have been ’wasted’ on suboptimal correlations and can contribute
instead to further lower the prediction error by exploiting more effective
correlations within a given class. Conversely, without classification, the
model would be trained on many molecules with various functional groups that
do not even occur in the test set, thus lowering the effective number of
training molecules that are relevant for the prediction of that test set.

It is also interesting to note that in the case of the QM7b results, even
though the classification was based only on the distribution of ZINDO HOMO-
LUMO gaps, the prediction errors also drop for predicting the GW gaps. As
such, there seems to be a certain transferability of the classification scheme
in the sense that it can be used across different levels of theory. But the
classification is also transferable between different properties, as shown in
Fig. 7. The greater improvement for LUMO energies than for HOMO energies is
likely due to correlation between gaps and LUMO energies being stronger than
between gaps and HOMO energies (see Fig. 3).

##  IV Conclusion

We have found that simple classification protocols, prior to training, can
dramatically improve the data-efficiency of QML models of HOMO-LUMO gaps in
the QM9 and QM7b data sets. The classification is based on chemical bonding
rules that allow us to define molecular classes based on structural input
features alone. Our frequency analysis reveals that the presence of functional
groups, such as aromatic rings and carbonyl groups, dominate sub-distributions
of HOMO-LUMO gaps, and can therefore be exploited for classification. After
classification, conventional kernel ridge regression based QML models afford
learning curves with systematically lower offsets than without
classifications. As a result, significantly fewer training molecules are
required to reach competitive prediction errors ($\sim$0.1 eV), e.g. 16k for
saturated molecules as compared to more than 64k training molecules necessary
when drawing at random from QM9. We have also shown that our SML approach can
be applied to related individual properties, i.e. the HOMO and LUMO energies
alone. Further analysis has indicated, that the scheme is robust across
different levels of theory for the labels, i.e. classification based on the
distribution of ZINDO gaps was shown to be transferable to train more
efficient QML models of GW gaps. Comparison to $\Delta$-ML results on the same
data set (QM7b) indicates that for HOMO-LUMO gaps, the classification approach
presented here within offers substantially more improvement.

The additional step of prior classification alone can already lower the
prediction errors in QML. The exploitation of simple relations between
molecular structure and the HOMO-LUMO gap were enough to improve learning
curves consistently. Our method can thus be considered to be a biased training
set selection Browning2017 ; Huang2020 , since the classification leads to
training sets that are specifically tailored towards the test sets. Our
results confirm that adequate training data is indeed crucial in order to
obtain the best possible performance in QML. Nevertheless, there is no
universally applicable classification that would work for any data-set. In our
case, the classifications for QM9 and QM7b are only similar because they
consist both of simple organic molecules, but in general, such a
classification depends on the chemical space a given data-set covers, the
property of interest and also the level theory at which the property is
calculated.

Similar to the B3LYP and ZINDO HOMO-LUMO gaps in QM9 and QM7b respectively,
other properties with multimodal distributions could also be investigated.
These could include properties related to the gap, such as excitation energies
Ramakrishnan2015a , but also properties of entirely different origin such a
highest vibrational frequencies ramakrishnan2015many  or NMR shifts. Similar
to the gaps, one should then identify the structural features that govern
these properties (well established for IR and NMR spectroscopy) in order to
define molecular classes within which these distributions become unimodal and
for which equal improvements in the data-efficiency of resulting QML models
should be expected. Other future work could also involve the use of
unsupervised ML methods to find new and potentially better classification
rules, based on more sophisticated combinations of functional groups, or other
molecular features. In summary, simple classification rules can lead to
substantial improvements in data-efficiency of QML models.

## Acknowledgements

We acknowledge support from the European Research Council (ERC-CoG grant QML
and H2020 projects BIG-MAP). This project has received funding from the
European Union’s Horizon 2020 research and innovation programme under Grant
Agreements #957189. This project has received funding from the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (grant agreement No. 772834). This result only reflects
the author’s view and the EU is not responsible for any use that may be made
of the information it contains. This work was partly supported by the NCCR
MARVEL, funded by the Swiss National Science Foundation. Some of the
computational results presented have been achieved using the Vienna Scientific
Cluster (VSC).

## Code Availability

The code used in this work is freely available from
https://github.com/b3rn4rdm/SelectedML.

## References

## References

  * (1) R. Ramakrishnan and O. A. von Lilienfeld, “Machine learning, quantum chemistry, and chemical space,” Reviews in Computational Chemistry, vol. 30, pp. 225–256, 2017.
  * (2) O. A. von Lilienfeld, “Quantum Machine Learning in Chemical Compound Space,” Angewandte Chemie (International ed.), vol. 57, no. 16, pp. 4164–4169, 2018.
  * (3) O. A. von Lilienfeld, K.-R. Müller, and A. Tkatchenko, “Exploring chemical compound space with quantum-based machine learning,” Nature Reviews Chemistry, pp. 1–12, 2020.
  * (4) O. A. von Lilienfeld and K. Burke, “Retrospective on a decade of machine learning for chemical discovery,” Nature Communications, vol. 11, no. 1, pp. 4895–4895, 2020.
  * (5) B. Huang and O. A. von Lilienfeld, “Ab initio machine learning in chemical compound space,” arXiv preprint arXiv:2012.07502, 2021.
  * (6) M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. Von Lilienfeld, “Fast and accurate modeling of molecular atomization energies with machine learning,” Physical Review Letters, vol. 108, no. 5, p. 058301, 2012.
  * (7) R. Ramakrishnan and O. A. von Lilienfeld, “Many molecular properties from one kernel in chemical space,” CHIMIA International Journal for Chemistry, vol. 69, no. 4, pp. 182–186, 2015.
  * (8) F. A. Faber, L. Hutchison, B. Huang, J. Gilmer, S. S. Schoenholz, G. E. Dahl, O. Vinyals, S. Kearnes, P. F. Riley, and O. A. Von Lilienfeld, “Prediction errors of molecular machine learning models lower than hybrid DFT error,” Journal of Chemical Theory and Computation, vol. 13, no. 11, pp. 5255–5264, 2017.
  * (9) A. Stuke, M. Todorović, M. Rupp, C. Kunkel, K. Ghosh, L. Himanen, and P. Rinke, “Chemical diversity in molecular orbital energy predictions with kernel ridge regression,” The Journal of Chemical Pysics, vol. 150, no. 20, pp. 204121–204121, 2019.
  * (10) A. Patra, R. Batra, A. Chandrasekaran, C. Kim, T. D. Huan, and R. Ramprasad, “A multi-fidelity information-fusion approach to machine learn and predict polymer bandgap,” Computational Materials Science, vol. 172, p. 109286, 2020.
  * (11) F. A. Faber, A. Lindmaa, O. A. Von Lilienfeld, and R. Armiento, “Machine Learning Energies of 2 Million Elpasolite (A B C 2 D 6) Crystals,” Physical Review Letters, vol. 117, no. 13, p. 135502, 2016.
  * (12) G. Pilania, J. E. Gubernatis, and T. Lookman, “Multi-fidelity machine learning models for accurate bandgap predictions of solids,” Computational Materials Science, vol. 129, pp. 156–163, 2017.
  * (13) O. Isayev, C. Oses, C. Toher, E. Gossett, S. Curtarolo, and A. Tropsha, “Universal fragment descriptors for predicting properties of inorganic crystals,” Nature Communications, vol. 8, p. 15679, 2017.
  * (14) W. Li, R. Jacobs, and D. Morgan, “Predicting the thermodynamic stability of perovskite oxides using machine learning models,” Computational Materials Science, vol. 150, pp. 454–463, 2018.
  * (15) K. T. Butler, D. W. Davies, H. Cartwright, O. Isayev, and A. Walsh, “Machine learning for molecular and materials science,” Nature (London), vol. 559, no. 7715, pp. 547–555, 2018.
  * (16) K. Hansen, G. Montavon, F. Biegler, S. Fazli, M. Rupp, M. Scheffler, O. A. Von Lilienfeld, A. Tkatchenko, and K.-R. Muller, “Assessment and validation of machine learning methods for predicting molecular atomization energies,” Journal of Chemical Theory and Computation, vol. 9, no. 8, pp. 3404–3419, 2013.
  * (17) J. Weinreich, N. J. Browning, and O. A. von Lilienfeld, “Machine learning of free energies in chemical compound space using ensemble representations: Reaching experimental uncertainty for solvation,” The Journal of Chemical Physics, vol. 154, no. 13, pp. 134113–134113, 2021.
  * (18) G. Montavon, M. Rupp, V. Gobre, A. Vazquez-Mayagoitia, K. Hansen, A. Tkatchenko, K.-R. Müller, and O. A. Von Lilienfeld, “Machine learning of molecular electronic properties in chemical compound space,” New Journal of Physics, vol. 15, no. 9, p. 095003, 2013.
  * (19) R. Ramakrishnan, M. Hartmann, E. Tapavicza, and O. A. von Lilienfeld, “Electronic spectra from TDDFT and machine learning in chemical space,” The Journal of Chemical Physics, vol. 143, no. 8, p. 084111, 2015.
  * (20) K. Schütt, M. Gastegger, A. Tkatchenko, K.-R. Müller, and R. J. Maurer, “Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions,” Nature Communications, vol. 10, no. 1, pp. 1–10, 2019.
  * (21) A. S. Christensen, F. A. Faber, and O. A. von Lilienfeld, “Operators in quantum machine learning: Response properties in chemical space,” The Journal of Chemical Physics, vol. 150, no. 6, p. 064105, 2019.
  * (22) E. O. Pyzer-Knapp, K. Li, and A. Aspuru-Guzik, “Learning from the harvard clean energy project: The use of neural networks to accelerate materials discovery,” Advanced Functional Materials, vol. 25, no. 41, pp. 6495–6502, 2015.
  * (23) M. A. Shandiz and R. Gauvin, “Application of machine learning methods for the prediction of crystal system of cathode materials in lithium-ion batteries,” Computational Materials Science, vol. 117, pp. 270–278, 2016\.
  * (24) R. Gómez-Bombarelli, J. Aguilera-Iparraguirre, T. D. Hirzel, D. Duvenaud, D. Maclaurin, M. A. Blood-Forsythe, H. S. Chae, M. Einzinger, D.-G. Ha, T. Wu, et al., “Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach,” Nature Materials, vol. 15, no. 10, pp. 1120–1127, 2016.
  * (25) A. D. Sendek, E. D. Cubuk, E. R. Antoniuk, G. Cheon, Y. Cui, and E. J. Reed, “Machine Learning-Assisted Discovery of Solid Li-Ion Conducting Materials,” Chemistry of materials, vol. 31, no. 2, pp. 342–352, 2019\.
  * (26) A. Zunger, “Inverse design in search of materials with target functionalities,” Nature Reviews Chemistry, vol. 2, no. 4, pp. 1–16, 2018\.
  * (27) P. B. Jørgensen, M. Mesta, S. Shil, J. M. García Lastra, K. W. Jacobsen, K. S. Thygesen, and M. N. Schmidt, “Machine learning-based screening of complex molecules for polymer solar cells,” The Journal of Chemical Physics, vol. 148, no. 24, p. 241735, 2018.
  * (28) “Jmol: an open-source Java viewer for chemical structures in 3D.” http://www.jmol.org/.
  * (29) F. Neese, “The ORCA program system,” Wiley interdisciplinary reviews. Computational molecular science, vol. 2, no. 1, pp. 73–78, 2012.
  * (30) J. Behler and M. Parrinello, “Generalized neural-network representation of high-dimensional potential-energy surfaces,” Physical Review Letters, vol. 98, no. 14, pp. 146401–146401, 2007.
  * (31) K. Schütt, H. Glawe, F. Brockherde, A. Sanna, K. Müller, and E. Gross, “How to represent crystal structures for machine learning: Towards fast prediction of electronic properties,” Physical Review B, vol. 89, no. 20, p. 205118, 2014.
  * (32) F. Pereira, K. Xiao, D. A. Latino, C. Wu, Q. Zhang, and J. Aires-de Sousa, “Machine learning methods to predict density functional theory B3LYP energies of HOMO and LUMO orbitals,” Journal of Chemical Information and Modeling, vol. 57, no. 1, pp. 11–21, 2017.
  * (33) O. T. Unke and M. Meuwly, “PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges,” Journal of Chemical Theory and Computation, vol. 15, no. 6, pp. 3678–3693, 2019.
  * (34) J. Westermayr, M. Gastegger, and P. Marquetand, “Combining SchNet and SHARC: The SchNarc Machine Learning Approach for Excited-State Dynamics,” The Journal of Physical Chemistry Letters, vol. 11, no. 10, pp. 3828–3834, 2020\.
  * (35) B. Huang and O. A. von Lilienfeld, “Quantum machine learning using atom-in-molecule-based fragments selected on the fly,” Nature Chemistry, pp. 1–7, 2020.
  * (36) F. A. Faber, A. S. Christensen, B. Huang, and O. A. von Lilienfeld, “Alchemical and structural distribution based representation for universal quantum machine learning,” The Journal of Chemical Physics, vol. 148, no. 24, p. 241717, 2018.
  * (37) A. S. Christensen, L. A. Bratholm, F. A. Faber, and O. Anatole von Lilienfeld, “FCHL revisited: Faster and more accurate quantum machine learning,” The Journal of Chemical Physics, vol. 152, no. 4, pp. 044107–044107, 2020.
  * (38) K. Hansen, F. Biegler, R. Ramakrishnan, W. Pronobis, O. A. Von Lilienfeld, K.-R. Müller, and A. Tkatchenko, “Machine learning predictions of molecular properties: Accurate many-body potentials and nonlocality in chemical space,” The Journal of Physical Chemistry Letters, vol. 6, no. 12, pp. 2326–2331, 2015.
  * (39) S. Chmiela, A. Tkatchenko, H. E. Sauceda, I. Poltavsky, K. T. Schütt, and K.-R. Müller, “Machine learning of accurate energy-conserving molecular force fields,” Science advances, vol. 3, no. 5, pp. e1603015–e1603015, 2017.
  * (40) S. Chmiela, H. E. Sauceda, K.-R. Müller, and A. Tkatchenko, “Towards exact molecular dynamics simulations with machine-learned force fields,” Nature Communications, vol. 9, no. 1, pp. 3887–3887, 2018.
  * (41) A. S. Christensen and O. A. von Lilienfeld, “On the role of gradients for machine learning of molecular energies and forces,” Machine Learning: Science and Technology, vol. 1, no. 4, 2020.
  * (42) N. J. Browning, R. Ramakrishnan, O. A. Von Lilienfeld, and U. Roethlisberger, “Genetic optimization of training sets for improved machine learning models of molecular properties,” The Journal of Physical Chemistry Letters, vol. 8, no. 7, pp. 1351–1359, 2017.
  * (43) S. Kubatkin, A. Danilov, M. Hjort, J. Cornil, J.-L. Bredas, N. Stuhr-Hansen, P. Hedegård, and T. Bjørnholm, “Single-electron transistor of a single organic molecule with access to several redox states,” Nature, vol. 425, no. 6959, pp. 698–701, 2003.
  * (44) J. Roncali, “Molecular engineering of the band gap of $\pi$-conjugated systems: Facing technological applications,” Macromolecular Rapid Communications, vol. 28, no. 17, pp. 1761–1775, 2007.
  * (45) M. Jurow, A. E. Schuckman, J. D. Batteas, and C. M. Drain, “Porphyrins as molecular electronic components of functional devices,” Coordination Chemistry Reviews, vol. 254, no. 19-20, pp. 2297–2310, 2010.
  * (46) P. M. Beaujuge and J. R. Reynolds, “Color control in $\pi$-conjugated organic polymers for use in electrochromic devices,” Chemical Reviews, vol. 110, no. 1, pp. 268–320, 2010.
  * (47) S. X. Tao, X. Cao, and P. A. Bobbert, “Accurate and efficient band gap predictions of metal halide perovskites using the DFT-1/2 method: GW accuracy with DFT expense,” Scientific Reports, vol. 7, no. 1, pp. 1–9, 2017.
  * (48) A. Stoliaroff and C. Latouche, “Accurate Ab Initio Calculations on Various PV-Based Materials: Which Functional to Be Used?,” The Journal of Physical Chemistry C, vol. 124, no. 16, pp. 8467–8478, 2020.
  * (49) L. C. Blum and J.-L. Reymond, “970 million druglike small molecules for virtual screening in the chemical universe database GDB-13,” Journal of the American Chemical Society, vol. 131, no. 25, pp. 8732–8733, 2009.
  * (50) R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld, “Big data meets quantum chemistry approximations: the $\Delta$-machine learning approach,” Journal of Chemical Theory and Computation, vol. 11, no. 5, pp. 2087–2096, 2015.
  * (51) J. Ridley and M. Zerner, “An intermediate neglect of differential overlap technique for spectroscopy: pyrrole and the azines,” Theoretica Chimica Acta, vol. 32, no. 2, pp. 111–134, 1973.
  * (52) M. C. Zerner, “Semiempirical molecular orbital methods,” Reviews in Computational Chemistry, vol. 2, pp. 313–365, 1991.
  * (53) L. Hedin, “New Method for Calculating the One-Particle Green’s Function with Application to the Electron-Gas Problem,” Physical Review, vol. 139, no. 3A, pp. 796–823, 1965.
  * (54) F. Aryasetiawan and O. Gunnarsson, “The GW method,” Reports on Progress in Physics, vol. 61, no. 3, pp. 237–312, 1998.
  * (55) L. Ruddigkeit, R. Van Deursen, L. C. Blum, and J.-L. Reymond, “Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17,” Journal of Chemical Information and Modeling, vol. 52, no. 11, pp. 2864–2875, 2012.
  * (56) R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. Von Lilienfeld, “Quantum chemistry structures and properties of 134 kilo molecules,” Scientific Data, vol. 1, no. 1, pp. 1–7, 2014.
  * (57) P. Hohenberg and W. Kohn, “Inhomogeneous electron gas,” Physical Review, vol. 136, no. 3B, p. B864, 1964.
  * (58) W. Kohn and L. J. Sham, “Self-consistent equations including exchange and correlation effects,” Physical Review, vol. 140, no. 4A, p. A1133, 1965\.
  * (59) A. D. Becke, “Density-functional thermochemistry. III. The role of exact exchange,” The Journal of Chemical Physics, vol. 98, no. 7, pp. 5648–5652, 1993.
  * (60) J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in International Conference on Machine Learning, pp. 1263–1272, PMLR, 2017.
  * (61) K. T. Schütt, H. E. Sauceda, P.-J. Kindermans, A. Tkatchenko, and K.-R. Müller, “SchNet–A deep learning architecture for molecules and materials,” The Journal of Chemical Physics, vol. 148, no. 24, p. 241722, 2018.
  * (62) C. Chen, W. Ye, Y. Zuo, C. Zheng, and S. P. Ong, “Graph networks as a universal machine learning framework for molecules and crystals,” Chemistry of Materials, vol. 31, no. 9, pp. 3564–3572, 2019.
  * (63) B. Anderson, T.-S. Hy, and R. Kondor, “Cormorant: Covariant molecular neural networks,” arXiv preprint arXiv:1906.04015, 2019.
  * (64) C. Lu, Q. Liu, C. Wang, Z. Huang, P. Lin, and L. He, “Molecular property prediction: A multilevel quantum interactions modeling perspective,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1052–1060, 2019.
  * (65) Z. Liu, L. Lin, Q. Jia, Z. Cheng, Y. Jiang, Y. Guo, and J. Ma, “Transferable multi-level attention neural network for accurate prediction of quantum chemistry properties via multi-task learning,” ChemRxiv, vol. 12588170, p. v1, 2020.
  * (66) D. Weininger, “SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules,” Journal of Chemical Information and Computer Sciences, vol. 28, no. 1, pp. 31–36, 1988.
  * (67) D. Weininger, A. Weininger, and J. L. Weininger, “SMILES. 2. Algorithm for generation of unique SMILES notation,” Journal of Chemical Information and Computer Sciences, vol. 29, no. 2, pp. 97–101, 1989.
  * (68) D. Weininger, “SMILES. 3. DEPICT. Graphical depiction of chemical structures,” Journal of Chemical Information and Computer Sciences, vol. 30, no. 3, pp. 237–243, 1990.
  * (69) “RDKit: Open-source cheminformatics.” http://www.rdkit.org, 2006.
  * (70) K.-R. Müller, S. Mika, G. Ratsch, K. Tsuda, and B. Schölkopf, “An introduction to kernel-based learning algorithms,” IEEE Transactions on Neural Networks, vol. 12, no. 2, pp. 181–201, 2001.
  * (71) B. Schölkopf, A. J. Smola, F. Bach, et al., Learning with kernels: support vector machines, regularization, optimization, and beyond.  MIT press, 2002.
  * (72) T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning.  Springer, 2009.
  * (73) C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning.  The MIT Press, 2006.
  * (74) C. Cortes, L. D. Jackel, S. A. Solla, V. Vapnik, and J. S. Denker, “Learning curves: Asymptotic values and rate of convergence,” in Advances in Neural Information Processing Systems, pp. 327–334, 1994.
  * (75) K.-R. Müller, M. Finke, N. Murata, K. Schulten, and S.-i. Amari, “A numerical study on learning curves in stochastic multilayer feedforward networks,” Neural Computation, vol. 8, no. 5, pp. 1085–1106, 1996.
  * (76) A. S. Christensen, F. A. Faber, B. Huang, L. A. Bratholm, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld, “QML: A Python Toolkit for Quantum Machine Learning.” https://github.com/qmlcode/qml, 2017.
  * (77) P. Zaspel, B. Huang, H. Harbrecht, and O. A. von Lilienfeld, “Boosting Quantum Machine Learning Models with a Multilevel Combination Technique: Pople Diagrams Revisited,” Journal of Chemical Theory and Computation, vol. 15, no. 3, pp. 1546–1559, 2018.
  * (78) Z. Liu, L. Lin, Q. Jia, Z. Cheng, Y. Jiang, Y. Guo, and J. Ma, “Transferable Multilevel Attention Neural Network for Accurate Prediction of Quantum Chemistry Properties via Multitask Learning,” Journal of Chemical Information and Modeling, vol. 61, no. 3, pp. 1066–1082, 2021.
  * (79) C. R. Collins, G. J. Gordon, O. A. Von Lilienfeld, and D. J. Yaron, “Constant size descriptors for accurate machine learning models of molecular properties,” The Journal of chemical physics, vol. 148, no. 24, p. 241718, 2018.
  * (80) J. Nigam, M. Willatt, and M. Ceriotti, “Equivariant representations for molecular Hamiltonians and N-center atomic-scale properties,” arXiv preprint arXiv:2109.12083, 2021.