# Impact of noise on inverse design: The case of NMR spectra matching

Dominik Lemm  University of Vienna, Faculty of Physics, Kolingasse 14-16,
AT-1090 Vienna, Austria  University of Vienna, Vienna Doctoral School in
Physics, Boltzmanngasse 5, AT-1090 Vienna, Austria  Guido Falk von Rudorff
University Kassel, Department of Chemistry, Heinrich-Plett-Str.40, 34132
Kassel, Germany  O. Anatole von Lilienfeld<EMAIL_ADDRESS>Departments of Chemistry, Materials Science and Engineering, and Physics,
University of Toronto, St. George Campus, Toronto, ON, Canada  Vector
Institute for Artificial Intelligence, Toronto, ON, M5S 1M1, Canada  Machine
Learning Group, Technische Universität Berlin and Institute for the
Foundations of Learning and Data, 10587 Berlin, Germany

###### Abstract

Despite its fundamental importance and widespread use for assessing reaction
success in organic chemistry, deducing chemical structures from nuclear
magnetic resonance (NMR) measurements has remained largely manual and time
consuming. To keep up with the accelerated pace of automated synthesis in self
driving laboratory settings, robust computational algorithms are needed to
rapidly perform structure elucidations. We analyse the effectiveness of
solving the NMR spectra matching task encountered in this inverse structure
elucidation problem by systematically constraining the chemical search space,
and correspondingly reducing the ambiguity of the matching task. Numerical
evidence collected for the twenty most common stoichiometries in the QM9-NMR
data base indicate systematic trends of more permissible machine learning
prediction errors in constrained search spaces. Results suggest that compounds
with multiple heteroatoms are harder to characterize than others. Extending
QM9 by $\sim$10 times more constitutional isomers with 3D structures generated
by Surge, ETKDG and CREST, we used ML models of chemical shifts trained on the
QM9-NMR data to test the spectra matching algorithms. Combining both
${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ shifts in the matching process
suggests twice as permissible machine learning prediction errors than for
matching based on ${}^{13}\mathrm{C}$ shifts alone. Performance curves
demonstrate that reducing ambiguity and search space can decrease machine
learning training data needs by orders of magnitude.

##  I Introduction

Figure 1: Schematic workflow for autonomous chemical discovery as well as
scaling of constitutional isomer space versus data availability in the
QM9ramakrishnan_quantum_2014  database. a) After the chemical synthesis of
molecular compounds, reaction products are characterized using spectroscopic
methods such as nuclear magnetic resonance (NMR). The measured
${}^{1}\mathrm{H}$ and ${}^{13}\mathrm{C}$ spectra are automatically processed
and potential candidate structures suggested via machine learning. b) Number
of constitutional isomers for 20 stoichiometries considered.  Figure 2:
Elucidation performance curves of C7O2H10, C5N3OH7, C8OH14 spectra using
Gaussian noise to control chemical shift accuracy in terms of mean absolute
error (MAE). a-b) ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ spectra matching.
Individual points were obtained by calculating the percentage of queries where
noisy and noise free query spectra have the lowest distance. All points have
been fitted using Eq.3. Solid curves correspond to candidate numbers $N_{QM9}$
from QM9ramakrishnan_quantum_2014 . Dashed curves are an extrapolation to
candidate numbers $N_{Surge}$ as obtained via graph enumerationsurge . The
legend corresponds to both a) and b), respectively. c) Spectra matching using
both ${}^{1}\mathrm{H}$ and ${}^{13}\mathrm{C}$ shifts. Dashed lines
correspond to the accuracy required to correctly elucidate 95$\%$ of queries
when only ${}^{1}\mathrm{H}$ or ${}^{13}\mathrm{C}$ spectra are being used,
respectively.  Figure 3: Trends in QM9ramakrishnan_quantum_2014  chemical
compound space to correctly elucidate queries at 95$\%$ accuracy. a)
Extrapolated MAE at candidate numbers $N_{Surge}$ of the 20 most common
stoichiometries in QM9ramakrishnan_quantum_2014 . b) MAE using only
${}^{13}\mathrm{C}$ spectra (${}^{13}\mathrm{C}_{single}$ ) against
${}^{13}\mathrm{C}$ and noise-free ${}^{1}\mathrm{H}$ spectra combined
(${}^{13}\mathrm{C}_{combined}$ ) at candidate numbers $N_{QM9}$ from
QM9ramakrishnan_quantum_2014 .  Figure 4: Elucidation accuracy of C7O2H10
spectra using machine learning ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$
shift predictions. Mean absolute error (MAE) refers to the predictive accuracy
of the machine learning models, respectively. a-b) ${}^{13}\mathrm{C}$ and
${}^{1}\mathrm{H}$ spectra matching at increasing search pool sizes $N$. The
inset depicts the decay of the elucidation accuracy of the best performing
machine learning model at increasing levels of Gaussian noise on query spectra
(MAEQ). c-d) Spectra matching accuracy when restricting the search pool to
contain only known functional groups. The inset in d) depicts the search pool
size $N$ restricted to compounds with similar functional groups as the query,
respectively. e) Spectra matching using ${}^{1}\mathrm{H}$ and
${}^{13}\mathrm{C}$ shifts combined. f) Accuracy required to reach 85$\%$
correct elucidation at increasing $N$ when using both ${}^{1}\mathrm{H}$ and
${}^{13}\mathrm{C}$ shifts combined.

Current development times of novel molecular materials can span several
decades from discovery to commercialization. In order for humanity to react to
global challenges, the digitizationlab_automation22 ; chemdigitization19 ;
GROMSKI20204 ; digisynth ; EAST_review  of molecular and materials discovery
aims to accelerate the process to a few years. Long experiment times severely
limit the coverage of the vastness of chemical space, making the development
of self driving laboratories for autonomous robotics experimentation crucial
for high throughput synthesis of novel compounds (Fig.1 a))alanbayesian ;
XIE2023101043 ; jiang_airobots22 ; robotscientist ; Burger2020 ; archemist22 ;
closedloopopt_22 . To keep the pace of automated synthesis, fast and reliable
characterization of reaction products through spectroscopic methods is
required, an often manual, time intense and possibly error prone task. One of
the most common methods to elucidate the structure of reaction products are
nuclear magnetic resonance (NMR) experiments.nmrrev15  Through relaxation of
nuclear spins after alignment in a magnetic field, an NMR spectrum,
characteristic of local atomic environments of a compound, i.e. functional
groups, can be recorded. In particular, ${}^{1}\mathrm{H}$ and
${}^{13}\mathrm{C}$ NMR experiments are routinely used by experimental
chemists to identify the chemical structure or relevant groups just from the
spectrum. For larger compounds, however, the inverse problem of mapping
spectrum to structure becomes increasingly difficult, ultimately requiring NMR
of additional nuclei, stronger magnets, or more advanced two-dimensional NMR
experimentsstructureelucidator04 ; nmrchallenges17 .

Computer-assisted structure elucidation algorithms aim to iteratively
automatize the structure identification processWilloughby2014-oh ;
case_review21 ; case20 ; dp4ai ; dp5 . Current workflows include repeated
predictions of chemical shifts for candidate structure inputs through
empirical or ab initio methodshose ; compNMR ; Jonas2022-ff . Albeit accurate
even in condensed phase through use of plane-waves dsmp  or QM/MM setup
qmmmnmr , the cost of density functional theory (DFT) calculations severely
limits the number of candidate structures that can be tested, leaving the
identification of unknown reaction products out of reach for all but the
smallest search spaces. Data driven machine learning models leveraging
experimental or theoretical NMR databasesnmrshiftdb ; csd ; qm9nmr ;
Bratholm2021  provide orders of magnitude of speedup over ab initio
calculations, reaching 1-2 ppm ${}^{13}\mathrm{C}$ mean-absolute-error (MAE)
w.r.t. experiment or theory, respectivelyRupp2015-ds ; nmrpred_kwon20 ;
Jonas2019-oy ; Han_nmr22 ; qm9nmr ; Paruzzo2018 ; Musil2019 . However, while
the stoichiometry of the reaction product is usually known, e.g. through prior
mass spectrometry experiments, the number of possible constitutional isomers
exhibits NP hard scaling in number of atoms, quickly spanning millions of
valid molecular graphs already for molecules of modest size (Fig.1 b)). As
such, the inverse problem of inferring the molecular structure from an NMR
spectrum still poses a major challenge even for rapid solvers.

Recent machine learning approaches tackle the inverse problem using a
combination of graph generation and subsequent chemical shift predictions for
candidate rankingnmrinverse_jonas19 ; nmrinverse_huang21 ;
nmrinverse_Sridharan22 . First explored by Jonasnmrinverse_jonas19 , a Top-1
ranking with 57$\%$ reconstruction success-rate was achieved using deep
imitation learning to predict bonds of molecular graphs. Sridharan et
al.nmrinverse_Sridharan22  used online Monte Carlo tree search to build
molecular graphs resulting in a similar Top-1 ranking of 57.2$\%$. Huang et
al.nmrinverse_huang21  relied on substructure predictions from which complete
graphs can be constructed, reaching 67.4$\%$ Top-1 accuracy by ranking
substructure profiles instead of shifts. A commonality between all algorithms
is the subsequent ranking of candidates using spectra matching or other
heuristics. Consequently, even though the correct query compound could be
detected early, similar candidates might be ranked higher, making the ranking
process as critical as the candidate search itself.

In this work, we analyse the effectiveness of the NMR spectra matching task
encountered in the inverse structure elucidation problem. As stagnating
improvementsJonas2022-ff  in chemical shift predictions due to limited public
NMR data aggravate candidate rankings, results suggest that both the
prediction error of machine learning models and the number of possible
candidates are crucial factors for elucidation success. By systematically
controlling the size of chemical search space and accuracy of chemical shifts,
we find that higher error levels become permissible in constrained search
spaces. Moreover, results indicate that increasing the uniqueness through
including both ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ shifts in the
matching process, rather than relying on a single type of shift, significantly
reduces ambiguity and enhances error tolerance. To evaluate the spectra
matching task throughout chemical compound space, we systematically control
the accuracy of 1D ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ chemical shifts
of the 20 most common stoichiometries in QM9-NMRramakrishnan_quantum_2014 ;
qm9nmr  by applying distinct levels of Gaussian white noise. Note that while
we focus on DFT based 1D NMR in this work, future studies could include
experimental data and 2D NMR information. Comparisons amongst stoichiometries
suggest that chemical spaces with increasing amounts of heteroatoms and number
of constitutional isomers are harder to characterize than others. To test the
spectra matching method on a large search space, we extended QM9-NMR to 56k
C7O2H10 constitutional isomers. Controlling the chemical shift accuracy
through machine learning models trained at increasing training set sizes,
performance curves again indicate a trade-off between search space and
accuracy. Hence, as less accurate shift predictions become useful, results
show that machine learning training data needs can be reduced by multiple
orders of magnitude.

##  II Theory & Methods

###  II.1 NMR Spectra Matching

Consider a query ${}^{13}\mathrm{C}$ or ${}^{1}\mathrm{H}$ spectrum with a set
of $N$ possible candidate constitutional isomer spectra. We chose the squared
euclidean distance as a metric to rank candidate spectra against the query
spectrum (see SI Fig.3 for comparison against other metrics):

$d(\delta_{q},\delta_{i})=\sum_{j=1}^{n}(\delta_{q,j}-\delta_{i,j})^{2},$ (1)

with $\delta$ being a sorted spectrum of $n$ chemical shifts
(${}^{13}\mathrm{C}$ or ${}^{1}\mathrm{H}$ ), $q$ being the query, $i$ being
the $i$-th of $N$ candidates, and $j$ being the $j$-th chemical shift in a
spectrum, respectively. To use both ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$
shifts simultaneously for spectra matching, a total distance can be calculated
as follows:

$d_{combined}=d(\delta^{13C}_{q},\delta^{13C}_{i})+\gamma\cdot
d(\delta^{1H}_{q},\delta^{1H}_{i}),$ (2)

with $\gamma=64$ being a scaling factor determined via cross-validation (see
SI Fig.1) to ensure similar weighting. Final rankings are obtained by sorting
all candidates by distance. The Top-1 accuracy is calculated as the proportion
of queries correctly ranked as the closest spectrum, respectively.

###  II.2 Elucidation performance curves

To analyse the spectra matching elucidation accuracy, we systematically
control the number of possible candidates $N$ and the accuracy of chemical
shifts, respectively. For each constitutional isomer set, we choose 10$\%$ as
queries and 90$\%$ as search pool, respectively. Next, we randomly sample $N$
spectra from the search pool, including the query spectrum. Each sample size
is drawn ten times and the Top-1 accuracy averaged across all runs. To control
the accuracy of chemical shifts, we apply Gaussian white noise (up to 1 or 10
$\sigma$ for ${}^{1}\mathrm{H}$ and ${}^{13}\mathrm{C}$ , respectively) or use
the machine learning error as a function of training set size (c.f. SI Fig.5
for learning curves). For each $N$ and chemical shift accuracy, results are
presented as elucidation performance curves (c.f. Fig.2 a-b)), showing the
elucidation success as a function of chemical shift accuracy in terms of mean
absolute error (MAE).

###  II.3 Chemical Shift Prediction

We relied on kernel ridge regression (KRR) for machine learning
${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ chemical shifts as presented in
Ref.qm9nmr . We use a Laplacian kernel and the local atomic Faber-Christensen-
Huang-Lilienfeld (FCHL19christensen_fchl_2020 ) representation with a radial
cutoffqm9nmr  of 4 $\mathrm{\text{\r{A}}}$. The kernel width and
regularization coefficient have been determined through 10-fold cross-
validation on a subset of 10’000 chemical shifts of the training set.

###  II.4 Data

The QM9-NMRqm9nmr ; ramakrishnan_quantum_2014  dataset was used in this work,
containing 130’831 small molecules up to nine heavy atoms (CONF) with chemical
shieldings at the mPW1PW91/6-311+G(2d,p)-level of theory. We used the 20 most
common stoichiometries (Fig.1 b)), having a minimum of 1.7k constitutional
isomers available in the dataset. To extend the QM9-NMR C7O2H10 constitutional
isomers space, we generated 54’641 SMILES using Surgesurge . 3D structures
have been generated using ETKDGriniker_better_2015  and CRESTcrest  using
GFN2-xTB/GFN-FF. Adding the structures to QM9, a total pool size of 56.95k
C7O2H10 isomers was obtained. For the training of chemical shift machine
learning models, we selected C8OH12, C8OH10, C8OH14, C7O2H8 and C7O2H12
constitutional isomers, yielding a total of 143k ${}^{13}\mathrm{C}$ and 214k
${}^{1}\mathrm{H}$ training points, respectively.

##  III Results & Discussion

###  III.1 Spectra matching accuracy with synthetic noise

To analyse the influence of noise and number of candidates on the elucidation
success, we applied Gaussian noise to ${}^{13}\mathrm{C}$ and
${}^{1}\mathrm{H}$ shifts of C7O2H10, C5N3OH7 and C8OH14 constitutional
isomers, respectively. Fig.2 a-b) depicts a sigmoidal shaped trend of Top-1
elucidation accuracies at increasing candidate pool sizes $N_{QM9}$ as a
function of mean absolute error (MAE). Note that increasing the maximum
candidate pool size leads to an offset of the trend towards less permissible
errors. A possible explanation is the correlation of the density of chemical
space with increasing numbers of candidate spectra $N$SML . As shift
predictions need to become more accurate, limiting $N$ through prior knowledge
of the chemical space could be beneficial. Similar findings have been reported
by Sridharan et al.nmrinverse_Sridharan22 , noting that brute force
enumerations of chemical space lead to worse rankings than constrained graph
generation. Note that while the trends in ${}^{13}\mathrm{C}$ and
${}^{1}\mathrm{H}$ elucidation are similar, less error is permissible when
using ${}^{1}\mathrm{H}$ shifts.

To further reduce the ambiguity, we include both ${}^{13}\mathrm{C}$ and
${}^{1}\mathrm{H}$ shifts into the matching problem as per Eq.2. Results
suggest 50$\%$ and $\sim 150\%$ more permissible ${}^{13}\mathrm{C}$ and
${}^{1}\mathrm{H}$ errors when both spectra are considered in the matching
process (Fig.2 c)). Similar to how chemists solve the elucidation problem, the
inclusion of more distinct properties increases the uniqueness and can improve
the elucidation success.

###  III.2 Extrapolating the search space

Due to the limited amount of constitutional isomers in databases compared to
the number of possible graphs faced during inverse design (Fig.1 b)),
assessing the chemical shift accuracy for successful elucidation is severely
limited. As such, we extrapolate elucidation performance curves to obtain
estimates about chemical shift accuracies in candidate pool sizes larger than
QM9. We fit each elucidation performance curve (Fig.2 a-b)), respectively,
using a smoothly broken power law function:

$f(x)=(1+(\frac{x}{x_{b}})^{d})^{\alpha}$ (3)

with $x_{b}$ controlling the upper bend and offset, $d$ changing the curvature
and $\alpha$ changing the tilt of the function (see SI Fig.2), respectively.
The parameters of Eq.3 as a function of $N$ can again be fitted using a power
law function (see SI Fig.2) and extrapolated to the total number of graphs
$N_{Surge}$, respectively.

Results of the extrapolation (Fig.2 a-b) dashed) indicate significant
differences in elucidation efficiency among stoichiometries. For instance,
C8OH14 queries are potentially easier to elucidate than C5N3OH7 structures.
Possible reasons are the limited number of C8OH14 graphs compared to millions
of C5N3OH7 isomers. Moreover, the number of heteroatoms of the C5N3OH7
stoichiometry might hamper the characterization when only relying on
${}^{13}\mathrm{C}$ or ${}^{1}\mathrm{H}$ , respectively. Hence, to solve the
inverse structure elucidation problem using experimental data of compounds
larger than QM9, reducing ambiguities through including both
${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ shifts as well as to reduce the
candidate space is critical for elucidation success.

###  III.3 Trends in chemical space

To analyse the elucidation efficiency throughout chemical space, we applied
the Gaussian noise and extrapolation procedure to the 20 most common
stoichiometries in QM9 (Fig.1 b)). Fig.3 a) shows the MAE required for 95$\%$
elucidation success as a function of $N_{Surge}$. Results suggest that less
error is permissible for stoichiometries with large $N_{Surge}$ and fewer
carbon atoms. As such, using only ${}^{13}\mathrm{C}$ shifts might not be
sufficient to fully characterize the compound. Again, similar to how chemists
use multiple NMR spectra to deduct chemical structures, additional information
such as ${}^{1}\mathrm{H}$ shifts are beneficial to extend the information
content.

In Fig. 3 b), the error permissiveness of spectra matching using only
${}^{13}\mathrm{C}$ (see SI Fig.4 for ${}^{1}\mathrm{H}$ ) versus combining
both ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ is being compared, revealing a
linear trend between both. Note that the C7NOH7 stoichiometry shows the
smallest benefit from adding additional ${}^{1}\mathrm{H}$ information.
Interestingly, a hierarchy for C7NOHX stoichiometries of different degrees of
unsaturation is visible, indicating an inverse correlation between number of
hydrogens and ${}^{13}\mathrm{C}_{single}$ MAE (Fig. 3 b) green). Similar
hierarchies are also observed for other stoichiometries such as C7O2HX and
C8OHX (Fig. 3 b) blue and orange). On average, the combination of
${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ for spectra matching increases the
error permissiveness of ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ by 85$\%$
and 261$\%$ (see SI Fig.4), respectively.

###  III.4 Comparison to machine learned shift predictions

To test the elucidation performance using machine learning predictions, we
trained ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ KRR models at increasing
training set sizes (see SI Fig.5 for learning curves) and predicted chemical
shifts of 56k C7O2H10 constitutional isomers. Results again show similar
trends as observed with Gaussian noise (Fig.4 a-b)), however, indicate more
permissive accuracy thresholds. For instance, KRR ${}^{13}\mathrm{C}$
predictions at 2 ppm MAE can identify 64$\%$ of queries rather than only
17$\%$ suggested by the Gaussian noise experiment. The difference could be
explained due the systematic, non uniform nature of the
QM9ramakrishnan_quantum_2014  chemical space, influencing the shape and
extrapolation of elucidation performance curves in Fig.2. Moreover, Gaussian
noise is applied to all shifts at random compared to possibly more systematic
machine learning predictions. Note that the trade-off between error and $N$ is
consistent and that the exact parameters will depend on the machine learning
model and the finite sampling of constitutional isomer space.

To model possible experimental noise on query spectra, we apply Gaussian noise
to query spectra and evaluate the elucidation performance of the best
performing machine learning model (see insets in Fig.4 a-b)). Results indicate
a halving of elucidation accuracy when the query spectrum contains up to 2 ppm
MAEQ in ${}^{13}\mathrm{C}$ and 0.15 ppm MAE in ${}^{1}\mathrm{H}$ error,
respectively. Thus, in the presence of experimental measurement noise even
higher prediction accuracies might be necessary. Combining both
${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ spectra for matching improves the
elucidation performance up to 90$\%$ (Fig.4 e)). Again, the combination of
spectra for elucidation highlights the effectiveness of reducing the ambiguity
of the matching problem by including additional properties.

Investigating potential strategies to reduce the constitutional isomer search
space, we constrained $N$ based on functional groups (see SI Table 1).
Randomly selecting functional groups present in each query, $N$ can be reduced
by 50$\%$ and 62$\%$ on average (see Fig.4 d) inset for distributions),
respectively. Results in Fig.4 c-d) indicate an increase of the elucidation
accuracy by 5$\%$ in ${}^{13}\mathrm{C}$ and up to 10$\%$ for
${}^{1}\mathrm{H}$ , respectively, in agreement with the elucidation
performance in Fig.4 a-b). Note that the knowledge of two functional groups
only led to marginal improvements. However, fragmentation could be more
beneficial for larger compounds than present in QM9ramakrishnan_quantum_2014 ,
as reported by Yao et al.Yao2023 . Using both ${}^{13}\mathrm{C}$ and
${}^{1}\mathrm{H}$ shifts on the reduced search space only lead to marginal
improvements of 0.5$\%$ over the results of the full search space.

Figure 5: Performance curves (red, blue) of the MAE permissible to correctly
identify 60, 70, 80, 90$\%$ of C7O2H10 query spectra at a given pool size $N$
using machine learning shifts predictions, respectively.
${}^{13}\mathrm{C}_{single}$ (red) only uses ${}^{13}\mathrm{C}$ shifts for
elucidation, whereas ${}^{13}\mathrm{C}_{combined}$ uses ${}^{13}\mathrm{C}$
and ${}^{1}\mathrm{H}$ spectra combined, assuming a ${}^{1}\mathrm{H}$ MAE of
0.15 ppm. The learning curve (orange) indicates the systematic improvement of
QM9ramakrishnan_quantum_2014  ${}^{13}\mathrm{C}$ chemical shift predictions
as a function of training set size $N_{train}$ using KRR with the
FCHL19christensen_fchl_2020  representation.

###  III.5 Balancing search space and accuracy

We use performance curves to analyse the relationship between the elucidation
performance of C7O2H10 queries, machine learning prediction errors and
candidate pool sizes $N$. The systematic decay of performance curves (Fig.5
red and blue) again demonstrates that constraining $N$ with prior knowledge
allows for less accurate shift predictions to be applicable. Extrapolating the
${}^{13}\mathrm{C}_{single}$ performance curves indicates a machine learning
MAE of 0.93 ppm to correctly rank 90$\%$ of queries out of 56k possible
candidates (Fig.5 red), 0.02 ppm lower than suggested by Gaussian noise. To
reach an MAE of 0.93 ppm, four million training instances are required (Fig.5
orange). Using both ${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ shifts requires
two orders of magnitude less training data (Fig.5 blue). As such, facing
expensive experimental measurements and ab initio calculations, more effective
inverse structure elucidation could be achieved by balancing machine learning
data needs through reduced search spaces and incorporation of additional
properties.

##  IV Conclusion

We have presented an analysis of the effectiveness of the NMR spectra matching
task encountered in the inverse structure elucidation problem. By
systematically controlling the predictive accuracy of ${}^{13}\mathrm{C}$ and
${}^{1}\mathrm{H}$ chemical shifts, we found consistent trends throughout
chemical compound space, suggesting that higher errors become permissible as
the number of possible candidates decreases. Note that while we relied on 1D
ab initio NMR data, similar analysis could be performed using 1D or 2D
experimental spectra. Applications to the most common constitutional isomers
in QM9 highlight that chemical spaces with many heteroatoms are harder to
characterize when only relying on a single type of chemical shift. Using both
${}^{13}\mathrm{C}$ and ${}^{1}\mathrm{H}$ chemical shifts increases the error
permissiveness by 85$\%$ and 261$\%$ on average, respectively. Machine
learning predictions for 56k C7O2H10 compounds showed that using both
${}^{13}\mathrm{C}$ or ${}^{1}\mathrm{H}$ shifts increased elucidation success
to 90$\%$ compared to only 64$\%$ and 36$\%$ when used alone, respectively.
The usefulness of the analysis is expressed via performance curves, showing
that training demands can be reduced by orders of magnitude compared to
relying on specific shifts alone.

We believe that as the accuracy of machine learning models to distinguish
spectra is limited, constrained search spaces or inclusion of more distinct
properties are necessary to improve candidate rankings. Rather than solely
relying on more accurate models, future approaches could include explicit
knowledge of chemical reactions, functional groups or data from mass
spectrometry, infrared- or Raman spectroscopyGastegger2021 ; McGill2021 ;
Grimme2013 ; Shrivastava2021 ; Jung2023 ; Pracht2020 , respectively. Finally,
explicitly accounting for atomic similarities and chemical shift uncertainties
via the DP5 probability might further increase the confidence in structure
assignmentsdp5 .

## Acknowledgement

O.A.v.L. has received funding from the European Research Council (ERC) under
the European Union’s Horizon 2020 research and innovation programme (grant
agreement No. 772834). O.A.v.L. has received support as the Ed Clark Chair of
Advanced Materials and as a Canada CIFAR AI Chair. Icons in Fig.1 from DBCLS,
Openclipart and Simon Dürr from bioicons.com under CC-BY 4.0 and CC0,
respectively.

## Data & Code Availability

The QM9-NMR dataset is openly available at
https://moldis.tifrh.res.in/data/QM9NMR. The code and additional data used in
this study is available at https://doi.org/10.5281/zenodo.8126380.

## Conflict of Interest

The authors have no conflict of interest.

## References

## References

  * (1) R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld, “Quantum chemistry structures and properties of 134 kilo molecules,” Scientific Data, vol. 1, pp. 1–7, Aug. 2014.  Number: 1 Publisher: Nature Publishing Group.
  * (2) B. D. McKay, M. A. Yirik, and C. Steinbeck, “Surge: a fast open-source chemical graph generator,” Journal of Cheminformatics, vol. 14, Apr. 2022\.
  * (3) J. Bai, L. Cao, S. Mosbach, J. Akroyd, A. A. Lapkin, and M. Kraft, “From platform to knowledge graph: Evolution of laboratory automation,” JACS Au, vol. 2, no. 2, pp. 292–309, 2022.
  * (4) S. Herres-Pawlis, O. Koepler, and C. Steinbeck, “Nfdi4chem: Shaping a digital and cultural change in chemistry,” Angewandte Chemie International Edition, vol. 58, no. 32, pp. 10766–10768, 2019.
  * (5) P. S. Gromski, J. M. Granda, and L. Cronin, “Universal chemical synthesis and discovery with ‘the chemputer’,” Trends in Chemistry, vol. 2, no. 1, pp. 4–12, 2020.
  * (6) I. W. Davies, “The digitization of organic synthesis,” Nature, vol. 570, pp. 175–181, June 2019.
  * (7) B. Huang, G. F. von Rudorff, and O. A. von Lilienfeld, “Towards self-driving laboratories in chemistry and materials sciences: The central role of dft in the era of ai,” 2023.
  * (8) R. J. Hickman, M. Aldeghi, F. Häse, and A. Aspuru-Guzik, “Bayesian optimization with known experimental and design constraints for chemistry applications,” Digital Discovery, vol. 1, pp. 732–744, 2022.
  * (9) Y. Xie, K. Sattari, C. Zhang, and J. Lin, “Toward autonomous laboratories: Convergence of artificial intelligence and experimental automation,” Progress in Materials Science, vol. 132, p. 101043, 2023.
  * (10) Y. Jiang, D. Salley, A. Sharma, G. Keenan, M. Mullin, and L. Cronin, “An artificial intelligence enabled chemical synthesis robot for exploration and optimization of nanomaterials,” Science Advances, vol. 8, no. 40, p. eabo2626, 2022.
  * (11) R. D. King, K. E. Whelan, F. M. Jones, P. G. K. Reiser, C. H. Bryant, S. H. Muggleton, D. B. Kell, and S. G. Oliver, “Functional genomic hypothesis generation and experimentation by a robot scientist,” Nature, vol. 427, pp. 247–252, Jan. 2004.
  * (12) B. Burger, P. M. Maffettone, V. V. Gusev, C. M. Aitchison, Y. Bai, X. Wang, X. Li, B. M. Alston, B. Li, R. Clowes, N. Rankin, B. Harris, R. S. Sprick, and A. I. Cooper, “A mobile robotic chemist,” Nature, vol. 583, pp. 237–241, July 2020.
  * (13) H. Fakhruldeen, G. Pizzuto, J. Glowacki, and A. I. Cooper, “Archemist: Autonomous robotic chemistry system architecture,” 2022.  arXiv:2204.13571.
  * (14) N. H. Angello, V. Rathore, W. Beker, A. Wołos, E. R. Jira, R. Roszak, T. C. Wu, C. M. Schroeder, A. Aspuru-Guzik, B. A. Grzybowski, and M. D. Burke, “Closed-loop optimization of general reaction conditions for heteroaryl suzuki-miyaura coupling,” Science, vol. 378, no. 6618, pp. 399–405, 2022\.
  * (15) M. Elyashberg, “Identification and structure elucidation by nmr spectroscopy,” TrAC Trends in Analytical Chemistry, vol. 69, pp. 88–97, 2015.
  * (16) M. E. Elyashberg, K. A. Blinov, A. J. Williams, S. G. Molodtsov, G. E. Martin, and E. R. Martirosian, “Structure elucidator: A versatile expert system for molecular structure elucidation from 1d and 2d nmr data and molecular fragments,” Journal of Chemical Information and Computer Sciences, vol. 44, no. 3, pp. 771–792, 2004.  PMID: 15154743.
  * (17) P. Giraudeau, “Challenges and perspectives in quantitative NMR,” Magn. Reson. Chem., vol. 55, pp. 61–69, Jan. 2017.
  * (18) P. H. Willoughby, M. J. Jansma, and T. R. Hoye, “A guide to small-molecule structure assignment through computation of (${}^{1}\mathrm{H}$ and ${}^{13}\mathrm{C}$) NMR chemical shifts,” Nat. Protoc., vol. 9, pp. 643–660, Mar. 2014.
  * (19) M. Elyashberg and D. Argyropoulos, “Computer assisted structure elucidation (CASE): Current and future perspectives,” Magn. Reson. Chem., vol. 59, pp. 669–690, July 2021.
  * (20) C. S. Kim, J. Oh, and T. H. Lee, “Structure elucidation of small organic molecules by contemporary computational chemistry methods,” Arch. Pharm. Res., vol. 43, pp. 1114–1127, Nov. 2020.
  * (21) A. Howarth, K. Ermanis, and J. M. Goodman, “DP4-AI automated NMR data analysis: straight from spectrometer to structure,” Chem. Sci., vol. 11, pp. 4351–4359, Mar. 2020.
  * (22) A. Howarth and J. M. Goodman, “The DP5 probability, quantification and visualisation of structural uncertainty in single molecules,” Chem. Sci., vol. 13, pp. 3507–3518, Mar. 2022.
  * (23) W. Bremser, “Hose — a novel substructure code,” Analytica Chimica Acta, vol. 103, pp. 355–365, Dec. 1978.
  * (24) M. W. Lodewyk, M. R. Siebert, and D. J. Tantillo, “Computational prediction of 1h and 13c chemical shifts: A useful tool for natural product, mechanistic, and synthetic organic chemistry,” Chemical Reviews, vol. 112, no. 3, pp. 1839–1862, 2012.  PMID: 22091891.
  * (25) E. Jonas, S. Kuhn, and N. Schlörer, “Prediction of chemical shift in NMR: A review,” Magn. Reson. Chem., vol. 60, pp. 1021–1031, Nov. 2022\.
  * (26) D. Sebastiani and M. Parrinello, “A new ab-initio approach for NMR chemical shifts in periodic systems,” The Journal of Physical Chemistry A, vol. 105, pp. 1951–1958, Feb. 2001.
  * (27) D. Sebastiani and U. Rothlisberger, “Nuclear magnetic resonance chemical shifts from hybrid DFT QM/MM calculations,” The Journal of Physical Chemistry B, vol. 108, pp. 2807–2815, Feb. 2004.
  * (28) S. Kuhn and N. E. Schlörer, “Facilitating quality control for spectra assignments of small organic molecules: nmrshiftdb2–a free in-house NMR database with integrated LIMS for academic service laboratories,” Magn. Reson. Chem., vol. 53, pp. 582–589, Aug. 2015.
  * (29) C. R. Groom, I. J. Bruno, M. P. Lightfoot, and S. C. Ward, “The cambridge structural database,” Acta Crystallogr. B Struct. Sci. Cryst. Eng. Mater., vol. 72, pp. 171–179, Apr. 2016.
  * (30) A. Gupta, S. Chakraborty, and R. Ramakrishnan, “Revving up ${}^{13}\mathrm{C}$ NMR shielding predictions across chemical space: benchmarks for atoms-in-molecules kernel machine learning with new data for 134 kilo molecules,” Machine Learning: Science and Technology, vol. 2, p. 035010, May 2021.
  * (31) L. A. Bratholm, W. Gerrard, B. Anderson, S. Bai, S. Choi, L. Dang, P. Hanchar, A. Howard, S. Kim, Z. Kolter, R. Kondor, M. Kornbluth, Y. Lee, Y. Lee, J. P. Mailoa, T. T. Nguyen, M. Popovic, G. Rakocevic, W. Reade, W. Song, L. Stojanovic, E. H. Thiede, N. Tijanic, A. Torrubia, D. Willmott, C. P. Butts, and D. R. Glowacki, “A community-powered search of machine learning strategy space to find NMR property prediction models,” PLOS ONE, vol. 16, p. e0253612, July 2021.
  * (32) M. Rupp, R. Ramakrishnan, and O. A. von Lilienfeld, “Machine learning for quantum mechanical properties of atoms in molecules,” J. Phys. Chem. Lett., vol. 6, pp. 3309–3313, Aug. 2015.
  * (33) Y. Kwon, D. Lee, Y.-S. Choi, M. Kang, and S. Kang, “Neural message passing for nmr chemical shift prediction,” Journal of Chemical Information and Modeling, vol. 60, no. 4, pp. 2024–2030, 2020.  PMID: 32250618.
  * (34) E. Jonas and S. Kuhn, “Rapid prediction of NMR spectral properties with quantified uncertainty,” J. Cheminform., vol. 11, p. 50, Aug. 2019.
  * (35) J. Han, H. Kang, S. Kang, Y. Kwon, D. Lee, and Y.-S. Choi, “Scalable graph neural network for nmr chemical shift prediction,” Phys. Chem. Chem. Phys., vol. 24, pp. 26870–26878, 2022.
  * (36) F. M. Paruzzo, A. Hofstetter, F. Musil, S. De, M. Ceriotti, and L. Emsley, “Chemical shifts in molecular solids by machine learning,” Nature Communications, vol. 9, Oct. 2018.
  * (37) F. Musil, M. J. Willatt, M. A. Langovoy, and M. Ceriotti, “Fast and accurate uncertainty estimation in chemical machine learning,” Journal of Chemical Theory and Computation, vol. 15, pp. 906–915, Jan. 2019.
  * (38) E. Jonas, “Deep imitation learning for molecular inverse problems,” in Advances in Neural Information Processing Systems, vol. 32, Curran Associates, Inc., 2019.
  * (39) Z. Huang, M. S. Chen, C. P. Woroch, T. E. Markland, and M. W. Kanan, “A framework for automated structure elucidation from routine nmr spectra,” Chem. Sci., vol. 12, pp. 15329–15338, 2021.
  * (40) B. Sridharan, S. Mehta, Y. Pathak, and U. D. Priyakumar, “Deep reinforcement learning for molecular inverse problem of nuclear magnetic resonance spectra to molecular structure,” The Journal of Physical Chemistry Letters, vol. 13, no. 22, pp. 4924–4933, 2022.  PMID: 35635003.
  * (41) A. S. Christensen, L. A. Bratholm, F. A. Faber, and O. Anatole von Lilienfeld, “FCHL revisited: Faster and more accurate quantum machine learning,” The Journal of Chemical Physics, vol. 152, p. 044107, Jan. 2020.  Publisher: American Institute of Physics.
  * (42) S. Riniker and G. A. Landrum, “Better Informed Distance Geometry: Using What We Know To Improve Conformation Generation,” Journal of Chemical Information and Modeling, vol. 55, pp. 2562–2574, Dec. 2015\.  Publisher: American Chemical Society.
  * (43) P. Pracht, F. Bohle, and S. Grimme, “Automated exploration of the low-energy chemical space with fast quantum chemical methods,” Phys. Chem. Chem. Phys., vol. 22, pp. 7169–7192, 2020.
  * (44) D. Lemm, G. F. von Rudorff, and O. A. von Lilienfeld, “Improved decision making with similarity based machine learning,” 2022.  arXiv:2205.05633.
  * (45) L. Yao, M. Yang, J. Song, Z. Yang, H. Sun, H. Shi, X. Liu, X. Ji, Y. Deng, and X. Wang, “Conditional molecular generation net enables automated structure elucidation based on 13c NMR spectra and prior knowledge,” Analytical Chemistry, vol. 95, pp. 5393–5401, Mar. 2023.
  * (46) M. Gastegger, K. T. Schütt, and K.-R. Müller, “Machine learning of solvent effects on molecular spectra and reactions,” Chemical Science, vol. 12, no. 34, pp. 11473–11483, 2021.
  * (47) C. McGill, M. Forsuelo, Y. Guan, and W. H. Green, “Predicting infrared spectra with message passing neural networks,” Journal of Chemical Information and Modeling, vol. 61, pp. 2594–2609, May 2021.
  * (48) S. Grimme, “Towards first principles calculation of electron impact mass spectra of molecules,” Angewandte Chemie International Edition, vol. 52, pp. 6306–6312, Apr. 2013.
  * (49) A. D. Shrivastava, N. Swainston, S. Samanta, I. Roberts, M. W. Muelas, and D. B. Kell, “MassGenie: A transformer-based deep learning method for identifying small molecules from their mass spectra,” Biomolecules, vol. 11, p. 1793, Nov. 2021.
  * (50) G. Jung, S. G. Jung, and J. M. Cole, “Automatic materials characterization from infrared spectra using convolutional neural networks,” Chemical Science, vol. 14, no. 13, pp. 3600–3609, 2023.
  * (51) P. Pracht, D. F. Grant, and S. Grimme, “Comprehensive assessment of GFN tight-binding and composite density functional theory methods for calculating gas-phase infrared spectra,” Journal of Chemical Theory and Computation, vol. 16, pp. 7044–7060, Oct. 2020.


# Human Biases Preventing The Widespread Adoption Of Self-Driving Cars

Benjamin Kahl

###### Abstract

Self-driving cars offer a plethora of safety advantages over our accustomed
human-driven ones, yet many individuals feel uneasy sharing the road with
these machines and entrusting their lives to their driving capabilities. Thus,
bringing about a widespread adoption of autonomous cars requires overcoming
these compulsions through careful planning and forethought. Here we break down
the three primary psychological barriers that may hamstring or even wholly
prevent their widespread adoption as well as how to tackle them.

##  I Introduction

Recent strides in autonomous driving technology have brought self-driving cars
ever closer to being regular, consumer-grade products. Their adoption promises
to bestow improvements in road safety, efficiency and convenience. But the
biggest challenge to autonomous cars may not be technological, but
psychological.

The vast majority of American citizens report feeling less safe at the
prospect of sharing the road with self-driving vehicles and 78% are afraid to
ride in one. [7]

This simple fact may have far reaching implications. Anxious consumers may be
susceptible to deterrence, provoke legislators into enacting suffocating
regulations, even prompt some into outlawing them entirely.

The underlying apprehension is inextricably linked to the notion of trust, and
all its associated hopes and fears. Autonomous cars will be required to
navigate complex traffic environments with the power of life and death over
pedestrians, drivers and even their own passengers. Given this tremendous
responsibility, consumers regard autonomous cars with trepidation and are
unwilling to yield any vulnerability to a machine they don’t trust.

Deconstructing this mistrust and fear as well as addressing it directly is
quintessential towards paving the way for a widespread adoption of self-
driving cars.

##  II Psychological Roadblocks to the adoption of self-driving cars

Nature magazine article Psychological roadblocks to the adoption of self-
driving vehicles [1] discusses three primary barriers impeding a widespread
adoption and proposes steps towards addressing them: The dilemma of autonomous
ethics, risk heuristics and the theory of the machine mind.

The subsequent chapters will present each as described by Shariff et al. in
the Nature article and provide further insight through data from a series of
additional studies and surveys.

Figure 1: Overview of the primary psychological roadblocks, as shown in
Psychological roadblocks to the adoption of self-driving vehicles [1]

##  III Dilemma of autonomous ethics

Computer-driven cars are being heralded as the future benefactors of a new
standard for road safety, in no small part thanks to their vitally short
reaction-times and pragmatic decision making capabilities.

When faced with split-second decisions, human drivers cannot be expected to
act in accordance of some written ruleset and thus have to rely on their
neural precepts and natural instincts.

For this reason, the ability to assess and act upon difficult predicaments
within a limited timespan has brought fourth a new set of moral conundrums to
be grappled with.

Traditionally described in terms of a tram, railway and lever, the trolley
problem has found itself in the center of discussion about how autonomous cars
should behave on the road, given that extreme traffic situations may lead to a
choice of either putting the lives of several pedestrians into peril, or
saving them by sacrificing its own passengers.

A cars modus operandi may adhere to a classical utilitarian doctrine by
prioritizing the minimization of potential casualties in any given situation.
Alternatively, the priority may lie in keeping the cars own passengers alive.

The dilemma of autonomous ethics refers to the fact that potential self-
driving vehicle purchasers are likelier to purchase self-protective models,
yet want other consumers to drive utilitarian ones.

Potential utilitarian AV111AVs (autonomous vehicles) may include other
vehicles than privately owned cars. For the sake of brevity the term AV will
be used synonymously to autonomous car throughout the remainder of this
document. buyers might be prompted to think twice when confronted with the
thought that their car may one day decide to harm them.

###  III-A Potential counter-measures

While government-imposed regulations may be required to enforce a utilitarian
doctrine, they may also bring unintended side effects, as most consumers
disapprove of such regulations [2].

In addition, disallowing generally preferred self-protective vehicles may
induce a delay in the widespread adoption of AVs, negating the potential lives
saved by utilitarian models in the first place.

Shariff et al. assert that, in order to alleviate consumer deterrence, people
need to feel both safe and virtuous about their AV ownership.

In order to invoke a sentiment of safety, it is imperative to keep the public
informed on the overall safety benefits that self-driving vehicles can provide
as well as highlighting the exceeding rarity of self-sacrifice scenarios.

To beseech a feeling of virtue, Shariff et al. suggest to mimic the appeal of
virtue signalling, which has already found success in the marketing strategies
of electric car manufacturers.

Vehicles such as the Toyota Prius and the BMW i-Series exhibit highly
recognizable, unique shapes and color-schemes on their exterior designs in
order to allow their owners to signal to the public their environmental
commitment. Analogously, utilitarian AVs may allow them to signal their
ethical commitment.

###  III-B Social Dilemma of AVs

This inherently social dilemma is further examined in the Science Journal
article The Social Dilemma of Autonomous Vehicles [2] through a series of 6
surveys that closely emulate the Moral Machine Experiment [3], although with a
significantly smaller sample size and somewhat more specific inquiries.

Figure 2: Results from studies 1-4 as presented in The social dilemma of
autonomous vehicles[2]

The results from the first and second study (see fig. 2) clearly demonstrate
the presence of overwhelming support for a utilitarian driving model, given
the large preference for loss-minimizing choices. When confronted with the
question of how AVs will actually behave, participants’ responses were far
more spread out, suggesting that AVs are believed to act in a less utilitarian
manner than they should be.

The third study examines consumer preference for purchasing minimize or self-
protective vehicles if the participant imagined themselves either alone, with
a coworker or their family in the car. The results depicted in fig. 2 bring
fourth following observations:

  * •

Despite overwhelming agreement with a loss-minimizing morality, the
willingness to buy such a car is considerably lower.

  * •

Agreement with loss-minimizing morality is palpably and negatively affected by
the presence of a coworker or family member in the car.

  * •

The willingness to purchase a self-protective vehicle increases in reverse
proportion to the willingness of buying a loss-minimizing car.

Despite the willingness to purchase self-protective cars surpassing the
willingness to buy minimize cars for multiple passengers, there is still a
slight a preference for utilitarian ones if the participants imagine
themselves alone.

This observation is of particular importance when considered in parity with
the fact that average car occupancy rates lie well below 1.5 [4], meaning most
cars are driven alone.

Furthermore, an increased amount of passengers also dramatically decreases the
likelihood of a self-sacrifice scenario, as an AV with 4 passengers would have
to put the lives of at least 4 pedestrians at risk in order to warrant such a
maneuver.

##  IV Risk heuristics

Ever increasing adoption rates and higher amounts of AVs on the road will
inevitably lead to to an increase in AV-related traffic accidents, despite
their substantial safety advantages over human-driven cars.

However, due to their novelty, AVs will find a spotlight shone upon any
traffic accident they are a part of (particularly lethal ones) in the form of
disproportionately outsized media coverage.

Similarly to coverage on airplane crashes, AVs may find themselves stigmatized
by the sheer amount of detrimental headlines and news articles which amplify
peoples’ fears by tapping into their

  * •

Availability bias, which is the tendency to perceive examples of risks that
come to mind easily as more representative than they genuinely are.

  * •

Affective bias, which causes risks to be perceived as larger if they evoke
vivid emotional reactions.

  * •

Algorithm aversion, the tendency of people to loose faith in erroneous
algorithms faster than in humans making similar mistakes.

  * •

Overconfidence in ones own performance, something highly prevalent in the
driving domain in particular.

The exalted anxiety caused by negative media coverage paired with
overconfidence in ones’ own ability to stay out of accidents may compromise
the financial feasibility of consumer-grade self-driving cars.

###  IV-A Potential counter-measures

Shariff et al. propose to prepare the public for the inevitability of AV
accidents by not promising infallibility while still emphasizing the safety
advantages AVs bring to the table.

Furthermore, legislators are to resist capitulating on peoples’ fears and
instead opt for the implementation of fear-placebos. (High-visibility, low-
cost concessions that do the most to quench public trepidation without posing
a real obstacle to AV manufacturers.)

###  IV-B Blame distribution in dual-error cases

2016 brought this problem into the foreground with the first level 2 AV
traffic fatality222As per the NHTSA[8]: A level 2 AV can itself control both
steering and breaking/accelerating. The human driver must continue to pay full
attention (“monitor the driving environment”) at all times and perform the
rest of the driving task. It is considered to be a form of automated driving,
but not fully autonomous driving. in America, where a Tesla Model S engaged in
Autopilot mode crashed into an 18-wheel tractor-trailer killing its owner and
only passenger.

The incident sparked a wave of news articles from every major news
organization in the US, a level of coverage that none of the over 40 thousand
US traffic fatalities that year came even close to achieving.

In their official statement addressing the crash, Tesla claimed that "neither
Autopilot nor the driver noticed the white side of the tractor trailer against
a brightly lit sky, so the brake was not applied"[9], suggesting that both the
human and machine driver should have taken action but neither did. Both
drivers made a mistake.

Despite this, a considerable amount of the published news articles were
remarkably skewed towards blaming the human driver. Unsubstantiated rumors
began circulating that the driver had been watching a movie[10][11][12] whilst
on the road and had gotten multiple previous warnings[13]. Even the victims
family themselves placed no blame on the car[14], despite having substantial
grounds for a lawsuit[15].

The exact nature of blame distribution in dual-error cases between human and
machine is examined in Blaming humans in autonomous vehicle accidents: Shared
responsibility across levels of automation by Awad et al. through a series of
studies.

The studies differentiate between a two types of drivers for each car: The
primary driver possesses general control over the vehicles’ steering and
acceleration at all times. If a mistake is made by the primary driver, it is
the secondary drivers obligation to intervene and prevent aforementioned
mistake by overriding the primary drivers actions.

Figure 3: Scenarios regarded within the studies of Blaming humans in
autonomous vehicle accidents: Shared responsibility across levels of
automation[5]

Figure 3 lists all permutations of the regarded 2-driver scenarios, where H
stands for human drivers and M for machine drivers. Thus, the MH scenario
(Machine primary, human secondary) would correspond to the aforementioned
Tesla Autopilot and the HM to the Toyota Guardian system. Non critical cases
(H, M, HH and MM) are merely taken for reference.

Fig. 3 also shows the two types of regarded accidents:

####  IV-B1 Bad Intervention

In a bad intervention accident the primary driver does not make a mistake.
Instead, the secondary driver issues a faulty override which ultimately causes
a crash.

As can be observed in fig. 4, the single mistake nature of this scenario leads
to the vast majority of test participants placing the blame on the secondary
driver.

####  IV-B2 Missed Intervention

A missed intervention accident follows the pattern of the above mentioned 2016
Tesla crash: The primary driver makes a mistake, putting the car in peril.
However the secondary driver fails to identify said mistake and thus does not
intervene.

The results shown in fig. 4 put on display that, no matter the type of
accident and the allocation of roles (HM/MH), participants had a clear
tendency to place more blame on the machine driver than the human driver. The
man-in-the-loop is blamed more than the machine-in-the-loop.

Figure 4: Results from the studies conducted in Blaming humans in autonomous
vehicle accidents: Shared responsibility across levels of automation[5]. S1
and S2 refer to different studies.

Awad et al. conclude that, "while there may be many psychological barriers to
self-driving car adoption, public over-reaction to dual-error cases is not
likely to be one of them. In fact, we should perhaps be concerned about public
under-reaction".

While bad intervention cases may be causal to the underlying psychological
roadblock, the diminished blame on AV manufacturers in missed intervention
cases may bring a different set of problems with it. It may, for instance,
prompt manufacturers to design their Guardian systems more passively, as to
avoid bad intervention failures and allocating traffic accidents caused by
their systems further towards the missed-intervention spectrum.

In addition to the lesser pressure on manufacturers to perform, the foregoing
disbalance may induce societal level responses to this trend. With the
majority of the blame being allocated on the human, people may train
themselves to drive so that, in the case of a crash, the blame would befall on
the machine. (By, for example, not attempting to correct a faulty bad
intervention.)

##  V Lack of transparency

The final roadblock presented by Shariff et al. is the lack of transparency
into the machine decision making process and the lack of trust inherently
caused by it.

A central part of this barrier is the information asymmetry of consumers being
highly aware of the individual flaws and failures within the AV domain, yet
blissfully unaware of the plethora of small successes and optimizations.

Partially fuelled by the black-box nature of machine learning algorithms, the
scarcity of exact knowledge into how AVs make their decisions turn them into
unpredictable agents. This causes not just potential customers but also other
road participants to approach AVs with some trepidation in regular traffic
environments.

On the other hand, providing the public with a full disclosure of information
about how AVs work internally would simply overwhelm the average consumer with
the tremendous amount of technical details and intricacies that go into
developing a self-driving vehicle.

The solution proposed by Shariff et al. is to conduct specialized research
into what type of information a consumer should be exposed to in order to form
trustworthy bonds between humans and machines, and what information should
remain obscured through a series of abstractions and simplifications.

To some extent, the lack of transparency and predictability of AV decision
making may also be linked to the disproportionate amount of attention placed
of fringe moral scenarios, such as the trolley problem.

###  V-A Moral preferences for AVs

The data portrayed in fig. 2 (Roadblock 1) seems to suggest that people have a
general preference for strictly utilitarian driving models, but here
participants were confronted with classical trolley-problem dilemmas where the
choice is binary and the outcome probability is absolute.

In the publication How Should Autonomous Cars Drive? A Preference for Defaults
in Moral Judgments Under Risk and Uncertainty[6] Meder at al. claim that the
abstract nature of such dilemmas are highly unrealistic and thus inadequate
for the purpose of modelling how AVs should drive.

The trolley problem is typically presented as a choice between action (pulling
a lever to prevent a larger number of deaths but actively cause a different
set of people to die) or inaction (abstain from any action that may cause harm
to someone but allow a greater number of people to be harmed). In the domain
of AVs, the problem can be modelled as either staying in lane and harming a
pedestrian that is crossing the street or swerving to the side and possibly
crash into a bystander standing on the sidewalk.

Neither human-driven nor computer-controlled cars can possibly assess the
exact outcome of any situation with flawless precision, thus these decisions
have to be made on a probabilistic basis at best. In their paper, Meder et al.
back this point up by quoting an engineer working on AVs:

"It takes some of the intellectual intrigue out of the [trolley] problem, but
the answer is almost always “slam on the brakes”…. You’re much more confident
about things directly in front of you, just because of how the system works,
but also your control is much more precise by slamming on the brakes than
trying to swerve into anything." (Hern, 2016)

This emphasizes a higher degree of uncertainty when certain actions (swerving)
are taken in contrast to others (breaking). In order to discover how
participants believe AVs should behave under such circumstances, the outcomes
of the choices offered in these surveys were only probabilistically known or
entirely unknown.

The study is divided into a set of two experiments:

####  V-A1 Experiment 1

Figure 5: Predicament described in the first experiment of How Should
Autonomous Cars Drive? A Preference for Defaults in Moral Judgments Under Risk
and Uncertainty[6]. Depiction adapted from The social dilemma of autonomous
vehicles[2]

The predicament described to participants in experiment 1 follows the
depiction in fig. 5:

A car (either human-driven or AV) is moving along a street when it suddenly
encounters a pedestrian on the street in front of it. The chances of collision
are assessed by the driver as either 20%, 50% or 80%.

The driver must now choose to either put on the brakes while staying in lane
and cause a lethal accident with the above mentioned probability, or
alternatively, it can try to swerve to the side and negate the chances of
colliding with the pedestrian entirely. Instead, if the car swerves, there is
either a 50% or an unknown chance to collide with an unrelated bystander.

The survey was taken by a total of 1648 participants of which 872 were used in
result analysis, as the remaining 776 either dropping out or were excluded on
grounds of insufficient language proficiency or previous participation.

Each test subject was asked a series of 5 questions:

  * •

Decision Preference: Should the car stay or swerve?

  * •

Moral Judgement: How morally acceptable is it to stay or swerve?

  * •

Swerving Threshold: Participants were asked to provide a minimum likelihood of
collision at which to swerve.

  * •

Probability Estimate: If the probability of collision with the bystander was
unknown, participants were asked to provide an estimate.

  * •

Decision Rule: What general rule should drivers follow in such situations?

Figure 6: Results to first and second questions of experiment 1.

Fig. 6 shows the distribution of answers given to the first question. As one
may expect, the proportion of people opting for the "swerve" option increases
in relation to the chances of collision with the pedestrian.

However, rudimentary utilitarian arithmetic would yield a graph passing
through the 50%-50% intersection, as the chances of killing either the
pedestrian or bystander are equal. Instead, participants showed a higher
preference for staying in lane and thus, evidently, did not adhere to a simple
consequentialist analysis.

Equally noteworthy is the fact that, if the chances of collision with
bystander were unknown, participants preferred the car to stay in lane, even
if likelihood of collision with pedestrian were unusually high (80%). In
addition, this only holds true for AVs and not human drivers, meaning that,
according to participants, AVs should be less prone to swerving maneuvers than
their human counterparts.

A similar trend can be observed in the results of the second question, with
almost all data-points hovering on the side of staying. Furthermore, most AV
data-points are above those of human drivers, meaning that participants found
it to be more moral for computer-driven cars to stay in lane than human-driven
ones.

Figure 7: Proportion of participants answering "never swerve" to the third
question of experiment 1.

Similarly, fig. 7 shows that the proportion of people saying to "never swerve"
was consistently higher for autonomous cars in the third question.

In light of the foregoing, it can be concluded that subjects placed a
particular weight on a default action (putting on the brakes and staying in
lane), even if this approach was not the loss-minimizing one.

####  V-A2 Experiment 2

The second experiment was meant to verify whether if the foregoing findings
held up with hindsight, meaning if these preferences remained when the outcome
of the scenario was already known.

The described scenario follows the same pattern, though here only AVs are
regarded and the estimated chances of collision were both either unknown or
50%. In addition the outcome (accident or no accident) with a taken action by
the car (staying or swerving) were known to participants.

Figure 8: Moral judgement for AV actions from experiment 2.

When prompted for a moral judgement, the distribution of answers follows fig.
8. Evidently, the decrease in moral acceptability between collision and no
collision is negligible if the car stayed in lane, yet decreased sharply if it
swayed, thus the hindsight effect is significantly stronger for swaying
scenarios.

The preference for default rules comes as a consequence of the underlying
dichotomy between two different consequentialist rulesets:

  * •

Simple Consequentialism: A simple consequentialist model functions in
accordance to simple utilitarian arithmetic. The moral act is the one that
statistically maximizes some social utility criterion. Only the direct
consequences of that one particular act are considered.

  * •

Rule Consequentialism: In a rule consequentilist model, the rightness of an
action derives from whether if the action maximizes utility in a whole class
of situations governed by a rule.

Applying a default rule such as "putting on the brakes and staying in lane"
throughout all trolley-like scenarios increases the predictability of AVs and
allows other drivers and pedestrians to adapt accordingly. If we instead
desire the statistical minimization of casualties above all else, AV driving
patterns become unpredictable and, as a result, untrustworthy, given that
pedestrians and drivers could never foresee when a car may decide to harm them
in pursuit of the greater good.

The tremendous amount of focus placed on highly unrealistic, statistically
unlikely predicaments such as the trolley problem, may be a primary cause to
the lack of trust, as it paints AVs in an unfamiliar, cold and calculating
light.

##  VI CONCLUSIONS

The future adoption-rate of self-driving cars will entirely depend on
customers’ willingness to purchase them. Perceiving AVs as either dangerous,
untrustworthy or immoral will cause these rates to suffer.

Disproportionate exposure to news articles and headlines on AV accidents may
paint a picture of a death trap on wheels, despite all the brazen promises and
claims on safety improvements. Mediating expectations may be the primary
amendment to this issue by not promising infallibility and emphasizing the
overall safety benefits.

Although consumer deterrence may come in consequence of extreme (bad
intervention) cases, the looming threat of negative press coverage also
ensures that manufacturers are kept under pressure to perform. The pedantic
gaze provided by a critical press implies that AV manufacturers cannot simply
shirk responsibility for their products’ lapses, as they will be severely
questioned about any and all shortcomings.

Furthermore, familiarizing AV driving patterns through the utilization of
default actions and rules may be a quintessential step in pursuit of a
trustworthy kinship between human and machine drivers.

## References

  * [1] Shariff, Azim & Bonnefon, Jean-François & Rahwan, Iyad. (2017). "Psychological roadblocks to the adoption of self-driving vehicles". Nature Human Behaviour. 1. 10.1038/s41562-017-0202-6. URL: https://www.researchgate.net/publication/319630815_Psychological_roadblocks_to_the_adoption_of_self-driving_vehicles
  * [2] Bonnefon, Jean-François and Shariff, Azim and Rahwan, Iyad (2016), "The social dilemma of autonomous vehicles", American Association for the Advancement of Science, URL: https://science.sciencemag.org/content/352/6293/1573
  * [3] Awad, Edmond & Dsouza, Sohan & Kim, Richard & Schulz, Jonathan & Henrich, Joseph & Shariff, Azim & Bonnefon, Jean-François & Rahwan, Iyad. (2018). "The Moral Machine Experiment". Nature. 563. 10.1038/s41586-018-0637-6.
  * [4] European Environment Agency - EEA, "Occupancy rates of passenger vehicles", 2015, URL: https://www.eea.europa.eu/data-and-maps/indicators/occupancy-rates-of-passenger-vehicles/occupancy-rates-of-passenger-vehicles
  * [5] Awad, Edmond & Levine, Sydney & Kleiman-Weiner, Max & Dsouza, Sohan & Tenenbaum, Josh & Shariff, Azim & Bonnefon, Jean-François & Rahwan, Iyad. (2020). Blaming humans in autonomous vehicle accidents: Shared responsibility across levels of automation. Nature Human Behaviour. 10.1038/s41562-019-0762-8. URL: https://arxiv.org/abs/1803.07170
  * [6] Meder, Björn & Fleischhut, Nadine & Krumnau, Nina-Carolin & Waldmann, Michael. (2018). "How Should Autonomous Cars Drive? A Preference for Defaults in Moral Judgments Under Risk and Uncertainty". Risk Analysis. 10.1111/risa.13178. URL: https://www.researchgate.net/publication/327314813_How_Should_Autonomous_Cars_Drive_A_Preference_for_Defaults_in_Moral_Judgments_Under_Risk_and_Uncertainty
  * [7] American Automobile Association (7 March 2017) "Americans Feel Unsafe Sharing the Road with Fully Self-Driving Vehicles.", URL: https://newsroom.aaa.com/2017/03/americans-feel-unsafe-sharing-road-fully-self-driving-cars/
  * [8] National Highway Traffic Safety Administration - NHTSA, "Automated Vehicles for Safety" URL: https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety
  * [9] The Tesla Team, "A Tragic Loss", June 30, 2016, URL: https://www.tesla.com/blog/tragic-loss
  * [10] Andrew Buncombe - The Independent, "Tesla crash: Driver who died while on Autopilot mode ‘was watching Harry Potter’", 1 July 2016 URL: https://www.independent.co.uk/news/world/americas/tesla-crash-driver-who-died-while-on-autopilot-mode-was-watching-harry-potter-a7114356.html
  * [11] Nick Allen - The Telegraph, "Tesla autopilot crash driver ’was watching Harry Potter movie’, witness claims", 1 July 2016 URL: https://www.telegraph.co.uk/news/2016/07/01/tesla-autopilot-crash-driver-was-watching-harry-potter-movie-wit/
  * [12] Sam Lewin, Nicky Woolf - The Guardian, "Tesla driver killed while using autopilot was watching Harry Potter, witness says", 1 July 2016 URL: https://www.theguardian.com/technology/2016/jul/01/tesla-driver-killed-autopilot-self-driving-car-harry-potter
  * [13] David Shepardson - Reuters, "Tesla driver in fatal ’Autopilot’ crash got numerous warnings: U.S. government", 19 July 2017 URL: https://www.reuters.com/article/us-tesla-crash/tesla-driver-in-fatal-autopilot-crash-got-numerous-warnings-u-s-government-idUSKBN19A2XC
  * [14] David Shepardson - Reuters, "Family of Tesla driver killed in ’autopilot’ crash does not blame car", 21 September 2017 URL: https://www.reuters.com/article/us-tesla-autopilot/family-of-tesla-driver-killed-in-autopilot-crash-does-not-blame-car-idUSKCN1BM22T
  * [15] Nicky Woolf - The Guardian, "Tesla fatal autopilot crash: family may have grounds to sue, legal experts say", 6 July 2016 URL: https://www.theguardian.com/technology/2016/jul/06/tesla-autopilot-crash-joshua-brown-family-potential-lawsuit