# Bimanual Manipulation of Steady Hand Eye Robots with Adaptive Sclera Force
Control: Cooperative vs. Teleoperation Strategies

Mojtaba Esfandiari1, Golchehr Amirkhani1, Graduate Student Member, IEEE,
Peter Gehlbach2, Member, IEEE, Russell H. Taylor3, Life Fellow, IEEE, Iulian
Iordachita1, Senior Member, IEEE *This work is supported by the U.S. National
Institutes of Health under the grant numbers 2R01EB023943-04A1, 1R01
EB025883-01A1, R01 EB034397, and partially by JHU internal funds.1 Mojtaba
Esfandiari, Golchehr Amirkhani, and Iulian Iordachita are with the Department
of Mechanical Engineering and also Laboratory for Computational Sensing and
Robotics at the Johns Hopkins University, Baltimore, MD, 21218, USA (e-mail:
mesfand2,gamirkh1, iordachita@jhu.edu). 2 Peter Gehlbach is with the Wilmer
Eye Institute, Johns Hopkins Hospital, Baltimore, MD, 21287, USA. (e-mail:
<EMAIL_ADDRESS>3Russell H. Taylor is with the Department of Computer
Science and also the Laboratory for Computational Sensing and Robotics at the
Johns Hopkins University, Baltimore, MD, 21218, USA<EMAIL_ADDRESS>
###### Abstract

Performing intricate eye microsurgery, such as retinal vein cannulation (RVC),
as a potential treatment for retinal vein occlusion (RVO), without the
assistance of a surgical robotic system is very challenging to do safely. The
main limitation has to do with the physiological hand tremor of surgeons.
Robot-assisted eye surgery technology may resolve the problems of hand tremors
and fatigue and improve the safety and precision of RVC. The Steady-Hand Eye
Robot (SHER) is an admittance-based robotic system that can filter out hand
tremors and enables ophthalmologists to manipulate a surgical instrument
inside the eye cooperatively. However, the admittance-based cooperative
control mode does not address crucial safety considerations, such as
minimizing contact force between the surgical instrument and the sclera
surface to prevent tissue damage. An adaptive sclera force control algorithm
was proposed to address this limitation using an FBG-based force-sensing tool
to measure and minimize the tool-sclera interaction force. Additionally,
features like haptic feedback or hand motion scaling, which can improve the
safety and precision of surgery, require a teleoperation control framework. A
single-manual adaptive teleoperation control mode was implemented on SHER 2.1
in previous work for the first time in robot-assisted retinal microsurgery. We
extended the previous work into a bimanual adaptive teleoperation (BMAT)
control mode using SHER 2.0 and SHER 2.1 and compared its performance with a
bimanual adaptive cooperative (BMAC) mode. Both BMAT and BMAC modes were
tested in sitting and standing postures during a vessel-following experiment
under a surgical microscope. It is shown, for the first time to the best of
our knowledge in robot-assisted retinal surgery, that integrating the adaptive
sclera force control algorithm with the bimanual teleoperation framework
enables surgeons to safely perform bimanual telemanipulation of the eye
without over-stretching it, even in the absence of registration between the
two robots. Experimental results demonstrate the effectiveness of the proposed
BMAT control framework.

##  I INTRODUCTION

Retinal vein occlusion (RVO) has been identified as the second most prevalent
potentially blinding retinal vascular disease, impacting an estimated 28
million individuals globally, according to a meta-analysis study [1]. This
condition stems from the occlusion of a retinal vein, leading to eye problems
including vision loss. Presently, there is no established standard for
directly treating RVO with a surgical approach. Retinal vein cannulation (RVC)
has been attempted in order to deliver a therapeutic agent directly into the
affected vein [2], however, its applicability is highly challenging due to the
micron scale of retinal veins and human hand tremor [3]. The problem of scale
is presently best addressed with the assistance of a surgical robotic system.
Of note, the diameter of the largest retinal veins is on the order of
$150\mu$m [4], and the root mean square (RMS) amplitude of hand tremor for an
ophthalmic surgeon is measured to be $182\mu$m [5].

Several surgical robotic systems have been developed to filter out
physiological hand tremors and improve the safety of retinal surgery. Some
examples of these systems include the Steady Hand Eye Robot (SHER) [6, 7],
Preceyes [8], to name a few [9, 10, 11].

Figure 1: Bimanual teleoperation architecture with adaptive force control
algorithm. The system consists of two Phantom Omni robots, the SHER 2.0 and
SHER 2.1, a surgical microscope, two FBG-based force-sensing surgical
instruments attached to the SHERs’ end-effector to measure the tip force and
sclera force, a monitor, and two foot pedals to activate and control the SHERs
impedance. The figure shows the high-level adaptive force controller, a mid-
level optimizer, and a low-level Galil joint velocity controller for SHER 2.0
and SHER 2.1.

The SHER 2.0 and SHER 2.1 are two robot versions developed at Johns Hopkins
University. The main control algorithm implemented on SHERs is an admittance-
based control where the user and the robot cooperatively manipulate a surgical
tool. Using a force/torque sensor attached to the robot handle, the exerted
force by the user is measured and converted to a desired end-effector velocity
to control the robot [7]. An important limitation of this control algorithm is
that the force/torque sensor is not sensitive enough to measure the minimal
interaction force existing on the surgical instrument at the sclera entry
(RCM), which could result in applying excessive force to the sclera distorting
and potentially causing damaging. To overcome this issue, Ebrahimi et al.
proposed an adaptive sclera force control algorithm [12] using an FBG-based
force-sensing tool to automatically minimize the sclera force and keep it
within a suggested safe range of 120 mN [13]. Notably, FBG sensors have
applications in flexible needles or continuum robots, serving purposes such as
shape sensing [14] or force sensing [15]. A comparison of the adaptive force
control algorithm and the admittance-based control algorithm performance was
conducted using a vessel-following experiment. An eye phantom was used where
the SHER 2.1 held the primary force-sensing surgical tool and cooperatively
manipulated the user’s dominant hand. Simultaneously, a conventional surgical
instrument was directly held by the non-dominant hand, serving as a secondary
tool for re-orienting the eye under a surgical microscope [12]. Despite
reducing the applied force to the sclera by the dominant hand compared to the
admittance-based control mode, controlling the force magnitude on the
secondary hand-held tool is impossible, and applying excessively unsafe force
to the sclera cannot be avoided. Also, this evaluation took place within a
single-manual cooperative control mode without teleoperation capability.
Integrating this adaptive force control algorithm in a teleoperation modality
has the potential to offer surgeons advanced capabilities, contributing to an
improvement in patient safety. Esfandiari et al. implemented this algorithm in
a teleoperation modality and compared its performance with a cooperative
control mode in a single-manual framework [16].

In this work, we extended the previous single-manual teleoperation framework
to a bimanual adaptive teleoperation control framework integrated with an
adaptive sclera force control algorithm on the SHER 2.0 and the SHER 2.1
systems using two force-sensing surgical instruments. This framework enables
surgeons to safely conduct bi-manual teleoperation on the eye, preventing any
stretching of the eye, even without registering the two robots. For the first
time in robot-assisted retinal microsurgery applications, we compared the
performance of the proposed bimanual adaptive teleoperation control mode and a
bimanual adaptive cooperative control mode integrated with the same adaptive
force control algorithm. This evaluation occurred during a vessel-following
experiment on an eye phantom under a surgical microscope for both sitting and
standing postures for each control mode. The objective was to assess and
compare the performance of the BMAT and BMAC control modes and also the effect
of user’s posture on their performance.

The two FBG-based force-sensing surgical instruments are calibrated using a
method outlined by [15]. This calibration enabled the estimation of the tip
force, sclera force, and insertion depth by measuring the wavelength shifts of
nine FBG sensors attached to each tool.

The remaining sections of this paper are organized as follows. Methods are
discussed in Section II. Section III describes the experimental setup and the
experiment procedure. The experimental results are analyzed and discussed in
Section IV. Finally, a conclusion is made in Section V.

Figure 2: Experiment setup includes the SHER 2.1 and the SHER 2.0, two PHANTOM
Omni robots, two ATI force sensors, two force-sensing tools (attached to the
SHERs’ handles) equipped with FBG sensors, two FBG interrogators, an eye
phantom with four colored vessels and four target points (pins) attached to
the end of each colored vessel, an armrest, and a surgical microscope.

##  II Methods

The objective of this experiment is to extend a previously developed adaptive
teleoperation control mode [16] to two eye robots, the SHER 2.0 and SHER 2.1,
and to assess the performance of the two control frameworks: bimanual adaptive
cooperative mode and bimanual adaptive teleoperation mode (see Fig. 1), in
both sitting and standing postures. In both control modes, the sclera force
magnitude is automatically controlled (without the user’s commands) by an
adaptive force control algorithm [12] to avoid stretching and damaging the
sclera. Following the suggestion of our clinical lead, we conducted a "vessel-
following" experiment, a prototypical task in retinal surgery [17]. The
experimental setup prepared to perform the vessel-following experiment
provides a measurement and recording of SHERs’ kinematic information, the
user’s hand force/torque during interaction with the SHERs’ handle, and the
force applied to the surgical tool at the sclera point and the tooltip.

###  II-A Adaptive Force Control of the SHER

In this control algorithm, if each of the measured sclera force components,
$F_{sx}$ and $F_{sy}$, surpasses the specified safety threshold, $T_{s}$, the
robot is directed along a safe sclera force trajectory [18]. By estimating the
stiffness of the sclera tissue, denoted by $\alpha_{i}$ $(i=x$ or $y)$, the
desired sclera force trajectories, $f_{dx}$ and $f_{dy}$, can be calculated
using Equations (1) and (2), respectively:

$\displaystyle f_{dx}=\frac{T_{s}sign(F_{sx})}{2}(e^{-(t-t_{x})}+1),$ (1)
$\displaystyle f_{dy}=\frac{T_{s}sign(F_{sy})}{2}(e^{-(t-t_{y})}+1),$ (2)

where $t_{i}$ represents the time at which $F_{si}$ surpasses the safe
threshold.

The desired linear velocities of the SHER’s end-effector along the $x$ and $y$
directions in the body frame $\\{B\\}$ can be determined by the adaptive force
controller using Equations (3) and (4):

$\displaystyle\dot{X}^{b}_{A_{i}}(t)=\alpha_{i}\dot{f}_{di}(t)-K_{fi}\Delta{f}_{i}(t),\quad
i=\\{x,y\\},$ (3)
$\displaystyle\dot{\alpha}_{i}=-\Gamma_{i}\dot{f}_{di}(t)\Delta{f}_{i}(t),\quad
i=\\{x,y\\}$ (4)

where $\Delta{f}_{i}=F_{si}-f_{di}$ denotes the sclera force error, and
$\Gamma_{i}$ and $K_{fi}$ represent constant gains for the adaptation law and
the force tracking error, respectively. $\alpha_{i}$ stands for the estimated
tissue compliance, and $\dot{f}_{di}$ is the derivative of $f_{di}$. This
adaptive sclera force control algorithm applies to both cooperative and
teleoperation control modalities. In the cooperative control mode, the
remaining components of the SHER end-effector velocity are generated by the
admittance-based controller, whereas in teleoperation control mode, they are
derived from the PHANTOM Omni’s end-effector velocity $\dot{X}_{o}^{b}$
(mapped to the SHER body coordinate $\\{B\\}$). The proposed teleoperation
control mode, integrated with an adaptive sclera force control algorithm,
consists of three primary components: a high-level controller, a mid-level
optimizer with joint constraints, and a low-level joint velocity controller
(Galil).

The high-level controller consists of two primary components: Firstly, there
is a kinematic controller that translates the end-effector velocity of the
Omni, denoted as $\dot{X}_{o}$, into the SHER’s body coordinate as
$\dot{X}_{o}^{b}$. It is then pre-multiplied by
$K=diag(\kappa_{1},\kappa_{2},\kappa_{3},\kappa_{4},\kappa_{5},\kappa_{6})$, a
user-defined motion scaling matrix, to provide an intuitive motion on the SHER
end-effector velocity, i.e., $\dot{X}_{des}^{b}=K\dot{X}_{o}^{b}$. Secondly,
an adaptive sclera force controller that receives input from the sclera force
components $F_{sx}$ and $F_{sy}$, which are measured in the body coordinate
$\\{B\\}$ by the force-sensing instrument. This controller generates the
desired adaptive end-effector velocity in the SHER body coordinate, denoted as
$\dot{X}_{A}^{b}$ in Equation (3).

Figure 3: Vessel-following experiment procedure for four combinations of
control modes and user posture including (a) standing BMAC, (b) standing BMAT,
(c) sitting BMAC, and (d) sitting BMAT. (e) The user activates each robot
using a foot pedal and follows a random order of colored vessels by the
dominant (right) hand and touches a target point at the end of each vessel by
the tooltip of the dominant hand and re-orients the eye phantom using the non-
dominant hand under a surgical microscope.

This hybrid kinematic-force control method determines the desired SHER
velocity by switching between $\dot{X}_{A}^{b}$ and $\dot{X}_{o}^{b}$ based on
the magnitude of the sclera force. When the scleral force components fall
below a safe threshold $T_{s}$, the SHER is controlled based on kinematic
teleoperation using the Omni velocity $\dot{X}_{o}^{b}$; otherwise, the
adaptive force controller takes charge, enforcing the SHER’s end-effector
velocity through $\dot{X}_{A}^{b}$ to minimize scleral force components. As a
result, the desired end-effector velocity of SHER, $\dot{X}_{des}^{b}$, is
generated based on the following hybrid kinematic-force control algorithm:

$\displaystyle\dot{X}_{des}^{b}=$
$\displaystyle\begin{bmatrix}(1-\Delta_{x})\kappa_{1}&0&\textbf{0}_{1\times
4}\\\ 0&(1-\Delta_{y})\kappa_{2}&\textbf{0}_{1\times 4}\\\ \textbf{0}_{4\times
2}&&diag(\kappa_{3},\kappa_{4},\kappa_{5},\kappa_{6})\end{bmatrix}\dot{X}_{o}^{b}$
$\displaystyle+$
$\displaystyle\begin{bmatrix}diag(\Delta_{x},\Delta_{y})&\textbf{0}_{2\times
4}\\\ \textbf{0}_{4\times 2}&\textbf{0}_{4\times
4}\end{bmatrix}\begin{bmatrix}\dot{X}_{A_{x}}^{b}\\\ \dot{X}_{A_{y}}^{b}\\\
\textbf{0}_{4\times 1}\end{bmatrix},$ (5)

in which $\Delta_{i}$ ($i=x$ or $y$) is a binary variable automatically set by
the control switching policy (Algorithm 1) based on the magnitude of sclera
force components $F_{si}$ ($i=x$ or $y$). $\Delta_{i}=1$ means the adaptive
force control algorithm for the corresponding sclera force component ($x$ or
$y$) is activated. Otherwise, when $\Delta_{i}=0$, the adaptive force control
algorithm gets deactivated, and Equation (5) becomes equivalent to the
kinematic control $\dot{X}_{des}^{b}=K\dot{X}_{o}^{b}$.

Algorithm 1 Hybrid Kinematic-Force Control Switching Policy

Input: Sclera force components $F_{si}$, $i=x,y$.

Output: $\Delta_{i}$ and $\dot{X}_{A_{i}}^{b}$, $i=x,y$.

$\Delta_{i}\leftarrow 0$;
if  _$F_{si}\geq T_{s}$_ then

$t_{i}\leftarrow t$; $\triangleright$ Save current time $t$
$\Delta_{i}\leftarrow 1$; $\triangleright$ Activate adaptive force control
while  _ $|F_{si}|>0.75T_{s}$_ do

$\Delta_{i}\leftarrow 1$; $\triangleright$ Keep adaptive force control
activated
$f_{di}\leftarrow$ (1), (2); $\triangleright$ Calculate desired force
$\dot{X}_{A_{i}}^{b}\leftarrow$ (3), (4); $\triangleright$ Calculate adaptive
velocities

end while

$\Delta_{i}\leftarrow 0$; $\triangleright$ Switch to kinematic control for
axis $i$

end if

The mid-level optimizer computes the optimal desired joint velocities, denoted
as $\dot{\Theta}_{des}$, for the SHER while accounting for its joint limit
constraints. Finally, the low-level Galil joint velocity controller generates
the necessary control commands to reach the desired joint velocities. Other
signals include $\dot{\Theta}_{SHER}$ (actual joint velocities of the SHER),
$\dot{X}_{SHER}^{s}$ (actual end-effector velocity in the spatial coordinate
of the SHER), and $F_{t}$ (tooltip force) (see Fig. 1).

##  III Experiments

###  III-A Experimental Setup

The setup components comprised the SHER 2.1 on the right holding the primary
force-sensing tool, the SHER 2.0 on the left holding the secondary force-
sensing tool, two robot controllers (Galil motion controllers, CA, USA), two
PHANTOM Omni robots (SensAble Technologies Inc., MA, USA), two ATI force
sensors (Nano17, ATI Industrial Automation, Apex, NC, USA), two force-sensing
tools with three optical fiber channels attached to each tool shaft and a
total of nine FBG sensors (Technica Optical Components, China) on the three
channels, an FBG interrogator (HYPERION si155, Luna) for the primary force-
sensing tool and another FBG interrogator (sm130, Micron Optics, Inc.) for the
secondary fore-sensing tool, an eye phantom, an armrest, and a surgical
microscope (Leica Microsystems) (see Fig.2). All these devices were
interconnected to two main computers for the two robots through a TCP-IP
connection.

Two 6-DoF ATI Nano 17 force sensors are integrated into the SHERs’ end-
effector to measure the user hand’s force and torque exerted on the robot
handle and to control the robot in the cooperative mode with an admittance-
based control algorithm [7]. For measuring the sclera force applied to the eye
phantom by the two surgical instruments, two FBG force-sensing tools are
employed. The tool shaft features three FBG fibers arranged at 120-degree
intervals, housing a total of nine FBG sensors distributed along the three
fibers [19] (see Fig. 2). Optical signals from these FBG sensors are received
by the two FBG interrogators, which then convert them into force readings at
the sclera point and the tooltip. The force read at the sclera point has two
components, $F_{sx}$ and $F_{sy}$, perpendicular to the tool shaft and are
used in the adaptive sclera force control algorithm to always maintain the
sclera force level below a safe threshold. The two PHANTOM Omnis serve as a
joystick controller for the bimanual teleoperation of SHER 2.0 and SHER 2.1.
The eye phantom, designed as an anatomical eye model replicating critical
features of a natural human eye, is utilized in this experiment.

###  III-B Experimental Procedure

To assess the performance of the two bimanual control modes, BMAC and BMAT,
the user is asked to conduct a vessel-following experiment in sitting and
standing postures for each control mode (see Fig. 3a-d). The task involves
tracking a colored trajectory on the retinal surface of an eye phantom that
simulates retinal veins. The colored trajectory comprises four different
colors: red, green, blue, and yellow. A pin is attached at the end of each
colored vessel to specify a target point and the user is asked to touch the
pin with the tooltip. This requires pinpoint accuracy in tool use by the user.
The user starts the experiment by approaching the tip of the primary tool,
manipulated by the dominant hand, to the start point at the intersection of
the four colored vessels, tracking the four colored vessels in a random order,
and touching the target pins at the end of each colored vessel. Once all
colors are tracked, the user returns to the intersection point (see Fig. 3e).
This procedure is considered one trial. More specific details of the procedure
are outlined in [17]. The experiment includes 5 sets of tests collected on
five different days. For each test, 10 trials are repeated following a random
order of colored vessels, resulting in 50 trials for each mode, namely,
standing BMAC, sitting BMAC, standing BMAT, and sitting BMAT. To optimize
visualization, all trials are conducted under a surgical microscope with an
illuminator.

In the bimanual cooperative control mode, the user holds the tool handle, and
the robots are manipulated in a direct cooperation manner (Figs. 3a, 3c). In
contrast, there is an indirect interaction between the user and the SHERs
through the PHANTOM Omni interfaces in the bimanual teleoperation mode. Two
foot-pedals separately activate each robot. The more the foot pedal is pushed,
the less stiffness is observed in the SHER handle and the surgical instrument.
During the sitting BMAC and sitting BMAT, the user’s hand is stabilized by an
armrest to further reduce the hand tremor (see Figs. 3c, 3d), whereas, this
armrest is lacking in the standing postures. Regardless of the control mode,
the user uses the dominant hand to manipulate the primary force-sensing tool
and the non-dominant hand to manipulate a secondary tool for re-orientation of
the eye phantom to reach optimum visualization under the microscope. It’s
noteworthy that bimanual manipulation is a standard method of performing
retinal surgery. All kinematic and force information measured from the robots,
the ATI force sensors, and the FBG-based force-sensing tools for each vessel-
following trial are recorded in separate .csv files and subsequently analyzed
using MATLAB. The statistical analyses were performed using a paired t-test,
MATLAB’s ttest2() function, assuming unequal variance (Welch’s t-test).
Statistical significance is inferred when p$<$0.05.

(a)

(b)

Figure 4: An example of the performance of the adaptive force control
algorithm on limiting the sclera force components $F_{sx}$ and $F_{sy}$ and
sclera force norm $F_{s}$ for the dominant hand (a) and the non-dominant hand
(b) during sitting bimanual adaptive teleoperation control mode. Upon reaching
the 100 mN threshold, the adaptive sclera force control algorithm is
triggered, initiating efforts to minimize the sclera force components, namely
$F_{sx}$ and $F_{sy}$, along a desired exponentially decaying force trajectory
(dashed curves). The objective of the adaptive force controller is to keep the
norm of sclera force $F_{s}$ below a safe threshold of 120 mN [13]. The four
peaks in the tip force for the dominant hand show that the four target points
(pins) at the end of the colored vessels are touched by the tooltip.

(a)

(b)

Figure 5: The mean of the tool-sclera interaction forces was calculated over
50 trials for the sitting BMAT mode for (a) dominant hand and (b) non-dominant
hand.  TABLE I: Mean and maximum sclera force, handle force/torque, mean
completion time, and the percentage of time the sclera force is over 120 mN.
The results are reported for the BMAC and the BMAT control modes, for both
sitting and standing postures and for both the dominant hand (right robot,
SHER 2.1) and non-dominant hand (left robot, SHER 2.0), during the vessel-
following experiment. The values inside the parenthesis are the standard
deviations.  TABLE II: The p-values for standing BMAC vs. sitting BMAC, for
mean and maximum sclera force, mean completion time, and the percentage of
time during which the sclera force is over 120 mN, for both the dominant hand
(DH) and the non-dominant hand (NDH).  Standing BMAC vs. Sitting BMAC
---
p-value |  Mean sclera force |  Max sclera force |  Mean completion time |  $\%$ of time force over 120 mN
DH |  0.0259 |  5.594e-05 |  0.0084 |  9.722e-05
NDH |  1.585e-4 |  5.777e-9 |  0.0084 |  8.679e-05
TABLE III: The p-values for standing BMAT vs. sitting BMAT.  Standing BMAT vs.
Sitting BMAT
---
p-value |  Mean sclera force |  Max sclera force |  Mean completion time |  $\%$ of time force over 120 mN
DH |  3.788e-10 |  1.779e-12 |  6.213e-18 |  0.0068
NDH |  2.579e-08 |  1.948e-15 |  6.213e-18 |  0.0016
TABLE IV: The p-values for sitting BMAC vs. sitting BMAT. Sitting BMAC vs.
Sitting BMAT
---
p-value |  Mean sclera force |  Max sclera force |  Mean completion time |  $\%$ of time force over 120 mN
DH |  0.0012 |  1.564e-08 |  0.0063 |  0.0040
NDH |  0.0187 |  0.1621 |  0.0063 |  0.1732

##  IV Experimental Results and Discussion

To the best of our knowledge, the implementation of a bimanual adaptive
teleoperation control mode integrated with an adaptive sclera force control
algorithm on a microsurgical robotic system used for retinal microsurgery
applications, in specific on the SHER 2.0 and SHER 2.1, is unprecedented. The
performance of the proposed BMAT mode is compared with a BMAC mode in sitting
and standing postures for the user by considering several important metrics
related to patient safety and surgeons’ comfort. For example, some factors,
having to do with patient safety, including mean and maximum sclera force and
the percentage of aggregate completion times in which the recorded sclera
force exceeded the safety threshold of 120 mN are reported for both dominant
and non-dominant hand for all four control modes. Other factors related to
surgeons’ intuition and comfort are also reported such as human-robot
interaction force/torque and task completion time (Table I). The completion
time is also of great importance which determines the general operational time
and cost. Of note, these factors are analyzed and reported for a single expert
user and may not be extrapolated to non-expert users who are not at their
learning curve plateau [20].

The user, in both sitting and standing postures, is supposed to follow a
random order of colored vessel trajectory while approaching the tooltip as
close as possible to the bottom of the eye phantom without touching it and
guide the tooltip toward a target pin attached to the end of each vessel and
slightly touch it by the tooltip.

###  IV-A Sclera Force Control Performance

The adaptive sclera force control algorithm dynamically adjusts the SHER 2.0
and SHER 2.1 end-effector velocity to maintain the sclera force below a safe
threshold of 120 mN. This feature enables the surgeon to safely manipulate the
eye without applying unsafe force to the sclera. Using this technique, it is
possible to safely perform bimanual robot manipulation of the eye without
registering the two robots, yet avoid stretching the eye during bimanual
manipulation. Figure 4 describes an example of the functionality of the sclera
force control algorithm during sitting BMAT control mode for both dominant and
non-dominant hands. It is observed that once the sclera force $F_{s}$ reaches
the 100 mN threshold, the adaptive sclera force control algorithm gets
activated and tries to minimize the sclera force components, $F_{sx}$ and
$F_{sy}$, along a desired exponentially decaying force trajectory. The
proposed safe limit for sclera force is 120 mN [13]. The 100 mN threshold is
selected for activating the adaptive force control to provide a 20 mN offset
before the safe limit. The presence of four peaks in the tip force data for
the dominant hand (Fig. 4a) indicates that the tooltip made contact with the
four target points (pins) located at the ends of the colored vessels. Also, an
example of the mean sclera force for all 50 trials for the sitting BMAT
control mode is provided in Fig. 5, which indicates that the user showed a
consistent performance in this mode for both dominant and non-dominant hands.
Similar performance is observed for the other three control/posture modes.

Here, a comparison is made between the standing and sitting postures for each
of the two BMAC and BMAT control modes, and also between the sitting BMAC and
the sitting BMAT (p-values are reported in Tables II – IV) to compare the
performance of BMAC and BMAT and to examine the effects of user’s posture on
their performance.

###  IV-B Standing BMAC vs. Sitting BMAC

For example, by comparing the standing BMAC and sitting BMAC, it is observed
that the user applied a significantly smaller maximum force of 185.14
$\pm$21.93 by his dominant hand during the sitting BMAC compared to this value
in the standing BMAC which is 245.57 $\pm$23.07 (p $\ll$ 0.05). This
difference is also significant for the non-dominant hand. The percentage of
aggregate time when the sclera force is above 120 mN for the dominant hand is
0.45 $\pm$ 0.07 for the sitting BMAC and 2.14 $\pm 0.06$ for the standing BMAC
(p $\ll$ 0.05). Task completion time is also significantly reduced from 30.81
$\pm$5.35 for the standing BMAC to 23.21 $\pm$14.8 for the sitting BMAC (p =
0.0084) (Tables I, II).

###  IV-C Standing BMAT vs. Sitting BMAT

Similarly, by comparing the sitting BMAT and standing BMAT, it is observed
that the user demonstrated better performance sitting. For example, the mean
sclera force for the non-dominant hand during sitting BMAT is 44.84 $\pm$5.39,
whereas this value for the standing BMAT is 62.77 $\pm$22.26 (p$\ll$0.05), or
the maximum sclera force for the dominant hand in the standing BMAT is 290.49
$\pm$38.89 while this value is reduced to 189.92 $\pm$17.64 for the sitting
BMAT (p$\ll$0.05). Similar improvement is also noticed for the mean completion
time and the percentage of aggregate time when sclera force is above 120 mN
(see Tables I, III). Based on these findings, we conclude that the user
outperforms in the sitting posture because the arms are stabilized using an
armrest, reducing hand tremors and fatigue. Furthermore, it is difficult to
handle both foot pedals in standing mode as the user is supposed to hold their
weight on the heel and pivot to push the pedals. Sometimes the user did not
handle the pedals efficiently, especially the left one which is related to the
non-dominant hand. This resulted in higher stiffness of the robots and,
therefore, higher maximum sclera force. Although, the sitting posture allows a
superior performance that makes it a preferred option for robot-assisted
retinal microsurgery, however, the standing posture also allows acceptable and
safe performance, which could potentially be used for other surgeries during
which the surgeons may decide to partly stand during the operation such as
microvascular Anastomosis [21].

###  IV-D Sitting BMAC vs. Sitting BMAT

Finally, by comparing the results of the BMAT and the BMAC modes in the
sitting posture (Tables I, IV), it is observed that bimanual adaptive
teleoperation showed comparable or better performance to the bimanual adaptive
cooperative mode. For example, the mean sclera force for the dominant hand for
the sitting BMAT is 32.04 $\pm$4.84, which is less than the value for the
sitting BMAC (35.79 $\pm$5.84, p = 0.0012). Also, the percentage of aggregate
time with a scleral force higher than 120 mN is less for the sitting BMAT
(0.01 $\pm$7.93e-4) as compared to the sitting BMAC (0.45 $\pm$0.07, p =
0.004). This demonstrates lower scleral forces with a sitting BMAT. This could
be, in part, due to the slower end-effector velocity during the teleoperation
mode compared to the cooperative mode, which in turn results in a longer task
completion time for the sitting BMAT mode (26.53 $\pm$ 6.47 s) compared to the
sitting BMAC mode (23.21 $\pm$ 14.8 s, p = 0.0063). The maximum sclera force
for both dominant and non-dominant hands for the sitting BMAT is higher than
those of the sitting BMAC. For example, for the dominant hand, the value for
the sitting BMAC is 185.14 $\pm$ 21.93 mN while this value for the sitting
BMAT is slightly higher, 189.92 $\pm$ 17.64 mN. Based on our understanding,
this has to do with the repositioning capability of the teleoperation mode. In
the cooperative mode, the user can finish the task by holding the pedals
during the entire vessel-following trial, whereas, during the teleoperation
mode, the motion scaling capability necessitates several repeated unclutching,
repositioning of the Omni handles, and clutching to follow a trajectory. This
results in a higher stiffness at the SHERs’ handle and the surgical instrument
at the beginning of each clutching, which resulted in a higher maximum sclera
force for the BMAT mode; however, it is still on the same level of magnitude
as the BMAC mode.

Therefore, based on the observed results, the adaptive bimanual teleoperation
control mode demonstrated the desired performance during the vessel-following
experiment and could be a promising technology with new capabilities to
improve surgeons’ comfort, dexterity, and patients’ safety. The effectiveness
of the proposed BMAT control mode should be further examined using other types
of biological models, such as chicken embryo or pig eye, in addition to this
phantom eye model. Moreover, adding haptic force feedback to the teleoperation
mode provides the surgeons with a better understanding of the sclera force
level and may improve the safety of surgery.

##  V Conclusion

This work performed a vessel-following experiment on a bimanual adaptive
teleoperation framework with an adaptive sclera force control algorithm on
SHER 2.0 and SHER 2.1 using two force-sensing tools. This was a first in
robot-assisted retinal surgery applications. The tools were equipped with FBG
sensors capable of measuring the force at the tool sclerotomy and the tooltip.
The performance of this bimanual teleoperation mode was compared with a
bimanual cooperative mode utilizing a single expert user by doing a vessel-
following experiment inside an eye phantom for two sitting and standing
postures. Both surgical tools, including the secondary tool used for rotating
the eyeball, are force-sensing tools that provide sclera force measurements
for the adaptive force control algorithm to adjust scleral forces
automatically to maintain them below a safe threshold. Using this force
control technique, it is possible to perform bimanual robotic surgery on the
eye without registering the two robots.

Also, a comparison between the bimanual teleoperation and the bimanual
cooperative modes in the sitting posture demonstrates that the bimanual
teleoperation mode shows comparable performance to the bimanual cooperative
mode and can potentially improve the safety and precision of surgery by
providing surgeons with additional capabilities, including motion scaling and
repositioning.

In the future, we intend to add haptic feedback to reflect the sclera force on
the user’s hand and conduct a multi-user study to further compare the BMAT
control mode equipped with haptic feedback and the BMAC control mode.

## References

  * [1] P. Song, Y. Xu, M. Zha, Y. Zhang, and I. Rudan, “Global epidemiology of retinal vein occlusion: a systematic review and meta-analysis of prevalence, incidence, and risk factors,” _Journal of global health_ , vol. 9, no. 1, 2019.
  * [2] K. Willekens, A. Gijbels, L. Schoevaerdts, L. Esteveny, T. Janssens, B. Jonckx, J. H. Feyen, C. Meers, D. Reynaerts, E. Vander Poorten _et al._ , “Robot-assisted retinal vein cannulation in an in vivo porcine retinal vein occlusion model,” _Acta ophthalmologica_ , vol. 95, no. 3, pp. 270–275, 2017.
  * [3] S. Singh and C. Riviere, “Physiological tremor amplitude during retinal microsurgery,” in _Proceedings of the IEEE 28th Annual Northeast Bioengineering Conference (IEEE Cat. No. 02CH37342)_. IEEE, 2002, pp. 171–172.
  * [4] D. Goldenberg, J. Shahar, A. Loewenstein, and M. Goldstein, “Diameters of retinal blood vessels in a healthy cohort as measured by spectral domain optical coherence tomography,” _Retina_ , vol. 33, no. 9, pp. 1888–1894, 2013.
  * [5] C. N. Riviere and P. S. Jensen, “A study of instrument motion in retinal microsurgery,” in _Proceedings of the 22nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Cat. No. 00CH37143)_ , vol. 1. IEEE, 2000, pp. 59–60.
  * [6] B. Mitchell, J. Koo, I. Iordachita, P. Kazanzides, A. Kapoor, J. Handa, G. Hager, and R. Taylor, “Development and application of a new steady-hand manipulator for retinal surgery,” in _Proceedings 2007 IEEE International Conference on Robotics and Automation_. IEEE, 2007, pp. 623–629.
  * [7] A. Üneri, M. A. Balicki, J. Handa, P. Gehlbach, R. H. Taylor, and I. Iordachita, “New steady-hand eye robot with micro-force sensing for vitreoretinal surgery,” in _2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics_. IEEE, 2010, pp. 814–819.
  * [8] T. Meenink, G. Naus, M. de Smet, M. Beelen, and M. Steinbuch, “Robot assistance for micrometer precision in vitreoretinal surgery,” _Investigative ophthalmology & visual science_, vol. 54, no. 15, pp. 5808–5808, 2013.
  * [9] Y. Ida, N. Sugita, T. Ueta, Y. Tamaki, K. Tanimoto, and M. Mitsuishi, “Microsurgical robotic system for vitreoretinal surgery,” _International journal of computer assisted radiology and surgery_ , vol. 7, pp. 27–34, 2012.
  * [10] M. A. Nasseri, M. Eder, S. Nair, E. Dean, M. Maier, D. Zapp, C. P. Lohmann, and A. Knoll, “The introduction of a new robot for assistance in ophthalmic surgery,” in _2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2013, pp. 5682–5685.
  * [11] E. Vander Poorten, C. N. Riviere, J. J. Abbott, C. Bergeles, M. A. Nasseri, J. U. Kang, R. Sznitman, K. Faridpooya, and I. Iordachita, “Robotic retinal surgery,” in _Handbook of robotic and image-guided surgery_. Elsevier, 2020, pp. 627–672.
  * [12] A. Ebrahimi, M. G. Urias, N. Patel, R. H. Taylor, P. Gehlbach, and I. Iordachita, “Adaptive control improves sclera force safety in robot-assisted eye surgery: A clinical study,” _IEEE Transactions on Biomedical Engineering_ , vol. 68, no. 11, pp. 3356–3365, 2021.
  * [13] A. Ebrahimi, C. He, M. Roizenblatt, N. Patel, S. Sefati, P. Gehlbach, and I. Iordachita, “Real-time sclera force feedback for enabling safe robot-assisted vitreoretinal surgery,” in _2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2018, pp. 3650–3655.
  * [14] G. Amirkhani, A. Goodridge, M. Esfandiari, H. Phalen, J. H. Ma, I. Iordachita, and M. Armand, “Design and fabrication of a fiber bragg grating shape sensor for shape reconstruction of a continuum manipulator,” _IEEE Sensors Journal_ , 2023.
  * [15] X. He, M. Balicki, P. Gehlbach, J. Handa, R. Taylor, and I. Iordachita, “A multi-function force sensing instrument for variable admittance robot control in retinal microsurgery,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014, pp. 1411–1418.
  * [16] M. Esfandiari, J. W. Kim, B. Zhao, G. Amirkhani, M. Hadi, P. Gehlbach, R. H. Taylor, and I. Iordachita, “Cooperative vs. teleoperation control of the steady hand eye robot with adaptive sclera force control: A comparative study,” _arXiv preprint arXiv:2312.01631_ , 2023.
  * [17] C. He, N. Patel, A. Ebrahimi, M. Kobilarov, and I. Iordachita, “Preliminary study of an rnn-based active interventional robotic system (airs) in retinal microsurgery,” _International journal of computer assisted radiology and surgery_ , vol. 14, pp. 945–954, 2019.
  * [18] N. Patel, M. Urias, A. Ebrahimi, R. H. Taylor, P. Gehlbach, and I. Iordachita, “Force-based control for safe robot-assisted retinal interventions: In vivo evaluation in animal studies,” _IEEE Transactions on Medical Robotics and Bionics_ , vol. 4, no. 3, pp. 578–587, 2022.
  * [19] C. He, E. Yang, and I. Iordachita, “Dual-stiffness force-sensing cannulation tool for retinal microsurgery,” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2019, pp. 3212–3216.
  * [20] B. Zhao, M. Esfandiari, D. E. Usevitch, P. Gehlbach, and I. Iordachita, “Human-robot interaction in retinal surgery: A comparative study of serial and parallel cooperative robots,” _arXiv preprint arXiv:2304.00213_ , 2023.
  * [21] C. M. Harris, A. O. Mitchell, and R. M. Laughlin, “Microvascular principles,” _Atlas of Operative Oral and Maxillofacial Surgery_ , pp. 895–900, 2022.


# AdsorbRL: Deep Multi-Objective Reinforcement Learning for Inverse Catalysts
Design

Romain Lacombe
Stanford University &Lucas Hendren
Stanford University &Khalid El-Awady
Stanford University &{rlacombe, hendren<EMAIL_ADDRESS>
###### Abstract

A central challenge of the clean energy transition is the development of
catalysts for low-emissions technologies. Recent advances in Machine Learning
for quantum chemistry drastically accelerate the computation of catalytic
activity descriptors such as adsorption energies. Here we introduce _AdsorbRL_
, a Deep Reinforcement Learning agent aiming to identify potential catalysts
given a multi-objective binding energy target, trained using offline learning
on the Open Catalyst 2020 and Materials Project data sets. We experiment with
Deep Q-Network agents to traverse the space of all $\sim$160,000 possible
unary, binary and ternary compounds of 55 chemical elements, with very sparse
rewards based on adsorption energy known for only between 2,000 and 3,000
catalysts per adsorbate. To constrain the actions space, we introduce Random
Edge Traversal and train a single-objective DQN agent on the known states
subgraph, which we find strengthens target binding energy by an average of 4.1
eV. We extend this approach to multi-objective, goal-conditioned learning, and
train a DQN agent to identify materials with the highest (respectively lowest)
adsorption energies for multiple simultaneous target adsorbates. We experiment
with Objective Sub-Sampling, a novel training scheme aimed at encouraging
exploration in the multi-objective setup, and demonstrate simultaneous
adsorption energy improvement across all target adsorbates, by an average of
0.8 eV. Overall, our results suggest strong potential for Deep Reinforcement
Learning applied to the inverse catalysts design problem.

##  1 Introduction: Challenges of Catalysts Design

A central challenge of the clean energy transition is the development of high-
performance catalysts for electrochemical and thermocatalytic energy
conversion [1] [2]. The problem of catalysts design [3], which seeks to
identify high-performance materials with increased intrinsic activity for any
desired reaction of interest, is a critical technology enabler for low-
emissions technologies, from H_2 production from water and renewable energy,
to the transformation of waste CO_2 into valuable non-fossil fuels and feed-
stock [4]. Rapid progress in Machine Learning (ML) methods for computational
quantum chemistry [5], along with recently available datasets of catalysts [6]
and their properties [7], could significantly accelerate the identification of
materials with enhanced catalytic activity.

A key descriptor of catalytic activity is adsorption energy, or the energy
with which the reagent species or reaction intermediate (‘adsorbate’), a small
molecule on a surface site (e.g. H2O$\star$), binds to the surface of the
heterogeneous catalyst (here an inorganic compound nano-particle). A core
tenet of heterogeneous catalysis science, the Sabatier Principle [8], holds
that the optimal catalyst should have a binding energy with the reactants that
is neither too weak nor too strong.

Identifying materials that best match a target energy profile for multiple
adsorbates, for instance strong binding with $\star$OH but weaker adsorption
of H2O$\star$, is thus particularly helpful for catalysts design [9].
Computational approaches to catalyst design have traditionally leveraged
advances in computational chemistry to screen large chemical spaces for
materials with optimal adsorption energies for key intermediates [10]. Despite
advances in Density Functional Theory (DFT) [5] computation, estimating
adsorption energy for a single (catalyst, adsorbate) pair still requires heavy
computational resources, which makes _in silico_ high-throughput screening of
catalysts costly [11].

Inverse design adopts the opposite approach: starting from the desired
property, the task aims to design catalysts from first principles so they best
fit that objective [12]. While ML techniques for inverse materials design show
strong promise, synthesizability and physical realization of discovered
materials is a challenge [13] compared to high-throughput screening of known
materials.

We turn to Reinforcement Learning (RL) [14] to propose a third way: training
an agent to traverse a space of materials, not by exhaustive search, but by
gravitating towards optima for the target property.

We introduce _AdsorbRL_ , a Deep Reinforcement Learning (DRL) [15] agent
trained to traverse a space of materials and identify promising catalysts
given a multi-objective binding energy target. Specifically, we use Offline RL
[16] on the Materials Project [7] and Open Catalyst 2020 [6] datasets of
adsorption energies to train a DRL agent to identify catalysts which bind the
strongest (lowest adsorption energy) or the weakest (highest adsorption
energy) with an array of target adsorbates, chosen for their importance for
the clean energy transition.

We hypothesize that RL can be especially helpful to navigate chemical space in
the multi-objective setting. By learning to traverse a sparse rewards
environment in a multi-objective goal-conditioned setup, our agent could help
identify materials with desirable properties for the multi-objective target at
hand, and serve as an in silico rapid screening mechanism to identify leads on
which to further focus computational chemistry resources (DFT computations, MD
simulations, etc.).

This paper presents our experiments with several Deep RL setups to traverse a
large space of compounds, and identify materials with the desired adsorption
energies profile for a set of adsorbates of interest. We introduce Multi-
Objective DQN with Sub-Sampling, and a novel algorithm to train such a
generalized multi-objective agent.

Our findings indicate the promise of Deep RL in navigating complex chemical
spaces, and present novel approaches to tackle goal-conditioned multi-
objective Reinforcement Learning for materials design. These methods could
serve as a foundation for more complex computational challenges in large
chemical spaces, and the development of novel materials for a wide range of
applications in heterogeneous catalysis, electrochemical energy conversion,
and low-emissions technologies.

Figure 1: Deep RL approach to the catalyst design problem. Images: Open
Catalyst Project.

##  2 Related Work

###  2.1 Machine Learning for Adsorption Energy Predictions

The Open Catalyst Project [6] is a research effort between Meta Fundamental AI
Research (FAIR) and Carnegie Mellon University using Artificial Intelligence
to model and discover new catalysts for renewable energy conversion and
storage. Quantum mechanical computational methods such as Density Functional
Theory [5] can help evaluate new catalysts, but are severely limited by the
very high computational cost of the higher precision levels of theory.

For that purpose, Open Catalyst Project has developed and released two public
datasets, _Open Catalyst 2020 (OC20)_ [6] and _Open Catalyst 2022 (OC22)_
[17], to train ML models to efficiently approximate these calculations. These
datasets together contain 1.3 million molecular relaxations with results from
over 260 million DFT calculations. Importantly, Lan et al. [11] recently
released _AdsorbML_ , an hybrid ML and computational DFT model trained to
predict the adsorption energy of an adsorbate onto the sorbent surface.
Inference greatly accelerates computation and achieves final energies within
$\sim$0.1 eV of ground truth.

###  2.2 Machine Learning for Catalysts Design

Rapid progress in Machine Learning has lead to increased interest in applying
data-driven learning techniques to the identification of catalysts. Zitnick et
al. [3] introduce the general problem of catalysts design, and provide an
overview of Machine Learning approaches to the problem to motivate the
introduction of the _Open Catalyst_ dataset. Seh et al. [1] provide an
overview of how chemists in the laboratory can combine experiment and theory
to design catalysts in order to increase their intrinsic activity or their
number of active sites.

Freeze et al. [12] review recent advances in inverse design for catalysts,
including approaches based on Machine Learning, as well as more general
optimization (gradient ascent, genetic algorithms), but do not point to
Reinforcement Learning. Zhu et al. [18] and Noh et al. [13] provide a
comprehensive review of the published literature on Machine Learning for
electrocatalysts and inorganic solid materials, respectively, including
approaches using _OC20_ data. While many different techniques have been
applied to this problem space, none based on Reinforcement Learning are cited.

###  2.3 Reinforcement Learning for Molecular and Materials Design

In the more general area of inverse materials design, a number of Deep RL-
based approaches have reported success. Zhou, et al. [19] apply the Deep
Q-Network (DQN) algorithm [20] to molecular optimization. Their work
demonstrates the effectiveness of a multi-objective reinforcement learning
network for organic materials design, via a step-by-step addition or removal
of elements or bonds.

Sridharan, et al. [21] explore the application of Deep RL to molecules
generation by training a model to reconstruct 2D molecular graph based Nuclear
Magnetic Resonance (NMR) spectroscopy. They designed an innovative framework
using Monte-Carlo-Tree-Search (MCTS) and Graph Convolutional Networks to
reconstruct the most likely molecular structure based on the chemical
information extracted from NMR spectroscopy output. They demonstrated that
applying their model in a RL setting helped consistently identify exact
molecular structures based only on spectrograms.

Pan, et al. [22] apply Deep Q-Networks [20] to inverse inorganic materials
design. This paper demonstrates the use of Deep RL for the identification of
new materials based on desired properties, and was especially novel in its
approach of inorganic materials design modeling, and how to integrate chemical
constraints through Lagrange multipliers to impose charge neutrality and
electro-negativity balance. This approach demonstrated an RL agent navigating
trajectories in state space via successive additions of elements to an
inorganic compound for the inverse design task.

Finally Sui, et al. [23] demonstrate the effectiveness of Deep RL for general
digital materials design. They specifically show how RL improves on
traditional approaches to find new design patterns even in vast design spaces,
and show an application to additive manufacturing via the selection of soft or
stiff voxels.

To our knowledge, Deep RL techniques have not been tried on sorbents and
electrocatalyst design to date, which is what we propose in this paper.

##  3 Dataset and Features

Our dataset is sourced from the Materials Project, an open-access database for
discovery of inorganic materials, which provides pre-computed properties for a
large number of candidate catalysts [7]. Specifically, we use the Catalysis
Explorer, an online application that provides pre-computed adsorption energies
for various catalysts in the Materials Project under different configurations,
sourced from the Open Catalyst Project OC20 [6] dataset. We explore catalysts
associated with the following 6 adsorbates, which are of major significance
for clean energy:

  * •

$\star$OH2: adsorbed water, a key reagent for the Oxygen Evolution Reaction
(OER) in H2 generation from water electrolysis [24], and a key product of the
Oxygen Reduction Reaction (ORR) reaction for H2 fuel cells [25];

  * •

$\star$OH: adsorbed hydroxyl radical, a key intermediate for OER [24] and ORR
[25], often a rate-limiting step requiring the application of electrochemical
over-potentials as its adsorption energy tends to scale linearly with those of
$\star$O and $\star$OOH [9];

  * •

$\star$CH4: adsorbed methane, of importance for direct air capture of natural
gas, and for the CO2 Reduction Reaction (CO2RR) for e.g. methanation of
captured carbon dioxide [26];

  * •

$\star$CH2: adsorbed methylene radical, another important intermediate for CO2
Reduction Reaction (CO2RR) for the ethylene electro-catalystic production
pathway, a key building block of the modern chemicals industry [4];

  * •

$\star$N2: adsorbed molecular nitrogen, a key reagent for the Nitrogen
Reduction Reaction (NRR) [27] for ammonia production, an essential ingredient
of synthetic fertilizers without which half the world population would not be
fed [28];

  * •

$\star$NH3: adsorbed ammonia, the desired product of Nitrogen reduction (NRR)
[27].

For each adsorbate compound, the Materials Project Catalysis Explorer provides
a set of properties including the compound, bulk formula, adsorption energy,
and Miller indices describing a particular lattice structure and orientation.
For tractability purposes, and reasoning that the lowest energy configuration
may be most representative of overall activity, we filter for all unique 1, 2,
and 3-element compounds and select the lowest adsorption energy configuration
for the target adsorbate among all combinations of stoichiometry and Miller
indices, ignoring lattice orientation, and coordination environment.

This simplifies the materials space exploration problem to traversal of the
space of all $\sim$160,000 possible unary, binary and ternary compounds of 55
chemical elements (Figure 4). Adsorption energies are known for only between
2,000 and 3,000 catalysts per adsorbate, providing a Given the very sparse
reward for our agent, with only 7,386 total unique catalysts in our dataset of
$\sim$160,000 possible compounds with known adsorption energy for at least one
adsorbate (Table 4). We use the $\star\ce{OH2}$ adsorbate for our single
objective experiments, and the 6 target adsorbates above for the multi-
objective setup. Finally, in experiments (3), (4) and (5), we further
constrain this problem to the _OC20_ -subgraph of compounds with known
adsorption energy for at least one target adsorbate.

Our dataset is available for download on the AdsorbRL GitHub repository.

##  4 Model: Reinforcement Learning Setting

Our Reinforcement Learning model comprises of the following elements,
illustrated in Figure 1.

States: $S$. States are unary, binary or ternary compounds of 55 elements
matching to catalysts from the Materials Project dataset, comprising of 1 to 3
atomic elements forming the compound (e.g. $\ce{SiC}$: Silicon carbide).
Catalysts are represented by a 55 dimensional one-hot vector in our full
generality setup (experiments (1), (3), (4), and (5)). In experiment (2),
states are simply a one-element square on the Periodic Table of Elements grid,
represented by their atomic number $Z$.

Actions: $a$. These are steps the agent can take to traverse the dataset of
materials. They are defined differently for each experimental setup:

  * •

Experiment (1): in the full states/full actions setup (section 5.1), actions
can be: addition of an element (subject to the total elements in the catalyst
being less than or equal to 3); removal of an element (subject to there being
at least one element in the catalyst); ‘do-nothing’, an action that leaves the
catalyst as is; and a terminate action that ends the episode.

  * •

Experiment (2): on the Periodic Table setup (section 5.2), only 5 actions are
allowed: move left by one element, move right by one element, move up by one
element, move down by one element, or stay in place. Trajectories
automatically terminate after 9 steps.

  * •

Experiment (3)—(5): in the full states/constrained actions setup (section 5.3
and 5.4), there are five possible actions: add a random element (‘random
edge’), remove the first, second, or third element in the catalyst, or
terminate the episode.

Reward: $r\propto f\left(E_{ads}(S_{i})\right)$. The reward for arriving in a
state is a function of its adsorption energy ($E_{ads}$) for the target
adsorbate. In single-objective experiments, we use $r=-E_{ads}$ and
$r=E_{ads}^{2}$ to target terminal states with the strongest binding possible
(measured by a large, negative energy value). For multi-objective setup we use
$r=-E_{ads}^{3}$ to encourage the agent to find extrema with strongest
(respectively, weakest) adsorption. Rewards functions for each experiment are
reported in Table 1.

Termination: we either offer termination as an action option to the agent, or
upon completion of a number of steps (e.g 9 steps).

Policy: $\pi(a|S)$. We train a policy to choose the next action at each state,
using the Q-Learning [29] and Deep Q-Network algorithms [20] with the
following Bellman equation [30]:

$\mathrm{Q}^{*}(a|S)=r(a|S)+\gamma\max_{a}\left(\mathrm{Q}^{*}(a|S^{\prime})\right)$

Multi-objective goal-conditioning. In this setup, we train our policy to
follow multiple objectives at once. We define an objective vector $\mathbf{g}$
as an array of +1 or -1 for each of the 6 adsorbates in our dataset, encoding
whether we seek to bind stronger (+1, minimize energy) or weaker (-1, maximize
energy). The reward becomes a function of $E_{ads}$ for each adsorbate and the
objective vector $\mathbf{g}$:

$\pi(a|S)=\pi(a|S,\mathbf{g})$
$\mathbf{g}=(g_{i})=(\dots,\\{-1,+1\\}_{i},\dots)$
$r(S,\mathbf{g})=f\left(E_{ads}(S_{t+1},\mathrm{ads}_{i})\right),\mathbf{g})$

The Bellman equation [30] in the goal-conditioned, multi-objective setup
becomes:

$\mathrm{Q}^{*}(a|S,\mathbf{g})=r(a|S,\mathbf{g})+\gamma\max_{a}\left(\mathrm{Q}^{*}(a|S^{\prime},\mathbf{g})\right)$

Evaluation metrics. To evaluate our trained policies, we run a number of roll-
outs from random initial states and compute the average adsorption energy of
all final states reached by the policy. The delta with the average energy of
initial states (respectively its inverse for weak binding targets in the
multi-objective setup) measures how well our agent solves the problem:

$\Delta=\frac{1}{N}\sum_{i\in\textrm{final}}-\left(E_{ads}(S_{i})\right)-\frac{1}{N}\sum_{j\in\textrm{initial}}-\left(E_{ads}(S_{j})\right)$

##  5 Experiments & Results

We present the setup for each of the five experiments we report in Table 1.

Exp. | Target adsorbates | States | Actions | Algorithm | Reward
---|---|---|---|---|---
(1) | Single ($\star$OH2) | All states | Graph traversal | DQN | $-E_{ads}$
(2) | Single ($\star$OH2) | Periodic table | $\leftarrow|\rightarrow|\uparrow|\downarrow$ | Q-Learning | $E_{ads}^{2}$
(3) | Single ($\star$OH2) |  _OC20_ -subgraph | Random edges | DQN | $-E_{ads}^{3}$
(4) | Multi-objective |  _OC20_ -subgraph | Random edges | DQN | $-E_{ads}\cdot\mathbf{g}$
(5) | Multi-objective |  _OC20_ -subgraph | Random edges | DQN with Sub-Sampling | $-E_{ads_{i}}g_{i}\quad i\sim\mathcal{U}(i=1\dots 6)$

Table 1: Summary of experiments (1) to (5).

###  5.1 Offline RL on Full State & Actions Space

#### Experiment (1).

Our first experiment attempts to train an agent to traverse the full compounds
space from a random initial catalyst to the lowest energy catalyst using
reward shaping [31]. Our offline data represents the set of all possible
tuples, $(s,a,r,s^{\prime})$ where $s$ and $s^{\prime}$ are existing catalysts
in the dataset (‘valid’ states) and $s^{\prime}$ is reachable from $s$ via a
valid action, $a$ (resulting in approximately 81,000 offline training tuples).
The reward for a valid state is its inverse adsorption energy ($r=-E_{ads}$).
We deem states with unknown adsorption energy ‘invalid’ and assign them a
penalty ($-\lambda$) reward. We use a DQN model [20] with 2 hidden layers of
sizes 512 and 64 and hyperparameters similar to those used later in sections
5.3 and 5.4 and train our network for 100,000 training steps.

We find that this approach does not yield a useful agent, and fails to
noticeably converge towards desired states (Figure 3). A high penalty is
needed to coax the agent to stay away from invalid states: adsorption energies
of valid states are in the range of $(0,-10)$ eV and we use a penalty
$\lambda\in(10,200)$. We find the agent mostly learns to move quickly to an
invalid state and terminate with an overall reward of $-\lambda$, and avoids
longer episodes that would accumulate multiple penalties.

This can be understood by noting the sparsity of the state space. Referring to
Figure 4, we see that the cardinality of the ‘valid’ set is around 2,379
catalysts, while it is $\simeq$160,000 for the whole space (all possible
3-element combos using one of 55 elements). This implies that over 98% of
possible states are invalid and we hypothesize that the sparsity of the data
makes it too hard to learn to navigate to the optimal state. Figure 3 reports
the low success rate for varying values of $\lambda$.

###  5.2 Simplified State & Actions Space: Periodic Table of Elements

#### Experiment (2).

To test our hypothesis about the sparse reward issues faced in experiment (1),
we aim to simplify our RL setup as much as possible to test whether agents
trained in a more tractable setup do exhibit lower average terminal state
energies. We implement the simplest environment for an agent to learn
fundamental chemical knowledge: _GridWorld_ [29] on the Periodic Table of
Elements [32]. This setup is novel to our knowledge and can serve as a
building block to learn more complex cheminformatics problems.

We define our Markov Decision Process (MDP) as follows:

  * •

86 single element states (all elements from atomic number $Z=1$ (H) to $Z=86$
(Rn);

  * •

5 actions: $\\{\\_|\leftarrow|\rightarrow|\downarrow|\uparrow\\}$ to move
between elements on the periodic table (see figure 4);

  * •

Episodes last for a set 9 steps duration (long enough to reach the optimal
element from any random starting point on the table).

We find that the Q-learning algorithm [29] on single elements with a reward
defined as $E^{2}_{ads}$ consistently reaches the lowest energy states. Table
5 in Appendix reports final states for 20 random roll-outs: the agent reaches
a -7.4 eV average terminal energy vs -1.5 eV for starting elements – see
figure (4) – and terminates on top-2 states for $\sim~{}95\%$ of roll-outs
starting from random states. We report 20 random roll-outs from a policy
learned on this environment in Table 5 in appendix.

###  5.3 Simplified Actions Space: Random Edge Traversal

#### Experiment (3): Single Objective DQN with Random Edge Traversal on _OC20_
-Subgraph.

In light of the results from experiment (2), we hypothesize that the periodic
table environment works better than the original Section 5.1 setup for two
reasons: (i) learning a limited set of actions is more tractable compared to
the full 55 possible actions in experiment (1), and (ii) valid states are too
sparse, and a negative reward on those invalid states leads to shorter
episodes which never reach valid states. In other words, we now traverse the
subgraph of _OC20_ -subgraph materials with known adsorption energy for our
target adsorbate.

To that end, we reduce the original action space (from Section 5.1) from 60
actions to 5 actions, and introduce the following modified setup: an action is
only taken if it leads to a ‘valid’ state. If it is invalid, the episode does
not end and no reward is given; the state just remains unchanged. Finally, we
introduce Random Edge Traversal: instead of separately enumerating each
element that could be added or removed, we add a random element if the agent
chooses to expand the chemical compound.

This method of constraining the action space is, to our knowledge, novel in
the literature, and introduces a hybrid between bandits (pull a lever to add a
random element) and reinforcement learning (the agent can backtrack, removing
elements previously added). Formally, the Markov Decision Process (MDP) is
defined as follows:

  * •

A state is still a 55-dimensional 1-hot vector representing the catalyst
chemical composition (up to three non-zero elements at any given time).

  * •

5 possible actions: <add> a random element (that hasn’t already been added to
the catalyst), remove the <first>, <second>, or <third> element in the
catalyst (if they exist), or <stop> the episode.

  * •

Rewards are returned when the agent terminates the episode or if we hit
maximum episode length (between 20 and 75 steps). The reward function is
$-E_{ads}^{3}$ to magnify the rewards towards the strongest (respectively
weakest) binding elements.

  * •

Transitions are guided by the actions as in Section 5.1, with the caveat that
the state remains unchanged change when an invalid action is selected. Initial
state is chosen at random, and discount factor is 0.9.

Our agent is a Double-DQN with two hidden layers with layer sizes 512 and 64.
To avoid exploding Q-values explode when the target update period is too low,
we use an update period of 300 steps, along with a learning rate of $10^{-3}$,
and we implement an epsilon-greedy policy with $\epsilon=0.1$.

Experimental results reveal that trajectory roll-outs tend to be much longer
than in Section 5.1. Common trajectory lengths range from 10-40 steps, while
trajectories between 40 and 75 steps (the maximum length) are slightly less
common. We report 9 random roll-outs from a policy learned on this environment
in Table 6 in appendix. We find that final states improve target adsorption
energy compared to random initial states by an average of 4.1 eV.

We report average terminal state energies over 50 roll-outs for single-
objective agents trained in experiments (1), (2) and (3) in table 2.

Experiments | Initial state | Exp (1) | Exp (2) | Exp (3)
---|---|---|---|---
Avg. $E_{ads}$ (eV) | -1.5 | -2.2 | -7.4 | -5.6
$\Delta$ (eV) | - | 0.7 | 5.9 | 4.1
Table 2: Experiments (1), (2) & (3) (single-objective). Average energies over
50 single-objective roll-outs. Objective: minimize adsorption energy (higher
$\Delta$ is better).

###  5.4 Multi-Element Multi-Objective DQN on _OC20_ -Subgraph

#### Experiment (4): Multi-Objective DQN.

In practical catalysis experiments, a wide variety of potential adsorbates and
possible reaction intermediates are present in the environment. Selectivity to
a given reaction product is a major challenge [10] [4], and as a result, ideal
catalyst design agents should learn to identify catalysts with stronger
binding energy with certain adsorbates, and weaker adsorption with others.
This is particularly important to try and break the ‘scaling relations’ such
as between $\star$O, $\star$OH, and $\star$OOH in ORR [24].

We extend our experimental setup to a multi-objective reinforcement learning
problem, where each adsorbate is a separate objective. In this setup, we train
an agent to find catalysts that minimize (respectively maximizes) adsorption
energies for each objective adsorbate. We traverse the _OC20_ -subgraph of
materials with known adsorption energy to at least one target adsorbate.

We first implement a standard approach whenever we have multiple objectives:
summing weighted rewards from each objective. In this scenario, the reward
from each objective is the energy, and the weights are either -1 or +1, based
on whether we seek to minimize or maximize the energy for that adsorbate. We
train a DQN (same hyperparameters as previously), with the MDP defined as
follows:

  * •

A state is the same 55-dimensional 1-hot vector representing the catalyst
compound.

  * •

5 possible actions: <add> a random element, remove the <first>, <second>, or
<third> element in the catalyst (if they exist), or <stop> the episode.

  * •

Rewards are returned when we either hit the maximum episode length, which is
now set to 20. The reward is the dot product of goal vector and adsorption
energies (we don’t use negative cube rewards since average adsorption energies
matter for the maximization cases):

$r=-E_{ads}\cdot\mathbf{g}$

#### Experiment (5): Multi-Objective DQN with Sub-Sampling.

We hypothesize that as the number of objectives increases in the previous
setup, there’s a higher chance that the agent gets stuck in local minima,
where a step in the environment may help optimize for one or two adsorbates
but hurt the rest, which might discourage exploration.

Considering that policies learned on a random mixture of objectives might
encourage exploration, we introduce Multi-Objective DQN with Sub-Sampling, a
new method by which we randomly sample one objective among the six for each
training roll-out, :

$r=-E_{ads_{i}}\times g_{i}\quad i_{\mathrm{roll-out}}\sim\mathcal{U}(i=1\dots
6)$

This roll-out-level objective sampling approach is, to our knowledge, a novel
contribution to literature. We evaluate this approach in our experimental
setup, by comparing sub-sampling pairs of objectives to a baseline from the
previous experimental setup where all objectives are used in the reward
computation for all roll-outs.

We report average final state energies over 50 roll-outs for experiments (4)
and (5) in table 3 (stable over several runs). Final states found with sub-
sampling are a better fit with their respective objectives for adsorbates 2,
3, and 4, but slightly worse for 1, 5, and 6. We notice that convergence with
sub-sampled objective rewards takes longer than the baseline, as one may
expect. Experimental results (reported in Table 3 and Figure (5) in appendix)
support the hypothesis that sub-sampling objectives helps encourage
exploration, with longer average trajectories than baseline.

Overall, we find that, in the difficult multi-objective setting, both baseline
and sub-sampling approaches improve final state adsorption energy towards the
desired direction (increase or decrease), simultaneously across all 6
adsorbates, by an average of 0.8 eV.

Adsorbate |  1: *CH2 |  2: *CH4 |  3: *N2 |  4: *NH3 |  5: *OH2 |  6: *OH
---|---|---|---|---|---|---
Objective |  Increase |  Increase |  Increase |  Decrease |  Decrease |  Decrease
Initial state |  -2.2 |  -3.3 |  -1.8 |  -1.6 |  -1.9 |  -1.9
Exp (4): Baseline |  -2.2 |  -3.0 |  -1.5 |  -1.9 |  -3.9 |  -3.9
Exp (5): Sub-Sampling |  -2.3 |  -3.0 |  -1.6 |  -2.1 |  -3.8 |  -3.8
Table 3: Experiment (4) and (5) (multi-objective). Average energies over 50
multi-objective roll-outs for the three experimental setups (rows) on each of
the 6 objectives (columns).

##  6 Analysis

###  6.1 Challenges of Sparse Known States Graph Traversal

A particular challenging aspect of or our setup is the very sparse nature of
the problem. The limited number of states for which adsorption energies are
known ($\sim 2,000$–$3000$) compared to the larger number of possible states
($55+55\times 54+55\times 54\times 53=160,345$), makes learning to converge to
low energy states difficult for our agent in the initial full state/actions
setup.

We found success after drastically simplifying our setup with the Periodic
Table _GridWorld_ environment, which reinforced the intuition that limiting
the number of actions is paramount to obtaining helpful results in a sprase
rewards setup. This encouraged us to explore Random Edge Traversal of the
_OC20_ -subgraph to significantly limit the number of actions a model must
learn, and our results show this may prove a helpful general principle to
traverse complex material spaces of sparsely documented properties.

###  6.2 Generalizing Inverse Catalyst Design with RL

Our results raise the question of whether the performance we report warrants
the cost and complexity of training Deep RL models, where more standard
optimization techniques would be more straightforward for offline learning on
known energy datasets, even in the multi-objective setting.

Using Deep RL to solve this class of problems despite its inherent complexity
enables us to address problems linear solvers cannot. While these experiments
use offline learning on a dataset where adsorption energies are known, we
envision using ML-based adsorption energy estimators as a critic in an actor-
critic RL setup, and use exploitation of states with known energy to direct
computational resources where exploring new unknown states is likely to be
most beneficial.

Another enticing approach enabled by RL would be to include additional sparse
signals to our reward model. For example, hard to model physical properties
whenever they are known (stability, selectivity), materials cost, patents, or
even human feedback (RLHF [33]), where candidate catalysts are ranked by
scientists based on their experience (e.g. manufacturing cost, experimental
complexity, industry preferences, etc.). Such a complex reward function or
multiple set of criteria would be hard to encode in a tractable way for a
direct optimization setup to solve, but would be a good fit for multi-
objective Deep RL approaches.

##  7 Conclusion

This paper presents our experiments with various Deep RL setups. We introduce
Multi-Objective DQN with Sub-Sampling and Random Edge Traversal, a novel
method to train a generalized multi-objective agent to traverse a large space
of possible catalysts with sparse known properties, and identify materials
with the desired adsorption energies profile for a set of target adsorbates of
interest.

We demonstrated that in the goal-conditioned multi-objective setting, Deep RL
can identify promising materials for any combination of target adsorbates
binding energies. In practice, conducting a large number of roll-outs and
identifying the most common terminal states would point to promising materials
on which to focus computational and experimental resources.

Our findings indicate the promise of Deep RL in navigating complex chemical
spaces, and present novel approaches to tackle goal-conditioned multi-
objective Reinforcement Learning for materials design. These methods could
serve as a foundation for more complex computational challenges in large
chemical spaces, and the development of novel materials for a wide range of
applications in heterogeneous catalysis, electrochemical energy conversion,
and low-emissions technologies.

### Known Limitations and Future Work

First, we report results on a single combination of 6 adsorbates, and a single
objective vector in the multi-objective setup. Further experimentations on a
larger set of adsorbates objective vectors would help results robustness,
especially for the novel objective sub-sampling approach we introduce.

Second, we report goal-conditioning only on extrema, and train agents seeking
to maximize or minimize adsorption energy. Other approaches to train our agent
to seek states with intermediate adsorption energy, such as scalar
conditioning (defining a target value for $E_{ads}$), would be particularly
helpful to find optimal catalysts, which usually have intermediate binding
strength on the activity-binding energy ‘volcano plots’ (Sabatier Principle
[10]).

Relabeling targets is a potent way to improve goal-conditioned agents, and
future work could focus on improving the performance of our multi-objective
agent through Hindsight Experience Replay [34], as well as Prioritized
Experience Replay (PER) [35]. Other offline algorithms such as Conservative
Q-Learning [36] may also prove helpful.

Lastly, our overall objective was to train an RL agent to traverse a vast
space of possible materials with multiple target adsorbates. While we used
offline RL on known adsorption energies datasets to facilitate
experimentation, we envision an actor-critic setup where an actor is trained
using our graph traversal setup, and a critic uses ML-based adsorption energy
evaluation models. An ML-based, lightweight critic such as _AdsorbML_ [11]
could perform approximate but fast energy evaluations for unknown states
traversed at roll-out time, and more exact DFT computations could be reserved
for frequent final states revealed by accumulated roll-outs as strong catalyst
candidates.

### Code & Data Access

The code implementation for our experiments, as well our datasets compiled
from the Materials Project [7] and Open Catalyst 2020 [6] datasets, are made
available for download for reproducibility purposes on the AdsorbRL GitHub
repository.

### Acknowledgements

The authors wish to thank Prof. Chelsea Finn, Dr. Karol Hausman, and Jonathan
Yang at Stanford University for guidance on this project, as well as Ajay
Kannan for his contribution to Random Edge Traversal and the multi-element,
multi-objective setup. We are grateful to Meta AI and the chemical engineers,
materials scientists and AI researchers who collaborated on the Open Catalyst
Project and Materials Project, and everyone whose research informed these data
sets. This work would not have been possible without them.

## References

  * [1] Zhi Wei Seh, Jakob Kibsgaard, Colin F. Dickens, Ib Chorkendorff, Jens K. Nørskov, and Thomas F. Jaramillo.  Combining theory and experiment in electrocatalysis: Insights into materials design.  Science, 355(6321):eaad4998, 2017.
  * [2] Steven Chu and Arun Majumdar.  Opportunities and challenges for a sustainable energy future.  Nature, 488(7411):294–303, 2012.
  * [3] C. L. Zitnick, L. Chanussot1, A. Das, S. Goyal, J. Heras-Domingo, C. Ho, W. Hu, T. Lavril, A. Palizhati, M. Rivière, M. Shuaibi, A. Sriram, K. Tran, B. Wood, J. Yoon, D. Parikh, and Z. Ulissi.  An introduction to electrocatalyst design using machine learning for renewable energy storage, 2020.  arXiv:2010.09435v1.
  * [4] Phil De Luna, Christopher Hahn, Drew Higgins, Shaffiq A Jaffer, Thomas F Jaramillo, and Edward H Sargent.  What would it take for renewably powered electrosynthesis to displace petrochemical processes?  Science, 364(6438):eaav3506, 2019.
  * [5] S. Kurth, M.A.L. Marques, and E.K.U. Gross.  Density-functional theory.  Encyclopedia of Condensed Matter Physics 2005, pages 395–402, 2005.
  * [6] Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi.  Open catalyst 2020 (oc20) dataset and community challenges.  ACS Catalysis, 11(10):6059–6072, May 2021.
  * [7] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson.  Commentary: The Materials Project: A materials genome approach to accelerating materials innovation.  APL Materials, 1(1):011002, 07 2013.
  * [8] Paul Sabatier.  The method of direct hydrogenation by catalysis.  Nobel Lecture, 1912.
  * [9] Javier Pérez-Ramírez and Núria López.  Strategies to break linear scaling relationships.  Nature Catalysis, 2:971–976, 2019.
  * [10] Jens K Nørskov, Thomas Bligaard, Jan Rossmeisl, and C. H. Christensen.  Towards the computational design of solid catalysts.  Nature Chemistry, 1:37–46, 2009.
  * [11] J. Lan, A. Palizhati, M. Shuaibi, B.M. Wood, B. Wander, A. Das, M. Uyttendaele, C. L. Zitnick, and Z. W. Ulissi.  Adsorbml: Accelerating adsorption energy calculations with machine learning, 2023.  arXiv:2211.16486.
  * [12] Jessica G. Freeze, H. Ray Kelly, and Victor S. Batista.  Search for catalysts by inverse design: Artificial intelligence, mountain climbers, and alchemists.  Chemical Reviews, 119(11):6595–6612, 2019.
  * [13] Juhwan Noh, Geun Ho Gu, Sungwon Kim, and Yousung Jung.  Machine-enabled inverse design of inorganic solid materials: promises and challenges.  Chemical Science, 11:4871–4881, 2020.  Minireview.
  * [14] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore.  Reinforcement learning: A survey.  Journal of artificial intelligence research, 4:237–285, 1996.
  * [15] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath.  Deep reinforcement learning: A brief survey.  IEEE Signal Processing Magazine, 34(6):26–38, 2017.
  * [16] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  arXiv preprint arXiv:2005.01643, 2020.
  * [17] Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M. Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, Anuroop Sriram, Félix Therrien, Jehad Abed, Oleksandr Voznyy, Edward H. Sargent, Zachary Ulissi, and C. Lawrence Zitnick.  The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts.  ACS Catalysis, 13(5):3066–3084, February 2023.
  * [18] S. Zhu, K. Jiang, B. Chen, and S. Zheng.  Data-driven design of electrocatalysts: principle, progress, and perspective.  J. Mater. Chem. A, 11:3849, 2023.
  * [19] Z. Zhou, S. Kearnes, and L. Li et al.  Optimization of molecules via deep reinforcement learning.  Sci Rep, 9:10752, 2019.
  * [20] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * [21] Sridharan B, Mehta S, Pathak Y, and Priyakumar UD.  Spectra to structure: Deep reinforcement learning for molecular inverse problem, 2021.  ChemRxiv. Cambridge: Cambridge Open Engage; 2021. doi:10.26434/chemrxiv-2021-4hc7k.
  * [22] Elton Pan, Christopher Karpovich, and Elsa Olivetti.  Deep reinforcement learning for inverse inorganic materials design, 2022.
  * [23] F. Sui, R. Guo, Z. Zhang, G. Gu, and L. Lin.  Deep reinforcement learning for digital materials design.  ACS Materials Lett., 3:1433–1439, 2021.
  * [24] Isabela C Man, Hai-Yan Su, Federico Calle-Vallejo, Heine A Hansen, José I Martínez, Nilay G Inoglu, John Kitchin, Thomas F Jaramillo, Jens K Nørskov, and Jan Rossmeisl.  Universality in oxygen evolution electrocatalysis on oxide surfaces.  ChemCatChem, 3(7):1159–1165, 2011.
  * [25] Minhua Shao, Qiaowan Chang, Jean-Pol Dodelet, and Regis Chenitz.  Recent advances in electrocatalysts for oxygen reduction reaction.  Chemical reviews, 116(6):3594–3657, 2016.
  * [26] Stephanie Nitopi, Erlend Bertheussen, Soren B Scott, Xinyan Liu, Albert K Engstfeld, Sebastian Horch, Brian Seger, Ifan EL Stephens, Karen Chan, Christopher Hahn, et al.  Progress and perspectives of electrochemical co2 reduction on copper in aqueous electrolyte.  Chemical reviews, 119(12):7610–7672, 2019.
  * [27] Aayush R Singh, Brian A Rohr, Jay A Schwalbe, Matteo Cargnello, Karen Chan, Thomas F Jaramillo, Ib Chorkendorff, and Jens K Nørskov.  Electrochemical ammonia synthesis: The selectivity challenge, 2017.
  * [28] Jan Willem Erisman, Mark A Sutton, James Galloway, Zbigniew Klimont, and Wilfried Winiwarter.  How a century of ammonia synthesis changed the world.  Nature Geoscience, 1(10):636–639, 2008.
  * [29] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [30] Richard Bellman.  The theory of dynamic programming.  Bulletin of the American Mathematical Society, 60(6):503–515, 1954.
  * [31] Y. Wu, G. Tucker, and O. Nachum.  Behavior regularized offline reinforcement learning, 2019.  arXiv:1911.11361v1.
  * [32] IUPAC.  Periodic table of elements.
  * [33] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862, 2022.
  * [34] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  Advances in neural information processing systems, 30, 2017.
  * [35] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  arXiv preprint arXiv:1511.05952, 2015.
  * [36] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  Advances in Neural Information Processing Systems, 33:1179–1191, 2020.

Adsorbate |  | Dataset size
---
(# of known catalysts)
$\star\ce{OH2}$ | 2,379
$\star\ce{CH2}$ | 2,759
$\star\ce{CH4}$ | 2,409
$\star\ce{N2}$ | 2,111
$\star\ce{NH3}$ | 2,473
$\star\ce{OH}$ | 2,655
All adsorbates | 7,386 (unique)
Table 4: Sparse rewards: number of catalysts for which adsorption energy is known for the corresponding adsorbates, out of $\sim$160,000 possible compounds. Does not sum up due to duplicates. Figure 2: 3-hop ego graph of the lowest energy state for $\ce{*OH_{2}}$ adsorbate (SiC). Red: SiC. Green: 1-hop ego network. Dark blue: 2-hope ego network. Light blue: 3-hop ego network. Figure 3: Experiment (1). Success rate in reaching the optimal valid state using reward shaping for each value of $\lambda$. The graph illustrates the difficulty in training a DQN agent that converges in our dataset with sparse rewards. Figure 4: Experiment (2). Left: Periodic table of elements, traversed in a GridWorld setting (stay | left | right | up | down). Elements present in _OC20_ are highlighted in blue. Right: average final energy after evaluation roll-outs while Q-learning the policy (NB: plot represents $-E_{ads}$, final adsorption energy is negative). Final state | Trajectory length | Energy (eV)
---|---|---
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Iron | 9 | -9.1
Carbon | 9 | -7.9
Iron | 9 | -9.1
Carbon | 9 | -7.9
Iron | 9 | -9.1
Manganese | 9 | -0.9
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Iron | 9 | -9.1
Iron | 9 | -9.1
Carbon | 9 | -7.9
Carbon | 9 | -7.9
Table 5: Experiment (2). Twenty random roll-outs from a policy learned on this environment. Lower energy is better. Average random single-element reward is -1.5eV. Final state | Trajectory length | Energy (eV)
---|---|---
Carbon | 24 | -7.9
Iron | 40 | -9.1
Sulfur and Vanadium | 5 | -6.3
Iron | 10 | -9.1
Iron | 24 | -9.1
Cesium and Hydrogen | 1 | -0.8
Tantalum and Vanadium | 9 | -2.1
Iron | 24 | -9.1
Iron | 18 | -9.1
Table 6: Experiment (3). Nine random roll-outs from a policy learned on this
environment. Lower energy is better. Average random single-element reward is
-1.5eV. Figure 5: Experiments (4) & (5): multi-objective DQN on OC20-Subgraph.
Number of episode steps at roll-out, full objective vs objective sub-sampling.