# Hybrid Magnonic Reservoir Computing

Cliff B. Abbott  Dmytro A. Bozhko  Department of Physics and Energy Science,
University of Colorado Colorado Springs, Colorado Springs, 80918 CO, USA

###### Abstract

Abstract: Magnonic systems have been a major area of research interest due to
their potential benefits in speed and lower power consumption compared to
traditional computing. One particular area that they may be of advantage is as
Physical Reservoir Computers in machine learning models. In this work, we
build on an established design for using an Auto-Oscillation Ring as a
reservoir computer by introducing a simple neural network midstream and
introduce an additional design using a spin wave guide with a scattering
regime for processing data with different types of inputs. We simulate these
designs on the new micro magnetic simulation software, Magnum.np, and show
that the designs are capable of performing on various real world data sets
comparably or better than traditional dense neural networks.

††preprint: APS/123-QED

##  I Introduction

Reservoir computing (RC) is a machine learning concept where a lower
dimensional input is passed to a system which maps to a higher dimensional
output via a complex, non-linear, but dynamically consistent process. Unlike
other machine learning models, the training process is regulated to the
output, or readout, layer(s) of the RC allowing for significant reductions in
training time Lukoševičius and Jaeger (2009). However, the implementation of a
RC can still be quite complicated and computationally expensive when done with
simulated neurons, like in the Recurrent Neural Network approach. Because of
this, Physical Reservoir Computers (PRC), where the complex, nonlinear
dynamics are done by exploiting natural systems dynamics rather than
simulated, have become a popular area of research Nakajima (2020). Of the many
physical systems available, magnonics is a promising candidate for next
generation computers due to the potentially fast computational speed, low
energy consumption compared to other electronics, and small physical size of
the systems Pirro  _et al._ (2021); Rana  _et al._ (2019); Rana and Otani
(2019); Chumak, Serga, and Hillebrands (2017); Chumak (2019); Sadovnikov,
Beginin, and Morozova (2016); Albisetti  _et al._ (2018); Mahmoud  _et al._
(2021); Heussner  _et al._ (2017). While much of the current work has been
focused on the implementation of spinwave based logic gates Schneider  _et
al._ (2008); Ustinov  _et al._ (2019); Chen  _et al._ (2021); Kostylev  _et
al._ (2005); Yu, Lan, and Xiao (2020); Khitun, Bao, and Wang (2008, 2010);
Zografos  _et al._ (2014), there have recently been several proposed designs
for magnonic RCs Nakane, Tanaka, and Hirose (2018); Körber  _et al._ (2023);
Nikitin  _et al._ (2022); Kondrashov  _et al._ (2022).

In this work, we build on the magnonic auto-oscillation ring (AOR) proposed in
Watt  _et al._ (2021) and model through numerical simulation Watt and Kostylev
(2024) by Watt et al. by adding a dense neural network midstream with the new
micromagnetic simulation package Magnum.np Bruckner  _et al._ (2023). Also, we
explore differences between encoding input information into either the
amplitude or phase of excited spin waves. Previous work has primarily been
focused on proving the design against theoretical benchmarks such as input
memory and parity memory. Here we use historical S&P stock market data to test
the design on real world applications.

In addition to the AOR, we introduce a design for a spin-wave-based system for
feature mixing we call a Parallel Input Scattering Model (PSM). The PSM
encodes features into two separate spin waves that are simultaneously sent
through a scattering region before being read out. We tested various
capabilities of this model on the Iris and Statlog datasets from the UCI
Machine Learning Repository Fisher (1988); Hofmann (1994) and on a 3rd custom
dataset.

The rest of the paper is structured as follows. Section II: Methods covers (A)
the setup of the simulation software, followed by (B) the design and operation
of the AOR as well as the data used and it’s preprocessing, and the final
subsection (C) covers the same material for the PSM. Section III: Results
reports the various testing results for (A) the AOR and (B) the PSM. Finally,
Section IV: Conclusions covers the insights from the simulations and
suggestions for future work.

##  II Methods

###  II.1 Simulation

For simulating the designs in this paper, we used the Magnum.np Python
package. This package is built on the PyTorch backend allowing python native
simulations. It also allows for seamless integration of Torch machine learning
models, back propagation methods, and built in GPU optimization methods. For
all simulations, a grid size $2.5\,\mathrm{nm^{3}}$, saturation magnetization
value of $140\,\mathrm{kA/m}$, exchange constant of $3.5\,\mathrm{pJ/m}$, and
Gilbert damping constant of $0.0002$ were used to approximate a Yttrium Iron
Garnet thin film. Performance criteria for all simulations is the percentage
accuracy at identifying the correct answer. The simulations were done on an
Nvidia DGX Workstation utilizing four A100 8GB GPUs.

###  II.2 Auto Oscillation Ring

Figure 1

Figure 1:  Layout of the AOR design used. The data is encoded into the input
signal which is then used to excite magnetostatic surface waves which
propagate down the wave guide. The output antenna converts the spin wave back
into an electronic signal which is then sent to 3 different channels: the
readout, the feedback line, and the amplification neural network. The feedback
line adds the output signal to the incoming signal at the input antenna. The
amplification neural network takes the output from the previous interval and
uses it to determine some additional amplification to the next intervals input
signal.

shows the layout of the AOR. A grid size of $1250\times 50\times
2.5\,\mathrm{nm}$ was used for this simulation. A $200\,\mathrm{mT}$ external
bias field was initialized in plane and perpendicular to the direction of
propagation for the waveguide and magnetostatic surface waves were excited at
the input antenna at a frequency of $14\,\mathrm{GHz}$ for $0.3$ nanoseconds
corresponding to 4 periods per input signal. The output is read per $1/100th$
of a nanosecond into the amplification neural network (ANN) such that the
output $a$ of the network with inputs $x$ and at input interval $i$ is
$a_{i}=(x_{i-1,0},x_{i-1,1},...,x_{i-1,30})$. The ANN is a 2 layer multi-layer
perceptron (MLP) with a first layer of 10 nodes and ReLU activation, followed
by a single node layer and sigmoid activation. The sigmoid activation was used
to ensure that the output values were between 0 and 1 so as to prevent the ANN
from blowing up the input signal amplitude to unrealistic values. For
analysis, the input signal before amplification by the ANN, the signal from
the output antenna, and the difference between the two are used.
Watt et al. Watt  _et al._ (2021); Watt and Kostylev (2024) used a test
measure of input memory (recover the input value j inputs ago) and a parity
memory (the binary sum of the previous j inputs). Here we used the closing
values for 250 consecutive trading days of the US S&P Stock Index taken from
Yahoo! Finance. The data was pre-processed into a percentage move from the
previous day’s closing value and re-scaled to a maximum and minimum of 1 and 0
respectively (Fig 2a). The target value was 1 if the next day was a positive
move or 0 if it was negative. The inputs $I$ were then encoded into input
signal $S$ as either a modulation of the amplitude $S=I\times Sin(2\pi ft)$ or
of the phase $S=Sin(2\pi ft+\pi I)$. The output of the simulation was
additionally processed in one of three ways:
$\bm{1.}$ A direct linear mapping

$Weights=TrainOutput^{-1}\cdot TrainTargets$

$Prediction=Weights\cdot TestOutput$

$\bm{2.}$ Using an ensemble of linear maps approach

$TrainOutput\rightarrow(TrainOut_{1},...,TrainOut_{N})$

$Weights_{1}=TrainOut_{1}^{-1}\cdot TrainTargets_{1}$

…

$Weights_{N}=TrainOutN^{-1}\cdot TrainTargets_{N}$

$Prediction=\frac{1}{N}\sum_{N}{Weights_{N}\cdot TestOutput}$

$\bm{3.}$ An additional small MLP.

Figure 2:  a) The result of the preprocessing on a sample of the stock data.
b) A comparison of the output behavior to the input values. Output values
normalized to be comparable to the input.

###  II.3 Parallel Input Scattering Model

The PSM (Fig 3) was a $1250\times 50\times 2.5\,\mathrm{nm}$ grid with a slice
at the excitation side cutting two channels into the wave guide. At the
output, the region was divided into 1, 2, or 3 output channels.

Figure 3:  Layout of the PSM design used. The input region is divided into 2
input channels. Spin waves are excited in the input channels propagating down
to the scattering region where the two waves mix. The output region was
divided into 1, 2, or 3 read out channels (2 readout channels shown here).

A $250\,\mathrm{mT}$ external bias field was applied out of plane and the
input antennas excited forward volume magnetostatic spin waves at 6.2 GHz.
Forward volume wave geometry was chosen for this design so that waves could
propagate isotropically in the scattering region. This allows the input waves
from the different channels to scatter into each other. The interaction and
interference of the scattered waves is analogous to the non-linear
connectivity in a dense neural network layer. The scattering was achieved by
applying a reduced saturation magnetization in randomized spots as would be
seen through local laser heating Vogel  _et al._ (2018). Multiple data sets
were used to test the PSM in different capacities. In all cases, the neural
network to analyze the outputs was a small 1 hidden layer MLP with varying
node counts.

The initial data used was the Iris dataset from the UCI Machine Learning
Repository Fisher (1988). The data has 3 classes of species (Setosa,
Veriscolor, and Virginica) and 4 features for each class (petal length, petal
width, sepal length, and sepal width). Using only 2 features, the classes can
be broken up into two sets of 2 classes, one easily separable and one only
mostly separable (Fig 4a). As with the stock data and all additional data
sets, the features were scaled to a maximum value of 1. The width values were
then inverted by subtracting their value from 1. This was done to prevent the
PSM from simply being able to tell the classes apart simply by the output
amplitude as both the width and length of the petals and sepals increased in
value from $Setosa\rightarrow Veriscolor\rightarrow Virginica$. Inverting one
of the feature measures forces the PSM to differentiate the classes through
the scattering process. The intention of this test was to gauge the potential
of the PSM for feature mixing before continuing to more complicated
structures.

The second data set was the Statlog (German Credit Data) set also from the UCI
Machine Learning Repository Hofmann (1994). This data set contains 24
features, 12 of which are binary and 12 of which are numerical. Only the 12
numerical features were used as inputs for the PSM. The labels were set to 0
for good credit risk and 1 for bad credit risk. The inputs were stacked into a
6x2 tensor and sent into the 2 PSM input channels 1 at a time for 6
consecutive intervals. Only the 1 output channel PSM was used for this test.
The intention of this test was to push the feature mixing and information
preservation capability on a single PSM.

The final data set, referred to here as the Dimensional Reduction set, was a
created dataset with 4 features and 3 classes based on a randomly chosen
function of those features. The features were broken up into sets of 2 and
each set sent to their own PSM with a single output channel. The output of the
two initial PSMs were fed into the input channels of a final PSM, also with a
single output channel. The intention of this test was to gauge the capability
of the PSM for dimensional reduction of data channels.

[a] [b]

Figure 4:  a) Iris classes plotted by petal length and width. The Setosa and
Veriscolor classes make up the easily separable set. The Veriscolor and
Virginica classes make up the mostly separable set. b) The scaled readout from
the 1 output channel simulation for two samples within the easily separable
set.

##  III Results

###  III.1 Auto Oscillation Ring

To get a representation of the model’s abilities, each output method (linear
map, ensemble, or neural network) was performed 1000 times each on a
randomized selection of the data determined using the SKlearn Train Test Split
module with splits of 10, 20, 30, 40, and 50 percent. This resulted in 5000
trials for each method and a total of 15000 trials. The direct linear mapping
and ensemble approaches were nearly identical and never better on the AOR
output than on the data directly. Improvements were seen using the neural
network method on the AOR output compared to a similar network fed the data
directly. As shown in Table 1,

Auto Oscillation Ring
Neural Network Method Results
Max Mean Std Reference - No AOR (NN) 80.0 61.6 11.75 Reference - No AOR (LM)
80.0 62.7 5.41 Amplitude Encoded - (NN) 88.0 63.2* 8.19 Amplitude Encoded -
(LM) 84.0 56.7 6.23 Amplitude Encoded - ANN (NN) 92.0 64.7* 6.58 Amplitude
Encoded - ANN (LM) 84.0 57.5 6.16 Phase Encoded - (NN) 88.0 61.7 9.10 Phase
Encoded - (LM) 84.0 54.5 6.52 Phase Encoded - ANN (NN) 88.0 61.8 9.12 Phase
Encoded - ANN (LM) 88.0 55.3 6.52

Table 1: AOR Results: NN - Neural Network output analysis. LM - Linear Mapping
output analysis. ANN - Amplification Neural Network was included. The results
show that the inclusion of the ANN generally improved results and that
amplification encoding produced the best results. *indicates an improvement
over reference numbers.

phase encoding had a minimal impact on performance whether or not the ANN was
included. There was, however, a statistically significant improvement in
performance with amplitude encoding. Additional improvement was also seen in
all cases when the ANN was included. The AOR with ANN and amplitude encoding
had the highest maximum accuracy, highest mean accuracy, and lowest standard
deviation.

We also found that training of the ANN was not possible. When using back
propagation methods, the parameter gradients were always zero. To verify this
result and ensure that the zero gradient was not due to a mistake in the code
detaching the gradient during simulation, we also attempted training using the
parameter shift method popular in quantum machine learning Cerezo  _et al._
(2021). This method simply uses the derivation of a derivative
$(f(x+\Delta)-f(x))/\Delta$ which in the limit $\Delta\rightarrow 0$ is
$\frac{df}{dx}$. It requires 1 additional forward run of the model per
parameter but is not susceptible to the detachment error. The zero gradient of
the ANN does fit within the RC theory as the method of non-linearity
introduced into the system is not relevant, only the amount and consistency.

###  III.2 Parallel Input Scattering Model

####  III.2.1 Iris Data

The Iris data represents an entry level test to determine if the PSM was worth
pursuing. The design performed quite well showing promise for further testing.
The PSM was able to clearly distinguish classes in most cases (see for example
Fig 4b). Inverting the data along one axis (flipping Fig 4a horizontally)
significantly improved results suggesting that the amplitude ratio between the
two input channels was more important than the overall amplitude. Results for
the inverted axis case are presented in table 2 and a comparison to other
machine learning methods (fig 5) shows that the PSM performed extremely well
even for such a basic task.

Figure 5:  Benchmark results taken from Fisher (1988) with PSM results added
for comparison. LM - Linear Mapping result (95.0%). NN - Neural Network result
(98.3%).

Parallel Input Scattering Model Results
Iris Data Accuracy
Output Channels 1 2 3 Setosa/Veriscolor(a) Linear Map 97.5 100 100 Neural
Network 100 100 100 Viginica/Veriscolor(b) Linear Map 85.0 90.0 90.0 Neural
Network 95.0 97.5 97.5 Setosa/Veriscolor/Viginica(c) Linear Map 90.0 95.0 95.0
Neural Network 92.0 98.3 97.1

Table 2: Accuracies for the Iris data based on the number of output channels.
(a) the easily distinguishable classes. (b) the mostly distinguishable
classes. (c) distinguishing between all 3 classes at the same time. Standard
deviation for all results was less than 1%.

####  III.2.2 Statlog Data

Figure 6: (a) Features are broken up into pairs. Each pair is sent to the PSM
one at a time. (b) Benchmark results taken from Hofmann (1994) with PSM
results added for comparison. Red Line - Linear Mapping result (67.7 +/-
9.6%). Blue Line - Neural Network result (74.7 +/- 2.0%).

The Statlog data demonstrated that the PSM was able to preserve information
from multiple features put into the system sequentially (Fig 6a). The waves
that were scattered continue to bounce around and effect the output channel
well after the initial wavefront has passed through. This appears to be
sufficient as a form of fading memory required by RC theory. Despite the fact
that the feature data had no temporal relation, the sequential input worked
well to maintain the information at the output channel. Because the benchmark
performances from Hofmann (1994) are assumed to be done on the full set of 24
features, reference benchmarks were done using the same subset of 12 features
(Table 3).

Parallel Input Scattering Model Results
Statlog Data Accuracy
Mean Std Reference Linear Mapping 71.6 1.9 Reference Neural Network 74.3 0.9
PSM Linear Mapping 67.7 9.6 PSM Neural Network 74.7 2.0

Table 3: Accuracies for the Statlog data. Reference accuracy tests were done
by performing the same output analysis directly on the 12 input features.

The accuracy from direct linear mapping decreased with the PSM compared to the
reference. However, the addition of a small MLP at the output recovered the
loss in accuracy and slightly improved the baseline performance. While some
information may have been lost due to inserting non-temporal data in a time
series fashion, it appears that the mixing from the scattering region did more
than make up for loss. These initial results suggest that the PSM is capable
of reducing multiple input features into a single output channel regardless of
whether they are time series data traditionally used in RC systems or not.

####  III.2.3 Dimensional Reduction Data

Figure 7: Workflow for dimensional reduction test. Features are separated to
different input channels on different devices. Output channels from previous
layer devices are passed to the input channels of the next layer until a
single output channel remains.

Parallel Input Scattering Model Results
(a) Dimensional Reduction Data Accuracy
Mean Std Reference Linear Mapping 37.9 1.7 Reference Neural Network 95.7 1.3
PSM Linear Mapping 65.9 6.1 PSM Neural Network 81.2 3.8 (b) Accuracy On Single
PSM
Mean Std One Output Channel PSM Linear Mapping 66.7 5.6 PSM Neural Network
96.8 1.7 Two Output Channels PSM Linear Mapping 69.8 5.4 PSM Neural Network
97.5 3.8

Table 4: Accuracies for the dimensional reduction data: (a) for the multiple
PSM approach where only two features are provided to each initial PSM and the
single channel outputs are fed into the input channels of the next layer PSM.
(b) for a single PSM where all features are provided in sequential pairs. Both
one and two output channel accuracy given.

The dimensional reduction data was created to test the ability of multiple
PSMs to take single value features and reduce them down to single channel
output channel (Fig 7). Outputs from the previous layer were also amplified to
be roughly the same strength as the original input layer. The data was created
by randomly choosing 3 points in a 4 dimensional space as class centers.
Additional sets of random 4 dimensional vectors were created and assigned a
class based on the closest class center. A loss of some information was seen
(-14.5 $\pm$ 5.1% accuracy) when comparing MLP accuracy’s of the output
channel compared to the data directly (Table 4 a). There was a significant
improvement in performance from direct linear mapping which is to be expected
from the non-linear mixing in the RC scattering regions. In order to confirm
that the information loss was due to the design and not a peculiarity in the
data, this dataset was also tested on a single PSM in the same manner as the
Statlog data (results in Table 4 b). The features were paired and given as
inputs to the PSM sequentially. The single PSM result was significantly better
than both the multiple PSM design and the baseline reference. This confirms
that sequential passing of features on a single PSM is more effective than
using multiple PSMs in a converging architecture.

##  IV Conclusions

The Auto Oscillation Ring produced a statistically relevant result for the
given task. The design showed marginal improvement with the addition of the
in-line Amplification Neural Network which needed no training. Additional
improvements in the design may increase it’s utility as part of a larger
machine learning system.

The Parallel Input Scattering Model was designed to mimic a series of dense
neural network layers by scattering a series of forward volume waves. This
design performed very well on the basic classification test with results
comparable to other leading machine learning methods. The results from the
feature capacity test are arguably the most promising for future use of the
PSM. The ability of the design to convert 12 input features into 1 output
channel is highly encouraging for possible use in reducing the size,
complexity, and cost of large machine learning models. The use of multiple
cascading PSMs was not effective. This is likely due to the loss in
information seen from the output of the PSM without an additional neural
network at the output. This loss is carried forward from the previous PSM to
the next.

The simulation size of all designs presented was much smaller than the size of
a likely physical implementation due to hardware and time constraints. While
the general physics and result trends should scale up, there may be noticeable
differences when physically implementing these designs. Future work would
likely consist of building a physical system and testing for similar
performance. Should the results be consistent with the simulated results,
finding a direct comparison of the speed and energy cost of magnonic systems
vs traditional neural networks would be of interest.

##  V Acknowledgements

Authors acknowledge the support from the National Science Foundation of the
United States by Grant No. ECCS-2138236 .

## References

  * Lukoševičius and Jaeger (2009) M. Lukoševičius and H. Jaeger, “Reservoir computing approaches to recurrent neural network training,” Computer science review 3, 127–149 (2009).
  * Nakajima (2020) K. Nakajima, “Physical reservoir computing—an introductory perspective,” Japanese Journal of Applied Physics 59, 060501 (2020).
  * Pirro  _et al._ (2021) P. Pirro, V. Vasyuchka, A. Serga, and B. Hillebrands, “Advances in coherent magnonics,” Nature Reviews Materials 6, 1114–1135 (2021).
  * Rana  _et al._ (2019) B. Rana, A. K. Mondal, S. Bandyopadhyay, al, Y. Fu, L. Yang, M. Cao, D. Lachance-Quirion, Y. Tabuchi, A. Gloppe, K. Usami, and Y. Nakamura, “Hybrid quantum systems based on magnonics,” Applied Physics Express 12, 070101 (2019).
  * Rana and Otani (2019) B. Rana and Y. Otani, “Towards magnonic devices based on voltage-controlled magnetic anisotropy,” Communications Physics 2, 90 (2019).
  * Chumak, Serga, and Hillebrands (2017) A. Chumak, A. Serga, and B. Hillebrands, “Magnonic crystals for data processing,” Journal of Physics D: Applied Physics 50, 244001 (2017).
  * Chumak (2019) A. Chumak, “Magnon spintronics: Fundamentals of magnon-based computing,”  (2019).
  * Sadovnikov, Beginin, and Morozova (2016) A. Sadovnikov, E. Beginin, and M. Morozova, “Nonlinear spin wave coupling in adjacent magnonic crystals,” Applied Physics Letters 109, 042407 (2016).
  * Albisetti  _et al._ (2018) E. Albisetti, D. Petti, G. Sala, R. Silvani, and S. Tacchi, “Nanoscale spin-wave circuits based on engineered reconfigurable spin-textures,” Communications Physics 1, 56 (2018).
  * Mahmoud  _et al._ (2021) A. N. Mahmoud, F. Vanderveken, C. Adelmann, F. Ciubotaru, S. Cotofana, and S. Hamdioui, “Spin wave normalization toward all magnonic circuits,” IEEE Transactions on Circuits and Systems I: Regular Papers 68, 536–549 (2021).
  * Heussner  _et al._ (2017) F. Heussner, A. A. Serga, T. Brächer, B. Hillebrands, and P. Pirro, “A switchable spin-wave signal splitter for magnonic networks,” Applied Physics Letters 111 (2017), 10.1063/1.4987007/33857.
  * Schneider  _et al._ (2008) T. Schneider, A. A. Serga, B. Leven, B. Hillebrands, R. L. Stamps, and M. P. Kostylev, “Realization of spin-wave logic gates,” Applied Physics Letters 92, 022505 (2008).
  * Ustinov  _et al._ (2019) A. B. Ustinov, E. Lähderanta, M. Inoue, and B. A. Kalinikos, “Nonlinear spin-wave logic gates,” IEEE Magnetics Letters 10 (2019), 10.1109/LMAG.2019.2950638.
  * Chen  _et al._ (2021) J. Chen, H. Wang, T. Hula, C. Liu, S. Liu, T. Liu, H. Jia, Q. Song, C. Guo, Y. Zhang, J. Zhang, X. Han, D. Yu, M. Wu, H. Schultheiss, and H. Yu, “Reconfigurable spin-wave interferometer at the nanoscale,” ACS Publications 21, 6244 (2021).
  * Kostylev  _et al._ (2005) M. Kostylev, A. Serga, T. Schneider, and B. Leven, “Spin-wave logical gates,” Applied Physics Letters 87, 153501 (2005).
  * Yu, Lan, and Xiao (2020) W. Yu, J. Lan, and J. Xiao, “Magnetic logic gate based on polarized spin waves,” Physical Review Applied 13 (2020), 10.1103/PHYSREVAPPLIED.13.024055.
  * Khitun, Bao, and Wang (2008) A. Khitun, M. Bao, and K. L. Wang, “Spin wave magnetic nanofabric: A new approach to spin-based logic circuitry,” IEEE Transactions on Magnetics 44, 2141–2152 (2008).
  * Khitun, Bao, and Wang (2010) A. Khitun, M. Bao, and K. L. Wang, “Magnonic logic circuits,” Journal of Physics D: Applied Physics 43, 264005 (2010).
  * Zografos  _et al._ (2014) O. Zografos, P. Raghavan, L. Amarú, B. Sorée, R. Lauwereins, I. Radu, D. Verkest, and A. Thean, “System-level assessment and area evaluation of spin wave logic circuits,” in  _Proceedings of the 2014 IEEE/ACM International Symposium on Nanoscale Architectures_, NANOARCH ’14 (Association for Computing Machinery, New York, NY, USA, 2014) p. 25–30.
  * Nakane, Tanaka, and Hirose (2018) R. Nakane, G. Tanaka, and A. Hirose, “Reservoir computing with spin waves excited in a garnet film,” IEEE access 6, 4462–4469 (2018).
  * Körber  _et al._ (2023) L. Körber, C. Heins, T. Hula, J.-V. Kim, S. Thlang, H. Schultheiss, J. Fassbender, and K. Schultheiss, “Pattern recognition in reciprocal space with a magnon-scattering reservoir,” Nature Communications 14, 3954 (2023).
  * Nikitin  _et al._ (2022) A. A. Nikitin, A. A. Nikitin, A. B. Ustinov, S. Watt, and M. P. Kostylev, “Theoretical model for nonlinear spin-wave transient processes in active-ring oscillators with variable gain and its application for magnonic reservoir computing,” Journal of Applied Physics 131 (2022), 10.1063/5.0081142.
  * Kondrashov  _et al._ (2022) A. V. Kondrashov, A. A. Nikitin, A. A. Nikitin, M. Kostylev, and A. B. Ustinov, “Numerical simulation of performance of magnonic reservoir computer based on active-ring oscillator,” Journal of Magnetism and Magnetic Materials 563 (2022), 10.1016/j.jmmm.2022.169968.
  * Watt  _et al._ (2021) S. Watt, M. Kostylev, A. B. Ustinov, and B. A. Kalinikos, “Implementing a magnonic reservoir computer model based on time-delay multiplexing,” Physical Review Applied 15, 064060 (2021).
  * Watt and Kostylev (2024) S. Watt and M. Kostylev, “Numerical simulations of a magnonic reservoir computer,” Journal of Applied Physics 135 (2024).
  * Bruckner  _et al._ (2023) F. Bruckner, S. Koraltan, C. Abert, and D. Suess, “magnum. np–a pytorch based gpu enhanced finite difference micromagnetic simulation framework for high level development and inverse design,” arXiv preprint arXiv:2302.08843 (2023).
  * Fisher (1988) R. A. Fisher, “Iris,” UCI Machine Learning Repository (1988), DOI: https://doi.org/10.24432/C56C76.
  * Hofmann (1994) H. Hofmann, “Statlog (German Credit Data),” UCI Machine Learning Repository (1994), DOI: https://doi.org/10.24432/C5NC77.
  * Vogel  _et al._ (2018) M. Vogel, R. Aßmann, P. Pirro, A. V. Chumak, B. Hillebrands, and G. von Freymann, “Control of spin-wave propagation using magnetisation gradients,” Scientific reports 8, 11099 (2018).
  * Cerezo  _et al._ (2021) M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, _et al._ , “Variational quantum algorithms,” Nature Reviews Physics 3, 625–644 (2021).


# Clinical Trials Protocol Authoring using LLMs

Morteza Maleki
College of Computing
Georgia Institute of Technology
Atlanta, Georgia
<EMAIL_ADDRESS>
###### Abstract

This report embarks on a mission to revolutionize clinical trial protocol
development through the integration of advanced AI technologies. With a focus
on leveraging the capabilities of generative AI, specifically GPT-4, this
initiative aimed to streamline and enhance the efficiency and accuracy of
clinical trial protocols. The methodology encompassed a detailed analysis and
preparation of comprehensive drug and study level metadata, followed by the
deployment of GPT-4 for automated protocol section generation. Results
demonstrated a significant improvement in protocol authoring, highlighted by
increases in efficiency, accuracy, and the customization of protocols to
specific trial requirements. Challenges encountered during model selection and
prompt engineering were systematically addressed, leading to refined
methodologies that capitalized on the advanced text generation capabilities of
GPT-4. This project not only showcases the practical applications and benefits
of generative AI in clinical trial design but also sets a foundation for
future innovations in the field.

##  1 Introduction

The landscape of clinical trial design is undergoing a transformative shift,
driven by the integration of advanced technological solutions aimed at
enhancing the efficiency, accuracy, and speed of trial protocol development.
Within this context, the "Intelligent Clinical Trial Design Support using
Generative AI" project represents a pioneering effort to harness the potential
of generative artificial intelligence (AI) to revolutionize the creation of
clinical trial protocols. This initiative is positioned at the forefront of
innovation in clinical research, aiming to leverage the capabilities of
generative AI to significantly improve the process of trial protocol
authoring.

Clinical trials are the cornerstone of medical research, providing the
necessary evidence for the safety and efficacy of new treatments before they
can be made available to the public. However, the design and development of
clinical trial protocols are often characterized by complex, time-consuming
tasks that require meticulous attention to detail and adherence to regulatory
standards.

The adoption of generative AI in this domain promises to automate and
streamline the protocol authoring process, reducing the manual effort involved
and minimizing the potential for human error. Specifically, we focused on
exploiting the advanced capabilities of GPT-4, a state-of-the-art language
model known for its ability to generate coherent and contextually relevant
text. Through the strategic application of this technology, we anticipated not
only to expedite the development of clinical trial protocols but also to
elevate the quality of the documents produced, thereby contributing to the
overall efficiency and effectiveness of clinical research.

This introduction sets the stage for the subsequent sections of the
manuscript, which will delve into the project’s background, methodology,
results, and the broader implications of our work. By aligning our efforts
with the evolving needs of clinical trial design, the project underscores the
vital role of generative AI in shaping the future of medical research,
offering new avenues for innovation and advancement in the field.

##  2 Materials and Methods

###  2.1 Clinical Trails

Clinical trials serve as pivotal research studies aimed at evaluating the
therapeutic effects, risks, and benefits of medical interventions,
encompassing drugs, devices, surgical procedures, and behavioral treatments.
Maleki et al. (2022); Maleki and Khan (2023) As the bedrock of evidence-based
medicine, these trials meticulously guide the introduction of new treatments
to ensure they are safe and effective for public use. Organized into phases I
through IV, each phase of a clinical trial is designed to answer specific
research questions, progressively leading to the development and market
approval of new therapeutic interventions.

###  2.2 Study Design in Clinical Trials

The intricacies of clinical trial design form the backbone of any research
study in this domain. This framework encompasses the strategic planning and
execution of the trial, including the selection of participants through well-
defined inclusion and exclusion criteria, ensuring a representative and
suitable subject pool. To mitigate bias, the study design incorporates methods
such as randomization and control groups, alongside blinding mechanisms where
both participants and researchers are unaware of the treatment allocations.
Furthermore, the establishment of primary and secondary endpoints, along with
rigorous data collection and analysis procedures, ensures the acquisition of
reliable and actionable findings.

###  2.3 Challenges and Pain Points in Study Design

The complexity of clinical trial design is underscored by several challenges,
ranging from ethical considerations and regulatory compliance to practical
issues like patient recruitment and data management. Ensuring the safety and
rights of participants while adhering to stringent health authority
regulations presents a continuous balancing act. Additionally, the recruitment
and retention of a sufficient number of participants remain a perennial
obstacle, compounded by the need to maintain the integrity and accuracy of
collected data. These challenges necessitate innovative solutions to enhance
operational efficiency and data quality.

###  2.4 Project Impact

The "Intelligent Clinical Trial Design Support using Generative AI" project
aims to redefine the landscape of clinical trial protocol development by
introducing automation through advanced AI models. This initiative focuses on
generating coherent and contextually relevant protocol sections, thereby
substantially reducing the manual labor involved in protocol authoring. By
leveraging extensive study and drug metadata, the project ensures the accuracy
and customization of generated protocols to meet specific trial requirements,
thereby enhancing efficiency, accuracy, and flexibility in clinical trial
design.

###  2.5 Literature Review

  1. 1.

Enhanced training techniques for long text generation

In the realm of long text generation, several groundbreaking studies have set
new benchmarks. Devlin et al Devlin et al. (2018) in "BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding" revolutionized
language modeling with BERT, introducing a bidirectional training method that
significantly enhanced contextual understanding in text generation. Building
upon this, Dai et al. Dai et al. (2019) addressed the limitations of
traditional Transformer models through "Transformer-XL: Attentive Language
Models Beyond a Fixed-Length Context." This paper was instrumental in
developing a recurrence mechanism to handle long-term dependencies, enabling
the generation of longer and more coherent text sequences. Another pivotal
contribution came from Keskar et al. Keskar et al. (2019) with "CTRL: A
Conditional Transformer Language Model for Controllable Generation." This work
shifted the focus towards controllable text generation, demonstrating how
specific prompts could steer text generation in desired thematic directions.
Lastly, Song et al. Song et al. (2019) proposed "MASS: Masked Sequence to
Sequence Pre-training for Language Generation," introducing a novel pre-
training method for sequence-to-sequence models. This approach laid the
groundwork for advanced techniques in generating long-form content,
highlighting the evolution of model training methodologies in this domain.

  2. 2.

Publications on using GPT prompts for long text generation

Complementing the advancements in model training, significant research has
also focused on the use of GPT prompts for long text generation. Petroni et
al. Petroni et al. (2019) in "Language Models as Knowledge Bases?" explored
the potential of GPT-2 as a knowledge base, highlighting its capability to
generate text based on embedded knowledge, thus underscoring the effectiveness
of prompt engineering. Building on this concept, Lester et al. Lester et al.
(2021) delved into the scalability of prompt tuning in large language models
like GPT-3 in "The Power of Scale for Parameter-Efficient Prompt Tuning." This
research provided valuable insights into how efficient prompting significantly
enhances a model’s ability to generate contextually relevant long texts.
Further exploring the control and flexibility in text generation, Dathathri et
al. Dathathri et al. (2019) presented "Plug and Play Language Models: A Simple
Approach to Controlled Text Generation." This study introduced a novel method
for controlling the style and content of generated texts using GPT-2,
showcasing a flexible, plug-and-play approach for generating long texts with
minimal fine-tuning.

###  2.6 Methodology

In order to carry out the task of protocol generation, various activities from
study and drug metadata collection, clinical trials protocol introduction and
study design sections extraction, data preprocessing, prompt engineering and
evaluation of the final results.

  * •

Protocol and Section Review: Analyzing Type II diabetes protocols from CT.GOV,
focusing on US-based sponsors and recent studies.

  * •

Section Selection for Automation: Choosing the Introduction and Study Design
sections based on their prevalence and significance in protocols.

  * •

Metadata Collection: Accumulating extensive metadata from CT.Gov and
TrialTrove for the selected protocols.

  * •

Technical Implementation:

    * –

LLM Training: Experimentation with T5 models, leading to insights about the
challenges in generating satisfactory outputs.

    * –

GPT-4 Prompt Engineering: Successfully leveraging GPT-4 for high-quality
generation of the Introduction section.

###  2.7 Data Sources and Analysis

####  2.7.1 Drug level metadata

The drug level metadata formed a foundational component of our AI model’s
training data. The dataset encompassed comprehensive details about each drug,
including:

  * •

Basic Information: Citeline Drug ID, Generic Drug Name, and a summary of the
drug.

  * •

Development Status: Detailed information on the development stage of the drug,
including global and regional statuses.

  * •

Therapeutic Application: Insight into the diseases targeted by the drug,
including whether these diseases are categorized as rare.

  * •

Company Profiles: Information about the companies involved in the development
and marketing of the drugs, including headquarters and subsidiary details.

  * •

Clinical and Scientific Details: In-depth information on the drug’s mechanism
of action, therapeutic class, delivery route, and molecular characteristics.

This metadata provided valuable insights into the drugs’ profiles,
significantly aiding in the generation of contextually accurate and relevant
protocol sections.

####  2.7.2 Study level metadata

The study level metadata enriched our dataset with detailed information on
various clinical trials. This data included:

  * •

Trial Information: Trial ID, title, phase, status, and detailed objectives.

  * •

Sponsorship Details: Information on the sponsor companies and their
geographical locations.

  * •

Patient and Disease Information: Detailed descriptions of the patient
segments, diseases studied, and inclusion/exclusion criteria.

  * •

Endpoints and Outcomes: Information on primary and secondary endpoints,
including details and dates of reporting.

This diverse dataset enabled the AI model to understand the nuances of
different clinical trials, ensuring that the generated protocol sections were
aligned with the specific requirements of each study. The metadata fields used
in this project are summarized in Table 1.

Metadata fields used in model training and prompt engineering:

Table 1: Metadata fields used in model training and prompt engineering. # |  Name |  Description
---|---|---
1 |  MeSH Term |  Medical subject headings that describe the content.
2 |  Primary Endpoint Group |  The main outcome that is measured to see if the treatment was effective.
3 |  Treatment Plan |  The schedule and dosage of the treatment administered.
4 |  Sponsor |  Organization or individual who oversees the clinical trial and is responsible for the data.
5 |  Start Date |  The date on which the trial began or is set to begin.
6 |  Study Keywords |  Keywords or phrases related to the study.
7 |  Protocol / Trial ID |  Unique identifier for the trial protocol.
8 |  Primary Tested Drug |  The main drug or treatment being tested in the trial.
9 |  Trial Tag / Attribute |  Additional attributes or tags related to the trial.
10 |  Primary Tested Drug:Mechanism of Action |  The biological mechanism through which the drug exerts its effect.
11 |  Decentralized (DCT) Attribute |  Indicates if the trial includes decentralized or virtual components.
12 |  Trial Title |  The official title of the clinical trial.
13 |  Primary Tested Drug: Target |  The specific biological target of the primary tested drug.
14 |  Patient Population |  The characteristics of the patient group being studied.
15 |  Drug Names |  Names of all drugs involved in the trial.
16 |  Trial Phase |  The phase of the clinical trial, indicating its stage in the development process.
17 |  Primary Tested Drug: Therapeutic Class |  The class of medical use the drug falls under.
18 |  Inclusion Criteria |  Criteria that must be met for an individual to be eligible for the trial.
19 |  Exclusion Criteria |  Criteria that disqualify an individual from participating in the trial.
20 |  Global Status |  The current status of the trial globally.
21 |  Therapeutic Area |  The area of medicine related to the treatment being tested.
22 |  Oncology Biomarker Common Use(s) |  Common uses of biomarkers in oncology trials.
23 |  Identified Sites |  Locations where the trial is being conducted.
24 |  Development Status |  The development stage of the drug within the trial.
25 |  Disease |  The disease or condition being studied in the trial.
26 |  Trial Objective |  The primary objective or goal of the trial.
27 |  Trial Region |  Geographic region where the trial is conducted.
28 |  Drug Disease |  Diseases targeted by the drugs being tested.
29 |  Phase I |  Indicates if the trial is in Phase I.
30 |  Phase II |  Indicates if the trial is in Phase II.
31 |  Phase III |  Indicates if the trial is in Phase III.
32 |  Preclinical |  Indicates if the trial is in the preclinical stage.
33 |  Therapeutic Class |  The broad category of the drug’s therapeutic use.
34 |  Therapeutic Class Status |  Status within its therapeutic class.
35 |  Drug Disease: Rare |  Indicates if the drug targets a rare disease.

###  2.8 Data Sources

The summary data, metadata, and code for this work can be found in the
following GitHub repository:
https://github.com/mprtrmrtz/protocol_authoring_using_LLM

###  2.9 Data processing and Preparation

The data processing for this project involved a comprehensive ETL process.
This process was meticulously designed to extract relevant data from three key
sources: protocol extractions, study metadata, and drug metadata. The primary
goal was to prepare this data for effective use in training and utilizing the
generative AI models for protocol generation.

####  2.9.1 Data Extraction

  * •

Protocol Extractions: The ‘protocol_extractions.xlsx‘ file was read into a
DataFrame, capturing essential information about various clinical trial
protocols.

  * •

Study Metadata: Similarly, data from ‘TT Study Level Metadata.xlsx‘ was
loaded, providing detailed metadata on clinical studies.

  * •

Drug Metadata: The ‘TT Drug Level Metadata.xlsx‘ file was processed to obtain
comprehensive data on drugs involved in these studies.

####  2.9.2 Data Pre-processing and Transformation

  * •

Field Selection: Key fields were identified for both study and drug metadata
to ensure relevancy to the project’s objectives. Fields such as ’Generic Drug
Name’, ’Drug Disease’, and ’Event Date’ for drugs, and ’Trial ID’, ’Disease’,
and ’Therapeutic Area’ for studies, were among those selected.

  * •

Duplicate Removal: A custom function ‘remove_duplicates‘ was used to eliminate
duplicate entries within specific columns of the drug metadata, ensuring data
cleanliness. \- Data Splitting and Exploding: In the study metadata, the
’Primary Tested Drug’ column was split and exploded to disaggregate the drug
names, facilitating more detailed analysis.

  * •

Text Replacement: Specific text replacements were made in the ’Primary Tested
Drug’ field to standardize drug names, enhancing data uniformity.

  * •

Dataframe Renaming and Merging: The drug metadata DataFrame was renamed for
consistency, and then merged with the study metadata to create a combined
DataFrame, aligning drugs with their corresponding trials.

  * •

NCT ID Extraction: The National Clinical Trial (NCT) ID was extracted from the
’Protocol/Trial ID’ column, which was a critical step to align each trial with
its specific protocol.

  * •

Final Data Merge: The combined study and drug metadata were merged with the
protocol extractions DataFrame using the NCT ID, culminating in a
comprehensive dataset that aligns protocol extractions with detailed study and
drug information.

####  2.9.3 Load Phase

The processed data was manually provided to the models in this work, however,
in the actual implementation, it is is expected that the processed data would
be provided through AWS or Database query for further analysis.

####  2.9.4 Data Preparation for Model Training

The final merged dataset, now rich with both protocol and associated metadata,
was primed for use in training the generative AI models. This dataset provides
the depth and context necessary for the AI to understand and generate accurate
protocol sections.
The ETL process employed in this project was critical in ensuring that the
data was not only comprehensive and relevant but also clean and structured in
a way that maximizes its utility in training effective AI models. This
meticulous approach to data processing underpins the success of the AI model
in generating precise and contextually appropriate clinical trial protocols.

##  3 Model Development and Evaluation

After data preprocessing step and preparation of metadata for text generation,
two approaches were taken for text generation:

  1. 1.

LLM Model Training: Involving T5 Small, Large and BioBart models training

  2. 2.

Open AI GPT Models Prompt Engineering: Involving gpt-3.5, gpt-4 and their
variations

###  3.1 Model Training Based Google LLM Models

The following models were used for training on the metadata and their
corresponding Introduction section of the protocol. Additional information
about the models below:

  1. 1.

T5 Small LLM Model

     * •

Model Overview: T5, or Text-to-Text Transfer Transformer, is a language model
developed by Google. The "small" variant is a more compact version of the
standard T5 model.

     * •

Model Size: The T5 small model typically has fewer parameters compared to its
larger counterparts. The exact number can vary but is significantly lower than
the base or large models.

     * •

Key Features:

       * –

Designed to convert all NLP tasks into a text-to-text format, meaning both the
input and output are always text strings.

       * –

Despite being a smaller variant, it retains the core functionality of the T5
architecture and is suitable for environments where computational resources
are limited.

       * –

Ideal for tasks like translation, summarization, question answering, and more,
but with a balance between performance and resource efficiency.

  2. 2.

T5 Large LLM Model

     * •

Model Overview: This is a larger variant of the T5 model, which means it has
more parameters and potentially better performance at the cost of increased
computational resource requirements.

     * •

Model Size: The T5 large model has significantly more parameters than the
small variant, enhancing its capacity to understand and generate text.

     * •

Key Features:

       * –

Like the small model, it treats all NLP tasks as text-to-text problems.

       * –

Due to its larger size, it generally performs better on complex tasks and can
handle more nuanced aspects of language processing.

       * –

Suitable for high-resource settings where advanced NLP capabilities are
needed.

  3. 3.

BioBART LLM Model

     * •

Model Overview: BioBART is a variant of the BART (Bidirectional and Auto-
Regressive Transformers) model, which is tailored for biomedical text.

     * •

Key Features:

       * –

BART is known for its effectiveness in text generation and comprehension
tasks, and BioBART adapts this for the biomedical domain.

       * –

It’s pre-trained on large-scale biomedical literature, making it adept at
handling specialized terminology and concepts found in medical texts.

       * –

Often used for tasks like biomedical text summarization, medical question
answering, and information extraction from clinical notes or research papers.

The T5 and BioBART models are examples of how language models can be adapted
to different scales (like small and large) and specialized domains (like
biomedicine). These adaptations allow for more effective application in
specific scenarios, balancing the trade-offs between computational resources
and task-specific performance.

It was realized that the text generation task using these models were not
efficient and accurate at all, the output generated text was short (400
characters) and did not encapsulate the full purpose of text generation in
this project. The work on this venue was immediately halted and a prompt
engineering path was explored as an alternative approach.

###  3.2 Open AI GPT Models

  1. 1.

GPT-3.5

     * •

Model Variants: GPT-3.5-turbo, GPT-3.5-turbo-1106, GPT-3.5-turbo-16k-0613

     * •

Key Features:

       * –

Iterative improvements on GPT-3, focusing on efficiency, speed, and fine-
tuning capabilities.

       * –

The ’turbo’ variants indicate models optimized for faster response times and
efficient processing, suitable for interactive applications.

       * –

These models continue to push the boundaries of language understanding and
generation, maintaining high performance across diverse tasks.

  2. 2.

GPT-4

     * •

Key Features:

       * –

Anticipated to be an advancement over GPT-3 with more parameters and improved
capabilities.

       * –

Expected to further enhance the model’s understanding, context handling, and
response generation.

       * –

Likely to continue OpenAI’s trend of creating more powerful and versatile
language models.

  3. 3.

GPT-4-1106-preview

     * •

Key Features:

       * –

A variant of GPT-4, which is a preview or a scaled-down version for specific
applications or testing.

       * –

Likely to offer improvements over GPT-3.5 while being a step towards the full
GPT-4 model.

Each iteration of GPT models represents a leap forward in natural language
processing, with improvements in text generation, comprehension, and
versatility in handling various language-based tasks. As AI research
progresses, these models continue to evolve, offering increasingly advanced
capabilities in understanding and generating human language.

##  4 Results

###  4.1 GPT4 Prompt Engineering

Leveraging GPT4, we observed a significant improvement in the quality of the
generated content. The model successfully created protocol sections that were
not only accurate but also contextually rich and coherent. This success marked
a pivotal point in the project, showcasing the potential of advanced
generative AI in automating complex tasks like protocol generation.
Prompt template provided to GPT models for text generation:

You are a clinical study designer, and you need to generate the introduction
section of a clinical trial protocol based on information about the study and
drug metadata. Follow the same format and writing style as the example below:

Example 1 Input:
…

Generated:
…

Example 2 Input:
..

Generated:
..

Metadata for prediction: ..

Now Begin!
Generate the introduction section:

###  4.2 Evaluation Metrics

The following evaluation metrics have been used to assess the output of the
gpt text generation models.

####  4.2.1 Cosine Similarity

Cosine similarity measures the cosine of the angle between two non-zero
vectors in a multi-dimensional space. In text analysis, these vectors often
represent text documents. It operates in the following paradigm:

  * •

Text data is first converted into vectors, often using techniques like TF-IDF
(Term Frequency-Inverse Document Frequency) or word embeddings.

  * •

Cosine similarity is then calculated between these vectors. The value ranges
from -1 (completely different) to 1 (exactly the same).

  * •

It’s used to compare the similarity between the generated text and a reference
text.

  * •

High cosine similarity indicates that the generated text closely matches the
reference in terms of content and context.

####  4.2.2 BLEU Scores

BLEU (Bilingual Evaluation Understudy) is a metric for evaluating a generated
sentence to a set of reference sentences. It operates in the following
paradigm:

  * •

BLEU-n Score: Measures the co-occurrence of n-grams (sequence of n words)
between the generated text and reference text. BLEU1 refers to unigrams (1
word), BLEU2 to bigrams (2 words), and so forth.

  * •

BLEU score calculates the precision of n-grams in the generated text relative
to the reference text, then combines these precisions using a geometric mean.

  * •

A brevity penalty is applied if the generated text is shorter than the
reference, to discourage overly terse outputs.

  * •

A higher BLEU score indicates better agreement between the machine’s output
and the human reference.

  * •

Used extensively in machine translation and can be applied to other text
generation tasks.

####  4.2.3 Rogue Scores

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics
used for evaluating automatic summarization and machine translation.

It operates in the following paradigm:

  * •

ROUGE-N: Measures the overlap of n-grams between the generated text and
reference text.

  * •

ROUGE-L: Considers the longest common subsequence between the generated and
reference text.

  * •

Recall: The proportion of the reference text’s n-grams found in the generated
text.

  * •

Precision: The proportion of the generated text’s n-grams that are found in
the reference text.

  * •

F1 Score: The harmonic mean of precision and recall, providing a balance
between the two.

  * •

ROUGE is particularly useful in evaluating summarization where capturing the
essence of the original text is crucial.

  * •

A higher ROUGE score indicates that the generated text more effectively
captures the key points of the reference text.

These metrics are valuable tools in assessing the quality of text generated by
AI models. They provide objective ways to measure aspects like similarity,
fluency, and relevance of the generated text compared to a human-written
reference.

Figures 1 - 7 and Table 2 show the results of analysis on model performance in
the task of Introduction section generation for all models and variation of
number of examples.

Table 2: Evaluation Metrics for GPT Models Model |  # | Time | In. Tok. | Out. Tok. | Error | Cos. Sim. | bleu1 | bleu2 | bleu3 | bleu4 | Precision | Recall | F1
---|---|---|---|---|---|---|---|---|---|---|---|---|---
gpt-3.5-turbo |  0 | 88.55 | 375 | 344 | – | 0.6 | 0.22 | 0.11 | 0.06 | 0.04 | 0.33 | 0.19 | 0.24
gpt-3.5-turbo |  1 | 146.87 | 1,188 | 477 | – | 0.82 | 0.53 | 0.4 | 0.33 | 0.27 | 0.62 | 0.5 | 0.55
gpt-3.5-turbo |  2 | – | – | – | Max Tok. | – | – | – | – | – | – | – | –
gpt-3.5-turbo |  3 | – | – | – | Max Tok. | – | – | – | – | – | – | – | –
gpt-3.5-turbo-1106 |  0 | 7.44 | 375 | 255 | – | 0.6 | 0.11 | 0.05 | 0.03 | 0.02 | 0.32 | 0.15 | 0.2
gpt-3.5-turbo-1106 |  1 | 9.69 | 1,970 | 299 | – | 0.55 | 0.19 | 0.1 | 0.06 | 0.04 | 0.43 | 0.23 | 0.3
gpt-3.5-turbo-1106 |  2 | 13.84 | 3,554 | 442 | – | 0.62 | 0.32 | 0.16 | 0.09 | 0.06 | 0.34 | 0.25 | 0.29
gpt-3.5-turbo-1106 |  3 | 13.37 | 4,356 | 373 | – | 0.78 | 0.35 | 0.25 | 0.19 | 0.14 | 0.62 | 0.37 | 0.47
gpt-3.5-turbo-16k-0613 |  0 | 70.47 | 375 | 361 | – | 0.61 | 0.23 | 0.11 | 0.06 | 0.03 | 0.32 | 0.19 | 0.24
gpt-3.5-turbo-16k-0613 |  1 | 153.33 | 1,970 | 693 | – | 0.63 | 0.39 | 0.2 | 0.1 | 0.06 | 0.27 | 0.26 | 0.26
gpt-3.5-turbo-16k-0613 |  2 | 719.03 | 1,990 | 398 | – | 0.81 | 0.43 | 0.32 | 0.26 | 0.21 | 0.71 | 0.47 | 0.57
gpt-3.5-turbo-16k-0613 |  3 | 70.06 | 4,356 | 296 | – | 0.63 | 0.18 | 0.11 | 0.08 | 0.06 | 0.41 | 0.22 | 0.29
gpt-4 |  0 | 20.39 | 375 | 303 | – | 0.58 | 0.17 | 0.07 | 0.03 | 0.01 | 0.27 | 0.16 | 0.2
gpt-4 |  1 | 34.16 | 1,970 | 467 | – | 0.58 | 0.32 | 0.15 | 0.08 | 0.04 | 0.29 | 0.24 | 0.26
gpt-4 |  2 | 22.54 | 2,772 | 297 | – | 0.74 | 0.22 | 0.15 | 0.11 | 0.08 | 0.53 | 0.32 | 0.4
gpt-4 |  3 | 9.29 | 4,356 | 94 | – | 0.52 | 0 | 0 | 0 | 0 | 0.57 | 0.13 | 0.22
gpt-4-1106-preview |  0 | 36.93 | 375 | 518 | – | 0.65 | 0.37 | 0.16 | 0.05 | 0.02 | 0.24 | 0.23 | 0.24
gpt-4-1106-preview |  1 | 37.32 | 1,970 | 611 | – | 0.64 | 0.41 | 0.18 | 0.08 | 0.03 | 0.26 | 0.29 | 0.27
gpt-4-1106-preview |  2 | 37.19 | 2,772 | 637 | – | 0.7 | 0.44 | 0.21 | 0.11 | 0.06 | 0.27 | 0.31 | 0.29
gpt-4-1106-preview |  3 | 24.24 | 3,574 | 409 | – | 0.67 | 0.31 | 0.15 | 0.08 | 0.05 | 0.3 | 0.25 | 0.27

##  5 Comparative Analysis

Generated Text for ’gpt-3.5-turbo-16k-0613’ model with two examples provided:

2\. Introduction

2.1 Study Rationale

Type 2 diabetes (T2D) is a metabolic disorder that is commonly associated with
comorbidities such as obesity, hypertension, and dyslipidemia. These
comorbidities increase the risk of chronic diabetic complications and make it
challenging to achieve adequate glycemic control (Garber 2012; Neeland et al.
2019; Dwivedi et al. 2020). Therefore, there is a need for therapeutic
interventions that can simultaneously target T2D and obesity to address this
unmet medical need (Lau and Teoh 2015; Aroda 2018; Pereira and Eriksson 2019;
Chun and Butts 2020). LY3209590 is a long-acting insulin, specifically a basal
insulin Fc-conjugated (basal insulin-FC), that is being developed to improve
glycemic control in patients with T2D, including those with comorbid obesity
and insulin-requiring patients (Eli Lilly). It is designed to provide a
prolonged and consistent release of insulin to maintain stable blood glucose
levels throughout the day.

2.2 Background

T2D is characterized by impaired glycemic control due to insulin resistance in
peripheral tissues and inadequate insulin secretion by beta cells (Zheng et
al. 2018). The use of insulin therapy is often necessary in the management of
T2D, especially in patients who cannot achieve glycemic control with oral
antidiabetic medications alone (Eli Lilly). The primary objective of this
phase 2, randomized, open-label trial is to evaluate the safety and efficacy
of LY3209590 in participants with T2D, including those with comorbid obesity
and insulin-requiring patients. The study will assess the impact of LY3209590
on body mass index, hemoglobin A1c percentage, and hypoglycemia incidence.
Participants will receive LY3209590 in combination with other antidiabetic
medications, including metformin, to evaluate its effectiveness in real-world
clinical practice.

2.3 Benefit/Risk Assessment

The safety and efficacy of LY3209590 have been assessed in preclinical and
early clinical trials. The known and expected benefits and risks of LY3209590,
including adverse events related to insulin therapy, will be evaluated in this
trial. The benefits of LY3209590 include improved glycemic control and
potential weight management benefits due to its Fc-conjugated formulation. The
risks associated with LY3209590 are primarily related to hypoglycemia and
adverse events commonly observed with insulin therapy. Based on the available
data, including data from early clinical trials and the known safety profile
of insulin therapy, the potential benefits of LY3209590 outweigh the
anticipated risks. The trial protocol includes measures to minimize the risk
to participants, such as regular monitoring of blood glucose levels and
appropriate adjustment of insulin dosage based on individual needs.

Overall, this phase 2 trial aims to further evaluate the safety and efficacy
of LY3209590 in a larger population of patients with T2D. The results of this
trial will provide valuable information to support the continued development
of LY3209590 in phase 3 clinical trials.

3\. Study Design

3.1 Overall Design

This trial is a phase 2, randomized, open-label trial to evaluate the safety
and efficacy of LY3209590 in participants with T2D. The trial will include
participants with comorbid obesity and insulin-requiring patients who have
inadequate glycemic control despite treatment with other antidiabetic
medications. Participants will be randomly assigned to receive either
LY3209590 or standard-of-care therapy. The primary endpoints of the trial
include changes in body mass index, hemoglobin A1c percentage, and
hypoglycemia incidence. Secondary endpoints include changes in fasting plasma
glucose levels, postprandial glucose excursions, and patient-reported
outcomes.

Participants will receive LY3209590 in combination with other antidiabetic
medications, including metformin. The study duration is anticipated to be
several months to allow for adequate evaluation of the short-term and long-
term efficacy and safety of LY3209590. The trial design incorporates measures
to minimize bias, including randomization, blinding of outcome assessors, and
appropriate statistical analysis. Safety evaluations will be conducted
throughout the trial period to monitor for adverse events and ensure
participant well-being.

Overall, this study design aims to provide robust evidence on the safety and
efficacy of LY3209590 in a real-world clinical setting, further informing its
future clinical development.

(Note: The additional sections of the protocol, such as the study procedures,
statistical analysis plan, and ethical considerations, have not been provided.
These sections would typically be included in the introduction of a clinical
trial protocol.

Actual introduction section:

Study Rationale

LY3209590 is a long-acting insulin receptor agonist with a half-life enabling
weekly dosing developed for the treatment of type 2 diabetes mellitus (T2DM).
This Phase 2 study of LY3209590 will: Evaluate the effects of LY3209590 on
glycemic control using 2 different dose individualization strategies compared
with insulin degludec in study participants previously treated with basal
insulin.

3.2. Background

LY3209590 was designed as a novel, once-weekly, long-acting insulin receptor
agonist that is intended for the treatment of hyperglycemia in study
participants with T2DM. A once weekly insulin receptor agonist has the
potential to increase willingness to initiate and compliance with an insulin
therapy in study participants with T2DM. In real life only about a quarter of
these study participants reach the target HbA1c of 7% (Blonde L et al. 2014)
and a once weekly insulin receptor agonist could help to increase this number.
The potential downside of such a long acting insulin receptor agonist could be
a higher risk for hypoglycemic events or a longer duration of such events. The
recently launched long acting insulin degludec has nevertheless shown a
reduced potential to induce hypoglycemia as compared to insulin glargine while
its half-life is twice as long as that of insulin glargine (Heise et al.
2012). The current phase 2 study will therefore evaluate the safety and
efficacy of LY3209590 as compared to insulin degludec while very closely
monitoring glycemic profiles using continuous glucose monitoring.

Data from the single-ascending dose (SAD) study demonstrated clear evidence of
glucose lowering in study participants with T2DM at single doses ranging from
CCI . The pharmacokinetics (PK) of LY3209590 following single doses
demonstrated prolonged time- action profile to support once-weekly
administration. LY3209590 reached maximum concentration approximately after
dosing, followed by a mean elimination half-life of approximately in study
participants with T2DM. With a long elimination half-life, following a weekly
fixed-dose regimen, PK steady-state is reached in approximately CCI and
results in approximately r concentration than after single dose. However,
using a loading dose strategy that is, on PK modeling. PK following a single
loading dose during the first week was demonstrated to be comparable to
concentration profile at Week based on preliminary data in the multiple
ascending dose (MAD) study. No clinically significant persistence in
hypoglycemia was observed in the study. Besides the known risk of
hypoglycemia, no relevant safety signals have been observed in the Phase 1
studies. Additional information about LY3209590 can be found in the
Investigator’s Brochure.

The toxicity profile of LY3209590 has been characterized in 6-week and 6-month
repeat-dose toxicology studies in rats and dogs, and reproductive toxicology
studies in rats and rabbits. CCI, CCI, CCI, LY3209590, CCI, CCI, CCI, CCI
I8H-MC-BDCM. These studies demonstrated a familiar spectrum of effects
typically associated with changes secondary to hypoglycemia and/or
hyperinsulinemia resulting from repeat dosing of exogenous insulin in
normoglycemic test systems. All findings were considered to be target-related
with no evidence of off-target effects in any of the parameters assessed.
Additional information about LY3209590 can be found in the Investigator’s
Brochure.

3.3. Benefit/Risk Assessment

The data from the SAD study and the ongoing MAD study (Studies BDCA and BDCB)
have shown that LY3209590 was well tolerated and the adverse drug reactions
are in line with those reported for long acting insulins.

Potential risks associated with LY3209590, derived from the known risks of
long acting insulins are hypoglycemia, hypersensitivity reactions (localized
allergy and/or systemic allergy), undesirable effects at the injection site
(injection-site reactions and lipodystrophy), and peripheral edema. An
additional risk based on preclinical data could be an elevation of liver
aminotransferases. To date, this has not been observed in study participants
or healthy volunteers exposed to LY3209590 for up to 6 weeks.

More information about the known and expected benefits, risks, Serious Adverse
Events (SAE) and reasonably anticipated AEs of LY3209590 are to be found in
the Investigator’s Brochure.

##  6 Discussion

As can be seen from the model results, providing examples to the prompt
significantly enhances the outcome and quality of text generation. It helps
with following the same format, writing style and inclusion of certain
information in the introduction section. Additionally, it can be observed that
gpt-4 and gpt-3.5-turbo-16k-063 significantly outperform other models in
almost all metrics measured. It can be observed that inclusion of two examples
is the optimal number of examples for the prompt to understand the writing
style and to learn what to include in the introduction section.

Inclusion of relevant examples to the prompt (metadata and introduction
section pair from the same Sponsor and at the same Trial Phase as the current
study design) significantly improved the accuracy of text generation and
resulted in cosine similarity of up to 0.8 and rouge precision of up to 0.7
which indicates conformance to the desired writing style.

As an enhancement to the Intelligent Clinical Trials (ICT) platform, this
feature can be simply integrated with the application and previous metadata
and protocol sections can be provided to the gpt LLM models for protocol
generation purposes.

###  6.1 Challenges and resolutions

The initial use of T5 models highlighted the challenges in generating long and
contextually rich content. Adjustments were made to focus on the GPT-4 model
and prompt engineering instead of model training, which significantly improved
the output quality for the Introduction section.

###  6.2 Accomplishments

The following achievements were accomplished as part of this research:

  1. 1.

40 Protocols Analyzed: Focused on Type II Diabetes, with specific criteria for
selection.

  2. 2.

Scraping and Collection: Efficient extraction of Introduction and Study Design
sections for all protocols.

  3. 3.

Comprehensive Metadata Collection: Gathering detailed metadata for each
protocol, enriching the training dataset from trial trove data repository.

  4. 4.

Drug Metadata Compilation: Obtaining data for 20 commonly referenced drugs in
the protocols.

  5. 5.

LLM Script Refinement: Adjusting existing LLM scripts to align with the
project’s requirements.

  6. 6.

Prompt Engineering and Evaluation: Appropriate gpt prompt has been developed
along with extensive experimentation and evaluation metrics for performance in
generation.

  7. 7.

Robust Text Generation: Introduction section of clinical protocol has been
generated using gpt models with high accuracy and reasonable performance
metrics assisting clinical trials designers with their manual task of protocol
writing.

Future efforts may include expanding the range of sections automated, refining
the model’s accuracy, and exploring further integrations within clinical
research frameworks.

##  7 Conclusion

The "Intelligent Clinical Trial Design Support using Generative AI" project
stands as a testament to the transformative potential of AI in clinical
research. The strategic use of generative AI models like GPT-4 not only
streamlined the protocol development process but also opened new avenues for
innovation in clinical trial design. The project demonstrated the immense
potential of generative AI in the realm of clinical trial design. The
successful integration of AI-generated protocol sections represents a
significant leap forward in optimizing the efficiency and accuracy of clinical
trial protocols. Future endeavors may explore broader applications of this
technology, further revolutionizing the field of clinical research.

## References

  * Dai et al. [2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  _arXiv preprint arXiv:1901.02860_ , 2019.
  * Dathathri et al. [2019] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.  Plug and play language models: A simple approach to controlled text generation.  _arXiv preprint arXiv:1912.02164_ , 2019.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Keskar et al. [2019] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_ , 2019.
  * Lester et al. [2021] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_ , 2021.
  * Maleki and Khan [2023] M Maleki and M Khan.  Covid-19 health equity & justice dashboard: A step towards countering health disparities among seniors and minority population.  _Social Science Research Network_ , 2023.
  * Maleki et al. [2022] Morteza Maleki, Mohsen Bahrami, Monica Menendez, and Jose Balsa-Barreiro.  Social behavior and covid-19: Analysis of the social factors behind compliance with interventions across the united states.  _International journal of environmental research and public health_ , 19(23):15716, 2022.
  * Petroni et al. [2019] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel.  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_ , 2019.
  * Song et al. [2019] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  Mass: Masked sequence to sequence pre-training for language generation.  _arXiv preprint arXiv:1905.02450_ , 2019.