# Sensitivity Analyses of Clinical Trial Designs: Selecting Scenarios and
Summarizing Operating Characteristics

Larry Han1, Andrea Arfè2, Lorenzo Trippa1,3

1 Department of Biostatistics,
Harvard T.H. Chan School of Public Health

2 Department of Epidemiology and Biostatistics,
Memorial Sloan Kettering Cancer Center

3 Department of Biostatistics and Computational Biology,
Dana-Farber Cancer Institute

###### Abstract

The use of simulation-based sensitivity analyses is fundamental to evaluate
and compare candidate designs for future clinical trials. In this context,
sensitivity analyses are especially useful to assess the dependence of
important design operating characteristics (OCs) with respect to various
unknown parameters (UPs). Typical examples of OCs include the likelihood of
detecting treatment effects and the average study duration, which depend on
UPs that are not known until after the onset of the clinical study, such as
the distributions of the primary outcomes and patient profiles. Two crucial
components of sensitivity analyses are (i) the choice of a set of plausible
simulation scenarios
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ and (ii) the list
of OCs of interest. We propose a new approach to choose the set of scenarios
for inclusion in design sensitivity analyses. Our approach balances the need
for simplicity and interpretability of OCs computed across several scenarios
with the need to faithfully summarize—through simulations—how the OCs vary
across all plausible values of the UPs. Our proposal also supports the
selection of the number of simulation scenarios to be included in the final
sensitivity analysis report. To achieve these goals, we minimize a loss
function $\mathcal{L}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})$
that formalizes whether a specific set of $K$ sensitivity scenarios
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ is adequate to
summarize how the OCs of the trial design vary across all plausible values of
the UPs. Then, we use optimization techniques to select the best set of
simulation scenarios to exemplify the OCs of the trial design.

Keywords: Clinical trial design; operating characteristics; sensitivity
analysis; function approximation; simulated annealing.

##  1 Introduction

Clinical trial designs are becoming increasingly complex to meet the
multifaceted needs and goals of precision medicine. Examples of complex
designs include adaptive seamless phase i/ii designs for evaluating, early in
the treatment development process, the dosing, safety, and activity of new
drugs (Hobbs et al., 2019). Also, adaptive randomized trials with frequent
interim looks at the data can evaluate one or more therapies simultaneously
while attempting to minimize trial duration and resources (Thorlund et al.,
2018; Berry et al., 2010). Additional examples of complex designs have been
implemented in biomarker-stratified trials to evaluate the efficacy of a
therapy and possible variations of treatment effects across patient subgroups
(Mehta et al., 2019).

When planning a new trial, it is necessary to predict and evaluate several
operating characteristics (OCs) $\mathbf{f}=(f_{1},\ldots,f_{R})$. Relevant
OCs can include the likelihood of selecting an effective dose with low
toxicity in a phase i/ii study, the probability of detecting treatment effects
in a randomized study, the expected trial duration, costs, and other metrics
to evaluate designs that often enroll patients from different subgroups.
Multiple OCs typically need to be examined jointly in order to evaluate the
relevant trade-offs achieved by candidate designs, such as balancing the
accuracy in estimating treatment effects and the expected study duration.

The obvious challenge for evaluating a candidate design is that the value of
$\mathbf{f}$, the vector of OCs of the study design, is not known, and it is
difficult to estimate before the onset of the trial. Indeed, $\mathbf{f}$ is
usually a function $\mathbf{f}=\mathbf{f}(\boldsymbol{\theta})$ of a vector
$\boldsymbol{\theta}=(\theta_{1},\ldots,\theta_{d})$ that contains unknown
parameters (UPs) which identify the distribution of all relevant variables
that will be captured during the trial. For example, UPs can include the
enrollment and drop-out rates, the magnitude of treatment effects, and the
prevalence of predictive biomarkers in the trial population. Uncertainty on
$\boldsymbol{\theta}$ makes it non-trivial to evaluate whether a candidate
design is appropriate for implementing the new study.

Sensitivity analyses are commonly used to account for uncertainty on UPs
$\boldsymbol{\theta}$ and OCs $\mathbf{f}(\boldsymbol{\theta})$ when
evaluating a candidate design. They typically proceed in three steps. First, a
set of plausible scenarios
$\\{\boldsymbol{\theta}_{1},\ldots,\boldsymbol{\theta}_{K}\\}$, i.e., specific
values of the vector $\boldsymbol{\theta}$ of UPs, is selected. Next, the
corresponding set of OCs are computed
$\\{\boldsymbol{\mathbf{f}}(\boldsymbol{\theta}_{1})$, $\ldots$,
$\boldsymbol{\mathbf{f}}(\boldsymbol{\theta}_{K})\\}$ using trial simulations
or analytic results. Finally, based on the computed OCs and their variations
across the set of $K$ scenarios, the investigators evaluate if the candidate
design is appropriate to achieve the aims of the study. Throughout the
manuscript, we use the terms sensitivity analysis or sensitivity report to
indicate a set of $K$ scenarios
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ and the associated
OCs $\\{\boldsymbol{\mathbf{f}}(\boldsymbol{\theta}_{1})$, $\ldots$,
$\boldsymbol{\mathbf{f}}(\boldsymbol{\theta}_{K})\\}$ which are computed to
illustrate how the OCs $\mathbf{f}(\boldsymbol{\theta})$ vary across plausible
values of UPs $\boldsymbol{\theta}$.

Producing a sensitivity report to effectively evaluate a study design presents
several challenges. Indeed, it can be difficult to select the set of UPs
$\\{\boldsymbol{\theta}_{1},\ldots,\boldsymbol{\theta}_{K}\\}$, especially if
the dimension $d$ of the vector $\boldsymbol{\theta}$ is moderate to high (say
$d\geq 5$). For the investigators, it might be unclear if the selected
scenarios are adequate to illustrate the variations of the OCs $\mathbf{f}$
across potential values of the UPs $\boldsymbol{\theta}$. Similarly, for
regulators, there may be skepticism as to whether the selected scenarios are
chosen to highlight positive aspects of the trial design without pointing at
its limitations and negative aspects (Razavi et al., 2021). Another subtle
challenge is the choice of the number of scenarios $K$. Indeed, a large number
of scenarios (say $K=100$) may simplify the task of representing how the OCs
$\mathbf{f}(\boldsymbol{\theta})$ vary across potential values of the UPs
$\boldsymbol{\theta}$, but a sensitivity report that contains too many
scenarios makes it difficult to interpret and communicate the included
results.

A common approach used to define the set of scenarios
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ is to vary a
single entry of $\boldsymbol{\theta}$ (say $\theta_{1}$) while fixing the
other UPs to some reference values $\hat{\theta}_{2},...,\hat{\theta}_{d}$,
which might be estimated from previous studies. In this case, the set of
scenarios becomes
$\\{\boldsymbol{\theta}_{1}=({\theta_{1,1},\hat{\theta}_{2},...,\hat{\theta}_{d}})),...,{\boldsymbol{\theta}_{K}}=({\theta_{K,1},\hat{\theta}_{2},...,\hat{\theta}_{d}})\\}$.
Similar perturbations of $\boldsymbol{\hat{\theta}}$ can be repeated for the
other entries of the vector. However, this approach can be inadequate if the
relation between one UP ($\theta_{1}$, say) and the OCs changes when we
consider distinct values of the other UPs ($\theta_{2},...,\theta_{d}$). In
such cases, this approach compromises the possibility of illustrating with
sufficient accuracy how the OCs $\mathbf{f}(\boldsymbol{\theta})$ vary across
plausible values of the UPs $\boldsymbol{\theta}$, for example through a table
of OCs
$\mathbf{f}(\boldsymbol{\theta}_{1}),...,\mathbf{f}(\boldsymbol{\theta}_{K})$
computed for a representative set of scenarios
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$.

We propose a method to choose an optimal set of scenarios
$\\{\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}\\}$ for a
sensitivity report that will provide relevant OCs
$\\{\mathbf{f}(\boldsymbol{\theta}_{1}^{*}),...,\mathbf{f}(\boldsymbol{\theta}_{K}^{*})\\}$.
This decision is based on a formal utility criterion $\mathcal{U}$. The
function
$\mathcal{U}:\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}\mapsto\mathbb{R}$
formalizes the ability of any set of scenarios
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ to represent the
map $\boldsymbol{\Theta}\rightarrow{\mathbf{f}}(\boldsymbol{\Theta})$, where
$\boldsymbol{\Theta}$ is the parameter space. In some cases, we will consider
a restriction $\boldsymbol{\Theta}^{\prime}\subset\boldsymbol{\Theta}$ to
focus only on plausible values of the UPs $\boldsymbol{\theta}$. This utility
criterion $\mathcal{U}$ assigns high (low) utility to
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ if the table of
potential UPs and OCs
$[(\boldsymbol{\theta}_{1},\boldsymbol{\mathbf{f}}(\boldsymbol{\theta}_{1}))$,
$\ldots$,
$(\boldsymbol{\theta}_{K},\boldsymbol{\mathbf{f}}(\boldsymbol{\theta}_{K}))]$
can be viewed as an accurate (inaccurate) summary of how the design’s OCs vary
across $\boldsymbol{\Theta}$ or the restricted parameter space
$\boldsymbol{\Theta}^{\prime}$. We call the set of scenarios that maximizes
$\mathcal{U}$ the Representative and Optimal Sensitivity Analysis (ROSA)
scenarios. To select ROSA scenarios
$\\{\boldsymbol{\theta}_{1}^{*},\ldots,\boldsymbol{\theta}_{K}^{*}\\}$, we
introduce a computational procedure that leverages (i) flexible regression
methods like neural networks (NNs) (Goodfellow et al., 2016) and (ii)
optimization algorithms like simulated annealing (SA) (Bélisle, 1992). Our
approach is applicable to any trial design, regardless of the number $d$ of
UPs in $\boldsymbol{\theta}$ and the number $R$ of OCs in
$\mathbf{f}(\boldsymbol{\theta})$.

To illustrate the method, we conduct sensitivity analyses for three trial
designs. The first is a two-arm randomized design that aims to test and
estimate the effects of an experimental treatment compared to the standard of
care (SOC). The second is a multi-stage randomized trial that leverages an
auxiliary outcome $S$ measured shortly after randomization for interim
decisions and a primary outcome $Y$ with a longer ascertainment time (Niewczas
et al., 2019). The third is a biomarker-adaptive enrichment design similar to
the design of the TAPPAS trial (Mehta et al., 2019), a randomized phase iii
trial comparing TRC105 and pazopanib versus pazopanib alone in patients with
advanced angiosarcoma (Jenkins et al., 2011; Jones et al., 2017). In the first
design, we consider a single UP and a single OC, whereas the latter two
designs consider multiple UPs and multiple OCs.

##  2 Selecting sensitivity scenarios

We introduce our procedure to select $K$ sensitivity scenarios
$\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\in\boldsymbol{\Theta}$,
where $\boldsymbol{\Theta}$ is the set of potential values of the UPs
$\boldsymbol{\theta}$. We assume that $\boldsymbol{\Theta}$ is a bounded
subset of $\mathbb{R}^{d}$ and use the notation $||\cdot||_{2}$ to indicate
the Euclidean norm on $\mathbb{R}^{d}$. We will restrict $\boldsymbol{\Theta}$
to a subset $\boldsymbol{\Theta}^{\prime}$ when there is sufficient prior
information from completed studies or clinical experience.

###  2.1 Optimization of a utility criterion

We identify ROSA scenarios
$\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}$ as the scenarios
that maximize a utility criterion $\mathcal{U}$

$\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}=\mathop{\rm\arg\\!\max}_{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}}\text{
}\mathcal{U}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}),$ (1)

where

$\mathcal{U}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})=-\max_{\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}}\left\\{\min_{k=1,...,K}D[\mathbf{f}(\boldsymbol{\theta}^{\prime}),\mathbf{f}(\boldsymbol{\theta}_{k})]\right\\}.$
(2)

We can symmetrically define the corresponding loss function
$\mathcal{L}=-\mathcal{U}$ by inverting the sign in equation (2). Here,
$D[\mathbf{f}(\boldsymbol{\theta}^{\prime}),\mathbf{f}(\boldsymbol{\theta}_{k})]$
is a metric between the OCs
$\mathbf{f}(\boldsymbol{\theta}^{\prime})=(f_{1}(\boldsymbol{\theta}^{\prime}),\ldots
f_{R}(\boldsymbol{\theta}^{\prime}))$ and
$\mathbf{f}(\boldsymbol{\theta}_{k})=(f_{1}(\boldsymbol{\theta}_{k},)\ldots,f_{R}(\boldsymbol{\theta}_{k}))$.
We will consider metrics of the form

$D[{\textbf{f}}(\boldsymbol{\theta}^{\prime}),{\textbf{f}}\left(\boldsymbol{\theta_{k}})\right]=\sum_{r=1}^{R}w_{r}||f_{r}(\boldsymbol{\theta}^{\prime})-f_{r}(\boldsymbol{\theta}_{k})||_{2},$

where $w_{1},\ldots,w_{R}$ are non-negative weights that sum to one.

We can now provide an explicit interpretation of the utility function
$\mathcal{U}$ in equation (2) and of the loss function
$\mathcal{L}=-\mathcal{U}$. Consider a set of scenarios
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ – the order of the
entries is not relevant – and $\boldsymbol{\theta}^{\prime}$ in
$\boldsymbol{\Theta}$. For $1\leq k\leq K$, the metric
$D[\mathbf{f}(\boldsymbol{\theta}^{\prime}),\mathbf{f}(\boldsymbol{\theta}_{k})]$
is a summary of the differences between the OCs $\mathbf{f}$ at
$\boldsymbol{\theta}^{\prime}$ and the same OCs when we consider the $k$-th
scenario $\boldsymbol{\theta}_{k}$. Therefore,
$\min_{k=1,...,K}D[\mathbf{f}(\boldsymbol{\theta}^{\prime}),\mathbf{f}(\boldsymbol{\theta}_{k})]$
can be viewed as an approximation error between
$\mathbf{f}(\boldsymbol{\theta}^{\prime})$ and a similar vector of OCs
selected among our $K$ options
$\mathbf{f}(\boldsymbol{\theta}_{1}),...,\mathbf{f}(\boldsymbol{\theta}_{K})$.
Expression (2) identifies through the maximization operator the worst-case,
with highest approximation error, that we can obtain by varying
$\boldsymbol{\theta}^{\prime}$ in $\boldsymbol{\Theta}$. We maximize the
utility function $\mathcal{U}$ and use
$\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}$ to indicate the
ROSA scenarios. Alternative utility criteria and loss functions are described
later in the manuscript.

To provide a geometric interpretation of the loss function $\mathcal{L}$, we
illustrate how one set of $K$ scenarios can be preferable to a different set
of $K$ scenarios (Figure 1). Specifically, suppose we aim to design a single-
arm trial with an interim analysis that allows for early-stopping for
futility. The goal of the trial is to compare the response rate of an
experimental drug $\theta_{1}$ with that of the SOC $\theta_{0}$ at the end of
the study. However, because study patients only receive the experimental drug,
the response rate under the SOC $\theta_{0}$ is estimated
($\widehat{\theta}_{0}$) before the onset of the study, for example using data
from a previous trial. At the interim analysis, the trial may stop for
futility if the preliminary evidence of positive treatment effects
$\Delta_{interim}$ is insufficient to continue the study. During the final
analysis, the null hypothesis $H_{0}:\theta_{1}\leq\widehat{\theta}_{0}$ (the
experimental therapy is not superior to the historical control) is tested
against the alternative hypothesis $H_{1}:\theta_{1}>\widehat{\theta}_{0}$
(the experimental therapy is superior to the historical control). The
discrepancy between $\theta_{0}$ and $\widehat{\theta}_{0}$ could impact the
OCs, such as the probability of stopping the study for futility (Vanderbeek et
al., 2019). In this design, $\boldsymbol{\theta}=(\theta_{0},\theta_{1})$ are
the UPs, and $\boldsymbol{\Theta}=[0,1]^{2}$. Suppose that there are two OCs
of interest: (i) $f_{1}$, power, defined as the probability of a true positive
result when the experimental drug has beneficial effects compared to the SOC
($f_{1}$ is equal to zero if the treatment effects are null or negative) and
(ii) $f_{2}$, the expected sample size.

Figure 1: Geometric representation of an arbitrary scenario
$\boldsymbol{\theta}^{\prime}$ and two proposed sets of scenarios. (Left)
Parameter space $\boldsymbol{\Theta}=[0,1]^{2}$ with arbitrary scenario
$\boldsymbol{\theta}^{\prime}$ (orange triangle) and two sets of proposed
scenarios $\\{\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{6}^{1}\\}$
(blue points) and
$\\{\boldsymbol{\theta}_{1}^{2},...,\boldsymbol{\theta}_{6}^{2}\\}$ (red
points). (Right) OC surface $\mathbf{f}(\boldsymbol{\Theta})$ with the
corresponding OCs for $\boldsymbol{\theta}^{\prime}$ and the two proposed sets
of scenarios. The radius of the dotted circles (with blue points as centers)
is the value of the loss $\mathcal{L}$ associated with the blue points. ROSA
scenarios minimize the loss $\mathcal{L}$, which in turn is equal to the
radius of the dotted circles that cover the OC surface
$\mathbf{f}(\boldsymbol{\Theta})$.

The left panel of Figure 1 is a representation of $\boldsymbol{\Theta}$. We
are interested in the two OCs of the single-arm design. Two sets of $K=6$
scenarios are proposed. The first set of scenarios
$\\{\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{6}^{1}\\}$ (blue
points) is chosen by varying both UPs at the same time, while the second set
$\\{\boldsymbol{\theta}_{1}^{2},...,\boldsymbol{\theta}_{6}^{2}\\}$ (red
points) is chosen by varying only $\theta_{0}$ while fixing the value of
$\theta_{1}$. The two sets of scenarios, the corresponding OCs, and associated
loss $\mathcal{L}$ are represented in the right panel of Figure 1. The first
set of scenarios (blue points) is preferred over the second set of scenarios
(red points) because it is more representative of the variation of the OCs
over $\boldsymbol{\Theta}$. Geometrically, the loss
$\mathcal{L}(\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{6}^{1})$
associated with the blue points is identical to the minimum radius of the
circles with centers
$\mathbf{f}(\boldsymbol{\theta}_{1}^{1}),...,\mathbf{f}(\boldsymbol{\theta}_{6}^{1})$
(see Figure 1) necessary to cover the OCs surface
$\mathbf{f}(\boldsymbol{\Theta})$.

###  2.2 Estimating the OCs

We describe an algorithm to numerically approximate the OCs
$\mathbf{f}(\boldsymbol{\theta})$ for every
$\boldsymbol{\theta}\in\boldsymbol{\Theta}$. This is necessary to solve the
optimization problem in equation (2). Indeed, in most cases the function
$\mathbf{f}(\boldsymbol{\theta})$ cannot be computed in closed form.

We briefly outline our four-step procedure. In the first step, we choose a
large number $J$ (say $J=1000$) of training scenarios
$\boldsymbol{\theta}_{1}^{t},...,\boldsymbol{\theta}_{J}^{t}.$ In the second
step, we use Monte Carlo simulations to obtain estimates
$\bar{\mathbf{f}}(\boldsymbol{\theta}_{1}^{t})$, …,
$\bar{\mathbf{f}}(\boldsymbol{\theta}_{J}^{t})$ of
$\mathbf{f}(\boldsymbol{\theta}_{1}^{t})$, …,
$\mathbf{f}(\boldsymbol{\theta}_{J}^{t})$. In the third step, we train a
flexible regression model – we use NNs in our implementation – based on the
data points
$(\boldsymbol{\theta}_{1}^{t},\bar{\mathbf{f}}(\boldsymbol{\theta}_{1}^{t}))$,
…,
$(\boldsymbol{\theta}_{J}^{t},\bar{\mathbf{f}}(\boldsymbol{\theta}_{J}^{t}))$.
The output of this step is a regression function
$\hat{\mathbf{f}}(\boldsymbol{\theta})$ that is easy to compute at any
$\boldsymbol{\theta}\in\boldsymbol{\Theta}$ and that approximates
$\mathbf{f}(\boldsymbol{\theta})$. In the fourth step, we validate the
regression model based on $J^{\prime}$ (say $J^{\prime}=200$) independent
simulations
$(\boldsymbol{\theta}_{1}^{v},\bar{\mathbf{f}}(\boldsymbol{\theta}_{1}^{v}))$,
…,
$(\boldsymbol{\theta}_{J^{\prime}}^{v},\bar{\mathbf{f}}(\boldsymbol{\theta}_{J^{\prime}}^{v}))$.
Steps 1-3 of this procedure are summarized in Algorithm 1. Step 4 is described
in Algorithm 2.

In more detail, in step 1, to select the training scenarios
$\boldsymbol{\theta}_{1}^{t},...,\boldsymbol{\theta}_{J}^{t}$, we randomly
select $J$ scenarios in $\boldsymbol{\Theta}$ using Latin hypercube sampling
(LHS) (McKay et al., 2000; Carnell and Carnell, 2016). LHS generates $J$
scenarios by first partitioning the $d$ UP dimensions into $J$ non-overlapping
intervals and selecting one value from each interval at random. The $J$ values
obtained for the first UP ${\theta}_{1}$ are randomly paired with the $J$
values obtained for the second ${\theta}_{2}$, and so on, for all $d$ UPs to
form $J$ $d$-tuples, which constitute the training scenarios
$\boldsymbol{\theta}_{1}^{t},...,\boldsymbol{\theta}_{J}^{t}$.

In step 2, we estimate the OCs of the trial design. For simplicity, we
consider OCs defined as expected values (e.g., bias, power, duration of the
trial, etc.), but the algorithm can be easily modified to consider other OCs.
Specifically, we assume that
${\mathbf{f}}(\boldsymbol{\theta})=\mathbb{E}_{\boldsymbol{\theta}}[\boldsymbol{\varphi}(Z,\boldsymbol{\theta})]$
for some function $\boldsymbol{\varphi}$, where the random vector $Z$
represents the data generated during the trial – including the collection of
treatment assignment indicators and realized patient outcomes – under scenario
$\boldsymbol{\theta}$. For example, $\boldsymbol{\varphi}$ can be the
indicator that captures if a null hypothesis of interest has been correctly
rejected at the end of the study, or the duration of the simulated trial. In
practice, to estimate ${\mathbf{f}}(\boldsymbol{\theta})$, we proceed as
follows. First, for each of the training scenarios
$\boldsymbol{\theta}_{j}^{t}$, $1\leq j\leq J$, we simulate $M$ (say $M=200$)
clinical trials following the trial design. We then use the $M$ scenario-
specific simulated trials to compute the estimate

$\bar{\mathbf{f}}(\boldsymbol{\theta}_{j}^{t})=M^{-1}\sum_{m=1}^{M}\boldsymbol{\varphi}(Z_{j,m},\boldsymbol{\theta}_{j}^{t}),\quad
1\leq j\leq J,$

where $Z_{j,m}$ is the $m^{th}$ trial dataset simulated under the $j^{th}$
training scenario $\boldsymbol{\theta}_{j}^{t}$.

In step 3, we have only two inputs, the scenarios
$\boldsymbol{\theta}_{j}^{t}$ and the estimates
$\bar{\mathbf{f}}(\boldsymbol{\theta}_{j}^{t})$, $1\leq j\leq J$, to fit a
function $\hat{\mathbf{f}}(\boldsymbol{\theta})$. For example, one could use
NNs, splines (Bookstein, 1989), or Gaussian processes (Rasmussen, 2003). We
use NN regression functions in our applications because these are easy to
compute using widely available software and have been demonstrated to have
good empirical performance (Leshno et al., 1993; Hornik, 1991; Goodfellow et
al., 2016).

1 Input: Trial design, Parameter space $\boldsymbol{\Theta}$, $J$, $M$

2Step 1: Select $J$ scenarios
$\boldsymbol{\theta}_{1}^{t},...,\boldsymbol{\theta}_{J}^{t}\in\boldsymbol{\Theta}$

3

4Step 2: for  _$j=1$ to $J$_ do

5 Simulate $M$ trials

6 Obtain approximate OCs
$\boldsymbol{\bar{f}}(\boldsymbol{\theta}_{j}^{t})=M^{-1}\sum_{m=1}^{M}\boldsymbol{\varphi}(Z_{j,m},\boldsymbol{\theta}_{j}^{t})$,
where $\boldsymbol{\varphi}$ is a function of $Z_{j,m}$, the $m^{th}$ trial
dataset simulated under the $j^{th}$ scenario, and the corresponding parameter
$\boldsymbol{\theta}_{j}$

7 end for

8

9Step 3: Obtain an approximation of the OCs $\hat{\boldsymbol{f}}$ by training
a regression algorithm, for example a NN model, and using the data points
$(\boldsymbol{\theta}_{j}^{t},\boldsymbol{\bar{f}}(\boldsymbol{\theta}_{j}^{t}))$,
$1\leq j\leq J$

10

Output: Function $\hat{\mathbf{f}}(\boldsymbol{\theta})$

Algorithm 1 Obtaining a function $\boldsymbol{\hat{f}}$ that approximates the
OC function $\mathbf{f}$

In step 4 (Algorithm 2), we investigate the differences between
$\hat{\boldsymbol{f}}$ and $\boldsymbol{f}$. Specifically, we first select at
random $J^{\prime}$ validation scenarios
$\boldsymbol{\theta}_{1}^{v},...,\boldsymbol{\theta}_{J^{\prime}}^{v}$
independently with respect to previous computations (step 1-3) and simulate
$M^{\prime}$ trials (say $M^{\prime}=500$) for each
$\boldsymbol{\theta}_{j^{\prime}}^{v}$, $1\leq j^{\prime}\leq J^{\prime}$.
Based on the results of the simulated trials, for each $j^{\prime}$, we then
compute Monte Carlo estimates
$\bar{\mathbf{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})=M^{\prime-1}\sum_{m^{\prime}=1}^{M\prime}\boldsymbol{\varphi}(Z_{j^{\prime},m^{\prime}},\boldsymbol{\theta}_{j^{\prime}}^{v})$
of the OCs $\mathbf{f}(\boldsymbol{\theta}_{j^{\prime}}^{v})$. For several
important OCs (e.g., average sample size, expected duration, power, type 1
error), the estimator
$\bar{\mathbf{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})=M^{\prime-1}\sum_{m^{\prime}=1}^{M\prime}\boldsymbol{\varphi}(Z_{j^{\prime},m^{\prime}},\boldsymbol{\theta}_{j^{\prime}}^{v})$
is unbiased. Finally, we compare the estimates
$\bar{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$ and the
independent estimates
$\hat{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$. We use summary
statistics and graphs to evaluate the differences
$\hat{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})-\bar{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$.
If the approximation
$\hat{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$ is not adequate,
we can use a different regression methodology, increase the number
$(M,M^{\prime})$ of trials, or increase the number $J$ of training scenarios
in Algorithm 1.

1 Input: Approximation of the OCs $\hat{\boldsymbol{f}}$, Trial design,
$J^{\prime}$

2Randomly select $J^{\prime}$ scenarios
$\boldsymbol{\theta}_{1}^{v},...,\boldsymbol{\theta}_{J^{\prime}}^{v}\in\boldsymbol{\Theta}$
independently from previous computations (Algorithm 1)

3

4for  _$j^{\prime}=1$ to $J^{\prime}$_ do

5 Simulate $M^{\prime}$ trials $Z_{j^{\prime},m^{\prime}}$

6 Compute
$\bar{\boldsymbol{f}}(\boldsymbol{\theta_{j^{\prime}}^{v}})=M^{\prime-1}\sum_{m^{\prime}=1}^{M^{\prime}}\boldsymbol{\varphi}(Z_{j^{\prime},m^{\prime}},\boldsymbol{\theta}_{j^{\prime}}^{v})$

7 Compute $\hat{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$

8 end for

9 Output: Set of differences
$\hat{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})-\bar{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$
and scatterplots to jointly visualize the OC estimates
$\bar{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$ and the
independent estimates
$\hat{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$, $1\leq
j^{\prime}\leq J^{\prime}$. Compute summaries of the differences (e.g.,
median, range, or other descriptive statistics).

Interpretation: Differences between
$\bar{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$ and the
independent estimates
$\hat{\boldsymbol{f}}(\boldsymbol{\theta}_{j^{\prime}}^{v})$, $1\leq
j^{\prime}\leq J^{\prime}$, consistently close to zero provide evidence that
$\hat{\boldsymbol{f}}$ is an accurate approximation of $\boldsymbol{f}$

Algorithm 2 Validating the approximation of the OCs $\hat{\boldsymbol{f}}$

###  2.3 Approximating the Loss Function

After computing $\hat{\boldsymbol{f}}$ (Algorithm 1) and validating its
accuracy (Algorithm 2), we use it to approximate the loss function
$\mathcal{L}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})$. To
proceed, we choose a diffuse and finite subset of the parameter space
$\boldsymbol{\Theta}^{F}\subset\boldsymbol{\Theta}$. For example
$\boldsymbol{\Theta}^{F}$ can include 100,000 random points from a
distribution with support $\boldsymbol{\Theta}$. When
$\boldsymbol{\Theta}^{F}$ contains a large number of random points that are
distributed over $\boldsymbol{\Theta}$, under minimal assumptions (e.g.,
compact $\boldsymbol{\Theta}$ and OCs with bounded range),

$\displaystyle\mathcal{L}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})$
$\displaystyle=\max_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\left\\{\min_{k=1,...,K}D[\boldsymbol{f}(\boldsymbol{\theta}),\boldsymbol{f}(\boldsymbol{\theta}_{k})]\right\\}$
$\displaystyle\approx\max_{\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}^{F}}\left\\{\min_{k=1,...,K}D[\boldsymbol{\hat{f}}(\boldsymbol{\theta}^{\prime}),\boldsymbol{\hat{f}}(\boldsymbol{\theta}_{k})]\right\\}=\hat{\mathcal{L}}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}).$

To summarize, we can approximate the loss function
$\mathcal{L}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})$ over the
entire parameter space $\boldsymbol{\Theta}$ by
$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})$ using
a diffuse and finite subset $\boldsymbol{\Theta}^{F}$.

###  2.4 Optimization

We now aim to approximately minimize the loss function $\hat{\mathcal{L}}$. To
illustrate the need for approximate solutions, consider the setting of a
single UP $(d=1)$, a finite ${\boldsymbol{\Theta}}$, and an easy-to-compute
loss function $\mathcal{L}$. Even in this simple setting, identifying
$\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}\in\boldsymbol{\Theta}$
can be challenging. For example, to select $K=10$ representative scenarios
$\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}$ from $1000$
points $\\{\boldsymbol{\theta}_{j};1\leq j\leq 1000\\}=\boldsymbol{\Theta}$,
the loss function $\hat{\mathcal{L}}$ would need to be calculated for
$2.63\times 10^{23}$ different possible sets
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$. In what follows,
we describe the use of SA (Algorithm 3), a simple strategy to reduce the
outlined computational burden, regardless if $\boldsymbol{\Theta}$ is finite
or not Kirkpatrick et al. (1983); Bélisle (1992); Spall (2005).

The SA algorithm proceeds as follows. First, initial scenarios
$\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{K}^{1}$ are proposed,
for example by sampling
$\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{K}^{1}$ from a
probability distribution with support $\boldsymbol{\Theta}$. Then, iteratively
for $1\leq i\leq I$, the current scenarios
$\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i}$ are perturbed by
adding to them Gaussian noise variables
$\boldsymbol{z}_{1}^{i},...,\boldsymbol{z}_{K}^{i}$, thus obtaining new
proposed scenarios
$\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime}$ (this
step is represented by the “Perturb” operator in Algorithm 3). At each
iteration, the proposed scenarios
$\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime}$ can
either be accepted (i.e.,
$[\boldsymbol{\theta}_{1}^{i+1},...,\boldsymbol{\theta}_{K}^{i+1}]\leftarrow[\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime}]$)
or rejected (i.e.,
$[\boldsymbol{\theta}_{1}^{i+1},...,\boldsymbol{\theta}_{K}^{i+1}]\leftarrow[\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i}]$).
The acceptance or rejection of the proposed scenarios is stochastic, with
probability $\rho_{i}$ (defined below), which is a function of
$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime})$
and
$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i})$.

The acceptance probability $\rho_{i}$ is equal to $1$ when
$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime})<\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i})$.
That is, if the proposed scenarios decrease the current loss value, then the
proposed scenarios are accepted. If instead
$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime})\geq\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i})$,
then $\rho_{i}$ is equal to

$\exp\left(\frac{\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i})-\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime})}{T_{i}}\right),$

where $T_{i},$ $0\leq i\leq I$, is a decreasing sequence of positive real
numbers often called the “cooling schedule” of the algorithm. A common cooling
schedule is $T_{i}=T_{0}\cdot r^{i-1}$, where $T_{0}$ is a constant and
$r\in(0,1)$ is a multiplicative contraction, but other forms are possible
(Spall, 2005). In our applications, we use a piecewise-constant cooling
schedule (Husmann et al., 2017).

After simulating the outlined Markov Chain for a fixed number $I$ of
iterations, the final set of scenarios
$\\{\boldsymbol{\theta}_{1}^{I+1},...,\boldsymbol{\theta}_{K}^{I+1}\\}$
approximately minimizes the loss function $\hat{\mathcal{L}}$ (Bélisle, 1992).
In our ROSA implementation, we use multiple independent replicates of
Algorithm 3, with different initial scenarios
$\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{K}^{1}$, to investigate
convergence of the Markov chain. Intuitively, if the independent chains
converge, then the corresponding loss values of the approximate optima
$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{I+1},...,\boldsymbol{\theta}_{K}^{I+1})$
should be nearly identical.

1 Initialize the values of
$\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{K}^{1}$, e.g., by
sampling from a distribution over $\boldsymbol{\Theta}$

2 Best proposal
$\leftarrow\boldsymbol{\theta}_{1}^{1},...,\boldsymbol{\theta}_{K}^{1}$

3 for  _$i=1$ to $I$_ do

4 New proposal
$\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime}\leftarrow\textrm{Perturb}(\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i})$

5 if
_$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime})\leq\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i})$
_ then

6 Define $\boldsymbol{\theta}_{j}^{i+1}=\boldsymbol{\theta}_{j}^{\prime}$ for
every $j=1,\ldots,K$ ;

7

8 else

9 Compute the acceptance probability
$\rho_{i}=\exp\left([\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{i},...,\boldsymbol{\theta}_{K}^{i})-\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{\prime},...,\boldsymbol{\theta}_{K}^{\prime})]/T_{i}\right)$

10 Sample $U_{i}\sim\textrm{Uniform}(0,1)$

11 If $U_{i}\leq\rho_{i}$, define
$\boldsymbol{\theta}_{j}^{i+1}=\boldsymbol{\theta}_{j}^{\prime}$ for every
$j=1,\ldots,K$;

12 Otherwise $\boldsymbol{\theta}_{j}^{i+1}=\boldsymbol{\theta}_{j}^{i}$ for
every $j=1,\ldots,K$.

13

14 end for

Output: $\boldsymbol{\theta}_{1}^{I+1},...,\boldsymbol{\theta}_{K}^{I+1}$,
$\hat{\mathcal{L}}(\boldsymbol{\theta}_{1}^{I+1},...,\boldsymbol{\theta}_{K}^{I+1})$

Algorithm 3 Pseudocode for SA to obtain ROSA scenarios

##  3 Applications: Sensitivity Analyses of Trial Designs

We illustrate the ROSA approach by performing sensitivity analyses for three
designs of different complexity levels. In each example, we describe the
design of the trial, the UPs, and the OCs of interest.

In the first example, we will only consider a single UP (i.e.,
$\boldsymbol{\theta}\in\mathbb{R}$) and a single OC $f(\boldsymbol{\theta})$
that can be computed analytically. In this case, the optimal set of scenarios
$\\{\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}\\}$ can be
computed exactly, without resorting to approximation methods. This simple and
stylized setting is useful to highlight the similarity of the approximations
and selected scenarios computed by ROSA with their exact counterparts.

In the second example, we consider sensitivity analyses with multiple UPs and
two OCs. We illustrate the use of our computational procedures, including the
OCs approximation procedure (Algorithm 1), the validation procedure (Algorithm
2), and the SA optimization procedure (Algorithm 3). We investigate whether it
is appropriate to fix the value of some of the UPs across all sensitivity
scenarios. Identical values for a subset of the UPs can simplify the
interpretation of the sensitivity analysis but can also introduce severe
limitations in faithfully representing how the OCs vary across plausible
values of the UPs.

In the third example, we discuss sensitivity analyses dedicated to an adaptive
trial with sub-populations defined by biomarkers, considering multiple UPs and
multiple OCs of interest. We examine the difference in the marginal losses

$\mathcal{L}_{r}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})=\max_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\left\\{\min_{k=1,...,K}||f_{r}(\boldsymbol{\theta})-f_{r}(\boldsymbol{\theta}_{k})||_{2}\right\\},\quad{}1\leq
r\leq R,$ (3)

when the set of scenarios are chosen by optimizing different loss functions.
For example, let $\mathcal{S}_{r}$ be the set of scenarios that minimize the
marginal loss $\mathcal{L}_{r}$ in (3). Similarly, let $\mathcal{S}$ be the
set of scenarios that minimize the joint loss $\mathcal{L}=-\mathcal{U}$ in
(2). Then it is intuitive that
$\mathcal{L}_{r}(\mathcal{S}_{r})\leq\mathcal{L}_{r}(\mathcal{S})$, $1\leq
r\leq R$. In different words, the marginal losses $\mathcal{L}_{r}$ tend to be
smaller when the set of scenarios is chosen to minimize $\mathcal{L}_{r}$
compared to a set of scenarios that minimizes $\mathcal{L}$ with the aim of
representing multiple OCs. If the discrepancy
$\mathcal{L}_{r}(\mathcal{S}_{r})-\mathcal{L}_{r}(\mathcal{S})$, $1\leq r\leq
R$, is relatively small for all $R$ total OCs, then this indicates that it is
reasonable to select a single set of scenarios $\mathcal{S}$ to illustrate how
the $R$ OCs vary jointly across $\boldsymbol{\Theta}$.

By illustrating the ROSA methodology in three trial designs, we show its
flexibility with potential applications to evaluate nearly any clinical trial
design. Indeed, ROSA only requires the possibility of simulating the trials
under potential UPs $\boldsymbol{\theta}\in\boldsymbol{\Theta}=\mathbb{R}^{d}$
and the definition of the OCs of interest.

###  3.1 Application 1: Two-arm RCT

We consider the design of a two-arm randomized trial (1:1 randomization ratio)
with a sample of $n=30$ patients. For each $i=1,\ldots,n$, we let $A_{i}=0$ or
$1$ if the $i$-th study patient is assigned to the control or experimental
arm. The outcomes of the $n$ study patients are $Y_{1},\ldots,Y_{n}$, which we
assume to be independent and normally distributed. If $A_{i}=a$ then $Y_{i}$
has mean $\boldsymbol{\mu}_{a}=100+15a$ and standard deviation $\sigma$ equal
to $30$. In the analysis of the study, a $z$-statistic will be used to test
the null hypothesis $H_{0}:\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{0}\leq 0$
against the alternative $H_{1}:\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{0}>0$ at
5% significance level.

The goal of the sensitivity analysis is to assesses the variation of the
probability of rejecting $H_{0}$, a function $f(\boldsymbol{\theta})$ of the
unknown treatment effect
$\boldsymbol{\theta}=\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{0}\in\boldsymbol{\Theta}=\mathbb{R}$.
For example, if we knew that $\boldsymbol{\theta}=13.5$, then
$f(\boldsymbol{\theta})=0.80$, but in general $\boldsymbol{\theta}$ is an
unknown value.

Suppose we aim to identify $K=3$ scenarios
$\boldsymbol{\theta}_{1}^{*},\boldsymbol{\theta}_{2}^{*},\boldsymbol{\theta}_{3}^{*}$
that maximize the utility $\mathcal{U}$, i.e.,

$\boldsymbol{\theta}_{1}^{*},\boldsymbol{\theta}_{2}^{*},\boldsymbol{\theta}_{3}^{*}=\mathop{\rm\arg\\!\max}_{\boldsymbol{\theta}_{1},\boldsymbol{\theta}_{2},\boldsymbol{\theta}_{3}\in\boldsymbol{\Theta}}\text{
}\mathcal{U}(\boldsymbol{\theta}_{1},\boldsymbol{\theta}_{2},\boldsymbol{\theta}_{3}),$
(4)

where
$\mathcal{U}(\boldsymbol{\theta}_{1},\boldsymbol{\theta}_{2},\boldsymbol{\theta}_{3})=-\max_{\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}}\min_{k=1,2,3}|f(\boldsymbol{\theta}^{\prime})-f(\boldsymbol{\theta}_{k})|.$

In this trial, we have a single UP (${\boldsymbol{\Theta}}=\mathbb{R}$), and
the OC of interest is monotone, continuous, invertible, and ranges from $0$ to
$1$. Therefore, it is straightforward to see that the optimal scenarios
$\boldsymbol{\theta}_{1}^{*},\boldsymbol{\theta}_{2}^{*},\boldsymbol{\theta}_{3}^{*}$
correspond to the OC values that evenly divide the interval $(0,1)$. To be
precise,
$\\{f(\boldsymbol{\theta}_{1}^{*}),f(\boldsymbol{\theta}_{2}^{*}),f(\boldsymbol{\theta}_{3}^{*})\\}=\\{1/6,3/6,5/6\\}$;
these are the three values of a regular grid on the interval $(0,1)$. Figure
2A illustrates the optimal set of scenarios when $K=\\{3,5,10\\}$. Since
$f(\boldsymbol{\theta)}$ can be calculated exactly, the optimal scenarios
$\boldsymbol{\theta}_{1}^{*},\boldsymbol{\theta}_{2}^{*},\boldsymbol{\theta}_{3}^{*}$
can be obtained by computing the inverse function $f^{-1}$ at the values
$1/6,3/6,$ and $5/6$. Specifically,

$\left\\{\boldsymbol{\theta}_{1}^{*},\boldsymbol{\theta}_{2}^{*},\boldsymbol{\theta}_{3}^{*}\right\\}=\left\\{\frac{\sigma(z_{f(\boldsymbol{\theta}_{1}^{*})}+z_{1-\alpha/2})}{\sqrt{n}},\frac{\sigma(z_{f(\boldsymbol{\theta}_{2}^{*})}+z_{1-\alpha/2})}{\sqrt{n}},\frac{\sigma(z_{f(\boldsymbol{\theta}_{3}^{*})}+z_{1-\alpha/2})}{\sqrt{n}}\right\\},$

where $z_{1-\alpha/2}$ is the $1-\alpha/2$ quantile of the standard normal
distribution. The corresponding optimal scenarios are illustrated as red
asterisks in Figure 2B.

The exact computation of the optimal set of scenarios provides a solid
benchmark for an initial evaluation of ROSA (Algorthms 1-3). We can compare
the exact solution with the results from ROSA, which has the advantage of
being applicable to other designs and OCs that are not available in closed
form.

We implement our ROSA approach to identify $K=3$ scenarios. We randomly select
$J=1000$ scenarios
$\boldsymbol{\theta}_{1}^{t},...,\boldsymbol{\theta}_{1000}^{t}$ with
independent samples from the $\textrm{Uniform}(-5,25)$ distribution. Note that
$f(-5)\approx 0$ and $f(25)\approx 1$. For each $\boldsymbol{\theta}_{j}^{t}$,
$1\leq j\leq 1000$, we simulate $M=200$ trials to compute the estimate
$\bar{f}(\boldsymbol{\theta}_{j}^{t})=200^{-1}\sum_{m=1}^{200}{\varphi}(Z_{j,m},\boldsymbol{\theta}_{j}^{t}),$
where ${\varphi}(Z_{j,m},\boldsymbol{\theta}_{j}^{t})\in\\{0,1\\}$ either
accepts or rejects the $H_{0}:\boldsymbol{\theta}_{j}^{t}\leq 0$ for trial $m$
and scenario $j$. Then, we compute a smooth function
$\hat{f}(\boldsymbol{\theta})$ using the independent estimates
$\bar{f}(\boldsymbol{\theta}_{j})$ and a NN with 3 hidden layers (8, 64, and
64 neurons respectively) and ReLU activation functions. Finally, to select
three sensitivity scenarios, we use a SA algorithm based on an initial
parameterization $T_{1}=1000$, temperature reduction factor $r=0.8$, and final
parameterization $T_{\min}=0.1$ (c.f. Algorithm 3). We repeat these three
steps (selection of scenarios, use of the NN, and optimization with SA) $20$
times, each time initializing
$\boldsymbol{\theta}_{1},\boldsymbol{\theta}_{2},\boldsymbol{\theta}_{3}$ with
independent random draws from the $\textrm{Uniform}(-5,25)$ distribution. The
results of the exact approach (red asterisks) compared with ROSA (blue points)
are shown in Figure 2B. The scenarios
$\boldsymbol{\theta}_{1}^{*},\boldsymbol{\theta}_{2}^{*},\boldsymbol{\theta}_{3}^{*}$
selected by SA (blue dots) are close to the exact solution (red asterisks).

Figure 2: Sensitivity analysis of a RCT (OC: probability of rejecting
$H_{0}$). Panel A: Exact solutions when $K=\\{3,5,10\\}$. Panel B: Comparison
of $K=3$ scenarios selected through exact calculation (red asterisks) and by
$20$ ROSA implementations with different initial proposals (blue points).
Panel C: Graphical tool to choose the number $K$ of sensitivity scenarios.

We ran ROSA with $K=2,3,4,5,6,7,8,9,10,20,$ or $30$, and compared the loss
$\mathcal{L}$ in the resulting set of scenarios with that of the exact
solution. The difference in the loss $\mathcal{L}$ of the exact and
approximate optima was less than $1\%$ across all $K$ values that we
considered (Figure 2C). Table 1 indicates that the computation time of the SA
algorithm scales well as $K$ increases and that, as expected, the loss
$\mathcal{L}$ decreases as $K$ increases. All analyses were run on a Windows
laptop with an Intel(R) Core(TM) i7-7700HQ 2.80 GHz processor, 16GB RAM, and
6MB of cache memory.

In practice, the decision regarding the number $K$ of scenarios to report is
left to the analyst. This choice can be supported by a graph like Figure 2C,
which allows the investigator to determine the minimum number $K$ of scenarios
needed to guarantee a loss
$\mathcal{L}(\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*})$ no
larger than a targeted threshold. For example, to guarantee a loss no larger
than $0.050$ in this example, we need to select at least $10$ scenarios for
the sensitivity report.

Number $K$ of Scenarios | Time (seconds) | ROSA Loss ${\mathcal{L}}$ | Min. Loss $\mathcal{L}$ | Rel. Diff.
---|---|---|---|---
5 | 8.8 | 0.101 | 0.100 | 1.0%
6 | 8.8 | 0.084 | 0.083 | 0.7%
7 | 9.1 | 0.072 | 0.071 | 0.8%
8 | 9.2 | 0.062 | 0.0625 | 0.7%
9 | 9.1 | 0.056 | 0.056 | 0.6%
10 | 9.1 | 0.050 | 0.050 | 0.2%
20 | 10.1 | 0.025 | 0.025 | 0.5%
30 | 10.2 | 0.017 | 0.0167 | 0.8%
Table 1: ROSA computation time, ROSA loss $\mathcal{L}$, minimum (exact) loss
$\mathcal{L}$, and relative difference in loss of ROSA scenarios compared to
the exact solutions.

###  3.2 Application 2: Interim decisions based on auxiliary outcomes

We consider a two-arm, two-stage randomized trial with a binary primary
outcome $Y$ and a binary auxiliary outcome $S$ (Niewczas et al., 2019). The
primary outcome $Y$ is available $T_{Y}$ months after randomization, while the
auxiliary outcome $S$ is available after $T_{S}<T_{Y}$ months. For example, in
glioblastoma trials, 12-month progression-free survival (PFS) and 24-month
overall survival (OS) have been used as auxiliary and primary outcomes,
respectively (Han et al., 2014). The approach that we illustrate is applicable
for any value of $T_{Y}$ and $T_{S}<T_{Y}$.

We let $N_{a}$ be the planned number of patients for arms $a=0,1$ (i.e.,
control and experimental arms) and indicate with $p_{a}$ the response
probability $P(Y=1\mid A=a)$. Similarly, let $n_{a}$ be the planned number of
patients assigned to arm $a$ before the interim analysis, and $q_{a}$ indicate
the response probability $P(S=1\mid A=a)$. The difference $\Delta=p_{1}-p_{0}$
is the treatment effect on $Y$. The primary aim of the trial is to test
$H_{0}:\Delta\leq 0$ versus $H_{1}:\Delta>0,$ at level $\alpha$. The final
analysis of the study involves only the primary outcome $Y$, and the trial
will use a standard $Z$-test,
$Z_{Y}=\frac{\hat{p}_{1}-\hat{p}_{0}}{\sqrt{\bar{p}(1-\bar{p})(N_{1}^{-1}+N_{0}^{-1})}},$
where $\hat{p}_{a}$ is the estimate of $p_{a}$ and $\bar{p}$ is a weighted
average of $\hat{p}_{1}$ and $\hat{p}_{0}$.

An interim analysis is conducted after the auxiliary outcomes $S$ become
available for $n_{a}$ patients for arms $a=0$ and $1$ (i.e., $T_{S}$ months
after the enrollment of $n_{a}$ patients on arms $a=0$ and $1$), with early-
stopping for futility or continuation based on a summary of the auxiliary
outcomes $S$. In several clinical settings, the treatment effect on $S$ tends
to be more pronounced than the treatment effect on $Y$. The interim analysis
is based on the summary
$Z_{S}=\frac{\hat{q}_{1}-\hat{q}_{0}}{\sqrt{\bar{q}(1-\bar{q})(n_{1}^{-1}+n_{0}^{-1})}},$
where $\hat{q}_{a}$ is the estimate of $q_{a}$ and $\bar{q}$ is a weighted
average of $\hat{q}_{1}$ and $\hat{q}_{0}$. We replicate the design of
Niewczas et al. (2019), which calculates at the interim analysis the
conditional power (CP) using the auxiliary outcome $S$ to determine whether to
stop the trial for futility or not. Specifically, the CP is calculated based
on $Z_{S}$ and the information fraction
$t_{S}=\frac{N_{1}^{-1}+N_{0}^{-1}}{n_{1}^{-1}+n_{0}^{-1}}$ as

$CP(t_{S})=1-\Phi\left(\frac{z_{1-\alpha}-Z_{S}t_{S}^{1/2}}{\sqrt{1-t_{S}}}\right),$

where $z_{1-\alpha}$ is the $1-\alpha$ quantile of the standard normal
distribution and $\Phi(\cdot)$ is the cumulative distribution function of the
standard normal distribution. Here, we set the cut-off point to be $0.5$ so
that the trial continues when $CP(t_{s})\geq 0.5$.

The complexity of the sensitivity report increases with $K$ (the number of
scenarios), $d$ (the number of entries of the UPs $\boldsymbol{\theta}$), and
$R$ (the number of OCs $\textbf{f}(\boldsymbol{\theta})$). Here the full set
of UPs $\boldsymbol{\Theta}\subset\mathbb{R}^{7}$ include the enrollment rate
$e\in(0,\infty)$, the response rates $p_{a}\in(0,1)$ for $Y$ in $A=a$, the
response rates $q_{a}\in(0,1)$ for $S$ in $A=a$, and the correlation between
$Y$ and $S$ in $A=a$, $\rho_{a}\in(-1,1)$.

Controlling the complexity of the sensitivity report is important to ensure
high interpretability of the report, which will be discussed by several
stakeholders. There are a few potential strategies to reduce the complexity of
the sensitivity report. First, it is often possible to consider only a subset
of the parameter space
$\boldsymbol{\Theta}^{\prime}\subset\boldsymbol{\Theta}$ based on prior
knowledge of plausible values of the UPs. For example, previous clinical
studies can indicate a plausible range for the enrollment rate $e$, the
response rates $p_{0}$ under the SOC, and other parameters that are expected
to have minimal variations across trials. In addition, we can also consider
fixing multiple entries of the $K$ vectors
$\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}$ to some reference
values. In this case the space from which we select scenarios
$\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}$ is further reduced to
$\boldsymbol{\Theta}^{\prime}_{re}\subset\boldsymbol{\Theta}^{\prime}$. For
example, if the OCs have low sensitivity with respect to the correlation
parameters $\rho_{a}$ or the enrollment rate $e$ of the study, then we can fix
these UPs to common values (i.e., estimates) across all $K$ scenarios.

ROSA allows us to evaluate whether it is appropriate to assign the same value
to one or more UPs (e.g., $\rho_{0}$ and $\rho_{1}$) across all $K$ scenarios.
In other words, we evaluate a sensitivity report with all scenarios in a
restricted subset
$\boldsymbol{\Theta}^{\prime}_{re}\subset\boldsymbol{\Theta}^{\prime}$. A
sensitivity report with scenarios in $\boldsymbol{\Theta}^{\prime}_{re}$ can
potentially be easier to interpret compared to a report in which all $d$
entries of $\boldsymbol{\theta}$ vary across scenarios by reducing the number
of dimensions $d$ of the UPs and pointing to the most relevant UPs when
discussing the variations of the OCs across $\boldsymbol{\Theta}^{\prime}$. We
can select scenarios from the restriction
$\boldsymbol{\Theta}_{re}^{\prime}\subset\boldsymbol{\Theta}^{\prime}$ only if
the capability of the sensitivity report of representing the OCs variations
across $\boldsymbol{\Theta}^{\prime}$ is preserved. Our case study
investigates this aspect.

The OCs of interest in our case study are the probability of rejecting the
null hypothesis of no treatment effect on $Y$ at the end of the study and the
average sample size. Using our procedure, we randomly select $J=1000$ training
scenarios using LHS and conduct $M=500$ Monte Carlo simulations for each of
the $J$ training scenarios to obtain estimates of the OCs across
$\boldsymbol{\Theta}^{\prime}$.

Here $\boldsymbol{\Theta}^{\prime}$ is a product space with the enrollment
rate $e\in(0.2,1)$, the response rates $p_{a}\in(0.2,0.4)$ for $Y$ in $A=a$,
the response rates $q_{a}\in(0.2,0.4)$ for $S$ in $A=a$, and the correlation
between $Y$ and $S$ in $A=a$, $\rho_{a}\in(0,0.6)$. For
$\boldsymbol{\Theta}^{\prime}_{re}$, we fix the enrollment rate $e=0.5$ and
the response rates $p_{0}=q_{0}=0.3$ in the control groups.

We use a NN to obtain an interpolation of the OCs. As described in Algorithm
4, to evaluate if the estimates of the OCs are accurate, we compare them to
independent Monte Carlo estimates of size M = 100,000 on a set of
$J^{\prime}=200$ uniformly-distributed validation points spanning the
plausible parameter space $\boldsymbol{\Theta}^{\prime}$. The coefficient of
determination $R^{2}$ in this comparison is $0.96$, suggesting that the NN
accurately estimates the OCs (Supplementary Materials).

We compare two sensitivity reports, and our goal is to provide stakeholders
the simplified version if it accurately describes the OCs. The first one
includes scenarios from $\boldsymbol{\Theta}^{\prime}\subset\mathbb{R}^{7}$
restricted by prior knowledge from completed studies and clinical experience
and the second includes scenarios from
$\boldsymbol{\Theta}^{\prime}_{re}\subset\boldsymbol{\Theta}^{\prime}$ further
restricted by fixing the value of some entries of $\boldsymbol{\theta}$ as
described above. We use SA to identify two sets of scenarios in
$\boldsymbol{\Theta}^{\prime}_{re}$ and $\boldsymbol{\Theta}^{\prime}$,
respectively. In both cases we minimize the same loss function $\mathcal{L}$
defined over K-tuples of $\boldsymbol{\Theta}^{\prime}$ points. We also
calculate the loss $\mathcal{L}$ associated with these two optimal sets of
scenarios from $\boldsymbol{\Theta}^{\prime}$ and
$\boldsymbol{\Theta}^{\prime}_{re}$. In Figure 3, we illustrate the difference
in loss $\mathcal{L}$ between these two optimal sets. As expected, the loss
$\mathcal{L}$ decreases as $K$ increases. We observe in Figure 3 that for any
value of $K$, the loss $\mathcal{L}$ associated with the optimal set of
scenarios restricted to $\boldsymbol{\Theta}^{\prime}_{re}$ is larger compared
to the optimal scenarios in $\boldsymbol{\Theta}^{\prime}$. However, the
difference is modest, and the gain in interpretability of a sensitivity
analysis report with fewer UPs may be worth the slightly larger loss. For
example, if an investigator requires the loss to be under a threshold of
$\mathcal{L}=0.2$, then it is sufficient to consider $K=10$ scenarios,
regardless of whether we consider scenarios selected from
$\boldsymbol{\Theta}^{\prime}$ or $\boldsymbol{\Theta}^{\prime}_{re}$.

Figure 3: Clinical trial design with an interim analysis and an auxiliary
endpoint. A graphical representation to choose the number of sensitivity
scenarios $K\in\\{2,5,10,15\\}$. We compare optimal sets of scenarios selected
from $\boldsymbol{\Theta}^{\prime}\subset\mathbb{R}^{7}$ and from the lower-
dimensional restriction
$\boldsymbol{\Theta}^{\prime}_{re}\subset\boldsymbol{\Theta}^{\prime}$.

###  3.3 Application 3: Biomarker-driven adaptive enrichment

In several oncology trials, a major decision is whether to restrict patient
enrollment to a targeted subgroup of patients (e.g., biomarker-positive
subgroup) or to enroll a broader patient population. Enrolling only a
biomarker-positive subgroup may deny a substantial number of patients access
to an effective therapy, whereas enrolling a larger population may compromise
the power to detect positive treatment effects. Several trial designs
discussed in the literature attempt to address the outlined problem through
interim looks at the data. Among them, we consider an adaptive two-stage
enrichment trial design with one-to-one randomization (Jenkins et al., 2011;
Jones et al., 2017; Mehta et al., 2019). The design is applicable in the
setting where a biomarker-positive subgroup of patients is hypothesized to
benefit more from the experimental treatment than the rest of the study
population.

The design includes a single interim analysis, and it uses progression-free
survival (PFS) for interim decision-making, while overall survival (OS) is the
endpoint for the final analysis, which occurs when a pre-specified number of
events is reached. The interim analysis uses the estimated PFS hazard ratio
(HR) to capture potential early signals of treatment effects. In the
implementation of Jenkins et al. (2011), which we replicate, the HR is
estimated for both the overall population ($\hat{\theta}_{HR}$) and the
biomarker-positive subgroup ($\hat{\theta}_{HR}^{+}$). An interim decision
determines which group is enrolled and tested during the second stage of the
trial:

A – Promising results in the biomarker-positive population. If the HR estimate
$\hat{\theta}_{HR}^{+}<0.6$ but $\hat{\theta}_{HR}\geq 0.8$, then the trial
will continue enrolling only biomarker-positive patients and the final
analysis will test $H_{0}^{+}$. Here $H_{0}^{+}$ is the null hypothesis of no
differences in OS between treatment and control groups in the biomarker-
positive population. The null hypothesis is rejected if
$\omega_{1}\Phi^{-1}(1-p_{1}^{+})+\omega_{2}\Phi^{-1}(1-p_{2}^{+})<1.96$,
where $p_{1}^{+}$ $(p_{2}^{+})$ is a log-rank p-value computed using only OS
data from patients randomized during the first (second) stage of the trial.
The weights $(\omega_{1},\omega_{2})$ and the standard normal cumulative
distribution function $\Phi$ are used to summarize evidence of treatment
effects from the two stages of the trial. We refer to Jenkins et al. (2011)
for details on the choice of $(\omega_{1},\omega_{2})$ and other aspects of
the final analysis.

B – Promising results in the overall population only. If
$\hat{\theta}_{HR}^{+}\geq 0.6$ but $\hat{\theta}_{HR}<0.8$, then the trial
will continue enrolling all patients and the final analysis will only test
$H_{0}^{O}$, the null hypothesis of no differences in OS in the overall
population. In this case the null hypothesis is tested using stage-specific OS
log-rank p-values $(p_{1}^{O},p_{2}^{O})$ and combining evidence from the two
stages of the trial.

C – Unpromising results. If $\hat{\theta}_{HR}^{+}\geq 0.6$ and
$\hat{\theta}_{HR}\geq 0.8$, then the trial stops early for futility.

D – Promising early results for both populations. Lastly, if the estimated HR
in the biomarker-positive subgroup $\hat{\theta}_{HR}^{+}<0.6$ and the overall
population $\hat{\theta}_{HR}<0.8$, then the trial will continue enrolling all
patients and testing efficacy both in the overall population and in the
biomarker-positive subgroup.

The potential conclusion at the final analysis are (i) to recommend the new
treatment for biomarker-positive patients, (ii) recommend the new treatment
for both biomarker-positive and biomarker-negative patients, or (iii) not
recommend the experimental treatment for future patients.

Sensitivity analysis; the definition of $\boldsymbol{\Theta}^{\prime}$. We
choose plausible intervals for the UPs based on prior literature.
Specifically, the recruitment rate $\theta_{1}\in(0.5,1)$ per week, the
prevalence of the biomarker-positive subgroup $\theta_{2}\in(0.15,0.25)$, the
PFS HR comparing the treatment and control groups in the biomarker-positive
subgroup $\theta_{3}\in(0.5,1.2)$, the PFS HR comparing treatment and control
in the biomarker-negative subgroup $\theta_{4}\in(0.6,1.2),$ the OS HR
comparing treatment and control in the biomarker-positive subgroup
$\theta_{5}\in(0.7,1.2),$ the OS HR comparing treatment and control groups in
the biomarker-negative subgroup $\theta_{6}\in(0.8,1.2),$ the correlation
between OS and PFS in the biomarker-positive subgroup
$\theta_{7}\in(0.3,0.6)$, and the correlation between OS and PFS in the
biomarker-negative subgroup $\theta_{8}\in(0.2,0.7)$. Marginal exponential
distributions using a mixture representation were used for simulating
correlated OS and PFS times (Michael and Schucany, 2002). More flexible models
such as the Weibull distribution can be considered.

Sensitivity analysis; the definition of $\boldsymbol{f}$. We focus on the
following three OCs: (i) $f_{1}$, the probability of enrolling only biomarker-
positive patients in the second stage, (ii) $f_{2}$, the probability of
enrolling both biomarker-positive and biomarker-negative patients in the
second stage, and (iii) $f_{3}$, the probability of no evidence of positive
treatment effects, which is equal to the probability of not rejecting the null
hypotheses.

For the outlined two-stage trial with biomarker populations, our ROSA pipeline
can be used to compute multiple sensitivity reports, varying both the list of
OCs $\boldsymbol{f}$ and the definition of $\boldsymbol{\Theta}^{\prime}$. For
example, one can fix the OS HRs in the biomarker-positive and negative
populations to focus on the design sensitivity to other parameters, such as
the PFS HRs. Similarly, the set of UPs $\boldsymbol{\Theta}^{\prime}$ can be
restricted to $\boldsymbol{\theta}$ values with positive effects only for the
biomarker-positive population. Importantly, one set of training simulations
can be re-utilized to compute multiple sensitivity tables where the
definitions of $\boldsymbol{f}$ and $\boldsymbol{\Theta}^{\prime}$ vary.

We describe the difference between the marginal losses $\mathcal{L}_{r}$,
$r=1,2,3$, when scenarios
$\boldsymbol{\theta}_{1},\ldots,\boldsymbol{\theta}_{K}$ in
$\boldsymbol{\Theta}^{\prime}$ are chosen by optimizing $\mathcal{L}_{r}$ in
(3) – optimum:
$\mathcal{S}_{r}=\boldsymbol{\theta}_{1,r}^{*},...,\boldsymbol{\theta}_{K,r}^{*}$
– or by optimizing $\mathcal{L}$ as in (2) – optimum:
$\mathcal{S}=\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}$.
Recall that $\mathcal{S}$ is computed with the goal of illustrating how
multiple OCs vary across $\boldsymbol{\Theta}^{\prime}$ while
$\mathcal{S}_{r}$ optimizes the representation of a single OC $f_{r}$. The
weights in (2) are $w_{1}=w_{2}=w_{3}=1/3$. In Figure 4 panel 1, we plot
$\mathcal{L}_{1}(\mathcal{S}_{1})$ in red and $\mathcal{L}_{1}(\mathcal{S})$
in blue. Similarly, in panel 2 we compare $\mathcal{L}_{2}(\mathcal{S}_{2})$
and $\mathcal{L}_{2}(\mathcal{S})$, and in panel 3 we compare
$\mathcal{L}_{3}(\mathcal{S}_{3})$ and $\mathcal{L}_{3}(\mathcal{S})$. Our
results indicate that for all three OCs,
$\mathcal{L}_{r}(\mathcal{S})>\mathcal{L}_{r}(\mathcal{S}_{r})$, $r=1,2,3$. As
expected, there is an increase of the marginal losses $\mathcal{L}_{r}$ when
the set of scenarios is selected to illustrate jointly the variations of
multiple OCs across $\boldsymbol{\Theta}^{\prime}$. However, this difference
is small $(<10\%)$ for all $K\in\\{2,5,10,15\\}$. Furthermore, for each
$K\in\\{2,5,10,15\\}$, the relative difference is similar across the three OCs
$f_{1},f_{2},f_{3}$ (Figure 4). This result supports the use of identical
weights and of a single sensitivity table, with the same set of scenarios
$\mathcal{S}$ to illustrate jointly all three OCs.

Figure 4: Marginal losses $\mathcal{L}_{r}$, $r=1,2,3$ of different sets of
scenarios $\mathcal{S}_{r}$ (red) and $\mathcal{S}$ (blue).

##  4 Discussion

During the design stage of a clinical trial, sensitivity reports are typically
produced to discuss sample size, interim analyses, and other major decisions
with various stakeholders. The sensitivity report consists of one or a few
tables dedicated to showcasing how major operating characteristics (OCs)
$\mathbf{f}(\boldsymbol{\theta})$ vary across potential values of UPs in
$\boldsymbol{\Theta}$. In most cases the analyst focuses on subsets of
plausible parameters $\boldsymbol{\Theta}^{\prime}\subset\boldsymbol{\Theta}$,
for example, values concordant with previous studies, or subsets of potential
$\boldsymbol{\theta}$ values of particular interest becouse of positive and
clinically relevant treatment effects. ROSA supports the choice of which and
how many scenarios to include in these sensitivity reports.

The evaluation of complex designs such as dose-finding studies (Iasonos et
al., 2015), factorial trials (Green et al., 2002), and response-adaptive
trials (Pallmann et al., 2018) focuses on multiple OCs, such as the level of
toxicities, the probability of selecting the correct treatment arm, or
frequentist OCs, including power and false positive probabilities.

Simulations are fundamental in the design of complex trials since OCs can
rarely be obtained analytically and are crucial in the assessment of study
designs for regulators, pharmaceutical companies and other stakeholders (Food
et al., 2020). However, a limited number of scenarios or poorly chosen
scenarios could be inadequate to highlight variations of the OCs across
plausible UPs and can result in sub-optimal decisions.

We focus on choosing an informative number $K$ of scenarios
$\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}$ among the plausible UPs
to summarize the variations of key OCs. Our approach minimizes an explicit
loss function and uses established techniques for functional approximation
(NNs) and numerical optimization (SA). We showcase our approach in three
trials. Importantly, our approach is general and can be applied to nearly any
clinical trial design. It only requires simulations to mimic the clinical
trial under hypothetical scenarios.

Although our approach is general, we focused on loss functions $\mathcal{L}$
of a specific form (2). It is possible to consider different loss functions.
For example, one could consider the loss function
$\tilde{\mathcal{L}}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})=E_{\boldsymbol{\theta}^{\prime}\sim
g(\cdot)}\left\\{\min_{k=1,...,K}D[\mathbf{f}(\boldsymbol{\theta}^{\prime}),\mathbf{f}(\boldsymbol{\theta}_{k})]\right\\},$
where $g(\cdot)$ is a probability distribution on $\boldsymbol{\Theta}$ (e.g.,
a posterior distribution obtained from previous data). The distribution $g$
could be used to incorporate prior information about the unknown UPs in the
selection of sensitivity scenarios. Moreover, the metric
$D:\boldsymbol{\Theta}^{2}\rightarrow\mathbb{R}$ can be extended to capture
both differences between OCs at plausible values
$\boldsymbol{\theta},\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}$ and
other aspects, such as the difference between expected values of the outcomes
$Y$ at $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{\prime}$.

One major challenge in the presentation of sensitivity reports is the need of
simplicity and interpretability of the results. To this end, we considered
fixing one or more UPs to identical values across the $K$ scenarios, which may
be reasonable when there is a priori knowledge on certain UPs. There are other
ways to simplify a sensitivity report, such as removing OCs that do not vary
across plausible UPs, or reporting only the range of the OCs across
$\boldsymbol{\Theta}$ instead of presenting the OCs for each representative
scenario.

Variations of the ROSA approach may also consider optimization algorithms
other than SA and regression methods alternative to NN for approximating the
OCs across $\boldsymbol{\Theta}$.

## Acknowledgements

The authors thank Cyrus Mehta and Christina Howe for helpful conversations and
feedback that greatly enhanced the paper. LH was supported by the Clinical
Orthopedic and Musculoskeletal Education and Training (COMET) Program, NIAMS
grant T32 AR055885. LT was supported by NIH grant R01LM013352.

## Supplementary Material

The supplementary material includes a table of notation used in the paper, and
a validation comparison of power estimates by MC simulations and our NN in the
two arm, two-stage randomized trial example.

Table 2: Notation $\boldsymbol{\Theta}$ | $\triangleq$ |  Unknown parameter (UP) space in $\mathbb{R}^{d}$
---|---|---
$\boldsymbol{\Theta}^{\prime}$ | $\triangleq$ |  Restricted UP subspace by prior knowledge in $\mathbb{R}^{d}$
$\boldsymbol{\Theta}^{\prime}_{re}$ | $\triangleq$ |  Restricted UP subspace by prior knowledge and fixing certain dimensions in $\mathbb{R}^{d}$
$\boldsymbol{\Theta}^{F}$ | $\triangleq$ |  Diffuse and finite UP subspace in $\mathbb{R}^{d}$
$\boldsymbol{\theta}=(\theta_{1},...,\theta_{d})$ | $\triangleq$ |  $d$-dimensional vector of UPs
$\boldsymbol{\theta}^{t}=(\boldsymbol{\theta}_{1}^{t},...,\theta_{d}^{t})$ | $\triangleq$ |  $d$-dimensional training vector of UPs
$\boldsymbol{\theta}^{v}=(\theta_{1}^{v},...,\theta_{d}^{v})$ | $\triangleq$ |  $d$-dimensional validation vector of UPs
$\\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K}\\}$ | $\triangleq$ |  A set of $K$ sensitivity scenarios
$\mathcal{S}=\\{\boldsymbol{\theta}_{1}^{*},...,\boldsymbol{\theta}_{K}^{*}\\}$ | $\triangleq$ |  The ROSA set of $K$ sensitivity scenarios optimizing loss $\mathcal{L}$
$\mathcal{S}_{r}=\\{\boldsymbol{\theta}_{1,r}^{*},...,\boldsymbol{\theta}_{K,r}^{*}\\}$ | $\triangleq$ |  The ROSA set of $K$ sensitivity scenarios optimizing marginal loss $\mathcal{L}_{r}$
$\mathbf{f}(\boldsymbol{\theta})$ | $\triangleq$ |  $R$-vector of operating characteristics (OCs) for UPs $\boldsymbol{\theta}$
$\hat{\mathbf{f}}(\boldsymbol{\theta})$ | $\triangleq$ |  Estimated $R$-vector of OCs for UPs $\boldsymbol{\theta}$
$\bar{\mathbf{f}}(\boldsymbol{\theta})$ | $\triangleq$ |  Average across $M$ simulations of the $R$-vector of OCs for UPs $\boldsymbol{\theta}$
$\boldsymbol{\varphi}(Z_{j,m},\boldsymbol{\theta}_{j})$ | $\triangleq$ |  Generic function to capture if a null hypothesis has been rejected, where $Z_{j,m}$ is the $m^{th}$ trial under the $j^{th}$ scenario, $\boldsymbol{\theta}_{j}$
$\mathcal{L}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})$ | $\triangleq$ |  Loss function
$\mathcal{U}(\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{K})$ | $\triangleq$ |  Utility criterion
$w_{1},...,w_{r}$ | $\triangleq$ |  Fixed non-negative weights for OCs $f_{1},...,f_{R}$
$\omega_{1},\omega_{2}$ | $\triangleq$ |  Weights for stage 1 and 2 p-values
$D[\cdot,\cdot]$ | $\triangleq$ |  Pre-specified distance metric
$\boldsymbol{z}_{1}^{i},...,\boldsymbol{z}_{K}^{i}$ | $\triangleq$ |  Gaussian noise in iteration $i$ of SA
$\rho_{i}$ | $\triangleq$ |  Acceptance probability in iteration $i$ of SA
$T_{0},T_{1},...,T_{I}$ | $\triangleq$ |  Decreasing sequence of positive numbers (cooling schedule of SA)
$r$ | $\triangleq$ |  Multiplicative reduction factor for SA in $(0,1)$
$U_{i}$ | $\triangleq$ |  Random variable distributed Uniform(0,1) for SA
$e$ | $\triangleq$ |  Enrollment rate in $(0,\infty)$
$N_{a}$ | $\triangleq$ |  Planned number of patients on arm $a=0,1$ at the final analysis
$n_{a}$ | $\triangleq$ |  Planned number of patients on arm $a=0,1$ at the interim analysis
$S$ | $\triangleq$ |  Binary auxiliary outcome
$Y$ | $\triangleq$ |  Primary outcome
$p_{a}$ | $\triangleq$ |  Response probability $P(Y=1\mid A=a)$
$\Delta=p_{1}-p_{0}$ | $\triangleq$ |  Treatment effect on $Y$
$q_{a}$ | $\triangleq$ |  Response probability $P(S=1\mid A=a)$
$\rho_{a}$ | $\triangleq$ |  Correlation between $Y$ and $S$ in $A=a$
Figure 5: Comparison of power estimates from $200$ randomly sampled validation
points spanning the parameter space by Monte Carlo simulation and NN.

## References

  * Bélisle (1992) Bélisle, C. J. (1992).  Convergence theorems for a class of simulated annealing algorithms on rd.  Journal of Applied Probability pages 885–895.
  * Berry et al. (2010) Berry, S. M., Carlin, B. P., Lee, J. J., and Muller, P. (2010).  Bayesian adaptive methods for clinical trials.  CRC press.
  * Bookstein (1989) Bookstein, F. L. (1989).  Principal warps: Thin-plate splines and the decomposition of deformations.  IEEE Transactions on pattern analysis and machine intelligence 11, 567–585.
  * Carnell and Carnell (2016) Carnell, R. and Carnell, M. R. (2016).  Package ‘lhs’.  CRAN. https://cran. rproject. org/web/packages/lhs/lhs. pdf .
  * Food et al. (2020) Food, Administration, D., et al. (2020).  Interacting with the fda on complex innovative trial designs for drugs and biological products.  FDA .
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., and Courville, A. (2016).  Deep learning.  MIT press.
  * Green et al. (2002) Green, S., Liu, P.-Y., and O’Sullivan, J. (2002).  Factorial design considerations.  Journal of Clinical Oncology 20, 3424–3430.
  * Han et al. (2014) Han, K., Ren, M., Wick, W., Abrey, L., Das, A., Jin, J., and Reardon, D. A. (2014).  Progression-free survival as a surrogate endpoint for overall survival in glioblastoma: a literature-based meta-analysis from 91 trials.  Neuro-oncology 16, 696–706.
  * Hobbs et al. (2019) Hobbs, B. P., Barata, P. C., Kanjanapan, Y., Paller, C. J., Perlmutter, J., Pond, G. R., Prowell, T. M., Rubin, E. H., Seymour, L. K., Wages, N. A., et al. (2019).  Seamless designs: current practice and considerations for early-phase drug development in oncology.  JNCI: Journal of the National Cancer Institute 111, 118–128.
  * Hornik (1991) Hornik, K. (1991).  Approximation capabilities of multilayer feedforward networks.  Neural networks 4, 251–257.
  * Husmann et al. (2017) Husmann, K., Lange, A., and Spiegel, E. (2017).  The r package optimization: Flexible global optimization with simulated-annealing.
  * Iasonos et al. (2015) Iasonos, A., Gönen, M., and Bosl, G. J. (2015).  Scientific review of phase i protocols with novel dose-escalation designs: how much information is needed?  Journal of Clinical Oncology 33, 2221\.
  * Jenkins et al. (2011) Jenkins, M., Stone, A., and Jennison, C. (2011).  An adaptive seamless phase ii/iii design for oncology trials with subpopulation selection using correlated survival endpoints.  Pharmaceutical statistics 10, 347–356.
  * Jones et al. (2017) Jones, R. L., Attia, S., Mehta, C. R., Liu, L., Sankhala, K. K., Robinson, S. I., Ravi, V., Penel, N., Stacchiotti, S., Tap, W. D., et al. (2017).  Tappas: An adaptive enrichment phase 3 trial of trc105 and pazopanib versus pazopanib alone in patients with advanced angiosarcoma (aas).  J. Clin. Oncol. 35, TPS11081.
  * Kirkpatrick et al. (1983) Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983).  Optimization by simulated annealing.  Science 220, 671–680.
  * Leshno et al. (1993) Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. (1993).  Multilayer feedforward networks with a nonpolynomial activation function can approximate any function.  Neural networks 6, 861–867.
  * McKay et al. (2000) McKay, M. D., Beckman, R. J., and Conover, W. J. (2000).  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code.  Technometrics 42, 55–61.
  * Mehta et al. (2019) Mehta, C., Liu, L., and Theuer, C. (2019).  An adaptive population enrichment phase iii trial of trc105 and pazopanib versus pazopanib alone in patients with advanced angiosarcoma (tappas trial).  Annals of Oncology 30, 103–108.
  * Michael and Schucany (2002) Michael, J. and Schucany, W. (2002).  The mixture approach for simulating new families of bivariate distributions with specified correlations.  The American Statistician 56, 48–54.
  * Niewczas et al. (2019) Niewczas, J., Kunz, C. U., and König, F. (2019).  Interim analysis incorporating short-and long-term binary endpoints.  Biometrical Journal 61, 665–687.
  * Pallmann et al. (2018) Pallmann, P., Bedding, A. W., Choodari-Oskooei, B., Dimairo, M., Flight, L., Hampson, L. V., Holmes, J., Mander, A. P., Odondi, L., Sydes, M. R., et al. (2018).  Adaptive designs in clinical trials: why use them, and how to run and report them.  BMC medicine 16, 1–15.
  * Rasmussen (2003) Rasmussen, C. E. (2003).  Gaussian processes in machine learning.  In Summer school on machine learning, pages 63–71. Springer.
  * Razavi et al. (2021) Razavi, S., Jakeman, A., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Piano, S. L., Iwanaga, T., Becker, W., et al. (2021).  The future of sensitivity analysis: an essential discipline for systems modeling and policy support.  Environmental Modelling & Software 137, 104954\.
  * Spall (2005) Spall, J. C. (2005).  Introduction to stochastic search and optimization: estimation, simulation, and control.  John Wiley & Sons.
  * Thorlund et al. (2018) Thorlund, K., Haggstrom, J., Park, J. J., and Mills, E. J. (2018).  Key design considerations for adaptive clinical trials: a primer for clinicians.  BMJ 360, k698.
  * Vanderbeek et al. (2019) Vanderbeek, A. M., Ventz, S., Rahman, R., Fell, G., Cloughesy, T. F., Wen, P. Y., Trippa, L., and Alexander, B. M. (2019).  To randomize, or not to randomize, that is the question: using data from prior clinical trials to guide future designs.  Neuro-oncology 21, 1239–1249.


# Audio-visual training for improved grounding in video-text LLMs

Shivprasad Sagare  Hemachandran S  Kinshuk Sarabhai  Prashant Ullegaddi
Rajeshkumar SA
Phronetic AI  Shivprasad Sagare, Hemachandran S., Kinshuk Sarabhai,
Prashant Ullegaddi, Rajeshkumar SA
PhroneticAI
Correspondence<EMAIL_ADDRESS>
###### Abstract

Recent advances in multimodal LLMs, have led to several video-text models
being proposed for critical video-related tasks. However, most of the previous
works support visual input only, essentially muting the audio signal in the
video. Few models that support both audio and visual input, are not explicitly
trained on audio data. Hence, the effect of audio towards video understanding
is largely unexplored. To this end, we propose a model architecture that
handles audio-visual inputs explicitly. We train our model with both audio and
visual data from a video instruction-tuning dataset. Comparison with vision-
only baselines, and other audio-visual models showcase that training on audio
data indeed leads to improved grounding of responses. For better evaluation of
audio-visual models, we also release a human-annotated benchmark dataset, with
audio-aware question-answer pairs.

Audio-visual training for improved grounding in video-text LLMs

Shivprasad Sagare, Hemachandran S., Kinshuk Sarabhai, Prashant Ullegaddi,
Rajeshkumar SA PhroneticAI Correspondence<EMAIL_ADDRESS>

##  1 Introduction

Conversational agents fueled by LLMs have made it possible for us to interact
in a new way with data from multiple modalitiesYin et al. (2024)Wadekar et al.
(2024). Image-text multimodal LLMs(MLLMs) like LLaVALiu et al. (2023) have
demonstrated the effectiveness of visual instruction-tuning(IT) data. Several
works like VideoChatGPTMaaz et al. (2023), VideoChatLi et al. (2024), PLLaVaXu
et al. (2024) have extended the image-text model architecture for video
related tasks.

Figure 1: An example of improved grounding in the video-text LLM outputs, due
to the additional audio signal as input.

However, most of the above works rely only on the visual input, and do not
consider audio signal for video understanding. In real world, listening to
audio while playing the video, adds immensely to our perception of the video.
We propose a video-text MLLM, with Phi-2Gunasekar et al. (2023) as the LLM
backbone. It supports both audio and visual inputs, using WhisperRadford et
al. (2022) and sigLIPZhai et al. (2023) encoders respectively. Unlike previous
works, we train the model using audio data explicitly, in addition to the
visual data. We aim to explore the role of audio in video understanding and if
audio input can be utilized for better grounding of video-text LLMs. We also
explore the creation of better benchmarks that encompass variety of question-
answer pairs. Evaluation on several benchmarks demonstrates the effectiveness
of audio as an additional signal in better understanding of the video content.

Overall we make the following key contributions: 1.We propose an efficient
video-text MLLM architecture consisting of separate encoders to process the
audio and visual inputs.
2.We train our video-text model using both audio and visual signals
simultaneously, aiming to explore the effect of audio input on model outputs.
3.We release a human-annotated benchmark dataset containing video instruction-
tuning samples, which are audio-aware.

Models | Visual | Audio | Audio-visual
---|---|---|---
VideoChatGPT | ✓ | – | –
LLaSM | – | ✓ | –
Video-LLaMA | ✓ | $\times$ | $\times$
NExT-GPT | ✓ | ✓ | $\times$
our | ✓ | ✓ | ✓
Table 1: Comparing MLLMs based on the input modalities supported, and the
training data. – indicates that the input modality isn’t supported. $\times$
indicates that the input modality is supported, but the model isn’t trained
using such data. ✓indicates that the model architecture supports the input
modality, and has also been explicitly trained on such data.

##  2 Related work

Vision-text MLLMs: LLaVALiu et al. (2023), MiniGPT4Zhu et al. (2023) have
showcased the efficacy of visual instruction-tuning datasets for image-text
tasks. BunnyHe et al. (2024) explores a similar idea but using lightweight LLM
backbones. Several works like PLLaVAXu et al. (2024) build on the top of
image-text MLLMs to support video input. VideoChatGPTMaaz et al. (2023)
extends the CLIP image encoderRadford et al. (2021) to videos by averaging the
representations across spatial and temporal dimensions.

Audio-text MLLMs: Similar to vision-text, there has been recent work in fusing
audio input features with text LLM for several audio-text tasksZhang et al.
(2023a). LLaSMShu et al. (2023) demonstrates the effectiveness of pretraining
the projector layers using speech-to-text data. Some previous works like
AudioGPTHuang et al. (2023) build on LLM-based planning and tool-use to solve
several audio tasks at once.

Audio-vision-text MLLMs Similar to our work, Video-LLaMAZhang et al. (2023b),
and NExT-GPTWu et al. (2023) support audio and visual input simultaneously,
both relying on unified modality encoder ImageBindGirdhar et al. (2023).
However, Video-LLaMA is trained only on visual IT datasets, assuming the audio
branch learns implicitly. NExT-GPT is trained using cross-modal IT dataset,
but doesn’t utilize audio-visual simultaneous input from videos. Unlike
previous works, we explore training using audio-visual input from videos
simultaneously, and explore the grounding effect it has on model outputs.

##  3 Model architecture

Following the idea of fusing the modality inputs into LLMLiu et al.
(2023)Zhang et al. (2023b), we build a video-text MLLM architecture consisting
of two separate branches for audio and visual inputs. Each branch consists of
modality encoder, projector layers to transform the encoder representations
into LLM embedding space, followed by the backbone LLM.

We use WhisperRadford et al. (2022) as an audio encoder, and use its last
hidden state as audio representationsShu et al. (2023). To encode the video,
we use sigLIP image encoderZhai et al. (2023). Following Maaz et al. (2023),
we treat video as a sequence of images, and compute frame representations
using sigLIP. We then compute spatial and temporal average of representations
across 100 uniformly sampled frames, and use it as a video representation.
Inspired from BunnyHe et al. (2024), we rely on low-cost, efficient,
lightweight LLM backbone with 2.7 Billion parameters, phi-2Gunasekar et al.
(2023). Projector layer for both vision and audio branch is mlp2x-geluHe et
al. (2024).

The exact flow of input data through both the audio and visual branches is
shown in the form of tensor dimensions, in figure 2. Audio and visual input is
converted into 64 and 829 token embeddings respectively. Audio, visual, and
text token embeddings are then concatenated before passing to the backbone
LLM.

Figure 2: Tensor dimensions in the figure denote the flow of data through the
encoder and projector layers. Audio encoder(Whisper) and video encoder(using
sigLIP) produce 64 and 829 token embeddings respectively, which are then
concatenated with the text token embeddings as the final input to the LLM.
Unlike previous works, we train both the audio and vision branch
simultaneously using a video instruction tuning dataset.

##  4 Training setup and datasets

Training different components of our model with appropriate data is a key
focus of our research. Typically, these MLLMs go through a pretraining stage,
followed by the finetuning stage.

Pretraining: Pretraining aims to align different modalities to text LLM space,
by training on some generic modality-to-text task. Only projector layer
weights are trained during this phase, while encoders, and LLM weights are
frozen. We pretrain our audio projector layers using a combination of Speech-
to-Text(STT) dataset(CommonVoiceArdila et al. (2020)) and audio captioning
dataset(AudioCapsKim et al. (2019)) with 50K samples each. We convert these
datasets into our instruction-tuning prompt template by creating 10
instructions each for transcription and captioning. Since our visual branch
relies on image encoder, we employ already trained checkpoint by BunnyHe et
al. (2024) to initialize vision projector layers. It has been trained on 2M
subset of an image-text dataset LAIONSchuhmann et al. (2022). We freeze the
vision branch while pretraining audio projector layers, and vice versa.

Finetuning: Finetuning or instruction tuning is aimed to train the LLM model
to follow the exact requests or questions in the user promptOuyang et al.
(2022). Unlike previous works, we explicitly train both the audio and visual
branches of the model simultaneously, using video instruction-tuning dataset
containing both the audio and visual data. We rely on VideoInstruct100KMaaz et
al. (2023) dataset with 100K samples containing video and question answer
pair. Although the dataset authors had used the dataset only for visual
instruction tuning, we extract the audios(wav format) from the videos(mp4
format) for our use-case.

We aim to explore if including audio features during training helps the model
to better understand the video. To measure this effect, we also train a
baseline vision-only model, without the audio branch. We train the vision
branch of the model, using the visual data from same dataset.

Experiment details We implement the audio and video functionality by extending
the codebases of Bunny and LLaSM. We use Whisper-small, siglip-
so400m-patch14-384, and phi-2 models from HuggingFace. Pretraining for audio
projector layer was done using A100, with global batch size of 128. Finetuning
was implemented using LoRA for training LLM weights, on A40 machine.

Metrics | visual-only model (our) | video-llama | audio-visual model (our)
---|---|---|---
Correctness of Information | 2.34 | 1.96 | 2.69
Detail Orientation | 2.35 | 2.18 | 2.49
Contextual Understanding | 2.74 | 2.16 | 3.04
Temporal Understanding | 1.97 | 1.82 | 2.22
Consistency | 2.45 | 1.79 | 2.71
Average | 2.37 | 1.98 | 2.63
Table 2: Results on VideoChatGPT evaluation framework. Our audio-visual training setup shows impressive results when compared with other audio-vision model(Video-LLaMA), as well our vision-only baseline. Metrics | visual-only model (our) | video-llama | audio-visual model (our)
---|---|---|---
Correctness of Information | 2.34 | 1.49 | 2.77
Detail Orientation | 2.36 | 1.7 | 2.44
Contextual Understanding | 2.75 | 1.92 | 3.04
Temporal Understanding | 2.17 | 1.4 | 2.4
Average | 2.40 | 1.62 | 2.66
Table 3: Results on our benchmark dataset. Results illustrate similar trend as
above, where training on audio signals helps the model to generate more
accurate responses. We haven’t yet incorporated evaluation for consistency
metric in our benchmark dataset.

##  5 Benchmark dataset

Several evaluation criteria and datasets have been introduced to benchmark the
vision-text MLLMsChen and Dolan (2011)Maaz et al. (2023)Heilbron et al.
(2015). VideoChatGPT has released a human verified benchmark dataset
consisting of 500 videos and corresponding question-answer pairs for video-
text tasks. However, these benchmarks do not consider audio information while
creating the question-answer pairs based on videos. Thus, it is challenging to
evaluate the capability of model to attend to both the audio and visual
signals while generating the output.

Therefore, we annotate such an audio-visual instruction-tuning dataset that
contains question-answer pairs based both on audio and visual information in
the video. We include both generic questions, like ’What is happening in the
video?’, as well as more specific questions related to the video. Answer of
each question is around 2 sentences, with most of the videos available on
YouTube. We release a set of 120 such samples, as we intend to scale the size
and quality of the data in future. Example samples from our benchmark dataset
are shown below.

Sample 1
Question: What is the man doing in the video?
Answer: In the video, the man fires his gun upwards, producing the sharp sound
of a bullet being shot. The echo reverberates through the air, adding tension
and intensity to the scene.

Sample 2
Question: What is the man on the stage mentoring about in the video?
Answer: The workshop leader, mentors a student on speaking louder for clarity.
He asks the student to raise the volume from level 3 to level 7. Finally, the
student earns an applause from the audience in the communication workshop.

##  6 Evaluation

We extensively evaluate our model using VideoChatGPT evaluation framework
across 5 key metrics. It relies on LLM-based evaluation(using GPT-3.5) which
rates the output on the scale of 1-5. We compare our audio-visual model with
the visual-only baseline that we have trained, as well as other audio-visual
model, Video-LLaMA. The evaluation results are summarized in the table 2.
Similarly, we evaluate on our benchmark dataset, and observe the similar
trends, as summarized in 3.

The audio-visual model clearly performs better than the vision-only baseline
by a margin. Interestingly, Video-LLaMA which is also an audio-visual model
performs poorly on both the benchmarks. Video-LLaMA does not utilize the audio
inputs explicitly, and instead rely on visual signals only during training. We
could not compare against another audio-visual model, NExT-GPT, as it relies
on LLaMA-v0 weights which couldn’t be available to us due to licensing.

Qualitative analysis of audio-visual model outputs demonstrate better overall
quality compared to vision-only model. We also analyze the model outputs at
intermediate stages, i.e. after pre-training. Our model could very well
generate the captions of audio data, which showed the efficacy of pretraining
step. There is scope for better encoding strategies and training regimes for
utilizing audio information even more.

##  7 Conclusion and future work

We performed several experiments and evaluations to specifically study how
audio signal can be utilized for better video understanding. Training the MLLM
simultaneously on audio-visual signals of the video indeed results in a better
performance, as seen in quantitative evaluation using several metrics. We also
contributed a benchmark dataset curated to evaluate the video-understanding
capability using both visual and audio information.

Based on these results, we are motivated to experiment with sophisticated ways
of incorporating audio and visual signals together for video related tasks.
Future work also consists of the extensive analysis of the type of question-
answer pairs in video IT datasets, and work on creating better evaluation
benchmarks catering to wide range of video-related use-cases.

## References

  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020.  Common voice: A massively-multilingual speech corpus.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4218–4222, Marseille, France. European Language Resources Association.
  * Chen and Dolan (2011) David Chen and William Dolan. 2011.  Collecting highly parallel data for paraphrase evaluation.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 190–200, Portland, Oregon, USA. Association for Computational Linguistics.
  * Girdhar et al. (2023) Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023.  Imagebind: One embedding space to bind them all.  _Preprint_ , arXiv:2305.05665.
  * Gunasekar et al. (2023) Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. 2023.  Textbooks are all you need.
  * He et al. (2024) Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. 2024.  Efficient multimodal learning from data-centric perspective.  _Preprint_ , arXiv:2402.11530.
  * Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  Activitynet: A large-scale video benchmark for human activity understanding.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 961–970.
  * Huang et al. (2023) Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe. 2023.  Audiogpt: Understanding and generating speech, music, sound, and talking head.  _Preprint_ , arXiv:2304.12995.
  * Kim et al. (2019) Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019.  AudioCaps: Generating captions for audios in the wild.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 119–132, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Li et al. (2024) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2024.  Videochat: Chat-centric video understanding.  _Preprint_ , arXiv:2305.06355.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual instruction tuning.  _Preprint_ , arXiv:2304.08485.
  * Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  _Preprint_ , arXiv:2306.05424.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  _Preprint_ , arXiv:2203.02155.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  _Preprint_ , arXiv:2103.00020.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.  _Preprint_ , arXiv:2212.04356.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Shu et al. (2023) Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. 2023.  Llasm: Large language and speech model.  _Preprint_ , arXiv:2308.15930.
  * Wadekar et al. (2024) Shakti N. Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello. 2024.  The evolution of multimodal model architectures.  _Preprint_ , arXiv:2405.17927.
  * Wu et al. (2023) Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023.  Next-gpt: Any-to-any multimodal llm.  _Preprint_ , arXiv:2309.05519.
  * Xu et al. (2024) Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024.  Pllava : Parameter-free llava extension from images to videos for video dense captioning.  _Preprint_ , arXiv:2404.16994.
  * Yin et al. (2024) Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024.  A survey on multimodal large language models.  _Preprint_ , arXiv:2306.13549.
  * Zhai et al. (2023) Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023.  Sigmoid loss for language image pre-training.  _Preprint_ , arXiv:2303.15343.
  * Zhang et al. (2023a) Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023a.  Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.  _Preprint_ , arXiv:2305.11000.
  * Zhang et al. (2023b) Hang Zhang, Xin Li, and Lidong Bing. 2023b.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _Preprint_ , arXiv:2306.02858.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _Preprint_ , arXiv:2304.10592.