11institutetext: Institute for Health and Society, Medical College of
Wisconsin, 8701 Watertown Plank Rd, Wauwatosa, WI, 53226 22institutetext:
Department of Statistics, University of Missouri, 600 S State St., Apt. 408
Bellingham, WA 98225

# Most Powerful Test Sequences with Early Stopping Options

Sergey Tarima  Nancy Flournoy

(Received: date / Accepted: date)

###### Abstract

Sequential likelihood ratio testing is found to be most powerful in sequential
studies with early stopping rules when grouped data come from the one-
parameter exponential family. First, to obtain this elusive result, the
probability measure of a group sequential design is constructed with support
for all possible outcome events, as is useful for designing an experiment
prior to having data. This construction identifies impossible events that are
not part of the support. The overall probability distribution is dissected
into stage specific components. These components are sub-densities of interim
test statistics first described by Armitage, McPherson and Rowe (1969) that
are commonly used to create stopping boundaries given an $\alpha$-spending
function and a set of interim analysis times. Likelihood expressions
conditional on reaching a stage are given to connect pieces of the probability
anatomy together.

The reduction of the support caused by the adoption of an early stopping rule
induces sequential truncation (not nesting) in the probability distributions
of possible events. Multiple testing induces mixtures on the adapted support.
Even asymptotic distributions of inferential statistics are mixtures of
truncated distributions. In contrast to the classical result on local
asymptotic normality (Le Cam 1960), statistics that are asymptotically normal
without stopping options have asymptotic distributions that are mixtures of
truncated normal distributions under local alternatives with stopping options;
under fixed alternatives, asymptotic distributions of test statistics are
degenerate.

###### Keywords:

Adaptive designs adapted support group sequential designs local asymptotics
interim hypothesis testing likelihood ratio tests

##  1 Introduction

We define a _sequential experiment_ to be one in which the decision to stop
collecting data is based on data collected previously in the study. Wetherill
and Glazebrook (1986) emphasize that ” _two aspects of a sequential procedure
must be clearly distinguished, the stopping rule, and the manner in which
inferences are made once observations are stopped. $\ldots$ in the design
problem, it is important to know how probable are various possible results.”_
We distinguish the probability framework underpinning these two activities,
and also a third - the probability framework underlying interim hypothesis
tests.

Dodge and Romig (1929) proposed the first known sequential test procedure in
which a decision to stop or continue collecting data was based on prior data,
recognizing that decisions to stop or continue a trial made based on prior
observations could substantially reduce the expected numbers of required
subjects; it was a two-stage design. Bartky (1943) devised a multiple
sequential testing procedure for binomial data based on Neyman and Pearson
(1933)’s likelihood ratio test that Wald (1947) cites as a ”forerunner” to his
more general _sequential probability ratio test_ (SPRT) procedure in which the
probabilities of type I and II errors are controlled. Extensions with stopping
decisions based on groups of subjects [_Group Sequential Designs (GSDs)_] are
given in Jennison and Turnbull (1999).

Neyman and Pearson (1933) show that likelihood ratio tests are most powerful
for testing a simple null versus a simple alternative hypotheses. In Ferguson
(2014), the Karlin-Rubin theorem is viewed as an extension of Neyman-Pearson
approach to most powerful testing of composite hypotheses. The Karlin-Rubin
theorem applies to the one-dimensional exponential family. With sequential
stopping options, some elements of the sample space become impossible, which
changes the distributions of statistics. Sufficient statistics become
dependent on the random sample size; see Blackwell (1947). If statistics
belong to the exponential family without early stopping options, then they
belong to a curved exponential family when exposed to early stopping options
(Efron (1975); Liu and Hall (1999); Liu et al. (2006)). Section 3.2 shows that
despite being from a curved exponential family, the sequential tests based on
likelihood ratios continue to be most powerful for any $\alpha$-spending
function.

###  1.1 Notation and A Simple Example

This example demonstrates a couple important repercussions of sequential
stopping rules:

  * •

The support is reduced,

  * •

Bivariate normal random variables become non-observable; the observable
bivariate random variable is a mixture of truncated normal random variables,

Let $X_{1}$ and $X_{2}$ be $N(\theta,1)$ random variables with an unknown
location parameter $\theta$.

####  1.1.1 Non-sequential experiments

If $X_{1}=x_{1}$ alone is observed, the log-likelihood

$l(\theta|X_{1}=x_{1})=-2\log\sqrt{2\pi}-0.5(x_{1}-\theta)^{2}$

is maximized at $\widehat{\theta}_{1}=x_{1}$.

If both $X_{1}=x_{1}$ and $X_{2}=x_{2}$ are observed independently, the random
variable $(X_{1},X_{2})$ is defined on the probability space
$\left(R^{2},{\cal{B}},P\right)$, where $R^{2}$ is the sample space
[$R=\left(-\infty,\infty\right)$], ${\cal{B}}$ is the Borel $\sigma$-algebra
on $R^{2}$ and $P$ is a bivariate normal distribution with mean vector
$(\theta,\theta)$, units variances and zero correlation.

The log-likelihood function is

$l(\theta|X_{1}=x_{1},X_{2}=x_{2})=-2\log\sqrt{2\pi}-0.5(x_{1}-\theta)^{2}-0.5(x_{2}-\theta)^{2}$

and the maximum likelihood estimator (MLE) of $\theta$ is
$\widehat{\theta}_{2}=(x_{1}+x_{2})/2$.

####  1.1.2 Sequential experiments: likelihood, support and probability
measures

What happens in sequential settings when $X_{2}$ is only observed if
$X_{1}<2.18$? Let $D$ denote the random stopping stage. Then
$D=1+I(X_{1}<2.18)$, where $I(\cdot)$ is an indicator function. In this simple
example, $D$ is also the random sample size.

The joint distribution of (X,D) and the marginal distribution of X are the
same:

$\displaystyle
f_{\textbf{X}}\left(\textbf{x}|\theta\right)=f_{\textbf{X},D}\left(\textbf{x},d|\theta\right)$
$\displaystyle=\phi(x_{1}-\theta)\left[\phi(x_{2}-\theta)\right]^{I\left(d=2\right)}$
(1)
$\displaystyle=[\phi(x_{1}-\theta)]^{I(d=1)}\left[\phi(x_{1}-\theta)\phi(x_{2}-\theta)\right]^{I\left(d=2\right)},$

where $\phi(\cdot)$ is the standard normal density. The representation of
$f_{\textbf{X}}\left(\textbf{x}|\theta\right)$ in the first line of (1)
partitions the density according to data collection stages, while the
representation in the second line partitions the density according to stopping
stages. In canonical form,

$\displaystyle f_{\textbf{X},D}\left(\textbf{x},d|\theta\right)$
$\displaystyle=$ $\displaystyle
h(\mathbf{x})\exp\Big{(}\left[x_{1}+I(d=2)x_{2}\right]\theta-\left[1+I(d=2)\right]\frac{\theta^{2}}{2}\Big{)},$
(2)

where
$h(\mathbf{x})=\exp\Big{(}-\frac{1}{2}\left[x_{1}^{2}+I(d=2)x_{2}^{2}+I(d=2)\log\left(\sqrt{\pi}\right)+\log\left(\sqrt{\pi}\right)\right]\Big{)}$.
Thus, the density (2) belongs to the curved exponential family with a
sufficient statistic

$\left(\sum_{k=1}^{d}x_{i},d\right)=\left[x_{1}+I(d=2)x_{2},\,1+I(d=2)\right].$

Curved exponential families were defined by Efron (1975) and the sufficient
statistic with a random number of summands $\sum_{k=1}^{D}x_{k}$ was derived
by Blackwell (1947). Probability distribution (2) is a special case of the
exponential family derived in Liu et al. (2006) [see their formula (2.6)]. A
more general probability distribution of $\mathbf{X}$ is presented in Section
2.

The log-likelihood function

$\displaystyle l(\theta|X_{1}=x_{1},X_{2}=x_{2},D=d)$
$\displaystyle=\begin{cases}[\phi(x_{1}-\theta)]^{I(d=1)}&\textrm{ if }D=1,\\\
\left[\phi(x_{1}-\theta)\phi(x_{2}-\theta)\right]^{I\left(d=2\right)}&\textrm{
if }D=2\end{cases}$

is maximized at $\widehat{\theta}=I(d=1)x_{1}+I(d=2)(x_{1}+x_{2})/2.$
Consequently, the score function, the MLE and the observed information are the
same as for the non-sequential experiment. What changes?

The joint support of $X_{1}$ and $X_{2}$ changes because $X_{2}$ becomes
impossible (not just missing) when $X_{1}\geq 2.18$. The random variable
$(X_{1},X_{2})$ is non-observable when $X_{1}\geq 2.18$ and the joint
distribution of $X_{1}$ and $X_{2}$ is therefore truncated and not normal.
Formally, the support for joint density can be decomposed into support for the
experiment stopping with $X_{1}$ and support for the experiment continuing to
observe $X_{2}$:

${\cal{T}}=\\{x_{1}\geq 2.18\\}\ \cup\ \\{\\{x_{1}<2.18\\}\cap\\{x_{2}\in
R\\}\\}\ \subset R^{2}.$

This ${\cal{T}}$ is a special case of the support formalized in Liu et al.
(2006). If $A\in{\cal{T}}$, then

$\displaystyle P_{(D)}=\text{Pr}((X_{1},X_{2})\in A)$ $\displaystyle=$
$\displaystyle\text{Pr}(X_{1}\geq 2.18)\text{Pr}(X_{1}\in A|X_{1}\geq 2.18)$
(3) $\displaystyle+$
$\displaystyle\text{Pr}(X_{1}<2.18)\text{Pr}((X_{1},X_{2})\in A|X_{1}<2.18)$

is a probability measure on the $\sigma$-algebra $\sigma({\cal{T}})$. Thus,
the observable random variable

$\displaystyle\mathbf{X}_{{\cal{T}}}=\begin{cases}X_{1}&\textrm{ if }$D=1$\\\
(X_{1},X_{2})&\textrm{ if }$D=2$.\end{cases}$

is defined on the probability space
$\left({\cal{T}},\sigma({\cal{T}}),P_{(D)}\right).$

In contrast to $(X_{1},X_{2})$, $\mathbf{X}_{{\cal{T}}}$ is observable for
this sequential experiment. The MLE, is a random variable defined on
$\left({\cal{T}},\sigma({\cal{T}}),P_{(D)}\right)$ and its probability
distribution is a mixture of the left truncated normal random variable
$\\{X_{1}|D=1\\}$ and an average of the right truncated normal random variable
$\\{X_{1}|D=2\\}$ and the normal random variable $X_{2}$.

Thus, in this sequential experiment, MLEs are not normal random variables as
illustrated in Figure 1. The first column shows the distribution of the MLE if
the experiment stopped at stage 1, which is left truncated normal. The seconds
of histograms shows the distribution of the MLE if the experiment proceeded to
stage 2. This distribution is a mixture of right truncated data from stage 1
and untruncated normal data from stage 2. The final column shows unconditional
distribution of the MLE.

(a) $x_{1}:x_{1}\geq 2.18$, $\theta=0$ (b) $\frac{x_{1}+x_{2}}{2}:x_{1}<2.18$,
$\theta=0$ (c) $\widehat{\theta}$, $\theta=0$

(d) $x_{1}:x_{1}\geq 2.18$, $\theta=2.18$ (e)
$\frac{x_{1}+x_{2}}{2}:x_{1}<2.18$, $\theta=2.18$ (f) $\widehat{\theta}$,
$\theta=2.18$

Figure 1:  Distribution of MLEs from the experiment described in Section 1.1.
$n_{1}=n_{2}=1$; $10^{6}$ Monte-Carlo simulations.

###  1.2 The Scope of this Paper

We consider sequential experiments having a small finite number of interim
decision points, that is, experimental set-ups for which Martingale central
limit theorems and Brownian theory are not suitable. Our interest is in
experiments that aim primarily on a hypothesis test of effect size. We focus
on characterizing the effect of sequential stopping rules on probability
distributions of test statistics.

In this manuscript, Section 2 introduces notation for GSDs with stopping rules
dependent on a parameter of interest [through the distributions of the test
statistics]. This section also presents distributions of cumulative test
statistics conditional on reaching a stage, conditional on stopping at a
stage, and unconditional defined on the probability space with truncation-
adapted support. All of these probability distributions are truncated or
truncated-mixtures. Section 3 presents likelihood-based inference on the
truncation-adapted probability space. In section 3.1 a local asymptotic
distribution of the MLEs is found to be non-degenerate; it is a mixture of
truncated normal distributions. Section 3.2 shows that the possibility of
early stopping does not change the monotonicity of likelihood ratios in the
one-parameter exponential family. Thus, stage-specific tests continue to be
uniformly most powerful by the Karlin-Rubin theorem, which makes sequential
tests based on monotone likelihood ratio uniformly most powerful. Throughout
Sections 2 and 3 theoretical results are illustrated by a two stage example,
Pocock (1977). Finally, Section 4 concludes this article with a summary and a
discussion of impact within the contemporary research environment.

##  2 Probability Distributions with Early Stopping

Let $X$ denote subjects’ outcome variable and assume that, when observed in
isolation, it has a probability distribution function or a probability mass
function $f_{X}=f_{X}\left(x|\theta\right)$. To simplify the material, the
term density is used to refer to probability measures without formally
distinguishing between them. Let a sequence of $X$s be observed with the
primary objective of testing the null hypothesis $H_{0}:\theta=0$ with overall
$\alpha$-level type 1 error and $1-\beta$ power at an alternative
$H_{1}:\theta=\theta_{1}$.

It is convenient to group the random sample into _stages_ separated by the
interim analysis times: $\mathbf{X}_{1},\mathbf{X}_{2},\ldots$, where
$\mathbf{X}_{k}=(X_{n_{(k-1)}+1},\ldots,X_{n_{(k)}})$;
$n_{(k)}=\sum_{j=1}^{k}n_{j}$; here for simplicity $n_{j}$ is a pre-specified
number of observations in stage $j$, $1\leq k\leq K$; $n_{(0)}=0$ and $K-1$ is
the maximum number of interim analyses permitted. Every stage is assumed to be
“reachable”, that is, there is a positive probability of reaching each stage.
Data collection at each stage is followed by a hypothesis test that results in
a decision to stop the study or to enroll a new group of patients; except that
if stage $K$ is reached, the experiment stops after $n_{(K)}$ observations
regardless of the last $n_{K}$ observations’ values.

###  2.1 Stopping Decisions

Let $T_{(k)}$ be a function of observations
$\mathbf{X}_{(k)}=(X_{1},\ldots,X_{n_{(k)}})$ that is compared against a
cutoff value $c_{k}$ to determine whether to stop at stage $k$ or continue
through stage $k+1$, $1\leq k<K-1$. These decisions are defined by the events
$\left\\{\cap_{j=1}^{k-1}\left\\{T_{(j)}\leq
c_{j}\right\\}\right\\}\cap\left\\{T_{(k)}>c_{k}\right\\}$ and
$\cap_{j=1}^{K-1}\left\\{T_{(j)}\leq c_{k}\right\\}$, respectively, and are
conveniently summarized by a random variable denoting the stopping stage:

$D=K\cdot I\left(\cap_{j=1}^{K-1}\left\\{T_{(j)}\leq
c_{j}\right\\}\right)+\sum_{k=1}^{K-1}k\cdot
I\left(\left(\cap_{j=1}^{k-1}\left\\{T_{(j)}\leq
c_{j}\right\\}\right)\cap\left\\{T_{(k)}>c_{k}\right\\}\right);$

$D\in\\{1,\ldots,K\\}$ will appear as random index such as in
$\mathbf{X}_{(D)}$ to emphasize that the stopping stage is unknown and is
described probabilistically though the random variable $D$. $\mathbf{X}_{(d)}$
is the random variable $\mathbf{X}_{(D)}$ conditioned on stopping with stage
$D=d,d=1,\ldots,K$.

It is important to account for $D$ in probability statements about $T_{(d)}$
because $D$ determines the observations’ probability support as illustrated in
Section 1.1.

###  2.2 After deciding to stop

(a) Sketch of disjoint support regions for subdensities by (scaleless) stage-
specific test statistics (b) Sketch of disjoint support regions for
subdensities by (scaleless) cumulative test statistics

Figure 2:  Support associated with different stopping decisions when $K=3$;
$c_{k}$ is the critical value for stopping at stage $k$.

If study stops at stage $D=d<K$, $H_{0}$ is rejected. If $d=K$, the final
hypothesis test determines acceptance or rejection of $H_{0}$. At the time of
the final analysis, the density of observations conditional on stopping at
stage $d$ (i.e., the density of $\mathbf{X}_{(d)}=\mathbf{X}_{(D)}|\\{D=d\\}$)
is

$\displaystyle
f_{\mathbf{X}_{(d)}}^{C}=f_{\mathbf{X}_{(D)}}\left(\textbf{{x}}_{(D)}|D=d,\theta\right)=\frac{I\left(D=d\right)}{\text{Pr}_{\theta}\left(D=d\right)}f_{\mathbf{X}_{(d)}}\left(\textbf{{x}}_{(d)}|\theta\right)=\frac{f_{\mathbf{X}_{(d)}}^{sub}}{\text{Pr}_{\theta}\left(D=d\right)},$
(4)

where
$f_{\mathbf{X}_{(d)}}^{sub}=[f_{\mathbf{X}_{(d)}}(\textbf{{x}}_{(d)}|\theta)]^{I\left(D=d\right)}$
denotes the sub-density with support defined by $D=d$ (see the exemplary
sketches in Figure 2). Similarly, for a statistic
$T_{(d)}=T(\mathbf{X}_{(d)})$, the conditional on $D=d$ density is
$f_{T_{(d)}}^{C}$.

In contrast, if the stopping rule is not random and the experiment stops with
$n_{(d)}$ observations, $\text{Pr}_{\theta}(D=d)\equiv 1$ and the observations
have density $f_{\mathbf{X}_{(d)}}$.

###  2.3 At the Time of Experimental Design

Prior to data collection, both $D$ and $X_{(D)}$ are unknown and the joint
density of $X_{(D)}$ can be written in several ways:

$\displaystyle f_{\mathbf{X}_{(D)}}\left(\textbf{{x}}_{(D)}|\theta\right)$
$\displaystyle=$
$\displaystyle\sum_{d=1}^{K}[f_{\mathbf{X}_{(d)}}(\textbf{{x}}_{(d)}|\theta)]^{I\left(D=d\right)}$
(5) $\displaystyle=$
$\displaystyle\sum_{d=1}^{K}f_{\mathbf{X}_{(d)}}^{sub}=\sum_{d=1}^{K}\text{Pr}_{\theta}\left(D=d\right)f_{\mathbf{X}_{(d)}}^{C}.$

The joint density is a mixture of densities corresponding to possible outcome
vectors; i.e., these densities are defined on non-overlapping regions of the
density’s support.

When a test statistic $T_{d}$ summarizes $d$th stage data, the density of
$T_{(D)}$ can be written analogous to Equation (5) as

$\displaystyle f_{T_{(D)}}\left({\textit{t}}_{(D)}|\theta\right)$
$\displaystyle=$
$\displaystyle\sum_{d=1}^{K}[f_{{T}_{(d)}}({\textit{t}}_{(d)}|\theta)]^{I\left(D=d\right)}$
(6) $\displaystyle=$
$\displaystyle\sum_{d=1}^{K}f_{{T}_{(d)}}^{sub}=\sum_{d=1}^{K}\text{Pr}_{\theta}\left(D=d\right)f_{{T}_{(d)}}^{C}.$

Even if every stage-specific test statistic $T_{d}$ is normally distributed,
the distribution of $T_{(D)}$ is not.

###  2.4 At Interim Hypothesis Testing

Suppose at stage $d-1$, the decision was made to continue sampling, the
support for the $d$th test statistic is characterized by $D\geq d$. The
density of $T_{(D)}$ conditional on $D\geq d$ is

$\displaystyle f_{T_{(D)}}\left(t_{(D)}|D\geq d,\theta\right)$
$\displaystyle=$
$\displaystyle\frac{\sum_{k=d}^{K}f_{T_{(k)}}^{sub}}{\text{Pr}_{\theta}\left(D\geq
d\right)}=\sum_{k=d}^{K}\frac{\text{Pr}_{\theta}\left(D=k\right)}{\text{Pr}_{\theta}\left(D\geq
d\right)}f_{T_{(k)}}^{C}.$ (7)

Again, even if each $T_{k}$ is normal, the distributions of $T_{(D)}|D\geq d$
are not.

####  2.4.1 Connection with Armitage’s algorithm

SAS’s popular SEQDESIGN procedure, R’s gsDesign package, Cytel’s EAST and
others assess type I and power properties using a recursive sub-density
formula (Armitage et al. (1969)) to evaluate the distribution of
$T_{(D)}|D\geq d$. Armitage’s subdensity is $f_{T_{(D)}|D\geq
d}^{sub}=\sum_{k=d}^{K}f_{T_{(k)}}^{sub}$ and

$\displaystyle f_{T_{(D)}}\left(t_{(D)}|D\geq d,\theta\right)$
$\displaystyle=$ $\displaystyle\frac{f_{T_{(D)}|D\geq
d}^{sub}}{\text{Pr}_{\theta}\left(D\geq d\right)}.$ (8)

For example, at $K=2$, the sub-density of $T_{(D)}|D\geq d$ is

$f^{sub}_{T_{(D)}|D\geq 2}(t|\theta)=\int_{-\infty}^{c_{1}}f_{T_{(D\geq
2)}|T_{(1)}}(t|t_{1},\theta)f_{T_{(1)}}(t_{1}|\theta)dt_{1}$ (9)

and its density is $f_{T_{(D)}|D\geq 2}(t|\theta)=f^{sub}_{T_{(D)}|D\geq
d}(t|\theta)\left(\int_{-\infty}^{c_{1}}f_{T_{(1)}}(t_{1}|\theta)dt_{1}\right)^{-1}$.
Recursively, the density conditional on reaching the $d$th interim analysis is

$f_{T_{(D)}}(t|D\geq d,\theta)=\frac{\int_{-\infty}^{c_{d-1}}f_{T_{(D\geq
d)}|T_{(d-1)}}(t|t_{d-1},\theta)f_{T_{(d-1)}}(t_{d-1}|\theta)dt_{d-1}}{\int_{-\infty}^{c_{d-1}}f_{T_{(d-1)}}(t_{d-1}|\theta)dt_{d-1}}.$
(10)

###  2.5 $\sigma-$fields and support defined by a set of critical values

At the time of $d$th hypothesis test, given $D\geq d$, the values
$\\{x_{j}:j>n_{(d)}\\}$ are not observable and hence do not contribute to the
density; indeed, they do not belong to the adaptation-rule driven sample
space, and consequently, they do not belong to a $\sigma$-field of the random
process being monitored: hence, they do not belong to the sequential
experiment as a whole. In this paper, by analogy with structural zeroes in
contingency tables, these values are excluded from the sample space.

The $d$th stage-specific test statistic $T_{d}=T\left(\textbf{X}_{d}\right)$
is defined on a probability space
$\left({\cal{T}}_{d},\sigma\left({\cal{T}}_{d}\right),P_{d}\right)$, where
${\cal{T}}_{d}$ is typically a real line (${\cal{R}}$),
$\sigma\left({\cal{T}}_{d}\right)$ is Borel $\sigma-$field and $P_{d}$ is a
probability measure on the measurable space
$\left({\cal{T}}_{d},\sigma\left({\cal{T}}_{d}\right)\right)$. A sequence of
nested $\sigma$-fields
${\cal{F}}_{(d)}:=\sigma\left({\cal{T}}_{1}\right)\times\cdots\times\sigma\left({\cal{T}}_{d}\right),$
$d=1,\ldots,K$, creates a filtration
$\textbf{F}=\left({\cal{F}}_{(d)}\right)_{d\leq K}$ on the product probability
space $\left({\cal{T}},{\cal{F}}_{(K)},P\right)$, where
${\cal{T}}={\cal{T}}_{1}\times\cdots\times{\cal{T}}_{K}$ and
$P=P_{1}\times\cdots\times P_{K}$. But in the presence of possible stopping,
not all combinations of $(T_{1},\ldots,T_{K})$ are possible. The cumulative
test statistics $T_{(d)}=T_{(d)}(T_{1},\ldots,T_{d})$ are defined only on a
subspace of the sample space ${\cal{T}}_{1}\times\cdots\times{\cal{T}}_{d}$.
Thus, probability environment substantially changes.

####  2.5.1 Interim hypothesis testing

The statistic $T_{(1)}=T_{1}$ conditional on reaching stage $1$ $(D\geq 1)$ is
defined on the sample space ${\cal{T}}_{(1)}={\cal{T}}_{1}$; so
$\sigma\left({\cal{T}}_{(1)}\right)={\cal{F}}_{(1)}$. The statistic $T_{(2)}$
conditional on $D\geq 2$ is defined on

${\cal{T}}_{(2)}=\left[(t_{1},t_{2}):\\{t_{1}\leq c_{1}\\}\right],$

and $\sigma\left({\cal{T}}_{(2)}\right)\subset{\cal{F}}_{(2)}$. Further, for
$d\in\\{3,\ldots,K\\}$, $T_{(d)}$ conditional on $D\geq d$ is defined on

${\cal{T}}_{(d)}=\left((t_{1},\ldots,t_{d}):\left\\{t_{(j)}\leq
c_{j}\right\\},j=1,\ldots,d-1\right).$

For all $d$, the support ${\cal{T}}_{(d)}$ and the $\sigma$-field
$\sigma\left({\cal{T}}_{(d)}\right)$ is reduced by the possibility of early
stopping: $\sigma\left({\cal{T}}_{(d)}\right)\subset{\cal{F}}_{(d)}$. This
creates new measurable spaces
$\left({\cal{T}}_{(d)},\sigma\left({\cal{T}}_{(d)}\right)\right)$ for interim
tests at every possible stage $1,\ldots,K$. Armitage’s recursive sub-density
formula [Armitage et al. (1969)] is defined on this measurable space; see
Equation (9) for $K=2$. Re-scaling yields a density function [see Equation
(10)] which defines a probability measure to complete the probability space
$\left({\cal{T}}_{(d)},\sigma\left({\cal{T}}_{(d)}\right),P_{(d)}\right)$,
where the probability measure $P_{(d)}$ is determined by density (10):
$P_{(d)}(A)=\int_{A}f_{T_{(D)}}(t|D\geq d,\theta)dt$,
$A\in\sigma\left({\cal{T}}_{(d)}\right)$.

####  2.5.2 After the stop decision is made

At each interim stage $d=1,\ldots,K-1$, the decision to reject or accept
$H_{0}$ splits ${\cal{T}}_{(d)}$ into two non-overlapping regions denoted
${\cal{T}}_{(d)}^{stop}$ and ${\cal{T}}_{(d)}^{cont}$, respectively. Since
${\cal{T}}_{(d)}^{stop}\cap{\cal{T}}_{(d+1)}=\emptyset$ and
${\cal{T}}_{(d)}^{cont}\subset{\cal{T}}_{(d+1)}$, then the sets
${\cal{T}}_{(1)}^{stop},\ldots,{\cal{T}}_{(K-1)}^{stop},$ and
${\cal{T}}_{(K)}$ make a partition of the sample space of $T_{(D)}$:

${\cal{T}}_{(K)}+\sum_{k=1}^{K-1}{\cal{T}}_{(k)}^{stop}=\left(\cup_{d=1}^{K-1}{\cal{T}}_{(d)}^{stop}\right)\cup{\cal{T}}_{(K)}=\cup_{d=1}^{K}{\cal{T}}_{(d)}\subset{\cal{T}}.$

We define a probability space for the observable random variable ${T}_{(d)}$
using the probability measures $P_{(d)}$ defined by $f_{T_{(d)}}^{C}$ on the
measurable space
$\left({\cal{T}}_{(d)}^{stop},\sigma\left({\cal{T}}_{(d)}^{stop}\right)\right)$.
At $D=K$, the probability measure $P_{(K)}$ defined by $f_{T_{(K)}}^{C}$
completes the probability space for a measurable space
$\left({\cal{T}}_{(K)},\sigma\left({\cal{T}}_{(K)}\right)\right)$.

Thus, at the end of the study, at $D=d$, the researcher operates with an
_observable_ random variable $T_{(d)}$.

####  2.5.3 At the design stage

The conditional random variables, $T_{(d)}$, defined on non-overlapping
$\sigma$-fields are combined together into the unconditional random variable
$T_{(D)}\sim f_{T_{(D)}}$ defined on the sample space
$\left(\cup_{d=1}^{K-1}{\cal{T}}_{(d)}^{stop}\right)\cup{\cal{T}}_{(K)}$. The
probability distribution defined on the $\sigma$-field on this sample space is
$P_{(D)}=\sum_{d=1}^{K}\text{Pr}\left(D=d\right)P_{(d)}$.

####  2.5.4 Impossible events

The set ${\cal{T}}_{0}={\cal{T}}\setminus\cup_{d=1}^{K}{\cal{T}}_{(d)}$
contains all impossible combinations of $(t_{1},\ldots,t_{K})$ under a chosen
stopping rule. If $K=2$, for example, then
${\cal{T}}\setminus\cup_{d=1}^{2}{\cal{T}}_{(d)}=\left\\{(t_{1},t_{2}):t_{1}>c_{1}\right\\}$.
Without the possibility of early stopping, all combinations of
$(t_{1},\ldots,t_{K})\in{\cal{T}}$ would be possible.

###  2.6 Pocock’s Example: One-Sided Two-Group Sequential Z-test

Pocock (1977) proposed a simple two-stage procedure for testing
$H_{0}:\theta=0$ with a pre-determined power at $H_{1}:\theta=\theta_{1}$ on
normal data. With $n_{1}=n_{2}$, $c_{1}=c_{2}=2.18$ is used to secure the
overall type 1 error rate $\alpha=0.025$ with a one-sided $z$ test. Let
$n_{0}=0$ and

$Z_{k}=\frac{1}{\sqrt{n_{k}}}\sum_{i=n_{(k-1)}+1}^{n_{(k)}}X_{i}=\sqrt{n_{k}}\cdot\bar{X}_{k}\overset{d}{=}\mathcal{N}(\theta,1),\quad
k=1,2.$

The study is stopped for efficacy at stage 1 if $Z_{1}\geq 2.18$ and proceeds
to stage 2 when $Z_{1}<2.18$. If the study is stopped at stage 1, the support
for $\bar{X}_{1}|Z_{1}\geq 2.18$ starts at $2.18/\sqrt{n_{1}}$ and stretches
to $+\infty$. If the study continues through stage 2, support for
$\bar{X}_{1}|Z_{1}<2.18$ ranges from $-\infty$ to $2.18/\sqrt{n_{1}}$ .

The test statistic for the second stage, under $Z_{1}<2.18$,

$Z_{(2)}=\frac{\sqrt{n_{1}}}{\sqrt{n_{1}+n_{2}}}Z_{1}+\frac{\sqrt{n_{2}}}{\sqrt{n_{1}+n_{2}}}Z_{2}=\frac{1}{\sqrt{n_{1}+n_{2}}}\sum_{i=1}^{n_{1}+n_{2}}X_{i}.$

The distribution of $Z_{(2)}$ is a mixture of a right-truncated normal
$Z_{1}|Z_{1}<2.18$ and the normal $Z_{2}$.

Figure 3 shows histograms of $Z_{1}|Z_{1}\geq 2.18$, $Z_{(2)}|Z_{1}<2.18$, and
$Z_{(D)}$ estimated from $100,000$ Monte-Carlo samples assuming $\theta=2.18$.
These histograms based on $n_{1}=n_{2}=100$ are almost identical to the
histograms in Figure 1 based on $n_{1}=n_{2}=1$. This highlights on the
important message that non-normality continues to be present even
asymptotically Tarima and Flournoy (2019a).

(a) $Z_{1}|Z_{1}\geq 2.18$, $\theta=2.18$ (b) $Z_{(2)}|Z_{1}<2.18,$
$\theta=2.18$ (c) $Z_{(D)},\theta=2.18$

Figure 3:  Histograms of test statistics from a two-stage Pocock’s experiment
with critical value $c_{1}=2.18$; $f_{\mathbf{X}_{k}}=\mathcal{N}(2.18,1)$ and
$n_{k}=100,k=1,2$.

##  3 Likelihood-based Inference with Early Stopping

If the stopping rule is not random and the experiment stopped with $n_{(d)}$
observations, $\text{Pr}_{\theta}(D=d)\equiv 1$, the likelihood is

$\displaystyle{\mathcal{L}}^{fix}\left(\theta|k,\textbf{{x}}_{(d)}\right)=f_{\mathbf{X}_{(d)}}.$
(11)

Considering the joint density (5) conditional on the data
$\left(d,\textbf{{x}}_{(d)}\right)$ observed at the end of a sequential
experiment, the likelihood is

$\displaystyle{\mathcal{L}}\left(\theta|d,\textbf{{x}}_{(d)}\right)=f_{\mathbf{X}_{(d)}}^{sub}.$
(12)

The indicator in (4) emphasizes that support for the random variables is
reduced by the conditioning; see this illustrated in Figure 2. Note also in
Figure 2 that the support conditional on stopping at one stage is disjoint
from the support conditional on stopping at another stage.

${\cal{L}}$ is a function of $\theta$ and the observed data
$(d,\textbf{{x}}_{(d)})$, but ${\cal{L}}$ is a continuous function only of
$\theta$, as discontinuities in $(d,\textbf{{x}}_{(d)})$ arise from the
mixture distribution of $\mathbf{X}_{(D)}$. These discontinuities are
inherited by MLEs and other statistics derived from them. Conditional on
$(d,\textbf{{x}}_{(d)})$, MLEs maximizing ${\mathcal{L}}$ are

$\displaystyle\widehat{\theta}$
$\displaystyle=\arg\max_{\theta}f_{\mathbf{X}_{(d)}}^{sub}.$

For every $\left(d,\textbf{{x}}_{(d)}\right)$,
$f_{\mathbf{X}_{(d)}}^{sub}=f_{\mathbf{X}_{(d)}}$ for all $\theta$;
consequently,

$\widehat{\theta}^{fix}:=\arg\max_{\theta}f_{\mathbf{X}_{(d)}}=\arg\max_{\theta}f_{\mathbf{X}_{(d)}}^{sub}=\widehat{\theta},$

that is, as is well known, making stopping decisions does not alter maximum
likelihood point estimates; they are the same whether obtained by maximizing
${\mathcal{L}}$ or ${\mathcal{L}}^{fix}$ and other observed statistics derived
from the likelihood (e.g., the score function and the observed information)
are unaffected as well.

The following example extends the simple one in Section 1.1 with
$n_{1}=n_{2}=1$ to arbitrary $n_{1}$ and $n_{2}$ to illustrate (as is proven
later) that, although maximum likelihood point estimates and test statistics
are unaffected by early stopping decisions, their probability distribution
does not tend to normality even with larger sample sizes.

###  3.1 Large Sample Properties

The $d$th stage-specific MLEs $\widehat{\theta}_{d}$ of $\theta$ and their
statistical models are called _regular_ if, without the possibility of early
stopping,

$\xi_{d}=\sqrt{n_{d}}\left(\widehat{\theta}_{d}-\theta\right)\overset{d}{\to}{\cal{N}}(0,\sigma^{2}),$
(13)

where $0<\sigma<\infty$. Assumption (13) was described in Tarima and Flournoy
(2019a) to include the more specific assumptions for

  * •

independent, identically distributed observations by Cramér [e.g., for
example, Ferguson (1996)],

  * •

independent not identically distributed observations [e.g., Philippou et al.
(1973)],

  * •

dependent observations [e.g., Crowder (1976)],

  * •

and densities whose support depends on parameters [e.g., Wang et al. (2014)].

All these specific sets of assumptions include assumptions of the existence
and consistency of the MLE.

With large samples, the MLE at the stage $d$ analysis can be approximated
recursively by

$\displaystyle\widehat{\theta}_{(d)}$
$\displaystyle\approx\frac{n_{(d-1)}}{n_{(d)}}\widehat{\theta}_{(d-1)}+\frac{n_{d}}{n_{(d)}}\widehat{\theta}_{d}\approx\sum_{j=1}^{d}\frac{n_{j}}{n_{(d)}}\widehat{\theta}_{d},$
(14)

where $\widehat{\theta}_{(d-1)}$ is an MLE based on cumulative data from
stages $1$ to $d-1$ and $\widehat{\theta}_{d}$ is the $d$th stage-specific
MLE. The standardized $d$th stage-specific MLE is so,

$T_{(d)}=\sqrt{n_{(d)}}\sum_{j=1}^{d}\frac{n_{j}}{n_{(d)}}\left(\frac{\widehat{\theta}_{j}-\theta}{\sigma}\right)=\sum_{j=1}^{d}\sqrt{\frac{n_{j}}{n_{(d)}}}\
\xi_{j}\approx\sqrt{n_{(d)}}\left(\frac{\widehat{\theta}_{(d)}-\theta}{\sigma}\right),$

that is, $T_{(d)}$ is (approximately) the standardized $d$th stage-specific
MLE. The asymptotic properties of $T_{(D)}$ depend on the existence and
distribution of a limiting random variable $r_{(D)}$ defined by

$\sum_{d=1}^{K}I(D=d)\frac{n_{(d)}}{n_{d}}\overset{d}{\to}\sum_{d=1}^{K}I(D=d)r_{(d)}=r_{(D)},$
(15)

where $r_{(d)}=\lim_{n_{d}\to\infty}n_{(d)}/n_{d}$ is the asymptotic ratio of
the $d$th cumulative-stage and stage-specific sample sizes [Theorem 1 in
Tarima and Flournoy (2019a)]; $r_{(D)}$ is a multinomial random variable with
support on $r_{(d)}$.

Assume the limits $n_{(d)}/n_{j}\to r_{(d)j}\in(0,\infty)$, $j\leq d$ exist
with $r_{(d)d}=r_{(d)}$. Then given $D=d$,

$T_{(d)}\to\sum_{j=1}^{d}\frac{\xi_{j}}{\sqrt{r_{(d)j}}}.$

While $\xi_{j}\overset{d}{\to}{\cal{N}}(0,1)$, the distributions used for the
final analysis, the interim analysis and experimental design, respectively,
are mixtures of truncated distributions:

$\displaystyle{\text{Pr}_{\theta}}\left(T_{(D)}<v|D=d\right)$
$\displaystyle\to$
$\displaystyle\text{Pr}_{\theta}\left(\sum_{j=1}^{d}\frac{\xi_{j}}{\sqrt{r_{(d)j}}}<v\Big{|}D=d\right),$
$\displaystyle{\text{Pr}_{\theta}}\left(T_{(D)}<v|D\geq d\right)$
$\displaystyle\to$
$\displaystyle\sum_{k=d}^{K}{\text{Pr}_{\theta}}\left(D=k\right){\text{Pr}_{\theta}}\left(\sum_{j=1}^{k}\frac{\xi_{j}}{\sqrt{r_{(k)j}}}<v\Big{|}D=k\right),$
$\displaystyle\text{Pr}_{\theta}\left(T_{(D)}<v\right)$ $\displaystyle\to$
$\displaystyle\sum_{k=1}^{K}{\text{Pr}_{\theta}}\left(D=k\right){\text{Pr}_{\theta}}\left(\sum_{j=1}^{k}\frac{\xi_{j}}{\sqrt{r_{(k)j}}}<v\Big{|}D=k\right).$

### Pocock’s Example: Large Sample Properties

Under assumption (13) with large sample sizes,
$\widehat{\theta}_{(D)}=I(D=1)\widehat{\theta}_{(1)}+I(D=2)\widehat{\theta}_{(2)}$
are standardized as

$\displaystyle T_{(D)}$
$\displaystyle=\sqrt{n_{(D)}}\left(\widehat{\theta}_{(D)}-\theta\right)/\sigma$
$\displaystyle=I(D=1)\sqrt{n_{1}}\left(\widehat{\theta}_{(1)}-\theta\right)/\sigma+I(D=2)\sqrt{n_{1}+n_{2}}\left(\widehat{\theta}_{(2)}-\theta\right)/\sigma.$
(16)

Assume the limit
$I(D=1)+I(D=2)\frac{n_{1}+n_{2}}{n_{2}}\overset{d}{\to}r_{(D)}$ exists as
$n_{1}\to\infty$, Then, adopting the local alternative hypothesis
$\theta=h/\sqrt{n_{1}}$ yields a non-degenerate limiting distribution of
$T_{(D)}$ that models both stages of the experiment:

$\displaystyle\text{Pr}_{\theta}\left(T_{(D)}<v\right)$ $\displaystyle\to$
$\displaystyle p_{1}\Phi\left(\left.v\right|D=1\right)$ (17) $\displaystyle+$
$\displaystyle(1-p_{1})\int_{-\infty}^{c_{1}}\Phi\left(\sqrt{r_{(D)}}v-\sqrt{r_{(D)}-1}y\right)\phi\left(y|D=2\right)dy,$

where $p_{1}=\lim_{n_{1}\to\infty}{\text{Pr}_{\theta}}\left(D=1\right)$ is the
limiting stage 1 stopping probability; $\phi$ and $\Phi$ denote the standard
normal density and cumulative distribution function, respectively. With a
fixed alternative $\theta$,
$\text{Pr}_{\theta}\left(T_{(D)}<v\right)\to\Phi(v)$ because $p_{1}\to 1$ with
probability 1, and modeling related to stage 2 data is lost.

In Pocock’s example,
$T_{(D)}=I(D=1)Z_{1}+I(D=2)\left(Z_{1}+Z_{2}\right)/\sqrt{2}$ and

$\displaystyle\text{Pr}_{\theta}\left(T_{(D)}<v\right)$ $\displaystyle\to$
$\displaystyle\text{Pr}_{\theta}\left(Z_{1}>c_{1}\right)\Phi\left(\left.v\right|Z_{1}>c_{1}\right)$
(18) $\displaystyle+$ $\displaystyle\text{Pr}_{\theta}\left(Z_{1}\leq
c_{1}\right)\int_{-\infty}^{c_{1}}\Phi\left(\sqrt{2}v-y\right)\frac{\phi\left(y\right)}{\text{Pr}\left(y\leq
c_{1}\right)}dy$

Note if $Z_{1}\leq c_{1}$, then

$\text{Pr}_{\theta}\left(\frac{Z_{1}+Z_{2}}{\sqrt{2}}<v|Z_{1}\leq
c_{1}\right)=\int_{-\infty}^{c_{1}}\Phi\left(\sqrt{2}v-y\right)\frac{\phi\left(y\right)}{\text{Pr}\left(y\leq
c_{1}\right)}dy,$

which is a continuous mixture of distributions.

###  3.2 Most Powerful Group Sequential Tests

Let $X\sim f_{X}(\theta)$, where $f$ belongs to the one-parameter exponential
family. Without a possibility of early stopping, the likelihood for a
realization $\textbf{{x}}=\left(x_{1},\ldots,x_{n}\right)$ of a random sample
$\textbf{X}=\left(X_{1},\ldots,X_{n}\right)$ is

$\displaystyle{\cal{L}}\left(\theta|\textbf{{x}}\right)=\prod_{i=1}^{n}f_{X}\left(\textbf{{x}}\right)$
$\displaystyle=h\left(\textbf{{x}}\right)g\left(T\left(\textbf{{x}}\right)|\theta\right)=h\left(\textbf{{x}}\right)e^{\eta(\theta)T\left(\textbf{{x}}\right)+A\left(\theta\right)}$

where all relevant information about $\theta$ is absorbed by a sufficient
statistic $T\left(\textbf{{x}}\right)$. Assume the test statistic $Z$ is a
one-to-one transformation of $T$.

If $LR(t)=g(t|\theta)/g(t|\theta_{0})$ has monotone likelihood ratio (MLR) in
$t$, then the Karlin-Rubin theorem provides uniformly most powerful (UMP)
tests.

In sequential testing settings, when $d$th stage is reached $\left(D\geq
d\right)$, the interim likelihood

$\displaystyle{\cal{L}}\left(\theta|D\geq d,\textbf{{x}}_{(d)}\right)$
$\displaystyle=$ $\displaystyle I\left(D\geq
d\right)h\left(\textbf{{x}}_{(d)}\right)e^{\eta(\theta)T\left(\textbf{{x}}_{(d)}\right)+A_{d}\left(\theta\right)}$
(19)

has the associated interim likelihood ratio

$\displaystyle LR(t|D\geq d)=\frac{{\cal{L}}\left(\theta|D\geq
d,\textbf{{x}}_{(d)}\right)}{{\cal{L}}\left(\theta_{0}|D\geq
d,\textbf{{x}}_{(d)}\right)}.$

For every $d$th stage hypothesis test, $D\geq d$ and

$LR(t|D\geq
d)=\exp\left[\left(\eta(\theta_{1})-\eta(\theta_{0})\right)T_{(d)}+\left(A_{d}(\theta_{1})-A_{d}(\theta_{0})\right)\right]$

which means that the MLR property is preserved with early stopping.

Definition: A sequence of $\alpha_{d}$-level interim tests
$\\{T_{(1)}>c_{1}\\},\ldots,\\{T_{(D)}>c_{D}\\}$ will be called the sequential
test.

###### Theorem 3.1

For any fixed $(\alpha_{1},\ldots,\alpha_{K})$ and
$\left(n_{1},\ldots,n_{K}\right)$ $(1)$ the interim LR test
$\\{T_{(d)}>c_{d}\\}$ is a UMP $\alpha_{d}$-level test and $(2)$ no sequential
test is more powerful than the sequential test based interim LRs.

Feature | Mathematical Definition
---|---
overall type $1$ error | $\alpha=\sum_{k=1}^{K}\alpha_{k}\prod_{j=1}^{k-1}\text{Pr}_{0}\left(Z_{(j)}\leq u_{j}\right)$
stage-specific type 1 error | $\alpha_{d}=\text{Pr}_{0}\left(Z_{(d)}>u_{d}\big{|}\cap_{j=1}^{d-1}\left\\{Z_{(j)}\leq u_{j}\right\\}\right)$
$\alpha$-spending function | $\alpha_{(d)}=\sum_{j=1}^{d}\alpha_{j}\prod_{i=1}^{j-1}\text{Pr}_{0}\left(Z_{(j)}\leq u_{j}\right)$
overall type 2 error | $\beta(\theta)=\sum_{j=1}^{K}\beta_{j}(\theta)\prod_{i=1}^{j-1}\text{Pr}_{\theta}\left(Z_{(j)}\leq u_{j}\right)$
stage-specific type 2 error | $\beta_{d}(\theta)=\text{Pr}_{\theta}\left(Z_{(d)}>u_{d}\big{|}\cap_{j=1}^{d-1}\left\\{Z_{(j)}\leq u_{j}\right\\}\right)$
$\beta$-spending function | $\beta_{(d)}\left(\theta\right)=\sum_{j=1}^{d}\beta_{j}(\theta)\prod_{i=1}^{j-1}\text{Pr}_{\theta}\left(Z_{(j)}\leq u_{j}\right)$
overall power | $1-\beta(\theta)$
stage-specific power | $1-\beta_{d}(\theta)$
cumulative power | $1-\beta_{(d)}\left(\theta\right)$
Table 1: Definitions of Operational Characteristics for Sequential Tests
$d=1,\ldots,K$; $\prod_{i=1}^{0}[\cdot]=1$.

Proof. $(1)$ As shown above, conditional on reaching stage $d$, $(D\geq d)$,
$T_{(d)}$ continues to be sufficient in exponential families and the MLR
property is preserved. By the Karlin-Rubin theorem, the test based on
$T_{(d)}$ is uniformly most powerful at $d^{th}$ stage.
$(2)$ Using Table 1 notation,

$1-\beta(\theta)=\sum_{k=1}^{K}\left(\prod_{j=1}^{k-1}\left[\beta_{j}(\theta)\right]\right)[1-\beta_{k}(\theta)]=1-\prod_{k=1}^{K}\beta_{k}(\theta).$

At $K=1$, $\alpha_{1}$ and $n_{1}$ uniquely define $c_{1}$ and
$\left\\{T_{(1)}>c_{1}\right\\}$ is a UMP by part 1 with a stage-specific
power curve $1-\beta_{1}(\theta)$. At an arbitrary stage $d$, $\alpha_{d}$ and
$n_{d}$ uniquely define $c_{d}$ and, by part 1 of the theorem, the stage-
specific power $1-\beta_{d}(\theta)$, is the highest. Thus, for any choice of
$(\alpha_{1},\ldots,\alpha_{K})$ and $\left(n_{1},\ldots,n_{K}\right)$, and
consequently $\\{T_{(d)}>c_{d}\\}$, the power of the sequential test based on
LRs is $1-\beta(\theta)=1-\prod_{k=1}^{K}\beta_{k}(\theta)$. This power is the
highest for any given $\theta$, because the stage specific type 2 errors
$\beta_{d}(\theta)$ are the lowest for each $d$ at any $\theta$. Q.E.D.

### Pocock’s Example: Likelihood Ratio

For $X_{i}\sim{\cal{N}}(\theta,1)$, $i=1,\ldots,n$, the likelihood ratio for
testing $H_{0}:\theta=\theta_{0}$ vs $H_{A}:\theta=\theta_{1}>\theta_{0}$ is

$\log LR(t)=-n\left(\log\theta_{0}-\log\theta_{1}\right)-t/\theta_{1},$

which has rejection region $\left\\{t>c\right\\}$, where
$t=\sum_{i=1}^{n}X_{i}$ is a sufficient statistic. Using $Z=t/\sqrt{n}\sim
f_{Z}={\cal{N}}(\theta_{1},1)$, the likelihood ratio test is
$\left\\{Z>c/\sqrt{n}\right\\}$.

With Pocock’s example, under $D\geq 1$, $Z_{1}$ is a normal random variable
used for stage 1 hypothesis testing, and

$\log LR(Z_{1}|D\geq
1)=-n_{1}\left(\log\theta_{0}-\log\theta_{1}\right)-\sqrt{n_{1}}Z_{1}/\theta_{1}.$

Then,

$\log LR(Z_{(2)}|D\geq
2)=-(n_{1}+n_{2})\left(\log\theta_{0}-\log\theta_{1}\right)-\sqrt{n_{1}+n_{2}}Z_{(2)}/\theta_{1}+C,$

where
$C=\log\text{Pr}\left(Z_{1}<c_{1}|\theta_{0}\right)-\log\text{Pr}\left(Z_{1}<c_{1}|\theta_{1}\right)$
is independent of data given $D\geq 2$. By Theorem 3.1, $\\{Z_{(d)}\geq
c_{d}\\}$ is the UMP test at stage $d$ given $\alpha_{d}$ and $n_{d}$, and no
other sequential test is more powerful overall.

##  4 Impact and Summary

To establish that a sequence of likelihood ratio tests are most powerful, we
began by constructing the joint probability distribution over the set of
possible events. The use of an early stopping criterion eliminates the
possibility of some realizations of cumulative test statistics
$\mathbf{Z}=\left(Z_{(1)},\ldots,Z_{(K)}\right)$. This makes an otherwise
normal random process $\mathbf{Z}$ unobservable. On its true stopping rule
adapted support, the distribution of $\mathbf{Z}$ is a mixture of truncated
distributions at each stage. Thus, unadapted distributional assumptions should
be modified to take into account the planned adaptation scheme. Liu and Hall
(1999) and Liu et al. (2006) recognized change in support in the one-parameter
exponential family and investigated bias estimation. Schou and Marschner
(2013) recognized presence of truncation in the joint distribution of stage-
specific test statistics, but they mostly focused on bias in meta-analytic
studies. The adapted support is critical for derivation MLEs’ and test
statistics’ distributions in Section 2.

In Section 2, distinct probability measures are derived for design
[unconditional], interim hypothesis testing [conditional on collecting data up
to the time of interim testing] and when the study is completed [conditional
on deciding to stop at a particular stage]. These probability distributions
formalized a new probabilistic framework (Section 2) which is used in Section
3.2 to show no testing sequence is more powerful than sequential likelihood
ratio tests.

Likelihood ratio tests are most powerful for testing simple hypotheses and,
under monotone likelihood ratio, for testing composite hypotheses; see Neyman
and Pearson (1933) and Karlin-Rubin theorem in Ferguson (2014). The Karlin-
Rubin theorem works with the one-parameter exponential family and Section 3.2
shows that most powerful testing continues to hold for sequential tests. Even
though distributions of sample-size-dependent sufficient statistics (Blackwell
(1947)) belong to a curved exponential family (Efron (1975); Liu and Hall
(1999); Liu et al. (2006)), the distributions conditional on the sample size
still belong to one-parameter exponential family. This fact is used in Section
3.2 to show that likelihood ratio sequential tests are most powerful tests
with any pre-determined $\alpha$-spending function. This result is applicable
to many common group sequential designs.

Treatment-effect-dependent (non-ancillary) stopping rules are part of many
common GSDs including Pocock Pocock (1977), O’Brien & Fleming O’Brien and
Fleming (1979), and Haybittle-Peto designs Haybittle (1971); Peto et al.
(1976). If the data follow a normal distribution with known variance (without
possibility of early stopping), then these designs are most powerful for their
$\alpha$-spending functions. Similarly, application of Simes test (see Simes
(1986)) and its recently proposed modification for sequential testing (see
Tamhane et al. (2020)) cannot have higher power than group sequential tests
from Jennison and Turnbull (1999) with the same $\alpha$-spending function.

Historically, many researchers have relied, and currently rely, on joint
normality (e.g., Jennison and Turnbull (1999), Proschan et al. (2006), Kunz et
al. (2020)). Critical adjustments ate often done with recursive sub-density
estimation; see Armitage et al. (1969). Section 2.4.1 places Armitage’s sub-
density formula in the broader probability framework of possible events and
confirms, from the prospective of this $\sigma$-algebra, that the common
current practice of using Armitage’s formula for calculating distributions of
interim test statistics is appropriate.

There is plenty of evidence that adaptive designs make statistics non-normally
distributed. Demets and Lan (1994) point out that the distribution of stage-
specific test statistics is not normal and should be estimated recursively.
Jennison and Turnbull (1999) plot the density of a normal test statistic used
in GSD settings [pages 174-177], where discontinuity points clearly show non-
normality. Li et al. (2002) find the joint density of stage 1 and stage 2
standardized test statistics not to be bivariate normal. Local asymptotic non-
normality was established following sample size recalculations (SSRs) that
depend on an interim observed treatment effect (Tarima and Flournoy (2019a,
b)); and a GSD with a single interim analysis can be viewed as a special case
of an SSR. MLEs converging to random mixtures of normal variables have been
found in other adaptive designs (Ivanova et al. (2000), Ivanova and Flournoy
(2001), May and Flournoy (2009), Lane and Flournoy (2012), Flournoy et al.
(2018)).

Milanzi et al. (2015) developed a likelihood approach that applies when the
early stopping rule does not depend on the parameter of interest. In this
case, sample size adaptation is ancillary to the treatment effect and
asymptotic normality of MLEs holds. Gnedenko and Korolev (1996); Bening et al.
(2012); Christoph et al. (2020); Korolev and Zeifman (2019) assume that the
distribution of the random sample size does not depend on previosly collected
data; Molenberghs et al. (2012) assumes this asymptotically.

Nevertheless, convergence of sample means to non-normal random variables was
shown even for ancillary random sample sizes. In Gnedenko and Korolev (1996),
Gnedenko and Korolev show convergence of standardized sums with random number
of summands of infinitely divisible random variables to mixtures of stationary
distributions. They give conditions for convergence to a mixture of normal
distributions. Bening et al. Bening et al. (2012) and Christoph et al.
Christoph et al. (2020) explore convergence to mixtures of normal
distributions and to Student’s limit distribution. When convergence is _mixed_
(see, for example Häusler and Luschgy (2015)), Lin et al. (2020) shows how
norming with the observed information can result in a normal limit. However,
the requirement for mixed convergence appears strong, and it does not cover
the limiting mixtures obtained in this paper.

The impact of early stopping is pervasive. It affects the probabilistic
characterisation of the tests (e.g., type I error and Fisher Information) as
well as the distributions of MLEs and test statistics. Its effect on Fisher
information, when stopped at different stages, is not widely recognized.

During the design phase, before observations are taken, the full form of the
joint density (5), accounting for all possible events, is appropriate. This
contrasts with the current practice of using the density assuming the
experiment will continue through stage $K$. More details on the differences
resulting from these two design approaches will be the subject of another
paper.

However, normality and asymptotic normality assumptions continue to be
directly used with non-normally distributed statistics. We identify two main
reasons for this.

  1. 1.

Many researchers consider large sample properties against a fixed treatment
effect independent of the sample size. From one point of view, a treatment
effect is a population quantity which does not change with sample size. But if
one develops an asymptotic approximation to the testing environment using a
fixed treatment effect, the statistical experiment stops at the first interim
analysis with probability one for any consistent test; test statistics
degenerate to a point mass; see Section 7.4 in Fleming and Harrington (1991).
Under a fixed treatment effect, the power converges to one and cannot be used
to compare different testing procedures. This issue triggered development of
various descriptions of asymptotic relative efficiency. The most popular
approach is Pitman asymptotic relative efficiency Pitman (1948), where
asymptotic power is evaluated under local alternatives; see Nikitin (1995). In
addition, local alternatives clearly reflect actual practice for experiment
planning. Experiments are never planned for a statistical power = 1. Small
sample size studies (pre-clinical, animal studies) are planned to detect large
effect sizes, moderate sample sizes (typical phase 3 studies) are used to
detect moderate effect sizes, and large sample sizes (epidemiological studies,
like vaccine studies) are used to detect small differences.

Koopmeiners et al. (2012) explored MLEs conditional on stopping, but assumed
asymptotic normality to evaluate their uncertainty. Martens and Logan (2018)
relied on asymptotic normality for evaluating regression coefficients under
the Fine–Gray model in GSD settings. Asendorf et al. (2019) evaluated
asymptotic properties with SSR under a fixed alternative for negative binomial
random variables.

  2. 2.

Some researchers investigate local asymptotic properties when early stopping
is not possible or is ancillary to the treatment effect. This also leads to
asymptotic normality. Scharfstein et al. (1997) show that without possibility
of early stopping “time-sequential joint distributions of many statistics
$\ldots$ are multivariate normal with an independent increments covariance
structure” under local alternatives. These results are generally consistent
with the classical results on local asymptotic normality of Le Cam (1960),
where both mean and variance of the limiting normal distribution depend on the
parameter of interest; see Chapter 7 of van der Vaart (1998). However, Section
3.1 shows that the possibility of early stopping destroys local asymptotic
normality: the limiting distribution of standardized test statistics is a
mixture of truncated normal distributions. Similar findings were previously
proved for non-ancillary sample size recalculations; see Tarima and Flournoy
(2019a).

Gao et al. (2013) is a rare exception in not making a normality assumption;
these authors mostly deal with set operations and probabilities and, using
stage-wise ordering of events, they calculate P-values, confidence intervals,
and a median unbiased estimate of the parameter of interest.

It is recommended that the full form of the joint density (5), accounting for
all possible events, be used for study design before observations are taken.
This contrasts with the current practice of using the density assuming the
experiment will continue through stage $K$.

## Conflict of interest

The authors declare that they have no conflict of interest.

## References

  * Armitage et al. (1969) Armitage, P., C. McPherson, and B. Rowe (1969).  Repeated significance tests on accumulating data.  Journal of the Royal Statistical Society: Series A 132, 235–244.
  * Asendorf et al. (2019) Asendorf, T., R. Henderson, H. Schmidli, and T. Friede (2019).  Sample size re-estimation for clinical trials with longitudinal negative binomial counts including time trends: Sample size re-estimation with longitudinal negative binomial counts.  Statistics in Medicine 38(9), 1503–1528.
  * Bartky (1943) Bartky, W. (1943).  Multiple sampling with constant probability.  The Annals of Matheamtical Staistics 14, 363–377.
  * Bening et al. (2012) Bening, V., N. Galieva, and R. Korolev (2012).  On rate of convergence in distribution of asymptotically normal statistics based on samples of random size.  In Annales Mathematicae et Informaticae, pp. 17–28.
  * Blackwell (1947) Blackwell, D. (1947, 03).  Conditional expectation and unbiased sequential estimation.  Ann. Math. Statist. 18(1), 105–110.
  * Christoph et al. (2020) Christoph, G., M. M. Monakhov, and V. V. Ulyanov (2020).  Second-order chebyshev–edgeworth and cornish–fisher expansions for distributions of statistics constructed from samples with random sizes.  Journal of Mathematical Sciences, 1–29.
  * Crowder (1976) Crowder, M. J. (1976).  Maximum likelihood estimation for dependent observations.  Journal of the Royal Statistical Society. Series B (Methodological) 38(1), 45–53.
  * Demets and Lan (1994) Demets, D. L. and K. K. G. Lan (1994).  Interim analysis: The alpha spending function approach.  Statistics in Medicine 13(13–14), 1341–1352.
  * Dodge and Romig (1929) Dodge, H. and H. Romig (1929).  A method of sampling inspection.  The Bell System Technical Journal 8, 613–631.
  * Efron (1975) Efron, B. (1975, 11).  Defining the curvature of a statistical problem (with applications to second order efficiency).  Ann. Statist. 3(6), 1189–1242.
  * Ferguson (1996) Ferguson, T. (1996).  A Course in Large Sample Theory.  New York: Routledge.
  * Ferguson (2014) Ferguson, T. S. (2014).  Mathematical statistics: A decision theoretic approach, Volume 1.  Academic press.
  * Fleming and Harrington (1991) Fleming, T. R. and D. P. Harrington (1991).  Counting Processes and Survival Analysis.  John Wiley & Sons.
  * Flournoy et al. (2018) Flournoy, N., C. May, and C. Tommasi (2018).  The effects of adaptation on inference for non-linear regression models with normal errors.  arXiv preprint arXiv:1812.03970.
  * Gao et al. (2013) Gao, P., L. Liu, and C. Mehta (2013).  Exact inference for adaptive group sequential designs.  Statistics in medicine 32(23), 3991–4005.
  * Gnedenko and Korolev (1996) Gnedenko, B. V. and V. Y. Korolev (1996).  Random summation: limit theorems and applications.  CRC press.
  * Häusler and Luschgy (2015) Häusler, E. and H. Luschgy (2015).  Stable convergence and stable limit theorems, Volume 74.  Springer.
  * Haybittle (1971) Haybittle, J. (1971).  Repeated assessment of results in clinical trials of cancer treatment.  The British journal of radiology 44(526), 793–797.
  * Ivanova and Flournoy (2001) Ivanova, A. and N. Flournoy (2001).  A birth and death urn for ternary outcomes: stochastic processes applied to urn models.  Chapman and Hall/CRC, Boca Raton.
  * Ivanova et al. (2000) Ivanova, A., W. F. Rosenberger, S. D. Durham, and N. Flournoy (2000).  A birth and death urn for randomized clinical trials: Asymptotic methods.  Sankhy, Series B 62(1), 104–118.
  * Jennison and Turnbull (1999) Jennison, C. and B. Turnbull (1999).  Group Sequential Methods with Applications to Clinical Trials.  Chapman & Hall/CRC Interdisciplinary Statistics. CRC Press.
  * Koopmeiners et al. (2012) Koopmeiners, J. S., Z. Feng, and M. S. Pepe (2012).  Conditional estimation after a two-stage diagnostic biomarker study that allows early termination for futility.  Statistics in Medicine 31(5), 420–435.
  * Korolev and Zeifman (2019) Korolev, V. and A. Zeifman (2019).  From asymptotic normality to heavy-tailedness via limit theorems for random sums and statistics with random sample sizes.  In Probability, Combinatorics and Control, pp. XX–XX. IntechOpen.
  * Kunz et al. (2020) Kunz, C. U., S. Jörgens, F. Bretz, N. Stallard, K. V. Lancker, D. Xi, S. Zohar, C. Gerlinger, and T. Friede (2020).  Clinical trials impacted by the covid-19 pandemic: Adaptive designs to the rescue?  Statistics in Biopharmaceutical Research 0(0), 1–17.
  * Lane and Flournoy (2012) Lane, A. and N. Flournoy (2012).  Two-stage adaptive optimal design with fixed first-stage sample size.  Journal of Probability and Statistics 2012, n/a–n/a.
  * Le Cam (1960) Le Cam, L. M. (1960).  Locally asymptotically normal families of distributions.  Univ. California Publ. Statist. 3, 37–98.
  * Li et al. (2002) Li, G., W. J. Shih, T. Xie, and J. Lu (2002).  A sample size adjustment procedure for clinical trials based on conditional power.  Biostatistics 3(2), 277–287.
  * Lin et al. (2020) Lin, Z., N. Flournoy, and W. F. Rosenberger (2020).  Random norming aids analysis of non-linear regression models with sequential informative dose selection.  Journal of Statistical Planning and Inference 206, 29–42.
  * Liu and Hall (1999) Liu, A. and W. Hall (1999).  Unbiased estimation following a group sequential test.  Biometrika 86(1), 71–78.
  * Liu et al. (2006) Liu, A., W. Hall, K. F. Yu, and C. Wu (2006).  Estimation following a group sequential test for distributions in the one-parameter exponential family.  Statistica Sinica 16(1), 165–181.
  * Martens and Logan (2018) Martens, M. J. and B. R. Logan (2018).  A group sequential test for treatment effect based on the Fine–Gray model.  Biometrics 74(3), 1006–1013.
  * May and Flournoy (2009) May, C. and N. Flournoy (2009).  Asymptotics in response-adaptive designs generated by a two-color, randomly reinforced urn.  The Annals of Statistics 37(2), 1058–1078.
  * Milanzi et al. (2015) Milanzi, E., G. Molenberghs, A. Alonso, M. G. Kenward, A. A. Tsiatis, M. Davidian, and G. Verbeke (2015).  Estimation after a group sequential trial.  Statistics in Biosciences 7(2), 187–205.
  * Molenberghs et al. (2012) Molenberghs, G., M. G. Kenward, M. Aerts, G. Verbeke, A. A. Tsiatis, M. Davidian, and D. Rizopoulos (2012).  On random sample size, ignorability, ancillarity, completeness, separability, and degeneracy: sequential trials, random sample sizes, and missing data.  Statistical Methods in Medical Research 23, 11–41.
  * Neyman and Pearson (1933) Neyman, J. and E. S. Pearson (1933).  Ix. on the problem of the most efficient tests of statistical hypotheses.  Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 231(694-706), 289–337.
  * Nikitin (1995) Nikitin, Y. (1995).  Asymptotic efficiency of nonparametric tests.  Cambridge University Press.
  * O’Brien and Fleming (1979) O’Brien, P. C. and T. R. Fleming (1979).  A multiple testing procedure for clinical trials.  Biometrics, 549–556.
  * Peto et al. (1976) Peto, R., M. Pike, P. Armitage, N. Breslow, D. Cox, S. V. Howard, N. Mantel, K. McPherson, J. Peto, and P. Smith (1976).  Design and analysis of randomized clinical trials requiring prolonged observation of each patient. i. introduction and design.  British journal of cancer 34(6), 585–612.
  * Philippou et al. (1973) Philippou, A. N., G. Roussas, et al. (1973).  Asymptotic distribution of the likelihood function in the independent not identically distributed case.  The Annals of Statistics 1(3), 454–471.
  * Pitman (1948) Pitman, E. (1948).  Lecture Notes on Nonparametric Statistics.  New York: Columbia University.
  * Pocock (1977) Pocock, S. J. (1977).  Group sequential methods in the design and analysis of clinical trials.  Biometrika 64(2), 191–199.
  * Proschan et al. (2006) Proschan, M. A., K. K. G. Lan, and J. T. Wittes (2006).  Statistical Monitoring of Clinical Trials: A Unified Approach.  Springer.
  * Scharfstein et al. (1997) Scharfstein, D. O., A. A. Tsiatis, and J. M. Robins (1997).  Semiparametric efficiency and its implication on the design and analysis of group-sequential studies.  Journal of the American Statistical Association 92(440), 1342–1350.
  * Schou and Marschner (2013) Schou, I. M. and I. C. Marschner (2013).  Meta-analysis of clinical trials with early stopping: an investigation of potential bias.  Statistics in Medicine 32(28), 4859–4874.
  * Simes (1986) Simes, R. J. (1986).  An improved bonferroni procedure for multiple tests of significance.  Biometrika 73(3), 751–754.
  * Tamhane et al. (2020) Tamhane, A. C., J. Gou, and A. Dmitrienko (2020).  Some drawbacks of the simes test in the group sequential setting.  Statistics in Biopharmaceutical Research 12(3), 390–393.
  * Tarima and Flournoy (2019a) Tarima, S. and N. Flournoy (2019a).  Asymptotic properties of maximum likelihood estimators with sample size recalculation.  Statistical Papers 60, 23–44.
  * Tarima and Flournoy (2019b) Tarima, S. and N. Flournoy (2019b).  Distribution theory following blinded and unblinded sample size re-estimation under parametric models.  Communications in Statistics - Simulation and Computation 0(0), 1–12.
  * van der Vaart (1998) van der Vaart, A. (1998).  Asymptotic Statistics.  Cambridge University Press.
  * Wald (1947) Wald, A. (1947).  Sequential Analysis.  New York: Wiley.
  * Wang et al. (2014) Wang, H., N. Flournoy, and E. Kpamegan (2014).  A new bounded log-linear regression model.  Metrika 77(5), 695–720.
  * Wetherill and Glazebrook (1986) Wetherill, G. and K. Glazebrook (1986).  Sequential Methods in Statistics (3 ed.).  New York: Chapman and Hall.