# Risk-based decision making: estimands for sequential prediction under
interventions

Kim Luijken  Department of Epidemiology, Julius Center for Health Sciences and
Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
Paweł Morzywołek  Department of Applied Mathematics, Computer Science and
Statistics, Ghent University, Ghent, Belgium  Department of Statistics,
University of Washington, Seattle, United States  Wouter van Amsterdam
Department of Epidemiology, Julius Center for Health Sciences and Primary
Care, University Medical Center Utrecht, Utrecht, the Netherlands  Giovanni
Cinà  Department of Medical Informatics, Amsterdam University Medical Centers,
Amsterdam, the Netherlands  Institute for Logic, Language and Computation,
University of Amsterdam, Amsterdam, the Netherlands  Pacmed, Amsterdam, the
Netherlands  Jeroen Hoogland  Department of Epidemiology and Data Science,
Amsterdam University Medical Centers, Amsterdam, the Netherlands  Ruth Keogh
Department of Medical Statistics, London School of Hygiene & Tropical
Medicine, Keppel Street, London, United Kingdom  Jesse Krijthe  Pattern
Recognition and Bio-Informatics Group, EEMCS, Delft University of Technology,
Delft, the Netherlands  Sara Magliacane  Amsterdam Machine Learning Lab,
University of Amsterdam, Amsterdam, the Netherlands  Thijs van Ommen
Department of Information and Computing Sciences, Utrecht University, Utrecht,
the Netherlands  Niels Peek  Division of Informatics, Imaging and Data
Science, Faculty of Biology, Medicine and Health, University of Manchester,
Manchester Academic Health Science Centre, Manchester, United Kingdom  Hein
Putter  Department of Biomedical Data Sciences, Leiden University Medical
Center, Leiden, the Netherlands  Maarten van Smeden  Department of
Epidemiology, Julius Center for Health Sciences and Primary Care, University
Medical Center Utrecht, Utrecht, the Netherlands  Matthew Sperrin  Division of
Informatics, Imaging and Data Science, Faculty of Biology, Medicine and
Health, University of Manchester, Manchester Academic Health Science Centre,
Manchester, United Kingdom  Junfeng Wang  Division of Pharmacoepidemiology and
Clinical Pharmacology, Department of Pharmaceutical Sciences, Utrecht
University, Utrecht, the Netherlands  Daniala Weir  Division of
Pharmacoepidemiology and Clinical Pharmacology, Department of Pharmaceutical
Sciences, Utrecht University, Utrecht, the Netherlands  Vanessa Didelez
Department of Biometry and Data Management, Leibniz Institute for Prevention
Research and Epidemiology - BIPS, Bremen, Germany  Faculty of
Mathematics/Computer Science, University of Bremen, Bremen, Germany  Nan van
Geloven

###### Abstract

Prediction models are used amongst others to inform medical decisions on
interventions. Typically, individuals with high risks of adverse outcomes are
advised to undergo an intervention while those at low risk are advised to
refrain from it. Standard prediction models do not always provide risks that
are relevant to inform such decisions: e.g., an individual may be estimated to
be at low risk because similar individuals in the past received an
intervention which lowered their risk. Therefore, prediction models supporting
decisions should target risks belonging to defined intervention strategies.
Previous works on prediction under interventions assumed that the prediction
model was used only at one time point to make an intervention decision. In
clinical practice, intervention decisions are rarely made only once: they
might be repeated, deferred and re-evaluated. This requires estimated risks
under interventions that can be reconsidered at several potential decision
moments. In the current work, we highlight key considerations for formulating
estimands in sequential prediction under interventions that can inform such
intervention decisions. We illustrate these considerations by giving examples
of estimands for a case study about choosing between vaginal delivery and
cesarean section for women giving birth. Our formalization of prediction tasks
in a sequential, causal, and estimand context provides guidance for future
studies to ensure that the right question is answered and appropriate causal
estimation approaches are chosen to develop sequential prediction models that
can inform intervention decisions.

The authors declare no conflict of interest and state that no additional data
was used.

## Risk-based decision making: estimands for sequential prediction under
interventions

Keywords: Counterfactual prediction - Estimand - Prediction model - Prediction
under interventions

##  1 Introduction

To enhance health and health care, there is a need for smart decision support
tools that can improve medical decision making by providing care professionals
and patients with person-specific information. Clinical prediction models
estimate risks of (future) outcomes conditional on patient characteristics and
thus have the potential to provide such information[1, 2, 3, 4]. Yet, standard
prediction models are generally not suitable for supporting intervention
decisions. Prediction models that are intended to inform decisions about
interventions need to answer questions like ‘what is the risk of outcome $Y$
under intervention option $a$ conditional on an individual’s characteristics
$\mathbf{X}$?’. Answering such ‘what if’ questions requires embedding in
causal reasoning[5, 6, 7, 8, 9, 10, 11, 12]. We refer to this task as
‘prediction under (hypothetical) interventions’iiiPrediction under
interventions has also been referred to as ‘counterfactual prediction’[13, 8,
10]. We use the term prediction under interventions instead because at the
moment of making the prediction the outcomes are still in the future and hence
risks need not be counterfactual[14]..

A number of studies have emphasized the need for prediction under
interventions and clarified how to estimate risks under interventions[5, 6, 7,
8, 9, 10, 11]. The focus of these studies was on intervention decisions that
were made at a single time point, but in clinical practice intervention
decisions are rarely made only once and might often be deferred or re-
evaluated. To support such sequential decisions, prediction models are needed
that can estimate risks under interventions at any potential decision moment.
By sequentially providing estimated outcome risks under intervention options
conditional on individual characteristics, the model functions as an assistive
decision support tool[15]. Methods for sequential predictions are well
established in a standard prediction context, but not yet for interventional
predictions[16, 17]. The current work clarifies how to formulate and formalize
sequential predictions under interventions.

The topic of the current work, sequential prediction under interventions, is
different from the related topic of optimal dynamic intervention regimes. A
dynamic intervention regime is a function that takes baseline covariates,
covariate history, and intervention history as inputs and returns an
intervention decision to be made next[18, 19, 20, 21, 14]. Methods exist for
deriving optimal dynamic intervention regimes which are optimal with respect
to a predefined utility function. Optimal dynamic intervention rules provide
advice on what intervention to assign at a moment in time. Methods for
deriving optimal dynamic decision rules assume that an evaluation function
exists to assess the value of candidate decision rules. In practice, the
challenge is often to come up with such a utility function: it will determine
everything. Our work can provide utility functions – in particular, those that
assess the risk of adverse health outcomes under a given candidate decision
rule. Simply providing information on outcome risks under interventions can
sometimes be more useful to end users than a suggested intervention decision.
This is because patients and care professionals can have different utility
functions and their value judgements and view on what is optimal can change
over time. An optimal intervention rule requires assignment of an intervention
as recommended by the decision rule during all stages in order to arrive at
the optimal outcome. Care professionals may be hesitant to consistently follow
an algorithm’s suggestions for multiple intervention decisions ahead. Instead,
having estimated risks of outcomes under intervention options on the table can
facilitate a conversation between care professional and patient so that
informed decisions can be made[22, 7].

The starting point of development of a prediction model under interventions is
formulating the risk questions that reflect the desired predictions, that is,
formulating the estimands. The current work highlights key considerations for
formulating estimands for sequential prediction under interventions. We
illustrate these considerations by giving examples of estimands for single-
stage and sequential prediction under interventions in a case study about
deciding on vaginal delivery or cesarean section for women giving birth. The
contribution of this paper is to define prediction tasks in a sequential,
causal, and estimand context. A clearly defined estimand is needed to ensure
answering the right question and choosing appropriate causal estimation
approaches.

##  2 How to formulate estimands for prediction under interventions

We start with providing general considerations for formulating estimands for
prediction under interventions and then highlight aspects that require
additional attention in the sequential setting.

An estimand is a precise definition of the target quantity of an analysis. In
prediction studies, an estimand captures the conditional outcome risk (or
value) of interest that is intended to be be used as information about a
particular individual. The conditional outcome risk can be interpreted as
individualized rather than individual, because they are formalized at a group
level given a covariate pattern[23, 24]. The term ‘estimand’ is rarely used in
prediction modelling studies, but prediction guidelines implicitly provide
recommendations on aspects that should be defined to specify the target of a
prediction study[25, 1, 26, 27]. By combining these recommendations with
roadmaps available from clinical trial literature[28] and causal inference
literature[29, 30], we arrive at the following elements that are required for
estimands for prediction under interventions:

  * •

Population: A characterisation of the target population to whom the prediction
model is to be applied, including the care setting;

  * •

Moment(s) of intended use: The moment(s) at which the prediction model is to
be used to inform the intervention decision;

  * •

Intervention options: The interventions considered at the moment(s) of
intended use and, if relevant, for how long they are administered;

  * •

Outcome and prediction horizon: The predicted outcome including the time since
the moment of prediction at which we consider it. The specified outcome and
prediction horizon should represent information that is needed in the
discussion among care professionals and patients to make the intervention
decision;

  * •

Predictor(s): The measurement(s) used for prediction that characterize the
individual, i.e., that individualize the predictions. The measurement(s) must
be available at the moment(s) of intended use and measurement procedures must
correspond to those in the setting of intended use.

###  2.1 Considerations for prediction under interventions at a single time
point

As an example of a model that provides predictions under interventions at a
single time point, an estimand could express the risk of a binary outcome
$Y\in\\{0,1\\}$ at a prediction horizon $h$, conditional on predictors
$\mathbf{X}$ measured at moment of intended use under specified intervention
options $A\in\\{0,1\\}$ in a given population. Introducing notation, we denote
regular time points by $k=0,1,2,\ldots,K$. Observed intervention status
$A_{k}$, predictors $\mathbf{X}_{k}$, and outcome status $Y_{k}$ are measured
at each time point. We assume the moment of intended use is time point $k=0$.
We let $\underline{a}_{0}=(a_{0},\dots,a_{K})$ be a specified joint
intervention, setting the intervention to a particular level $a_{k}$ for the
times between $0$ and $K$. Here, $\underline{a}_{0}=\mathbf{0}$ indicates not
initiating the intervention at the moment of prediction and continuing to do
so until time point $K$ and $\underline{a}_{0}=\mathbf{1}$ indicates
initiating the intervention at the moment of prediction and continuing to do
so until time point $K$. An intervention option might also refer to a well-
defined standard of care. We define the potential outcome
$Y_{h}^{\underline{a}_{0}}$ as the outcome at prediction horizon $h$ if an
individual is assigned intervention sequence $\underline{a}_{0}$. By fixing
the intervention to a certain strategy, an estimand for prediction under
intervention at a single time point can be formally defined as:

$\Pr{\left[Y^{\underline{a}_{0}}_{h}=1|\mathbf{X_{0}}\right]},$

e.g., for $\underline{a}_{0}=\mathbf{0}$ this would be the conditional risk of
outcome $Y$ at prediction horizon $h$ for an individual with predictor values
$\mathbf{X_{0}}$ if they would not initiate the intervention up to the
prediction horizon.

To precisely define the intervention options for the estimand, it is helpful
to make a distinction between different types of interventions. Point
interventions are administered once or for a (very) short duration, like
single-dose pharmacological treatments or surgery. Sustained interventions are
administered during longer disease episodes and can take the form of a static
or dynamic regime[31, 18]. A static regime specifies a fixed sequence of the
intervention status for a pre-defined duration. For instance, “an individual
starts the intervention of interest and continues to use it during a year”.
Static regimes can be seen as joint interventions as they do not take
information into account that occurs during the course of time. Dynamic
regimes were mentioned in the introduction and propose an intervention
decision based on the (time-varying) health state of an individual. For
instance, “an individual starts the intervention of interest when marker X
drops below a certain value”. Point interventions, static intervention
regimes, and dynamic intervention regimes can all be used as intervention
options in an estimand for prediction under interventions.

###  2.2 Considerations for sequential prediction under interventions

When an intervention decision can be deferred or re-evaluated, a prediction
model capable of sequentially estimating risks under hypothetical
interventions is needed.

We consider a setting in which predictions of a binary outcome are made at
regular time points $k=0,1,2,\ldots$. The prediction horizon can either stay
fixed for each $k$, i.e., $h_{k}=h$, or can be defined relative to the moment
of prediction, e.g., $h_{k}=k+w$. Let $\underline{a}_{k}=(a_{k},\dots,a_{K})$
be the specified intervention sequence of interest. We let
${\mathbf{\overline{X}}}_{k}=(\mathbf{X}_{0},\mathbf{X}_{1},\mathbf{X}_{2},\ldots,\mathbf{X}_{k})$
denote the observed predictor history up to $k$,
${\overline{A}}_{k}=({A}_{0},{A}_{1},{A}_{2},\ldots,{A}_{k})$ denote the
observed intervention history up to $k$, and $Y_{k}$ denote whether the
outcome has occurred before time $k$. An estimand for sequential prediction
under an intervention option can then be defined as:

$\Pr{\left[Y^{\underline{a}_{k}}_{h_{k}}=1|\overline{\mathbf{{X}}}_{k},Y_{k}=0,{\overline{A}}_{k}\right]},$

e.g., for $\underline{a}_{k}=\mathbf{0}$ and $\overline{A}_{k}=\mathbf{0}$,
this estimand reflects the conditional outcome risk at time $k+w$ for an
individual with a predictor history $\mathbf{\overline{X}}_{k}$ who did not
experience the outcome or initiate the intervention before time $k$, if they
would not initiate the intervention up to the prediction horizon by deciding
on ‘no intervention’ each time the need for intervening is re-evaluated.

An estimand for sequential prediction under interventions incorporates updated
predictor information. The time until the next prediction moment informs how
long the intervention options could be fixed and at what time point the
prediction horizon could be set. For instance, if prediction moments follow
each other at short intervals, it might be more reasonable to define a short-
term prediction horizon and to fix the intervention option to a certain level
until that prediction horizon. This is likely less reasonable if predictions
under interventions are made infrequently. An overview of key considerations
regarding estimands for sequential prediction under interventions is given in
Table 1.

Estimand element |  Questions that help in formulating the estimand element
---|---
Population |  • To which individuals will the prediction model be applied? • In which health care setting will the prediction model be applied?
Moment(s) of intended use |  • At which moment(s) is the prediction model (re)consulted to inform the intervention decision?
Intervention options |  • Which intervention options are relevant at the moment(s) of making the intervention decision? • For how long should the intervention strategy be fixed? • Should the duration to fix the intervention option be aligned with the time till next moment of prediction?
Outcome and prediction horizon |  • Which outcome(s) are most informative for the intervention decision? • What prediction horizon provides important information for the intervention decision: a short-term or long-term horizon? • Should the outcome be defined differently because of the specified intervention option(s)? • Should the prediction horizon be aligned with the time till next moment of prediction?
Predictor(s) |  • Which predictors are predictive of the outcome of interest? • Based on which characteristics should the outcome risks be individualized? • Which measurements are available at the moment(s) of intended use?
Table 1: Considerations to define estimands for sequential prediction under
interventions.

##  3 Case study: mode of delivery in women with high-risk pregnancies

We use a clinical case study to illustrate the required considerations for
formulating estimands for sequential prediction under interventions. We define
several example estimands for a prediction model that informs the intervention
decision for mode of delivery in women giving birth. We start out with a
setting in which the prediction model is only used once to inform the
decision. For this single-stage prediction, we formulate four example
estimands that vary in their intervention options. The example is then
extended to sequential prediction under interventions with three example
estimands that vary in intervention options and prediction horizon.

###  3.1 Clinical context

The clinical case study is inspired by a prediction model developed by Schuit
and colleagues that was proposed to support medical decisions during labor of
high-risk pregnant women[32]. Different from the original model, in the
current study we define the outcome as a composite of adverse events in the
mother and child. To simplify the discussion, we focus on the interventions
vaginal delivery and cesarean section only, and implicitly include other
interventions, like ways to induce natural vaginal delivery or instrumental
vaginal delivery, under vaginal delivery.

As a brief background on the clinical setting based on Schuit and
colleagues[32]; some pregnancies are classified as ‘high-risk’ because they
are complicated by pre-existing maternal disease or complications during
pregnancy. High-risk pregnant women are typically monitored by gynaecologists
in secondary care and are admitted to hospital to give birth. Birth can take
place via different modes of delivery including natural vaginal delivery and
cesarean section. Cesarean section can be a way to prevent adverse neonatal
outcomes due to fetal distress, but it comes with risks for the mother like
increased blood loss, incontinence, and infection[33]. The decision to give
birth via cesarean section can be made before start of labor so that a planned
cesarean section can be performed at an elected time[34]. When high-risk
pregnant women not scheduled for planned cesarean section go into labor, the
decision about mode of delivery is re-evaluated at start of labor and
repeatedly during labor. The decision to perform cesarean section is based on
a weighting of benefits and harms of the procedure and preferences of care
professionals and the woman giving birth[35, 34].

###  3.2 Elements shared across the example estimands

Common elements of the estimands formulated in the case study are:

  * •

Population: Pregnant women at a gestational age of 36 weeks or over with
preexisting maternal disease or complications during pregnancy admitted to
hospital to give birth;

  * •

Moment(s) of intended use: At the start of labor for the single-stage
prediction and hourly from start of labor for the sequential prediction;

  * •

Intervention options: Intervention options considered are vaginal delivery and
cesarean section, where the duration of fixing the intervention varies across
estimands;

  * •

Outcome and prediction horizon: The outcome is defined as a composite of any
adverse neonatal or maternal outcomes. Adverse neonatal outcomes include
stillbirth, early neonatal death, or requiring admission to a neonatal
intensive care unit. Adverse maternal outcomes include maternal death and
postpartum admission to intensive care. The prediction horizon varies across
example estimands;

  * •

Predictors: Fetal heart rate, dilatation, maternal systolic blood pressure,
maternal diastolic blood pressure, maternal age, parity, and history of
preterm birth. For the sequential prediction, we additionally use hourly-
updated information of the time-varying predictors (fetal heart rate,
dilatation, maternal systolic blood pressure, and maternal diastolic blood
pressure).

###  3.3 Single-stage prediction under interventions

In the single-stage prediction setting, we assume the prediction model is used
only at the start of labor to estimate risks of adverse outcomes 72 hours
later under different intervention options. We vary the defined intervention
options across four example estimands (Figure 1). Note that a single-stage
prediction assumes a single time point at which predictions are made to inform
the intervention decision, but these predictions can apply to a sequence of
(hypothetical) future interventions.

Figure 1: Illustrations of estimands for single-stage prediction under
intervention options in the case study. The shaded rectangles indicate the
intervention options under which predictions are made: (1) Light grey for
cesarean section at time point 0, which is irreversible; (2) Black for vaginal
mode of delivery only; (3) Black for setting mode of delivery to vaginal
delivery at time point 0, and diagonal stripes for the option of cesarean
section as in usual care; (4) Dark grey for the dynamic intervention rule
‘performing a cesarean section as soon as fetal heart rate is abnormal’.

#### Notation

We introduce some further notation specific to the case study. Let
$k=0,1,2,\ldots,72$ be time points of one hour apart, where $k=0$ denotes the
start of labor and $h=72$ denotes the prediction horizon. Let
$\underline{a}_{0}=(a_{0},a_{1},a_{2},\ldots,a_{71})$ denote an intervention
option for the 72 time intervals, where $a_{k}\in\\{0,1\\}$, $a_{k}=1$ denotes
cesarean section at time $k$ and $a_{k}=0$ denotes deciding on vaginal mode of
delivery at time $k$. Because cesarean section is irreversible, $a_{k}$ stays
$1$ after it was set to $1$ for the first time by the nature of the
intervention. We use $\underline{A}_{k}=(A_{k},\ldots,A_{72})$ to denote a
vector of observed intervention status between time $k$ and time $72$. Let
$\mathbf{X_{0}}$ denote the set of seven predictors available at the start of
labor: fetal heart rate, dilatation, maternal systolic blood pressure,
maternal diastolic blood pressure, maternal age, parity, and history of
preterm birth. Let $Y_{72}\in\\{0,1\\}$ denote the outcome status at the
prediction horizon, where $Y_{72}=1$ represents any of the defined adverse
neonatal or maternal outcomes having occurred and $Y_{72}=0$ represents
completion of birth in absence of the adverse outcomes. Because all women can
be expected to have given birth after 72 hours, the chosen prediction horizon
implies that the outcome reflects all adverse events.

#### Estimand 1: Immediate cesarean section

To formulate the outcome risk when cesarean section is performed immediately
at the start of labor, we define the intervention option as setting mode of
delivery to cesarean section at $k=0$, i.e., ${a}_{0}=1$. Then,

$\Pr{\left[Y^{{a_{0}}=1}_{72}=1|\mathbf{X_{0}}\right]}$ (1)

expresses the conditional risk of adverse neonatal or maternal outcomes 72
hours after start of labor under immediate cesarean section at start of labor
given the predictors at start of labor. Note that predictors after time point
0 cannot be considered, because this would imply that model users would need
to input information about the future at moment of intended use.

#### Estimand 2: Vaginal mode of delivery only

Another intervention option would be a sustained static intervention regime
from the start of labor until the prediction horizon, e.g.,
$\underline{a}_{0}=\mathbf{0}$. Then,

$\Pr{\left[Y^{\underline{a}_{0}=\mathbf{0}}_{72}=1|\mathbf{X_{0}}\right]}$ (2)

expresses the conditional risk of adverse neonatal or maternal outcomes 72
hours after start of labor under vaginal mode of delivery only, given the
predictors at start of labor.

#### Estimand 3: Deciding on vaginal mode of delivery at moment of prediction
and thereafter intervention policy “as usual”

Estimand (2) informs the intervention decision by predicting outcomes under
perfect adherence to the intervention. The woman in labor and health care
professionals may prefer a vaginal delivery but the intervention strategy
‘vaginal mode of delivery only’ likely does not reflect actual practice. In
practice, women and care professionals re-evaluate the need for intervening
and may decide to change the mode of delivery. Information on the conditional
risk of adverse events under practice as usual can be obtained, for example,
by setting an intervention option only for the moment at which the prediction
is made, without committing to a particular intervention choice or decision
rule at future time points. Predicting under usual practice refers to the
typical practice and decision rules used in the development/training data.

To estimate such risks, we define the intervention option as setting mode of
delivery to vaginal delivery between $k=0$ and $k=1$, i.e., $a_{0}=0$, and
leaving the option of choosing cesarean section later if indicated according
to usual intervention assignment during all subsequent times $k$, i.e.,
$\underline{A}_{1}$[22]${}^{\text{, Ch.~{}5}}$. This implies that $a_{0}$ is
fixed to a value, whereas $\underline{A}_{1}$ is a random variable taking on
the ”natural” values, i.e. what would happen under the policy used in the
development/training data. Then,

$\Pr{\left[Y^{a_{0}=0,{\underline{A}_{1}}}_{72}=1|\mathbf{X_{0}}\right]}$ (3)

expresses the conditional risk of adverse neonatal or maternal outcomes 72
hours after the start of labor under deciding on vaginal mode of delivery for
the first hour after start of labor and the option of cesarean section at
subsequent time points according to usual intervention policy, given the
predictors at start of labor. The way that the intervention option is
specified in Estimand (3) relates closely to the way this is done in the
intervention contrast of an intention to treat effect.

The appeal of Estimand (3) is that the intervention option is similar to the
way a decision might be made in clinical practice: an intervention option is
now chosen, but there is no commitment to a particular intervention option or
decision rule at future points in time. The downside of this is that the
conditional outcome risks are dependent on the intervention assignment that
was observed in the development/training data. This implies that estimated
risks will only generalize to settings with similar intervention assignment
policies[7]. Predictions under a predefined static intervention or (optimal)
dynamic intervention do not rely on this assumption.

#### Estimand 4: Performing a cesarean section as soon as fetal heart rate is
abnormal

At the start of labour, a possible decision is to perform a cesarean section
as soon as but only if there are signs of fetal hear rate abnormality. More
specifically, we define a dynamic intervention rule that cesarean section is
performed the first time fetal heart rate is less than 110 beats per minute
for 3 minutes (persistent fetal bradycardia) or higher than 160 beats per
minute (fetal tachycardia). This is an example of a basic dynamic intervention
rule, in which an intervention is assigned based on the history of a single
covariate up to that stage. More generally, dynamic intervention rules can
also be based on multiple covariates such as taking the mother‘s health status
into account. It could be an optimal dynamic rule as described in the
introduction as well.

Let $X_{k}$ denote fetal heart rate at time point $k$ and $g_{k}$ an indicator
function that flags abnormal fetal heart rate.

$g_{k}({x}_{k})=\begin{cases}0&\text{if $110\leq x_{k}\leq 160$}\\\ 1&\text{if
$x_{k}<110\text{ for 3 minutes in a row or }x_{k}>160$}\end{cases}$

The dynamic intervention sequence $d$ equals 0 until the first time point $k$
where $g_{k}(X_{k})=1$, and 1 from that point on wards. As the $X_{k}$’s are
random, the sequence $d$ is random as well, we only fix the rule on how to
respond to variations in $X_{k}$. The estimand of interest can be specified as

$\Pr{\left[Y^{\underline{a}_{0}=d}_{72}=1|\mathbf{X}_{0}\right]}.$ (4)

This estimand expresses the conditional risk of adverse neonatal or maternal
outcomes under the intervention option to perform cesarean section as soon as
the fetal heart rate is abnormal, given the predictors at start of labor.

It deserves clarification as to why we consider a dynamic intervention
strategy for the single-stage prediction setting, which may be a confusing
classification. The dynamic intervention rule states that based on fetal heart
rate during the course of labor, cesarean section can be performed. This means
that the need for intervening is re-evaluated over time. However, the choice
to commit to this rule is made only once, at the start of labor. In other
words, the action (of starting cesarean section) is re-evaluated, the decision
is not. To support the decision, we only need a single-stage prediction of the
conditional outcome risk under the dynamic intervention rule (Figure 1-4).
Next, we will consider estimands where the prediction under intervention
options is re-evaluated over time.

###  3.4 Sequential prediction under interventions

Predictions under interventions can be considered several times during the
course of labor to inform the decision on mode of delivery. We continue the
example by assuming that the prediction model is used at the start of labor
and thereafter revisited every hour to update the risks. When defining an
estimand that is suitable for sequential prediction under interventions, it is
important to consider the time-varying information up to the moment of making
the prediction as well as the time axes for the intervention options and
outcome after the moment of making the prediction. We discuss these
considerations using three examples of relevant estimands.

#### Further notation

We now assume that predictions to inform the decision about mode of delivery
are made every hour $k=0,1,2,\ldots$. We keep the prediction horizon at 72
hours after start of labor. In other settings, the prediction horizon may be
defined relative to the moment of making the prediction, as explained in
Section 2.2. The predictors $\mathbf{X}_{k}$ are measured at each time
pointiiiiiiWe realize that maternal age, parity, and history of preterm birth
are time-fixed predictors, but do not explicate this in the notation for
simplicity.. Let
${\overline{\mathbf{X}}}_{k}=(\mathbf{X}_{0},\mathbf{X}_{1},\mathbf{X}_{2},\ldots,\mathbf{X}_{k})$
denote the predictor history up to time point $k$. We denote whether the woman
has given birth at time $k$ by introducing an ‘at risk’ indicator
$Z_{k}\in\\{0,1\\}$, where $Z_{k}=1$ denotes that the woman is still in labor
and no adverse outcome has occurred yet and $Z_{k}=0$ denotes that the woman
has given birth or an adverse outcome has occurred.

#### Estimand 5: Vaginal mode of delivery only, using updated predictor
information

We first extend Estimand (2) to a sequential prediction setting using updated
information at the moment of prediction. The starting point is to consider for
which individuals the intervention decision still is to be made, i.e., which
individuals form the ‘risk set’ at time $k$. In this example, the intervention
decision on mode of delivery is to be made in women who are still in labor and
who have not experienced an adverse event. We use $Z_{k}=1$ to select the risk
set of women who are still in labor at time $k$.

Predictor information up to the moment of making the prediction is
$\overline{\mathbf{X}}_{k}$. We define the intervention option as setting mode
of delivery to vaginal delivery from time $k$ onwards for the remaining time
of labor, i.e., $\underline{a}_{k}=\mathbf{0}$. Then,

$\Pr{\left[Y^{\underline{a}_{k}=\mathbf{0}}_{72}=1|\overline{\mathbf{X}}_{k},Z_{k}=1\right]}$
(5)

expresses the conditional risk of adverse neonatal or maternal outcomes 72
hours after start of labor under vaginal mode of delivery only given the
predictor history up to time $k$ and that the woman is still in labor and no
adverse outcome has occurred at time $k$ (Figure 2). We retrieve Estimand 2 as
the special case where the moment of intended use if restricted to $k=0$.

Figure 2: Depiction of sequential prediction under interventions for Estimand
5. The black rectangles indicate the intervention option under which
predictions are made: vaginal delivery, which is sustained throughout labour.
The moment of making the prediction and prediction horizon shift over time to
provide sequential updated predictions.

#### Estimand 6: Vaginal mode of delivery at moment of prediction and
thereafter intervention policy “as usual”

A consideration in sequential prediction is to define the duration of fixing
the intervention strategy under which predictions are made. As explained in
the discussion of Estimand (3), fixing the intervention strategy for the
entire prediction window has benefits in terms of generalizability, but it
might not be an intervention option that corresponds closely to actual
clinical practice. In the current example, it might be preferable to fix the
intervention for one hour because the prediction model is revisited every
hour.

Define the intervention option as setting mode of delivery to vaginal delivery
for one hour at time $k$, i.e., $a_{k}=0$, and follow usual intervention
policy during all subsequent times, i.e., $\underline{A}_{k+1}$. Then,

$\Pr{\left[Y^{a_{k}=0,{\underline{A}_{k+1}}}_{72}=1|\overline{\mathbf{X}}_{k},Z_{k}=1\right]}$
(6)

expresses the conditional risk of adverse neonatal or maternal outcomes 72
hours after start of labor under vaginal delivery for the next hour and having
the option of cesarean section at subsequent time points according to usual
intervention policy. This risk is conditional on the predictor history up to
time $k$ and that the woman is still in labor and no adverse outcome has
occurred at time $k$. The duration over which the intervention option is fixed
is in this case informed by the time till next decision moment. We retrieve
Estimand 3 as the special case where the moment of intended use if restricted
to $k=0$.

#### Estimand 7: Vaginal mode of delivery at moment of prediction and a short
prediction horizon

Finally, estimands for sequential prediction require that the time axis of the
outcome, i.e., the prediction horizon, is defined. This element of the
estimand determines what information the prediction model provides to inform
the intervention decision. In the example, the prediction model is used to
estimate risks of relatively acute outcomes (within 72 hours) for the mother
and child. In other cases a long prediction horizon may be apt, for instance
when predicting (side) effects that might develop months or years later. When
the predictions are revisited regularly, it might be preferable to estimate a
conditional outcome risk until the next moment the prediction model is
consulted. We could for example define the prediction horizon at $k+1$. Then,

$\Pr{\left[Y^{a_{k}=0}_{k+1}=1|\overline{\mathbf{X}}_{k},Z_{k}=1\right]}$ (7)

expresses the conditional risk of adverse neonatal or maternal outcomes within
one hour after consulting the prediction model under vaginal delivery for the
next hour given the predictor history up to time $k$ and that the woman is
still in labor and no adverse outcome has occurred at time $k$. The duration
of fixing the intervention and the prediction horizon are in this case
informed by the time till next decision moment. Because of the short
prediction horizon, it might also be informative to define an intermediate
outcome instead, such as fetal hypoxia.

##  4 Remarks on identification and estimation

After estimands for prediction under interventions are defined in a study, the
subsequent steps are assessing identifiability given the observed data and
choosing an appropriate estimator. Identifiability and estimation are beyond
the scope of the current work, but we provide a few pointers here.

Assessing whether the observed information is sufficient to identify the
desired estimand for prediction under interventions involves evaluating
whether causal identification assumptions are met. The identifiability
assumptions typically include consistency, (conditional and sequential)
exchangeability, and (sequential) positivity[36, 37, 38]. We refer the reader
to other material for discussions on how to assess identifiability, such as
[36, 37, 38], but make two remarks relevant to prediction under interventions
in particular. The predictor(s) that are defined as part of an estimand are
typically chosen based on (bedside) availability and prognostic value. They
are not selected with causal assumptions in mind. Evaluation of (sequential)
exchangeability might reveal that additional (time-varying) covariates need to
be taken into account during estimation for confounding adjustment. Regarding
(sequential) positivity, it needs to be possible to observe individuals under
all formulated intervention options for all covariate patterns given the
predictors in the model. This is more stringent compared to (sequential)
positivity for average treatment effects and often implies that more data is
needed or parametric assumptions need to be made. Alternatively, one could
take a pragmatic approach and focus on intervention options that are
identifiable given the available data[38].

If an estimand is identifiable given observed data, we can then select an
estimator from many available existing methods to estimate the formulated
estimands, mainly stemming from causal inference literature on
dynamic/conditional treatment effect estimation[18, 39, 40]. Such methods can
be used to develop a prediction model under the hypothetical intervention
scenario(s) as if all individuals had followed the defined intervention
option[7, 5, 11, 9]. In the sequential prediction context, additional care
should be given to the way the history of the predictors up to the moment of
prediction is summarized[16, 17, 41, 42]. Further methodological work is
needed to develop dedicated methods for estimating and validation of
prediction under interventions[38], for example combining the selection of
predictors from a set of candidate predictors while remaining sufficient
adjustment for confounding.

##  5 Discussion

The current work provides recommendations about how to formulate estimands for
sequential prediction under interventions. We combined guidance for
formulating estimands in prediction, clinical trial and causal inference
literature and discussed key considerations (Table 1). The considerations were
illustrated by formalizing seven estimands in a clinical case study about the
decision on mode of delivery in women giving birth. These insights on
estimands extend previous work on predictions under interventions for single-
stage decisions and link principles of dynamic prediction modelling to
prediction under intervention options.

In empirical studies focusing on sequential prediction under interventions, it
is likely that multiple estimands are relevant for the prediction problem at
hand. How to align the moments of intended use, intervention duration, and
prediction horizon should depend on context-specific knowledge and preferences
of different model users. Ultimately, the selected (set of) estimand(s) should
capture the identifiable risks under intervention options that provide
relevant information for decision making. The choice of moments of intended
use of a prediction model under interventions can also be determined by
estimating the optimal timing of risk assessments[43].

We discussed estimands for an example of an assistive prediction model that
can provide information on risks for mother and child under the defined
intervention options. These risks are one piece of information that could
inform the decision on mode of delivery and leaves weighting of other sources
of information to arrive at a decision to the model users. As such, assistive
prediction models fit the process of shared decision making more naturally
compared to optimal dynamic intervention rules (which are sometimes referred
to as directive decision support tools[15]).

The transition from outcome risk estimation to a particular intervention or
triage decision is typically described to take place only after model
development and validation, in a so-called impact analysis study[15, 44]. In
an impact analysis, the impact of the use of a prediction model on patient
outcomes or (cost) efficiency of care is evaluated in a setting in which some
healthcare professionals are assigned to use the prediction model when making
decisions about an intervention – preferably by randomization. The current
work links predictions to decisions already in the model development phase,
which ensures that predictions are aligned to their intended clinical impact.
We recommend that development of such prediction models always starts by
defining relevant estimands. The need for other good practices in prediction
modelling, like external validation and impact studies, remains[45, 38, 37].

Causal research closely related to prediction under interventions focuses on
estimation of individualized intervention effects conditional on covariates
with the goal to investigate heterogeneity in intervention effects in
randomized controlled trials[23, 46, 47, 48, 49]. Still, most work in this
area seems to assume that an intervention decision is made at a single point
in time, at the moment of randomization, which often does not align with
decision making in clinical practice. Furthermore, there is debate whether a
conditional difference in means should be used to assign interventions based
on who benefits most[50, 51]. Our work focuses on individualized outcome risks
under certain intervention options rather than estimating intervention
contrasts so that the prediction model can be used as an assistive tool in
making an intervention decision.

Our clinical case study did not address all intricacies of decisions to be
made when defining estimands for sequential prediction under interventions.
For example, others have proposed to set the intervention for the first time
point and afterwards assume an optimal intervention regime[52]. The use case
contained some specific features that do not occur in general such as
irreversibility of the intervention ‘cesarean section’. We specified a
composite outcome, but knowing the risk of each outcome separately might be
more helpful in practice to weigh the risks for the mother and neonate. The
prediction horizon was set at a time point of 72 hours, which was relevant in
the clinical context, but may complicate the estimation when a single model is
used for prediction windows of different duration. Because our focus was on
defining estimands, we kept the discussion of estimation strategies to a
minimum. Yet, estimating conditional outcome risks under interventions might
be particularly challenging in a sequential setting.

Defining a suitable estimand for sequential prediction under interventions is
far from trivial, but is a pivotal starting point for development of any
prediction model intended to inform intervention decisions at multiple time
points. Different estimands can produce risk estimates that are relatively
similar, but in some situations estimates can differ substantially[7, 53].
Formulating estimands prevents intervention decisions from being misguided by
information from prediction models. The current work illustrates how estimands
for sequential prediction under interventions can be formulated.

## References

  * [1] Karel GM Moons, Douglas G Altman, Johannes B Reitsma, John PA Ioannidis, Petra Macaskill, Ewout W Steyerberg, Andrew J Vickers, David F Ransohoff, and Gary S Collins.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration.  Annals of internal medicine, 162(1):W1–W73, 2015.
  * [2] Karel GM Moons, Patrick Royston, Yvonne Vergouwe, Diederick E Grobbee, and Douglas G Altman.  Prognosis and prognostic research: What, why, and how?  the BMJ, 338, 2009.
  * [3] Steyerberg EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating.  New York: Springer, 2009.
  * [4] Richard D Riley, Danielle van der Windt, Peter Croft, and Karel GM Moons.  Prognosis Research in Healthcare: Concepts, Methods, and Impact.  Oxford University Press, 2019.
  * [5] Matthew Sperrin, Glen P Martin, Alexander Pate, Tjeerd Van Staa, Niels Peek, and Iain Buchan.  Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models.  Statistics in Medicine, 37(28):4142–4154, 2018.
  * [6] Matthew Sperrin, David Jenkins, Glen P Martin, and Niels Peek.  Explicit causal reasoning is needed to prevent prognostic models being victims of their own success.  Journal of the American Medical Informatics Association, 26(12):1675–1676, 2019.
  * [7] Nan van Geloven, Sonja A Swanson, Chava L Ramspek, Kim Luijken, Merel van Diepen, Tim P Morris, Rolf HH Groenwold, Hans C van Houwelingen, Hein Putter, and Saskia le Cessie.  Prediction meets causal inference: The role of treatment in clinical prediction models.  European Journal of Epidemiology, 35:619–630, 2020.
  * [8] Barbra A Dickerman and Miguel A Hernán.  Counterfactual prediction is not only for causal inference.  European Journal of Epidemiology, 35:615–617, 2020.
  * [9] Barbra A Dickerman, Issa J Dahabreh, Krystal V Cantos, Roger W Logan, Sara Lodi, Christopher T Rentsch, Amy C Justice, and Miguel A Hernán.  Predicting counterfactual risks under hypothetical treatment strategies: An application to hiv.  European Journal of Epidemiology, 37(4):367–376, 2022.
  * [10] Amanda Coston, Alan Mishler, Edward H Kennedy, and Alexandra Chouldechova.  Counterfactual risk assessments, evaluation, and fairness.  In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 582–593, 2020.
  * [11] Lijing Lin, Matthew Sperrin, David A Jenkins, Glen P Martin, and Niels Peek.  A scoping review of causal methods enabling predictions under hypothetical interventions.  Diagnostic and Prognostic Research, 5:1–16, 2021.
  * [12] Nan van Geloven, Ruth Keogh, Wouter van Amsterdam, Giovanni Cina, Jesse Krijthe, Niels Peek, Kim Luijken, Sara Magliacane, Pawel Morzywolek, Thijs van Ommen, Hein Putter, Matthew Sperrin, Junfeng Wang, Daniala Weir, and Vanessa Didelez.  Causal blind spots when using prediction models for medical decision making [forthcoming].
  * [13] Miguel A Hernán, John Hsu, and Brian Healy.  A second chance to get causal inference right: A classification of data science tasks.  Chance, 32(1):42–49, 2019.
  * [14] A Philip Dawid and Vanessa Didelez.  Identifying the consequences of dynamic treatment strategies: A decision theoretic overview.  Statistics Surveys, 4:184–231, 2010.
  * [15] Brendan M Reilly and Arthur T Evans.  Translating clinical research into clinical practice: Impact of using prediction rules to make decisions.  Annals of Internal Medicine, 144(3):201–209, 2006.
  * [16] Dimitris Rizopoulos.  Joint Models for Longitudinal and Time-to-event Data: With Applications in R.  CRC press, 2012.
  * [17] Hans van Houwelingen and Hein Putter.  Dynamic Prediction in Clinical Survival Analysis.  CRC Press, 2011.
  * [18] Bibhas Chakraborty and Erica EM Moodie.  Statistical Methods for Dynamic Treatment Regimes: Reinforcement Learning, Causal Inference, and Personalized Medicine, volume 76.  Springer Science & Business Media, 2013.
  * [19] Erica EM Moodie, Thomas S Richardson, and David A Stephens.  Demystifying optimal dynamic treatment regimes.  Biometrics, 63(2):447–455, 2007.
  * [20] Susan A Murphy.  Optimal dynamic treatment regimes.  Journal of the Royal Statistical Society Series B: Statistical Methodology, 65(2):331–355, 2003.
  * [21] James M Robins.  Optimal structural nested models for optimal sequential decisions.  In Proceedings of the Second Seattle Symposium in Biostatistics: Analysis of correlated data, pages 189–326. Springer, 2004.
  * [22] Morzywolek, Pawel.  Causal inference methods to optimize clinical decision-making in treatment initiation based on routinely collected data.  PhD thesis, Ghent University, 2023.
  * [23] Jeroen Hoogland, Joanna IntHout, Michail Belias, Maroeska M Rovers, Richard D Riley, Frank E. Harrell Jr, Karel GM Moons, Thomas PA Debray, and Johannes B Reitsma.  A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint.  Statistics in Medicine, 40(26):5961–5981, 2021.
  * [24] Michael C Knaus, Michael Lechner, and Anthony Strittmatter.  Machine learning estimation of heterogeneous causal effects: Empirical monte carlo evidence.  The Econometrics Journal, 24(1):134–161, 2021.
  * [25] Karel GM Moons, Joris AH de Groot, Walter Bouwmeester, Yvonne Vergouwe, Susan Mallett, Douglas G Altman, Johannes B Reitsma, and Gary S Collins.  Critical appraisal and data extraction for systematic reviews of prediction modelling studies: The charms checklist.  PLoS medicine, 11(10):e1001744, 2014.
  * [26] Rebecca Whittle, Kara-Louise Royle, Kelvin P Jordan, Richard D Riley, Christian D Mallen, and George Peat.  Prognosis research ideally should measure time-varying predictors at their intended moment of use.  Diagnostic and Prognostic Research, 1:1–9, 2017.
  * [27] Kim Luijken, Rolf HH Groenwold, Ben Van Calster, Ewout W Steyerberg, and Maarten van Smeden.  Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective.  Statistics in Medicine, 38(18):3444–3459, 2019.
  * [28] ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials.  https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e9-r1-addendum-estimands-sensitivity-analysis-clinical-trials-guideline-statistical-principles_en.pdf.  Accessed: 2023-09-12.
  * [29] Maya L Petersen and Mark J van der Laan.  Causal models and learning from data: Integrating causal modeling and statistical estimation.  Epidemiology (Cambridge, Mass.), 25(3):418, 2014.
  * [30] Els Goetghebeur, Saskia le Cessie, Bianca De Stavola, Erica EM Moodie, Ingeborg Waernbaum, and ” the topic group Causal Inference (TG7) of the STRATOS initiative.  Formulating causal questions and principled statistical answers.  Statistics in Medicine, 39(30):4922–4948, 2020.
  * [31] Jonathan AC Sterne, Miguel A Hernán, Barnaby C Reeves, Jelena Savović, Nancy D Berkman, Meera Viswanathan, David Henry, Douglas G Altman, Mohammed T Ansari, Isabelle Boutron, et al.  Robins-i: A tool for assessing risk of bias in non-randomised studies of interventions.  the BMJ, 355, 2016.
  * [32] Ewoud Schuit, Anneke Kwee, Michelle EMH Westerhuis, Hendrikus JHM Van Dessel, Giuseppe CM Graziosi, Jan MM Van Lith, Jan G Nijhuis, S Guid Oei, Herman P Oosterbaan, Nico WE Schuitemaker, et al.  A clinical prediction model to assess the risk of operative delivery.  BJOG: An International Journal of Obstetrics & Gynaecology, 119(8):915–923, 2012.
  * [33] National Institute for Health and Care Excellence.  Caesarean birth, 2021.
https://www.nice.org.uk/guidance/ng192.

  * [34] National Institute for Health and Care Excellence.  Intrapartum care for women with existing medical conditions or obstetric complications and their babies, 2019.
https://www.nice.org.uk/guidance/ng121.

  * [35] National Institute for Health and Care Excellence.  Intrapartum care, 2023.
https://www.nice.org.uk/guidance/ng235.

  * [36] Miguel A Hernán and James M Robins.  Causal Inference: What If.  Chapman & Hall/CRC, Boca Raton, 2020.
  * [37] Christopher B Boyer, Issa J Dahabreh, and Jon A Steingrimsson.  Assessing model performance for counterfactual predictions.  arXiv preprint arXiv:2308.13026, 2023.
  * [38] Ruth H Keogh and Nan van Geloven.  Prediction under hypothetical interventions: Evaluation of performance using longitudinal observational data.  arXiv preprint arXiv:2304.10005, 2023.
  * [39] Pawel Morzywolek, Johan Decruyenaere, and Stijn Vansteelandt.  On a general class of orthogonal learners for the estimation of heterogeneous treatment effects.  arXiv preprint arXiv:2303.12687, 2023.
  * [40] Jon Michael Gran, Kjetil Røysland, Marcel Wolbers, Vanessa Didelez, Jonathan AC Sterne, Bruno Ledergerber, Hansjakob Furrer, Viktor Von Wyl, and Odd O Aalen.  A sequential cox approach for estimating the causal effect of treatment in the presence of time-dependent confounding applied to data from the swiss hiv cohort study.  Statistics in Medicine, 29(26):2757–2768, 2010.
  * [41] Hein Putter and Hans C van Houwelingen.  Landmarking 2.0: Bridging the gap between joint models and landmarking.  Statistics in Medicine, 41(11):1901–1917, 2022.
  * [42] Dimitris Rizopoulos, Jeremy MG Taylor, Grigorios Papageorgiou, and Todd M Morgan.  Using joint models for longitudinal and time-to-event data to investigate the causal effect of salvage therapy after prostatectomy.  arXiv preprint arXiv:2309.02115, 2023.
  * [43] Francesca Gasperoni, Christopher H Jackson, Angela M Wood, Michael J Sweeting, Paul J Newcombe, David Stevens, and Jessica K Barrett.  Optimal risk-assessment scheduling for primary prevention of cardiovascular disease.  arXiv preprint arXiv:2302.04992, 2023.
  * [44] Karel GM Moons, Douglas G Altman, Yvonne Vergouwe, and Patrick Royston.  Prognosis and prognostic research: Application and impact of prognostic models in clinical practice.  the BMJ, 338, 2009.
  * [45] Anne AH de Hond, Artuur M Leeuwenberg, Lotty Hooft, Ilse MJ Kant, Steven WJ Nijman, Hendrikus JA van Os, Jiska J Aardoom, Thomas PA Debray, Ewoud Schuit, Maarten van Smeden, et al.  Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: A scoping review.  NPJ Digital Medicine, 5(1):2, 2022.
  * [46] David M Kent, Jessica K Paulus, David Van Klaveren, Ralph D’Agostino, Steve Goodman, Rodney Hayward, John PA Ioannidis, Bray Patrick-Lake, Sally Morton, Michael Pencina, et al.  The predictive approaches to treatment effect heterogeneity (PATH) statement.  Annals of Internal Medicine, 172(1):35–45, 2020.
  * [47] David M Kent, David Van Klaveren, Jessica K Paulus, Ralph D’Agostino, Steve Goodman, Rodney Hayward, John PA Ioannidis, Bray Patrick-Lake, Sally Morton, Michael Pencina, et al.  The predictive approaches to treatment effect heterogeneity (PATH) statement: Explanation and elaboration.  Annals of Internal Medicine, 172(1):W1–W25, 2020.
  * [48] David M Kent, Ewout Steyerberg, and David van Klaveren.  Personalized evidence based medicine: Predictive approaches to heterogeneous treatment effects.  the BMJ, 363, 2018.
  * [49] Mingyang Cai, Stef van Buuren, and Gerko Vink.  How to relate potential outcomes: Estimating individual treatment effects under a given specified partial correlation.  arXiv preprint arXiv:2208.12931, 2022.
  * [50] Tyler J VanderWeele, Alex R Luedtke, Mark J van der Laan, and Ronald C Kessler.  Selecting optimal subgroups for treatment using many covariates.  Epidemiology (Cambridge, Mass.), 30(3):334, 2019.
  * [51] Bas BL Penning de Vries, Rolf HH Groenwold, and Alex Luedtke.  Re. selecting optimal subgroups for treatment using many covariates.  Epidemiology (Cambridge, Mass.), 31(4):e33, 2020.
  * [52] Stijn Vansteelandt and Marshall Joffe.  Structural nested models and g-estimation: The partially realized promise.  Statistical Science, pages 707–731, 2014.
  * [53] Ilaria Prosepe, Rolf HH Groenwold, Rachel Knevel, Romin Pajouheshnia, and Nan van Geloven.  The disconnect between development and intended use of clinical prediction models for covid-19: A systematic review and real-world data illustration.  Frontiers in Epidemiology, 2:899589, 2022.

### Conflict of interest

The authors declare no conflict of interest.

### Data availability statement

The authors state that no additional data was used for this study.

### Funding

No funding was received for this study.