The finite sample performance of instrumental variable-based estimators of the
Local Average Treatment Effect when controlling for covariates
Hugo Bodory+, Martin Huber++, Michael Lechner*,**

\+ University of St.Gallen, Vice-President’s Board (Research & Faculty)
++ University of Fribourg, Dept. of Economics
* University of St.Gallen, Dept. of Economics

###### Abstract

This paper investigates the finite sample performance of a range of
parametric, semi-parametric, and non-parametric instrumental variable
estimators when controlling for a fixed set of covariates to evaluate the
local average treatment effect. Our simulation designs are based on empirical
labor market data from the US and vary in several dimensions, including effect
heterogeneity, instrument selectivity, instrument strength, outcome
distribution, and sample size. Among the estimators and simulations
considered, non-parametric estimation based on the random forest (a machine
learner controlling for covariates in a data-driven way) performs competitive
in terms of the average coverage rates of the (bootstrap-based) 95% confidence
intervals, while also being relatively precise. Non-parametric kernel
regression as well as certain versions of semi-parametric radius matching on
the propensity score, pair matching on the covariates, and inverse probability
weighting also have a decent coverage, but are less precise than the random
forest-based method. In terms of the average root mean squared error of LATE
estimation, kernel regression performs best, closely followed by the random
forest method, which has the lowest average absolute bias.

Keywords: Instrumental variables, local average treatment effects, Empirical
Monte Carlo Study.

JEL Classification: C21, C26.

Correspondence to: Hugo Bodory, University of St. Gallen, Varnbüelstrasse 14,
CH-9000 St. Gallen<EMAIL_ADDRESS>Martin Huber, University of
Fribourg, Bd. de Pérolles 90, CH-1700 Fribourg<EMAIL_ADDRESS>Michael
Lechner, University of St. Gallen, Varnbüelstrasse 14, CH-9000 St. Gallen,
<EMAIL_ADDRESS>**: Michael Lechner is also affiliated with CEPR and
PSI, London, CESIfo, Munich, IAB, Nuremberg, and IZA, Bonn.

##  1 Introduction

The evaluation of the causal effect of a treatment (e.g., fertility) on an
outcome (e.g., labor supply) is frequently complicated by endogeneity,
implying that the treatment is associated with unobserved characteristics
affecting the outcome (e.g. personality traits, preferences, and values
concerning family and working life). One may nevertheless assess treatment
effects in the presence of an instrumental variable (IV) which affects the
treatment of (at least) some subjects in a monotonic way, does not directly
affect the outcome (other than through treatment) and is as good as randomly
assigned. Under these conditions, the local average treatment effect (LATE) on
the compliers, the subpopulation whose treatment state reacts positively to
the instrument, is identified, as discussed in Imbens+94 and Angrist+96. In
many empirical contexts, it may seem unlikely that the IV assumptions hold
unconditionally, in particular when the treatment evaluation relies on
observational data in which the instrument is not explicitly randomized like
in an experiment. Depending on the application, it might, however, appear
plausible that the IV assumptions hold conditional on covariates observed in
the data. In this case, the LATE is identified and can be consistently
estimated under certain conditions, see the discussions in Abadie00, Tan2006,
and Froel2007.

This paper assesses the finite sample performance of various parametric, semi-
parametric, and non-parametric IV estimators when controlling for a fixed
(i.e., pre-defined and low-dimensional) set of covariates by Monte Carlo
simulations that are based on empirical labor market data from Angrist+98. The
latter study assesses the effect of fertility, defined as having at least
three vs. two children, on mother’s labor supply (for instance, a binary
employment status or weeks employed per year), using twins at the second birth
as instrument. The intuition for this IV strategy is that if a mother with one
child get twins at the second birth, then fertility immediately increases to
three rather than two children, implying a first stage effect of the twins
instrument on the treatment. In the spirit of HuLeWu13, our empirical Monte
Carlo simulation makes to a certain extent use of the empirical associations
in the labour market data when assessing the various IV estimators, with the
aim that our analysis is more closely linked to real world applications.

We vary the simulation designs in several dimensions, including treatment
effect heterogeneity, instrument selectivity across observed covariates
(namely age, race, and quarter of birth), instrument strength, the outcome
distribution, and sample size. We analyse the performance of a range of
estimators commonly considered in treatment and policy evaluation, including
two stage least squares, inverse probability weighting, matching, doubly
robust estimation, and nonparametric regression. We find that overall, non-
parametric estimation based on the random forest, a machine learning algorithm
controlling for covariates in a data-driven way, performs best in terms of
coverage rates. The latter are defined as the share of simulations in which
the true LATE is included in the 95% confidence interval of a LATE estimator,
where the standard error required for the construction of the confidence
interval is obtained by the non-parametric bootstrap. Furthermore, the random
forest-based estimator is relatively precise, implying that the confidence
interval is comparably short, which (conditional on having a decent coverage)
appears desirable from the perspective of statistical power. Non-parametric
kernel regression as well as certain versions of semi-parametric radius
matching on the propensity score, pair matching on the covariates, and inverse
probability weighting also have a decent coverage, but are less precise than
the random forest-based method. Concerning the average root mean squared error
of LATE estimation, kernel regression performs best (and also has the smallest
average standard deviations), closely followed by the random forest method,
which has the lowest average absolute bias.

The remainder of this paper is organized as follows. Section 2 discusses the
identifying assumptions for IV-based LATE evaluation in the presence of
covariates. Section 3 introduces various parametric, semi-parametric, and non-
parametric LATE estimators, as well as a bootstrap procedure for computing
standard errors. Section 4 presents our empirical Monte Carlo simulation
approach, namely the empirical data and the simulation designs. Section 5
presents the results on the finite sample performance of the LATE estimators.
Section 6 concludes.

##  2 Identification of the LATE

In this section, we present the assumptions underlying the identification of
the Local Average Treatment Effect (LATE) when controlling for covariates. To
formalize the discussion, let us denote by $D_{i}$ a possibly endogenous
treatment received by unit $i$, and by $Y_{i}$ the outcome variable based on
which the treatment effect is to be evaluated. In their seminal paper,
Imbens+94 define the LATE as the mean effect of $Y_{i}$ in response to a
change in $D_{i}$ among the compliers, a subgroup whose $D_{i}$ reacts to an
exogenous shift in the instrumental variable, which is denoted by $Z_{i}$. To
discuss the identification of the LATE, we make use of the potential outcomes
framework introduced by Rubin74, which expresses causal effects as differences
between potential outcomes under treatment and non-treatment. We adapt this
concept to our instrumental variable setting with binary indicators $D_{i}$
and $Z_{i}$, and define potential outcome and treatment variables for unit $i$
in the following way:

$\displaystyle Y_{i,z}^{d}$ $\displaystyle=$ $\displaystyle
Y_{i}(D_{i}=d_{i},Z_{i}=z_{i})$ (1) $\displaystyle D_{i,z}$ $\displaystyle=$
$\displaystyle D_{i}(Z_{i}=z_{i}),$ (2)

with $d_{i},z_{i}\in\\{0,1\\}$. Using this framework, Angrist+96 show that
units can be divided into two subgroups, compliers and noncompliers. Compliers
are those induced to take the treatment when being assigned to it. Formally,
this type of units is characterized by $D_{i,1}-D_{i,0}=1$. The subgroup of
noncompliers may consist of three further types, namely always-takers with
$D_{i,1}=D_{i,0}=1$, never-takers with $D_{i,1}=D_{i,0}=0$, and defiers with
$D_{i,1}-D_{i,0}=-1$. Note that the type of a single unit cannot be identified
because the counterfactual potential treatment (that would have occurred under
the alternative, rather than the factual instrument assignment) is not
observed.

Abadie00, Tan2006, and Froel2007 consider non-parametric LATE identification
and estimation when controlling for observed covariates, denoted by $X_{i}$.
We subsequently present the identifying assumptions in this context, which
consist of (i) a monotonicity restriction on the treatment, (ii) the existence
of compliers, (iii) conditional independence of the instrument and the share
of compliance types, (iv) conditional mean independence of the outcome and the
instrument, and (v) common support.

######  Assumption 1 (Monotonicity).

$P(D_{i,0}>D_{i,1})=0$.

######  Assumption 2 (Existence of compliers).

$P(D_{i,0}<D_{i,1})>0$.

######  Assumption 3 (Unconfounded type).

$P(\tau_{i}=t|X_{i}=x_{i},Z_{i}=0)=P(\tau_{i}=t|X_{i}=x_{i},Z_{i}=1)$ for
$t\in\\{a,n,c\\}$.
The types $\tau$ include always-takers $a$, never-takers $n$, and compliers
$c$.

######  Assumption 4 (Conditional mean independence of the outcome).

$E[Y_{i,Z_{i}}^{0}|X_{i}=x_{i},Z_{i}=0,\tau_{i}=t]=E[Y_{i,Z_{i}}^{0}|X_{i}=x_{i},Z_{i}=1,\tau_{i}=t]$
for $t\in\\{n,c\\}$,
$E[Y_{i,Z_{i}}^{1}|X_{i}=x_{i},Z_{i}=0,\tau_{i}=t]=E[Y_{i,Z_{i}}^{1}|X_{i}=x_{i},Z_{i}=1,\tau_{i}=t]$
for $t\in\\{a,c\\}$.

######  Assumption 5 (Common support).

$Supp(X_{i}|Z_{i}=1)=Supp(X_{i}|Z_{i}=0)$.

Assumption 1 rules out the presence of defiers, a type whose treatment never
complies with the instrument. Assumption 2 implies that the subgroup of
compliers exists. Due to the conditional independence of the instrument and
the shares of compliers, always-takers, and never-takers stated in Assumption
3, the first stage effect of the instrument on the treatment is identified
conditional on covariates, such that any variables affecting both the
instrument and the treatment are controlled for. The conditional mean
independence in Assumption 4 rules out a direct average effect of the
instrument on the outcome (exclusion restriction) and unobservables that
jointly affect the instrument and the outcome when controlling for covariates.
Finally, Assumption 5 ensures that for all covariate values occurring in the
population, either instrument value $Z_{i}\in\\{0,1\\}$ exists such that the
instrument is not deterministic in the covariates.

Under Assumptions 1 to 5, the LATE, denoted as
$\theta=E[Y_{i,Z_{i}}^{1}-Y_{i,Z_{i}}^{0}|D_{i,1}-D_{i,0}=1]$, is identified
by

$\theta=\frac{E_{X}[E[Y_{i}|Z_{i}=1,X_{i}]-E[Y_{i}|Z_{i}=0,X_{i}]]}{E_{X}[E[D_{i}|Z_{i}=1,X_{i}]-E[D_{i}|Z_{i}=0,X_{i}]]}.$
(3)

Based on the insights of rosenbaum1983, Froel2007 shows that identification is
also obtained by conditioning on the instrument propensity score
$p(x):=P(Z_{i}=1|X_{i}=x)$ rather than the covariates, because it possesses
the so-called ‘balancing property’. That is, conditioning on the one-
dimensional propensity score balances the distribution of the covariates
across the states of the instrument. For this reason, the LATE is
alternatively identified by

$\theta=\frac{E_{X}[E[Y_{i}|Z_{i}=1,p(X_{i})]-E[Y_{i}|Z_{i}=0,p(X_{i})]]}{E_{X}[E[D_{i}|Z_{i}=1,p(X_{i})]-E[D_{i}|Z_{i}=0,p(X_{i})]]}.$
(4)

##  3 Estimation and inference

In this section, we present parametric, semi-parametric, and non-parametric
methods for estimating the LATE parameter $\theta$ introduced in Section 2. We
also discuss a trimming rule that tackles limited common support in covariate
values across instrument states, based on dropping observations which would
obtain large weights in the estimator because their covariate values occur
(almost) exclusively in only one of the instrument states. Finally, we provide
an bootstrap procedure for estimating the standard errors of the LATE
estimators.

###  3.1 Estimation

One method for the estimation of $\theta$ frequently applied in empirical work
is two-stage least-squares (2SLS), which is easy to implement and
computationally fast. However, the linearity assumption of the 2SLS estimator
implies effect homogeneity, a restriction that may not hold in empirical
studies. We consider 2SLS as a benchmark method, but also include more general
LATE estimators that allow for effect heterogeneity of the LATE across values
of the covariates.

Equations 3 and 4 imply that $\theta$ can be expressed as the ratio of two
treatment effect estimators that account for covariate differences in the
presence and absence of the instrument. The numerator gives the reduced form
effect of $Z_{i}$ on $Y_{i}$ and the denominator the first stage effect of
$Z_{i}$ on $D_{i}$. Thus, a natural choice for the construction of estimators
for $\theta$ is to substitute the expressions in the numerators and
denominators of Equations 3 and 4 by estimators standardly applied in
treatment or policy evaluation, see for instance the surveys by Imbens03 and
ImWo08.

Many treatment effect estimators are semi-parametric in the sense that
(parametric) propensity score estimation is combined with non-parametric
treatment effect estimation, using weighting, matching, or doubly robust
methods. A growing number of simulation studies has investigated the finite
sample behavior of such treatment effect estimators when the treatment is
exogenous conditional on covariates, see for instance Froe00a, Zh04,
LuncefordDavidian2004, BuDNMC09, HuLeWu13, and FrHuWi14. We consider such
methods to estimate the LATE based on estimates of the instrument propensity
score. We also vary the degree of flexibility of the estimators and implement
parametric, semi-parametric, and non-parametric approaches to compute the
reduced form and first stage effects in the numerators and denominators of
Equations 3 and 4.

SmithTodd00, among others, regard treatment effect estimators as weighted
differences in outcomes. We apply this definition to the Wald formula and
express the LATE as:

$\hat{\theta}=\frac{\frac{1}{n_{1}}\sum_{i=1}^{n}z_{i}\hat{w}_{i}y_{i}-\frac{1}{n_{0}}\sum_{j=1}^{n}(1-z_{j})\hat{w}_{j}y_{j}}{\frac{1}{n_{1}}\sum_{i=1}^{n}z_{i}\hat{w}_{i}d_{i}-\frac{1}{n_{0}}\sum_{j=1}^{n}(1-z_{j})\hat{w}_{j}d_{j}}.$
(5)

$n$ denotes the size of an i.i.d. sample of realizations of
$\\{Y_{i},D_{i},Z_{i},X_{i}\\}$ with observation $i\in{1,...,n}$.
$n_{1}=\sum_{i=1}^{n}Z_{i}$ is the size of the subsample of those with
$Z_{i}=1$, $n_{0}=n-n_{1}$, and $\hat{w}_{i}$ are weights that may depend on
$X_{i}$ or $\hat{p}(x)$, an estimate of the propensity score $p(x)$. Next, we
discuss different methods of estimating $\hat{p}(x)$ and $\hat{w}_{i}$.

###  3.2 Instrument propensity scores

We consider two different approaches to balance the covariates across groups
for units with $Z_{i}=0$ and $Z_{i}=1$. One is to directly control for
covariates $X_{i}$, but some LATE estimators alternatively control for
estimates of $p(x)$, which is motivated by the propensity score’s balancing
properties discussed in rosenbaum1983. Their results imply that $p(x)$ is
capable of equalizing the covariate distributions across instrument states,
such that the instrument is conditionally independent of potential outcomes
and treatments given the propensity score whenever independence holds
conditional on the covariates. A practical advantage of controlling for the
propensity score (rather than a vector of covariates) is that it is one-
dimensional and thus, avoids the curse of dimensionality.

We compute $\hat{p}(x)$ in three different ways. Firstly, we specify a probit
model to estimate the conditional probability $P(Z_{i}=1|X_{i}=x_{i})$ by

$\hat{p}(x)^{probit}=\Phi(x_{i}^{T}\tilde{\beta}_{ML}),$ (6)

where $\tilde{\beta}_{ML}$ denotes the estimated probit coefficients based on
maximum likelihood and $\Phi(x_{i}^{T}\tilde{\beta}_{ML})$ is the cumulative
distribution function of the standard normal distribution evaluated at
$X_{i}^{T}\tilde{\beta}_{ML}$.

Secondly, we apply the covariate balancing propensity score (CBPS) method by
ImaiRatkovic2014 to compute $\hat{p}(x)$. This methodology maximizes covariate
balancing when predicting treatment assignment using the generalized method-
of-moments (GMM) framework. ImaiRatkovic2014 show that the CBPS method is
robust to mild misspecifications of the propensity score model, which is
estimated by the following expression:

$\hat{p}(x)^{CBPS}=\Lambda(x_{i}^{T}\tilde{\beta}_{GMM}),$ (7)

where $\tilde{\beta}_{GMM}$ are coefficients estimated by GMM and
$\Lambda(x_{i}^{T}\tilde{\beta}_{GMM})$ is the cumulative distribution
function of the standard logistic distribution evaluated at
$x_{i}^{T}\tilde{\beta}_{GMM}$. We use the overidentified version of CBPS,
with more moment conditions (based on the covariate balancing condition and
the score of a logit model) than coefficients $\beta_{GMM}$, which are
estimated by continuously updated GMM estimation:

$\tilde{\beta}_{GMM}=\arg\min_{\beta}\bar{g}_{\beta}(Z,X)^{T}\Sigma_{\beta}(Z,X)^{-1}\bar{g}_{\beta}(Z,X).$
(8)

$\bar{g}_{\beta}(Z,X)$ is the sample mean of the moment conditions and
$\Sigma_{\beta}(Z,X)$ is a consistent variance estimator, described in more
detail in Chapter 2.2 of ImaiRatkovic2014.

Our third estimator of the instrument propensity score is fully non-parametric
and based on kernel regression:

$\hat{p}(x)^{lc}=\frac{\sum_{i=1}^{n}z_{i}K\Big{(}\frac{x_{i}-x}{h}\Big{)}}{\sum_{i=1}^{n}K\Big{(}\frac{x_{i}-x}{h}\Big{)}}.$
(9)

Equation 9 corresponds to the Nadaraya-Watson (local constant) kernel
estimator, where $K$ denotes the Epanechnikov kernel and bandwidth $h$ is
chosen by least-squares cross-validation, i.e., by minimizing the least
squares cross validation error w.r.t. $h$, see LiRacine06. As an alternative
to using $\hat{p}(x)^{lc}$ as weighting function, we also apply the Nadaraya-
Watson estimator for estimating the outcome and treatment models in Equation
3, see our discussion on non-parametric estimation methods in Chapter 3.7.

A practically relevant issue of treatment effect methods is thin or lacking
common support (or overlap) in the propensity score, which may compromise
estimation due to a non-comparability across groups, see the discussions in
Imbens03, ImWo08, and LeSt19. If specific propensity score values among one
group are either very rare (thin common support) or absent (lack of common
support) among the opposite group, as it may occur close to the boundaries of
the propensity score, some units may receive a very large weight $\hat{w}_{i}$
in LATE estimation as provided in Equation 5. In the case of thin common
support, these observations could dominate the estimator of the LATE which may
potentially entail an explosion of the variance. In the case of lacking common
support, this even introduces asymptotic bias by giving a large weight to
observations that are not comparable to observations in the opposite group in
terms of the propensity score.

HuLeWu13 and Bodory20 consider a trimming procedure to tackle common support
issues in the sample also discussed in Imbens03, which is asymptotically
unbiased if common support holds asymptotically. It is based on setting the
weights of those observations to zero whose relative share of all weights
within either instrument state in Equation 5 exceeds a particular threshold
value in % (denoted by $t$):

$\hat{w}_{i}=\hat{w}_{i}\mathbb{I}\left\\{\frac{z_{i}/\hat{p}(x)^{lc}}{\sum_{j=1}^{n}z_{j}/\hat{p}(x)^{lc}}+\frac{(1-z_{i})/(1-\hat{p}(x)^{lc})}{\sum_{j=1}^{n}(1-z_{j})/(1-\hat{p}(x)^{lc})}\leq
t\%\right\\}.$ (10)

We set the threshold $t$ to 5% and trim observations based on the weights of
normalized IPW, see (3.3), irrespective of the LATE estimator considered. This
changes (in finite samples) the target parameter due to discarding
observations with extreme weights, but ensures common support prior to
estimation. Note that our bootstrap variance estimators discussed in Section
3.8 account for the stochastic nature of trimming.

###  3.3 Inverse probability weighting (IPW)

Inverse probability weighting (IPW) reweighs (instrument) group-specific
outcomes such that the distribution of the covariates in the total population
is matched, see Hirano+00 for a more detailed discussion. We consider a
normalized IPW estimator in our simulations, which performed well in several
simulation studies on conditionally exogenous treatments, see for instance
HuLeWu13 and BuDNMC09. The IPW-based LATE estimator corresponds to

$\hat{\theta}_{\text{IPW}}=\frac{\sum_{i=1}^{n}z_{i}y_{i}\left\\{\frac{\frac{1}{\hat{p}(x_{i})}}{\sum_{j=1}^{n}\frac{z_{j}}{\hat{p}(x_{j})}}\right\\}-\sum_{i=1}^{n}(1-z_{i})y_{i}\left\\{\frac{\frac{1}{1-\hat{p}(x_{i})}}{\sum_{j=1}^{n}\frac{1-z_{j}}{1-\hat{p}(x_{j})}}\right\\}}{\sum_{i=1}^{n}z_{i}d_{i}\left\\{\frac{\frac{1}{\hat{p}(x_{i})}}{\sum_{j=1}^{n}\frac{z_{j}}{\hat{p}(x_{j})}}\right\\}-\sum_{i=1}^{n}(1-z_{i})d_{i}\left\\{\frac{\frac{1}{1-\hat{p}(x_{i})}}{\sum_{j=1}^{n}\frac{1-z_{j}}{1-\hat{p}(x_{j})}}\right\\}}.$
(11)

The normalizations $\sum_{j=1}^{n}\frac{z_{j}}{\hat{p}(x_{j})}$ and
$\sum_{j=1}^{n}\frac{1-z_{j}}{1-\hat{p}(x_{j})}$ ensure that the weights in
curly brackets add up to one. It is easy to see that (11) corresponds to (5)
when setting $\hat{w}_{i}$ in the latter to
$z_{i}n_{1}\left\\{\frac{\frac{1}{\hat{p}(x_{i})}}{\sum_{j=1}^{n}\frac{z_{j}}{\hat{p}(x_{j})}}\right\\}+(1-z_{i})n_{0}\left\\{\frac{\frac{1}{1-\hat{p}(x_{i})}}{\sum_{j=1}^{n}\frac{1-z_{j}}{1-\hat{p}(x_{j})}}\right\\}$.
IPW possesses the desirable property that it can attain the semiparametric
efficiency bound (implying the smallest possible asymptotic variance) derived
by Ha98, if the propensity score is estimated non-parametrically (while this
is generally not the case for parametric propensity scores). Furthermore, it
is computationally inexpensive and easy to implement. However, evidence in the
treatment effect literature suggests that IPW also has an important drawback:
at the boundaries of the support of the propensity score, estimation may be
unstable and the variance may explode in finite samples, see Froe00a and
KhTa07.

###  3.4 Doubly robust estimation

Doubly robust (DR) estimation combines IPW with outcome regression. It
reweighs outcome models for different instrument states by the inverse of the
propensity scores. Denoting the conditional mean outcomes in the presence and
absence of the instrument by $\mu_{z}^{y}(x):=E[Y_{i}|Z=z_{i},X_{i}=x_{i}]$
and $\mu_{z}^{d}(x):=E[D_{i}|Z_{i}=z_{i},X_{i}=x_{i}]$, the DR LATE estimator
corresponds to

$\hat{\theta}_{\text{DR}}=\frac{\frac{1}{n}\sum_{i=1}^{n}\left(\hat{\mu}_{1}^{y}(x)+\frac{z_{i}(y_{i}-\hat{\mu}_{1}^{y}(x))}{\hat{p}(x)}-\hat{\mu}_{0}^{y}(x)-\frac{(1-z_{i})(y_{i}-\hat{\mu}_{0}^{y}(x))}{1-\hat{p}(x)}\right)}{\frac{1}{n}\sum_{i=1}^{n}\left(\hat{\mu}_{1}^{d}(x)+\frac{z_{i}(d_{i}-\hat{\mu}_{1}^{d}(x))}{\hat{p}(x)}-\hat{\mu}_{0}^{d}(x)-\frac{(1-z_{i})(d_{i}-\hat{\mu}_{0}^{d}(x))}{1-\hat{p}(x)}\right)}.$
(12)

For non-binary outcomes, we run OLS regression to compute
$\hat{\mu}_{z}^{y}(x)=x_{i}^{T}\hat{\beta}_{z,OLS}$. For binary outcome and
treatment variables, we apply probit regression to compute
$\hat{\mu}_{z}(x)=\Phi(x_{i}^{T}\hat{\beta}_{z,ML})$. The coefficients
$\beta_{z}$ are estimated in the subgroups with $Z_{i}\in\\{0,1\\}$.
Differently to IPW, which exclusively relies on reweighing by the propensity
score, the DR estimator remains consistent even if either $\hat{p}(x)$ or
$\hat{\mu}_{z}(x)$ is misspecified, as it makes use of both, the treatment and
outcome models. If both are correctly specified, the DR estimator is semi-
parametrically efficient, as discussed in RobinsRotnitzkyZhao1994.

###  3.5 Matching

Matching is based on assigning (matching) to each observation in one
instrument state one or more units in the other instrument state with
comparable covariates, in order to estimate the LATE based on the ratio of
average differences in the outcome and the treatment across units with and
without instrument in the matched sample. We implement multiple variants of
two types of matching methods, pair and radius matching, to estimate $\theta$.

Pair (or one-to-one) matching with replacement (implying that an observation
may be matched several times) as discussed in Ru73a matches to each reference
observation exactly the observation with the most similar covariates in the
opposite instrument state. This implies the following weights in Equation 5:

$\varpi_{i,j}=\mathbb{I}\left\\{\
|\hat{f}(x_{i},x_{j})|=\underset{k:Z_{k}\in\\{0,1\\}}{\min}|\hat{f}(x_{i},x_{k})|\right\\}.$
(13)

$\varpi_{i,j}$ is the weight of the outcome (or treatment) of observation $j$
in one instrument group (e.g., $Z_{j}=0$) when matched to unit $i$ in the
opposite group (e.g., $Z_{i}=1$), with $Z_{k}=1-Z_{i}$.
$\mathbb{I}\\{\cdot\\}$ is the indicator function, which is one if its
argument is true and zero otherwise. $\hat{f}(\cdot)$ is a function of the
difference in covariates between observations $i$ and $j$. For example, the
function could be defined as the difference in propensity score estimates of
observations $i$ and $j$ in the case of propensity score matching or as a
distance metric w.r.t. the covariate values of $i$ and $j$ like the Euclidean
distance in the case of matching directly on the covariates. In pair matching,
all weights are zero except for the observation $j$ with the smallest
difference with reference unit $i$, which receives a weight of one. For
propensity score matching, we base the weights on the distance of the one-
dimensional propensity score, while for direct matching, we use a normalized
Euclidean distance metric, where differences in the covariates are weighed by
the inverse of the variances of $X_{i}$. Because only one observation is
matched to each unit irrespective of the sample size and the potential
availability of several suitable matches with similar covariates, pair
matching is not efficient (i.e., does not attain the smallest possible
variance asymptotically). On the other hand, it is likely more robust to
propensity score misspecification than IPW, in particular if the misspecified
propensity score model is only a monotone transformation of the true model,
see for instance Zh08, MiTc09, Waernbaum2012, and HuLeWu13.

Radius matching as discussed in RosenbaumRubin1985 and DehejiaWahba99 uses
_all_ matches with propensity scores within a predefined radius around the
reference unit, which trades off some bias in order to increase efficiency (or
precision). This approach expectedly works relatively well if several
comparable potential matches are available for a reference unit. In the
simulations, we consider the radius matching algorithm of LeMiWu11, which
performed well in HuLeWu13, who also provide details on the radius matching-
related weighting function $\hat{w_{i}}$ in Equation 5. The estimator combines
distance-weighted radius matching, where units within the radius are weighted
proportionally to the inverse of their distance to the reference unit, with a
regression-based bias correction, see Ru79 and AbIm11. For the bias
correction, we apply an OLS regression adjustment for $Y$ and a probit
regression adjustment for $D$ to remove small and large sample bias due to
mismatches. HuLeSt2014 provide a detailed description of the estimator. As in
LeMiWu11, the radius size in our simulations is defined as a function of the
distribution of distances between reference units and matches in pair
matching. Namely, it is set to 3 times the maximum pair matching distance.
Note that we include radius matching both with and without conditioning on the
covariate ‘age at first birth’ in addition to the propensity score to account
for this influential confounder.

###  3.6 Parametric regression estimators

In our simulations, IPW, DR estimation, and matching are implemented with
various degrees of flexibility in terms of parametric assumptions. We consider
both semi-parametric versions based on parametric propensity score models,
$\hat{p}(x)^{probit}$ and $\hat{p}(x)^{CBPS}$, as well as fully non-parametric
estimators using the non-parametric propensity scores $\hat{p}(x)^{lc}$ (based
on a local constant kernel regression) or when directly conditioning on
$X_{i}$. For non-parametric DR estimation, also the conditional means of the
binary treatment and binary (or non-binary) outcome $\hat{\mu}_{z}(x)$ are
estimated by local constant (or local linear) kernel regressions.

In addition, we also consider several parametric treatment effect estimators.
The first parametric approach computes the LATE by differences in the
conditional mean functions $\hat{\mu}_{z}(x)$, which are estimated by OLS
regressions for non-binary outcomes and by probit regressions for the
treatment and binary outcome variables (see Section 3.4). Formally, this
regression-based LATE estimator corresponds to the following expression:

$\hat{\theta}_{\text{REGR}}=\frac{\frac{1}{n}\sum_{i=1}^{n}(\hat{\mu}_{1}^{y}(x)-\hat{\mu}_{0}^{y}(x))}{\frac{1}{n}\sum_{i=1}^{n}(\hat{\mu}_{1}^{d}(x)-\hat{\mu}_{0}^{d}(x))}.$
(14)

Furthermore, we apply two-stage least-squares (2SLS) estimation, which was
also applied by Angrist+98 for analysing the data our simulations are based
on. 2SLS may be regarded as a benchmark method for instrumental variable
estimation under the assumption of homogeneous treatment effects. Formally,
the 2SLS estimator is given by

$\displaystyle\hat{\theta}_{\text{2SLS}}=\left[\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{x}_{i}^{T}\tilde{z}_{i}\right)\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{z}_{i}^{T}\tilde{z}_{i}\right)^{-1}\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{z}_{i}^{T}\tilde{x}_{i}\right)\right]^{-1}$
$\displaystyle\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{x}_{i}^{T}\tilde{z}_{i}\right)\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{z}_{i}^{T}\tilde{z}_{i}\right)^{-1}\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{z}_{i}^{T}y_{i}\right),$
(15)

where $\tilde{x}_{i}:=(1,x_{i,1},\cdots,x_{i,K})$,
$\tilde{z}_{i}:=(\tilde{x}_{i},z_{i})$, and $K$ denotes the number of
covariates $X_{i}$. Note that in our just-identified settings with one
treatment and one instrumental variable, the 2SLS estimator is numerically
identical to the limited information maximum likelihood (LIML) estimator.

###  3.7 Further non-parametric estimators

We analyze the performance of three further non-parametric estimation methods
that do not impose any functional form assumptions on the regression functions
of the outcome or the treatment.

Firstly, we apply the generalized random forest (GRF) method, a non-parametric
estimator introduced by ATW18. GRF is a variant of random forest algorithms, a
machine learning approach, see for instance the discussion in LUW20 and
citations therein. As described in Breiman2001, random forests consist of
averaging the predictions of many decision trees applied to different
subsamples that are repeatedly drawn from the original data. In each of these
samples, a decision tree partitions the space of $X_{i}$ into a set of
rectangles and computes the fitted value of $Y_{i}$ as the average outcome in
each of the rectangles. The partitions are chosen in a data-driven way such
that the predictive performance is maximized (e.g. by minimizing the squared
residuals based on the fitted values in each rectangle). A popular estimation
algorithm for decision trees is CART (classification and regression tree), see
for instance Chapter 9.2 in the textbook of hastie01statisticallearning.

GRF shares the core features of ‘traditional’ random forest algorithms like
recursive partitioning, subsampling from the original data, and the random
selection of a subset of covariates at each partitioning step. However, as a
methodological twist, GRF uses a gradient-based partitioning scheme and a
particular (so-called ‘honest’) sample splitting technique (within any of the
drawn sub-samples) that avoids overfitting the predictive models to the
specificities of the data, see WagerAthey2018. Using the conditional
expectation function $\mu_{z}(X_{i})$ in Section 3.4 and applying the GRF to
estimate the latter for the outcome and the treatment to obtain
$\hat{\mu}_{z,RF}^{Y}(x)$ and $\hat{\mu}_{z,RF}^{D}(x)$ for $z\in\\{0,1\\}$
(where the subscript RF indicates the random forest approach), we compute the
LATE as follows:

$\hat{\theta}_{\text{RF}}=\frac{\frac{1}{n}\sum_{i=1}^{n}(\hat{\mu}_{1,RF}^{Y}(x)-\hat{\mu}_{0,RF}^{Y}(x))}{\frac{1}{n}\sum_{i=1}^{n}(\hat{\mu}_{1,RF}^{D}(x)-\hat{\mu}_{0,RF}^{D}(x))}.$
(16)

Algorithm 1 in ATW18 provides more details on the GRF method. We estimate the
conditional expectations in Equation 16 using the default options of the
causal_forest function of the grf package for the statistical software R, see
grf2020.

Alternatively, we could have estimated the predictions $\hat{\mu}_{z,RF}(x)$
by standard Breiman-type random forests [Breiman2001], or considered
double/debiased machine learning estimators based on Neyman orthogonal scores
[DML18] or alternative causal forest algorithms [MCF19] for estimation. Such
methods would also be appropriate to evaluate the finite sample performance of
LATE estimators in high-dimensional settings with many potential covariates,
as they are capable of selecting control variables in a data-driven way, an
interesting topic that we leave for future research.

Secondly, we use non-parametric kernel regression to estimate the conditional
mean functions $\hat{\mu}_{z}(x)$ defined in Section 3.4, see the subscript NP
in the respective estimates in Equation 17. For non-binary outcomes,
$\hat{\mu}_{z,NP}^{y}(x)$ is estimated by local linear kernel regression, for
the binary outcome and treatment variables, $\hat{\mu}_{z,NP}^{y}(x)$ and
$\hat{\mu}_{z,NP}^{d}(x)$ are estimated by local constant kernel regression.

$\hat{\theta}_{\text{NP}}=\frac{\frac{1}{n}\sum_{i=1}^{n}(\hat{\mu}_{1,NP}^{y}(x)-\hat{\mu}_{0,NP}^{y}(x))}{\frac{1}{n}\sum_{i=1}^{n}(\hat{\mu}_{1,NP}^{d}(x)-\hat{\mu}_{0,NP}^{d}(x))}.$
(17)

Finally, we consider a (naive) LATE estimator that is based on the mean
differences of the outcome and treatment variables, respectively, across
instrument states, which in contrast to the other methods does not control for
the covariates. Therefore, the consistency of this approach provided in
Equation 18 generally requires that the IV assumptions hold unconditionally,
i.e., without conditioning on $X$.

$\hat{\theta}_{\text{MEANS}}=\frac{\frac{1}{n_{1}}\sum_{i=1}^{n}y_{i}z_{i}-\frac{1}{n_{0}}\sum_{i=1}^{n}y_{i}(1-z_{i})}{\frac{1}{n_{1}}\sum_{i=1}^{n}d_{i}z_{i}-\frac{1}{n_{0}}\sum_{i=1}^{n}d_{i}(1-z_{i})}.$
(18)

Table 3.7 summarizes the LATE estimators analysed in our simulation study
along with the corresponding conditioning sets.

Table 1: Point estimators

tabularl|ccccc Estimators Conditioning sets
propensity scores covariates none
local
probit cbps constant
Inverse probability weighting $ipw^{probit}$ $ipw^{cbps}$ $ipw^{lc}$
Doubly robust $dr^{probit}$ $dr^{cbps}$ $dr^{lc}$
Pair matching $pairmatch^{probit}$ $pairmatch^{cbps}$ $pairmatch^{lc}$
$pairmatch^{x}$
Radius matching on propensity score $radmatch^{probit}$ $radmatch^{cbps}$
$radmatch^{lc}$
Radius matching on propensity score + covariate $radmatchx^{probit}$
$radmatchx^{cbps}$ $radmatchx^{lc}$
Parametric regressions $reg$
2SLS $tsls$
Random forests $randforest$
Non-parametric regressions $reg^{kernel}$
Mean differences (ignoring covariates) $means$

###  3.8 Inference

Treatment effect estimation frequently relies on the non-parametric bootstrap
for statistical inference [Ef79, Horowitz2001]. In an extensive simulation
study with a conditionally exogenous treatment, Bodory20 find evidence that
variance estimation of treatment effect estimators based on bootstrap
procedures outperforms asymptotic variance approximations in terms of
rejection and coverage probabilities in finite samples. These results even
hold for matching estimators in small samples, despite the inconsistency of
the non-parametric bootstrap for the (non-smooth) pair matching estimator, see
the discussion in AbadieImbens06.

For this reason, we apply the non-parametric bootstrap to estimate the
standard errors of all LATE estimators. This algorithm randomly draws $B$
bootstrap samples of size $n$ (the size of a simulation sample) with
replacement out of each simulation sample and estimates the LATE in every
draw. Denoting the $B$ bootstrapped LATE estimators by $\hat{\theta}^{b}$,
with $b\in\\{1,2,\dots,B\\}$, we estimate the standard error $\sigma$ of a
LATE estimator by

$\hat{\sigma}=\sqrt{\frac{1}{B-1}\sum_{b=1}^{B}\left(\hat{\theta}^{b}-\frac{1}{B}\sum_{b=1}^{B}\hat{\theta}^{b}\right)^{2}}.$
(19)

In line with Bodory20, we set $B=199$. Bootstrapping naturally accounts for
heteroscedasticity as well as uncertainty due to trimming of influential
observations and propensity score estimation.

##  4 Simulation design with empirical data

Simulations often rely on randomly generated data drawn from a probability
distribution that is selected by the researcher. However, the data generating
processes (DGPs) of such simulations may appear somewhat arbitrary in the
sense that they might be far from reflecting typical associations between
variables in empirical data. To improve upon this caveat, HuLeWu13 suggest a
simulation design based on empirical data, also called Empirical Monte Carlo
Study (EMCS), an idea that has been subsequently applied in several papers,
see for instance FrHuWi14, HuLeMe2016, and Bodory20, among others. Briefly,
the idea of an EMCS is to randomly draw small samples from large real data
sets while relying as much as possible on the empirical associations between
the variables when generating the simulation designs.

Our study follows this EMCS approach to evaluate the properties of various IV
estimators of the LATE, with the aim that the simulation designs are more
closely linked to real world data. However, we point out that also in an EMCS,
several important choices about the simulation features are to be made by the
researcher such that the DGPs are not fully determined by the data, see the
caveats raised by AdSl2013. The remainder of this section describes the
implementation of our EMCS. We first present the empirical labor market data
underlying our simulations and then provide the steps for generating the
various simulation designs.

###  4.1 Database

Our simulations are based on empirical data analysed in Angrist+98, who aim at
exploiting exogenous variation in family size to evaluate the treatment effect
of fertility, defined as having at least three vs. two children, on female
labor supply. This database is well suited to analyze the finite sample
properties of IV estimators by means of an EMCS for several reasons. First,
the data set is large, as it comprises 394840 observations and therefore
easily allows one to draw many different random subsamples. Furthermore, the
data contains a strong instrument that importantly affects fertility, namely
twins at second birth.111There may be cases where the randomness of twin
births is violated, see FGV18 for a discussion on dizygotic twinning. In our
simulation study, we artificially generate random and non-random instrument
assignments. Finally, it provides demographic information on the mothers,
which may be used as covariates to control for potential confounders of the
instrument and the outcome.

Coming from the 1980 Census Public Use Micro Samples (PUMS), the data set
contains information on young mothers aged 21 to 35, all of which gave birth
to at least two children. Our analysis considers two different outcomes, the
number of weeks worked within one year (with 43% zeros) and an indicator for
being employed at all in that year. The binary treatment variable indicates if
a mother has more than two kids (treatment is one) or two kids (treatment is
zero). The binary instrumental variable is one if a mother gave birth to twins
at second birth and zero otherwise. The covariates considered in our
simulation include mother’s age, mother’s age at first birth, race, and
quarter of birth.

Table 2: Descriptive statistics of the full sample

tabularl|cccccccccc Variables More than two kids Twins at second birth
No Yes No Yes
mean std mean std st.diff mean std mean std st.diff
Outcomes
Weeks worked / 10 2.32 2.26 1.73 2.13 19 2.09 2.23 1.89 2.19 6
Worked for pay (binary) 0.61 0.49 17 0.57 0.52 7
Treatment
More than two kids (binary) 0.40 1 123
Instrument
Twins at second birth (binary) 0 0.02 15
Covariates: Mothers’ characteristics
Age / 10 2.98 0.36 3.05 0.34 14 3.01 0.35 3.04 0.34 5
Age at first birth /10 2.06 0.30 1.94 0.27 29 2.01 0.29 2.04 0.30 5
African American (binary) 0.10 0.15 11 0.12 0.15 6
Other race (binary) 0.18 0.18 1 0.18 0.20 3
First quarter of birth (binary) 0.24 0.24 1 0.24 0.24 1
Second quarter of birth (binary) 0.24 0.25 1 0.24 0.24 0
Third quarter of birth (binary) 0.27 0.27 0 0.27 0.27 0
Fourth quarter of birth (binary) 0.25 0.43 0.25 0.43 0 0.25 0.43 0.26 0.44 1
$X_{i}\tilde{\beta}$ -0.35 0.36 -0.14 0.36 42 -2.39 0.05 -2.38 0.05 10
$\Phi(X_{i}\tilde{\beta})$ 0.37 0.13 0.45 0.14 42 0.01 0 0.01 0 10
Number of obs., Pseudo-R2 in % 236089 158751 10.5 391460 3380 0.2

Notes: The statistics mean, std, and st.diff stand for mean, standard
deviation, and standardized difference in percent, respectively. The
standardized difference is defined as the absolute difference of means
normalized by the square root of the sum of estimated variances of the
particular variables in both subsamples (see e.g. Imbens and Wooldridge
(2009),p. 24). $\tilde{\beta}$ denotes the estimated probit coeffcients and
$\Phi(X_{i}\tilde{\beta})$ is the cumulative distribution function of the
standard normal distribution evaluated at $X_{i}\tilde{\beta}$. Pseudo-R2 is
the so-called Nagelkerke’s R2:
$(1-exp(-(-2(l_{0}-l_{1}))/n)/(1-exp(-(-2l_{0})/n)$, where $l_{0}$ and $l_{1}$
are the log likelihoods for the null and full model, respectively, and $n$
denotes the number of observations.

Table 4.1 reports descriptive statistics of the database, by treatment
indicator (more than two kids) and the instrument (twins at second birth). The
upper part presents descriptives for the two labor market outcomes ‘weeks
worked’ (in weeks) and ‘worked for pay’ (binary). There are large differences
between the outcomes of the treated and non-treated in terms of the
standardized difference statistic as suggested by RosenbaumRubin1985 (the
literature considers values around 20 and above as severely unbalanced). The
line underneath the outcomes in Table 4.1 gives details on the treatment
variable. Not surprisingly, the treatment fully complies with the instrument
if the latter equals one, because all mothers with twins at second birth
($Z_{i}=1$) necessarily have more than two children ($D_{i}=1$). The
subsequent row of Table 4.1 provides information on the instrument. It reveals
that 2% of women with at least three children have twins at their second
birth. Considering the covariates, the standardized differences show that
mothers’ characteristics are partly unbalanced across treatment states,
whereas they are well balanced across instrument states, in line with a
randomly assigned instrument. The randomness of the instrument is also
supported by the pseudo-R2 statistic with a value of 0.2% when regressing the
instrument on the covariates.

###  4.2 Simulation designs

Data generating processes (DGPs) may differ in (infinitely) many dimensions.
We select ten practically relevant dimensions for varying the specifications
of our simulation models. These dimensions include: effect homogeneity vs.
heterogeneity, randomness vs. non-randomness of the instrument, varying levels
of instrument strength, binary vs. non-binary outcome distributions, and
different sample sizes. Summary statistics of all DPGs are presented in Table
4.2.

We start by assuming homogeneous treatment effects with a randomly assigned
instrument and the empirically observed instrument strength. To evaluate the
performance of the estimators under these conditions, we define a new
population for which the true LATE is equal to zero. To this end, we drop all
3380 observations from the database who receive the instrument ($Z_{i}=1$).
Among the remaining 391460 observations with instrument state $Z_{i}=0$ (no
twins at second birth), there is no reduced form effect of the instrument on
the outcome or first stage effect of the instrument on the treatment, such
that there exists no LATE. After that, we create a pseudo-instrument and
artificially assign $Z_{i}=1$ to those who are similar to the 3380 discarded
observations in terms of observed characteristics. This similarity is
determined by $1:M$ matching on the covariates without replacement. By setting
$M=58$, we assign $Z_{i}=1$ to approximately half of the observations, see
column 4 of Table 4.2. In addition, we set the treatment state of everyone
with $Z_{i}=1$ to $D_{i}=1$ (as in the original database) to maintain the
empirically observed instrument strength. Finally, we draw small samples from
our new population to compare the finite sample properties of alternative LATE
estimators.

To simulate specifications with a weaker instrument, we reduce the first stage
effect by lowering the impact of $Z_{i}$ on $D_{i}$. Instead of setting all
observations with $Z_{i}=1$ to $D_{i}=1$, we change the treatment status from
zero to one only for those with $Z_{i}=1$ for which the condition
$D_{i}=\mathbbm{1}(u_{i}>1.25)$ holds. $\mathbbm{1}(\cdot)$ denotes the
indicator function which is one if its argument is true, otherwise it is zero,
and $u_{i}$ is a standard normally distributed random variable. Column 7 of
Table 4.2 displays the first stage coefficients for the different DGPs.

The randomness of the instrument implies that the covariates are balanced
across groups. To mimic a non-random assignment of the instrument, we increase
the magnitude of instrument selectivity in the following way. We first
estimate the propensity score $\hat{p}(1.5X_{i})^{probit}$ (see Equation 6)
using the original database. Then, we change the instrument status $Z_{i}$
from zero to one for observations with characteristics similar to the 3380
observations dropped from the original database (with $Z_{i}=1$). We obtain
such similar matches by $1:M$ matching on the estimated propensity score
$\hat{p}(1.5X_{i})^{probit}$, with $M=22$. Next, we assign $D_{i}=1$ to all
observations with $Z_{i}=1$. Based on this modified data set, we increase the
selection into the instrument by discarding the best matches for the newly
created observations with $Z_{i}=1$ among observation with $Z_{i}=0$. To find
the best matches to be discarded, we apply $1:M$ matching on a newly estimated
propensity score $\hat{p}(X_{i})^{probit}$ (with the modified instrument
assignments), where $M=3$. The selectivity of the instrument is provided in
columns 5 and 6 in Table 4.2.

To model a scenario with non-constant treatment effects, we introduce effect
heterogeneity with respect to age and race as follows. We add to the existing
control variables squared and cubic terms of both age variables (‘age’ and
‘age at first birth’) and interact the unmodified age variables with the
indicator variable for African Americans. This new set of control variables
for settings with effect heterogeneity is denoted by $X_{i}^{het}$ for each
unit $i$. We generate $Y_{i}$ and $D_{i}$ in each simulation sample according
to the rules $Y_{i}=Y_{i,1}^{d}Z_{i}+Y_{i,0}^{d}(1-Z_{i})$ and
$D_{i}=D_{i,1}Z_{i}+D_{i,1}(1-Z_{i})$. To this end, we compute the non-binary
potential outcomes based on the equation
$Y_{i,z}^{d}=X_{i}^{het}\hat{\beta}_{OLS}+\hat{\sigma}v_{i}$, where $v_{i}$ is
a standard normally distributed random variable. $\hat{\beta}_{OLS}$ and
$\hat{\sigma}$ are the coefficients and residual standard deviation of OLS
regressions in subsamples by instrument state $Z_{i}\in\\{0,1\\}$ of our new
population. The binary potential outcomes are computed based on
$Y_{i,z}^{d}=\mathbbm{1}(X_{i}^{het}\hat{\beta}_{probit}+v_{i}>0)$, where
$\mathbbm{1}(\cdot)$ is the indicator function and $\hat{\beta}_{probit}$ are
the coefficients estimated from probit models in subsamples by instrument
state of our new population. The potential treatments $D_{i,1}$ are set to
one, whereas $D_{i,0}$ is computed analogously to $Y_{i,z}^{d}$ in the binary
outcome case.

We combine these variations in the DGPs with respect to effect heterogeneity,
instrument strength, and instrument selectivity with smaller and larger sample
sizes of 1000 and 2000, respectively, and with binary and non-binary outcome
distributions. We run 2000 simulations for the smaller and 1000 simulations
for the larger samples. Table 4.2 presents summary statistics of the DGPs
considered in our simulation study.

Table 3: Summary statistics (DGPs)

tabularllrrrrrrrrr Random Empiri- Share Share St.diff. Pseudo- First LATE
Trimming
selec- cally (%) (%) (%) R2 (%) stage 1000 2000 (%)
tion observed More Twins-2 Twins-2 Twins-2 (%) 1000 2000
Twins-2strength than 2 $P(Z|X)$$P(Z|X)$Twins-2
Twins-2 children
Effect homogeneity (Weeks worked)
Yes Yes 70 50 23 4 60 0 0 0.03 0.02
Yes No 43 50 23 4 6 0 0 0.03 0.02
No Yes 70 50 51 16 61 0 0 0.06 0.03
No No 43 50 51 16 7 0 0 0.06 0.03
Effect homogeneity (Worked for pay)
Yes Yes 0 0
Yes No 0 0
No Yes 0 0
No No 0 0
Effect heterogeneity (Weeks worked)
Yes Yes 71 50 23 4 57 -0.91 0.27 0.03 0.02
Yes No 85 50 23 4 30 0.52 0.51 0.03 0.02
No Yes 71 50 51 16 58 -0.09 -0.09 0.06 0.05
No No 82 50 51 16 34 -0.21 -0.21 0.06 0.05
Effect heterogeneity (Worked for pay)
Yes Yes 0.00 0.00
Yes No 0.01 0.01
No Yes -0.01 -0.01
No No -0.02 -0.02

Note: The numbers in columns 3-9 are averages over all simulation replications
with samples of sizes 1000 and 2000. ’St.diff’: Standardized difference
defined as absolute mean difference normalized by the square root of the sum
of the estimated variances of the particular variables in both subsamples
[*][p. 24]ImWo08. ’std’: standard deviation. Pseudo-R2 is the so-called
Nagelkerke’s R2: $(1-exp(-(-2(l_{0}-l_{1}))/n)/(1-exp(-(-2l_{0})/n)$, where
$l_{0}$ and $l_{1}$ are the log likelihoods for the null and full model,
respectively, and $n$ is the sample size. Trimming: Share of dropped units due
to the trimming rule in Equation 10 or non-convergence of propensity scores.
All statistics that do not dependend on the outcome variables are only
presented for Weeks worked.

##  5 Results

This section presents results about the finite sample performance of various
LATE estimators across different DGPs. We rank the estimators by their
coverage rates, which are defined as the share of simulations in which the
true LATE is included in the 95% confidence interval of the respective LATE
estimator. We recall that the standard errors for computing those confidence
intervals come from the non-parametric bootstrap, as discussed in Section 3.8.
For the sake of brevity, we subsequently only discuss a selection of our
results, which conveys the main message of our findings. In the Appendix, we
include more detailed results.

Table 4: Coverage rates and intervals

point | coverage | diff | confidence | diff
---|---|---|---|---
estimators | rates | (pp) | intervals | (%)
$randforest$ | 95.0 | 0.0 | 228.8 | 8.0
$radmatch^{probit}$ | 95.1 | 0.1 | 356.5 | 68.3
$reg^{kernel}$ | 94.8 | 0.2 | 329.4 | 55.5
$pairmatch^{x}$ | 94.6 | 0.4 | 265.6 | 25.4
$ipw^{cbps}$ | 95.5 | 0.5 | 290.4 | 37.1
$radmatchx^{cbps}$ | 94.4 | 0.6 | 284.5 | 34.3
$radmatchx^{probit}$ | 94.1 | 0.9 | 265.6 | 25.4
$reg$ | 96.0 | 1.0 | 211.8 | 0.0
$tsls$ | 96.0 | 1.0 | 260.9 | 23.2
$dr^{probit}$ | 96.0 | 1.0 | 257.5 | 21.6
$means$ | 90.5 | 4.5 | 234.5 | 10.7

  * •

Notes: ‘diff’ indicates the difference to the left best performer in
percentage points (pp) or in percent (%).

Table 4 provides the average coverage rates and lengths of confidence
intervals across all DGPs of any parametric, semi-parametric, or non-
parametric LATE estimator which performs best (in terms of coverage) in at
least one of the ten DGPs discussed in Section 4.2. We find that only the non-
parametric random forest-based LATE estimator described in Equation 16 attains
exactly the nominal coverage size of 95% on average. Furthermore, its average
length of confidence intervals is the second shortest among the estimators
analyzed in Table 4, 8% larger than the average interval of the parametric
regression estimator, the nominal size of which is 96%. Conditional on
obtaining a decent coverage, a short confidence interval is desirable in terms
of precision, as it implies a lower estimation uncertainty. Three out of the
four LATE estimators whose average coverage rates come closest to 95% are non-
parametric, with those of non-parametric kernel regression (94.8%) and pair
matching on the covariates (94.6%) having a minor under-coverage. Also semi-
parametric radius matching on the propensity score performs decent in terms of
coverage rates, with the probit-based version attaining an average rate of
95.1%, and two further versions achieving 94.4% and 94.1%, respectively.
Furthermore, also IPW using the CBPS method for propensity score estimation
reaches a satisfactory average coverage rate of 95.5%. However, the average
length of the confidence intervals of radius matching, kernel regression, pair
matching, and IPW is substantially larger than that of parametric regression
or of the random forest.

Table 5: Average absolute biases, standard deviations, and root mean square
errors

point | bias | diff | sd | diff | rmse | diff | bias | diff
---|---|---|---|---|---|---|---|---
estimators | effect | (%) |  | (%) |  | (%) | se | (%)
$randforest$ | 0.6 | 0.0 | 8.0 | 14.1 | 8.0 | 11.9 | 1.7 | 0.0
$radmatch^{probit}$ | 1.3 | 100.1 | 12.0 | 70.0 | 12.0 | 67.3 | 10.8 | 540.7
$reg^{kernel}$ | 1.4 | 124.8 | 7.0 | 0.0 | 7.2 | 0.0 | 4.2 | 149.3
$pairmatch^{x}$ | 0.8 | 21.4 | 9.9 | 40.3 | 9.9 | 37.6 | 5.8 | 247.4
$ipw^{cbps}$ | 1.2 | 85.4 | 7.4 | 5.4 | 7.5 | 4.7 | 4.4 | 162.4
$radmatchx^{cbps}$ | 1.1 | 76.9 | 8.6 | 22.9 | 8.7 | 21.5 | 4.1 | 142.8
$radmatchx^{probit}$ | 1.2 | 86.4 | 9.7 | 38.1 | 9.8 | 36.4 | 3.4 | 100.1
$reg$ | 0.7 | 10.1 | 7.3 | 3.7 | 7.3 | 1.9 | 4.0 | 140.0
$tsls$ | 0.7 | 11.3 | 7.9 | 12.7 | 7.9 | 10.6 | 3.8 | 124.0
$dr^{probit}$ | 0.8 | 31.9 | 7.6 | 8.6 | 7.7 | 6.9 | 3.9 | 133.3
$means$ | 3.4 | 440.1 | 7.9 | 11.7 | 8.8 | 21.9 | 5.6 | 235.3

  * •

Notes: ‘bias effect’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘bias se’ indicates the median bias of the estimated
bootstrap standard error. ‘diff’ indicates the difference to the left best
performer in percent (%).

The coverage accuracy of the different estimation methods is related to the
bias and variance of the LATE estimators, as well as the bias of the
bootstrap-based standard error. Table 5 provides details on these statistics.
We find that two non-parametric methods perform best when considering the bias
of LATE estimation, the standard deviation, and the mean squared error (i.e.,
the sum of the squared bias and the variance), as well as the bias of the
standard error (relative to a LATE estimator’s true standard deviation). The
random forest-based LATE estimator has on average the smallest deviation from
the true LATE, with its absolute bias amounting to 0.6. The non-parametric
kernel regression estimator has the smallest average standard deviation among
the estimators in Table 5, amounting to 7.0. It also performs best in terms of
root mean squared errors across DGPs with an average value of 7.2. When
considering the median bias of the bootstrap standard errors relative to the
true standard deviations of the respective LATE estimators, the inference
method of the random forest-based LATE estimator performs best, with an
average median bias of 1.7. The averages of the median biases of its
competitors are on average at least 100% larger.

Our findings suggest that the coverage accuracy is mainly driven by a LATE
estimator’s bias. This is for instance the reason why the mean differences
estimator (which ignores covariates), the bias of which exceeds the bias of
the random forest-based LATE estimator by 440.1%, shows a poor coverage in
Table 4. Also the OLS estimator performs poorly in terms of coverage, due to
its high bias, while its variance is small (results not presented but
available on request). The performance of all LATE estimators by DGP is
presented in Section A.3 of the Appendix. Tables 18-49 provide details on the
coverage rates, biases, standard deviations, and root mean squared errors.

Table 6: Best performing estimators in terms of coverage rates

DGP feature | point estimator
---|---
| parametric | semi-parametric | non-parametric
effect homogeneity | $tsls$ | $radmatchx^{probit}$ | $means$
effect heterogeneity | $tsls$ | $dr^{probit}$ | $randforest$
standard selection | $reg$ | $radmatchx^{cbps}$ | $randforest$
strong selection | $tsls$ | $ipw^{cbps}$ | $randforest$
weaker first stage | $reg$ | $dr^{probit}$ | $randforest$
strong first stage | $reg$ | $radmatchx^{probit}$ | $pairmatch^{x}$
non-binary outcome | $reg$ | $radmatch^{probit}$ | $randforest$
binary outcome | $tsls$ | $radmatch^{probit}$ | $randforest$
small sample size | $reg$ | $radmatch^{probit}$ | $reg^{kernel}$
larger sample size | $tsls$ | $radmatch^{probit}$ | $reg^{kernel}$

  * •

Notes: The best performing LATE estimator under each DGP feature is shown in
bold print.

Table 6 lists the best performing LATE estimators in terms of average
coverage, separately for each of ten DGP features (see the rows) as well as
for parametric, semi-parametric, and non-parametric methods (see the columns).
The results suggest that semi-parametric and non-parametric estimators come
closest to the nominal coverage rate of 95%. The radius matching algorithm of
LeMiWu11 (with or without controlling for a covariate in addition to the
propensity score) most frequently performs best both among the semi-parametric
LATE estimators (in 70% of cases) and overall (in 50% of cases). Radius
matching achieves the best average coverage in settings with effect
homogeneity, a strong instrument, non-binary and binary outcomes, and under a
larger sample size. For specifications with standard and strong selection into
the instrument, the non-parametric random forest-based estimator is closest to
the nominal size. Considering scenarios with effect heterogeneity, a weaker
instrument, and a small sample size, the best performers are LATE estimators
based on 2SLS, DR, and non-parametric regression, respectively.

The performance of the best performing LATE estimators across the ten DGP
features is presented in Section A.2 of the Appendix. Tables 8-17 provide
information on the average coverage rates, biases, standard deviations, and
root mean squared errors.

##  6 Conclusion

This paper presented a simulation study based on empirical labor market data
to investigate the finite sample properties of a range of point estimators of
the local average treatment effect (LATE) when controlling for a fixed (and
low-dimensional) set of covariates. The structure of these estimators is
inspired by the Wald estimator, consisting of the ratio of the estimated
reduced form effect of the instrument on the outcome and the estimated first
stage effect of the instrument on the treatment. Furthermore, we applied the
non-parametric bootstrap to estimate the standard errors and the 95%
confidence intervals of the LATE estimators. We find that among the LATE
estimators considered, non-parametric kernel regression has the smallest
average root mean squared error across the different simulations, closely
followed by the random forest-based approach, which has the lowest average
absolute bias. The random forest method also performs very competitive in
terms of average coverage rates, while at the same time having relatively
narrow confidence intervals, which is attractive in terms of precision.
Specific versions of semi-parametric radius matching on the propensity score,
nonparametric kernel regression, inverse probability weighting, and pair
matching on the covariates perform decently in terms of coverage, too, but
have substantially wider confidence intervals.

## References

  * [1] [Abadie]Abadie2003Abadie00 Abadie, A. (2003): “Semiparametric instrumental Variable estimation of treatment response models,” _Journal of Econometrics_ , 113, 231–263.
  * [2] [Abadie and Imbens]Abadie and Imbens2008AbadieImbens06 Abadie, A., and G. W. Imbens (2008): “On the Failure of the Bootstrap for Matching Estimators,” _Econometrica_ , 76, 1537–1557.
  * [3] [Abadie and Imbens]Abadie and Imbens2011AbIm11  (2011): “Bias-Corrected Matching Estimators for Average Treatment Effects,” _Journal of Business and Economic Statistics_ , 29, 1–11.
  * [4] [Advani and Sloczyński]Advani and Sloczyński2013AdSl2013 Advani, A., and T. Sloczyński (2013): “Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies,” _IZA Discussion Paper No. 7874_.
  * [5] [Angrist and Evans]Angrist and Evans1998Angrist+98 Angrist, J., and W. Evans (1998): “Children and their parents labor supply: Evidence from exogeneous variation in family size,” _American Economic Review_ , 88, 450–477.
  * [6] [Angrist, Imbens, and Rubin]Angrist, Imbens, and Rubin1996Angrist+96 Angrist, J., G. Imbens, and D. Rubin (1996): “Identification of Causal Effects using Instrumental Variables,” _Journal of American Statistical Association_ , 91, 444–472 (with discussion).
  * [7] [Athey, Tibshirani, and Wager]Athey, Tibshirani, and Wager2019ATW18 Athey, S., J. Tibshirani, and S. Wager (2019): “Generalized random forests,” _The Annals of Statistics_ , 47(2), 1148 – 1178.
  * [8] [Bodory, Camponovo, Huber, and Lechner]Bodory, Camponovo, Huber, and Lechner2020Bodory20 Bodory, H., L. Camponovo, M. Huber, and M. Lechner (2020): “The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators,” _Journal of Business & Economic Statistics_, 38(1), 183–200.
  * [9] [Breiman]Breiman2001Breiman2001 Breiman, L. (2001): “Random Forests,” _Machine Learning_ , 45, 5–32.
  * [10] [Busso, DiNardo, and McCrary]Busso, DiNardo, and McCrary2014BuDNMC09 Busso, M., J. DiNardo, and J. McCrary (2014): “New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators,” _Review of Economics and Statistics_ , 96, 885–897.
  * [11] [Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey, and Robins]Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey, and Robins2018DML18 Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins (2018): “Double/debiased machine learning for treatment and structural parameters,” _The Econometrics Journal_ , 21(1), C1–C68.
  * [12] [Dehejia and Wahba]Dehejia and Wahba1999DehejiaWahba99 Dehejia, R. H., and S. Wahba (1999): “Causal Effects in Non-experimental Studies: Reevaluating the Evaluation of Training Programmes,” _Journal of American Statistical Association_ , 94, 1053–1062.
  * [13] [Efron]Efron1979Ef79 Efron, B. (1979): “Bootstrap Methods: Another Look at the Jackknife,” _The Annals of Statistics_ , 7, 1–26.
  * [14] [Farbmacher, Guber, and Vikström]Farbmacher, Guber, and Vikström2018FGV18 Farbmacher, H., R. Guber, and J. Vikström (2018): “Increasing the credibility of the Twin birth instrument,” _Journal of Applied Econometrics_ , 33(3), 457–472.
  * [15] [Frölich]Frölich2004Froe00a Frölich, M. (2004): “Finite Sample Properties of Propensity-Score Matching and Weighting Estimators,” _The Review of Economics and Statistics_ , 86, 77–90.
  * [16] [Frölich]Frölich2007Froel2007 Frölich, M. (2007): “Nonparametric IV estimation of local average treatment effects with covariates,” _Journal of Econometrics_ , 139(1), 35–75.
  * [17] [Frölich, Huber, and Wiesenfarth]Frölich, Huber, and Wiesenfarth2017FrHuWi14 Frölich, M., M. Huber, and M. Wiesenfarth (2017): “The finite sample performance of semi- and nonparametric estimators for treatment effects and policy evaluation,” _Computational Statistics and Data Analysis_ , 115, 91–102.
  * [18] [Hahn]Hahn1998Ha98 Hahn, J. (1998): “On the role of the propensity score in efficient semiparametric estimation of average treatment effects,” _Econometrica_ , 66(2), 315–331.
  * [19] [Hastie, Tibshirani, and Friedman]Hastie, Tibshirani, and Friedman2001hastie01statisticallearning Hastie, T., R. Tibshirani, and J. Friedman (2001): _The Elements of Statistical Learning_ , Springer Series in Statistics. Springer New York Inc., New York, NY, USA.
  * [20] [Hirano, Imbens, and Ridder]Hirano, Imbens, and Ridder2003Hirano+00 Hirano, K., G. W. Imbens, and G. Ridder (2003): “Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score,” _Econometrica_ , 71, 1161–1189.
  * [21] [Horowitz]Horowitz2001Horowitz2001 Horowitz, J. L. (2001): “The Bootstrap,” in _Handbook of Econometrics_ , ed. by J. J. Heckman, and E. Learer, pp. 3159–3228. North-Holland.
  * [22] [Huber, Lechner, and Mellace]Huber, Lechner, and Mellace2016HuLeMe2016 Huber, M., M. Lechner, and G. Mellace (2016): “The Finite Sample Performance of Estimators for Mediation Analysis Under Sequential Conditional Independence,” _Journal of Business & Economic Statistics_, 34(1), 139–160.
  * [23] [Huber, Lechner, and Steinmayr]Huber, Lechner, and Steinmayr2014HuLeSt2014 Huber, M., M. Lechner, and A. Steinmayr (2014): “Radius matching on the propensity score with bias adjustment: tuning parameters and finite sample behaviour,” _forthcoming in Empirical Economics_.
  * [24] [Huber, Lechner, and Wunsch]Huber, Lechner, and Wunsch2013HuLeWu13 Huber, M., M. Lechner, and C. Wunsch (2013): “The performance of estimators based on the propensity score,” _Journal of Econometrics_ , 175, 1–21.
  * [25] [Imai and Ratkovic]Imai and Ratkovic2014ImaiRatkovic2014 Imai, K., and M. Ratkovic (2014): “Covariate balancing propensity score,” _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 76, 243–263.
  * [26] [Imbens]Imbens2004Imbens03 Imbens, G. W. (2004): “Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review,” _The Review of Economics and Statistics_ , 86, 4–29.
  * [27] [Imbens and Angrist]Imbens and Angrist1994Imbens+94 Imbens, G. W., and J. Angrist (1994): “Identification and Estimation of Local Average Treatment Effects,” _Econometrica_ , 62, 467–475.
  * [28] [Imbens and Wooldridge]Imbens and Wooldridge2009ImWo08 Imbens, G. W., and J. M. Wooldridge (2009): “Recent Developments in the Econometrics of Program Evaluation,” _Journal of Economic Literature_ , 47, 5–86.
  * [29] [Khan and Tamer]Khan and Tamer2010KhTa07 Khan, S., and E. Tamer (2010): “Irregular Identification, Support Conditions, and Inverse Weight Estimation,” _Econometrica_ , 78, 2021–2042.
  * [30] [Lechner and Mareckova]Lechner and Mareckova2022MCF19 Lechner, M., and J. Mareckova (2022): “Modified Causal Forest,” working paper, University of St. Gallen, School of Economics and Political Science.
  * [31] [Lechner, Miquel, and Wunsch]Lechner, Miquel, and Wunsch2011LeMiWu11 Lechner, M., R. Miquel, and C. Wunsch (2011): “Long-run Effects of Public Sector Sponsored Training in West Germany,” _Journal of the European Economic Association_ , 9, 742–784.
  * [32] [Lechner and Strittmatter]Lechner and Strittmatter2019LeSt19 Lechner, M., and A. Strittmatter (2019): “Practical procedures to deal with common support problems in matching estimation,” _Econometric Reviews_ , 38(2), 193–207.
  * [33] [Lee, Ullah, and Wang]Lee, Ullah, and Wang2020LUW20 Lee, T.-H., A. Ullah, and R. Wang (2020): _Bootstrap Aggregating and Random Forest_ pp. 389–429. Springer.
  * [34] [Li and Racine]Li and Racine2006LiRacine06 Li, Q., and J. S. Racine (2006): _Nonparametric Econometrics: Theory and Practice_ , no. 8355 in Economics Books. Princeton University Press.
  * [35] [Lunceford and Davidian]Lunceford and Davidian2004LuncefordDavidian2004 Lunceford, J. K., and M. Davidian (2004): “Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study,” _Statistics in Medicine_ , 23, 2937–2960.
  * [36] [Millimet and Tchernis]Millimet and Tchernis2009MiTc09 Millimet, D., and R. Tchernis (2009): “On the specification of propensity scores, with applications to the analysis of trade policies.,” _Journal of Business & Economic Statistics_, 27, 297–315.
  * [37] [Robins, Rotnitzky, and Zhao]Robins, Rotnitzky, and Zhao1994RobinsRotnitzkyZhao1994 Robins, J. M., A. Rotnitzky, and L. P. Zhao (1994): “Estimation of regression coefficients when some regressors are not always observed,” _Journal of the American Statistical Association_ , 89, 846–866.
  * [38] [Rosenbaum and Rubin]Rosenbaum and Rubin1983rosenbaum1983 Rosenbaum, P. R., and D. B. Rubin (1983): “The Central Role of the Propensity Score in Observational Studies for Causal Effects,” _Biometrika_ , 70(1), 41–55.
  * [39] [Rosenbaum and Rubin]Rosenbaum and Rubin1985RosenbaumRubin1985  (1985): “Constructing a control group using multivariate matched sampling methods that incorporate the propensity score.,” _The American Statistician_ , 39, 33–38.
  * [40] [Rubin]Rubin1973Ru73a Rubin, D. B. (1973): “Matching to Remove Bias in Observational Studies,” _Biometrics_ , 29, 159–183.
  * [41] [Rubin]Rubin1974Rubin74  (1974): “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies,” _Journal of Educational Psychology_ , 66, 688–701.
  * [42] [Rubin]Rubin1979Ru79  (1979): “Using Multivariate Matched Sampling and Regression Adjustment to Control Bias in Observational Studies,” _Journal of the American Statistical Association_ , 74, 318–328.
  * [43] [Smith and Todd]Smith and Todd2005SmithTodd00 Smith, J., and P. Todd (2005): “Does matching overcome LaLonde’s critique of nonexperimental estimators?,” _Journal of Econometrics_ , 125, 305–353.
  * [44] [Tan]Tan2006Tan2006 Tan, Z. (2006): “Regression and Weighting Methods for Causal Inference Using Instrumental Variables,” _Journal of the American Statistical Association_ , 101, 1607–1618.
  * [45] [Tibshirani, Athey, Friedberg, Hadad, Hirshberg, Miner, Sverdrup, Wager, and Wright]Tibshirani, Athey, Friedberg, Hadad, Hirshberg, Miner, Sverdrup, Wager, and Wright2020grf2020 Tibshirani, J., S. Athey, R. Friedberg, V. Hadad, D. Hirshberg, L. Miner, E. Sverdrup, S. Wager, and M. Wright (2020): _grf: Generalized Random Forests_ R package version 4.0.2).
  * [46] [Waernbaum]Waernbaum2012Waernbaum2012 Waernbaum, I. (2012): “Model misspecification and robustness in causal inference: comparing matching with doubly robust estimation,” _Statistics in Medicine_ , 31, 1572–1581.
  * [47] [Wager and Athey]Wager and Athey2018WagerAthey2018 Wager, S., and S. Athey (2018): “Estimation and Inference of Heterogeneous Treatment Effects using Random Forests,” _Journal of the American Statistical Association_ , 113(523), 1228–1242.
  * [48] [Zhao]Zhao2004Zh04 Zhao, Z. (2004): “Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence,” _Review of Economics and Statistics_ , 86, 91–107.
  * [49] [Zhao]Zhao2008Zh08  (2008): “Sensitivity of Propensity Score Methods to the Specifications,” _Economics Letters_ , 98, 309–319.
  * [50]

##  Appendix A Appendix

###  A.1 DGPs

Table A.1 summarizes the 32 data generating processes of our Empirical Monte
Carlo study. They differ in several dimensions, namely with respect to effect
heterogeneity, instrument selectivity, instrument strength, outcome
distribution, and sample size.

Table 7: Numbered list of DGPs

tabularl|ccccc Effect Strong Empirically Binary Sample
hetero- instrument observed outcome size
geneity selection instrument
DGP strength
1 0 0 1 0 1000
2 1 0 0 0 1000
3 0 1 1 0 1000
4 0 0 0 0 1000
5 1 1 0 0 1000
6 1 0 1 0 1000
7 0 1 0 0 1000
8 1 1 1 0 1000
9 0 0 1 1 1000
10 1 0 0 1 1000
11 0 1 1 1 1000
12 0 0 0 1 1000
13 1 1 0 1 1000
14 1 0 1 1 1000
15 0 1 0 1 1000
16 1 1 1 1 1000
17 0 0 1 0 2000
18 1 0 0 0 2000
19 0 1 1 0 2000
20 0 0 0 0 2000
21 1 1 0 0 2000
22 1 0 1 0 2000
23 0 1 0 0 2000
24 1 1 1 0 2000
25 0 0 1 1 2000
26 1 0 0 1 2000
27 0 1 1 1 2000
28 0 0 0 1 2000
29 1 1 0 1 2000
30 1 0 1 1 2000
31 0 1 0 1 2000
32 1 1 1 1 2000

###  A.2 Local average treatment effects averaged over different dimensions

The following Tables 8-17 evaluate the LATE estimators with respect to
different properties like the absolute bias or the root mean squared error.
The tables refer to different scenarios that vary in terms of effect
heterogeneity, instrument selectivity, instrument strength, outcome
distribution, and sample size. The estimators are sorted according to their
performance in terms of coverage rates for 95% confidence intervals.

Table 8: Effect homogeneity

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$radmatchx^{probit}$ | 96.0 | 6 | 525.3 | 26 | 8 | 2.0 | 94 | 6 | 17.8 | 41 | 9 | 18.0 | 40 | 9 | 6.9 | 99 | 9
$radmatchx^{cbps}$ | 96.1 | 6 | 563.3 | 35 | 8 | 1.9 | 80 | 6 | 15.7 | 25 | 8 | 15.8 | 23 | 8 | 8.3 | 140 | 8
$means$ | 93.8 | 9 | 464.1 | 11 | 4 | 5.7 | 447 | 11 | 14.4 | 15 | 3 | 15.8 | 23 | 7 | 11.3 | 227 | 5
$pairmatch^{x}$ | 96.3 | 7 | 525.6 | 26 | 8 | 1.3 | 27 | 4 | 18.1 | 44 | 8 | 18.2 | 41 | 8 | 11.8 | 242 | 8
$randforest$ | 96.4 | 6 | 452.6 | 8 | 3 | 1.0 | 0 | 4 | 14.7 | 17 | 6 | 14.7 | 14 | 5 | 3.5 | 0 | 6
$reg^{kernel}$ | 96.5 | 8 | 653.7 | 56 | 6 | 2.4 | 130 | 9 | 12.6 | 0 | 5 | 12.9 | -0 | 5 | 8.6 | 147 | 6
$radmatch^{probit}$ | 96.5 | 5 | 706.5 | 69 | 10 | 2.2 | 113 | 7 | 22.2 | 76 | 10 | 22.3 | 73 | 10 | 21.7 | 526 | 9
$ipw^{cbps}$ | 97.0 | 6 | 575.7 | 38 | 6 | 2.0 | 90 | 7 | 13.5 | 7 | 4 | 13.7 | 6 | 4 | 8.9 | 156 | 5
$tsls$ | 97.1 | 4 | 516.6 | 23 | 4 | 1.2 | 18 | 4 | 14.5 | 15 | 4 | 14.6 | 13 | 4 | 7.6 | 118 | 5
$reg$ | 97.1 | 4 | 418.5 | 0 | 4 | 1.2 | 14 | 4 | 13.2 | 5 | 4 | 13.3 | 4 | 3 | 8.1 | 134 | 4
$dr^{probit}$ | 97.2 | 5 | 509.8 | 22 | 5 | 1.4 | 38 | 5 | 13.9 | 11 | 4 | 14.0 | 9 | 4 | 7.9 | 127 | 3

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 9: Effect heterogeneity

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$tsls$ | 94.9 | 2 | 5.1 | 4 | 3 | 0.2 | 0 | 4 | 1.3 | 3 | 2 | 1.3 | 0 | 1 | -0.0 | 5 | 3
$reg$ | 94.9 | 2 | 5.2 | 6 | 5 | 0.2 | 14 | 4 | 1.3 | 5 | 4 | 1.3 | 2 | 2 | -0.0 | 6 | 3
$dr^{probit}$ | 94.8 | 2 | 5.2 | 7 | 6 | 0.2 | 28 | 5 | 1.4 | 5 | 4 | 1.3 | 3 | 3 | -0.0 | 0 | 4
$ipw^{cbps}$ | 94.0 | 4 | 5.1 | 5 | 5 | 0.4 | 106 | 8 | 1.3 | 4 | 4 | 1.4 | 4 | 4 | -0.0 | 13 | 3
$randforest$ | 93.6 | 6 | 5.0 | 2 | 2 | 0.2 | 26 | 5 | 1.4 | 7 | 6 | 1.4 | 4 | 4 | -0.1 | 244 | 7
$radmatch^{probit}$ | 93.6 | 6 | 6.4 | 31 | 11 | 0.3 | 72 | 5 | 1.8 | 37 | 11 | 1.8 | 35 | 10 | -0.1 | 383 | 8
$reg^{kernel}$ | 93.2 | 7 | 5.1 | 5 | 5 | 0.4 | 150 | 9 | 1.5 | 15 | 6 | 1.5 | 16 | 7 | -0.2 | 503 | 6
$pairmatch^{x}$ | 92.8 | 8 | 5.7 | 16 | 9 | 0.2 | 17 | 4 | 1.6 | 24 | 9 | 1.6 | 21 | 9 | -0.1 | 405 | 10
$radmatchx^{cbps}$ | 92.7 | 8 | 5.7 | 18 | 9 | 0.3 | 103 | 6 | 1.6 | 24 | 8 | 1.6 | 24 | 8 | -0.1 | 358 | 8
$radmatchx^{probit}$ | 92.2 | 9 | 5.8 | 18 | 10 | 0.3 | 88 | 5 | 1.6 | 26 | 10 | 1.6 | 26 | 9 | -0.2 | 422 | 10
$means$ | 87.1 | 11 | 4.9 | 0 | 2 | 1.1 | 539 | 11 | 1.3 | 0 | 2 | 1.7 | 29 | 8 | -0.0 | 43 | 3

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 10: Empirically observed instrument selection

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$randforest$ | 95.0 | 7 | 202.8 | 0 | 1 | 0.6 | 0 | 4 | 7.5 | 5 | 6 | 7.5 | 5 | 4 | 2.0 | 0 | 8
$radmatchx^{cbps}$ | 95.0 | 6 | 337.4 | 66 | 9 | 0.7 | 30 | 3 | 9.3 | 31 | 8 | 9.3 | 31 | 8 | 3.9 | 94 | 8
$pairmatch^{x}$ | 94.7 | 8 | 245.7 | 21 | 8 | 0.9 | 66 | 4 | 9.7 | 37 | 9 | 9.8 | 37 | 9 | 5.1 | 152 | 10
$radmatch^{probit}$ | 95.4 | 5 | 292.2 | 44 | 10 | 1.0 | 85 | 5 | 9.5 | 33 | 10 | 9.5 | 33 | 9 | 8.3 | 315 | 7
$reg^{kernel}$ | 95.5 | 7 | 264.5 | 30 | 6 | 1.5 | 160 | 10 | 7.6 | 7 | 6 | 7.7 | 8 | 6 | 4.7 | 135 | 6
$radmatchx^{probit}$ | 94.5 | 8 | 320.4 | 58 | 9 | 0.7 | 29 | 2 | 10.4 | 47 | 9 | 10.4 | 46 | 9 | 3.3 | 62 | 9
$ipw^{cbps}$ | 95.9 | 4 | 315.1 | 55 | 6 | 1.1 | 97 | 9 | 8.8 | 24 | 5 | 8.9 | 24 | 5 | 4.3 | 112 | 4
$dr^{probit}$ | 96.0 | 4 | 297.9 | 47 | 5 | 0.9 | 67 | 7 | 7.8 | 10 | 3 | 7.8 | 10 | 3 | 4.0 | 98 | 4
$reg$ | 96.0 | 3 | 209.8 | 3 | 4 | 0.9 | 62 | 6 | 7.1 | 0 | 3 | 7.1 | 0 | 2 | 4.5 | 125 | 3
$tsls$ | 96.0 | 4 | 252.6 | 25 | 3 | 1.0 | 70 | 6 | 8.1 | 15 | 3 | 8.2 | 15 | 3 | 4.2 | 107 | 4
$means$ | 93.0 | 10 | 207.0 | 2 | 3 | 2.8 | 406 | 11 | 7.9 | 12 | 3 | 8.5 | 20 | 8 | 5.8 | 187 | 4

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 11: Strong instrument selection

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$randforest$ | 95.1 | 5 | 254.7 | 21 | 4 | 0.7 | 55 | 5 | 8.6 | 43 | 6 | 8.6 | 39 | 4 | 1.4 | 0 | 6
$ipw^{cbps}$ | 95.1 | 6 | 265.7 | 26 | 5 | 1.2 | 174 | 7 | 6.0 | 0 | 3 | 6.2 | -0 | 4 | 4.6 | 237 | 4
$radmatch^{probit}$ | 94.7 | 5 | 420.7 | 100 | 11 | 1.5 | 229 | 7 | 14.4 | 140 | 11 | 14.5 | 135 | 11 | 13.2 | 875 | 10
$pairmatch^{x}$ | 94.4 | 7 | 285.5 | 36 | 8 | 0.6 | 33 | 4 | 10.0 | 66 | 8 | 10.0 | 62 | 8 | 6.6 | 388 | 8
$reg^{kernel}$ | 94.2 | 8 | 394.3 | 87 | 4 | 1.4 | 205 | 9 | 6.5 | 8 | 5 | 6.7 | 8 | 6 | 3.7 | 171 | 6
$tsls$ | 96.0 | 3 | 269.1 | 28 | 4 | 0.5 | 0 | 2 | 7.7 | 28 | 3 | 7.7 | 25 | 2 | 3.4 | 150 | 4
$reg$ | 96.0 | 3 | 213.9 | 2 | 5 | 0.5 | 7 | 2 | 7.5 | 25 | 4 | 7.5 | 22 | 3 | 3.5 | 162 | 4
$dr^{probit}$ | 96.0 | 3 | 217.1 | 3 | 6 | 0.7 | 61 | 3 | 7.5 | 24 | 6 | 7.5 | 22 | 4 | 3.9 | 186 | 3
$radmatchx^{cbps}$ | 93.8 | 8 | 231.7 | 10 | 8 | 1.5 | 233 | 9 | 8.0 | 32 | 8 | 8.1 | 32 | 8 | 4.3 | 214 | 8
$radmatchx^{probit}$ | 93.7 | 8 | 210.7 | 0 | 9 | 1.6 | 261 | 8 | 9.0 | 50 | 9 | 9.2 | 49 | 9 | 3.5 | 157 | 10
$means$ | 87.9 | 11 | 262.0 | 24 | 2 | 4.0 | 782 | 11 | 7.8 | 30 | 2 | 9.0 | 45 | 7 | 5.5 | 307 | 4

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 12: Weaker instrument strength

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$dr^{probit}$ | 94.8 | 2 | 5.4 | 6 | 6 | 0.2 | 28 | 5 | 1.4 | 5 | 4 | 1.4 | 2 | 4 | 0.0 | 1327 | 3
$reg$ | 94.7 | 3 | 5.4 | 5 | 5 | 0.2 | 13 | 4 | 1.4 | 5 | 4 | 1.4 | 1 | 2 | -0.0 | 0 | 3
$tsls$ | 94.7 | 3 | 5.3 | 4 | 3 | 0.2 | 0 | 4 | 1.4 | 3 | 2 | 1.4 | 0 | 1 | -0.0 | 1563 | 4
$ipw^{cbps}$ | 94.1 | 3 | 5.3 | 4 | 5 | 0.4 | 105 | 8 | 1.4 | 4 | 4 | 1.4 | 4 | 4 | -0.0 | 3852 | 3
$radmatch^{probit}$ | 93.6 | 6 | 6.7 | 31 | 11 | 0.3 | 81 | 6 | 1.8 | 38 | 11 | 1.8 | 34 | 11 | -0.1 | 141524 | 8
$randforest$ | 93.4 | 7 | 5.2 | 1 | 2 | 0.2 | 20 | 4 | 1.4 | 7 | 6 | 1.4 | 3 | 4 | -0.1 | 96925 | 7
$reg^{kernel}$ | 93.1 | 7 | 5.3 | 4 | 5 | 0.4 | 143 | 9 | 1.5 | 15 | 6 | 1.6 | 15 | 6 | -0.2 | 193929 | 6
$pairmatch^{x}$ | 92.9 | 8 | 5.9 | 15 | 9 | 0.2 | 14 | 4 | 1.6 | 24 | 9 | 1.6 | 20 | 8 | -0.1 | 156350 | 9
$radmatchx^{cbps}$ | 92.6 | 8 | 6.0 | 17 | 9 | 0.3 | 93 | 6 | 1.6 | 25 | 9 | 1.7 | 23 | 8 | -0.1 | 147726 | 9
$radmatchx^{probit}$ | 92.3 | 9 | 6.0 | 17 | 10 | 0.3 | 83 | 5 | 1.7 | 27 | 10 | 1.7 | 25 | 9 | -0.1 | 172399 | 10
$means$ | 87.4 | 11 | 5.1 | 0 | 2 | 1.1 | 541 | 11 | 1.3 | 0 | 2 | 1.8 | 29 | 8 | -0.0 | 8467 | 3

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 13: Empirically observed instrument strength

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$radmatchx^{probit}$ | 95.9 | 6 | 525.1 | 26 | 8 | 2.0 | 93 | 6 | 17.8 | 41 | 9 | 17.9 | 40 | 9 | 6.9 | 100 | 8
$pairmatch^{x}$ | 96.2 | 7 | 525.4 | 26 | 8 | 1.3 | 27 | 4 | 18.1 | 44 | 9 | 18.1 | 42 | 8 | 11.8 | 243 | 9
$radmatchx^{cbps}$ | 96.3 | 6 | 563.1 | 35 | 8 | 1.9 | 80 | 6 | 15.7 | 25 | 8 | 15.8 | 23 | 7 | 8.3 | 141 | 7
$means$ | 93.5 | 10 | 463.9 | 11 | 3 | 5.7 | 441 | 11 | 14.4 | 15 | 4 | 15.8 | 23 | 7 | 11.3 | 228 | 5
$reg^{kernel}$ | 96.5 | 8 | 653.5 | 56 | 6 | 2.4 | 129 | 9 | 12.6 | 0 | 5 | 12.8 | 0 | 6 | 8.5 | 148 | 5
$radmatch^{probit}$ | 96.6 | 5 | 706.2 | 69 | 10 | 2.2 | 110 | 6 | 22.1 | 76 | 10 | 22.2 | 73 | 10 | 21.7 | 530 | 9
$randforest$ | 96.6 | 5 | 452.4 | 8 | 3 | 1.0 | 0 | 4 | 14.7 | 17 | 6 | 14.7 | 15 | 5 | 3.4 | 0 | 6
$ipw^{cbps}$ | 96.9 | 6 | 575.5 | 38 | 6 | 2.0 | 88 | 7 | 13.5 | 7 | 4 | 13.6 | 6 | 4 | 8.8 | 157 | 5
$reg$ | 97.2 | 4 | 418.3 | 0 | 4 | 1.2 | 13 | 4 | 13.2 | 5 | 4 | 13.3 | 4 | 3 | 8.1 | 135 | 3
$dr^{probit}$ | 97.3 | 5 | 509.6 | 22 | 5 | 1.4 | 37 | 5 | 13.9 | 11 | 4 | 14.0 | 9 | 4 | 7.9 | 128 | 4
$tsls$ | 97.3 | 4 | 516.4 | 23 | 4 | 1.2 | 17 | 4 | 14.5 | 15 | 4 | 14.5 | 14 | 4 | 7.5 | 119 | 5

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 14: Discrete outcome

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$radmatch^{probit}$ | 95.0 | 6 | 695.6 | 68 | 11 | 2.4 | 100 | 6 | 21.0 | 62 | 10 | 21.2 | 59 | 10 | 23.5 | 460 | 8
$randforest$ | 94.9 | 5 | 449.1 | 9 | 2 | 1.2 | 0 | 4 | 14.8 | 14 | 6 | 14.8 | 12 | 4 | 4.2 | 0 | 7
$ipw^{cbps}$ | 95.2 | 5 | 569.1 | 37 | 6 | 2.3 | 86 | 8 | 13.1 | 1 | 4 | 13.3 | 1 | 4 | 10.0 | 139 | 4
$reg^{kernel}$ | 94.5 | 7 | 646.1 | 56 | 5 | 2.8 | 127 | 9 | 13.0 | 0 | 6 | 13.3 | 0 | 6 | 9.0 | 115 | 7
$pairmatch^{x}$ | 94.4 | 8 | 519.0 | 25 | 8 | 1.5 | 23 | 4 | 18.5 | 43 | 9 | 18.6 | 40 | 8 | 12.2 | 190 | 9
$dr^{probit}$ | 95.9 | 4 | 504.0 | 22 | 5 | 1.6 | 33 | 5 | 14.2 | 9 | 4 | 14.3 | 7 | 3 | 8.4 | 101 | 4
$reg$ | 95.9 | 3 | 413.9 | 0 | 4 | 1.4 | 12 | 4 | 13.9 | 7 | 3 | 13.9 | 5 | 3 | 8.3 | 99 | 3
$radmatchx^{cbps}$ | 94.0 | 7 | 556.0 | 34 | 9 | 2.2 | 80 | 6 | 15.7 | 21 | 8 | 15.8 | 19 | 7 | 9.2 | 120 | 8
$tsls$ | 96.0 | 3 | 509.9 | 23 | 4 | 1.4 | 13 | 3 | 14.0 | 8 | 3 | 14.1 | 6 | 2 | 8.8 | 111 | 4
$radmatchx^{probit}$ | 93.8 | 8 | 518.3 | 25 | 9 | 2.3 | 89 | 6 | 17.5 | 35 | 9 | 17.7 | 33 | 9 | 8.1 | 93 | 9
$means$ | 87.6 | 10 | 460.2 | 11 | 3 | 6.7 | 452 | 11 | 15.2 | 17 | 3 | 16.9 | 28 | 9 | 11.3 | 170 | 5

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 15: Binary outcome

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$radmatch^{probit}$ | 95.1 | 5 | 17.3 | 106 | 11 | 0.1 | 188 | 6 | 2.9 | 420 | 10 | 2.9 | 406 | 10 | -1.9 | 23346 | 8
$randforest$ | 95.1 | 7 | 8.4 | 0 | 2 | 0.0 | 52 | 4 | 1.3 | 130 | 6 | 1.3 | 124 | 4 | -0.8 | 10024 | 7
$radmatchx^{cbps}$ | 94.8 | 7 | 13.1 | 55 | 8 | 0.0 | 24 | 6 | 1.6 | 192 | 8 | 1.6 | 184 | 8 | -1.1 | 12892 | 8
$reg^{kernel}$ | 95.2 | 7 | 12.7 | 50 | 5 | 0.1 | 152 | 10 | 1.1 | 97 | 5 | 1.1 | 92 | 6 | -0.6 | 7446 | 5
$pairmatch^{x}$ | 94.7 | 7 | 12.3 | 45 | 8 | 0.0 | 0 | 4 | 1.2 | 115 | 8 | 1.2 | 109 | 8 | -0.5 | 5728 | 9
$radmatchx^{probit}$ | 94.4 | 8 | 12.8 | 52 | 9 | 0.1 | 73 | 5 | 1.9 | 246 | 9 | 1.9 | 236 | 9 | -1.3 | 16419 | 10
$ipw^{cbps}$ | 95.8 | 4 | 11.7 | 38 | 5 | 0.1 | 155 | 8 | 1.7 | 207 | 5 | 1.7 | 200 | 4 | -1.2 | 14503 | 4
$tsls$ | 96.0 | 4 | 11.8 | 40 | 4 | 0.0 | 4 | 5 | 1.8 | 227 | 3 | 1.8 | 218 | 3 | -1.3 | 15757 | 4
$reg$ | 96.0 | 3 | 9.7 | 15 | 5 | 0.0 | 7 | 4 | 0.7 | 34 | 4 | 0.7 | 30 | 3 | -0.2 | 2870 | 3
$dr^{probit}$ | 96.1 | 3 | 11.0 | 30 | 6 | 0.0 | 40 | 5 | 1.1 | 97 | 5 | 1.1 | 92 | 4 | -0.6 | 7033 | 4
$means$ | 93.3 | 10 | 8.8 | 4 | 2 | 0.1 | 235 | 11 | 0.6 | 0 | 2 | 0.6 | 0 | 6 | -0.0 | 0 | 3

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 16: Small sample size

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$reg^{kernel}$ | 94.8 | 7 | 556.6 | 68 | 6 | 1.5 | 141 | 10 | 9.8 | 3 | 6 | 9.9 | 4 | 6 | 8.4 | 68 | 7
$radmatch^{probit}$ | 95.2 | 5 | 457.0 | 38 | 10 | 1.3 | 102 | 6 | 14.3 | 49 | 10 | 14.3 | 49 | 10 | 17.2 | 244 | 8
$randforest$ | 94.8 | 7 | 351.8 | 6 | 2 | 0.6 | 0 | 5 | 10.3 | 7 | 5 | 10.3 | 7 | 4 | 5.0 | 0 | 7
$radmatchx^{cbps}$ | 94.5 | 7 | 435.4 | 32 | 8 | 0.9 | 51 | 6 | 11.0 | 15 | 8 | 11.0 | 15 | 8 | 8.6 | 73 | 8
$pairmatch^{x}$ | 94.5 | 8 | 362.5 | 10 | 8 | 0.8 | 22 | 3 | 12.4 | 29 | 9 | 12.4 | 29 | 9 | 12.5 | 150 | 9
$ipw^{cbps}$ | 95.5 | 4 | 469.7 | 42 | 6 | 1.2 | 90 | 8 | 10.8 | 13 | 4 | 10.9 | 14 | 4 | 8.2 | 65 | 4
$radmatchx^{probit}$ | 94.3 | 7 | 390.2 | 18 | 8 | 1.1 | 70 | 5 | 12.4 | 29 | 9 | 12.4 | 30 | 9 | 7.9 | 59 | 9
$reg$ | 95.8 | 4 | 330.5 | -0 | 4 | 0.7 | 6 | 4 | 9.6 | 0 | 3 | 9.6 | 0 | 2 | 8.7 | 74 | 3
$dr^{probit}$ | 95.8 | 4 | 420.3 | 27 | 6 | 0.8 | 36 | 5 | 10.4 | 9 | 5 | 10.4 | 9 | 4 | 8.2 | 63 | 3
$tsls$ | 95.9 | 3 | 429.1 | 30 | 5 | 0.7 | 9 | 4 | 11.1 | 16 | 3 | 11.1 | 16 | 3 | 7.9 | 58 | 4
$means$ | 92.5 | 10 | 356.1 | 8 | 3 | 3.4 | 446 | 11 | 11.5 | 20 | 3 | 12.0 | 26 | 7 | 10.2 | 105 | 4

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

Table 17: Larger sample size

| cov | r | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$radmatch^{probit}$ | 94.9 | 6 | 255.9 | 176 | 11 | 1.3 | 98 | 7 | 9.7 | 141 | 10 | 9.8 | 134 | 10 | 4.3 | 27420 | 9
$reg^{kernel}$ | 94.9 | 7 | 102.3 | 10 | 5 | 1.3 | 109 | 9 | 4.2 | 6 | 6 | 4.5 | 7 | 6 | -0.0 | 0 | 5
$randforest$ | 95.3 | 6 | 105.8 | 14 | 3 | 0.6 | 0 | 3 | 5.8 | 44 | 6 | 5.8 | 39 | 5 | -1.6 | 10318 | 7
$pairmatch^{x}$ | 94.6 | 7 | 168.7 | 82 | 8 | 0.8 | 21 | 4 | 7.4 | 84 | 9 | 7.4 | 77 | 8 | -0.8 | 5196 | 9
$ipw^{cbps}$ | 95.5 | 5 | 111.1 | 20 | 5 | 1.2 | 81 | 7 | 4.0 | 0 | 4 | 4.2 | 0 | 4 | 0.6 | 3665 | 4
$radmatchx^{cbps}$ | 94.3 | 7 | 133.7 | 44 | 9 | 1.3 | 102 | 6 | 6.3 | 57 | 9 | 6.4 | 54 | 8 | -0.5 | 2893 | 8
$tsls$ | 96.1 | 4 | 92.6 | 0 | 3 | 0.7 | 13 | 4 | 4.7 | 18 | 3 | 4.8 | 15 | 2 | -0.4 | 2354 | 4
$radmatchx^{probit}$ | 93.9 | 8 | 140.9 | 52 | 9 | 1.3 | 102 | 6 | 7.1 | 77 | 10 | 7.2 | 72 | 9 | -1.2 | 7523 | 10
$reg$ | 96.2 | 3 | 93.1 | 1 | 4 | 0.7 | 14 | 4 | 5.0 | 25 | 4 | 5.1 | 21 | 3 | -0.6 | 3883 | 3
$dr^{probit}$ | 96.2 | 4 | 94.7 | 2 | 5 | 0.8 | 28 | 5 | 4.9 | 21 | 4 | 4.9 | 18 | 3 | -0.3 | 1993 | 4
$means$ | 88.4 | 10 | 112.9 | 22 | 3 | 3.4 | 435 | 11 | 4.3 | 6 | 2 | 5.5 | 31 | 8 | 1.0 | 6554 | 4

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the average rank.

###  A.3 Local average treatment effects for each DGP

The following Tables 18-49 provide the performance of the LATE estimators
separately by each DGP. The estimators are sorted according to their
performance in terms of coverage rates of 95% confidence intervals.

Table 18: DGP 1

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$pairmatch^{probit}$ | 94.6 | 10.8 | 24 | 18 | 0.3 | 257 | 10 | 2.8 | 16 | 16 | 2.8 | 16 | 16 | -0.0 | 0 | 1 | 2000 | 2000
$pairmatch^{cbps}$ | 94.3 | 10.8 | 24 | 19 | 0.3 | 319 | 14 | 2.8 | 18 | 18 | 2.8 | 18 | 18 | -0.1 | 306 | 2 | 2000 | 2000
$radmatch^{cbps}$ | 93.8 | 10.3 | 18 | 15 | 0.3 | 323 | 15 | 2.8 | 16 | 17 | 2.8 | 16 | 17 | -0.1 | 973 | 12 | 2000 | 2000
$ipw^{cbps}$ | 93.8 | 9.2 | 6 | 7 | 0.4 | 384 | 16 | 2.4 | 1 | 8 | 2.4 | 2 | 6 | -0.1 | 426 | 7 | 2000 | 2000
$reg$ | 93.7 | 9.1 | 5 | 3 | 0.3 | 254 | 9 | 2.4 | 0 | 2 | 2.4 | 0 | 2 | -0.1 | 397 | 5 | 2000 | 2000
$ipw^{probit}$ | 93.7 | 9.1 | 5 | 6 | 0.3 | 242 | 7 | 2.4 | 0 | 5 | 2.4 | 0 | 4 | -0.1 | 426 | 8 | 2000 | 2000
$dr^{probit}$ | 93.7 | 9.1 | 5 | 5 | 0.3 | 263 | 11 | 2.4 | 0 | 4 | 2.4 | 0 | 5 | -0.1 | 428 | 9 | 2000 | 2000
$tsls$ | 93.6 | 9.1 | 5 | 2 | 0.3 | 237 | 5 | 2.4 | 0 | 1 | 2.4 | 0 | 1 | -0.1 | 385 | 4 | 2000 | 2000
$radmatch^{probit}$ | 93.6 | 10.3 | 18 | 14 | 0.3 | 244 | 8 | 2.7 | 15 | 15 | 2.7 | 15 | 14 | -0.1 | 820 | 10 | 2000 | 2000
$dr^{cbps}$ | 93.6 | 9.1 | 5 | 4 | 0.3 | 264 | 12 | 2.4 | 0 | 3 | 2.4 | 0 | 3 | -0.1 | 420 | 6 | 2000 | 2000
$dr^{lc}$ | 96.8 | 18.5 | 113 | 21 | 0.5 | 546 | 18 | 2.4 | 1 | 9 | 2.5 | 3 | 8 | 0.3 | 1964 | 17 | 2000 | 1955
$ipw^{lc}$ | 96.8 | 14.8 | 70 | 20 | 0.8 | 931 | 20 | 2.4 | 1 | 6 | 2.5 | 6 | 10 | 0.3 | 2089 | 18 | 2000 | 1993
$radmatchx^{cbps}$ | 92.7 | 9.7 | 12 | 11 | 0.2 | 120 | 3 | 2.7 | 12 | 12 | 2.7 | 11 | 11 | -0.2 | 1329 | 13 | 2000 | 2000
$reg^{kernel}$ | 92.7 | 9.2 | 6 | 9 | 0.6 | 607 | 19 | 2.5 | 4 | 11 | 2.5 | 6 | 9 | -0.1 | 828 | 11 | 2000 | 1968
$randforest$ | 92.3 | 8.7 | 0 | 1 | 0.3 | 240 | 6 | 2.4 | 2 | 10 | 2.5 | 2 | 7 | -0.2 | 1587 | 16 | 2000 | 2000
$radmatchx^{probit}$ | 92.3 | 9.7 | 12 | 10 | 0.1 | 69 | 2 | 2.7 | 13 | 13 | 2.7 | 12 | 12 | -0.2 | 1535 | 15 | 2000 | 2000
$pairmatch^{x}$ | 92.3 | 9.8 | 12 | 12 | 0.2 | 177 | 4 | 2.7 | 14 | 14 | 2.7 | 13 | 13 | -0.2 | 1513 | 14 | 2000 | 2000
$means$ | 90.2 | 9.2 | 6 | 8 | 1.4 | 1633 | 21 | 2.4 | 1 | 7 | 2.8 | 16 | 15 | -0.1 | 307 | 3 | 2000 | 2000
$pairmatch^{lc}$ | 80.2 | 10.3 | 19 | 16 | 0.3 | 312 | 13 | 4.5 | 91 | 19 | 4.5 | 90 | 19 | -1.9 | 14427 | 19 | 2000 | 2000
$radmatchx^{lc}$ | 75.4 | 9.9 | 13 | 13 | 0.1 | 0 | 1 | 4.7 | 99 | 20 | 4.7 | 98 | 20 | -2.2 | 16864 | 20 | 2000 | 2000
$radmatch^{lc}$ | 73.6 | 10.4 | 19 | 17 | 0.4 | 406 | 17 | 5.2 | 119 | 21 | 5.2 | 118 | 21 | -2.6 | 19571 | 21 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 19: DGP 2

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$dr^{probit}$ | 94.8 | 15.8 | 4 | 7 | 0.3 | 171 | 11 | 4.0 | 1 | 6 | 4.0 | 1 | 5 | 0.1 | 96 | 9 | 2000 | 2000
$ipw^{probit}$ | 94.8 | 15.7 | 4 | 5 | 0.3 | 144 | 9 | 4.0 | 1 | 3 | 4.0 | 0 | 2 | 0.0 | 81 | 4 | 2000 | 2000
$dr^{cbps}$ | 94.8 | 15.7 | 4 | 6 | 0.3 | 175 | 13 | 4.0 | 1 | 5 | 4.0 | 1 | 4 | 0.0 | 86 | 5 | 2000 | 2000
$reg$ | 94.8 | 15.7 | 4 | 4 | 0.3 | 165 | 10 | 4.0 | 1 | 4 | 4.0 | 0 | 3 | 0.0 | 87 | 6 | 2000 | 2000
$tsls$ | 94.8 | 15.7 | 4 | 3 | 0.3 | 143 | 8 | 3.9 | 0 | 2 | 3.9 | 0 | 1 | 0.0 | 61 | 2 | 2000 | 2000
$ipw^{cbps}$ | 94.7 | 15.8 | 4 | 8 | 0.5 | 310 | 17 | 4.0 | 1 | 7 | 4.0 | 1 | 6 | 0.1 | 94 | 8 | 2000 | 2000
$radmatch^{cbps}$ | 94.6 | 17.7 | 17 | 17 | 0.4 | 249 | 15 | 4.5 | 16 | 14 | 4.6 | 16 | 15 | -0.0 | 80 | 3 | 2000 | 2000
$pairmatch^{probit}$ | 95.6 | 18.6 | 23 | 19 | 0.2 | 35 | 2 | 4.6 | 18 | 17 | 4.6 | 17 | 16 | 0.1 | 249 | 11 | 2000 | 2000
$reg^{kernel}$ | 94.4 | 15.9 | 5 | 9 | 0.9 | 585 | 19 | 4.1 | 4 | 11 | 4.2 | 6 | 10 | -0.0 | 88 | 7 | 2000 | 1970
$pairmatch^{cbps}$ | 95.7 | 18.6 | 23 | 18 | 0.5 | 266 | 16 | 4.6 | 18 | 16 | 4.6 | 17 | 18 | 0.1 | 290 | 12 | 2000 | 2000
$radmatchx^{cbps}$ | 94.2 | 16.8 | 11 | 11 | 0.3 | 130 | 7 | 4.5 | 14 | 12 | 4.5 | 13 | 12 | -0.2 | 596 | 13 | 2000 | 2000
$radmatch^{probit}$ | 94.2 | 17.7 | 17 | 16 | 0.2 | 61 | 3 | 4.6 | 16 | 15 | 4.6 | 15 | 14 | -0.1 | 181 | 10 | 2000 | 2000
$pairmatch^{x}$ | 93.8 | 17.1 | 13 | 13 | 0.3 | 128 | 6 | 4.6 | 18 | 18 | 4.6 | 17 | 17 | -0.3 | 1045 | 16 | 2000 | 2000
$randforest$ | 93.7 | 15.1 | 0 | 1 | 0.4 | 227 | 14 | 4.1 | 4 | 10 | 4.1 | 4 | 8 | -0.2 | 791 | 15 | 2000 | 2000
$radmatchx^{probit}$ | 93.6 | 16.8 | 11 | 12 | 0.1 | 0 | 1 | 4.5 | 14 | 13 | 4.5 | 14 | 13 | -0.2 | 736 | 14 | 2000 | 2000
$ipw^{lc}$ | 96.6 | 21.8 | 44 | 20 | 1.2 | 879 | 20 | 4.0 | 1 | 8 | 4.1 | 5 | 9 | 0.6 | 2101 | 17 | 2000 | 1989
$dr^{lc}$ | 96.9 | 31.7 | 110 | 21 | 0.8 | 502 | 18 | 4.0 | 3 | 9 | 4.1 | 4 | 7 | 0.6 | 2193 | 18 | 2000 | 1954
$means$ | 91.6 | 15.5 | 3 | 2 | 2.1 | 1604 | 21 | 3.9 | 0 | 1 | 4.5 | 13 | 11 | 0.0 | 0 | 1 | 2000 | 2000
$pairmatch^{lc}$ | 79.2 | 17.4 | 15 | 14 | 0.3 | 110 | 4 | 8.4 | 114 | 20 | 8.4 | 112 | 20 | -4.0 | 14979 | 19 | 2000 | 2000
$radmatchx^{lc}$ | 75.3 | 16.8 | 11 | 10 | 0.3 | 126 | 5 | 8.2 | 110 | 19 | 8.2 | 109 | 19 | -4.0 | 15086 | 20 | 2000 | 2000
$radmatch^{lc}$ | 74.2 | 17.5 | 16 | 15 | 0.3 | 174 | 12 | 9.2 | 136 | 21 | 9.3 | 134 | 21 | -4.8 | 18271 | 21 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 20: DGP 3

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$tsls$ | 95.0 | 9.5 | 4 | 5 | 0.1 | 0 | 1 | 2.5 | 4 | 4 | 2.5 | 0 | 1 | -0.1 | 17 | 7 | 2000 | 2000
$dr^{cbps}$ | 94.4 | 9.8 | 7 | 8 | 0.2 | 189 | 3 | 2.5 | 6 | 8 | 2.6 | 2 | 5 | -0.1 | 2 | 2 | 2000 | 2000
$ipw^{probit}$ | 94.3 | 9.6 | 5 | 6 | 0.4 | 437 | 9 | 2.5 | 4 | 5 | 2.5 | 1 | 3 | -0.1 | 6 | 4 | 2000 | 2000
$dr^{probit}$ | 94.2 | 9.8 | 7 | 9 | 0.2 | 189 | 4 | 2.6 | 6 | 9 | 2.6 | 2 | 6 | -0.1 | 0 | 1 | 2000 | 2000
$reg$ | 94.1 | 9.7 | 6 | 7 | 0.1 | 73 | 2 | 2.5 | 5 | 6 | 2.5 | 1 | 2 | -0.1 | 10 | 5 | 2000 | 2000
$pairmatch^{probit}$ | 94.0 | 12.6 | 38 | 19 | 0.6 | 605 | 12 | 3.3 | 38 | 15 | 3.4 | 34 | 15 | -0.1 | 78 | 8 | 2000 | 2000
$ipw^{cbps}$ | 93.8 | 9.5 | 4 | 4 | 0.6 | 662 | 14 | 2.5 | 4 | 3 | 2.6 | 3 | 7 | -0.1 | 13 | 6 | 2000 | 2000
$randforest$ | 92.7 | 9.1 | 0 | 1 | 0.3 | 237 | 7 | 2.5 | 5 | 7 | 2.5 | 2 | 4 | -0.2 | 244 | 10 | 2000 | 2000
$pairmatch^{cbps}$ | 92.7 | 12.5 | 37 | 18 | 0.6 | 605 | 11 | 3.3 | 38 | 16 | 3.4 | 35 | 16 | -0.2 | 153 | 9 | 2000 | 2000
$radmatch^{probit}$ | 92.6 | 12.4 | 36 | 17 | 0.6 | 686 | 15 | 3.4 | 42 | 18 | 3.5 | 39 | 18 | -0.3 | 353 | 12 | 2000 | 2000
$radmatch^{cbps}$ | 92.0 | 12.3 | 35 | 16 | 0.6 | 632 | 13 | 3.4 | 40 | 17 | 3.4 | 37 | 17 | -0.3 | 331 | 11 | 2000 | 2000
$radmatchx^{probit}$ | 91.8 | 10.8 | 18 | 15 | 0.6 | 697 | 17 | 3.0 | 26 | 14 | 3.1 | 24 | 13 | -0.3 | 387 | 14 | 2000 | 2000
$reg^{kernel}$ | 91.8 | 9.5 | 4 | 3 | 0.7 | 767 | 19 | 2.9 | 22 | 12 | 3.0 | 21 | 11 | -0.5 | 765 | 16 | 2000 | 1866
$pairmatch^{x}$ | 91.6 | 10.5 | 15 | 13 | 0.3 | 221 | 6 | 2.9 | 22 | 11 | 3.0 | 18 | 10 | -0.3 | 368 | 13 | 2000 | 2000
$radmatchx^{cbps}$ | 91.5 | 10.7 | 18 | 14 | 0.7 | 740 | 18 | 3.0 | 25 | 13 | 3.1 | 23 | 12 | -0.3 | 393 | 15 | 2000 | 2000
$ipw^{lc}$ | 99.9 | 69.9 | 665 | 20 | 1.0 | 1202 | 20 | 2.5 | 3 | 2 | 2.7 | 7 | 8 | 6.4 | 10272 | 20 | 2000 | 1644
$dr^{lc}$ | 100.0 | 405.2 | 4337 | 21 | 0.6 | 690 | 16 | 2.6 | 9 | 10 | 2.7 | 8 | 9 | 36.0 | 58554 | 21 | 2000 | 1264
$means$ | 85.6 | 9.2 | 1 | 2 | 2.1 | 2531 | 21 | 2.4 | 0 | 1 | 3.2 | 27 | 14 | -0.1 | 4 | 3 | 2000 | 2000
$pairmatch^{lc}$ | 66.8 | 10.4 | 14 | 11 | 0.2 | 207 | 5 | 6.1 | 155 | 20 | 6.1 | 146 | 20 | -3.5 | 5582 | 18 | 2000 | 2000
$radmatchx^{lc}$ | 65.5 | 10.0 | 9 | 10 | 0.3 | 263 | 8 | 5.9 | 146 | 19 | 5.9 | 137 | 19 | -3.4 | 5406 | 17 | 2000 | 2000
$radmatch^{lc}$ | 60.0 | 10.4 | 14 | 12 | 0.4 | 451 | 10 | 6.6 | 173 | 21 | 6.6 | 164 | 21 | -3.9 | 6296 | 19 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 21: DGP 4

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$radmatch^{lc}$ | 95.4 | 4500.7 | 94 | 20 | 3.2 | 155 | 5 | 119.8 | 108 | 20 | 119.8 | 107 | 20 | 104.1 | 265 | 14 | 1970 | 1968
$pairmatch^{lc}$ | 96.3 | 3517.3 | 52 | 11 | 1.3 | 0 | 1 | 107.7 | 87 | 19 | 107.7 | 86 | 19 | 120.2 | 321 | 16 | 1973 | 1962
$pairmatch^{probit}$ | 99.7 | 5040.8 | 118 | 21 | 5.2 | 312 | 8 | 88.4 | 53 | 17 | 88.5 | 53 | 17 | 128.9 | 352 | 18 | 1980 | 1978
$randforest$ | 99.8 | 2478.2 | 7 | 3 | 3.0 | 138 | 3 | 67.1 | 16 | 6 | 67.2 | 16 | 6 | 41.5 | 46 | 2 | 1992 | 1974
$radmatchx^{probit}$ | 99.8 | 3787.2 | 64 | 13 | 3.0 | 136 | 2 | 78.8 | 37 | 12 | 78.9 | 37 | 12 | 76.5 | 168 | 5 | 1987 | 1984
$ipw^{cbps}$ | 99.8 | 3934.4 | 70 | 15 | 7.6 | 501 | 16 | 74.4 | 29 | 10 | 74.7 | 29 | 10 | 82.1 | 188 | 9 | 1991 | 1990
$means$ | 99.8 | 2314.1 | 0 | 1 | 16.7 | 1221 | 21 | 80.5 | 40 | 13 | 82.2 | 42 | 14 | 83.0 | 191 | 11 | 1985 | 1975
$radmatch^{cbps}$ | 99.8 | 3696.9 | 60 | 12 | 6.2 | 390 | 12 | 85.1 | 48 | 16 | 85.3 | 48 | 16 | 124.8 | 338 | 17 | 1987 | 1977
$reg$ | 99.8 | 2441.8 | 6 | 2 | 5.3 | 319 | 9 | 60.6 | 5 | 2 | 60.8 | 5 | 2 | 77.2 | 171 | 6 | 1995 | 1978
$radmatch^{probit}$ | 99.8 | 3225.3 | 39 | 8 | 8.4 | 563 | 17 | 88.8 | 54 | 18 | 89.1 | 54 | 18 | 109.9 | 286 | 15 | 1990 | 1979
$radmatchx^{cbps}$ | 99.8 | 3971.1 | 72 | 18 | 3.1 | 143 | 4 | 66.3 | 15 | 5 | 66.3 | 15 | 4 | 82.3 | 189 | 10 | 1985 | 1981
$dr^{cbps}$ | 99.8 | 3002.7 | 30 | 5 | 4.7 | 271 | 7 | 57.6 | 0 | 1 | 57.8 | 0 | 1 | 88.5 | 210 | 12 | 1991 | 1983
$ipw^{lc}$ | 99.8 | 3947.6 | 71 | 16 | 11.4 | 804 | 20 | 70.8 | 23 | 8 | 71.7 | 24 | 8 | 195.7 | 586 | 21 | 1995 | 1985
$tsls$ | 99.8 | 3150.0 | 36 | 6 | 6.1 | 382 | 11 | 67.5 | 17 | 7 | 67.8 | 17 | 7 | 80.4 | 182 | 7 | 1993 | 1986
$ipw^{probit}$ | 99.8 | 3952.7 | 71 | 17 | 4.2 | 234 | 6 | 72.7 | 26 | 9 | 72.9 | 26 | 9 | 73.4 | 157 | 3 | 1993 | 1992
$dr^{probit}$ | 99.8 | 3873.1 | 67 | 14 | 5.6 | 342 | 10 | 65.8 | 14 | 3 | 66.0 | 14 | 3 | 74.1 | 160 | 4 | 1993 | 1993
$reg^{kernel}$ | 99.9 | 3313.5 | 43 | 9 | 9.9 | 688 | 19 | 66.1 | 15 | 4 | 66.8 | 16 | 5 | 81.8 | 187 | 8 | 1994 | 1959
$pairmatch^{x}$ | 99.9 | 2775.0 | 20 | 4 | 6.8 | 438 | 15 | 80.5 | 40 | 14 | 80.8 | 40 | 13 | 91.3 | 220 | 13 | 1990 | 1978
$pairmatch^{cbps}$ | 99.9 | 4201.6 | 82 | 19 | 6.6 | 423 | 14 | 84.0 | 46 | 15 | 84.2 | 46 | 15 | 153.8 | 439 | 20 | 1980 | 1975
$dr^{lc}$ | 100.0 | 3203.3 | 38 | 7 | 8.5 | 571 | 18 | 75.5 | 31 | 11 | 76.0 | 31 | 11 | 149.5 | 424 | 19 | 1995 | 1955
$radmatchx^{lc}$ | 89.7 | 3488.8 | 51 | 10 | 6.2 | 395 | 13 | 127.0 | 121 | 21 | 127.2 | 120 | 21 | -28.5 | 0 | 1 | 1975 | 1973

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 22: DGP 5

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$tsls$ | 94.7 | 14.4 | 11 | 4 | 0.7 | 37 | 2 | 3.6 | 11 | 3 | 3.7 | 0 | 1 | 0.0 | 928 | 7 | 2000 | 2000
$reg$ | 94.3 | 14.8 | 15 | 10 | 0.8 | 59 | 3 | 3.7 | 15 | 7 | 3.8 | 4 | 3 | 0.0 | 345 | 5 | 2000 | 2000
$dr^{probit}$ | 93.8 | 15.1 | 17 | 12 | 0.8 | 75 | 4 | 3.8 | 17 | 9 | 3.9 | 6 | 6 | 0.0 | 199 | 3 | 2000 | 2000
$pairmatch^{probit}$ | 93.8 | 19.7 | 53 | 19 | 1.2 | 158 | 12 | 5.1 | 57 | 16 | 5.2 | 43 | 16 | -0.2 | 5624 | 10 | 2000 | 2000
$dr^{cbps}$ | 93.8 | 15.0 | 17 | 11 | 0.8 | 78 | 6 | 3.8 | 17 | 8 | 3.9 | 6 | 5 | 0.0 | 221 | 4 | 2000 | 2000
$ipw^{probit}$ | 93.7 | 14.6 | 13 | 8 | 1.1 | 130 | 8 | 3.7 | 13 | 6 | 3.9 | 5 | 4 | 0.0 | 0 | 1 | 2000 | 2000
$radmatch^{probit}$ | 93.6 | 19.3 | 50 | 17 | 1.3 | 170 | 13 | 5.2 | 60 | 18 | 5.4 | 45 | 18 | -0.4 | 13333 | 12 | 2000 | 2000
$pairmatch^{cbps}$ | 93.5 | 19.4 | 51 | 18 | 1.2 | 148 | 10 | 5.0 | 54 | 15 | 5.1 | 40 | 15 | -0.1 | 4095 | 8 | 2000 | 2000
$randforest$ | 93.3 | 13.9 | 8 | 2 | 0.9 | 94 | 7 | 3.7 | 13 | 5 | 3.8 | 3 | 2 | -0.1 | 5373 | 9 | 2000 | 2000
$radmatch^{cbps}$ | 92.6 | 19.0 | 48 | 16 | 1.2 | 152 | 11 | 5.1 | 57 | 17 | 5.3 | 43 | 17 | -0.4 | 13262 | 11 | 2000 | 2000
$ipw^{cbps}$ | 92.4 | 14.4 | 12 | 5 | 1.4 | 185 | 14 | 3.7 | 12 | 4 | 3.9 | 6 | 7 | -0.0 | 22 | 2 | 2000 | 2000
$pairmatch^{x}$ | 92.3 | 16.5 | 28 | 13 | 0.8 | 77 | 5 | 4.6 | 40 | 12 | 4.6 | 26 | 11 | -0.4 | 14139 | 13 | 2000 | 2000
$reg^{kernel}$ | 91.8 | 14.2 | 10 | 3 | 1.4 | 196 | 16 | 4.1 | 26 | 11 | 4.3 | 18 | 10 | -0.5 | 18965 | 16 | 2000 | 1871
$radmatchx^{probit}$ | 90.3 | 16.7 | 30 | 15 | 1.5 | 209 | 17 | 4.7 | 44 | 14 | 4.9 | 34 | 14 | -0.5 | 17905 | 15 | 2000 | 2000
$ipw^{lc}$ | 99.8 | 126.3 | 880 | 20 | 2.0 | 312 | 20 | 3.5 | 7 | 2 | 4.0 | 9 | 8 | 9.6 | 359462 | 20 | 2000 | 1589
$radmatchx^{cbps}$ | 90.1 | 16.6 | 29 | 14 | 1.5 | 223 | 19 | 4.6 | 42 | 13 | 4.9 | 32 | 13 | -0.4 | 15508 | 14 | 2000 | 2000
$dr^{lc}$ | 100.0 | 696.7 | 5306 | 21 | 1.4 | 193 | 15 | 3.8 | 18 | 10 | 4.1 | 11 | 9 | 148.5 | 5558002 | 21 | 2000 | 1362
$means$ | 83.6 | 12.9 | 0 | 1 | 3.3 | 594 | 21 | 3.3 | 0 | 1 | 4.7 | 27 | 12 | 0.0 | 824 | 6 | 2000 | 2000
$pairmatch^{lc}$ | 65.6 | 14.5 | 13 | 7 | 1.1 | 130 | 9 | 9.9 | 205 | 20 | 10.0 | 172 | 20 | -6.2 | 233179 | 18 | 2000 | 2000
$radmatchx^{lc}$ | 64.8 | 14.4 | 12 | 6 | 0.5 | 0 | 1 | 8.9 | 174 | 19 | 8.9 | 143 | 19 | -5.3 | 196777 | 17 | 2000 | 2000
$radmatch^{lc}$ | 59.2 | 14.6 | 13 | 9 | 1.5 | 218 | 18 | 10.0 | 206 | 21 | 10.1 | 174 | 21 | -6.3 | 234489 | 19 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 23: DGP 6

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$ipw^{cbps}$ | 94.8 | 8.1 | 5 | 7 | 0.3 | 259 | 17 | 2.7 | 1 | 6 | 2.1 | 1 | 6 | -0.6 | 77 | 6 | 2000 | 2000
$pairmatch^{cbps}$ | 95.3 | 9.5 | 23 | 19 | 0.2 | 224 | 16 | 3.0 | 10 | 15 | 2.4 | 18 | 18 | -0.5 | 47 | 3 | 2000 | 2000
$tsls$ | 94.7 | 8.0 | 3 | 2 | 0.2 | 123 | 9 | 2.7 | 0 | 1 | 2.0 | 0 | 1 | -0.6 | 77 | 7 | 2000 | 2000
$dr^{probit}$ | 94.7 | 8.0 | 4 | 5 | 0.2 | 132 | 11 | 2.7 | 0 | 4 | 2.0 | 0 | 3 | -0.7 | 78 | 10 | 2000 | 2000
$reg$ | 94.7 | 8.0 | 4 | 3 | 0.2 | 133 | 12 | 2.7 | 0 | 2 | 2.0 | 0 | 2 | -0.7 | 77 | 8 | 2000 | 2000
$ipw^{probit}$ | 94.6 | 8.0 | 4 | 6 | 0.2 | 112 | 7 | 2.7 | 0 | 5 | 2.0 | 0 | 5 | -0.7 | 77 | 9 | 2000 | 2000
$dr^{cbps}$ | 94.6 | 8.0 | 4 | 4 | 0.2 | 137 | 13 | 2.7 | 0 | 3 | 2.0 | 0 | 4 | -0.7 | 79 | 11 | 2000 | 2000
$radmatch^{cbps}$ | 94.4 | 9.1 | 17 | 15 | 0.2 | 215 | 15 | 2.9 | 9 | 14 | 2.4 | 16 | 14 | -0.6 | 70 | 5 | 2000 | 2000
$pairmatch^{probit}$ | 95.6 | 9.5 | 23 | 18 | 0.1 | 24 | 3 | 3.0 | 12 | 18 | 2.4 | 17 | 17 | -0.6 | 58 | 4 | 2000 | 2000
$radmatch^{probit}$ | 94.0 | 9.0 | 17 | 14 | 0.1 | 51 | 4 | 3.0 | 10 | 17 | 2.3 | 16 | 13 | -0.7 | 81 | 12 | 2000 | 2000
$radmatchx^{cbps}$ | 93.9 | 8.6 | 11 | 11 | 0.2 | 117 | 8 | 2.9 | 8 | 12 | 2.3 | 13 | 11 | -0.7 | 95 | 14 | 2000 | 2000
$reg^{kernel}$ | 93.8 | 8.4 | 8 | 9 | 0.4 | 458 | 19 | 2.8 | 4 | 11 | 2.2 | 6 | 9 | -0.7 | 96 | 15 | 2000 | 1973
$pairmatch^{x}$ | 93.8 | 8.7 | 12 | 12 | 0.2 | 107 | 6 | 3.0 | 10 | 16 | 2.4 | 17 | 16 | -0.8 | 107 | 17 | 2000 | 2000
$ipw^{lc}$ | 96.3 | 12.1 | 56 | 20 | 0.7 | 767 | 20 | 2.7 | 1 | 7 | 2.2 | 6 | 10 | -0.4 | -0 | 1 | 2000 | 1991
$randforest$ | 93.6 | 7.7 | 0 | 1 | 0.2 | 201 | 14 | 2.7 | 2 | 10 | 2.1 | 4 | 8 | -0.8 | 110 | 18 | 2000 | 2000
$dr^{lc}$ | 96.7 | 15.8 | 104 | 21 | 0.4 | 407 | 18 | 2.7 | 2 | 9 | 2.1 | 4 | 7 | -0.4 | 5 | 2 | 2000 | 1959
$radmatchx^{probit}$ | 93.0 | 8.6 | 11 | 10 | 0.1 | 0 | 1 | 2.9 | 8 | 13 | 2.3 | 13 | 12 | -0.7 | 98 | 16 | 2000 | 2000
$means$ | 91.0 | 8.1 | 5 | 8 | 1.2 | 1415 | 21 | 2.7 | 1 | 8 | 2.4 | 16 | 15 | -0.7 | 82 | 13 | 2000 | 2000
$pairmatch^{lc}$ | 80.7 | 9.1 | 17 | 16 | 0.1 | 16 | 2 | 4.5 | 68 | 19 | 4.1 | 104 | 19 | -2.2 | 500 | 19 | 2000 | 2000
$radmatchx^{lc}$ | 76.0 | 8.7 | 12 | 13 | 0.2 | 124 | 10 | 4.6 | 71 | 20 | 4.2 | 108 | 20 | -2.4 | 555 | 20 | 2000 | 2000
$radmatch^{lc}$ | 74.8 | 9.1 | 18 | 17 | 0.2 | 100 | 5 | 5.0 | 87 | 21 | 4.7 | 131 | 21 | -2.7 | 637 | 21 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 24: DGP 7

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$pairmatch^{lc}$ | 94.1 | 4598.4 | 146 | 15 | 2.4 | 2656 | 3 | 131.0 | 156 | 19 | 131.0 | 155 | 19 | 109.2 | 165 | 14 | 1974 | 1971
$radmatch^{lc}$ | 93.6 | 5664.4 | 203 | 20 | 1.5 | 1602 | 2 | 132.0 | 158 | 20 | 132.0 | 157 | 20 | 106.1 | 157 | 13 | 1962 | 1962
$pairmatch^{probit}$ | 99.7 | 4874.0 | 161 | 16 | 10.6 | 12005 | 19 | 95.7 | 87 | 17 | 96.3 | 87 | 17 | 172.5 | 318 | 18 | 1982 | 1978
$radmatchx^{probit}$ | 99.7 | 2225.9 | 19 | 2 | 10.2 | 11453 | 18 | 71.2 | 39 | 11 | 71.9 | 40 | 11 | 72.8 | 76 | 8 | 1991 | 1984
$radmatch^{cbps}$ | 99.8 | 5302.2 | 184 | 17 | 7.3 | 8222 | 13 | 99.3 | 94 | 18 | 99.6 | 94 | 18 | 170.7 | 314 | 17 | 1978 | 1976
$radmatch^{probit}$ | 99.8 | 3838.6 | 106 | 13 | 8.0 | 9041 | 14 | 95.6 | 87 | 16 | 95.9 | 87 | 16 | 178.5 | 333 | 19 | 1986 | 1981
$tsls$ | 99.8 | 3493.4 | 87 | 12 | 2.8 | 3060 | 4 | 66.2 | 29 | 7 | 66.2 | 29 | 6 | 67.4 | 63 | 5 | 1994 | 1993
$randforest$ | 99.8 | 2989.8 | 60 | 8 | 4.2 | 4656 | 7 | 72.8 | 42 | 12 | 72.9 | 42 | 12 | 41.2 | 0 | 1 | 1995 | 1994
$dr^{probit}$ | 99.8 | 2641.5 | 42 | 4 | 5.3 | 5910 | 10 | 67.6 | 32 | 9 | 67.8 | 32 | 9 | 66.8 | 62 | 4 | 1990 | 1972
$dr^{cbps}$ | 99.8 | 2421.6 | 30 | 3 | 4.4 | 4874 | 8 | 66.1 | 29 | 5 | 66.2 | 29 | 5 | 72.0 | 75 | 7 | 1989 | 1980
$pairmatch^{x}$ | 99.8 | 2815.7 | 51 | 7 | 3.0 | 3358 | 6 | 83.9 | 64 | 14 | 83.9 | 63 | 14 | 115.2 | 179 | 15 | 1989 | 1980
$radmatchx^{cbps}$ | 99.8 | 2754.3 | 48 | 6 | 8.1 | 9155 | 15 | 66.7 | 30 | 8 | 67.2 | 31 | 7 | 73.7 | 79 | 9 | 1991 | 1983
$reg^{kernel}$ | 99.9 | 5359.0 | 187 | 18 | 8.7 | 9747 | 17 | 58.7 | 15 | 3 | 59.3 | 16 | 3 | 62.5 | 52 | 2 | 1991 | 1892
$pairmatch^{cbps}$ | 99.9 | 4426.6 | 137 | 14 | 5.6 | 6262 | 11 | 93.3 | 82 | 15 | 93.5 | 82 | 15 | 159.7 | 287 | 16 | 1981 | 1979
$reg$ | 99.9 | 2657.9 | 42 | 5 | 2.9 | 3228 | 5 | 65.0 | 27 | 4 | 65.1 | 27 | 4 | 66.5 | 61 | 3 | 1992 | 1987
$ipw^{lc}$ | 99.9 | 5380.6 | 188 | 19 | 13.7 | 15458 | 20 | 66.1 | 29 | 6 | 67.4 | 31 | 8 | 463.4 | 1024 | 20 | 1994 | 1980
$ipw^{probit}$ | 100.0 | 3144.4 | 68 | 9 | 4.7 | 5269 | 9 | 51.2 | 0 | 1 | 51.4 | -0 | 1 | 76.2 | 85 | 10 | 1991 | 1990
$ipw^{cbps}$ | 100.0 | 3364.2 | 80 | 11 | 6.6 | 7404 | 12 | 56.7 | 11 | 2 | 57.1 | 11 | 2 | 68.6 | 66 | 6 | 1991 | 1990
$dr^{lc}$ | 100.0 | 8846.7 | 374 | 21 | 8.3 | 9347 | 16 | 70.0 | 37 | 10 | 70.5 | 37 | 10 | 513.6 | 1146 | 21 | 1994 | 1889
$means$ | 100.0 | 3212.4 | 72 | 10 | 24.8 | 28138 | 21 | 78.6 | 54 | 13 | 82.4 | 60 | 13 | 82.0 | 99 | 11 | 1991 | 1984
$radmatchx^{lc}$ | 79.2 | 1866.5 | 0 | 1 | 0.1 | 0 | 1 | 138.9 | 172 | 21 | 138.9 | 170 | 21 | -95.0 | 130 | 12 | 1968 | 1967

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 25: DGP 8

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$tsls$ | 95.0 | 8.5 | 5 | 5 | 0.2 | 1 | 2 | 2.2 | 5 | 3 | 2.2 | 0 | 1 | 0.0 | 969 | 6 | 2000 | 2000
$reg$ | 94.4 | 8.7 | 6 | 7 | 0.3 | 26 | 3 | 2.2 | 7 | 6 | 2.2 | 2 | 2 | -0.0 | 0 | 1 | 2000 | 2000
$dr^{probit}$ | 94.4 | 8.8 | 7 | 9 | 0.4 | 55 | 4 | 2.2 | 8 | 9 | 2.2 | 3 | 6 | 0.0 | 624 | 4 | 2000 | 2000
$dr^{cbps}$ | 94.3 | 8.7 | 7 | 8 | 0.4 | 56 | 5 | 2.2 | 8 | 8 | 2.2 | 3 | 4 | 0.0 | 282 | 2 | 2000 | 2000
$ipw^{probit}$ | 94.0 | 8.6 | 5 | 6 | 0.5 | 123 | 9 | 2.2 | 6 | 5 | 2.2 | 3 | 3 | 0.0 | 691 | 5 | 2000 | 2000
$pairmatch^{cbps}$ | 94.0 | 11.2 | 38 | 18 | 0.7 | 193 | 10 | 2.9 | 41 | 15 | 3.0 | 37 | 15 | -0.1 | 6688 | 9 | 2000 | 2000
$pairmatch^{probit}$ | 93.6 | 11.3 | 39 | 19 | 0.7 | 194 | 11 | 2.9 | 42 | 17 | 3.0 | 38 | 16 | -0.1 | 6507 | 8 | 2000 | 2000
$randforest$ | 93.3 | 8.3 | 2 | 2 | 0.4 | 86 | 7 | 2.2 | 7 | 7 | 2.2 | 3 | 5 | -0.1 | 8662 | 10 | 2000 | 2000
$ipw^{cbps}$ | 93.2 | 8.5 | 5 | 4 | 0.7 | 196 | 12 | 2.2 | 5 | 4 | 2.3 | 5 | 7 | 0.0 | 327 | 3 | 2000 | 2000
$radmatch^{cbps}$ | 93.0 | 11.0 | 35 | 16 | 0.7 | 204 | 14 | 2.9 | 42 | 16 | 3.0 | 38 | 17 | -0.2 | 14189 | 12 | 2000 | 2000
$radmatch^{probit}$ | 92.8 | 11.1 | 37 | 17 | 0.7 | 212 | 16 | 3.0 | 45 | 18 | 3.1 | 41 | 18 | -0.2 | 17701 | 14 | 2000 | 2000
$reg^{kernel}$ | 92.7 | 8.5 | 5 | 3 | 0.7 | 208 | 15 | 2.3 | 10 | 11 | 2.4 | 9 | 10 | -0.1 | 10309 | 11 | 1999 | 1880
$pairmatch^{x}$ | 92.2 | 9.5 | 16 | 13 | 0.4 | 58 | 6 | 2.6 | 27 | 13 | 2.6 | 22 | 11 | -0.2 | 20258 | 15 | 2000 | 2000
$radmatchx^{cbps}$ | 91.6 | 9.6 | 18 | 14 | 0.8 | 243 | 19 | 2.6 | 27 | 12 | 2.7 | 25 | 12 | -0.2 | 16330 | 13 | 2000 | 2000
$radmatchx^{probit}$ | 91.3 | 9.6 | 18 | 15 | 0.8 | 229 | 18 | 2.6 | 29 | 14 | 2.8 | 27 | 13 | -0.2 | 20345 | 16 | 2000 | 2000
$ipw^{lc}$ | 99.9 | 57.0 | 599 | 20 | 1.1 | 351 | 20 | 2.1 | 3 | 2 | 2.4 | 9 | 9 | 5.4 | 497997 | 20 | 2000 | 1648
$dr^{lc}$ | 100.0 | 350.1 | 4193 | 21 | 0.7 | 196 | 13 | 2.2 | 9 | 10 | 2.3 | 8 | 8 | 32.6 | 3026625 | 21 | 1999 | 1276
$means$ | 84.4 | 8.2 | 0 | 1 | 2.0 | 729 | 21 | 2.1 | 0 | 1 | 2.9 | 31 | 14 | 0.0 | 1802 | 7 | 2000 | 2000
$pairmatch^{lc}$ | 68.2 | 9.2 | 13 | 11 | 0.5 | 120 | 8 | 5.6 | 171 | 20 | 5.6 | 157 | 20 | -3.2 | 297620 | 18 | 2000 | 2000
$radmatchx^{lc}$ | 64.8 | 8.8 | 8 | 10 | 0.2 | 0 | 1 | 5.2 | 155 | 19 | 5.2 | 141 | 19 | -3.0 | 276921 | 17 | 2000 | 2000
$radmatch^{lc}$ | 62.9 | 9.2 | 13 | 12 | 0.8 | 217 | 17 | 5.9 | 189 | 21 | 6.0 | 175 | 21 | -3.6 | 332536 | 19 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 26: DGP 9

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$dr^{probit}$ | 95.0 | 0.2 | 4 | 6 | 0.0 | 199 | 9 | 0.1 | 0 | 4 | 0.1 | 0 | 4 | -0.0 | 57 | 7 | 2000 | 2000
$ipw^{cbps}$ | 95.0 | 0.2 | 4 | 8 | 0.0 | 325 | 14 | 0.1 | 0 | 8 | 0.1 | 1 | 6 | -0.0 | 9 | 2 | 2000 | 2000
$dr^{cbps}$ | 95.0 | 0.2 | 4 | 3 | 0.0 | 199 | 10 | 0.1 | 0 | 3 | 0.1 | 0 | 3 | -0.0 | 61 | 9 | 2000 | 2000
$pairmatch^{cbps}$ | 95.0 | 0.2 | 22 | 18 | 0.0 | 327 | 15 | 0.1 | 18 | 18 | 0.1 | 18 | 18 | -0.0 | 51 | 5 | 2000 | 2000
$radmatch^{cbps}$ | 95.0 | 0.2 | 17 | 16 | 0.0 | 316 | 13 | 0.1 | 16 | 16 | 0.1 | 16 | 16 | -0.0 | 534 | 12 | 2000 | 2000
$reg$ | 95.0 | 0.2 | 4 | 2 | 0.0 | 189 | 6 | 0.1 | 0 | 1 | 0.1 | 0 | 1 | -0.0 | 42 | 4 | 2000 | 2000
$pairmatch^{probit}$ | 95.1 | 0.2 | 23 | 19 | 0.0 | 194 | 7 | 0.1 | 16 | 17 | 0.1 | 16 | 17 | 0.0 | 0 | 1 | 2000 | 2000
$ipw^{probit}$ | 94.9 | 0.2 | 4 | 7 | 0.0 | 215 | 12 | 0.1 | 0 | 6 | 0.1 | 0 | 5 | -0.0 | 56 | 6 | 2000 | 2000
$tsls$ | 94.8 | 0.2 | 4 | 4 | 0.0 | 206 | 11 | 0.1 | 0 | 2 | 0.1 | 0 | 2 | -0.0 | 33 | 3 | 2000 | 2000
$reg^{kernel}$ | 94.5 | 0.2 | 5 | 9 | 0.0 | 654 | 19 | 0.1 | 2 | 11 | 0.1 | 3 | 10 | -0.0 | 91 | 10 | 2000 | 2000
$radmatch^{probit}$ | 94.0 | 0.2 | 17 | 15 | 0.0 | 198 | 8 | 0.1 | 15 | 15 | 0.1 | 14 | 15 | -0.0 | 367 | 11 | 2000 | 2000
$pairmatch^{x}$ | 93.8 | 0.2 | 12 | 13 | 0.0 | 103 | 4 | 0.1 | 13 | 14 | 0.1 | 13 | 14 | -0.0 | 909 | 16 | 2000 | 2000
$radmatchx^{cbps}$ | 93.7 | 0.2 | 11 | 10 | 0.0 | 44 | 3 | 0.1 | 11 | 12 | 0.1 | 11 | 12 | -0.0 | 822 | 13 | 2000 | 2000
$means$ | 93.6 | 0.2 | 4 | 5 | 0.0 | 1321 | 21 | 0.1 | 0 | 5 | 0.1 | 6 | 11 | -0.0 | 60 | 8 | 2000 | 2000
$radmatchx^{probit}$ | 93.5 | 0.2 | 11 | 11 | 0.0 | 0 | 1 | 0.1 | 12 | 13 | 0.1 | 12 | 13 | -0.0 | 874 | 14 | 2000 | 2000
$randforest$ | 93.4 | 0.2 | 0 | 1 | 0.0 | 178 | 5 | 0.1 | 2 | 10 | 0.1 | 2 | 7 | -0.0 | 884 | 15 | 2000 | 2000
$ipw^{lc}$ | 97.4 | 0.3 | 60 | 20 | 0.0 | 804 | 20 | 0.1 | 0 | 7 | 0.1 | 3 | 9 | 0.0 | 1915 | 18 | 2000 | 1992
$dr^{lc}$ | 97.6 | 0.4 | 91 | 21 | 0.0 | 531 | 17 | 0.1 | 1 | 9 | 0.1 | 2 | 8 | 0.0 | 1815 | 17 | 2000 | 1984
$pairmatch^{lc}$ | 78.4 | 0.2 | 17 | 14 | 0.0 | 481 | 16 | 0.1 | 101 | 19 | 0.1 | 101 | 19 | -0.0 | 13476 | 19 | 2000 | 2000
$radmatchx^{lc}$ | 74.1 | 0.2 | 12 | 12 | 0.0 | 10 | 2 | 0.1 | 106 | 20 | 0.1 | 106 | 20 | -0.1 | 15054 | 20 | 2000 | 2000
$radmatch^{lc}$ | 71.7 | 0.2 | 17 | 17 | 0.0 | 582 | 18 | 0.1 | 133 | 21 | 0.1 | 133 | 21 | -0.1 | 18336 | 21 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 27: DGP 10

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$pairmatch^{probit}$ | 95.5 | 0.5 | 22 | 19 | 0.0 | 72 | 3 | 0.1 | 18 | 16 | 0.1 | 16 | 17 | -0.0 | 0 | 1 | 2000 | 2000
$pairmatch^{cbps}$ | 94.5 | 0.5 | 22 | 18 | 0.0 | 315 | 13 | 0.1 | 18 | 18 | 0.1 | 17 | 18 | -0.0 | 322 | 2 | 2000 | 2000
$dr^{probit}$ | 94.4 | 0.4 | 4 | 8 | 0.0 | 262 | 8 | 0.1 | 2 | 7 | 0.1 | 0 | 5 | -0.0 | 1222 | 3 | 2000 | 2000
$dr^{cbps}$ | 94.4 | 0.4 | 4 | 7 | 0.0 | 265 | 9 | 0.1 | 2 | 6 | 0.1 | 0 | 4 | -0.0 | 1329 | 6 | 2000 | 2000
$tsls$ | 94.4 | 0.4 | 3 | 3 | 0.0 | 289 | 11 | 0.1 | 1 | 2 | 0.1 | 0 | 1 | -0.0 | 1656 | 8 | 2000 | 2000
$radmatch^{probit}$ | 94.3 | 0.5 | 16 | 16 | 0.0 | 128 | 4 | 0.1 | 16 | 14 | 0.1 | 14 | 14 | -0.0 | 4754 | 11 | 2000 | 2000
$reg$ | 94.3 | 0.4 | 4 | 6 | 0.0 | 244 | 7 | 0.1 | 2 | 5 | 0.1 | 0 | 3 | -0.0 | 1224 | 4 | 2000 | 2000
$ipw^{probit}$ | 94.2 | 0.4 | 3 | 4 | 0.0 | 285 | 10 | 0.1 | 2 | 4 | 0.1 | 0 | 2 | -0.0 | 1306 | 5 | 2000 | 2000
$ipw^{cbps}$ | 94.1 | 0.4 | 4 | 5 | 0.0 | 477 | 15 | 0.1 | 2 | 8 | 0.1 | 1 | 6 | -0.0 | 1621 | 7 | 2000 | 2000
$radmatch^{cbps}$ | 94.1 | 0.5 | 16 | 17 | 0.0 | 310 | 12 | 0.1 | 16 | 15 | 0.1 | 15 | 15 | -0.0 | 5478 | 12 | 2000 | 2000
$reg^{kernel}$ | 94.0 | 0.4 | 5 | 9 | 0.0 | 1014 | 19 | 0.1 | 4 | 10 | 0.1 | 4 | 10 | -0.0 | 2700 | 10 | 2000 | 2000
$ipw^{lc}$ | 96.2 | 0.5 | 36 | 20 | 0.0 | 1250 | 20 | 0.1 | 1 | 3 | 0.1 | 2 | 8 | 0.0 | 16720 | 17 | 2000 | 1989
$radmatchx^{cbps}$ | 93.6 | 0.4 | 10 | 11 | 0.0 | 177 | 5 | 0.1 | 14 | 12 | 0.1 | 12 | 12 | -0.0 | 9528 | 13 | 2000 | 2000
$dr^{lc}$ | 96.5 | 0.7 | 80 | 21 | 0.0 | 835 | 18 | 0.1 | 2 | 9 | 0.1 | 2 | 7 | 0.0 | 19120 | 18 | 2000 | 1979
$radmatchx^{probit}$ | 93.3 | 0.4 | 11 | 12 | 0.0 | 0 | 1 | 0.1 | 14 | 13 | 0.1 | 12 | 13 | -0.0 | 9795 | 14 | 2000 | 2000
$means$ | 93.2 | 0.4 | 1 | 2 | 0.0 | 1983 | 21 | 0.1 | 0 | 1 | 0.1 | 4 | 11 | -0.0 | 2423 | 9 | 2000 | 2000
$pairmatch^{x}$ | 92.8 | 0.4 | 12 | 13 | 0.0 | 190 | 6 | 0.1 | 18 | 17 | 0.1 | 16 | 16 | -0.0 | 13635 | 16 | 2000 | 2000
$randforest$ | 92.6 | 0.4 | 0 | 1 | 0.0 | 353 | 14 | 0.1 | 5 | 11 | 0.1 | 3 | 9 | -0.0 | 11295 | 15 | 2000 | 2000
$pairmatch^{lc}$ | 78.6 | 0.5 | 14 | 14 | 0.0 | 564 | 16 | 0.2 | 107 | 20 | 0.2 | 104 | 20 | -0.1 | 144307 | 19 | 2000 | 2000
$radmatchx^{lc}$ | 75.0 | 0.4 | 9 | 10 | 0.0 | 3 | 2 | 0.2 | 105 | 19 | 0.2 | 102 | 19 | -0.1 | 147467 | 20 | 2000 | 2000
$radmatch^{lc}$ | 71.9 | 0.5 | 14 | 15 | 0.0 | 715 | 17 | 0.2 | 131 | 21 | 0.2 | 128 | 21 | -0.1 | 179991 | 21 | 2000 | 2000

  * •

Notes: ‘cov’ and ‘int’ refer to coverage rate and length of confidence
interval, ‘bias’ denotes the absolute bias from the true treatment effect,
‘sd’ is the standard deviation of the estimator, ‘rmse’ stands for root mean
squared error, and ‘seb’ indicates the median bias of the estimated bootstrap
standard error. ‘diff’ indicates the difference to the left best performer in
percent (%) and ‘r’ is the rank. ‘nsimp’ shows the number of simulations with
point estimations smaller than $|10^{10}|$ and ‘nsimse’ denotes the number of
simulations used to compute ‘cov’, ‘int’, and ‘seb’. Conditional on ‘nsimp’,
simulations with standard errors greater than 150 times the (true) standard
error are not considered (i.e.simulations with non-converging bootstrap
standard errors are dropped).

Table 28: DGP 11

| cov | int | d | r | bias | d | r | sd | d | r | rmse | d | r | seb | d | r | nsimp | nsimse
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$dr^{probit}$ | 95.0 | 0.2 | 7 | 9 | 0.0 | 133 | 3 | 0.1 | 6 | 10 | 0.1 | 1 | 8 | -0.0 | 8 | 2 | 2000 | 2000
$ipw^{cbps}$ | 94.9 | 0.2 | 5 | 4 | 0.0 | 1556 | 15 | 0.1 | 4 | 3 | 0.1 | 0 | 1 | -0.0 | 12 | 4 | 2000 | 2000
$ipw^{probit}$ | 95.2 | 0.2 | 6 | 7 | 0.0 | 667 | 6 | 0.1 | 5 | 5 | 0.1 | 0 | 3 | -0.0 | 0 | 1 | 2000 | 2000
$pairmatch^{cbps}$ | 94.8 | 0.3 | 36 | 18 | 0.0 | 816 | 9 | 0.1 | 37 | 15 | 0.1 | 31 | 15 | -0.0 | 163 | 9 | 2000 | 2000
$dr^{cbps}$ | 95.2 | 0.2 | 7 | 8 | 0.0 | 97 | 2 | 0.1 | 6 | 8 | 0.1 | 1 | 6 | -0.0 | 12 | 3 | 2000 | 2000
$reg$ | 94.8 | 0.2 | 5 | 6 | 0.0 | 596 | 5 | 0.1 | 5 | 6 | 0.1 | 1 | 4 | -0.0 | 43 | 7 | 2000 | 2000
$pairmatch^{probit}$ | 94.7 | 0.3 | 37 | 19 | 0.0 | 896 | 11 | 0.1 | 37 | 16 | 0.1 | 31 | 16 | -0.0 | 65 | 8 | 2000 | 2000