# Generative invariance: causal extrapolation without exogeneity

Carlos García<EMAIL_ADDRESS>[ David Ríos
<EMAIL_ADDRESS>[ Instituto de Ciencias Matemáticaspresep=, ]e1

###### Abstract

We present a new estimator for predicting outcomes in different distributional
settings under hidden confounding without relying on instruments or exogenous
variables. The population definition of our estimator identifies causal
parameters, whose empirical version is plugged into a generative model capable
of replicating the conditional law within a test environment. We check that
the probabilistic affinity between our proposal and test distributions is
invariant across interventions. This work enhances the current statistical
comprehension of causality by demonstrating that predictions in a test
environment can be made without the need for exogenous variables and without
specific assumptions regarding the strength of perturbations or the overlap of
distributions.

62A,

62J,

###### keywords:

[class=MSC]

,

##  1 Introduction

In exploring the underlying mechanisms of real random variables $X$ and $Y$
following a joint distribution $P$, we consider the following structural
causal model (SCM):

$X\leftarrow ZV+H\alpha_{*}+\varepsilon^{\prime}_{X}\textrm{ and }Y\leftarrow
X\beta_{*}+H\gamma_{*}+\varepsilon^{\prime}_{Y}$

where $Z$, $\varepsilon^{\prime}_{X}$, $\varepsilon^{\prime}_{Y}$, $H$, and
$V$ are mutually independent random variables. In this model, all variables
except $Z$, which takes binary values 0 and 1 representing training and
testing phases, are Gaussian-distributed. The error term
$\varepsilon^{\prime}_{Y}$ has a mean of zero. The graphical structure of the
model is depicted next.

$X$$Z$$Y$$H$

We face the problem of providing predictions for covariate values
$X_{1},\ldots,X_{n}$ drawn after setting $Z=1$, based on observations
$(X_{1},Y_{1}),\ldots(X_{n},Y_{n})$ we obtained upon having fixed $Z=0$.

The previous set of assumptions is sufficient for making the classical
instrumental variable (IV) assumptions hold [16, 1], with $Z$ acting as an
instrument. The IV method, a cornerstone in countering hidden confounding
within observational studies, hinges on the instrument $Z$ being orthogonal to
the residuals of the response $Y$ after adjusting for the covariate $X$
through the causal parameter $\beta_{*}$. This orthogonality criterion
underpins the IV method’s fundamental premise. Under the three IV assumptions,
our objective encompasses minimizing the risk below with respect to
$\beta\in\mathbb{R}$.

$\mathbb{E}\left[\mathbb{E}^{2}\left[Y-X\beta\mid Z\right]\right]$

Conditional expectation can be understood as a projection in an $L^{2}$ space
of random variables, where it represents the best approximation of a random
variable given the conditioning one in terms of expected squared distance.
This minimization process constitutes the extent to which the conditioning
variable explains the variance of the other, capturing the part that can be
predicted or explained by $Z$. The minimization problem above looks for the
$\beta$ that hinders $Z$ the most when trying to explain the variability of
the residuals. The technical nuance that the square is taken outside the inner
expectation elucidates the rationale behind the feasibility of consistently
estimating the causal parameter. This subtlety is pivotal from a formal
viewpoint.

Given the increasing interest in utilizing exogenous variables for domain
generalization [9, 2, 10, 3, 5, 11, 14, 12, 7, 6], there is a notable drive
towards marrying robustness with causal inference principles. This ambition is
reflected in the efforts to generalize findings from observational settings to
uncharted interventional distributions.

Drawing inspiration from the rich landscape of extrapolation in statistical
analysis- see e.g. [13, 14]\- we propose a novel estimator that identifies the
parameters indexing a generator class that can replicate the conditional law
of test environments with an arbitrarily shifted covariate distribution.

The important novelty of our approach is that its performance does not depend
on the strength of perturbations and, importantly, it does not need to use
exogenous variables. This marks a significant departure from conventional
causal inference strategies, which have historically hinged on the collection
of data across varied observational contexts to exploit the constancy of
conditional distributions of response variables within a precise causal
framework [8]. Our approach, therefore, presents a paradigm shift in
statistical thinking regarding causal inference, moving away from traditional
reliance on data diversity to ensure estimation viability.

Let $(X^{z},Y^{z})$ denote the random variable following the conditional
distribution of $(X,Y)$ given intervention $Z=z\in\\{0,1\\}$. Denote these
distributions by $P^{0}$ and $P^{1}$ respectively. This allows to detach our
initial SCM into a pair of SCMs representing the observational and test
distributions respectively

$\displaystyle P^{0}:$ $\displaystyle X^{0}$
$\displaystyle\leftarrow\varepsilon_{X},$ $\displaystyle P^{1}:$
$\displaystyle X^{1}$ $\displaystyle\leftarrow\varepsilon_{X}+ZV,$
$\displaystyle Y^{0}$ $\displaystyle\leftarrow
X^{0}\beta_{*}+\varepsilon_{Y},$ $\displaystyle Y^{1}$
$\displaystyle\leftarrow X^{1}\beta_{*}+\varepsilon_{Y}.$

where $\varepsilon_{X}:=H\alpha_{*}+\varepsilon^{\prime}_{X}$ and
$\varepsilon_{Y}:=H\gamma_{*}+\varepsilon^{\prime}_{Y}$. It is important to
note that the newly defined residuals are not uncorrelated anymore:
$\operatorname{Cov}(\varepsilon_{X},\varepsilon_{Y})=\operatorname{Cov}(H\alpha_{*}+\varepsilon^{\prime}_{X},H\gamma_{*}+\varepsilon^{\prime}_{Y})=\gamma_{*}\alpha_{*}\operatorname{Var}H=:K^{*}$,
accounting for hidden confounding. Notice that the assumptions at the
beginning imply that there are no interventions on the response or on the
residuals.

Figure 1: A sample from $P^{0}$, the training environment, is depicted in
blue. Next, we sample from 5 different test environments $P^{1}$ which are
colored in black. We fit an ordinary linear model to the blue data and launch
red predictions. Notice how the more different the test covariates are with
respect to the training set, the worse OLS performs. Our estimator can
reproduce the probabilistic structure of the test environments by looking at
the blue data and just at the covariate values in each test sample.

The fundamental problem here is that we just observe
$(X^{1}_{1},\texttt{NA}),\ldots,(X^{1}_{n},\texttt{NA})$ from $P^{1}$.
According to IV theory, using the test data as a second environment would
suffice to identify the causal parameter. However, we do not have access to
the values of $Y^{1}$ in a sample generated by $P^{1}$.

We will see that a positive sample variance of the test covariate distribution
fundamentally enables the generation of samples that convincingly appear to
originate from $P^{1}$, bridging the gap between theoretical assumptions and
practical applications. This perspective acknowledges the common scenario in
applied fields, where predictions are unusually made for a solitary individual
in a novel domain, thereby enhancing our methodology’s applicability and
relevance.

The utilization of the causal parameter for predictive purposes, while
invaluable in understanding underlying mechanisms, may not always be the most
effective strategy, especially under conditions of minimal perturbation. We
exemplify this with the scenario where train and test distributions are the
same. Although the ordinary least squares (OLS) estimator using $P^{0}$ is
biased because of being the assumption $\varepsilon_{Y}\perp X^{0}$ violated,
by definition we have that:

$\mathbb{E}\left[\left(Y^{0}-X^{0}\beta_{*}\right)^{2}\right]\geq\mathbb{E}\left[\left(Y^{0}-X^{0}\beta_{\operatorname{OLS}}\right)^{2}\right]$

where

$\beta_{\operatorname{OLS}}:=\arg\\!\min_{\beta}\mathbb{E}\left[(Y^{0}-X^{0}\beta)^{2}\right]=\frac{\operatorname{Cov}(X^{0},Y^{0})}{\operatorname{Var}X^{0}}\stackrel{{\scriptstyle
K_{*}=0}}{{=}}\beta_{*}$

Therefore, there exist situations where using the causal parameter leads to
suboptimal predictive performance. Nevertheless, OLS produces an arbitrarily
high mean square error when the test distribution involves a strong departure
from the training one [11].

##  2 Population generative invariance

Assume $\operatorname{Var}X^{1}>0$. Let $\mathcal{K}$ be the real interval
centered around zero with length
$\sqrt{\operatorname{Var}X^{1}\operatorname{Var}\varepsilon_{Y}}$. Also define
for each $K\in\mathcal{K}$

$g_{K}:(x,\xi)\in\mathbb{R}^{2}\longmapsto(x-\mathbb{E}X^{1})\frac{K}{\operatorname{Var}X^{1}}+\xi\sqrt{\operatorname{Var}\varepsilon_{Y}-\frac{K^{2}}{\operatorname{Var}X^{1}}}$

We show that evaluations of these functions on $X^{1}$ and noise
$\xi\sim\mathcal{N}(0,1)$ are intimately connected to the joint probabilistic
structure of the covariate and the response residuals at test phase.

######  Lemma 2.1.

For $K\in\mathcal{K}$, $g_{K}(X^{1},\xi)$ is a Gaussian random variable such
that

  1. 1.

$\mathbb{E}g_{K}(X^{1},\xi)=0$

  2. 2.

$\operatorname{Var}g_{K}(X^{1},\xi)=\operatorname{Var}\varepsilon_{Y}$

  3. 3.

$\mathbb{E}(X^{1}-\mathbb{E}X^{1})g(X^{1},\xi)=K$

As a consequence, by setting $K=K_{*}$ we can recover a copy of
$(X^{1},Y^{1})$

######  Corollary 2.1.

$(X^{1},g_{K_{*}}(X^{1},\xi)+X^{1}\beta_{*})\sim P^{1}$

Let us set up the model

$\mathcal{M}=\\{m_{\theta}:(x,\xi)\in\mathbb{R}^{2}\longmapsto
g_{K}(x,\xi)+x\beta\colon\theta=(\beta,K)\in\Theta:=\mathbb{R}\times\mathcal{K}\\}$

Functions in $\mathcal{M}$ ought to be regarded as generative models. Each
element in the function class is a function $m_{\theta}$ that takes covariates
$x$ and noise $\xi$, is indexed by a tuple $\theta=(\beta,K)\in\Theta$ and
induces a conditional distribution: for $\theta\in\Theta$, draw
$\xi\sim\mathcal{N}(0,1)$ and consider the conditional distribution of
$m_{\theta}(X^{1},\xi)$ given $X^{1}=x$. In this sense, our extrapolation
procedure involves a generative model consisting of two phases: first sample
from the distribution of $X^{1}$ and then sample $Y^{1}$ from the conditional
distribution given $X_{1}$ induced by the generator.

######  Remark 2.1.

The do-interventional distribution of $Y^{1}$ under do$(X^{1}=x)$ is the same
as that of $m_{(\beta_{*},0)}(x,\xi)$, $\xi\sim\mathcal{N}(0,1)$.

In addressing the estimation of model parameters, we posit the determination
of these true values as a theoretical risk minimization problem. This problem,
intriguingly, possesses a solution that is both unique and attainable with
minimal assumptions.

$\beta^{opt},K^{opt}:=\arg\\!\min_{(\beta,K)\in\Theta}\mathbb{E}\left[(Y^{0}-X^{0}\beta-(X^{0}-\mathbb{E}X^{0})K)^{2}\right]$

Expanding the square inside of the expectation and setting derivatives to zero
we arrive at the following result.

######  Lemma 2.2.

Whenever they exist and are unique, $\beta^{opt},K^{opt}$ need to satisfy that

${\operatorname{Var}X^{0}}K^{opt}={(\beta_{*}-\beta_{opt})\operatorname{Var}X^{0}+K_{*}}$

and

$(\operatorname{Var}X^{0}+\mathbb{E}^{2}X^{0})(\beta_{*}-\beta_{opt})={K_{opt}\operatorname{Var}X^{0}-K_{*}}$

######  Corollary 2.2.

If $\mathbb{E}X^{0}\neq 0$ and $\operatorname{Var}X^{0}\neq 0$ then
$\beta^{opt},K^{opt}$ exist and are unique. Furthermore,

$\beta^{opt}=\beta_{*}\textrm{ and }K^{opt}\operatorname{Var}X^{0}=K_{*}$

######  Remark 2.2.

If there was access to an exogenous variable or anchor $Z$ pointing to more
than one environment such that we have access to samples of both covariates
and responses, we would proceed by linearly regressing out $Z$ from $Y$ and
$X$ and considering the residuals resulting from these two regressions. This
is equivalent to adjusting for the effect of the exogenous variable, sometimes
referred to as partialling out. Assuming that an observation $(X^{z},Y^{z})$
comes from environment $z\in\\{1,\ldots,N\\}$ with equal probability, we have
that
$\mathbb{E}X=\frac{1}{\operatorname{|ImZ|}}\sum_{z=1}^{\operatorname{|ImZ|}}\mathbb{E}X^{z}$.
The theoretical risk to minimize would therefore be

$\mathbb{E}\left[\left[(1-P_{Z})(Y-X\beta-(X-\mathbb{E}X)K)\right]^{2}\right]$

##  3 Empirical generative invariance

We advance our analysis by replacing $P^{0}$ with its empirical measure in the
integral of the theoretical risk’s expectation, leveraging a sample from the
training environment $(X^{0},Y^{0}),\ldots,(X^{0}_{n},Y^{0}_{n})\sim P^{0}$.
Let $\widehat{\mu}=\frac{1}{n}\sum_{i=1}^{n}X_{i}.$

Let our estimators be defined according to

$\widehat{\beta},\widehat{K}:=\arg\\!\min_{\beta,K}\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-X_{i}\beta-(X_{i}-\widehat{\mu})K)^{2}=\frac{1}{n}\sum_{i=1}^{n}\lVert
Y-X\beta-(X-\widehat{\mu})K\rVert^{2}$

######  Lemma 3.1 (Normal equations of empirical generative invariance).

Let $X=(X^{1},\ldots,X_{n})\in\mathbb{R}^{n\times 1}$ ,
$Y=(Y^{1},\ldots,Y_{n})\in\mathbb{R}^{n\times 1}$ and
$1\widehat{\mu}\in\mathbb{R}^{n\times 1}$. Then $\widehat{\beta},\widehat{K}$
must satisfy

$X^{T}X\widehat{\beta}+X^{T}(X-1\widehat{\mu})\widehat{K}=X^{T}Y$

$X^{T}(X-1\widehat{\mu})\widehat{\beta}+(X-1\widehat{\mu})^{T}(X-1\widehat{\mu})\widehat{K}=(X-1\widehat{\mu})^{T}Y$

######  Corollary 3.1.

$\widehat{\beta}$ and $\widehat{K}$ exist and are unique if $\widehat{\mu}\neq
0$ and $\frac{1}{n}X^{T}X-\widehat{\mu}^{2}\neq 0$

######  Remark 3.1.

If $\operatorname{Var}X^{0}=0$, this is translated in that $X^{0}=x$ for a
deterministic value $x$, what constitutes a hard intervention. We cannot
recover the do-interventional distribution (involving $\beta_{*}$) in this
case.

######  Remark 3.2.

If $\operatorname{Var}X^{1}=0$, we can still estimate $\beta^{*}$ and $K^{*}$
but the best we can do in terms of prediction is to multiply $\widehat{\beta}$
by the only covariate observation we have. Sampling from the covariate
distribution in the test environment is what opens access to creating
artificially correlated residuals and therefore optimal predictions. This is
the reason why many methodology needs some sort of “causal regularization”
when making predictions over a singleton $x$ and they proceed by modifying the
proposed estimated value for the slope.

##  4 Experiments

First, we need to clarify how point estimates of the OLS parameter and the
causal parameter $\beta^{*}$ lead to generators.

  * •

Empirical OLS induces a generative model defined by

$(x,\xi)\longmapsto
x\widehat{\beta}_{\operatorname{OLS}}+\widehat{\sigma}^{2}\xi$

where $\widehat{\sigma}^{2}$ is empirical squared residual standard error

  * •

Empirical causal parameter induces another generative model defined by

$(x,\xi)\longmapsto x\widehat{\beta}+\widehat{\sigma_{*}}^{2}\xi$

where $\widehat{\sigma_{*}}^{2}=\frac{1}{n}\sum_{i=1}^{n}\lVert
Y-X\widehat{\beta}-(X-\widehat{\mu})\widehat{K}\rVert^{2}$ is the minimum
empirical risk.

We consider a grid of possible second-order moments for $V$. For each value
$s$ in the grid, we sample the variance of $V$ from runif(0,s). Then we get
the mean by subtracting this value from $s$. Then we generate one test dataset
using this distribution for the perturbations. We hide the response values and
produce predictions using our proposed generator besides the two defined
above. Finally, we compute three energy scores: one corresponding to comparing
the original test data with the sample synthesized by each generator. We do
this for 50 times for each $s$ and average the energy scores. The results of
this experiment are visible in Figure 2.

###  4.1 Energy statistics

For the simulations, we need a measure of similarity between
$(X^{1}_{1},Y^{1}_{1})\ldots,(X^{1}_{n},Y^{1}_{n})$ and an artificial sample.
Energy distance between the distribution of two random variables $Z$ and $W$
taking values in the same metric space, $\mathcal{E}(Z,W)$, is a statistical
measure used to determine the similarity between two distributions [15].
Originating from the concept of Newton’s gravitational potential energy,
energy statistics leverage distances between data points in metric spaces. The
energy of a dataset is akin to gravitational potential energy, relying on
distances to infer statistical properties.

As a consequence of Proposition 3 in [15], for $Z,W\in\mathbb{R}^{2}$ and
$\delta$ the Euclidean distance therein, we have that

$\mathbb{E}(Z,W)=2\mathbb{E}\delta(Z,W)-\mathcal{E}\delta\left(Z,Z^{\prime}\right)-\mathbb{E}\delta\left(W,W^{\prime}\right)=0$

holds if and only if $Z$ and $W$ are identically distributed. Tilde stands for
an iid copy. For us, $Z=(X^{1},Y^{1})$ and $W=(X^{1},\widetilde{Y}^{1})$,
where $\widetilde{Y}^{1}$ follows the distribution of $m_{\theta}(X^{1},\xi)$
conditional to $X^{1}$ for some $\theta\in\Theta$.

Suppose that $Z_{1},\ldots,Z_{n_{1}}$ and $W_{1},\ldots,W_{n_{2}}$ are
independent random samples from the distributions of $Z$ and $W$,
respectively. The two-sample energy statistic corresponding to the energy
distance $\mathcal{E}(Z,W)$ is

$\displaystyle\widehat{\mathcal{E}}_{n_{1},n_{2}}=$
$\displaystyle\frac{2}{n_{1}n_{2}}\sum_{i=1}^{n_{1}}\sum_{m=1}^{n_{2}}\left|Z_{i}-W_{m}\right|$
$\displaystyle-\frac{1}{n_{1}^{2}}\sum_{i=1}^{n_{1}}\sum_{j=1}^{n_{1}}\left|Z_{i}-Z_{j}\right|-\frac{1}{n_{2}^{2}}\sum_{\ell=1}^{n_{2}}\sum_{m=1}^{n_{2}}\left|W_{\ell}-W_{m}\right|.$

For us $n_{1}=n_{2}$ because both samples from $Z$ and $W$ involve the same
test covariate observations.

Figure 2: We fix one sampled training dataset and estimate the generator
parameters using it. Then, for each different test interventional distribution
of the covariate, we compute energy distance between the true sample and three
artificial ones corresponding to each line type. For each fixed pertubation
strength, we average across 50 test datasets. Notice how the causal parameter
performs worse than OLS when the test covariate distribution is similar to
that of the training set (small perturbations). Also, notice how OLS degrades
arbitrarily as the strength of interventions grows - notice the log scale in
the horizontal axis. Finally, our proposal seems to perform uniformly well in
terms of imitating predictions uniformly well across interventions. This
served as inspiration for the new concept of generative invariance with
respect to energy statistics.

##  5 Conclusions

Our work identifies causal parameters without relying on instruments, and
constructs a generative model capable of making predictions with covariate
observations from a test domain, without assumptions on perturbation strength.
Unlike distributional robustness literature, which requires assumptions about
perturbation magnitude, our estimator is free from tunable hyperparameters.
Although it assumes real Gaussian variables and no interventions on the
confounders and the response during testing, these constraints simplify the
mathematical framework without detracting from the significance of our
contributions. We are not making any assumption on how much the training and
test distributions overlap, as often needed in the literature (see e.g [4]).
Our analysis does not include a comparison with alternative methods as they
typically require data on both the covariate and the response for at least two
distinct values of $Z$, a condition not requisite for our approach.

A crucial direction for immediate future work is the derivation of asymptotic
properties. This step is essential to fully understand the behavior and
reliability of our estimators as sample sizes increase. Also the derivation of
approximate identifiability results when there are interventions in the latent
variables or the response at test by bounding
$\lVert\beta^{opt}-\beta_{*}\rVert_{\infty}$ and $\lVert
K^{opt}-K_{*}\rVert_{\infty}$

Future work offers promising avenues, including extending our approach to
high-dimensional settings where $X$ is p-dimensional, and exploring nonlinear
models. The class $\mathcal{M}$ looks like a semiparametric model when
replacing $x\beta$ by a richer family of functions. The part of the model
concerning $g_{K}$ is always going to be parametric, very likely of the same
dimension as the number of covariates. These directions not only promise to
broaden the applicability of our methodology but also to enhance its
predictive power in complex data landscapes.

## References

  * [1] J. D. Angrist, G. W. Imbens, and D. B. Rubin.  Identification of causal effects using instrumental variables.  Journal of the American statistical Association, 91(434):444–455, 1996.
  * [2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz.  Invariant risk minimization.  arXiv preprint arXiv:1907.02893, 2019.
  * [3] R. Christiansen, N. Pfister, M. E. Jakobsen, N. Gnecco, and J. Peters.  A causal framework for distribution generalization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6614–6630, 2021.
  * [4] C. García-Meixide and M. Matabuena.  Causal survival embeddings: non-parametric counterfactual inference under censoring, 2023.
  * [5] A. Henzi, X. Shen, M. Law, and P. Bühlmann.  Invariant probabilistic prediction.  arXiv preprint arXiv:2309.10083, 2023.
  * [6] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In International Conference on Machine Learning, pages 5815–5826. PMLR, 2021.
  * [7] Y. Lin, H. Dong, H. Wang, and T. Zhang.  Bayesian invariant risk minimization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16021–16030, 2022.
  * [8] J. Peters, P. Bühlmann, and N. Meinshausen.  Causal inference by using invariant prediction: identification and confidence intervals.  Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947–1012, 2016.
  * [9] M. Rojas-Carulla, B. Schölkopf, R. Turner, and J. Peters.  Invariant models for causal transfer learning.  The Journal of Machine Learning Research, 19(1):1309–1342, 2018\.
  * [10] D. Rothenhäusler, P. Bühlmann, and N. Meinshausen.  Causal dantzig: Fast inference in linear structural equation models with hidden variables under additive interventions.  The Annals of Statistics, 47(3):1688–1722, 2019.
  * [11] D. Rothenhäusler, N. Meinshausen, P. Bühlmann, and J. Peters.  Anchor regression: Heterogeneous data meet causality.  Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(2):215–246, 2021.
  * [12] S. Saengkyongam, L. Henckel, N. Pfister, and J. Peters.  Exploiting independent instruments: Identification and distribution generalization.  In International Conference on Machine Learning, pages 18935–18958. PMLR, 2022.
  * [13] S. Saengkyongam, E. Rosenfeld, P. Ravikumar, N. Pfister, and J. Peters.  Identifying representations for intervention extrapolation.  arXiv preprint arXiv:2310.04295, 2023.
  * [14] X. Shen, P. Bühlmann, and A. Taeb.  Causality-oriented robustness: exploiting general additive interventions.  arXiv preprint arXiv:2307.10299, 2023.
  * [15] G. J. Székely and M. L. Rizzo.  The energy of data.  Annual Review of Statistics and Its Application, 4:447–479, 2017\.
  * [16] P. G. Wright.  The tariff on animal and vegetable oils.  Number 26. Macmillan, 1928.


* Daniel et al. (2018) Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh.  Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions.  _ACM Comput. Surv._ , 51(1), jan 2018.  ISSN 0360-0300.  doi: 10.1145/3148148.  URL https://doi.org/10.1145/3148148.
  * Das and Cui (2019) Manirupa Das and Renhao Cui.  Comparison of quality indicators in user-generated content using social media and scholarly text.  _arXiv_ , 2019.  doi: 10.48550/arXiv.1910.11399.  URL https://doi.org/10.48550/arXiv.1910.11399.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv_ , 2019.  doi: 10.48550/arXiv.1810.04805.  URL https://doi.org/10.48550/arXiv.1810.04805.
  * Douglas et al. (2023) Benjamin D. Douglas, Patrick J. Ewell, and Markus Brauer.  Data quality in online human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona.  _PLOS ONE_ , 18(3):1–17, 03 2023.  doi: 10.1371/journal.pone.0279720.  URL https://doi.org/10.1371/journal.pone.0279720.
  * Dubay (2004) William Dubay.  The principles of readability.  _CA_ , 92627949:631–3309, 01 2004.
  * DuBay (2007) William H. DuBay.  _Smart language: readers, readability, and the grading of text_.  Impact Information, Costa Mesa, Calif., 2007.  ISBN 978-1-4196-5439-8.
  * Feigenbaum (2003) Edward A Feigenbaum.  Some challenges and grand challenges for computational intelligence.  _Journal of the ACM (JACM)_ , 50(1):32–40, 2003\.  ISSN 0004-5411.  doi: 10.1145/602382.602400.
  * Flesch (1948) Rudolph Flesch.  A new readability yardstick.  _Journal of Applied Psychology_ , 32(3):p221 – 233, June 1948.  ISSN 0021-9010.  URL http://libezproxy.open.ac.uk/login?url=http://search.ebscohost.com.libezproxy.open.ac.uk/login.aspx?direct=true&db=pdh&AN=apl-32-3-221&site=ehost-live&scope=site.
  * Gamblin et al. (2017) Bradlee Gamblin, Matthew Winslow, Benjamin Lindsay, Andrew Newsom, and Andre Kehn.  Comparing in-person, sona, and mechanical turk measurements of three prejudice-relevant constructs.  _Current Psychology_ , 36, 06 2017.  doi: 10.1007/s12144-015-9403-1.
  * Gao et al. (2023) Catherine A. Gao, Frederick M. Howard, Nikolay S. Markov, Emma C. Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T. Pearson.  Comparing scientific abstracts generated by chatgpt to real abstracts with detectors and blinded human reviewers.  _npj Digital Medicine_ , 6(1):75, 2023.  ISSN 2398-6352.  doi: 10.1038/s41746-023-00819-6.  URL https://doi.org/10.1038/s41746-023-00819-6.
  * Gilson et al. (2022) Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, R. Andrew Taylor, and David Chartash.  How does chatgpt perform on the medical licensing exams? the implications of large language models for medical education and knowledge assessment.  _medRxiv_ , page 2022.12.23.22283901, 2022.  doi: 10.1101/2022.12.23.22283901.  URL https://www.medrxiv.org/content/medrxiv/early/2022/12/26/2022.12.23.22283901.full.pdf.
  * Godinho et al. (2020) Alexandra Godinho, Christina Schell, and John A. Cunningham.  Out damn bot, out: Recruiting real people into substance use studies on the internet.  _Substance Abuse_ , 41(1):3–5, 2020.  doi: 10.1080/08897077.2019.1691131.  URL https://journals.sagepub.com/doi/abs/10.1080/08897077.2019.1691131.
  * Gomez Adorno et al. (2018) Helena Gomez Adorno, Germán Rios, Juan Posadas Durán, Grigori Sidorov, and Gerardo Sierra.  Stylometry-based approach for detecting writing style changes in literary texts.  _Computación y Sistemas_ , 22, 03 2018.  doi: 10.13053/cys-22-1-2882.  URL https://doi.org/10.13053/cys-22-1-2882.
  * Griffin et al. (2022) Marybec Griffin, Richard J. Martino, Caleb LoSchiavo, Camilla Comer-Carruthers, Kristen D. Krause, Christopher B. Stults, and Perry N. Halkitis.  Ensuring survey research data integrity in the era of internet bots.  _Quality & Quantity_, 56(4):2841–2852, 2022\.  ISSN 1573-7845.  doi: 10.1007/s11135-021-01252-1.  URL https://doi.org/10.1007/s11135-021-01252-1.
  * Gunning (1952) R. Gunning.  _The Technique of Clear Writing_.  McGraw-Hill, 1952.  ISBN 9787000014190.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  _arXiv_ , 2023.  doi: 10.48550/arXiv.2301.07597.  URL https://doi.org/10.48550/arXiv.2301.07597.
  * Hamby and Taylor (2016) Tyler Hamby and Wyn Taylor.  Survey satisficing inflates reliability and validity measures: An experimental comparison of college and amazon mechanical turk samples.  _Educational and Psychological Measurement_ , 76, 01 2016.  doi: 10.1177/0013164415627349.
  * Hämäläinen et al. (2023) Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari.  Evaluating large language models in generating synthetic hci research data: a case study.  In _CHI ’23: CHI Conference on Human Factors in Computing Systems_ , pages 1–19, Hamburg Germany, 2023. ACM.  ISBN 978-1-4503-9421-5.  doi: 10.1145/3544548.3580688.  URL https://dl.acm.org/doi/10.1145/3544548.3580688.
  * Irfan et al. (2018) Bahar Irfan, James Kennedy, Séverin Lemaignan, Fotios Papadopoulos, Emmanuel Senft, and Tony Belpaeme.  Social psychology and human-robot interaction: An uneasy marriage.  In _Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’18, page 13–20, New York, NY, USA, 2018. Association for Computing Machinery.  ISBN 9781450356152.  doi: 10.1145/3173386.3173389.  URL https://doi.org/10.1145/3173386.3173389.
  * Kerr (1998) Norbert Kerr.  Harking: Hypothesizing after the results are known.  _Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc_ , 2:196–217, 02 1998.  doi: 10.1207/s15327957pspr0203˙4.
  * Kincaid et al. (1975) J. Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, and Brad S. Chissom.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.  _Institute for Simulation and Training, University of Central Florida_ , 1975.  URL https://stars.library.ucf.edu/istlibrary/56/?utm_sourc.
  * Krosnick (1991) Jon A. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys.  _Applied Cognitive Psychology_ , 5(3):213–236, 1991.  ISSN 08884080, 10990720.  doi: 10.1002/acp.2350050305.  URL https://doi.org/10.1002/acp.2350050305.
  * Kuek et al. (2015) Siou Chew Kuek, Cecilia Paradi-Guilford, Toks Fayomi, Saori Imaizumi, Panos Ipeirotis, Patricia Pina, and Manpreet Singh.  The Global Opportunity in Online Outsourcing.  World Bank Publications - Reports 22284, The World Bank Group, June 2015\.  URL https://ideas.repec.org/p/wbk/wboper/22284.html.
  * Kumarage et al. (2023) Tharindu Kumarage, Joshua Garland, Amrita Bhattacharjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu.  Stylometric detection of ai-generated text in twitter timelines.  _arXiv_ , 2023.  doi: 10.48550/arXiv.2303.03697.
  * Kung et al. (2023) Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng.  Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models.  _PLOS Digital Health_ , 2(2):e0000198, 2023.  doi: 10.1371/journal.pdig.0000198.  URL https://doi.org/10.1371/journal.pdig.0000198.
  * Leichtmann and Nitsch (2020a) Benedikt Leichtmann and Verena Nitsch.  How much distance do humans keep toward robots? literature review, meta-analysis, and theoretical considerations on personal space in human-robot interaction.  _Journal of Environmental Psychology_ , 68:101386, 2020a.  doi: 10.1016/j.jenvp.2019.101386.  URL https://doi.org/10.1016/j.jenvp.2019.101386.
  * Leichtmann and Nitsch (2020b) Benedikt Leichtmann and Verena Nitsch.  Is the social desirability effect in human–robot interaction overestimated? a conceptual replication study indicates less robust effects.  _International Journal of Social Robotics_ , 13:1013 – 1031, 2020b.  doi: 10.1007/s12369-020-00688-z.  URL https://doi.org/10.1007/s12369-020-00688-z.
  * Leichtmann et al. (2022) Benedikt Leichtmann, Verena Nitsch, and Martina Mara.  Crisis ahead? why human-robot interaction user studies may have replicability problems and directions for improvement.  _Frontiers in Robotics and AI_ , 9:838116, 03 2022.  doi: 10.3389/frobt.2022.838116.
  * Li (2015) Jamy Li.  The benefit of being physically present: A survey of experimental works comparing copresent robots, telepresent robots and virtual agents.  _International Journal of Human-Computer Studies_ , 77, 01 2015.  doi: 10.1016/j.ijhcs.2015.01.001.
  * Lund et al. (2023) Brady Lund, Ting Wang, Nishith Reddy Mannuru, Bing Nie, Somipam Shimray, and Ziang Wang.  Chatgpt and a new academic reality: Artificial intelligence-written research papers and the ethics of the large language models in scholarly publishing.  _Journal of the Association for Information Science and Technology_ , 03 2023.  doi: 10.1002/asi.24750.  URL https://doi.org/10.1002%2Fasi.24750.
  * Mao et al. (2013) Andrew Mao, Ece Kamar, Yiling Chen, Eric Horvitz, Megan Schwamb, Chris Lintott, and Arfon Smith.  Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , volume 1, pages 94–102, 2013.  doi: 10.1609/hcomp.v1i1.13075.  URL https://doi.org/10.1609/hcomp.v1i1.13075.
  * Mc Laughlin (1969) G Harry Mc Laughlin.  Smog grading-a new readability formula.  _Journal of reading_ , 12(8):639–646, 1969.  URL https://www.jstor.org/stable/40011226.
  * Mitrovic et al. (2023) Sandra Mitrovic, Davide Andreoletti, and Omran Ayoub.  Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text.  _arXiv_ , 01 2023.  doi: 10.48550/arXiv.2301.13852.  URL https://doi.org/10.48550/arXiv.2301.13852.
  * Moor (2001) J. H Moor.  The status and future of the turing test.  _Minds and Machines_ , 11(1):77–93, 2001.  ISSN 1572-8641.  doi: 10.1023/A:1011218925467.
  * Naglieri et al. (2004) Jack Naglieri, Fritz Drasgow, Mark Schmit, Len Handler, Aurelio Prifitera, Amy Margolis, and Roberto Velasquez.  Psychological testing on the internet: New problems, old issues.  _The American psychologist_ , 59:150–62, 05 2004.  doi: 10.1037/0003-066X.59.3.150.
  * Nov et al. (2023) Oded Nov, Nina Singh, and Devin Mann.  Putting chatgpt’s medical advice to the (turing) test.  _medRxiv_ , 01 2023.  doi: 10.1101/2023.01.23.23284735.  URL https://dx.doi.org/10.2139/ssrn.4413305.
  * Oliveira et al. (2020) Raquel Oliveira, Patricia Arriaga, Fernando Santos, Samuel Mascarenhas, and Ana Paiva.  Towards prosocial design: A scoping review of the use of robots and virtual agents to trigger prosocial behaviour.  _Computers in Human Behavior_ , 114, 09 2020.  doi: 10.1016/j.chb.2020.106547.
  * Open Science Collaboration (2015) Open Science Collaboration.  Estimating the reproducibility of psychological science.  _Science_ , 349(6251):aac4716, 2015.  doi: 10.1126/science.aac4716.  URL https://doi.org/10.1126/science.aac4716.  doi: 10.1126/science.aac4716.
  * Peer et al. (2017) Eyal Peer, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti.  Beyond the turk: Alternative platforms for crowdsourcing behavioral research.  _Journal of Experimental Social Psychology_ , 70:153–163, 05 2017.  doi: 10.1016/j.jesp.2017.01.006.
  * Pegoraro et al. (2023) Alessandro Pegoraro, Kavita Kumari, Hossein Fereidooni, and Ahmad-Reza Sadeghi.  To chatgpt, or not to chatgpt: That is the question!  _arXiv_ , 2023.  doi: 10.48550/arXiv.2304.01487.  URL https://doi.org/10.48550/arXiv.2304.01487.
  * Peng (2011) Roger D. Peng.  Reproducible research in computational science.  _Science_ , 334(6060):1226–1227, 2011.  doi: 10.1126/science.1213847.  URL https://www.science.org/doi/abs/10.1126/science.1213847.
  * Powers et al. (2007) Aaron Powers, Sara Kiesler, Susan Fussell, and Cristen Torrey.  Comparing a computer agent with a humanoid robot.  In _Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’07, page 145–152, New York, NY, USA, 2007. Association for Computing Machinery.  ISBN 9781595936172.  doi: 10.1145/1228716.1228736.  URL https://doi.org/10.1145/1228716.1228736.
  * Pozzar et al. (2020) R. Pozzar, M. J. Hammer, M. Underhill-Blazey, A. A. Wright, J. A. Tulsky, F. Hong, D. A. Gundersen, and D. L. Berry.  Threats of bots and other bad actors to data quality following research participant recruitment through social media: Cross-sectional questionnaire.  _J Med Internet Res_ , 22(10):e23021, 2020.  ISSN 1439-4456 (Print) 1438-8871.  doi: 10.2196/23021.
  * Rahman and Watanobe (2023) Md. Mostafizer Rahman and Yutaka Watanobe.  Chatgpt for education and research: Opportunities, threats, and strategies.  _Applied Sciences_ , 13(9):5783, May 2023.  ISSN 2076-3417.  doi: 10.3390/app13095783.  URL http://dx.doi.org/10.3390/app13095783.
  * Rudolph et al. (2023) Jürgen Rudolph, Samson Tan, and Shannon Tan.  Chatgpt: Bullshit spewer or the end of traditional assessments in higher education?  _Journal of Applied Learning and Teaching_ , 6(1), 2023\.  doi: 10.37074/jalt.2023.6.1.9.  URL https://doi.org/10.37074/jalt.2023.6.1.9.
  * Searles et al. (2023) Andrew Searles, Yoshimichi Nakatsuka, Ercan Ozturk, Andrew Paverd, Gene Tsudik, and Ai Enkoji.  An empirical study & evaluation of modern captchas.  _arXiv_ , 2023.  doi: 10.48550/arXiv.2307.12108.
  * Si and Callan (2001) Luo Si and Jamie Callan.  A statistical model for scientific readability.  In _Proceedings of the Tenth International Conference on Information and Knowledge Management_ , CIKM ’01, page 574–576, New York, NY, USA, 2001. Association for Computing Machinery.  ISBN 1581134363.  doi: 10.1145/502585.502695.  URL https://doi.org/10.1145/502585.502695.
  * Simmons et al. (2011) Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn.  False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.  _Psychological Science_ , 22(11):1359–1366, 2011\.  ISSN 0956-7976.  doi: 10.1177/0956797611417632.  URL https://doi.org/10.1177/0956797611417632.  doi: 10.1177/0956797611417632.
  * Smith and Senter (1967) E. A. Smith and R. J. Senter.  Automated readability index.  _Amrl tr_ , pages 1–14, 1967.  Smith, E A Senter, R J Journal Article United States 1967/05/01 AMRL TR. 1967 May:1-14.
  * Strait et al. (2020) Megan Strait, Florian Lier, Jasmin Bernotat, Sven Wachsmuth, Friederike Eyssel, Robert Goldstone, and Selma Šabanović.  A three-site reproduction of the joint simon effect with the nao robot.  In _Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’20, page 103–111, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450367462.  doi: 10.1145/3319502.3374783.  URL https://doi.org/10.1145/3319502.3374783.
  * Susnjak (2022) Teo Susnjak.  Chatgpt: The end of online exam integrity?  _arXiv_ , 12 2022.  doi: 10.48550/arXiv.2212.09292.
  * Swiatkowski and Dompnier (2017) Wojciech Swiatkowski and Benoit Dompnier.  Replicability crisis in social psychology: Looking at the past to find new pathways for the future.  _International Review of Social Psychology_ , 30:111–124, 05 2017.  doi: https://doi.org/10.5334/irsp.66.
  * Teitcher et al. (2015) J. E. Teitcher, W. O. Bockting, J. A. Bauermeister, C. J. Hoefer, M. H. Miner, and R. L. Klitzman.  Detecting, preventing, and responding to ”fraudsters” in internet research: ethics and tradeoffs.  _J Law Med Ethics_ , 43(1):116–33, 2015.  ISSN 1073-1105 (Print) 1073-1105.  doi: 10.1111/jlme.12200.
  * Tenney et al. (2021) Elizabeth R. Tenney, Elaine Costa, Aurélien Allard, and Simine Vazire.  Open science and reform practices in organizational behavior research over time (2011 to 2019).  _Organizational Behavior and Human Decision Processes_ , 162:218–223, 2021.  ISSN 0749-5978.  doi: https://doi.org/10.1016/j.obhdp.2020.10.015.  URL https://www.sciencedirect.com/science/article/pii/S0749597820304015.
  * Thellman et al. (2016) Sam Thellman, Annika Silvervarg, Agneta Gulz, and Tom Ziemke.  Physical vs. virtual agent embodiment and effects on social interaction.  In _International Conference on Intelligent Virtual Agents_ , volume 10011, pages 412–415, 09 2016.  ISBN 978-3-319-47664-3.  doi: 10.1007/978-3-319-47665-0˙44.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _arXiv_ , 2023.  doi: 10.48550/arXiv.2302.13971.  URL https://doi.org/10.48550/arXiv.2302.13971.
  * Turing et al. (2004) A. Turing, R. Braithwaite, G. Jefferson, and M. Newman.  Can automatic calculating machines be said to think? (1952).  In Jack Copeland, editor, _The Essential Turing_ , pages 487–515. Clarendon Press, 2004.
  * Turing (1950) A. M. Turing.  Computing machinery and intelligence.  _Mind_ , LIX(236):433–460, 1950.  ISSN 0026-4423.  doi: 10.1093/mind/LIX.236.433.  URL https://doi.org/10.1093/mind/LIX.236.433.
  * Turing (2004) A. M. Turing.  Intelligent machinery.  In Jack Copeland, editor, _The Essential Turing_ , National Physical Laboratory Report, pages 395–432. Clarendon Press, 2004.  ISBN 0191606863.
  * Ullman and Malle (2017) Daniel Ullman and Bertram F. Malle.  Human-robot trust: Just a button press away.  In _Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’17, page 309–310, New York, NY, USA, 2017. Association for Computing Machinery.  ISBN 9781450348850.  doi: 10.1145/3029798.3038423.  URL https://doi.org/10.1145/3029798.3038423.
  * Ullman et al. (2021) Daniel Ullman, Salomi Aladia, and Bertram F. Malle.  Challenges and opportunities for replication science in hri: A case study in human-robot trust.  In _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’21, page 110–118, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450382892.  doi: 10.1145/3434073.3444652.  URL https://doi.org/10.1145/3434073.3444652.
  * Uzun (2023) Levent Uzun.  Chatgpt and academic integrity concerns: Detecting artificial intelligence generated content.  _Language Education and Technology_ , 3:45–54, 04 2023\.  URL http://www.langedutech.com/letjournal/index.php/let/article/view/49.
  * Vonasch et al. (2022) A. Vonasch, R. Mofradidoost, and K. Gray.  When people reject free money: Phantom costs and the psychology of economic exchange.  _PsyArXiv_ , 2022.  doi: 10.31234/osf.io/fcery.
  * Wu et al. (2022) Meng-Jia Wu, Kelly Zhao, and Francisca Fils-Aime.  Response rates of online surveys in published research: A meta-analysis.  _Computers in Human Behavior Reports_ , 7:100206, 05 2022\.  doi: 10.1016/j.chbr.2022.100206.
  * Yarrish et al. (2019) Christina Yarrish, Laurie Groshon, Juliet Daisy Mitchell, Ashlyn Appelbaum, Samantha Klock, Taylor Winternitz, and Dara G. Friedman-Wheeler.  Finding the signal in the noise: Minimizing responses from bots and inattentive humans in online research.  _the Behavior Therapist_ , 42(7):235–242, 2019\.  ISSN 0278-8403(Electronic).