plain

# Functional Generalized Empirical Likelihood Estimation for
Conditional Moment Restrictions

Heiner Kremer  Max Planck Institute for Intelligent Systems, Tübingen, Germany
Jia-Jie Zhu  Weierstrass Institute for Applied Analysis and Stochastics,
Berlin, Germany  Krikamol Muandet  Max Planck Institute for Intelligent
Systems, Tübingen, Germany  Bernhard Schölkopf

###### Abstract

Important problems in causal inference, economics, and, more generally, robust
machine learning can be expressed as conditional moment restrictions, but
estimation becomes challenging as it requires solving a continuum of
unconditional moment restrictions. Previous works addressed this problem by
extending the generalized method of moments (GMM) to continuum moment
restrictions. In contrast, generalized empirical likelihood (GEL) provides a
more general framework and has been shown to enjoy favorable small-sample
properties compared to GMM-based estimators. To benefit from recent
developments in machine learning, we provide a functional reformulation of GEL
in which arbitrary models can be leveraged. Motivated by a dual formulation of
the resulting infinite dimensional optimization problem, we devise a practical
method and explore its asymptotic properties. Finally, we provide kernel- and
neural network-based implementations of the estimator, which achieve state-of-
the-art empirical performance on two conditional moment restriction problems.

##  1 Introduction

Moment restrictions identify a parameter of interest by restricting the
expectation value of so-called moment functions, which depend on the parameter
and random variables representing the underlying noisy data generating
process. Important problems in causal inference, economics, and generally
robust machine learning can be cast in this form (Newey, 1993; Ai and Chen,
2003; Bennett and Kallus, 2023; Dikkala et al., 2020). Particularly
challenging are problems formulated as _conditional_ moment restrictions
(CMR), which constrain the conditional expectation of the moment function.
Such problems appear, e.g., in instrumental variable (IV) regression (Newey
and Powell, 2003; Angrist and Pischke, 2008), where the expectation of the
residual of the prediction conditioned on so-called instruments is restricted
to be zero. Other applications are policy learning (Bennett and Kallus, 2020)
and off-policy evaluation in reinforcement learning (Kallus and Uehara, 2020;
Bennett et al., 2021; Chen et al., 2022) as well as double/debiased machine
learning (Chernozhukov et al., 2016, 2017, 2018).

As conditional moment restrictions are difficult to handle directly, a common
approach is to transform them into an infinite number of corresponding
unconditional moment restrictions Bierens (1982). Generalizing the
corresponding estimation methods from the finite-dimensional case to the
infinite-dimensional case is an active area of research (Carrasco and Florens,
2000; Carrasco et al., 2007; Chaussé, 2012; Carrasco and Kotchoni, 2017;
Muandet et al., 2020; Bennett and Kallus, 2023; Zhang et al., 2023).

One of the most popular approaches to learning with moment restrictions is
Hansen’s celebrated generalized method of moments (GMM) (Hansen, 1982). In
order to improve the small sample properties of GMM estimators, alternative
methods have been proposed and are generally known as generalized empirical
likelihood (GEL) estimators (Smith, 1997, 2011; Newey and Smith, 2004). GEL
generalizes the original empirical likelihood framework developed by Owen
(1988, 1990) and Qin and Lawless (1994) to different divergence functions and
contains many related estimators as special cases. While closely related to
GMM, the estimators from the GEL family have been shown theoretically to
exhibit smaller higher-order-biases than those of GMM (Newey and Smith, 2004)
and therefore promise to have favorable small sample properties. With
increasing number of over-identifying restrictions, i.e., when the number of
restrictions exceeds the number of parameters, this advantage has been shown
theoretically to become more significant (Newey and Smith, 2002; Donald et
al., 2003). Therefore, we expect the framework to be particularly suited for
the case of infinitely many restrictions. We leverage this potential for
conditional moment restrictions by developing the theoretical foundation for a
GEL framework with continua of moment restrictions.

##### Our contributions

First, we extend the GEL framework to conditional moment restrictions by
generalizing it to _functional_ moment restrictions. Second, building on a
result from infinite optimization, we derive a dual form which allows us to
employ modern machine learning models in the GEL context. This generalizes
existing results not only to functional moment restrictions but also to
general $\varphi$-divergences beyond the Cressie-Reed family. Third, we
provide the asymptotic properties of our estimator, showing that it is
consistent for functional moment restrictions. Then, we show how the result
for the functional case can be translated to the conditional case yielding an
efficient estimator for conditional moment restrictions. Finally, we discuss
the relation to existing methods and provide experimental results.

Compared to previous extensions of GEL (Kitamura et al., 2004; Tang and Leng,
2010; Chaussé, 2012; Carrasco and Kotchoni, 2017), our approach combines the
idea of a continuum generalization of GEL (Chaussé, 2012; Carrasco and
Kotchoni, 2017) with the flexibility of machine learning models such as neural
networks and kernel methods. Our general framework contains related estimators
such as a one-step/continuous updating version of the variational method of
moments (VMM) estimator (Bennett and Kallus, 2023) as special cases. In
contrast to VMM, our method allows the use of divergences other than the
$\chi^{2}$-divergence.

The remainder of this paper is organized as follows. Section 2 introduces the
method of moments framework (Hall, 2004) and two popular relaxations. Section
3 presents our main contributions, the theoretical development of our FGEL
estimator, followed by experimental results in Section 4. Finally, we discuss
related works in Section 5.

Compared to the initial conference version, this paper has been updated using
recent results of Kremer et al. (2023) to derive an explicit convergence rate
for the estimator, as well as to prove its semi-parametric efficiency.

##  2 Learning with Moment Restrictions

Let $X$ be a random variable taking values in
$\mathcal{X}\subseteq\mathbb{R}^{r}$ with distribution $P_{0}$ and let
$\psi:\mathcal{X}\times\Theta\rightarrow\mathbb{R}^{m}$ denote a vector of $m$
functions, the so-called moment functions, with parameters
$\theta\in\Theta\subset\mathbb{R}^{p}$. We denote with $E_{P}[\cdot]$ the
expectation over all random variables that are not conditioned on with respect
to a distribution $P$ and refer to the population distribution $P_{0}$
whenever we omit the subscript. Assume that there exists a unique parameter
$\theta_{0}\in\Theta$ such that $E[\psi(X;\theta_{0})]=0$. For instance,
$E[X-\theta_{0}]=0$ characterizes the mean of $P_{0}$. Our goal is to estimate
$\theta_{0}$ based on a sample $\\{x_{i}\\}_{i=1}^{n}$ from $P_{0}$. The
corresponding empirical moment restrictions read

$\displaystyle E_{\hat{P}_{n}}[\psi(X;\theta)]=0,\quad\theta\in\Theta,$ (1)

where $\hat{P}_{n}=\sum_{i=1}^{n}\frac{1}{n}\delta_{x_{i}}$ is the empirical
distribution. This is a system of $m$ estimating equations for $p$ parameters
which can be fulfilled exactly as long as $m\leq p$. For example,
$E_{\hat{P}_{n}}[X-\theta]=0$ gives $\theta=\frac{1}{n}\sum_{i=1}^{n}x_{i}$ as
an empirical estimate of the mean of $P_{0}$. However, in the over-identified
case, i.e., when the number of non-redundant moment restrictions exceeds the
number of parameters ($m>p$), it is generally impossible to fulfill all moment
restrictions (1) exactly. To obtain a feasible problem, the constraints (1)
need to be relaxed. Below we discuss two popular approaches, namely, the
generalized method of moments (Hansen, 1982) and maximum (generalized)
empirical likelihood estimation (Owen, 1988, 1990; Qin and Lawless, 1994).

##### Generalized Method of Moments (GMM)

The generalized method of moments relaxes the constraint (1) into a
minimization of a quadratic form of the average of the moment functions, i.e.,
$\theta^{\mathrm{GMM}}_{W}=\operatorname*{arg\,min}_{\theta\in\Theta}\,\hat{\psi}(\theta)^{T}W\hat{\psi}(\theta)$,
where $\hat{\psi}(\theta):=E_{\hat{P}_{n}}[\psi(X;\theta)]$ and
$W\in\mathbb{R}^{m\times m}$ denotes the so-called weighting matrix.
Asymptotic normality theory shows that an efficient estimator, i.e., an
estimator with minimal asymptotic variance among the class of GMM estimators,
is obtained by choosing $W$ as the inverse covariance matrix of the moment
functions, $W=\widehat{\Omega}_{\theta}^{-1}$, where
$\widehat{\Omega}_{\theta}:=E_{\hat{P}_{n}}[\psi(X;\theta)\psi(X;\theta)^{T}]$,
which itself a function of $\theta$ (Hansen, 1982). The resulting estimator,
i.e.,

$\displaystyle\theta^{\mathrm{CUE}}=\operatorname*{arg\,min}_{\theta\in\Theta}\;\hat{\psi}(\theta)^{T}\widehat{\Omega}_{\theta}^{-1}\hat{\psi}(\theta),$
(2)

is the continuous updating estimator (CUE) of Hansen et al. (1996) which
results from a non-convex optimization problem and can exhibit unfavorable
convergence properties if $\widehat{\Omega}_{\theta}$ is ill-conditioned
(Hall, 2004). Therefore, one often resorts to a 2-step procedure: first, an
inefficient but consistent estimate $\tilde{\theta}$ of $\theta_{0}$ is
obtained, e.g., by setting $W=I$. Second, this estimate is used to compute
$\widehat{\Omega}_{\tilde{\theta}}^{-1}$ which is kept fixed during the second
optimization step. This yields the so-called optimally weighted GMM estimator
(Hansen, 1982),

$\displaystyle\theta^{\mathrm{OWGMM}}=\operatorname*{arg\,min}_{\theta\in\Theta}\;\hat{\psi}(\theta)^{T}\widehat{\Omega}_{\tilde{\theta}}^{-1}\hat{\psi}(\theta).$
(3)

A more in-depth exposition of the GMM framework can be found in Hall (2004).

##### Generalized Empirical Likelihood (GEL)

The empirical likelihood framework (Owen, 1988, 1990; Qin and Lawless, 1994)
relaxes the restrictions (1) by requiring $E_{P}[\psi(X;\theta)]=0$ to be
fulfilled exactly but allowing the distribution $P$ to deviate from the
empirical distribution $\hat{P}_{n}$. For a continuous function
$\varphi:\mathbb{R}\rightarrow\mathbb{R}$ we define the $\varphi$-divergence
between distributions $P$ and $Q$ as
$D_{\varphi}(P||Q)=\int\varphi\big{(}\frac{\mathrm{d}P}{\mathrm{d}Q}\big{)}\mathrm{d}Q$,
where $\frac{\mathrm{d}P}{\mathrm{d}Q}$ denotes the Radon-Nikodym derivative
of $P$ with respect to $Q$. Then, we can define the profile divergence with
respect to this $\varphi$-divergence as

$\displaystyle R(\theta)=\inf_{P\ll\hat{P}_{n}}\
D_{\varphi}(P||\hat{P}_{n})\quad\mathrm{s.t.}\quad
E_{P}[\psi(X;\theta)]=0,\quad E_{P}[1]=1,$ (4)

where $P\ll\hat{P}_{n}$ describes the set of positive measures $P$ that are
absolutely continuous with respect to the empirical distribution
$\hat{P}_{n}$. In other words, $P$ describes multinomial distributions on the
sample, i.e., re-weightings of the data points. The maximum empirical
likelihood estimator (MELE) for $\theta$ is then given by
$\theta^{\mathrm{EL}}=\operatorname*{arg\,min}_{\theta\in\Theta}R(\theta)$.
The framework, originally proposed as empirical likelihood by Owen for the
case $\varphi(p)=-2\log(p)$, has been generalized to other divergence measures
for which it is known as minimum discrepancy (MD) (Corcoran, 1998) or
generalized empirical likelihood (Smith, 1997, 2005). The latter corresponds
to its dual formulation. It contains many related estimators as special cases.
For example, by choosing the function $\varphi$ from the Cressie-Read family
of non-parametric discrepancy measures (Cressie and Read, 1984),

$\displaystyle\varphi_{\gamma}(p)=\frac{1}{\gamma(\gamma+1)}\left(p^{\gamma+1}-1\right),$
(5)

one retrieves the CUE for $\gamma=1$ (Newey and Smith, 2004), the exponential
tilting estimator for $\gamma\rightarrow 0$ (Kitamura and Stutzer, 1997) and
finally the original empirical likelihood estimator for $\gamma\rightarrow-1$
(Qin and Lawless, 1994). Detailed exposition of the GEL framework can be found
in Smith (1997) and Owen (2001).

##  3 Functional Generalized Empirical Likelihood

In this work, we are concerned with problems that can be expressed by
infinitely many moment restrictions, especially those that arise from
_conditional_ moment restrictions (CMR) (Newey, 1993; Ai and Chen, 2003) of
the form

$\displaystyle E[\psi(X;\theta_{0})|Z]=0,\quad P_{Z}\textrm{-a.s.},$ (6)

where $Z\in\mathcal{Z}$ is an additional random variable with marginal
distribution $P_{Z}$. By the law of iterated expectation, the CMR (6) can be
expressed in terms of infinitely many unconditional moment restrictions
(Bierens, 1982)

$\displaystyle E[\psi(X;\theta_{0})^{T}h(Z)]=0,\quad\forall h\in\mathcal{H},$
(7)

where $\mathcal{H}$ denotes a space of measurable functions
$h:\mathcal{Z}\rightarrow\mathbb{R}^{m}$, i.e., the space of square integrable
functions $L^{2}(P_{Z})$. As (7) has to hold for all functions in
$\mathcal{H}$, this implies an uncountable infinite number, i.e., a continuum,
of moment restrictions ($m=\infty$). For example, the instrumental variable
regression problem can be described by a CMR via $E[Y-f(X;\theta_{0})|Z]=0$
where $Z$ is an instrumental variable and $\theta\in\Theta$ parameterizes a
function $f:\mathcal{X}\rightarrow\mathcal{Y}$. Motivated by this example, in
the following, we will refer to $Z$ and $h$ as instrument and instrument
function, respectively, in the context of general CMR.

###  3.1 Our Method

Maximum empirical likelihood estimation is based on minimizing a profile
divergence $R:\Theta\rightarrow\mathbb{R}$ over a parameter space $\Theta$.
Let $\mathcal{P}:=\\{P\ll\hat{P}_{n}:E_{P}[1]=1\\}$ denote the set of
distributions that are absolutely continuous with respect to the empirical
distribution. For conditional moment restrictions of the form (6), we can
define the profile divergence as

$\displaystyle
R(\theta):=\min_{P\in\mathcal{P}}D_{\varphi}(P||\hat{P}_{n})\quad\mathrm{s.t.}\quad
E_{P}[\psi(X;\theta)|Z]=0\quad P_{Z}\textrm{-a.s.},$ (8)

where $D_{\varphi}$ is defined in terms of the $\varphi$-divergence (see Table
1). Let $\mathcal{H}$ be a sufficiently large Hilbert space of functions such
that (7) implies (6) and let $\mathcal{H}_{1}=\\{h\in\mathcal{H}:\|h\|\leq
1\\}$ denote the unit ball in $\mathcal{H}$. Let $\mathcal{H}^{\ast}$ be the
corresponding dual space of functionals $\mathcal{H}\rightarrow\mathbb{R}$
equipped with the dual norm $\|\cdot\|_{\mathcal{H}^{\ast}}$ defined for
$\Psi\in\mathcal{H}^{\ast}$ as
$\|\Psi\|_{\mathcal{H}^{\ast}}=\sup_{h\in\mathcal{H}_{1}}\Psi(h)$. Then, we
can define the _moment functional_ , a statistical functional
$\Psi(X,Z;\theta)\in\mathcal{H}^{\ast}$, as

$\displaystyle\Psi(X,Z;\theta):\ \ $ $\displaystyle\mathcal{H}\rightarrow\
\mathbb{R}$ $\displaystyle h\mapsto\
\Psi(X,Z;\theta)(h)=\psi(X;\theta)^{T}h(Z),$

which can be seen as a weighted evaluation functional with respect to the
conditioning variable $Z$. With this definition, we can express (7) as the
functional constraint
$\|E_{P_{0}}[\Psi(X,Z;\theta_{0})]\|_{\mathcal{H}^{\ast}}=0$. The computation
of the profile likelihood thus becomes a _functionally-constrained_
optimization problem

$\displaystyle
R(\theta)=\inf_{P\in\mathcal{P}}D_{\varphi}(P||\hat{P}_{n})\quad\mathrm{s.t.}\quad\left\|E_{P}[\Psi(X,Z;\theta_{0})]\right\|_{\mathcal{H}^{\ast}}=0.$
(9)

The FGEL problem arises from the dual formulation of (9). For the case of
finite dimensional moment restrictions, the duality relationship has been
extensively explored by numerous works (Smith, 1997, 2011; Kitamura et al.,
2004; Newey and Smith, 2004). However, as shown by Borwein (1993) these
duality results do not carry over to infinite dimensional restrictions.
Following the approach of Borwein (1993) and Carrasco and Kotchoni (2017), we
define a relaxed version of the functionally constrained profile likelihood
(9) with relaxation parameter $\lambda>0$ as

$\displaystyle
R_{\lambda}(\theta):=\inf_{P\in\mathcal{P}}D_{\varphi}(P||\hat{P}_{n})\quad\mathrm{s.t.}\quad\left\|E_{P}[\Psi(X,Z;\theta)]\right\|_{\mathcal{H}^{\ast}}\leq\lambda.$
(10)

With this relaxation, a constraint qualification condition holds and (10)
admits a strongly dual form as formalized in the following theorem.

######  Theorem 3.1.

Let $\varphi^{\ast}(v)=\sup_{p\in\mathbb{R}^{n}}\langle v,p\rangle-\varphi(p)$
denote the Legendre-Fenchel conjugate function of a strongly convex function
$\varphi$. Then the problem

$\displaystyle
R_{\lambda}(\theta)=\inf_{p\in\mathbb{R}^{n}}\sum_{i=1}^{n}\frac{1}{n}f(np_{i})\quad\mathrm{s.t.}\quad\left\|\frac{1}{n}\sum_{i=1}^{n}p_{i}\Psi(x_{i},z_{i};\theta)\right\|_{\mathcal{H}^{\ast}}\leq\lambda,\
\ \sum_{i=1}^{n}p_{i}=1$

admits the dual form

$\displaystyle R_{\lambda}(\theta)=\sup_{\begin{subarray}{c}h\in\mathcal{H}\\\
\mu\in\mathbb{R}\end{subarray}}\mu-\frac{1}{n}\sum_{i=1}^{n}\varphi^{\ast}\left(\Psi(x_{i},z_{i};\theta)(h)+\mu\right)-\lambda\|h\|_{\mathcal{H}}$
(11)

and strong duality holds between these formulations. Moreover, the unique
minimizer of the primal problem is given by

$p_{i}=\left(\frac{d}{dv}\varphi^{\ast}\right)\left(\Psi(x_{i},z_{i};\theta)(\hat{h})+\hat{\mu}\right),$

where $\hat{h}$, $\hat{\mu}$ are any solutions of the dual problem. Moreover,
as $\lambda\rightarrow 0$, $R_{\lambda}(\theta)\rightarrow R(\theta)$.

######  Remark 3.2.

Theorem 3.1 can be seen as a generalization of the duality result of Newey and
Smith (2004) not only to functional-valued moment restrictions but also to
general strongly convex divergence functions beyond the Cressie-Reed family.

Equation (11) provides a regularized functional generalization of the profile
divergence. Based on this result, we define our functional generalized
empirical likelihood estimator by making two modifications: first, we
substitute the norm term in (11) for a differentiable quadratic version. This
modification is solely to simplify the analysis. Later we will choose the
regularization parameter to be $\lambda_{n}=o_{p}(1)$ and find that
$\|h\|_{\mathcal{H}}=o_{p}(1)$. Hence we can always find a $\chi^{\prime}>0$
and $\lambda^{\prime}_{n}=O_{p}(n^{-\chi^{\prime}})$ such that
$\lambda_{n}/2\|h\|^{2}\rightarrow 0$ and
$\lambda^{\prime}_{n}\|h\|\rightarrow 0$ at the same rate which implies that
the formulations are asymptotically equivalent. Second, we drop the Lagrange
parameter $\mu$ corresponding to the normalization constraint
$\sum_{i=1}^{n}p_{i}=1$. This is motivated by several observations. From a
theoretical point of view, it simplifies the problem at no cost as we will
show later in Theorems 3.8-3.10 that setting $\mu=0$ still yields a consistent
and efficient estimator. From a practical aspect it facilitates the
implementation of the estimator using stochastic gradient methods. To see
this, consider exemplarily the FGEL estimator with $\chi^{2}$-divergence. Then
we can carry out the supremum over $\mu\in\mathbb{R}$ in closed form and
obtain

$\displaystyle R^{\chi^{2}}_{\lambda}(\theta)=$
$\displaystyle\sup_{h}E_{\hat{P}_{n}}[\psi(X;\theta)^{T}h(Z)]$
$\displaystyle-\frac{1}{2}E_{\hat{P}_{n}}\left[\left(\psi(X;\theta)^{T}h(Z)-E_{\hat{P}_{n}}[\psi(X;\theta)^{T}h(Z)]-1\right)^{2}\right]-\lambda\|h\|_{\mathcal{H}}.$

This contains two expectation operators combined in a non-linear way, which
generally leads to biased gradient estimates and thus renders mini-batch
stochastic gradient descent optimization complicated. In addition, by setting
$\mu=0$, our estimator has the same form as the finite dimensional GEL
estimator proposed by Smith (1997) and Kitamura and Stutzer (1997) with the
difference of involving an optimization over functions $h\in\mathcal{H}$ and
an additional regularization term for $h$. With these modifications, we define
our FGEL estimator as follows.

Table 1: Common choices for the $\varphi$-divergence and the corresponding convex conjugate $\varphi^{\ast}(v)=\sup_{p}p^{T}v-\varphi(p)$ and its domain. A GEL function $\phi$ can be defined for each $\varphi$-divergence as $\phi(v)=-\varphi^{\ast}(v)$. $\varphi(p)$ | $\varphi^{\ast}(v)$ | $\operatorname{dom}(\varphi^{\ast})$
---|---|---
$\frac{1}{2}(p-1)^{2}$ | $\frac{1}{2}(1+v)^{2}$ | $\mathbb{R}$
$-\log(p)$ | $\log(1-v)$ | $\big{(}-\infty,1-\frac{1}{n}\big{]}$
$p\log(p)$ | $e^{v}$ | $\mathbb{R}$

######  Definition 3.3.

Let $V\subseteq\mathbb{R}$ be an open interval containing zero and
$\phi:V\rightarrow\mathbb{R}$ be a twice differentiable concave function with
first and second derivatives $\phi_{1}(0)\neq 0$ and $\phi_{2}(0)<0$. Then we
define the empirical FGEL objective
$G:\Theta\times\widehat{\mathcal{H}}(\theta)\rightarrow\mathbb{R}$ as

$G_{\lambda_{n}}(\theta,h):=\frac{1}{n}\sum_{i=1}^{n}\phi\left(\Psi(x_{i},z_{i};\theta)(h)\right)-\frac{\lambda_{n}}{2}\|h\|_{\mathcal{H}}^{2},$
(12)

where $\Psi(x_{i},z_{i};\theta)(h)=\psi(x_{i};\theta)^{T}h(z_{i})$ and
$\widehat{\mathcal{H}}(\theta):=\\{h\in\mathcal{H}:\
\psi(x_{i};\theta)^{T}h(z_{i})\in\operatorname{dom}(\phi),\ 1\leq i\leq n\\}$.
The FGEL estimate $\hat{\theta}$ of $\theta_{0}$ results from a saddle point
of $G_{\lambda_{n}}(\theta,h)$

$\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}\sup_{h\in\widehat{\mathcal{H}}(\theta)}G_{\lambda_{n}}(\theta,h).$
(13)

######  Remark 3.4.

While the choice of $\phi$ can be motivated by the duality relationship of
Theorem 3.1, i.e., $\phi=-\varphi^{\ast}$ with
$\varphi^{\ast}(v)=\sup_{p}v^{T}p-\varphi(p)$ for $\varphi$ defined via some
$\varphi$-divergence, GEL estimators can be defined for any function $\phi$
which fulfills the corresponding conditions of Definition 3.3.

Within the FGEL framework, the regularization term is responsible for
regularizing an originally ill-posed operator estimation problem, which
results from the optimization of $G_{\lambda_{n}}(\theta,h)$ over the
instrument functions $h\in\mathcal{H}$. We will demonstrate this here
exemplarily for the $\chi^{2}$-divergence, which admits a closed form
solution. Note that a similar argument has been provided earlier by Carrasco
and Kotchoni (2017). Let
$\Psi_{i}(\theta):=\Psi(x_{i},z_{i};\theta)\in\mathcal{H}^{\ast}$ and
$\Psi_{i}^{\ast}\in\mathcal{H}$ denote its dual which can be identified with a
function in $\mathcal{H}$ by the self-duality property of Hilbert spaces
(Zeidler, 2012). Then the first order condition for $h$ reads

$\displaystyle 0$
$\displaystyle=-\frac{1}{n}\sum_{i=1}^{n}\left[\Psi_{i}^{\ast}(\theta)-\left(\Psi_{i}^{\ast}(\theta)\Psi_{i}(\theta)+\lambda_{n}I\otimes
I\right)(h)\right]$ $\displaystyle\Rightarrow
h=-\left(\widehat{\Omega}_{\theta}+\lambda_{n}I\otimes
I\right)^{-1}\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}^{\ast}(\theta),$

where
$\widehat{\Omega}_{\theta}=\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}^{\ast}(\theta)\Psi_{i}(\theta)$
denotes the empirical covariance operator of the moment functional. For any
$i=1,\ldots,n$, the operator $\Psi_{i}^{\ast}(\theta)\Psi_{i}(\theta)$ has at
most rank $1$ and thus as the sum of $n$ such operators
$\widehat{\Omega}_{\theta}$ can have most rank $n$. As
$\widehat{\Omega}_{\theta}$ is an infinite dimensional linear operator
$\mathcal{H}\rightarrow\mathcal{H}$ it thus must be rank deficient for any
finite $n$ and therefore singular and non-invertible. This highlights the fact
that the regularization parameter in the FGEL framework is not merely an
artefact of the restoration of the strong duality between the primal and dual
GEL problems, but a fundamental requirement for any definition of a
functional/continuum GEL extension. Note that by the uniform weak law of large
number and the continuous mapping theorem we have for $\Psi(\theta)$
continuous in $\theta$ and $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$
that
$\widehat{\Omega}(\hat{\theta})\overset{p}{\rightarrow}\Omega_{0}=E[\Psi(X,Z;\theta_{0})^{\ast}\Psi(X,Z;\theta_{0})]$.
In Theorem 3.8 we show that for the conditional case under the mild assumption
that $V(Z):=E[\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}|Z]$ is non-singular
with probability $1$ together with assumptions on the instrument functions
class $\mathcal{H}$ it follows that $\Omega_{0}$ is non-singular and thus in
the limit $n\rightarrow\infty$ the first order conditions for $h$ remain well
posed as $\lambda_{n}\rightarrow 0$.

The general formulation (13) allows us to employ a wide range of function
classes $\mathcal{H}$ and generally for finite samples, the choice of
$\mathcal{H}$ will influence the obtained estimator. Building on recent
developments in machine learning, we can represent $h$ by a flexible deep
neural network (Hartford et al., 2017; Lewis and Syrgkanis, 2018) or a random
forest model (Athey et al., 2019), for example. In this work, we mainly focus
our discussion on instrument functions from reproducing kernel Hilbert spaces
for their favorable theoretical properties but also consider neural network
function classes.

######  Remark 3.5.

The FGEL framework admits an interesting relation to _distributionally robust
optimization_ and as such can be used for (distributionally) robust learning.
Refer to Section A.1 of the appendix for a more detailed account of this
connection.

###  3.2 Asymptotic Properties

In this section, we establish asymptotic properties of our estimator given in
(13). The proofs generalize the ones of Newey and Smith (2004) for the GEL
estimator with finite dimensional moment restrictions to our regularized
problem with functional-valued moment restrictions. Let in the following
$\mathcal{H}_{1}$ denote the unit ball in $\mathcal{H}$.

######  Theorem 3.6 (Consistency).

Assume that a) $\theta_{0}\in\Theta$ is the unique solution to
$\|E[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}=0$; b) $\Theta$, $\mathcal{X}$,
$\mathcal{Z}$ are compact; c) $\Psi(x,z;\theta)$ is continuous in $x,z$ and
$\theta$ everywhere;
d)
$E[\left(\sup_{\theta\in\Theta}\|\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}\right)^{\nu}]<\infty$
for some $\nu>2$; e)
$\Omega_{0}=E[\Psi(X,Z;\theta_{0})\otimes\Psi(X,Z;\theta_{0})]$ is non-
singular; f) $\lambda_{n}=O_{p}(n^{-\xi})$ with $0<\xi<1/2$; g) $\phi$ is
twice continuously differentiable in a neighborhood of zero and
$\phi_{1}(0)\neq 0$, $\phi_{2}(0)<0$; h) the class of functions
$\\{\Psi(\cdot;\theta)(h):\ \theta\in\Theta,\ h\in\mathcal{H}_{1}\\}$ is
$P_{0}$-Donsker. Let $\hat{\theta}$ denote the FGEL estimator for
$\theta_{0}$, then $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$ and
$\|E[\Psi(X,Z;\hat{\theta})]\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$.

If additionally i) $\theta_{0}\in\operatorname{int}(\Theta)$; j)
$\Psi(x,z;\theta)$ is continuously differentiable in a neighborhood
$\bar{\Theta}$ of $\theta_{0}$
and
$E[\sup_{\theta\in\bar{\Theta}}\|\nabla_{\theta}\Psi(X,Z;\theta)\|_{\mathcal{H}^{\ast}}^{2}]<\infty$;
and k) $\Sigma_{0}:=\left\langle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})],E[\nabla_{\theta^{T}}\Psi(X,Z;\theta_{0})]\right\rangle_{\mathcal{H}^{\ast}}$
is a non-singular matrix in $\mathbb{R}^{p\times p}$, we have
$\|\hat{\theta}-\theta_{0}\|_{2}^{2}=O_{p}(n^{-1/2})$.

The following theorem shows that the limiting distributions of the variables
follow a normal distribution $N$ with covariance matrix $\Sigma_{\theta}$ and
Gaussian process $\mathcal{N}$ with kernel $\Sigma_{h}$ respectively.

######  Theorem 3.7 (Asymptotic normality).

Let the assumptions of Theorem 3.6 be satisfied and define the parameter-
gradient of the moment functional as
$\nabla_{\theta}\Psi_{0}:=E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]$. Then,

$\displaystyle\sqrt{n}(\hat{\theta}-\theta_{0})\overset{d}{\rightarrow}N(0,\Sigma_{\theta}),\quad\sqrt{n}(\hat{h}-h)\overset{d}{\rightarrow}\mathcal{N}(0,\Sigma_{h}),$

where
$\Sigma_{\theta}=((\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast}))^{-1}$
and
$\Sigma_{h}=\Omega_{0}^{-1}-\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast})\Sigma_{\theta}(\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}$.

###  3.3 From Functional to Conditional Moment Restrictions

Theorems 3.6 and 3.7 show that FGEL provides a $n^{-1/2}$-consistent and
asymptotically normal estimator for _functional_ moment restrictions of the
form $\|E[\Psi(X,Z;\theta_{0})]\|_{\mathcal{H}^{\ast}}=0$. While this is of
independent interest, often functional moment restrictions are merely a way to
express a set of corresponding conditional moment restrictions
$E[\psi(X;\theta_{0})|Z]=0\ P_{Z}\mathrm{-a.s.}$ using a sufficiently
expressive function space $\mathcal{H}$ in the corresponding variational form
(7). The following theorems show how the results on the functional formulation
translate to the case of conditional moment restrictions. These results were
not contained in the conference version of this manuscript and are originally
due to Kremer et al. (2023).

######  Theorem 3.8 (Consistency).

Let $\mathcal{H}\subseteq L^{2}(\mathcal{Z},\mathbb{R}^{m},P_{Z})$ be a
Hilbert space of locally Lipschitz functions which is sufficiently rich such
that equivalence between (6) and (7) holds. Further assume that a)
$\theta_{0}\in\Theta$ is the unique solution to $E[\psi(X;\theta)|Z]=0\
P_{Z}\text{-a.s.}$; b) $\Theta\subset\mathbb{R}^{p}$ as well as
$\mathcal{X}\times\mathcal{Z}$ are compact; c) $\psi(x;\theta)$ is continuous
in $x$ and $\theta$ everywhere; d)
$E[\sup_{\theta\in\Theta}\|\psi(X;\theta)\|^{2}_{2}|Z]<\infty$ w.p.1; e)
$V_{0}(Z):=E[\psi(X;\theta_{0})\psi(X;\theta_{0})|Z]$ is non-singular w.p.1;
and f) $\lambda_{n}=O_{p}(n^{-\xi})$ with $0<\xi<1/2$; and g) $\phi$ is twice
continuously differentiable in a neighborhood of zero and $\phi_{1}(0)\neq 0$,
$\phi_{2}(0)<0$ and h) the function classes
$\\{\psi(\cdot;\theta):\theta\in\Theta\\}$ and $\mathcal{H}_{1}$ are
$P_{0}$-Donsker. Then for the FGEL estimator $\hat{\theta}$ we have
$\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$.

Assume additionally i) $\theta_{0}\in\operatorname{int}(\Theta)$; j)
$\psi(x;\theta)$ is continuously differentiable in a neighborhood
$\bar{\Theta}$ of $\theta_{0}$
and
$E[\sup_{\theta\in\bar{\Theta}}\|\nabla_{\theta}\psi(X;\theta)\|^{2}|Z]<\infty$
w.p.1; as well as k)
$\operatorname{rank}\left(E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\right)=p$
w.p.1. Then we have $\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$.

######  Theorem 3.9 (Asymptotic Normality).

Let the assumptions of Theorem 3.8 be satisfied. Then, for the FGEL estimator
$\hat{\theta}$ we have

$\displaystyle\sqrt{n}(\hat{\theta}-\theta_{0})\overset{d}{\rightarrow}N(0,\Xi_{0})$

where $\Xi_{0}=E\left[E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\ V_{0}^{-1}(Z)\
E[\nabla_{\theta}\psi(X;\theta_{0})|Z]\right]^{-1}$.

The asymptotic variance in Theorem 3.9 agrees with the semi-parametric
efficiency bound of Chamberlain (1987). This implies that FGEL provides an
efficient estimator for conditional moment restrictions.

######  Corollary 3.10 (Efficiency).

Let the assumptions of Theorem 3.8 be satisfied. Then the FGEL estimate
$\hat{\theta}$ is an efficient estimator for $\theta_{0}$, i.e., it has the
smallest asymptotic variance among all estimators based solely on the
conditional moment restrictions $E[\psi(X;\theta_{0})|Z]=0\
P_{Z}\text{-a.s.}$.

In order to translate the results for the functional estimator to conditional
moment restrictions by applying Theorems 3.8-3.10 one needs to choose a space
of instrument functions $\mathcal{H}$ which fulfills the corresponding
conditions. In the following we show that the reproducing kernel Hilbert space
of certain kind of kernel fulfills these assumptions.

###  3.4 Kernel FGEL

The definition of our FGEL estimator contains a supremum over a function space
$\mathcal{H}$. In order to address the conditional moment restriction problem,
the function space must be expressive enough to exhibit an equivalent
unconditional formulation. At the same time, optimization over function spaces
is generally intractable and thus requires approximations. Selecting
instrument functions from a reproducing kernel Hilbert space, one obtains a
computationally efficient formulation involving finite dimensional parameters.

##### Reproducing kernel Hilbert spaces

Let $\mathcal{X}$ be a non-empty set and $\mathcal{H}$ a Hilbert space of
functions $f:\mathcal{X}\rightarrow\mathbb{R}$. Let
$\langle\cdot,\cdot\rangle_{\mathcal{H}}$ and $\|\cdot\|_{\mathcal{H}}$ denote
the inner product and norm on $\mathcal{H}$ respectively. Then $\mathcal{H}$
is called a reproducing kernel Hilbert space (RKHS) if there exists a
symmetric function $k:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ such
that $k(x,\cdot)\in\mathcal{H}$ for all $x\in\mathcal{X}$ and $\langle
f,k(x,\cdot)\rangle_{\mathcal{H}}=f(x)$ for all $f\in\mathcal{H}$ and
$x\in\mathcal{X}$. Every positive (semi-)definite kernel is the unique
reproducing kernel of an RKHS. We call a reproducing kernel $k$ _integrally
strictly positive definite_ (ISPD) if additionally for any $f\in\mathcal{H}$
with $0<\|f\|_{2}^{2}<\infty$ we have
$\int_{\mathcal{X}}f(x)k\left(x,x^{\prime}\right)f\left(x^{\prime}\right)\mathrm{d}x\mathrm{~{}d}x^{\prime}>0$.
See, e.g., Schölkopf and Smola (2002) for a comprehensive introduction.

Let $\mathcal{H}=\bigoplus_{i=1}^{m}\mathcal{H}_{i}$ denote the direct sum of
$m$ RKHS of universal kernels $k_{i}$ (Micchelli et al., 2006). The following
theorem which is based on Theorem 3.2 of Muandet et al. (2020) shows that the
RKHS corresponding to a universal ISPD kernel (e.g., Gaussian kernel) is
expressive enough to represent the conditional moment restriction (6) in terms
of a continuum of unconditional restrictions.

######  Theorem 3.11.

Let $\mathcal{H}=\bigoplus_{i=1}^{m}\mathcal{H}_{i}$ denote the direct sum of
$m$ RKHS unit balls $\mathcal{H}_{i}$ corresponding to ISPD kernels $k_{i}$,
$i=1,\ldots,m$. Let $P$ denote a distribution over random variables
$X\in\mathcal{X}$ and $Z\in\mathcal{Z}$ with marginal distributions $P_{X}$
and $P_{Z}$. Then

$\displaystyle E_{P}[\psi(X;\theta)|Z]=0\ \ P_{Z}\textrm{-a.s.},$ (14)

if and only if

$\displaystyle E_{P}[\psi(X;\theta)^{T}h(Z)]=0\ \ \forall h\in\mathcal{H}.$
(15)

With this result at hand, we can apply Theorems 3.8-3.10 to show that FGEL
combined with such an RKHS as instrument functions space, which we term Kernel
FGEL, provides an efficient estimator for CMR problems as formalized in the
following corollary.

######  Corollary 3.12.

The RKHS of a universal ISPD kernel satisfies the assumptions of Theorems
3.8-3.10. Thus, under Assumptions a)-j) of Theorem 3.8 Kernel FGEL provides a
$\frac{1}{\sqrt{n}}$-consistent, asymptotically normal and semi-parametrically
efficient estimator for conditional moment restriction problems.

Applying the representer theorem (Schölkopf et al., 2001) to the supremum over
the instrument functions $h$ in equation (13) allows us to represent the RKHS
function in terms of finite dimensional parameters
$\alpha_{r}\in\mathbb{R}^{n}$, $r=1,\ldots,m$, and yields a finite dimensional
and convex optimization problem as formalized by the following lemma.

######  Lemma 3.13.

Let $\mathcal{H}=\bigoplus_{i=1}^{m}\mathcal{H}_{i}$ be an RKHS corresponding
to $m$ universal kernels $k_{i}$, $i=1,\ldots,m$. Let
$K_{r}\in\mathbb{R}^{n\times n}$, $r=1,\ldots,m$ denote the kernel matrices
and let $\alpha=\\{\alpha_{r}\\}_{r=1}^{m}$ with
$\alpha_{r}\in\mathbb{R}^{n}$. Then the maximization over the instrument
functions in the FGEL objective (13) can be expressed as

$\displaystyle R_{\lambda_{n}}(\theta):=\max_{\alpha\in\widehat{A}_{\theta}}$
$\displaystyle\Bigg{\\{}\frac{1}{n}\sum_{i=1}^{n}\phi\left(v_{i}(\theta,\alpha)\right)-\frac{\lambda_{n}}{2}\sum_{r=1}^{m}\alpha_{r}^{T}K_{r}\alpha_{r}\Bigg{\\}},$

with $v_{i}=\sum_{r=1}^{m}(\alpha_{r}^{T}K_{r})_{i}\psi_{r}(x_{i};\theta)$ and
$\widehat{A}_{\theta}=\\{\alpha:v_{i}\in\operatorname{dom}(\phi),1\leq i\leq
n\\}$. The Kernel FGEL estimator is then defined as the solution of
$\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}R_{\lambda_{n}}(\theta)$.

We provide details on the optimization algorithm in Section A.2 of the
appendix.

###  3.5 Neural FGEL

As expressed by universal approximation theorems (e.g., (Yarotsky, 2017)),
neural networks can represent arbitrarily large function classes and have
shown state-of-the-art performance on related tasks (Hartford et al., 2017;
Lewis and Syrgkanis, 2018; Bennett et al., 2019). As such, they provide a
particularly interesting choice of instrument function class. Let
$h_{\omega}:\mathcal{Z}\rightarrow\mathbb{R}^{m}$ denote a feed-forward neural
network with parameters $\omega$. Then we can define the Neural FGEL estimator
as a saddle point of

$\displaystyle
G_{\lambda_{n}}(\theta,\omega):=\frac{1}{n}\sum_{i=1}^{n}\phi\left(\psi(x_{i};\theta)^{T}h_{\omega}(z_{i})\right)-\frac{\lambda_{n}}{2n}\sum_{i=1}^{n}\|h_{\omega}(z_{i})\|_{\mathbb{R}^{m}}^{2},$

where the regularization term penalizes the magnitude of the output as in
Dikkala et al. (2020) and Bennett and Kallus (2023). We leave the theoretical
analysis of the Neural FGEL estimator for future work.

###  3.6 Other Instrument Function Classes

FGEL estimators can be defined for arbitrary instrument function classes
$\mathcal{H}$ under mild conditions: Let $P$ denote a reference measure over
$X\in\mathcal{X}$, then we can place a class $\mathcal{H}$ of functions
$f:\mathcal{X}\rightarrow\mathbb{R}^{m}$ into the Hilbert space of square-
integrable functions $L^{2}(\mathcal{H},P)$ as long as any $f\in\mathcal{H}$
is bounded on any set with non-zero measure, which is a realistic assumption
for many model classes. The corresponding norm with respect to the empirical
measure is then given via
$\|h\|_{\mathcal{H}}^{2}=\frac{1}{n}\sum_{i=1}^{n}\|h(z_{i})\|_{\mathbb{R}^{m}}^{2}$.
If the underlying problem of interest is a conditional moment restriction
(instead of a general functional moment restriction), $\mathcal{H}$
additionally must be expressive enough such that an equivalence between the
conditional (6) and unconditional (7) formulations holds.

###  3.7 Choice of Divergence Function

In this section, we discuss various choices of divergences and establish
connections to existing methods. In the finite dimensional case, it is well
known that for any quadratic discrepancy function the GEL estimator coincides
with the continuous updating GMM (CUE) estimator (Newey and Smith, 2004). An
interesting special choice of divergence function is given below.

######  Proposition 3.14.

Choosing the GEL function as $\phi(v)=-(1\pm\frac{v}{2})^{2}$ and rescaling
the regularization parameter $\tilde{\lambda}_{n}=2\lambda_{n}$, the FGEL
estimator becomes equivalent to the solution of the optimization problem

$\displaystyle\min_{\theta\in\Theta}\sup_{h\in\mathcal{H}}\Big{\\{}E_{\hat{P}_{n}}[\psi(X;\theta)^{T}h(X)]-\frac{1}{4}E_{\hat{P}_{n}}\left[\left(\psi(X;\theta)^{T}h(X)\right)^{2}\right]-\frac{\tilde{\lambda}_{n}}{4}\|h\|_{\mathcal{H}}^{2}\Big{\\}}.$

This resembles the objective of the VMM estimator of Bennett and Kallus (2023)
with the only difference that the covariance term contains the decision
variable $\theta$ instead of a first-stage estimate $\tilde{\theta}$. In this
sense, with this special choice of divergence function our FGEL estimator and
the VMM estimator are related in the same way as the continuous updating
estimator (CUE) (2) and the optimally weighted 2-step GMM estimator (3). With
the kernel version of our FGEL estimator, we can carry out the optimization
over $h\in\mathcal{H}$ in closed form and similarly obtain a continuous
updating version of the Kernel VMM estimator.

A functional generalization of the original empirical likelihood estimator is
retrieved by setting $\phi(v)=-\log(1-v)$. The empirical likelihood estimator
has many desirable properties. It has been shown by Newey and Smith (2004)
that the ordinary EL estimator has the smallest higher order bias among the
family of GEL estimators (including GMM). Further, Corcoran (1998) shows that
confidence intervals constructed from the EL-based profile likelihood admit a
Bartlett correction which by a simple subtraction allows to reduce the
coverage error from $O(n^{-1})$ to $O(n^{-2})$. This property of the EL
framework is unique among the family of GEL estimators (Corcoran, 1998).

Using the GEL function corresponding to the Kullback-Leibler (KL) divergence
$\phi(v)=-e^{v}$ one obtains a functional generalization of the exponential
tilting estimator of Kitamura and Stutzer (1997) and Imbens et al. (1998)
which shows good empirical performance on many tasks (Imbens et al., 1998). In
contrast to the $\chi^{2}$-divergence, the KL-divergence enjoys great
popularity as a distributional divergence measure in machine learning (Blei et
al., 2017). Therefore, a functional moment restriction estimator based on the
KL-divergence instead of the dominating $\chi^{2}$-divergence (GMM) could be
of particular interest.

Figure 1: Estimation error over sample size for the heteroskedastic regression
experiment. Lines and shaded regions represent the MSE of the estimated
parameters and the standard error averaged over $70$ runs respectively. Figure
2: Comparison of different divergence functions. Lines and shaded regions
represent the MSE of the estimated parameters and the standard error averaged
over $70$ runs respectively.

##  4 Experiments

For all experiments we use radial basis function kernels
$k_{i}(x,x^{\prime})=\exp(-\gamma\|x-x^{\prime}\|^{2})$, $i=1,\ldots,m$ and
set the bandwidth parameter $\gamma$ via the common median heuristic
(Schölkopf and Smola, 2002; Garreau et al., 2017). If not stated otherwise, we
tune the remaining hyperparameters of all methods by evaluating the MMR
objective
$\ell(\theta)=1/n^{2}\sum_{i,j=1}^{n}\psi(x_{i};\theta)^{T}K_{ij}\psi(x_{j};\theta)$
(Zhang et al., 2023) on a validation set of the same size as the training set
(refer to Section A.3 of the appendix for details). We compare the performance
of our kernel- and neural network-based methods with ordinary least-squares
(LSQ), sieve minimum distance (SMD) (Ai and Chen, 2003), kernel maximum moment
restrictions (MMR) (Zhang et al., 2023) and the kernel- and neural network
versions of the variational method of moments (K-VMM and NN-VMM) (Bennett and
Kallus, 2023; Bennett et al., 2019) on two conditional moment restriction
problems. Code for reproducing the experimental results is available at
https://github.com/HeinerKremer/Functional-GEL.

###  4.1 Linear Regression under Heteroskedastic Noise

We define a simple data generating process for a one-dimensional estimation
problem. Let $\theta=1.7\in\mathbb{R}$ and

$\displaystyle y=x^{T}\theta+\varepsilon,\ \ \ \
x\sim\operatorname{Uniform}([-1.5,1.5]),$

where $\varepsilon$ describes heteroskedastic noise such that
$\varepsilon|x\sim\mathcal{N}(0,\sigma=5x^{2})$. We can formulate the
regression task as the conditional moment restriction $E[Y-X^{T}\theta|X]=0\
P_{X}\text{-a.s.}$. As $\varepsilon$ is a mean zero random variable, here, we
can simply use the mean squared error on a validation set as an unbiased
validation metric to tune the hyperparameters of all methods.

Figure 1 shows the mean-squared error (MSE) of the estimated parameters using
different versions of FGEL and other state-of-the-art estimators for
conditional moment restrictions in dependence on the sample size. Here we
treat the choice of divergence as an additional hyperparameter. We observe
that both our methods yield the lowest parameter MSE and even slightly
outperform the recently proposed state-of-the-art VMM estimator (Bennett and
Kallus, 2023). In Figure 2 we evaluate the effect of the divergence function.
We observe that while the average performance is largely independent of the
choice of divergence function, a comparison with the results shown in Figure 1
reveals that for any fixed sample the different divergences yield estimators
of different quality. Thus, treating the divergence as hyperparameter and
choosing the estimator with the lowest validation loss, allows us to exceed
the performance of the FGEL estimator with fixed divergence. As GMM-based
methods implicitly build on the $\chi^{2}$-divergence, this highlights an
advantage of our method which can leverage any $\varphi$-divergences. Note
that for any _fixed_ divergence function the performance of our FGEL
estimators are roughly on par with the corresponding VMM estimators.

###  4.2 Instrumental Variable Regression

We adopt a slightly modified version of the IV regression experiment of Lewis
and Syrgkanis (2018), which has also been used by Bennett et al. (2019) and
Zhang et al. (2023). Let the data generating process be given by

$\displaystyle y=f_{0}(x)+e+\delta,$ $\displaystyle x=z+e+\gamma,$
$\displaystyle z\sim\operatorname{Uniform}([-3,3]),$ $\displaystyle e\sim
N(0,1),$ $\displaystyle\gamma,\delta\sim N(0,0.1),$

where $f_{0}$ is picked from the following simple functions

$\displaystyle\operatorname{sin:}f_{0}(x)=\sin(x),$
$\displaystyle\operatorname{abs:}f_{0}(x)=|x|,$
$\displaystyle\operatorname{linear:}f_{0}(x)=x,$
$\displaystyle\operatorname{step:}f_{0}(x)=I_{\\{x\geq 0\\}}.$

We approximate $f_{0}$ by a shallow neural network $f_{\theta}(x)$ with 2
layers of $[20,3]$ units and leaky ReLU activation functions and base the
estimation on the conditional moment restrictions $E[Y-f_{\theta}(X)|Z]=0\ \
P_{Z}\text{-a.s.}$. As generally the true model is not contained in this model
class, this provides a typical case of model misspecification and theoretical
properties of the our method (and equally all baseline methods) for this
setting have yet to be developed (see Dikkala et al. (2020) for recent
progress in this direction). We use training and validation sets of size
$n=2000$ and evaluate the prediction error on a test set of $20000$ samples.
The results are visualized in Table 2. We observe that with the exception of
one task the FGEL and VMM estimators outperform all other baselines. Compared
to each other NN-FGEL seems to be preferable over NN-VMM but the kernel
versions of both methods exhibit similar performance without showing a clear
advantage of one over the other for this task.

Table 2: Prediction MSE for the instrumental variable task. Mean and standard
deviation of the mean are computed over 50 random runs and multiplied by $10$
for ease of presentation.

| LSQ | SMD | MMR | K-VMM | NN-VMM | K-FGEL | NN-FGEL
---|---|---|---|---|---|---|---
abs | $3.72\pm 0.30$ | $2.97\pm 0.97$ | $2.78\pm 0.60$ | $0.43\pm 0.15$ | $0.45\pm 0.10$ | $\bf{0.17\pm 0.01}$ | $0.23\pm 0.06$
step | $3.03\pm 0.03$ | $0.37\pm 0.04$ | $0.71\pm 0.03$ | $\bf{0.31\pm 0.01}$ | $0.41\pm 0.01$ | $0.41\pm 0.03$ | $0.34\pm 0.01$
sin | $3.28\pm 0.04$ | $\bf{1.01\pm 0.06}$ | $3.61\pm 0.07$ | $1.55\pm 0.12$ | $1.72\pm 0.11$ | $1.97\pm 0.16$ | $1.66\pm 0.12$
linear | $2.76\pm 0.06$ | $0.97\pm 0.72$ | $1.98\pm 0.38$ | $0.31\pm 0.06$ | $0.34\pm 0.05$ | $0.32\pm 0.05$ | $\bf{0.20\pm 0.03}$

Our experiments show that the FGEL estimator is a viable alternative to
previously proposed continuum method of moments estimators for conditional
moment restrictions and can surpass the previous state-of-the-art on some
tasks. However, further empirical evidence needs to be collected to verify its
predicted superior finite sample properties for infinitely many moment
restrictions. We leave a comprehensive experimental evaluation to future work.

##  5 Related Work

Learning with conditional or infinite dimensional moment restrictions
respectively has been an active field of research in econometrics and more
recently in machine learning. In the former context, seminal work on extending
the generalized method of moments to continua of moment restrictions has been
carried out by Carrasco and Florens (2000); Carrasco et al. (2007) by placing
the constraints in an RKHS. In the machine learning community, GMM-related
estimators have been developed by casting the infinite dimensional moment
restriction problem as a minimax game and representing the adversarial player
by an RKHS function (Zhang et al., 2023; Bennett and Kallus, 2023) or a
flexible neural network (Hartford et al., 2017; Lewis and Syrgkanis, 2018;
Dikkala et al., 2020; Bennett et al., 2019). While the neural network-based
methods often achieve good performance in practice, they generally are
computationally more expensive and lack the theoretical properties of
traditional GMM estimators. In contrast, Bennett and Kallus (2023)’s Kernel
VMM estimator comes with strong theoretical guarantees but results from a
2-step procedure and thus depends on an initial parameter estimate. As
discussed in Section 3.7, our framework contains a continuous updating version
of VMM as a special case but allows for using alternative $\varphi$-divergence
functions.

As an alternative to GMM estimation, sieve-based methods (Newey and Powell,
2003; Donald et al., 2003; Ai and Chen, 2003; Chen and Pouzo, 2012) address
conditional moment restrictions by growing the number of unconditional
restrictions with the sample size by manually selecting an increasing number
of basis functions. While these often come with desirable efficiency results,
in practice they can be hard to tune and computationally demanding (Bennett
and Kallus, 2023). Another line of work implicitly estimates optimal
instrument functions via a kernel-smoothed localized empirical likelihood
function (Tripathi and Kitamura, 2003; Kitamura et al., 2004). Their use of
kernels is different from our approach as we do not smooth the profile
divergence but use RKHS functions (and other function classes) as instrument
functions.

Several works extended the generalized empirical likelihood framework to
handle infinite dimensional moment restrictions and thus conditional moment
restrictions (Donald et al., 2003; Chaussé, 2012; Carrasco and Kotchoni,
2017). The GEL estimator of Chaussé (2012) is based on approximately imposing
a continuum of moment restrictions using a parameterized basis of functions
and solving a regularized version of the GEL first order conditions. While it
is theoretically closely related to our method, the regularization scheme and
computational approach differs from ours. Similarly, closely related to our
method is the regularized GEL estimator of Carrasco and Kotchoni (2017), which
is defined via a set of optimality conditions and solved using a procedure
motivated by the Three-Steps Euclidean Likelihood procedure of Antoine et al.
(2007). In contrast to these methods, our estimator is defined as a saddle
point of an objective function and thus benefits from recent advances in mini-
max optimization (Daskalakis et al., 2017; Lin et al., 2020). To the best of
our knowledge, our work is the first to combine GEL estimation with modern
machine learning and in particular kernel methods and neural networks.

##  6 Conclusion

Several long-established problems in machine learning can naturally be
expressed as a risk minimization problem. On the other hand, emerging areas
such as causal inference, algorithmic decision making, and robust learning
often involve problems that are formulated as (potentially infinite) moment
restrictions and require different algorithmic frameworks for estimation and
inference. Recent works have advanced this development by combining classical
techniques from econometrics such as generalized method of moments (GMM) with
modern machine learning models such as deep neural networks and kernel
machines. Likewise, our work contributes to this endeavour by equipping the
more general generalized empirical likelihood (GEL) framework with such
powerful models. While the econometrics community enjoys the new class of
algorithms, we believe the machine learning community will likewise benefit
from new perspectives on causal inference and robust learning which will be
explored in future works.

This paper laid the theoretical foundation of the functional GEL framework,
but there remain open questions that impede real-world applications. Firstly,
more efficient optimization procedures need to be developed that allow for
large scale applications. Secondly, theoretical properties of the framework
with specific function classes need to be explored. Lastly, the framework
needs to be tested for the training of more complex models for real-world
applications (e.g. robust learning). Our goal is to address some of these
problems in future work.

## Acknowledgements

We thank Simon Buchholz and Yassine Nemmour for helpful discussions and
feedback on an earlier version of the manuscript. This work was supported by
the German Federal Ministry of Education and Research (BMBF): Tübingen AI
Center, FKZ: 01IS18039B.

## References

  * Ai and Chen (2003) C. Ai and X. Chen.  Efficient estimation of models with conditional moment restrictions containing unknown functions.  _Econometrica_ , 71(6):1795–1843, 2003.
  * Angrist and Pischke (2008) J. D. Angrist and J.-S. Pischke.  _Mostly harmless econometrics_.  Princeton university press, 2008.
  * Antoine et al. (2007) B. Antoine, H. Bonnal, and E. Renault.  On the efficient use of the informational content of estimating equations: Implied probabilities and euclidean empirical likelihood.  _Journal of Econometrics_ , 138(2):461–487, 2007\.
  * Athey et al. (2019) S. Athey, J. Tibshirani, and S. Wager.  Generalized random forests.  _The Annals of Statistics_ , 47(2):1148–1178, 04 2019.
  * Ben-Tal et al. (2013) A. Ben-Tal, D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen.  Robust solutions of optimization problems affected by uncertain probabilities.  _Management Science_ , 59(2):341–357, 2013.
  * Bennett and Kallus (2020) A. Bennett and N. Kallus.  Efficient policy learning from surrogate-loss classification reductions.  In _International Conference on Machine Learning_ , pages 788–798. PMLR, 2020.
  * Bennett and Kallus (2023) A. Bennett and N. Kallus.  The variational method of moments.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 85(3):810–841, 2023.
  * Bennett et al. (2019) A. Bennett, N. Kallus, and T. Schnabel.  Deep generalized method of moments for instrumental variable analysis.  _Advances in neural information processing systems_ , 32, 2019.
  * Bennett et al. (2021) A. Bennett, N. Kallus, L. Li, and A. Mousavi.  Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1999–2007. PMLR, 2021.
  * Bernstein (2009) D. S. Bernstein.  Matrix mathematics.  In _Matrix Mathematics_. Princeton university press, 2009.
  * Bierens (1982) H. J. Bierens.  Consistent model specification tests.  _Journal of Econometrics_ , 20(1):105–134, 1982\.
  * Blei et al. (2017) D. M. Blei, A. Kucukelbir, and J. D. McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American Statistical Association_ , 112(518):859–877, Apr 2017.
  * Borwein (1993) J. M. Borwein.  On the failure of maximum entropy reconstruction for fredholm equations and other infinite systems.  _Mathematical programming_ , 61(1):251–261, 1993\.
  * Carrasco and Florens (2000) M. Carrasco and J.-P. Florens.  Generalization of gmm to a continuum of moment conditions.  _Econometric Theory_ , 16(6):797–834, 2000.
  * Carrasco and Kotchoni (2017) M. Carrasco and R. Kotchoni.  Regularized generalized empirical likelihood estimators.  Technical report, Technical report, 2017.
  * Carrasco et al. (2007) M. Carrasco, M. Chernov, J.-P. Florens, and E. Ghysels.  Efficient estimation of general dynamic models with a continuum of moment conditions.  _Journal of econometrics_ , 140(2):529–573, 2007\.
  * Chamberlain (1987) G. Chamberlain.  Asymptotic efficiency in estimation with conditional moment restrictions.  _Journal of Econometrics_ , 34(3):305–334, 1987\.
  * Chaussé (2012) P. Chaussé.  Generalized empirical likelihood for a continuum of moment conditions.  _Technical Report_ , 2012.
  * Chen and Pouzo (2012) X. Chen and D. Pouzo.  Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals.  _Econometrica_ , 80(1):277–321, 2012.
  * Chen et al. (2022) Y. Chen, L. Xu, C. Gulcehre, T. L. Paine, A. Gretton, N. De Freitas, and A. Doucet.  On instrumental variable regression for deep offline policy evaluation.  _The Journal of Machine Learning Research_ , 23(1):13635–13674, 2022.
  * Chernozhukov et al. (2016) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins.  Double/debiased machine learning for treatment and causal parameters.  _arXiv preprint arXiv:1608.00060_ , 2016.
  * Chernozhukov et al. (2017) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, and W. Newey.  Double/debiased/neyman machine learning of treatment effects.  _American Economic Review_ , 107(5):261–65, 2017\.
  * Chernozhukov et al. (2018) V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins.  Double/debiased machine learning for treatment and structural parameters, 2018.
  * Corcoran (1998) S. A. Corcoran.  Bartlett adjustment of empirical discrepancy statistics.  _Biometrika_ , 85(4):967–972, 12 1998.
  * Cressie and Read (1984) N. Cressie and T. R. C. Read.  Multinomial goodness-of-fit tests.  _Journal of the Royal Statistical Society. Series B (Methodological)_ , 46(3):440–464, 1984.
  * Danskin (1966) J. M. Danskin.  The theory of max-min, with applications.  _SIAM Journal on Applied Mathematics_ , 14(4):641–664, 1966.
  * Daskalakis et al. (2017) C. Daskalakis, A. Ilyas, V. Syrgkanis, and H. Zeng.  Training gans with optimism.  _arXiv preprint arXiv:1711.00141_ , 2017.
  * Dikkala et al. (2020) N. Dikkala, G. Lewis, L. Mackey, and V. Syrgkanis.  Minimax estimation of conditional moment models.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 12248–12262. Curran Associates, Inc., 2020.
  * Donald et al. (2003) S. Donald, G. Imbens, and W. Newey.  Empirical likelihood estimation and consistent tests with conditional moment restrictions.  _Journal of Econometrics_ , 117:55–93, 11 2003.
  * Duchi et al. (2021) J. C. Duchi, P. W. Glynn, and H. Namkoong.  Statistics of robust optimization: A generalized empirical likelihood approach.  _Mathematics of Operations Research_ , 46(3):946–969, 2021.
  * Garreau et al. (2017) D. Garreau, W. Jitkrittum, and M. Kanagawa.  Large sample analysis of the median heuristic.  _arXiv preprint arXiv:1707.07269_ , 2017.
  * Goodfellow et al. (2014) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Greenfeld and Shalit (2020) D. Greenfeld and U. Shalit.  Robust learning with the hilbert-schmidt independence criterion.  In _International Conference on Machine Learning_ , pages 3759–3768. PMLR, 2020.
  * Hall (2004) A. Hall.  _Generalized method of moments_.  Wiley Online Library, 2004.
  * Hansen (1982) L. P. Hansen.  Large sample properties of generalized method of moments estimators.  _Econometrica_ , 50(4):1029–1054, 1982.
  * Hansen et al. (1996) L. P. Hansen, J. Heaton, and A. Yaron.  Finite-sample properties of some alternative gmm estimators.  _Journal of Business & Economic Statistics_, 14(3):262–280, 1996.
  * Hartford et al. (2017) J. Hartford, G. Lewis, K. Leyton-Brown, and M. Taddy.  Deep iv: A flexible approach for counterfactual prediction.  In _International Conference on Machine Learning_ , pages 1414–1423. PMLR, 2017.
  * Heinze-Deml and Meinshausen (2021) C. Heinze-Deml and N. Meinshausen.  Conditional variance penalties and domain shift robustness.  _Machine Learning_ , 110(2):303–348, 2021.
  * Imbens et al. (1998) G. W. Imbens, R. H. Spady, and P. Johnson.  Information theoretic approaches to inference in moment condition models.  _Econometrica_ , 66(2):333–357, 1998.
  * Kallus and Uehara (2020) N. Kallus and M. Uehara.  Double reinforcement learning for efficient off-policy evaluation in markov decision processes.  _J. Mach. Learn. Res._ , 21(167):1–63, 2020.
  * Kitamura and Stutzer (1997) Y. Kitamura and M. Stutzer.  An information-theoretic alternative to generalized method of moments estimation.  _Econometrica_ , 65(4):861–874, 1997.
  * Kitamura et al. (2004) Y. Kitamura, G. Tripathi, and H. Ahn.  Empirical likelihood-based inference in conditional moment restriction models.  _Econometrica_ , 72(6):1667–1714, 2004.
  * Kosorok (2008) M. R. Kosorok.  _Introduction to empirical processes and semiparametric inference_ , volume 61.  Springer, 2008.
  * Kremer et al. (2023) H. Kremer, Y. Nemmour, B. Schölkopf, and J.-J. Zhu.  Estimation beyond data reweighting: Kernel method of moments.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 17745–17783. PMLR, 23–29 Jul 2023.
  * Lam (2019) H. Lam.  Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization.  _Operations Research_ , 67(4):1090–1105, 2019\.
  * Lewis and Syrgkanis (2018) G. Lewis and V. Syrgkanis.  Adversarial generalized method of moments.  _arXiv preprint arXiv:1803.07164_ , 2018.
  * Lin et al. (2020) T. Lin, C. Jin, and M. Jordan.  On gradient descent ascent for nonconvex-concave minimax problems.  In _International Conference on Machine Learning_ , pages 6083–6093. PMLR, 2020.
  * Micchelli et al. (2006) C. Micchelli, Y. Xu, and H. Zhang.  Universal kernels.  _Mathematics_ , 7, 12 2006.
  * Muandet et al. (2020) K. Muandet, W. Jitkrittum, and J. Kübler.  Kernel conditional moment test via maximum moment restriction.  In _Conference on Uncertainty in Artificial Intelligence_ , pages 41–50. PMLR, 2020.
  * Newey and Smith (2002) W. Newey and R. Smith.  Asymptotic bias and equivalence of gmm and gel estimators.  _Econometrica_ , 72, 10 2002.
  * Newey (1993) W. K. Newey.  Efficient estimation of models with conditional moment restrictions.  In _Econometrics_ , volume 11 of _Handbook of Statistics_ , pages 419–454. Elsevier, 1993.
  * Newey and Powell (2003) W. K. Newey and J. L. Powell.  Instrumental variable estimation of nonparametric models.  _Econometrica_ , 71(5):1565–1578, 2003.
  * Newey and Smith (2004) W. K. Newey and R. J. Smith.  Higher order properties of gmm and generalized empirical likelihood estimators.  _Econometrica_ , 72(1):219–255, 2004.
  * Owen (1990) A. Owen.  Empirical likelihood ratio confidence regions.  _The Annals of Statistics_ , 18(1):90–120, 1990\.
  * Owen (1988) A. B. Owen.  Empirical likelihood ratio confidence intervals for a single functional.  _Biometrika_ , 75(2):237–249, 1988.
  * Owen (2001) A. B. Owen.  _Empirical likelihood_.  Chapman and Hall/CRC, 2001.
  * Qin and Lawless (1994) J. Qin and J. Lawless.  Empirical likelihood and general estimating equations.  _The Annals of Statistics_ , 22(1):300–325, 1994\.
  * Rothenhäusler et al. (2021) D. Rothenhäusler, N. Meinshausen, P. Bühlmann, and J. Peters.  Anchor regression: Heterogeneous data meet causality.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 83(2):215–246, 2021.
  * Schölkopf and Smola (2002) B. Schölkopf and A. J. Smola.  _Learning with kernels: support vector machines, regularization, optimization, and beyond_.  MIT press, 2002.
  * Schölkopf et al. (2001) B. Schölkopf, R. Herbrich, and A. J. Smola.  A generalized representer theorem.  In _Computational Learning Theory_ , pages 416–426, 2001.
  * Smith (1997) R. J. Smith.  Alternative semi-parametric likelihood approaches to generalised method of moments estimation.  _The Economic Journal_ , 107(441):503–519, 1997\.
  * Smith (2005) R. J. Smith.  Local GEL methods for conditional moment restrictions.  Technical report, cemmap working paper, 2005.
  * Smith (2011) R. J. Smith.  GEL criteria for moment condition models.  _Econometric Theory_ , 27(6):1192–1235, 2011\.
  * Tang and Leng (2010) C. Y. Tang and C. Leng.  Penalized high-dimensional empirical likelihood.  _Biometrika_ , 97(4):905–920, 2010.
  * Tripathi and Kitamura (2003) G. Tripathi and Y. Kitamura.  Testing conditional moment restrictions.  _The Annals of Statistics_ , 31(6):2059–2095, 2003.
  * Yarotsky (2017) D. Yarotsky.  Error bounds for approximations with deep relu networks.  _Neural Networks_ , 94:103–114, 2017.
  * Zeidler (2012) E. Zeidler.  _Applied functional analysis: applications to mathematical physics_ , volume 108.  Springer Science & Business Media, 2012.
  * Zhang et al. (2023) R. Zhang, M. Imaizumi, B. Schölkopf, and K. Muandet.  Instrumental variable regression via kernel maximum moment loss.  _Journal of Causal Inference_ , 11(1):20220073, 2023.

##  Appendix A Additional Information

###  A.1 Distributional Robustness of FGEL

It is well-known that the profile divergence is a dual formulation to the
distributionally robust optimization (DRO) formulation (Lam, 2019; Duchi et
al., 2021). In the context of this paper, one can show that
$R_{\lambda_{n}}(\theta)\leq\rho$ if and only if

$\lambda_{n}\geq\inf_{P\in\mathcal{P}}\|E_{P}[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}\;\
\mathrm{s.t.}\,D_{\varphi}(P||\hat{P}_{n})\leq\rho.$

However, we do not simply rely on the divergence-ball centered at the
empirical data distribution $\\{P|\ D_{\varphi}(P||\hat{P}_{n})\leq\rho\\}$
(referred to as an ambiguity set in the DRO literature) for robustness. Since
that robustness is often used to account for the statistical error due to
finite samples. Instead, we are concerned with a second and stronger layer of
robustness.

First, note that the quantity $E_{P}[\Psi(X,Z;\theta)]$ is used to approximate
the conditional moment constraint in our original formulation (8). Since the
goal of FGEL is to satisfy the the conditional moment restrictions
$E_{P}[\psi(X;\theta)\,|\,Z]=0$ almost everywhere in the domain, in robust
optimization terms, we are robustifying against the instrument $Z$. The
instrument $Z$ can create much stronger distribution shifts in the data-
generating process than the mere statistical fluctuation described by
divergence-ball-based DRO works following Ben-Tal et al. (2013) and Duchi et
al. (2021). We leave an alternative DRO algorithm against such strong
distribution shifts for future work.

From another perspective, our method can also be seen as enforcing
independence between $Z$ and the moment restriction, e.g., for IV regression
the residual $\psi(X;\theta)=Y-f_{\theta}(X)$. Intuitively, we want the
residual $Y-f_{\theta}(X)$ to be small and invariant to transformations of the
$Z$ variable (marginal shift). This kind of robust learning strategy has also
been studied in works by Greenfeld and Shalit (2020); Rothenhäusler et al.
(2021); Heinze-Deml and Meinshausen (2021).

###  A.2 Computing the FGEL Estimator

Problem (13) is generally a non-convex-convex min-max problem in the parameter
$\theta$ and function $h$. Let $h=h_{\alpha}$ be described by a finite
dimensional set of parameters $\alpha\in A$, which is the case, e.g., for
neural network function classes or RKHS after using a representer theorem.
Furthermore let the set of parameters $A$ be compact. If additionally the
parameterization leaves the convexity of the inner problem intact (e.g., in
the case of kernel-FGEL) we can use a simplified version of Danskin’s theorem
(Danskin, 1966) to compute gradients of
$R_{\lambda_{n}}(\theta):=\sup_{h\in\widehat{\mathcal{H}}_{\theta}}G_{\lambda_{n}}(\theta,h)$
in a principled way.

######  Lemma A.1 (Danskin).

Let $\hat{h}(\theta)$ denote the solution of the inner convex optimization
over $h\in\mathcal{H}_{\theta}$ such that
$\hat{h}(\theta)=\operatorname*{arg\,max}_{h\in\widehat{\mathcal{H}}_{\theta}}G_{\lambda_{n}}(\theta,h)$.
Then the gradient of the profile divergence $R_{\lambda_{n}}(\theta)$ with
respect to the parameters $\theta\in\Theta$ is given by

$\nabla R_{\lambda_{n}}(\theta)=\nabla
G_{\lambda_{n}}(\theta,\hat{h}(\theta)).$

Therefore, we can adopt a gradient-based strategy for the outer optimization
problem over $\theta$ using in each step the gradient estimate obtained from
the solution of the inner maximization over $h$. Depending on the GEL function
$\phi$ the optimization of both, the outer and inner problem can then be
solved efficiently with an off-the-shelf solver e.g. using LBFGS (cf.
Algorithm 1).

Algorithm 1 Kernel-FGEL

Input: data $(x_{i},y_{i},z_{i})$, hyperparameter $\lambda$

while not converged do

while not converged do

$\alpha\leftarrow\operatorname{LBFGS}(G_{\lambda}(\theta,h_{\alpha}))$

end while

$\theta\leftarrow\operatorname{LBFGS}(G_{\lambda}(\theta,h_{\alpha}))$

end while

Output: Parameter estimate $\theta$

For the case of a neural network instrument function classes, we build on the
recent progress in mini-max optimization and employ the optimistic Adam
optimizer (Daskalakis et al., 2017) which has been developed to solve similar
saddle point problems for training generative adversarial networks (Goodfellow
et al., 2014) (cf. Algorithm 2). Implementations of both approaches are
available under https://github.com/HeinerKremer/Functional-GEL.

Algorithm 2 Neural-FGEL

Input: data $(x_{i},y_{i},z_{i})$, hyperparameter $\lambda$

while not converged do

$\alpha\leftarrow\operatorname{OAdam}(G_{\lambda}(\theta,h_{\alpha}))$

$\theta\leftarrow\operatorname{OAdam}(G_{\lambda}(\theta,h_{\alpha}))$

end while

Output: Parameter estimate $\theta$

###  A.3 Hyperparameter selection

Tuning the hyperparameter of our method, i.e., the regularization parameter
$\lambda_{n}$ (and, e.g., learning rates) requires a data-driven performance
measure of the obtained model parameters. We know that for the true
distribution $P_{0}$ and true parameter $\theta_{0}$ we obtain
$\|E_{P_{0}}[\Psi(X,Z;\theta_{0})]\|_{\mathcal{H}^{\ast}}^{2}=0$. Let $\beta$
denote the set of hyperparameters and $\hat{\theta}(\beta)$ the corresponding
solution to (13). Then we can define a performance measure of the solution
candidate $\hat{\theta}(\beta)$ as
$\ell(\beta)=\|E_{P_{0}}[\Psi(X,Z;\hat{\theta}(\beta))]\|_{\mathcal{H}^{\ast}}^{2}$.
As we do not have access to the true distribution $P_{0}$ we can define a
natural surrogate loss $\hat{\ell}$ using a validation set with empirical
distribution $\hat{P}_{\text{val}}$ as

$\displaystyle\hat{\ell}(\beta)=\|E_{\hat{P}_{\text{val}}}[\Psi(X,Z;\hat{\theta}(\beta))]\|_{\mathcal{H}^{\ast}}^{2}$
(16)

Choosing $\mathcal{H}$ as an RKHS, this can be expressed as the kernel maximum
of moment restriction objective of Muandet et al. (2020) and Zhang et al.
(2023) evaluated on the validation data as shown by the following lemma.

######  Lemma A.2.

Let $\\{x_{i},z_{i}\\}_{i=1}^{n}$ denote the validation data and define
$\boldsymbol{\psi}_{j}(\boldsymbol{x};\theta)=\operatorname{vec}(\\{\psi_{j}(x_{i};\theta)\\}_{i=1}^{n})$.
Let $K_{j}$ denote the kernel Gram matrix with entries
$\left(K_{j}\right)_{pq}=k_{j}(z_{p},z_{q})$, $p,q=1,\ldots,n$,
$j=1,\ldots,m$. Then we can express (16) as

$\displaystyle\hat{\ell}(\beta)=\frac{1}{n^{2}}\sum_{j=1}^{m}\boldsymbol{\psi}_{j}(\boldsymbol{x};\theta)^{T}K_{j}\boldsymbol{\psi}_{j}(\boldsymbol{x};\theta).$

Here we assume that possible hyperparameters of the kernel are already set via
commonly employed heuristics like the median heuristic (Schölkopf and Smola,
2002; Garreau et al., 2017) for the kernel bandwidth and only tune the
remaining parameters of our method.

##  Appendix B Proofs

###  B.1 Preliminaries

For ease of notation we define some expressions first. Define
$\Psi_{i}(\theta):=\Psi(x_{i},z_{i};\theta)$ and denote
$\phi_{i}(v)=\frac{d^{i}}{(dv)^{i}}\phi(v)$ and $\phi_{i}=\phi_{i}(0)$.
Without loss of generality we assume that $\phi_{1}(0)=\phi_{2}(0)=-1$, as any
$\phi$ with $\phi_{1}\neq 0$ and $\phi_{2}<0$ can be rescaled to achieve this
(see Newey and Smith (2004)). Define the empirical objective as
$\widehat{G}_{\lambda_{n}}(\theta,h)=\sum_{i=1}^{n}\phi(\Psi(x_{i},z_{i};\theta)(h))-\lambda\|h\|_{\mathcal{H}}^{2}$
and the empirical constraint set as
$\widehat{\mathcal{H}}_{n}(\theta)=\\{h\in\mathcal{H}:\Psi(x_{i},z_{i};\theta)(h)\in\operatorname{dom}(\phi)\
\ \forall(x_{i},z_{i}),\ i=1,\ldots,n\\}$. Throughout the proofs we will make
use of functional derivatives and a functional version of Taylor’s theorem
with Lagrange remainder, which we define and state next, respectively.

######  Definition B.1 (Functional Derivative).

Let $\mathcal{H}$ be a vector space of functions. For a functional
$G:\mathcal{H}\rightarrow\mathbb{R}$ and a pair of functions
$h,\tilde{h}\in\mathcal{H}$, we define the derivative operator
$D_{h}G(h)[\tilde{h}]=\left.\frac{d}{dt}G(h+t\tilde{h})\right|_{t=0}$.
Likewise, we define

$\displaystyle
D_{h}^{k}G(h)\left[h_{1},\ldots,h_{k}\right]=\left.\frac{\partial^{k}}{\partial
t_{1}\ldots\partial
t_{k}}G\left(h+t_{1}h_{1}+\ldots+t_{k}h_{k}\right)\right|_{t_{1}=\cdots=t_{k}=0}.$

Similarly, when considering a function of a vector-valued parameter,
$G:\Theta\subseteq\mathbb{R}^{p}\rightarrow\mathbb{R}$, we denote the $k$-th
standard directional derivative at $\theta\in\Theta$ as
$D^{k}_{\theta}G(\theta)(\theta_{1},\ldots,\theta_{k})$. Alternatively we use
$\nabla_{\theta}G(\theta)$ to denote a row vector in $\mathbb{R}^{p}$ (the
gradient) and $\nabla_{\theta^{T}}G(\theta)$ to denote the corresponding
column vector in the dual space.

######  Proposition B.2 (Taylor’s theorem).

Let $G:\mathcal{H}\rightarrow\mathbb{R}$, where $\mathcal{H}$ is a vector
space of functions. For any $h,h^{\prime}\in\mathcal{H}$, if $t\mapsto
G\left(th+(1-t)h^{\prime}\right)$ is $(k+1)$-times differentiable over an open
interval containing $[0,1]$, then there exists
$\bar{h}\in\operatorname{conv}\left(\left\\{h,h^{\prime}\right\\}\right)$ such
that

$\displaystyle G\left(h^{\prime}\right)=$ $\displaystyle\
G(h)+\sum_{i=1}^{k}\frac{1}{i!}D_{h}^{i}G(h)[\underbrace{h^{\prime}-h,\ldots,h^{\prime}-h}_{i\text{
times }}]$
$\displaystyle+\frac{1}{(k+1)!}D_{h}^{k+1}G(\bar{h})[\underbrace{h^{\prime}-h,\ldots,h^{\prime}-h}_{k+1\text{
times }}].$

Equally, using the notation of Definition B.1 the same result holds for
functions of vector-valued parameters
$G:\Theta\subseteq\mathbb{R}^{p}\rightarrow\mathbb{R}$.

Our duality result builds on Theorem 3.1 of Borwein (1993). For completeness,
we will state it here adapted to our notation. Note that while the theorem is
already closely related to our result, a direct application of the theorem to
our case is impeded as we additionally need to take into account the
normalization constraint for $p$, i.e., $\sum_{i=1}^{n}p_{i}=1$.

######  Proposition B.3 (Borwein’s theorem).

For the problem

$\displaystyle
P=\inf_{p\in\mathbb{R}^{n}}\sum_{i=1}^{n}\frac{1}{n}f(np_{i})\quad\mathrm{s.t.}\quad\|\frac{1}{n}\sum_{i=1}^{n}p_{i}\Psi(x_{i},z_{i};\theta)\|_{\mathcal{H}^{\ast}}=0$
(17)

where we assume the infimum is attained (when finite), consider for
$\lambda>0$ the relaxed problem

$\displaystyle
P_{\lambda}=\min_{p\in\mathbb{R}^{n}}\sum_{i=1}^{n}\frac{1}{n}f(np_{i})\quad\mathrm{s.t.}\quad\|\frac{1}{n}\sum_{i=1}^{n}p_{i}\Psi(x_{i},z_{i};\theta)\|_{\mathcal{H}^{\ast}}\leq\lambda.$
(18)

Then the value $P_{\lambda}$ equals the value of the dual program

$\displaystyle
D_{\lambda}=\max_{h\in\mathcal{H}}-\frac{1}{n}\sum_{i=1}^{n}f^{\ast}(\Psi(x_{i},z_{i};\theta)(h))-\lambda\|h\|_{\mathcal{H}},$
(19)

and the unique optimal solution of (18) is given by

$\displaystyle(p_{\lambda})_{i}=\left(\frac{d}{dv}f^{\ast}\right)\left(\Psi(x_{i},z_{i};\theta)(\hat{h})\right),\quad
i=1,\ldots,n,$

where $\hat{h}$ is any solution of (19). Moreover, as $\lambda\rightarrow 0$,
$p_{\lambda}$ converges in mean to the unique solution of (17) and
$P_{\lambda}\rightarrow P$.

For the proofs of the asymptotic properties of our estimator, we need the
following results.

######  Lemma B.4 (Corollary 9.31, Kosorok (2008)).

Let $\mathcal{F}$ and $\mathcal{G}$ be Donsker classes of functions. Then
$\mathcal{F}+\mathcal{G}$ is Donsker. Further if additionally $\mathcal{F}$
and $\mathcal{G}$ are uniformly bounded, then $\mathcal{F}\cdot\mathcal{G}$ is
Donsker.

######  Lemma B.5 (Lemma 18, Bennett and Kallus (2023)).

Suppose that $\mathcal{G}$ is a class of functions of the form
$g:\Xi\rightarrow\mathbb{R}$, and that $\mathcal{G}$ is $P$-Donsker in the
sense of Kosorok (2008). Then we have

$\displaystyle\sup_{g\in\mathcal{G}}E_{\hat{P}_{n}}[g(\xi)]-E[g(\xi)]=O_{p}(n^{-1/2}).$

##### Proof of Theorem 3.1

###### Proof.

The proof follows almost directly from application of Proposition B.3 by
taking into account the additional constraint $\sum_{i=1}^{n}p_{i}=1$.

The dual problem can be derived by introducing Lagrange parameters $\nu>0$ and
$\mu\in\mathbb{R}$ and defining the Lagrangian

$\displaystyle
L(\theta,p,\mu,\nu)=\sum_{i=1}^{n}\frac{1}{n}f(np_{i})-\mu\left(\sum_{i=1}^{n}p_{i}-1\right)+\nu\left(\|\sum_{i=1}^{n}p_{i}\Psi(x_{i},z_{i};\theta)\|_{\mathcal{H}^{\ast}}-\lambda\right).$

Using the definition of the dual norm and the fact that trivially
$\lambda=\max_{\|h\|=1}\|h\|\lambda$, we have

$\displaystyle
L=\sum_{i=1}^{n}\frac{1}{n}f(np_{i})-\mu\left(\sum_{i=1}^{n}p_{i}-1\right)+\sup_{\|\tilde{h}\|_{\mathcal{H}}=1}\left(\sum_{i=1}^{n}\langle\nu\tilde{h},p_{i}\Psi(x_{i},z_{i};\theta)\rangle-\|\nu\tilde{h}\|_{\mathcal{H}}\lambda\right).$

By defining new dual Lagrange parameters $h=\nu\tilde{h}\in\mathcal{H}$, we
thus obtain

$\displaystyle
L(\theta,p,\mu,h)=\sum_{i=1}^{n}\frac{1}{n}f(np_{i})-\mu\left(\sum_{i=1}^{n}p_{i}-1\right)+\sum_{i=1}^{n}\langle
h,p_{i}\Psi(x_{i},z_{i};\theta)\rangle-\|h\|_{\mathcal{H}}\lambda.$

Now, redefining $p_{i}\rightarrow np_{i}$ and optimizing the Lagrangian with
respect to p we get

$\displaystyle\min_{p}\Big{\\{}\mu-\frac{1}{n}\sum_{i=1}^{n}\Big{[}(\mu-\Psi(x_{i},z_{i};\theta)(h))p_{i}-f(p_{i})\Big{]}-\lambda\|h\|_{\mathcal{H}}\Big{\\}}$
$\displaystyle=$ $\displaystyle\
\mu-\frac{1}{n}\sum_{i=1}^{n}\max_{p_{i}}\Big{\\{}(\mu-\Psi(x_{i},z_{i};\theta)(h))p_{i}-f(p_{i})\Big{\\}}-\lambda\|h\|_{\mathcal{H}}$
$\displaystyle=$ $\displaystyle\
\mu-\frac{1}{n}\sum_{i=1}^{n}f^{\ast}(\mu-\Psi(x_{i},z_{i};\theta)(h))-\lambda\|h\|_{\mathcal{H}},$

where we used the definition of the Legrendre-Fenchel (convex) conjugate
function $f^{\ast}(v)=\sup_{x}\langle v,x\rangle-f(x)$. As for any
$h\in\mathcal{H}$, $-h\in\mathcal{H}$, we can redefine $h\rightarrow-h$ and
finally obtain the result. Finally from Proposition B.3 it follows that strong
duality holds and the unique minimizer of the primal problem is given by

$p_{i}=\left(\frac{d}{dv}f^{\ast}\right)\left(\Psi(x_{i},z_{i};\theta)(\hat{h})+\hat{\mu}\right),\quad
i=1,\ldots,n,$

where $\hat{h}$, $\hat{\mu}$ are any solutions of the dual problem. ∎

###  B.2 Asymptotic Properties of FGEL

####  B.2.1 Proof of Theorem 3.6 (Consistency)

For the proof of Lemma B.7 we will need the following result whose proof
closely follows a similar result for vector-valued moment restrictions of Owen
(1990) and Kitamura et al. (2004) (Lemma D.2):

######  Lemma B.6.

Let X be a random variable taking values in
$\mathcal{X}\subseteq\mathbb{R}^{r}$. For a functional
$\Psi:\mathcal{X}\times\Theta\times\mathcal{H}\rightarrow\mathbb{R}$ with
$E\left[\left(\sup_{\theta\in\Theta}\|\Psi(X;\theta)\|_{\mathcal{H}^{\ast}}\right)^{m}\right]<\infty$,
it follows that $\max_{1\leq j\leq
n}\sup_{\theta\in\Theta}\left\|\Psi\left(x_{j};\theta\right)\right\|_{\mathcal{H}^{\ast}}=o\left(n^{1/m}\right)$
with probability 1.

###### Proof.

For ease of notation define the random variable
$Y:=\sup_{\theta\in\Theta}\left\|\Psi\left(X;\theta\right)\right\|_{\mathcal{H}^{\ast}}$
and let for $i\in\mathbb{N}$, $Y_{i}$ denote independent copies of $Y$. Then
as $E[Y^{m}]<\infty$, we must have that
$\sum_{i=1}^{\infty}P(Y_{i}^{m}>n)<\infty$ or equivalently
$\sum_{i=1}^{\infty}P(Y_{i}>n^{1/m})<\infty$. Hence by the Borel-Cantelli
Lemma the event $Y_{i}>n^{1/m}$ happens only finitely often with probability
$1$ which likewise implies $Z_{n}:=\max_{1\leq i\leq n}Y_{i}>n^{1/m}$ happens
only finitely often with probability $1$. By the same argument the event
$Z_{n}>\epsilon n^{1/m}$ happens only finitely often for any $\epsilon>0$ and
thus

$\limsup Z_{n}/n^{1/m}\leq\epsilon$

with probability $1$ and thus $Z_{n}=o(n^{1/m})$ with probability $1$. ∎

The following Lemma shows that if we constrain the space of the dual parameter
to a ball of radius $\zeta$ with $1/\nu<\zeta<1/2-\xi$, the largest value the
empirical moment functional evaluated on the dual parameter can take converges
to zero in probability. Furthermore any such ball is contained in the
empirical constraint set $\widehat{\mathcal{H}}(\theta)=\\{h\in\mathcal{H}:\
\psi(x_{i};\theta)^{T}h(z_{i})\in\operatorname{dom}(\phi),\ 1\leq i\leq n\\}$.

######  Lemma B.7.

Let the assumptions of Theorem 3.6 be satisfied, then for any $\zeta$ with
$1/\nu<\zeta<1/2$ define
$\mathcal{H}_{n}=\\{h\in\mathcal{H}:\|h\|_{\mathcal{H}}\leq n^{-\zeta}\\}$.
Then $\sup_{\theta\in\Theta,h\in\mathcal{H}_{n},1\leq i\leq
n}|\Psi(x_{i},z_{i};\theta)(h)|\overset{p}{\rightarrow}0$ and w.p.a.1,
$\mathcal{H}_{n}\subseteq\widehat{\mathcal{H}}(\theta)$ for all
$\theta\in\Theta$.

###### Proof.

Using the Cauchy-Schwarz inequality together with Lemma B.6 we have

$\displaystyle\sup_{\theta\in\Theta,h\in\mathcal{H}_{n},1\leq i\leq
n}|\Psi(x_{i},z_{i};\theta)(h)|$
$\displaystyle\leq\sup_{\theta\in\Theta,h\in\mathcal{H}_{n},1\leq i\leq
n}\left(\|h\|_{\mathcal{H}}\cdot\|\Psi(x_{i},z_{i};\theta)\|_{\mathcal{H}^{\ast}}\right)$
$\displaystyle\leq n^{-\zeta}\sup_{\theta\in\Theta,1\leq i\leq
n}\|\Psi(x_{i},z_{i};\theta)\|_{\mathcal{H}^{\ast}}$
$\displaystyle=O_{p}(n^{-\zeta+1/\nu})\overset{p}{\rightarrow}0.$

As $V=\operatorname{dom}(\phi)$ is an open interval containing zero it follows
that $\Psi(x_{i},z_{i};\theta)(h)\in\operatorname{dom}(\phi)$ w.p.a.1 for all
$\theta\in\Theta$ and $h\in\mathcal{H}_{n}$. ∎

######  Lemma B.8.

Let the assumptions of Theorem 3.6 be satisfied and assume
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$ with
$E[\|\psi(X;\bar{\theta})-\psi(X;\theta_{0})\|_{\infty}]=O_{p}(n^{-\rho})$
with $0<\rho<1/2$. Define the operators

$\displaystyle\Omega(\theta)$
$\displaystyle=E[\Psi(X,Z;{\theta})\otimes\Psi(X,Z;{\theta})]$
$\displaystyle\widehat{\Omega}(\theta)$
$\displaystyle=E_{\hat{P}_{n}}[\Psi(X,Z;{\theta})\otimes\Psi(X,Z;{\theta})].$

Then we have
$\|\widehat{\Omega}(\bar{\theta})-\Omega(\theta_{0})\|=O_{p}(n^{-\rho})$.

###### Proof.

The proof follows the proof of Lemma 20 of Bennett and Kallus (2023). Using
the triangle inequality we have,

$\displaystyle\|\widehat{\Omega}(\bar{\theta})-\Omega_{0}\|\leq\|\widehat{\Omega}(\bar{\theta})-\Omega(\bar{\theta})\|+\|\Omega(\bar{\theta})-\Omega(\theta_{0})\|.$

For the first term we have

$\displaystyle\|\widehat{\Omega}(\bar{\theta})-\Omega(\bar{\theta})\|$
$\displaystyle=\sup_{h,h^{\prime}\in\mathcal{H}_{1}}E_{\hat{P}_{n}}[h(Z)^{T}\psi(X;\bar{\theta})\psi(X;\bar{\theta})^{T}h^{\prime}(Z)]-E[h(Z)^{T}\psi(X;\bar{\theta})\psi(X;\bar{\theta})^{T}h^{\prime}(Z)]$
$\displaystyle=\sup_{g\in\mathcal{G}^{2}}E_{\hat{P}_{n}}[g(X,Z)]-E[g(X,Z)],$

where

$\displaystyle\mathcal{G}$
$\displaystyle=\\{g:g(x,z)=h(z)^{T}\psi(x;\bar{\theta}),\
h\in\mathcal{H}_{1}\\},$ $\displaystyle\mathcal{G}^{2}$
$\displaystyle=\\{g:g(x,z)=g_{1}(x,z)g_{2}(x,z),\
g_{1},g_{2}\in\mathcal{G}\\}.$

Now by Assumption h), $\mathcal{G}$ is $P_{0}$-Donsker and uniformly bounded
by continuity of $\Psi$ and compactness of its domain. Therefore
$\mathcal{G}^{2}$ is $P_{0}$-Donsker by Lemma B.4 which lets us employ Lemma
B.5 to conclude that
$\|\widehat{\Omega}(\bar{\theta})-\Omega(\bar{\theta})\|=O_{p}(n^{-1/2})$.

For the second term we have

$\displaystyle\|\Omega(\bar{\theta})-\Omega(\theta_{0})\|=$
$\displaystyle\sup_{h,h^{\prime}\in\mathcal{H}}E[h(Z)^{T}\left(\psi(X;\bar{\theta})\psi(X;\bar{\theta})^{T}-\psi(X;\theta_{0})\psi(X;\theta_{0})^{T}\right)h^{\prime}(Z)]$
$\displaystyle\leq$
$\displaystyle\sum_{i,j=1}^{m}\sup_{h,h^{\prime}\in\mathcal{H}}E[h_{i}(Z)h^{\prime}_{j}(Z)\psi_{i}(X;\bar{\theta})\left(\psi_{j}(X;\bar{\theta})-\psi_{j}(X;\theta_{0})\right)]$
$\displaystyle+\sum_{i,j=1}^{m}\sup_{h,h^{\prime}\in\mathcal{H}}E[h_{i}(Z)h^{\prime}_{j}(Z)\psi_{j}(X;\theta_{0})\left(\psi_{i}(X;\bar{\theta})-\psi_{i}(X;\theta_{0})\right)]$
$\displaystyle\leq$ $\displaystyle
2m^{2}C_{h}^{2}C_{\psi}E[\|\psi(X;\bar{\theta})-\psi(X;\theta_{0})\|_{\infty}]$
$\displaystyle\leq$ $\displaystyle O_{p}(n^{-\rho}).$

Putting things together we get
$\|\widehat{\Omega}(\bar{\theta})-\Omega_{0}\|=O_{p}(n^{-\rho})$. ∎

######  Lemma B.9.

Let the assumptions of Theorem 3.6 be satisfied and assume
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$ with
$E[\|\psi(X;\bar{\theta})-\psi(X;\theta_{0})\|_{\infty}]=O_{p}(n^{-\rho})$.
Then for $\lambda_{n}=O_{p}(n^{-\xi})$ with $0<\xi<\rho$ and any
$\dot{h}\in\operatorname{conv}(\\{0,\bar{h}\\})$ with
$\bar{h}\in\mathcal{H}_{n}$, as defined in Lemma B.7, the operator

$\displaystyle\widehat{\Omega}_{\lambda_{n}}(\dot{h},\bar{\theta})=-\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\bar{\Psi}_{i}(\dot{h}))(\bar{\Psi}_{i}\otimes\bar{\Psi}_{i})+\lambda_{n}I\otimes
I$

is non-singular w.p.a.1 with largest eigenvalue $C<\infty$.

###### Proof.

Define
$\widehat{\Omega}(\bar{\theta})=-\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\Psi(x_{i},z_{i};\bar{\theta})(\dot{h}))\left(\Psi(x_{i},z_{i};\bar{\theta})\otimes\Psi(x_{i},z_{i};\bar{\theta})\right)$.
As $\dot{h}$ lies in between $0$ and $\bar{h}$ and
$\bar{h}\in\mathcal{H}_{n}$, we have $\dot{h}\in\mathcal{H}_{n}$ and thus by
Lemma B.7 we have $\sup_{\theta\in\Theta}\max_{1\leq i\leq
n}|\phi_{2}(\Psi_{i}(\theta)(\dot{h}))+1|\overset{p}{\rightarrow}0$ and we
have
$\widehat{\Omega}(\bar{\theta})=\frac{1}{n}\sum_{i=1}^{n}\Psi(x_{i},z_{i};\bar{\theta})\otimes\Psi(x_{i},z_{i};\bar{\theta})$
w.p.a.1. By Lemma B.8, $\widehat{\Omega}(\bar{\theta})$ converges to the non-
singular operator $\Omega_{0}$ at rate $O_{p}(n^{-\rho})$ and as
$\lambda_{n}=O_{p}(n^{-\xi})$ with $0<\xi<\rho$, we have that
$\widehat{\Omega}_{\lambda_{n}}(\dot{h},\bar{\theta})$ is non-singular
w.p.a.1.

To bound the largest eigenvalue of
$\widehat{\Omega}_{\lambda_{n}}:=\widehat{\Omega}_{\lambda_{n}}(\dot{h},\bar{\theta})$
consider any $h\in\mathcal{H}$ and

$\displaystyle\langle
h,\widehat{\Omega}_{\lambda_{n}}h\rangle_{\mathcal{H}}=\frac{1}{n}\sum_{i=1}^{n}\|\Psi(x_{i},z_{i};\bar{\theta})(h)\|^{2}\leq\frac{1}{n}\sum_{i=1}^{n}\|\Psi(x_{i},z_{i};\bar{\theta})\|_{\mathcal{H}^{\ast}}^{2}\|h\|_{\mathcal{H}}^{2}=C_{\psi}^{2}\|h\|_{\mathcal{H}}^{2},$

where we used that by assumption for any $\theta\in\Theta$ and
$h\in\mathcal{H}_{1}$, $(x,z)\mapsto\psi(x;\theta)^{T}h(z)$ is a continuous
function on a compact domain and therefore bounded. Thus the largest
eigenvalue of $\widehat{\Omega}_{\lambda_{n}}$ must be bounded by some
constant $C<\infty$. ∎

The following lemma generalizes Lemma A2 of Newey and Smith (2004) to our
regularized continuum formulation. It is different from a similar proof of
Chaussé (2012) (Lemma 2), as our regularization procedure differs from theirs.

######  Lemma B.10.

Let the assumptions of Theorem 3.6 be satisfied. Additionally assume there is
a consistent parameter estimate $\bar{\theta}\in\Theta$,
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$, with
$\|E_{\hat{P}_{n}}[\Psi(X,Z;\bar{\theta})]\|_{\mathcal{H}^{\ast}}=O_{p}\left(n^{-1/2}\right)$
as well as
$E[\|\psi(X;\bar{\theta})-\psi(X;\theta_{0})\|_{\infty}]=O_{p}(n^{-1/2})$.
Further let $\lambda_{n}=O_{p}(n^{-\xi})$ where $0<\xi<1/2-1/\nu$. Then
$\bar{h}=$
$\operatorname*{arg\,max}_{h\in\widehat{\mathcal{H}}(\bar{\theta})}\widehat{G}_{\lambda_{n}}(\bar{\theta},h)$
exists w.p.a.1, $\|\bar{h}\|_{\mathcal{H}}=O_{p}\left(n^{-1/2}\right)$, and
$\widehat{G}_{\lambda_{n}}(\bar{\theta},\bar{h})\leq\phi(0)+O_{p}\left(n^{-1}\right)$.

###### Proof.

Let $\bar{\Psi}_{i}:=\Psi_{i}(\bar{\theta}):=\Psi(x_{i},z_{i};\bar{\theta})$
and $\bar{\Psi}=\frac{1}{n}\sum_{i=1}^{n}\bar{\Psi}_{i}$. By Lemma B.7 and
twice continuous differentiability of $\phi(v)$ in a neighborhood of zero,
$\widehat{G}_{\lambda_{n}}(\bar{\theta},h)$ is twice continuously
differentiable on $\mathcal{H}_{n}=\\{h:\|h\|_{\mathcal{H}}\leq n^{-\zeta}\\}$
w.p.a.1. Then
$\tilde{h}=\operatorname*{arg\,max}_{h\in\mathcal{H}_{n}}\widehat{G}_{\lambda_{n}}(\bar{\theta},h)$
exists w.p.a.1. Using Taylor’s theorem (Proposition B.2) we can expand the
regularized GEL objective about $h=0$ and obtain

$\displaystyle\phi_{0}$
$\displaystyle=\widehat{G}_{\lambda_{n}}(\bar{\theta},0)$
$\displaystyle\leq\widehat{G}_{\lambda_{n}}(\bar{\theta},\tilde{h})$
$\displaystyle=\phi_{0}-\bar{\Psi}(\tilde{h})-\frac{1}{2}\underbrace{\left[-\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\bar{\Psi}_{i}(\dot{h}))(\bar{\Psi}_{i}\otimes\bar{\Psi}_{i})+\lambda_{n}I\otimes
I\right]}_{=:\widehat{\Omega}_{\lambda_{n}}(\dot{h},\bar{\theta})}(\tilde{h},\tilde{h})$

for some $\dot{h}$ on the line between $0$ and $\tilde{h}$. Now, by Lemma B.9
the regularized covariance operator
$\widehat{\Omega}_{\lambda_{n}}(\dot{h},\bar{\theta})$ is positive definite
with smallest eigenvalue $C>0$ bounded away from zero w.p.a.1. Using this and
subtracting $\phi_{0}$ on both sides yields

$\displaystyle
0=\bar{\Psi}(\tilde{h})-\frac{1}{2}\langle\tilde{h},\widehat{\Omega}_{\lambda_{n}}(\bar{\theta})\tilde{h}\rangle_{\mathcal{H}}\leq\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}\|\tilde{h}\|_{\mathcal{H}}-\frac{1}{2}C\|\tilde{h}\|_{\mathcal{H}}^{2},$

where in the second line we used the Cauchy-Schwarz inequality for the first
term. This means we have
$\frac{1}{2}C\|\tilde{h}\|_{\mathcal{H}}\leq\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}$
w.p.a.1. As by assumption $\|\bar{\Psi}\|=O_{p}(n^{-1/2})$ it follows that
$\|\tilde{h}\|=O_{p}(n^{-1/2})$. Now, as $n^{-1/2}\leq n^{-\zeta}$ we have
$\tilde{h}\in\operatorname{int}(\mathcal{H}_{n})$ w.p.a.1. Then, as
$\tilde{h}$ is a maximizer contained in the interior of the domain
$\mathcal{H}_{n}$, it must correspond to a stationary point of
$\widehat{G}_{\lambda_{n}}$, i.e.,
$\left(\partial\widehat{G}_{\lambda_{n}}/\partial
h\right)(\bar{\theta},\tilde{h})=0$. However, from Lemma B.7 it follows that
w.p.a.1 $\tilde{h}\in\widehat{\mathcal{H}}(\bar{\theta})$ and as
$\widehat{G}_{\lambda_{n}}(\bar{\theta},h)$ is concave and
$\widehat{\mathcal{H}}(\bar{\theta})$ is convex we must have
$\widehat{G}_{\lambda_{n}}(\bar{\theta},\tilde{h})=\sup_{h\in\widehat{\mathcal{H}}(\bar{\theta})}\widehat{G}_{\lambda_{n}}(\bar{\theta},h)$,
which directly implies $\bar{h}=\tilde{h}$ and proves the first conclusion.
The second conclusion follows directly as
$\bar{h}\in\operatorname{int}(\mathcal{H}_{n})$ and thus
$\bar{h}=O_{p}(n^{-\zeta})$. Finally as
$\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$ by assumption, we have
$\widehat{G}_{\lambda_{n}}(\bar{\theta},\bar{h})\leq\phi_{0}+\|\bar{\Psi}\|_{\mathcal{H}^{\ast}}\|\bar{h}\|_{\mathcal{H}}-\frac{1}{2}C\|\bar{h}\|_{\mathcal{H}}^{2}=\phi_{0}+O_{p}\left(n^{-1}\right)$,
which completes the proof. ∎

The following lemma uses Lemmas B.7 and B.10 to show that the empirical moment
functional $E_{\hat{P}_{n}}[\Psi(X,Z;\hat{\theta})]$ evaluated at the FGEL
estimator $\hat{\theta}$ converges to zero in the dual norm. The proof closely
follows the proof of Lemma A3 of Newey and Smith (2004).

######  Lemma B.11.

Let the assumptions of Theorem 3.6 be satisfied and denote $\hat{\theta}$ the
corresponding FGEL estimator
$\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}\sup_{h\in\widehat{\mathcal{H}}(\theta)}\widehat{G}_{\lambda_{n}}({\theta},h)$.
Then
$\|E_{\hat{P}_{n}}[\Psi(X,Z;\hat{\theta})]\|_{\mathcal{H}^{\ast}}=O_{p}\left(n^{-1/2}\right)$.

###### Proof.

Define
$\hat{\Psi}_{i}:=\Psi_{i}(\hat{\theta}):=\Psi(x_{i},z_{i};\hat{\theta})$ and
$\hat{\Psi}=\frac{1}{n}\sum_{i=1}^{n}\hat{\Psi}_{i}$. Let $\mu(\hat{\Psi})$ be
the Riesz representer of $\hat{\Psi}\in\mathcal{H}^{\ast}$ in $\mathcal{H}$.
Further, let $1/\nu<\zeta<1/2$ be defined as in Lemma B.7 and consider
$\tilde{h}=-n^{-\zeta}\mu(\hat{\Psi})/\|\mu(\hat{\Psi})\|_{\mathcal{H}}$,
which implies $\tilde{h}\in\mathcal{H}_{n}$ and therefore by Lemma B.7
$\max_{1\leq i\leq n}\|\hat{\Psi}_{i}(\tilde{h})\|\overset{p}{\rightarrow}0$
and $\tilde{h}\in\widehat{\mathcal{H}}(\hat{\theta})$ w.p.a.1. Using the same
steps as in the proof of Lemma B.10 we can Taylor expand the empirical FGEL
objective about $h=0$,

$\displaystyle\widehat{G}_{\lambda_{n}}(\hat{\theta},\tilde{h})=\phi(0)-\hat{\Psi}(\tilde{h})-\frac{1}{2}\langle\tilde{h},\widehat{\Omega}_{\lambda_{n}}(\dot{h},\hat{\theta})\tilde{h}\rangle_{\mathcal{H}},$

for some $\dot{h}$ on the line between 0 and $\tilde{h}$. Now by Lemma B.9 the
largest eigenvalue of the regularized covariance operator
$\widehat{\Omega}_{\lambda_{n}}(\dot{h},\hat{\theta})$ can be bounded by a
constant $\tilde{C}>0$ w.p.a.1. Therefore, we have w.p.a.1,

$\displaystyle\widehat{G}_{\lambda_{n}}(\hat{\theta},\tilde{h})\geq\phi(0)+n^{-\zeta}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}-Cn^{-2\zeta},$

where for the second term we have used the definition of $\tilde{h}$ and the
notation $C=\frac{1}{2}\tilde{C}$. Consider $\bar{\theta}=\theta_{0}$ in Lemma
B.10, for which the requirements are fulfilled as
$\|E_{\hat{P}_{n}}[\Psi(X,Z;{\theta_{0}})]\|_{\mathcal{H}^{\ast}}=O_{p}\left(n^{-1/2}\right)$
by Lemma B.5 using Assumption h). Moreover, being the solution to the mini-max
problem, $(\hat{\theta},\hat{h})$ correspond to a saddle point of the
_empirical_ FGEL objective $\widehat{G}_{\lambda_{n}}$. Using this and Lemma
B.10 we have

$\displaystyle\phi(0)+n^{-\zeta}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}-Cn^{-2\zeta}\leq\widehat{G}_{\lambda_{n}}(\hat{\theta},\tilde{h})\leq\widehat{G}_{\lambda_{n}}(\hat{\theta},\hat{h})\leq\sup_{h\in\widehat{\mathcal{H}}(\theta_{0})}\widehat{G}_{\lambda_{n}}(\theta_{0},h)\leq\phi(0)+O_{p}(n^{-1}).$

Now, subtracting $\phi(0)$ on both sides and solving for
$\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}$, we obtain

$\displaystyle\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}\leq
O_{p}(n^{\zeta-1})+Cn^{-\zeta}=O_{p}(n^{-\zeta}),$ (20)

which follows as $1/\nu<\zeta<1/2$ and therefore $\zeta-1<-1/2<-\zeta$.
Consider any $\epsilon_{n}\rightarrow 0$ and let
$\tilde{h}=-\epsilon_{n}\mu(\hat{\Psi})$. Then by (20),
$\tilde{h}=o_{p}(n^{-\zeta})$ and therefore $\tilde{h}\in\mathcal{H}_{n}$
w.p.a.1. Then as previously we have

$\displaystyle\phi(0)-\hat{\Psi}(\tilde{h})-C\|h\|_{\mathcal{H}}^{2}=\
\phi(0)+\epsilon_{n}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}-C\epsilon_{n}^{2}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}\leq\
\phi(0)+O_{p}(n^{-1}).$

As $1-\epsilon_{n}C$ is bounded away from zero, for all $n$ large enough, we
have $\epsilon_{n}\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}^{2}=O_{p}(n^{-1})$. As
this holds for all $\epsilon_{n}\rightarrow 0$, it follows that
$\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$. ∎

##### Proof of Theorem 3.6

###### Proof.

Define $\hat{\Psi}_{i}=\Psi(x_{i},z_{i};\hat{\theta})$ and
$\hat{\Psi}=\frac{1}{n}\sum_{i=1}^{n}\hat{\Psi}_{i}$. By Assumption h) and
Lemma B.5, we have
$\|\hat{\Psi}(\theta)-E[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$
for any $\theta\in\Theta$. From Lemma B.11 we also have
$\|\hat{\Psi}\|_{\mathcal{H}^{\ast}}=O_{p}(n^{-1/2})$ and thus using the
triangle inequality we get

$\displaystyle\left\|E[\Psi(X,Z;\hat{\theta})]\right\|_{\mathcal{H}^{\ast}}$
$\displaystyle=\left\|E[\Psi(\hat{\theta})]-\hat{\Psi}+\hat{\Psi}\right\|_{\mathcal{H}^{\ast}}$
$\displaystyle\leq\left\|E[\Psi(X,Z;\hat{\theta})]-\hat{\Psi}\right\|_{\mathcal{H}^{\ast}}+\left\|\hat{\Psi}\right\|_{\mathcal{H}^{\ast}}$
$\displaystyle=O_{p}(n^{-1/2})\overset{p}{\rightarrow}0.$

As by Assumption a) $\theta_{0}$ is the unique parameter for which
$\|E[\Psi(X,Z;\theta)]\|_{\mathcal{H}^{\ast}}=0$ it follows that
$\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$.

Following the proof of Theorem A.1 of Kremer et al. (2023) we can use this
result to translate the convergence rate of the moment functional to a
convergence rate of the FGEL estimator $\hat{\theta}$ using Assumptions i)-k).
The proof is identical to the one provided by Kremer et al. (2023) and we
state it here merely for completeness.

By the mean value theorem, there exists
$\bar{\theta}\in\operatorname{conv}(\\{\theta_{0},\hat{\theta}\\})$ such that

$\displaystyle\Psi(X,Z;\hat{\theta})=\Psi(X,Z;\theta_{0})+(\hat{\theta}-\theta_{0})^{T}\nabla_{\theta}\Psi(X,Z;\bar{\theta}).$

Using this we have

$\displaystyle\|E[\Psi(X,Z;\hat{\theta})]\|^{2}_{\mathcal{H}^{\ast}}$
$\displaystyle=\|\underbrace{E[\Psi(X,Z;\theta_{0})]}_{=0}+(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})]\|^{2}_{\mathcal{H}^{\ast}}$
$\displaystyle=\left\langle(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})],(\hat{\theta}-\theta_{0})^{T}E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})]\right\rangle_{\mathcal{H}^{\ast}}$
$\displaystyle=(\hat{\theta}-\theta_{0})^{T}\underbrace{\left\langle
E[\nabla_{\theta}\Psi(X,Z;\bar{\theta})],E[\nabla_{\theta^{T}}\Psi(X,Z;\bar{\theta})]\right\rangle_{\mathcal{H}^{\ast}}}_{=:\Sigma(\bar{\theta})}(\hat{\theta}-\theta_{0})$
$\displaystyle\geq\lambda_{\text{min}}\left(\Sigma(\bar{\theta})\right)\|\hat{\theta}-\theta_{0}\|^{2}_{2}$

Now as $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$ and
$\bar{\theta}\in\operatorname{conv}(\\{\theta_{0},\hat{\theta}\\})$ we have
$\bar{\theta}\overset{p}{\rightarrow}\theta_{0}$ and thus
$\Sigma(\bar{\theta})\overset{p}{\rightarrow}\Sigma(\theta_{0})=:\Sigma_{0}$
by the continuous mapping theorem. By the non-negativity of the norm
$\Sigma_{0}$ is positive-semi definite and non-singular by Assumption k), thus
the smallest eigenvalue of $\Sigma(\bar{\theta})$,
$\lambda_{\text{min}}(\Sigma(\bar{\theta}))$, is positive and bounded away
from zero w.p.a.1. Finally as $\|E[\Psi(X,Z;\hat{\theta})]\|=O_{p}(n^{-1/2})$
taking the square-root on both sides we have
$\|\hat{\theta}-\theta_{0}\|=O_{p}(n^{-1/2})$.

∎

####  B.2.2 Proof of Theorem 3.7 (Asymptotic Normality)

######  Lemma B.12.

Let
$\Omega_{0}=E[\Psi(X,Z;\theta_{0})^{\ast}\Psi(X,Z;\theta_{0})]\in\mathcal{H}\times\mathcal{H}$
and $\Sigma_{0}=\left\langle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})],E[\nabla_{\theta^{T}}\Psi(X,Z;\theta_{0})]\right\rangle_{\mathcal{H}^{\ast}}\in\mathbb{R}^{p\times
p}$ be non-singular. Then the matrix

$\displaystyle M=-\begin{pmatrix}0&\nabla_{\theta}\Psi_{0}\\\
\nabla_{\theta^{T}}\Psi_{0}^{\ast}&\Omega_{0}\end{pmatrix}.$

is invertible and its inverse is given by

$\displaystyle
M^{-1}=\begin{pmatrix}\Xi&-\Xi\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\\\
-\Omega_{0}^{-1}\left(\nabla_{\theta^{T}}\Psi_{0}^{\ast}\right)\Xi&\Omega_{0}^{-1}+\Omega_{0}^{-1}\left(\nabla_{\theta^{T}}\Psi_{0}^{\ast}\right)\Xi\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\end{pmatrix}$

with
$\Xi=\left(\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\left(\nabla_{\theta^{T}}\Psi_{0}^{\ast}\right)\right)^{-1}\in\mathbb{R}^{p\times
p}$.

###### Proof.

In order to find the inverse of $M$ we resort to standard blockmatrix algebra.
Note that the inverse of a matrix

$\displaystyle P=\begin{pmatrix}A&B\\\ C&D\end{pmatrix}$

can be derived as

$\displaystyle
P^{-1}=\begin{pmatrix}(A-BD^{-1}C)^{-1}&-(A-BD^{-1}C)^{-1}BD^{-1}\\\
-D^{-1}C(A-BD^{-1}C)^{-1}&D^{-1}+D^{-1}C(A-BD^{-1}C)^{-1}BD^{-1}\end{pmatrix}$

whenever $D$ and its Schur complement in $P$, $P/D=A-BD^{-1}C$ are invertible
(see e.g., Bernstein (2009)).

The Schur complement of $\Omega_{0}$ in $M$ is a matrix
$\Gamma\in\mathbb{R}^{p\times p}$ defined as

$\displaystyle\Gamma:=M/\Omega_{0}=-\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\left(\nabla_{\theta^{T}}\Psi_{0}^{\ast}\right).$

Now, as $\Omega_{0}$ is a positive definite operator by assumption, its
smallest eigenvalue is bounded away from zero. It immediately follows that the
smallest eigenvalue of $\Omega^{-1}_{0}$,
$\lambda_{\mathrm{min}}(\Omega_{0}^{-1})>0$ is bounded away from zero and thus
we have for any $\theta\in\Theta$ with $\|\theta\|_{2}>0$,

$\displaystyle\theta^{T}\Gamma\theta$
$\displaystyle=-\theta^{T}\left(\nabla_{\theta}\Psi_{0}\right)\Omega_{0}^{-1}\left(\nabla_{\theta_{T}}\Psi_{0}^{\ast}\right)\theta$
$\displaystyle\leq-\lambda_{\mathrm{min}}(\Omega_{0}^{-1})\theta^{T}\underbrace{\left\langle
E[\nabla_{\theta}\Psi(X,Z;\theta_{0})],E[\nabla_{\theta^{T}}\Psi(X,Z;\theta_{0})]\right\rangle_{\mathcal{H}^{\ast}}}_{=\Sigma_{0}}\theta$
$\displaystyle\leq-\lambda_{\mathrm{min}}(\Omega_{0}^{-1})\lambda_{\mathrm{min}}(\Sigma_{0})\|\theta\|_{2}^{2}$
$\displaystyle<0,$

where we used that $\Sigma_{0}$ is symmetric by construction and non-singular
by assumption and therefore positive definite so its smallest eigenvalue
$\lambda_{\mathrm{min}}(\Sigma_{0})>0$. Now as $\Gamma\in\mathbb{R}^{p\times
p}$ is a strictly negative definite matrix it is non-singular and thus
invertible. Finally, as $\Omega_{0}$ and its Schur complement in $M$,
$\Gamma$, are invertible, we can employ the standard blockmatrix inversion
formula to arrive at the result. ∎

##### Proof of Theorem 3.7

The proof generalizes Theorem 3.2 of Newey and Smith (2004) to our regularized
continuum estimator.

###### Proof.

Define $\Psi_{i}(\theta):=\Psi_{i}(x_{i},z_{i};\theta)$ and
$\hat{\Psi}=\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}(\theta)$ and analogous
$\Psi^{\ast}_{i}(\theta)=\Psi^{\ast}_{i}(x_{i},z_{i};\theta)$ and
$\Psi^{\ast}(\theta)=\frac{1}{n}\sum_{i=1}^{n}\Psi^{\ast}_{i}(\theta)$. Let
$\hat{\theta},\hat{h}$ denote the FGEL estimates of the parameters $\theta$
and Lagrange multiplier function $h$. The first order optimality conditions of
the saddle point objective (12) are given by

$\displaystyle D_{h}\widehat{G}_{\lambda_{n}}(\hat{\theta},\hat{h})$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\phi_{1}(\Psi_{i}(\hat{\theta})(\hat{h}))\Psi_{i}^{\ast}(\hat{\theta})-\lambda_{n}\hat{h}=0$
(21)
$\displaystyle\nabla_{\theta}\widehat{G}_{\lambda_{n}}(\hat{\theta},\hat{h})$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\phi_{1}(\Psi_{i}(\hat{\theta})(\hat{h}))(\nabla_{\theta}\Psi_{i}(\hat{\theta}))(\hat{h})=0,$
(22)

where $\nabla_{\theta}\Psi_{i}(\theta)\in\Theta\times\mathcal{H}^{\ast}$ is
the gradient of the function $\theta\mapsto\Psi_{i}(\theta)$ w.r.t. $\theta$.
Define $\beta=(\theta,h)$ then using Taylor’s theorem (Proposition B.2) we can
linearize the first order conditions about the true parameters
$\beta_{0}=(\theta_{0},0)$ which yields for the first condition (21)

$\displaystyle 0=$
$\displaystyle-\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}^{\ast}(\theta_{0})+\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\Psi_{i}(\dot{\theta})(\dot{h}))\Psi_{i}^{\ast}(\dot{\theta})\Psi_{i}(\dot{\theta})(\hat{h})$
$\displaystyle-\lambda_{n}\hat{h}+\frac{1}{n}\sum_{i=1}^{n}\phi_{1}(\Psi_{i}(\dot{\theta})(\dot{h}))\nabla_{\theta^{T}}\Psi_{i}^{\ast}(\dot{\theta})(\hat{\theta}-\theta_{0})$
$\displaystyle+\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\Psi_{i}(\dot{\theta})(\dot{h}))\Psi_{i}^{\ast}(\dot{\theta})(\nabla_{\theta^{T}}\Psi_{i}(\dot{\theta}))(\dot{h})(\hat{\theta}-\theta_{0}),$

where $(\dot{\theta},\dot{h})$ lies on the line between
$(\hat{\theta},\hat{h})$ and $(\theta_{0},0)$. For the second condition (22)
we obtain

$\displaystyle 0=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\phi_{1}(\Psi_{i}(\bar{\theta})(\bar{h}))(\nabla_{\theta}\Psi_{i}(\bar{\theta}))(\hat{h})$
$\displaystyle+\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\Psi_{i}(\bar{\theta})(\bar{h}))(\nabla_{\theta}\Psi_{i}(\bar{\theta}))(\bar{h})\Psi_{i}(\bar{\theta})(\hat{h})$
$\displaystyle+\Bigg{\\{}\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\Psi_{i}(\bar{\theta})(\bar{h}))(\nabla_{\theta}\Psi_{i}(\bar{\theta}))(\bar{h})(\nabla_{\theta}\Psi_{i}(\bar{\theta}))(\bar{h})$
$\displaystyle+\frac{1}{n}\sum_{i=1}^{n}\phi_{1}(\Psi_{i}(\bar{\theta})(\bar{h}))D^{2}_{\theta}(\Psi_{i}(\bar{\theta}))(\bar{h})\Bigg{\\}}(\hat{\theta}-\theta_{0}),$

where again $(\bar{\theta},\bar{h})$ lies on the line between
$(\hat{\theta},\hat{h})$ and $(\theta_{0},0)$. Now as $\hat{h}=o_{p}(1)$ we
have $\bar{h}=o_{p}(1)$ and $\dot{h}=o_{p}(1)$. Therefore for
$n\rightarrow\infty$ most terms go to zero and we are left with the first
condition (21) reducing to

$\displaystyle 0=$
$\displaystyle-\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}^{\ast}(\theta_{0})+\frac{1}{n}\sum_{i=1}^{n}\phi_{2}(\Psi_{i}(\dot{\theta})(\dot{h}))\Psi_{i}^{\ast}(\dot{\theta})\Psi_{i}(\dot{\theta})(\hat{h})$
$\displaystyle-\lambda_{n}\hat{h}+\frac{1}{n}\sum_{i=1}^{n}\phi_{1}(\Psi_{i}(\dot{\theta})(\dot{h}))\nabla_{\theta^{T}}\Psi_{i}^{\ast}(\dot{\theta})(\hat{\theta}-\theta_{0})$
$\displaystyle+o_{p}(1)$

and the second (22) to

$\displaystyle 0=$
$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\phi_{1}(\Psi_{i}(\bar{\theta})(\bar{h}))(\nabla_{\theta}\Psi_{i}(\bar{\theta}))(\hat{h})+o_{p}(1).$

As $\hat{h}=O_{p}(n^{-1/2})$ and $\bar{h},\dot{h}$ lie between $\hat{h}$ and
$0$, the conditions of Lemma B.7 are fulfilled, i.e.,
$\bar{h},\dot{h}\in\mathcal{H}_{n}$, and hence $\max_{1\leq i\leq
n}|\Psi_{i}(\bar{\theta})(\bar{h})|\overset{p}{\rightarrow}0$ and $\max_{1\leq
i\leq
n}|\phi_{1}(\Psi_{i}(\bar{\theta})(\bar{h}))+1|\overset{p}{\rightarrow}0$ as
well as $\max_{1\leq i\leq
n}|\phi_{2}(\Psi_{i}(\bar{\theta})(\bar{h}))+1|\overset{p}{\rightarrow}0$ and
the same equivalently holds for $\dot{\theta}$ and $\dot{h}$.

Define $\hat{\beta}=(\hat{\theta},\hat{h})$ and $\beta_{0}=(\theta_{0},0)$,
then, we can write the conditions in matrix form as

$\displaystyle\begin{pmatrix}0\\\ 0\end{pmatrix}=\begin{pmatrix}0\\\
-\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}^{\ast}(\theta_{0})\end{pmatrix}+\underbrace{\begin{pmatrix}0&-\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}\Psi_{i}(\dot{\theta})\\\
-\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta^{T}}\Psi_{i}^{\ast}(\bar{\theta})&-\widehat{\Omega}_{\lambda_{n}}(\dot{h},\dot{\theta})\end{pmatrix}}_{=:M_{n}}(\hat{\beta}-\beta_{0})+o_{p}(1)$
(23)

Now by the weak law of large numbers and the continuous mapping theorem, we
have
$\widehat{\Omega}_{\lambda_{n}}(\dot{h},\dot{\theta})\overset{p}{\rightarrow}\Omega(\theta_{0})=:\Omega_{0}$,
which is a non-singular and thus invertible operator. Further for the off-
diagonal elements, as $\hat{\theta}\overset{p}{\rightarrow}\theta_{0}$ and by
continuity of $\theta\mapsto\nabla_{\theta}\Psi(x,z;\theta)$ we have by the
continuous mapping theorem and the weak law of large numbers
$-\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}\Psi_{i}(\dot{\theta})\overset{p}{\rightarrow}-E[\nabla_{\theta}\Psi(X,Z;\theta_{0})]=:-\nabla_{\theta}\Psi_{0}$.
Therefore, as $n\rightarrow\infty$ we have $M_{n}\rightarrow M$ with

$\displaystyle M=-\begin{pmatrix}0&\nabla_{\theta}\Psi_{0}\\\
\nabla_{\theta^{T}}\Psi^{\ast}_{0}&\Omega_{0}\end{pmatrix}.$

Using Assumptions e) and k) of Theorem 3.6, it follows from Lemma B.12 that
the blockoperator $M$ is non-singular and thus invertible with inverse

$\displaystyle M^{-1}=-\begin{pmatrix}-B&C\\\ C^{\ast}&D\end{pmatrix}$

with
$B=((\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast}))^{-1}$,
$C=B(\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}$ and
$D=\Omega_{0}^{-1}-\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast})B(\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}$.

With this at hand we can solve (23) for $\hat{\beta}-\beta_{0}$ which yields

$\displaystyle\sqrt{n}(\hat{\theta}-\theta)$
$\displaystyle=-\sqrt{n}C\left(\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}^{\ast}(\theta_{0})\right)+o_{p}(1)$
$\displaystyle\sqrt{n}(\hat{h}-h)$
$\displaystyle=-\sqrt{n}D\left(\frac{1}{n}\sum_{i=1}^{n}\Psi_{i}^{\ast}(\theta_{0})\right)+o_{p}(1)$

Finally by the Donsker property of $\Psi$ we have
$\sqrt{n}\left(\frac{1}{n}\sum_{i=1}^{n}\Psi(X,Z;\theta_{0})\right)\sim\mathcal{N}(0,\Omega_{0})$
and thus we get

$\displaystyle\sqrt{n}(\hat{\theta}-\theta_{0})\sim\mathcal{N}(0,C\Omega_{0}C^{\ast})$

where

$\displaystyle C\Omega_{0}C^{\ast}$
$\displaystyle=((\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast}))^{-1}(\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}\Omega_{0}\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast})((\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast}))^{-1}$
$\displaystyle=(\nabla_{\theta}\Psi_{0})\Omega_{0}^{-1}(\nabla_{\theta^{T}}\Psi_{0}^{\ast}))^{-1}$

∎

####  B.2.3 Asymptotic Properties for CMR

The asymptotic properties of the FGEL estimator for conditional moment
restrictions follow by expressing the conditional moment restrictions (6) in
terms of the equivalent variational/functional formulation (7) and translating
the assumptions of Theorems 3.8 and 3.9 into the conditions required for
Theorems 3.6 and 3.7 respectively. The proofs are almost identical to the ones
provided by Kremer et al. (2023) for translating the results on their KMM
estimator for functional moment restrictions to the conditional moment
restriction case. Therefore, we only state the differences here and refer to
the proofs of Theorems 3.4-3.6 of Kremer et al. (2023) for details.

##### Proof of Theorem 3.8

The proof follows directly from the proof of Theorem 3.4 of Kremer et al.
(2023), ignoring their assumption f) which is not required here and instead
using that the additional Assumption g) of the FGEL estimator is fulfilled by
the identical Assumption g) in Theorem 3.6. Further the Donsker property h) of
Theorem 3.6 is fulfilled by the corresponding Assumption h) and Lemma B.4
using that both $\psi$ and $h$ are uniformly bounded as continuous functions
on compact domains.

##### Proof of Theorem 3.9

The proof is identical to the one provided by Kremer et al. (2023) for their
Theorem 3.5.

##### Proof of Theorem 3.10

Efficiency follows immediately from Theorem 3.9 as the asymptotic variance of
the FGEL estimator agrees with the semi-parametric efficiency bound for CMR
estimators by Chamberlain (1987).

###  B.3 Kernel FGEL

##### Proof of Theorem 3.11

###### Proof.

The proof follows the proof of Theorem 3.2 in Muandet et al. (2020). Equation
(15) follows from (14) directly by the law of iterated expectation. To see
this, assume $E[\psi(X;\theta)|Z]=0\ P_{Z}\textrm{-a.s.}$, then $\forall
h\in\mathcal{H}$

$\displaystyle E[\Psi(X,Z;\theta)(h)]$ $\displaystyle=E[\psi(X;\theta)h(Z)]$
$\displaystyle=E[E[\psi(X;\theta)h(Z)|Z]]$
$\displaystyle=E[E[\psi(X;\theta)|Z]h(Z)]$ $\displaystyle=0.$

For the other direction, note that $E[\Psi(X,Z;\theta)(h)]=0$ $\forall
h\in\mathcal{H}$ implies $\sup_{h\in\mathcal{H}}E[\Psi(X,Z;\theta)(h)]=0$ and
thus

$\displaystyle 0$ $\displaystyle=\sup_{h\in\mathcal{H}}E[\Psi(X,Z;\theta)(h)]$
$\displaystyle=\sum_{j=1}^{m}\sup_{\|h_{j}\|_{\mathcal{H}}\leq
1}E[\psi_{j}(X;\theta)h_{j}(Z)]$
$\displaystyle=\sum_{j=1}^{m}\sup_{\|h_{j}\|_{\mathcal{H}}\leq 1}\langle
E[\psi_{j}(X;\theta)k_{j}(Z,\cdot)],h_{j}\rangle$
$\displaystyle=\sum_{j=1}^{m}\|E[\psi_{j}(X;\theta)k_{j}(Z,\cdot)]\|_{\mathcal{H}}$
$\displaystyle=\sum_{j=1}^{m}\|E_{Z}[\underbrace{E_{X}[\psi_{j}(X;\theta)|Z]}_{:=\xi_{j}(Z)}k_{j}(Z,\cdot)]\|_{\mathcal{H}}$
$\displaystyle=\sum_{j=1}^{m}\|\int_{\mathcal{Z}}\xi_{j}(z)k_{j}(z,\cdot)p(z)dz\|_{\mathcal{H}}$

As each element of the sum is non-negative, we must have for $j=1,\ldots,m$,

$\displaystyle 0$
$\displaystyle=\|\int_{\mathcal{Z}}\xi_{j}(z)k_{j}(z,\cdot)p(z)\mathrm{d}z\|_{\mathcal{H}}$
$\displaystyle=\|\int_{\mathcal{Z}}\xi_{j}(z)k_{j}(z,\cdot)p(z)\mathrm{d}z\|_{\mathcal{H}}^{2}$
$\displaystyle=\int_{\mathcal{Z}\times\mathcal{Z}}\xi_{j}(z)\langle
k_{j}(z,\cdot),k_{j}(z^{\prime},\cdot)\rangle_{\mathcal{H}}\xi_{j}(z^{\prime})p(z)p(z^{\prime})\mathrm{d}z\mathrm{d}z^{\prime}$
$\displaystyle=\int_{\mathcal{Z}\times\mathcal{Z}}\xi_{j}(z)k_{j}(z,z^{\prime})\xi_{j}(z^{\prime})p(z)p(z^{\prime})\mathrm{d}z\mathrm{d}z^{\prime}.$

By definition of ISPD kernels (see Section 3) this directly implies
$\|\xi_{j}(z)p(z)\|_{2}^{2}=0$. It follows that $\xi_{j}(z)=0\ \
\mathrm{a.e.}$ on the support of $p(z)$ and thus
$P_{Z}(\\{z\in\mathcal{Z}:\xi_{j}(z)=0\\})=1$. Finally this implies

$\xi_{j}(Z)=E[\psi_{j}(X;\theta)|Z]=0\ \ P_{Z}\textrm{-a.s.},\ \
j=1,\ldots,m,$

which completes the equivalence between (14) and (15). ∎

##### Proof of Corollary 3.12

###### Proof.

Equivalence between (6) and (7) holds for any RKHS corresponding to a
universal ISPD kernels by Theorem 3.11. Moreover, the local Lipschitz property
is fulfilled as in any RKHS the evaluation functional is bounded, which
implies that for any $h\in\mathcal{H}$

$\displaystyle\|h(z_{1})-h(z_{2})\|$ $\displaystyle=\|\langle
h,k(z_{1},\cdot)\rangle-\langle h,k(z_{2},\cdot)\rangle\|$
$\displaystyle\leq\|h\|\|k(z_{1},\cdot)-k(z_{2},\cdot)\|$ $\displaystyle\leq
C\left(\|k(z_{1},\cdot)\|_{\mathcal{H}}+\|k(z_{2},\cdot)\|_{\mathcal{H}}\right)$
$\displaystyle\leq L$

where we used the Cauchy-Schwarz and triangle inequalities. Finally the
Donsker property of $\mathcal{H}_{1}$ follows from Lemma 17 of Bennett and
Kallus (2023). ∎

##### Proof of Lemma 3.13

###### Proof.

The profile divergence can be written as

$\displaystyle
R_{\lambda_{n}}(\theta)=\inf_{h\in\widehat{\mathcal{H}}}-\sum_{i=1}^{n}\phi(\Psi(x_{i},z_{i};\theta)(h)+\frac{\lambda_{n}}{2}\|h\|_{\mathcal{H}}.$

As $-\phi$ is a convex function and $\widehat{\mathcal{H}}$ is convex it
follows that this is a convex optimization problem. Therefore, we can employ
the representer theorem Schölkopf et al. (2001) and express each component $r$
of the $m$-dimensional vector of RKHS functions as
$h_{r}(\cdot)=\sum_{i=1}^{n}(\alpha_{r})_{i}k_{r}(z_{i},\cdot)$, with
$\alpha_{r}\in\mathbb{R}^{n}$. Therefore, we get

$\displaystyle\Psi(x_{i},z_{i};\theta)(h)=\sum_{r=1}^{m}\sum_{i,j=1}^{n}(\alpha_{r})_{j}\left(K_{r}\right)_{ji}\psi_{r}(x_{i},z_{i};\theta)$
$\displaystyle\|h\|_{\mathcal{H}}^{2}=\sum_{r=1}^{m}\sum_{i,j=1}^{n}(\alpha_{r})_{i}\langle
k_{r}(z_{i},\cdot),k_{r}(z_{j},\cdot)\rangle(\alpha_{r})_{j}=\sum_{r=1}^{m}\alpha_{r}^{T}K_{r}\alpha_{r}$

Inserting this back into $R_{\lambda_{n}}(\theta)$ yields the result. ∎

###  B.4 Additional Proofs

##### Proof of Proposition 3.14

###### Proof.

The result follows directly by inserting
$\phi(v)=\left(1\pm\frac{v}{2}\right)^{2}$ into (12) and using that as
$\mathcal{H}$ is a vector space, for every $h\in\mathcal{H}$, its negative
$-h$ is also contained in $\mathcal{H}$. Therefore the first order conditions
agree for the positive and negative sign in $\phi$. ∎